Repository: ComplianceAsCode/content
Pull request #13029: OPENSCAP-4118 - Add script to build tests
Author: login: Mab879
Labels: name: Infrastructure
description: Our content build system
Created: 2025-02-12T00:00:33Z | Merged: 2025-03-12T07:36:35Z

#### Description:

This is a very rough draft a script that renders the Automatus tests for a given product. The script will be clean up once the general form is agreed upon. 

#### Rationale:

Make running the Automatus tests possible with out Automatus script.

#### Review Hints:
1. `export ADDITIONAL_CMAKE_OPTION="-DSSG_BUILT_TESTS_ENABLED:BOOL=ON"`
1. ` ./build_product  rhel10`



Diff:
diff --git a/CMakeLists.txt b/CMakeLists.txt
index 2c717290f307..8763166e76c4 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -68,6 +68,7 @@ option(SSG_JINJA2_CACHE_ENABLED "If enabled, the jinja2 templating files will be
 option(SSG_SCE_ENABLED "If enabled, additional SCE audit content will be enabled alongside OVAL-based auditing." FALSE)
 option(SSG_SRG_XLSX_EXPORT "If enabled, an XLSX of SRG Export will be ceated." FALSE)
 option(SSG_SEPARATE_SCAP_FILES_ENABLED "If enabled, separate SCAP files (OVAL, XCCDF, CPE dict, ...) will be installed alongside the source data-streams" TRUE)
+option(SSG_BUILT_TESTS_ENABLED "If enabled, Automatus tests will be built" FALSE)
 # Validation Options
 option(SSG_BATS_TESTS_ENABLED "If enabled, bats will be used to run unit-tests of bash remediations." TRUE)
 option(SSG_LINKCHECKER_VALIDATION_ENABLED "If enabled, linkchecker will be used to validate URLs in all the HTML guides and tables." TRUE)
@@ -290,8 +291,9 @@ else()
     message(STATUS "jinja2 cache: disabled")
 endif()
 message(STATUS "Separate SCAP files: ${SSG_SEPARATE_SCAP_FILES_ENABLED}")
-message(STATUS "STIG Delta Taloring files: ${SSG_BUILD_DISA_DELTA_FILES}")
+message(STATUS "STIG Delta Tailoring files: ${SSG_BUILD_DISA_DELTA_FILES}")
 message(STATUS "Thin data streams: ${SSG_THIN_DS}")
+message(STATUS "Pre-build Automatus tests: ${SSG_BUILT_TESTS_ENABLED}")
 
 message(STATUS " ")
 message(STATUS "Validation options:")
diff --git a/build-scripts/build_tests.py b/build-scripts/build_tests.py
new file mode 100755
index 000000000000..27bc73de635a
--- /dev/null
+++ b/build-scripts/build_tests.py
@@ -0,0 +1,245 @@
+#!/usr/bin/env python3
+
+import argparse
+import logging
+import os
+import pathlib
+import sys
+from typing import TypeVar, Generator, Set, Dict
+import multiprocessing
+
+import ssg.constants
+import ssg.environment
+import ssg.jinja
+import ssg.utils
+import ssg.yaml
+import ssg.templates
+
+SSG_ROOT = str(pathlib.Path(__file__).resolve().parent.parent.absolute())
+JOB_COUNT = multiprocessing.cpu_count()
+T = TypeVar("T")
+
+
+def _create_arg_parser() -> argparse.ArgumentParser:
+    parser = argparse.ArgumentParser(
+        description="Converts built content tests to be rendered.")
+    parser.add_argument("--build-config-yaml", required=True, type=str,
+                        help="YAML file with information about the build configuration. "
+                             "e.g.: ~/scap-security-guide/build/build_config.yml")
+    parser.add_argument("--product-yaml", required=True, type=str,
+                        help="YAML file with information about the product we are building. "
+                             "e.g.: ~/scap-security-guide/rhel10/product.yml")
+    parser.add_argument("--output", required=True, type=str,
+                        help="Output path"
+                             "e.g.:  ~/scap-security-guide/build/rhel10/tests")
+    parser.add_argument("--resolved-rules-dir", required=True, type=str,
+                        help="Directory with <rule-id>.yml resolved rule YAMLs "
+                             "e.g.: ~/scap-security-guide/build/rhel10/rules")
+    parser.add_argument("--log-level", action="store", type=str, default="ERROR",
+                        choices=["ERROR", "WARNING", "INFO", "DEBUG", "TRACE"],
+                        help="What level to log at. Defaults to ERROR.")
+    parser.add_argument("--root", default=SSG_ROOT,
+                        help=f"Path to the project. Defaults to {SSG_ROOT}")
+    parser.add_argument("--jobs", "-j", type=int, default=JOB_COUNT,
+                        help=f"Number of cores to use. Defaults to {JOB_COUNT} on this system.")
+    return parser
+
+
+def _write_path(file_contents: str, output_path: os.PathLike) -> None:
+    with open(output_path, "w") as file:
+        file.write(file_contents)
+        file.write("\n")
+
+
+def _is_test_file(filename: str) -> bool:
+    return filename.endswith(('.pass.sh', '.fail.sh', '.notapplicable.sh'))
+
+
+def _get_deny_templated_scenarios(test_config_path: pathlib.Path) -> Set[str]:
+    if test_config_path.exists():
+        test_config = ssg.yaml.open_raw(str(test_config_path.absolute()))
+        deny_templated_scenarios = test_config.get('deny_templated_scenarios', set())
+        return deny_templated_scenarios
+    return set()
+
+
+def _process_shared_file(env_yaml: dict, file: pathlib.Path, shared_output_path: pathlib.Path) \
+        -> None:
+    file_contents = ssg.jinja.process_file_with_macros(str(file.absolute()), env_yaml)
+    shared_script_path = shared_output_path / file.name
+    _write_path(file_contents, shared_script_path)
+
+
+def _copy_and_process_shared(env_yaml: dict, output_path: pathlib.Path, root_path: pathlib.Path) \
+        -> None:
+    tests_shared_root = root_path / "tests" / "shared"
+    shared_output_path = output_path / "shared"
+    shared_output_path.mkdir(parents=True, exist_ok=True)
+    for file in tests_shared_root.iterdir():  # type: pathlib.Path
+        # We only support one level deep, this avoids recursive functions
+        if file.is_dir():
+            for sub_file in file.iterdir():
+                shared_output_path_sub = shared_output_path / file.name
+                shared_output_path_sub.mkdir(parents=True, exist_ok=True)
+                _process_shared_file(env_yaml, sub_file, shared_output_path_sub)
+        else:
+            _process_shared_file(env_yaml, file, shared_output_path)
+
+
+def _get_platform_from_file_contents(file_contents: str) -> str:
+    # Some tests don't have an explict platform assume always applicable
+    platform = "multi_platform_all"
+    for line in file_contents.split("\n"):
+        if line.startswith('# platform'):
+            platform_parts = line.split('=')
+            if len(platform_parts) == 2:
+                platform = platform_parts[1]
+                break
+    return platform.strip()
+
+
+def _process_local_tests(product: str, env_yaml: dict, rule_output_path: pathlib.Path,
+                         rule_tests_root: pathlib.Path) -> None:
+    logger = logging.getLogger()
+    for test in rule_tests_root.iterdir():  # type: pathlib.Path
+        if test.is_dir():
+            logger.warning("Skipping directory %s in rule %s", test.name,
+                           rule_output_path.name)
+            continue
+        if not _is_test_file(test.name):
+            file_contents = ssg.jinja.process_file_with_macros(str(test.absolute()),
+                                                               env_yaml)
+            output_file = rule_output_path / test.name
+            rule_output_path.mkdir(parents=True, exist_ok=True)
+            _write_path(file_contents, output_file)
+        file_contents = test.read_text()
+        platform = _get_platform_from_file_contents(file_contents)
+        if ssg.utils.is_applicable_for_product(platform, product):
+            content = ssg.jinja.process_file_with_macros(str(test.absolute()), env_yaml)
+            rule_output_path.mkdir(parents=True, exist_ok=True)
+            _write_path(content, rule_output_path / test.name)
+
+
+def _should_skip_templated_tests(deny_templated_scenarios, test):
+    return not test.name.endswith(".sh") or test.name in deny_templated_scenarios
+
+
+def _process_templated_tests(env_yaml: Dict, rendered_rule_obj: Dict, templates_root: pathlib.Path,
+                             rule_output_path: pathlib.Path):
+    logger = logging.getLogger()
+    rule_path = pathlib.Path(rendered_rule_obj['definition_location'])
+    product = rule_output_path.parent.parent.name
+    rule_id = rule_path.parent.name
+    if "name" not in rendered_rule_obj["template"]:
+        raise ValueError(f"Invalid template config on rule {rule_id}")
+    template_name = rendered_rule_obj["template"]["name"]
+    template_root = templates_root / template_name
+    template_tests_root = template_root / "tests"
+
+    rule_root = rule_path.parent
+    rule_tests_root = rule_root / "tests"
+
+    if not template_tests_root.exists():
+        logger.debug("Template %s doesn't have tests. Skipping for rule %s.",
+                     template_name, rule_id)
+        return
+    test_config_path = rule_tests_root / "test_config.yml"
+    deny_templated_scenarios = _get_deny_templated_scenarios(test_config_path)
+    for test in template_tests_root.iterdir():  # type: pathlib.Path
+        if _should_skip_templated_tests(deny_templated_scenarios, test):
+            logger.warning("Skipping %s for %s as it is a denied test scenario",
+                           test.name, rule_id)
+            continue
+        template = ssg.templates.Template.load_template(str(templates_root.absolute()),
+                                                        template_name)
+        rendered_rule_obj["template"]["vars"]["_rule_id"] = rule_id
+        template_parameters = template.preprocess(rendered_rule_obj["template"]["vars"], "test")
+        env_yaml = env_yaml.copy()
+        jinja_dict = ssg.utils.merge_dicts(env_yaml, template_parameters)
+        file_contents = ssg.jinja.process_file_with_macros(str(test.absolute()), jinja_dict)
+        platform = _get_platform_from_file_contents(file_contents)
+        if ssg.utils.is_applicable_for_product(platform, product):
+            rule_output_path.mkdir(parents=True, exist_ok=True)
+            test_output_path = rule_output_path / test.name
+            _write_path(file_contents, test_output_path)
+            logger.debug("Wrote scenario %s for rule %s", test.name, rule_id)
+        else:
+            logger.warning("Skipping scenario %s for rule %s as it not applicable to %s",
+                           test.name, rule_id, product)
+
+
+def _process_rules(env_yaml: Dict, output_path: pathlib.Path,
+                   templates_root: pathlib.Path, product_rules: list,
+                   resolved_root: pathlib.Path) -> None:
+    product = resolved_root.parent.name
+    for rule_id in product_rules:
+        rule_file = resolved_root / f'{rule_id}.yml'
+
+        rendered_rule_obj = ssg.yaml.open_raw(str(rule_file))
+        rule_path = pathlib.Path(rendered_rule_obj["definition_location"])
+        rule_root = rule_path.parent
+
+        rule_tests_root = rule_root / "tests"
+        rule_output_path = output_path / rule_id
+
+        if rule_tests_root.exists():
+            _process_local_tests(product, env_yaml, rule_output_path, rule_tests_root)
+        if rendered_rule_obj["template"] is not None:
+            _process_templated_tests(env_yaml, rendered_rule_obj, templates_root, rule_output_path)
+
+
+def _get_benchmark_cpes(env_yaml) -> Set[str]:
+    benchmark_cpes = set()
+    for cpe in env_yaml["cpes"]:
+        for _, data in cpe.items():
+            benchmark_cpes.add(data["name"])
+    return benchmark_cpes
+
+
+def _get_rules_in_profile(built_profiles_root) -> Generator[str, None, None]:
+    for profile_file in built_profiles_root.iterdir():  # type: pathlib.Path
+        if not profile_file.name.endswith(".profile"):
+            continue
+        profile_data = ssg.yaml.open_raw(str(profile_file.absolute()))
+        for selection in profile_data["selections"]:
+            if "=" not in selection:
+                yield selection
+
+
+def main() -> int:
+    args = _create_arg_parser().parse_args()
+    logging.basicConfig(level=logging.getLevelName(args.log_level))
+    env_yaml = ssg.environment.open_environment(args.build_config_yaml, args.product_yaml)
+    root_path = pathlib.Path(args.root).resolve()
+    output_path = pathlib.Path(args.output).resolve()
+    resolved_rules_dir = pathlib.Path(args.resolved_rules_dir)
+    if not resolved_rules_dir.exists() or not resolved_rules_dir.is_dir():
+        logging.error("Unable to find product at %s", str(resolved_rules_dir))
+        logging.error("Is the product built?")
+        return 1
+
+    output_path.mkdir(parents=True, exist_ok=True)
+    _copy_and_process_shared(env_yaml, output_path, root_path)
+
+    built_profiles_root = resolved_rules_dir.parent / "profiles"
+    rules_in_profiles = list(set(_get_rules_in_profile(built_profiles_root)))
+
+    templates_root = root_path / "shared" / "templates"
+    processes = list()
+    for chunk in range(args.jobs):
+        process_args = (env_yaml, output_path, templates_root,
+                        rules_in_profiles[chunk::args.jobs], resolved_rules_dir)
+        process = multiprocessing.Process(target=_process_rules, args=process_args)
+        processes.append(process)
+        process.start()
+    for process in processes:
+        process.join()
+    # Write a file for CMake
+    # So we don't have a dependency on a folder
+    done_file = output_path / ".test_done"
+    done_file.touch()
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())
diff --git a/cmake/SSGCommon.cmake b/cmake/SSGCommon.cmake
index 0aa78e31700d..211ca7c394b9 100644
--- a/cmake/SSGCommon.cmake
+++ b/cmake/SSGCommon.cmake
@@ -671,6 +671,22 @@ macro(ssg_build_disa_delta PRODUCT PROFILE)
             DESTINATION ${SSG_TAILORING_INSTALL_DIR})
 endmacro()
 
+macro(ssg_build_tests PRODUCT)
+    add_custom_command(
+        OUTPUT "${CMAKE_BINARY_DIR}/${PRODUCT}/tests/.tests_done"
+        COMMAND env "PYTHONPATH=$ENV{PYTHONPATH}" "${PYTHON_EXECUTABLE}" "${CMAKE_SOURCE_DIR}/build-scripts/build_tests.py"  --build-config-yaml "${CMAKE_BINARY_DIR}/build_config.yml" --resolved-rules-dir "${CMAKE_CURRENT_BINARY_DIR}/rules" --output  "${CMAKE_CURRENT_BINARY_DIR}/tests" --product-yaml "${CMAKE_SOURCE_DIR}/products/${PRODUCT}/product.yml"
+        # Actually we mean that it depends on resolved rules - see also ssg_build_templated_content
+        DEPENDS ${PRODUCT}-content
+        COMMENT "[${PRODUCT}-tests] generating tests"
+    )
+
+    add_custom_target(
+        ${PRODUCT}-tests
+        DEPENDS "${CMAKE_BINARY_DIR}/${PRODUCT}/tests/.tests_done"
+    )
+endmacro()
+
+
 # Top-level macro to build all output artifacts for the specified product.
 # Ensures the various targets we create in the above macros are linked to
 # the default build target when applicable and handles installation steps.
@@ -885,6 +901,11 @@ macro(ssg_build_product PRODUCT)
         set_tests_properties("scapval-${PRODUCT}" PROPERTIES LABELS scapval)
     endif()
 
+    if(SSG_BUILT_TESTS_ENABLED)
+        ssg_build_tests(${PRODUCT})
+        add_dependencies(${PRODUCT} ${PRODUCT}-tests)
+    endif()
+
     # grab all the kickstarts (if any) and install them
     file(GLOB KICKSTART_FILES "${CMAKE_CURRENT_SOURCE_DIR}/kickstart/ssg-${PRODUCT}-*-ks.cfg")
     install(FILES ${KICKSTART_FILES}
diff --git a/docs/manual/developer/02_building_complianceascode.md b/docs/manual/developer/02_building_complianceascode.md
index 3ff5e1cf7b32..d4152675d7e4 100644
--- a/docs/manual/developer/02_building_complianceascode.md
+++ b/docs/manual/developer/02_building_complianceascode.md
@@ -521,6 +521,12 @@ bash/rhel9-script-hipaa.sh
 ...
 ```
 
+### Automatus Tests
+
+You can build the Automatus tests so they can be run directly.
+To enable build with `-DSSG_BUILT_TESTS_ENABLED:BOOL=ON`
+The tests will be `$PRODUCT/tests/` with directory for each rule.
+
 ## Testing
 
 To ensure validity of built artifacts prior to installation, we recommend
diff --git a/tests/automatus.py b/tests/automatus.py
index e0749d5d29bf..ece63a1cb2b7 100755
--- a/tests/automatus.py
+++ b/tests/automatus.py
@@ -16,7 +16,7 @@
 ssg_dir = os.path.join(os.path.dirname(__file__), "..")
 sys.path.append(ssg_dir)
 
-from ssg_test_suite.log import LogHelper
+from tests.ssg_test_suite.log import LogHelper
 import ssg_test_suite.oscap
 import ssg_test_suite.test_env
 import ssg_test_suite.profile
@@ -74,6 +74,7 @@ def parse_args():
             help="DEPRECATED: Use --remove-platforms instead; "
             "Find all CPEs that are present in local OpenSCAP's CPE dictionary "
             "that match the provided regex, "
+
             "and add them as platforms to all data stream benchmarks. "
             "If the regex doesn't match anything, it will be treated "
             "as a literal CPE, and added as a platform. "
diff --git a/tests/ssg_test_suite/common.py b/tests/ssg_test_suite/common.py
index 44b2fd519f7e..6e4aac57d6ca 100644
--- a/tests/ssg_test_suite/common.py
+++ b/tests/ssg_test_suite/common.py
@@ -21,7 +21,7 @@
 from ssg.products import product_yaml_path, load_product_yaml
 from ssg.rules import get_rule_dir_yaml
 from ssg.utils import mkdir_p
-from ssg_test_suite.log import LogHelper
+from tests.ssg_test_suite.log import LogHelper
 
 import ssg.templates
 
@@ -386,6 +386,7 @@ def send_scripts(test_env, test_content_by_rule_id):
     archive_file_basename = os.path.basename(archive_file)
     remote_archive_file = os.path.join(remote_dir, archive_file_basename)
     logging.debug("Uploading scripts.")
+    logging.debug(f"LogHelper.LOG_DIR {LogHelper.LOG_DIR}")
     log_file_name = os.path.join(LogHelper.LOG_DIR, "env-preparation.log")
 
     with open(log_file_name, 'a') as log_file:
diff --git a/tests/ssg_test_suite/oscap.py b/tests/ssg_test_suite/oscap.py
index b884e2c23f9b..c6ab3876a7f8 100644
--- a/tests/ssg_test_suite/oscap.py
+++ b/tests/ssg_test_suite/oscap.py
@@ -15,9 +15,9 @@
 
 from ssg.constants import OSCAP_PROFILE_ALL_ID
 
-from ssg_test_suite.log import LogHelper
-from ssg_test_suite import test_env
-from ssg_test_suite import common
+from tests.ssg_test_suite.log import LogHelper
+from tests.ssg_test_suite import test_env
+from tests.ssg_test_suite import common
 
 from ssg.shims import input_func
 
diff --git a/tests/ssg_test_suite/rule.py b/tests/ssg_test_suite/rule.py
index 7826b3888914..f2e25e77109d 100644
--- a/tests/ssg_test_suite/rule.py
+++ b/tests/ssg_test_suite/rule.py
@@ -17,11 +17,11 @@
 from ssg.jinja import process_file_with_macros
 from ssg.rules import is_rule_dir
 
-from ssg_test_suite import oscap
-from ssg_test_suite import xml_operations
-from ssg_test_suite import test_env
-from ssg_test_suite import common
-from ssg_test_suite.log import LogHelper
+from tests.ssg_test_suite import oscap
+from tests.ssg_test_suite import xml_operations
+from tests.ssg_test_suite import test_env
+from tests.ssg_test_suite import common
+from tests.ssg_test_suite.log import LogHelper
 
 
 Rule = collections.namedtuple(
diff --git a/tests/ssg_test_suite/test_env.py b/tests/ssg_test_suite/test_env.py
index 26aefb94a416..ae24092c125a 100644
--- a/tests/ssg_test_suite/test_env.py
+++ b/tests/ssg_test_suite/test_env.py
@@ -9,9 +9,8 @@
 import sys
 import time
 
-import ssg_test_suite
-from ssg_test_suite import common
-from ssg_test_suite.log import LogHelper
+from tests.ssg_test_suite import common
+from tests.ssg_test_suite.log import LogHelper
 
 
 class SavedState(object):
diff --git a/tests/unit/ssg_test_suite/test_rule.py b/tests/unit/ssg_test_suite/test_rule.py
index 8e97fd8310b3..afcf8a3b5e5d 100644
--- a/tests/unit/ssg_test_suite/test_rule.py
+++ b/tests/unit/ssg_test_suite/test_rule.py
@@ -1,7 +1,5 @@
 import os
-import pytest
 
-from ssg_test_suite import rule
 from tests.ssg_test_suite.rule import Scenario
 from ssg.constants import OSCAP_PROFILE_ALL_ID
 


Reviews:
user: login: comps
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-02-14T21:17:29Z
user: login: comps
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-02-14T21:28:23Z
user: login: comps
body: I left some limited comments on the python code (as I am not familiar with the build system overall). Feel free to ignore any of them, as I don't know what is the level of code quality you want to maintain in this project.

Aside from the comments, consider (for an extra pedantic exercise):
* `typing` consistency - most of the code seems untyped, but some functions have types despite being `_` (a.k.a. internal, not part of API). I'm also not sure if `done_file: pathlib.Path = output_path / ".test_done"` does anything (with the inlined type)
* Overall `"` vs `'` consistency, they seem to be intermixed randomly
* Lack of comments (especially in big functions like `_process_rules()`
* Inconsistent indent (`ssg.jinja.process_file_with_macros` in `_process_shared_file()` seems to squash two arguments at the end, `ssg.environment.open_environment` in `main()` uses a different multi-line function call syntax.
* f-strings for `logger.*` calls - note that you don't have to use the prehistoric `%s` syntax, if you pass only one argument, `logging` takes it as verbatim, so it can be an f-string like one of those you use for exceptions.

I'll test the functionality tomorrow, looking forward to it. :slightly_smiling_face: 
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-06T21:58:49Z
user: login: evgenyz
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-06T22:15:23Z
user: login: Mab879
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-06T22:28:28Z
user: login: Mab879
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-06T22:28:45Z
user: login: comps
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-06T22:39:57Z
user: login: jan-cerny
state: CHANGES_REQUESTED
author_association: COLLABORATOR
submitted_at: 2025-03-07T09:58:36Z
user: login: Mab879
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-07T13:30:33Z
user: login: jan-cerny
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-10T13:38:03Z
user: login: jan-cerny
body: Looks good to me now. It works for me as I expect.

@comps please check it if it's good for you
state: APPROVED
author_association: COLLABORATOR
submitted_at: 2025-03-11T08:59:18Z

Review comments:
diff_hunk: @@ -0,0 +1,218 @@
+#!/usr/bin/python3
+
+import argparse
+import logging
+import pathlib
+import sys
+from typing import TypeVar, Iterator, List, Iterable, Set, Dict
+import multiprocessing
+
+import ssg.environment
+import ssg.jinja
+import ssg.utils
+import ssg.yaml
+import ssg.templates
+import tests.ssg_test_suite.common
+import tests.ssg_test_suite.rule
+
+SSG_ROOT = str(pathlib.Path(__file__).resolve().parent.parent.absolute())
+JOB_COUNT = multiprocessing.cpu_count()
+T = TypeVar('T')
+
+
+def _create_arg_parser() -> argparse.ArgumentParser:
+    parser = argparse.ArgumentParser(
+        description="Converts builds content tests to be rendered"
+    )
+    parser.add_argument(
+        "--build-config-yaml", required=True,
+        help="YAML file with information about the build configuration. "
+             "e.g.: ~/scap-security-guide/build/build_config.yml"
+    )
+    parser.add_argument(
+        "--product-yaml", required=True,
+        help="YAML file with information about the product we are building. "
+             "e.g.: ~/scap-security-guide/rhel10/product.yml"
+    )
+    parser.add_argument(
+        "--output", required=True,
+        help="Output path"
+             "e.g.:  ~/scap-security-guide/build/rhel10/tests"
+    )
+    parser.add_argument(
+        "--resolved-rules-dir", required=True,
+        help="Directory with <rule-id>.yml resolved rule YAMLs"
+             "e.g.: ~/scap-security-guide/build/rhel10/rules"
+    )
+    parser.add_argument("--verbose", action="store_true", default=False)
+    parser.add_argument("--root", default=SSG_ROOT,
+                        help=f"Path to the project. Defaults to {SSG_ROOT}")
+    parser.add_argument("--jobs", "-j", type=int, default=JOB_COUNT,
+                        help=f"Number of cores to use. Defaults to {JOB_COUNT} on this system.")
+    return parser
+
+
+def _is_test_file(filename: str) -> bool:
+    return (filename.endswith('.pass.sh')
+            or filename.endswith('.fail.sh') or
+            filename.endswith('.notapplicable.sh'))
path: build-scripts/build_tests.py
user: login: comps
body: `str.endswith()` takes a tuple, so you can probably just
```suggestion
    return filename.endswith(('.pass.sh', '.fail.sh', '.notapplicable.sh'))
```
created_at: 2025-02-14T21:17:29Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,218 @@
+#!/usr/bin/python3
+
+import argparse
+import logging
+import pathlib
+import sys
+from typing import TypeVar, Iterator, List, Iterable, Set, Dict
+import multiprocessing
+
+import ssg.environment
+import ssg.jinja
+import ssg.utils
+import ssg.yaml
+import ssg.templates
+import tests.ssg_test_suite.common
+import tests.ssg_test_suite.rule
+
+SSG_ROOT = str(pathlib.Path(__file__).resolve().parent.parent.absolute())
+JOB_COUNT = multiprocessing.cpu_count()
+T = TypeVar('T')
+
+
+def _create_arg_parser() -> argparse.ArgumentParser:
+    parser = argparse.ArgumentParser(
+        description="Converts builds content tests to be rendered"
+    )
+    parser.add_argument(
+        "--build-config-yaml", required=True,
+        help="YAML file with information about the build configuration. "
+             "e.g.: ~/scap-security-guide/build/build_config.yml"
+    )
+    parser.add_argument(
+        "--product-yaml", required=True,
+        help="YAML file with information about the product we are building. "
+             "e.g.: ~/scap-security-guide/rhel10/product.yml"
+    )
+    parser.add_argument(
+        "--output", required=True,
+        help="Output path"
+             "e.g.:  ~/scap-security-guide/build/rhel10/tests"
+    )
+    parser.add_argument(
+        "--resolved-rules-dir", required=True,
+        help="Directory with <rule-id>.yml resolved rule YAMLs"
+             "e.g.: ~/scap-security-guide/build/rhel10/rules"
+    )
+    parser.add_argument("--verbose", action="store_true", default=False)
+    parser.add_argument("--root", default=SSG_ROOT,
+                        help=f"Path to the project. Defaults to {SSG_ROOT}")
+    parser.add_argument("--jobs", "-j", type=int, default=JOB_COUNT,
+                        help=f"Number of cores to use. Defaults to {JOB_COUNT} on this system.")
+    return parser
+
+
+def _is_test_file(filename: str) -> bool:
+    return (filename.endswith('.pass.sh')
+            or filename.endswith('.fail.sh') or
+            filename.endswith('.notapplicable.sh'))
+
+
+def _divide_iters(iterator: Iterable[T], n: int) -> List[Iterator[T]]:
+    items = list(iterator)
+    length = len(items)
+
+    chunk_size = length // n
+    remainder = length % n
+
+    inters = []
+    start = 0
+
+    for i in range(n):
+        end = start + chunk_size + (1 if i < remainder else 0)
+        inters.append(iter(items[start:end]))
+        start = end
+    return inters
path: build-scripts/build_tests.py
user: login: comps
body: If you don't mind interleaving the split, you could use python's array slicing "step" property.

Given 3 workers and a total of 10 rules, you can assign rules to them like this:
```python
>>> a = [1,2,3,4,5,6,7,8,9,10]
>>> a[0::3]
[1, 4, 7, 10]
>>> a[1::3]
[2, 5, 8]
>>> a[2::3]
[3, 6, 9]
```
where the first (`start`) number is worker index, and the second (`step`) number is the total number of workers.

This is what we use in Contest to avoid any off-by-one problems and ensure no rule is left behind, while making things just simpler and evenly distributing rule complexity across all workers (as complex/simple rules tend to alphabetically bunch up).

https://github.com/RHSecurityCompliance/contest/blob/1204f812956a855eb15d26b1014cb28e5306ee05/per-rule/test.py#L93-L97
created_at: 2025-02-14T21:28:23Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,237 @@
+#!/usr/bin/env python3
+
+import argparse
+import logging
+import pathlib
+from typing import TypeVar, List, Iterable, Set, Dict
+import multiprocessing
+
+import ssg.environment
+import ssg.jinja
+import ssg.utils
+import ssg.yaml
+import ssg.templates
+import tests.ssg_test_suite.common
+import tests.ssg_test_suite.rule
+
+SSG_ROOT = str(pathlib.Path(__file__).resolve().parent.parent.absolute())
+JOB_COUNT = multiprocessing.cpu_count()
+T = TypeVar('T')
+
+
+def _create_arg_parser() -> argparse.ArgumentParser:
+    parser = argparse.ArgumentParser(
+        description="Converts built content tests to be rendered."
+    )
+    parser.add_argument(
+        "--build-config-yaml", required=True,
+        help="YAML file with information about the build configuration. "
+             "e.g.: ~/scap-security-guide/build/build_config.yml"
+    )
+    parser.add_argument(
+        "--product-yaml", required=True,
+        help="YAML file with information about the product we are building. "
+             "e.g.: ~/scap-security-guide/rhel10/product.yml"
+    )
+    parser.add_argument(
+        "--output", required=True,
+        help="Output path"
+             "e.g.:  ~/scap-security-guide/build/rhel10/tests"
+    )
+    parser.add_argument(
+        "--resolved-rules-dir", required=True,
+        help="Directory with <rule-id>.yml resolved rule YAMLs"
+             "e.g.: ~/scap-security-guide/build/rhel10/rules"
+    )
+    parser.add_argument("--log-level", action="store", type=str, default="ERROR",
+                        choices=["ERROR", "WARNING", "INFO", "DEBUG", "TRACE"],
+                        help="What level to log at. Defaults to ERROR.")
+    parser.add_argument("--root", default=SSG_ROOT,
+                        help=f"Path to the project. Defaults to {SSG_ROOT}")
+    parser.add_argument("--jobs", "-j", type=int, default=JOB_COUNT,
+                        help=f"Number of cores to use. Defaults to {JOB_COUNT} on this system.")
+    return parser
+
+
+def _is_test_file(filename: str) -> bool:
+    return filename.endswith(('.pass.sh', '.fail.sh', '.notapplicable.sh'))
+
+
+def _get_deny_templated_scenarios(test_config_path):
+    deny_templated_scenarios = list()
+    if test_config_path.exists():
+        test_config = ssg.yaml.open_raw(str(test_config_path.absolute()))
+        if 'deny_templated_scenarios' in test_config:
+            deny_templated_scenarios = test_config['deny_templated_scenarios']
+    return deny_templated_scenarios
path: build-scripts/build_tests.py
user: login: comps
body: Maybe
```suggestion
def _get_deny_templated_scenarios(test_config_path):
    if test_config_path.exists():
        test_config = ssg.yaml.open_raw(str(test_config_path.absolute()))
        if 'deny_templated_scenarios' in test_config:
            return set(test_config['deny_templated_scenarios'])
    return set()
```
?
created_at: 2025-03-06T21:18:24Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,237 @@
+#!/usr/bin/env python3
+
+import argparse
+import logging
+import pathlib
+from typing import TypeVar, List, Iterable, Set, Dict
+import multiprocessing
+
+import ssg.environment
+import ssg.jinja
+import ssg.utils
+import ssg.yaml
+import ssg.templates
+import tests.ssg_test_suite.common
+import tests.ssg_test_suite.rule
+
+SSG_ROOT = str(pathlib.Path(__file__).resolve().parent.parent.absolute())
+JOB_COUNT = multiprocessing.cpu_count()
+T = TypeVar('T')
+
+
+def _create_arg_parser() -> argparse.ArgumentParser:
+    parser = argparse.ArgumentParser(
+        description="Converts built content tests to be rendered."
+    )
+    parser.add_argument(
+        "--build-config-yaml", required=True,
+        help="YAML file with information about the build configuration. "
+             "e.g.: ~/scap-security-guide/build/build_config.yml"
+    )
+    parser.add_argument(
+        "--product-yaml", required=True,
+        help="YAML file with information about the product we are building. "
+             "e.g.: ~/scap-security-guide/rhel10/product.yml"
+    )
+    parser.add_argument(
+        "--output", required=True,
+        help="Output path"
+             "e.g.:  ~/scap-security-guide/build/rhel10/tests"
+    )
+    parser.add_argument(
+        "--resolved-rules-dir", required=True,
+        help="Directory with <rule-id>.yml resolved rule YAMLs"
+             "e.g.: ~/scap-security-guide/build/rhel10/rules"
+    )
+    parser.add_argument("--log-level", action="store", type=str, default="ERROR",
+                        choices=["ERROR", "WARNING", "INFO", "DEBUG", "TRACE"],
+                        help="What level to log at. Defaults to ERROR.")
+    parser.add_argument("--root", default=SSG_ROOT,
+                        help=f"Path to the project. Defaults to {SSG_ROOT}")
+    parser.add_argument("--jobs", "-j", type=int, default=JOB_COUNT,
+                        help=f"Number of cores to use. Defaults to {JOB_COUNT} on this system.")
+    return parser
+
+
+def _is_test_file(filename: str) -> bool:
+    return filename.endswith(('.pass.sh', '.fail.sh', '.notapplicable.sh'))
+
+
+def _get_deny_templated_scenarios(test_config_path):
+    deny_templated_scenarios = list()
+    if test_config_path.exists():
+        test_config = ssg.yaml.open_raw(str(test_config_path.absolute()))
+        if 'deny_templated_scenarios' in test_config:
+            deny_templated_scenarios = test_config['deny_templated_scenarios']
+    return deny_templated_scenarios
+
+
+def _process_shared_file(env_yaml, file, shared_output_path):
+    file_contents = ssg.jinja.process_file_with_macros(str(file.absolute()),
+                                                       env_yaml)
+    shared_script_path = shared_output_path / file.name
+    with open(shared_script_path, 'w') as file:
+        file.write(file_contents)
+        file.write('\n')
path: build-scripts/build_tests.py
user: login: comps
body: ```suggestion
    _write_path(file_contents, shared_script_path)
```
?

(And in other similar cases.)
created_at: 2025-03-06T21:36:20Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,237 @@
+#!/usr/bin/env python3
+
+import argparse
+import logging
+import pathlib
+from typing import TypeVar, List, Iterable, Set, Dict
+import multiprocessing
+
+import ssg.environment
+import ssg.jinja
+import ssg.utils
+import ssg.yaml
+import ssg.templates
+import tests.ssg_test_suite.common
+import tests.ssg_test_suite.rule
+
+SSG_ROOT = str(pathlib.Path(__file__).resolve().parent.parent.absolute())
+JOB_COUNT = multiprocessing.cpu_count()
+T = TypeVar('T')
+
+
+def _create_arg_parser() -> argparse.ArgumentParser:
+    parser = argparse.ArgumentParser(
+        description="Converts built content tests to be rendered."
+    )
+    parser.add_argument(
+        "--build-config-yaml", required=True,
+        help="YAML file with information about the build configuration. "
+             "e.g.: ~/scap-security-guide/build/build_config.yml"
+    )
+    parser.add_argument(
+        "--product-yaml", required=True,
+        help="YAML file with information about the product we are building. "
+             "e.g.: ~/scap-security-guide/rhel10/product.yml"
+    )
+    parser.add_argument(
+        "--output", required=True,
+        help="Output path"
+             "e.g.:  ~/scap-security-guide/build/rhel10/tests"
+    )
+    parser.add_argument(
+        "--resolved-rules-dir", required=True,
+        help="Directory with <rule-id>.yml resolved rule YAMLs"
+             "e.g.: ~/scap-security-guide/build/rhel10/rules"
+    )
+    parser.add_argument("--log-level", action="store", type=str, default="ERROR",
+                        choices=["ERROR", "WARNING", "INFO", "DEBUG", "TRACE"],
+                        help="What level to log at. Defaults to ERROR.")
+    parser.add_argument("--root", default=SSG_ROOT,
+                        help=f"Path to the project. Defaults to {SSG_ROOT}")
+    parser.add_argument("--jobs", "-j", type=int, default=JOB_COUNT,
+                        help=f"Number of cores to use. Defaults to {JOB_COUNT} on this system.")
+    return parser
+
+
+def _is_test_file(filename: str) -> bool:
+    return filename.endswith(('.pass.sh', '.fail.sh', '.notapplicable.sh'))
+
+
+def _get_deny_templated_scenarios(test_config_path):
+    deny_templated_scenarios = list()
+    if test_config_path.exists():
+        test_config = ssg.yaml.open_raw(str(test_config_path.absolute()))
+        if 'deny_templated_scenarios' in test_config:
+            deny_templated_scenarios = test_config['deny_templated_scenarios']
+    return deny_templated_scenarios
+
+
+def _process_shared_file(env_yaml, file, shared_output_path):
+    file_contents = ssg.jinja.process_file_with_macros(str(file.absolute()),
+                                                       env_yaml)
+    shared_script_path = shared_output_path / file.name
+    with open(shared_script_path, 'w') as file:
+        file.write(file_contents)
+        file.write('\n')
+
+
+def _copy_and_process_shared(env_yaml, output_path, root_path):
+    tests_shared_root = root_path / "tests" / "shared"
+    shared_output_path = output_path / "shared"
+    shared_output_path.mkdir(parents=True, exist_ok=True)
+    for file in tests_shared_root.iterdir():  # type: pathlib.Path
+        # We only support one level deep, this avoids recursive functions
+        if file.is_dir():
+            for sub_file in file.iterdir():
+                shared_output_path_sub = shared_output_path / file.name
+                shared_output_path_sub.mkdir(parents=True, exist_ok=True)
+                _process_shared_file(env_yaml, sub_file, shared_output_path_sub)
+        else:
+            _process_shared_file(env_yaml, file, shared_output_path)
+
+
+def _process_local_tests(benchmark_cpes, env_yaml, rule_output_path, rule_tests_root):
+    logger = logging.getLogger()
+    for test in rule_tests_root.iterdir():  # type: pathlib.Path
+        if not test.name.endswith(".sh"):
+            logger.debug("Skipping file %s in rule %s is it doesn't end with .sh.",
+                         test.name, rule_output_path.name)
+            continue
+        if not _is_test_file(test.name):
+            file_contents = ssg.jinja.process_file_with_macros(str(test.absolute()),
+                                                               env_yaml)
+            output_file = rule_output_path / test.name
+            rule_output_path.mkdir(parents=True, exist_ok=True)
+            with open(output_file, 'w') as file:
+                file.write(file_contents)
+                file.write('\n')
+        file_contents = test.read_text()
+        scenario = tests.ssg_test_suite.rule.Scenario(test.name, file_contents)
+        if scenario.matches_platform(benchmark_cpes):
+            content = ssg.jinja.process_file_with_macros(str(test.absolute()), env_yaml)
+            rule_output_path.mkdir(parents=True, exist_ok=True)
+            with open(rule_output_path / test.name, 'w') as file:
+                file.write(content)
+                file.write('\n')
+
+
+def _write_path(file_contents, output_path):
+    with open(output_path, 'w') as file:
+        file.write(file_contents)
+        file.write('\n')
+
+
+def _process_rules(benchmark_cpes: Set[str], env_yaml: Dict, output_path: pathlib.Path,
+                   rules: Iterable[pathlib.Path], templates_root: pathlib.Path,
+                   product_rules: list) -> None:
+    logger = logging.getLogger()
+    for rule_file in rules:  # type: pathlib.Path
+        rendered_rule_obj = ssg.yaml.open_raw(str(rule_file))
+        rule_path = pathlib.Path(rendered_rule_obj["definition_location"])
+        rule_root = rule_path.parent
+        rule_id = rule_root.name
+        if rule_id not in product_rules:
+            logger.debug("Skipping %s since it is not in the product", rule_id)
+            continue
+        rule_tests_root = rule_root / "tests"
+        rule_output_path = output_path / rule_id
+
+        if rule_tests_root.exists():
+            _process_local_tests(benchmark_cpes, env_yaml, rule_output_path, rule_tests_root)
+        if rendered_rule_obj["template"] is not None:
+            if "name" not in rendered_rule_obj["template"]:
+                raise ValueError(f"Invalid template config on rule {rule_id}")
+            template_name = rendered_rule_obj["template"]["name"]
+            template_root = templates_root / template_name
+            template_tests_root = template_root / "tests"
+            if not template_tests_root.exists():
+                logger.debug("Template %s doesn't have tests. Skipping for rule %s.",
+                             template_name, rule_id)
+                continue
+            test_config_path = rule_tests_root / "test_config.yml"
+            deny_templated_scenarios = _get_deny_templated_scenarios(test_config_path)
+            for test in template_tests_root.iterdir():  # type: pathlib.Path
+                if not test.name.endswith(".sh") or test.name in deny_templated_scenarios:
+                    logging.warning("Skipping % for %s", test.name, rule_id)
+                    continue
+                template = ssg.templates.Template.load_template(str(templates_root.absolute()),
+                                                                template_name)
+                rendered_rule_obj["template"]["vars"]["_rule_id"] = rule_id
+                template_parameters = template.preprocess(
+                    rendered_rule_obj["template"]["vars"], "test")
+                env_yaml = env_yaml.copy()
+                jinja_dict = ssg.utils.merge_dicts(env_yaml, template_parameters)
+                file_contents = ssg.jinja.process_file_with_macros(str(test.absolute()),
+                                                                   jinja_dict)
+                scenario = tests.ssg_test_suite.rule.Scenario(test.name, file_contents)
+                if scenario.matches_platform(benchmark_cpes):
+                    rule_output_path.mkdir(parents=True, exist_ok=True)
+                    test_output_path = rule_output_path / test.name
+                    _write_path(file_contents, test_output_path)
+                    logger.debug('Wrote scenario %s for rule %s', test.name, rule_id)
+                else:
+                    logger.debug('Skipping scenario %s for rule %s has it not applicable',
+                                 test.name, rule_id)
+
+
+def _get_benchmark_cpes(env_yaml):
+    benchmark_cpes = set()
+    for cpe in env_yaml["cpes"]:
+        for _, data in cpe.items():
+            benchmark_cpes.add(data["name"])
+    return benchmark_cpes
+
+
+def _get_rules_in_profile(built_profiles_root):
+    rules_in_profile = list()
+    for profile_file in built_profiles_root.iterdir():  # type: pathlib.Path
+        if not profile_file.name.endswith('.profile'):
+            continue
+        profile_data = ssg.yaml.open_raw(str(profile_file.absolute()))
+        for selection in profile_data["selections"]:
+            if '=' not in selection:
+                rules_in_profile.append(selection)
+    return rules_in_profile
+
+
+def main() -> int:
+    args = _create_arg_parser().parse_args()
+    logging.basicConfig(level=logging.getLevelName(args.log_level))
+    env_yaml = ssg.environment.open_environment(
+        args.build_config_yaml, args.product_yaml)
+
+    root_path = pathlib.Path(args.root).resolve().absolute()
+    output_path = pathlib.Path(args.output).resolve().absolute()
path: build-scripts/build_tests.py
user: login: comps
body: `.resolve()` already makes the path absolute:
```
$ pydoc3 pathlib.Path.resolve | cat
Help on function resolve in pathlib.Path:

pathlib.Path.resolve = resolve(self, strict=False)
    Make the path absolute, resolving all symlinks on the way and also
    normalizing it.
```
created_at: 2025-03-06T21:42:43Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,237 @@
+#!/usr/bin/env python3
+
+import argparse
+import logging
+import pathlib
+from typing import TypeVar, List, Iterable, Set, Dict
+import multiprocessing
+
+import ssg.environment
+import ssg.jinja
+import ssg.utils
+import ssg.yaml
+import ssg.templates
+import tests.ssg_test_suite.common
+import tests.ssg_test_suite.rule
+
+SSG_ROOT = str(pathlib.Path(__file__).resolve().parent.parent.absolute())
+JOB_COUNT = multiprocessing.cpu_count()
+T = TypeVar('T')
+
+
+def _create_arg_parser() -> argparse.ArgumentParser:
+    parser = argparse.ArgumentParser(
+        description="Converts built content tests to be rendered."
+    )
+    parser.add_argument(
+        "--build-config-yaml", required=True,
+        help="YAML file with information about the build configuration. "
+             "e.g.: ~/scap-security-guide/build/build_config.yml"
+    )
+    parser.add_argument(
+        "--product-yaml", required=True,
+        help="YAML file with information about the product we are building. "
+             "e.g.: ~/scap-security-guide/rhel10/product.yml"
+    )
+    parser.add_argument(
+        "--output", required=True,
+        help="Output path"
+             "e.g.:  ~/scap-security-guide/build/rhel10/tests"
+    )
+    parser.add_argument(
+        "--resolved-rules-dir", required=True,
+        help="Directory with <rule-id>.yml resolved rule YAMLs"
+             "e.g.: ~/scap-security-guide/build/rhel10/rules"
+    )
+    parser.add_argument("--log-level", action="store", type=str, default="ERROR",
+                        choices=["ERROR", "WARNING", "INFO", "DEBUG", "TRACE"],
+                        help="What level to log at. Defaults to ERROR.")
+    parser.add_argument("--root", default=SSG_ROOT,
+                        help=f"Path to the project. Defaults to {SSG_ROOT}")
+    parser.add_argument("--jobs", "-j", type=int, default=JOB_COUNT,
+                        help=f"Number of cores to use. Defaults to {JOB_COUNT} on this system.")
+    return parser
+
+
+def _is_test_file(filename: str) -> bool:
+    return filename.endswith(('.pass.sh', '.fail.sh', '.notapplicable.sh'))
+
+
+def _get_deny_templated_scenarios(test_config_path):
+    deny_templated_scenarios = list()
+    if test_config_path.exists():
+        test_config = ssg.yaml.open_raw(str(test_config_path.absolute()))
+        if 'deny_templated_scenarios' in test_config:
+            deny_templated_scenarios = test_config['deny_templated_scenarios']
+    return deny_templated_scenarios
+
+
+def _process_shared_file(env_yaml, file, shared_output_path):
+    file_contents = ssg.jinja.process_file_with_macros(str(file.absolute()),
+                                                       env_yaml)
+    shared_script_path = shared_output_path / file.name
+    with open(shared_script_path, 'w') as file:
+        file.write(file_contents)
+        file.write('\n')
+
+
+def _copy_and_process_shared(env_yaml, output_path, root_path):
+    tests_shared_root = root_path / "tests" / "shared"
+    shared_output_path = output_path / "shared"
+    shared_output_path.mkdir(parents=True, exist_ok=True)
+    for file in tests_shared_root.iterdir():  # type: pathlib.Path
+        # We only support one level deep, this avoids recursive functions
+        if file.is_dir():
+            for sub_file in file.iterdir():
+                shared_output_path_sub = shared_output_path / file.name
+                shared_output_path_sub.mkdir(parents=True, exist_ok=True)
+                _process_shared_file(env_yaml, sub_file, shared_output_path_sub)
+        else:
+            _process_shared_file(env_yaml, file, shared_output_path)
+
+
+def _process_local_tests(benchmark_cpes, env_yaml, rule_output_path, rule_tests_root):
+    logger = logging.getLogger()
+    for test in rule_tests_root.iterdir():  # type: pathlib.Path
+        if not test.name.endswith(".sh"):
+            logger.debug("Skipping file %s in rule %s is it doesn't end with .sh.",
+                         test.name, rule_output_path.name)
+            continue
+        if not _is_test_file(test.name):
+            file_contents = ssg.jinja.process_file_with_macros(str(test.absolute()),
+                                                               env_yaml)
+            output_file = rule_output_path / test.name
+            rule_output_path.mkdir(parents=True, exist_ok=True)
+            with open(output_file, 'w') as file:
+                file.write(file_contents)
+                file.write('\n')
+        file_contents = test.read_text()
+        scenario = tests.ssg_test_suite.rule.Scenario(test.name, file_contents)
+        if scenario.matches_platform(benchmark_cpes):
+            content = ssg.jinja.process_file_with_macros(str(test.absolute()), env_yaml)
+            rule_output_path.mkdir(parents=True, exist_ok=True)
+            with open(rule_output_path / test.name, 'w') as file:
+                file.write(content)
+                file.write('\n')
+
+
+def _write_path(file_contents, output_path):
+    with open(output_path, 'w') as file:
+        file.write(file_contents)
+        file.write('\n')
+
+
+def _process_rules(benchmark_cpes: Set[str], env_yaml: Dict, output_path: pathlib.Path,
+                   rules: Iterable[pathlib.Path], templates_root: pathlib.Path,
+                   product_rules: list) -> None:
+    logger = logging.getLogger()
+    for rule_file in rules:  # type: pathlib.Path
+        rendered_rule_obj = ssg.yaml.open_raw(str(rule_file))
+        rule_path = pathlib.Path(rendered_rule_obj["definition_location"])
+        rule_root = rule_path.parent
+        rule_id = rule_root.name
+        if rule_id not in product_rules:
+            logger.debug("Skipping %s since it is not in the product", rule_id)
+            continue
+        rule_tests_root = rule_root / "tests"
+        rule_output_path = output_path / rule_id
+
+        if rule_tests_root.exists():
+            _process_local_tests(benchmark_cpes, env_yaml, rule_output_path, rule_tests_root)
+        if rendered_rule_obj["template"] is not None:
+            if "name" not in rendered_rule_obj["template"]:
+                raise ValueError(f"Invalid template config on rule {rule_id}")
+            template_name = rendered_rule_obj["template"]["name"]
+            template_root = templates_root / template_name
+            template_tests_root = template_root / "tests"
+            if not template_tests_root.exists():
+                logger.debug("Template %s doesn't have tests. Skipping for rule %s.",
+                             template_name, rule_id)
+                continue
+            test_config_path = rule_tests_root / "test_config.yml"
+            deny_templated_scenarios = _get_deny_templated_scenarios(test_config_path)
+            for test in template_tests_root.iterdir():  # type: pathlib.Path
+                if not test.name.endswith(".sh") or test.name in deny_templated_scenarios:
+                    logging.warning("Skipping % for %s", test.name, rule_id)
+                    continue
+                template = ssg.templates.Template.load_template(str(templates_root.absolute()),
+                                                                template_name)
+                rendered_rule_obj["template"]["vars"]["_rule_id"] = rule_id
+                template_parameters = template.preprocess(
+                    rendered_rule_obj["template"]["vars"], "test")
+                env_yaml = env_yaml.copy()
+                jinja_dict = ssg.utils.merge_dicts(env_yaml, template_parameters)
+                file_contents = ssg.jinja.process_file_with_macros(str(test.absolute()),
+                                                                   jinja_dict)
+                scenario = tests.ssg_test_suite.rule.Scenario(test.name, file_contents)
+                if scenario.matches_platform(benchmark_cpes):
+                    rule_output_path.mkdir(parents=True, exist_ok=True)
+                    test_output_path = rule_output_path / test.name
+                    _write_path(file_contents, test_output_path)
+                    logger.debug('Wrote scenario %s for rule %s', test.name, rule_id)
+                else:
+                    logger.debug('Skipping scenario %s for rule %s has it not applicable',
+                                 test.name, rule_id)
+
+
+def _get_benchmark_cpes(env_yaml):
+    benchmark_cpes = set()
+    for cpe in env_yaml["cpes"]:
+        for _, data in cpe.items():
+            benchmark_cpes.add(data["name"])
+    return benchmark_cpes
+
+
+def _get_rules_in_profile(built_profiles_root):
+    rules_in_profile = list()
+    for profile_file in built_profiles_root.iterdir():  # type: pathlib.Path
+        if not profile_file.name.endswith('.profile'):
+            continue
+        profile_data = ssg.yaml.open_raw(str(profile_file.absolute()))
+        for selection in profile_data["selections"]:
+            if '=' not in selection:
+                rules_in_profile.append(selection)
+    return rules_in_profile
+
+
+def main() -> int:
+    args = _create_arg_parser().parse_args()
+    logging.basicConfig(level=logging.getLevelName(args.log_level))
+    env_yaml = ssg.environment.open_environment(
+        args.build_config_yaml, args.product_yaml)
+
+    root_path = pathlib.Path(args.root).resolve().absolute()
+    output_path = pathlib.Path(args.output).resolve().absolute()
+    resolved_rules_dir = pathlib.Path(args.resolved_rules_dir)
+    if not resolved_rules_dir.exists() or not resolved_rules_dir.is_dir():
+        logging.error("Unable to find product at %s", str(resolved_rules_dir))
+        logging.error("Is the product built?")
+        return 1
+
+    output_path.mkdir(parents=True, exist_ok=True)
+    _copy_and_process_shared(env_yaml, output_path, root_path)
+    benchmark_cpes = _get_benchmark_cpes(env_yaml)
+
+    built_profiles_root = resolved_rules_dir.parent / "profiles"
+    rules_in_profiles = _get_rules_in_profile(built_profiles_root)
+
+    templates_root = root_path / "shared" / "templates"
+    all_resolved_rules = list(resolved_rules_dir.iterdir())
+    processes = list()
+    for chunk in range(args.jobs):
+        process_args = (benchmark_cpes, env_yaml, output_path,
+                        all_resolved_rules[chunk::args.jobs], templates_root, rules_in_profiles,)
+        process = multiprocessing.Process(target=_process_rules, args=process_args)
+        processes.append(process)
+        process.start()
+    for process in processes:
+        process.join()
+    # Write a file for CMake
+    # So we don't have dependency on a folder
+    done_file: pathlib.Path = output_path / ".test_done"
+    done_file.touch()
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())
path: build-scripts/build_tests.py
user: login: comps
body: I would honestly just let python interpreter defaults handle exit code - raise an exception if something bad happens, otherwise let it finish cleanly.

If you really want an exception-less exit on `resolved_rules_dir` not existing (seems like an arbitrary choice?), just do `sys.exit(1)` in that one case. `sys` will take care of the exit-related exception handling.
created_at: 2025-03-06T21:49:32Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,237 @@
+#!/usr/bin/env python3
+
+import argparse
+import logging
+import pathlib
+from typing import TypeVar, List, Iterable, Set, Dict
+import multiprocessing
+
+import ssg.environment
+import ssg.jinja
+import ssg.utils
+import ssg.yaml
+import ssg.templates
+import tests.ssg_test_suite.common
+import tests.ssg_test_suite.rule
+
+SSG_ROOT = str(pathlib.Path(__file__).resolve().parent.parent.absolute())
+JOB_COUNT = multiprocessing.cpu_count()
+T = TypeVar('T')
+
+
+def _create_arg_parser() -> argparse.ArgumentParser:
+    parser = argparse.ArgumentParser(
+        description="Converts built content tests to be rendered."
+    )
+    parser.add_argument(
+        "--build-config-yaml", required=True,
+        help="YAML file with information about the build configuration. "
+             "e.g.: ~/scap-security-guide/build/build_config.yml"
+    )
+    parser.add_argument(
+        "--product-yaml", required=True,
+        help="YAML file with information about the product we are building. "
+             "e.g.: ~/scap-security-guide/rhel10/product.yml"
+    )
+    parser.add_argument(
+        "--output", required=True,
+        help="Output path"
+             "e.g.:  ~/scap-security-guide/build/rhel10/tests"
+    )
+    parser.add_argument(
+        "--resolved-rules-dir", required=True,
+        help="Directory with <rule-id>.yml resolved rule YAMLs"
+             "e.g.: ~/scap-security-guide/build/rhel10/rules"
+    )
+    parser.add_argument("--log-level", action="store", type=str, default="ERROR",
+                        choices=["ERROR", "WARNING", "INFO", "DEBUG", "TRACE"],
+                        help="What level to log at. Defaults to ERROR.")
+    parser.add_argument("--root", default=SSG_ROOT,
+                        help=f"Path to the project. Defaults to {SSG_ROOT}")
+    parser.add_argument("--jobs", "-j", type=int, default=JOB_COUNT,
+                        help=f"Number of cores to use. Defaults to {JOB_COUNT} on this system.")
+    return parser
+
+
+def _is_test_file(filename: str) -> bool:
+    return filename.endswith(('.pass.sh', '.fail.sh', '.notapplicable.sh'))
+
+
+def _get_deny_templated_scenarios(test_config_path):
+    deny_templated_scenarios = list()
+    if test_config_path.exists():
+        test_config = ssg.yaml.open_raw(str(test_config_path.absolute()))
+        if 'deny_templated_scenarios' in test_config:
+            deny_templated_scenarios = test_config['deny_templated_scenarios']
+    return deny_templated_scenarios
path: build-scripts/build_tests.py
user: login: evgenyz
body: The `.get()` method is always shorter and in most of the cases faster:
```suggestion
def _get_deny_templated_scenarios(test_config_path):
    deny_templated_scenarios = set()
    if test_config_path.exists():
        test_config = ssg.yaml.open_raw(str(test_config_path.absolute()))
        deny_templated_scenarios |= test_config.get('deny_templated_scenarios', [])
    return deny_templated_scenarios
    
```
created_at: 2025-03-06T22:15:23Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,237 @@
+#!/usr/bin/env python3
+
+import argparse
+import logging
+import pathlib
+from typing import TypeVar, List, Iterable, Set, Dict
+import multiprocessing
+
+import ssg.environment
+import ssg.jinja
+import ssg.utils
+import ssg.yaml
+import ssg.templates
+import tests.ssg_test_suite.common
+import tests.ssg_test_suite.rule
+
+SSG_ROOT = str(pathlib.Path(__file__).resolve().parent.parent.absolute())
+JOB_COUNT = multiprocessing.cpu_count()
+T = TypeVar('T')
+
+
+def _create_arg_parser() -> argparse.ArgumentParser:
+    parser = argparse.ArgumentParser(
+        description="Converts built content tests to be rendered."
+    )
+    parser.add_argument(
+        "--build-config-yaml", required=True,
+        help="YAML file with information about the build configuration. "
+             "e.g.: ~/scap-security-guide/build/build_config.yml"
+    )
+    parser.add_argument(
+        "--product-yaml", required=True,
+        help="YAML file with information about the product we are building. "
+             "e.g.: ~/scap-security-guide/rhel10/product.yml"
+    )
+    parser.add_argument(
+        "--output", required=True,
+        help="Output path"
+             "e.g.:  ~/scap-security-guide/build/rhel10/tests"
+    )
+    parser.add_argument(
+        "--resolved-rules-dir", required=True,
+        help="Directory with <rule-id>.yml resolved rule YAMLs"
+             "e.g.: ~/scap-security-guide/build/rhel10/rules"
+    )
+    parser.add_argument("--log-level", action="store", type=str, default="ERROR",
+                        choices=["ERROR", "WARNING", "INFO", "DEBUG", "TRACE"],
+                        help="What level to log at. Defaults to ERROR.")
+    parser.add_argument("--root", default=SSG_ROOT,
+                        help=f"Path to the project. Defaults to {SSG_ROOT}")
+    parser.add_argument("--jobs", "-j", type=int, default=JOB_COUNT,
+                        help=f"Number of cores to use. Defaults to {JOB_COUNT} on this system.")
+    return parser
+
+
+def _is_test_file(filename: str) -> bool:
+    return filename.endswith(('.pass.sh', '.fail.sh', '.notapplicable.sh'))
+
+
+def _get_deny_templated_scenarios(test_config_path):
+    deny_templated_scenarios = list()
+    if test_config_path.exists():
+        test_config = ssg.yaml.open_raw(str(test_config_path.absolute()))
+        if 'deny_templated_scenarios' in test_config:
+            deny_templated_scenarios = test_config['deny_templated_scenarios']
+    return deny_templated_scenarios
+
+
+def _process_shared_file(env_yaml, file, shared_output_path):
+    file_contents = ssg.jinja.process_file_with_macros(str(file.absolute()),
+                                                       env_yaml)
+    shared_script_path = shared_output_path / file.name
+    with open(shared_script_path, 'w') as file:
+        file.write(file_contents)
+        file.write('\n')
+
+
+def _copy_and_process_shared(env_yaml, output_path, root_path):
+    tests_shared_root = root_path / "tests" / "shared"
+    shared_output_path = output_path / "shared"
+    shared_output_path.mkdir(parents=True, exist_ok=True)
+    for file in tests_shared_root.iterdir():  # type: pathlib.Path
+        # We only support one level deep, this avoids recursive functions
+        if file.is_dir():
+            for sub_file in file.iterdir():
+                shared_output_path_sub = shared_output_path / file.name
+                shared_output_path_sub.mkdir(parents=True, exist_ok=True)
+                _process_shared_file(env_yaml, sub_file, shared_output_path_sub)
+        else:
+            _process_shared_file(env_yaml, file, shared_output_path)
+
+
+def _process_local_tests(benchmark_cpes, env_yaml, rule_output_path, rule_tests_root):
+    logger = logging.getLogger()
+    for test in rule_tests_root.iterdir():  # type: pathlib.Path
+        if not test.name.endswith(".sh"):
+            logger.debug("Skipping file %s in rule %s is it doesn't end with .sh.",
+                         test.name, rule_output_path.name)
+            continue
+        if not _is_test_file(test.name):
+            file_contents = ssg.jinja.process_file_with_macros(str(test.absolute()),
+                                                               env_yaml)
+            output_file = rule_output_path / test.name
+            rule_output_path.mkdir(parents=True, exist_ok=True)
+            with open(output_file, 'w') as file:
+                file.write(file_contents)
+                file.write('\n')
+        file_contents = test.read_text()
+        scenario = tests.ssg_test_suite.rule.Scenario(test.name, file_contents)
+        if scenario.matches_platform(benchmark_cpes):
+            content = ssg.jinja.process_file_with_macros(str(test.absolute()), env_yaml)
+            rule_output_path.mkdir(parents=True, exist_ok=True)
+            with open(rule_output_path / test.name, 'w') as file:
+                file.write(content)
+                file.write('\n')
+
+
+def _write_path(file_contents, output_path):
+    with open(output_path, 'w') as file:
+        file.write(file_contents)
+        file.write('\n')
+
+
+def _process_rules(benchmark_cpes: Set[str], env_yaml: Dict, output_path: pathlib.Path,
+                   rules: Iterable[pathlib.Path], templates_root: pathlib.Path,
+                   product_rules: list) -> None:
+    logger = logging.getLogger()
+    for rule_file in rules:  # type: pathlib.Path
+        rendered_rule_obj = ssg.yaml.open_raw(str(rule_file))
+        rule_path = pathlib.Path(rendered_rule_obj["definition_location"])
+        rule_root = rule_path.parent
+        rule_id = rule_root.name
+        if rule_id not in product_rules:
+            logger.debug("Skipping %s since it is not in the product", rule_id)
+            continue
+        rule_tests_root = rule_root / "tests"
+        rule_output_path = output_path / rule_id
+
+        if rule_tests_root.exists():
+            _process_local_tests(benchmark_cpes, env_yaml, rule_output_path, rule_tests_root)
+        if rendered_rule_obj["template"] is not None:
+            if "name" not in rendered_rule_obj["template"]:
+                raise ValueError(f"Invalid template config on rule {rule_id}")
+            template_name = rendered_rule_obj["template"]["name"]
+            template_root = templates_root / template_name
+            template_tests_root = template_root / "tests"
+            if not template_tests_root.exists():
+                logger.debug("Template %s doesn't have tests. Skipping for rule %s.",
+                             template_name, rule_id)
+                continue
+            test_config_path = rule_tests_root / "test_config.yml"
+            deny_templated_scenarios = _get_deny_templated_scenarios(test_config_path)
+            for test in template_tests_root.iterdir():  # type: pathlib.Path
+                if not test.name.endswith(".sh") or test.name in deny_templated_scenarios:
+                    logging.warning("Skipping % for %s", test.name, rule_id)
+                    continue
+                template = ssg.templates.Template.load_template(str(templates_root.absolute()),
+                                                                template_name)
+                rendered_rule_obj["template"]["vars"]["_rule_id"] = rule_id
+                template_parameters = template.preprocess(
+                    rendered_rule_obj["template"]["vars"], "test")
+                env_yaml = env_yaml.copy()
+                jinja_dict = ssg.utils.merge_dicts(env_yaml, template_parameters)
+                file_contents = ssg.jinja.process_file_with_macros(str(test.absolute()),
+                                                                   jinja_dict)
+                scenario = tests.ssg_test_suite.rule.Scenario(test.name, file_contents)
+                if scenario.matches_platform(benchmark_cpes):
+                    rule_output_path.mkdir(parents=True, exist_ok=True)
+                    test_output_path = rule_output_path / test.name
+                    _write_path(file_contents, test_output_path)
+                    logger.debug('Wrote scenario %s for rule %s', test.name, rule_id)
+                else:
+                    logger.debug('Skipping scenario %s for rule %s has it not applicable',
+                                 test.name, rule_id)
+
+
+def _get_benchmark_cpes(env_yaml):
+    benchmark_cpes = set()
+    for cpe in env_yaml["cpes"]:
+        for _, data in cpe.items():
+            benchmark_cpes.add(data["name"])
+    return benchmark_cpes
+
+
+def _get_rules_in_profile(built_profiles_root):
+    rules_in_profile = list()
+    for profile_file in built_profiles_root.iterdir():  # type: pathlib.Path
+        if not profile_file.name.endswith('.profile'):
+            continue
+        profile_data = ssg.yaml.open_raw(str(profile_file.absolute()))
+        for selection in profile_data["selections"]:
+            if '=' not in selection:
+                rules_in_profile.append(selection)
+    return rules_in_profile
+
+
+def main() -> int:
+    args = _create_arg_parser().parse_args()
+    logging.basicConfig(level=logging.getLevelName(args.log_level))
+    env_yaml = ssg.environment.open_environment(
+        args.build_config_yaml, args.product_yaml)
+
+    root_path = pathlib.Path(args.root).resolve().absolute()
+    output_path = pathlib.Path(args.output).resolve().absolute()
+    resolved_rules_dir = pathlib.Path(args.resolved_rules_dir)
+    if not resolved_rules_dir.exists() or not resolved_rules_dir.is_dir():
+        logging.error("Unable to find product at %s", str(resolved_rules_dir))
+        logging.error("Is the product built?")
+        return 1
+
+    output_path.mkdir(parents=True, exist_ok=True)
+    _copy_and_process_shared(env_yaml, output_path, root_path)
+    benchmark_cpes = _get_benchmark_cpes(env_yaml)
+
+    built_profiles_root = resolved_rules_dir.parent / "profiles"
+    rules_in_profiles = _get_rules_in_profile(built_profiles_root)
+
+    templates_root = root_path / "shared" / "templates"
+    all_resolved_rules = list(resolved_rules_dir.iterdir())
+    processes = list()
+    for chunk in range(args.jobs):
+        process_args = (benchmark_cpes, env_yaml, output_path,
+                        all_resolved_rules[chunk::args.jobs], templates_root, rules_in_profiles,)
+        process = multiprocessing.Process(target=_process_rules, args=process_args)
+        processes.append(process)
+        process.start()
+    for process in processes:
+        process.join()
+    # Write a file for CMake
+    # So we don't have dependency on a folder
+    done_file: pathlib.Path = output_path / ".test_done"
+    done_file.touch()
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())
path: build-scripts/build_tests.py
user: login: Mab879
body: I can move to `sys.exit`. However, under the covers this is how `sys.exit` works.
created_at: 2025-03-06T22:28:28Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,237 @@
+#!/usr/bin/env python3
+
+import argparse
+import logging
+import pathlib
+from typing import TypeVar, List, Iterable, Set, Dict
+import multiprocessing
+
+import ssg.environment
+import ssg.jinja
+import ssg.utils
+import ssg.yaml
+import ssg.templates
+import tests.ssg_test_suite.common
+import tests.ssg_test_suite.rule
+
+SSG_ROOT = str(pathlib.Path(__file__).resolve().parent.parent.absolute())
+JOB_COUNT = multiprocessing.cpu_count()
+T = TypeVar('T')
+
+
+def _create_arg_parser() -> argparse.ArgumentParser:
+    parser = argparse.ArgumentParser(
+        description="Converts built content tests to be rendered."
+    )
+    parser.add_argument(
+        "--build-config-yaml", required=True,
+        help="YAML file with information about the build configuration. "
+             "e.g.: ~/scap-security-guide/build/build_config.yml"
+    )
+    parser.add_argument(
+        "--product-yaml", required=True,
+        help="YAML file with information about the product we are building. "
+             "e.g.: ~/scap-security-guide/rhel10/product.yml"
+    )
+    parser.add_argument(
+        "--output", required=True,
+        help="Output path"
+             "e.g.:  ~/scap-security-guide/build/rhel10/tests"
+    )
+    parser.add_argument(
+        "--resolved-rules-dir", required=True,
+        help="Directory with <rule-id>.yml resolved rule YAMLs"
+             "e.g.: ~/scap-security-guide/build/rhel10/rules"
+    )
+    parser.add_argument("--log-level", action="store", type=str, default="ERROR",
+                        choices=["ERROR", "WARNING", "INFO", "DEBUG", "TRACE"],
+                        help="What level to log at. Defaults to ERROR.")
+    parser.add_argument("--root", default=SSG_ROOT,
+                        help=f"Path to the project. Defaults to {SSG_ROOT}")
+    parser.add_argument("--jobs", "-j", type=int, default=JOB_COUNT,
+                        help=f"Number of cores to use. Defaults to {JOB_COUNT} on this system.")
+    return parser
+
+
+def _is_test_file(filename: str) -> bool:
+    return filename.endswith(('.pass.sh', '.fail.sh', '.notapplicable.sh'))
+
+
+def _get_deny_templated_scenarios(test_config_path):
+    deny_templated_scenarios = list()
+    if test_config_path.exists():
+        test_config = ssg.yaml.open_raw(str(test_config_path.absolute()))
+        if 'deny_templated_scenarios' in test_config:
+            deny_templated_scenarios = test_config['deny_templated_scenarios']
+    return deny_templated_scenarios
+
+
+def _process_shared_file(env_yaml, file, shared_output_path):
+    file_contents = ssg.jinja.process_file_with_macros(str(file.absolute()),
+                                                       env_yaml)
+    shared_script_path = shared_output_path / file.name
+    with open(shared_script_path, 'w') as file:
+        file.write(file_contents)
+        file.write('\n')
+
+
+def _copy_and_process_shared(env_yaml, output_path, root_path):
+    tests_shared_root = root_path / "tests" / "shared"
+    shared_output_path = output_path / "shared"
+    shared_output_path.mkdir(parents=True, exist_ok=True)
+    for file in tests_shared_root.iterdir():  # type: pathlib.Path
+        # We only support one level deep, this avoids recursive functions
+        if file.is_dir():
+            for sub_file in file.iterdir():
+                shared_output_path_sub = shared_output_path / file.name
+                shared_output_path_sub.mkdir(parents=True, exist_ok=True)
+                _process_shared_file(env_yaml, sub_file, shared_output_path_sub)
+        else:
+            _process_shared_file(env_yaml, file, shared_output_path)
+
+
+def _process_local_tests(benchmark_cpes, env_yaml, rule_output_path, rule_tests_root):
+    logger = logging.getLogger()
+    for test in rule_tests_root.iterdir():  # type: pathlib.Path
+        if not test.name.endswith(".sh"):
+            logger.debug("Skipping file %s in rule %s is it doesn't end with .sh.",
+                         test.name, rule_output_path.name)
+            continue
+        if not _is_test_file(test.name):
+            file_contents = ssg.jinja.process_file_with_macros(str(test.absolute()),
+                                                               env_yaml)
+            output_file = rule_output_path / test.name
+            rule_output_path.mkdir(parents=True, exist_ok=True)
+            with open(output_file, 'w') as file:
+                file.write(file_contents)
+                file.write('\n')
+        file_contents = test.read_text()
+        scenario = tests.ssg_test_suite.rule.Scenario(test.name, file_contents)
+        if scenario.matches_platform(benchmark_cpes):
+            content = ssg.jinja.process_file_with_macros(str(test.absolute()), env_yaml)
+            rule_output_path.mkdir(parents=True, exist_ok=True)
+            with open(rule_output_path / test.name, 'w') as file:
+                file.write(content)
+                file.write('\n')
+
+
+def _write_path(file_contents, output_path):
+    with open(output_path, 'w') as file:
+        file.write(file_contents)
+        file.write('\n')
+
+
+def _process_rules(benchmark_cpes: Set[str], env_yaml: Dict, output_path: pathlib.Path,
+                   rules: Iterable[pathlib.Path], templates_root: pathlib.Path,
+                   product_rules: list) -> None:
+    logger = logging.getLogger()
+    for rule_file in rules:  # type: pathlib.Path
+        rendered_rule_obj = ssg.yaml.open_raw(str(rule_file))
+        rule_path = pathlib.Path(rendered_rule_obj["definition_location"])
+        rule_root = rule_path.parent
+        rule_id = rule_root.name
+        if rule_id not in product_rules:
+            logger.debug("Skipping %s since it is not in the product", rule_id)
+            continue
+        rule_tests_root = rule_root / "tests"
+        rule_output_path = output_path / rule_id
+
+        if rule_tests_root.exists():
+            _process_local_tests(benchmark_cpes, env_yaml, rule_output_path, rule_tests_root)
+        if rendered_rule_obj["template"] is not None:
+            if "name" not in rendered_rule_obj["template"]:
+                raise ValueError(f"Invalid template config on rule {rule_id}")
+            template_name = rendered_rule_obj["template"]["name"]
+            template_root = templates_root / template_name
+            template_tests_root = template_root / "tests"
+            if not template_tests_root.exists():
+                logger.debug("Template %s doesn't have tests. Skipping for rule %s.",
+                             template_name, rule_id)
+                continue
+            test_config_path = rule_tests_root / "test_config.yml"
+            deny_templated_scenarios = _get_deny_templated_scenarios(test_config_path)
+            for test in template_tests_root.iterdir():  # type: pathlib.Path
+                if not test.name.endswith(".sh") or test.name in deny_templated_scenarios:
+                    logging.warning("Skipping % for %s", test.name, rule_id)
+                    continue
+                template = ssg.templates.Template.load_template(str(templates_root.absolute()),
+                                                                template_name)
+                rendered_rule_obj["template"]["vars"]["_rule_id"] = rule_id
+                template_parameters = template.preprocess(
+                    rendered_rule_obj["template"]["vars"], "test")
+                env_yaml = env_yaml.copy()
+                jinja_dict = ssg.utils.merge_dicts(env_yaml, template_parameters)
+                file_contents = ssg.jinja.process_file_with_macros(str(test.absolute()),
+                                                                   jinja_dict)
+                scenario = tests.ssg_test_suite.rule.Scenario(test.name, file_contents)
+                if scenario.matches_platform(benchmark_cpes):
+                    rule_output_path.mkdir(parents=True, exist_ok=True)
+                    test_output_path = rule_output_path / test.name
+                    _write_path(file_contents, test_output_path)
+                    logger.debug('Wrote scenario %s for rule %s', test.name, rule_id)
+                else:
+                    logger.debug('Skipping scenario %s for rule %s has it not applicable',
+                                 test.name, rule_id)
+
+
+def _get_benchmark_cpes(env_yaml):
+    benchmark_cpes = set()
+    for cpe in env_yaml["cpes"]:
+        for _, data in cpe.items():
+            benchmark_cpes.add(data["name"])
+    return benchmark_cpes
+
+
+def _get_rules_in_profile(built_profiles_root):
+    rules_in_profile = list()
+    for profile_file in built_profiles_root.iterdir():  # type: pathlib.Path
+        if not profile_file.name.endswith('.profile'):
+            continue
+        profile_data = ssg.yaml.open_raw(str(profile_file.absolute()))
+        for selection in profile_data["selections"]:
+            if '=' not in selection:
+                rules_in_profile.append(selection)
+    return rules_in_profile
+
+
+def main() -> int:
+    args = _create_arg_parser().parse_args()
+    logging.basicConfig(level=logging.getLevelName(args.log_level))
+    env_yaml = ssg.environment.open_environment(
+        args.build_config_yaml, args.product_yaml)
+
+    root_path = pathlib.Path(args.root).resolve().absolute()
+    output_path = pathlib.Path(args.output).resolve().absolute()
path: build-scripts/build_tests.py
user: login: Mab879
body: I can clean that up.
created_at: 2025-03-06T22:28:45Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,237 @@
+#!/usr/bin/env python3
+
+import argparse
+import logging
+import pathlib
+from typing import TypeVar, List, Iterable, Set, Dict
+import multiprocessing
+
+import ssg.environment
+import ssg.jinja
+import ssg.utils
+import ssg.yaml
+import ssg.templates
+import tests.ssg_test_suite.common
+import tests.ssg_test_suite.rule
+
+SSG_ROOT = str(pathlib.Path(__file__).resolve().parent.parent.absolute())
+JOB_COUNT = multiprocessing.cpu_count()
+T = TypeVar('T')
+
+
+def _create_arg_parser() -> argparse.ArgumentParser:
+    parser = argparse.ArgumentParser(
+        description="Converts built content tests to be rendered."
+    )
+    parser.add_argument(
+        "--build-config-yaml", required=True,
+        help="YAML file with information about the build configuration. "
+             "e.g.: ~/scap-security-guide/build/build_config.yml"
+    )
+    parser.add_argument(
+        "--product-yaml", required=True,
+        help="YAML file with information about the product we are building. "
+             "e.g.: ~/scap-security-guide/rhel10/product.yml"
+    )
+    parser.add_argument(
+        "--output", required=True,
+        help="Output path"
+             "e.g.:  ~/scap-security-guide/build/rhel10/tests"
+    )
+    parser.add_argument(
+        "--resolved-rules-dir", required=True,
+        help="Directory with <rule-id>.yml resolved rule YAMLs"
+             "e.g.: ~/scap-security-guide/build/rhel10/rules"
+    )
+    parser.add_argument("--log-level", action="store", type=str, default="ERROR",
+                        choices=["ERROR", "WARNING", "INFO", "DEBUG", "TRACE"],
+                        help="What level to log at. Defaults to ERROR.")
+    parser.add_argument("--root", default=SSG_ROOT,
+                        help=f"Path to the project. Defaults to {SSG_ROOT}")
+    parser.add_argument("--jobs", "-j", type=int, default=JOB_COUNT,
+                        help=f"Number of cores to use. Defaults to {JOB_COUNT} on this system.")
+    return parser
+
+
+def _is_test_file(filename: str) -> bool:
+    return filename.endswith(('.pass.sh', '.fail.sh', '.notapplicable.sh'))
+
+
+def _get_deny_templated_scenarios(test_config_path):
+    deny_templated_scenarios = list()
+    if test_config_path.exists():
+        test_config = ssg.yaml.open_raw(str(test_config_path.absolute()))
+        if 'deny_templated_scenarios' in test_config:
+            deny_templated_scenarios = test_config['deny_templated_scenarios']
+    return deny_templated_scenarios
+
+
+def _process_shared_file(env_yaml, file, shared_output_path):
+    file_contents = ssg.jinja.process_file_with_macros(str(file.absolute()),
+                                                       env_yaml)
+    shared_script_path = shared_output_path / file.name
+    with open(shared_script_path, 'w') as file:
+        file.write(file_contents)
+        file.write('\n')
+
+
+def _copy_and_process_shared(env_yaml, output_path, root_path):
+    tests_shared_root = root_path / "tests" / "shared"
+    shared_output_path = output_path / "shared"
+    shared_output_path.mkdir(parents=True, exist_ok=True)
+    for file in tests_shared_root.iterdir():  # type: pathlib.Path
+        # We only support one level deep, this avoids recursive functions
+        if file.is_dir():
+            for sub_file in file.iterdir():
+                shared_output_path_sub = shared_output_path / file.name
+                shared_output_path_sub.mkdir(parents=True, exist_ok=True)
+                _process_shared_file(env_yaml, sub_file, shared_output_path_sub)
+        else:
+            _process_shared_file(env_yaml, file, shared_output_path)
+
+
+def _process_local_tests(benchmark_cpes, env_yaml, rule_output_path, rule_tests_root):
+    logger = logging.getLogger()
+    for test in rule_tests_root.iterdir():  # type: pathlib.Path
+        if not test.name.endswith(".sh"):
+            logger.debug("Skipping file %s in rule %s is it doesn't end with .sh.",
+                         test.name, rule_output_path.name)
+            continue
+        if not _is_test_file(test.name):
+            file_contents = ssg.jinja.process_file_with_macros(str(test.absolute()),
+                                                               env_yaml)
+            output_file = rule_output_path / test.name
+            rule_output_path.mkdir(parents=True, exist_ok=True)
+            with open(output_file, 'w') as file:
+                file.write(file_contents)
+                file.write('\n')
+        file_contents = test.read_text()
+        scenario = tests.ssg_test_suite.rule.Scenario(test.name, file_contents)
+        if scenario.matches_platform(benchmark_cpes):
+            content = ssg.jinja.process_file_with_macros(str(test.absolute()), env_yaml)
+            rule_output_path.mkdir(parents=True, exist_ok=True)
+            with open(rule_output_path / test.name, 'w') as file:
+                file.write(content)
+                file.write('\n')
+
+
+def _write_path(file_contents, output_path):
+    with open(output_path, 'w') as file:
+        file.write(file_contents)
+        file.write('\n')
+
+
+def _process_rules(benchmark_cpes: Set[str], env_yaml: Dict, output_path: pathlib.Path,
+                   rules: Iterable[pathlib.Path], templates_root: pathlib.Path,
+                   product_rules: list) -> None:
+    logger = logging.getLogger()
+    for rule_file in rules:  # type: pathlib.Path
+        rendered_rule_obj = ssg.yaml.open_raw(str(rule_file))
+        rule_path = pathlib.Path(rendered_rule_obj["definition_location"])
+        rule_root = rule_path.parent
+        rule_id = rule_root.name
+        if rule_id not in product_rules:
+            logger.debug("Skipping %s since it is not in the product", rule_id)
+            continue
+        rule_tests_root = rule_root / "tests"
+        rule_output_path = output_path / rule_id
+
+        if rule_tests_root.exists():
+            _process_local_tests(benchmark_cpes, env_yaml, rule_output_path, rule_tests_root)
+        if rendered_rule_obj["template"] is not None:
+            if "name" not in rendered_rule_obj["template"]:
+                raise ValueError(f"Invalid template config on rule {rule_id}")
+            template_name = rendered_rule_obj["template"]["name"]
+            template_root = templates_root / template_name
+            template_tests_root = template_root / "tests"
+            if not template_tests_root.exists():
+                logger.debug("Template %s doesn't have tests. Skipping for rule %s.",
+                             template_name, rule_id)
+                continue
+            test_config_path = rule_tests_root / "test_config.yml"
+            deny_templated_scenarios = _get_deny_templated_scenarios(test_config_path)
+            for test in template_tests_root.iterdir():  # type: pathlib.Path
+                if not test.name.endswith(".sh") or test.name in deny_templated_scenarios:
+                    logging.warning("Skipping % for %s", test.name, rule_id)
+                    continue
+                template = ssg.templates.Template.load_template(str(templates_root.absolute()),
+                                                                template_name)
+                rendered_rule_obj["template"]["vars"]["_rule_id"] = rule_id
+                template_parameters = template.preprocess(
+                    rendered_rule_obj["template"]["vars"], "test")
+                env_yaml = env_yaml.copy()
+                jinja_dict = ssg.utils.merge_dicts(env_yaml, template_parameters)
+                file_contents = ssg.jinja.process_file_with_macros(str(test.absolute()),
+                                                                   jinja_dict)
+                scenario = tests.ssg_test_suite.rule.Scenario(test.name, file_contents)
+                if scenario.matches_platform(benchmark_cpes):
+                    rule_output_path.mkdir(parents=True, exist_ok=True)
+                    test_output_path = rule_output_path / test.name
+                    _write_path(file_contents, test_output_path)
+                    logger.debug('Wrote scenario %s for rule %s', test.name, rule_id)
+                else:
+                    logger.debug('Skipping scenario %s for rule %s has it not applicable',
+                                 test.name, rule_id)
+
+
+def _get_benchmark_cpes(env_yaml):
+    benchmark_cpes = set()
+    for cpe in env_yaml["cpes"]:
+        for _, data in cpe.items():
+            benchmark_cpes.add(data["name"])
+    return benchmark_cpes
+
+
+def _get_rules_in_profile(built_profiles_root):
+    rules_in_profile = list()
+    for profile_file in built_profiles_root.iterdir():  # type: pathlib.Path
+        if not profile_file.name.endswith('.profile'):
+            continue
+        profile_data = ssg.yaml.open_raw(str(profile_file.absolute()))
+        for selection in profile_data["selections"]:
+            if '=' not in selection:
+                rules_in_profile.append(selection)
+    return rules_in_profile
+
+
+def main() -> int:
+    args = _create_arg_parser().parse_args()
+    logging.basicConfig(level=logging.getLevelName(args.log_level))
+    env_yaml = ssg.environment.open_environment(
+        args.build_config_yaml, args.product_yaml)
+
+    root_path = pathlib.Path(args.root).resolve().absolute()
+    output_path = pathlib.Path(args.output).resolve().absolute()
+    resolved_rules_dir = pathlib.Path(args.resolved_rules_dir)
+    if not resolved_rules_dir.exists() or not resolved_rules_dir.is_dir():
+        logging.error("Unable to find product at %s", str(resolved_rules_dir))
+        logging.error("Is the product built?")
+        return 1
+
+    output_path.mkdir(parents=True, exist_ok=True)
+    _copy_and_process_shared(env_yaml, output_path, root_path)
+    benchmark_cpes = _get_benchmark_cpes(env_yaml)
+
+    built_profiles_root = resolved_rules_dir.parent / "profiles"
+    rules_in_profiles = _get_rules_in_profile(built_profiles_root)
+
+    templates_root = root_path / "shared" / "templates"
+    all_resolved_rules = list(resolved_rules_dir.iterdir())
+    processes = list()
+    for chunk in range(args.jobs):
+        process_args = (benchmark_cpes, env_yaml, output_path,
+                        all_resolved_rules[chunk::args.jobs], templates_root, rules_in_profiles,)
+        process = multiprocessing.Process(target=_process_rules, args=process_args)
+        processes.append(process)
+        process.start()
+    for process in processes:
+        process.join()
+    # Write a file for CMake
+    # So we don't have dependency on a folder
+    done_file: pathlib.Path = output_path / ".test_done"
+    done_file.touch()
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())
path: build-scripts/build_tests.py
user: login: comps
body: I guess either way works. I remember some problems with raising `SystemExit()` with non-CPython intepreters, but we don't use those here, and those problems are probably fixed by now anyway. I would still leave the `0` exit to be done by default instead of `SystemExit(0)`, but that's probably bikeshedding.
created_at: 2025-03-06T22:39:56Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,230 @@
+#!/usr/bin/env python3
path: build-scripts/build_tests.py
user: login: jan-cerny
body: I don't like that the PR description says in rationale "Make testing with thin data streams possible." because testing with thin data stream has been possible for many months and people are testing with thin data streams daily. Please reword the rationale.
created_at: 2025-03-07T08:47:31Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,230 @@
+#!/usr/bin/env python3
+
path: build-scripts/build_tests.py
user: login: jan-cerny
body: This script is slow, it takes more than 2 minutes to build scenarios for RHEL 9. We need to think to speed up.
created_at: 2025-03-07T08:53:13Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,230 @@
+#!/usr/bin/env python3
+
+import argparse
+import logging
+import os
+import pathlib
+import sys
+from typing import TypeVar, Generator, Iterable, Set, Dict
+import multiprocessing
+
+import ssg.environment
+import ssg.jinja
+import ssg.utils
+import ssg.yaml
+import ssg.templates
+import tests.ssg_test_suite.common
+import tests.ssg_test_suite.rule
+
+SSG_ROOT = str(pathlib.Path(__file__).resolve().parent.parent.absolute())
+JOB_COUNT = multiprocessing.cpu_count()
+T = TypeVar("T")
+
+
+def _create_arg_parser() -> argparse.ArgumentParser:
+    parser = argparse.ArgumentParser(
+        description="Converts built content tests to be rendered."
+    )
+    parser.add_argument(
+        "--build-config-yaml", required=True,
+        help="YAML file with information about the build configuration. "
+             "e.g.: ~/scap-security-guide/build/build_config.yml"
+    )
+    parser.add_argument(
+        "--product-yaml", required=True,
+        help="YAML file with information about the product we are building. "
+             "e.g.: ~/scap-security-guide/rhel10/product.yml"
+    )
+    parser.add_argument(
+        "--output", required=True,
+        help="Output path"
+             "e.g.:  ~/scap-security-guide/build/rhel10/tests"
+    )
+    parser.add_argument(
+        "--resolved-rules-dir", required=True,
+        help="Directory with <rule-id>.yml resolved rule YAMLs"
+             "e.g.: ~/scap-security-guide/build/rhel10/rules"
+    )
+    parser.add_argument("--log-level", action="store", type=str, default="ERROR",
+                        choices=["ERROR", "WARNING", "INFO", "DEBUG", "TRACE"],
+                        help="What level to log at. Defaults to ERROR.")
+    parser.add_argument("--root", default=SSG_ROOT,
+                        help=f"Path to the project. Defaults to {SSG_ROOT}")
+    parser.add_argument("--jobs", "-j", type=int, default=JOB_COUNT,
+                        help=f"Number of cores to use. Defaults to {JOB_COUNT} on this system.")
+    return parser
+
+
+def _write_path(file_contents: str, output_path: os.PathLike) -> None:
+    with open(output_path, "w") as file:
+        file.write(file_contents)
+        file.write("\n")
+
+
+def _is_test_file(filename: str) -> bool:
+    return filename.endswith(('.pass.sh', '.fail.sh', '.notapplicable.sh'))
+
+
+def _get_deny_templated_scenarios(test_config_path: pathlib.Path) -> Set[str]:
+    if test_config_path.exists():
+        test_config = ssg.yaml.open_raw(str(test_config_path.absolute()))
+        deny_templated_scenarios = test_config.get('deny_templated_scenarios', set())
+        return deny_templated_scenarios
+    return set()
+
+
+def _process_shared_file(env_yaml: dict, file: pathlib.Path, shared_output_path: pathlib.Path) \
+        -> None:
+    file_contents = ssg.jinja.process_file_with_macros(str(file.absolute()), env_yaml)
+    shared_script_path = shared_output_path / file.name
+    _write_path(file_contents, shared_script_path)
+
+
+def _copy_and_process_shared(env_yaml: dict, output_path: pathlib.Path, root_path: pathlib.Path) \
+        -> None:
+    tests_shared_root = root_path / "tests" / "shared"
+    shared_output_path = output_path / "shared"
+    shared_output_path.mkdir(parents=True, exist_ok=True)
+    for file in tests_shared_root.iterdir():  # type: pathlib.Path
+        # We only support one level deep, this avoids recursive functions
+        if file.is_dir():
+            for sub_file in file.iterdir():
+                shared_output_path_sub = shared_output_path / file.name
+                shared_output_path_sub.mkdir(parents=True, exist_ok=True)
+                _process_shared_file(env_yaml, sub_file, shared_output_path_sub)
+        else:
+            _process_shared_file(env_yaml, file, shared_output_path)
+
+
+def _process_local_tests(benchmark_cpes: Set[str], env_yaml: dict, rule_output_path: pathlib.Path,
+                         rule_tests_root: pathlib.Path) -> None:
+    logger = logging.getLogger()
+    for test in rule_tests_root.iterdir():  # type: pathlib.Path
+        if not test.name.endswith(".sh"):
+            logger.debug("Skipping file %s in rule %s is it doesn't end with .sh.",
+                         test.name, rule_output_path.name)
+            continue
+        if not _is_test_file(test.name):
+            file_contents = ssg.jinja.process_file_with_macros(str(test.absolute()),
+                                                               env_yaml)
+            output_file = rule_output_path / test.name
+            rule_output_path.mkdir(parents=True, exist_ok=True)
+            _write_path(file_contents, output_file)
+        file_contents = test.read_text()
+        scenario = tests.ssg_test_suite.rule.Scenario(test.name, file_contents)
+        if scenario.matches_platform(benchmark_cpes):
+            content = ssg.jinja.process_file_with_macros(str(test.absolute()), env_yaml)
+            rule_output_path.mkdir(parents=True, exist_ok=True)
+            _write_path(content, rule_output_path / test.name)
+
+
+def _process_rules(benchmark_cpes: Set[str], env_yaml: Dict, output_path: pathlib.Path,
+                   rules: Iterable[pathlib.Path], templates_root: pathlib.Path,
+                   product_rules: list) -> None:
+    logger = logging.getLogger()
+    for rule_file in rules:  # type: pathlib.Path
+        rendered_rule_obj = ssg.yaml.open_raw(str(rule_file))
+        rule_path = pathlib.Path(rendered_rule_obj["definition_location"])
+        rule_root = rule_path.parent
+        rule_id = rule_root.name
+        if rule_id not in product_rules:
+            logger.debug("Skipping %s since it is not in the product", rule_id)
+            continue
+        rule_tests_root = rule_root / "tests"
+        rule_output_path = output_path / rule_id
+
+        if rule_tests_root.exists():
+            _process_local_tests(benchmark_cpes, env_yaml, rule_output_path, rule_tests_root)
+        if rendered_rule_obj["template"] is not None:
+            if "name" not in rendered_rule_obj["template"]:
+                raise ValueError(f"Invalid template config on rule {rule_id}")
+            template_name = rendered_rule_obj["template"]["name"]
+            template_root = templates_root / template_name
+            template_tests_root = template_root / "tests"
+            if not template_tests_root.exists():
+                logger.debug("Template %s doesn't have tests. Skipping for rule %s.",
+                             template_name, rule_id)
+                continue
+            test_config_path = rule_tests_root / "test_config.yml"
+            deny_templated_scenarios = _get_deny_templated_scenarios(test_config_path)
+            for test in template_tests_root.iterdir():  # type: pathlib.Path
+                if not test.name.endswith(".sh") or test.name in deny_templated_scenarios:
+                    logging.warning("Skipping % for %s", test.name, rule_id)
+                    continue
+                template = ssg.templates.Template.load_template(str(templates_root.absolute()),
+                                                                template_name)
+                rendered_rule_obj["template"]["vars"]["_rule_id"] = rule_id
+                template_parameters = template.preprocess(
+                    rendered_rule_obj["template"]["vars"], "test")
+                env_yaml = env_yaml.copy()
+                jinja_dict = ssg.utils.merge_dicts(env_yaml, template_parameters)
+                file_contents = ssg.jinja.process_file_with_macros(str(test.absolute()),
+                                                                   jinja_dict)
+                scenario = tests.ssg_test_suite.rule.Scenario(test.name, file_contents)
+                if scenario.matches_platform(benchmark_cpes):
+                    rule_output_path.mkdir(parents=True, exist_ok=True)
+                    test_output_path = rule_output_path / test.name
+                    _write_path(file_contents, test_output_path)
+                    logger.debug("Wrote scenario %s for rule %s", test.name, rule_id)
+                else:
+                    logger.debug("Skipping scenario %s for rule %s has it not applicable",
+                                 test.name, rule_id)
+
+
+def _get_benchmark_cpes(env_yaml) -> Set[str]:
+    benchmark_cpes = set()
+    for cpe in env_yaml["cpes"]:
+        for _, data in cpe.items():
+            benchmark_cpes.add(data["name"])
+    return benchmark_cpes
+
+
+def _get_rules_in_profile(built_profiles_root) -> Generator[str, None, None]:
+    for profile_file in built_profiles_root.iterdir():  # type: pathlib.Path
+        if not profile_file.name.endswith(".profile"):
+            continue
+        profile_data = ssg.yaml.open_raw(str(profile_file.absolute()))
+        for selection in profile_data["selections"]:
+            if "=" not in selection:
+                yield selection
+
+
+def main() -> int:
+    args = _create_arg_parser().parse_args()
+    logging.basicConfig(level=logging.getLevelName(args.log_level))
+    env_yaml = ssg.environment.open_environment(args.build_config_yaml, args.product_yaml)
+    root_path = pathlib.Path(args.root).resolve().absolute()
+    output_path = pathlib.Path(args.output).resolve().absolute()
+    resolved_rules_dir = pathlib.Path(args.resolved_rules_dir)
+    if not resolved_rules_dir.exists() or not resolved_rules_dir.is_dir():
+        logging.error("Unable to find product at %s", str(resolved_rules_dir))
+        logging.error("Is the product built?")
+        return 1
+
+    output_path.mkdir(parents=True, exist_ok=True)
+    _copy_and_process_shared(env_yaml, output_path, root_path)
+    benchmark_cpes = _get_benchmark_cpes(env_yaml)
+
+    built_profiles_root = resolved_rules_dir.parent / "profiles"
+    rules_in_profiles = list(_get_rules_in_profile(built_profiles_root))
path: build-scripts/build_tests.py
user: login: jan-cerny
body: This should be a set because many rules are present in more than 1 profile so you have a lot of duplicates in this list. Converting to a set will automatically eliminate duplicates because each item can be only once in a set.
created_at: 2025-03-07T08:59:11Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,230 @@
+#!/usr/bin/env python3
+
+import argparse
+import logging
+import os
+import pathlib
+import sys
+from typing import TypeVar, Generator, Iterable, Set, Dict
+import multiprocessing
+
+import ssg.environment
+import ssg.jinja
+import ssg.utils
+import ssg.yaml
+import ssg.templates
+import tests.ssg_test_suite.common
+import tests.ssg_test_suite.rule
+
+SSG_ROOT = str(pathlib.Path(__file__).resolve().parent.parent.absolute())
+JOB_COUNT = multiprocessing.cpu_count()
+T = TypeVar("T")
+
+
+def _create_arg_parser() -> argparse.ArgumentParser:
+    parser = argparse.ArgumentParser(
+        description="Converts built content tests to be rendered."
+    )
+    parser.add_argument(
+        "--build-config-yaml", required=True,
+        help="YAML file with information about the build configuration. "
+             "e.g.: ~/scap-security-guide/build/build_config.yml"
+    )
+    parser.add_argument(
+        "--product-yaml", required=True,
+        help="YAML file with information about the product we are building. "
+             "e.g.: ~/scap-security-guide/rhel10/product.yml"
+    )
+    parser.add_argument(
+        "--output", required=True,
+        help="Output path"
+             "e.g.:  ~/scap-security-guide/build/rhel10/tests"
+    )
+    parser.add_argument(
+        "--resolved-rules-dir", required=True,
+        help="Directory with <rule-id>.yml resolved rule YAMLs"
+             "e.g.: ~/scap-security-guide/build/rhel10/rules"
+    )
+    parser.add_argument("--log-level", action="store", type=str, default="ERROR",
+                        choices=["ERROR", "WARNING", "INFO", "DEBUG", "TRACE"],
+                        help="What level to log at. Defaults to ERROR.")
+    parser.add_argument("--root", default=SSG_ROOT,
+                        help=f"Path to the project. Defaults to {SSG_ROOT}")
+    parser.add_argument("--jobs", "-j", type=int, default=JOB_COUNT,
+                        help=f"Number of cores to use. Defaults to {JOB_COUNT} on this system.")
+    return parser
+
+
+def _write_path(file_contents: str, output_path: os.PathLike) -> None:
+    with open(output_path, "w") as file:
+        file.write(file_contents)
+        file.write("\n")
+
+
+def _is_test_file(filename: str) -> bool:
+    return filename.endswith(('.pass.sh', '.fail.sh', '.notapplicable.sh'))
+
+
+def _get_deny_templated_scenarios(test_config_path: pathlib.Path) -> Set[str]:
+    if test_config_path.exists():
+        test_config = ssg.yaml.open_raw(str(test_config_path.absolute()))
+        deny_templated_scenarios = test_config.get('deny_templated_scenarios', set())
+        return deny_templated_scenarios
+    return set()
+
+
+def _process_shared_file(env_yaml: dict, file: pathlib.Path, shared_output_path: pathlib.Path) \
+        -> None:
+    file_contents = ssg.jinja.process_file_with_macros(str(file.absolute()), env_yaml)
+    shared_script_path = shared_output_path / file.name
+    _write_path(file_contents, shared_script_path)
+
+
+def _copy_and_process_shared(env_yaml: dict, output_path: pathlib.Path, root_path: pathlib.Path) \
+        -> None:
+    tests_shared_root = root_path / "tests" / "shared"
+    shared_output_path = output_path / "shared"
+    shared_output_path.mkdir(parents=True, exist_ok=True)
+    for file in tests_shared_root.iterdir():  # type: pathlib.Path
+        # We only support one level deep, this avoids recursive functions
+        if file.is_dir():
+            for sub_file in file.iterdir():
+                shared_output_path_sub = shared_output_path / file.name
+                shared_output_path_sub.mkdir(parents=True, exist_ok=True)
+                _process_shared_file(env_yaml, sub_file, shared_output_path_sub)
+        else:
+            _process_shared_file(env_yaml, file, shared_output_path)
+
+
+def _process_local_tests(benchmark_cpes: Set[str], env_yaml: dict, rule_output_path: pathlib.Path,
+                         rule_tests_root: pathlib.Path) -> None:
+    logger = logging.getLogger()
+    for test in rule_tests_root.iterdir():  # type: pathlib.Path
+        if not test.name.endswith(".sh"):
+            logger.debug("Skipping file %s in rule %s is it doesn't end with .sh.",
+                         test.name, rule_output_path.name)
+            continue
+        if not _is_test_file(test.name):
+            file_contents = ssg.jinja.process_file_with_macros(str(test.absolute()),
+                                                               env_yaml)
+            output_file = rule_output_path / test.name
+            rule_output_path.mkdir(parents=True, exist_ok=True)
+            _write_path(file_contents, output_file)
+        file_contents = test.read_text()
+        scenario = tests.ssg_test_suite.rule.Scenario(test.name, file_contents)
+        if scenario.matches_platform(benchmark_cpes):
+            content = ssg.jinja.process_file_with_macros(str(test.absolute()), env_yaml)
+            rule_output_path.mkdir(parents=True, exist_ok=True)
+            _write_path(content, rule_output_path / test.name)
+
+
+def _process_rules(benchmark_cpes: Set[str], env_yaml: Dict, output_path: pathlib.Path,
+                   rules: Iterable[pathlib.Path], templates_root: pathlib.Path,
+                   product_rules: list) -> None:
+    logger = logging.getLogger()
+    for rule_file in rules:  # type: pathlib.Path
+        rendered_rule_obj = ssg.yaml.open_raw(str(rule_file))
+        rule_path = pathlib.Path(rendered_rule_obj["definition_location"])
+        rule_root = rule_path.parent
+        rule_id = rule_root.name
+        if rule_id not in product_rules:
+            logger.debug("Skipping %s since it is not in the product", rule_id)
path: build-scripts/build_tests.py
user: login: jan-cerny
body: You don't have to open the resolved rule yaml file if it is skipped. You can get the rule id from the file name. It will save some time if you don't open resolved rules that aren't going to be processed.
created_at: 2025-03-07T09:06:17Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,230 @@
+#!/usr/bin/env python3
+
+import argparse
+import logging
+import os
+import pathlib
+import sys
+from typing import TypeVar, Generator, Iterable, Set, Dict
+import multiprocessing
+
+import ssg.environment
+import ssg.jinja
+import ssg.utils
+import ssg.yaml
+import ssg.templates
+import tests.ssg_test_suite.common
+import tests.ssg_test_suite.rule
+
+SSG_ROOT = str(pathlib.Path(__file__).resolve().parent.parent.absolute())
+JOB_COUNT = multiprocessing.cpu_count()
+T = TypeVar("T")
+
+
+def _create_arg_parser() -> argparse.ArgumentParser:
+    parser = argparse.ArgumentParser(
+        description="Converts built content tests to be rendered."
+    )
+    parser.add_argument(
+        "--build-config-yaml", required=True,
+        help="YAML file with information about the build configuration. "
+             "e.g.: ~/scap-security-guide/build/build_config.yml"
+    )
+    parser.add_argument(
+        "--product-yaml", required=True,
+        help="YAML file with information about the product we are building. "
+             "e.g.: ~/scap-security-guide/rhel10/product.yml"
+    )
+    parser.add_argument(
+        "--output", required=True,
+        help="Output path"
+             "e.g.:  ~/scap-security-guide/build/rhel10/tests"
+    )
+    parser.add_argument(
+        "--resolved-rules-dir", required=True,
+        help="Directory with <rule-id>.yml resolved rule YAMLs"
+             "e.g.: ~/scap-security-guide/build/rhel10/rules"
+    )
+    parser.add_argument("--log-level", action="store", type=str, default="ERROR",
+                        choices=["ERROR", "WARNING", "INFO", "DEBUG", "TRACE"],
+                        help="What level to log at. Defaults to ERROR.")
+    parser.add_argument("--root", default=SSG_ROOT,
+                        help=f"Path to the project. Defaults to {SSG_ROOT}")
+    parser.add_argument("--jobs", "-j", type=int, default=JOB_COUNT,
+                        help=f"Number of cores to use. Defaults to {JOB_COUNT} on this system.")
+    return parser
+
+
+def _write_path(file_contents: str, output_path: os.PathLike) -> None:
+    with open(output_path, "w") as file:
+        file.write(file_contents)
+        file.write("\n")
+
+
+def _is_test_file(filename: str) -> bool:
+    return filename.endswith(('.pass.sh', '.fail.sh', '.notapplicable.sh'))
+
+
+def _get_deny_templated_scenarios(test_config_path: pathlib.Path) -> Set[str]:
+    if test_config_path.exists():
+        test_config = ssg.yaml.open_raw(str(test_config_path.absolute()))
+        deny_templated_scenarios = test_config.get('deny_templated_scenarios', set())
+        return deny_templated_scenarios
+    return set()
+
+
+def _process_shared_file(env_yaml: dict, file: pathlib.Path, shared_output_path: pathlib.Path) \
+        -> None:
+    file_contents = ssg.jinja.process_file_with_macros(str(file.absolute()), env_yaml)
+    shared_script_path = shared_output_path / file.name
+    _write_path(file_contents, shared_script_path)
+
+
+def _copy_and_process_shared(env_yaml: dict, output_path: pathlib.Path, root_path: pathlib.Path) \
+        -> None:
+    tests_shared_root = root_path / "tests" / "shared"
+    shared_output_path = output_path / "shared"
+    shared_output_path.mkdir(parents=True, exist_ok=True)
+    for file in tests_shared_root.iterdir():  # type: pathlib.Path
+        # We only support one level deep, this avoids recursive functions
+        if file.is_dir():
+            for sub_file in file.iterdir():
+                shared_output_path_sub = shared_output_path / file.name
+                shared_output_path_sub.mkdir(parents=True, exist_ok=True)
+                _process_shared_file(env_yaml, sub_file, shared_output_path_sub)
+        else:
+            _process_shared_file(env_yaml, file, shared_output_path)
+
+
+def _process_local_tests(benchmark_cpes: Set[str], env_yaml: dict, rule_output_path: pathlib.Path,
+                         rule_tests_root: pathlib.Path) -> None:
+    logger = logging.getLogger()
+    for test in rule_tests_root.iterdir():  # type: pathlib.Path
+        if not test.name.endswith(".sh"):
+            logger.debug("Skipping file %s in rule %s is it doesn't end with .sh.",
+                         test.name, rule_output_path.name)
+            continue
+        if not _is_test_file(test.name):
+            file_contents = ssg.jinja.process_file_with_macros(str(test.absolute()),
+                                                               env_yaml)
+            output_file = rule_output_path / test.name
+            rule_output_path.mkdir(parents=True, exist_ok=True)
+            _write_path(file_contents, output_file)
+        file_contents = test.read_text()
+        scenario = tests.ssg_test_suite.rule.Scenario(test.name, file_contents)
+        if scenario.matches_platform(benchmark_cpes):
+            content = ssg.jinja.process_file_with_macros(str(test.absolute()), env_yaml)
+            rule_output_path.mkdir(parents=True, exist_ok=True)
+            _write_path(content, rule_output_path / test.name)
+
+
+def _process_rules(benchmark_cpes: Set[str], env_yaml: Dict, output_path: pathlib.Path,
+                   rules: Iterable[pathlib.Path], templates_root: pathlib.Path,
+                   product_rules: list) -> None:
+    logger = logging.getLogger()
+    for rule_file in rules:  # type: pathlib.Path
+        rendered_rule_obj = ssg.yaml.open_raw(str(rule_file))
+        rule_path = pathlib.Path(rendered_rule_obj["definition_location"])
+        rule_root = rule_path.parent
+        rule_id = rule_root.name
+        if rule_id not in product_rules:
+            logger.debug("Skipping %s since it is not in the product", rule_id)
+            continue
+        rule_tests_root = rule_root / "tests"
+        rule_output_path = output_path / rule_id
+
+        if rule_tests_root.exists():
+            _process_local_tests(benchmark_cpes, env_yaml, rule_output_path, rule_tests_root)
+        if rendered_rule_obj["template"] is not None:
+            if "name" not in rendered_rule_obj["template"]:
+                raise ValueError(f"Invalid template config on rule {rule_id}")
+            template_name = rendered_rule_obj["template"]["name"]
+            template_root = templates_root / template_name
+            template_tests_root = template_root / "tests"
+            if not template_tests_root.exists():
+                logger.debug("Template %s doesn't have tests. Skipping for rule %s.",
+                             template_name, rule_id)
+                continue
+            test_config_path = rule_tests_root / "test_config.yml"
+            deny_templated_scenarios = _get_deny_templated_scenarios(test_config_path)
+            for test in template_tests_root.iterdir():  # type: pathlib.Path
+                if not test.name.endswith(".sh") or test.name in deny_templated_scenarios:
+                    logging.warning("Skipping % for %s", test.name, rule_id)
+                    continue
+                template = ssg.templates.Template.load_template(str(templates_root.absolute()),
+                                                                template_name)
+                rendered_rule_obj["template"]["vars"]["_rule_id"] = rule_id
+                template_parameters = template.preprocess(
+                    rendered_rule_obj["template"]["vars"], "test")
+                env_yaml = env_yaml.copy()
+                jinja_dict = ssg.utils.merge_dicts(env_yaml, template_parameters)
+                file_contents = ssg.jinja.process_file_with_macros(str(test.absolute()),
+                                                                   jinja_dict)
+                scenario = tests.ssg_test_suite.rule.Scenario(test.name, file_contents)
+                if scenario.matches_platform(benchmark_cpes):
+                    rule_output_path.mkdir(parents=True, exist_ok=True)
+                    test_output_path = rule_output_path / test.name
+                    _write_path(file_contents, test_output_path)
+                    logger.debug("Wrote scenario %s for rule %s", test.name, rule_id)
+                else:
+                    logger.debug("Skipping scenario %s for rule %s has it not applicable",
+                                 test.name, rule_id)
+
+
+def _get_benchmark_cpes(env_yaml) -> Set[str]:
+    benchmark_cpes = set()
+    for cpe in env_yaml["cpes"]:
+        for _, data in cpe.items():
+            benchmark_cpes.add(data["name"])
+    return benchmark_cpes
+
+
+def _get_rules_in_profile(built_profiles_root) -> Generator[str, None, None]:
+    for profile_file in built_profiles_root.iterdir():  # type: pathlib.Path
+        if not profile_file.name.endswith(".profile"):
+            continue
+        profile_data = ssg.yaml.open_raw(str(profile_file.absolute()))
+        for selection in profile_data["selections"]:
+            if "=" not in selection:
+                yield selection
+
+
+def main() -> int:
+    args = _create_arg_parser().parse_args()
+    logging.basicConfig(level=logging.getLevelName(args.log_level))
+    env_yaml = ssg.environment.open_environment(args.build_config_yaml, args.product_yaml)
+    root_path = pathlib.Path(args.root).resolve().absolute()
+    output_path = pathlib.Path(args.output).resolve().absolute()
+    resolved_rules_dir = pathlib.Path(args.resolved_rules_dir)
+    if not resolved_rules_dir.exists() or not resolved_rules_dir.is_dir():
+        logging.error("Unable to find product at %s", str(resolved_rules_dir))
+        logging.error("Is the product built?")
+        return 1
+
+    output_path.mkdir(parents=True, exist_ok=True)
+    _copy_and_process_shared(env_yaml, output_path, root_path)
+    benchmark_cpes = _get_benchmark_cpes(env_yaml)
+
+    built_profiles_root = resolved_rules_dir.parent / "profiles"
+    rules_in_profiles = list(_get_rules_in_profile(built_profiles_root))
+
+    templates_root = root_path / "shared" / "templates"
+    all_resolved_rules = list(resolved_rules_dir.iterdir())
+    processes = list()
+    for chunk in range(args.jobs):
+        process_args = (benchmark_cpes, env_yaml, output_path,
+                        all_resolved_rules[chunk::args.jobs], templates_root, rules_in_profiles,)
path: build-scripts/build_tests.py
user: login: jan-cerny
body: I would split the rules_in_profiles into chunks instead of all_resolved_rules because the all_resolved_rules contain rules that aren't going to be processed because they are not present in the built content so the subprocesses won't be load balanced. If you split rules_in_profiles to chunks each subprocess would get the same amout of rules to be processed, which would be better load balancing.
created_at: 2025-03-07T09:19:44Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,230 @@
+#!/usr/bin/env python3
+
+import argparse
+import logging
+import os
+import pathlib
+import sys
+from typing import TypeVar, Generator, Iterable, Set, Dict
+import multiprocessing
+
+import ssg.environment
+import ssg.jinja
+import ssg.utils
+import ssg.yaml
+import ssg.templates
+import tests.ssg_test_suite.common
+import tests.ssg_test_suite.rule
+
+SSG_ROOT = str(pathlib.Path(__file__).resolve().parent.parent.absolute())
+JOB_COUNT = multiprocessing.cpu_count()
+T = TypeVar("T")
+
+
+def _create_arg_parser() -> argparse.ArgumentParser:
+    parser = argparse.ArgumentParser(
+        description="Converts built content tests to be rendered."
+    )
+    parser.add_argument(
+        "--build-config-yaml", required=True,
+        help="YAML file with information about the build configuration. "
+             "e.g.: ~/scap-security-guide/build/build_config.yml"
+    )
+    parser.add_argument(
+        "--product-yaml", required=True,
+        help="YAML file with information about the product we are building. "
+             "e.g.: ~/scap-security-guide/rhel10/product.yml"
+    )
+    parser.add_argument(
+        "--output", required=True,
+        help="Output path"
+             "e.g.:  ~/scap-security-guide/build/rhel10/tests"
+    )
+    parser.add_argument(
+        "--resolved-rules-dir", required=True,
+        help="Directory with <rule-id>.yml resolved rule YAMLs"
+             "e.g.: ~/scap-security-guide/build/rhel10/rules"
+    )
+    parser.add_argument("--log-level", action="store", type=str, default="ERROR",
+                        choices=["ERROR", "WARNING", "INFO", "DEBUG", "TRACE"],
+                        help="What level to log at. Defaults to ERROR.")
+    parser.add_argument("--root", default=SSG_ROOT,
+                        help=f"Path to the project. Defaults to {SSG_ROOT}")
+    parser.add_argument("--jobs", "-j", type=int, default=JOB_COUNT,
+                        help=f"Number of cores to use. Defaults to {JOB_COUNT} on this system.")
+    return parser
+
+
+def _write_path(file_contents: str, output_path: os.PathLike) -> None:
+    with open(output_path, "w") as file:
+        file.write(file_contents)
+        file.write("\n")
+
+
+def _is_test_file(filename: str) -> bool:
+    return filename.endswith(('.pass.sh', '.fail.sh', '.notapplicable.sh'))
+
+
+def _get_deny_templated_scenarios(test_config_path: pathlib.Path) -> Set[str]:
+    if test_config_path.exists():
+        test_config = ssg.yaml.open_raw(str(test_config_path.absolute()))
+        deny_templated_scenarios = test_config.get('deny_templated_scenarios', set())
+        return deny_templated_scenarios
+    return set()
+
+
+def _process_shared_file(env_yaml: dict, file: pathlib.Path, shared_output_path: pathlib.Path) \
+        -> None:
+    file_contents = ssg.jinja.process_file_with_macros(str(file.absolute()), env_yaml)
+    shared_script_path = shared_output_path / file.name
+    _write_path(file_contents, shared_script_path)
+
+
+def _copy_and_process_shared(env_yaml: dict, output_path: pathlib.Path, root_path: pathlib.Path) \
+        -> None:
+    tests_shared_root = root_path / "tests" / "shared"
+    shared_output_path = output_path / "shared"
+    shared_output_path.mkdir(parents=True, exist_ok=True)
+    for file in tests_shared_root.iterdir():  # type: pathlib.Path
+        # We only support one level deep, this avoids recursive functions
+        if file.is_dir():
+            for sub_file in file.iterdir():
+                shared_output_path_sub = shared_output_path / file.name
+                shared_output_path_sub.mkdir(parents=True, exist_ok=True)
+                _process_shared_file(env_yaml, sub_file, shared_output_path_sub)
+        else:
+            _process_shared_file(env_yaml, file, shared_output_path)
+
+
+def _process_local_tests(benchmark_cpes: Set[str], env_yaml: dict, rule_output_path: pathlib.Path,
+                         rule_tests_root: pathlib.Path) -> None:
+    logger = logging.getLogger()
+    for test in rule_tests_root.iterdir():  # type: pathlib.Path
+        if not test.name.endswith(".sh"):
+            logger.debug("Skipping file %s in rule %s is it doesn't end with .sh.",
+                         test.name, rule_output_path.name)
+            continue
+        if not _is_test_file(test.name):
+            file_contents = ssg.jinja.process_file_with_macros(str(test.absolute()),
+                                                               env_yaml)
+            output_file = rule_output_path / test.name
+            rule_output_path.mkdir(parents=True, exist_ok=True)
+            _write_path(file_contents, output_file)
+        file_contents = test.read_text()
+        scenario = tests.ssg_test_suite.rule.Scenario(test.name, file_contents)
+        if scenario.matches_platform(benchmark_cpes):
+            content = ssg.jinja.process_file_with_macros(str(test.absolute()), env_yaml)
+            rule_output_path.mkdir(parents=True, exist_ok=True)
+            _write_path(content, rule_output_path / test.name)
+
+
+def _process_rules(benchmark_cpes: Set[str], env_yaml: Dict, output_path: pathlib.Path,
+                   rules: Iterable[pathlib.Path], templates_root: pathlib.Path,
+                   product_rules: list) -> None:
+    logger = logging.getLogger()
+    for rule_file in rules:  # type: pathlib.Path
+        rendered_rule_obj = ssg.yaml.open_raw(str(rule_file))
+        rule_path = pathlib.Path(rendered_rule_obj["definition_location"])
+        rule_root = rule_path.parent
+        rule_id = rule_root.name
+        if rule_id not in product_rules:
+            logger.debug("Skipping %s since it is not in the product", rule_id)
+            continue
+        rule_tests_root = rule_root / "tests"
+        rule_output_path = output_path / rule_id
+
+        if rule_tests_root.exists():
+            _process_local_tests(benchmark_cpes, env_yaml, rule_output_path, rule_tests_root)
+        if rendered_rule_obj["template"] is not None:
+            if "name" not in rendered_rule_obj["template"]:
+                raise ValueError(f"Invalid template config on rule {rule_id}")
+            template_name = rendered_rule_obj["template"]["name"]
+            template_root = templates_root / template_name
+            template_tests_root = template_root / "tests"
+            if not template_tests_root.exists():
+                logger.debug("Template %s doesn't have tests. Skipping for rule %s.",
+                             template_name, rule_id)
+                continue
+            test_config_path = rule_tests_root / "test_config.yml"
+            deny_templated_scenarios = _get_deny_templated_scenarios(test_config_path)
+            for test in template_tests_root.iterdir():  # type: pathlib.Path
+                if not test.name.endswith(".sh") or test.name in deny_templated_scenarios:
+                    logging.warning("Skipping % for %s", test.name, rule_id)
+                    continue
+                template = ssg.templates.Template.load_template(str(templates_root.absolute()),
+                                                                template_name)
+                rendered_rule_obj["template"]["vars"]["_rule_id"] = rule_id
+                template_parameters = template.preprocess(
+                    rendered_rule_obj["template"]["vars"], "test")
+                env_yaml = env_yaml.copy()
+                jinja_dict = ssg.utils.merge_dicts(env_yaml, template_parameters)
+                file_contents = ssg.jinja.process_file_with_macros(str(test.absolute()),
+                                                                   jinja_dict)
+                scenario = tests.ssg_test_suite.rule.Scenario(test.name, file_contents)
+                if scenario.matches_platform(benchmark_cpes):
path: build-scripts/build_tests.py
user: login: jan-cerny
body: This is very inefficient.

The method "matches_platform" loads all CPEs for all platforms. It calls `common.matches_platform` which calls `common._get_platform_cpes` which iterates over all matched products and loads all CPEs for these products.

Therefore for many test scenarios we load many CPEs, including CPEs for other products and we repeat the loading for each test scenario. The result is that this script spends most of the time in loading CPEs.

We need to create a different and simplified way for test scenarios platform matching here.
created_at: 2025-03-07T09:49:44Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,230 @@
+#!/usr/bin/env python3
+
+import argparse
+import logging
+import os
+import pathlib
+import sys
+from typing import TypeVar, Generator, Iterable, Set, Dict
+import multiprocessing
+
+import ssg.environment
+import ssg.jinja
+import ssg.utils
+import ssg.yaml
+import ssg.templates
+import tests.ssg_test_suite.common
+import tests.ssg_test_suite.rule
+
+SSG_ROOT = str(pathlib.Path(__file__).resolve().parent.parent.absolute())
+JOB_COUNT = multiprocessing.cpu_count()
+T = TypeVar("T")
+
+
+def _create_arg_parser() -> argparse.ArgumentParser:
+    parser = argparse.ArgumentParser(
+        description="Converts built content tests to be rendered."
+    )
+    parser.add_argument(
+        "--build-config-yaml", required=True,
+        help="YAML file with information about the build configuration. "
+             "e.g.: ~/scap-security-guide/build/build_config.yml"
+    )
+    parser.add_argument(
+        "--product-yaml", required=True,
+        help="YAML file with information about the product we are building. "
+             "e.g.: ~/scap-security-guide/rhel10/product.yml"
+    )
+    parser.add_argument(
+        "--output", required=True,
+        help="Output path"
+             "e.g.:  ~/scap-security-guide/build/rhel10/tests"
+    )
+    parser.add_argument(
+        "--resolved-rules-dir", required=True,
+        help="Directory with <rule-id>.yml resolved rule YAMLs"
+             "e.g.: ~/scap-security-guide/build/rhel10/rules"
+    )
+    parser.add_argument("--log-level", action="store", type=str, default="ERROR",
+                        choices=["ERROR", "WARNING", "INFO", "DEBUG", "TRACE"],
+                        help="What level to log at. Defaults to ERROR.")
+    parser.add_argument("--root", default=SSG_ROOT,
+                        help=f"Path to the project. Defaults to {SSG_ROOT}")
+    parser.add_argument("--jobs", "-j", type=int, default=JOB_COUNT,
+                        help=f"Number of cores to use. Defaults to {JOB_COUNT} on this system.")
+    return parser
+
+
+def _write_path(file_contents: str, output_path: os.PathLike) -> None:
+    with open(output_path, "w") as file:
+        file.write(file_contents)
+        file.write("\n")
+
+
+def _is_test_file(filename: str) -> bool:
+    return filename.endswith(('.pass.sh', '.fail.sh', '.notapplicable.sh'))
+
+
+def _get_deny_templated_scenarios(test_config_path: pathlib.Path) -> Set[str]:
+    if test_config_path.exists():
+        test_config = ssg.yaml.open_raw(str(test_config_path.absolute()))
+        deny_templated_scenarios = test_config.get('deny_templated_scenarios', set())
+        return deny_templated_scenarios
+    return set()
+
+
+def _process_shared_file(env_yaml: dict, file: pathlib.Path, shared_output_path: pathlib.Path) \
+        -> None:
+    file_contents = ssg.jinja.process_file_with_macros(str(file.absolute()), env_yaml)
+    shared_script_path = shared_output_path / file.name
+    _write_path(file_contents, shared_script_path)
+
+
+def _copy_and_process_shared(env_yaml: dict, output_path: pathlib.Path, root_path: pathlib.Path) \
+        -> None:
+    tests_shared_root = root_path / "tests" / "shared"
+    shared_output_path = output_path / "shared"
+    shared_output_path.mkdir(parents=True, exist_ok=True)
+    for file in tests_shared_root.iterdir():  # type: pathlib.Path
+        # We only support one level deep, this avoids recursive functions
+        if file.is_dir():
+            for sub_file in file.iterdir():
+                shared_output_path_sub = shared_output_path / file.name
+                shared_output_path_sub.mkdir(parents=True, exist_ok=True)
+                _process_shared_file(env_yaml, sub_file, shared_output_path_sub)
+        else:
+            _process_shared_file(env_yaml, file, shared_output_path)
+
+
+def _process_local_tests(benchmark_cpes: Set[str], env_yaml: dict, rule_output_path: pathlib.Path,
+                         rule_tests_root: pathlib.Path) -> None:
+    logger = logging.getLogger()
+    for test in rule_tests_root.iterdir():  # type: pathlib.Path
+        if not test.name.endswith(".sh"):
+            logger.debug("Skipping file %s in rule %s is it doesn't end with .sh.",
+                         test.name, rule_output_path.name)
+            continue
+        if not _is_test_file(test.name):
+            file_contents = ssg.jinja.process_file_with_macros(str(test.absolute()),
+                                                               env_yaml)
+            output_file = rule_output_path / test.name
+            rule_output_path.mkdir(parents=True, exist_ok=True)
+            _write_path(file_contents, output_file)
+        file_contents = test.read_text()
+        scenario = tests.ssg_test_suite.rule.Scenario(test.name, file_contents)
+        if scenario.matches_platform(benchmark_cpes):
path: build-scripts/build_tests.py
user: login: jan-cerny
body: here we also use the problematic method
created_at: 2025-03-07T09:54:15Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,230 @@
+#!/usr/bin/env python3
+
+import argparse
+import logging
+import os
+import pathlib
+import sys
+from typing import TypeVar, Generator, Iterable, Set, Dict
+import multiprocessing
+
+import ssg.environment
+import ssg.jinja
+import ssg.utils
+import ssg.yaml
+import ssg.templates
+import tests.ssg_test_suite.common
+import tests.ssg_test_suite.rule
+
+SSG_ROOT = str(pathlib.Path(__file__).resolve().parent.parent.absolute())
+JOB_COUNT = multiprocessing.cpu_count()
+T = TypeVar("T")
+
+
+def _create_arg_parser() -> argparse.ArgumentParser:
+    parser = argparse.ArgumentParser(
+        description="Converts built content tests to be rendered."
+    )
+    parser.add_argument(
+        "--build-config-yaml", required=True,
+        help="YAML file with information about the build configuration. "
+             "e.g.: ~/scap-security-guide/build/build_config.yml"
+    )
+    parser.add_argument(
+        "--product-yaml", required=True,
+        help="YAML file with information about the product we are building. "
+             "e.g.: ~/scap-security-guide/rhel10/product.yml"
+    )
+    parser.add_argument(
+        "--output", required=True,
+        help="Output path"
+             "e.g.:  ~/scap-security-guide/build/rhel10/tests"
+    )
+    parser.add_argument(
+        "--resolved-rules-dir", required=True,
+        help="Directory with <rule-id>.yml resolved rule YAMLs"
+             "e.g.: ~/scap-security-guide/build/rhel10/rules"
+    )
+    parser.add_argument("--log-level", action="store", type=str, default="ERROR",
+                        choices=["ERROR", "WARNING", "INFO", "DEBUG", "TRACE"],
+                        help="What level to log at. Defaults to ERROR.")
+    parser.add_argument("--root", default=SSG_ROOT,
+                        help=f"Path to the project. Defaults to {SSG_ROOT}")
+    parser.add_argument("--jobs", "-j", type=int, default=JOB_COUNT,
+                        help=f"Number of cores to use. Defaults to {JOB_COUNT} on this system.")
+    return parser
+
+
+def _write_path(file_contents: str, output_path: os.PathLike) -> None:
+    with open(output_path, "w") as file:
+        file.write(file_contents)
+        file.write("\n")
+
+
+def _is_test_file(filename: str) -> bool:
+    return filename.endswith(('.pass.sh', '.fail.sh', '.notapplicable.sh'))
+
+
+def _get_deny_templated_scenarios(test_config_path: pathlib.Path) -> Set[str]:
+    if test_config_path.exists():
+        test_config = ssg.yaml.open_raw(str(test_config_path.absolute()))
+        deny_templated_scenarios = test_config.get('deny_templated_scenarios', set())
+        return deny_templated_scenarios
+    return set()
+
+
+def _process_shared_file(env_yaml: dict, file: pathlib.Path, shared_output_path: pathlib.Path) \
+        -> None:
+    file_contents = ssg.jinja.process_file_with_macros(str(file.absolute()), env_yaml)
+    shared_script_path = shared_output_path / file.name
+    _write_path(file_contents, shared_script_path)
+
+
+def _copy_and_process_shared(env_yaml: dict, output_path: pathlib.Path, root_path: pathlib.Path) \
+        -> None:
+    tests_shared_root = root_path / "tests" / "shared"
+    shared_output_path = output_path / "shared"
+    shared_output_path.mkdir(parents=True, exist_ok=True)
+    for file in tests_shared_root.iterdir():  # type: pathlib.Path
+        # We only support one level deep, this avoids recursive functions
+        if file.is_dir():
+            for sub_file in file.iterdir():
+                shared_output_path_sub = shared_output_path / file.name
+                shared_output_path_sub.mkdir(parents=True, exist_ok=True)
+                _process_shared_file(env_yaml, sub_file, shared_output_path_sub)
+        else:
+            _process_shared_file(env_yaml, file, shared_output_path)
+
+
+def _process_local_tests(benchmark_cpes: Set[str], env_yaml: dict, rule_output_path: pathlib.Path,
+                         rule_tests_root: pathlib.Path) -> None:
+    logger = logging.getLogger()
+    for test in rule_tests_root.iterdir():  # type: pathlib.Path
+        if not test.name.endswith(".sh"):
+            logger.debug("Skipping file %s in rule %s is it doesn't end with .sh.",
path: build-scripts/build_tests.py
user: login: jan-cerny
body: This will skip files like for example `linux_os/guide/auditing/auditd_configure_rules/audit_file_modification/audit_rules_unsuccessful_file_modification/tests/test_audit.rules` which are needed for the test scenarios to work.
created_at: 2025-03-07T09:58:19Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,230 @@
+#!/usr/bin/env python3
path: build-scripts/build_tests.py
user: login: Mab879
body: I have updated it.
created_at: 2025-03-07T13:30:33Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,244 @@
+#!/usr/bin/env python3
+
+import argparse
+import logging
+import os
+import pathlib
+import sys
+from typing import TypeVar, Generator, Set, Dict
+import multiprocessing
+
+import ssg.constants
+import ssg.environment
+import ssg.jinja
+import ssg.utils
+import ssg.yaml
+import ssg.templates
+
+SSG_ROOT = str(pathlib.Path(__file__).resolve().parent.parent.absolute())
+JOB_COUNT = multiprocessing.cpu_count()
+T = TypeVar("T")
+
+
+def _create_arg_parser() -> argparse.ArgumentParser:
+    parser = argparse.ArgumentParser(
+        description="Converts built content tests to be rendered."
+    )
+    parser.add_argument(
+        "--build-config-yaml", required=True,
+        help="YAML file with information about the build configuration. "
+             "e.g.: ~/scap-security-guide/build/build_config.yml"
+    )
+    parser.add_argument(
+        "--product-yaml", required=True,
+        help="YAML file with information about the product we are building. "
+             "e.g.: ~/scap-security-guide/rhel10/product.yml"
+    )
+    parser.add_argument(
+        "--output", required=True,
+        help="Output path"
+             "e.g.:  ~/scap-security-guide/build/rhel10/tests"
+    )
+    parser.add_argument(
+        "--resolved-rules-dir", required=True,
+        help="Directory with <rule-id>.yml resolved rule YAMLs"
+             "e.g.: ~/scap-security-guide/build/rhel10/rules"
+    )
+    parser.add_argument("--log-level", action="store", type=str, default="ERROR",
+                        choices=["ERROR", "WARNING", "INFO", "DEBUG", "TRACE"],
+                        help="What level to log at. Defaults to ERROR.")
+    parser.add_argument("--root", default=SSG_ROOT,
+                        help=f"Path to the project. Defaults to {SSG_ROOT}")
+    parser.add_argument("--jobs", "-j", type=int, default=JOB_COUNT,
+                        help=f"Number of cores to use. Defaults to {JOB_COUNT} on this system.")
+    return parser
+
+
+def _write_path(file_contents: str, output_path: os.PathLike) -> None:
+    with open(output_path, "w") as file:
+        file.write(file_contents)
+        file.write("\n")
+
+
+def _is_test_file(filename: str) -> bool:
+    return filename.endswith(('.pass.sh', '.fail.sh', '.notapplicable.sh'))
+
+
+def _get_deny_templated_scenarios(test_config_path: pathlib.Path) -> Set[str]:
+    if test_config_path.exists():
+        test_config = ssg.yaml.open_raw(str(test_config_path.absolute()))
+        deny_templated_scenarios = test_config.get('deny_templated_scenarios', set())
+        return deny_templated_scenarios
+    return set()
+
+
+def _process_shared_file(env_yaml: dict, file: pathlib.Path, shared_output_path: pathlib.Path) \
+        -> None:
+    file_contents = ssg.jinja.process_file_with_macros(str(file.absolute()), env_yaml)
+    shared_script_path = shared_output_path / file.name
+    _write_path(file_contents, shared_script_path)
+
+
+def _copy_and_process_shared(env_yaml: dict, output_path: pathlib.Path, root_path: pathlib.Path) \
+        -> None:
+    tests_shared_root = root_path / "tests" / "shared"
+    shared_output_path = output_path / "shared"
+    shared_output_path.mkdir(parents=True, exist_ok=True)
+    for file in tests_shared_root.iterdir():  # type: pathlib.Path
+        # We only support one level deep, this avoids recursive functions
+        if file.is_dir():
+            for sub_file in file.iterdir():
+                shared_output_path_sub = shared_output_path / file.name
+                shared_output_path_sub.mkdir(parents=True, exist_ok=True)
+                _process_shared_file(env_yaml, sub_file, shared_output_path_sub)
+        else:
+            _process_shared_file(env_yaml, file, shared_output_path)
+
+
+def _get_platform_from_file_contents(file_contents: str) -> str:
+    # Some tests don't have an explict platform assume always applicable
+    platform = "multi_platform_all"
+    for line in file_contents.split("\n"):
+        if not line.startswith('#'):
+            # The loop is now in the main test content, stop processing the file.
+            break
path: build-scripts/build_tests.py
user: login: jan-cerny
body: The test scenario `linux_os/guide/system/accounts/accounts-session/accounts_tmout/tests/multiline_profile_d.fail.sh` which is marked as `# platform = multi_platform_sle` gets build when I build RHEL 9 content. It shouldn't be there, it's for SUSE. The reason seems to be that this test scenario has empty line before the `# platform = multi_platform_sle` line and therefore it's skipped by this code here.
created_at: 2025-03-10T12:42:50Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,244 @@
+#!/usr/bin/env python3
+
+import argparse
+import logging
+import os
+import pathlib
+import sys
+from typing import TypeVar, Generator, Set, Dict
+import multiprocessing
+
+import ssg.constants
+import ssg.environment
+import ssg.jinja
+import ssg.utils
+import ssg.yaml
+import ssg.templates
+
+SSG_ROOT = str(pathlib.Path(__file__).resolve().parent.parent.absolute())
+JOB_COUNT = multiprocessing.cpu_count()
+T = TypeVar("T")
+
path: build-scripts/build_tests.py
user: login: jan-cerny
body: Now it's really fast! It takes me less than 3 seconds on my laptop to build RHEL 9 tests. Great improvement!
created_at: 2025-03-10T13:08:56Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,244 @@
+#!/usr/bin/env python3
+
+import argparse
+import logging
+import os
+import pathlib
+import sys
+from typing import TypeVar, Generator, Set, Dict
+import multiprocessing
+
+import ssg.constants
+import ssg.environment
+import ssg.jinja
+import ssg.utils
+import ssg.yaml
+import ssg.templates
+
+SSG_ROOT = str(pathlib.Path(__file__).resolve().parent.parent.absolute())
+JOB_COUNT = multiprocessing.cpu_count()
+T = TypeVar("T")
+
+
+def _create_arg_parser() -> argparse.ArgumentParser:
+    parser = argparse.ArgumentParser(
+        description="Converts built content tests to be rendered."
+    )
+    parser.add_argument(
+        "--build-config-yaml", required=True,
+        help="YAML file with information about the build configuration. "
+             "e.g.: ~/scap-security-guide/build/build_config.yml"
+    )
+    parser.add_argument(
+        "--product-yaml", required=True,
+        help="YAML file with information about the product we are building. "
+             "e.g.: ~/scap-security-guide/rhel10/product.yml"
+    )
+    parser.add_argument(
+        "--output", required=True,
+        help="Output path"
+             "e.g.:  ~/scap-security-guide/build/rhel10/tests"
+    )
+    parser.add_argument(
+        "--resolved-rules-dir", required=True,
+        help="Directory with <rule-id>.yml resolved rule YAMLs"
path: build-scripts/build_tests.py
user: login: jan-cerny
body: missing space after YAMLs
created_at: 2025-03-10T13:11:23Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,244 @@
+#!/usr/bin/env python3
+
+import argparse
+import logging
+import os
+import pathlib
+import sys
+from typing import TypeVar, Generator, Set, Dict
+import multiprocessing
+
+import ssg.constants
+import ssg.environment
+import ssg.jinja
+import ssg.utils
+import ssg.yaml
+import ssg.templates
+
+SSG_ROOT = str(pathlib.Path(__file__).resolve().parent.parent.absolute())
+JOB_COUNT = multiprocessing.cpu_count()
+T = TypeVar("T")
+
+
+def _create_arg_parser() -> argparse.ArgumentParser:
+    parser = argparse.ArgumentParser(
+        description="Converts built content tests to be rendered."
+    )
+    parser.add_argument(
+        "--build-config-yaml", required=True,
+        help="YAML file with information about the build configuration. "
+             "e.g.: ~/scap-security-guide/build/build_config.yml"
+    )
+    parser.add_argument(
+        "--product-yaml", required=True,
+        help="YAML file with information about the product we are building. "
+             "e.g.: ~/scap-security-guide/rhel10/product.yml"
+    )
+    parser.add_argument(
+        "--output", required=True,
+        help="Output path"
+             "e.g.:  ~/scap-security-guide/build/rhel10/tests"
+    )
+    parser.add_argument(
+        "--resolved-rules-dir", required=True,
+        help="Directory with <rule-id>.yml resolved rule YAMLs"
+             "e.g.: ~/scap-security-guide/build/rhel10/rules"
+    )
+    parser.add_argument("--log-level", action="store", type=str, default="ERROR",
+                        choices=["ERROR", "WARNING", "INFO", "DEBUG", "TRACE"],
+                        help="What level to log at. Defaults to ERROR.")
+    parser.add_argument("--root", default=SSG_ROOT,
+                        help=f"Path to the project. Defaults to {SSG_ROOT}")
+    parser.add_argument("--jobs", "-j", type=int, default=JOB_COUNT,
+                        help=f"Number of cores to use. Defaults to {JOB_COUNT} on this system.")
+    return parser
+
+
+def _write_path(file_contents: str, output_path: os.PathLike) -> None:
+    with open(output_path, "w") as file:
+        file.write(file_contents)
+        file.write("\n")
+
+
+def _is_test_file(filename: str) -> bool:
+    return filename.endswith(('.pass.sh', '.fail.sh', '.notapplicable.sh'))
+
+
+def _get_deny_templated_scenarios(test_config_path: pathlib.Path) -> Set[str]:
+    if test_config_path.exists():
+        test_config = ssg.yaml.open_raw(str(test_config_path.absolute()))
+        deny_templated_scenarios = test_config.get('deny_templated_scenarios', set())
+        return deny_templated_scenarios
+    return set()
+
+
+def _process_shared_file(env_yaml: dict, file: pathlib.Path, shared_output_path: pathlib.Path) \
+        -> None:
+    file_contents = ssg.jinja.process_file_with_macros(str(file.absolute()), env_yaml)
+    shared_script_path = shared_output_path / file.name
+    _write_path(file_contents, shared_script_path)
+
+
+def _copy_and_process_shared(env_yaml: dict, output_path: pathlib.Path, root_path: pathlib.Path) \
+        -> None:
+    tests_shared_root = root_path / "tests" / "shared"
+    shared_output_path = output_path / "shared"
+    shared_output_path.mkdir(parents=True, exist_ok=True)
+    for file in tests_shared_root.iterdir():  # type: pathlib.Path
+        # We only support one level deep, this avoids recursive functions
+        if file.is_dir():
+            for sub_file in file.iterdir():
+                shared_output_path_sub = shared_output_path / file.name
+                shared_output_path_sub.mkdir(parents=True, exist_ok=True)
+                _process_shared_file(env_yaml, sub_file, shared_output_path_sub)
+        else:
+            _process_shared_file(env_yaml, file, shared_output_path)
+
+
+def _get_platform_from_file_contents(file_contents: str) -> str:
+    # Some tests don't have an explict platform assume always applicable
+    platform = "multi_platform_all"
+    for line in file_contents.split("\n"):
+        if not line.startswith('#'):
+            # The loop is now in the main test content, stop processing the file.
+            break
+        if line.startswith('# platform'):
+            platform_parts = line.split('=')
+            if len(platform_parts) == 2:
+                platform = platform_parts[1]
+                break
+    return platform.strip()
+
+
+def _process_local_tests(product: str, env_yaml: dict, rule_output_path: pathlib.Path,
+                         rule_tests_root: pathlib.Path) -> None:
+    logger = logging.getLogger()
+    for test in rule_tests_root.iterdir():  # type: pathlib.Path
+        if test.is_dir():
+            logger.warning("Skipping directory %s in rule %s", test.name,
+                           rule_output_path.name)
+            continue
+        if not _is_test_file(test.name):
+            file_contents = ssg.jinja.process_file_with_macros(str(test.absolute()),
+                                                               env_yaml)
+            output_file = rule_output_path / test.name
+            rule_output_path.mkdir(parents=True, exist_ok=True)
+            _write_path(file_contents, output_file)
+        file_contents = test.read_text()
+        platform = _get_platform_from_file_contents(file_contents)
+        if ssg.utils.is_applicable_for_product(platform, product):
+            content = ssg.jinja.process_file_with_macros(str(test.absolute()), env_yaml)
+            rule_output_path.mkdir(parents=True, exist_ok=True)
+            _write_path(content, rule_output_path / test.name)
+
+
+def _process_rules(env_yaml: Dict, output_path: pathlib.Path,
path: build-scripts/build_tests.py
user: login: jan-cerny
body: Code Climate complains about code complexity of this function. Please split it to multiple functions. You can for example extract code related to templated tests to a new function _process_templated_tests which would be consistent with _process_local_tests.
created_at: 2025-03-10T13:15:26Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,244 @@
+#!/usr/bin/env python3
+
+import argparse
+import logging
+import os
+import pathlib
+import sys
+from typing import TypeVar, Generator, Set, Dict
+import multiprocessing
+
+import ssg.constants
+import ssg.environment
+import ssg.jinja
+import ssg.utils
+import ssg.yaml
+import ssg.templates
+
+SSG_ROOT = str(pathlib.Path(__file__).resolve().parent.parent.absolute())
+JOB_COUNT = multiprocessing.cpu_count()
+T = TypeVar("T")
+
+
+def _create_arg_parser() -> argparse.ArgumentParser:
+    parser = argparse.ArgumentParser(
+        description="Converts built content tests to be rendered."
+    )
+    parser.add_argument(
+        "--build-config-yaml", required=True,
+        help="YAML file with information about the build configuration. "
+             "e.g.: ~/scap-security-guide/build/build_config.yml"
+    )
+    parser.add_argument(
+        "--product-yaml", required=True,
+        help="YAML file with information about the product we are building. "
+             "e.g.: ~/scap-security-guide/rhel10/product.yml"
+    )
+    parser.add_argument(
+        "--output", required=True,
+        help="Output path"
+             "e.g.:  ~/scap-security-guide/build/rhel10/tests"
+    )
+    parser.add_argument(
+        "--resolved-rules-dir", required=True,
+        help="Directory with <rule-id>.yml resolved rule YAMLs"
+             "e.g.: ~/scap-security-guide/build/rhel10/rules"
+    )
+    parser.add_argument("--log-level", action="store", type=str, default="ERROR",
+                        choices=["ERROR", "WARNING", "INFO", "DEBUG", "TRACE"],
+                        help="What level to log at. Defaults to ERROR.")
+    parser.add_argument("--root", default=SSG_ROOT,
+                        help=f"Path to the project. Defaults to {SSG_ROOT}")
+    parser.add_argument("--jobs", "-j", type=int, default=JOB_COUNT,
+                        help=f"Number of cores to use. Defaults to {JOB_COUNT} on this system.")
+    return parser
+
+
+def _write_path(file_contents: str, output_path: os.PathLike) -> None:
+    with open(output_path, "w") as file:
+        file.write(file_contents)
+        file.write("\n")
+
+
+def _is_test_file(filename: str) -> bool:
+    return filename.endswith(('.pass.sh', '.fail.sh', '.notapplicable.sh'))
+
+
+def _get_deny_templated_scenarios(test_config_path: pathlib.Path) -> Set[str]:
+    if test_config_path.exists():
+        test_config = ssg.yaml.open_raw(str(test_config_path.absolute()))
+        deny_templated_scenarios = test_config.get('deny_templated_scenarios', set())
+        return deny_templated_scenarios
+    return set()
+
+
+def _process_shared_file(env_yaml: dict, file: pathlib.Path, shared_output_path: pathlib.Path) \
+        -> None:
+    file_contents = ssg.jinja.process_file_with_macros(str(file.absolute()), env_yaml)
+    shared_script_path = shared_output_path / file.name
+    _write_path(file_contents, shared_script_path)
+
+
+def _copy_and_process_shared(env_yaml: dict, output_path: pathlib.Path, root_path: pathlib.Path) \
+        -> None:
+    tests_shared_root = root_path / "tests" / "shared"
+    shared_output_path = output_path / "shared"
+    shared_output_path.mkdir(parents=True, exist_ok=True)
+    for file in tests_shared_root.iterdir():  # type: pathlib.Path
+        # We only support one level deep, this avoids recursive functions
+        if file.is_dir():
+            for sub_file in file.iterdir():
+                shared_output_path_sub = shared_output_path / file.name
+                shared_output_path_sub.mkdir(parents=True, exist_ok=True)
+                _process_shared_file(env_yaml, sub_file, shared_output_path_sub)
+        else:
+            _process_shared_file(env_yaml, file, shared_output_path)
+
+
+def _get_platform_from_file_contents(file_contents: str) -> str:
+    # Some tests don't have an explict platform assume always applicable
+    platform = "multi_platform_all"
+    for line in file_contents.split("\n"):
+        if not line.startswith('#'):
+            # The loop is now in the main test content, stop processing the file.
+            break
+        if line.startswith('# platform'):
+            platform_parts = line.split('=')
+            if len(platform_parts) == 2:
+                platform = platform_parts[1]
+                break
+    return platform.strip()
+
+
+def _process_local_tests(product: str, env_yaml: dict, rule_output_path: pathlib.Path,
+                         rule_tests_root: pathlib.Path) -> None:
+    logger = logging.getLogger()
+    for test in rule_tests_root.iterdir():  # type: pathlib.Path
+        if test.is_dir():
+            logger.warning("Skipping directory %s in rule %s", test.name,
+                           rule_output_path.name)
+            continue
+        if not _is_test_file(test.name):
+            file_contents = ssg.jinja.process_file_with_macros(str(test.absolute()),
+                                                               env_yaml)
+            output_file = rule_output_path / test.name
+            rule_output_path.mkdir(parents=True, exist_ok=True)
+            _write_path(file_contents, output_file)
+        file_contents = test.read_text()
+        platform = _get_platform_from_file_contents(file_contents)
+        if ssg.utils.is_applicable_for_product(platform, product):
+            content = ssg.jinja.process_file_with_macros(str(test.absolute()), env_yaml)
+            rule_output_path.mkdir(parents=True, exist_ok=True)
+            _write_path(content, rule_output_path / test.name)
+
+
+def _process_rules(env_yaml: Dict, output_path: pathlib.Path,
+                   templates_root: pathlib.Path, product_rules: list,
+                   resolved_root: pathlib.Path) -> None:
+    logger = logging.getLogger()
+    product = resolved_root.parent.name
+    for rule_id in product_rules:
+        rule_file = resolved_root / f'{rule_id}.yml'
+
+        rendered_rule_obj = ssg.yaml.open_raw(str(rule_file))
+        rule_path = pathlib.Path(rendered_rule_obj["definition_location"])
+        rule_root = rule_path.parent
+
+        rule_tests_root = rule_root / "tests"
+        rule_output_path = output_path / rule_id
+
+        if rule_tests_root.exists():
+            _process_local_tests(product, env_yaml, rule_output_path, rule_tests_root)
+        if rendered_rule_obj["template"] is not None:
+            if "name" not in rendered_rule_obj["template"]:
+                raise ValueError(f"Invalid template config on rule {rule_id}")
+            template_name = rendered_rule_obj["template"]["name"]
+            template_root = templates_root / template_name
+            template_tests_root = template_root / "tests"
+            if not template_tests_root.exists():
+                logger.debug("Template %s doesn't have tests. Skipping for rule %s.",
+                             template_name, rule_id)
+                continue
+            test_config_path = rule_tests_root / "test_config.yml"
+            deny_templated_scenarios = _get_deny_templated_scenarios(test_config_path)
+            for test in template_tests_root.iterdir():  # type: pathlib.Path
+                if not test.name.endswith(".sh") or test.name in deny_templated_scenarios:
+                    logging.warning("Skipping %s for %s as it is a denied test scenario",
path: build-scripts/build_tests.py
user: login: jan-cerny
body: logger
created_at: 2025-03-10T13:26:11Z
author_association: COLLABORATOR
diff_hunk: @@ -671,6 +671,22 @@ macro(ssg_build_disa_delta PRODUCT PROFILE)
             DESTINATION ${SSG_TAILORING_INSTALL_DIR})
 endmacro()
 
+macro(ssg_build_tests PRODUCT)
+    add_custom_command(
+        OUTPUT "${CMAKE_BINARY_DIR}/${PRODUCT}/tests/.tests_done"
+        COMMAND env "PYTHONPATH=$ENV{PYTHONPATH}" "${PYTHON_EXECUTABLE}" "${CMAKE_SOURCE_DIR}/build-scripts/build_tests.py"  --build-config-yaml "${CMAKE_BINARY_DIR}/build_config.yml" --resolved-rules-dir "${CMAKE_CURRENT_BINARY_DIR}/rules" --output  "${CMAKE_CURRENT_BINARY_DIR}/tests" --product-yaml "${CMAKE_SOURCE_DIR}/products/${PRODUCT}/product.yml"
path: cmake/SSGCommon.cmake
user: login: jan-cerny
body: works fine for me :+1: 
created_at: 2025-03-10T13:36:47Z
author_association: COLLABORATOR

PR comments:
user: login: openshift-ci[bot]
created_at: 2025-02-12T00:00:37Z
body: Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with `/test all`
author_association: NONE
user: login: Mab879
created_at: 2025-02-14T20:59:27Z
body: Things left to do:
1. Check on Automatus runs. They seem to be broken. 
Commands like `./automatus.py rule --datastream ../build/ssg-rhel10-ds.xml --libvirt qemu:///system automatus_rhel10 root_permissions_syslibrary_files` are failing.

1. I need to doc how to turn it on.
2.  Possible turn down the jobs number when building many products
3. Verify the correct tests are being ran
author_association: MEMBER
user: login: qlty-cloud-legacy[bot]
created_at: 2025-03-10T21:42:54Z
body: Code Climate has analyzed commit fd08c5a3 and detected **3 issues** on this pull request.

Here's the issue category breakdown:

| Category | Count |
|:---:|:---:|
| Complexity | 3 |


The test coverage on the diff in this pull request is **100.0%** (**50%** is the threshold).

This pull request will bring the total coverage in the repository to **62.0%** (**0.0%** change).

View more on [Code Climate](https://codeclimate.com/github/ComplianceAsCode/content/pull/13029).

author_association: NONE
user: login: comps
created_at: 2025-03-11T19:24:31Z
body: Looks good to me, builds (at least on my RHEL-9) as intended.

I don't see any empty rule directories anymore (so I'm assuming rules without tests are not present inside `build/rhel9/tests/`), non-`sh` files are also included, no `{{` or `}}` found via greps, it all looks legit.


I already used it to find quite a lot of "broken" tests, `.sh` scripts that don't have a shebang at the top (presumably Automatus calls them via `bash script.pass.sh` explicitly).

```
$ cd build/rhel9/tests/
$ (IFS=$'\n'; for f in $(find . -type f -name '*.sh'); do h=$(head -n1 "$f"); [[ $h != '#!/bin/bash' ]] && echo $f; done) | wc -l
266
```
author_association: COLLABORATOR