Repository: oscal-compass/compliance-trestle (python)

Issue:
Add Implementation part headers by default for main component in SSP Markdown
## Issue description / feature objectives

Implementation part headers (e.g `### Implementation part a.`) appear in the SSP Markdown for components when rules are attached. There may be use cases when a high level response is needed at the statement level for the overall system with no rules attached. The main component `This System` is an example of this.

## Caveats / Assumptions

- Assumption - `This System` would be used to provide overall system responses and context so the user would need to be able to provide that at the statement level.
- Assumption - `This System` would not have rules attached

## Completion Criteria

- The `### Implementation part <statement label>` would appear by default for the `This System` component when generating SSP Markdown.


Pull request:
feat: adds implementation parts to This System component in markdown
## Types of changes

<!--- What types of changes does your code introduce? Put an `x` in all the boxes that apply: -->

- [ ] Hot fix (emergency fix and release)
- [ ] Bug fix (non-breaking change which fixes an issue)
- [X] New feature (non-breaking change which adds functionality)
- [x] Documentation (change which affects the documentation site)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
- [ ] Release (`develop` -> `main`)

## Quality assurance (all should be covered).

- [X] My code follows the code style of this project.
- [x] Documentation for my change is up to date?
- [X] My PR meets testing requirements.
- [X] All new and existing tests passed.
- [X] All commits are signed-off.

## Summary

Alters SSP markdown to write the heading for the implementation part if it exists in the control for This System

> Note: This has been tested with the component markdown to ensure the original behavior is not altered and parts are only written when rules are attached

The default behavior when writing SSP is not changed. It can be altered by using the `include-all-parts` flag.
The `--include-all-parts` flag controls `This System` is written in the markdown. When `--include-all-parts` is set, all control parts will be written with the main component present under the `## Implementation for part <part>` section. All other components will still be conditionally added based on the presence of rules.

Closes #1527 

## Key links:

- [Sonar coverage](https://sonarcloud.io/dashboard?id=compliance-trestle)

# Before you merge

- Ensure it is a 'squash commit' if not a release.
- Ensure CI is currently passing
- Check sonar. If you are working for a fork a maintainer will reach out, if required.


Patch:
diff --git a/docs/tutorials/ssp_profile_catalog_authoring/ssp_profile_catalog_authoring.md b/docs/tutorials/ssp_profile_catalog_authoring/ssp_profile_catalog_authoring.md
index 5679e173..9a1d2acb 100644
--- a/docs/tutorials/ssp_profile_catalog_authoring/ssp_profile_catalog_authoring.md
+++ b/docs/tutorials/ssp_profile_catalog_authoring/ssp_profile_catalog_authoring.md
@@ -1049,6 +1049,8 @@ In addition, this is the only control markdown where the moustache (`{{}}`) item
 
 The markdown can have guidance per-component in the control, as shown by the line, `### ACME Component`.  Any prose directly under a `##` implementation section will apply to the overall system component, but sections in a sub-header of the form `###` will only apply to that particular component.
 
+The `--include-all-parts` flag controls how the main system component (`This System`) is written in the markdown. By default, control parts are not written out unless a component has a rule associated with that particular control part or `statement`. When `--include-all-parts` is set, all control parts will be written with the main component present under the `## Implementation for part <part>` section. All other components will still be conditionally added based on the presence of rules.
+
 After generating the markdown for the resolved profile catalog you may then edit the files and provide text in the sections with `Add control implementation...` in them.  But do not remove the horizontal rule
 lines or modify/remove the lines with `### ` in them, corresponding to system components.
 
diff --git a/trestle/common/const.py b/trestle/common/const.py
index 381aff39..48f1ecd4 100644
--- a/trestle/common/const.py
+++ b/trestle/common/const.py
@@ -328,6 +328,11 @@ HELP_MARKDOWN_NAME = 'Name of the output generated profile markdown folder'
 
 HELP_COMPDEFS = 'Comma-separated list of component-definitions for the ssp.'
 
+HELP_INCLUDE_ALL_PARTS = (
+    'Include all control parts when generating SSP markdown.'
+    + ' By default, only controls and parts with an attached rule will be written out.'
+)
+
 FILTER_BY_PROFILE = 'filter-by-profile'
 
 FILTER_BY_COMPONENTS = 'filter-by-components'
diff --git a/trestle/core/catalog/catalog_reader.py b/trestle/core/catalog/catalog_reader.py
index f887ea19..4ed76d13 100644
--- a/trestle/core/catalog/catalog_reader.py
+++ b/trestle/core/catalog/catalog_reader.py
@@ -201,30 +201,45 @@ class CatalogReader():
                 ssp.control_implementation.implemented_requirements
             )
             ssp.control_implementation.implemented_requirements.append(control_imp_req)
+        CatalogReader._add_statement_to_impl_requirement(control_imp_req, statement_id)
+        return control_imp_req
+
+    @staticmethod
+    def _add_statement_to_impl_requirement(impl_req: ossp.ImplementedRequirement, statement_id: str) -> ossp.Statement:
+        """Update the implemented requirement with a new statement."""
         statement = gens.generate_sample_model(ossp.Statement)
         statement.statement_id = statement_id
         statement.by_components = None
-        control_imp_req.statements = as_list(control_imp_req.statements)
-        control_imp_req.statements.append(statement)
-        return control_imp_req
+        impl_req.statements = as_list(impl_req.statements)
+        impl_req.statements.append(statement)
+        return statement
 
     @staticmethod
     def _get_by_comp_from_imp_req(
         imp_req: ossp.ImplementedRequirement, statement_id: str, comp_uuid: str
     ) -> ossp.ByComponent:
         if statement_id:
+            found_statement: Optional[ossp.Statement] = None
             for statement in as_list(imp_req.statements):
                 if statement.statement_id == statement_id:
-                    for by_comp in as_list(statement.by_components):
-                        if by_comp.component_uuid == comp_uuid:
-                            return by_comp
-                    # didnt find bycomp so need to make one
-                    by_comp = gens.generate_sample_model(ossp.ByComponent)
-                    by_comp.component_uuid = comp_uuid
-                    by_comp.implementation_status = com.ImplementationStatus(state=const.STATUS_PLANNED)
-                    statement.by_components = as_list(statement.by_components)
-                    statement.by_components.append(by_comp)
-                    return by_comp
+                    found_statement = statement
+                    break
+
+            # If the statement doesn't exist, create it
+            if not found_statement:
+                found_statement = CatalogReader._add_statement_to_impl_requirement(imp_req, statement_id)
+            else:
+                for by_comp in as_list(found_statement.by_components):
+                    if by_comp.component_uuid == comp_uuid:
+                        return by_comp
+
+            # didnt find bycomp or new statement so need to make one
+            by_comp = gens.generate_sample_model(ossp.ByComponent)
+            by_comp.component_uuid = comp_uuid
+            by_comp.implementation_status = com.ImplementationStatus(state=const.STATUS_PLANNED)
+            found_statement.by_components = as_list(found_statement.by_components)
+            found_statement.by_components.append(by_comp)
+            return by_comp
         else:
             for by_comp in as_list(imp_req.by_components):
                 if by_comp.component_uuid == comp_uuid:
@@ -235,7 +250,6 @@ class CatalogReader():
             imp_req.by_components = as_list(imp_req.by_components)
             imp_req.by_components.append(by_comp)
             return by_comp
-        raise TrestleError(f'Internal error seeking by_comp for component {comp_uuid} and statement {statement_id}')
 
     @staticmethod
     def _read_comp_info_from_md(control_file_path: pathlib.Path,
diff --git a/trestle/core/commands/author/ssp.py b/trestle/core/commands/author/ssp.py
index 065e75c2..e129e53f 100644
--- a/trestle/core/commands/author/ssp.py
+++ b/trestle/core/commands/author/ssp.py
@@ -81,6 +81,14 @@ class SSPGenerate(AuthorCommonCommand):
             action='store_true',
             default=False
         )
+        self.add_argument(
+            '-iap',
+            '--include-all-parts',
+            help=const.HELP_INCLUDE_ALL_PARTS,
+            required=False,
+            action='store_true',
+            default=False
+        )
 
     def _run(self, args: argparse.Namespace) -> int:
         try:
@@ -110,7 +118,8 @@ class SSPGenerate(AuthorCommonCommand):
                 md_path,
                 yaml_header,
                 args.overwrite_header_values,
-                args.force_overwrite
+                args.force_overwrite,
+                args.include_all_parts
             )
 
         except Exception as e:  # pragma: no cover
@@ -125,7 +134,8 @@ class SSPGenerate(AuthorCommonCommand):
         md_path: pathlib.Path,
         yaml_header: Dict[str, Any],
         overwrite_header_values: bool,
-        force_overwrite: bool
+        force_overwrite: bool,
+        include_all_parts: bool
     ) -> int:
         """
         Generate the ssp markdown from the profile and compdefs.
@@ -157,6 +167,7 @@ class SSPGenerate(AuthorCommonCommand):
         context.overwrite_header_values = overwrite_header_values
         context.allowed_sections = None
         context.comp_def_name_list = compdef_name_list
+        context.include_all_parts = include_all_parts
 
         # if file not recognized as URI form, assume it represents name of file in trestle directory
         profile_in_trestle_dir = '://' not in profile_name_or_href
diff --git a/trestle/core/control_context.py b/trestle/core/control_context.py
index 560c6b54..a2ccb977 100644
--- a/trestle/core/control_context.py
+++ b/trestle/core/control_context.py
@@ -44,6 +44,7 @@ class ControlContext:
     trestle_root: pathlib.Path
     md_root: pathlib.Path
     prompt_responses: bool
+    include_all_parts: bool
     overwrite_header_values: bool
     set_parameters_flag: bool
     cli_yaml_header: Optional[Dict[Any, Any]] = None
@@ -71,6 +72,7 @@ class ControlContext:
         trestle_root: pathlib.Path,
         md_root: pathlib.Path,
         prompt_responses: bool = False,
+        include_all_parts: bool = False,
         overwrite_header_values: bool = False,
         set_parameters_flag: bool = False,
         cli_yaml_header: Optional[Dict[Any, Any]] = None,
@@ -97,6 +99,7 @@ class ControlContext:
             trestle_root,
             md_root,
             prompt_responses,
+            include_all_parts,
             overwrite_header_values,
             set_parameters_flag,
             cli_yaml_header=cli_yaml_header,
@@ -132,6 +135,7 @@ class ControlContext:
             context.trestle_root,
             context.md_root,
             context.prompt_responses,
+            context.include_all_parts,
             context.overwrite_header_values,
             context.set_parameters_flag,
             cli_yaml_header=copy.deepcopy(context.cli_yaml_header),
diff --git a/trestle/core/control_writer.py b/trestle/core/control_writer.py
index 8d9fbae3..481e2393 100644
--- a/trestle/core/control_writer.py
+++ b/trestle/core/control_writer.py
@@ -142,6 +142,22 @@ class ControlWriter():
         else:
             self._insert_status(ImplementationStatus(state=const.STATUS_PLANNED), level)
 
+    def _process_main_component(self, part_id: str, part_label: str, comp_dict: CompDict) -> None:
+        """Handle prompts for This System component."""
+        self._md_file.new_paragraph()
+        self._md_file.new_header(3, const.SSP_MAIN_COMP_NAME)
+        self._md_file.new_paragraph()
+        prose = f'{const.SSP_ADD_THIS_SYSTEM_IMPLEMENTATION_FOR_CONTROL_TEXT}: {part_id} -->'
+        status = ImplementationStatus(state=const.STATUS_PLANNED)
+        sys_comp_dict: Dict[str, ComponentImpInfo] = comp_dict.get(const.SSP_MAIN_COMP_NAME, {})
+        if sys_comp_dict and part_label in sys_comp_dict:
+            comp_info = sys_comp_dict[part_label]
+            if comp_info.prose:
+                prose = comp_info.prose
+            status = comp_info.status
+        self._md_file.new_paraline(prose)
+        self._insert_status(status, 4)
+
     def _add_component_control_prompts(self, control_id: str, comp_dict: CompDict, context: ControlContext) -> bool:
         """Add prompts to the markdown for the control itself, per component."""
         if context.purpose not in [ContextPurpose.COMPONENT, ContextPurpose.SSP]:
@@ -151,19 +167,10 @@ class ControlWriter():
         did_write = False
         # do special handling for This System
         if context.purpose == ContextPurpose.SSP:
-            self._md_file.new_paragraph()
-            self._md_file.new_header(3, const.SSP_MAIN_COMP_NAME)
-            self._md_file.new_paragraph()
-            prose = f'{const.SSP_ADD_THIS_SYSTEM_IMPLEMENTATION_FOR_CONTROL_TEXT}: {control_id} -->'
-            status = ImplementationStatus(state=const.STATUS_PLANNED)
-            if const.SSP_MAIN_COMP_NAME in comp_dict:
-                comp_info = list(comp_dict[const.SSP_MAIN_COMP_NAME].values())[0]
-                if comp_info.prose:
-                    prose = comp_info.prose
-                status = comp_info.status
-            self._md_file.new_paraline(prose)
-            self._insert_status(status, 4)
+            # Passing in the control id and the empty key ('') for the label
+            self._process_main_component(control_id, '', comp_dict)
             did_write = True
+
         sorted_comp_names = sorted(comp_dict.keys())
         for comp_name in sorted_comp_names:
             dic = comp_dict[comp_name]
@@ -209,14 +216,11 @@ class ControlWriter():
                         # if no label guess the label from the sub-part id
                         part_label = ControlInterface.get_label(prt)
                         part_label = prt.id.split('.')[-1] if not part_label else part_label
-                        # only write out part if rules apply to it
-                        rules_apply = False
-                        for _, dic in comp_dict.items():
-                            if part_label in dic and dic[part_label].rules:
-                                rules_apply = True
-                                break
-                        if not rules_apply:
+
+                        # Determine if this part should be written out
+                        if self._skip_part(context, part_label, comp_dict):
                             continue
+
                         if not did_write_part:
                             self._md_file.new_line(const.SSP_MD_LEAVE_BLANK_TEXT)
                             # insert extra line to make mdformat happy
@@ -224,11 +228,17 @@ class ControlWriter():
                         self._md_file.new_hr()
                         self._md_file.new_header(level=2, title=f'Implementation for part {part_label}')
                         wrote_label_content = False
+                        # To reduce verbosity by default, the main component
+                        # is conditionally written for each part dependant on the control context.
+                        if context.purpose == ContextPurpose.SSP and context.include_all_parts:
+                            self._process_main_component(prt.id, part_label, comp_dict)
+                            wrote_label_content = True
                         sorted_comp_names = sorted(comp_dict.keys())
                         for comp_name in sorted_comp_names:
-                            dic = comp_dict[comp_name]
+                            # This System already handled
                             if comp_name == const.SSP_MAIN_COMP_NAME:
                                 continue
+                            dic = comp_dict[comp_name]
                             if part_label in dic:
                                 # insert the component name for ssp but not for comp_def
                                 # because there should only be one component in generated comp_def markdown
@@ -256,6 +266,24 @@ class ControlWriter():
                     self._insert_comp_info(part_label, dic, context)
         self._md_file.new_hr()
 
+    def _skip_part(self, context: ControlContext, part_label: str, comp_dict: CompDict) -> bool:
+        """
+        Check if a part should be skipped based on rules and context.
+
+        Notes: The default logic is to conditionally add control parts based
+        on whether the component has rules associated with that part. This can be
+        changed using the control context for SSP markdown.
+        """
+        if context.purpose == ContextPurpose.SSP and context.include_all_parts:
+            return False
+        else:
+            no_applied_rules = True
+            for _, dic in comp_dict.items():
+                if part_label in dic and dic[part_label].rules:
+                    no_applied_rules = False
+                    break
+        return no_applied_rules
+
     def _dump_subpart_infos(self, level: int, part: Dict[str, Any]) -> None:
         name = part['name']
         title = self._sections_dict.get(name, name) if self._sections_dict else name
diff --git a/trestle/core/repository.py b/trestle/core/repository.py
index dfbff4fa..a873c07d 100644
--- a/trestle/core/repository.py
+++ b/trestle/core/repository.py
@@ -622,6 +622,7 @@ class AgileAuthoring(Repository):
         compdefs: str,
         leveraged_ssp: str = '',
         force_overwrite: bool = False,
+        include_all_parts: bool = False,
         yaml_header: str = '',
         overwrite_header_values: bool = False
     ) -> bool:
@@ -637,6 +638,7 @@ class AgileAuthoring(Repository):
             leveraged_ssp=leveraged_ssp,
             trestle_root=self.root_dir,
             force_overwrite=force_overwrite,
+            include_all_parts=include_all_parts,
             yaml_header=yaml_header,
             overwrite_header_values=overwrite_header_values,
             verbose=verbose


Test patch:
diff --git a/tests/test_utils.py b/tests/test_utils.py
index 354d17a9..29c746ad 100644
--- a/tests/test_utils.py
+++ b/tests/test_utils.py
@@ -484,6 +484,7 @@ def setup_for_ssp(
         output=output_name,
         verbose=0,
         overwrite_header_values=False,
+        include_all_parts=False,
         yaml_header=yaml_path,
         allowed_sections=None,
         force_overwrite=None
@@ -510,6 +511,7 @@ def setup_for_ssp_fedramp(
         output=output_name,
         verbose=0,
         overwrite_header_values=False,
+        include_all_parts=False,
         yaml_header=None,
         allowed_sections=None,
         force_overwrite=None
diff --git a/tests/trestle/core/commands/author/ssp_test.py b/tests/trestle/core/commands/author/ssp_test.py
index b2afbfc9..b4fe2c5e 100644
--- a/tests/trestle/core/commands/author/ssp_test.py
+++ b/tests/trestle/core/commands/author/ssp_test.py
@@ -1168,3 +1168,138 @@ def test_ssp_gen_and_assemble_add_props(tmp_trestle_dir: pathlib.Path) -> None:
     assert len(smt_a.props) == 1
     assert smt_a.props[0].name == 'smt_prop'
     assert smt_a.props[0].value == 'smt prop'
+
+
+def test_ssp_gen_and_assemble_implementation_parts(tmp_trestle_dir: pathlib.Path, monkeypatch: MonkeyPatch) -> None:
+    """Test ssp generate and assemble edit implementation parts."""
+    gen_args, _ = setup_for_ssp(tmp_trestle_dir, prof_name, ssp_name)
+    args_compdefs = gen_args.compdefs
+    gen_args.include_all_parts = True
+
+    ssp_gen = SSPGenerate()
+    assert ssp_gen._run(gen_args) == 0
+
+    prose_sys = 'My response for This System'
+    prose_aa = 'My response for comp aa'
+    prose_sys_a = 'My response for This System part a.'
+    prose_aa_a = 'My response for comp aa part a.'
+
+    # ac-1 edit
+    ac_1_path = tmp_trestle_dir / ssp_name / 'ac/ac-1.md'
+    assert test_utils.substitute_text_in_file(
+        ac_1_path, '<!-- Add implementation prose for the main This System component for control: ac-1 -->', prose_sys
+    )
+    assert test_utils.substitute_text_in_file(
+        ac_1_path,
+        '<!-- Add implementation prose for the main This System component for control: ac-1_smt.a -->',
+        prose_sys_a
+    )
+    assert test_utils.substitute_text_in_file(ac_1_path, 'imp req prose for ac-1 from comp aa', prose_aa)
+    assert test_utils.substitute_text_in_file(ac_1_path, 'statement prose for part a. from comp aa', prose_aa_a)
+    # change status for sys comp
+    assert test_utils.substitute_text_in_file(ac_1_path, 'Status: planned', 'Status: alternative')
+
+    part_a_text_edited = """## Implementation for part a.
+
+### This System
+
+My response for This System part a.
+
+#### Implementation Status: planned
+
+### comp_aa
+
+My response for comp aa part a.
+
+#### Rules:
+
+  - comp_rule_aa_1
+
+#### Implementation Status: partial
+
+### comp_ab
+
+<!-- Add control implementation description here for item a. -->
+
+#### Rules:
+
+  - comp_rule_ab_1
+
+#### Implementation Status: partial
+
+______________________________________________________________________
+"""
+
+    md_api = MarkdownAPI()
+    _, tree = md_api.processor.process_markdown(ac_1_path)
+    node = tree.get_node_for_key('## Implementation for part a.')
+    assert node.content.raw_text == part_a_text_edited
+
+    # now assemble the edited controls into json ssp
+    ssp_assemble = SSPAssemble()
+    args = argparse.Namespace(
+        trestle_root=tmp_trestle_dir,
+        markdown=ssp_name,
+        output=ssp_name,
+        verbose=0,
+        regenerate=False,
+        version='',
+        name=None,
+        compdefs=args_compdefs
+    )
+    assert ssp_assemble._run(args) == 0
+
+    # Verify the correct information is in the assembled ssp
+    ssp, _ = ModelUtils.load_model_for_class(tmp_trestle_dir, ssp_name, ossp.SystemSecurityPlan, FileContentType.JSON)
+    imp_reqs = ssp.control_implementation.implemented_requirements
+    ac_1_imp_req = next((i_req for i_req in imp_reqs if i_req.control_id == 'ac-1'), None)
+    assert ac_1_imp_req is not None
+
+    # comp_aa is the first component in the list and This System is the last
+    assert ac_1_imp_req.by_components[0].description == prose_aa  # type: ignore
+    assert ac_1_imp_req.by_components[2].implementation_status.state == 'alternative'  # type: ignore
+    assert ac_1_imp_req.by_components[2].description == prose_sys  # type: ignore
+    ac_1_a_smt = next((smt for smt in ac_1_imp_req.statements if smt.statement_id == 'ac-1_smt.a'), None)
+    assert ac_1_a_smt is not None
+    assert ac_1_a_smt.by_components[0].description == prose_aa_a  # type: ignore
+    assert ac_1_a_smt.by_components[2].description == prose_sys_a  # type: ignore
+
+    # Regeneration checks to make sure the markdown is not overwritten
+    assert ssp_gen._run(gen_args) == 0
+    _, tree = md_api.processor.process_markdown(ac_1_path)
+    node = tree.get_node_for_key('## Implementation for part a.')
+    assert node.content.raw_text == part_a_text_edited
+    assert test_utils.confirm_text_in_file(ac_1_path, const.SSP_MD_IMPLEMENTATION_QUESTION, prose_sys)
+
+
+def test_ssp_generate_no_cds_include_all_parts(tmp_trestle_dir: pathlib.Path) -> None:
+    """Test the ssp generator with no comp defs and include all parts are true."""
+    args, _ = setup_for_ssp(tmp_trestle_dir, prof_name, ssp_name)
+
+    args.compdefs = None
+    args.include_all_parts = True
+    ssp_cmd = SSPGenerate()
+    assert ssp_cmd._run(args) == 0
+    md_dir = tmp_trestle_dir / ssp_name
+    ac_1 = md_dir / 'ac/ac-1.md'
+    assert ac_1.exists()
+    at_2 = md_dir / 'at/at-2.md'
+    assert at_2.exists()
+
+    md_api = MarkdownAPI()
+    header, tree = md_api.processor.process_markdown(ac_1)
+    assert header[const.TRESTLE_GLOBAL_TAG][const.SORT_ID] == 'ac-01'
+
+    part_a_text_no_comp = """## Implementation for part a.
+
+### This System
+
+<!-- Add implementation prose for the main This System component for control: ac-1_smt.a -->
+
+#### Implementation Status: planned
+
+______________________________________________________________________
+"""
+
+    node = tree.get_node_for_key('## Implementation for part a.')
+    assert node.content.raw_text == part_a_text_no_comp