Repository: oscal-compass/compliance-trestle (2024-06-13 ~ 2024-06-18)

problem_statement: trestle validate component-definitions incorrectly failing on port-ranges
## Describe the bug

After upgrading to trestle 3.0.1, a ComponentDefinition file that was valid under trestle 2.6.1 started to fail with messages about the `protocols -> port-ranges` `start` and `end` values not being valid dicts. They should be non-negative integers.

## To Reproduce

Steps to reproduce the behavior:

Attempt to validate this minimum reproduction case:

```json
{
  "component-definition": {
    "uuid": "e7ba3d37-2ba0-4db9-9cf8-f5ec1214ffd5",
    "metadata": {
      "title": "Generic component-definition created by trestle named replication.",
      "last-modified": "2024-06-11T21:16:49.175968+00:00",
      "version": "0.0.0",
      "oscal-version": "1.1.2"
    },
    "components": [
      {
        "uuid": "4cef9684-578e-460e-a2d1-6eb78b8a76ca",
        "type": "REPLACE_ME",
        "title": "REPLACE_ME",
        "description": "REPLACE_ME",
        "protocols": [
          {
            "uuid": "0130b4ef-4561-4363-bab5-1e20b9888e49",
            "name": "https",
            "title": "Transport Layer Security",
            "port-ranges": [
              {
                "start": 443,
                "end": 443,
                "transport": "TCP"
              }
            ]
          }
        ],
        "control-implementations": []
      }
    ]
  }
}
```

## Expected behavior

`trestle validate` should return `VALID`.

## Screenshots / Logs.

<img width="1242" alt="Screenshot 2024-06-11 at 5 40 29 PM" src="https://github.com/oscal-compass/compliance-trestle/assets/285842/41618055-6437-4179-92d1-0ff322c82ddb">

## Environment

- OS: docker - python:3.11.9-slim base image.
- Python version: 3.11.9
- Installed packages: just trestle 3.0.1

date: 2024-06-13
patch: diff --git a/scripts/oscal_normalize.py b/scripts/oscal_normalize.py
index cd5fe724..73a37b29 100644
--- a/scripts/oscal_normalize.py
+++ b/scripts/oscal_normalize.py
@@ -1033,7 +1033,13 @@ def kill_roots(file_classes):
                 for ii in range(1, len(c.lines)):
                     line = c.lines[ii]
                     for name, body in new_root_classes.items():
-                        # handle special case
+                        # handle special cases
+                        if 'NonNegativeIntegerDatatype' in line:
+                            line = line.replace('NonNegativeIntegerDatatype', 'conint(ge=0, multiple_of=1)', 1)
+                        if 'PositiveIntegerDatatype' in line:
+                            line = line.replace('PositiveIntegerDatatype', 'conint(ge=1, multiple_of=1)', 1)
+                        if 'IntegerDatatype' in line:
+                            line = line.replace('IntegerDatatype', 'conint(multiple_of=1)', 1)
                         if c.name == 'OscalVersion':
                             if any(token in line for token in [' __root__: StringDatatype']):
                                 line = line.replace(name, body, 1)
diff --git a/trestle/oscal/common.py b/trestle/oscal/common.py
index 2908b765..71a463c8 100644
--- a/trestle/oscal/common.py
+++ b/trestle/oscal/common.py
@@ -408,6 +408,25 @@ class PortRangeValidValues(Enum):
     UDP = 'UDP'
 
 
+class PortRange(OscalBaseModel):
+    """
+    Where applicable this is the IPv4 port range on which the service operates.
+    """
+
+    class Config:
+        extra = Extra.forbid
+
+    start: Optional[conint(ge=0, multiple_of=1)] = Field(
+        None, description='Indicates the starting port number in a port range', title='Start'
+    )
+    end: Optional[conint(ge=0, multiple_of=1)] = Field(
+        None, description='Indicates the ending port number in a port range', title='End'
+    )
+    transport: Optional[PortRangeValidValues] = Field(
+        None, description='Indicates the transport type.', title='Transport'
+    )
+
+
 class PartyUuid(OscalBaseModel):
     __root__: UUIDDatatype = Field(
         ..., description='Reference to a party by UUID.', title='Party Universally Unique Identifier Reference'
@@ -982,9 +1001,9 @@ class AtFrequency(OscalBaseModel):
     class Config:
         extra = Extra.forbid
 
-    period: PositiveIntegerDatatype = Field(
-        ..., description='The task must occur after the specified period has elapsed.', title='Period'
-    )
+    period: conint(
+        ge=1, multiple_of=1
+    ) = Field(..., description='The task must occur after the specified period has elapsed.', title='Period')
     unit: TimeUnitValidValues = Field(..., description='The unit of time for the period.', title='Time Unit')
 
 
@@ -1557,25 +1576,6 @@ class Published(OscalBaseModel):
     )
 
 
-class PortRange(OscalBaseModel):
-    """
-    Where applicable this is the IPv4 port range on which the service operates.
-    """
-
-    class Config:
-        extra = Extra.forbid
-
-    start: Optional[NonNegativeIntegerDatatype] = Field(
-        None, description='Indicates the starting port number in a port range', title='Start'
-    )
-    end: Optional[NonNegativeIntegerDatatype] = Field(
-        None, description='Indicates the ending port number in a port range', title='End'
-    )
-    transport: Optional[PortRangeValidValues] = Field(
-        None, description='Indicates the transport type.', title='Transport'
-    )
-
-
 class Protocol(OscalBaseModel):
     """
     Information about the protocol used to provide a service.

test_patch: diff --git a/tests/data/validate/component-definitions/x1/component-definition.json b/tests/data/validate/component-definitions/x1/component-definition.json
new file mode 100644
index 00000000..0dc7ec5f
--- /dev/null
+++ b/tests/data/validate/component-definitions/x1/component-definition.json
@@ -0,0 +1,20 @@
+{
+  "component-definition": {
+    "uuid": "e7ba3d37-2ba0-4db9-9cf8-f5ec1214ffd5",
+    "metadata": {
+      "title": "Generic component-definition created by trestle named replication.",
+      "last-modified": "2024-06-11T21:16:49.175968+00:00",
+      "version": "0.0.0",
+      "oscal-version": "1.0.4"
+    },
+    "components": [
+      {
+        "uuid": "4cef9684-578e-460e-a2d1-6eb78b8a76ca",
+        "type": "REPLACE_ME",
+        "title": "REPLACE_ME",
+        "description": "REPLACE_ME",
+        "control-implementations": []
+      }
+    ]
+  }
+}
\ No newline at end of file
diff --git a/tests/data/validate/component-definitions/x2/component-definition.json b/tests/data/validate/component-definitions/x2/component-definition.json
new file mode 100644
index 00000000..69c85ee2
--- /dev/null
+++ b/tests/data/validate/component-definitions/x2/component-definition.json
@@ -0,0 +1,34 @@
+{
+  "component-definition": {
+    "uuid": "e7ba3d37-2ba0-4db9-9cf8-f5ec1214ffd5",
+    "metadata": {
+      "title": "Generic component-definition created by trestle named replication.",
+      "last-modified": "2024-06-11T21:16:49.175968+00:00",
+      "version": "0.0.0",
+      "oscal-version": "1.0.4"
+    },
+    "components": [
+      {
+        "uuid": "4cef9684-578e-460e-a2d1-6eb78b8a76ca",
+        "type": "REPLACE_ME",
+        "title": "REPLACE_ME",
+        "description": "REPLACE_ME",
+        "protocols": [
+          {
+            "uuid": "0130b4ef-4561-4363-bab5-1e20b9888e49",
+            "name": "https",
+            "title": "Transport Layer Security",
+            "port-ranges": [
+              {
+                "start": 443,
+                "end": 443,
+                "transport": "TCP"
+              }
+            ]
+          }
+        ],
+        "control-implementations": []
+      }
+    ]
+  }
+}
\ No newline at end of file
diff --git a/tests/data/validate/component-definitions/x3/component-definition.json b/tests/data/validate/component-definitions/x3/component-definition.json
new file mode 100644
index 00000000..1fadb711
--- /dev/null
+++ b/tests/data/validate/component-definitions/x3/component-definition.json
@@ -0,0 +1,34 @@
+{
+  "component-definition": {
+    "uuid": "e7ba3d37-2ba0-4db9-9cf8-f5ec1214ffd5",
+    "metadata": {
+      "title": "Generic component-definition created by trestle named replication.",
+      "last-modified": "2024-06-11T21:16:49.175968+00:00",
+      "version": "0.0.0",
+      "oscal-version": "1.0.4"
+    },
+    "components": [
+      {
+        "uuid": "4cef9684-578e-460e-a2d1-6eb78b8a76ca",
+        "type": "REPLACE_ME",
+        "title": "REPLACE_ME",
+        "description": "REPLACE_ME",
+        "protocols": [
+          {
+            "uuid": "0130b4ef-4561-4363-bab5-1e20b9888e49",
+            "name": "https",
+            "title": "Transport Layer Security",
+            "port-ranges": [
+              {
+                "start": -443,
+                "end": 443,
+                "transport": "TCP"
+              }
+            ]
+          }
+        ],
+        "control-implementations": []
+      }
+    ]
+  }
+}
\ No newline at end of file
diff --git a/tests/trestle/core/commands/validate_test.py b/tests/trestle/core/commands/validate_test.py
index 4f14a089..8eb02f45 100644
--- a/tests/trestle/core/commands/validate_test.py
+++ b/tests/trestle/core/commands/validate_test.py
@@ -30,6 +30,7 @@ from tests.test_utils import setup_for_ssp
 import trestle.common.const as const
 import trestle.core.generators as gens
 import trestle.oscal.assessment_plan as ap
+import trestle.oscal.common as common
 import trestle.oscal.profile as prof
 import trestle.oscal.ssp as ossp
 from trestle import cli
@@ -407,3 +408,65 @@ def test_validate_ssp_with_no_profile(tmp_trestle_dir: pathlib.Path, monkeypatch
 
     new_ssp.import_profile.href = original_href
     ModelUtils.save_top_level_model(new_ssp, tmp_trestle_dir, ssp_name, FileContentType.JSON)
+
+
+def test_period(tmp_trestle_dir: pathlib.Path, monkeypatch: MonkeyPatch) -> None:
+    """Test period."""
+    unit = common.TimeUnitValidValues.seconds
+    _ = common.AtFrequency(period=1, unit=unit)
+    try:
+        _ = common.AtFrequency(period=0, unit=unit)
+        raise RuntimeError('must be positive integer')
+    except Exception:
+        pass
+
+
+def test_validate_component_definition(tmp_trestle_dir: pathlib.Path, monkeypatch: MonkeyPatch) -> None:
+    """Test validation of Component Definition."""
+    jfile = 'component-definition.json'
+
+    sdir = test_data_dir / 'validate' / 'component-definitions' / 'x1'
+    spth = sdir / f'{jfile}'
+
+    tdir = tmp_trestle_dir / test_utils.COMPONENT_DEF_DIR / 'my_test_model'
+    tpth = tdir / f'{jfile}'
+
+    (tdir).mkdir(exist_ok=True, parents=True)
+
+    shutil.copyfile(spth, tpth)
+    validate_command = f'trestle validate -f {tpth}'
+    test_utils.execute_command_and_assert(validate_command, 0, monkeypatch)
+
+
+def test_validate_component_definition_ports(tmp_trestle_dir: pathlib.Path, monkeypatch: MonkeyPatch) -> None:
+    """Test validation of ports in Component Definition."""
+    jfile = 'component-definition.json'
+
+    sdir = test_data_dir / 'validate' / 'component-definitions' / 'x2'
+    spth = sdir / f'{jfile}'
+
+    tdir = tmp_trestle_dir / test_utils.COMPONENT_DEF_DIR / 'my_test_model'
+    tpth = tdir / f'{jfile}'
+
+    (tdir).mkdir(exist_ok=True, parents=True)
+
+    shutil.copyfile(spth, tpth)
+    validate_command = f'trestle validate -f {tpth}'
+    test_utils.execute_command_and_assert(validate_command, 0, monkeypatch)
+
+
+def test_validate_component_definition_ports_invalid(tmp_trestle_dir: pathlib.Path, monkeypatch: MonkeyPatch) -> None:
+    """Test validation of ports in Component Definition."""
+    jfile = 'component-definition.json'
+
+    sdir = test_data_dir / 'validate' / 'component-definitions' / 'x3'
+    spth = sdir / f'{jfile}'
+
+    tdir = tmp_trestle_dir / test_utils.COMPONENT_DEF_DIR / 'my_test_model'
+    tpth = tdir / f'{jfile}'
+
+    (tdir).mkdir(exist_ok=True, parents=True)
+
+    shutil.copyfile(spth, tpth)
+    validate_command = f'trestle validate -f {tpth}'
+    test_utils.execute_command_and_assert(validate_command, 1, monkeypatch)


----

problem_statement: ` _CsvMgr.get_rule_key() ` is supplied with incorrect argument
## Describe the bug

<img width="904" alt="image" src="https://github.com/oscal-compass/compliance-trestle/assets/78010183/a7126df8-fcba-4b15-820f-c555101a60f7">
The above code snippet is from `csv-to-oscal-cd.py`.  The highlighted portion has a potential bug. The `get_rule_key()` expects `component_title`, but `component_description` is passed in. But, this error somehow doesn't generate any serious bug whether when  `component_title` has the same value as `component_description`(in the majority cases) or when they are different. However, the expected key is not in the `_csv_rules_map` dict.

## To Reproduce

Steps to reproduce the behavior:
1. Refer to the below test case:

```
def test_execute_different_component_title_and_description(tmp_path: pathlib.Path) -> None:
    """Test execute missing value."""
    _, section = _get_config_section_init(tmp_path, 'test-csv-to-oscal-cd.config')
    # inject error
    rows = _get_rows('tests/data/csv/ocp4-user.v2.csv')
    row = rows[2]
    assert row[7] == 'OSCO'
    assert row[8] == 'OSCO'
    row[8] = 'IAM'
    assert row[8] == 'IAM'
    component_title = row[7]
    component_description = row[8]
    component_type = row[5]
    rule_id = row[1]
    with mock.patch('trestle.tasks.csv_to_oscal_cd.csv.reader') as mock_csv_reader:
        mock_csv_reader.return_value = rows
        expected_key = (component_title, component_type, rule_id)
        wrong_key = (component_description, component_type, rule_id)
        tgt = csv_to_oscal_cd.CsvToOscalComponentDefinition(section)
        retval = tgt.execute()
        assert retval == TaskOutcome.SUCCESS
        assert wrong_key in tgt._csv_mgr.get_rule_keys()
        assert expected_key in tgt._csv_mgr.get_rule_keys()
```

## Expected behavior

A clear and concise description of what you expected to happen.

## Screenshots / Logs.

If applicable, add screenshots to help explain your problem.

## Environment

- OS: \[e.g. iOS\]
- Python version:
- Installed packages:

date: 2024-06-13
patch: diff --git a/trestle/tasks/csv_to_oscal_cd.py b/trestle/tasks/csv_to_oscal_cd.py
index 455b6131..367359ca 100644
--- a/trestle/tasks/csv_to_oscal_cd.py
+++ b/trestle/tasks/csv_to_oscal_cd.py
@@ -1545,7 +1545,7 @@ class _CsvMgr():
             component_description = self.get_row_value(row, f'{COMPONENT_DESCRIPTION}')
             rule_id = self.get_row_value(row, f'{RULE_ID}')
             # rule sets
-            key = _CsvMgr.get_rule_key(component_description, component_type, rule_id)
+            key = _CsvMgr.get_rule_key(component_title, component_type, rule_id)
             if key in self._csv_rules_map:
                 text = f'row "{row_num}" contains duplicate {RULE_ID} "{rule_id}"'
                 raise RuntimeError(text)

test_patch: diff --git a/tests/trestle/tasks/csv_to_oscal_cd_test.py b/tests/trestle/tasks/csv_to_oscal_cd_test.py
index e1d075ec..e2ebc1fa 100644
--- a/tests/trestle/tasks/csv_to_oscal_cd_test.py
+++ b/tests/trestle/tasks/csv_to_oscal_cd_test.py
@@ -885,6 +885,34 @@ def test_execute_param_duplicate_value(tmp_path: pathlib.Path) -> None:
         assert retval == TaskOutcome.FAILURE
 
 
+def test_execute_correct_rule_key(tmp_path: pathlib.Path) -> None:
+    """Test execute missing value."""
+    _, section = _get_config_section_init(tmp_path, 'test-csv-to-oscal-cd.config')
+    rows = _get_rows('tests/data/csv/ocp4-user.v2.csv')
+    row = rows[2]
+    # ensure component_title and component_description are different
+    assert row[7] == 'OSCO'
+    assert row[8] == 'OSCO'
+    row[8] = 'IAM'
+    assert row[8] == 'IAM'
+    component_title = row[7]
+    component_description = row[8]
+    component_type = row[5]
+    rule_id = row[1]
+    with mock.patch('trestle.tasks.csv_to_oscal_cd.csv.reader') as mock_csv_reader:
+        mock_csv_reader.return_value = rows
+        # perform transformation
+        tgt = csv_to_oscal_cd.CsvToOscalComponentDefinition(section)
+        retval = tgt.execute()
+        assert retval == TaskOutcome.SUCCESS
+        # insure expected key exists
+        expected_key = (component_title, component_type, rule_id)
+        assert expected_key in tgt._csv_mgr.get_rule_keys()
+        # insure unexpected key does not exist
+        unexpected_key = (component_description, component_type, rule_id)
+        assert unexpected_key not in tgt._csv_mgr.get_rule_keys()
+
+
 def test_execute_missing_param_default_value(tmp_path: pathlib.Path) -> None:
     """Test execute missing param default_value."""
     _, section = _get_config_section_init(tmp_path, 'test-csv-to-oscal-cd-bp.config')


----

problem_statement: Trestle author -tv flag doesn't enforce the version format (e.g. 0.0.1) correctly
## Describe the bug
Invalid version formats (e.g. 1a2b3) are not captured when using the `trestle author header -tv` command. This behavior is due to the . character is not escaped properly in `is_valid_version()` under `template_versioning.py`.

A fix might look like this:
<img width="406" alt="image" src="https://github.com/oscal-compass/compliance-trestle/assets/78010183/685f98cf-8a83-4f0f-8b41-8e2c60ba622e">


## To Reproduce

Steps to reproduce the behavior:
` trestle author headers setup -tn my_task -tv 1a2b3`

## Expected behavior

The program should throw an error, and the task folder should not be created.

## Screenshots / Logs.
After executing the above command:
<img width="303" alt="image" src="https://github.com/oscal-compass/compliance-trestle/assets/78010183/07754145-6e73-4b0b-bb38-05576d01d559">

## Environment

- OS: \[e.g. iOS\]
- Python version:
- Installed packages:

date: 2024-06-18
patch: diff --git a/trestle/core/commands/author/versioning/template_versioning.py b/trestle/core/commands/author/versioning/template_versioning.py
index e9c954d8..43f52483 100644
--- a/trestle/core/commands/author/versioning/template_versioning.py
+++ b/trestle/core/commands/author/versioning/template_versioning.py
@@ -212,7 +212,7 @@ class TemplateVersioning:
             return True  # we can have empty version
         if template_version == '0.0.0':
             return False
-        version_regex = r'^[0-9]+.[0-9]+.[0-9]+$'
+        version_regex = r'^[0-9]+\.[0-9]+\.[0-9]+$'
         pattern = re.compile(version_regex)
         if pattern.search(template_version):
             return True

test_patch: diff --git a/tests/trestle/core/commands/author/versioning/template_versioning_test.py b/tests/trestle/core/commands/author/versioning/template_versioning_test.py
index 8d7fbb10..10f04bb2 100644
--- a/tests/trestle/core/commands/author/versioning/template_versioning_test.py
+++ b/tests/trestle/core/commands/author/versioning/template_versioning_test.py
@@ -222,6 +222,7 @@ def test_valid_version() -> None:
     assert not TemplateVersioning.is_valid_version('0.1')
     assert not TemplateVersioning.is_valid_version('1')
     assert not TemplateVersioning.is_valid_version('0.0.0.1')
+    assert not TemplateVersioning.is_valid_version('0a0b1')
 
 
 def test_empty_folder_is_not_created(tmp_path: pathlib.Path) -> None: