Repository: fkie-cad/Logprep (language: python)

Issue:
Auto Rule Tester throws AttributeError after ProcessorStats Refactoring
The auto rule tester is currently not operational because it tries to access a processor attribute that was removed by the ProcesorStats refactoring.

**Expected behavior**
The auto rule tester should iterate over the rule tests and output the results.

**Current behavior**
An AttributeError is being printed `AttributeError: 'Normalizer' object has no attribute 'ps'`.

**Steps to reproduce**

1. Create a rule test in the exampledata directory, for example `exampledata/rules/normalizer/generic/example_rule_test.json`
```json
[
  {
    "raw": {
      "event": "foo"
    },
    "processed": {
      "event": "foo"
    }
  }
]
```
3. Start logprep with `PYTHONPATH='.' python logprep/run_logprep.py quickstart/exampledata/config/pipeline.yml --auto-test`
4. See error `AttributeError: 'Normalizer' object has no attribute 'ps'`

**Environment**

Logprep version: ba49d341
Python version: 3.9.13

**Possible solution**
Remove all reference to to the removed processor stats object and write more tests for the auto rule tester.


Discussion / hints:

Fix patch:
diff --git a/CHANGELOG.md b/CHANGELOG.md
index e1e39e7a4..0da285c64 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -9,6 +9,7 @@
 
 * Fix performance of the metrics tracking. Due to a store metrics statement at the wrong position
 the logprep performance was dramatically decreased when tracking metrics was activated.
+* Fix Auto Rule Tester which tried to access processor stats that do not exist anymore. 
 
 ### Breaking
 
diff --git a/logprep/processor/clusterer/processor.py b/logprep/processor/clusterer/processor.py
index 04671f9df..52f14a865 100644
--- a/logprep/processor/clusterer/processor.py
+++ b/logprep/processor/clusterer/processor.py
@@ -17,6 +17,20 @@
     Criteria 1: { "message": "A sample message", "tags": ["clusterable", ...], ... }
     Criteria 2: { "message": "A sample message", "clusterable": true, ... }
     Criteria 3: { "message": "A sample message", "syslog": { "facility": <number> }, "event": { "severity": <string> }, ... }
+
+
+Example
+^^^^^^^
+..  code-block:: yaml
+    :linenos:
+
+    - clusterername:
+        type: clusterer
+        specific_rules:
+            - tests/testdata/rules/specific/
+        generic_rules:
+            - tests/testdata/rules/generic/
+        output_field_name: target_field
 """
 
 from logging import Logger


Test patch:
diff --git a/logprep/util/auto_rule_tester.py b/logprep/util/auto_rule_tester.py
index 41333c573..b4529944c 100644
--- a/logprep/util/auto_rule_tester.py
+++ b/logprep/util/auto_rule_tester.py
@@ -442,8 +442,6 @@ def _eval_file_rule_test(self, rule_test: dict, processor: Processor, r_idx: int
             if test.get("target_rule_idx") is not None and test.get("target_rule_idx") != r_idx:
                 continue
 
-            # Setup arrays according to rule count, but here it's always one
-            processor.ps.setup_rules(["placeholder"])
             try:
                 extra_output = processor.process(test["raw"])
             except BaseException as error:
diff --git a/tests/testdata/auto_tests/clusterer/rules/generic/rule_with_custom_tests.yml b/tests/testdata/auto_tests/clusterer/rules/generic/rule_with_custom_tests.yml
new file mode 100644
index 000000000..d433ad94e
--- /dev/null
+++ b/tests/testdata/auto_tests/clusterer/rules/generic/rule_with_custom_tests.yml
@@ -0,0 +1,9 @@
+filter: message
+clusterer:
+  target: message
+  pattern: '(bytes|Bytes|Byte)'
+  repl: 'byte'
+description: '...'
+tests:
+  raw:    'Byte is a Bytes is a bytes is a byte'
+  result: 'byte is a byte is a byte is a byte'
\ No newline at end of file
diff --git a/tests/testdata/auto_tests/clusterer/rules/specific/rule_with_custom_tests.yml b/tests/testdata/auto_tests/clusterer/rules/specific/rule_with_custom_tests.yml
new file mode 100644
index 000000000..d433ad94e
--- /dev/null
+++ b/tests/testdata/auto_tests/clusterer/rules/specific/rule_with_custom_tests.yml
@@ -0,0 +1,9 @@
+filter: message
+clusterer:
+  target: message
+  pattern: '(bytes|Bytes|Byte)'
+  repl: 'byte'
+description: '...'
+tests:
+  raw:    'Byte is a Bytes is a bytes is a byte'
+  result: 'byte is a byte is a byte is a byte'
\ No newline at end of file
diff --git a/tests/testdata/auto_tests/dropper/rules/drop_field.json b/tests/testdata/auto_tests/dropper/rules/generic/drop_field.json
similarity index 100%
rename from tests/testdata/auto_tests/dropper/rules/drop_field.json
rename to tests/testdata/auto_tests/dropper/rules/generic/drop_field.json
diff --git a/tests/testdata/auto_tests/dropper/rules/drop_field_test.json b/tests/testdata/auto_tests/dropper/rules/generic/drop_field_test.json
similarity index 100%
rename from tests/testdata/auto_tests/dropper/rules/drop_field_test.json
rename to tests/testdata/auto_tests/dropper/rules/generic/drop_field_test.json
diff --git a/tests/testdata/auto_tests/dropper/rules/specific/drop_field.json b/tests/testdata/auto_tests/dropper/rules/specific/drop_field.json
new file mode 100644
index 000000000..d95447ce4
--- /dev/null
+++ b/tests/testdata/auto_tests/dropper/rules/specific/drop_field.json
@@ -0,0 +1,4 @@
+[{
+  "filter": "drop_me",
+  "drop":  ["drop_me"]
+}]
\ No newline at end of file
diff --git a/tests/testdata/auto_tests/dropper/rules/specific/drop_field_test.json b/tests/testdata/auto_tests/dropper/rules/specific/drop_field_test.json
new file mode 100644
index 000000000..a39057dd4
--- /dev/null
+++ b/tests/testdata/auto_tests/dropper/rules/specific/drop_field_test.json
@@ -0,0 +1,4 @@
+[{
+  "raw": {"drop_me":  "some_value", "keep_me":  "some_value"},
+  "processed": {"keep_me":  "some_value"}
+}]
\ No newline at end of file
diff --git a/tests/testdata/auto_tests/labeler/rules/auto_test_labeling_match.json b/tests/testdata/auto_tests/labeler/rules/generic/auto_test_labeling_match.json
similarity index 100%
rename from tests/testdata/auto_tests/labeler/rules/auto_test_labeling_match.json
rename to tests/testdata/auto_tests/labeler/rules/generic/auto_test_labeling_match.json
diff --git a/tests/testdata/auto_tests/labeler/rules/auto_test_labeling_match_test.json b/tests/testdata/auto_tests/labeler/rules/generic/auto_test_labeling_match_test.json
similarity index 100%
rename from tests/testdata/auto_tests/labeler/rules/auto_test_labeling_match_test.json
rename to tests/testdata/auto_tests/labeler/rules/generic/auto_test_labeling_match_test.json
diff --git a/tests/testdata/auto_tests/labeler/rules/auto_test_labeling_mismatch.json b/tests/testdata/auto_tests/labeler/rules/specific/auto_test_labeling_mismatch.json
similarity index 100%
rename from tests/testdata/auto_tests/labeler/rules/auto_test_labeling_mismatch.json
rename to tests/testdata/auto_tests/labeler/rules/specific/auto_test_labeling_mismatch.json
diff --git a/tests/testdata/auto_tests/labeler/rules/auto_test_labeling_mismatch_test.json b/tests/testdata/auto_tests/labeler/rules/specific/auto_test_labeling_mismatch_test.json
similarity index 100%
rename from tests/testdata/auto_tests/labeler/rules/auto_test_labeling_mismatch_test.json
rename to tests/testdata/auto_tests/labeler/rules/specific/auto_test_labeling_mismatch_test.json
diff --git a/tests/testdata/auto_tests/labeler/rules/auto_test_labeling_no_test_.json b/tests/testdata/auto_tests/labeler/rules/specific/auto_test_labeling_no_test_.json
similarity index 100%
rename from tests/testdata/auto_tests/labeler/rules/auto_test_labeling_no_test_.json
rename to tests/testdata/auto_tests/labeler/rules/specific/auto_test_labeling_no_test_.json
diff --git a/tests/testdata/auto_tests/normalizer/rules/generic_or_specific/auto_test_normalizer_match.json b/tests/testdata/auto_tests/normalizer/rules/generic/auto_test_normalizer_match.json
similarity index 100%
rename from tests/testdata/auto_tests/normalizer/rules/generic_or_specific/auto_test_normalizer_match.json
rename to tests/testdata/auto_tests/normalizer/rules/generic/auto_test_normalizer_match.json
diff --git a/tests/testdata/auto_tests/normalizer/rules/generic_or_specific/auto_test_normalizer_match_test.json b/tests/testdata/auto_tests/normalizer/rules/generic/auto_test_normalizer_match_test.json
similarity index 100%
rename from tests/testdata/auto_tests/normalizer/rules/generic_or_specific/auto_test_normalizer_match_test.json
rename to tests/testdata/auto_tests/normalizer/rules/generic/auto_test_normalizer_match_test.json
diff --git a/tests/testdata/auto_tests/normalizer/rules/generic_or_specific/auto_test_normalizer_mismatch.json b/tests/testdata/auto_tests/normalizer/rules/specific/auto_test_normalizer_mismatch.json
similarity index 100%
rename from tests/testdata/auto_tests/normalizer/rules/generic_or_specific/auto_test_normalizer_mismatch.json
rename to tests/testdata/auto_tests/normalizer/rules/specific/auto_test_normalizer_mismatch.json
diff --git a/tests/testdata/auto_tests/normalizer/rules/generic_or_specific/auto_test_normalizer_mismatch_test.json b/tests/testdata/auto_tests/normalizer/rules/specific/auto_test_normalizer_mismatch_test.json
similarity index 100%
rename from tests/testdata/auto_tests/normalizer/rules/generic_or_specific/auto_test_normalizer_mismatch_test.json
rename to tests/testdata/auto_tests/normalizer/rules/specific/auto_test_normalizer_mismatch_test.json
diff --git a/tests/testdata/auto_tests/normalizer/rules/generic_or_specific/auto_test_normalizer_no_test_.json b/tests/testdata/auto_tests/normalizer/rules/specific/auto_test_normalizer_no_test_.json
similarity index 100%
rename from tests/testdata/auto_tests/normalizer/rules/generic_or_specific/auto_test_normalizer_no_test_.json
rename to tests/testdata/auto_tests/normalizer/rules/specific/auto_test_normalizer_no_test_.json
diff --git a/tests/testdata/auto_tests/pre_detector/rules_auto_tests/auto_test_pre_detector_match.json b/tests/testdata/auto_tests/pre_detector/rules/generic/auto_test_pre_detector_match.json
similarity index 100%
rename from tests/testdata/auto_tests/pre_detector/rules_auto_tests/auto_test_pre_detector_match.json
rename to tests/testdata/auto_tests/pre_detector/rules/generic/auto_test_pre_detector_match.json
diff --git a/tests/testdata/auto_tests/pre_detector/rules_auto_tests/auto_test_pre_detector_match_test.json b/tests/testdata/auto_tests/pre_detector/rules/generic/auto_test_pre_detector_match_test.json
similarity index 100%
rename from tests/testdata/auto_tests/pre_detector/rules_auto_tests/auto_test_pre_detector_match_test.json
rename to tests/testdata/auto_tests/pre_detector/rules/generic/auto_test_pre_detector_match_test.json
diff --git a/tests/testdata/auto_tests/pre_detector/rules_auto_tests/auto_test_pre_detector_mismatch.json b/tests/testdata/auto_tests/pre_detector/rules/specific/auto_test_pre_detector_mismatch.json
similarity index 100%
rename from tests/testdata/auto_tests/pre_detector/rules_auto_tests/auto_test_pre_detector_mismatch.json
rename to tests/testdata/auto_tests/pre_detector/rules/specific/auto_test_pre_detector_mismatch.json
diff --git a/tests/testdata/auto_tests/pre_detector/rules_auto_tests/auto_test_pre_detector_mismatch_test.json b/tests/testdata/auto_tests/pre_detector/rules/specific/auto_test_pre_detector_mismatch_test.json
similarity index 100%
rename from tests/testdata/auto_tests/pre_detector/rules_auto_tests/auto_test_pre_detector_mismatch_test.json
rename to tests/testdata/auto_tests/pre_detector/rules/specific/auto_test_pre_detector_mismatch_test.json
diff --git a/tests/testdata/auto_tests/pre_detector/rules_auto_tests/auto_test_pre_detector_no_test_.json b/tests/testdata/auto_tests/pre_detector/rules/specific/auto_test_pre_detector_no_test_.json
similarity index 100%
rename from tests/testdata/auto_tests/pre_detector/rules_auto_tests/auto_test_pre_detector_no_test_.json
rename to tests/testdata/auto_tests/pre_detector/rules/specific/auto_test_pre_detector_no_test_.json
diff --git a/tests/testdata/auto_tests/pseudonymizer/rules/generic_or_specific/auto_test_pseudonymizer_match.json b/tests/testdata/auto_tests/pseudonymizer/rules/generic/auto_test_pseudonymizer_match.json
similarity index 100%
rename from tests/testdata/auto_tests/pseudonymizer/rules/generic_or_specific/auto_test_pseudonymizer_match.json
rename to tests/testdata/auto_tests/pseudonymizer/rules/generic/auto_test_pseudonymizer_match.json
diff --git a/tests/testdata/auto_tests/pseudonymizer/rules/generic_or_specific/auto_test_pseudonymizer_match_test.json b/tests/testdata/auto_tests/pseudonymizer/rules/generic/auto_test_pseudonymizer_match_test.json
similarity index 100%
rename from tests/testdata/auto_tests/pseudonymizer/rules/generic_or_specific/auto_test_pseudonymizer_match_test.json
rename to tests/testdata/auto_tests/pseudonymizer/rules/generic/auto_test_pseudonymizer_match_test.json
diff --git a/tests/testdata/auto_tests/pseudonymizer/rules/generic_or_specific/auto_test_pseudonymizer_mismatch.json b/tests/testdata/auto_tests/pseudonymizer/rules/specific/auto_test_pseudonymizer_mismatch.json
similarity index 100%
rename from tests/testdata/auto_tests/pseudonymizer/rules/generic_or_specific/auto_test_pseudonymizer_mismatch.json
rename to tests/testdata/auto_tests/pseudonymizer/rules/specific/auto_test_pseudonymizer_mismatch.json
diff --git a/tests/testdata/auto_tests/pseudonymizer/rules/generic_or_specific/auto_test_pseudonymizer_mismatch_test.json b/tests/testdata/auto_tests/pseudonymizer/rules/specific/auto_test_pseudonymizer_mismatch_test.json
similarity index 100%
rename from tests/testdata/auto_tests/pseudonymizer/rules/generic_or_specific/auto_test_pseudonymizer_mismatch_test.json
rename to tests/testdata/auto_tests/pseudonymizer/rules/specific/auto_test_pseudonymizer_mismatch_test.json
diff --git a/tests/testdata/auto_tests/pseudonymizer/rules/generic_or_specific/auto_test_pseudonymizer_no_test_.json b/tests/testdata/auto_tests/pseudonymizer/rules/specific/auto_test_pseudonymizer_no_test_.json
similarity index 100%
rename from tests/testdata/auto_tests/pseudonymizer/rules/generic_or_specific/auto_test_pseudonymizer_no_test_.json
rename to tests/testdata/auto_tests/pseudonymizer/rules/specific/auto_test_pseudonymizer_no_test_.json
diff --git a/tests/testdata/auto_tests/template_replacer/rules/template_replacer.json b/tests/testdata/auto_tests/template_replacer/rules/generic/template_replacer.json
similarity index 100%
rename from tests/testdata/auto_tests/template_replacer/rules/template_replacer.json
rename to tests/testdata/auto_tests/template_replacer/rules/generic/template_replacer.json
diff --git a/tests/testdata/auto_tests/template_replacer/rules/template_replacer_test.json b/tests/testdata/auto_tests/template_replacer/rules/generic/template_replacer_test.json
similarity index 100%
rename from tests/testdata/auto_tests/template_replacer/rules/template_replacer_test.json
rename to tests/testdata/auto_tests/template_replacer/rules/generic/template_replacer_test.json
diff --git a/tests/testdata/auto_tests/template_replacer/rules/specific/template_replacer.json b/tests/testdata/auto_tests/template_replacer/rules/specific/template_replacer.json
new file mode 100644
index 000000000..051249872
--- /dev/null
+++ b/tests/testdata/auto_tests/template_replacer/rules/specific/template_replacer.json
@@ -0,0 +1,5 @@
+[{
+  "filter": "winlog.provider_name: \"the provider\" AND winlog.event_id: 123",
+  "template_replacer": {},
+  "description": ""
+}]
\ No newline at end of file
diff --git a/tests/testdata/auto_tests/template_replacer/rules/specific/template_replacer_test.json b/tests/testdata/auto_tests/template_replacer/rules/specific/template_replacer_test.json
new file mode 100644
index 000000000..e85378f33
--- /dev/null
+++ b/tests/testdata/auto_tests/template_replacer/rules/specific/template_replacer_test.json
@@ -0,0 +1,18 @@
+[{
+  "raw": {
+    "winlog": {
+      "provider_name": "the provider",
+      "event_id": 123,
+      "channel": 123
+    },
+    "dotted": {"message": "replace me!"}
+  },
+  "processed": {
+    "winlog": {
+      "provider_name": "the provider",
+      "event_id": 123,
+      "channel": 123
+    },
+    "dotted": {"message": "replaced message"}
+  }
+}]
\ No newline at end of file
diff --git a/tests/testdata/config/config-auto-tests.yml b/tests/testdata/config/config-auto-tests.yml
index eb0388efb..d5e570a28 100644
--- a/tests/testdata/config/config-auto-tests.yml
+++ b/tests/testdata/config/config-auto-tests.yml
@@ -3,28 +3,32 @@ timeout: 0.1
 
 pipeline:
   - labelername:
-        type: labeler
-        schema: tests/testdata/auto_tests/labeler/schema.json
-        include_parent_labels: True
-        specific_rules:
-            - tests/testdata/auto_tests/labeler/rules/specific/
-        generic_rules:
-          - tests/testdata/auto_tests/labeler/rules/generic/
+      type: labeler
+      schema: tests/testdata/auto_tests/labeler/schema.json
+      include_parent_labels: True
+      specific_rules:
+          - tests/testdata/auto_tests/labeler/rules/specific/
+      generic_rules:
+        - tests/testdata/auto_tests/labeler/rules/generic/
   - normalizer:
       type: normalizer
       specific_rules:
-        - tests/testdata/auto_tests/normalizer/rules/generic_or_specific/
+        - tests/testdata/auto_tests/normalizer/rules/specific/
       generic_rules:
-        - ''
+        - tests/testdata/auto_tests/normalizer/rules/generic/
       regex_mapping: tests/testdata/auto_tests/normalizer/regex_mapping.yml
   - dropper:
       type: dropper
-      rules:
-        - tests/testdata/auto_tests/dropper/rules/
+      specific_rules:
+        - tests/testdata/auto_tests/dropper/rules/specific/
+      generic_rules:
+        - tests/testdata/auto_tests/dropper/rules/generic/
   - pre_detector:
       type: pre_detector
-      rules:
-        - tests/testdata/auto_tests/pre_detector/rules
+      specific_rules:
+        - tests/testdata/auto_tests/pre_detector/rules/specific/
+      generic_rules:
+        - tests/testdata/auto_tests/pre_detector/rules/generic/
       pre_detector_topic: sre
   - pseudonymizer:
       type: pseudonymizer
@@ -34,9 +38,29 @@ pipeline:
       hash_salt: a_secret_tasty_ingredient
       pseudonyms_topic: pseudonyms
       specific_rules:
-        - tests/testdata/auto_tests/pseudonymizer/rules/generic_or_specific/
+        - tests/testdata/auto_tests/pseudonymizer/rules/generic/
       generic_rules:
-        - ''
+        - tests/testdata/auto_tests/pseudonymizer/rules/specific/
       max_cached_pseudonyms: 1000000
       max_caching_days: 1
-      tld_list: '-'
+  - templatereplacername:
+      type: template_replacer
+      specific_rules:
+        - tests/testdata/auto_tests/template_replacer/rules/specific/
+      generic_rules:
+        - tests/testdata/auto_tests/template_replacer/rules/generic/
+      template: tests/testdata/unit/template_replacer/replacer_template.yml
+      pattern:
+          delimiter: ","
+          fields:
+              - field.name.a
+              - field.name.b
+          allowed_delimiter_field: field.name.b
+          target_field: target.field
+  - clusterername:
+      type: clusterer
+      specific_rules:
+          - tests/testdata/auto_tests/clusterer/rules/specific/
+      generic_rules:
+          - tests/testdata/auto_tests/clusterer/rules/generic/
+      output_field_name: target_field
diff --git a/tests/testdata/unit/auto_rule_tester/normalizer/regex_mapping.yml b/tests/testdata/unit/auto_rule_tester/normalizer/regex_mapping.yml
deleted file mode 100644
index a8922be72..000000000
--- a/tests/testdata/unit/auto_rule_tester/normalizer/regex_mapping.yml
+++ /dev/null
@@ -1,4 +0,0 @@
-RE_IP_PORT_CAP: .*(?P<IP>[\d.]+):(?P<PORT>\d+).*
-RE_FULL_CAP: (?P<ALL>.*)
-RE_ONLY_THIS_CAP: .*(?P<ONLY_THIS>Only this!).*
-RE_SWITCH_CAP: (?P<SECOND>Second).*(?P<FIRST>First)
\ No newline at end of file
diff --git a/tests/testdata/unit/auto_rule_tester/normalizer/rules/generic/auto_test_normalizer_match.json b/tests/testdata/unit/auto_rule_tester/normalizer/rules/generic/auto_test_normalizer_match.json
deleted file mode 100644
index e0e765794..000000000
--- a/tests/testdata/unit/auto_rule_tester/normalizer/rules/generic/auto_test_normalizer_match.json
+++ /dev/null
@@ -1,7 +0,0 @@
-[{
-  "filter": "some_field",
-  "normalize": {
-    "some_field": "some_normalized_field"
-  },
-  "description": "Test-rule with matching auto-test"
-}]
\ No newline at end of file
diff --git a/tests/testdata/unit/auto_rule_tester/normalizer/rules/generic/auto_test_normalizer_match_test.json b/tests/testdata/unit/auto_rule_tester/normalizer/rules/generic/auto_test_normalizer_match_test.json
deleted file mode 100644
index 1531c1d8d..000000000
--- a/tests/testdata/unit/auto_rule_tester/normalizer/rules/generic/auto_test_normalizer_match_test.json
+++ /dev/null
@@ -1,11 +0,0 @@
-[
-  {
-    "raw": {
-      "some_field": "doesnt_matter"
-    },
-    "processed": {
-      "some_field": "doesnt_matter",
-      "some_normalized_field": "doesnt_matter"
-    }
-  }
-]
\ No newline at end of file
diff --git a/tests/testdata/unit/auto_rule_tester/normalizer/rules/specific/auto_test_normalizer_match.json b/tests/testdata/unit/auto_rule_tester/normalizer/rules/specific/auto_test_normalizer_match.json
deleted file mode 100644
index e0e765794..000000000
--- a/tests/testdata/unit/auto_rule_tester/normalizer/rules/specific/auto_test_normalizer_match.json
+++ /dev/null
@@ -1,7 +0,0 @@
-[{
-  "filter": "some_field",
-  "normalize": {
-    "some_field": "some_normalized_field"
-  },
-  "description": "Test-rule with matching auto-test"
-}]
\ No newline at end of file
diff --git a/tests/testdata/unit/auto_rule_tester/normalizer/rules/specific/auto_test_normalizer_match_test.json b/tests/testdata/unit/auto_rule_tester/normalizer/rules/specific/auto_test_normalizer_match_test.json
deleted file mode 100644
index 1531c1d8d..000000000
--- a/tests/testdata/unit/auto_rule_tester/normalizer/rules/specific/auto_test_normalizer_match_test.json
+++ /dev/null
@@ -1,11 +0,0 @@
-[
-  {
-    "raw": {
-      "some_field": "doesnt_matter"
-    },
-    "processed": {
-      "some_field": "doesnt_matter",
-      "some_normalized_field": "doesnt_matter"
-    }
-  }
-]
\ No newline at end of file
diff --git a/tests/unit/util/test_auto_rule_tester.py b/tests/unit/util/test_auto_rule_tester.py
index 83a1da18d..ce6063533 100644
--- a/tests/unit/util/test_auto_rule_tester.py
+++ b/tests/unit/util/test_auto_rule_tester.py
@@ -3,41 +3,20 @@
 # pylint: disable=protected-access
 # pylint: disable=no-self-use
 # pylint: disable=broad-except
+# pylint: disable=line-too-long
 import logging
 from unittest import mock
 
 import pytest
 
-from yaml import safe_dump
 from logprep.util.auto_rule_tester import AutoRuleTester
 
 LOGGER = logging.getLogger()
 
 
 @pytest.fixture(name="auto_rule_tester")
-def fixture_auto_rule_tester(tmp_path):
-    config = {
-        "process_count": 1,
-        "timeout": 0.1,
-        "pipeline": [
-            {
-                "normalizer_name": {
-                    "type": "normalizer",
-                    "specific_rules": [
-                        "tests/testdata/unit/auto_rule_tester/normalizer/rules/specific"
-                    ],
-                    "generic_rules": [
-                        "tests/testdata/unit/auto_rule_tester/normalizer/rules/generic"
-                    ],
-                    "regex_mapping": "tests/testdata/unit/auto_rule_tester/normalizer/"
-                    "regex_mapping.yml",
-                }
-            }
-        ],
-    }
-    config_path = str(tmp_path / "logprep_config.yml")
-    with open(config_path, "w", encoding="utf-8") as config_file:
-        safe_dump(config, config_file)
+def fixture_auto_rule_tester():
+    config_path = "tests/testdata/config/config-auto-tests.yml"
     return AutoRuleTester(config_path)
 
 
@@ -113,7 +92,7 @@ def test_coverage_half_rule_files_have_tests(self, auto_rule_tester):
         coverage = auto_rule_tester._check_which_rule_files_miss_tests(rules_pn)
         assert coverage == 50
 
-    def test_does_not_run_if_no_rules_exist(self, auto_rule_tester):
+    def test_does_not_run_if_no_rules_exist(self, auto_rule_tester, capsys):
         rules_pn = {
             "normalizer_name": {
                 "rules": [],
@@ -123,12 +102,24 @@ def test_does_not_run_if_no_rules_exist(self, auto_rule_tester):
 
         try:
             auto_rule_tester._run_if_any_rules_exist(rules_pn)
+            captured = capsys.readouterr()
+            assert "There are no rules within any of the rules directories!" in captured.out
         except Exception as error:
             assert False, f"test_does_not_run_if_no_rules_exist has raised an exception: {error}"
 
     def test_does_run_if_rules_exist(self, auto_rule_tester):
+        auto_rule_tester._config_yml["pipeline"] = [
+            {
+                "normalizer": {
+                    "type": "normalizer",
+                    "specific_rules": ["tests/testdata/auto_tests/normalizer/rules/specific/"],
+                    "generic_rules": ["tests/testdata/auto_tests/normalizer/rules/generic/"],
+                    "regex_mapping": "tests/testdata/auto_tests/normalizer/regex_mapping.yml",
+                }
+            }
+        ]
         rules_pn = {
-            "normalizer_name": {
+            "normalizer": {
                 "rules": [
                     {
                         "file": "rule",
@@ -197,3 +188,52 @@ def test_list_comparison_specific_setup_called_on_add_rules_from_directory(
         mock_init_rules_list_comparison.assert_called_once()
         auto_rule_tester._add_rules_from_directory(processor, "specific_rules")
         assert mock_init_rules_list_comparison.call_count == 2
+
+    def test_full_auto_rule_test_run(self, auto_rule_tester, capsys):
+        with pytest.raises(SystemExit):
+            auto_rule_tester.run()
+        expected_rules_with_tests = [
+            "RULES WITH TESTS:",
+            "tests/testdata/auto_tests/labeler/rules/generic/auto_test_labeling_match.json",
+            "tests/testdata/auto_tests/labeler/rules/specific/auto_test_labeling_mismatch.json",
+            "tests/testdata/auto_tests/normalizer/rules/generic/auto_test_normalizer_match.json",
+            "tests/testdata/auto_tests/normalizer/rules/specific/auto_test_normalizer_mismatch.json",
+            "tests/testdata/auto_tests/dropper/rules/generic/drop_field.json",
+            "tests/testdata/auto_tests/dropper/rules/specific/drop_field.json",
+            "tests/testdata/auto_tests/pre_detector/rules/generic/auto_test_pre_detector_match.json",
+            "tests/testdata/auto_tests/pre_detector/rules/specific/auto_test_pre_detector_mismatch.json",
+            "tests/testdata/auto_tests/pseudonymizer/rules/specific/auto_test_pseudonymizer_mismatch.json",
+            "tests/testdata/auto_tests/pseudonymizer/rules/generic/auto_test_pseudonymizer_match.json",
+            "tests/testdata/auto_tests/template_replacer/rules/generic/template_replacer.json",
+            "tests/testdata/auto_tests/template_replacer/rules/specific/template_replacer.json",
+        ]
+        expected_rules_without_tests = [
+            "RULES WITHOUT TESTS:",
+            "tests/testdata/auto_tests/labeler/rules/specific/auto_test_labeling_no_test_.json",
+            "tests/testdata/auto_tests/normalizer/rules/specific/auto_test_normalizer_no_test_.json",
+            "tests/testdata/auto_tests/pre_detector/rules/specific/auto_test_pre_detector_no_test_.json",
+            "tests/testdata/auto_tests/pseudonymizer/rules/specific/auto_test_pseudonymizer_no_test_.json",
+        ]
+        expected_rules_with_custom_tests = [
+            "RULES WITH CUSTOM TESTS:",
+            "tests/testdata/auto_tests/clusterer/rules/specific/rule_with_custom_tests.yml",
+        ]
+        expected_overall_results = [
+            "Results:",
+            "Failed tests: 7",
+            "Successful tests: 8",
+            "Total tests: 15",
+            "Rule Test Coverage: 75.00%",
+            "Warnings: 2",
+        ]
+        captured = capsys.readouterr()
+        expected_sample_lines = (
+            expected_rules_with_tests
+            + expected_rules_without_tests
+            + expected_rules_with_custom_tests
+            + expected_overall_results
+        )
+        for line in expected_sample_lines:
+            assert line in captured.out
+
+        assert "inverse check, this text should not be in captured out" not in captured.out