Repository: fkie-cad/Logprep (language: python)

Issue:
Dissector Rule needs additional placeholders for matching strings at the beginning/end of input
It can occur, that matching rulesets need an additional placeholder `%{}` when matching defined strings at the end of an input event.
Trying to match:

```json
{"message" : "Jan 31 08:07:43 hostname sshd[18539]: Connection closed by authenticating user username 5.5.5.5 port 49588 [preauth]"}
```

  * with the following rule:

```yml
filter: "message"
dissector:
  mapping:      
    message: "%{timestamp} %{+(\ )timestamp} %{+(\ )timestamp} %{hostname} sshd[%{}]: %{sshd_event} %{+(\ )sshd_event} by authenticating user %{user} %{remotehost} %{} %{} [%{sshd_status}]"
```

**Expected behavior**
The desired field `sshd_status` should get the value `preauth` after the pipeline. 

**Current behavior**
An Error is raised that the given pattern doesn't comply with the given Regex structure:

```python
ValueError: ("'mapping' must match regex '^(([^%]+))?(%\\\\{((\\\\+\\\\([^%]+\\\\))|(\\\\+(?!\\\\([^%]))|([&\\\\?]))?[^\\\\}\\\\%\\\\{\\\\}\\\\+]*(\\\\/\\\\d*)?\\\\}([^%]+))+%\\\\{((\\\\+\\\\([^%]+\\\\))|(\\\\+(?!\\\\([^%]))|([&\\\\?]))?[^\\\\}\\\\%\\\\{\\\\}\\\\+]*(\\\\/\\\\d*)?\\\\}$' ('%{timestamp} %{+( )timestamp} %{+( )timestamp} %{hostname} sshd[%{}]: %{sshd_event} %{+( )sshd_event} by authenticating user %{user} %{remotehost} %{} %{} [%{sshd_status}]' doesn't)", Attribute(name='mapping', default=Factory(factory=<class 'dict'>, takes_self=False), validator=_AndValidator(_validators=(<instance_of validator for type <class 'dict'>>, <deep_mapping validator for objects mapping <instance_of validator for type <class 'str'>> to <matches_re validator for pattern re.compile('^(([^%]+))?(%\\{((\\+\\([^%]+\\))|(\\+(?!\\([^%]))|([&\\?]))?[^\\}\\%\\{\\}\\+]*(\\/\\d*)?\\}([^%]+))+%\\{((\\+\\([^%]+\\))|(\\+(?!\\([^%]))|([&\\?]))?[^\\}\\%\\{\\}\\+]*(\\/\\d*)?\\}$')>>)), repr=True, eq=True, eq_key=None, order=True, order_key=None, hash=None, init=True, metadata=mappingproxy({}), type=<class 'dict'>, converter=None, kw_only=True, inherited=False, on_setattr=None), re.compile('^(([^%]+))?(%\\{((\\+\\([^%]+\\))|(\\+(?!\\([^%]))|([&\\?]))?[^\\}\\%\\{\\}\\+]*(\\/\\d*)?\\}([^%]+))+%\\{((\\+\\([^%]+\\))|(\\+(?!\\([^%]))|([&\\?]))?[^\\}\\%\\{\\}\\+]*(\\/\\d*)?\\}$'), '%{timestamp} %{+( )timestamp} %{+( )timestamp} %{hostname} sshd[%{}]: %{sshd_event} %{+( )sshd_event} by authenticating user %{user} %{remotehost} %{} %{} [%{sshd_status}]')
```

The error can be mitigated by adding `%{}` to the dissector pattern which is not very intuitive:

```
%{timestamp} %{+(\ )timestamp} %{+(\ )timestamp} %{hostname} sshd[%{}]: %{sshd_event} %{+(\ )sshd_event} by authenticating user %{user} %{remotehost} %{} %{} [%{sshd_status}]%{}"
```

**Environment**

Logprep version: 4.0.0+70.gdc33637
Python version: 3.9.16


**Possible solution**
The trailing placeholders should be added by the processor itself when it's necessary as this behaviour is really counter-intuitive to give the user a comprehensible pattern logic to match individual input events.

**Possible implementation**
I didn't look into the implementation yet. Maybe this behaviour is desired and this issue could be closed, otherwise I could provide a suggestion for an implementation.



Discussion / hints:

Fix patch:
diff --git a/logprep/processor/dissector/rule.py b/logprep/processor/dissector/rule.py
index 4df521528..2d209609e 100644
--- a/logprep/processor/dissector/rule.py
+++ b/logprep/processor/dissector/rule.py
@@ -119,7 +119,7 @@ class Config(FieldManagerRule.Config):
                 validators.deep_mapping(
                     key_validator=validators.instance_of(str),
                     value_validator=validators.matches_re(
-                        rf"^({DELIMETER})?({DISSECT}{DELIMETER})+{DISSECT}$"
+                        rf"^({DELIMETER})?({DISSECT}{DELIMETER})+({DISSECT})?$"
                     ),
                 ),
             ],


Test patch:
diff --git a/tests/unit/processor/dissector/test_dissector.py b/tests/unit/processor/dissector/test_dissector.py
index 5f49694ed..94e1c1479 100644
--- a/tests/unit/processor/dissector/test_dissector.py
+++ b/tests/unit/processor/dissector/test_dissector.py
@@ -266,7 +266,7 @@
             "filter": "message",
             "dissector": {
                 "mapping": {
-                    "message": "%{+( )extracted/4} %{+( )extracted/3} %{+( )extracted/2} %{+extracted/1}"
+                    "message": "%{+( )extracted/4} %{+( )extracted/3} %{+( )extracted/2} %{+extracted/1}"  # pylint: disable=line-too-long
                 }
             },
         },
@@ -279,7 +279,7 @@
             "filter": "message",
             "dissector": {
                 "mapping": {
-                    "message": "%{+( )extracted/4} %{+( )extracted/3} %{+( )extracted/2} %{+( )extracted/1}"
+                    "message": "%{+( )extracted/4} %{+( )extracted/3} %{+( )extracted/2} %{+( )extracted/1}"  # pylint: disable=line-too-long
                 }
             },
         },
@@ -465,7 +465,7 @@
         "Appending with special field separator",
         {
             "filter": "message",
-            "dissector": {"mapping": {"message": "INFO#%{+(\()date}#%{+(\))date}#MOREINFO%{}"}},
+            "dissector": {"mapping": {"message": r"INFO#%{+(\()date}#%{+(\))date}#MOREINFO%{}"}},
         },
         {
             "message": "INFO#2022 12 06 15:12:30:534#+0100#MOREINFO",
@@ -475,6 +475,12 @@
             "date": "(2022 12 06 15:12:30:534)+0100",
         },
     ),
+    (
+        "Dissection with delimeter ending",
+        {"filter": "message", "dissector": {"mapping": {"message": "this is %{target}."}}},
+        {"message": "this is the message."},
+        {"message": "this is the message.", "target": "the message"},
+    ),
 ]
 failure_test_cases = [  # testcase, rule, event, expected
     (
diff --git a/tests/unit/processor/dissector/test_dissector_rule.py b/tests/unit/processor/dissector/test_dissector_rule.py
index 5bd561519..7a0d147bb 100644
--- a/tests/unit/processor/dissector/test_dissector_rule.py
+++ b/tests/unit/processor/dissector/test_dissector_rule.py
@@ -82,16 +82,16 @@ def test_create_from_dict_returns_dissector_rule(self):
                     "filter": "message",
                     "dissector": {"mapping": {"field": "%{ts}:"}},
                 },
-                ValueError,
-                "must match regex",
+                None,
+                None,
             ),
             (
                 {
                     "filter": "message",
                     "dissector": {"mapping": {"field": "%{ts}:%{ts} "}},
                 },
-                ValueError,
-                "must match regex",
+                None,
+                None,
             ),
             (
                 {