Repository: fail2ban/fail2ban (language: python)

Issue:
[FR]: Exim: lines from encrypted connections (TLS, log entries X= and CV=) not matched
### Environment:

- Fail2Ban version <!-- including any possible distribution suffixes --> : 0.10.2-2.1
- OS, including release name/version : Debian 10.12 (buster)

#### Service, project or product which log or journal should be monitored

- Name of filter or jail in Fail2Ban (if already exists) : exim, exim-spam
- Service, project or product name, including release name/version : exim4 4.92-8+deb10u6

#### Any additional information

Exim log lines are _normally_ matched correctly, but not when the remote has successfully issued a STARTTLS command. Log lines then have `X=(TLS cipher suite)` and `CV=(certificate validation status)` entries that are not currently matched.

### Relevant lines from monitored log files:

```
2022-04-22 06:40:33 H=(pmhtcr) [<HOST>] F=<<EMAIL>> rejected RCPT <<EMAIL>>: relay not permitted
2022-04-22 00:23:03 H=(3E64W6mP) [<HOST>] X=TLS1.2:ECDHE_ECDSA_AES_256_GCM_SHA384:256 CV=no F=<<EMAIL>> rejected RCPT <<EMAIL>>: relay not permitted
```

(I redacted the IP and email addresses in those lines, using `<HOST>` and `<EMAIL>` placeholders, respectively)

The second line doesn't match because `exim-common.conf` only looks for `I=(local interface used)`, `U=(local user or RFC 1413 identity)`, `P=(protocol used)` and `F=(sender address)` entries:

```
host_info_suf = (?::\d+)?(?: I=\[\S+\](:\d+)?)?(?: U=\S+)?(?: P=e?smtp)?(?: F=(?:<>|[^@]+@\S+))?\s
```

The `X=` and `CV=` entries fit between `P=` and `F=`; the following definition for `host_info_suf` fixes the issue for me (added to my filters.d/exim-common.local file):

```
host_info_suf = (?::\d+)?(?: I=\[\S+\](:\d+)?)?(?: U=\S+)?(?: P=e?smtps?)?(?: X=\S+)?(?: CV=(?:yes|dane|no))?(?: F=(?:<>|[^@]+@\S+))?\s
```

I used https://www.exim.org/exim-html-current/doc/html/spec_html/ch-log_files.html as the basis for the field definitions.

### Further considerations

* See the relevant [source code in exim4's `src/receive.c`](https://github.com/Exim/exim/blob/97c83a31f1269ac154408a571b9207c6f3552fc9/src/src/receive.c#L4044-L4061)
* I disregarded the `DN=` and `SNI=` entries, as they are optional, while `tls_cipher` and `tls_certificate_verified` are on by default.
* On a separate note: the `P=(protocol used)` regex should probably be extended to also cover `esmpts` (extended  smpt over SSL). I tacked on a `s?` to my `(?: P=e?smtps?)?` non-capturing group, above.


Discussion / hints:

Fix patch:
diff --git a/ChangeLog b/ChangeLog
index e0358d030c..19a4446070 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -28,9 +28,11 @@ ver. 1.0.3-dev-1 (20??/??/??) - development nightly edition
   (value read from `/proc/sys/net/ipv6/conf/all/disable_ipv6`) if available, otherwise seeks over local IPv6 from network interfaces
   if available for platform and uses DNS to find local IPv6 as a fallback only
 * improve `ignoreself` by considering all local addresses from network interfaces additionally to IPs from hostnames (gh-3132)
+* `filter.d/exim.conf`:
+  - rewrite host line regex for all varied exim's log_selector states (gh-3263)
+  - fixed "dropped: too many ..." regex, also matching unrecognized commands now (gh-3502)
 * `action.d/mikrotik.conf` - new action for mikrotik routerOS, adds and removes entries from address lists on the router (gh-2860)
 * `action.d/smtp.py` - added optional support for TLS connections via the `ssl` arg.
-* `filter.d/exim.conf` - fixed "dropped: too many ..." regex, also matching unrecognized commands now (gh-3502)
 * `filter.d/nginx-forbidden.conf` - new filter to ban forbidden locations, e. g. using `deny` directive (gh-2226)
 * `filter.d/sshd.conf`:
   - avoid double counting for "maximum authentication attempts exceeded" (gh-3502)
diff --git a/config/filter.d/exim-common.conf b/config/filter.d/exim-common.conf
index 36644e9433..e16debf99d 100644
--- a/config/filter.d/exim-common.conf
+++ b/config/filter.d/exim-common.conf
@@ -9,12 +9,33 @@ after = exim-common.local
 
 [Definition]
 
-host_info_pre = (?:H=([\w.-]+ )?(?:\(\S+\) )?)?
-host_info_suf = (?::\d+)?(?: I=\[\S+\](:\d+)?)?(?: U=\S+)?(?: P=e?smtp)?(?: F=(?:<>|[^@]+@\S+))?\s
-host_info = %(host_info_pre)s\[<HOST>\]%(host_info_suf)s
+_fields_grp = (?: (?!H=)[A-Za-z]{1,4}(?:=\S+)?)*
+host_info = %(_fields_grp)s (?:H=)?(?:[\w.-]+)? ?(?:\(\S+\))? ?\[<ADDR>\](?::\d+)?%(_fields_grp)s
 pid = (?: \[\d+\]| \w+ exim\[\d+\]:)?
 
-# DEV Notes:
-# From exim source code: ./src/receive.c:add_host_info_for_log
-#
-# Author:  Daniel Black
+
+# DEV Notes
+# ------------
+# Host string happens:
+# H=[ip address]
+# H=(helo_name) [ip address]
+# H=host_name [ip address]
+# H=host_name (helo_name) [ip address]
+# flags H=host_name (helo_name) [ip address] flags
+# where only [ip address] always visible, ignore ident
+# From exim source code:
+#   src/src/host.c:host_and_ident()
+#   src/receive.c:add_host_info_for_log()
+
+# Substitution of `_fields_grp` bypasses all flags but H
+# Summary of Fields in Log Lines depending on log_selector
+# https://www.exim.org/exim-html-current/doc/html/spec_html/ch-log_files.html
+# at version exim-4.97.1
+# ---
+
+# Authors:
+#   Cyril Jaquier
+#   Daniel Black (rewrote with strong regexs)
+#   Sergey G. Brester aka sebres (optimization, rewrite to prefregex, reviews)
+#   Martin O'Neal (added additional regexs to detect authentication failures, protocol errors, and drops)
+#   Varlamov Vladimir (host line definition)
diff --git a/config/filter.d/exim-spam.conf b/config/filter.d/exim-spam.conf
index 1038cc0b91..0a283f4678 100644
--- a/config/filter.d/exim-spam.conf
+++ b/config/filter.d/exim-spam.conf
@@ -26,9 +26,9 @@ before = exim-common.conf
 
 [Definition]
 
-failregex =  ^%(pid)s \S+ F=(<>|\S+@\S+) %(host_info)srejected by local_scan\(\): .{0,256}$
-             ^%(pid)s %(host_info)sF=(<>|[^@]+@\S+) rejected RCPT [^@]+@\S+: .*dnsbl.*\s*$
-             ^%(pid)s \S+ %(host_info)sF=(<>|[^@]+@\S+) rejected after DATA: This message contains a virus \(\S+\)\.\s*$
+failregex =  ^%(pid)s \S+%(host_info)s rejected by local_scan\(\): .{0,256}$
+             ^%(pid)s%(host_info)s rejected RCPT [^@]+@\S+: .*dnsbl.*\s*$
+             ^%(pid)s \S+%(host_info)s rejected after DATA: This message contains a virus \(\S+\)\.\s*$
              ^%(pid)s \S+ SA: Action: flagged as Spam but accepted: score=\d+\.\d+ required=\d+\.\d+ \(scanned in \d+/\d+ secs \| Message-Id: \S+\)\. From \S+ \(host=\S+ \[<HOST>\]\) for <honeypot>$
              ^%(pid)s \S+ SA: Action: silently tossed message: score=\d+\.\d+ required=\d+\.\d+ trigger=\d+\.\d+ \(scanned in \d+/\d+ secs \| Message-Id: \S+\)\. From \S+ \(host=(\S+ )?\[<HOST>\]\) for \S+$
 
@@ -43,8 +43,6 @@ ignoreregex =
 
 honeypot = trap@example.com
 
-# DEV Notes:
-# The %(host_info) definition contains a <HOST> match
-#
-# Author: Cyril Jaquier
-#         Daniel Black (rewrote with strong regexs)
+# DEV Notes
+# -----------
+# The %(host_info) definition contains a <HOST> match. No space before. See exim-common.conf
diff --git a/config/filter.d/exim.conf b/config/filter.d/exim.conf
index 854c8ba7ce..7d4144a761 100644
--- a/config/filter.d/exim.conf
+++ b/config/filter.d/exim.conf
@@ -14,16 +14,16 @@ before = exim-common.conf
 [Definition]
 
 # Fre-filter via "prefregex" is currently inactive because of too different failure syntax in exim-log (testing needed):
-#prefregex = ^%(pid)s <F-CONTENT>\b(?:\w+ authenticator failed|([\w\-]+ )?SMTP (?:(?:call|connection) from|protocol(?: synchronization)? error)|no MAIL in|(?:%(host_info_pre)s\[[^\]]+\]%(host_info_suf)s(?:sender verify fail|rejected RCPT|dropped|AUTH command))).+</F-CONTENT>$
-
-failregex = ^%(pid)s %(host_info)ssender verify fail for <\S+>: (?:Unknown user|Unrouteable address|all relevant MX records point to non-existent hosts)\s*$
-            ^%(pid)s \w+ authenticator failed for (?:[^\[\( ]* )?(?:\(\S*\) )?\[<HOST>\](?::\d+)?(?: I=\[\S+\](:\d+)?)?: 535 Incorrect authentication data( \(set_id=.*\)|: \d+ Time\(s\))?\s*$
-            ^%(pid)s %(host_info)srejected RCPT [^@]+@\S+: (?:relay not permitted|Sender verify failed|Unknown user|Unrouteable address)\s*$
-            ^%(pid)s SMTP protocol synchronization error \([^)]*\): rejected (?:connection from|"\S+") %(host_info)s(?:next )?input=".*"\s*$
-            ^%(pid)s SMTP call from (?:[^\[\( ]* )?%(host_info)sdropped: too many (?:(?:nonmail|unrecognized) commands|syntax or protocol errors)
-            ^%(pid)s SMTP protocol error in "[^"]+(?:"+[^"]*(?="))*?" %(host_info)sAUTH command used when not advertised\s*$
-            ^%(pid)s no MAIL in SMTP connection from (?:[^\[\( ]* )?(?:\(\S*\) )?%(host_info)sD=\d\S*s(?: C=\S*)?\s*$
-            ^%(pid)s (?:[\w\-]+ )?SMTP connection from (?:[^\[\( ]* )?(?:\(\S*\) )?%(host_info)sclosed by DROP in ACL\s*$
+#prefregex = ^%(pid)s <F-CONTENT>\b(?:\w+ authenticator failed|([\w\-]+ )?SMTP (?:(?:call|connection) from|protocol(?: synchronization)? error)|no MAIL in|(?:%(host_info)s(?:sender verify fail|rejected RCPT|dropped|AUTH command))).+</F-CONTENT>$
+
+failregex = ^%(pid)s%(host_info)s sender verify fail for <\S+>: (?:Unknown user|Unrouteable address|all relevant MX records point to non-existent hosts)\s*$
+            ^%(pid)s \w+ authenticator failed for%(host_info)s: 535 Incorrect authentication data(?: \(set_id=.*\)|: \d+ Time\(s\))?\s*$
+            ^%(pid)s%(host_info)s rejected RCPT [^@]+@\S+: (?:relay not permitted|Sender verify failed|Unknown user|Unrouteable address)\s*$
+            ^%(pid)s SMTP protocol synchronization error \([^)]*\): rejected (?:connection from|"\S+")%(host_info)s (?:next )?input=".*"\s*$
+            ^%(pid)s SMTP call from%(host_info)s dropped: too many (?:(?:nonmail|unrecognized) commands|syntax or protocol errors)
+            ^%(pid)s SMTP protocol error in "[^"]+(?:"+[^"]*(?="))*?"%(host_info)s AUTH command used when not advertised\s*$
+            ^%(pid)s no MAIL in SMTP connection from%(host_info)s
+            ^%(pid)s (?:[\w\-]+ )?SMTP connection from%(host_info)s closed by DROP in ACL\s*$
             <mdre-<mode>>
 
 mdre-aggressive = ^%(pid)s no host name found for IP address <HOST>$
@@ -42,13 +42,10 @@ mode = normal
 
 ignoreregex = 
 
-# DEV Notes:
-# The %(host_info) definition contains a <HOST> match
+# DEV Notes
+# -----------
+# The %(host_info) definition contains a <HOST> match. No space before. See exim-common.conf
 #
 # SMTP protocol synchronization error \([^)]*\)  <- This needs to be non-greedy
 # to void capture beyond ")" to avoid a DoS Injection vulnerability as input= is
 # user injectable data.
-#
-# Author: Cyril Jaquier
-#         Daniel Black (rewrote with strong regexs)
-#         Martin O'Neal (added additional regexs to detect authentication failures, protocol errors, and drops)


Test patch:
diff --git a/fail2ban/tests/files/logs/exim b/fail2ban/tests/files/logs/exim
index 04a47af70e..94bca0852f 100644
--- a/fail2ban/tests/files/logs/exim
+++ b/fail2ban/tests/files/logs/exim
@@ -106,3 +106,8 @@
 2017-12-03 08:32:00 no host name found for IP address 192.0.2.8
 # failJSON: { "time": "2017-12-03T08:51:35", "match": true , "host": "192.0.2.9", "desc": "no IP found for host" }
 2017-12-03 08:51:35 no IP address found for host test.example.com (during SMTP connection from [192.0.2.9])
+
+# failJSON: { "time": "2024-03-21T19:26:06", "match": true , "host": "194.169.175.1" }
+2024-03-21 19:26:06 dovecot_login authenticator failed for (User) [194.169.175.1]:21298 I=[22.33.44.55]:465 Ci=30416: 535 Incorrect authentication data (set_id=uaf589@example.com)
+# failJSON: { "time": "2024-03-21T09:18:51", "match": true , "host": "9.12.1.21" }
+2024-03-21 09:18:51 H=m05.horp.tld [9.12.1.21]:43030 I=[194.169.175.2]:25 Ci=7326 CV=no SNI=mail.leone.tld F=<user@example.tld> rejected RCPT <locus@leone.tld>: relay not permitted