Repository: fail2ban/fail2ban

Issue:
fail2ban-regex broken against journal in 0.9.6
### Environment:
- Fail2Ban version (including any possible distribution suffixes): 0.9.6
- OS, including release name/version: Fedora 25
- [ X] Fail2Ban installed via OS/distribution mechanisms
- [ ] You have not applied any additional foreign patches to the codebase
- [ ] Some customizations were done to the configuration (provide details below is so)

### The issue:

```
breaks fail2ban-regex when testing against systemd-journal.

fail2ban-regex "systemd-journal" /etc/fail2ban/filter.d/sshd.conf

Running tests
=============

Use   failregex filter file : sshd, basedir: /etc/fail2ban
Use         maxlines : 10
Use    journal match : _SYSTEMD_UNIT=sshd.service + _COMM=sshd

Traceback (most recent call last):
  File "/bin/fail2ban-regex", line 34, in <module>
    exec_command_line()
  File "/usr/lib/python3.5/site-packages/fail2ban/client/fail2banregex.py", line 599, in exec_command_line
    if not fail2banRegex.start(opts, args):
  File "/usr/lib/python3.5/site-packages/fail2ban/client/fail2banregex.py", line 538, in start
    self.process(test_lines)
  File "/usr/lib/python3.5/site-packages/fail2ban/client/fail2banregex.py", line 370, in process
    for line_no, line in enumerate(test_lines):
  File "/usr/lib/python3.5/site-packages/fail2ban/client/fail2banregex.py", line 93, in journal_lines_gen
    yield FilterSystemd.formatJournalEntry(entry)
TypeError: formatJournalEntry() missing 1 required positional argument: 'logentry'
```
#### Any additional information

https://bodhi.fedoraproject.org/updates/FEDORA-2017-33e3a599fa
dovecot filter doesn't match
### Environment:

- Fail2Ban version 0.9.3-1 all
- Ubuntu 16.04LTS, xenial
- dovecot  2.2.22
- [x] Fail2Ban installed via OS/distribution mechanisms
- [x] You have not applied any additional foreign patches to the codebase
- [ ] Some customizations were done to the configuration (provide details below is so)

### The issue:

Following log lines from mail.log are not matched by current filter

```
Nov 26 11:11:21 hostname dovecot: imap-login: Disconnected: Too many invalid commands (tried to use disallowed plaintext auth): user=<willi>, rip=xx.xx.xxx.xx, lip=yy.yyy.yy.yy, session=<S5dIdTFCDKUWWMbU>

Nov 26 11:12:19 hostname dovecot: imap-login: Disconnected: Too many invalid commands (auth failed, 1 attempts in 17 secs): user=<willi>, method=PLAIN, rip=xx.xx.xxx.xx, lip=yy.yyy.yy.yy, TLS, session=<g3ZKeDECFqlWWMbU>
```
Relevant part: "Too many invalid commands"

#### Any additional information
Changing the relevant part of failregex to:

```
failregex = ^%(__prefix_line)s(pop3|imap)-login: (Info: )?(Aborted login|Disconnected)(: Inactivity|: Too many invalid commands)? \(((auth failed, \d+ attempts)( in \d+ secs)?|tried to use (disabled|disallowed) \S+ auth)\):( user=<\S*>,)?( method=\S+,)? rip=<HOST>(, lip=(\d{1,3}\.){3}\d{1,3})?(, TLS( handshaking(: SSL_accept\(\) failed: error:[\dA-F]+:SSL routines:[TLS\d]+_GET_CLIENT_HELLO:unknown protocol)?)?(: Disconnected)?)?(, session=<\S+>)?\s*$
```

helps to catch them all. I'm no regex expert, so better check it twice. 


Discussion:

Resolving patch:
diff --git a/fail2ban/client/fail2banregex.py b/fail2ban/client/fail2banregex.py
--- a/fail2ban/client/fail2banregex.py
+++ b/fail2ban/client/fail2banregex.py
@@ -41,12 +41,12 @@
 from ConfigParser import NoOptionError, NoSectionError, MissingSectionHeaderError
 
 try: # pragma: no cover
-	from systemd import journal
 	from ..server.filtersystemd import FilterSystemd
 except ImportError:
-	journal = None
+	FilterSystemd = None
 
 from ..version import version
+from .jailreader import JailReader
 from .filterreader import FilterReader
 from ..server.filter import Filter, FileContainer
 from ..server.failregex import RegexException
@@ -80,7 +80,7 @@ def pprint_list(l, header=None):
 		s = ''
 	output( s + "|  " + "\n|  ".join(l) + '\n`-' )
 
-def journal_lines_gen(myjournal): # pragma: no cover
+def journal_lines_gen(flt, myjournal): # pragma: no cover
 	while True:
 		try:
 			entry = myjournal.get_next()
@@ -88,7 +88,7 @@ def journal_lines_gen(myjournal): # pragma: no cover
 			continue
 		if not entry:
 			break
-		yield FilterSystemd.formatJournalEntry(entry)
+		yield flt.formatJournalEntry(entry)
 
 def get_opt_parser():
 	# use module docstring for help output
@@ -524,25 +524,22 @@ def start(self, opts, args):
 			except IOError as e:
 				output( e )
 				return False
-		elif cmd_log == "systemd-journal": # pragma: no cover
-			if not journal:
+		elif cmd_log.startswith("systemd-journal"): # pragma: no cover
+			if not FilterSystemd:
 				output( "Error: systemd library not found. Exiting..." )
 				return False
-			myjournal = journal.Reader(converters={'__CURSOR': lambda x: x})
+			output( "Use         systemd journal" )
+			output( "Use         encoding : %s" % self._encoding )
+			backend, beArgs = JailReader.extractOptions(cmd_log)
+			flt = FilterSystemd(None, **beArgs)
+			flt.setLogEncoding(self._encoding)
+			myjournal = flt.getJournalReader()
 			journalmatch = self._journalmatch
 			self.setDatePattern(None)
 			if journalmatch:
-				try:
-					for element in journalmatch:
-						if element == "+":
-							myjournal.add_disjunction()
-						else:
-							myjournal.add_match(element)
-				except ValueError:
-					output( "Error: Invalid journalmatch: %s" % shortstr(" ".join(journalmatch)) )
-					return False
+				flt.addJournalMatch(journalmatch)
 			output( "Use    journal match : %s" % " ".join(journalmatch) )
-			test_lines = journal_lines_gen(myjournal)
+			test_lines = journal_lines_gen(flt, myjournal)
 		else:
 			output( "Use      single line : %s" % shortstr(cmd_log) )
 			test_lines = [ cmd_log ]
diff --git a/fail2ban/server/filtersystemd.py b/fail2ban/server/filtersystemd.py
--- a/fail2ban/server/filtersystemd.py
+++ b/fail2ban/server/filtersystemd.py
@@ -178,6 +178,14 @@ def delJournalMatch(self, match=None):
 	def getJournalMatch(self):
 		return self.__matches
 
+	##
+	# Get journal reader
+	#
+	# @return journal reader
+
+	def getJournalReader(self):
+		return self.__journal
+
 	##
 	# Format journal log entry into syslog style
 	#


Test patch:
diff --git a/fail2ban/tests/files/logs/apache-modsecurity b/fail2ban/tests/files/logs/apache-modsecurity
--- a/fail2ban/tests/files/logs/apache-modsecurity
+++ b/fail2ban/tests/files/logs/apache-modsecurity
@@ -1,5 +1,5 @@
 # failJSON: { "time": "2013-12-23T13:12:31", "match": true , "host": "173.255.225.101" }
 [Mon Dec 23 13:12:31 2013] [error] [client 173.255.225.101] ModSecurity:  [file "/etc/httpd/modsecurity.d/activated_rules/modsecurity_crs_21_protocol_anomalies.conf"] [line "47"] [id "960015"] [rev "1"] [msg "Request Missing an Accept Header"] [severity "NOTICE"] [ver "OWASP_CRS/2.2.8"] [maturity "9"] [accuracy "9"] [tag "OWASP_CRS/PROTOCOL_VIOLATION/MISSING_HEADER_ACCEPT"] [tag "WASCTC/WASC-21"][tag "OWASP_TOP_10/A7"] [tag "PCI/6.5.10"] Access denied with code 403 (phase 2). Operator EQ matched 0 at REQUEST_HEADERS. [hostname "www.mysite.net"] [uri "/"] [unique_id "Urf@f12qgHIAACrFOlgAAABA"]
 
-# failJSON: { "time": "2013-12-28T09:18:05", "match": true , "host": "32.65.254.69" }
-[Sat Dec 28 09:18:05 2013] [error] [client 32.65.254.69] ModSecurity:  [file "/etc/httpd/modsecurity.d/10_asl_rules.conf"] [line "635"] [id "340069"] [rev "4"] [msg "Atomicorp.com UNSUPPORTED DELAYED Rules: Web vulnerability scanner"] [severity "CRITICAL"] Access denied with code 403 (phase 2). Pattern match "(?:nessus(?:_is_probing_you_|test)|^/w00tw00t\\\\.at\\\\.)" at REQUEST_URI. [hostname "192.81.249.191"] [uri "/w00tw00t.at.blackhats.romanian.anti-sec:)"] [unique_id "4Q6RdsBR@b4AAA65LRUAAAAA"] 
+# failJSON: { "time": "2013-12-28T09:18:05", "match": true , "host": "32.65.254.69", "desc": "additional entry (and exact one space)" }
+[Sat Dec 28 09:18:05 2013] [error] [client 32.65.254.69] ModSecurity: [file "/etc/httpd/modsecurity.d/10_asl_rules.conf"] [line "635"] [id "340069"] [rev "4"] [msg "Atomicorp.com UNSUPPORTED DELAYED Rules: Web vulnerability scanner"] [severity "CRITICAL"] Access denied with code 403 (phase 2). Pattern match "(?:nessus(?:_is_probing_you_|test)|^/w00tw00t\\\\.at\\\\.)" at REQUEST_URI. [hostname "192.81.249.191"] [uri "/w00tw00t.at.blackhats.romanian.anti-sec:)"] [unique_id "4Q6RdsBR@b4AAA65LRUAAAAA"] 
diff --git a/fail2ban/tests/files/logs/dovecot b/fail2ban/tests/files/logs/dovecot
--- a/fail2ban/tests/files/logs/dovecot
+++ b/fail2ban/tests/files/logs/dovecot
@@ -73,3 +73,8 @@ Jul 02 13:49:32 hostname dovecot[442]: pop3-login: Disconnected (no auth attempt
 
 # failJSON: { "time": "2005-03-23T06:10:52", "match": true , "host": "52.37.139.121" }
 Mar 23 06:10:52 auth: Info: ldap(dog,52.37.139.121,): invalid credentials
+
+# failJSON: { "time": "2005-07-26T11:11:21", "match": true , "host": "192.0.2.1" }
+Jul 26 11:11:21 hostname dovecot: imap-login: Disconnected: Too many invalid commands (tried to use disallowed plaintext auth): user=<test>, rip=192.0.2.1, lip=192.168.1.1, session=<S5dIdTFCDKUWWMbU>
+# failJSON: { "time": "2005-07-26T11:12:19", "match": true , "host": "192.0.2.2" }
+Jul 26 11:12:19 hostname dovecot: imap-login: Disconnected: Too many invalid commands (auth failed, 1 attempts in 17 secs): user=<test>, method=PLAIN, rip=192.0.2.2, lip=192.168.1.1, TLS, session=<g3ZKeDECFqlWWMbU>
diff --git a/fail2ban/tests/files/logs/mongodb-auth b/fail2ban/tests/files/logs/mongodb-auth
new file mode 100644
--- /dev/null
+++ b/fail2ban/tests/files/logs/mongodb-auth
@@ -0,0 +1,30 @@
+# failJSON: { "match": false }
+2016-11-20T00:04:00.110+0100 [conn1] Failed to authenticate root@admin with mechanism MONGODB-CR: AuthenticationFailed UserNotFound Could not find user root@admin
+# failJSON: { "time": "2016-11-20T00:04:00", "match": true , "host": "192.0.2.35" }
+2016-11-20T00:04:00.111+0100 [conn1] end connection 192.0.2.35:53276 (0 connections now open)
+
+# failJSON: { "match": false }
+2016-11-20T00:24:00.110+0100 [conn5] Failed to authenticate root@admin with mechanism MONGODB-CR: AuthenticationFailed UserNotFound Could not find user root@admin
+# failJSON: { "time": "2016-11-20T00:24:00", "match": true , "host": "192.0.2.171" }
+2016-11-20T00:24:00.111+0100 [conn5] end connection 192.0.2.171:53276 (0 connections now open)
+
+# failJSON: { "match": false }
+2016-11-20T00:24:00.110+0100 [conn334] Failed to authenticate root@admin with mechanism MONGODB-CR: AuthenticationFailed key mismatch
+# failJSON: { "time": "2016-11-20T00:24:00", "match": true , "host": "192.0.2.176" }
+2016-11-20T00:24:00.111+0100 [conn334] end connection 192.0.2.176:53276 (0 connections now open)
+
+# failJSON: { "match": false }
+2016-11-20T00:24:00.110+0100 [conn56] Failed to authenticate root@admin with mechanism MONGODB-CR: AuthenticationFailed key mismatch
+# failJSON: { "time": "2016-11-20T00:24:00", "match": true , "host": "192.0.2.1" }
+2016-11-20T00:24:00.111+0100 [conn56] end connection 192.0.2.1:53276 (0 connections now open)
+
+# failJSON: { "match": false }
+2016-11-20T12:54:02.370+0100 [initandlisten] connection accepted from 127.0.0.1:58774 #2261 (1 connection now open)
+# failJSON: { "match": false }
+2016-11-20T12:54:02.370+0100 [conn2261] end connection 127.0.0.1:58774 (0 connections now open)
+
+# failJSON: { "match": false }
+2016-11-20T13:07:49.781+0100 [conn2271]  authenticate db: admin { authenticate: 1, nonce: "xxx", user: "root", key: "xxx" }
+# failJSON: { "time": "2016-11-20T13:07:49", "match": false , "host": "192.0.2.178" }
+2016-11-20T13:07:49.834+0100 [conn2271] end connection 192.0.2.178:60268 (3 connections now open)
+
diff --git a/fail2ban/tests/files/logs/sshd b/fail2ban/tests/files/logs/sshd
--- a/fail2ban/tests/files/logs/sshd
+++ b/fail2ban/tests/files/logs/sshd
@@ -19,8 +19,10 @@ Jan  5 01:31:41 www sshd[1643]: ROOT LOGIN REFUSED FROM 1.2.3.4
 Jan  5 01:31:41 www sshd[1643]: ROOT LOGIN REFUSED FROM ::ffff:1.2.3.4
 
 #4
-# failJSON: { "time": "2005-07-20T14:42:11", "match": true , "host": "211.114.51.213" }
-Jul 20 14:42:11 localhost sshd[22708]: Invalid user ftp from 211.114.51.213
+# failJSON: { "time": "2005-07-20T14:42:11", "match": true , "host": "192.0.2.1", "desc": "Invalid user" }
+Jul 20 14:42:11 localhost sshd[22708]: Invalid user ftp from 192.0.2.1
+# failJSON: { "time": "2005-07-20T14:42:12", "match": true , "host": "192.0.2.2", "desc": "Invalid user with port" }
+Jul 20 14:42:12 localhost sshd[22708]: Invalid user ftp from 192.0.2.2 port 37220
 
 #5 new filter introduced after looking at 44087D8C.9090407@bluewin.ch
 # yoh: added ':' after [sshd] since the case without is not really common any more