Repository: fail2ban/fail2ban

Issue:
SSH bans despite NOFAIL prefix

### Environment:
- Fail2Ban version (including any possible distribution suffixes):
`Fail2Ban v0.10.0a2`
- OS, including release name/version: 
```
Linux 4.4.30-v7+ armv7l GNU/Linux
Raspbian GNU/Linux 8
```
- [ ] Fail2Ban installed via OS/distribution mechanisms
- [x] You have not applied any additional foreign patches to the codebase
- [ ] Some customizations were done to the configuration (provide details below is so)

### The issue:
I'm getting frequent SSH bans, due to an SSH client misbehaving and terminating connections in an abrupt way. The condition is indeed specified in the jail config, but with NOFAIL prefix.

#### Steps to reproduce
Set findtime and a number of attempts low in the jail configuration.
Connect via ssh, terminate it abruptly.

#### Expected behavior
Clients aren't banned as the F-NOFAIL is set.

#### Observed behavior
Clients are banned according to jail config.

#### Any additional information
I may be misunderstanding the concept of F-NOFAIL. Thanks for explaining.

### Configuration, dump and another helpful excerpts
sshd.conf filter line being triggered
`^<F-NOFAIL>Received disconnect</F-NOFAIL> from <HOST>: 11:`

ssh jail.local definition
```
[ssh]
enabled  = true
port     = 111
filter   = sshd
mode     = normal
logpath  = /var/log/auth.log
maxretry = 2
findtime = 5m
bantime  = 2h
```

#### Relevant lines from monitored log files in question:
Running auth.log vs the filter
```
sudo fail2ban-regex /var/log/auth.log sshd.conf

Running tests
=============

Use   failregex filter file : sshd, basedir: /etc/fail2ban
Use         maxlines : 1
Use      datepattern : Default Detectors
Use         log file : /var/log/auth.log
Use         encoding : UTF-8


Results
=======

Failregex: 25 total
|-  #) [# of hits] regular expression
|  17) [24] ^<F-NOFAIL>Received disconnect</F-NOFAIL> from <HOST>: 11:
|  18) [1] ^<F-NOFAIL>Connection closed</F-NOFAIL> by <HOST>(?: \[preauth\])?\s*$
`-

Ignoreregex: 0 total

Date template hits:
|- [# of hits] date format
|  [1050] {^LN-BEG}(?:DAY )?MON Day 24hour:Minute:Second(?:\.Microseconds)?(?: ExYear)?
`-

Lines: 1050 lines, 0 ignored, 25 matched, 1025 missed
[processed in 0.76 sec]
```


Discussion:
I don't understand what you want say us with this issue.

Indeed, both RE's ("Received disconnect" as well as "Connection closed" are specified, but they a just a helpers to get the host (IP) for the real failures (for another entries in log-file without IP address, with the same connection ID).
The fact is both above-mentioned strings may occur also by the absolutely legal ssh-connections (that make no failures at all).

In mode "normal" sshd-jail observes the authentication failures only (that you probably did not have).
Otherwise you should provide the example log-lines (or better excerpts `... | grep "sshd[connid]"` that contains such lines).

If you want use ddos, extra or aggressive modes, you should specify it e. g. as aggressive (and supply it also by `fail2ban-regex`):
```diff
- sudo fail2ban-regex /var/log/auth.log sshd.conf
+ sudo fail2ban-regex /var/log/auth.log sshd[mode=aggressive]
```
BTW I would increase `maxretry` in this case (because the value 2 by using of ddos-rules may be too small).

What I was trying to say was **I'm getting banned from the server, just because of NOFAIL ssh rules. Is this the right behavior?** I don't know. There was no such concept in 0.8 :)

Examples:
Log lines triggering the ban (ssh "normal" mode only).
```
|- Matched line(s):
|  Mar 19 12:38:34 v317 sshd[9516]: Received disconnect from 94.254.131.8: 11: Bye Bye [preauth]
|  Mar 19 12:38:57 v317 sshd[9519]: Received disconnect from 94.254.131.8: 11: Bye Bye [preauth]
|  Mar 19 12:46:06 v317 sshd[9579]: Received disconnect from 95.211.230.94: 11: Bye Bye
|  Mar 19 13:37:55 v317 sshd[8636]: Received disconnect from 94.254.131.8: 11: Bye Bye
|  Mar 19 13:49:23 v317 sshd[9255]: Received disconnect from 94.254.131.8: 11: Bye Bye [preauth]
|  Mar 19 13:49:42 v317 sshd[9264]: Received disconnect from 94.254.131.8: 11: Bye Bye
|  Mar 19 14:15:48 v317 sshd[10477]: Received disconnect from 37.220.35.202: 11: Bye Bye
|  Mar 19 14:17:11 v317 sshd[10567]: Received disconnect from 37.220.35.202: 11: Bye Bye
|  Mar 19 15:25:36 v317 sshd[11994]: Received disconnect from 93.115.95.201: 11: Bye Bye
|  Mar 19 15:28:40 v317 sshd[12453]: Received disconnect from 93.115.95.201: 11: Bye Bye
|  Mar 19 18:00:51 v317 sshd[15936]: Received disconnect from 94.254.131.8: 11: Bye Bye
|  Mar 19 18:17:48 v317 sshd[20774]: Received disconnect from 94.254.131.8: 11: Bye Bye
|  Mar 19 18:52:53 v317 sshd[21694]: Received disconnect from 94.254.131.8: 11: Bye Bye
|  Mar 19 21:06:16 v317 sshd[28262]: Received disconnect from 89.76.80.135: 11: Bye Bye
|  Mar 20 09:51:11 v317 sshd[31528]: Received disconnect from 94.254.131.8: 11: Bye Bye
|  Mar 20 10:25:56 v317 sshd[2505]: Received disconnect from 94.254.131.8: 11: Bye Bye
|  Mar 20 10:27:59 v317 sshd[3050]: Received disconnect from 94.254.131.8: 11:
|  Mar 20 10:33:47 v317 sshd[3139]: Connection closed by 94.254.131.8 [preauth]
|  Mar 20 12:09:45 v317 sshd[5789]: Received disconnect from 94.254.131.8: 11: Bye Bye
|  Mar 20 12:30:27 v317 sshd[6304]: Received disconnect from 94.254.131.8: 11: Bye Bye
|  Mar 20 15:02:51 v317 sshd[8897]: Received disconnect from 77.247.181.165: 11: Bye Bye
|  Mar 20 16:52:55 v317 sshd[11222]: Received disconnect from 94.254.131.8: 11: Bye Bye
|  Mar 20 17:01:22 v317 sshd[13373]: Received disconnect from 94.254.131.8: 11: Bye Bye
|  Mar 20 17:03:06 v317 sshd[14457]: Received disconnect from 94.254.131.8: 11: Bye Bye
|  Mar 20 17:43:35 v317 sshd[14624]: Received disconnect from 65.19.167.130: 11: Bye Bye
|  Mar 20 18:07:02 v317 sshd[15282]: Received disconnect from 94.254.131.8: 11: Bye Bye
|  Mar 20 18:15:17 v317 sshd[16576]: Received disconnect from 94.254.131.8: 11: Bye Bye
|  Mar 20 18:31:40 v317 sshd[2244]: Received disconnect from 94.254.131.8: 11: Bye Bye
```
The aggressive mode adds only one line to it:
`|  Mar 20 10:12:27 v317 sshd[639]: fatal: Read from socket failed: Connection reset by peer [preauth]`

Auth log filtered using a few example connection IDs:
```
cat /var/log/auth.log | grep 9516
Mar 19 12:38:34 v317 sshd[9516]: Received disconnect from 94.254.131.8: 11: Bye Bye [preauth]
cat /var/log/auth.log | grep 9519
Mar 19 12:38:57 v317 sshd[9519]: Received disconnect from 94.254.131.8: 11: Bye Bye [preauth]
cat /var/log/auth.log | grep 9579
Mar 19 12:46:06 v317 sshd[9579]: Received disconnect from 95.211.230.94: 11: Bye Bye
```

There's nothing else suspicious that was happening. So either NO-FAIL doesn't fully work or I'm not getting the concept.
Hmm, strange...
Can you please provide an output of:
```
?sudo? python -c "ip='94.254.131.8'; db='/var/lib/fail2ban/fail2ban.sqlite3';  import sys, logging; logging.basicConfig(stream=sys.stdout, level=logging.ERROR); from fail2ban.server.database import Fail2BanDb; db = Fail2BanDb(db); t = db.getBansMerged(ip=ip); print(('%d attempts, matches:\n  %s' % (t.getAttempt(), '\n  '.join(t.getMatches())) ) if t else 'NOT FOUND')"
?sudo? python -c "ip='95.211.230.94'; db='/var/lib/fail2ban/fail2ban.sqlite3';  import sys, logging; logging.basicConfig(stream=sys.stdout, level=logging.ERROR); from fail2ban.server.database import Fail2BanDb; db = Fail2BanDb(db); t = db.getBansMerged(ip=ip); print(('%d attempts, matches:\n  %s' % (t.getAttempt(), '\n  '.join(t.getMatches())) ) if t else 'NOT FOUND')"
```
I could reproduce it, so trying to fix now...
Thanks. If it helps anything now, both queries return NOT FOUND.

On Mon, Mar 20, 2017, 20:18 Serg G. Brester <notifications@github.com>
wrote:

> I could reproduce it, so trying to fix now...
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/fail2ban/fail2ban/issues/1727#issuecomment-287868567>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AL7yLczhWTBC4XpP5hY6d2MqHxqGV6ifks5rntD_gaJpZM4Mit74>
> .
>


Resolving patch:
diff --git a/fail2ban/client/jailreader.py b/fail2ban/client/jailreader.py
--- a/fail2ban/client/jailreader.py
+++ b/fail2ban/client/jailreader.py
@@ -220,8 +220,8 @@ def convert(self, allow_no_files=False):
 		if self.__filter:
 			stream.extend(self.__filter.convert())
 		for opt, value in self.__opts.iteritems():
-			if opt == "logpath" and	\
-					not self.__opts.get('backend', None).startswith("systemd"):
+			if opt == "logpath":
+				if self.__opts.get('backend', None).startswith("systemd"): continue
 				found_files = 0
 				for path in value.split("\n"):
 					path = path.rsplit(" ", 1)
diff --git a/fail2ban/server/filter.py b/fail2ban/server/filter.py
--- a/fail2ban/server/filter.py
+++ b/fail2ban/server/filter.py
@@ -550,24 +550,29 @@ def ignoreLine(self, tupleLines):
 
 	def _mergeFailure(self, mlfid, fail, failRegex):
 		mlfidFail = self.mlfidCache.get(mlfid) if self.__mlfidCache else None
+		# if multi-line failure id (connection id) known:
 		if mlfidFail:
 			mlfidGroups = mlfidFail[1]
-			# if current line not failure, but previous was failure:
-			if fail.get('nofail') and not mlfidGroups.get('nofail'):
-				del fail['nofail'] # remove nofail flag - was already market as failure
-				self.mlfidCache.unset(mlfid) # remove cache entry
-			# if current line is failure, but previous was not:
-			elif not fail.get('nofail') and mlfidGroups.get('nofail'):
-				del mlfidGroups['nofail'] # remove nofail flag
-				self.mlfidCache.unset(mlfid) # remove cache entry
+			# update - if not forget (disconnect/reset):
+			if not fail.get('mlfforget'):
+				mlfidGroups.update(fail)
+			else:
+				self.mlfidCache.unset(mlfid) # remove cached entry
+			# merge with previous info:
 			fail2 = mlfidGroups.copy()
 			fail2.update(fail)
+			if not fail.get('nofail'): # be sure we've correct current state
+				try:
+					del fail2['nofail']
+				except KeyError:
+					pass
 			fail2["matches"] = fail.get("matches", []) + failRegex.getMatchedTupleLines()
 			fail = fail2
-		elif fail.get('nofail'):
-			fail["matches"] = failRegex.getMatchedTupleLines()
+		elif not fail.get('mlfforget'):
 			mlfidFail = [self.__lastDate, fail]
 			self.mlfidCache.set(mlfid, mlfidFail)
+			if fail.get('nofail'):
+				fail["matches"] = failRegex.getMatchedTupleLines()
 		return fail
 
 
@@ -683,6 +688,11 @@ def findFailure(self, tupleLine, date=None):
 				mlfid = fail.get('mlfid')
 				if mlfid is not None:
 					fail = self._mergeFailure(mlfid, fail, failRegex)
+					# bypass if no-failure case:
+					if fail.get('nofail'):
+						logSys.log(7, "Nofail by mlfid %r in regex %s: %s",
+							mlfid, failRegexIndex, fail.get('mlfforget', "waiting for failure"))
+						if not self.checkAllRegex: return failList
 				else:
 					# matched lines:
 					fail["matches"] = fail.get("matches", []) + failRegex.getMatchedTupleLines()
@@ -702,18 +712,16 @@ def findFailure(self, tupleLine, date=None):
 					host = fail.get('dns')
 					if host is None:
 						# first try to check we have mlfid case (cache connection id):
-						if fid is None:
-							if mlfid:
-								fail = self._mergeFailure(mlfid, fail, failRegex)
-							else:
+						if fid is None and mlfid is None:
 								# if no failure-id also (obscure case, wrong regex), throw error inside getFailID:
 								fid = failRegex.getFailID()
 						host = fid
 						cidr = IPAddr.CIDR_RAW
 				# if mlfid case (not failure):
 				if host is None:
-					if not self.checkAllRegex: # or fail.get('nofail'):
-					 	return failList
+					logSys.log(7, "No failure-id by mlfid %r in regex %s: %s",
+						mlfid, failRegexIndex, fail.get('mlfforget', "waiting for identifier"))
+					if not self.checkAllRegex: return failList
 					ips = [None]
 				# if raw - add single ip or failure-id,
 				# otherwise expand host to multiple ips using dns (or ignore it if not valid):


Test patch:
diff --git a/fail2ban/tests/files/logs/sshd b/fail2ban/tests/files/logs/sshd
--- a/fail2ban/tests/files/logs/sshd
+++ b/fail2ban/tests/files/logs/sshd
@@ -113,6 +113,11 @@ May 27 00:16:33 host sshd[2364]: Received disconnect from 198.51.100.76: 11: Bye
 # failJSON: { "time": "2004-09-29T16:28:02", "match": true , "host": "127.0.0.1" }
 Sep 29 16:28:02 spaceman sshd[16699]: Failed password for dan from 127.0.0.1 port 45416 ssh1
 
+# failJSON: { "match": false, "desc": "no failure, just cache mlfid (conn-id)" }
+Sep 29 16:28:05 localhost sshd[16700]: Connection from 192.0.2.5
+# failJSON: { "match": false, "desc": "no failure, just covering mlfid (conn-id) forget" }
+Sep 29 16:28:05 localhost sshd[16700]: Connection closed by 192.0.2.5 [preauth]
+
 # failJSON: { "time": "2004-09-29T17:15:02", "match": true , "host": "127.0.0.1" }
 Sep 29 17:15:02 spaceman sshd[12946]: Failed hostbased for dan from 127.0.0.1 port 45785 ssh2: RSA 8c:e3:aa:0f:64:51:02:f7:14:79:89:3f:65:84:7c:30, client user "dan", client host "localhost.localdomain"
 
@@ -233,4 +238,6 @@ Nov 26 13:03:30 srv sshd[45]: fatal: Unable to negotiate with 192.0.2.2 port 554
 # failJSON: { "match": false }
 Nov 26 15:03:30 host sshd[22440]: Connection from 192.0.2.3 port 39678 on 192.168.1.9 port 22
 # failJSON: { "time": "2004-11-26T15:03:31", "match": true , "host": "192.0.2.3", "desc": "Multiline - no matching key exchange method" }
-Nov 26 15:03:31 host sshd[22440]: fatal: Unable to negotiate a key exchange method [preauth]
\ No newline at end of file
+Nov 26 15:03:31 host sshd[22440]: fatal: Unable to negotiate a key exchange method [preauth]
+# failJSON: { "time": "2004-11-26T15:03:32", "match": true , "host": "192.0.2.3", "filter": "sshd", "desc": "Second attempt within the same connect" }
+Nov 26 15:03:32 host sshd[22440]: fatal: Unable to negotiate a key exchange method [preauth]
diff --git a/fail2ban/tests/samplestestcase.py b/fail2ban/tests/samplestestcase.py
--- a/fail2ban/tests/samplestestcase.py
+++ b/fail2ban/tests/samplestestcase.py
@@ -182,6 +182,9 @@ def _testMissingSamples():
 				try:
 					ret = self.filter.processLine(line)
 					if not ret:
+						# Bypass if filter constraint specified:
+						if faildata.get('filter') and name != faildata.get('filter'):
+							continue
 						# Check line is flagged as none match
 						self.assertFalse(faildata.get('match', True),
 							"Line not matched when should have")
@@ -200,13 +203,13 @@ def _testMissingSamples():
 					self.assertEqual(len(ret), 1,
 						"Multiple regexs matched %r" % (map(lambda x: x[0], ret)))
 
-					# Fallback for backwards compatibility (previously no fid, was host only):
-					if faildata.get("host", None) is not None and fail.get("host", None) is None:
-						fail["host"] = fid
 					# Verify match captures (at least fid/host) and timestamp as expected
 					for k, v in faildata.iteritems():
-						if k not in ("time", "match", "desc"):
+						if k not in ("time", "match", "desc", "filter"):
 							fv = fail.get(k, None)
+							# Fallback for backwards compatibility (previously no fid, was host only):
+							if k == "host" and fv is None:
+								fv = fid
 							self.assertEqual(fv, v)
 
 					t = faildata.get("time", None)