Repository: fail2ban/fail2ban

Issue:
FAIL: testExecuteTimeoutWithNastyChildren
```
testExecuteTimeoutWithNastyChildren (fail2ban.tests.actiontestcase.CommandActionTest) ...  2015-08-10 14:15:58,450 140008252741376 Set action Test timeout = 60
 2015-08-10 14:15:58,451 140008252741376 Set actionstart =
 2015-08-10 14:15:58,451 140008252741376 Set actionban =
 2015-08-10 14:15:58,451 140008252741376 Set actionunban =
 2015-08-10 14:15:58,451 140008252741376 Set actioncheck =
 2015-08-10 14:15:58,451 140008252741376 Set actionstop =
 2015-08-10 14:15:58,451 140008252741376 Created <class 'fail2ban.server.action.CommandAction'>
FAIL
 2015-08-10 14:15:58,656 140008252741376
 2015-08-10 14:15:58,656 140008252741376 Nothing to do
testReplaceTag (fail2ban.tests.actiontestcase.CommandActionTest) ...  2015-08-10 14:15:58,657 140008252741376 Set action Test timeout = 60
 2015-08-10 14:15:58,657 140008252741376 Set actionstart =
 2015-08-10 14:15:58,657 140008252741376 Set actionban =
 2015-08-10 14:15:58,657 140008252741376 Set actionunban =
 2015-08-10 14:15:58,657 140008252741376 Set actioncheck =
 2015-08-10 14:15:58,657 140008252741376 Set actionstop =
 2015-08-10 14:15:58,657 140008252741376 Created <class 'fail2ban.server.action.CommandAction'>
 2015-08-10 14:15:58,657 140008252741376
 2015-08-10 14:15:58,657 140008252741376 Nothing to do
ok

======================================================================
FAIL: testExecuteTimeoutWithNastyChildren (fail2ban.tests.actiontestcase.CommandActionTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "./fail2ban/tests/actiontestcase.py", line 226, in testExecuteTimeoutWithNastyChildren
    RuntimeError, CommandAction.executeCmd, 'bash %s' % tmpFilename, timeout=.1)
AssertionError: RuntimeError not raised

----------------------------------------------------------------------
Ran 14 tests in 4.091s

FAILED (failures=1)
E: Failed with  /usr/bin/python2.7
```

Just starting to dig into this...

fail2ban crash with some fqdn
Hi,

When a attack come from a fqdn who has lot of ip-address we find fail2ban stuck. He don't really crash but until we remove the line inside the log file, fail2ban do nothing.

The fqdn is unassigned.infomir.com.ua, when you try a nslookup on this name you got...1362 ip-address.

Regards.



Discussion:
Hi,
I think, can be a timeout, possible to short (on busy or slow system).
I had twice another error with timeout, executed 1 second longer as expected (have already fixed it, for example here - https://github.com/sebres/fail2ban/commit/6ecd7ddddf1b9425cfac466a015f0c5f4255f1a6)

nah - that one is a different issue... Here timeout is short but the point is that the script itself should sleep VERRY LONG (10000 sec), so whatever timeout time is specified, it should be shorter, thus that process must be killed and thus exception raised.  I only see a possibility that command does execute without sleeping for some reason, and completes without being killed.  in mycase -- this test didn't fail even when rebuilt across all debian/ubuntus -- so something more fun happening here 

> and completes without being killed

I mean the timeout is so short (bash so slow), that the bash does not yet creates pipes or something other goes wrong (not in while cycle), and variable `retcode` has wrong value, because first will set here - https://github.com/fail2ban/fail2ban/blob/cb101e9f4abb4a7886ebd917f735c688ea758a00/fail2ban/server/action.py#L572

I think I understand what is happening, and it seems expected.  So I have no idea why the test wasn't failing elsewhere.  In the bash script you are trapping the TERM signal:

```
            f.write("""#!/bin/bash
        trap : HUP EXIT TERM

        echo "$$" > %s.pid
        echo "my pid $$ . sleeping lo-o-o-ong"
        sleep 10000
        """ % tmpFilename)
```

So when you kill the process group, only the sleep process dies directly due to SIGTERM.  The bash process continues and ends up exiting with a return code of 143:

```
32009 nanosleep({10000, 0},  <unfinished ...>
32009 <... nanosleep resumed> NULL)     = ? ERESTART_RESTARTBLOCK (Interrupted by signal)
32008 <... wait4 resumed> 0x7ffe3c52f8f0, 0, NULL) = ? ERESTARTSYS (To be restarted if SA_RESTART is set)
32009 --- SIGTERM {si_signo=SIGTERM, si_code=SI_USER, si_pid=31982, si_uid=1744} ---
32008 --- SIGTERM {si_signo=SIGTERM, si_code=SI_USER, si_pid=31982, si_uid=1744} ---
32008 rt_sigreturn({mask=[CHLD]})       = -1 EINTR (Interrupted system call)
32008 wait4(-1,  <unfinished ...>
32009 +++ killed by SIGTERM +++
32008 <... wait4 resumed> [{WIFSIGNALED(s) && WTERMSIG(s) == SIGTERM}], 0, NULL) = 32009
....
32008 write(2, "Terminated\n", 11)      = 11
32008 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
32008 --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_KILLED, si_pid=32009, si_uid=1744, si_status=SIGTERM, si_utime=0, si_stime=0} ---
32008 wait4(-1, 0x7ffe3c52f310, WNOHANG, NULL) = -1 ECHILD (No child processes)
...
32008 exit_group(143)                   = ?
32008 +++ exited with 143 +++
```

From bash's man page:
    When a command terminates on a fatal signal N, bash uses the value of 128+N as the exit status.
So bash is reporting that sleep exited with signal 15.  But poll() in action.py:executeCmd() then returns 143, and so no RuntimeError is generated.

So, why isn't this test failing on debian/ubuntu?  And I really don't know what type of situation is desired to be handled here.

Well, I think the `executeCmd` should always `return False` in case of error (retcode != 0), and this artificial test should be rewritten to `assertFalse` instead of `assertRaises`, because another real call of it always checks return code (https://github.com/fail2ban/fail2ban/search?utf8=%E2%9C%93&q=executeCmd). So throwing an error is really unexpected situation.

So, if I change it to assertFalse(), I then get:

```
FAIL: testExecuteTimeoutWithNastyChildren (fail2ban.tests.actiontestcase.CommandActionTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "./fail2ban/tests/actiontestcase.py", line 230, in testExecuteTimeoutWithNastyChildren
    self.assertTrue(self._is_logged('killed with SIGTERM'))
AssertionError: False is not true
```

Which brings up something that made debugging this harder than it might have been.  None of the logSys() messages from executeCmd appear to get logged.

I would be damned -- issue replicates for me on Debian as well now ;) not sure how it all passed by before when I have built/tested all across, oh well

@yarikoptic I can not reproduce it on debian's ... :) where you had it? which debian (or python)?

I believe it just happened for me straight on my laptop (mix of Debian testing/sid).  Now, upon initial run I also got a new one:

```
$> bin/fail2ban-testcases testExecuteTimeoutWithNastyChildren
Fail2ban 0.9.3.dev test suite. Python 2.7.10 (default, Jul  1 2015, 10:54:53) [GCC 4.9.2]. Please wait...
...........E..
======================================================================
ERROR: testExecuteTimeoutWithNastyChildren (fail2ban.tests.actiontestcase.CommandActionTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "./fail2ban/tests/actiontestcase.py", line 228, in testExecuteTimeoutWithNastyChildren
    self.assertFalse(pid_exists(getnastypid()))  # process should have been killed
  File "./fail2ban/tests/actiontestcase.py", line 221, in getnastypid
    with open(tmpFilename + '.pid') as f:
IOError: [Errno 2] No such file or directory: '/home/yoh/.tmp/fail2ban_Prh1iC.sh.pid'

----------------------------------------------------------------------
Ran 14 tests in 4.046s

FAILED (errors=1)
```

but then tested on another box, running jessie (with neurodebian etc), there that good old issue reproduces consistently

```
$> bin/fail2ban-testcases testExecuteTimeoutWithNastyChildren
Fail2ban 0.9.3.dev test suite. Python 2.7.9 (default, Mar  1 2015, 12:57:24) [GCC 4.9.2]. Please wait...
...........F..
======================================================================
FAIL: testExecuteTimeoutWithNastyChildren (fail2ban.tests.actiontestcase.CommandActionTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "./fail2ban/tests/actiontestcase.py", line 226, in testExecuteTimeoutWithNastyChildren
    RuntimeError, CommandAction.executeCmd, 'bash %s' % tmpFilename, timeout=.1)
AssertionError: RuntimeError not raised

----------------------------------------------------------------------
Ran 14 tests in 4.088s

FAILED (failures=1)
```

Any progress here?  I'm currently waiting for this to get fixed before upgrading to 0.9.3 in Fedora.

On Sat, 12 Sep 2015, Orion Poplawski wrote:

> Any progress here? I'm currently waiting for this to get fixed before upgrading
> to 0.9.3 in Fedora.

I will look into it over the weekend some time
sorry about the delay ;)

fixed now in 0b5eacec0934238e86b88e02555f234bf932a968, as test case use following code, returns a large list of IPs...

``` bash
python -c "from fail2ban.server.filter import DNSUtils; print(DNSUtils.dnsToIp('unassigned.infomir.com.ua'))"
```

@fail2ban/contributors, @fail2ban/developers if someone knows dns that resolves similar amounts of IPs, but persistent (in the long term stay so large), I will add a test case.

PR #1184 not yet merged

@jas01 note that making fail2ban parse/react to logs with names instead of IPs is not recommended (although we still do allow it by default).  E.g. in this case -- banning 1000 of ips might be a bit too rushed and counter productive. Theoretically with current code (what was your version of fail2ban?)  it should simply not ban that IP since resolving it fails (at least for me) using socket.gethostbyname_ex:

```
2015-09-08 14:27:02,077 fail2ban.filter         [1498]: WARNING Unable to find a corresponding IP address for unassigned.infomir.com.ua: [Errno 1] Unknown host
```

So it would be interesting to discover what do you mean by "stuck"?  if you had fail2ban loglevel at DEBUG level -- what were the last log msgs?  was it stuck at busy 100%?  did fail2ban-client could still communicate, etc

 Le 08/09/2015 à 11:29:20-0700, Yaroslav Halchenko a écrit

> @jas01 note that making fail2ban parse/react to logs with names instead of IPs
> is not recommended (although we still do allow it by default). E.g. in this

Ok.

> case -- banning 1000 of ips might be a bit too rushed and counter productive.
> Theoretically (didn't try to simulate attack from such a domain name) with
> current code (what was your version of fail2ban?) it should simply not ban that
> IP since resolving it fails (at least for me) using socket.gethostbyname_ex:
> 
> 2015-09-08 14:27:02,077 fail2ban.filter         [1498]: WARNING Unable to find a corresponding IP address for unassigned.infomir.com.ua: [Errno 1] Unknown host
> 
> So it would be interesting to discover what do you mean by "stuck"? if you had
> fail2ban loglevel at DEBUG level -- what were the last log msgs? was it stuck
> at busy 100%? did fail2ban-client could still communicate, etc

Well, stuck mean nothing append until we get ride of the line with the
name.

The daemon still running but no rules was added anymore, no message in the
log either.

Regards.

> The daemon still running but no rules was added anymore, no message in the log either.

All jails? or only this one?

@jas01 if it was persistent/reproducible, would you mind running

```
python -c "import socket; print(socket.gethostbyname_ex('unassigned.infomir.com.ua'))"
```

and (since we still don't know what system/fail2ban)

```
python3 -c "import socket; print(socket.gethostbyname_ex('unassigned.infomir.com.ua'))"
```

and pasting output from those two commands

most probably sebres nailed it down but would be very nice to confirm that stall happens due to gethostbyname_ex


Resolving patch:
diff --git a/fail2ban/server/action.py b/fail2ban/server/action.py
--- a/fail2ban/server/action.py
+++ b/fail2ban/server/action.py
@@ -560,32 +560,33 @@ def executeCmd(realCmd, timeout=60):
 			return True
 
 		_cmd_lock.acquire()
-		try: # Try wrapped within another try needed for python version < 2.5
+		try:
+			retcode = None  # to guarantee being defined upon early except
 			stdout = tempfile.TemporaryFile(suffix=".stdout", prefix="fai2ban_")
 			stderr = tempfile.TemporaryFile(suffix=".stderr", prefix="fai2ban_")
-			try:
-				popen = subprocess.Popen(
-					realCmd, stdout=stdout, stderr=stderr, shell=True,
-					preexec_fn=os.setsid  # so that killpg does not kill our process
-				)
-				stime = time.time()
+
+			popen = subprocess.Popen(
+				realCmd, stdout=stdout, stderr=stderr, shell=True,
+				preexec_fn=os.setsid  # so that killpg does not kill our process
+			)
+			stime = time.time()
+			retcode = popen.poll()
+			while time.time() - stime <= timeout and retcode is None:
+				time.sleep(0.1)
 				retcode = popen.poll()
-				while time.time() - stime <= timeout and retcode is None:
-					time.sleep(0.1)
-					retcode = popen.poll()
-				if retcode is None:
-					logSys.error("%s -- timed out after %i seconds." %
-						(realCmd, timeout))
-					pgid = os.getpgid(popen.pid)
-					os.killpg(pgid, signal.SIGTERM)  # Terminate the process
+			if retcode is None:
+				logSys.error("%s -- timed out after %i seconds." %
+				    (realCmd, timeout))
+				pgid = os.getpgid(popen.pid)
+				os.killpg(pgid, signal.SIGTERM)  # Terminate the process
+				time.sleep(0.1)
+				retcode = popen.poll()
+				if retcode is None:  # Still going...
+					os.killpg(pgid, signal.SIGKILL)  # Kill the process
 					time.sleep(0.1)
 					retcode = popen.poll()
-					if retcode is None: # Still going...
-						os.killpg(pgid, signal.SIGKILL)  # Kill the process
-						time.sleep(0.1)
-						retcode = popen.poll()
-			except OSError, e:
-				logSys.error("%s -- failed with %s" % (realCmd, e))
+		except OSError as e:
+			logSys.error("%s -- failed with %s" % (realCmd, e))
 		finally:
 			_cmd_lock.release()
 
@@ -603,15 +604,16 @@ def executeCmd(realCmd, timeout=60):
 			return True
 		elif retcode is None:
 			logSys.error("%s -- unable to kill PID %i" % (realCmd, popen.pid))
-		elif retcode < 0:
-			logSys.error("%s -- killed with %s" %
-				(realCmd, signame.get(-retcode, "signal %i" % -retcode)))
+		elif retcode < 0 or retcode > 128:
+			# dash would return negative while bash 128 + n
+			sigcode = -retcode if retcode < 0 else retcode - 128
+			logSys.error("%s -- killed with %s (return code: %s)" %
+				(realCmd, signame.get(sigcode, "signal %i" % sigcode), retcode))
 		else:
 			msg = _RETCODE_HINTS.get(retcode, None)
 			logSys.error("%s -- returned %i" % (realCmd, retcode))
 			if msg:
 				logSys.info("HINT on %i: %s"
 							% (retcode, msg % locals()))
-			return False
-		raise RuntimeError("Command execution failed: %s" % realCmd)
+		return False
 


Test patch:
diff --git a/fail2ban/tests/actiontestcase.py b/fail2ban/tests/actiontestcase.py
--- a/fail2ban/tests/actiontestcase.py
+++ b/fail2ban/tests/actiontestcase.py
@@ -196,11 +196,10 @@ def testExecuteIncorrectCmd(self):
 	def testExecuteTimeout(self):
 		stime = time.time()
 		# Should take a minute
-		self.assertRaises(
-			RuntimeError, CommandAction.executeCmd, 'sleep 60', timeout=2)
+		self.assertFalse(CommandAction.executeCmd('sleep 60', timeout=2))
 		# give a test still 1 second, because system could be too busy
 		self.assertTrue(time.time() >= stime + 2 and time.time() <= stime + 3)
-		self.assertTrue(self._is_logged('sleep 60 -- timed out after 2 seconds') 
+		self.assertTrue(self._is_logged('sleep 60 -- timed out after 2 seconds')
 			or self._is_logged('sleep 60 -- timed out after 3 seconds'))
 		self.assertTrue(self._is_logged('sleep 60 -- killed with SIGTERM'))
 
@@ -222,17 +221,16 @@ def getnastypid():
 				return int(f.read())
 
 		# First test if can kill the bastard
-		self.assertRaises(
-			RuntimeError, CommandAction.executeCmd, 'bash %s' % tmpFilename, timeout=.1)
+		self.assertFalse(CommandAction.executeCmd(
+		                 'bash %s' % tmpFilename, timeout=.1))
 		# Verify that the process itself got killed
 		self.assertFalse(pid_exists(getnastypid()))  # process should have been killed
 		self.assertTrue(self._is_logged('timed out'))
 		self.assertTrue(self._is_logged('killed with SIGTERM'))
 
 		# A bit evolved case even though, previous test already tests killing children processes
-		self.assertRaises(
-			RuntimeError, CommandAction.executeCmd, 'out=`bash %s`; echo ALRIGHT' % tmpFilename,
-			timeout=.2)
+		self.assertFalse(CommandAction.executeCmd(
+			'out=`bash %s`; echo ALRIGHT' % tmpFilename, timeout=.2))
 		# Verify that the process itself got killed
 		self.assertFalse(pid_exists(getnastypid()))
 		self.assertTrue(self._is_logged('timed out'))