Repository: fail2ban/fail2ban

Issue:
whois lookup used in DoS attacks against Fail2Ban
Remote admins can use whois redirects to a server that accepts the connection and hangs on to it forever.  If using a whois client that doesn't time out by default, it causes fail2ban to not proceed, and intruders can continue brute-forcing the server.

This is currently observed in the wild.

A workaround for this until someone fixes the whois callouts in fail2ban is to replace the whois binary with (or for EL, make the whois symlink point to) a script that calls the real whois with a timeout.
I use this:
# !/bin/sh

timeout 8 /usr/bin/jwhois "$@"



Discussion:
since b6059f47735f59ea3d9db3248afa1271e01e8e9b (0.9.0a1~11) we have `timeout` parameter for every action, which defaults to 60 (seconds).  Which version are you using? We might indeed reduce default to some smaller value I guess, and also modify it specifically for whois actions.  Just want to make clearer first what is the situation here ;-)

It is not fixed in 9.0.2 (latest stable for Gentoo at the time of posting).
Five different scripts in /etc/fail2ban/action.d do direct callouts of `/usr/bin/whois <ip>` without any timeout parameter.
Similar for several other callouts, like ADDRESS=$(dig ...)

When calling anything that uses outside access, there needs to be a timeout on the call.
And yes, this is a security bug, and one that's being actively exploited, so a notice to distros and maintainers like epel might be warranted.  That way, the fixes can be backported to older versions too, like those in long term stable distros like the Enterprise Linux family and Ubuntu LTS.

On Fri, 24 Jul 2015, arth1 wrote:

>    It is not fixed in 9.0.2 (latest stable for Gentoo at the time of
>    posting).
>    Five different scripts in /etc/fail2ban/action.d do direct callouts of
>    /usr/bin/whois <ip> without any timeout parameter.

Explicit timeout parameter is not necessary, as I have mentioned by
default should time out in 60sec for any external command

>    When calling anything that uses outside access, there needs to be a
>    timeout on the call.
>    And yes, this is a security bug, and one that's being actively exploited,
>    so a notice to distros and maintainers like epel might be warranted. That
>    way, the fixes can be backported to older versions too, like those in long
>    term stable distros like the Enterprise Linux family and Ubuntu LTS.

Care to contribute such a notice to the users list?

## 

Yaroslav O. Halchenko, Ph.D.
http://neuro.debian.net http://www.pymvpa.org http://www.fail2ban.org
Research Scientist,            Psychological and Brain Sciences Dept.
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834                       Fax: +1 (603) 646-1419
WWW:   http://www.linkedin.com/in/yarik        

Killing the parent action process does NOT kill subshell processes called with backticks or $().
The whois (and possibly dig) processes called by the actions will then continue to run.

When fail2ban encounters a "rogue" whois entry on a system with a whois that allows redirection (like jwhois), redirecting to a server that never closes the connection, the following occurs:

For 0.8.x, fail2ban itself will freeze up and no longer block (or release).
For 0.9.x, processes will accumulate until oom conditions occur, and the kernel starts killing other processes.  Or it runs out of sockets, whichever comes first.

I would say that this is not an improvement; rather the opposite.
When doing a callouts through a shell to processes that may fail to complete, you absolutely have to time out the callouts themselves.

gotcha -- thanks for spelling it out.  So, would need to prefix `timeout <timeout>` with subshell processes (backticks or $()) calling out to whois (any other to implicate here talking outside) only, correct?

timeout comes from coreutils so I expect being readily available across linuxes, but I see it not installed on OSX box I have access to.  So we would need to allow to parametrize that beast itself being optional with a FAT WARNING

Yep, that would be a safe thing to do.
You could have a `"<TIMEOUT> /usr/bin/whois <IP>"`, and have TIMEOUT set to something
sensible (including parameters), and set to nothing if there's no timeout command present.
(but with the FAT WARNING, as you say).

There are some 3rd party timeout like utilities that can be used too, if timeout is not available.
cratimeout http://cracauer-forum.cons.org/forum/viewtopic.php?t=17 might build on a mac.

If I'm looking at the `Popen.poll()` calls added in b6059f4 correctly, we shouldn't need `timeout` (the elf) prefixed to the direct calls to whois (or any executables...the intention in b6059f4 wrapping any action's executeCmd calls). Or is using `Popen.poll()` not effective (see below)?

> Killing the parent action process does NOT kill subshell processes called with backticks or $().
> The whois (and possibly dig) processes called by the actions will then continue to run.

@arth1 Can you provide a `ps -ef` or something similar to show what you're seeing? More specifically, I'm asking mostly since `/usr/bin/whois <ip>` shouldn't have, iirc, any backticks or $() - so seeing the full commands and process tree would be helpful.

Side note: Long-term, it may be worth looking into having fail2ban drop privileges for certain commands (like whois). Then they wouldn't get the oom-killer's root bonus points and be more likely to be eligible to be killed should something like this pop up again (and just as a general least-privilege best-practice).

Interesting....

```
actionban = sleep 50
            sleep 150
```

> 2015-07-25 13:31:35,575 fail2ban.transmitter    [8580]: DEBUG   Command: ['set', 'asterisk', 'banip', '1.2.3.90']
> 2015-07-25 13:32:36,678 fail2ban.actions        [8580]: ERROR   Failed to execute ban jail 'asterisk' action 'iptables-multiport' info 'CallingMap({'failures': 10, 'ipfailures': <function Actions.__checkBan.<locals>.<lambda> at 0x7f59dd054a60>, 'ipmatches': <function Actions.__checkBan.<locals>.<lambda> at 0x7f59dd054620>, 'matches': '', 'ipjailfailures': <function Actions.__checkBan.<locals>.<lambda> at 0x7f59dd054b70>, 'time': 1437845496.434967, 'ipjailmatches': <function Actions.__checkBan.<locals>.<lambda> at 0x7f59dd054840>, 'ip': '1.2.3.90'})': Command execution failed: sleep 50
> sleep 150
>     return self.executeCmd(realCmd, self.timeout)
> RuntimeError: Command execution failed: sleep 50
> sleep 150

```
# ps -ef | grep sleep
root     16568  8580  0 13:37 ?        00:00:00 /bin/sh -c sleep 50 sleep 150
root     16569 16568  0 13:37 ?        00:00:00 sleep 50

# ps -ef | grep sleep
root     16568  8580  0 13:37 ?        00:00:00 /bin/sh -c sleep 50 sleep 150
root     16591 16568  0 13:38 ?        00:00:00 sleep 150

# ps -ef | grep sleep
root     16591     1  0 13:38 ?        00:00:00 sleep 150
```

It would seem the real folly is that multiple actionban commands (or potentially any lists of commands) get munged together into a single 'string' (unexpected behavior from config parsing?). If each command were executed individually, the timeout in `executeCmd` would work as expected and order of execution would be maintained. Otherwise, the commands could also be executed concurrently (and their individual timeouts would be adhered to as expected with the existing code).

@yarikoptic Should the apparent root cause (configparse, etc or at least `executeCmd`) be fixed or just band-aid a `<timeout>` in front of every command execution fail2ban makes (and may make in the future or already exist in custom actions now and in the future, etc)?

@leeclemens, I think you misunderstand the problem.  The problem isn't that the reaping of the action isn't working, or that an action itself is called with backticks or $(), but that the action in turn does a shell callout using backticks or $() when embedding the output from a command into text.   So you get two (or more) levels of shells, and killing the top level does not kill commands running under the bottom level.

And the implication isn't that the system gets hacked, but either of:
- For 0.8.x, where there is no timeout, the fail2ban daemon hangs waiting, which not only prevent bans from taking place, but unbans and alerts too.
- For 0.9.x, the system itself can go down - fail2ban becomes a new vector for DoS attacks.

To see the subshell callouts do a grep for a backtick or $( in the action.d directory.

```
egrep '`/|\$\([^\(]' /etc/fail2ban/action.d/*
```

On my system with 0.9.2, I count two dig and four whois callouts, which will run under subshells.

To see how killing a parent process does not kill a subprocess, you can do this:

```
cat <<EOT >foo
#!/bin/sh
echo "bar returns \$(./bar)"
EOT
cat <<EOT >bar
#!/bin/sh
sleep 60
echo "bar"
EOT
chmod +x foo bar
./foo&
killall foo
ps -f $(pgrep bar)
```

When you kill just the parent process, the callout will still be running, reassigned to pid 1 because the parent is gone.  The above snippet will demonstrate this normal behavior as a minimal test case.

To see whether your system is affected by the current whois attacks (sporiadic, so far), replace /usr/bin/whois with something like

```
#!/bin/sh
realwhois=/usr/bin/jwhois
logfile=/var/log/whois.log
timeout 30 $realwhois "$@"
if [ $? -eq 124 ]; then
   touch $logfile
   chmod 666 $logfile
   echo "$(date) - timeout for $realwhois $*" >>$logfile
fi
```

On one system, I have four logged entries since yesterday.

To safely kill a shell script which has callouts, you need to kill the entire process group (or freeze the parents, find children, freeze those, and finally do a killing spree).

In the code, you have:

```
os.kill(popen.pid, signal.SIGTERM) # Terminate the process
```

Unfortunately, that is not a pgroup kill, but a kill of just the process.  You need os.killpg(), with a fallback to an os.kill() loop for systems that don't have killpg().

However, I would still recommend always doing a timeout for any command where the runtime can potentially be controlled by a different system, whether it's whois, dig, curl or other.
And with non-root privileges too, as far as possible.  Both are just good practices, and not specific to fail2ban, so if the shell code is copied/pasted into somewhere else, it will still be safe.

Killing just the parent without reaping the children is worse than doing nothing, because that allows new persistent subtasks to be spawned over and over again.  And worse, they run as root with 1 as the parent process, effectively turning them into daemons that won't be the first reaped when running out of memory or sockets.

With the old design, the worst that could happen was that fail2ban would hang.  Which is certainly bad, and being exploited in the wild as we speak.  With the new design, the whole system can be DoSed and potentially taken down.  Which is worse.

It is a very interesting topic.
I suggest not to use `timeout` but recursively kill child processes.

@arth1 Are you able to provide the output of `ps -ef` showing the issue on your system (I asked before, but you may have missed it.) Thanks!

@yarikoptic I can split this line of thinking off to another Issue, but it may be best to take the multi-line action-related commands and put them in little scripts that accept parameters (which we can pass in, replace per usual). Then we can read and process action-related commands as lists (and set Popen(shell=False) :+1:). Seems like the first step towards solving this, but just thinking out loud.

@arth1 Just to be clear and avoid inflammatory language, when you say "exploited in the wild", it's really just that your server is being ~~attacked~~ password-brute-forced by remote servers, and when you run `whois` querying those remote IP addresses it does not exit, correct?  It isn't fail2ban's fault whois does not exit gracefully or timeout properly (TCP RFC) and it is not targeted, in that the remote servers being banned aren't necessarily controlling their WHOIS servers (at least no evidence of that).

When I say "exploited in the wild", I mean exactly what I say.

The way the whois protocol works, it allows for owners of an assigned resource (address block in this case) to supply the registry with the address of a whois server that supplies decentralized information.  See RFC-2167 for more detals.  The whois query to the information center will then be answered with a redirect message, which the client follows.  The largest professional Linux install base in the world is RHEL or distros that have Red Hat as their upstream, including Red Hat Enterprise Linux, CentOS, Scientific Linux and Oracle Linux.  They all use jwhois, which supports whois redirects by default.
The exploits that are happening as we speak is that ssh root attacks are made from IP ranges which are under the control of the attacker.  They have redirects for whois that point to a server under the attacker's control, which accepts the request without proceeding.   This fake whois server hangs on to the connection, and prevents whois from completing.
There is nothing wrong going on at the IP level - there are plenty of services that depend on sockets being kept open for quite long time.  So the connection should not be closed by the TCP/IP stack.   The closing of connections needs to be initiated from a higher level.
An example of an IP that _did_ tarpit the connection is 50.61.248.118.  It still redirects, but the rwhois server is no longer serving, so that IP is no longer a problem.  The attackers move on.

No, I will not provide a ps output of a system where whois (and its parent, fail2ban) hangs.  I have patched all my internet facing systems on three continents to avoid the problem.   Asking me to unpatch my systems is an unreasonable request.  It's like an antivirus vendor requesting that a user unpatch his systems so it can be infected so they can see it.  Not reasonable at all.

I have provided VERY detailed explanations of exactly what goes on, and reproducible minimal case scenarios that demonstrate the design problem with your reaper (including ps output).   And answered any questions.  @yarikoptic understood the problem.
If that is not enough for you, and you don't trust that I tell the truth about the processes hanging and that I saw the ps output, and are unwilling to set up a scenario that gives you the ps output you want, you're the obstacle here, not me.

Quite frankly, you're not avoiding inflammatory language here, you're enticing it.  Ask and ye shall receive.

I gave you more than one chance - I went above and beyond what you should expect of anyone, despite you going out on a tangent, explaining in detail what goes wrong, why, how to reproduce, and multiple ways how to fix it.   MORE than enough information was provided.   If you expect more, no.
If it's beyond your capability to set up your own rwhois server to replicate the problem in-house, and you fail to understand the above explanations of why killing just a paret process is a bad thing that leads to DoS attacks, perhaps you should not contribute to a security related product.

My concern isn't you, it's my systems that are under constant attack, and my fellow administrators, who now will be alerted to a very real DoS problem that isn't a problem if they uninstall fail2ban.
It's very reproducible.

1:  Change the whois callout to call a script that doesn't return (just add a sleep 5000 or similar if you don't want to set up your own rwhois teergrube).   Or the same for dig with a tarpitted DNS server.
2:  Attack the fail2ban server until it calls one of the actions with a whois callout.
3:  Watch fail2ban (for 0.8.x as in LTS distributions) hang, or (for 0.9.x) detach the process and accumulate detached processes until the entire system eventually goes down.

It's not exactly rocket science to see what is wrong here, and why.   And no, it's not whois hanging - that is just a trigger.  The real problem is the bad design of fail2ban that doesn't take into account that callouts may not return (0.8.x) or, worse, kills only the parent process (0.9.x) in the false belief that this will make things better.  In reality, it opens up for a new way to DoS the entire system, not just fail2ban.

But in order to save everyone's time, I'll leave now, unsubscribe from this issue, and submit this to one of the security groups I have worked with before, and get a CERT advisory, so admins can protect themselves.  If they cannot patch all the callouts themselves, my advice is to downgrade to 0.8.x or uninstall fail2ban, so they at least won't risk the system being taken down due to a design flaw in fail2ban.
Your input here has been very valuable in showing me that going elsewhere is what I should have done in the first place, instead of working with the vendors like I normally do.  Apologies to Yaroslav.

Good luck!

@arth1 Thanks again for the response. if CVE gets issued, I would appreciate if you post it here (although I guess I will get notified independently as well).

@szepeviktor finding out kids recursively might be often not reliable, e.g. consider a script which starts process in a new shell which starts new kid in the background (and then obviously exits):

```
$> cat /tmp/runbg.sh
#!/bin/bash

echo "I am PID $$ running now background task in the same shell"
sleep 100 &
echo "Current ps:"
ps -eo pid,ppid,command | grep -e 'slee[p]' -e "^$$"

echo "I am PID $$ running now background task in a new shell"
( sleep 100 & )
echo "Current ps:"
ps -eo pid,ppid,command | grep -e 'slee[p]' -e "^$$"
echo "PID $$ exiting"
```

so output will be

```
I am PID 13262 running now background task in the same shell
Current ps:
13262 13644 /bin/bash /tmp/runbg.sh
13263 13262 sleep 100
I am PID 13262 running now background task in a new shell
Current ps:
13262 13644 /bin/bash /tmp/runbg.sh
13263 13262 sleep 100
13267     1 sleep 100
PID 13262 exiting
```

and with additional ps

```
$> ps -eo pid,ppid,command | grep -e 'slee[p]'
13263     1 sleep 100
13267     1 sleep 100
```

we see that even a new sleep lost its parent, so went back to the top init process.  The point is that we wouldn't be able to track them easily without lots of overloads etc.  Adding call to `timeout` is indeed the easiest and most reliable solution here.

@leeclemens there is no special treatment to multiline commands ATM (i.e. joining them into a single line). Just that in the ps output you just saw that the newline symbol is not "printed out" that is why it looks like they are 'mungled', but as you saw -- functions correctly. So  they are all "legit" and shouldn't be a culprit of any problem AFAIK.

@arth1 (since you might no longer be subscribed, I hope you don't mind): if by any chance you still have any IP/name which causes *whois on your (release?) system to hang -- I would appreciate -- for a sample to see it myself.  Meanwhile I will prep the 'timeout ' patch

@szepeviktor on a second though though -- since it is our scripts which invoke those vulnerable to hanging processes, and if we don't call out to awkward wrappers similar to the one I hypothesized,  we could indeed probably just kill all of the kids of the hanging PID and live happily after.  @arth1 -- do you see any problem with that?  I think it might indeed be a cleaner and more generic solution than injecting timeouts at this point. 

If I write the PR:
- should only network tools get `timeout` added?
- should all binary get timeout? one for network tools, another for others?
- what about `sendmail`?
- should I add limit `grep -m 1000` also?
- what else to limit in time, length etc.?

I will try – I have left one prod system with a whois wrapper that logs the timeout instead of using timeout in the action.  The system I first discovered it on.  If the attacker’s rwhois server is still live (for small values of “live”) when I check the logs, I’ll forward you the IP of both.

A problem is, of course, that many of modern attacking systems are subject to the schorched earth tactic, and auto-wiped by a dead man switch when the game is up and the puppeteer isn’t able to connect to them anymore.  So I’ll try to be quick if I see one.

Regards,

## 

*Art

From: Yaroslav Halchenko [mailto:notifications@github.com] 
Sent: Sunday, July 26, 2015 6:44 PM
To: fail2ban/fail2ban
Cc: arth1
Subject: Re: [fail2ban] whois lookup used in DoS attacks against Fail2Ban (#1129)

@arth1 https://github.com/arth1  (since you might no longer be subscribed, I hope you don't mind): if by any chance you still have any IP/name which causes *whois on your (release?) system to hang -- I would appreciate -- for a sample to see it myself. Meanwhile I will prep the 'timeout ' patch

—
Reply to this email directly or view it on GitHub https://github.com/fail2ban/fail2ban/issues/1129#issuecomment-125046247 .  https://github.com/notifications/beacon/ABvUqkvnHEaQmT8vMY7R4MChw6s9lQvyks5ohVolgaJpZM4FfLnQ.gif 

@szepeviktor sorry -- I wasn't clear... I will take care about it -- crafted a patch already (#1134) -- needs tests though.  Wanna take a crack at those?  I need to attend to the family for a bit ;)

No, that’s good - switching to os.killpg(pid,signal) or os.kill(-pid,signal) instead of os.kill(pid,signal) should do the job nicely.

Note that there are a couple of small caveats:
-          A proc might have spawned a subproc durin the duration of killpg()
-          A proc might catch TERM and discard it

In either case, following up with another killpg with a KILL signal would take care of those corner cases.

That only leaves the minimal risk of something else getting the old pid and inadvertently being killed.  Which can mostly be avoided by not running as root.  :)

while github accepts my other reply via email.... indeed with killpg we might not be able to kill the hanging child process, but then again, going back to our use-case, do you @arth1 know if hanging whois/jwhois would die from TERM or would really require KILL (thus this quick fix wouldn't be really sufficient)?

@yarikoptic I've provided plenty of evidence. (I know what a newline character is, thanks.) /proc/<pid>/cmdline further supports what I have observed. The diatribe from @\arth1, while entertaining, adds no value. As I mentioned a while ago, we should address the root-cause. But so be it, you've gone the other direction. My fork has diverged quite a bit by now anyhow and I already unsubscribed from this Issue :/

As far as I can tell with a quick test (killing the process before the lookup is complete) jwhois, bsd whois and dig all terminate nicely with sigterm.

With sendmail, at least one incarnation ignores it, but as long as nobody slaps on an -odi to the sendmail parameters to get it to deliver in the foreground, it shouldn’t be an issue.  The queue runners time out on their own.

Regards,

## 

*Art

From: Yaroslav Halchenko [mailto:notifications@github.com] 
Sent: Sunday, July 26, 2015 7:39 PM
To: fail2ban/fail2ban
Cc: arth1
Subject: Re: [fail2ban] whois lookup used in DoS attacks against Fail2Ban (#1129)

while github accepts my other reply via email.... indeed with killpg we might not be able to kill the hanging child process, but then again, going back to our use-case, do you @arth1 https://github.com/arth1  know if hanging whois/jwhois would die from TERM or would really require KILL (thus this quick fix wouldn't be really sufficient)?

—
Reply to this email directly or view it on GitHub https://github.com/fail2ban/fail2ban/issues/1129#issuecomment-125048366 .  https://github.com/notifications/beacon/ABvUqiMsi4BLDepod0seOUfjvwDgEEPSks5ohWcCgaJpZM4FfLnQ.gif 

Limit log lines: https://github.com/fail2ban/fail2ban/pull/1135

ok.  Then I will neither overenginneer (seek/collect all kids and check them as well), nor possibly shot myself in the foot (send killpg with KILL for a good measure since as you have pointed out -- another process might take it on a VERY busy system I guess).  But need to look into how to not share the same pgrp with the parent fail2ban process ;) otherwise everything goes kaboom ;-)

@szepeviktor what #1135 has to do with this one? if you want to establish relationship back to this one, just mention this issue in that PR.

> what #1135 has to do with this one?

You're limiting time. #1135 limits size.

@leeclemens Ok -- rereading helped and I guess I have misunderstood a bit what you were saying... So you are suggesting to introduce handling separate lines in the commands individually.  Although feasible in general, and indeed possibly comes with benefits, I see it as a major undertaking (would need to provide sharing the state for new stuff which you get from one line execution to the next ones etc).  You can prove me wrong with a PR which provide such a refactoring (which also would refactor our existing commands).  Although possible as a RF for future, for this fix we need a concise clean fix which could be adopted by distributions' maintainers of Fail2Ban.

edition #1: moreover it doesn't really address concern raised here -- to kill off all possibly hanging children processes of a process we started

> ```
>  what #1135 has to do with this one?
> ```
> 
>    You're limiting time. #1135 limits size.

does it help to resole "whois lookup used in DoS attacks against Fail2Ban" ?
;)  So indeed related, but not directly relevant (thus mere reference
from within #1135 is imho what is appropriate... just for future).


Resolving patch:
diff --git a/fail2ban/server/action.py b/fail2ban/server/action.py
--- a/fail2ban/server/action.py
+++ b/fail2ban/server/action.py
@@ -565,7 +565,9 @@ def executeCmd(realCmd, timeout=60):
 			stderr = tempfile.TemporaryFile(suffix=".stderr", prefix="fai2ban_")
 			try:
 				popen = subprocess.Popen(
-					realCmd, stdout=stdout, stderr=stderr, shell=True)
+					realCmd, stdout=stdout, stderr=stderr, shell=True,
+					preexec_fn=os.setsid  # so that killpg does not kill our process
+				)
 				stime = time.time()
 				retcode = popen.poll()
 				while time.time() - stime <= timeout and retcode is None:
@@ -574,11 +576,12 @@ def executeCmd(realCmd, timeout=60):
 				if retcode is None:
 					logSys.error("%s -- timed out after %i seconds." %
 						(realCmd, timeout))
-					os.kill(popen.pid, signal.SIGTERM) # Terminate the process
+					pgid = os.getpgid(popen.pid)
+					os.killpg(pgid, signal.SIGTERM)  # Terminate the process
 					time.sleep(0.1)
 					retcode = popen.poll()
 					if retcode is None: # Still going...
-						os.kill(popen.pid, signal.SIGKILL) # Kill the process
+						os.killpg(pgid, signal.SIGKILL)  # Kill the process
 						time.sleep(0.1)
 						retcode = popen.poll()
 			except OSError, e:


Test patch:
diff --git a/fail2ban/tests/actiontestcase.py b/fail2ban/tests/actiontestcase.py
--- a/fail2ban/tests/actiontestcase.py
+++ b/fail2ban/tests/actiontestcase.py
@@ -24,12 +24,14 @@
 __copyright__ = "Copyright (c) 2004 Cyril Jaquier"
 __license__ = "GPL"
 
+import os
 import time
+import tempfile
 
 from ..server.action import CommandAction, CallingMap
 
 from .utils import LogCaptureTestCase
-
+from .utils import pid_exists
 
 class CommandActionTest(LogCaptureTestCase):
 
@@ -202,6 +204,44 @@ def testExecuteTimeout(self):
 			or self._is_logged('sleep 60 -- timed out after 3 seconds'))
 		self.assertTrue(self._is_logged('sleep 60 -- killed with SIGTERM'))
 
+	def testExecuteTimeoutWithNastyChildren(self):
+		# temporary file for a nasty kid shell script
+		tmpFilename = tempfile.mktemp(".sh", "fail2ban_")
+		# Create a nasty script which would hang there for a while
+		with open(tmpFilename, 'w') as f:
+			f.write("""#!/bin/bash
+		trap : HUP EXIT TERM
+
+		echo "$$" > %s.pid
+		echo "my pid $$ . sleeping lo-o-o-ong"
+		sleep 10000
+		""" % tmpFilename)
+
+		def getnastypid():
+			with open(tmpFilename + '.pid') as f:
+				return int(f.read())
+
+		# First test if can kill the bastard
+		self.assertRaises(
+			RuntimeError, CommandAction.executeCmd, 'bash %s' % tmpFilename, timeout=.1)
+		# Verify that the proccess itself got killed
+		self.assertFalse(pid_exists(getnastypid()))  # process should have been killed
+		self.assertTrue(self._is_logged('timed out'))
+		self.assertTrue(self._is_logged('killed with SIGTERM'))
+
+		# A bit evolved case even though, previous test already tests killing children processes
+		self.assertRaises(
+			RuntimeError, CommandAction.executeCmd, 'out=`bash %s`; echo ALRIGHT' % tmpFilename,
+			timeout=.2)
+		# Verify that the proccess itself got killed
+		self.assertFalse(pid_exists(getnastypid()))
+		self.assertTrue(self._is_logged('timed out'))
+		self.assertTrue(self._is_logged('killed with SIGTERM'))
+
+		os.unlink(tmpFilename)
+		os.unlink(tmpFilename + '.pid')
+
+
 	def testCaptureStdOutErr(self):
 		CommandAction.executeCmd('echo "How now brown cow"')
 		self.assertTrue(self._is_logged("'How now brown cow\\n'"))
diff --git a/fail2ban/tests/utils.py b/fail2ban/tests/utils.py
--- a/fail2ban/tests/utils.py
+++ b/fail2ban/tests/utils.py
@@ -237,3 +237,30 @@ def getLog(self):
 
 	def printLog(self):
 		print(self._log.getvalue())
+
+# Solution from http://stackoverflow.com/questions/568271/how-to-check-if-there-exists-a-process-with-a-given-pid
+# under cc by-sa 3.0
+if os.name == 'posix':
+	def pid_exists(pid):
+		"""Check whether pid exists in the current process table."""
+		import errno
+		if pid < 0:
+			return False
+		try:
+			os.kill(pid, 0)
+		except OSError as e:
+			return e.errno == errno.EPERM
+		else:
+			return True
+else:
+	def pid_exists(pid):
+		import ctypes
+		kernel32 = ctypes.windll.kernel32
+		SYNCHRONIZE = 0x100000
+
+		process = kernel32.OpenProcess(SYNCHRONIZE, 0, pid)
+		if process != 0:
+			kernel32.CloseHandle(process)
+			return True
+		else:
+			return False
\ No newline at end of file