Repository: fail2ban/fail2ban

Issue:
Infinite loop on _escapedTags match in substituteRecursiveTags
Line 399 of server/action.py continues if an escaped tag matches; but this just repeats the surrounding 'while m:' loop without updating m. It therefore loops forever. This prevents using the blocklist_de action on Fedora 21, as it includes <matches> in the actionban string.

This patch seems to work for me:

```
------------------------------------------------------
commit c86c6b16e548911339ba3333d5242979cdb2f8a5
Author: Tony Thompson <athompson@samarc.org.uk>
Date:   Sun Jan 4 16:28:14 2015 +0000

    Step to next on _escapedTag match

    Previous code just repeated the while() loop with no change to the
    condition, and so looped forever. Step to the next match instead.

diff --git a/fail2ban/server/action.py b/fail2ban/server/action.py
index da7517f..9d8e9c8 100644
--- a/fail2ban/server/action.py
+++ b/fail2ban/server/action.py
@@ -395,8 +395,8 @@ class CommandAction(ActionBase):
                                        #logSys.log(5, 'recursion fail tag: %s value: %s' % (tag, value) )
                                        return False
                                elif found_tag in cls._escapedTags:
-                                       # Escaped so won't match
-                                       continue
+                                       # Escaped so won't match; move to next match
+                                       m = t.search(value, m.start() + 1)
                                else:
                                        if tags.has_key(found_tag):
                                                value = value.replace('<%s>' % found_tag , tags[found_tag])
------------------------------------------------------
```

Patch is available in branch master of https://github.com/TonyThompson/fail2ban-patch.git, if this looks like an appropriate fix.



Discussion:
do you have a sample which demonstrates the problem?  we should encode it into the unittest.  If you could come up with a "full blown" PR -- that would be the best

For an example of the problem, see config/action.d/blocklist_de.conf. This sets:

`actionban = curl --fail --data-urlencode 'server=<email>' --data 'apikey=<apikey>' --data 'service=<service>' --data 'ip=<ip>' --data-urlencode 'logs=<matches>' --data 'format=text' --user-agent "fail2ban v0.8.12" "https://www.blocklist.de/en/httpreports.html"`

To reproduce the infinite loop, I removed all fail2ban packages from my Fedora 21 installation, then ran 'sudo python setup.py install'. I used this /etc/fail2ban/jail.local file:

```
[sshd]
enabled = true
logpath = /var/log/secure
blocklist_de_apikey = deadbeef
action = %(action_blocklist_de)s
```

Then just start the server, e.g. `sudo python2.7 /usr/bin/fail2ban-server -f`, and run `top` to see the 100% load.

I can make running the testcases hang with this:

```
diff --git a/fail2ban/tests/actiontestcase.py b/fail2ban/tests/actiontestcase.py
index 5a58149..36e0ddb 100644
--- a/fail2ban/tests/actiontestcase.py
+++ b/fail2ban/tests/actiontestcase.py
@@ -59,6 +59,8 @@ class CommandActionTest(LogCaptureTestCase):
                self.assertEqual(CommandAction.substituteRecursiveTags({'A': '<C>'}), {'A': '<C>'})
                self.assertEqual(CommandAction.substituteRecursiveTags({'A': '<C> <D> <X>','X':'fun'}), {'A': '<C> <D> fun', 'X':'fun'})
                self.assertEqual(CommandAction.substituteRecursiveTags({'A': '<C> <B>', 'B': 'cool'}), {'A': '<C> cool', 'B': 'cool'})
+               # Escaped tags should be ignored
+               self.assertEqual(CommandAction.substituteRecursiveTags({'A': '<matches> <B>', 'B': 'cool'}), {'A': '<matches> cool', 'B': 'cool'})
                # Multiple stuff on same line is ok
                self.assertEqual(CommandAction.substituteRecursiveTags({'failregex': 'to=<honeypot> fromip=<IP> evilperson=<honeypot>', 'honeypot': 'pokie', 'ignoreregex': ''}),
                                                                { 'failregex': "to=pokie fromip=<IP> evilperson=pokie",
```

but since this is the first time I've touched Python, I'm not sure what the simplest test case would be. Also, should the test include a timeout, or is hanging OK as a failure indication?

The tests complete with my original patch.

hi,
so, this can be the reason why... ?
https://github.com/fail2ban/fail2ban/issues/885

sincerely i never tried to disable that blocklist_de, so I didn't know that was the problem... but in my case not only always 130% cpu load all the time, but neither it banned as usual, it didn't work at all... is that the source of these issues? I had to open a bug report... thank you

Note I only closed this by accident (hence the immediate reopening). Any feedback on whether I've provided enough information?

I'm just waiting for the next release on CentOS, to directly bypass the bugged 0.9.1-2.el7, 'cause I don't want to manually replace many files and ATM the version i'm using works fine enough... sincerely I can't understand why they release a not well tested version like that

@TonyThompson good catch, thanks.
I have fixed it now, commit follows...


Resolving patch:
diff --git a/fail2ban/server/action.py b/fail2ban/server/action.py
--- a/fail2ban/server/action.py
+++ b/fail2ban/server/action.py
@@ -394,20 +394,16 @@ def substituteRecursiveTags(cls, tags):
 					# recursive definitions are bad
 					#logSys.log(5, 'recursion fail tag: %s value: %s' % (tag, value) )
 					return False
-				elif found_tag in cls._escapedTags:
-					# Escaped so won't match
+				if found_tag in cls._escapedTags or not tags.has_key(found_tag):
+					# Escaped or missing tags - just continue on searching after end of match
+					# Missing tags are ok - cInfo can contain aInfo elements like <HOST> and valid shell
+					# constructs like <STDIN>.
+					m = t.search(value, m.end())
 					continue
-				else:
-					if tags.has_key(found_tag):
-						value = value.replace('<%s>' % found_tag , tags[found_tag])
-						#logSys.log(5, 'value now: %s' % value)
-						done.append(found_tag)
-						m = t.search(value, m.start())
-					else:
-						# Missing tags are ok so we just continue on searching.
-						# cInfo can contain aInfo elements like <HOST> and valid shell
-						# constructs like <STDIN>.
-						m = t.search(value, m.start() + 1)
+				value = value.replace('<%s>' % found_tag , tags[found_tag])
+				#logSys.log(5, 'value now: %s' % value)
+				done.append(found_tag)
+				m = t.search(value, m.start())
 			#logSys.log(5, 'TAG: %s, newvalue: %s' % (tag, value))
 			tags[tag] = value
 		return tags


Test patch:
diff --git a/fail2ban/tests/actiontestcase.py b/fail2ban/tests/actiontestcase.py
--- a/fail2ban/tests/actiontestcase.py
+++ b/fail2ban/tests/actiontestcase.py
@@ -59,6 +59,8 @@ def testSubstituteRecursiveTags(self):
 		self.assertEqual(CommandAction.substituteRecursiveTags({'A': '<C>'}), {'A': '<C>'})
 		self.assertEqual(CommandAction.substituteRecursiveTags({'A': '<C> <D> <X>','X':'fun'}), {'A': '<C> <D> fun', 'X':'fun'})
 		self.assertEqual(CommandAction.substituteRecursiveTags({'A': '<C> <B>', 'B': 'cool'}), {'A': '<C> cool', 'B': 'cool'})
+		# Escaped tags should be ignored
+		self.assertEqual(CommandAction.substituteRecursiveTags({'A': '<matches> <B>', 'B': 'cool'}), {'A': '<matches> cool', 'B': 'cool'})
 		# Multiple stuff on same line is ok
 		self.assertEqual(CommandAction.substituteRecursiveTags({'failregex': 'to=<honeypot> fromip=<IP> evilperson=<honeypot>', 'honeypot': 'pokie', 'ignoreregex': ''}),
 								{ 'failregex': "to=pokie fromip=<IP> evilperson=pokie",