Repository: fail2ban/fail2ban

Issue:
PyInotify fails after log rotation
Sorry to be a pest but I'd surely like to see this work. This is basically the same thing as reported in bug 44. I see that several others complained about it after the bug was closed as fixed so I thought it best if this was put in a new bug report.The software I'm running:
fail2ban: head branch checked out 4/11/2013
pyinotify:  0.8.9
python:  2.5.2
Linux:  2.6.30
Libc:  2.7
Distro:  Debian 5 (lenny).

Testcases:

```
Testing using /usr/bin/python2.5
Fail2ban 0.8.8 test suite. Python 2.5.2 (r252:60911, Jan 24 2010, 14:53:14) [GCC 4.3.2]. Please wait...
I: Skipping gamin backend testing. Got exception 'No module named gamin'
...................................................F................FFFFF.....warning: unhandled exception
..
======================================================================
FAIL: testNewChangeViaIsModified (testcases.filtertestcase.LogFileMonitor)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/srv/home/jon/fail2ban-0.8.8+20130411/testcases/filtertestcase.py", line 245, in testNewChangeViaIsModified
    self.assertTrue(self.isModified())
AssertionError

======================================================================
FAIL: test_delLogPath (testcases.filtertestcase.MonitorFailures<FilterPoll>(/tmp/monitorfailures_FilterPollXgtt-cfail2ban))
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/srv/home/jon/fail2ban-0.8.8+20130411/testcases/filtertestcase.py", line 496, in test_delLogPath
    self.assert_correct_last_attempt(GetFailures.FAILURES_01)
  File "/srv/home/jon/fail2ban-0.8.8+20130411/testcases/filtertestcase.py", line 408, in assert_correct_last_attempt
    self.assertTrue(self.isFilled(20)) # give Filter a chance to react
AssertionError

======================================================================
FAIL: test_grow_file (testcases.filtertestcase.MonitorFailures<FilterPoll>(/tmp/monitorfailures_FilterPollXgtt-cfail2ban))
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/srv/home/jon/fail2ban-0.8.8+20130411/testcases/filtertestcase.py", line 424, in test_grow_file
    self.assertTrue(self.isFilled(6))
AssertionError

======================================================================
FAIL: test_move_file (testcases.filtertestcase.MonitorFailures<FilterPoll>(/tmp/monitorfailures_FilterPollXgtt-cfail2ban))
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/srv/home/jon/fail2ban-0.8.8+20130411/testcases/filtertestcase.py", line 463, in test_move_file
    self.assertEqual(self.filter.failManager.getFailTotal(), 2)
AssertionError: 0 != 2

======================================================================
FAIL: test_new_bogus_file (testcases.filtertestcase.MonitorFailures<FilterPoll>(/tmp/monitorfailures_FilterPollXgtt-cfail2ban))
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/srv/home/jon/fail2ban-0.8.8+20130411/testcases/filtertestcase.py", line 481, in test_new_bogus_file
    self.assert_correct_last_attempt(GetFailures.FAILURES_01)
  File "/srv/home/jon/fail2ban-0.8.8+20130411/testcases/filtertestcase.py", line 408, in assert_correct_last_attempt
    self.assertTrue(self.isFilled(20)) # give Filter a chance to react
AssertionError

======================================================================
FAIL: test_rewrite_file (testcases.filtertestcase.MonitorFailures<FilterPoll>(/tmp/monitorfailures_FilterPollXgtt-cfail2ban))
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/srv/home/jon/fail2ban-0.8.8+20130411/testcases/filtertestcase.py", line 445, in test_rewrite_file
    self.assert_correct_last_attempt(GetFailures.FAILURES_01)
  File "/srv/home/jon/fail2ban-0.8.8+20130411/testcases/filtertestcase.py", line 408, in assert_correct_last_attempt
    self.assertTrue(self.isFilled(20)) # give Filter a chance to react
AssertionError

----------------------------------------------------------------------
Ran 80 tests in 101.366s

FAILED (failures=6)
E: Failed with  /usr/bin/python2.5

----
```

I'm gathering debug logs.



Discussion:
so "fail2ban-0.8.8+20130411" means that you grabbed GIT snapshot few
days back?

I guess the best solution for you on lenny would be to switch to polling
for now

## 

Yaroslav O. Halchenko, Ph.D.
http://neuro.debian.net http://www.pymvpa.org http://www.fail2ban.org
Senior Research Associate,     Psychological and Brain Sciences Dept.
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834                       Fax: +1 (603) 646-1419
WWW:   http://www.linkedin.com/in/yarik

Yes, I grabbed it from git a few days ago. Here is the logrotate config:

```
/var/log/auth.log
{
        create
        rotate 4
        weekly
        missingok
        notifempty
        compress
        delaycompress
        sharedscripts
        postrotate
                invoke-rc.d rsyslog reload > /dev/null
        endscript
}
```

And jail.conf:

```
[DEFAULT]
ignoreip  = 127.0.0.1/8
backend   = pyinotify
bantime   = 3600
findtime  = 3600
maxretry  = 5
destemail = root@localhost
banaction = iptables-multiport
mta       = sendmail
protocol  = tcp
chain     = INPUT
action = %(banaction)s[name=%(__name__)s, port="%(port)s", protocol="%(protocol)s", chain="%(chain)s"]

[vsftpd]
enabled = true
port = ftp,ftp-data
filter = vsftpd
logpath = /var/log/auth.log
bantime = 604800
maxretry = 3
findtime = 86400
```

I'll be switching all servers back to polling as soon as I get the debug log from the next rotation.

@jafcobend @yarikoptic Looking at the test failures above, it appears that _poll_ is failing, not _pyinotify_...

@kwirk but, go figure, polling works and PyInotify works until the log rolls.

heh -- I was a good boy and upgraded all of them from lenny (it is not
supported by Debian project for some time already) to squeeze -- so
would not be able to reproduce :-/

is there any particular about that box when you run tests?  e.g. busy
with CPU or IO?

On Mon, 22 Apr 2013, Yaroslav Halchenko wrote:

> >    [1]@jafcobend [2]@yarikoptic Looking at the test failures above, it
> >    appears that poll is failing, not pyinotify...
> 
> doh -- indeed!  I got biased by the subject ;)
> 
> So it is weird/bad that it fails there -- I will also check on
> 
> ## some elderly box I have available
> 
> Yaroslav O. Halchenko, Ph.D.
> http://neuro.debian.net http://www.pymvpa.org http://www.fail2ban.org
> Senior Research Associate,     Psychological and Brain Sciences Dept.
> Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
> Phone: +1 (603) 646-9834                       Fax: +1 (603) 646-1419
> WWW:   http://www.linkedin.com/in/yarik

Yaroslav Halchenko wrote:

> [...]
> is there any particular about that box when you run tests? e.g. busy
> with CPU or IO?
> [...] 
> Not really. I have two boxes that I rely on fail2ban the most. Both have
> the same issue. The software specs are the same across both machines.
> One is a dual Mac G5 PowerPC 2.5GHz w/ 2GB of RAM. The other is a dual
> Xeon 2.0GHz with 6GB of RAM running Debian's "bigmem" kernel. Neither of
> them spend much time busy. They average less then a load of 0.50. One is
> a mail server (smtp, imap, pop3). The other is a web / ftp server.

The tests were run on the Intel box.

## 

Sent from my Debian Linux laptop -- http://www.debian.org/intro/about

Jon Foster
JF Possibilities, Inc.
jon@jfpossibilities.com
541-410-2760
Making computers work for you!

First off I should mention that I tried a patch similar to what was proposed in bug 44. Here it is:

```
--- filterpyinotify.py~ 2013-04-16 11:20:22.000000000 -0700
+++ filterpyinotify.py  2013-04-16 11:20:22.000000000 -0700
@@ -103,7 +103,7 @@


        def _addFileWatcher(self, path):
-               wd = self.__monitor.add_watch(path, pyinotify.IN_MODIFY)
+               wd = self.__monitor.add_watch(path, pyinotify.IN_MODIFY | pyinotify.IN_CREATE)
               self.__watches.update(wd)
                logSys.debug("Added file watcher for %s", path)
                # process the file since we did get even
```

Anyhow this resulted in the following pyinotify events in the log, after Sunday morning's rotation:

```
2013-04-28 04:29:01,842 fail2ban.filter : DEBUG  Default Callback for Event: <Event dir=False mask=0x2 maskname=IN_MODIFY name='' path=/var/log/auth.log pathname=/var/log/auth.log wd=2 >
2013-04-28 04:29:01,842 fail2ban.filter : DEBUG  Default Callback for Event: <Event dir=False mask=0x2 maskname=IN_MODIFY name='' path=/var/log/auth.log pathname=/var/log/auth.log wd=2 >
2013-04-28 04:29:04,081 fail2ban.filter : DEBUG  Default Callback for Event: <Event dir=False mask=0x100 maskname=IN_CREATE name=auth.log.0.gz path=/var/log pathname=/var/log/auth.log.0.gz wd=1 >
2013-04-28 04:29:04,081 fail2ban.filter : DEBUG  Default Callback for Event: <Event dir=False mask=0x100 maskname=IN_CREATE name=auth.log.0.gz path=/var/log pathname=/var/log/auth.log.0.gz wd=1 >
2013-04-28 04:29:04,081 fail2ban.filter : DEBUG  Ignoring creation of /var/log/auth.log.0.gz we do not monitor
2013-04-28 04:29:04,082 fail2ban.filter : DEBUG  Ignoring creation of /var/log/auth.log.0.gz we do not monitor
2013-04-28 04:29:04,148 fail2ban.filter : DEBUG  Default Callback for Event: <Event dir=False mask=0x100 maskname=IN_CREATE name=auth.log.new path=/var/log pathname=/var/log/auth.log.new wd=1 >
2013-04-28 04:29:04,149 fail2ban.filter : DEBUG  Default Callback for Event: <Event dir=False mask=0x100 maskname=IN_CREATE name=auth.log.new path=/var/log pathname=/var/log/auth.log.new wd=1 >
2013-04-28 04:29:04,149 fail2ban.filter : DEBUG  Ignoring creation of /var/log/auth.log.new we do not monitor
2013-04-28 04:29:04,149 fail2ban.filter : DEBUG  Ignoring creation of /var/log/auth.log.new we do not monitor
2013-04-28 04:29:04,152 fail2ban.filter : DEBUG  Default Callback for Event: <Event dir=False mask=0x100 maskname=IN_CREATE name=auth.log.0 path=/var/log pathname=/var/log/auth.log.0 wd=1 >
2013-04-28 04:29:04,152 fail2ban.filter : DEBUG  Default Callback for Event: <Event dir=False mask=0x100 maskname=IN_CREATE name=auth.log.0 path=/var/log pathname=/var/log/auth.log.0 wd=1 >
2013-04-28 04:29:04,153 fail2ban.filter : DEBUG  Ignoring creation of /var/log/auth.log.0 we do not monitor
2013-04-28 04:29:04,153 fail2ban.filter : DEBUG  Ignoring creation of /var/log/auth.log.0 we do not monitor
```

I think this shows that logrotate is creating the new file first as "auth.log.new", which is ignored since its not monitored. Then moving it to auth.log, later, which isn't triggering any events. I have not worked with pyinotify but I'm going to add the IN_MOVED_TO and IN_MOVED_FROM specifications to the watch list and see what gets logged next Sunday. I'm guessing that the IN_MOVED_TO might give me what I want... not sure if the existing fail2ban code can swallow it but one thing at a time.

wholly molly -- indeed we must monitor both IN_MOVED_(TO|FROM) .... let me craft the unittest and probably the fix... thanks!


Resolving patch:
diff --git a/server/filterpyinotify.py b/server/filterpyinotify.py
--- a/server/filterpyinotify.py
+++ b/server/filterpyinotify.py
@@ -66,7 +66,7 @@ def __init__(self, jail):
 	def callback(self, event, origin=''):
 		logSys.debug("%sCallback for Event: %s", origin, event)
 		path = event.pathname
-		if event.mask & pyinotify.IN_CREATE:
+		if event.mask & ( pyinotify.IN_CREATE | pyinotify.IN_MOVED_TO ):
 			# skip directories altogether
 			if event.mask & pyinotify.IN_ISDIR:
 				logSys.debug("Ignoring creation of directory %s", path)
@@ -130,7 +130,7 @@ def _addLogPath(self, path):
 		if not (path_dir in self.__watches):
 			# we need to watch also  the directory for IN_CREATE
 			self.__watches.update(
-				self.__monitor.add_watch(path_dir, pyinotify.IN_CREATE))
+				self.__monitor.add_watch(path_dir, pyinotify.IN_CREATE | pyinotify.IN_MOVED_TO))
 			logSys.debug("Added monitor for the parent directory %s", path_dir)
 
 		self._addFileWatcher(path)
diff --git a/testcases/filtertestcase.py b/testcases/filtertestcase.py
--- a/testcases/filtertestcase.py
+++ b/testcases/filtertestcase.py
@@ -475,6 +475,40 @@ def test_move_file(self):
 			self.assertEqual(self.filter.failManager.getFailTotal(), 6)
 
 
+		def _test_move_into_file(self, interim_kill=False):
+			# if we move a new file into the location of an old (monitored) file
+			self.file1 = _copy_lines_between_files(GetFailures.FILENAME_01, self.name,
+												  n=100)
+			# make sure that it is monitored first
+			self.assert_correct_last_attempt(GetFailures.FAILURES_01)
+			self.assertEqual(self.filter.failManager.getFailTotal(), 3)
+
+			if interim_kill:
+				_killfile(None, self.name)
+				time.sleep(0.2)				  # let them know
+
+			# now create a new one to override old one
+			self.file = _copy_lines_between_files(GetFailures.FILENAME_01,
+												   self.name + '.new', n=100)
+			os.rename(self.name + '.new', self.name)
+			self.assert_correct_last_attempt(GetFailures.FAILURES_01)
+			self.assertEqual(self.filter.failManager.getFailTotal(), 6)
+
+			# and to make sure that it now monitored for changes
+			_copy_lines_between_files(GetFailures.FILENAME_01, self.name, n=100)
+			self.assert_correct_last_attempt(GetFailures.FAILURES_01)
+			self.assertEqual(self.filter.failManager.getFailTotal(), 9)
+
+
+		def test_move_into_file(self):
+			self._test_move_into_file(interim_kill=False)
+
+		def test_move_into_file_after_removed(self):
+			# exactly as above test + remove file explicitly
+			# to test against possible drop-out of the file from monitoring
+		    self._test_move_into_file(interim_kill=True)
+
+
 		def test_new_bogus_file(self):
 			# to make sure that watching whole directory does not effect
 			_copy_lines_between_files(GetFailures.FILENAME_01, self.name, n=100)


Test patch: