Repository: fail2ban/fail2ban

Issue:
asyncore: (9, 'Bad file descriptor')
```
Testing using /usr/bin/python2.6
Fail2ban 0.8.8 test suite. Python 2.6.8 (unknown, Jan 26 2013, 14:35:25) [GCC 4.7.2]. Please wait...
.........................................Exception in thread Thread-61:
Traceback (most recent call last):
  File "/usr/lib/python2.6/threading.py", line 532, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.6/threading.py", line 484, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/yoh/deb/gits/fail2ban/server/asyncserver.py", line 144, in start
    asyncore.loop(use_poll = False) # fixes the "Unexpected communication problem" issue on Python 2.6 and 3.0
  File "/usr/lib/python2.6/asyncore.py", line 210, in loop
    poll_fun(timeout, map)
  File "/usr/lib/python2.6/asyncore.py", line 140, in poll
    r, w, e = select.select(r, w, e, timeout)
error: (9, 'Bad file descriptor')

.....................................F....
```

Never ran into this one before (or at least not for a while ;) )



Discussion:
I will just consider it a fluke -- never ran into it again

Consistent on kfreebsd Debian boxes:

```
yoh@pons:~/deb/gits/fail2ban$ ./fail2ban-testcases
Fail2ban 0.8.8 test suite. Python 2.7.3rc2 (default, Apr 22 2012, 22:31:46) [GCC 4.6.3]. Please wait...
I: Skipping pyinotify backend testing. Got exception 'Couldn't find any inotify binding'
EEEEEEEEEEEEEEEEEEEEEEE......................Exception in thread Thread-2:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 551, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 504, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/yoh/deb/gits/fail2ban/server/asyncserver.py", line 146, in start
    asyncore.loop(use_poll = False) # fixes the "Unexpected communication problem" issue on Python 2.6 and 3.0
  File "/usr/lib/python2.7/asyncore.py", line 216, in loop
    poll_fun(timeout, map)
  File "/usr/lib/python2.7/asyncore.py", line 145, in poll
    r, w, e = select.select(r, w, e, timeout)
error: (9, 'Bad file descriptor')
```

Yikes. Doesn't look good. Wonder if it works with `asynccore.loop(use_poll = True)`?

Worse is that with e70d01b .. 281d310 (presumably unrelated) changes it became consistent for 2.6 and 2.7 on my laptop and occurred for 2.7 on travis:  https://travis-ci.org/fail2ban/fail2ban/jobs/7042490

Checking now if indeed just switching to use_poll=True would "work" on unittests without glitches but I am not sure if that is the way to go since original use_poll=False came out of the existing problem:
see fd898c9eacf35fd3d6b2fe647da11100be24a8de and e09fad559a8aea08f493d27c2e3345474d50455e  so additional care should be taken to introduce such a change pretty much reverting previously introduced fix

use_poll=True seems to relief this issue while running unittests

but actually I was misguided by travis (which haven't failed where it "should" have) -- this issue came up with 
https://github.com/fail2ban/fail2ban/commit/26715d5e5e4b1671ce6cbfabe13a6495a9f64b75
so running "setup.py install" is causing the failure (need to fix the test anyways since now it doesn't use the same python interpreter) and will dig from there

@yarikoptic Not sure if you noticed, but it appears that the failure is related to Socket test case.

```
python2 fail2ban-testcases Socket Setup
```

...raises the error, where independently they seem to both work fine. Strange...

On Fri, 10 May 2013, Steven Hiscocks wrote:

>    [1]@yarikoptic Not sure if you noticed, but it appears that the failure is
>    related to Socket test case.
>  python2 fail2ban-testcases Socket Setup
>    ...raises the error, where independently they seem to both work fine.
>    Strange...

ah -- good catch -- I haven't spotted it before ;)

I guess threads do no die that easily ...

## 

Yaroslav O. Halchenko, Ph.D.
http://neuro.debian.net http://www.pymvpa.org http://www.fail2ban.org
Senior Research Associate,     Psychological and Brain Sciences Dept.
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834                       Fax: +1 (603) 646-1419
WWW:   http://www.linkedin.com/in/yarik

yeap -- if I .join() (i.e. remove 1 sec timeout) -- that tandem proceeds
fine (but takes very long time for join() to return!  why?)

will run now on full battery to see if it comes again

On Fri, 10 May 2013, Steven Hiscocks wrote:

>    [1]@yarikoptic Not sure if you noticed, but it appears that the failure is
>    related to Socket test case.
> 
>  python2 fail2ban-testcases Socket Setup
> 
>    ...raises the error, where independently they seem to both work fine.
> 
> ##    Strange...
> 
> Yaroslav O. Halchenko, Ph.D.
> http://neuro.debian.net http://www.pymvpa.org http://www.fail2ban.org
> Senior Research Associate,     Psychological and Brain Sciences Dept.
> Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
> Phone: +1 (603) 646-9834                       Fax: +1 (603) 646-1419
> WWW:   http://www.linkedin.com/in/yarik

@yarikoptic Just came here to post the exact same about remove timeout from join... I wonder why the thread is hanging on so long...

On Fri, 10 May 2013, Steven Hiscocks wrote:

>    [1]@yarikoptic Just came here to post the exact same about remove timeout
>    from join... I wonder why the thread is hanging on so long...

but also it brought another backtrace:

```
testSocket (testcases.sockettestcase.Socket) ... setup  1
 2013-05-10 22:21:54,060 140697923151616 Detected Python 2.6 or greater. asyncore.loop() not using poll
 2013-05-10 22:21:55,063 140697967265536 Removed socket file /home/yoh/.tmp/socketQzWth_fail2ban.sock
 2013-05-10 22:21:55,063 140697967265536 Socket shutdown
ok
testSocketForce (testcases.sockettestcase.Socket) ... teardown  1
setup  1
 2013-05-10 22:22:25,086 140697967265536 Fail2ban seems to be already running
 2013-05-10 22:22:25,087 140697923151616 Fail2ban seems to be already running
 2013-05-10 22:22:25,087 140697967265536 Removed socket file /home/yoh/.tmp/sockethg6oQufail2ban.sock
 2013-05-10 22:22:25,088 140697967265536 Socket shutdown
 2013-05-10 22:22:25,088 140697923151616 Forcing execution of the server
Exception in thread Thread-62:
Traceback (most recent call last):
  File "/usr/lib/python2.6/threading.py", line 532, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.6/threading.py", line 484, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/yoh/deb/gits/fail2ban/server/asyncserver.py", line 125, in start
    os.remove(sock)
OSError: [Errno 2] No such file or directory: '/home/yoh/.tmp/sockethg6oQufail2ban.sock'

ok
testFormatExceptionConvertArgs (testcases.misctestcase.HelpersTest) ... ok
testFormatExceptionInfoBasic (testcases.misctestcase.HelpersTest) ... 
```

NB  -- those 'setup/teardown  1' print number of active threads and as should
just 1 there.  Before it was growing... if we make it work correctly without
timeout -- might be worth adding a check on threading.active_count() == 1

## 

Yaroslav O. Halchenko, Ph.D.
http://neuro.debian.net http://www.pymvpa.org http://www.fail2ban.org
Senior Research Associate,     Psychological and Brain Sciences Dept.
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834                       Fax: +1 (603) 646-1419
WWW:   http://www.linkedin.com/in/yarik

@yarikoptic The default time out on async.loop is 30 seconds which explains the long pause, but why is it timing out...
I'm not seeing the other error your having...

the delay is because of the asyncore.loop's timeout (defaults to
30 sec).  Setting it to 1 makes things fast again ;)

but not sure if that should be the way to go -- there might be a
better/legit way to shut it down which we have missed... need to get
better understanding of things here -- but removing timeout=1 for 'join'
- controlling number of active threads is probably the good thing
  anyways.

On Fri, 10 May 2013, Steven Hiscocks wrote:

>    [1]@yarikoptic Just came here to post the exact same about remove timeout
>    from join... I wonder why the thread is hanging on so long...
> 
>    —
>    Reply to this email directly or [2]view it on GitHub.
> 
> References
> 
>    Visible links
> 1. https://github.com/yarikoptic
> 2. https://github.com/fail2ban/fail2ban/issues/161#issuecomment-17742417

## 

Yaroslav O. Halchenko, Ph.D.
http://neuro.debian.net http://www.pymvpa.org http://www.fail2ban.org
Senior Research Associate,     Psychological and Brain Sciences Dept.
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834                       Fax: +1 (603) 646-1419
WWW:   http://www.linkedin.com/in/yarik

@yarikoptic I was confused why AsyncServer writable method would return False. Changed to return True (or just delete it as default  inherited server return this) and all seems to work nice a nippy again.

lol :laughing:  On python2.6, works well with writable returning True, still slow on python2.7, and python2.4 and 2.5 are fast but spam error messages...

Had another look at this, and found the following thread: http://stackoverflow.com/questions/5039608/poll-cant-detect-event-when-socket-is-closed-locally. 
This seems to resolve the time out issue when using _poll_.
I also added a _readable_ method which returns False once the socket is closed. These combined both seem to help when not using poll, but can get stuck in the time out still.

I've tried rolling back python2.6 as far back as python2.6.3 (earlier versions fail to compile), and cannot recreate origianl bug with poll = True. I wonder if the original poll issue is limited to python2.6.[0-2]?

Diff of code I'm testing with: https://gist.github.com/kwirk/5560548

sorry -- hadn't look at this but also got interested to recover the history how it came all about
- use_poll=True came up as a resolution to "error: (514, 'Unknown error 514')" while fail2ban would die out in 1 to 2 days (so might be some something unlikely to catch with unittests ;-) ).  At that time python <= 2.5 was in use
- python 2.6, when came out in e.g. Ubuntu,  caused fail2ban to not function correctly:  https://bugs.launchpad.net/ubuntu/+source/fail2ban/+bug/372304 .  Sure thing such information as python version was considered irrelevant and not posted, but looking at http://old-releases.ubuntu.com/ubuntu/dists/jaunty/main/binary-amd64/Packages it sees to be 2.6.2-0ubuntu1.  
  @buanzo (thanks again!) came up with a fix -- to not use poll ;-) and everyone became happy again! ;)

I wondered to run unittests with 2.6.2 -- ideally the same as that in jaunty -- so  initiated now

```
sudo debootstrap --arch=i386 --include=python2.6 jaunty  /tmp/old-python2.6 http://old-releases.ubuntu.com/ubuntu
```

to get myself such environment -- will post results of tests there (with use_poll=True for all) later

Nothing what our tests would pick up (besides an irrelevant tiny tune up I have just committed (2147488).

After the https://github.com/fail2ban/fail2ban/pull/280 we would not be able to ignore this one any longer, thus allocating it to be fixed for the next milestone! ;)

ping

pong... issue is still there and we all avoided it (I can't even find some testing code like from @kwirk) which I remember using for troubleshooting this one

pong...
It is today extremely reproducible for me with python 2.7.6, with ...

```
PYTHONPATH=. ./bin/fail2ban-testcases Socket Helpers Setup
```

And because of <a href="http://bugs.python.org/issue10878">issue10878</a>, resp. because of <blockquote>I am re-closing as there is no defined bug to fix, and 2.7 is closed to enhancements</blockquote>... we can only workaround this one with `use_poll = True` for 2.7.
Ready, will commit it after all test cases done.


Resolving patch:
diff --git a/fail2ban/server/asyncserver.py b/fail2ban/server/asyncserver.py
--- a/fail2ban/server/asyncserver.py
+++ b/fail2ban/server/asyncserver.py
@@ -149,8 +149,11 @@ def start(self, sock, force):
 		self.__init = True
 		# TODO Add try..catch
 		# There's a bug report for Python 2.6/3.0 that use_poll=True yields some 2.5 incompatibilities:
-		logSys.debug("Detected Python 2.6 or greater. asyncore.loop() not using poll")
-		asyncore.loop(use_poll=False) # fixes the "Unexpected communication problem" issue on Python 2.6 and 3.0
+		if sys.version_info >= (2, 7) and sys.version_info < (2, 8): # if python 2.7 ...
+			logSys.debug("Detected Python 2.7. asyncore.loop() using poll")
+			asyncore.loop(use_poll=True) # workaround for the "Bad file descriptor" issue on Python 2.7, gh-161
+		else:
+			asyncore.loop(use_poll=False) # fixes the "Unexpected communication problem" issue on Python 2.6 and 3.0
 	
 	##
 	# Stops the communication server.


Test patch:
diff --git a/fail2ban/tests/misctestcase.py b/fail2ban/tests/misctestcase.py
--- a/fail2ban/tests/misctestcase.py
+++ b/fail2ban/tests/misctestcase.py
@@ -113,7 +113,7 @@ def recursive_glob(treeroot, pattern):
 			# clean up
 			shutil.rmtree(tmp)
 			# remove build directory
-			os.system("%s %s clean --all >/dev/null"
+			os.system("%s %s clean --all >/dev/null 2>&1"
 					  % (sys.executable, self.setup))
 
 class TestsUtilsTest(unittest.TestCase):