Pylons/waitress — pull request #62 by bertjwregeer (2014-05-14 22:08:23+00:00)
Title: Fix: filewrapper leak

@mcdonc I have added work on top of your already existing work...

---

The biggest fight I've been having is with the `multiprocessing.Queue()` on OS X, because of some resource leak within it (when started from python setup.py nosetests) you run out of file descriptors pretty fast as each test run eats another 5 file descriptors.

This is with self.queue.close() when close()'ing the socket.

test_it (waitress.tests.test_functional.SleepyThreadTests) ...
New socket:
ok
test_long_body (waitress.tests.test_functional.TcpBadContentLengthTests) ...
New socket:
ok
test_short_body (waitress.tests.test_functional.TcpBadContentLengthTests) ...
New socket:
ok
test_bad_host_header (waitress.tests.test_functional.TcpEchoTests) ...
New socket:

Here we see what FD was available before a call to `multiprocessing.Queue()` and what FD was the next available after it.

test_it (waitress.tests.test_functional.SleepyThreadTests) ... usable fd found before mp.Queue(): 4
useable fd found after mp.Queue(): 8
ERROR
test_long_body (waitress.tests.test_functional.TcpBadContentLengthTests) ... usable fd found before mp.Queue(): 9
useable fd found after mp.Queue(): 13
ERROR
test_short_body (waitress.tests.test_functional.TcpBadContentLengthTests) ... usable fd found before mp.Queue(): 14
useable fd found after mp.Queue(): 18
ERROR
test_bad_host_header (waitress.tests.test_functional.TcpEchoTests) ... usable fd found before mp.Queue(): 19
useable fd found after mp.Queue(): 23
ERROR

Code for this:

fd = os.dup(1)
print("FD found: ".format(fd))
os.close(fd)

In this case, `multiprocessing.Queue()` used fd 4, 5, 6, 7. Then `proc.start()` uses yet another fd, that is also not returned even if you `proc.join()`. I am not sure where it is going.

If on the first run the socket created is socket number 10, then that is correctly shut down, on the next run 5 FD's are used (10, 11, 12, 13, 14), and a socket that does return it's fd is created at 15. That is visible in the first log I posted.

---

The other was the code stolen from gevent, on OS X at least, it was throwing errno connection reset by peer when the remote had already hung up and you attempted to read from it, this was not being handled properly in the tests. I have replaced it with using the errno symbols, but I have a bad feeling that on some system it is going to break... you know just because. Not sure what the best way to go about it is, are the numbers associated with the errno's POSIX and the same across all systems?