celery/billiard — pull request #68 by Kronuz (2013-08-03 17:01:50+00:00)
Title: Python 2.7 compatibility by using _connection3

I'm using *Celery 3.1.0rc3* and *Billiard 3.3.0rc1* (both from GitHub's master) with *Python 2.7*. After finding some problems with `billiard.connection` when I try to start a non-solo celery worker with multiple processes (it was saying `_multiprocessing.Connection` doesn't have `setblocking` attribute)... I first "fixed" that by also monkeypatching `Connection` to add `setblocking` (the same way it's done for pypy in the module `billiard.connection`). However, I then encountered another (arguably bigger) error which segfaults 11 the whole thing when it tries to access the connection pool during the start up of the processes.

I went ahead and hunted the problem, it's definitely in the `Connection` object. When `celery.concurrency.processes.AsynPool`, initializes the pool, while it creates the `inq` and `outq` queues (in `create_process_queues()`) by instantiating `_SimpleQueue`, it segfaults when it tries to access the `poll` attribute of `self._reader` during `billiard.queues._SimpleQueue.__init__()` (which is a `_billiard.Connection` instance object).

What I did was making `billiard.connection` use Python's 3 `_connection3` and made a few minor adjustments in that module to make it *Python 2.7* compatible. Using the resulting `_connection3` with *Python 2.7* fixes all the problems for me now, and this pull request contains the modifications I'm using.

Are there any drawbacks or performance issues when using the `_connection3.Connection` instead of `_billiard.Connection`?