celery/kombu — pull request #330 by bmbouter (2014-03-25 17:12:46+00:00)
Title: Implementation to add Qpid Transport for Kombu

This PR adds support for Qpid as a valid transport type for Kombu, and includes code, unit tests, and doc changes, and dependency updates.

==The code==
The bulk of the code itself is contained in the kombu.transport.qpid module.

NOTE: The code currently still does monkey patching of qpid.messaging. Current versions of qpid.messaging that are out in the wild (pypi, rhel6, fedora, etc) are missing two important patches, [QPID-5637](https://issues.apache.org/jira/browse/QPID-5637) and [QPID-5557](https://issues.apache.org/jira/browse/QPID-5557). This codebase introduces those patches at runtime, and eventually will be removed.

The monkey patching has been tested currently to work with clients using 0.22, 0.24, and 0.26 versions of qpid.messaging, so it should produce the expected behavior no matter what version of tools it is patching.

The code is flake8 compliant, in preparation for this to be accepted upstream. You should be able to run everything as described on [this celery page](http://celery.readthedocs.org/en/latest/contributing.html#pyflakes-pep8) and it should look clean.

==The tests==
This code includes unit tests, and has been tested with both Kombu functional tests and Celery functional tests.

===Unit Tests===
The functional tests are all located in the kombu.tests.transport.test_qpid module.

You can run all just these new tests using:
> nosetests --with-coverage --cover-html -v kombu.tests.transport.test_qpid

NOTE: these tests use threading, and occasionally running the tests hang, which causes Mock trouble sometimes. If the tests hang, Ctr-C them and rerun the test runner. You shouldn't have to do this more than once or twice to run all tests.

===Kombu Functional Tests===
The existing kombu functional tests are used without modification. The only addition is to have a test file that inherits from the standard Transort test object and names qpid as the backend module to test. You can run them using:

> nosetests -v funtests.tests.test_qpid

===Celery Functional Tests===
The existing celery 'stress' functional tests can be used, but require a small patch to have celery know to test the Qpid transport. Here is the diff to apply:

> diff --git a/funtests/stress/stress/templates.py b/funtests/stress/stress/templates.py
> index 2235bf6..2a2e034 100644
> --- a/funtests/stress/stress/templates.py
> +++ b/funtests/stress/stress/templates.py
> @@ -82,6 +82,12 @@ class redistore(default):
>
>
> @template()
> +class qpid(default):
> + BROKER_URL = os.environ.get('CSTRESS_BROKER', 'qpid://')
> + CELERY_RESULT_BACKEND = 'mongodb://127.0.0.1/'
> +
> +
> +@template()
> class acks_late(default):
> CELERY_ACKS_LATE = True

After applying the diff you need to start qpidd using --auth=no by running something like:
> sudo -u qpidd qpidd --auth=no

Then you need to start a 'stress test worker'
> cd funtests/stress
> celery -A stress worker -Z qpid

Then you need to start a 'stress test dispatcher'
> cd funtests/stress
> python -m stress -Z qpid

Watch the tests complete.

==The Docs==
The code itself contains a huge number of docstrings. They follow the syntax expectations of Kombu/Celery (line width, strict PEP8, etc).

The more general documentation has been updated in two important ways.

1. The transport has its own section which is partially from the docstrings. This contains important things like instructions on how to download dependencies.

2. Updates to things that are no longer true. For instance the docs use to say things like "only rabbitMQ supports SSL" which is no longer true.

You can build the docs using [these instructions](http://celery.readthedocs.org/en/latest/contributing.html#building-the-documentation).

You also should build the API reference since most of the docs live in there. You can follow [these instructions](http://celery.readthedocs.org/en/latest/contributing.html#api-reference)

==Dependencies==
This library relies on the pure-python packages named 'qpidtoollibs' and 'qpid.messaging'. Kombu expects all pure python dependencies to be put onto pypi, and they can be "bundled" into Kombu as a group of optional dependencies.

qpidtoollibs is currently published on pypi at version 0.26 [here](https://pypi.python.org/pypi/qpid-tools).
qpid.messaging is currently published on pypi at version 0.26 [here](https://pypi.python.org/pypi/qpid-python).

A bundle in Kombu has been created called 'qpid' which should allow someone to install the bundle using kombu[qpid]. The docs also explain how to do this.