Repository: ckan/ckan

Issue:
Missing requirement for Celery
`requirements.in` and `requirements.txt` currently don't contain an entry for `celery`. While use of Celery [is optional](https://github.com/ckan/ckan/blob/727fe76a4da67fb590e87ce75c5284f145a6e495/ckan/model/__init__.py#L191) it is the [documented background job system for CKAN](http://docs.ckan.org/en/ckan-2.6.0/maintaining/background-tasks.html) as of CKAN 2.6.

The problem is that CKAN's Celery code is incompatible with newer versions of Celery. For example, `pip install celery` pulls in 4.0.2, and using that within CKAN raises the following exception when initializing the database:
```$ /usr/lib/ckan/default/bin/paster --plugin=ckan db init -c /etc/ckan/default/production.ini
Traceback (most recent call last):
  File "/usr/lib/ckan/default/bin/paster", line 11, in <module>
    sys.exit(run())
  File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/paste/script/command.py", line 102, in run
    invoke(command, command_name, options, args[1:])
  File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/paste/script/command.py", line 141, in invoke
    exit_code = runner.run(args)
  File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/paste/script/command.py", line 236, in run
    result = self.command()
  File "/usr/lib/ckan/default/src/ckan/ckan/lib/cli.py", line 223, in command
    model.repo.init_db()
  File "/usr/lib/ckan/default/src/ckan/ckan/model/__init__.py", line 192, in init_db
    import ckan.lib.celery_app as celery_app
  File "/usr/lib/ckan/default/src/ckan/ckan/lib/celery_app.py", line 65, in <module>
    celery.loader.conf.update(default_config)
AttributeError: 'NoneType' object has no attribute 'update'
```
Reverting to the latest Celery 3.x version (3.1.25) solves the problem.

We should therefore add a requirement for Celery and pin it to version 3.1.25. Given that CKAN 2.7 introduces a new system based on RQ and deprecates the Celery system I don't think we need to update our Celery code to work with Celery 4.x.


Discussion:
I'm -1 for adding the celery requirement if we haven't required up until now and are eventually going to drop support for it. We can either document that you need to `pip install celery==3.x.y` or update the code to support 4.x. The actual line that is failing seems [trivial](http://docs.celeryproject.org/en/latest/whatsnew-4.0.html#modules) to update (there might be other more subtle changes needed).
Hm. I'm -1 on relying on documentation when all other requirements are taken care off automatically. A possible alternative would be to raise an `ImportError` in `celery_app` upon import if Celery's version is 4.x or later. That way nothing changes for people that don't use Celery and those who do get a meaningful error message.

Changing the code to support 4.x may look trivial but would need to be properly tested. Since there are basically no tests for CKAN's celery system, adding support for 4.x might break non-obvious things. In my opinion that's too much risk or effort for a system that's going to be deprecated in the next version.
OK, failing if Celery >= 4.x is very reasonable and sounds like the best option.

We can use `pkg_resources` or check for a Celery 4 only property

```python
import pkg_resources
pkg_resources.get_distribution("construct").version
```
Or we simply use `celery.__version__` :wink:
duh, I was being slow first thing I tried was `from celery import Celery; Celery.__version__` :upside_down_face: 

Resolving patch:
diff --git a/ckan/lib/celery_app.py b/ckan/lib/celery_app.py
--- a/ckan/lib/celery_app.py
+++ b/ckan/lib/celery_app.py
@@ -10,10 +10,14 @@
 import logging
 import os
 
-from celery import Celery
 from ckan.common import config as ckan_config
 from pkg_resources import iter_entry_points, VersionConflict
 
+from celery import __version__ as celery_version, Celery
+if not celery_version.startswith(u'3.'):
+    raise ImportError(u'Only Celery version 3.x is supported.')
+
+
 log = logging.getLogger(__name__)
 
 log.warning('ckan.lib.celery_app is deprecated, use ckan.lib.jobs instead.')


Test patch:
diff --git a/ckan/tests/lib/test_celery_app.py b/ckan/tests/lib/test_celery_app.py
new file mode 100644
--- /dev/null
+++ b/ckan/tests/lib/test_celery_app.py
@@ -0,0 +1,27 @@
+# encoding: utf-8
+
+u'''
+Tests for ``ckan.lib.celery_app``.
+'''
+
+from nose.tools import raises
+import mock
+
+
+class TestCeleryVersion(object):
+    u'''
+    Make sure that Celery's version is checked.
+    '''
+    @mock.patch.dict(u'sys.modules', {u'celery': mock.MagicMock()})
+    def check_celery_version(self, version):
+        import celery
+        celery.__version__ = version
+        import ckan.lib.celery_app
+        reload(ckan.lib.celery_app)
+
+    def test_3x(self):
+        self.check_celery_version(u'3.1.25')
+
+    @raises(ImportError)
+    def test_4x(self):
+        self.check_celery_version(u'4.0.0')