Repository: ckan/ckan

Issue:
JobTimeoutException: Job exceeded maximum timeout value (180 seconds)
### CKAN Version if known (or site URL)
CKAN 2.7

### Please describe the expected behaviour
expected behavior is to be able to configure background jobs timeout

### Please describe the actual behaviour
There should be a way to extend the timeout 

### What steps can be taken to reproduce the issue? 
1. I created a background job following the ckan documentation  [docs](http://docs.ckan.org/en/latest/maintaining/background-tasks.html)
2. Tried to pass `timeout=3600` as a parameter
3. Throws the following error:

```
(default) samuel@ckan:~$ paster --plugin=ckanext-gather2_integration gather2_integration load-data --config=/etc/ckan/default/development.ini
2018-01-23 10:52:41,374 DEBUG [ckanext.validation.plugin] Validation tables exist
Loading gather2 data
Traceback (most recent call last):
  File "/usr/lib/ckan/default/bin/paster", line 11, in <module>
    sys.exit(run())
  File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/paste/script/command.py", line 102, in run
    invoke(command, command_name, options, args[1:])
  File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/paste/script/command.py", line 141, in invoke
    exit_code = runner.run(args)
  File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/paste/script/command.py", line 236, in run
    result = self.command()
  File "/usr/lib/ckan/default/src/ckanext-gather2_integration/ckanext/gather2_integration/commands.py", line 47, in command
    self.run_load_data()
  File "/usr/lib/ckan/default/src/ckanext-gather2_integration/ckanext/gather2_integration/commands.py", line 59, in run_load_data
    get_action(u'gather2_integration_run')({'load-data':'true'})
  File "/usr/lib/ckan/default/src/ckan/ckan/logic/__init__.py", line 457, in wrapped
    result = _action(context, data_dict, **kw)
  File "/usr/lib/ckan/default/src/ckanext-gather2_integration/ckanext/gather2_integration/logic.py", line 20, in gather2_integration_run
    enqueue_job(run_gather2_integration,[], timeout=3600)
  File "/usr/lib/ckan/default/src/ckanext-gather2_integration/ckanext/gather2_integration/logic.py", line 13, in enqueue_job
    return t.enqueue_job(*args, **kwargs)
TypeError: enqueue() got an unexpected keyword argument 'timeout'

```
4. Without the parameter it defaults to 180 seconds with the following error:
```
 r = requests.get(url, headers=headers)
  File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/requests/api.py", line 70, in get
    return request('get', url, params=params, **kwargs)
  File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/requests/api.py", line 56, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/requests/sessions.py", line 488, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/requests/sessions.py", line 609, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/requests/adapters.py", line 423, in send
    timeout=timeout
  File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 345, in _make_request
    self._validate_conn(conn)
  File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 844, in _validate_conn
    conn.connect()
  File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/requests/packages/urllib3/connection.py", line 326, in connect
    ssl_context=context)
  File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/requests/packages/urllib3/util/ssl_.py", line 324, in ssl_wrap_socket
    return context.wrap_socket(sock, server_hostname=server_hostname)
  File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/requests/packages/urllib3/contrib/pyopenssl.py", line 436, in wrap_socket
    cnx.do_handshake()
  File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/OpenSSL/SSL.py", line 1715, in do_handshake
    result = _lib.SSL_do_handshake(self._ssl)
  File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/rq/timeouts.py", line 51, in handle_death_penalty
    'value ({0} seconds)'.format(self._timeout))
JobTimeoutException: Job exceeded maximum timeout value (180 seconds)
```
5. I need help.


Discussion:
Yes, it looks like we aren't passing the timeout value through to RQ's enqueue_call in https://github.com/ckan/ckan/blob/master/ckan/lib/jobs.py#L126

Should be a very small change, would you mind submitting a PR?
Sure.
Another possibility would be to add a configuration option to globally specify these timeouts. 
![why not both?](http://i0.kym-cdn.com/photos/images/original/000/538/731/0fc.gif)

@wardi Good idea: A default timeout that is configurable via the INI and can be overridden using a parameter while enqueuing a job.
@torfsen Makes sense to do it that way. Should have the PR ready when I have a minute today. Thanks
@s-chand thanks! I've also been bitten by this
Just been bitten by this one whilst using [ckanext-rq](https://github.com/ckan/ckanext-rq) and found this existing issue. It seems to have gone pretty stale since the start of the year, did anyone submit anything to resolve this in the end (to either here or to [ckanext-rq](https://github.com/ckan/ckanext-rq))? From looking around it doesn't look like it but thought it was worth making sure before I look at submitting a PR for it.
@jrdh AFAIK there hasn't been any work on this issue, so feel free to submit a PR. You can probably reuse most of the code for a PR against CKAN for a similar one against ckanext-rq.
:+1: thanks @torfsen, and yeah that's what I would expect too.

Resolving patch:
diff --git a/ckan/lib/jobs.py b/ckan/lib/jobs.py
--- a/ckan/lib/jobs.py
+++ b/ckan/lib/jobs.py
@@ -36,6 +36,7 @@
 log = logging.getLogger(__name__)
 
 DEFAULT_QUEUE_NAME = u'default'
+DEFAULT_JOB_TIMEOUT = config.get(u'ckan.jobs.timeout', 180)
 
 # RQ job queues. Do not use this directly, use ``get_queue`` instead.
 _queues = {}
@@ -123,7 +124,8 @@ def get_queue(name=DEFAULT_QUEUE_NAME):
         return queue
 
 
-def enqueue(fn, args=None, kwargs=None, title=None, queue=DEFAULT_QUEUE_NAME):
+def enqueue(fn, args=None, kwargs=None, title=None, queue=DEFAULT_QUEUE_NAME,
+            timeout=DEFAULT_JOB_TIMEOUT):
     u'''
     Enqueue a job to be run in the background.
 
@@ -141,6 +143,9 @@ def enqueue(fn, args=None, kwargs=None, title=None, queue=DEFAULT_QUEUE_NAME):
     :param string queue: Name of the queue. If not given then the
         default queue is used.
 
+    :param integer timeout: The timeout, in seconds, to be passed
+        to the background job via rq.
+
     :returns: The enqueued job.
     :rtype: ``rq.job.Job``
     '''
@@ -148,7 +153,8 @@ def enqueue(fn, args=None, kwargs=None, title=None, queue=DEFAULT_QUEUE_NAME):
         args = []
     if kwargs is None:
         kwargs = {}
-    job = get_queue(queue).enqueue_call(func=fn, args=args, kwargs=kwargs)
+    job = get_queue(queue).enqueue_call(func=fn, args=args, kwargs=kwargs,
+                                        timeout=timeout)
     job.meta[u'title'] = title
     job.save()
     msg = u'Added background job {}'.format(job.id)


Test patch:
diff --git a/ckan/tests/lib/test_jobs.py b/ckan/tests/lib/test_jobs.py
--- a/ckan/tests/lib/test_jobs.py
+++ b/ckan/tests/lib/test_jobs.py
@@ -74,6 +74,18 @@ def test_enqueue_queue(self):
         assert_equal(all_jobs[1].origin,
                      jobs.add_queue_name_prefix(u'my_queue'))
 
+    def test_enqueue_timeout(self):
+        self.enqueue()
+        self.enqueue(timeout=0)
+        self.enqueue(timeout=-1)
+        self.enqueue(timeout=3600)
+        all_jobs = self.all_jobs()
+        assert_equal(len(all_jobs), 4)
+        assert_equal(all_jobs[0].timeout, 180)
+        assert_equal(all_jobs[1].timeout, 180)
+        assert_equal(all_jobs[2].timeout, -1)
+        assert_equal(all_jobs[3].timeout, 3600)
+
 
 class TestGetAllQueues(RQTestBase):