Repository: ckan/ckan

Issue:
DatapusherPlugin implementation of notify() can call 'datapusher_submit' multiple times
Hi,

Currently `DatapusherPlugin` implements `notify()` from `IDomainObjectModification` & `IResourceUrlChange`.

On our CKAN 2.3 portal this this means it's called twice; the first time, because operation == model.domain_object.DomainObjectOperation.new (IDomainObjectModification) and the second time because operation is not passed (IResourceUrlChange).

It's been like that for a [long time](https://github.com/ckan/ckan/blame/8f298cd9e658d788b031a4059271210aff6a3d94/ckanext/datapusher/plugin.py) and I wonder it this has not been picked up before because it has not been tested with extensions that implement other interfaces leading to notify being called multiple times ? 

In my case implementing IResourceUrlChange sounds like a quick fix, but I'm wondering if this doesn't highlight that the logic in DatapusherPlugin should be re-considered?

What do you think?



Discussion:
I think I've come up with a better answer to that problem. The datapusher doesn't check is there is a pending task before submitting a new one, so rather than fixing the hook, I've fixed that.

``` diff
diff --git a/ckanext/datapusher/logic/action.py b/ckanext/datapusher/logic/action.py
index 1af23c3..3ced61e 100644
--- a/ckanext/datapusher/logic/action.py
+++ b/ckanext/datapusher/logic/action.py
@@ -64,19 +64,23 @@ def datapusher_submit(context, data_dict):
         'error': '{}',
     }
     try:
-        task_id = p.toolkit.get_action('task_status_show')(context, {
+        existing_task = p.toolkit.get_action('task_status_show')(context, {
             'entity_id': res_id,
             'task_type': 'datapusher',
             'key': 'datapusher'
-        })['id']
-        task['id'] = task_id
+        })
+        
+        if existing_task is not None and existing_task.get('state') == 'pending':
+            log.info("A pending task was found %r, ignoring" % existing_task.get('id'))
+            return True
+        
+        task['id'] = existing_task.get('id')
     except logic.NotFound:
         pass
-
+    
     context['ignore_auth'] = True
-    result = p.toolkit.get_action('task_status_update')(context, task)
-    task_id = result['id']
-
+    p.toolkit.get_action('task_status_update')(context, task)
+    
     try:
         r = requests.post(
             urlparse.urljoin(datapusher_url, 'job'),
```

What do you think? Will you take a PR for that?


Resolving patch:
diff --git a/ckanext/datapusher/logic/action.py b/ckanext/datapusher/logic/action.py
--- a/ckanext/datapusher/logic/action.py
+++ b/ckanext/datapusher/logic/action.py
@@ -84,18 +84,34 @@ def datapusher_submit(context, data_dict):
         'error': '{}',
     }
     try:
-        task_id = p.toolkit.get_action('task_status_show')(context, {
+        existing_task = p.toolkit.get_action('task_status_show')(context, {
             'entity_id': res_id,
             'task_type': 'datapusher',
             'key': 'datapusher'
-        })['id']
-        task['id'] = task_id
+        })
+        assume_task_stale_after = datetime.timedelta(seconds=int(
+            config.get('ckan.datapusher.assume_task_stale_after', 3600)))
+        if existing_task.get('state') == 'pending':
+            updated = datetime.datetime.strptime(
+                existing_task['last_updated'], '%Y-%m-%dT%H:%M:%S.%f')
+            time_since_last_updated = datetime.datetime.now() - updated
+            if time_since_last_updated > assume_task_stale_after:
+                # it's been a while since the job was last updated - it's more
+                # likely something went wrong with it and the state wasn't
+                # updated than its still in progress. Let it be restarted.
+                log.info('A pending task was found %r, but it is only %s hours'
+                         'old', existing_task['id'], time_since_last_updated)
+            else:
+                log.info('A pending task was found %s for this resource, so '
+                         'skipping this duplicate task', existing_task['id'])
+                return False
+
+        task['id'] = existing_task['id']
     except logic.NotFound:
         pass
 
     context['ignore_auth'] = True
-    result = p.toolkit.get_action('task_status_update')(context, task)
-    task_id = result['id']
+    p.toolkit.get_action('task_status_update')(context, task)
 
     try:
         r = requests.post(


Test patch:
diff --git a/ckanext/datapusher/tests/test_action.py b/ckanext/datapusher/tests/test_action.py
--- a/ckanext/datapusher/tests/test_action.py
+++ b/ckanext/datapusher/tests/test_action.py
@@ -3,6 +3,7 @@
 import datetime
 
 from nose.tools import eq_
+import mock
 
 import ckan.plugins as p
 from ckan.tests import helpers, factories
@@ -278,3 +279,21 @@ def test_does_not_resubmit_if_a_dataset_field_changes_in_the_meantime(
 
         # Not called
         eq_(len(mock_datapusher_submit.mock_calls), 1)
+
+    def test_duplicated_tasks(self):
+        def submit(res, user):
+            return helpers.call_action(
+                'datapusher_submit', context=dict(user=user['name']),
+                resource_id=res['id'])
+
+        user = factories.User()
+        res = factories.Resource(user=user)
+        with mock.patch('requests.post') as r_mock:
+            r_mock().json = mock.Mock(
+                side_effect=lambda: dict.fromkeys(
+                    ['job_id', 'job_key']))
+            r_mock.reset_mock()
+            submit(res, user)
+            submit(res, user)
+
+            eq_(1, r_mock.call_count)