Repository: ckan/ckan

Issue:
Add dataset button displays an Internal Server Error
### CKAN Version if known (or site URL)
git master branch

### Please describe the expected behaviour
I have configured docker-compose ckan based on the url below.

https://docs.ckan.org/en/2.8/maintaining/installing/install-from-docker-compose.html


There was no problem with datastore when adding ckan plugin.
```bash
ckan.plugins = stats text_view image_view recline_view datastore
```

However, when I added datapusher and pressed the add datasets button, I could see the following screen.

'Internal Server Error'

```bash
ckan.plugins = stats text_view image_view recline_view datastore datapusher

# Define which views should be created by default
# (plugins must be loaded in ckan.plugins)
ckan.views.default_views = image_view text_view recline_view

ckan.datapusher.formats = csv xls xlsx tsv application/csv application/vnd.ms-excel application/vnd.openxmlformats-officedocument.spreadsheetml.sheet
```

### Please describe the actual behaviour

I checked the docker-compose logs ckan command and it showed the following output:


```
 2019-12-31 01:52:57,225 ERROR [ckan.config.middleware.flask_app] 'NoneType' object has no attribute 'transaction'
ckan          | Traceback (most recent call last):
ckan          |   File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/flask/app.py", line 1949, in full_dispatch_request
ckan          |     rv = self.dispatch_request()
ckan          |   File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/flask/app.py", line 1935, in dispatch_request
ckan          |     return self.view_functions[rule.endpoint](**req.view_args)
ckan          |   File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/flask/views.py", line 89, in view
ckan          |     return self.dispatch_request(*args, **kwargs)
ckan          |   File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/flask/views.py", line 163, in dispatch_request
ckan          |     return meth(*args, **kwargs)
ckan          |   File "/usr/lib/ckan/venv/src/ckan/ckan/views/resource.py", line 242, in post
ckan          |     get_action(u'resource_create')(context, data)
ckan          |   File "/usr/lib/ckan/venv/src/ckan/ckan/logic/__init__.py", line 469, in wrapped
ckan          |     result = _action(context, data_dict, **kw)
ckan          |   File "/usr/lib/ckan/venv/src/ckan/ckan/logic/action/create.py", line 327, in resource_create
ckan          |     model.repo.commit()
ckan          |   File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/sqlalchemy/orm/scoping.py", line 162, in do
ckan          |     return getattr(self.registry(), name)(*args, **kwargs)
ckan          |   File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 1027, in commit
ckan          |     self.transaction.commit()
ckan          |   File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 494, in commit
ckan          |     self._prepare_impl()
ckan          |   File "/usr/lib/ckan/venv/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 464, in _prepare_impl
ckan          |     stx = self.session.transaction
ckan          | AttributeError: 'NoneType' object has no attribute 'transaction'
ckan          | 2019-12-31 01:52:57,341 INFO  [ckan.config.middleware.flask_app]  /dataset/testdataset07/resource/new render time 3.881 seconds
ckan          | 2019-12-31 01:52:57,624 INFO  [ckan.config.middleware.flask_app]  /api/i18n/en render time 0.001 seconds
ckan          | db:5432 - accepting connections
ckan          | /usr/lib/ckan/venv/local/lib/python2.7/site-packages/webassets/loaders.py:162: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
ckan          |   obj = self.yaml.load(f) or {}
ckan          | Initialising DB: SUCCESS
ckan          | 2019-12-31 01:53:31,960 INFO  [ckan.config.environment] Loading static files from public
ckan          | /usr/lib/ckan/venv/local/lib/python2.7/site-packages/webassets/loaders.py:162: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
ckan          |   obj = self.yaml.load(f) or {}
ckan          | 2019-12-31 01:53:32,004 INFO  [ckan.config.environment] Loading templates from /usr/lib/ckan/venv/src/ckan/ckan/templates
```

this is datapusher log
```
datapusher    | Deleting "8da3ff38-7c78-45ad-bff0-cd84ee76546e" from datastore.
datapusher    | Determined headers and types: [{'type': u'text', 'id': u'Region'}, {'type': u'text', 'id': u'Country'}, {'type': u'text', 'id': u'Item Type'}, {'type': u'text', 'id': u'Sales Channel'}, {'type': u'text', 'id': u'Order Priority'}, {'type': u'timestamp', 'id': u'Order Date'}, {'type': u'numeric', 'id': u'Order ID'}, {'type': u'timestamp', 'id': u'Ship Date'}, {'type': u'numeric', 'id': u'Units Sold'}, {'type': u'numeric', 'id': u'Unit Price'}, {'type': u'numeric', 'id': u'Unit Cost'}, {'type': u'numeric', 'id': u'Total Revenue'}, {'type': u'numeric', 'id': u'Total Cost'}, {'type': u'numeric', 'id': u'Total Profit'}]
datapusher    | Saving chunk 0
datapusher    | Successfully pushed 100 entries to "8da3ff38-7c78-45ad-bff0-cd84ee76546e".
```

### What steps can be taken to reproduce the issue?




Discussion:
I solved it by changing it to a stable version.
```
git checkout tags/ckan-2.8.2
```

Resolving patch:
diff --git a/ckanext/datapusher/plugin.py b/ckanext/datapusher/plugin.py
--- a/ckanext/datapusher/plugin.py
+++ b/ckanext/datapusher/plugin.py
@@ -37,7 +37,7 @@ class DatapusherPlugin(p.SingletonPlugin):
     p.implements(p.IActions)
     p.implements(p.IAuthFunctions)
     p.implements(p.IResourceUrlChange)
-    p.implements(p.IDomainObjectModification, inherit=True)
+    p.implements(p.IResourceController, inherit=True)
     p.implements(p.ITemplateHelpers)
     p.implements(p.IBlueprint)
 
@@ -65,60 +65,80 @@ def configure(self, config):
                     format(config_option)
                 )
 
-    def notify(self, entity, operation=None):
-        if isinstance(entity, model.Resource):
-            if (
-                operation == model.domain_object.DomainObjectOperation.new
-                or not operation
-            ):
-                # if operation is None, resource URL has been changed, as
-                # the notify function in IResourceUrlChange only takes
-                # 1 parameter
-                context = {
-                    u'model': model,
-                    u'ignore_auth': True,
-                    u'defer_commit': True
+    # IResourceUrlChange
+
+    def notify(self, resource):
+        context = {
+            u'model': model,
+            u'ignore_auth': True,
+        }
+        resource_dict = toolkit.get_action(u'resource_show')(
+            context, {
+                u'id': resource.id,
+            }
+        )
+        self._submit_to_datapusher(resource_dict)
+
+    # IResourceController
+
+    def after_create(self, context, resource_dict):
+
+        self._submit_to_datapusher(resource_dict)
+
+    def _submit_to_datapusher(self, resource_dict):
+
+        context = {
+            u'model': model,
+            u'ignore_auth': True,
+            u'defer_commit': True
+        }
+
+        resource_format = resource_dict.get('format')
+
+        submit = (
+            resource_format
+            and resource_format.lower() in self.datapusher_formats
+            and resource_dict.get('url_type') != u'datapusher'
+        )
+
+        if not submit:
+            return
+
+        try:
+            task = toolkit.get_action(u'task_status_show')(
+                context, {
+                    u'entity_id': resource_dict['id'],
+                    u'task_type': u'datapusher',
+                    u'key': u'datapusher'
+                }
+            )
+
+            if task.get(u'state') in (u'pending', u'submitting'):
+                # There already is a pending DataPusher submission,
+                # skip this one ...
+                log.debug(
+                    u'Skipping DataPusher submission for '
+                    u'resource {0}'.format(resource_dict['id'])
+                )
+                return
+        except toolkit.ObjectNotFound:
+            pass
+
+        try:
+            log.debug(
+                u'Submitting resource {0}'.format(resource_dict['id']) +
+                u' to DataPusher'
+            )
+            toolkit.get_action(u'datapusher_submit')(
+                context, {
+                    u'resource_id': resource_dict['id']
                 }
-                if (
-                    entity.format
-                    and entity.format.lower() in self.datapusher_formats
-                    and entity.url_type != u'datapusher'
-                ):
-
-                    try:
-                        task = toolkit.get_action(u'task_status_show')(
-                            context, {
-                                u'entity_id': entity.id,
-                                u'task_type': u'datapusher',
-                                u'key': u'datapusher'
-                            }
-                        )
-                        if task.get(u'state') == u'pending':
-                            # There already is a pending DataPusher submission,
-                            # skip this one ...
-                            log.debug(
-                                u'Skipping DataPusher submission for '
-                                u'resource {0}'.format(entity.id)
-                            )
-                            return
-                    except toolkit.ObjectNotFound:
-                        pass
-
-                    try:
-                        log.debug(
-                            u'Submitting resource {0}'.format(entity.id) +
-                            u' to DataPusher'
-                        )
-                        toolkit.get_action(u'datapusher_submit')(
-                            context, {
-                                u'resource_id': entity.id
-                            }
-                        )
-                    except toolkit.ValidationError as e:
-                        # If datapusher is offline want to catch error instead
-                        # of raising otherwise resource save will fail with 500
-                        log.critical(e)
-                        pass
+            )
+        except toolkit.ValidationError as e:
+            # If datapusher is offline want to catch error instead
+            # of raising otherwise resource save will fail with 500
+            log.critical(e)
+            pass
 
     def get_actions(self):
         return {


Test patch:
diff --git a/ckanext/datapusher/tests/test.py b/ckanext/datapusher/tests/test.py
--- a/ckanext/datapusher/tests/test.py
+++ b/ckanext/datapusher/tests/test.py
@@ -11,6 +11,7 @@
 import ckan.model as model
 import ckan.plugins as p
 import ckan.tests.legacy as tests
+from ckan.tests import factories
 import ckanext.datastore.backend.postgres as db
 from ckan.common import config
 from ckanext.datastore.tests.helpers import set_url_type
@@ -286,3 +287,70 @@ def test_custom_callback_url_base(self, app):
             data["result_url"]
             == "https://ckan.example.com/api/3/action/datapusher_hook"
         )
+
+    @responses.activate
+    @pytest.mark.ckan_config(
+        "ckan.datapusher.callback_url_base", "https://ckan.example.com"
+    )
+    @pytest.mark.ckan_config(
+        "ckan.datapusher.url", "http://datapusher.ckan.org"
+    )
+    @pytest.mark.ckan_config("ckan.plugins", "datastore datapusher")
+    @pytest.mark.usefixtures("with_plugins")
+    def test_create_resource_hooks(self, app):
+
+        responses.add(
+            responses.POST,
+            "http://datapusher.ckan.org/job",
+            content_type="application/json",
+            body=json.dumps({"job_id": "foo", "job_key": "barloco"}),
+        )
+        responses.add_passthru(config["solr_url"])
+
+        dataset = factories.Dataset()
+        resource = tests.call_action_api(
+            app,
+            "resource_create",
+            apikey=self.sysadmin_user.apikey,
+            package_id=dataset['id'],
+            format='CSV',
+        )
+
+    @responses.activate
+    @pytest.mark.ckan_config(
+        "ckan.datapusher.callback_url_base", "https://ckan.example.com"
+    )
+    @pytest.mark.ckan_config(
+        "ckan.datapusher.url", "http://datapusher.ckan.org"
+    )
+    @pytest.mark.ckan_config("ckan.plugins", "datastore datapusher")
+    @pytest.mark.usefixtures("with_plugins")
+    def test_update_resource_url_hooks(self, app):
+
+        responses.add(
+            responses.POST,
+            "http://datapusher.ckan.org/job",
+            content_type="application/json",
+            body=json.dumps({"job_id": "foo", "job_key": "barloco"}),
+        )
+        responses.add_passthru(config["solr_url"])
+
+        dataset = factories.Dataset()
+        resource = tests.call_action_api(
+            app,
+            "resource_create",
+            apikey=self.sysadmin_user.apikey,
+            package_id=dataset['id'],
+            url='http://example.com/old.csv',
+            format='CSV',
+        )
+
+        resource = tests.call_action_api(
+            app,
+            "resource_update",
+            apikey=self.sysadmin_user.apikey,
+            id=resource['id'],
+            url='http://example.com/new.csv',
+            format='CSV',
+        )
+        assert resource