Repository: ckan/ckan

Issue:
Datapusher error when root_path is set
Hi,

After setting **root_path** variable in CKAN config file, Datapusher no longer works. We always get:

```
Error: Process completed but unable to post to result_url 
```

Contents of datapusher.error.log:

```
[Fri Feb 05 09:20:00.905675 2016] [:error] [pid 5997:tid 140341959157632] Exception AttributeError: "'NoneType' object has no attribute 'Error'" in <generator object raw at 0x7fa3d9384410> ignored
[Fri Feb 05 11:16:52.104411 2016] [:error] [pid 1502:tid 140224457529088] /usr/lib/ckan/datapusher/lib/python2.7/site-packages/sqlalchemy/sql/sqltypes.py:185: SAWarning: Unicode type received non-unicode bind param value 'ca8c86c6-ca87-4b36-b6bb-3...'. (this warning may be suppressed after 10 occurrences)
[Fri Feb 05 11:16:52.119255 2016] [:error] [pid 1502:tid 140224457529088]   (util.ellipses_string(value),))
[Fri Feb 05 11:16:57.624008 2016] [:error] [pid 1502:tid 140224457529088] Job "push_to_datastore (trigger: RunTriggerNow, run = True, next run at: None)" raised an exception
[Fri Feb 05 11:16:57.624065 2016] [:error] [pid 1502:tid 140224457529088] Traceback (most recent call last):
[Fri Feb 05 11:16:57.624096 2016] [:error] [pid 1502:tid 140224457529088]   File "/usr/lib/ckan/datapusher/lib/python2.7/site-packages/apscheduler/scheduler.py", line 512, in _run_job
[Fri Feb 05 11:16:57.624106 2016] [:error] [pid 1502:tid 140224457529088]     retval = job.func(*job.args, **job.kwargs)
[Fri Feb 05 11:16:57.624116 2016] [:error] [pid 1502:tid 140224457529088]   File "/usr/lib/ckan/datapusher/src/datapusher/datapusher/jobs.py", line 287, in push_to_datastore
[Fri Feb 05 11:16:57.624125 2016] [:error] [pid 1502:tid 140224457529088]     resource = get_resource(resource_id, ckan_url, api_key)
[Fri Feb 05 11:16:57.624134 2016] [:error] [pid 1502:tid 140224457529088]   File "/usr/lib/ckan/datapusher/src/datapusher/datapusher/jobs.py", line 233, in get_resource
[Fri Feb 05 11:16:57.624143 2016] [:error] [pid 1502:tid 140224457529088]     check_response(r, url, 'CKAN')
[Fri Feb 05 11:16:57.624151 2016] [:error] [pid 1502:tid 140224457529088]   File "/usr/lib/ckan/datapusher/src/datapusher/datapusher/jobs.py", line 144, in check_response
[Fri Feb 05 11:16:57.624160 2016] [:error] [pid 1502:tid 140224457529088]     response=response.text)
[Fri Feb 05 11:16:57.624168 2016] [:error] [pid 1502:tid 140224457529088] HTTPError
```

If we don't use the root_path variable and set our subpath in site_url then Datapusher works again but some CKAN extensions like **archiver/qa** get broken as they need the **root_path** variable to get the subpath.

We think this is related to #2599

Any help to fix this would be appreciated. Thank you in advance.



Discussion:
The `result_url` is the callback URL that CKAN tells the DataPusher to ping once it has finished its importing. It should point to the  `/api/3/action/datapusher_hook` endpoint. Right now this URL is constructed using [`ckan.site_url`](https://github.com/ckan/ckan/blob/master/ckanext/datapusher/logic/action.py#L52:L53) and my gut feeling is that this is the correct approach.

I think there is definitely some confusion regarding `ckan.site_url` and `ckan.root_path`.
@wardi [here](https://github.com/ckan/ckanext-qa/issues/20#issuecomment-153342639) you seem to suggest that `ckan.site_url` should not have path components but that's not what the [docs](http://docs.ckan.org/en/ckan-2.5.1/maintaining/configuration.html#ckan-site-url) say and a quick grep over the code shows that is generally used as being the full base url of the CKAN site ([eg](https://github.com/ckan/ckan/blob/master/ckan/lib/mailer.py#L138:L139)).

I'm not sure what `ckan.root_path` was intended for (I guess for some i18n stuff) but I think that if extensions are relying on it to build URLs that's probably wrong

http://docs.ckan.org/en/ckan-2.5.1/maintaining/configuration.html#ckan-site-url

@amercader here's where I learned about root_path: https://github.com/ckan/ckan/pull/2599#issuecomment-134595167
Given that routes supports defining the structure of the complete url from hostname down with this variable, it's incorrect to build urls with site_url alone.  We need to use url_for which should take root_path into account. I'd say the docs and other places in the code where urls are constructed manually need to be updated.

So, which is the best approach? Setting `ckan.site_url` (then some extensions don't work) or setting `ckan.root_path` (then DataPusher fails)?

I'm facing with the same issue.


Resolving patch:
diff --git a/ckanext/datapusher/logic/action.py b/ckanext/datapusher/logic/action.py
--- a/ckanext/datapusher/logic/action.py
+++ b/ckanext/datapusher/logic/action.py
@@ -9,6 +9,7 @@
 
 import requests
 
+import ckan.lib.helpers as h
 import ckan.lib.navl.dictization_functions
 import ckan.logic as logic
 import ckan.plugins as p
@@ -59,8 +60,8 @@ def datapusher_submit(context, data_dict):
 
     datapusher_url = config.get('ckan.datapusher.url')
 
-    site_url = config['ckan.site_url']
-    callback_url = site_url.rstrip('/') + '/api/3/action/datapusher_hook'
+    site_url = h.url_for('/', qualified=True)
+    callback_url = h.url_for('/api/3/action/datapusher_hook', qualified=True)
 
     user = p.toolkit.get_action('user_show')(context, {'id': context['user']})
 


Test patch: