Repository: ckan/ckan

Issue:
datapusher and ckan in docker-compose
CKAN Version 2.8.3

I've been following the changes in #4879 as I have the same problem running datapusher and ckan both in docker-compose cluster where the public ckan url is not available. 

When running datapusher in a docker cluster generated by docker-compose the public ckan site url is not available to the local datapusher server - neither is it localhost.  So we need to give the datapusher the correct callback url to update ckan. 

Currently it always uses the site url and this fails to update.

datapusher is still using the site_url instead of the url set by 

    CKAN_DATAPUSHER_URL=http://datapusher:8800
    CKAN__DATAPUSHER__CALLBACK_URL_BASE=http://ckan:5000

which is the internal docker-compose name (and works)

Error log is 

    datapusher    | --------------------------------------------------------------------------------
    datapusher    | ERROR in scheduler [/usr/lib/python2.7/site-packages/apscheduler/scheduler.py:520]:
    datapusher    | Job "push_to_datastore (trigger: RunTriggerNow, run = True, next run at: None)" raised an exception
    datapusher    | --------------------------------------------------------------------------------
    datapusher    | ConnectionError: HTTPConnectionPool(host='ckan.dev.pfr.co.nz', port=5000): Max retries exceeded with url: /api/3/action/resource_show (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7feb6b45f5d0>: Failed to establish a new connection: [Errno 113] Host is unreachable',))

CKAN__DATAPUSHER__CALLBACK_URL_BASE is read into the system in 
**Action.py datapusher_submit**

    datapusher_url = config.get('ckan.datapusher.url')
    callback_url_base = config.get('ckan.datapusher.callback_url_base')
        if callback_url_base:
            callback_url = urlparse.urljoin(
                callback_url_base.rstrip('/'), '/api/3/action/datapusher_hook')
        else:
            callback_url = h.url_for(
                '/api/3/action/datapusher_hook', qualified=True)

So assuming the config is read correctly callback_url_base = http://ckan:5000
and thus callback_url = http://ckan:5000/api/3/action/datapusher_hook

Adding log statements we get 
ckan          | 2019-07-25 11:46:02,882 INFO  [ckanext.datapusher.logic.action] http://ckan:5000
ckan          | 2019-07-25 11:46:02,882 INFO  [ckanext.datapusher.logic.action] http://ckan:5000/api/3/action/datapusher_hook

confirming value of callback_url

Next a POST request is generated to the datapusher with a body 

    data=json.dumps({
                    'api_key': user['apikey'],
                    'job_type': 'push_to_datastore',
                    'result_url': callback_url,
                    'metadata': {
                        'ignore_hash': data_dict.get('ignore_hash', False),
                        'ckan_url': site_url,
                        'resource_id': res_id,
                        'set_url_type': data_dict.get('set_url_type', False),
                        'task_created': task['last_updated'],
                        'original_url': resource_dict.get('url'),
                    }
                }))

Note that result_url = callback_url and ckan_url = site_url which is ckan.dev.pfr.co.nz:5000

This body is then processed in the datapusher service
**jobs.py - push_to_datastore**

These values are pulled from the input body

data = input['metadata']
    ckan_url = data['ckan_url']
    resource_id = data['resource_id']
    api_key = input.get('api_key')

so ckan_url is still http://ckan.dev.pfr.co.nz:5000 and not http://ckan:5000

Where is the value of 'callback_url' read or used?
as far as I can see 'result_url' is never used. ckan_url is used exclusively.

Is there a different version of datapusher I should be using that makes use of the callback_url or has this bit not been done yet?

Thanks Andrew




 
datapusher and ckan in docker-compose
CKAN Version 2.8.3

I've been following the changes in #4879 as I have the same problem running datapusher and ckan both in docker-compose cluster where the public ckan url is not available. 

When running datapusher in a docker cluster generated by docker-compose the public ckan site url is not available to the local datapusher server - neither is it localhost.  So we need to give the datapusher the correct callback url to update ckan. 

Currently it always uses the site url and this fails to update.

datapusher is still using the site_url instead of the url set by 

    CKAN_DATAPUSHER_URL=http://datapusher:8800
    CKAN__DATAPUSHER__CALLBACK_URL_BASE=http://ckan:5000

which is the internal docker-compose name (and works)

Error log is 

    datapusher    | --------------------------------------------------------------------------------
    datapusher    | ERROR in scheduler [/usr/lib/python2.7/site-packages/apscheduler/scheduler.py:520]:
    datapusher    | Job "push_to_datastore (trigger: RunTriggerNow, run = True, next run at: None)" raised an exception
    datapusher    | --------------------------------------------------------------------------------
    datapusher    | ConnectionError: HTTPConnectionPool(host='ckan.dev.pfr.co.nz', port=5000): Max retries exceeded with url: /api/3/action/resource_show (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7feb6b45f5d0>: Failed to establish a new connection: [Errno 113] Host is unreachable',))

CKAN__DATAPUSHER__CALLBACK_URL_BASE is read into the system in 
**Action.py datapusher_submit**

    datapusher_url = config.get('ckan.datapusher.url')
    callback_url_base = config.get('ckan.datapusher.callback_url_base')
        if callback_url_base:
            callback_url = urlparse.urljoin(
                callback_url_base.rstrip('/'), '/api/3/action/datapusher_hook')
        else:
            callback_url = h.url_for(
                '/api/3/action/datapusher_hook', qualified=True)

So assuming the config is read correctly callback_url_base = http://ckan:5000
and thus callback_url = http://ckan:5000/api/3/action/datapusher_hook

Adding log statements we get 
ckan          | 2019-07-25 11:46:02,882 INFO  [ckanext.datapusher.logic.action] http://ckan:5000
ckan          | 2019-07-25 11:46:02,882 INFO  [ckanext.datapusher.logic.action] http://ckan:5000/api/3/action/datapusher_hook

confirming value of callback_url

Next a POST request is generated to the datapusher with a body 

    data=json.dumps({
                    'api_key': user['apikey'],
                    'job_type': 'push_to_datastore',
                    'result_url': callback_url,
                    'metadata': {
                        'ignore_hash': data_dict.get('ignore_hash', False),
                        'ckan_url': site_url,
                        'resource_id': res_id,
                        'set_url_type': data_dict.get('set_url_type', False),
                        'task_created': task['last_updated'],
                        'original_url': resource_dict.get('url'),
                    }
                }))

Note that result_url = callback_url and ckan_url = site_url which is ckan.dev.pfr.co.nz:5000

This body is then processed in the datapusher service
**jobs.py - push_to_datastore**

These values are pulled from the input body

data = input['metadata']
    ckan_url = data['ckan_url']
    resource_id = data['resource_id']
    api_key = input.get('api_key')

so ckan_url is still http://ckan.dev.pfr.co.nz:5000 and not http://ckan:5000

Where is the value of 'callback_url' read or used?
as far as I can see 'result_url' is never used. ckan_url is used exclusively.

Is there a different version of datapusher I should be using that makes use of the callback_url or has this bit not been done yet?

Thanks Andrew




 


Discussion:
Hi,
Facing the same issue here. Tried to fix it but got stuck.

Replacing  `'ckan_url': site_url`,  in ckanext/datapusher/logic/action.py with `ckan_url': callback_url_base` does not work as `callback_url_base = config.get('ckan.datapusher.callback_url_base')`  assigns None.

Fixed it by adding  `'ckan.datapusher.callback_url_base': 'CKAN__DATAPUSHER__CALLBACK_URL_BASE'`  at line 157 in in ckan/config/environment.py

However in datapusher repo 
File: datapusher/jobs.py
Method:  push_to_datastore
 ```
  try:
        resource = get_resource(resource_id, ckan_url, api_key)
    except util.JobError, e:
        #try again in 5 seconds just incase CKAN is slow at adding resource
        time.sleep(5)
        resource = get_resource(resource_id, ckan_url, api_key)
    # check if the resource url_type is a datastore
    if resource.get('url_type') == 'datastore':
        logger.info('Dump files are managed with the Datastore API')
        return
    # check scheme
    url = resource.get('url')
    scheme = urlparse.urlsplit(url).scheme 
```
`url = resource.get('url')` is called for pushing which still contains the site_url. Not sure about this fix.


Just noting for myself that until this is fixed I have to set the expected ckan ip in /etc/hosts/ of the datapusher container
Hi,
Facing the same issue here. Tried to fix it but got stuck.

Replacing  `'ckan_url': site_url`,  in ckanext/datapusher/logic/action.py with `ckan_url': callback_url_base` does not work as `callback_url_base = config.get('ckan.datapusher.callback_url_base')`  assigns None.

Fixed it by adding  `'ckan.datapusher.callback_url_base': 'CKAN__DATAPUSHER__CALLBACK_URL_BASE'`  at line 157 in in ckan/config/environment.py

However in datapusher repo 
File: datapusher/jobs.py
Method:  push_to_datastore
 ```
  try:
        resource = get_resource(resource_id, ckan_url, api_key)
    except util.JobError, e:
        #try again in 5 seconds just incase CKAN is slow at adding resource
        time.sleep(5)
        resource = get_resource(resource_id, ckan_url, api_key)
    # check if the resource url_type is a datastore
    if resource.get('url_type') == 'datastore':
        logger.info('Dump files are managed with the Datastore API')
        return
    # check scheme
    url = resource.get('url')
    scheme = urlparse.urlsplit(url).scheme 
```
`url = resource.get('url')` is called for pushing which still contains the site_url. Not sure about this fix.


Just noting for myself that until this is fixed I have to set the expected ckan ip in /etc/hosts/ of the datapusher container

Resolving patch:
diff --git a/ckanext/datapusher/logic/action.py b/ckanext/datapusher/logic/action.py
--- a/ckanext/datapusher/logic/action.py
+++ b/ckanext/datapusher/logic/action.py
@@ -61,13 +61,13 @@ def datapusher_submit(context, data_dict):
 
     datapusher_url = config.get('ckan.datapusher.url')
 
-    site_url = h.url_for('/', qualified=True)
-
     callback_url_base = config.get('ckan.datapusher.callback_url_base')
     if callback_url_base:
+        site_url = callback_url_base
         callback_url = urljoin(
             callback_url_base.rstrip('/'), '/api/3/action/datapusher_hook')
     else:
+        site_url = h.url_for('/', qualified=True)
         callback_url = h.url_for(
             '/api/3/action/datapusher_hook', qualified=True)
 


Test patch: