Repository: ckan/ckan

Issue:
Column name length limit for datastore upload
I'm running a pretty vanilla instance of CKAN 2.3, and am happily uploading CSV files into the datastore.

I then hit this error on one of my CSV files:

Error: CKAN DataStore bad response. Status code: 409 Conflict. At: http://localhost:8080/api/3/action/datastore_create. 
HTTP status code: 409 
Response: {"help": "http://localhost:8080/api/3/action/help_show?name=datastore_create", "success": false, "error": {"records": ["row \"1\" has extra keys \"yjFai PSsU iTcJHt (mnCceKYD mO czqaAIa NboYAkitvNiU R... 
Requested URL: http://localhost:8080/api/3/action/datastore_create

As it turns out, one of the column headings exceeded 63 characters. When I reduce the number of characters for that column, it uploads fine.

The CSV I am using is a bit sensitive, so I have sanitized it. The structure however is preserved and this particular CSV causes the problem also:

YJdpEvm,EoaU,sduZzPFs,PkQpw gnTwE YpbXCM (Fysaa zjYbgaF) (f),yjFai PSsU iTcJHt (mnCceKYD mO czqaAIa NboYAkitvNiU Ropv) (FUEnda)
pIBYzL,19 WFKNmKL 1417,MFTZU,2.71,1.74
tvqVeL,35 lAcFOPY 3899,PEpOtKH PcE,7,-5.2
NLLKtc,32 BPUlrhQ 7605,RqYiTmB ymWTc,3.79,8.36
qNByi,78 ngSvYFO 1078,JMikZ jIEMHQc,4.3,7.35
UOvnE,94 XwwsknD 3620,CksWVV lOri,9.8,0.00
Zpg,PrZZB 8694,IAlZxEm,3.55,-4.8
CpzUty,56 JdDKDqHQ 4203,uJtxSFw uuq,3.1,2.4
lWmVUT,62 fhDDVahO 1141,LHHP ltvn,2,-9.63

If you change the last word in the heading line (FUEnda) to simply (F), it uploads fine.

So, two questions:

Is there a hard limit for column headings for the datastorer function?
If so, is that configurable?

Much thanks for any assistance.

Logs are below:

Snippet from datapusher.error.log:
[Mon Dec 21 11:45:26 2015] [error] Deleting "indicator-4-2-0-5-1" from datastore.
[Mon Dec 21 11:45:26 2015] [error] Determined headers and types: [{'type': u'text', 'id': u'YJdpEvm'}, {'type': u'text', 'id': u'EoaU'}, {'type': u'text', 'id': u'sduZzPFs'}, {'type': u'numeric', 'id': u'PkQpw gnTwE YpbXCM (Fysaa zjYbgaF) (f)'}, {'type': u'numeric', 'id': u'yjFai PSsU iTcJHt (mnCceKYD mO czqaAIa NboYAkitvNiU Ropv) (FUEnda)'}]
[Mon Dec 21 11:45:26 2015] [error] Saving chunk 0
[Mon Dec 21 11:45:26 2015] [error] Job "push_to_datastore (trigger: RunTriggerNow, run = True, next run at: None)" raised an exception
[Mon Dec 21 11:45:26 2015] [error] Traceback (most recent call last):
[Mon Dec 21 11:45:26 2015] [error]   File "/usr/lib/ckan/datapusher/lib/python2.7/site-packages/apscheduler/scheduler.py", line 512, in _run_job
[Mon Dec 21 11:45:26 2015] [error]     retval = job.func(_job.args, *_job.kwargs)
[Mon Dec 21 11:45:26 2015] [error]   File "/usr/lib/ckan/datapusher/src/datapusher/datapusher/jobs.py", line 387, in push_to_datastore
[Mon Dec 21 11:45:26 2015] [error]     records, api_key, ckan_url)
[Mon Dec 21 11:45:26 2015] [error]   File "/usr/lib/ckan/datapusher/src/datapusher/datapusher/jobs.py", line 203, in send_resource_to_datastore
[Mon Dec 21 11:45:26 2015] [error]     check_response(r, url, 'CKAN DataStore')
[Mon Dec 21 11:45:26 2015] [error]   File "/usr/lib/ckan/datapusher/src/datapusher/datapusher/jobs.py", line 137, in check_response
[Mon Dec 21 11:45:26 2015] [error]     request_url=request_url, response=response.text)
[Mon Dec 21 11:45:26 2015] [error] HTTPError

Snippet from ckan_default.error.log:
[Mon Dec 21 11:49:56 2015] [error] 2015-12-21 11:49:56,820 ERROR [ckan.controllers.api] Validation error: '{\'records\': [u\'row "1" has extra keys "yjFai PSsU iTcJHt (mnCceKYD mO czqaAIa NboYAkitvNiU Ropv) (FUEnda)"\'], \'__type\': \'Validation Error\'}'
[Mon Dec 21 11:49:56 2015] [error] 2015-12-21 11:49:56,821 INFO  [ckan.lib.base]  /api/3/action/datastore_create render time 0.051 seconds
[Mon Dec 21 11:49:56 2015] [error] 2015-12-21 11:49:56,885 INFO  [ckan.lib.base]  /dataset/soe2015-storm-tide-inundation-4-2-0-5/resource_data/indicator-4-2-0-5-1 render time 0.290 seconds

...

[Mon Dec 21 11:50:04 2015] [error] [client 127.0.0.1] mod_wsgi (pid=19910): Exception occurred processing WSGI script '/etc/ckan/default/apache.wsgi'.
[Mon Dec 21 11:50:04 2015] [error] [client 127.0.0.1] IOError: failed to write data
[Mon Dec 21 11:50:04 2015] [error] [client 127.0.0.1] mod_wsgi (pid=19911): Exception occurred processing WSGI script '/etc/ckan/default/apache.wsgi'.
[Mon Dec 21 11:50:04 2015] [error] [client 127.0.0.1] IOError: failed to write data
[Mon Dec 21 11:50:04 2015] [error] [client 127.0.0.1] mod_wsgi (pid=19911): Exception occurred processing WSGI script '/etc/ckan/default/apache.wsgi'.
[Mon Dec 21 11:50:04 2015] [error] [client 127.0.0.1] IOError: failed to write data
[Mon Dec 21 11:50:04 2015] [error] [client 127.0.0.1] mod_wsgi (pid=19911): Exception occurred processing WSGI script '/etc/ckan/default/apache.wsgi'.
[Mon Dec 21 11:50:04 2015] [error] [client 127.0.0.1] IOError: failed to write data



Discussion:
The postgres maximum length for a column name is 63 characters, this probably isn't a coincidence. Not sure what could be done about it, other than warning the user.

Thanks. Good to know.

During the tech call we decided that pretty much all we can do about this is to clarify the error. I'll leave it open for now in case somebody would like to contribute a fix.

Hey folks, what about ckan shows a fat error message in the UI saying "Column headers cannot contain more than 63 characters" when a user attempts uploading such a file. I just hit this, and it sure wasn't easy tracking the errors all the way here. Thanks!

@rossjones I'll work on this.

Resolving patch:
diff --git a/ckanext/datastore/backend/postgres.py b/ckanext/datastore/backend/postgres.py
--- a/ckanext/datastore/backend/postgres.py
+++ b/ckanext/datastore/backend/postgres.py
@@ -855,6 +855,20 @@ def create_table(context, data_dict):
     field_ids = _pluck('id', supplied_fields)
     records = data_dict.get('records')
 
+    fields_errors = []
+
+    for field_id in field_ids:
+        # Postgres has a limit of 63 characters for a column name
+        if len(field_id) > 63:
+            message = 'Column heading "{0}" exceeds limit of 63 '\
+                'characters.'.format(field_id)
+            fields_errors.append(message)
+
+    if fields_errors:
+        raise ValidationError({
+            'fields': fields_errors
+        })
+
     # if type is field is not given try and guess or throw an error
     for field in supplied_fields:
         if 'type' not in field:


Test patch:
diff --git a/ckanext/datastore/tests/test_create.py b/ckanext/datastore/tests/test_create.py
--- a/ckanext/datastore/tests/test_create.py
+++ b/ckanext/datastore/tests/test_create.py
@@ -240,6 +240,21 @@ def test_sets_datastore_active_on_resource_on_delete(self):
 
         assert_equal(resource['datastore_active'], False)
 
+    @raises(p.toolkit.ValidationError)
+    def test_create_exceeds_column_name_limit(self):
+        package = factories.Dataset()
+        data = {
+            'resource': {
+                'package_id': package['id']
+            },
+            'fields': [{
+                'id': 'This is a really long name for a column. Column names '
+                'in Postgres have a limit of 63 characters',
+                'type': 'text'
+            }]
+        }
+        result = helpers.call_action('datastore_create', **data)
+
 
 class TestDatastoreCreate(tests.WsgiAppCase):
     sysadmin_user = None