Repository: ckan/ckan

Issue:
Latin characters not accepted in JSON DataStore fields
### CKAN version if known (or site URL)

2.3 (private deployment)
### Please describe the expected behaviour

The following dataset upsert should return `200 OK`:

```
> curl -X POST -k -H "Content-Type: application/json" -H "Accept: application/json" -H "Authorization: MY_API_KEY" -H "Cache-Control:  no-cache" -d '{
  "resource_id": "bba47654-df03-4c43-b2ff-b0f144223101",
  "records": [
    {
      "recvTime": "2016-04-06T16:42:02.533333",
      "temperature": "ñ"
    }
  ],
  "method": "insert",
  "force": "true"
}' http://10.0.0.2:5055/api/3/action/datastore_upsert
```
### Please describe the actual behaviour

```
500 Inernal Server Error

<html>
    <head>
    <title>Server Error</title>

    </head>
    <body>
    <h1>Server Error</h1>
    An internal server error occurred

    </body>
</html>
```

The following traces were taken from the log:

```
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] Error - <type 'exceptions.UnicodeEncodeError'>: 'ascii' codec can't encode character u'\\xf1' in position 0: ordinal not in range(128)
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] URL: https://192.168.21.64:8443/api/3/action/datastore_upsert
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] File '/usr/local/ckan/venv_valencia/lib/python2.6/site-packages/weberror/errormiddleware.py', line 162 in __call__
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   app_iter = self.application(environ, sr_checker)
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] File '/usr/local/ckan/venv_valencia/lib/python2.6/site-packages/webob/dec.py', line 147 in __call__
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   resp = self.call_func(req, *args, **self.kwargs)
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] File '/usr/local/ckan/venv_valencia/lib/python2.6/site-packages/webob/dec.py', line 208 in call_func
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   return self.func(req, *args, **kwargs)
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] File '/usr/local/ckan/venv_valencia/lib/python2.6/site-packages/fanstatic/publisher.py', line 234 in __call__
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   return request.get_response(self.app)
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] File '/usr/local/ckan/venv_valencia/lib/python2.6/site-packages/webob/request.py', line 1053 in get_response
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   application, catch_exc_info=False)
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] File '/usr/local/ckan/venv_valencia/lib/python2.6/site-packages/webob/request.py', line 1022 in call_application
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   app_iter = application(self.environ, start_response)
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] File '/usr/local/ckan/venv_valencia/lib/python2.6/site-packages/webob/dec.py', line 147 in __call__
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   resp = self.call_func(req, *args, **self.kwargs)
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] File '/usr/local/ckan/venv_valencia/lib/python2.6/site-packages/webob/dec.py', line 208 in call_func
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   return self.func(req, *args, **kwargs)
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] File '/usr/local/ckan/venv_valencia/lib/python2.6/site-packages/fanstatic/injector.py', line 54 in __call__
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   response = request.get_response(self.app)
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] File '/usr/local/ckan/venv_valencia/lib/python2.6/site-packages/webob/request.py', line 1053 in get_response
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   application, catch_exc_info=False)
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] File '/usr/local/ckan/venv_valencia/lib/python2.6/site-packages/webob/request.py', line 1022 in call_application
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   app_iter = application(self.environ, start_response)
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] File '/usr/local/ckan/venv_valencia/lib/python2.6/site-packages/beaker/middleware.py', line 73 in __call__
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   return self.app(environ, start_response)
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] File '/usr/local/ckan/venv_valencia/lib/python2.6/site-packages/beaker/middleware.py', line 155 in __call__
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   return self.wrap_app(environ, session_start_response)
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] File '/usr/local/ckan/venv_valencia/lib/python2.6/site-packages/routes/middleware.py', line 131 in __call__
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   response = self.app(environ, start_response)
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] File '/usr/local/ckan/venv_valencia/lib/python2.6/site-packages/pylons/wsgiapp.py', line 125 in __call__
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   response = self.dispatch(controller, environ, start_response)
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] File '/usr/local/ckan/venv_valencia/lib/python2.6/site-packages/pylons/wsgiapp.py', line 324 in dispatch
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   return controller(environ, start_response)
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] File '/usr/local/ckan/venv_valencia/src/ckan/ckan/controllers/api.py', line 78 in __call__
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   result =  base.BaseController.__call__(self, environ, start_response)
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] File '/usr/local/ckan/venv_valencia/src/ckan/ckan/lib/base.py', line 338 in __call__
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   res = WSGIController.__call__(self, environ, start_response)
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] File '/usr/local/ckan/venv_valencia/lib/python2.6/site-packages/pylons/controllers/core.py', line 221 in __call__
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   response = self._dispatch_call()
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] File '/usr/local/ckan/venv_valencia/lib/python2.6/site-packages/pylons/controllers/core.py', line 172 in _dispatch_call
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   response = self._inspect_call(func)
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] File '/usr/local/ckan/venv_valencia/lib/python2.6/site-packages/pylons/controllers/core.py', line 107 in _inspect_call
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   result = self._perform_call(func, args)
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] File '/usr/local/ckan/venv_valencia/lib/python2.6/site-packages/pylons/controllers/core.py', line 60 in _perform_call
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   return func(**args)
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] File '/usr/local/ckan/venv_valencia/src/ckan/ckan/controllers/api.py', line 208 in action
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   result = function(context, request_data)
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] File '/usr/local/ckan/venv_valencia/src/ckan/ckan/logic/__init__.py', line 424 in wrapped
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   result = _action(context, data_dict, **kw)
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] File '/usr/local/ckan/venv_valencia/src/ckan/ckanext/datastore/logic/action.py', line 212 in datastore_upsert
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   result = db.upsert(context, data_dict)
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] File '/usr/local/ckan/venv_valencia/src/ckan/ckanext/datastore/db.py', line 1121 in upsert
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   upsert_data(context, data_dict)
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] File '/usr/local/ckan/venv_valencia/src/ckan/ckanext/datastore/db.py', line 642 in upsert_data
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   row.append(_to_full_text(fields, record))
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] File '/usr/local/ckan/venv_valencia/src/ckan/ckanext/datastore/db.py', line 805 in _to_full_text
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   full_text.extend(json_get_values(value))
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] File '/usr/local/ckan/venv_valencia/src/ckan/ckanext/datastore/db.py', line 258 in json_get_values
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   current_list.append(str(obj))
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] UnicodeEncodeError: 'ascii' codec can't encode character u'\\xf1' in position 0: ordinal not in range(128)
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] CGI Variables
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] -------------
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   CKAN_CURRENT_URL: '/api/3/action/datastore_upsert'
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   CKAN_LANG: 'en'
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   CKAN_LANG_IS_DEFAULT: True
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   CONTENT_LENGTH: '206'
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   CONTENT_TYPE: 'application/json; charset=utf-8'
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   DOCUMENT_ROOT: '/var/www/html'
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   GATEWAY_INTERFACE: 'CGI/1.1'
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   HTTPS: '1'
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   HTTP_ACCEPT: 'application/json'
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   HTTP_AUTHORIZATION: '3d20cdae-44ad-4eeb-a8b7-8de485140133'
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   HTTP_CACHE_CONTROL: 'no-cache'
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   HTTP_CONNECTION: 'close'
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   HTTP_HOST: '192.168.21.64:8443'
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   HTTP_USER_AGENT: 'curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.18 Basic ECC zlib/1.2.3 libidn/1.18 libssh2/1.4.2'
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   PATH_INFO: '/api/3/action/datastore_upsert'
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   PATH_TRANSLATED: '/etc/ckan/valencia/apache.wsgi/api/3/action/datastore_upsert'
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   REMOTE_ADDR: '10.0.0.11'
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   REMOTE_PORT: '52839'
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   REQUEST_METHOD: 'POST'
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   REQUEST_URI: '/api/3/action/datastore_upsert'
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   SCRIPT_FILENAME: '/etc/ckan/valencia/apache.wsgi'
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   SERVER_ADDR: '10.0.0.27'
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   SERVER_ADMIN: 'root@localhost'
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   SERVER_NAME: '192.168.21.64'
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   SERVER_PORT: '8443'
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   SERVER_PROTOCOL: 'HTTP/1.0'
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   SERVER_SOFTWARE: 'Apache'
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] WSGI Variables
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] --------------
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   application: <fanstatic.publisher.Delegator object at 0x7f69d1c19250>
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   beaker.cache: <beaker.cache.CacheManager object at 0x7f69d1c19190>
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   beaker.get_session: <bound method SessionMiddleware._get_session of <beaker.middleware.SessionMiddleware object at 0x7f69d19a0750>>
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   beaker.session: {'_accessed_time': 1460985026.8013899, '_creation_time': 1460985026.8013899}
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   fanstatic.needed: <fanstatic.core.NeededResources object at 0x7f69d1c18f10>
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   mod_ssl.is_https: <built-in method ssl_is_https of mod_wsgi.Adapter object at 0x7f69cdc2e0a8>
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   mod_ssl.var_lookup: <built-in method ssl_var_lookup of mod_wsgi.Adapter object at 0x7f69cdc2e0a8>
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   mod_wsgi.application_group: 'ckan-valencia.ost:8443|'
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   mod_wsgi.callable_object: 'application'
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   mod_wsgi.handler_script: ''
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   mod_wsgi.input_chunked: '0'
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   mod_wsgi.listener_host: ''
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   mod_wsgi.listener_port: '443'
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   mod_wsgi.process_group: ''
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   mod_wsgi.request_handler: 'wsgi-script'
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   mod_wsgi.script_reloading: '1'
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   mod_wsgi.version: (3, 2)
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   paste.cookies: (<SimpleCookie: >, '')
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   paste.registry: <paste.registry.Registry object at 0x7f69d1c18c90>
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   paste.throw_errors: True
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   pylons.action_method: <bound method ApiController.action of <ckan.controllers.api.ApiController object at 0x7f69d1f51c10>>
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   pylons.controller: <ckan.controllers.api.ApiController object at 0x7f69d1f51c10>
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   pylons.environ_config: {'session': 'beaker.session', 'cache': 'beaker.cache'}
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   pylons.pylons: <pylons.util.PylonsContext object at 0x7f69d1c19210>
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   pylons.routes_dict: {'action': u'action', 'controller': u'api', 'ver': 3, 'logic_function': u'datastore_upsert'}
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   pylons.status_code_redirect: True
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   repoze.who.api: <repoze.who.api.API object at 0x7f69d1c18cd0>
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   repoze.who.logger: <logging.Logger instance at 0x7f69d1b2e7a0>
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   repoze.who.plugins: {'ckan.lib.authenticator:UsernamePasswordAuthenticator': <ckan.lib.authenticator.UsernamePasswordAuthenticator object at 0x7f69d1f51cd0>, 'friendlyform': <FriendlyFormPlugin 140092465969168>, 'auth_tkt': <CkanAuthTktCookiePlugin 140092459979728>}
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   routes.route: <routes.route.Route object at 0x7f69d1f22350>
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   routes.url: <routes.util.URLGenerator object at 0x7f69d1c19490>
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   webob._parsed_query_vars: (GET([]), '')
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   webob.adhoc_attrs: {'response': <Response at 0x7f69d1ac3050 200 OK>, 'language': 'en-us'}
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   webob.is_body_seekable: True
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   wsgi process: 'Multiprocess'
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   wsgi.file_wrapper: <built-in method file_wrapper of mod_wsgi.Adapter object at 0x7f69cdc2e0a8>
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   wsgi.version: (1, 1)
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11]   wsgiorg.routing_args: (<routes.util.URLGenerator object at 0x7f69d1c19490>, {'action': u'action', 'controller': u'api', 'ver': 3, 'logic_function': u'datastore_upsert'})
[Mon Apr 18 15:10:26 2016] [error] [client 10.0.0.11] ------------------------------------------------------------
```
### What steps can be taken to reproduce the issue?

Data upsert using a latin character: `ñ`, `ç`, `á`, etc.



Discussion:
@frbattid Can you check if the following patch fixes the issue for you?

``` diff
diff --git a/ckanext/datastore/db.py b/ckanext/datastore/db.py
index 1121c9e..895971c 100644
--- a/ckanext/datastore/db.py
+++ b/ckanext/datastore/db.py
@@ -255,7 +255,7 @@ def json_get_values(obj, current_list=None):
     elif isinstance(obj, dict):
         json_get_values(obj.items(), current_list)
     elif obj:
-        current_list.append(str(obj))
+        current_list.append(unicode(obj))
     return current_list
```

Also can you share the table structure of that DataStore resource? Something like:

```
> curl -H "Authorization: MY_API_KEY" http://10.0.0.2:5055/api/3/action/datastore_search
```

It's really bizarre because I can see the problem on the stack trace you pasted, but I didn't manage to reproduce the issue, ie I couldn't make that code execute with the same request.

@amercader this is what we got:

```
curl -h "Authorization: MY_API_KEY" http://10.0.0.2:5055/api/action/datastore_search?resource_id=bba47654-df03-4c43-b2ff-b0f144223101&limit=1

{
  "help": "https://192.168.21.64:8443/api/3/action/help_show?name=datastore_search",
  "success": true,
  "result": {
    "resource_id": "bba47654-df03-4c43-b2ff-b0f144223101",
    "fields": [
      {
        "type": "int4",
        "id": "_id"
      },
      {
        "type": "timestamp",
        "id": "recvTime"
      },
      {
        "type": "json",
        "id": "fiwareServicePath"
      },
      {
        "type": "json",
        "id": "entityType"
      },
      {
        "type": "json",
        "id": "entityId"
      },
      {
        "type": "timestamp",
        "id": "TimeInstant"
      },
      {
        "type": "json",
        "id": "temperature"
      },
      {
        "type": "json",
        "id": "temperature_md"
      },
      {
        "type": "json",
        "id": "pressure"
      },
      {
        "type": "json",
        "id": "pressure_md"
      },
      {
        "type": "json",
        "id": "presence"
      },
      {
        "type": "json",
        "id": "presence_md"
      },
      {
        "type": "json",
        "id": "humidity"
      },
      {
        "type": "json",
        "id": "humidity_md"
      },
      {
        "type": "json",
        "id": "fillLevel"
      },
      {
        "type": "json",
        "id": "fillLevel_md"
      }
    ],
    "records": [
      {
        "TimeInstant": null,
        "recvTime": "2016-03-15T11:52:37.670000",
        "temperature": "5701",
        "presence": null,
        "pressure_md": null,
        "fillLevel": null,
        "entityType": "room",
        "humidity": null,
        "pressure": null,
        "humidity_md": null,
        "fiwareServicePath": "electricidad",
        "entityId": "room008",
        "_id": 1,
        "temperature_md": null,
        "presence_md": null,
        "fillLevel_md": null
      }
    ],
    "_links": {
      "start": "/api/action/datastore_search?limit=1&resource_id=bba47654-df03-4c43-b2ff-b0f144223101",
      "next": "/api/action/datastore_search?offset=1&limit=1&resource_id=bba47654-df03-4c43-b2ff-b0f144223101"
    },
    "limit": 1,
    "total": 81534
  }
}
```

@frbattid That what I imagined, as the fields have a "json" type. Is there a reason why fields like "Temperature" are stored as JSON? Are they actually JSON values?

Of course the original issue remains, so can you confirm if the patch I pasted on my last message fixes it?

@frbattid @manucarrace see above

@amercader our fields can be a simple value or a compund value (json value), which is the case we need the type json.

On the other hand, I confirm the patch works for us. Do you plan to include it in further releases?

Thank you in advance


Resolving patch:
diff --git a/ckanext/datastore/db.py b/ckanext/datastore/db.py
--- a/ckanext/datastore/db.py
+++ b/ckanext/datastore/db.py
@@ -255,7 +255,7 @@ def json_get_values(obj, current_list=None):
     elif isinstance(obj, dict):
         json_get_values(obj.items(), current_list)
     elif obj:
-        current_list.append(str(obj))
+        current_list.append(unicode(obj))
     return current_list
 
 


Test patch:
diff --git a/ckanext/datastore/tests/test_upsert.py b/ckanext/datastore/tests/test_upsert.py
--- a/ckanext/datastore/tests/test_upsert.py
+++ b/ckanext/datastore/tests/test_upsert.py
@@ -1,3 +1,4 @@
+# -*- coding: utf-8 -*-
 import json
 import nose
 import datetime
@@ -9,6 +10,9 @@
 import ckan.lib.create_test_data as ctd
 import ckan.model as model
 import ckan.tests.legacy as tests
+import ckan.tests.helpers as helpers
+import ckan.tests.factories as factories
+
 
 import ckanext.datastore.db as db
 from ckanext.datastore.tests.helpers import rebuild_all_dbs, set_url_type
@@ -16,6 +20,62 @@
 assert_equal = nose.tools.assert_equal
 
 
+class TestDatastoreUpsertNewTests(object):
+    @classmethod
+    def setup_class(cls):
+        if not p.plugin_loaded('datastore'):
+            p.load('datastore')
+
+    @classmethod
+    def teardown_class(cls):
+        p.unload('datastore')
+        helpers.reset_db()
+
+    def test_upsert_doesnt_crash_with_json_field(self):
+        resource = factories.Resource()
+        data = {
+            'resource_id': resource['id'],
+            'force': True,
+            'primary_key': 'id',
+            'fields': [{'id': 'id', 'type': 'text'},
+                       {'id': 'book', 'type': 'json'},
+                       {'id': 'author', 'type': 'text'}],
+        }
+        helpers.call_action('datastore_create', **data)
+        data = {
+            'resource_id': resource['id'],
+            'force': True,
+            'method': 'insert',
+            'records': [
+                {'id': '1',
+                 'book': {'code': 'A', 'title': u'ñ'},
+                 'author': 'tolstoy'}],
+        }
+        helpers.call_action('datastore_upsert', **data)
+
+    def test_upsert_doesnt_crash_with_json_field_with_string_value(self):
+        resource = factories.Resource()
+        data = {
+            'resource_id': resource['id'],
+            'force': True,
+            'primary_key': 'id',
+            'fields': [{'id': 'id', 'type': 'text'},
+                       {'id': 'book', 'type': 'json'},
+                       {'id': 'author', 'type': 'text'}],
+        }
+        helpers.call_action('datastore_create', **data)
+        data = {
+            'resource_id': resource['id'],
+            'force': True,
+            'method': 'insert',
+            'records': [
+                {'id': '1',
+                 'book': u'ñ',
+                 'author': 'tolstoy'}],
+        }
+        helpers.call_action('datastore_upsert', **data)
+
+
 class TestDatastoreUpsert(tests.WsgiAppCase):
     sysadmin_user = None
     normal_user = None