Repository: ckan/ckan

Issue:
Downloading data with null value as JSON causes an Internal Server Error
**CKAN version**
2.8.4 and 2.8.10

**Describe the bug**
When downloading data (which is stored in the datastore) as JSON, an internal server error occurs if the data contains a null value. The error occurs when each row has at least one column with a null value.

**Steps to reproduce**
The bug can be reproduced via ckan API:
1. Create resource: http://myserver:myport/api/3/action/resource_create
    with body form data: package_id (ID of an existing dataset) and name (name of resource)
2. Create datastore and add data: http://myserver:myport/api/3/action/datastore_create
    with body raw data:
`{
  "resource_id": "54317d3a-3a26-4911-911d-57c615639756",
  "force": "true",
  "fields": [
    {
      "id": "id",
      "type": "numeric"
    },
    {
      "id": "name",
      "type": "text"
    }
  ],
  "primary_key": "id",
  "records": [
    {
      "id": "1",
      "name": null
    }
  ]
}`
3. Download data as JSON: http://myserver:myport/datastore/dump/54317d3a-3a26-4911-911d-57c615639756?format=json

**Expected behavior**
A download should be possible without error. In this particular case the resulting json file will be empty. (In case of multiple rows of data, ckan already ignors the rows with null values and downloads the rest of data successfully.)

**Additional details**
Stack trace:
![image](https://user-images.githubusercontent.com/111954802/186346787-383ff280-ad23-416e-a195-a8501b54da4a.png)



Discussion:
Hi @wardi @bpeXX , should we allow users to upload files with `null` values? I tried to reproduce the issue but received a different error that is associated with the #6713  issue, I've fixed issue #6713 and will raise the PR soon.

 But I found that when downloading a resource in json format, [this](https://github.com/ckan/ckan/blob/master/ckanext/datastore/backend/postgres.py#L1372-#L1378) query runs. I found that when concatenating with the `||` operator, the values from a row that contains a `null` value, the output for that particular row changes into a blank row, I've tested with multiple queries containing  `null` values, I believe this is the default behavior of PostgreSQL.

We can have a different approach where we can replace the `null` value with another value, but that would be an alteration of the file itself.
I believe a better approach would be to raise a ValidationError message if a `null` value is found in the file.

@wardi, I'd like to know your opinion on this.




Hi @bpeXX - Just wondering if you tested this issue with CKAN 2.9 (2.9.5 or master)?
Hi @kowh-ai, unfortunately I did not test the issue with CKAN 2.9 since our application is based on CKAN 2.8.4.
@bpeXX @kowh-ai I am having the same issue on CKAN 2.9.7
left a comment in issue page:
https://github.com/ckan/ckan/issues/6713#issuecomment-1507271171

Resolving patch:
diff --git a/ckanext/datastore/backend/postgres.py b/ckanext/datastore/backend/postgres.py
--- a/ckanext/datastore/backend/postgres.py
+++ b/ckanext/datastore/backend/postgres.py
@@ -1934,26 +1934,28 @@ def datastore_search(
             rank_columns)
         where = _where_clauses(data_dict, fields_types)
         select_cols = []
-        records_format = data_dict.get(u'records_format')
+        records_format = data_dict.get('records_format')
         for field_id in field_ids:
-            fmt = u'to_json({0})' if records_format == u'lists' else u'{0}'
+            fmt = '{0}'
+            if records_format == 'lists':
+                fmt = "coalesce(to_json({0}),'null')"
             typ = fields_types.get(field_id, '')
-            if typ == u'nested':
-                fmt = u'({0}).json'
-            elif typ == u'timestamp':
-                fmt = u"to_char({0}, 'YYYY-MM-DD\"T\"HH24:MI:SS')"
-                if records_format == u'lists':
-                    fmt = u"to_json({0})".format(fmt)
-            elif typ.startswith(u'_') or typ.endswith(u'[]'):
-                fmt = u'array_to_json({0})'
+            if typ == 'nested':
+                fmt = "coalesce(({0}).json,'null')"
+            elif typ == 'timestamp':
+                fmt = "to_char({0}, 'YYYY-MM-DD\"T\"HH24:MI:SS')"
+                if records_format == 'lists':
+                    fmt = f"coalesce(to_json({fmt}), 'null')"
+            elif typ.startswith('_') or typ.endswith('[]'):
+                fmt = "coalesce(array_to_json({0}),'null')"
 
             if field_id in rank_columns:
                 select_cols.append((fmt + ' as {1}').format(
                     rank_columns[field_id], identifier(field_id)))
                 continue
 
-            if records_format == u'objects':
-                fmt += u' as {0}'
+            if records_format == 'objects':
+                fmt += ' as {0}'
             select_cols.append(fmt.format(identifier(field_id)))
 
         query_dict['distinct'] = data_dict.get('distinct', False)


Test patch:
diff --git a/ckanext/datastore/tests/test_search.py b/ckanext/datastore/tests/test_search.py
--- a/ckanext/datastore/tests/test_search.py
+++ b/ckanext/datastore/tests/test_search.py
@@ -2181,3 +2181,46 @@ def test_fts_on_field_calculates_ranks_when_full_text_is_given(self):
         ranks = [r["rank"] for r in result["records"]]
         assert len(result["records"]) == 2
         assert len(set(ranks)) == 2
+
+    @pytest.mark.ckan_config("ckan.plugins", "datastore")
+    @pytest.mark.usefixtures("clean_datastore", "with_plugins")
+    def test_results_with_nulls(self):
+        ds = factories.Dataset()
+        r = helpers.call_action(
+            "datastore_create",
+            resource={"package_id": ds["id"]},
+            fields=[
+                {"id": "num", "type": "numeric"},
+                {"id": "dt", "type": "timestamp"},
+                {"id": "txt", "type": "text"},
+                {"id": "lst", "type": "_text"},
+            ],
+            records=[
+                {"num": 10, "dt": "2020-01-01", "txt": "aaab", "lst": ["one", "two"]},
+                {"num": 9, "dt": "2020-01-02", "txt": "aaab"},
+                {"num": 9, "txt": "aaac", "lst": ["one", "two"]},
+                {},  # all nulls
+            ],
+        )
+        assert helpers.call_action(
+            "datastore_search",
+            resource_id=r["resource_id"],
+            records_format=u"lists",
+            sort=u"num nulls last, dt nulls last",
+        )["records"] == [
+            [2, 9, "2020-01-02T00:00:00", "aaab", None],
+            [3, 9, None, "aaac", ["one", "two"]],
+            [1, 10, "2020-01-01T00:00:00", "aaab", ["one", "two"]],
+            [4, None, None, None, None],
+        ]
+        assert helpers.call_action(
+            "datastore_search",
+            resource_id=r["resource_id"],
+            records_format=u"objects",
+            sort=u"num nulls last, dt nulls last",
+        )["records"] == [
+            {"_id": 2, "num": 9, "dt": "2020-01-02T00:00:00", "txt": "aaab", "lst": None},
+            {"_id": 3, "num": 9, "dt": None, "txt": "aaac", "lst": ["one", "two"]},
+            {"_id": 1, "num": 10, "dt": "2020-01-01T00:00:00", "txt": "aaab", "lst": ["one", "two"]},
+            {"_id": 4, "num": None, "dt": None, "txt": None, "lst": None},
+        ]