Repository: ckan/ckan

Issue:
Datastore upsert update not working with index _id
**CKAN version** 2.9.5

**Describe the bug**
A clear and concise description of what the bug is.
datastore/upsert method=update does not work with _id 

**Steps to reproduce**
Steps to reproduce the behavior:
Create resource in datastore
insert values
read values => resulting records have unique field _id, as expected
update command with _id fails on "undefined primary_key"
{
    "error": {
        "__type": "Validation Error",
        "table": [
            "table does not have a unique key defined"
        ]
    },

**Expected behavior**
A clear and concise description of what you expected to happen.
Record number _id can be used as primary key, as in regular SQL database

**Additional details**
If possible, please provide the full stack trace of the error raised,  or add screenshots to help explain your problem.

Use case: update resource with timeseries data which by design do not come with an inherent primary key. So primary key must be database id. However, this cannot be enforced on datastore_create()

Maybe you consider this not a bug - but we have an id in the data, so not allowing to use it is quite close ... 




Discussion:
Hey, @digital-codes,  I've tested this and it doesn't work as you think. 

When using `datastore_create` action, you have `param: primary_key` this `primary_key` is optional. 

When you set your `fields/columns` e.g like this: `"fields": [{"id": "name"}, {"id": "age"}, {"id": "email"} ]` and your records like: `"records": [ {"name": "Andreas", "age": 30, "email": "andreas@digital-codes.com"} ]` you can set `primary_key` to be your `"email"` this would be your `unique_key` in your `fields/columns`.

Then you can use `datastore_upsert` (lets say you want to update your name) like this: 

curl -X POST http://localhost:5000/api/3/action/datastore_upsert -H "Authorization: your-key" -d '{"resource_id": "{resource.id}", "force": "True", "records": [{"name": "Kugel", "email": "andreas@digital-codes.com"}], "method": "update" }'.

If you don’t set your `primary_key` when creating, you cannot use `datastore_upsert` as all three methods require a `unique key`.

Thanks
I agree with the reporter. The `_id` field should be usable as a primary key for `datastore_upsert`. I'm happy to review a PR or help work on a fix for this issue.
Thanks for taking this issue. I'm not too familiar with CKAN internals nor Postgres, but if I'll try to help. I looked at backend/postgres.py and there is 

```
elif method in [_UPDATE, _UPSERT]:
        unique_keys = _get_unique_key(context, data_dict)
        if len(unique_keys) < 1:
            raise ValidationError({
                'table': [u'table does not have a unique key defined']
            })

```
which is probably giving the error. But I don't know why postgres doesn't accept the _id as a unique key. 

I looked further and *create_table* goes like:

```
def create_table(context: Context, data_dict: dict[str, Any]):
  ...
    datastore_fields = [
        {'id': '_id', 'type': 'serial primary key'},
        {'id': '_full_text', 'type': 'tsvector'},
    ]

```

 
Which seems to set _id as a primary key. Or is this a typo and it should be _primary_key_ instead of _primary_  and _key_ ?

Hmmm, maybe it's the way how the unique keys are looked up:

```
def _get_unique_key(context: Context, data_dict: dict[str, Any]) -> list[str]:
    sql_get_unique_key = '''
    SELECT
        a.attname AS column_names
    FROM
        pg_class t,
        pg_index idx,
        pg_attribute a
    WHERE
        t.oid = idx.indrelid
        AND a.attrelid = t.oid
        AND a.attnum = ANY(idx.indkey)
        AND t.relkind = 'r'
        AND idx.indisunique = true
        AND idx.indisprimary = false
        AND t.relname = %s
    '''

```
The line 
        AND idx.indisprimary = false
will probably exclude the _id ...


> The line AND idx.indisprimary = false will probably exclude the _id ...

Yep thats true
So how do we proceed? Do you consider it safe to just remove that line? 
I don't have a running local installation, just access to a production system, so I can't really test the modification and create a PR. Can one of you do that? 
Hey @wardi, I'd like to work on this issue, currently, I have a fix though might not seem the best solution, if you don't agree with this please give me some pointers.

Here is what I've done: 
I removed this line: `AND idx.indisprimary = false` from `_get_unique_key()` 
with that line removed, from now we will always receive the `_id` when calling `_get_unique_key()` 

i will comment my changes with `# this line is added by me` so you will know what has been added
```python
for num, record in enumerate(records):
      # remove the _id if not provided when calling update method
      if '_id' in unique_keys and '_id' not in record: # this line is added by me
          unique_keys.remove('_id') # this line is added by me
      # all key columns have to be defined
      missing_fields = [field for field in unique_keys
                        if field not in record]
      if missing_fields:
          raise ValidationError({
              'key': [u'''fields "{fields}" are missing
                  but needed as key'''.format(
                      fields=', '.join(missing_fields))]
          })

      for field in fields:
          value = record.get(field['id'])
          if value is not None and field['type'].lower() == 'nested':
              # a tuple with an empty second value
              record[field['id']] = (json.dumps(value), '')

      if "_id" in record: # this line is added by me
          # _id is not a field_name so we need to append
          field_names.append('_id') # this line is added by me
      non_existing_filed_names = [field for field in record
                                  if field not in field_names]
````

the original code you can find here https://github.com/ckan/ckan/blob/2e63e7fbaf7208f122e8c90a419f7c71485423b9/ckanext/datastore/backend/postgres.py#L1170

Thanks
@TomeCirun thanks, would you mind creating a pull request instead?

Resolving patch:
diff --git a/ckanext/datastore/backend/postgres.py b/ckanext/datastore/backend/postgres.py
--- a/ckanext/datastore/backend/postgres.py
+++ b/ckanext/datastore/backend/postgres.py
@@ -1162,37 +1162,48 @@ def upsert_data(context: Context, data_dict: dict[str, Any]):
 
     elif method in [_UPDATE, _UPSERT]:
         unique_keys = _get_unique_key(context, data_dict)
-        if len(unique_keys) < 1:
-            raise ValidationError({
-                'table': [u'table does not have a unique key defined']
-            })
 
         for num, record in enumerate(records):
-            # all key columns have to be defined
-            missing_fields = [field for field in unique_keys
-                              if field not in record]
-            if missing_fields:
+            if not unique_keys and '_id' not in record:
                 raise ValidationError({
-                    'key': [u'''fields "{fields}" are missing
-                        but needed as key'''.format(
-                            fields=', '.join(missing_fields))]
+                    'table': [u'unique key must be passed for update/upsert']
                 })
 
+            elif '_id' not in record:
+                # all key columns have to be defined
+                missing_fields = [field for field in unique_keys
+                                  if field not in record]
+                if missing_fields:
+                    raise ValidationError({
+                        'key': [u'''fields "{fields}" are missing
+                            but needed as key'''.format(
+                                fields=', '.join(missing_fields))]
+                    })
+
             for field in fields:
                 value = record.get(field['id'])
                 if value is not None and field['type'].lower() == 'nested':
                     # a tuple with an empty second value
                     record[field['id']] = (json.dumps(value), '')
 
-            non_existing_filed_names = [field for field in record
-                                        if field not in field_names]
-            if non_existing_filed_names:
+            non_existing_field_names = [
+                field for field in record
+                if field not in field_names and field != '_id'
+            ]
+            if non_existing_field_names:
                 raise ValidationError({
                     'fields': [u'fields "{0}" do not exist'.format(
-                        ', '.join(non_existing_filed_names))]
+                        ', '.join(non_existing_field_names))]
                 })
 
-            unique_values = [record[key] for key in unique_keys]
+            if '_id' in record:
+                unique_values = [record['_id']]
+                pk_sql = '"_id"'
+                pk_values_sql = '%s'
+            else:
+                unique_values = [record[key] for key in unique_keys]
+                pk_sql = ','.join([identifier(part) for part in unique_keys])
+                pk_values_sql = ','.join(['%s'] * len(unique_keys))
 
             used_fields = [field for field in fields
                            if field['id'] in record]
@@ -1203,19 +1214,18 @@ def upsert_data(context: Context, data_dict: dict[str, Any]):
 
             if method == _UPDATE:
                 sql_string = u'''
-                    UPDATE "{res_id}"
+                    UPDATE {res_id}
                     SET ({columns}, "_full_text") = ({values}, NULL)
                     WHERE ({primary_key}) = ({primary_value});
                 '''.format(
-                    res_id=data_dict['resource_id'],
+                    res_id=identifier(data_dict['resource_id']),
                     columns=u', '.join(
                         [identifier(field)
                          for field in used_field_names]).replace('%', '%%'),
                     values=u', '.join(
                         ['%s' for _ in used_field_names]),
-                    primary_key=u','.join(
-                        [u'"{0}"'.format(part) for part in unique_keys]),
-                    primary_value=u','.join(["%s"] * len(unique_keys))
+                    primary_key=pk_sql.replace('%', '%%'),
+                    primary_value=pk_values_sql,
                 )
                 try:
                     results = context['connection'].execute(
@@ -1233,24 +1243,23 @@ def upsert_data(context: Context, data_dict: dict[str, Any]):
 
             elif method == _UPSERT:
                 sql_string = u'''
-                    UPDATE "{res_id}"
+                    UPDATE {res_id}
                     SET ({columns}, "_full_text") = ({values}, NULL)
                     WHERE ({primary_key}) = ({primary_value});
-                    INSERT INTO "{res_id}" ({columns})
+                    INSERT INTO {res_id} ({columns})
                            SELECT {values}
-                           WHERE NOT EXISTS (SELECT 1 FROM "{res_id}"
+                           WHERE NOT EXISTS (SELECT 1 FROM {res_id}
                                     WHERE ({primary_key}) = ({primary_value}));
                 '''.format(
-                    res_id=data_dict['resource_id'],
-                    columns=u', '.join([
-                        u'"{0}"'.format(field.replace('%', '%%'))
-                        for field in used_field_names]),
+                    res_id=identifier(data_dict['resource_id']),
+                    columns=u', '.join(
+                        [identifier(field)
+                         for field in used_field_names]).replace('%', '%%'),
                     values=u', '.join(['%s::nested'
                                        if field['type'] == 'nested' else '%s'
                                        for field in used_fields]),
-                    primary_key=u','.join([u'"{0}"'.format(part)
-                                           for part in unique_keys]),
-                    primary_value=u','.join(["%s"] * len(unique_keys))
+                    primary_key=pk_sql.replace('%', '%%'),
+                    primary_value=pk_values_sql,
                 )
                 try:
                     context['connection'].execute(
diff --git a/ckanext/datastore/logic/action.py b/ckanext/datastore/logic/action.py
--- a/ckanext/datastore/logic/action.py
+++ b/ckanext/datastore/logic/action.py
@@ -219,14 +219,14 @@ def datastore_upsert(context: Context, data_dict: dict[str, Any]):
 
     *upsert*
         Update if record with same key already exists, otherwise insert.
-        Requires unique key.
+        Requires unique key or _id field.
     *insert*
         Insert only. This method is faster that upsert, but will fail if any
         inserted record matches an existing one. Does *not* require a unique
         key.
     *update*
         Update only. An exception will occur if the key that should be updated
-        does not exist. Requires unique key.
+        does not exist. Requires unique key or _id field.
 
 
     :param resource_id: resource id that the data is going to be stored under.


Test patch:
diff --git a/ckanext/datastore/tests/test_upsert.py b/ckanext/datastore/tests/test_upsert.py
--- a/ckanext/datastore/tests/test_upsert.py
+++ b/ckanext/datastore/tests/test_upsert.py
@@ -552,6 +552,67 @@ def test_calculate_record_count(self):
         last_analyze = when_was_last_analyze(resource["id"])
         assert last_analyze is not None
 
+    @pytest.mark.ckan_config("ckan.plugins", "datastore")
+    @pytest.mark.usefixtures("clean_datastore", "with_plugins")
+    def test_no_pk_update(self):
+        resource = factories.Resource()
+        data = {
+            "resource_id": resource["id"],
+            "force": True,
+            "fields": [
+                {"id": "book", "type": "text"},
+            ],
+            "records": [{"book": u"El Niño"}],
+        }
+        helpers.call_action("datastore_create", **data)
+
+        data = {
+            "resource_id": resource["id"],
+            "force": True,
+            "method": "upsert",
+            "records": [
+                {"_id": "1", "book": u"The boy"}
+            ],
+        }
+        helpers.call_action("datastore_upsert", **data)
+
+        search_result = _search(resource["id"])
+        assert search_result["total"] == 1
+        assert search_result["records"][0]["book"] == "The boy"
+
+    @pytest.mark.ckan_config("ckan.plugins", "datastore")
+    @pytest.mark.usefixtures("clean_datastore", "with_plugins")
+    def test_id_instead_of_pk_update(self):
+        resource = factories.Resource()
+        data = {
+            "resource_id": resource["id"],
+            "force": True,
+            "primary_key": "pk",
+            "fields": [
+                {"id": "pk", "type": "text"},
+                {"id": "book", "type": "text"},
+                {"id": "author", "type": "text"},
+            ],
+            "records": [{"pk": "1000", "book": u"El Niño", "author": "Torres"}],
+        }
+        helpers.call_action("datastore_create", **data)
+
+        data = {
+            "resource_id": resource["id"],
+            "force": True,
+            "method": "upsert",
+            "records": [
+                {"_id": "1", "book": u"The boy", "author": u"F Torres"}
+            ],
+        }
+        helpers.call_action("datastore_upsert", **data)
+
+        search_result = _search(resource["id"])
+        assert search_result["total"] == 1
+        assert search_result["records"][0]["pk"] == "1000"
+        assert search_result["records"][0]["book"] == "The boy"
+        assert search_result["records"][0]["author"] == "F Torres"
+
 
 @pytest.mark.usefixtures("with_request_context")
 class TestDatastoreInsert(object):
@@ -829,3 +890,64 @@ def test_update_non_existing_field(self):
         with pytest.raises(ValidationError) as context:
             helpers.call_action("datastore_upsert", **data)
         assert u'fields "dummy" do not exist' in str(context.value)
+
+    @pytest.mark.ckan_config("ckan.plugins", "datastore")
+    @pytest.mark.usefixtures("clean_datastore", "with_plugins")
+    def test_no_pk_update(self):
+        resource = factories.Resource()
+        data = {
+            "resource_id": resource["id"],
+            "force": True,
+            "fields": [
+                {"id": "book", "type": "text"},
+            ],
+            "records": [{"book": u"El Niño"}],
+        }
+        helpers.call_action("datastore_create", **data)
+
+        data = {
+            "resource_id": resource["id"],
+            "force": True,
+            "method": "update",
+            "records": [
+                {"_id": "1", "book": u"The boy"}
+            ],
+        }
+        helpers.call_action("datastore_upsert", **data)
+
+        search_result = _search(resource["id"])
+        assert search_result["total"] == 1
+        assert search_result["records"][0]["book"] == "The boy"
+
+    @pytest.mark.ckan_config("ckan.plugins", "datastore")
+    @pytest.mark.usefixtures("clean_datastore", "with_plugins")
+    def test_id_instead_of_pk_update(self):
+        resource = factories.Resource()
+        data = {
+            "resource_id": resource["id"],
+            "force": True,
+            "primary_key": "pk",
+            "fields": [
+                {"id": "pk", "type": "text"},
+                {"id": "book", "type": "text"},
+                {"id": "author", "type": "text"},
+            ],
+            "records": [{"pk": "1000", "book": u"El Niño", "author": "Torres"}],
+        }
+        helpers.call_action("datastore_create", **data)
+
+        data = {
+            "resource_id": resource["id"],
+            "force": True,
+            "method": "update",
+            "records": [
+                {"_id": "1", "book": u"The boy", "author": u"F Torres"}
+            ],
+        }
+        helpers.call_action("datastore_upsert", **data)
+
+        search_result = _search(resource["id"])
+        assert search_result["total"] == 1
+        assert search_result["records"][0]["pk"] == "1000"
+        assert search_result["records"][0]["book"] == "The boy"
+        assert search_result["records"][0]["author"] == "F Torres"