Repository: ckan/ckan

Issue:
datastore doesn't report row number for errors on upsert/create
## CKAN version
all

## Describe the bug
`datastore_create` and `datastore_upsert` can fail while loading data and will report an error but do not indicate which row was being processed when the error occurred.

### Steps to reproduce
e.g.
```bash
ckanapi action package_create name=foo
ckanapi action datastore_create \
  resource:'{"package_id":"foo"}' records:'[{"a":1}, {"a":"two"}]'
```
raises:
```python
ckan.logic.ValidationError: None - {
  'records': ['invalid input syntax for type integer: "two"']
}
```

### Expected behavior
There should be an indication of which row triggered the failure to make it easier to find bad data in large updates

### Additional details
There's no established pattern for returning the position of an error within a list in other ckan api calls, other than doing something like (for an error in the 5th row):

```python
ckan.logic.ValidationError: None - {
  'records': [
    {},{},{},{},{"a":['invalid input syntax for type integer: "two"']}
  ]
}
```
But:
 - we don't get back information from postgres on the column name for type errors
 - if upserting hundreds of rows we might need to create lists with hundreds of empty dicts just to indicate a position
 - this is not backwards compatible for clients expecting a list of strings in `records`

I'm proposing to do this instead (for an error in the 5th row):
```python
ckan.logic.ValidationError: None - {
  'records': ['invalid input syntax for type integer: "two"'],
  'records_row': 4
  ]
}
```
i.e. add a 0-based `records_row` value when returning an error where we know which row in records triggered the database error.

Love to hear your thoughts on this this change.


Discussion:
note this was accidentally partially implemented in #3428 as part of other changes but not discussed in the ticket, and the key used was `_records_row`. This ticket is to properly implement `records_row` for `insert` and `create` failures and to update the name (remove leading hyphen) for other cases.

Resolving patch:
diff --git a/ckanext/datastore/backend/postgres.py b/ckanext/datastore/backend/postgres.py
--- a/ckanext/datastore/backend/postgres.py
+++ b/ckanext/datastore/backend/postgres.py
@@ -1124,6 +1124,7 @@ def upsert_data(context: Context, data_dict: dict[str, Any]):
     records = data_dict['records']
     sql_columns = ", ".join(
         identifier(name) for name in field_names)
+    num = -1
 
     if method == _INSERT:
         rows = []
@@ -1151,8 +1152,10 @@ def upsert_data(context: Context, data_dict: dict[str, Any]):
         try:
             context['connection'].execute(sql_string, rows)
         except (DatabaseError, DataError) as err:
-            raise ValidationError(
-                {u'records': [_programming_error_summary(err)]})
+            raise ValidationError({
+                'records': [_programming_error_summary(err)],
+                'records_row': num,
+            })
 
     elif method in [_UPDATE, _UPSERT]:
         unique_keys = _get_unique_key(context, data_dict)
@@ -1228,8 +1231,9 @@ def upsert_data(context: Context, data_dict: dict[str, Any]):
                         sql_string, used_values + unique_values)
                 except sqlalchemy.exc.DatabaseError as err:
                     raise ValidationError({
-                        u'records': [_programming_error_summary(err)],
-                        u'_records_row': num})
+                        'records': [_programming_error_summary(err)],
+                        'records_row': num,
+                    })
 
                 # validate that exactly one row has been updated
                 if results.rowcount != 1:
@@ -1263,8 +1267,9 @@ def upsert_data(context: Context, data_dict: dict[str, Any]):
                         (used_values + unique_values) * 2)
                 except sqlalchemy.exc.DatabaseError as err:
                     raise ValidationError({
-                        u'records': [_programming_error_summary(err)],
-                        u'_records_row': num})
+                        'records': [_programming_error_summary(err)],
+                        'records_row': num,
+                    })
 
 
 def validate(context: Context, data_dict: dict[str, Any]):


Test patch:
diff --git a/ckanext/datastore/tests/test_create.py b/ckanext/datastore/tests/test_create.py
--- a/ckanext/datastore/tests/test_create.py
+++ b/ckanext/datastore/tests/test_create.py
@@ -1368,7 +1368,10 @@ def test_create_trigger_exception(self, app):
                     records=[{u"spam": u"spam"}, {u"spam": u"EGGS"}],
                     triggers=[{u"function": u"spamexception_trigger"}],
                 )
-        assert error.value.error_dict == {u"records": [u'"EGGS"? Yeeeeccch!']}
+        assert error.value.error_dict == {
+            "records": ['"EGGS"? Yeeeeccch!'],
+            "records_row": 1,
+        }
 
     @pytest.mark.ckan_config("ckan.plugins", "datastore")
     @pytest.mark.usefixtures("clean_datastore", "with_plugins")
@@ -1402,5 +1405,6 @@ def test_upsert_trigger_exception(self, app):
                     records=[{u"spam": u"spam"}, {u"spam": u"BEANS"}],
                 )
             assert error.value.error_dict == {
-                u"records": [u'"BEANS"? Yeeeeccch!']
+                "records": ['"BEANS"? Yeeeeccch!'],
+                "records_row": 1,
             }