Repository: ckan/ckan

Issue:
Unreliable ordering of datastore results
Hi,

In the absence of an "ORDER BY" clause, PostgreSQL does not make any promises regarding the ordering of rows. In particular if you use "OFFSET/LIMIT" clauses the ordering might actually **change** between two subsequent queries. This is from the PostgreSQL documentation:

> The query planner takes LIMIT into account when generating a query plan, so you are very likely to get different plans (yielding different row orders) depending on what you use for LIMIT and OFFSET. Thus, using different LIMIT/OFFSET values to select different subsets of a query result will give inconsistent results unless you enforce a predictable result ordering with ORDER BY. This is not a bug; it is an inherent consequence of the fact that SQL does not promise to deliver the results of a query in any particular order unless ORDER BY is used to constrain the order. 

http://www.postgresql.org/docs/current/static/sql-select.html

This issue affects datastore queries (I have witnessed it). While this is not technically a CKAN bug (the problem can be moved upwards to the API consumers), it is an unexpected behaviour. I would suggest the best place to address this is in the datastore itself, to always ensure queries include an ORDER BY term on a unique key (eg. _id)

I can do a pull request if people agree this is a good thing to do.



Discussion:
Note that this can be fixed in an extension. (Though I really think it should be fixed in ckan - data scientists using the API shouldn't have to know about issues like these!)

@aliceh75 dev meeting agrees with you :)

Just to confirm, is the `_id` field created by us and indexed?

Yes, the datastore always creates a `_id` field which is a primary key. See:
https://github.com/ckan/ckan/blob/master/ckanext/datastore/db.py#L295 

This cannot be overriden by users as fields that start with an underscore are forbidden via the API:
https://github.com/ckan/ckan/blob/master/ckanext/datastore/db.py#L81

Postgresql always indexes the primary key as far as I understand:

> Adding a primary key will automatically create a unique btree index on the column or group of columns used in the primary key.
> http://www.postgresql.org/docs/9.1/static/ddl-constraints.html

So I will create a PR (next week, I'm away until Monday!)

ref #2309 - in CKAN 2.3 sometimes  _id field is displayed in preview, and other times it is missing (and on the same dataset).

Not sure why this got closed by me merging a remote PR into a remote repo :( I've re-opened and submitted the PR into CKAN itself.
Is there any progress according to this issue?

We've stumbled into the same problem and would be happy about a fix. The PR seems to be pretty close - where it's getting stuck?
We just need a new PR to review

Resolving patch:
diff --git a/ckanext/datastore/backend/postgres.py b/ckanext/datastore/backend/postgres.py
--- a/ckanext/datastore/backend/postgres.py
+++ b/ckanext/datastore/backend/postgres.py
@@ -1319,6 +1319,9 @@ def search_data(context, data_dict):
     else:
         distinct = ''
 
+    if not sort and not distinct:
+        sort = ['_id']
+
     if sort:
         sort_clause = 'ORDER BY %s' % (', '.join(sort)).replace('%', '%%')
     else:


Test patch: