Repository: web2py/web2py

Issue:
Pagination problem with user-sorted grid
_From [ABasta..._at_gmail.com](https://code.google.com/u/103720714265511973715/) on October 11, 2013 03:03:54_

Using SQLFORM.grid (or .smartgrid), once the initial grid has loaded, if the user clicks on a column header to change the sort order, only that single field is used in the SQL ORDER BY clause. However, if the column used for sorting contains non-unique values, pagination may not work correctly, because the order in which the RDBMS returns records is not guaranteed to be the same with each query. For proper pagination, the records should be sorted on one or more columns that generate a single unique ordering.

Note, this problem occurs regardless of what is specified in the "orderby" argument to .grid() because the "orderby" argument is ignored when the user clicks on a grid header to sort the records.

A possible fix is to add the table "id" field to the ORDER BY clause in this case, which will guarantee a unique ordering (and shouldn't add too much processing time given that it is an indexed field). This is already done on the initial grid load if no "orderby" argument has been specified (due to the default orderby_on_limitby=True argument of the .select() method).

There is a more general problem with pagination in the grid. If the developer specifies an "orderby" argument in order to control the initial ordering of the displayed records, the table "id" field is no longer added to the ORDER BY clause, which could result in broken pagination. Since this is unlikely to be the intent in this case, perhaps the grid should always add the "id" fields to the ORDER BY. For the rare case where someone might want to prevent that for complete control over the ORDER BY clause, we could add an argument such as auto_pagination=True (set it to False to prevent the automatic behavior).

See https://groups.google.com/d/msg/web2py-developers/nklB1tRh_zU/ZbezJNLs4zgJ for further details.

_Original issue: http://code.google.com/p/web2py/issues/detail?id=1712_



Discussion:
_From [ABasta..._at_gmail.com](https://code.google.com/u/103720714265511973715/) on January 22, 2014 05:23:55_

To summarize, the proposal is to add an auto_pagination parameter to the grid that defaults to True. When True, the primary key(s) of the table(s) will always be appended to the orderby (or used to create an orderby). This will ensure proper pagination in all cases.

_From [massimo...._at_gmail.com](https://code.google.com/u/111721297562664670951/) on February 23, 2014 21:02:18_

Was this even done? I remember yes. Can you help me check?

_From [ABasta..._at_gmail.com](https://code.google.com/u/103720714265511973715/) on February 24, 2014 04:57:10_

We had an long discussion on the developers list, but I don't think this was actually implemented.


Resolving patch:
diff --git a/gluon/sqlhtml.py b/gluon/sqlhtml.py
--- a/gluon/sqlhtml.py
+++ b/gluon/sqlhtml.py
@@ -1958,7 +1958,8 @@ def grid(query,
              noconfirm=False,
              cache_count=None,
              client_side_delete=False,
-             ignore_common_filters=None):
+             ignore_common_filters=None,
+             auto_pagination=True):
 
         formstyle = formstyle or current.response.formstyle
 
@@ -2052,6 +2053,28 @@ def fetch_count(dbset):
                 nrows = 0
             return nrows
 
+        def fix_orderby(orderby):
+            if not auto_pagination:
+                return orderby
+            # enforce always an ORDER clause to avoid
+            # pagination errors. field_id is needed anyhow,
+            # is unique and usually indexed. See issue #679
+            if not orderby:
+                orderby = field_id
+            else:
+                if isinstance(orderby, Expression):
+                    if orderby.first:
+                        # here we're with a DESC order on a field
+                        # stored as orderby.first
+                        if orderby.first is not field_id:
+                            orderby = orderby | field_id
+                    else:
+                        # here we're with an ASC order on a field
+                        # stored as orderby
+                        if orderby is not field_id:
+                            orderby = orderby | field_id
+            return orderby
+
         def url(**b):
             b['args'] = args + b.get('args', [])
             localvars = request.get_vars.copy()
@@ -2313,6 +2336,8 @@ def linsert(lst, i, x):
                     else:
                         orderby = (order[:1] == '~' and ~sort_field) or sort_field
 
+            orderby = fix_orderby(orderby)
+
             expcolumns = [str(f) for f in columns]
             selectable_columns = [str(f) for f in columns if not isinstance(f, Field.Virtual)]
             if export_type.endswith('with_hidden_cols'):
@@ -2524,6 +2549,9 @@ def linsert(lst, i, x):
             limitby = (paginate * page, paginate * (page + 1))
         else:
             limitby = None
+
+        orderby = fix_orderby(orderby)
+
         try:
             table_fields = [field for field in fields
                             if (field.tablename in tablenames and


Test patch: