Repository: mathesar-foundation/mathesar

Issue:
Ordering by a non-orderable column (implicitly or explicitly) causes a backend crash
## Bug

Brent [writes](https://github.com/centerofci/mathesar/pull/2253#pullrequestreview-1251718380):

> At one point, you note in a comment that there's a bug whenever we have internal columns of JSON type, and no primary key in a table. This bug is still present for _any_ non-orderable column type if there is no primary key in the table (e.g., if the user attaches to a preexisting DB). It's easy to figure out whether or not a type is orderable in PostgreSQL. See the [docs page for ORDER BY](https://www.postgresql.org/docs/current/queries-order.html#id-1.5.6.9.5.10) and the [docs page for Operator Classes](https://www.postgresql.org/docs/9.5/indexes-opclass.html).

Brent is referring to [this comment](https://github.com/centerofci/mathesar/blob/8c215b2efdd740da0d530b624763cf260e12ca90/db/records/operations/sort.py#L47).

The error message received upon such a crash is: `could not identify an ordering operator for type json` (`json` is the type of the unorderable column in this case.

## Solution

The solution is, as explained by Brent, to make sure that every column we order by (implicitly or explicitly) is orderable.

## Severity

Because of this, a JSON column (it's non-orderable), can make a query/table view unrenderable, but only when the table/query does not have a primary key, which cannot happen in a Mathesar-created table. It can only happen when Mathesar is used to access a pre-existing database.


Discussion:
Hi 
Can i work on this issue ?

@Hatilar420 sure, I'll assign you!
Assigned myself by mistake. Meant to assign @Hatilar420.
sure np @dmos62 
Hi @dmos62, Can you help me in reproducing this bug ? are there any pre-existing tables or any query that i can use to reproduce this bug ? 
@Hatilar420 I don't have a dataset, but to manufacture one yourself you can create a Postgres table via `psql` that 1) doesn't have a primary key (PK) and 2) has a non-orderable (e.g. JSON) column. You can't use Mathesar to create the table, because it can't create a table without a PK.

This should break our backend, because it strives to provide total ordering by ordering by all columns, ignoring the fact that some columns cannot be ordered by (e.g. JSON columns).
sure, working on it. I am very sorry for any delays.
Hi @dmos62 while solving this issue, I came across this method. Can you give me an example or tell me how we are using this method since I see no declarations of it? and what is the parameter `relation` we passing to it? 

![image](https://user-images.githubusercontent.com/29446639/218450296-d746adc5-28d5-4336-ac3a-5971fde9ee9e.png)



@Hatilar420 to answer these questions you can check out the superclass `Transform` and how it's used in the codebase. A `Transform` is basically a transformation that can be applied to an SQL SELECT query. Try searching the tests as well.
 Hi, @dmos62  is this the desired response? I am getting a `200` response from it. After I implemented the solution. Since Mathesar doesn't support JSON yet. It is displaying the data like that.
If it is not desired, then what should be the desired response? 
![image](https://user-images.githubusercontent.com/29446639/218479771-30426f82-dc84-4a6d-b89e-322add38b90c.png)


Resolving patch:
diff --git a/db/records/operations/sort.py b/db/records/operations/sort.py
--- a/db/records/operations/sort.py
+++ b/db/records/operations/sort.py
@@ -44,27 +44,37 @@ def _build_order_by_all_columns_clause(relation):
     To be used when we have failed to find any other ordering criteria,
     since ordering by all columns is inherently inefficient.
 
-    Note the filtering out of internal columns. Before applying this fix, psycopg was throwing an error
-    like "could not identify an ordering operator for type json", because we were trying to
-    sort by an internal column like `__mathesar_group_metadata`, which has type `json`, which
-    requires special handling to be sorted. The problem is bypassed by not attempting to sort on
-    internal columns.
+    Note the filtering out some columns, namely internal columns and non-orderable columns. See
+    their docstrings for details.
     """
     return [
         {'field': col, 'direction': 'asc'}
         for col
         in relation.columns
-        if not _is_internal_column(col)
+        if _is_col_orderable(col) and not _is_internal_column(col)
     ]
 
 
 def _is_internal_column(col):
     """
+    Columns that Mathesar adds for its own devices and does not expose to the user. We don't want
+    to sort by these.
+
     Might not be exhaustive, take care.
     """
     return col.name == '__mathesar_group_metadata'
 
 
+def _is_col_orderable(col):
+    """
+    Some columns are not orderable (or at least don't have a non-ambiguous way to define order
+    without additional logic). We only want to order by orderably columns.
+    """
+    data_type = col.type
+    non_orderable_type = ['Binary', 'LargeBinary', 'PickleType', 'ARRAY', 'JSON', 'JSONB']
+    return str(data_type) not in non_orderable_type
+
+
 def apply_relation_sorting(relation, sort_spec):
     order_by_list = [
         _get_sorted_column_obj_from_spec(relation, spec) for spec in sort_spec


Test patch:
diff --git a/db/tests/conftest.py b/db/tests/conftest.py
--- a/db/tests/conftest.py
+++ b/db/tests/conftest.py
@@ -21,6 +21,7 @@
 MAGNITUDE_SQL = os.path.join(RESOURCES, "magnitude_testing_create.sql")
 ARRAY_SQL = os.path.join(RESOURCES, "array_create.sql")
 JSON_SQL = os.path.join(RESOURCES, "json_sort.sql")
+JSON_WITHOUT_PKEY_SQL = os.path.join(RESOURCES, "json_without_pkey.sql")
 BOOKS_FROM_SQL = os.path.join(RESOURCES, "books_import_from.sql")
 BOOKS_TARGET_SQL = os.path.join(RESOURCES, "books_import_target.sql")
 
@@ -34,6 +35,15 @@ def engine_with_roster(engine_with_schema):
     yield engine, schema
 
 
+@pytest.fixture
+def engine_with_JSON_without_pkey(engine_with_schema):
+    engine, schema = engine_with_schema
+    with engine.begin() as conn, open(JSON_WITHOUT_PKEY_SQL) as f:
+        conn.execute(text(f"SET search_path={schema}"))
+        conn.execute(text(f.read()))
+    yield engine, schema
+
+
 @pytest.fixture
 def engine_with_uris(engine_with_schema):
     engine, schema = engine_with_schema
@@ -120,6 +130,11 @@ def roster_table_name():
     return "Roster"
 
 
+@pytest.fixture(scope='session')
+def json_without_pkey_name():
+    return "json_without_pkey"
+
+
 @pytest.fixture(scope='session')
 def uris_table_name():
     return "uris"
@@ -212,6 +227,14 @@ def roster_table_obj(engine_with_roster, roster_table_name):
     return table, engine
 
 
+@pytest.fixture
+def json_without_pkey_table_obj(engine_with_JSON_without_pkey, json_without_pkey_name):
+    engine, schema = engine_with_JSON_without_pkey
+    metadata = MetaData(bind=engine)
+    table = Table(json_without_pkey_name, metadata, schema=schema, autoload_with=engine)
+    return table, engine
+
+
 @pytest.fixture
 def magnitude_table_obj(engine_with_magnitude, magnitude_table_name):
     engine, schema = engine_with_magnitude
diff --git a/db/tests/resources/json_without_pkey.sql b/db/tests/resources/json_without_pkey.sql
new file mode 100644
--- /dev/null
+++ b/db/tests/resources/json_without_pkey.sql
@@ -0,0 +1,7 @@
+CREATE TABLE "json_without_pkey" (
+    "json_object" json
+);
+
+INSERT INTO "json_without_pkey" VALUES
+('{"name": "John", "age": 30}'::json),
+('{"name": "Earl James", "age": 30}'::json);
diff --git a/db/tests/transforms/test_json_without_pkey.py b/db/tests/transforms/test_json_without_pkey.py
new file mode 100644
--- /dev/null
+++ b/db/tests/transforms/test_json_without_pkey.py
@@ -0,0 +1,7 @@
+from db.records.operations.select import get_records
+
+
+def test_default_ordering(json_without_pkey_table_obj):
+    table, engine = json_without_pkey_table_obj
+    records = get_records(table, engine, fallback_to_default_ordering=True)
+    assert len(records) == 2