Repository: mathesar-foundation/mathesar

Issue:
Undesirable record grouping behaviours
## Description
Record grouping has a set of behaviours, that are not desirable.
* It considers order_by, which leads to formation of incorrect query on the backend, if we don't group by the sorted column.
![Screenshot 2021-07-13 at 12 05 01 AM](https://user-images.githubusercontent.com/11850603/125338760-3630af00-e36e-11eb-9240-68cd524c4eec.png)

* It considers limit and offset. These apply on the grouped result itself, and is unrelated to the record limit & offset.
![Screenshot 2021-07-13 at 12 02 20 AM](https://user-images.githubusercontent.com/11850603/125338797-45aff800-e36e-11eb-8845-91c128a21606.png)
![Screenshot 2021-07-13 at 12 02 42 AM](https://user-images.githubusercontent.com/11850603/125338811-49dc1580-e36e-11eb-8f5e-ca0560b41239.png)


## Expected behavior
* It should not consider order_by.
* It should not consider limit and offset.

We could also probably have a dedicated API for this. It could also obtain the values for columns, to filter the grouped results. Having it as part of records API makes less sense, since the group count is not a reflection of the record results.


Discussion:
Do we also want the `filter` parameter to not affect the returned counts? 
The `filter` parameter has to affect the returned counts.
@pavish A couple of questions:

(1) Are you suggesting that we move this functionality to a separate API or that we fix the current API? It's unclear from the issue description.

For context, the idea behind keeping it in the record list API was that we could return group counts for a given query. We would only look at the list of records that the passed-in limit, offset, sorts, and filters affect and return group counts for that query. For example, if you were displaying page 2 of a table with 50 records per page, we would show you the group counts relevant to groups visible on that page.

@eito-fis I assume this matches your understanding.

(2) If we do move it to a separate API, I'm not sure what you mean by "It could also obtain the values for columns, to filter the grouped results." Could you explain further, including how you would use it?
@kgodey 

(1) I am open to both options. I suggested moving to separate API, as the group_counts currently are not related to the record results.

> We would only look at the list of records that the passed-in limit, offset, sorts, and filters affect and return group counts for that query

This would solve all the problems at hand. I also assumed this to be the behaviour initially, but this isn't the current behaviour. For this, we would need to fetch the records first, then find the values for which we obtain the grouped counts, with only the filters and not anything else. But I believe currently, it is directly applying all the query params on the `group by` query. 

(2) If we had a dedicated API, we would need something like `/v0/tables/1/record_groups/?fields=["Center"]&filter={Center:'Name of center'}`. This would only return the group 'Name of center'. The behaviour is akin to how filters work with records. This will not be needed if we could do the former.
Let's go with keeping it in the current API and fixing it to work the way I described. @eito-fis let me know if you have any questions.
To be clear, its not that we don't want to consider limit or offset at all, rather we want to limit and offset before grouping. Is this correct?
@eito-fis Yes.
@kgodey @eito-fis  There is one other change, it's more on the convention side of the API. 

Currently the results are in this format, when **there are more than one grouped column**:
```
"NASA Kennedy Space Center,Application": 29,
"NASA Kennedy Space Center,Issued": 58,
"NASA Johnson Space Center,Application": 54,
"NASA Ames Research Center,Application": 38,
"NASA Ames Research Center,Issued": 100,
```

I think it would make more sense to change it to: 
```
"NASA Kennedy Space Center": {
  "Application": 29,
  "Issued": 58
},
"NASA Johnson Space Center": {
  "Application": 54
},
"NASA Ames Research Center": {
  "Application": 38,
  "Issued": 100
},
```

I am siding with the latter, since column names could have commas in them.
@pavish I find that format confusing, since both `NASA Kennedy Space Center` and `Application` are field values, nesting them implies that one is the parent and one is a child, which is incorrect. I do agree that comma separation could end up with unparseable strings. 

I suggest we do something like:

```json
{
    "results": [
        {
            "columns": ["NASA Kennedy Space Center", "Application"],
            "count": 29
        },
        {
            "columns": ["NASA Kennedy Space Center", "Issued"],
            "count": 58
        }
    ]
}
```
>  if you were displaying page 2 of a table with 50 records per page, we would show you the group counts relevant to groups visible on that page.

@kgodey In this scenario, if page 2 has only one group 'Group A' which in total consists of 90 records, I assumed this would return the group name 'Group A' with count 90.

But currently limit is still considered, and it returns 50. We would need to re-open this issue.
I think this was my error with speccing it out, I didn't think through what considering limits would mean. Since @eito-fis is out this week, I'll prioritize fixing it.
Actually, since this isn't blocking @pavish (the counts will just be wrong for now), I'm not going to prioritize it myself.

Resolving patch:
diff --git a/db/records.py b/db/records.py
--- a/db/records.py
+++ b/db/records.py
@@ -4,7 +4,6 @@
 from sqlalchemy_filters import apply_filters, apply_sort
 from sqlalchemy_filters.exceptions import FieldNotFound
 
-from db.constants import ID
 
 logger = logging.getLogger(__name__)
 
@@ -27,18 +26,32 @@ def _get_primary_key_column(table):
 
 def _create_col_objects(table, column_list):
     return [
-        table.columns[col] if type(col) == str else col
+        table.columns[col] if type(col) == str else table.columns[col.name]
         for col in column_list
     ]
 
 
+def _get_query(table, limit, offset, order_by, filters):
+    query = select(table).limit(limit).offset(offset)
+    if order_by is not None:
+        query = apply_sort(query, order_by)
+    if filters is not None:
+        query = apply_filters(query, filters)
+    return query
+
+
+def _execute_query(query, engine):
+    with engine.begin() as conn:
+        records = conn.execute(query).fetchall()
+        return records
+
+
 def get_record(table, engine, id_value):
     primary_key_column = _get_primary_key_column(table)
     query = select(table).where(primary_key_column == id_value)
-    with engine.begin() as conn:
-        result = conn.execute(query).fetchall()
-        assert len(result) <= 1
-        return result[0] if result else None
+    result = _execute_query(query, engine)
+    assert len(result) <= 1
+    return result[0] if result else None
 
 
 def get_records(
@@ -59,13 +72,8 @@ def get_records(
                   field, in addition to an 'value' field if appropriate.
                   See: https://github.com/centerofci/sqlalchemy-filters#filters-format
     """
-    query = select(table).limit(limit).offset(offset)
-    if order_by is not None:
-        query = apply_sort(query, order_by)
-    if filters is not None:
-        query = apply_filters(query, filters)
-    with engine.begin() as conn:
-        return conn.execute(query).fetchall()
+    query = _get_query(table, limit, offset, order_by, filters)
+    return _execute_query(query, engine)
 
 
 def get_group_counts(
@@ -96,35 +104,51 @@ def get_group_counts(
         if field_name not in table.c:
             raise GroupFieldNotFound(f"Group field {field} not found in {table}.")
 
-    query = (
-        select(table)
-        .limit(limit)
-        .offset(offset)
+    table_columns = _create_col_objects(table, group_by)
+    count_query = (
+        select(*table_columns, func.count(table_columns[0]))
+        .group_by(*table_columns)
     )
-    if order_by is not None:
-        query = apply_sort(query, order_by)
     if filters is not None:
-        query = apply_filters(query, filters)
-    subquery = query.subquery()
+        count_query = apply_filters(count_query, filters)
+    filtered_count_query = _get_filtered_group_by_count_query(
+        table, engine, group_by, limit, offset, order_by, filters, count_query
+    )
+    if filtered_count_query is not None:
+        records = _execute_query(filtered_count_query, engine)
+        # Last field is the count, preceding fields are the group by fields
+        counts = {(*record[:-1],): record[-1] for record in records}
+    else:
+        counts = {}
+    return counts
 
-    group_by = [
-        subquery.columns[col] if type(col) == str else subquery.columns[col.name]
-        for col in group_by
-    ]
-    query = select(*group_by, func.count(subquery.c[ID])).group_by(*group_by)
-    with engine.begin() as conn:
-        records = conn.execute(query).fetchall()
 
-    # Last field is the count, preceding fields are the group by fields
-    counts = {
-        (*record[:-1],): record[-1]
-        for record in records
-    }
-    return counts
+def _get_filtered_group_by_count_query(
+        table, engine, group_by, limit, offset, order_by, filters, count_query
+):
+    # Get the list of groups that we should count.
+    # We're considering limit and offset here so that we only count relevant groups
+    relevant_subtable_query = _get_query(table, limit, offset, order_by, filters)
+    relevant_subtable_cte = relevant_subtable_query.cte()
+    cte_columns = _create_col_objects(relevant_subtable_cte, group_by)
+    distinct_tuples = get_distinct_tuple_values(cte_columns, engine, output_table=table)
+    if distinct_tuples:
+        limited_filters = [
+            {
+                "or": [
+                    distinct_tuples_to_filter(distinct_tuple_spec)
+                    for distinct_tuple_spec in distinct_tuples
+                ]
+            }
+        ]
+        filtered_count_query = apply_filters(count_query, limited_filters)
+    else:
+        filtered_count_query = None
+    return filtered_count_query
 
 
 def get_distinct_tuple_values(
-        column_list, engine, table=None, limit=None, offset=None,
+        column_list, engine, table=None, limit=None, offset=None, output_table=None
 ):
     """
     Returns distinct tuples from a given list of columns.
@@ -155,9 +179,10 @@ def get_distinct_tuple_values(
         .limit(limit)
         .offset(offset)
     )
-    with engine.begin() as conn:
-        res = conn.execute(query).fetchall()
-    return [tuple(zip(column_objects, row)) for row in res]
+    result = _execute_query(query, engine)
+    if output_table is not None:
+        column_objects = [output_table.columns[col.name] for col in column_objects]
+    return [tuple(zip(column_objects, row)) for row in result]
 
 
 def distinct_tuples_to_filter(distinct_tuples):


Test patch:
diff --git a/db/tests/records/test_grouping.py b/db/tests/records/test_grouping.py
--- a/db/tests/records/test_grouping.py
+++ b/db/tests/records/test_grouping.py
@@ -47,11 +47,10 @@ def test_get_group_counts_limit_offset_ordering(roster_table_obj, limit, offset)
     group_by = [roster.c["Grade"]]
     counts = records.get_group_counts(roster, engine, group_by, limit=limit,
                                       offset=offset, order_by=order_by)
-
     query = select(group_by[0])
     query = apply_sort(query, order_by)
     with engine.begin() as conn:
-        all_records = list(conn.execute(query))
+        all_records = conn.execute(query).fetchall()
     if limit is None:
         end = None
     elif offset is None:
@@ -59,7 +58,12 @@ def test_get_group_counts_limit_offset_ordering(roster_table_obj, limit, offset)
     else:
         end = limit + offset
     limit_offset_records = all_records[offset:end]
-    manual_count = Counter(limit_offset_records)
+    values_to_count = set([record["Grade"] for record in limit_offset_records])
+
+    manual_all_count = Counter(all_records).items()
+    manual_count = {
+        k: v for k, v in manual_all_count if int(k[0]) in values_to_count
+    }
 
     assert len(counts) == len(manual_count)
     for value, count in manual_count.items():