Repository: mathesar-foundation/mathesar

Issue:
Undesirable record grouping behaviours
## Description
Record grouping has a set of behaviours, that are not desirable.
* It considers order_by, which leads to formation of incorrect query on the backend, if we don't group by the sorted column.
![Screenshot 2021-07-13 at 12 05 01 AM](https://user-images.githubusercontent.com/11850603/125338760-3630af00-e36e-11eb-9240-68cd524c4eec.png)

* It considers limit and offset. These apply on the grouped result itself, and is unrelated to the record limit & offset.
![Screenshot 2021-07-13 at 12 02 20 AM](https://user-images.githubusercontent.com/11850603/125338797-45aff800-e36e-11eb-8845-91c128a21606.png)
![Screenshot 2021-07-13 at 12 02 42 AM](https://user-images.githubusercontent.com/11850603/125338811-49dc1580-e36e-11eb-8f5e-ca0560b41239.png)


## Expected behavior
* It should not consider order_by.
* It should not consider limit and offset.

We could also probably have a dedicated API for this. It could also obtain the values for columns, to filter the grouped results. Having it as part of records API makes less sense, since the group count is not a reflection of the record results.


Discussion:
Do we also want the `filter` parameter to not affect the returned counts? 
The `filter` parameter has to affect the returned counts.
@pavish A couple of questions:

(1) Are you suggesting that we move this functionality to a separate API or that we fix the current API? It's unclear from the issue description.

For context, the idea behind keeping it in the record list API was that we could return group counts for a given query. We would only look at the list of records that the passed-in limit, offset, sorts, and filters affect and return group counts for that query. For example, if you were displaying page 2 of a table with 50 records per page, we would show you the group counts relevant to groups visible on that page.

@eito-fis I assume this matches your understanding.

(2) If we do move it to a separate API, I'm not sure what you mean by "It could also obtain the values for columns, to filter the grouped results." Could you explain further, including how you would use it?
@kgodey 

(1) I am open to both options. I suggested moving to separate API, as the group_counts currently are not related to the record results.

> We would only look at the list of records that the passed-in limit, offset, sorts, and filters affect and return group counts for that query

This would solve all the problems at hand. I also assumed this to be the behaviour initially, but this isn't the current behaviour. For this, we would need to fetch the records first, then find the values for which we obtain the grouped counts, with only the filters and not anything else. But I believe currently, it is directly applying all the query params on the `group by` query. 

(2) If we had a dedicated API, we would need something like `/v0/tables/1/record_groups/?fields=["Center"]&filter={Center:'Name of center'}`. This would only return the group 'Name of center'. The behaviour is akin to how filters work with records. This will not be needed if we could do the former.
Let's go with keeping it in the current API and fixing it to work the way I described. @eito-fis let me know if you have any questions.
To be clear, its not that we don't want to consider limit or offset at all, rather we want to limit and offset before grouping. Is this correct?
@eito-fis Yes.
@kgodey @eito-fis  There is one other change, it's more on the convention side of the API. 

Currently the results are in this format, when **there are more than one grouped column**:
```
"NASA Kennedy Space Center,Application": 29,
"NASA Kennedy Space Center,Issued": 58,
"NASA Johnson Space Center,Application": 54,
"NASA Ames Research Center,Application": 38,
"NASA Ames Research Center,Issued": 100,
```

I think it would make more sense to change it to: 
```
"NASA Kennedy Space Center": {
  "Application": 29,
  "Issued": 58
},
"NASA Johnson Space Center": {
  "Application": 54
},
"NASA Ames Research Center": {
  "Application": 38,
  "Issued": 100
},
```

I am siding with the latter, since column names could have commas in them.

Resolving patch:
diff --git a/db/records.py b/db/records.py
--- a/db/records.py
+++ b/db/records.py
@@ -96,10 +96,8 @@ def get_group_counts(
         if field_name not in table.c:
             raise GroupFieldNotFound(f"Group field {field} not found in {table}.")
 
-    group_by = _create_col_objects(table, group_by)
     query = (
-        select(*group_by, func.count(table.c[ID]))
-        .group_by(*group_by)
+        select(table)
         .limit(limit)
         .offset(offset)
     )
@@ -107,6 +105,13 @@ def get_group_counts(
         query = apply_sort(query, order_by)
     if filters is not None:
         query = apply_filters(query, filters)
+    subquery = query.subquery()
+
+    group_by = [
+        subquery.columns[col] if type(col) == str else subquery.columns[col.name]
+        for col in group_by
+    ]
+    query = select(*group_by, func.count(subquery.c[ID])).group_by(*group_by)
     with engine.begin() as conn:
         records = conn.execute(query).fetchall()
 
diff --git a/mathesar/pagination.py b/mathesar/pagination.py
--- a/mathesar/pagination.py
+++ b/mathesar/pagination.py
@@ -67,7 +67,8 @@ def paginate_queryset(self, queryset, request, table_id,
                 filters=filters, order_by=order_by
             )
             # Convert the tuple keys into strings so it can be converted to JSON
-            group_count = {','.join(k): v for k, v in group_count.items()}
+            group_count = [{"values": list(cols), "count": count}
+                           for cols, count in group_count.items()]
             self.group_count = {
                 'group_count_by': group_count_by,
                 'results': group_count,


Test patch:
diff --git a/db/tests/records/test_grouping.py b/db/tests/records/test_grouping.py
--- a/db/tests/records/test_grouping.py
+++ b/db/tests/records/test_grouping.py
@@ -3,6 +3,7 @@
 
 import pytest
 from sqlalchemy import select
+from sqlalchemy_filters import apply_sort, apply_filters
 
 from db import records
 from db.records import GroupFieldNotFound, BadGroupFormat
@@ -32,35 +33,38 @@ def test_get_group_counts_mixed_str_col_field(filter_sort_table_obj):
     assert ("string1", 1) in counts
 
 
-def test_get_group_counts_limit_ordering(filter_sort_table_obj):
-    filter_sort, engine = filter_sort_table_obj
-    limit = 50
-    order_by = [{"field": "numeric", "direction": "desc", "nullslast": True}]
-    group_by = [filter_sort.c.numeric]
-    counts = records.get_group_counts(filter_sort, engine, group_by, limit=limit,
-                                      order_by=order_by)
-    assert len(counts) == 50
-    for i in range(1, 100):
-        if i > 50:
-            assert (i,) in counts
-        else:
-            assert (i,) not in counts
-
-
-def test_get_group_counts_limit_offset_ordering(filter_sort_table_obj):
-    filter_sort, engine = filter_sort_table_obj
-    offset = 25
-    limit = 50
-    order_by = [{"field": "numeric", "direction": "desc", "nullslast": True}]
-    group_by = [filter_sort.c.numeric]
-    counts = records.get_group_counts(filter_sort, engine, group_by, limit=limit,
+limit_offset_test_list = [
+    (limit, offset)
+    for limit in [None, 0, 25, 50, 100]
+    for offset in [None, 0, 25, 50, 100]
+]
+
+
+@pytest.mark.parametrize("limit,offset", limit_offset_test_list)
+def test_get_group_counts_limit_offset_ordering(roster_table_obj, limit, offset):
+    roster, engine = roster_table_obj
+    order_by = [{"field": "Grade", "direction": "desc", "nullslast": True}]
+    group_by = [roster.c["Grade"]]
+    counts = records.get_group_counts(roster, engine, group_by, limit=limit,
                                       offset=offset, order_by=order_by)
-    assert len(counts) == 50
-    for i in range(1, 100):
-        if i > 25 and i <= 75:
-            assert (i,) in counts
-        else:
-            assert (i,) not in counts
+
+    query = select(group_by[0])
+    query = apply_sort(query, order_by)
+    with engine.begin() as conn:
+        all_records = list(conn.execute(query))
+    if limit is None:
+        end = None
+    elif offset is None:
+        end = limit
+    else:
+        end = limit + offset
+    limit_offset_records = all_records[offset:end]
+    manual_count = Counter(limit_offset_records)
+
+    assert len(counts) == len(manual_count)
+    for value, count in manual_count.items():
+        assert value in counts
+        assert counts[value] == count
 
 
 count_values_test_list = itertools.chain(*[
@@ -84,8 +88,38 @@ def test_get_group_counts_count_values(roster_table_obj, group_by):
         all_records = conn.execute(select(*cols)).fetchall()
     manual_count = Counter(all_records)
 
-    for key, value in counts.items():
-        assert manual_count[key] == value
+    for value, count in manual_count.items():
+        assert value in counts
+        assert counts[value] == count
+
+
+filter_values_test_list = itertools.chain(*[
+    itertools.combinations([
+        {"field": "Student Name", "op": "ge", "value": "Test Name"},
+        {"field": "Student Email", "op": "le", "value": "Test Email"},
+        {"field": "Teacher Email", "op": "like", "value": "%gmail.com"},
+        {"field": "Subject", "op": "eq", "value": "Non-Existent Subject"},
+        {"field": "Grade", "op": "ne", "value": 99}
+    ], i) for i in range(1, 3)
+])
+
+
+@pytest.mark.parametrize("filter_by", filter_values_test_list)
+def test_get_group_counts_filter_values(roster_table_obj, filter_by):
+    roster, engine = roster_table_obj
+    group_by = ["Student Name"]
+    counts = records.get_group_counts(roster, engine, group_by, filters=filter_by)
+
+    cols = [roster.c[f] for f in group_by]
+    query = select(*cols)
+    query = apply_filters(query, filter_by)
+    with engine.begin() as conn:
+        all_records = conn.execute(query).fetchall()
+    manual_count = Counter(all_records)
+
+    for value, count in manual_count.items():
+        assert value in counts
+        assert counts[value] == count
 
 
 exceptions_test_list = [
diff --git a/mathesar/tests/views/api/test_record_api.py b/mathesar/tests/views/api/test_record_api.py
--- a/mathesar/tests/views/api/test_record_api.py
+++ b/mathesar/tests/views/api/test_record_api.py
@@ -116,14 +116,18 @@ def test_record_list_sort(create_table, client):
 
 
 def _test_record_list_group(table, client, group_count_by, expected_groups):
+    order_by = [
+        {'field': 'Center', 'direction': 'desc'},
+        {'field': 'Case Number', 'direction': 'asc'},
+    ]
+    json_order_by = json.dumps(order_by)
     json_group_count_by = json.dumps(group_count_by)
+    query_str = f'group_count_by={json_group_count_by}&order_by={json_order_by}'
 
     with patch.object(
         records, "get_group_counts", side_effect=records.get_group_counts
     ) as mock_infer:
-        response = client.get(
-            f'/api/v0/tables/{table.id}/records/?group_count_by={json_group_count_by}'
-        )
+        response = client.get(f'/api/v0/tables/{table.id}/records/?{query_str}')
         response_data = response.json()
 
     assert response.status_code == 200
@@ -133,10 +137,13 @@ def _test_record_list_group(table, client, group_count_by, expected_groups):
     assert 'group_count' in response_data
     assert response_data['group_count']['group_count_by'] == group_count_by
     assert 'results' in response_data['group_count']
+    assert 'values' in response_data['group_count']['results'][0]
+    assert 'count' in response_data['group_count']['results'][0]
 
     results = response_data['group_count']['results']
+    returned_groups = {tuple(group['values']) for group in results}
     for expected_group in expected_groups:
-        assert expected_group in results
+        assert expected_group in returned_groups
 
     assert mock_infer.call_args is not None
     assert mock_infer.call_args[0][2] == group_count_by
@@ -147,8 +154,8 @@ def test_record_list_group_single_column(create_table, client):
     table = create_table(table_name)
     group_count_by = ['Center']
     expected_groups = [
-        'NASA Ames Research Center',
-        'NASA Kennedy Space Center'
+        ('NASA Marshall Space Flight Center',),
+        ('NASA Stennis Space Center',)
     ]
     _test_record_list_group(table, client, group_count_by, expected_groups)
 
@@ -158,8 +165,8 @@ def test_record_list_group_multi_column(create_table, client):
     table = create_table(table_name)
     group_count_by = ['Center', 'Status']
     expected_groups = [
-        'NASA Ames Research Center,Issued',
-        'NASA Kennedy Space Center,Issued',
+        ('NASA Marshall Space Flight Center', 'Issued'),
+        ('NASA Stennis Space Center', 'Issued'),
     ]
     _test_record_list_group(table, client, group_count_by, expected_groups)