Repository: mathesar-foundation/mathesar

Issue:
Filters not applied when calculating count of items within group
## Reproduce

1. Go to the Library Management schema.
1. Load the Table Page for the Publications table.
1. Group by "Publication Year".
1. Observe the first group, for year 1900, to contain 10 records and to display a "Count" of 10. Good.
1. Add a filter condition requiring Title to contain the string "To".
1. Observe the first group, for year 1900, to contain 2 records.
1. Expect "Count" to display 2.
1. Observe "Count" displays 10.

![image](https://user-images.githubusercontent.com/42411/188676046-46fe3997-5c5e-433c-9f5e-f0f73bf36bb8.png)



Discussion:
Interesting. I consider this to be the expected behavior. If I wanted to count the number of results after a filter, I'd use the Data Explorer. For me, filtering the records in this view isn't supposed to change calculated statistics like "count", it's supposed to let me find the row I'm interested in working with by reducing the number of results. 

From a more technical perspective, for ranged grouping, filtering before grouping (which is your apparent expectation) changes the parameters of the given groups. If I filter a numeric column to cut of its upper range (i.e., something like `WHERE mycol < 10`), then the bounds of the automatically generated groups would change accordingly. This would (I think) be kind of confusing.

OTOH, if one wants to count based on ranged grouping, one can't do that in the data explorer until we have the relevant features implemented...I don't think we should indulge in over-complicating the table records view based on that deficiency, though.

To some extent, I consider showing the count in this view _at all_ to be extraneous, and potentially confusing. I'm becoming less and less convinced that grouping makes sense in this view (as opposed to bringing any relevant functionality into the data explorer).

@ghislaineguerin @kgodey Any opinion here? Is the user supposed to be using the table view for querying (albeit with a reduced feature set), or for viewing/editing data? Or both?

If both, we may want to figure out how to handle these order of operations problems and their implications more clearly. I see from the UI (should have noticed earlier, sorry) that the implication is that we first filter, then sort, then group (going from left to right). But, the actual implementation goes group, then sort, then filter on the back end. The order of these operations changes the output.

Edit: Another thought. What about limit/offset? Should the count of a group that goes into the next page be the total number of rows, or just the number of rows shown?
Can we just remove `count`?
I'd like not to remove `count`, and the count reflecting the filtered results.

> I consider this to be the expected behavior. If I wanted to count the number of results after a filter, I'd use the Data Explorer. 

We do not show a grouped result to the user in the table page, what we show is an ordered set of rows sectioned by their groups. I think from the user perspective, it is important to reflect the filtered count in each of those sections.

> For me, filtering the records in this view isn't supposed to change calculated statistics like "count", it's supposed to let me find the row I'm interested in working with by reducing the number of results.

I would agree with this if we had an UX which has filters per group, and not a common filter. I'd expect the common filter to have a higher precedence over the groups.
I agree that filters should take precedence over grouping. It is confusing to see a group count that doesn't match the data you see in front of you.

> Is the user supposed to be using the table view for querying (albeit with a reduced feature set), or for viewing/editing data? Or both?

Both. 

> we may want to figure out how to handle these order of operations problems and their implications more clearly.

I agree we should have a defined order of operations, ~here's my response to the [email thread](https://groups.google.com/a/mathesar.org/g/mathesar-developers/c/mhuc2cWLPXM/m/Om-jcOxgBwAJ)~ [here's the latest response](https://groups.google.com/a/mathesar.org/g/mathesar-developers/c/mhuc2cWLPXM/m/UPdhhzpiBwAJ).

---

I don't think we should spend any time or energy focusing on rethinking existing features (like whether grouping makes sense where it is) until we have released our first alpha release. 

Resolving patch:
diff --git a/db/transforms/operations/apply.py b/db/transforms/operations/apply.py
--- a/db/transforms/operations/apply.py
+++ b/db/transforms/operations/apply.py
@@ -35,14 +35,14 @@ def apply_transformations_deprecated(
 
     transforms = []
 
+    if filter:
+        transforms.append(base.Filter(filter))
     if duplicate_only:
         transforms.append(base.DuplicateOnly(duplicate_only))
     if group_by:
         transforms.append(base.Group(group_by))
     if order_by:
         transforms.append(base.Order(order_by))
-    if filter:
-        transforms.append(base.Filter(filter))
     if search:
         transforms.append(base.Search([search, limit]))
     if columns_to_select:


Test patch:
diff --git a/db/tests/tables/operations/test_select.py b/db/tests/tables/operations/test_select.py
--- a/db/tests/tables/operations/test_select.py
+++ b/db/tests/tables/operations/test_select.py
@@ -224,7 +224,7 @@ def test_get_description_from_table(roster_table_name, engine_with_roster):
     roster_table_oid = ma_sel.get_oid_from_table(roster_table_name, schema, engine)
     expect_comment = 'my super comment'
     with engine.begin() as conn:
-        conn.execute(text(f'''COMMENT ON TABLE "{roster_table_name}" IS '{expect_comment}';'''))
+        conn.execute(text(f'''COMMENT ON TABLE "{schema}"."{roster_table_name}" IS '{expect_comment}';'''))
 
     actual_comment = ma_sel.get_table_description(roster_table_oid, engine)
 
diff --git a/mathesar/tests/api/test_record_api.py b/mathesar/tests/api/test_record_api.py
--- a/mathesar/tests/api/test_record_api.py
+++ b/mathesar/tests/api/test_record_api.py
@@ -669,6 +669,36 @@ def _retuple_bound_tuples(bound_tuple_list):
     _test_group_equality(grouping_dict['groups'], expected_groups)
 
 
+def test_group_filter_combo_order(create_patents_table, client):
+    table_name = 'NASA Record List Group Filter'
+    table = create_patents_table(table_name)
+    name_id_map = table.get_column_name_id_bidirectional_map()
+
+    raw_grouping = {'columns': ['Center']}
+    raw_filter = {
+        "contains": [
+            {"column_id": [name_id_map["Case Number"]]}, {"literal": ["11"]}
+        ]
+    }
+    raw_order_by = [{'field': name_id_map['id'], 'direction': 'asc'}]
+    group_by_col_ids = [name_id_map[col_name] for col_name in raw_grouping['columns']]
+    ids_converted_group_by = {**raw_grouping, 'columns': group_by_col_ids}
+
+    grouping = json.dumps(ids_converted_group_by)
+    filter_ = json.dumps(raw_filter)
+    order_by = json.dumps(raw_order_by)
+
+    limit = 10
+    query_str = f'grouping={grouping}&order_by={order_by}&limit={limit}&filter={filter_}'
+
+    response = client.get(f'/api/db/v0/tables/{table.id}/records/?{query_str}')
+    response_data = response.json()
+
+    expect_group_counts = [2, 3, 2, 1, 7]
+    actual_group_counts = [g['count'] for g in response_data['grouping']['groups']]
+    assert actual_group_counts == expect_group_counts
+
+
 def test_record_list_pagination_limit(create_patents_table, client):
     table_name = 'NASA Record List Pagination Limit'
     table = create_patents_table(table_name)