Repository: mathesar-foundation/mathesar

Issue:
Something is wrong with the data explorer....
<!-- A clear and concise description of what the bug is. -->
I'm trying to make the Overdue Books report and ended up with this:
<img width="1151" alt="Screen Shot 2022-12-20 at 7 23 10 PM" src="https://user-images.githubusercontent.com/287034/208792427-f8b7bc2a-7228-4c11-90d0-d95766bcd7e4.png">

Aaron Black has not checked out the same book over and over again, here's what his checkouts actually look like.
![Screenshot 2022-12-20 at 19-27-52 Checkouts Library Management Mathesar](https://user-images.githubusercontent.com/287034/208792546-da7f06f7-1adc-4f04-879a-23de904da309.png)

If I try to recreate the exploration, everything looks fine - I add Patron name, email, and book title (I selected do not summarize) and get this:
![Screenshot 2022-12-20 at 19-29-36 Data Explorer Library Management Mathesar](https://user-images.githubusercontent.com/287034/208792856-46012765-44e5-4c9d-8e61-aa91329bfbb4.png)

Then I add Due Date from Checkouts and everything goes nuts

![Screenshot 2022-12-20 at 19-30-01 Data Explorer Library Management Mathesar](https://user-images.githubusercontent.com/287034/208792901-6d923733-569d-464b-9da4-691b33955c75.png)



Discussion:
This is a backend bug. Multiple rows are being returned by the API. @dmos62 
Hm, given base table `Patrons`, if you pull in as initial columns Patron id and Checkout id twice, you'd expect both Checkout id columns to be identical, but they're not, and when the second one is added, the number of records goes from 105 to 471. 

![2022-12-21_15-31](https://user-images.githubusercontent.com/2715476/208917432-be1814e4-3af8-4691-ac95-a9287910c20c.png)

Edit:

Two more observations:

- Seem to only be able to reproduce with the library dataset (`demo/resources/library_*.sql`);
- Bug reproduces when base table is patrons and joining in from the checkouts table through a reverse link, but not the other way around.

Edit2:

Diff of SQL queries with:
- base Checkouts and initial columns Checkouts.id, Patrons.id, Patrons.id (regular link, bug not reproduced),
- and, with base Patrons and initial columns Patrons.id, Checkouts.id, Checkouts.id (reverse link, bug reproduced):

``` diff
--- checkouts.tmp	2022-12-21 16:13:18.383151814 +0200
+++ patrons.tmp	2022-12-21 16:13:23.167205590 +0200
@@ -1,30 +1,30 @@
 WITH anon_3 AS (
   SELECT 
-    public."Checkouts".id AS "Checkouts_id", 
-    "Patrons_1".id AS "Patrons_id", 
-    "Patrons_2".id AS "Patrons_id_1" 
+    public."Patrons".id AS "Patrons_id", 
+    "Checkouts_1".id AS "Checkouts_id", 
+    "Checkouts_2".id AS "Checkouts_id_1" 
   FROM 
-    public."Checkouts" 
-    LEFT OUTER JOIN public."Patrons" AS "Patrons_1" ON public."Checkouts"."Patron" = "Patrons_1".id 
-    LEFT OUTER JOIN public."Patrons" AS "Patrons_2" ON public."Checkouts"."Patron" = "Patrons_2".id
+    public."Patrons" 
+    LEFT OUTER JOIN public."Checkouts" AS "Checkouts_1" ON public."Patrons".id = "Checkouts_1"."Patron" 
+    LEFT OUTER JOIN public."Checkouts" AS "Checkouts_2" ON public."Patrons".id = "Checkouts_2"."Patron"
 ), 
 anon_2 AS (
   SELECT 
-    anon_3."Checkouts_id" AS "Checkouts_id", 
     anon_3."Patrons_id" AS "Patrons_id", 
-    anon_3."Patrons_id_1" AS "Patrons_id_1" 
+    anon_3."Checkouts_id" AS "Checkouts_id", 
+    anon_3."Checkouts_id_1" AS "Checkouts_id_1" 
   FROM 
     anon_3 
   ORDER BY 
-    anon_3."Patrons_id_1" ASC, 
+    anon_3."Checkouts_id_1" ASC, 
     anon_3."Patrons_id" ASC, 
     anon_3."Checkouts_id" ASC
 ) 
 SELECT 
-  anon_2."Checkouts_id", 
   anon_2."Patrons_id", 
-  anon_2."Patrons_id_1" 
+  anon_2."Checkouts_id", 
+  anon_2."Checkouts_id_1" 
 FROM 
   anon_2 
 LIMIT 
   : param_1
```
I think the result of the current query makes sense from an sql perspective. We need to update the sql query to suit the behaviour we want in the backend.

In the mentioned query, 
When the base table is Checkouts:
* Assume we have 100 checkouts and we add columns from the checkouts table. There are now 100 rows, one checkout per result row.
* Assume we have 2 checkouts per patron.
* When performing the first join with Patrons, the result still contains 100 rows, with one checkout per result row. The join is made from checkouts to patrons, resulting in 1 associated patron per row. The patron columns may be duplicated but the row count will remain 100.
* When performing the second join with Patrons, it results in the same 100 rows, since the previous result has 100 rows.

When the base table is Patrons:
* Assume we have 10 patrons and we add columns from the patrons table. We have 10 rows, 1 unique patron per result row.
* Assume we have 2 checkouts per patron.
* We now perform the first join with Checkouts, the result contains 200 rows (100 * 2), with duplicated patrons, because the checkout is unique. We have the same patrons twice in the results.
* We perform the second join with Checkouts, which happens on the previous result with 200 rows, which joins with the patron id from each of the 200 result rows, resulting in 200 * 2 = 400 rows.
> -    public."Checkouts" 
> -    LEFT OUTER JOIN public."Patrons" AS "Patrons_1" ON public."Checkouts"."Patron" = "Patrons_1".id 
> -    LEFT OUTER JOIN public."Patrons" AS "Patrons_2" ON public."Checkouts"."Patron" = "Patrons_2".id

@dmos62 It is working as expected, a Patron can have more than one checkout and since you are doing multiple join, each of the joins refers to any one of the checkouts associated with the Patron.
The initial course of action is to avoid duplicates of the same join path to the same table.

I see that there is some implementation around this made by @mathemancer, but there is a possible bug in that implementation, or I'm mistaken and it's for something else.

Notice this comment:
```
# We cache this to avoid copies of the same join path to a given table
 jp_path_alias_map = {(): base_table}
```
within `DBQuery.initial_relation` in `mathesar/db/queries/base.py`.
Here's how I expect the report to work:

Start here https://library.mathesar.org/mathesar_tables/70/data-explorer/, start with Patrons

- Add Patron email, Patron first name, Patron last name (there should be 250 rows at this point)
- Add Books > Title and select "Do Not Summarize" (there should be 14629 rows at this point)
- Add Checkouts > Due Date (there should still be 14629 rows at this point)
- Add Checkouts > Check in Time (there should still be 14629 rows at this point)
- Add a filter for Due Date before today (there should be 14459 rows at this point)
- Add a filter for Check in Time is empty  (there should be 180 rows at this point)
- Hide columns Due Date and Check in Time
- Summarize based on patron email (there should be LESS than 180 rows at this point, and only "Book Title" should show up as a List)

Final report should look something like

| Patron email | Patron first name | Patron last name | Books list |
|-|-|-|-|
| kriti@example.com | Kriti | Godey | To Kill a Mockingbird, The Hunger Games |
| bob@example.com | Bob | Smith | The Lean Startup |

---

Login credentials for the `library.mathesar.org` server are in [this private Google Doc](https://docs.google.com/document/d/1qFYnDlHEn6PQhbQXwiHrwRPsHJ2KLoxw9leo1rZBNPA/).

Resolving patch:
diff --git a/db/queries/base.py b/db/queries/base.py
--- a/db/queries/base.py
+++ b/db/queries/base.py
@@ -179,6 +179,7 @@ def initial_relation(self):
         from_clause = base_table
         # We cache this to avoid copies of the same join path to a given table
         jp_path_alias_map = {(): base_table}
+        jp_path_unique_set = set()
 
         def _get_table(oid):
             """
@@ -199,15 +200,22 @@ def _process_initial_column(col):
 
             for i, jp in enumerate(jp_path):
                 left = jp_path_alias_map[jp_path[:i]]
-                right = _get_table(jp[1][0]).alias()
-                jp_path_alias_map[jp_path[:i + 1]] = right
+                right_table = jp_path[:i + 1]
+                if right_table in jp_path_alias_map:
+                    right = jp_path_alias_map[right_table]
+                else:
+                    right = _get_table(jp[1][0]).alias()
+                    jp_path_alias_map[jp_path[:i + 1]] = right
                 left_col_name = _get_column_name(jp[0][0], jp[0][1])
                 right_col_name = _get_column_name(jp[1][0], jp[1][1])
                 left_col = left.columns[left_col_name]
                 right_col = right.columns[right_col_name]
-                from_clause = from_clause.join(
-                    right, onclause=left_col == right_col, isouter=True,
-                )
+                join_columns = f"{left_col}, {right_col}"
+                if join_columns not in jp_path_unique_set:
+                    jp_path_unique_set.add(join_columns)
+                    from_clause = from_clause.join(
+                        right, onclause=left_col == right_col, isouter=True,
+                    )
 
             return right.columns[col_name].label(col.alias)
 


Test patch: