Repository: edgedb/edgedb

Issue:
Inconsistent query results with filter
- EdgeDB Version: [4.5+22da9c8 Docker (rev 0c24ca886)](https://hub.docker.com/layers/edgedb/edgedb/latest/images/sha256-5d37f1f815f0c195da038f04746f58030f591e1413a8414542925ee1b1f12839)
- EdgeDB CLI Version: 4.0.2+b26c502
- OS Version: Docker 24.0.6 (build ed223bc) with Windows 10 22H2 (Build 19045.3930) as Host OS

## Steps to Reproduce:

1. Setup an EdgeDB instance
2. Migrate to the below schema
3. Pipe contents of the `Seed` section below to `edgedb`
4. Run the query `select Item filter (.parent ?= <Item>{}) and (.category = <Category>"Products")` and observe no results being returned, despite there being items that match the filters.
5. Flip the conditions and run `select Item filter (.category = <Category>"Products") and (.parent ?= <Item>{})` and observe that this time there are items being returned and they're ones that match both filters.

**Interestingly** the query from step 4 also works if `parent` is included in the Item shape, as in:
`select Item { id, name, parent } filter (.parent ?= <Item>{}) and (.category = <Category>"Products")`.

Below you can find the query, their results and their generated SQL for both queries.

<details>
  <summary>Broken query</summary>

  ```edgeql
  select Item { id, name } filter (.parent ?= <Item>{}) and (.category = <Category>"Products")
  ```
  ```edgeql
  []
  ```
  ```sql
WITH
"t_schema::ObjectType~1" AS NOT MATERIALIZED
    (
    (SELECT/*<pg.SelectStmt at 0x7f86ea6632d0>*/
            "expr-4~2"."ObjectType_value~1" AS "expr~4_identity~1",
            "expr-4~2"."name_value~1" AS "name_value~2"
        FROM
            LATERAL
            (SELECT/*<pg.SelectStmt at 0x7f86ea663790>*/
                    "ObjectType~2".id AS "ObjectType_value~1",
                    "ObjectType~2"."9f6dd92a-0a42-5341-b599-fbf5494fd973" AS "name_value~1"    
                FROM
                    edgedbstd."2662a1b4-4f3f-5875-b6eb-ce52101a90a3_t" AS "ObjectType~2"       
                    CROSS JOIN LATERAL
                    (SELECT/*<pg.SelectStmt at 0x7f86ea670b10>*/
                            ("expr-1~2"."expr~1_value~1" OR
                             "expr-2~2"."expr~2_value~1") AS "expr~3_value~1"
                        FROM
                            LATERAL
                            (SELECT/*<pg.SelectStmt at 0x7f86ea671210>*/
                                    (NOT "ObjectType~2"."101d5f8c-251a-5fee-b30d-c0a3ed3ded56") AS "expr~1_value~1"
                            ) AS "expr-1~2"
                            CROSS JOIN (SELECT (NULL)::uuid AS "v~2") AS "v~1"
                            CROSS JOIN LATERAL
                            (SELECT/*<pg.SelectStmt at 0x7f86ea671710>*/
                                    ("ObjectType~2".id IS NOT DISTINCT FROM "v~1"."v~2") AS "expr~2_value~1"
                            ) AS "expr-2~2"
                    ) AS "expr-3~2"
                WHERE
                    "expr-3~2"."expr~3_value~1"
            ) AS "expr-4~2"
    ))
SELECT/*<pg.SelectStmt at 0x7f86ea63e8d0>*/
        (
        (SELECT/*<pg.SelectStmt at 0x7f86ea630b50>*/
                "str___tname__~2"."expr~10_value~1" AS "__tname___serialized~1"
            FROM
                LATERAL
                (SELECT/*<pg.SelectStmt at 0x7f86ea660b90>*/
                        "expr-10~2"."name_value~6" AS "expr~10_value~1"
                    FROM
                        LATERAL
                        (SELECT/*<pg.SelectStmt at 0x7f86ea661990>*/
                                "str_name~2"."name_value~5" AS "name_value~6"
                            FROM
                                LATERAL
                                (SELECT/*<pg.SelectStmt at 0x7f86ea662290>*/
                                        "q~3"."name_value~4" AS "name_value~5"
                                    FROM 
                                        LATERAL
                                        (SELECT/*<pg.SelectStmt at 0x7f86ea662690>*/
                                                "q~2"."name_value~3" AS "name_value~4"
                                            FROM
                                                LATERAL
                                                (SELECT/*<pg.SelectStmt at 0x7f86ea662cd0>*/   
                                                        "Item~2".__type__ AS "__type___identity~1"
                                                ) AS "q~1"
                                                JOIN LATERAL
                                                (SELECT/*<pg.SelectStmt at 0x7f86ea663650>*/   
                                                        "t~1"."expr~4_identity~1" AS "expr~4_identity~2",
                                                        "t~1"."name_value~2" AS "name_value~3" 
                                                    FROM
                                                        "t_schema::ObjectType~1" AS "t~1"      
                                                ) AS "q~2"
                                                    ON ("q~1"."__type___identity~1" = "q~2"."expr~4_identity~2")
                                        ) AS "q~3"
                                ) AS "str_name~2"
                            LIMIT
                            (SELECT/*<pg.SelectStmt at 0x7f86ea6782d0>*/
                                    (101)::int8 AS "expr~9_value~1"
                            )
                        ) AS "expr-10~2"
                    LIMIT
                    (SELECT/*<pg.SelectStmt at 0x7f86ea661910>*/
                            (101)::int8 AS "expr~11_value~1"
                    )
                ) AS "str___tname__~2"
        ),
        (SELECT/*<pg.SelectStmt at 0x7f86ea63e5d0>*/
                "uuid___tid__~2"."expr~6_value~1" AS "__tid___serialized~1"
            FROM
                LATERAL
                (SELECT/*<pg.SelectStmt at 0x7f86ea679350>*/
                        "expr-6~2"."__type___value~2" AS "expr~6_value~1"
                    FROM
                        LATERAL
                        (SELECT/*<pg.SelectStmt at 0x7f86ea679cd0>*/
                                "uuid_id~3"."__type___value~1" AS "__type___value~2"
                            FROM
                                LATERAL
                                (SELECT/*<pg.SelectStmt at 0x7f86ea67a850>*/
                                        "Item~2".__type__ AS "__type___value~1"
                                ) AS "uuid_id~3"
                            LIMIT
                            (SELECT/*<pg.SelectStmt at 0x7f86ea67af50>*/
                                    (101)::int8 AS "expr~5_value~1"
                            )
                        ) AS "expr-6~2"
                    LIMIT
                    (SELECT/*<pg.SelectStmt at 0x7f86ea679b50>*/
                            (101)::int8 AS "expr~7_value~1"
                    )
                ) AS "uuid___tid__~2"
        ), "Item~2".id, "Item~2"."b6c13820-b9ec-11ee-8b43-5963961d2db8") AS "Item_serialized~1" FROM
        edgedbpub."b6bd9168-b9ec-11ee-8fc8-1bf3579dccc0_t" AS "Item~2"
        CROSS JOIN LATERAL
        (SELECT/*<pg.SelectStmt at 0x7f86ea680210>*/
                ("expr-13~2"."expr~13_value~1" AND
                 "expr-16~2"."expr~16_value~1") AS "expr~17_value~1"
            FROM
                LATERAL
                (SELECT/*<pg.SelectStmt at 0x7f86ea680dd0>*/
                        "q~6"."parent_value~3" AS "parent_value~4",
                        "q~6"."category_value~2" AS "category_value~3"
                    FROM
                        LATERAL
                        (SELECT/*<pg.SelectStmt at 0x7f86ea681390>*/
                                ("m~1" = first_value("m~1") OVER ()) AS "m~1",
                                "q~5"."parent_value~1" AS "parent_value~3",
                                "q~5"."category_value~1" AS "category_value~2"
                            FROM
                                LATERAL
                                (
                                (SELECT/*<pg.SelectStmt at 0x7f86ea6819d0>*/
                                        1 AS "m~1",
                                        "Item~3".id AS "parent_value~1",
                                        "Item~2"."b321254d-bcbd-11ee-bf39-c90aa86b4895" AS "category_value~1"
                                    FROM
                                        LATERAL
                                        (SELECT/*<pg.SelectStmt at 0x7f86ea681250>*/
                                                "Item~2"."b320ba21-bcbd-11ee-b574-8d6ccb22a5a6" AS "parent_identity~1"
                                        ) AS "q~4"
                                        JOIN edgedbpub."b6bd9168-b9ec-11ee-8fc8-1bf3579dccc0_t" AS "Item~3"
                                            ON ("q~4"."parent_identity~1" = "Item~3".id)       
                                ) UNION ALL
                                (SELECT/*<pg.SelectStmt at 0x7f86ea681a50>*/
                                        2 AS "m~1",
                                        "v~3"."id~1" AS "parent_value~2",
                                        NULL AS "null~1"
                                    FROM 
                                        (SELECT (NULL)::uuid AS "id~1") AS "v~3"
                                )
                                ) AS "q~5"
                        ) AS "q~6"
                    WHERE
                        "q~6"."m~1"
                ) AS "Item_parent~2"
                CROSS JOIN (SELECT (NULL)::uuid AS "id~2") AS "v~4"
                CROSS JOIN LATERAL
                (SELECT/*<pg.SelectStmt at 0x7f86ea680750>*/
                        ("Item_parent~2"."parent_value~4" IS NOT DISTINCT FROM "v~4"."id~2") AS "expr~13_value~1"
                ) AS "expr-13~2"
                CROSS JOIN LATERAL
                (SELECT/*<pg.SelectStmt at 0x7f86ea688810>*/
                        ("Item_parent~2"."category_value~3" = "expr-15~2"."expr~15_value~1") AS "expr~16_value~1"
                    FROM
                        LATERAL
                        (SELECT/*<pg.SelectStmt at 0x7f86ea689990>*/
                                (($1)::text)::edgedbpub."8d8ca2b5-bcaa-11ee-bbdb-8302b87d90c3_domain" AS "expr~15_value~1"
                        ) AS "expr-15~2"
                ) AS "expr-16~2"
        ) AS "expr-17~2"
    WHERE
        "expr-17~2"."expr~17_value~1"
    LIMIT
    (SELECT/*<pg.SelectStmt at 0x7f86ea680190>*/
            (101)::int8 AS "expr~18_value~1"
    )
  ```
</details>

<details>
  <summary>Working query</summary>

  ```edgeql
  select Item { id, name } filter (.category = <Category>"Products") and (.parent ?= <Item>{})
  ```
  ```edgeql
  [
    {
      "id": "a61e09ee-bcbe-11ee-95df-83d5c2b58749",
      "name": "Shirt"
    },
    {
      "id": "a61e4cce-bcbe-11ee-95df-dbe77209a58b",
      "name": "Computer"
    }
]
  ```
  ```sql
WITH
"t_schema::ObjectType~1" AS NOT MATERIALIZED
    (
    (SELECT/*<pg.SelectStmt at 0x7fb096b9af90>*/
            "expr-4~2"."ObjectType_value~1" AS "expr~4_identity~1",
            "expr-4~2"."name_value~1" AS "name_value~2"
        FROM
            LATERAL
            (SELECT/*<pg.SelectStmt at 0x7fb096b9b490>*/
                    "ObjectType~2".id AS "ObjectType_value~1",
                    "ObjectType~2"."9f6dd92a-0a42-5341-b599-fbf5494fd973" AS "name_value~1"    
                FROM
                    edgedbstd."2662a1b4-4f3f-5875-b6eb-ce52101a90a3_t" AS "ObjectType~2"       
                    CROSS JOIN LATERAL
                    (SELECT/*<pg.SelectStmt at 0x7fb096bbc7d0>*/
                            ("expr-1~2"."expr~1_value~1" OR
                             "expr-2~2"."expr~2_value~1") AS "expr~3_value~1"
                        FROM
                            LATERAL
                            (SELECT/*<pg.SelectStmt at 0x7fb096bbced0>*/
                                    (NOT "ObjectType~2"."101d5f8c-251a-5fee-b30d-c0a3ed3ded56") AS "expr~1_value~1"
                            ) AS "expr-1~2"
                            CROSS JOIN (SELECT (NULL)::uuid AS "v~2") AS "v~1"
                            CROSS JOIN LATERAL
                            (SELECT/*<pg.SelectStmt at 0x7fb096bbd2d0>*/
                                    ("ObjectType~2".id IS NOT DISTINCT FROM "v~1"."v~2") AS "expr~2_value~1"
                            ) AS "expr-2~2"
                    ) AS "expr-3~2"
                WHERE
                    "expr-3~2"."expr~3_value~1"
            ) AS "expr-4~2"
    ))
SELECT/*<pg.SelectStmt at 0x7fb097fca710>*/
        (
        (SELECT/*<pg.SelectStmt at 0x7fb096b7f350>*/
                "str___tname__~2"."expr~10_value~1" AS "__tname___serialized~1"
            FROM
                LATERAL
                (SELECT/*<pg.SelectStmt at 0x7fb096b98450>*/
                        "expr-10~2"."name_value~6" AS "expr~10_value~1"
                    FROM
                        LATERAL
                        (SELECT/*<pg.SelectStmt at 0x7fb096b99290>*/
                                "str_name~2"."name_value~5" AS "name_value~6"
                            FROM
                                LATERAL
                                (SELECT/*<pg.SelectStmt at 0x7fb096b99e10>*/
                                        "q~3"."name_value~4" AS "name_value~5"
                                    FROM
                                        LATERAL
                                        (SELECT/*<pg.SelectStmt at 0x7fb096b9a310>*/
                                                "q~2"."name_value~3" AS "name_value~4"
                                            FROM
                                                LATERAL
                                                (SELECT/*<pg.SelectStmt at 0x7fb096b9a950>*/   
                                                        "Item~2".__type__ AS "__type___identity~1"
                                                ) AS "q~1"
                                                JOIN LATERAL
                                                (SELECT/*<pg.SelectStmt at 0x7fb096b9b310>*/   
                                                        "t~1"."expr~4_identity~1" AS "expr~4_identity~2",
                                                        "t~1"."name_value~2" AS "name_value~3" 
                                                    FROM
                                                        "t_schema::ObjectType~1" AS "t~1"      
                                                ) AS "q~2"
                                                    ON ("q~1"."__type___identity~1" = "q~2"."expr~4_identity~2")
                                        ) AS "q~3"
                                ) AS "str_name~2"
                            LIMIT
                            (SELECT/*<pg.SelectStmt at 0x7fb096bbfe10>*/
                                    (101)::int8 AS "expr~9_value~1"
                            )
                        ) AS "expr-10~2"
                    LIMIT
                    (SELECT/*<pg.SelectStmt at 0x7fb096b991d0>*/
                            (101)::int8 AS "expr~11_value~1"
                    )
                ) AS "str___tname__~2"
        ),
        (SELECT/*<pg.SelectStmt at 0x7fb096d75c50>*/
                "uuid___tid__~2"."expr~6_value~1" AS "__tid___serialized~1"
            FROM
                LATERAL
                (SELECT/*<pg.SelectStmt at 0x7fb096bc8d90>*/
                        "expr-6~2"."__type___value~2" AS "expr~6_value~1"
                    FROM
                        LATERAL
                        (SELECT/*<pg.SelectStmt at 0x7fb096bc9750>*/
                                "uuid_id~3"."__type___value~1" AS "__type___value~2"
                            FROM
                                LATERAL
                                (SELECT/*<pg.SelectStmt at 0x7fb096bca2d0>*/
                                        "Item~2".__type__ AS "__type___value~1"
                                ) AS "uuid_id~3"
                            LIMIT
                            (SELECT/*<pg.SelectStmt at 0x7fb096bca990>*/
                                    (101)::int8 AS "expr~5_value~1"
                            )
                        ) AS "expr-6~2"
                    LIMIT
                    (SELECT/*<pg.SelectStmt at 0x7fb096bc95d0>*/
                            (101)::int8 AS "expr~7_value~1"
                    )
                ) AS "uuid___tid__~2"
        ), "Item~2".id, "Item~2"."b6c13820-b9ec-11ee-8b43-5963961d2db8") AS "Item_serialized~1" FROM
        edgedbpub."b6bd9168-b9ec-11ee-8fc8-1bf3579dccc0_t" AS "Item~2"
        CROSS JOIN LATERAL
        (SELECT/*<pg.SelectStmt at 0x7fb096bcbb10>*/
                ("expr-15~2"."expr~15_value~1" AND
                 "expr-16~2"."expr~16_value~1") AS "expr~17_value~1"
            FROM
                LATERAL
                (SELECT/*<pg.SelectStmt at 0x7fb096bd0050>*/
                        ("Item~2"."b321254d-bcbd-11ee-bf39-c90aa86b4895" = "expr-14~2"."expr~14_value~1") AS "expr~15_value~1"
                    FROM
                        LATERAL
                        (SELECT/*<pg.SelectStmt at 0x7fb096bd0a10>*/
                                (($1)::text)::edgedbpub."8d8ca2b5-bcaa-11ee-bbdb-8302b87d90c3_domain" AS "expr~14_value~1"
                        ) AS "expr-14~2"
                ) AS "expr-15~2"
                CROSS JOIN LATERAL
                (SELECT/*<pg.SelectStmt at 0x7fb096bd1910>*/
                        "q~6"."parent_value~3" AS "parent_value~4"
                    FROM
                        LATERAL
                        (SELECT/*<pg.SelectStmt at 0x7fb096bd1ed0>*/
                                ("m~1" = first_value("m~1") OVER ()) AS "m~1",
                                "q~5"."parent_value~1" AS "parent_value~3"
                            FROM
                                LATERAL
                                (
                                (SELECT/*<pg.SelectStmt at 0x7fb096bd2510>*/
                                        1 AS "m~1",
                                        "Item~3".id AS "parent_value~1"
                                    FROM
                                        LATERAL
                                        (SELECT/*<pg.SelectStmt at 0x7fb096bd1d90>*/
                                                "Item~2"."b320ba21-bcbd-11ee-b574-8d6ccb22a5a6" AS "parent_identity~1"
                                        ) AS "q~4"
                                        JOIN edgedbpub."b6bd9168-b9ec-11ee-8fc8-1bf3579dccc0_t" AS "Item~3"
                                            ON ("q~4"."parent_identity~1" = "Item~3".id)       
                                ) UNION ALL
                                (SELECT/*<pg.SelectStmt at 0x7fb096bd2590>*/
                                        2 AS "m~1",
                                        "v~3"."id~1" AS "parent_value~2"
                                    FROM
                                        (SELECT (NULL)::uuid AS "id~1") AS "v~3"
                                )
                                ) AS "q~5"
                        ) AS "q~6"
                    WHERE
                        "q~6"."m~1"
                ) AS "Item_parent~2"
                CROSS JOIN (SELECT (NULL)::uuid AS "id~2") AS "v~4"
                CROSS JOIN LATERAL
                (SELECT/*<pg.SelectStmt at 0x7fb096bd1110>*/
                        ("Item_parent~2"."parent_value~4" IS NOT DISTINCT FROM "v~4"."id~2") AS "expr~16_value~1"
                ) AS "expr-16~2"
        ) AS "expr-17~2"
    WHERE
        "expr-17~2"."expr~17_value~1"
    LIMIT
    (SELECT/*<pg.SelectStmt at 0x7fb096bcb9d0>*/
            (101)::int8 AS "expr~18_value~1"
    )
  ```
</details>

## Seed:
```edgeql
with
  shirt := (insert Item { name := "Shirt", category := "Products" }),
  computer := (insert Item { name := "Computer", category := "Products" }),
  keyboard := (insert Item { name := "Keyboard", category := "Products", parent := computer })
select (
  shirt,
  computer,
  keyboard
);
```

## Schema:
```esdl
module default {
  scalar type Category extending enum<Blueprints, Materials, Products>;

  type Item {
    required name: str;

    optional parent: Item;
    required category: Category;
  };
};
```


Discussion:
This is a weird one. For what it's worth, I believe this query gets at the same objects you want to filter but seems to return the correct result consistently:

```edgeql
select Item filter (not exists .parent) and (.category = <Category>"Products");
```

Resolving patch:
diff --git a/edb/pgsql/compiler/relctx.py b/edb/pgsql/compiler/relctx.py
--- a/edb/pgsql/compiler/relctx.py
+++ b/edb/pgsql/compiler/relctx.py
@@ -792,9 +792,18 @@ def update_scope(
         assert p.path_id is not None
         ctx.path_scope[p.path_id] = stmt
 
+
+def update_scope_masks(
+        ir_set: irast.Set, rvar: pgast.PathRangeVar, *,
+        ctx: context.CompilerContextLevel) -> None:
+
+    if not isinstance(rvar, pgast.RangeSubselect):
+        return
+    stmt = rvar.subquery
+
     # Mark any paths under the scope tree as masked, so that they
     # won't get picked up by pull_path_namespace.
-    for child_path in scope_tree.get_all_paths():
+    for child_path in ctx.scope_tree.get_all_paths():
         pathctx.put_path_id_mask(stmt, child_path)
 
     # If this is an optional scope node, we need to be certain that
@@ -807,11 +816,11 @@ def update_scope(
     # with some DML linkprop cases (probably easy to fix) and a number
     # of materialization cases (possibly hard to fix), so I'm going
     # with a more conservative approach.
-    if scope_tree.is_optional(ir_set.path_id):
+    if ctx.scope_tree.is_optional(ir_set.path_id):
         # Since compilation is done, anything visible to us *will* be
         # up on the spine. Anything tucked away under a node must have
         # been pulled up.
-        for anc in scope_tree.ancestors:
+        for anc in ctx.scope_tree.ancestors:
             for direct_child in anc.path_children:
                 if not direct_child.optional:
                     pathctx.put_path_id_mask(stmt, direct_child.path_id)
diff --git a/edb/pgsql/compiler/relgen.py b/edb/pgsql/compiler/relgen.py
--- a/edb/pgsql/compiler/relgen.py
+++ b/edb/pgsql/compiler/relgen.py
@@ -240,6 +240,7 @@ def get_set_rvar(
                 null_query, (pgast.SelectStmt, pgast.NullRelation))
             null_query.where_clause = pgast.BooleanConstant(val=False)
 
+        relctx.update_scope_masks(ir_set, rvars.main.rvar, ctx=subctx)
         result_rvar = _include_rvars(rvars, scope_stmt=scope_stmt, ctx=subctx)
         for aspect in rvars.main.aspects:
             pathctx.put_path_rvar_if_not_exists(


Test patch:
diff --git a/tests/test_edgeql_scope.py b/tests/test_edgeql_scope.py
--- a/tests/test_edgeql_scope.py
+++ b/tests/test_edgeql_scope.py
@@ -4112,3 +4112,22 @@ async def test_edgeql_scope_linkprop_assert_03(self):
                 }
             ],
         )
+
+    async def test_edgeql_scope_filter_qeq_01(self):
+        await self.assert_query_result(
+            r'''
+            select User filter .avatar ?= <Card>{} and .name = 'Bob';
+            ''',
+            [
+                {},
+            ],
+        )
+
+        await self.assert_query_result(
+            r'''
+            select User filter .name = 'Bob' and .avatar ?= <Card>{}
+            ''',
+            [
+                {},
+            ],
+        )