Repository: edgedb/edgedb

Issue:
`for a in b union (a ?? (insert X))` doesn't insert new items
<!-- Please search existing issues to avoid creating duplicates. -->

<!--
For the EdgeDB Version: run `edgedb query 'select sys::get_version_as_str()'` from your project directory (or run `select sys::get_version_as_str();` in the EdgeDB interactive shell).
For the EdgeDB CLI Version: Run `edgedb --version` from anywhere
-->

- EdgeDB Version: https://packages.edgedb.com/archive/x86_64-unknown-linux-gnu/edgedb-server-4.1%2B2194ddc.tar.zst
- EdgeDB CLI Version: 4.1.0-dev
- OS Version: x86_64 GNU/Linux 6.6.1-arch1-1

```edgeql
with
   my_num := <int64>$my_num,
   chats := (
      select ChatBase { members: { * }, messages: { * } } 
      filter my_num in .members.num
      order by assert_single(( select .messages order by .created_at limit 1 ))
   ),
   memberships := distinct (
      for chat in chats union (
         (select ChatMembership filter .chat = chat and .user.num = my_num) ?? (insert ChatMembership { chat := chat, user := (select User filter .num = my_num) })
      )
   ),
select {
   chats := chats { * },
   memberships := memberships { * }
}
```
When executing query, `memberships` is always empty even if `chats` is not, and new entries are not being inserted in database
Replacing `memberships := ...` with:
```
memberships := distinct (
  for chat in chats union (
     insert ChatMembership {
        chat := chat,
        user := (select User filter .num = my_num)
     } unless conflict on ((.user, .chat))
     else (select ChatMembership)
  )
),
```
make it work as intended.

Sample schema:
```
scalar type UsersNumericSequence extending sequence;
type User {
   required num: UsersNumericSequence { readonly := true; constraint exclusive; };
}
scalar type ChatsNumericSequence extending sequence;
abstract type ChatBase {
   required num: ChatsNumericSequence { readonly := true; constraint exclusive; };
   multi members: User { on target delete allow; };
   index on (.num);
}
type ChatMembership {
   required user: User { on target delete delete source; };
   required chat: ChatBase { on target delete delete source; };
   constraint exclusive on ((.chat, .user));
}
```


Discussion:
A minimized test case against our `insert` test schema:
```
insert Subordinate { name := "foo" };
```
then
```
for sub in Subordinate union (                                             
  (select InsertTest filter .sub = sub) ??                                 
  (insert InsertTest { l2 := -1, sub := sub })                             
);      
```
returns `{}`

Or this one is a bit more minimized, since `sub` has a link property on it so it gets stored twice:
```
for sub in Subordinate union (                                             
  (select Note filter .subject = sub) ??                                   
  (insert Note { name := "", subject := sub })                             
);
```
Reduces to a problem with (internal) optional for:
```
for sub in Subordinate union (
  for optional x in (<Note>{},) union (
    select 1
  )
);
```
gives an empty set

The issue is that the identity of the `sub` iterator is being lost in the NULL optional fallback case.

Resolving patch:
diff --git a/edb/pgsql/compiler/relctx.py b/edb/pgsql/compiler/relctx.py
--- a/edb/pgsql/compiler/relctx.py
+++ b/edb/pgsql/compiler/relctx.py
@@ -818,7 +818,7 @@ def update_scope(
     # with some DML linkprop cases (probably easy to fix) and a number
     # of materialization cases (possibly hard to fix), so I'm going
     # with a more conservative approach.
-    if scope_tree.optional:
+    if scope_tree.is_optional(ir_set.path_id):
         # Since compilation is done, anything visible to us *will* be
         # up on the spine. Anything tucked away under a node must have
         # been pulled up.


Test patch:
diff --git a/tests/test_edgeql_for.py b/tests/test_edgeql_for.py
--- a/tests/test_edgeql_for.py
+++ b/tests/test_edgeql_for.py
@@ -1107,6 +1107,25 @@ async def test_edgeql_for_fake_group_02(self):
             [{"key": "Earth"}, {"key": "Water"}]
         )
 
+    async def test_edgeql_for_tuple_optional_01(self):
+        await self.assert_query_result(
+            r'''
+                for user in User union (
+                  ((select (1,) filter false) ?? (2,)).0
+                );
+            ''',
+            [2, 2, 2, 2],
+        )
+
+        await self.assert_query_result(
+            r'''
+                for user in User union (
+                  ((select (1,) filter user.name = 'Alice') ?? (2,)).0
+                );
+            ''',
+            tb.bag([1, 2, 2, 2]),
+        )
+
     async def test_edgeql_for_optional_01(self):
         # Lol FOR OPTIONAL doesn't work for object-type iterators
         # but it does work for 1-ary tuples
@@ -1163,6 +1182,39 @@ async def test_edgeql_for_optional_01(self):
             [{}],
         )
 
+        await self.assert_query_result(
+            r'''
+                for user in (select User filter .name = 'Alice') union (
+                  for optional x in (<Card>{},) union (
+                    1
+                  )
+                );
+            ''',
+            [1],
+        )
+
+        await self.assert_query_result(
+            r'''
+                for user in (select User filter .name = 'Alice') union (
+                  for optional x in (<Card>{},) union (
+                    user.name
+                  )
+                );
+            ''',
+            ['Alice'],
+        )
+
+        await self.assert_query_result(
+            r'''
+                for user in (select User filter .name = 'Alice') union (
+                  for optional x in (<Card>{},) union (
+                    user.name ++ (x.0.name ?? "!")
+                  )
+                );
+            ''',
+            ['Alice!'],
+        )
+
     @test.xerror('''
         FOR OPTIONAL is disabled for object-type iterators
     ''')
diff --git a/tests/test_edgeql_insert.py b/tests/test_edgeql_insert.py
--- a/tests/test_edgeql_insert.py
+++ b/tests/test_edgeql_insert.py
@@ -6462,3 +6462,46 @@ async def test_edgeql_insert_coalesce_04(self):
             ],
             variables=([0, 1, 2, 3],)
         )
+
+    async def test_edgeql_insert_coalesce_05(self):
+        await self.con.execute('''
+            insert Subordinate { name := "foo" };
+        ''')
+
+        Q = '''
+        for sub in Subordinate union (
+          (select Note filter .subject = sub) ??
+          (insert Note { name := "", subject := sub })
+        );
+        '''
+
+        await self.assert_query_result(
+            Q,
+            [{}],
+        )
+        await self.assert_query_result(
+            Q,
+            [{}],
+        )
+        await self.assert_query_result(
+            'select count(Note)',
+            [1],
+        )
+
+        await self.con.execute('''
+            insert Subordinate { name := "bar" };
+            insert Subordinate { name := "baz" };
+        ''')
+
+        await self.assert_query_result(
+            Q,
+            [{}] * 3,
+        )
+        await self.assert_query_result(
+            Q,
+            [{}] * 3,
+        )
+        await self.assert_query_result(
+            'select count(Note)',
+            [3],
+        )