Repository: edgedb/edgedb

Issue:
ISE on basic grouping
I've discovered this ISE by just playing with the shape of `group`

<!-- Please search existing issues to avoid creating duplicates. -->

<!--
For the EdgeDB Version: run `edgedb query 'select sys::get_version_as_str()'` from your project directory (or run `select sys::get_version_as_str();` in the EdgeDB interactive shell).
For the EdgeDB CLI Version: Run `edgedb --version` from anywhere
-->

- EdgeDB Version: 4.2+7038dfc / 4.2+8778fc1
- EdgeDB CLI Version: 4.0.2+da2788e / 4.0.2+b26c502
- OS Version: macOS 11.7.10 / Windows 10

Steps to Reproduce:

```
select (group Foo
by .detail).key.detail
```

<!-- If the issue is about a query error, please also provide your schema -->

Schema: 

```
type Foo {
    property detail -> str;
}
```

Problem:

`InternalServerError: Can't compile ref to visible binding (__derived__::detail~1@w~1)`



Discussion:
Interesting, this is working well for me on Windows. Maybe someone on a Mac can give it a try.

Any chance you can start a fresh instance and try this again to see if it still gives the same behavior? (Your super simple schema does look like a fresh instance, but just in case...)
Here I shared a stripped down version of my current schema, excluding other 49 types :) (I assume here should be absolutely no crosstalk between different tables with a simple `select/group` like this)
Just to make sure, I created a brand new project with a fresh out-of-the-oven instance, including nothing but this `Foo` type, on Windows 10, but I still experience the same problem.
Okay, I'll try again when I'm in front of the computer again but in the
meantime I wonder if it's related to the name of the property, because when
I tried the example it was with an identical structure but different type
and property names. Sometimes we've seen errors coming from the compiler
getting confused by a property with the same name as a keyword.

2023년 12월 4일 (월) 오후 6:48, themajashurka ***@***.***>님이 작성:

> Here I shared a stripped down of my current schema, excluding other 49
> types :) (I assume here should be absolutely no crosstalk between different
> tables with a simple select/group like this)
> Just to make sure, I created a brand new project with a fresh
> out-of-the-oven instance, including nothing but this Foo type, on Windows
> 10 (happened to work on two machines at the same time), but I still
> experience the same problem.
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/edgedb/edgedb/issues/6557#issuecomment-1838183672>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ANP2GL3BL6YPCQXV3B3MRRLYHWL7ZAVCNFSM6AAAAABAE47QACVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMZYGE4DGNRXGI>
> .
> You are receiving this because you commented.Message ID:
> ***@***.***>
>

Weird! I can't get the same error to reproduce regardless of if I give it the same property names or paste in your schema exactly as you have it. And I'm on Windows 11 which isn't that different from Windows 10 which you've confirmed does generate this error. I'll see if anyone else has any idea of what it might be.
I've tried with different property names, but it doesn't make a difference. It's weird indeed. Is there any database/instance level configuration to check?
A `using` doesn't help:
```
select (group Foo
using bar := .detail
by bar).key.bar
```

`InternalServerError: Can't compile ref to visible binding (__derived__::bar@w~1)`
In the meantime here is another idea: do you see anything interesting when you set RUST_LOG to trace? It'll be pages and pages of output (including every keystroke) but might be something in there of note.
How am I able to do that?
> How am I able to do this?

Oh sorry, just do this:

MacOS (probably): `set RUSTLOG = "trace"`

Windows in Powershell: `$env:RUST_LOG = "trace"`

Then you'll see oodles of output like this:
![image](https://github.com/edgedb/edgedb/assets/56599343/1e2c4daf-ff84-40e4-8a41-34488898ef22)

Okay, I'm about to lose it :) Until now, I've been trying querying the UI, but now because you suggested to check errors in the CLI, I've just done that, and it works. I mean, there are no errors, no ISE, just works.
UI repl error:
![image](https://github.com/edgedb/edgedb/assets/81056844/97c55892-5cd9-4803-8f0d-f360fdc310b4)

UI query editor error:
![image](https://github.com/edgedb/edgedb/assets/81056844/3214ffea-ae81-4e51-908b-f216c3e0726e)

I've inserted some dummy data, but the CLI finally does it's thing: 
![image](https://github.com/edgedb/edgedb/assets/81056844/0dba8c16-5a55-4022-af6b-3d6bb643847e)

Ah ha! That narrows it down at least to a UI issue and I guess we can move
it to there. I'll see if I can reproduce it too in a few minutes.

2023년 12월 4일 (월) 오후 11:18, themajashurka ***@***.***>님이 작성:

> Some dummy data, but the CLI finally does it's thing:
> image.png (view on web)
> <https://github.com/edgedb/edgedb/assets/81056844/0dba8c16-5a55-4022-af6b-3d6bb643847e>
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/edgedb/edgedb/issues/6557#issuecomment-1838732084>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ANP2GL5APLKYDBYUR57HBDLYHXLSVAVCNFSM6AAAAABAE47QACVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMZYG4ZTEMBYGQ>
> .
> You are receiving this because you commented.Message ID:
> ***@***.***>
>

Yes! I'm getting the same error in the UI, you are not alone.
> I've inserted some dummy data, but the CLI finally does it's thing: ![image](https://private-user-images.githubusercontent.com/81056844/287726460-0dba8c16-5a55-4022-af6b-3d6bb643847e.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTEiLCJleHAiOjE3MDE3MDIwMjMsIm5iZiI6MTcwMTcwMTcyMywicGF0aCI6Ii84MTA1Njg0NC8yODc3MjY0NjAtMGRiYThjMTYtNWE1NS00MDIyLWFmNmItM2Q2YmI2NDM4NDdlLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFJV05KWUFYNENTVkVINTNBJTJGMjAyMzEyMDQlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjMxMjA0VDE0NTUyM1omWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPThiZDFjYzQ1MGUzMjRiZTJlM2I1MGM4YTY1MTQ5MzU3NTRkZDE2MGY3YjlkNmEwYWE5MmYzMjFmMDBhMjhlMGYmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.OgTqzflAJ-VGnDXNJYg9PtI4fZVeMM1Y6CNTzvd56Ig)

Okay, looks like the issue is going to stay here - James mentioned that it's probably not a UI issue but a something the UI uses issue and related to the compiler:

>Since the error message is an ISE, I think it's probably not directly a UI issue, but a compiler bug related to one of the implicit typename/typeid/objectid injection flags the UI uses and the CLI/bindings don't.
The weird non reproducibility seems to be based on whether an implicit limit is injected into the query, which is kind of wild.

By default the CLI has a limit of 100; maybe the UI doesn't have one.
(I disabled the limit on my CLI so I reproduced it immediately)

Resolving patch:
diff --git a/edb/pgsql/compiler/expr.py b/edb/pgsql/compiler/expr.py
--- a/edb/pgsql/compiler/expr.py
+++ b/edb/pgsql/compiler/expr.py
@@ -769,6 +769,9 @@ def _compile_shape(
             name=el.name,
             val=ser_val
         ))
+        # Don't let the elements themselves leak out, since they may
+        # be wrapped in arrays.
+        pathctx.put_path_id_mask(ctx.rel, el.path_id)
 
     ser_result = pgast.TupleVar(elements=ser_elements, named=True)
     sval = output.serialize_expr(
diff --git a/edb/pgsql/compiler/group.py b/edb/pgsql/compiler/group.py
--- a/edb/pgsql/compiler/group.py
+++ b/edb/pgsql/compiler/group.py
@@ -209,6 +209,8 @@ def _compile_group(
         # Now we compile the bindings
         groupctx.path_scope = subjctx.path_scope.new_child()
         groupctx.path_scope[stmt.group_binding.path_id] = None
+        if stmt.grouping_binding:
+            groupctx.path_scope[stmt.grouping_binding.path_id] = None
 
         # Compile all the 'using' items
         for _alias, (value, using_card) in stmt.using.items():
@@ -359,6 +361,9 @@ def _get_volatility_ref() -> Optional[pgast.BaseExpr]:
         return vol_ref
 
     with ctx.new() as outctx:
+        # Inherit the path_scope we made earlier (with the GROUP bindings
+        # removed), so that we'll always look for those in the right place.
+        outctx.path_scope = groupctx.path_scope
 
         outctx.volatility_ref += (lambda stmt, xctx: _get_volatility_ref(),)
 
@@ -368,6 +373,8 @@ def _get_volatility_ref() -> Optional[pgast.BaseExpr]:
         clauses.compile_output(stmt.result, ctx=outctx)
 
     with ctx.new() as ictx:
+        ictx.path_scope = groupctx.path_scope
+
         # FILTER and ORDER BY need to have the base result as a
         # volatility ref.
         clauses.setup_iterator_volatility(stmt.result, ctx=ictx)
diff --git a/edb/pgsql/compiler/relgen.py b/edb/pgsql/compiler/relgen.py
--- a/edb/pgsql/compiler/relgen.py
+++ b/edb/pgsql/compiler/relgen.py
@@ -1132,6 +1132,8 @@ def process_set_as_path(
 
     elif not source_is_visible:
         with ctx.subrel() as srcctx:
+            srcctx.expr_exposed = False
+
             get_set_rvar(ir_source, ctx=srcctx)
 
             if is_inline_primitive_ref:
@@ -1341,12 +1343,14 @@ def process_set_as_subquery(
         if source_is_visible and (
             ir_source.path_id not in ctx.skippable_sources
         ):
-            source_set_rvar = get_set_rvar(ir_source, ctx=ctx)
-            # Force a source rvar so that trivial computed pointers
-            # on erroneous objects (like a bad array deref) fail.
-            # (Most sensible computables will end up requiring the
-            # source rvar anyway.)
-            ensure_source_rvar(ir_source, stmt, ctx=ctx)
+            with ctx.new() as sctx:
+                sctx.expr_exposed = False
+                source_set_rvar = get_set_rvar(ir_source, ctx=sctx)
+                # Force a source rvar so that trivial computed pointers
+                # on erroneous objects (like a bad array deref) fail.
+                # (Most sensible computables will end up requiring the
+                # source rvar anyway.)
+                ensure_source_rvar(ir_source, stmt, ctx=sctx)
     else:
         ir_source = None
         source_is_visible = False


Test patch:
diff --git a/tests/test_edgeql_group.py b/tests/test_edgeql_group.py
--- a/tests/test_edgeql_group.py
+++ b/tests/test_edgeql_group.py
@@ -1381,3 +1381,21 @@ async def test_edgeql_group_backlink(self):
             ''',
             [{"a": {}}, {"a": {}}],
         )
+
+    async def test_edgeql_group_destruct_immediately_01(self):
+        await self.assert_query_result(
+            r'''
+            WITH MODULE cards
+            select (group Card by .element).key.element
+            ''',
+            {"Fire", "Earth", "Water", "Air"},
+        )
+
+    async def test_edgeql_group_destruct_immediately_02(self):
+        await self.assert_query_result(
+            r'''
+            WITH MODULE cards
+            select (group Card by .element).grouping
+            ''',
+            ["element", "element", "element", "element"],
+        )
diff --git a/tests/test_edgeql_select.py b/tests/test_edgeql_select.py
--- a/tests/test_edgeql_select.py
+++ b/tests/test_edgeql_select.py
@@ -5930,6 +5930,14 @@ async def test_edgeql_select_bigint_index_03(self):
             ['[]'],
         )
 
+    async def test_edgeql_select_multi_property_shape_01(self):
+        await self.assert_query_result(
+            r"""
+            select (BooleanTest { tags }).tags
+            """,
+            tb.bag(['red', 'black', 'red', 'green', 'red']),
+        )
+
     async def test_edgeql_select_tuple_01(self):
         await self.assert_query_result(
             r"""