Repository: edgedb/edgedb

Issue:
SQL mode COPY includes internal columns, produces columns in nondiscoverable order
```
scope=# copy "User" to stdout;
27995	898	1149	0	1149	(0,1)	Alice	f5ab708e-1534-11ee-8439-6fca9e7c1d0c	ecb718b1-1534-11ee-b27d-ab578c0a5e0c	f5ab7086-1534-11ee-8439-973fa5acc1b4
...
```
I don't think we should include `ctid` and friends. They don't get included when running `copy` on normal postgres.

Additionally, the order that the columns are reported out does not correspond to the `ordinal_position`s reported when querying `information_schema`/`pg_catalog`

It also includes `__type__`, while `information_schema` omits it. (It does so deliberately, but I think it should probably start including it?).

I think that I am indifferent to whether the column order produced by `COPY` is changed to match `ordinal_position` or vice-versa.

(Assigning to Victor because I know he's looking at another SQL thing right now, but you should feel free to kick it over to @aljazerzen if you think that makes more sense)


Discussion:
https://github.com/edgedb/edgedb/pull/5715 contains a test of COPY that needed to be updated when a column got moved around. A good test for this issue would be to actually query `information_schema` to figure out what the correct index is.

Resolving patch:
diff --git a/edb/pgsql/resolver/command.py b/edb/pgsql/resolver/command.py
--- a/edb/pgsql/resolver/command.py
+++ b/edb/pgsql/resolver/command.py
@@ -42,19 +42,26 @@ def resolve_CopyStmt(stmt: pgast.CopyStmt, *, ctx: Context) -> pgast.CopyStmt:
         # cannot be copied as a table, but needs to be wrapped into a
         # `SELECT * FROM view`.
         relation, table = dispatch.resolve_relation(stmt.relation, ctx=ctx)
+        col_map: Dict[str, str] = {
+            col.name: col.reference_as
+            for col in table.columns
+            if col.name and col.reference_as
+        }
         if stmt.colnames:
-            col_map: Dict[str, str] = {
-                col.name: col.reference_as
-                for col in table.columns
-                if col.name and col.reference_as
-            }
             col_names = [col_map[name] for name in stmt.colnames]
 
         if relation.schemaname == 'edgedbpub':
             # This is probably a view based on edgedb schema, so wrap it into
             # a select query.
             if not col_names:
-                target_list: Sequence[str | pgast.Star] = [pgast.Star()]
+                # Avoid adding the system columns here. Also order the columns
+                # the same way as in the information_schema: id first, then
+                # alphabetically.
+                target_list = [col_map['id']]
+                for col_name, real_col in sorted(col_map.items()):
+                    if col_name not in {'id', 'tableoid', 'xmin', 'cmin',
+                                        'xmax', 'cmax', 'ctid'}:
+                        target_list.append(real_col)
             else:
                 target_list = col_names
             query = pgast.SelectStmt(


Test patch:
diff --git a/tests/test_sql_query.py b/tests/test_sql_query.py
--- a/tests/test_sql_query.py
+++ b/tests/test_sql_query.py
@@ -900,11 +900,17 @@ async def test_sql_query_copy_01(self):
             "Movie", output=out, format="csv", delimiter="\t"
         )
         out = io.StringIO(out.getvalue().decode("utf-8"))
-        # FIXME(#5716): Once COPY and information_schema are
-        # harmonized to agree on the order of columns, we should query
-        # information_schema to get the column number instead of
-        # hardcoding it.
-        names = set(row[7] for row in csv.reader(out, delimiter="\t"))
+        # Get the columns order from the information_schema.
+        res = await self.squery_values(
+            r'''
+            SELECT column_name, ordinal_position
+            FROM information_schema.columns cols
+            WHERE cols.table_name = 'Movie'
+            '''
+        )
+        col_map = {name: num - 1 for name, num in res}
+        names = set(
+            row[col_map['title']] for row in csv.reader(out, delimiter="\t"))
         self.assertEqual(names, {"Forrest Gump", "Saving Private Ryan"})
 
     async def test_sql_query_error_01(self):