Repository: ibis-project/ibis

Issue:
bug: to_parquet() writes incorrect data without previous .cache()
### What happened?

OK, I don't have a nice repro for this because I'm not sure exactly what is causing it, and it's too complicated for me to share it all. If you could help me figure out where exactly to look then I gladly will.

I am loading some data, doing a bunch of processing, and then writing `.to_parquet()`. It writes fine, but several columns are always null, when they should have contents. If I call .cache() before writing, then they write with the correct contents.

Here is the output of `ibis.show_sql(table)` just before the `.to_parquet()` if that is helpful: [log.sql](https://github.com/ibis-project/ibis/files/11327266/log.txt)

When I do this, I get these warnings, if you think that could be related. Using `sqlglot==11.5.7`
```
WARNING:sqlglot:REGEXP_EXTRACT does not support arg(s) ['position']
WARNING:sqlglot:REGEXP_EXTRACT does not support arg(s) ['position']
WARNING:sqlglot:REGEXP_EXTRACT does not support arg(s) ['position']
WARNING:sqlglot:REGEXP_EXTRACT does not support arg(s) ['position']
WARNING:sqlglot:REGEXP_EXTRACT does not support arg(s) ['position']
WARNING:sqlglot:REGEXP_EXTRACT does not support arg(s) ['position']
WARNING:sqlglot:REGEXP_EXTRACT does not support arg(s) ['position']
```

Let me know how you think I can debug this further.

### What version of ibis are you using?

5.1.0

### What backend(s) are you using, if any?

duckdb

### Relevant log output

_No response_

### Code of Conduct

- [X] I agree to follow this project's Code of Conduct


Discussion:
Hey @NickCrews -- sorry for the delay in getting to this, a few of us were at a conference this last week.

It's obviously hard to debug without a reproducer, but the fact that `cache` seems to make a difference is a little suspicious.

Can you also post the `show_sql` of your query with and without the `cache` call?  (or a diff of the two)

The "sometimes null sometimes not" issue does bring to mind arrow RecordBatchReader's getting exhausted, but from the sql output I don't think you'd be hitting any of the `pyarrow` read paths
Another thing that would be useful is knowing the schema of the input CSV, i.e. the types of these columns:

```
   3   │     t5."Date" AS "Date",
   4   │     t5."Payment Type" AS "Payment Type",
   5   │     t5."Amount" AS "Amount",
   6   │     t5."Last/Business Name" AS "Last/Business Name",
   7   │     t5."First Name" AS "First Name",
   8   │     t5."Address" AS "Address",
   9   │     t5."City" AS "City",
  10   │     t5."State" AS "State",
  11   │     t5."Zip" AS "Zip",
  12   │     t5."Country" AS "Country",
  13   │     t5."Occupation" AS "Occupation",
  14   │     t5."Employer" AS "Employer",
  15   │     t5."Election Name" AS "Election Name",
  16   │     t5."Municipality" AS "Municipality",
  17   │     t5."Office" AS "Office",
  18   │     t5."Filer Type" AS "Filer Type",
  19   │     t5."Name" AS "Name",
  20   │     t5."Report Year" AS "Report Year",
  21   │     t5."Submitted" AS "Submitted"
```
Hi @NickCrews, can you provide some more context here ^^?
Yeah sorry, I realize this is in my court, I just haven't found time to dig back into this.

In the original post I already posted the SQL without the .cache(). I'm not exactly how to show the SQL with the .cache(), isn't that going to be something useless like `SELECT * FROM _ibis_cached_adfasdfasdfasdf`? What I really want is to show is the SQL from the DDL statement `CREATE TEMP TABLE ....`, but how do I do that?
I think we need the schema from the columns in my previous comment to be able to try to generate some fake data, so that we can run the SQL you posted against the fake data and try to reproduce.
@NickCrews -- have you seen this happen again with the recent Ibis (6.0) and DuckDB (8.1) releases?  Otherwise we'll close it out.  Thanks!
I'm on vacation until next Tuesday, I'll take a look next week. Thanks!
OK, my code has evolved so much that I can't get the exact same error, but I get something similar, in the sense that things work with .cache(), and fail without cache(). This time, I get an error about not being able to find a memtable.

I'm still using ibis 5.1.0, but duckdb 0.8.0. I can update to ibis 6.0.0 if you think that's significant.

If I try to write to parquet without a .cache(), I get
```
sqlalchemy.exc.ProgrammingError: (duckdb.CatalogException) Catalog Error: Table with name _ibis_pyarrow_memtable_3vsypavcsr7399y9a3cwbk8bt does not exist!
Did you mean "pg_tablespace"?
```
from the attached SQL.
[without_cache.txt](https://github.com/ibis-project/ibis/files/12045466/err_sql.txt)

I still don't know how to show the SQL of the DDL statement that happens within the .cache(), as I assume that is going to be different from the above sql.
Hey @NickCrews -- I have a fix incoming for this (or at least, I have a reproducer for that error that I can fix).

Resolving patch:
diff --git a/ibis/backends/duckdb/__init__.py b/ibis/backends/duckdb/__init__.py
--- a/ibis/backends/duckdb/__init__.py
+++ b/ibis/backends/duckdb/__init__.py
@@ -821,6 +821,7 @@ def to_parquet(
         >>> # partition on multiple columns
         >>> con.to_parquet(penguins, "penguins_hive_dir", partition_by=("year", "island"))  # doctest: +SKIP
         """
+        self._run_pre_execute_hooks(expr)
         query = self._to_sql(expr, params=params)
         args = ["FORMAT 'parquet'", *(f"{k.upper()} {v!r}" for k, v in kwargs.items())]
         copy_cmd = f"COPY ({query}) TO {str(path)!r} ({', '.join(args)})"
@@ -855,6 +856,7 @@ def to_csv(
         **kwargs
             DuckDB CSV writer arguments. https://duckdb.org/docs/data/csv.html#parameters
         """
+        self._run_pre_execute_hooks(expr)
         query = self._to_sql(expr, params=params)
         args = [
             "FORMAT 'csv'",


Test patch:
diff --git a/ibis/backends/tests/test_export.py b/ibis/backends/tests/test_export.py
--- a/ibis/backends/tests/test_export.py
+++ b/ibis/backends/tests/test_export.py
@@ -254,11 +254,42 @@ def test_roundtrip_partitioned_parquet(tmp_path, con, backend, awards_players):
 
     # Reingest and compare schema
     reingest = con.read_parquet(outparquet / "*" / "*")
+    reingest = reingest.cast({"yearID": "int64"})
 
     # avoid type comparison to appease duckdb: as of 0.8.0 it returns large_string
     assert reingest.schema().names == awards_players.schema().names
 
-    backend.assert_frame_equal(awards_players.to_pandas(), awards_players.to_pandas())
+    reingest = reingest.order_by(["yearID", "playerID", "awardID", "lgID"])
+    awards_players = awards_players.order_by(["yearID", "playerID", "awardID", "lgID"])
+
+    backend.assert_frame_equal(reingest.to_pandas(), awards_players.to_pandas())
+
+
+@pytest.mark.notimpl(
+    ["dask", "impala", "pyspark"], reason="No support for exporting files"
+)
+@pytest.mark.notimpl(
+    ["datafusion"],
+    reason="No memtable support",
+)
+@pytest.mark.parametrize("ftype", ["csv", "parquet"])
+def test_memtable_to_file(tmp_path, con, ftype, monkeypatch):
+    """
+    Tests against a regression spotted in #6091 where a `memtable` that is
+    created and then immediately exported to `parquet` (or csv) will error
+    because we weren't registering the in-memory table before trying to export
+    it.
+    """
+    outfile = tmp_path / f"memtable.{ftype}"
+    assert not outfile.is_file()
+
+    monkeypatch.setattr(ibis.options, "default_backend", con)
+
+    memtable = ibis.memtable({"col": [1, 2, 3, 4]})
+
+    getattr(con, f"to_{ftype}")(memtable, outfile)
+
+    assert outfile.is_file()
 
 
 @pytest.mark.notimpl(["dask", "impala", "pyspark"])