Repository: ibis-project/ibis

Issue:
bug: several example datasets broken on postgres
### What happened?

```python
[ins] In [1]: import ibis

[ins] In [2]: con = ibis.postgres.connect(
         ...:     user="postgres",
         ...:     password="postgres",
         ...:     host="0.0.0.0",
         ...:     port=5432,
         ...:     database="ibis_testing",
         ...: )

[ins] In [3]: ibis.set_backend(con)

[ins] In [4]: ibis.examples.starwars.fetch()
---------------------------------------------------------------------------
NumericValueOutOfRange                    Traceback (most recent call last)
Cell In[4], line 1
----> 1 ibis.examples.starwars.fetch()

File ~/github.com/ibis-project/ibis/ibis/examples/__init__.py:85, in Example.fetch(self, table_name, backend)
     81 # TODO: It should be possible to avoid this memtable call, once all
     82 # backends support passing a `pyarrow.Table` to `create_table`
     83 # directly.
     84 obj = ibis.memtable(table)
---> 85 return backend.create_table(table_name, obj, temp=True, overwrite=True)

File ~/github.com/ibis-project/ibis/ibis/backends/postgres/__init__.py:692, in Backend.create_table(self, name, obj, schema, database, temp, overwrite)
    689     else:
    690         table = obj
--> 692     self._run_pre_execute_hooks(table)
    694     query = self._to_sqlglot(table)
    695 else:

File ~/github.com/ibis-project/ibis/ibis/backends/__init__.py:1030, in BaseBackend._run_pre_execute_hooks(self, expr)
   1028 self._define_udf_translation_rules(expr)
   1029 self._register_udfs(expr)
-> 1030 self._register_in_memory_tables(expr)

File ~/github.com/ibis-project/ibis/ibis/backends/sql/__init__.py:257, in SQLBackend._register_in_memory_tables(self, expr)
    255 def _register_in_memory_tables(self, expr: ir.Expr) -> None:
    256     for memtable in expr.op().find(ops.InMemoryTable):
--> 257         self._register_in_memory_table(memtable)

File ~/github.com/ibis-project/ibis/ibis/backends/postgres/__init__.py:155, in Backend._register_in_memory_table(self, op)
    153 with self.begin() as cur:
    154     cur.execute(create_stmt_sql)
--> 155     execute_batch(cur, sql, data, 128)

File /nix/store/xglf45k8zzyfsk3a18c0f8j3kv77gkh7-python3-3.12.2-env/lib/python3.12/site-packages/psycopg2/extras.py:1216, in execute_batch(cur, sql, argslist, page_size)
   1214 for page in _paginate(argslist, page_size=page_size):
   1215     sqls = [cur.mogrify(sql, args) for args in page]
-> 1216     cur.execute(b";".join(sqls))

NumericValueOutOfRange: bigint out of range
```

`penguins` is also broken.  For `starwars`, at least, the issue is that the `height` column is defined as `bigint` when it should _definitely_ be a float.

### What version of ibis are you using?

main @ b3e27eb9d

### What backend(s) are you using, if any?

postgres

### Relevant log output

_No response_

### Code of Conduct

- [X] I agree to follow this project's Code of Conduct


Discussion:
I tried this out on 8.0.0 and it's also broken there, so not a new regression related to the sqlglot shift.
I was poking at this a little bit, just trying to narrow down the problem.

The immediate cause appears to be the pyarrow CSV reader -- if the backend isn't one of duckdb or polars, then we load the csv using pyarrow.csv, then make it into a memtable, then pass it to `create_table`.

`pyarrow.csv` is either not sniffing enough lines of the csv (probably this), but it could be something else.

I will note that this impacts most SQL backends that have strong opinions about types (so all of them except sqlite).
We could probably get around this by converting all of the example datasets to parquet instead using duckdb (and just replacing those on the pinboard)
@gforsyth Thanks for the pointer!

I dug a little deeper into what you had said above and I believe the issue arises when there are null values in a column. In the `starwars` example, the column `height` is breaking the import. Specifically, we convert the data into a pandas df and use `.itertuples()` to generate the SQL, and depending on how the `nan` value gets executed it may throw an error. In Postgres `nan` gets turned into `'NaN'::float` and that throws and error in a column that is supposed to be integers. In mysql it throws an exception like `ProgrammingError: nan can not be used with MySQL`. In Pyspark it throws ``PySparkTypeError: [CANNOT_ACCEPT_OBJECT_IN_TYPE] `LongType()` can not accept object `172.0` in type `float`.``

Can we replace these NaN values with None/NULL?

I wasn't able to test all of the backends (because I have trouble setting some of them up) but this is working for bigquery, clickhouse, dask, datafusion, pandas, sqlite, trino. Broken in mysql, pyspark.

I can raise a PR to fix this if we're on the same page.
I think it's definitely fine to convert NaNs to NULL -- the important thing here is for the backends to load some example data, so anything in service of that seems good to me!

Resolving patch:
diff --git a/ibis/backends/mysql/__init__.py b/ibis/backends/mysql/__init__.py
--- a/ibis/backends/mysql/__init__.py
+++ b/ibis/backends/mysql/__init__.py
@@ -11,6 +11,7 @@
 from typing import TYPE_CHECKING, Any
 from urllib.parse import parse_qs, urlparse
 
+import numpy as np
 import pymysql
 import sqlglot as sg
 import sqlglot.expressions as sge
@@ -481,6 +482,9 @@ def _register_in_memory_table(self, op: ops.InMemoryTable) -> None:
 
             columns = schema.keys()
             df = op.data.to_frame()
+            # nan can not be used with MySQL
+            df = df.replace(np.nan, None)
+
             data = df.itertuples(index=False)
             cols = ", ".join(
                 ident.sql(self.name)
diff --git a/ibis/backends/postgres/__init__.py b/ibis/backends/postgres/__init__.py
--- a/ibis/backends/postgres/__init__.py
+++ b/ibis/backends/postgres/__init__.py
@@ -11,8 +11,11 @@
 from typing import TYPE_CHECKING, Any, Callable
 from urllib.parse import parse_qs, urlparse
 
+import numpy as np
+import pandas as pd
 import sqlglot as sg
 import sqlglot.expressions as sge
+from pandas.api.types import is_float_dtype
 
 import ibis
 import ibis.common.exceptions as com
@@ -144,6 +147,16 @@ def _register_in_memory_table(self, op: ops.InMemoryTable) -> None:
 
             columns = schema.keys()
             df = op.data.to_frame()
+            # nan gets compiled into 'NaN'::float which throws errors in non-float columns
+            # In order to hold NaN values, pandas automatically converts integer columns
+            # to float columns if there are NaN values in them. Therefore, we need to convert
+            # them to their original dtypes (that support pd.NA) to figure out which columns
+            # are actually non-float, then fill the NaN values in those columns with None.
+            convert_df = df.convert_dtypes()
+            for col in convert_df.columns:
+                if not is_float_dtype(convert_df[col]):
+                    df[col] = df[col].replace(np.nan, None)
+
             data = df.itertuples(index=False)
             cols = ", ".join(
                 ident.sql(self.dialect)


Test patch: