Repository: ibis-project/ibis

Issue:
feat: Uint dtypes from duckdb engine
### Is your feature request related to a problem?

I just tried to upgrade to ibis 6.0.0, and am running into a new problem trying to open a parquet that contains a Uint8. It worked in 5.1.0.

I have duckdb-engine==0.7.3 installed, and so the dtype of this column is interpreted as `<class 'duckdb_engine.datatypes.UInt8'>` ([added about a year ago to duckdb-engine](https://github.com/Mause/duckdb_engine/blame/beac20b5362a0f14aeee9d8a6f7281caf9d5b724/duckdb_engine/datatypes.py)). It looks like this type isn't included in the registry of types in `ibis/backends/duckdb/datatypes.py`, and so the lookup to an Ibis type fails:

```
Traceback (most recent call last):
  File "/Users/nickcrews/Documents/projects/scg/kobuk/kobuk/_yukon/add_financials.py", line 182, in <module>
    cli.fire()
  File "/Users/nickcrews/Documents/projects/scg/noatak/noatak/mysplink/api.py", line 202, in fire
    return fire.Fire({"train": self.train, "run": self.run})
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/nickcrews/Documents/projects/scg/kobuk/.venv/lib/python3.11/site-packages/fire/core.py", line 141, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/nickcrews/Documents/projects/scg/kobuk/.venv/lib/python3.11/site-packages/fire/core.py", line 475, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
                                ^^^^^^^^^^^^^^^^^^^^
  File "/Users/nickcrews/Documents/projects/scg/kobuk/.venv/lib/python3.11/site-packages/fire/core.py", line 691, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
                ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/nickcrews/Documents/projects/scg/noatak/noatak/mysplink/api.py", line 296, in train
    runner = self.make_runner(
             ^^^^^^^^^^^^^^^^^
  File "/Users/nickcrews/Documents/projects/scg/noatak/noatak/mysplink/api.py", line 330, in make_runner
    left, right = self._load(left, right, head)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/nickcrews/Documents/projects/scg/noatak/noatak/mysplink/api.py", line 343, in _load
    left_new, right_new = self.ioer.loader()
                          ^^^^^^^^^^^^^^^^^^
  File "/Users/nickcrews/Documents/projects/scg/kobuk/kobuk/_yukon/add_financials.py", line 164, in load_data
    return load_receipts(), load_financials()
           ^^^^^^^^^^^^^^^
  File "/Users/nickcrews/Documents/projects/scg/kobuk/kobuk/_yukon/add_financials.py", line 125, in load_receipts
    return ibis.read_parquet(p)
           ^^^^^^^^^^^^^^^^^^^^
  File "/Users/nickcrews/Documents/projects/scg/kobuk/.venv/lib/python3.11/site-packages/ibis/expr/api.py", line 963, in read_parquet
    return con.read_parquet(sources, table_name=table_name, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/nickcrews/Documents/projects/scg/kobuk/.venv/lib/python3.11/site-packages/ibis/backends/duckdb/__init__.py", line 428, in read_parquet
    return self.table(table_name)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/nickcrews/Documents/projects/scg/kobuk/.venv/lib/python3.11/site-packages/ibis/backends/base/sql/alchemy/__init__.py", line 608, in table
    schema = self._schema_from_sqla_table(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/nickcrews/Documents/projects/scg/kobuk/.venv/lib/python3.11/site-packages/ibis/backends/base/sql/alchemy/__init__.py", line 519, in _schema_from_sqla_table
    dtype = cls.compiler.translator_class.get_ibis_type(
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/nickcrews/Documents/projects/scg/kobuk/.venv/lib/python3.11/site-packages/ibis/backends/base/sql/alchemy/translator.py", line 81, in get_ibis_type
    return cls.type_mapper.to_ibis(sqla_type, nullable=nullable)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/nickcrews/Documents/projects/scg/kobuk/.venv/lib/python3.11/site-packages/ibis/backends/duckdb/datatypes.py", line 131, in to_ibis
    return super().to_ibis(typ, nullable=nullable)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/nickcrews/Documents/projects/scg/kobuk/.venv/lib/python3.11/site-packages/ibis/backends/base/sql/alchemy/datatypes.py", line 316, in to_ibis
    raise TypeError(f"Unable to convert type: {typ!r}")
TypeError: Unable to convert type: UInt8()
```

### Describe the solution you'd like

Probably some compat code so that Ibis works with multiple versions of duckdb-engine, either with or without the Uint types.

### What version of ibis are you running?

worked in 5.1.0, fails in 6.1.0.

### What backend(s) are you using, if any?

duckdb

### Code of Conduct

- [X] I agree to follow this project's Code of Conduct


Discussion:
Thanks for reporting this @NickCrews!

I can't reproduce this by using Ibis to create a parquet file with `UInt8` dtypes and then re-ingesting it.  Do you have a sample file that errors that you can share?  I think I know what the problem is, but it would be helpful to confirm.
[This file](https://github.com/ibis-project/ibis/files/12074667/600_load_err.parquet.zip) loads ok with `ibis.read_parquet(path)` in 5.1.0, but I get the above error in 6.0.0. It loads fine with pandas and with pyarrow.parquet.

Oddly, the parquet file doesn't actually contain Uint8s, just other uint types. Note that the dtypes don't totally make sense, eg donor__middle_name is an int32 when it should be string, because I made this with `pd.read_parquet().head().to_parquet()`, which seems to mess up the dtypes from the original. But that isn't relevant for this bug.
```
ibis.Schema {
  donation__date           timestamp
  donation__method         string
  donation__amount         float32
  donor__prefix            int32
  donor__first_name        string
  donor__middle_name       int32
  donor__last_name         string
  donor__suffix            int32
  donor__nickname          int32
  donor__street            string
  donor__country           string
  donor__occupation        string
  donor__employer          string
  contest__municipality    int32
  recipient__type          string
  recipient__display_name  string
  report__year             int16
  donation__id             int32
  contest__year            float64
  contest__level           int32
  contest__office          int32
  election__stage          int32
  donor__phone_home        int32
  donor__phone_cell        int32
  donor__dob               timestamp
  donor__gender            int32
  donor__race              int32
  donor__email             int32
  street_extended          int32
  donor__city              string
  donor__state             string
  donor__zipcode           string
  donor__latitude          float64
  donor__longitude         float64
  donor__org_name          int32
  donor__street_extended   int32
}
```


The rest of the installed deps remain the same, only the ibis version changed. Output of `pip freeze`:

```
affinegap==1.12
altair==5.0.1
annotated-types==0.5.0
appnope==0.1.3
arrow==1.2.3
asttokens==2.2.1
async-generator==1.10
atpublic==3.1.2
attrs==23.1.0
autopep8==2.0.2
backcall==0.2.0
beautifulsoup4==4.12.2
bidict==0.22.1
black==23.7.0
bs4==0.0.1
cachetools==5.3.1
certifi==2023.5.7
charset-normalizer==3.2.0
click==8.1.5
click-default-group==1.2.2
comm==0.1.3
dateparser==1.1.8
debugpy==1.6.7
decorator==5.1.1
DoubleMetaphone==1.1
duckdb==0.8.1
duckdb-engine==0.9.1
editdistance==0.6.2
editdistpy==0.1.3
et-xmlfile==1.1.0
exceptiongroup==1.1.2
executing==1.2.0
filelock==3.12.2
fire==0.5.0
future==0.18.3
gender-extractor==0.1.3
geocoder==1.38.1
get-chrome-driver==1.3.12
google-api-core==2.11.1
google-api-python-client==2.93.0
google-auth==2.22.0
google-auth-httplib2==0.1.0
google-auth-oauthlib==1.0.0
google-cloud-bigquery==3.11.3
google-cloud-core==2.3.3
google-crc32c==1.5.0
google-resumable-media==2.5.0
googleapis-common-protos==1.59.1
greenlet==2.0.2
grpcio==1.56.0
grpcio-status==1.56.0
gspread==5.10.0
h11==0.14.0
httplib2==0.22.0
ibis-framework==5.1.0
idna==3.4
ipykernel==6.24.0
ipython==8.14.0
ipywidgets==8.0.7
jedi==0.18.2
Jinja2==3.1.2
jsonschema==4.18.3
jsonschema-specifications==2023.6.1
jupyter_client==8.3.0
jupyter_core==5.3.1
jupyterlab-widgets==3.0.8
Levenshtein==0.21.1
lxml==4.9.2
markdown-it-py==3.0.0
MarkupSafe==2.1.3
matplotlib-inline==0.1.6
mdurl==0.1.2
Metaphone==0.6
multimethod==1.9.1
multipledispatch==0.6.0
mypy==1.4.1
mypy-extensions==1.0.0
nameparser==1.1.2
nbqa==1.7.0
nest-asyncio==1.5.6
networkx==3.1
nicknames==0.1.6
numpy==1.25.1
oauthlib==3.2.2
openpyxl==3.1.2
outcome==1.2.0
packaging==23.1
pandas==2.0.3
pandera==0.14.5
parso==0.8.3
parsy==2.1
pathspec==0.11.1
pexpect==4.8.0
pgeocode==0.4.0
phonetics==1.0.5
pickleshare==0.7.5
platformdirs==3.8.1
pluggy==1.2.0
pony==0.7.16
pooch==1.7.0
probableparsing==0.0.1
probablepeople==0.5.4
prompt-toolkit==3.0.39
proto-plus==1.22.3
protobuf==4.23.4
psutil==5.9.5
ptyprocess==0.7.0
pure-eval==0.2.2
pyarrow==11.0.0
pyasn1==0.5.0
pyasn1-modules==0.3.0
pybaum==0.1.3
pycodestyle==2.10.0
pydantic==2.0.3
pydantic_core==2.3.0
Pygments==2.15.1
pygraphviz==1.11
pyparsing==3.1.0
pyrsistent==0.19.3
PySocks==1.7.1
pytask==0.3.2
python-crfsuite==0.9.9
python-dateutil==2.8.2
python-dotenv==1.0.0
python-Levenshtein==0.21.1
pytz==2023.3
PyYAML==6.0
pyzmq==25.1.0
rapidfuzz==3.1.1
ratelim==0.1.6
referencing==0.29.1
regex==2023.6.3
requests==2.31.0
requests-oauthlib==1.3.1
rich==13.4.2
rpds-py==0.8.10
rsa==4.9
ruff==0.0.278
selenium==4.10.0
six==1.16.0
sniffio==1.3.0
sortedcontainers==2.4.0
soupsieve==2.4.1
splink==3.9.3
SQLAlchemy==2.0.18
sqlalchemy-views==0.3.2
sqlglot==11.4.1
stack-data==0.6.2
symspellpy==6.7.7
termcolor==2.3.0
thefuzz==0.19.0
tokenize-rt==5.1.0
tomli==2.0.1
toolz==0.12.0
tornado==6.3.2
tqdm==4.65.0
traitlets==5.9.0
trio==0.22.2
trio-websocket==0.10.3
typer==0.9.0
typing-inspect==0.9.0
typing_extensions==4.7.1
tzdata==2023.3
tzlocal==5.0.1
Unidecode==1.3.6
uritemplate==4.1.1
urllib3==1.26.16
usaddress==0.5.10
usaddress-scourgify==0.5.1
wcwidth==0.2.6
widgetsnbextension==4.0.8
wrapt==1.15.0
wsproto==1.2.0
xxhash==3.2.0
yaml-config==0.1.5
```

I can't reproduce the problem. It may have been fixed since 6.0.0 was released.
> Note that the dtypes don't totally make sense, eg donor__middle_name is an int32 when it should be string, because I made this with pd.read_parquet().head().to_parquet()

See https://github.com/duckdb/duckdb/issues/7149. In particular:

> This applies not only to query results, but also e.g. for our own tables, or when exporting data to Parquet or similar.

I would recommend `expr.cast`-ing columns to their desired type before calling `to_parquet`, if you expect that all the values in the column may be null.
hmm, I _can_ reproduce the problem, which is confusing given that @cpcloud and I have _highly_ similar environments.  Anyway, I'll take a look.
I am a bad person and was testing on a different branch than `master` (`renovate/lock-file-maintenance`) and on `master` it reproduces for me.

This suggests a dependency or related interaction is in play. @gforsyth don't let me derail you though.

Resolving patch:
diff --git a/ibis/backends/duckdb/datatypes.py b/ibis/backends/duckdb/datatypes.py
--- a/ibis/backends/duckdb/datatypes.py
+++ b/ibis/backends/duckdb/datatypes.py
@@ -101,9 +101,17 @@ def parse(text: str, default_decimal_parameters=(18, 3)) -> dt.DataType:
     ducktypes.SmallInteger: dt.Int16,
     ducktypes.Integer: dt.Int32,
     ducktypes.BigInteger: dt.Int64,
+    ducktypes.HugeInteger: dt.Int64,
+    # The following dtypes are present in duckdb_engine and will show up
+    # in _some_ tables, but it isn't clear _why_.
+    # Please don't remove them.
+    ducktypes.UInt8: dt.UInt8,
     ducktypes.UTinyInteger: dt.UInt8,
+    ducktypes.UInt16: dt.UInt16,
     ducktypes.USmallInteger: dt.UInt16,
+    ducktypes.UInt32: dt.UInt32,
     ducktypes.UInteger: dt.UInt32,
+    ducktypes.UInt64: dt.UInt64,
     ducktypes.UBigInteger: dt.UInt64,
 }
 


Test patch:
diff --git a/ibis/backends/duckdb/tests/test_datatypes.py b/ibis/backends/duckdb/tests/test_datatypes.py
--- a/ibis/backends/duckdb/tests/test_datatypes.py
+++ b/ibis/backends/duckdb/tests/test_datatypes.py
@@ -133,3 +133,23 @@ def test_generate_quoted_struct():
     result = typ.compile(dialect=duckdb_engine.Dialect())
     expected = 'STRUCT("in come" TEXT, "my count" BIGINT, thing INTEGER)'
     assert result == expected
+
+
+def test_read_uint8_from_parquet(tmp_path):
+    import ibis
+
+    con = ibis.connect("duckdb://:memory:")
+
+    t = ibis.memtable({"a": [1, 2, 3, 4]})
+    # There is an incorrect mapping in duckdb-engine from UInteger -> UInt8
+    # In order to get something that reads as a UInt8, we cast to UInt32 (UInteger)
+    t = t.mutate(a=t.a.cast("uint32"))
+
+    parqpath = tmp_path / "uint.parquet"
+
+    t.to_parquet(parqpath)
+
+    # If this doesn't fail, then things are working
+    t2 = con.read_parquet(parqpath)
+
+    assert t2.a.type() == dt.uint8