Repository: duckdb/duckdb (language: python)

Issue:
Arrow datatype Extension("arrow.json", Utf8, Some("")) not supported by Polars exception
### What happens?

with duckdb 1.1.0 I get the following error when selecting over a json file. multiple different json files. also some json files work.

`pyo3_runtime.PanicException: Arrow datatype Extension("arrow.json", Utf8, Some("")) not supported by Polars. You probably need to activate that data-type feature.`

with duckdb 1.0.0 everything works fine

### To Reproduce

```
import duckdb
duckdb.sql("SELECT unnest(entry, recursive := true) FROM read_json_auto('test.json', filename=true)").pl()
```

### OS:

Ubuntu 22.04

### DuckDB Version:

1.1.0

### DuckDB Client:

Python

### Hardware:

_No response_

### Full Name:

Daniel Gut

### Affiliation:

Aveniq

### What is the latest build you tested with? If possible, we recommend testing with the latest nightly build.

I have tested with a stable release

### Did you include all relevant data sets for reproducing the issue?

No - I cannot share the data sets because they are confidential

### Did you include all code required to reproduce the issue?

- [ ] Yes, I have

### Did you include all relevant configuration (e.g., CPU architecture, Python version, Linux distribution) to reproduce the issue?

- [X] Yes, I have


Discussion / hints:
Hi @OneCyrus, thanks for reporting this issue. Is there a chance you can reduce the confidential data set to a non-confidential reproducer? That would go help a lot in getting this resolved.
@szarnyasg looks like the issue is with the null property.

i reduced the dataset to the following:

```
{
    "entry": [
        {
            "content": {
                "ManagedSystem": {
                    "test": null
                }
            }
        }
    ]
}
```

```
from duckdb import connect

db = connect(":memory:")
test = db.sql(
    "SELECT entry FROM read_json_auto('test.json', filename=true)"
).pl()
```
Thanks, what's the Polars version?
I can reproduce it 👍 

```
➜  duckdb git:(main) ✗ python3 tmp/polars_json2.py
thread '<unnamed>' panicked at crates/polars-core/src/datatypes/field.rs:189:19:
Arrow datatype Extension("arrow.json", Utf8, Some("")) not supported by Polars. You probably need to activate that data-type feature.
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Traceback (most recent call last):
  File "/Users/thijs/DuckDBLabs/duckdb/tmp/polars_json2.py", line 6, in <module>
    ).pl()
      ^^^^
  File "/Users/thijs/.pyenv/versions/3.12.1/lib/python3.12/site-packages/polars/dataframe/frame.py", line 394, in __init__
    self._df = arrow_to_pydf(
               ^^^^^^^^^^^^^^
  File "/Users/thijs/.pyenv/versions/3.12.1/lib/python3.12/site-packages/polars/_utils/construction/dataframe.py", line 1209, in arrow_to_pydf
    pydf = PyDataFrame.from_arrow_record_batches(tbl.to_batches())
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
pyo3_runtime.PanicException: Arrow datatype Extension("arrow.json", Utf8, Some("")) not supported by Polars. You probably need to activate that data-type feature.
```

```
>>> import polars
>>> polars.__version__
'1.5.0'
```
first had the same issue as above with arrow.uuid instead of arrow.json. but can't reproduce it now. but now i have a consistent issue which shows the following polars type exception.

```
Arrow datatype Map(Field { name: "entries", dtype: Struct([Field { name: "key", dtype: Utf8, is_nullable: false, metadata: {} }, Field { name: "value", dtype: Utf8, is_nullable: true, metadata: {} }]), is_nullable: false, metadata: {} }, false) not supported by Polars. You probably need to activate that data-type feature.
```
@Tishj ok, got a reproduction with an empty object

```
[
    {
        "ipspace": {
            "_links": {}
        }
    }
]
```

```
thread 'polars-0' panicked at crates/polars-core/src/datatypes/field.rs:188:19:
Arrow datatype Map(Field { name: "entries", dtype: Struct([Field { name: "key", dtype: Utf8, is_nullable: false, metadata: {} }, Field { name: "value", dtype: Utf8, is_nullable: true, metadata: {} }]), is_nullable: false, metadata: {} }, false) not supported by Polars. You probably need to activate that data-type feature.
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Traceback (most recent call last):
  File "test.py", line 6, in <module>
    ).pl()
      ^^^^
  File "/workspaces/automation-data/.venv/lib/python3.11/site-packages/polars/dataframe/frame.py", line 395, in __init__
    self._df = arrow_to_pydf(
               ^^^^^^^^^^^^^^
  File "/workspaces/automation-data/.venv/lib/python3.11/site-packages/polars/_utils/construction/dataframe.py", line 1209, in arrow_to_pydf
    pydf = PyDataFrame.from_arrow_record_batches(tbl.to_batches())
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
pyo3_runtime.PanicException: Arrow datatype Map(Field { name: "entries", dtype: Struct([Field { name: "key", dtype: Utf8, is_nullable: false, metadata: {} }, Field { name: "value", dtype: Utf8, is_nullable: true, metadata: {} }]), is_nullable: false, metadata: {} }, false) not supported by Polars. You probably need to activate that data-type feature.
```

Fix patch:
diff --git a/src/common/arrow/arrow_converter.cpp b/src/common/arrow/arrow_converter.cpp
index 3524dc89888f..851d45b5beba 100644
--- a/src/common/arrow/arrow_converter.cpp
+++ b/src/common/arrow/arrow_converter.cpp
@@ -166,7 +166,7 @@ void SetArrowFormat(DuckDBArrowSchemaHolder &root_holder, ArrowSchema &child, co
 		break;
 	}
 	case LogicalTypeId::VARCHAR:
-		if (type.IsJSONType()) {
+		if (type.IsJSONType() && options.arrow_lossless_conversion) {
 			auto schema_metadata = ArrowSchemaMetadata::MetadataFromName("arrow.json");
 			root_holder.metadata_info.emplace_back(schema_metadata.SerializeMetadata());
 			child.metadata = root_holder.metadata_info.back().get();


Test patch:
diff --git a/tools/pythonpkg/tests/fast/arrow/test_canonical_extensions.py b/tools/pythonpkg/tests/fast/arrow/test_canonical_extensions.py
index 6fa4806029c4..a357df023214 100644
--- a/tools/pythonpkg/tests/fast/arrow/test_canonical_extensions.py
+++ b/tools/pythonpkg/tests/fast/arrow/test_canonical_extensions.py
@@ -10,14 +10,94 @@
 from arrow_canonical_extensions import UuidType, JSONType, UHugeIntType, HugeIntType
 
 
+class UuidTypeWrong(pa.ExtensionType):
+    def __init__(self):
+        pa.ExtensionType.__init__(self, pa.binary(4), "arrow.uuid")
+
+    def __arrow_ext_serialize__(self):
+        # since we don't have a parameterized type, we don't need extra
+        # metadata to be deserialized
+        return b''
+
+    @classmethod
+    def __arrow_ext_deserialize__(cls, storage_type, serialized):
+        # return an instance of this subclass given the serialized
+        # metadata.
+        return UuidTypeWrong()
+
+
+class JSONTypeWrong(pa.ExtensionType):
+    def __init__(self):
+        pa.ExtensionType.__init__(self, pa.int32(), "arrow.json")
+
+    def __arrow_ext_serialize__(self):
+        # since we don't have a parameterized type, we don't need extra
+        # metadata to be deserialized
+        return b''
+
+    @classmethod
+    def __arrow_ext_deserialize__(cls, storage_type, serialized):
+        # return an instance of this subclass given the serialized
+        # metadata.
+        return JSONTypeWrong()
+
+
+"""
+    These fixtures make sure that the extension_type is registered at the start of the function,
+    and unregistered at the end.
+    
+    No matter if an error occurred or the function ended early for whatever reason
+"""
+
+
+@pytest.fixture(scope='function')
+def arrow_duckdb_hugeint():
+    pa.register_extension_type(HugeIntType())
+    yield
+    pa.unregister_extension_type("duckdb.hugeint")
+
+
+@pytest.fixture(scope='function')
+def arrow_duckdb_uhugeint():
+    pa.register_extension_type(UHugeIntType())
+    yield
+    pa.unregister_extension_type("duckdb.uhugeint")
+
+
+@pytest.fixture(scope='function')
+def arrow_json():
+    pa.register_extension_type(JSONType())
+    yield
+    pa.unregister_extension_type("arrow.json")
+
+
+@pytest.fixture(scope='function')
+def arrow_json_wrong():
+    pa.register_extension_type(JSONTypeWrong())
+    yield
+    pa.unregister_extension_type("arrow.json")
+
+
+@pytest.fixture(scope='function')
+def arrow_uuid():
+    pa.register_extension_type(UuidType())
+    yield
+    pa.unregister_extension_type("arrow.uuid")
+
+
+@pytest.fixture(scope='function')
+def arrow_uuid_wrong():
+    pa.register_extension_type(UuidTypeWrong())
+    yield
+    pa.unregister_extension_type("arrow.uuid")
+
+
 class TestCanonicalExtensionTypes(object):
 
-    def test_uuid(self):
+    def test_uuid(self, arrow_uuid):
         duckdb_cursor = duckdb.connect()
         duckdb_cursor.execute("SET arrow_lossless_conversion = true")
 
-        pa.register_extension_type(UuidType())
-
         storage_array = pa.array([uuid.uuid4().bytes for _ in range(4)], pa.binary(16))
         uuid_type = UuidType()
         storage_array = uuid_type.wrap_array(storage_array)
@@ -28,14 +108,10 @@ def test_uuid(self):
 
         assert duck_arrow.equals(arrow_table)
 
-        pa.unregister_extension_type("arrow.uuid")
-
-    def test_uuid_from_duck(self):
+    def test_uuid_from_duck(self, arrow_uuid):
         duckdb_cursor = duckdb.connect()
         duckdb_cursor.execute("SET arrow_lossless_conversion = true")
 
-        pa.register_extension_type(UuidType())
-
         arrow_table = duckdb_cursor.execute("select uuid from test_all_types()").fetch_arrow_table()
 
         assert arrow_table.to_pylist() == [
@@ -59,29 +135,10 @@ def test_uuid_from_duck(self):
         ]
         assert duckdb_cursor.execute("FROM arrow_table").fetchall() == [(UUID('00000000-0000-0000-0000-000000000100'),)]
 
-        pa.unregister_extension_type("arrow.uuid")
-
-    def test_uuid_exception(self):
-        class UuidTypeWrong(pa.ExtensionType):
-            def __init__(self):
-                pa.ExtensionType.__init__(self, pa.binary(4), "arrow.uuid")
-
-            def __arrow_ext_serialize__(self):
-                # since we don't have a parameterized type, we don't need extra
-                # metadata to be deserialized
-                return b''
-
-            @classmethod
-            def __arrow_ext_deserialize__(self, storage_type, serialized):
-                # return an instance of this subclass given the serialized
-                # metadata.
-                return UuidTypeWrong()
-
+    def test_uuid_exception(self, arrow_uuid_wrong):
         duckdb_cursor = duckdb.connect()
         duckdb_cursor.execute("SET arrow_lossless_conversion = true")
 
-        pa.register_extension_type(UuidTypeWrong())
-
         storage_array = pa.array(['aaaa'], pa.binary(4))
         uuid_type = UuidTypeWrong()
         storage_array = uuid_type.wrap_array(storage_array)
@@ -91,11 +148,7 @@ def __arrow_ext_deserialize__(self, storage_type, serialized):
         with pytest.raises(duckdb.InvalidInputException, match="arrow.uuid must be a fixed-size binary of 16 bytes"):
             duck_arrow = duckdb_cursor.execute('FROM arrow_table').arrow()
 
-        pa.unregister_extension_type("arrow.uuid")
-
-    def test_json(self, duckdb_cursor):
-        pa.register_extension_type(JSONType())
-
+    def test_json(self, duckdb_cursor, arrow_json):
         data = {"name": "Pedro", "age": 28, "car": "VW Fox"}
 
         # Convert dictionary to JSON string
@@ -107,30 +160,12 @@ def test_json(self, duckdb_cursor):
 
         arrow_table = pa.Table.from_arrays([storage_array], names=['json_col'])
 
+        duckdb_cursor.execute("SET arrow_lossless_conversion = true")
         duck_arrow = duckdb_cursor.execute('FROM arrow_table').arrow()
 
         assert duck_arrow.equals(arrow_table)
 
-        pa.unregister_extension_type("arrow.json")
-
-    def test_json_throw(self, duckdb_cursor):
-        class JSONTypeWrong(pa.ExtensionType):
-            def __init__(self):
-                pa.ExtensionType.__init__(self, pa.int32(), "arrow.json")
-
-            def __arrow_ext_serialize__(self):
-                # since we don't have a parameterized type, we don't need extra
-                # metadata to be deserialized
-                return b''
-
-            @classmethod
-            def __arrow_ext_deserialize__(self, storage_type, serialized):
-                # return an instance of this subclass given the serialized
-                # metadata.
-                return JSONTypeWrong()
-
-        pa.register_extension_type(JSONTypeWrong())
-
+    def test_json_throw(self, duckdb_cursor, arrow_json_wrong):
         storage_array = pa.array([32], pa.int32())
         json_type = JSONTypeWrong()
         storage_array = json_type.wrap_array(storage_array)
@@ -139,7 +174,6 @@ def __arrow_ext_deserialize__(self, storage_type, serialized):
 
         with pytest.raises(duckdb.InvalidInputException, match="arrow.json must be of a varchar format "):
             duck_arrow = duckdb_cursor.execute('FROM arrow_table').arrow()
-        pa.unregister_extension_type("arrow.json")
 
     def test_uuid_no_def(self):
         duckdb_cursor = duckdb.connect()
@@ -181,9 +215,7 @@ def test_uuid_no_def_stream(self):
             (None,),
         ]
 
-    def test_uuid_udf_registered(self, duckdb_cursor):
-        pa.register_extension_type(UuidType())
-
+    def test_uuid_udf_registered(self, arrow_uuid):
         def test_function(x):
             print(x.type.__class__)
             return x
@@ -194,19 +226,17 @@ def test_function(x):
         rel = con.sql("select ? as x", params=[uuid.UUID('ffffffff-ffff-ffff-ffff-ffffffffffff')])
         rel.project("test(x) from t").fetchall()
 
-        pa.unregister_extension_type("arrow.uuid")
-
     def test_uuid_udf_unregistered(self):
-        duckdb_cursor = duckdb.connect()
-        duckdb_cursor.execute("SET arrow_lossless_conversion = true")
+        con = duckdb.connect()
+        con.execute("SET arrow_lossless_conversion = true")
 
         def test_function(x):
             print(x.type.__class__)
             return x
 
-        duckdb_cursor.create_function('test', test_function, ['UUID'], 'UUID', type='arrow')
+        con.create_function('test', test_function, ['UUID'], 'UUID', type='arrow')
 
-        rel = duckdb_cursor.sql("select ? as x", params=[uuid.UUID('ffffffff-ffff-ffff-ffff-ffffffffffff')])
+        rel = con.sql("select ? as x", params=[uuid.UUID('ffffffff-ffff-ffff-ffff-ffffffffffff')])
         with pytest.raises(duckdb.Error, match="It seems that you are using the UUID arrow canonical extension"):
             rel.project("test(x) from t").fetchall()
 
@@ -219,11 +249,9 @@ def __arrow_ext_serialize__(self):
                 return b''
 
             @classmethod
-            def __arrow_ext_deserialize__(self, storage_type, serialized):
+            def __arrow_ext_deserialize__(cls, storage_type, serialized):
                 return UuidTypeWrong()
 
-                pa.register_extension_type(UuidType())
-
         storage_array = pa.array(['pedro'], pa.binary(5))
         my_type = MyType()
         storage_array = my_type.wrap_array(storage_array)
@@ -233,12 +261,10 @@ def __arrow_ext_deserialize__(self, storage_type, serialized):
         with pytest.raises(duckdb.NotImplementedException, match=" Arrow Type with extension name: pedro.binary"):
             duck_arrow = duckdb_cursor.execute('FROM arrow_table').arrow()
 
-    def test_hugeint(self):
-        duckdb_cursor = duckdb.connect()
-
-        duckdb_cursor.execute("SET arrow_lossless_conversion = true")
+    def test_hugeint(self, arrow_duckdb_hugeint):
+        con = duckdb.connect()
 
-        pa.register_extension_type(HugeIntType())
+        con.execute("SET arrow_lossless_conversion = true")
 
         storage_array = pa.array([b'\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff'], pa.binary(16))
         hugeint_type = HugeIntType()
@@ -246,19 +272,15 @@ def test_hugeint(self):
 
         arrow_table = pa.Table.from_arrays([storage_array], names=['numbers'])
 
-        assert duckdb_cursor.execute('FROM arrow_table').fetchall() == [(-1,)]
-
-        assert duckdb_cursor.execute('FROM arrow_table').arrow().equals(arrow_table)
+        assert con.execute('FROM arrow_table').fetchall() == [(-1,)]
 
-        duckdb_cursor.execute("SET arrow_lossless_conversion = false")
+        assert con.execute('FROM arrow_table').arrow().equals(arrow_table)
 
-        assert not duckdb_cursor.execute('FROM arrow_table').arrow().equals(arrow_table)
+        con.execute("SET arrow_lossless_conversion = false")
 
-        pa.unregister_extension_type("duckdb.hugeint")
-
-    def test_uhugeint(self, duckdb_cursor):
-        pa.register_extension_type(UHugeIntType())
+        assert not con.execute('FROM arrow_table').arrow().equals(arrow_table)
 
+    def test_uhugeint(self, duckdb_cursor, arrow_duckdb_uhugeint):
         storage_array = pa.array([b'\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff\xff'], pa.binary(16))
         uhugeint_type = UHugeIntType()
         storage_array = uhugeint_type.wrap_array(storage_array)
@@ -267,25 +289,23 @@ def test_uhugeint(self, duckdb_cursor):
 
         assert duckdb_cursor.execute('FROM arrow_table').fetchall() == [(340282366920938463463374607431768211455,)]
 
-        pa.unregister_extension_type("duckdb.uhugeint")
-
     def test_bit(self):
-        duckdb_cursor = duckdb.connect()
+        con = duckdb.connect()
 
-        res_blob = duckdb_cursor.execute("SELECT '0101011'::BIT str FROM range(5) tbl(i)").arrow()
+        res_blob = con.execute("SELECT '0101011'::BIT str FROM range(5) tbl(i)").arrow()
 
-        duckdb_cursor.execute("SET arrow_lossless_conversion = true")
+        con.execute("SET arrow_lossless_conversion = true")
 
-        res_bit = duckdb_cursor.execute("SELECT '0101011'::BIT str FROM range(5) tbl(i)").arrow()
+        res_bit = con.execute("SELECT '0101011'::BIT str FROM range(5) tbl(i)").arrow()
 
-        assert duckdb_cursor.execute("FROM res_blob").fetchall() == [
+        assert con.execute("FROM res_blob").fetchall() == [
             (b'\x01\xab',),
             (b'\x01\xab',),
             (b'\x01\xab',),
             (b'\x01\xab',),
             (b'\x01\xab',),
         ]
-        assert duckdb_cursor.execute("FROM res_bit").fetchall() == [
+        assert con.execute("FROM res_bit").fetchall() == [
             ('0101011',),
             ('0101011',),
             ('0101011',),
@@ -294,15 +314,15 @@ def test_bit(self):
         ]
 
     def test_timetz(self):
-        duckdb_cursor = duckdb.connect()
+        con = duckdb.connect()
 
-        res_time = duckdb_cursor.execute("SELECT '02:30:00+04'::TIMETZ str FROM range(1) tbl(i)").arrow()
+        res_time = con.execute("SELECT '02:30:00+04'::TIMETZ str FROM range(1) tbl(i)").arrow()
 
-        duckdb_cursor.execute("SET arrow_lossless_conversion = true")
+        con.execute("SET arrow_lossless_conversion = true")
 
-        res_tz = duckdb_cursor.execute("SELECT '02:30:00+04'::TIMETZ str FROM range(1) tbl(i)").arrow()
+        res_tz = con.execute("SELECT '02:30:00+04'::TIMETZ str FROM range(1) tbl(i)").arrow()
 
-        assert duckdb_cursor.execute("FROM res_time").fetchall() == [(datetime.time(2, 30),)]
-        assert duckdb_cursor.execute("FROM res_tz").fetchall() == [
+        assert con.execute("FROM res_time").fetchall() == [(datetime.time(2, 30),)]
+        assert con.execute("FROM res_tz").fetchall() == [
             (datetime.time(2, 30, tzinfo=datetime.timezone(datetime.timedelta(seconds=14400))),)
         ]
diff --git a/tools/pythonpkg/tests/fast/arrow/test_polars.py b/tools/pythonpkg/tests/fast/arrow/test_polars.py
index 78c74564e091..7a2ae5008f0a 100644
--- a/tools/pythonpkg/tests/fast/arrow/test_polars.py
+++ b/tools/pythonpkg/tests/fast/arrow/test_polars.py
@@ -57,3 +57,19 @@ def test_empty_polars_dataframe(self, duckdb_cursor):
             duckdb.InvalidInputException, match='Provided table/dataframe must have at least one column'
         ):
             duckdb_cursor.sql("from polars_empty_df")
+
+    def test_polars_from_json(self, duckdb_cursor):
+        from io import StringIO
+
+        duckdb_cursor.sql("set arrow_lossless_conversion=false")
+        string = StringIO("""{"entry":[{"content":{"ManagedSystem":{"test":null}}}]}""")
+        res = duckdb_cursor.read_json(string).pl()
+        assert str(res['entry'][0][0]) == "{'content': {'ManagedSystem': {'test': None}}}"
+
+    def test_polars_from_json_error(self, duckdb_cursor):
+        from io import StringIO
+
+        duckdb_cursor.sql("set arrow_lossless_conversion=true")
+        string = StringIO("""{"entry":[{"content":{"ManagedSystem":{"test":null}}}]}""")
+        with pytest.raises(pl.exceptions.PanicException):
+            res = duckdb_cursor.read_json(string).pl()