CVE-2023-47248 (Python, published 2023-11-09T09:15Z)
Description: value: Deserialization of untrusted data in IPC and Parquet readers in PyArrow versions 0.14.0 to 14.0.0 allows arbitrary code execution. An application is vulnerable if it reads Arrow IPC, Feather or Parquet data from untrusted sources (for example user-supplied input files).

This vulnerability only affects PyArrow, not other Apache Arrow implementations or bindings.

It is recommended that users of PyArrow upgrade to 14.0.1. Similarly, it is recommended that downstream libraries upgrade their dependency requirements to PyArrow 14.0.1 or later. PyPI packages are already available, and we hope that conda-forge packages will be available soon.

If it is not possible to upgrade, we provide a separate package `pyarrow-hotfix` that disables the vulnerability on older PyArrow versions. See  https://pypi.org/project/pyarrow-hotfix/  for instructions.


Severity: nan (CVSS v3 base score: 9.8)
Weakness: CWE-502 \u2014 Deserialization of Untrusted Data
The product deserializes untrusted data without sufficiently verifying that the resulting data will be valid.
Repository: https://github.com/apache/arrow
Files changed:
docs/source/python/extending_types.rst
python/pyarrow/tests/test_cffi.py
python/pyarrow/tests/test_extension_type.py
python/pyarrow/tests/test_pandas.py
python/pyarrow/types.pxi
Commit message: GH-38607: [Python] Disable PyExtensionType autoload (#38608)

### Rationale for this change

PyExtensionType autoload is really a misfeature. It creates PyArrow-specific extension types, though using ExtensionType is almost the same complexity while allowing deserialization from non-PyArrow software.

### What changes are included in this PR?

* Disable PyExtensionType autoloading and deprecate PyExtensionType instantiation.
* Update the docs to emphasize ExtensionType.

### Are these changes tested?

Yes.

### Are there any user-facing changes?

Yes.

* Closes: #38607

Authored-by: Antoine Pitrou <antoine@python.org>
Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>

Vulnerable code:
Pyarrow allows you to define such extension types from Python.

There are currently two ways:

* Subclassing :class:`PyExtensionType`: the (de)serialization is based on pickle.
  This is a good option for an extension type that is only used from Python.
* Subclassing :class:`ExtensionType`: this allows to give a custom
  Python-independent name and serialized metadata, that can potentially be
  recognized by other (non-Python) Arrow implementations such as PySpark.
be represented as ``FixedSizeBinary`` type with 16 bytes.
Using the first approach, we create a ``UuidType`` subclass, and implement the
``__reduce__`` method to ensure the class can be properly pickled::
    class UuidType(pa.PyExtensionType):
            pa.PyExtensionType.__init__(self, pa.binary(16))
        def __reduce__(self):
            return UuidType, ()
    'arrow.py_extension_type'
another Python process. The custom UUID type will be preserved there, as long
as the definition of the class is available (the type can be unpickled).
    UuidType(extension<arrow.py_extension_type>)

We can define the same type using the other option::

    class UuidType(pa.ExtensionType):

        def __init__(self):
            pa.ExtensionType.__init__(self, pa.binary(16), "my_package.uuid")

        def __arrow_ext_serialize__(self):
            # since we don't have a parameterized type, we don't need extra
            # metadata to be deserialized
            return b''

        @classmethod
        def __arrow_ext_deserialize__(self, storage_type, serialized):
            # return an instance of this subclass given the serialized
            # metadata.
            return UuidType()

This is a slightly longer implementation (you need to implement the special
methods ``__arrow_ext_serialize__`` and ``__arrow_ext_deserialize__``), and the
extension type needs to be registered to be received through IPC (using
:func:`register_extension_type`), but it has
now a unique name::

    >>> uuid_type = UuidType()
    >>> uuid_type.extension_name
    'my_package.uuid'

    >>> pa.register_extension_type(uuid_type)
the extension type as a "uuid" type, if it has implemented its own extension
type to receive it.
If the type is not registered in the receiving application, it will fall back
to the storage type.
            pa.ExtensionType.__init__(self, pa.int64(), 'my_package.period')
            # return an instance of this subclass given the serialized
Note that, once created, the data type instance is considered immutable. If,
in the example above, the ``freq`` parameter would change after instantiation,
the reconstruction of the type instance after IPC will be incorrect.
Parameterized extension types are also possible using the pickle-based type
subclassing :class:`PyExtensionType`. The equivalent example for the period
data type from above would look like::

    class PeriodType(pa.PyExtensionType):

        def __init__(self, freq):
            self._freq = freq
            pa.PyExtensionType.__init__(self, pa.int64())

        @property
        def freq(self):
            return self._freq

        def __reduce__(self):
            return PeriodType, (self.freq,)

Also the storage type does not need to be fixed but can be parameterized.

    class Point3DType(pa.PyExtensionType):
            pa.PyExtensionType.__init__(self, pa.list_(pa.float32(), 3))
        def __reduce__(self):
            return Point3DType, ()
extension array class will be preserved (as long as the definitions of the classes above
are available).

The same ``__arrow_ext_class__`` specialization can be used with custom types defined
by subclassing :class:`ExtensionType`.
    class Point3DType(pa.PyExtensionType):
            pa.PyExtensionType.__init__(self, pa.list_(pa.float32(), 3))
        def __reduce__(self):
            return Point3DType, ()
class ParamExtType(pa.PyExtensionType):
        pa.PyExtensionType.__init__(self, pa.binary(width))
    def __reduce__(self):
        return ParamExtType, (self.width,)
def check_export_import_schema(schema_factory):
    assert schema_new == schema_factory()
    check_export_import_schema(make_extension_schema)
    check_export_import_batch(make_extension_batch)
class TinyIntType(pa.PyExtensionType):
        pa.PyExtensionType.__init__(self, pa.int8())
    def __reduce__(self):
        return TinyIntType, ()
class IntegerType(pa.PyExtensionType):
        pa.PyExtensionType.__init__(self, pa.int64())
    def __reduce__(self):
        return IntegerType, ()
class IntegerEmbeddedType(pa.PyExtensionType):
        pa.PyExtensionType.__init__(self, IntegerType())
    def __reduce__(self):
        return IntegerEmbeddedType, ()
class UuidType(pa.PyExtensionType):
        pa.PyExtensionType.__init__(self, pa.binary(16))

    def __reduce__(self):
        return UuidType, ()
class UuidType2(pa.PyExtensionType):
        pa.PyExtensionType.__init__(self, pa.binary(16))
    def __reduce__(self):
        return UuidType2, ()
class LabelType(pa.PyExtensionType):
        pa.PyExtensionType.__init__(self, pa.string())
    def __reduce__(self):
        return LabelType, ()
class ParamExtType(pa.PyExtensionType):
        pa.PyExtensionType.__init__(self, pa.binary(width))
    def __reduce__(self):
        return ParamExtType, (self.width,)
class MyStructType(pa.PyExtensionType):
        pa.PyExtensionType.__init__(self, self.storage_type)
    def __reduce__(self):
        return MyStructType, ()
class MyListType(pa.PyExtensionType):
        pa.PyExtensionType.__init__(self, storage_type)
    def __reduce__(self):
        return MyListType, (self.storage_type,)
class AnnotatedType(pa.PyExtensionType):
        super().__init__(storage_type)
        return AnnotatedType, (self.storage_type, self.annotation)
    assert ty.extension_name == "arrow.py_extension_type"
    expected = "extension<arrow.py_extension_type<IntegerType>>"
        assert ty.extension_name == "arrow.py_extension_type"
    msg = ("Casting from 'extension<arrow.py_extension_type<TinyIntType>>' "
           "'extension<arrow.py_extension_type<IntegerType>>' not permitted. "
def check_example_batch(batch):
    assert isinstance(arr, pa.ExtensionArray)
    assert arr.type.storage_type == pa.binary(3)
    assert arr.storage.to_pylist() == [b"foo", b"bar"]
def test_ipc():
    arr = check_example_batch(batch)
    assert arr.type == ParamExtType(3)
def test_ipc_unknown_type():
    batch = example_batch()
    buf = ipc_write_batch(batch)
    del batch

    orig_type = ParamExtType
    try:
        # Simulate the original Python type being unavailable.
        # Deserialization should not fail but return a placeholder type.
        del globals()['ParamExtType']
        arr = check_example_batch(batch)
        assert isinstance(arr.type, pa.UnknownExtensionType)

        # Can be serialized again
        buf2 = ipc_write_batch(batch)
        del batch, arr

        batch = ipc_read_batch(buf2)
        arr = check_example_batch(batch)
        assert isinstance(arr.type, pa.UnknownExtensionType)
    finally:
        globals()['ParamExtType'] = orig_type

    # Deserialize again with the type restored
    batch = ipc_read_batch(buf2)
    arr = check_example_batch(batch)
    assert arr.type == ParamExtType(3)
    assert table.column('structs').type == mystruct_array.type
    assert table.column('lists').type == mylist_array.type
    assert table == orig_table

    with pytest.raises(pa.ArrowInvalid, match='without all of its fields'):
        pq.ParquetFile(filename).read(columns=['structs.left'])
    assert table.column(0).type == struct_array.type
    assert table == orig_table
    assert table.column(0).type == list_array.type
    assert table == orig_table
    assert table.column(0).type == list_array.type
    assert table == orig_table
    assert table.column(0).type == mylist_array.type
    assert table == orig_table
class DummyExtensionType(pa.PyExtensionType):
        pa.PyExtensionType.__init__(self, pa.int64())
    def __reduce__(self):
        return DummyExtensionType, ()
class MyCustomIntegerType(pa.PyExtensionType):
        pa.PyExtensionType.__init__(self, pa.int64())
    def __reduce__(self):
        return MyCustomIntegerType, ()

    Examples
    --------
    Define a UuidType extension type subclassing PyExtensionType:

    >>> import pyarrow as pa
    >>> class UuidType(pa.PyExtensionType):
    ...     def __init__(self):
    ...         pa.PyExtensionType.__init__(self, pa.binary(16))
    ...     def __reduce__(self):
    ...         return UuidType, ()
    ...

    Create an instance of UuidType extension type:

    >>> uuid_type = UuidType() # doctest: +SKIP
    >>> uuid_type # doctest: +SKIP
    UuidType(FixedSizeBinaryType(fixed_size_binary[16]))

    Inspect the extension type:

    >>> uuid_type.extension_name # doctest: +SKIP
    'arrow.py_extension_type'
    >>> uuid_type.storage_type # doctest: +SKIP
    FixedSizeBinaryType(fixed_size_binary[16])

    Wrap an array as an extension array:

    >>> import uuid
    >>> storage_array = pa.array([uuid.uuid4().bytes for _ in range(4)],
    ...                          pa.binary(16)) # doctest: +SKIP
    >>> uuid_type.wrap_array(storage_array) # doctest: +SKIP
    <pyarrow.lib.ExtensionArray object at ...>
    [
      ...
    ]

    Or do the same with creating an ExtensionArray:

    >>> pa.ExtensionArray.from_storage(uuid_type,
    ...                                storage_array) # doctest: +SKIP
    <pyarrow.lib.ExtensionArray object at ...>
    [
      ...
    ]

Fixed code:
Pyarrow allows you to define such extension types from Python by subclassing
:class:`ExtensionType` and giving the derived class its own extension name
and serialization mechanism. The extension name and serialized metadata
can potentially be recognized by other (non-Python) Arrow implementations
such as PySpark.
be represented as ``FixedSizeBinary`` type with 16 bytes::
    class UuidType(pa.ExtensionType):
            super().__init__(pa.binary(16), "my_package.uuid")

        def __arrow_ext_serialize__(self):
            # Since we don't have a parameterized type, we don't need extra
            # metadata to be deserialized
            return b''
        @classmethod
        def __arrow_ext_deserialize__(cls, storage_type, serialized):
            # Sanity checks, not required but illustrate the method signature.
            assert storage_type == pa.binary(16)
            assert serialized == b''
            # Return an instance of this subclass given the serialized
            # metadata.
            return UuidType()

The special methods ``__arrow_ext_serialize__`` and ``__arrow_ext_deserialize__``
define the serialization of an extension type instance. For non-parametric
types such as the above, the serialization payload can be left empty.
    'my_package.uuid'
another Python process. The receiving process must explicitly register the
extension type for deserialization, otherwise it will fall back to the
storage type::

    >>> pa.register_extension_type(UuidType())
    UuidType(FixedSizeBinaryType(fixed_size_binary[16]))
the extension type as a "my_package.uuid" type, if it has implemented its own
extension type to receive it. If the type is not registered in the receiving
application, it will fall back to the storage type.
            super().__init__(pa.int64(), 'my_package.period')
            # Return an instance of this subclass given the serialized
Note that, once created, the data type instance is considered immutable.
    class Point3DType(pa.ExtensionType):
            super().__init__(pa.list_(pa.float32(), 3), "my_package.Point3DType")
        def __arrow_ext_serialize__(self):
            return b''

        @classmethod
        def __arrow_ext_deserialize__(cls, storage_type, serialized):
            return Point3DType()
extension array class will be preserved (as long as the receiving process registers
the extension type using :func:`register_extension_type` before reading the IPC data).
    from collections import namedtuple

    class Point3DType(pa.ExtensionType):
            super().__init__(pa.list_(pa.float32(), 3), "my_package.Point3DType")
        def __arrow_ext_serialize__(self):
            return b''

        @classmethod
        def __arrow_ext_deserialize__(cls, storage_type, serialized):
            return Point3DType()
import contextlib
@contextlib.contextmanager
def registered_extension_type(ext_type):
    pa.register_extension_type(ext_type)
    try:
        yield
    finally:
        pa.unregister_extension_type(ext_type.extension_name)


class ParamExtType(pa.ExtensionType):
        super().__init__(pa.binary(width),
                         "pyarrow.tests.test_cffi.ParamExtType")
    def __arrow_ext_serialize__(self):
        return str(self.width).encode()

    @classmethod
    def __arrow_ext_deserialize__(cls, storage_type, serialized):
        width = int(serialized.decode())
        return cls(width)
def make_extension_storage_schema():
    # Should be kept in sync with make_extension_schema
    return pa.schema([('ext', ParamExtType(3).storage_type)],
                     metadata={b'key1': b'value1'})


def check_export_import_schema(schema_factory, expected_schema_factory=None):
    if expected_schema_factory is None:
        expected_schema_factory = schema_factory

    assert schema_new == expected_schema_factory()
    # Extension type is unregistered => the storage type is imported
    check_export_import_schema(make_extension_schema,
                               make_extension_storage_schema)

    # Extension type is registered => the extension type is imported
    with registered_extension_type(ParamExtType(1)):
        check_export_import_schema(make_extension_schema)
    with registered_extension_type(ParamExtType(1)):
        check_export_import_batch(make_extension_batch)
import contextlib
@contextlib.contextmanager
def registered_extension_type(ext_type):
    pa.register_extension_type(ext_type)
    try:
        yield
    finally:
        pa.unregister_extension_type(ext_type.extension_name)


@contextlib.contextmanager
def enabled_auto_load():
    pa.PyExtensionType.set_auto_load(True)
    try:
        yield
    finally:
        pa.PyExtensionType.set_auto_load(False)


class TinyIntType(pa.ExtensionType):
        super().__init__(pa.int8(), 'pyarrow.tests.TinyIntType')
    def __arrow_ext_serialize__(self):
        return b''

    @classmethod
    def __arrow_ext_deserialize__(cls, storage_type, serialized):
        assert serialized == b''
        assert storage_type == pa.int8()
        return cls()
class IntegerType(pa.ExtensionType):
        super().__init__(pa.int64(), 'pyarrow.tests.IntegerType')
    def __arrow_ext_serialize__(self):
        return b''

    @classmethod
    def __arrow_ext_deserialize__(cls, storage_type, serialized):
        assert serialized == b''
        assert storage_type == pa.int64()
        return cls()
class IntegerEmbeddedType(pa.ExtensionType):
        super().__init__(IntegerType(), 'pyarrow.tests.IntegerType')
    def __arrow_ext_serialize__(self):
        # XXX pa.BaseExtensionType should expose C++ serialization method
        return self.storage_type.__arrow_ext_serialize__()

    @classmethod
    def __arrow_ext_deserialize__(cls, storage_type, serialized):
        deserialized_storage_type = storage_type.__arrow_ext_deserialize__(
            serialized)
        assert deserialized_storage_type == storage_type
        return cls()
class UuidType(pa.ExtensionType):
        super().__init__(pa.binary(16), 'pyarrow.tests.UuidType')
    def __arrow_ext_serialize__(self):
        return b''
    @classmethod
    def __arrow_ext_deserialize__(cls, storage_type, serialized):
        return cls()


class UuidType2(pa.ExtensionType):
        super().__init__(pa.binary(16), 'pyarrow.tests.UuidType2')
    def __arrow_ext_serialize__(self):
        return b''

    @classmethod
    def __arrow_ext_deserialize__(cls, storage_type, serialized):
        return cls()
class LabelType(pa.ExtensionType):
        super().__init__(pa.string(), 'pyarrow.tests.LabelType')
    def __arrow_ext_serialize__(self):
        return b''

    @classmethod
    def __arrow_ext_deserialize__(cls, storage_type, serialized):
        return cls()
class ParamExtType(pa.ExtensionType):
        super().__init__(pa.binary(width), 'pyarrow.tests.ParamExtType')
    def __arrow_ext_serialize__(self):
        return str(self._width).encode()

    @classmethod
    def __arrow_ext_deserialize__(cls, storage_type, serialized):
        width = int(serialized.decode())
        assert storage_type == pa.binary(width)
        return cls(width)
class MyStructType(pa.ExtensionType):
        super().__init__(self.storage_type, 'pyarrow.tests.MyStructType')
    def __arrow_ext_serialize__(self):
        return b''
    @classmethod
    def __arrow_ext_deserialize__(cls, storage_type, serialized):
        assert serialized == b''
        assert storage_type == cls.storage_type
        return cls()

class MyListType(pa.ExtensionType):
        assert isinstance(storage_type, pa.ListType)
        super().__init__(storage_type, 'pyarrow.tests.MyListType')
    def __arrow_ext_serialize__(self):
        return b''

    @classmethod
    def __arrow_ext_deserialize__(cls, storage_type, serialized):
        assert serialized == b''
        return cls(storage_type)
class AnnotatedType(pa.ExtensionType):
        super().__init__(storage_type, 'pyarrow.tests.AnnotatedType')

    def __arrow_ext_serialize__(self):
        return b''

    @classmethod
    def __arrow_ext_deserialize__(cls, storage_type, serialized):
        assert serialized == b''
        return cls(storage_type)


class LegacyIntType(pa.PyExtensionType):

    def __init__(self):
        pa.PyExtensionType.__init__(self, pa.int8())
        return LegacyIntType, ()
    assert ty.extension_name == "pyarrow.tests.UuidType"
    expected = "extension<pyarrow.tests.IntegerType<IntegerType>>"
        assert ty.extension_name == "pyarrow.tests.UuidType"
    msg = ("Casting from 'extension<.*?<TinyIntType>>' "
           "'extension<.*?<IntegerType>>' not permitted. "
def check_example_batch(batch, *, expect_extension):
    if expect_extension:
        assert isinstance(arr, pa.ExtensionArray)
        assert arr.type.storage_type == pa.binary(3)
        assert arr.storage.to_pylist() == [b"foo", b"bar"]
    else:
        assert arr.type == pa.binary(3)
        assert arr.to_pylist() == [b"foo", b"bar"]
def test_ipc_unregistered():
    batch.validate(full=True)
    check_example_batch(batch, expect_extension=False)
def test_ipc_registered():
    with registered_extension_type(ParamExtType(1)):
        batch = example_batch()
        buf = ipc_write_batch(batch)
        del batch
        batch.validate(full=True)
        arr = check_example_batch(batch, expect_extension=True)
        assert arr.type == ParamExtType(3)
    result.validate(full=True)
    result.validate(full=True)
    # Unregistered
    table.validate(full=True)
    assert table.column('structs').type == struct_array.type
    assert table.column('structs').combine_chunks() == struct_array
    assert table.column('lists').type == list_array.type
    assert table.column('lists').combine_chunks() == list_array

    # Registered
    with registered_extension_type(mystruct_array.type):
        with registered_extension_type(mylist_array.type):
            table = pq.read_table(filename)
            table.validate(full=True)
            assert table.column('structs').type == mystruct_array.type
            assert table.column('lists').type == mylist_array.type
            assert table == orig_table

            # Cannot select a subfield of an extension type with
            # a struct storage type.
            with pytest.raises(pa.ArrowInvalid,
                               match='without all of its fields'):
                pq.ParquetFile(filename).read(columns=['structs.left'])
    table.validate(full=True)
    assert table.column(0).type == pa.struct({'ints': pa.int64(),
                                              'exts': pa.int64()})
    with registered_extension_type(ext_type):
        table = pq.read_table(filename)
        table.validate(full=True)
        assert table.column(0).type == struct_array.type
        assert table == orig_table
    table.validate(full=True)
    assert table.column(0).type == pa.list_(pa.int64())
    with registered_extension_type(ext_type):
        table = pq.read_table(filename)
        table.validate(full=True)
        assert table.column(0).type == list_array.type
        assert table == orig_table
    table.validate(full=True)
    assert table.column(0).type == pa.large_list(pa.int64())
    with registered_extension_type(ext_type):
        table = pq.read_table(filename)
        table.validate(full=True)
        assert table.column(0).type == list_array.type
        assert table == orig_table
    assert table.column(0).type == pa.list_(pa.int64())
    with registered_extension_type(mylist_array.type):
        with registered_extension_type(inner_ext_array.type):
            table = pq.read_table(filename)
            assert table.column(0).type == mylist_array.type
            assert table == orig_table


def test_legacy_int_type():
    with pytest.warns(FutureWarning, match="PyExtensionType is deprecated"):
        ext_ty = LegacyIntType()
    arr = pa.array([1, 2, 3], type=ext_ty.storage_type)
    ext_arr = pa.ExtensionArray.from_storage(ext_ty, arr)
    batch = pa.RecordBatch.from_arrays([ext_arr], names=['ext'])
    buf = ipc_write_batch(batch)

    with pytest.warns(
            RuntimeWarning,
            match="pickle-based deserialization of pyarrow.PyExtensionType "
                  "subclasses is disabled by default"):
        batch = ipc_read_batch(buf)
        assert isinstance(batch.column(0).type, pa.UnknownExtensionType)

    with enabled_auto_load():
        with pytest.warns(FutureWarning, match="PyExtensionType is deprecated"):
            batch = ipc_read_batch(buf)
            assert isinstance(batch.column(0).type, LegacyIntType)
            assert batch.column(0) == ext_arr
class DummyExtensionType(pa.ExtensionType):
        super().__init__(pa.int64(),
                         'pyarrow.tests.test_pandas.DummyExtensionType')
    def __arrow_ext_serialize__(self):
        return b''

    @classmethod
    def __arrow_ext_deserialize__(cls, storage_type, serialized):
        assert serialized == b''
        assert storage_type == pa.int64()
        return cls()
class MyCustomIntegerType(pa.ExtensionType):
        super().__init__(pa.int64(),
                         'pyarrow.tests.test_pandas.MyCustomIntegerType')
    def __arrow_ext_serialize__(self):
        return b''
        The underlying storage type for the extension type.
        A unique name distinguishing this extension type. The name will be
        used when deserializing IPC data.
_py_extension_type_auto_load = False


    .. warning::
       This class is deprecated and its deserialization is disabled by default.
       :class:`ExtensionType` is recommended instead.

        warnings.warn(
            "pyarrow.PyExtensionType is deprecated "
            "and will refuse deserialization by default. "
            "Instead, please derive from pyarrow.ExtensionType and implement "
            "your own serialization mechanism.",
            FutureWarning)
        if not _py_extension_type_auto_load:
            warnings.warn(
                "pickle-based deserialization of pyarrow.PyExtensionType subclasses "
                "is disabled by default; if you only ingest "
                "trusted data files, you may re-enable this using "
                "`pyarrow.PyExtensionType.set_auto_load(True)`.\n"
                "In the future, Python-defined extension subclasses should "
                "derive from pyarrow.ExtensionType (not pyarrow.PyExtensionType) "
                "and implement their own serialization mechanism.\n",
                RuntimeWarning)
            return UnknownExtensionType(storage_type, serialized)
    # XXX Cython marks extension types as immutable, so cannot expose this
    # as a writable class attribute.
    @classmethod
    def set_auto_load(cls, value):
        """
        Enable or disable auto-loading of serialized PyExtensionType instances.

        Parameters
        ----------
        value : bool
            Whether to enable auto-loading.
        """
        global _py_extension_type_auto_load
        assert isinstance(value, bool)
        _py_extension_type_auto_load = value