Repository: data-engineering-collective/plateau (language: python)

Issue:
Support pyarrow=5.*

Support pyarrow=5.*



Discussion / hints:

Fix patch:
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index f32a4010..80a50060 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -31,6 +31,10 @@ jobs:
             os: "ubuntu-latest"
             pyarrow: "4.0.1"
             python: "3.10"
+          - numfocus_nightly: false
+            os: "ubuntu-latest"
+            pyarrow: "5.0.0"
+            python: "3.9"
           - numfocus_nightly: false
             os: "ubuntu-latest"
             pyarrow: "6.0.1"
diff --git a/CHANGES.rst b/CHANGES.rst
index 8e0fd305..d11328f5 100644
--- a/CHANGES.rst
+++ b/CHANGES.rst
@@ -5,7 +5,7 @@ Changelog
 Plateau 4.1.0 (2022-09-??)
 ==========================
 
-* Support for ``pyarrow`` 6, 7, 8, 9
+* Support for ``pyarrow`` 5, 6, 7, 8, 9
 * Support for ``numpy`` 1.23
 * Support for ``dask>=2022.4.2``
 * Support Python 3.8-3.10
diff --git a/plateau/serialization/_parquet.py b/plateau/serialization/_parquet.py
index 5e2e075b..f7c4b6d4 100644
--- a/plateau/serialization/_parquet.py
+++ b/plateau/serialization/_parquet.py
@@ -42,12 +42,12 @@
 EPOCH_ORDINAL = datetime.date(1970, 1, 1).toordinal()
 MAX_NB_RETRIES = 6  # longest retry backoff = BACKOFF_TIME * 2**(MAX_NB_RETRIES - 2)
 BACKOFF_TIME = 0.01  # 10 ms
-PYARROW_LT_5 = version.parse(pa.__version__) < version.parse("5")
+PYARROW_LT_6 = version.parse(pa.__version__) < version.parse("6")
 PYARROW_LT_8 = version.parse(pa.__version__) < version.parse("8")
 
-# Since pyarrow 5, the Parquet version/features can be selected more granular.
+# Since pyarrow 6, the Parquet version/features can be selected more granular.
 # Version 2.0 is equal to 2.4 but 2.4 doesn't trigger deprecation warnings.
-if PYARROW_LT_5:
+if PYARROW_LT_6:
     PARQUET_VERSION = "2.0"
 else:
     PARQUET_VERSION = "2.4"
diff --git a/reference-data/arrow-compat/5.0.0.parquet b/reference-data/arrow-compat/5.0.0.parquet
new file mode 100644
index 00000000..7bee788d
Binary files /dev/null and b/reference-data/arrow-compat/5.0.0.parquet differ


Test patch:
diff --git a/tests/serialization/test_arrow_compat.py b/tests/serialization/test_arrow_compat.py
index 174f374a..fa0b6e30 100644
--- a/tests/serialization/test_arrow_compat.py
+++ b/tests/serialization/test_arrow_compat.py
@@ -23,6 +23,7 @@
     "2.0.0",
     "3.0.0",
     "4.0.1",
+    "5.0.0",
     "6.0.1",
     "7.0.0",
     "8.0.1",