Repository: TileDB-Inc/TileDB-Py
Pull request #2185 (2025-04-15 17:56:56)

Remove support for HDFS
Follow-up to https://github.com/TileDB-Inc/TileDB-Py/pull/2184. HDFS support was removed upstream in TileDB via https://github.com/TileDB-Inc/TileDB/pull/5496. However, this change hasn't been released yet, so I leave it to @kounelisagis to decide when it makes sense to merge this PR.

kounelisagis: All of the changes should be made conditionally based on the libtiledb version. The same should also have been done for https://github.com/TileDB-Inc/TileDB-Py/pull/2184.
jdblischak: > All of the changes should be made conditionally based on the libtiledb version.

Doesn't TileDB-Py only support a single minor release of TileDB at a time? I don't see the value in adding conditional logic to support TileDB<2.28 if that is not supported.

> The same should also have been done for #2184.

I'm not overly worried about #2184. That test was super basic. Do you want me to send a follow-up PR so restore that test for TileDB<=2.27?
kounelisagis: The test is failing because the token is missing after forking.

Patch:
diff --git a/.github/workflows/ci-tiledb-from-source.yml b/.github/workflows/ci-tiledb-from-source.yml
index b3e990af..3ffd5ee5 100644
--- a/.github/workflows/ci-tiledb-from-source.yml
+++ b/.github/workflows/ci-tiledb-from-source.yml
@@ -30,7 +30,6 @@ jobs:
             -DTILEDB_S3=ON \
             -DTILEDB_AZURE=ON \
             -DTILEDB_GCS=ON \
-            -DTILEDB_HDFS=ON \
             -DTILEDB_SERIALIZATION=ON \
             -DTILEDB_WEBP=ON \
             -DTILEDB_TESTS=OFF \
diff --git a/misc/pypi_linux/Dockerfile.aarch64.manylinux2014 b/misc/pypi_linux/Dockerfile.aarch64.manylinux2014
index febf9e34..16576972 100644
--- a/misc/pypi_linux/Dockerfile.aarch64.manylinux2014
+++ b/misc/pypi_linux/Dockerfile.aarch64.manylinux2014
@@ -46,7 +46,7 @@ RUN cd /home/tiledb/ && \
   cd build && \
   $CMAKE -DTILEDB_S3=ON -DTILEDB_AZURE=ON \
   -DTILEDB_SERIALIZATION=ON \
-  -DTILEDB_CPP_API=ON -DTILEDB_HDFS=ON -DTILEDB_TESTS=OFF \
+  -DTILEDB_CPP_API=ON -DTILEDB_TESTS=OFF \
   -DTILEDB_GCS=ON \
   -DTILEDB_FORCE_ALL_DEPS:BOOL=ON \
   -DTILEDB_LOG_OUTPUT_ON_FAILURE:BOOL=ON \
diff --git a/misc/pypi_linux/Dockerfile2014 b/misc/pypi_linux/Dockerfile2014
index fde748d3..fdb2cbf4 100644
--- a/misc/pypi_linux/Dockerfile2014
+++ b/misc/pypi_linux/Dockerfile2014
@@ -52,7 +52,7 @@ RUN cd /home/tiledb/ && \
   mkdir build && \
   cd build && \
   $CMAKE -DTILEDB_S3=ON -DTILEDB_AZURE=ON -DTILEDB_GCS=ON \
-  -DTILEDB_CPP_API=ON -DTILEDB_HDFS=ON -DTILEDB_TESTS=OFF \
+  -DTILEDB_CPP_API=ON -DTILEDB_TESTS=OFF \
   -DTILEDB_FORCE_ALL_DEPS:BOOL=ON \
   -DTILEDB_LOG_OUTPUT_ON_FAILURE:BOOL=ON \
   -DSANITIZER="OFF;-DCOMPILER_SUPPORTS_AVX2:BOOL=FALSE" \
diff --git a/tiledb/vfs.py b/tiledb/vfs.py
index 7614a1d8..21177ed2 100644
--- a/tiledb/vfs.py
+++ b/tiledb/vfs.py
@@ -122,7 +122,7 @@ class VFS(lt.VFS):
     def supports(self, scheme: str) -> bool:
         """Returns true if the given URI scheme (storage backend) is supported.
 
-        :param str scheme: scheme component of a VFS resource URI (ex. 'file' / 'hdfs' / 's3')
+        :param str scheme: scheme component of a VFS resource URI (ex. 'file' / 's3')
         :rtype: bool
         :return: True if the linked libtiledb version supports the storage backend, False otherwise
         :raises ValueError: VFS storage backend is not supported
@@ -135,11 +135,14 @@ class VFS(lt.VFS):
             "s3": lt.FileSystem.S3,
             "azure": lt.FileSystem.AZURE,
             "gcs": lt.FileSystem.GCS,
-            "hdfs": lt.FileSystem.HDFS,
-        }
+        } | (
+            {"hdfs": lt.FileSystem.HDFS}
+            if tiledb.libtiledb.version() < (2, 28, 0)
+            else {}
+        )
 
         if scheme not in scheme_to_fs_type:
-            raise ValueError(f"Unsupported VFS scheme '{scheme}://'")
+            return False
 
         return self._ctx.is_supported_fs(scheme_to_fs_type[scheme])
 


Test patch:
diff --git a/tiledb/tests/test_libtiledb.py b/tiledb/tests/test_libtiledb.py
index 94123c0a..8e0c3d0b 100644
--- a/tiledb/tests/test_libtiledb.py
+++ b/tiledb/tests/test_libtiledb.py
@@ -3956,6 +3956,11 @@ class TestAsBuilt(DiskTestCase):
         else:
             assert x["gcs"]["enabled"] == False
 
+        if vfs.supports("hdfs"):
+            assert x["hdfs"]["enabled"] == True
+        else:
+            assert x["hdfs"]["enabled"] == False
+
         if vfs.supports("s3"):
             assert x["s3"]["enabled"] == True
         else:
diff --git a/tiledb/tests/test_vfs.py b/tiledb/tests/test_vfs.py
index 5089f4d9..46e826d8 100644
--- a/tiledb/tests/test_vfs.py
+++ b/tiledb/tests/test_vfs.py
@@ -23,8 +23,8 @@ class TestVFS(DiskTestCase):
         self.assertIsInstance(vfs.supports("gcs"), bool)
         self.assertIsInstance(vfs.supports("azure"), bool)
 
-        with self.assertRaises(ValueError):
-            vfs.supports("invalid")
+        # an invalid scheme should return False
+        self.assertFalse(vfs.supports("invalid"))
 
     def test_vfs_config(self):
         opt = {"region": "us-west-x1234"}