Repository: vaexio/vaex

Issue:
[FEATURE-REQUEST] Read files from ADLS Gen 2 (Cloud support for Azure Data Lake)
Hello,

in addition to support for S3 and GCS, it'd be awesome if it'd also be possible to read files directly from ADLS Gen 2, since some of us are restricted in the choice of cloud platforms we can use.

Thanks,


Discussion:
Hi,

thanks for opening the issue, Using fsspec, this can already work by installing:
https://github.com/dask/adlfs
And looking at this example:
https://vaex.io/blog/a-hybrid-apache-arrow-numpy-dataframe-with-vaex-version-4#cloud-storage
Let us know if this works for you.
We could possibly add support for this by recognising `"adl://"`, but lets first see if you can get this working.

cheers,

Maarten
thanks so much @maartenbreddels , what you are pointing to makes sense - will give it a shot, and report back

recognizing `adl://` natively would be awesome :)
Hi,

I'm also trying to read from Azure storage. I tried with adlfs but I'm failing to make it work. May be I'm doing something wrong. Here's a code snippet:

```
...
fs = adlfs.AzureBlobFileSystem(...)
df = vaex.open(f'abfs://{container}/{name}', fs=fs)
```

This raises an error from pyarrow `Unrecognized filesystem: <class 'pyarrow.fs.FSSpecHandler'>. `filesystem` argument must be a FileSystem instance or a valid file system URI'`.
After a deeper look in the code, in [vaex/file/__init__.py](https://github.com/vaexio/vaex/blob/c2348c34bc9fdd6e0ddd503e1a0124176d20743d/packages/vaex-core/vaex/file/__init__.py#L192), the fs object provided is encapsulated in FSSpecHandler :
```
            if isinstance(fs, fsspec.AbstractFileSystem):
                fs = pa.fs.FSSpecHandler(fs)
```

which is then forwarded up to pyarrow/fs.py and then fails on method [_ensure_filesystem](https://github.com/apache/arrow/blob/27d21d4d3b18280805ed7118c089d930fd018f11/python/pyarrow/fs.py#L106) which expect an `fsspec.AbstractFileSystem`but get an `FSSpecHandler`

Am I doing something wrong?


Thanks for this report, I can reproduce this error, and I'll fix this soon. A workaround for now should be:
```python
import adlfs
import vaex
import pyarrow as pa
import pyarrow.fs
fs = adlfs.AzureBlobFileSystem(...)
fs = pa.fs.FSSpecHandler(fs)
fs = pa.fs.PyFileSystem(fs)
container = 'foo'
name = 'bar'
df = vaex.open(f'{container}/{name}', fs=fs)
```

Resolving patch:
diff --git a/packages/vaex-core/vaex/file/__init__.py b/packages/vaex-core/vaex/file/__init__.py
--- a/packages/vaex-core/vaex/file/__init__.py
+++ b/packages/vaex-core/vaex/file/__init__.py
@@ -190,6 +190,8 @@ def parse(path, fs_options={}, fs=None, for_arrow=False):
             import fsspec
             if isinstance(fs, fsspec.AbstractFileSystem):
                 fs = pa.fs.FSSpecHandler(fs)
+            if for_arrow:
+                fs = pyarrow.fs.PyFileSystem(fs)
         return fs, path
     if isinstance(path, (list, tuple)):
         scheme, _ = split_scheme(path[0])


Test patch:
diff --git a/tests/cloud_dataset_test.py b/tests/cloud_dataset_test.py
--- a/tests/cloud_dataset_test.py
+++ b/tests/cloud_dataset_test.py
@@ -1,5 +1,6 @@
 import vaex
 import pytest
+import fsspec
 
 
 fs_options = {'anonymous': 'true'}
@@ -52,3 +53,9 @@ def test_cloud_concat(base_url):
     df = df1.concat(df2)
 
     assert df.x.tolist() == [1, None, 1, 2]
+
+
+def test_fsspec_arrow():
+    fs = fsspec.filesystem('s3', anon=True)
+    df = vaex.open('vaex/testing/xys-masked.parquet', fs=fs)
+    assert df.x.tolist() == [1, None]