Repository: fsspec/adlfs (language: python)

Issue:
Support virtual directory stubs with uppercase "Hdi_isfolder" metadata
## Background
adlfs checks blobs for, among other things, the `hdi_isfolder` metadata key, to determine if the blob is a stub for a virtual directory, i.e. there are other files "nested under it."

https://github.com/fsspec/adlfs/blob/092685f102c5cd215550d10e8347e5bce0e2b93d/adlfs/spec.py#L857-L859

## Problem

I have encountered some bugs in upstream blob writers where they create this stub blob with a capital-H `Hdi_isfolder` metadata key.  Examples include:
- https://github.com/Azure/azure-sdk-for-go/issues/17850
- I also see it happen when writing to Azure Blob Storage locations that are mounted as virtual volumes from inside containerized Azure Batch tasks (i.e. inside a Docker container), on a Batch Pool.  I saw the capitalized metadata keys begin appearing between August 28 and August 31, 2023, and I have an internal Azure support ticket open for this.

Anyway, when the metadata key is `Hdi_isfolder` (capital-H), adfs does not detect that the blob is a directory stub, but rather thinks it's a file containing data.  Then it errors:

```py
import adlfs
import pyarrow
from pyarrow.dataset import dataset

abfs = adlfs.AzureBlobFileSystem(account_name="example", account_key="XXXX")
pqdata = dataset("path/to/stub", filesystem=abfs)
```
```pytb
ArrowInvalid: Error creating dataset. Could not read schema from 'path/to/stub'.
  Is this a 'parquet' file?:
    Could not open Parquet input source 'path/to/stub':
      Parquet file size is 0 bytes

  File "pip/pyarrow==13.0.0/pyarrow/dataset.py", line 773, in dataset
    return _filesystem_dataset(source, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "pip/pyarrow==13.0.0/pyarrow/dataset.py", line 466, in _filesystem_dataset
    return factory.finish(schema)
           ^^^^^^^^^^^^^^^^^^^^^^

  File "pyarrow/_dataset.pyx", line 2941, in pyarrow._dataset.DatasetFactory.finish
  File "pyarrow/error.pxi", line 144, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 100, in pyarrow.lib.check_status
```

## Request

I am of the opinion that this is a bug in the writers of these stub blobs, and should be fixed there.  However the damage has already been done and these stub blobs with capitalized metadata key already exist in the wild.  So I am wondering if we can have adlfs become case-insensitive in its metadata check.

## Beware

This bug is especially difficult to diagnose. One reason is that Azure Storage Explorer automatically shows all metadata names in lowercase, even if they are not actually in lowercase. I've found that I have to use the 'Containers' app in Azure Portal to view correct metadata names on stub blobs. 



Discussion / hints:

Fix patch:
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 2990da37..9b602f64 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,5 +1,10 @@
 **Change Log**
 
+Unreleased
+----------
+
+- Handle mixed casing for `hdi_isfolder` metadata when determining whether a blob should be treated as a folder.
+
 2023.10.0
 ---------
 - Added support for timeout/connection_timeout/read_timeout
diff --git a/adlfs/spec.py b/adlfs/spec.py
index 026897ba..5297325e 100644
--- a/adlfs/spec.py
+++ b/adlfs/spec.py
@@ -858,7 +858,11 @@ async def _details(
                     data["size"] = None
                 elif data["metadata"].get("is_directory") == "false":
                     data["type"] = "file"
-                elif data["metadata"].get("hdi_isfolder") == "true":
+                elif (
+                    # In some cases Hdi_isfolder is capitalized, see #440
+                    data["metadata"].get("hdi_isfolder") == "true"
+                    or data["metadata"].get("Hdi_isfolder") == "true"
+                ):
                     data["type"] = "directory"
                     data["size"] = None
             if return_glob:


Test patch:
diff --git a/adlfs/tests/test_spec.py b/adlfs/tests/test_spec.py
index 2a451043..27277a40 100644
--- a/adlfs/tests/test_spec.py
+++ b/adlfs/tests/test_spec.py
@@ -1,6 +1,7 @@
 import datetime
 import tempfile
 
+import azure.storage.blob
 import dask.dataframe as dd
 import numpy as np
 import pandas as pd
@@ -1860,3 +1861,16 @@ async def test_put_file_timeout(storage, mocker, tmp_path):
         connection_timeout=12,
         read_timeout=13,
     )
+
+
+@pytest.mark.parametrize("key", ["hdi_isfolder", "Hdi_isfolder"])
+def test_hdi_isfolder_case(storage: azure.storage.blob.BlobServiceClient, key: str):
+    cc = storage.get_container_client("data")
+    cc.upload_blob(b"folder", b"", metadata={key: "true"}, overwrite=True)
+
+    fs = AzureBlobFileSystem(
+        account_name=storage.account_name, connection_string=CONN_STR
+    )
+
+    result = fs.info("data/folder")
+    assert result["type"] == "directory"