fsspec: support fsspec>=2023.12.0 glob changes
Opened by pmrowla (closed)

- adds support for the `fs.glob` changes introduced in `fsspec==2023.12.0` and unpins the current upper bound

Should close #6644
Should close #6645

The `test_data_files` glob/pattern tests pass for me in:
- `fsspec==2023.10.0` (the pinned max version in datasets `main`)
- `fsspec==2023.12.0` (#6644)
- `fsspec==2024.2.0` (#6645)

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6687). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Looking into the CI failure, this PR is incompatible with `huggingface-hub>=0.20.0`. It looks like there were several changes made to HfFileSystem in 0.20.0, @lhoestq any ideas on what the issue might be in particular?

a bisect indicates that it's related to https://github.com/huggingface/huggingface_hub/pull/1815

It looks like huggingface-hub's `HfFileSystem.glob` is broken for exact string matches (that don't contain glob wildcards) when combining `huggingface-hub>=0.20.0` and `fsspec>=2023.12.0`.

I did a quick test with huggingface-hub `main`, and adding this test case to `tests/test_hf_filesystem::HfFileSystemTests::test_glob` (https://github.com/huggingface/huggingface_hub/blob/main/tests/test_hf_file_system.py) passes with `fsspec==2023.10.0` and fails with `fsspec==2023.12.0`
```python
        self.assertEqual(
            sorted(self.hffs.glob(self.hf_path + "/.gitattributes")),
            sorted([self.hf_path + "/.gitattributes"]),
        )

```

the `hffs.glob()` call with a pattern that does not contain any wildcards returns an empty list:
```
E       AssertionError: Lists differ: [] != ['datasets/__DUMMY_TRANSFORMERS_USER__/rep[35 chars]tes']
E
E       Second list contains 1 additional elements.
E       First extra element 0:
E       'datasets/__DUMMY_TRANSFORMERS_USER__/repo-7d0ae9-17091013467064/.gitattributes'
E
E       - []
E       + ['datasets/__DUMMY_TRANSFORMERS_USER__/repo-7d0ae9-17091013467064/.gitattributes']
```
(and with the compatible/passing older fsspec versions the glob call returns the single exact file match as expected)

So it looks like the CI failure here isn't directly related to this PR. The failing patterns that don't contain any `*` wildcards are generated by `datasets` with or without this PR, but now that this PR installs the incompatible fsspec version, the underlying `HfFileSystem.glob()` call ends up failing.

I just opened https://github.com/huggingface/huggingface_hub/pull/2056 to fix this.

Do you mind if I continue this PR to run the CI against `huggingface_hub@main` until the fix is released ?

EDIT: the fix has been released in `huggingface_hub` 0.21.2 - I removed my commits that were using `huggingface_hub@main`

I just added two additional patterns to cover cases like `test-data/xxx.csv` and `data-test/xxx.csv`

<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005461 / 0.011353 (-0.005892) | 0.003861 / 0.011008 (-0.007148) | 0.063252 / 0.038508 (0.024744) | 0.031474 / 0.023109 (0.008364) | 0.250321 / 0.275898 (-0.025577) | 0.275198 / 0.323480 (-0.048282) | 0.003275 / 0.007986 (-0.004710) | 0.002874 / 0.004328 (-0.001454) | 0.049499 / 0.004250 (0.045248) | 0.045334 / 0.037052 (0.008282) | 0.266347 / 0.258489 (0.007858) | 0.308974 / 0.293841 (0.015133) | 0.027742 / 0.128546 (-0.100804) | 0.010274 / 0.075646 (-0.065373) | 0.207516 / 0.419271 (-0.211755) | 0.036538 / 0.043533 (-0.006995) | 0.247949 / 0.255139 (-0.007190) | 0.268986 / 0.283200 (-0.014214) | 0.019842 / 0.141683 (-0.121841) | 1.117547 / 1.452155 (-0.334607) | 1.175813 / 1.492716 (-0.316903) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.103661 / 0.018006 (0.085655) | 0.331023 / 0.000490 (0.330534) | 0.000240 / 0.000200 (0.000040) | 0.000041 / 0.000054 (-0.000013) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.019767 / 0.037411 (-0.017645) | 0.061500 / 0.014526 (0.046974) | 0.075899 / 0.176557 (-0.100658) | 0.122240 / 0.737135 (-0.614895) | 0.074621 / 0.296338 (-0.221717) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.287501 / 0.215209 (0.072292) | 2.794737 / 2.077655 (0.717082) | 1.505362 / 1.504120 (0.001242) | 1.379481 / 1.541195 (-0.161713) | 1.394836 / 1.468490 (-0.073654) | 0.545803 / 4.584777 (-4.038974) | 2.364167 / 3.745712 (-1.381545) | 2.800923 / 5.269862 (-2.468939) | 1.723910 / 4.565676 (-2.841766) | 0.061270 / 0.424275 (-0.363005) | 0.005006 / 0.007607 (-0.002601) | 0.334952 / 0.226044 (0.108908) | 3.367122 / 2.268929 (1.098194) | 1.839822 / 55.444624 (-53.604803) | 1.553774 / 6.876477 (-5.322703) | 1.583585 / 2.142072 (-0.558488) | 0.624680 / 4.805227 (-4.180547) | 0.116364 / 6.500664 (-6.384300) | 0.042412 / 0.075469 (-0.033057) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.975207 / 1.841788 (-0.866580) | 11.843126 / 8.074308 (3.768818) | 9.418537 / 10.191392 (-0.772855) | 0.130648 / 0.680424 (-0.549775) | 0.013747 / 0.534201 (-0.520454) | 0.288195 / 0.579283 (-0.291088) | 0.269861 / 0.434364 (-0.164503) | 0.326732 / 0.540337 (-0.213606) | 0.441256 / 1.386936 (-0.945680) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005185 / 0.011353 (-0.006168) | 0.003836 / 0.011008 (-0.007172) | 0.050057 / 0.038508 (0.011549) | 0.030929 / 0.023109 (0.007820) | 0.263558 / 0.275898 (-0.012340) | 0.284553 / 0.323480 (-0.038927) | 0.004331 / 0.007986 (-0.003655) | 0.002815 / 0.004328 (-0.001513) | 0.050187 / 0.004250 (0.045936) | 0.048431 / 0.037052 (0.011379) | 0.271005 / 0.258489 (0.012515) | 0.304749 / 0.293841 (0.010908) | 0.029286 / 0.128546 (-0.099260) | 0.010598 / 0.075646 (-0.065048) | 0.058111 / 0.419271 (-0.361160) | 0.053665 / 0.043533 (0.010132) | 0.257574 / 0.255139 (0.002436) | 0.285802 / 0.283200 (0.002602) | 0.018917 / 0.141683 (-0.122766) | 1.206517 / 1.452155 (-0.245638) | 1.220572 / 1.492716 (-0.272144) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.122466 / 0.018006 (0.104460) | 0.567887 / 0.000490 (0.567397) | 0.000321 / 0.000200 (0.000121) | 0.000049 / 0.000054 (-0.000006) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.022120 / 0.037411 (-0.015292) | 0.075456 / 0.014526 (0.060931) | 0.086385 / 0.176557 (-0.090171) | 0.126045 / 0.737135 (-0.611091) | 0.087502 / 0.296338 (-0.208837) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.304847 / 0.215209 (0.089638) | 3.008095 / 2.077655 (0.930441) | 1.726178 / 1.504120 (0.222058) | 1.592332 / 1.541195 (0.051138) | 1.603714 / 1.468490 (0.135224) | 0.576875 / 4.584777 (-4.007902) | 2.450884 / 3.745712 (-1.294828) | 2.719073 / 5.269862 (-2.550789) | 1.775261 / 4.565676 (-2.790415) | 0.063144 / 0.424275 (-0.361131) | 0.005122 / 0.007607 (-0.002485) | 0.350004 / 0.226044 (0.123960) | 3.467146 / 2.268929 (1.198218) | 2.062907 / 55.444624 (-53.381717) | 1.798793 / 6.876477 (-5.077684) | 1.921204 / 2.142072 (-0.220868) | 0.651832 / 4.805227 (-4.153396) | 0.122326 / 6.500664 (-6.378338) | 0.041396 / 0.075469 (-0.034073) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.024859 / 1.841788 (-0.816928) | 12.569744 / 8.074308 (4.495436) | 10.448487 / 10.191392 (0.257095) | 0.131529 / 0.680424 (-0.548895) | 0.014853 / 0.534201 (-0.519348) | 0.287683 / 0.579283 (-0.291600) | 0.289814 / 0.434364 (-0.144550) | 0.323935 / 0.540337 (-0.216403) | 0.425208 / 1.386936 (-0.961728) |

</details>
</details>

![](https://cml.dev/watermark.png#ba71e92c59c9bd9d1ee6168691977f0c4728ed6e "CML watermark")


> EDIT: the fix has been released in `huggingface_hub` 0.21.2 - I removed my commits that were using `huggingface_hub@main`

Please note that people using `huggingface_hub` < 0.21.2 and latest `fsspec` will have issues when using `datasets`:
- https://github.com/huggingface/lighteval/actions/runs/8139147047/job/22241658122?pr=86
- https://github.com/huggingface/lighteval/pull/84

CC: @clefourrier