Pin supported upper version of fsspec
State: closed | Author: albertvillanova (MEMBER)

Pin upper version of `fsspec` to avoid disruptions introduced by breaking changes (and the need of urgent patch releases with hotfixes) on each release on their side. See:
- #6331
- #6210 
- #5731
- #5617
- #5447

I propose that we explicitly test, introduce fixes and support each new `fsspec` version release.

CC: @LysandreJik 

Comments: <details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006915 / 0.011353 (-0.004438) | 0.004110 / 0.011008 (-0.006898) | 0.084392 / 0.038508 (0.045884) | 0.079649 / 0.023109 (0.056540) | 0.305760 / 0.275898 (0.029862) | 0.343968 / 0.323480 (0.020488) | 0.005402 / 0.007986 (-0.002584) | 0.003342 / 0.004328 (-0.000986) | 0.064774 / 0.004250 (0.060523) | 0.055919 / 0.037052 (0.018866) | 0.315194 / 0.258489 (0.056705) | 0.355014 / 0.293841 (0.061173) | 0.032140 / 0.128546 (-0.096406) | 0.008865 / 0.075646 (-0.066781) | 0.287684 / 0.419271 (-0.131588) | 0.053504 / 0.043533 (0.009971) | 0.306852 / 0.255139 (0.051713) | 0.331125 / 0.283200 (0.047925) | 0.023476 / 0.141683 (-0.118207) | 1.506590 / 1.452155 (0.054435) | 1.574508 / 1.492716 (0.081792) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.239987 / 0.018006 (0.221981) | 0.459144 / 0.000490 (0.458654) | 0.008509 / 0.000200 (0.008309) | 0.000335 / 0.000054 (0.000280) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.028353 / 0.037411 (-0.009058) | 0.082345 / 0.014526 (0.067819) | 0.499524 / 0.176557 (0.322967) | 0.152896 / 0.737135 (-0.584239) | 0.096978 / 0.296338 (-0.199360) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.404855 / 0.215209 (0.189646) | 4.053103 / 2.077655 (1.975448) | 2.069638 / 1.504120 (0.565518) | 1.917354 / 1.541195 (0.376159) | 2.035816 / 1.468490 (0.567326) | 0.480358 / 4.584777 (-4.104419) | 3.594316 / 3.745712 (-0.151396) | 3.582952 / 5.269862 (-1.686910) | 2.101142 / 4.565676 (-2.464535) | 0.057004 / 0.424275 (-0.367271) | 0.007715 / 0.007607 (0.000108) | 0.487417 / 0.226044 (0.261372) | 4.863100 / 2.268929 (2.594172) | 2.569038 / 55.444624 (-52.875587) | 2.187167 / 6.876477 (-4.689310) | 2.270034 / 2.142072 (0.127962) | 0.578095 / 4.805227 (-4.227132) | 0.133283 / 6.500664 (-6.367381) | 0.060164 / 0.075469 (-0.015305) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.269120 / 1.841788 (-0.572667) | 19.493072 / 8.074308 (11.418764) | 14.560576 / 10.191392 (4.369184) | 0.167440 / 0.680424 (-0.512984) | 0.018493 / 0.534201 (-0.515708) | 0.392774 / 0.579283 (-0.186509) | 0.420903 / 0.434364 (-0.013461) | 0.461904 / 0.540337 (-0.078433) | 0.643104 / 1.386936 (-0.743832) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006985 / 0.011353 (-0.004368) | 0.004246 / 0.011008 (-0.006762) | 0.066246 / 0.038508 (0.027738) | 0.080757 / 0.023109 (0.057648) | 0.391774 / 0.275898 (0.115876) | 0.424957 / 0.323480 (0.101478) | 0.005575 / 0.007986 (-0.002411) | 0.003447 / 0.004328 (-0.000881) | 0.066565 / 0.004250 (0.062315) | 0.057597 / 0.037052 (0.020544) | 0.394663 / 0.258489 (0.136174) | 0.430310 / 0.293841 (0.136469) | 0.032746 / 0.128546 (-0.095800) | 0.008783 / 0.075646 (-0.066863) | 0.071940 / 0.419271 (-0.347331) | 0.048877 / 0.043533 (0.005344) | 0.390269 / 0.255139 (0.135130) | 0.411867 / 0.283200 (0.128668) | 0.024101 / 0.141683 (-0.117582) | 1.507370 / 1.452155 (0.055215) | 1.585810 / 1.492716 (0.093093) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.222796 / 0.018006 (0.204790) | 0.459035 / 0.000490 (0.458546) | 0.005322 / 0.000200 (0.005122) | 0.000099 / 0.000054 (0.000045) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.033237 / 0.037411 (-0.004174) | 0.098244 / 0.014526 (0.083718) | 0.106654 / 0.176557 (-0.069903) | 0.159675 / 0.737135 (-0.577460) | 0.108470 / 0.296338 (-0.187869) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.429085 / 0.215209 (0.213876) | 4.281206 / 2.077655 (2.203551) | 2.320492 / 1.504120 (0.816372) | 2.153218 / 1.541195 (0.612024) | 2.287122 / 1.468490 (0.818632) | 0.497307 / 4.584777 (-4.087470) | 3.799541 / 3.745712 (0.053828) | 3.380053 / 5.269862 (-1.889809) | 2.100009 / 4.565676 (-2.465667) | 0.057988 / 0.424275 (-0.366287) | 0.007381 / 0.007607 (-0.000226) | 0.506843 / 0.226044 (0.280798) | 5.071286 / 2.268929 (2.802357) | 2.750487 / 55.444624 (-52.694137) | 2.415613 / 6.876477 (-4.460864) | 2.667144 / 2.142072 (0.525072) | 0.624889 / 4.805227 (-4.180338) | 0.134191 / 6.500664 (-6.366473) | 0.060704 / 0.075469 (-0.014765) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.353074 / 1.841788 (-0.488714) | 20.507074 / 8.074308 (12.432766) | 14.911788 / 10.191392 (4.720396) | 0.149248 / 0.680424 (-0.531176) | 0.020593 / 0.534201 (-0.513608) | 0.398458 / 0.579283 (-0.180825) | 0.434846 / 0.434364 (0.000482) | 0.478853 / 0.540337 (-0.061484) | 0.648072 / 1.386936 (-0.738864) |

</details>
</details>

![](https://cml.dev/watermark.png#b30c72a2d3d9c191a590e0f0a6b3a6363ab15e8f "CML watermark")


In particular I expect fsspec to do another breaking change in the next release (switch to glob.glob)

_The documentation is not available anymore as the PR was closed or merged._

see https://github.com/huggingface/datasets/pull/6338

Yes, unfortunately breaking changes are quite usual from their part.

<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006099 / 0.011353 (-0.005253) | 0.003672 / 0.011008 (-0.007336) | 0.083095 / 0.038508 (0.044587) | 0.059607 / 0.023109 (0.036498) | 0.319591 / 0.275898 (0.043693) | 0.351945 / 0.323480 (0.028465) | 0.004785 / 0.007986 (-0.003201) | 0.002965 / 0.004328 (-0.001364) | 0.062907 / 0.004250 (0.058657) | 0.049122 / 0.037052 (0.012070) | 0.344641 / 0.258489 (0.086152) | 0.361519 / 0.293841 (0.067678) | 0.027254 / 0.128546 (-0.101292) | 0.008081 / 0.075646 (-0.067565) | 0.261569 / 0.419271 (-0.157702) | 0.045101 / 0.043533 (0.001568) | 0.313645 / 0.255139 (0.058506) | 0.337843 / 0.283200 (0.054644) | 0.020968 / 0.141683 (-0.120715) | 1.438450 / 1.452155 (-0.013705) | 1.507567 / 1.492716 (0.014850) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.230826 / 0.018006 (0.212820) | 0.434363 / 0.000490 (0.433873) | 0.008210 / 0.000200 (0.008010) | 0.000212 / 0.000054 (0.000157) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.025278 / 0.037411 (-0.012133) | 0.073659 / 0.014526 (0.059133) | 0.085147 / 0.176557 (-0.091409) | 0.145451 / 0.737135 (-0.591684) | 0.086400 / 0.296338 (-0.209939) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.429887 / 0.215209 (0.214678) | 4.292626 / 2.077655 (2.214971) | 2.266824 / 1.504120 (0.762704) | 2.090472 / 1.541195 (0.549277) | 2.186477 / 1.468490 (0.717987) | 0.503684 / 4.584777 (-4.081093) | 3.100791 / 3.745712 (-0.644921) | 3.008938 / 5.269862 (-2.260923) | 1.885559 / 4.565676 (-2.680118) | 0.057434 / 0.424275 (-0.366841) | 0.006639 / 0.007607 (-0.000969) | 0.506579 / 0.226044 (0.280535) | 5.058905 / 2.268929 (2.789977) | 2.708321 / 55.444624 (-52.736304) | 2.367388 / 6.876477 (-4.509089) | 2.422660 / 2.142072 (0.280587) | 0.587562 / 4.805227 (-4.217665) | 0.125260 / 6.500664 (-6.375404) | 0.061856 / 0.075469 (-0.013613) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.280495 / 1.841788 (-0.561292) | 17.968873 / 8.074308 (9.894565) | 13.922838 / 10.191392 (3.731446) | 0.149907 / 0.680424 (-0.530517) | 0.016736 / 0.534201 (-0.517465) | 0.333417 / 0.579283 (-0.245866) | 0.367710 / 0.434364 (-0.066654) | 0.389648 / 0.540337 (-0.150690) | 0.535625 / 1.386936 (-0.851311) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006237 / 0.011353 (-0.005116) | 0.003787 / 0.011008 (-0.007221) | 0.062536 / 0.038508 (0.024028) | 0.062335 / 0.023109 (0.039226) | 0.455209 / 0.275898 (0.179311) | 0.488961 / 0.323480 (0.165482) | 0.004875 / 0.007986 (-0.003111) | 0.002961 / 0.004328 (-0.001368) | 0.063045 / 0.004250 (0.058795) | 0.048624 / 0.037052 (0.011571) | 0.455743 / 0.258489 (0.197254) | 0.494024 / 0.293841 (0.200183) | 0.028690 / 0.128546 (-0.099856) | 0.008147 / 0.075646 (-0.067499) | 0.069479 / 0.419271 (-0.349792) | 0.041613 / 0.043533 (-0.001919) | 0.460472 / 0.255139 (0.205333) | 0.475606 / 0.283200 (0.192406) | 0.020600 / 0.141683 (-0.121083) | 1.464960 / 1.452155 (0.012805) | 1.540942 / 1.492716 (0.048226) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.214558 / 0.018006 (0.196552) | 0.410482 / 0.000490 (0.409992) | 0.005539 / 0.000200 (0.005339) | 0.000076 / 0.000054 (0.000021) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.027044 / 0.037411 (-0.010367) | 0.081512 / 0.014526 (0.066986) | 0.101963 / 0.176557 (-0.074593) | 0.146686 / 0.737135 (-0.590449) | 0.092676 / 0.296338 (-0.203663) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.468766 / 0.215209 (0.253557) | 4.680514 / 2.077655 (2.602859) | 2.562454 / 1.504120 (1.058334) | 2.383692 / 1.541195 (0.842497) | 2.481820 / 1.468490 (1.013330) | 0.509122 / 4.584777 (-4.075655) | 3.201597 / 3.745712 (-0.544115) | 2.853539 / 5.269862 (-2.416323) | 1.891535 / 4.565676 (-2.674141) | 0.058594 / 0.424275 (-0.365681) | 0.006448 / 0.007607 (-0.001159) | 0.535950 / 0.226044 (0.309906) | 5.388239 / 2.268929 (3.119311) | 2.999986 / 55.444624 (-52.444638) | 2.733291 / 6.876477 (-4.143186) | 2.841548 / 2.142072 (0.699475) | 0.602388 / 4.805227 (-4.202840) | 0.126369 / 6.500664 (-6.374295) | 0.061519 / 0.075469 (-0.013951) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.322746 / 1.841788 (-0.519042) | 17.940825 / 8.074308 (9.866517) | 14.679559 / 10.191392 (4.488167) | 0.146481 / 0.680424 (-0.533943) | 0.018060 / 0.534201 (-0.516141) | 0.334924 / 0.579283 (-0.244359) | 0.384735 / 0.434364 (-0.049629) | 0.391834 / 0.540337 (-0.148503) | 0.540011 / 1.386936 (-0.846925) |

</details>
</details>

![](https://cml.dev/watermark.png#d82f3c2264436ef60fac8c397fb11c80175c5132 "CML watermark")