Temporarily pin fsspec < 2023.10.0
State: closed | Author: albertvillanova (MEMBER)

Temporarily pin fsspec < 2023.10.0 until permanent solution is found.

Hot fix #6330.

See: https://github.com/huggingface/datasets/actions/runs/6610904287/job/17953774987
```
...
ERROR tests/test_iterable_dataset.py::test_iterable_dataset_from_file - NotImplementedError: Loading a dataset cached in a LocalFileSystem is not supported.
= 373 failed, 2055 passed, 17 skipped, 8 warnings, 6 errors in 228.14s (0:03:48) =
```

Comments: _The documentation is not available anymore as the PR was closed or merged._

<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.009605 / 0.011353 (-0.001747) | 0.004864 / 0.011008 (-0.006144) | 0.114605 / 0.038508 (0.076097) | 0.090874 / 0.023109 (0.067765) | 0.429203 / 0.275898 (0.153305) | 0.489888 / 0.323480 (0.166408) | 0.006542 / 0.007986 (-0.001443) | 0.004585 / 0.004328 (0.000257) | 0.090251 / 0.004250 (0.086001) | 0.066612 / 0.037052 (0.029560) | 0.437491 / 0.258489 (0.179002) | 0.515196 / 0.293841 (0.221355) | 0.047756 / 0.128546 (-0.080791) | 0.013587 / 0.075646 (-0.062059) | 0.376960 / 0.419271 (-0.042311) | 0.069701 / 0.043533 (0.026168) | 0.430850 / 0.255139 (0.175711) | 0.475061 / 0.283200 (0.191861) | 0.034800 / 0.141683 (-0.106883) | 1.799947 / 1.452155 (0.347793) | 1.941863 / 1.492716 (0.449147) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.316685 / 0.018006 (0.298679) | 0.595098 / 0.000490 (0.594608) | 0.015447 / 0.000200 (0.015247) | 0.000463 / 0.000054 (0.000409) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.039364 / 0.037411 (0.001953) | 0.091295 / 0.014526 (0.076769) | 0.109380 / 0.176557 (-0.067177) | 0.185454 / 0.737135 (-0.551681) | 0.104476 / 0.296338 (-0.191862) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.626291 / 0.215209 (0.411082) | 5.869948 / 2.077655 (3.792293) | 2.466267 / 1.504120 (0.962147) | 2.183572 / 1.541195 (0.642377) | 2.208286 / 1.468490 (0.739796) | 0.817175 / 4.584777 (-3.767602) | 5.255141 / 3.745712 (1.509429) | 4.878668 / 5.269862 (-0.391193) | 2.917020 / 4.565676 (-1.648657) | 0.104995 / 0.424275 (-0.319280) | 0.008687 / 0.007607 (0.001080) | 0.678993 / 0.226044 (0.452948) | 7.004983 / 2.268929 (4.736054) | 3.444040 / 55.444624 (-52.000584) | 2.745075 / 6.876477 (-4.131402) | 2.720151 / 2.142072 (0.578078) | 0.995803 / 4.805227 (-3.809424) | 0.205928 / 6.500664 (-6.294736) | 0.077053 / 0.075469 (0.001584) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.587354 / 1.841788 (-0.254434) | 23.843227 / 8.074308 (15.768919) | 21.355771 / 10.191392 (11.164379) | 0.225593 / 0.680424 (-0.454831) | 0.029054 / 0.534201 (-0.505147) | 0.469676 / 0.579283 (-0.109607) | 0.582619 / 0.434364 (0.148255) | 0.576932 / 0.540337 (0.036594) | 0.946182 / 1.386936 (-0.440754) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.009819 / 0.011353 (-0.001534) | 0.005562 / 0.011008 (-0.005446) | 0.075512 / 0.038508 (0.037004) | 0.084294 / 0.023109 (0.061185) | 0.549516 / 0.275898 (0.273618) | 0.550364 / 0.323480 (0.226884) | 0.006603 / 0.007986 (-0.001383) | 0.004587 / 0.004328 (0.000259) | 0.084040 / 0.004250 (0.079789) | 0.066815 / 0.037052 (0.029762) | 0.549224 / 0.258489 (0.290735) | 0.556213 / 0.293841 (0.262372) | 0.048538 / 0.128546 (-0.080008) | 0.014050 / 0.075646 (-0.061596) | 0.088955 / 0.419271 (-0.330317) | 0.062393 / 0.043533 (0.018860) | 0.528770 / 0.255139 (0.273631) | 0.564854 / 0.283200 (0.281655) | 0.033976 / 0.141683 (-0.107707) | 1.858558 / 1.452155 (0.406403) | 1.894616 / 1.492716 (0.401899) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.378597 / 0.018006 (0.360591) | 0.650586 / 0.000490 (0.650097) | 0.033179 / 0.000200 (0.032979) | 0.000477 / 0.000054 (0.000423) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.031779 / 0.037411 (-0.005632) | 0.103393 / 0.014526 (0.088867) | 0.119810 / 0.176557 (-0.056747) | 0.192188 / 0.737135 (-0.544948) | 0.114545 / 0.296338 (-0.181794) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.623571 / 0.215209 (0.408362) | 6.350249 / 2.077655 (4.272594) | 3.207773 / 1.504120 (1.703653) | 2.861118 / 1.541195 (1.319923) | 2.864445 / 1.468490 (1.395955) | 0.827451 / 4.584777 (-3.757326) | 5.323860 / 3.745712 (1.578148) | 4.569197 / 5.269862 (-0.700665) | 2.967595 / 4.565676 (-1.598081) | 0.090926 / 0.424275 (-0.333349) | 0.007820 / 0.007607 (0.000213) | 0.731610 / 0.226044 (0.505565) | 7.342651 / 2.268929 (5.073723) | 3.781727 / 55.444624 (-51.662897) | 3.222100 / 6.876477 (-3.654377) | 3.546145 / 2.142072 (1.404073) | 1.030500 / 4.805227 (-3.774728) | 0.226563 / 6.500664 (-6.274101) | 0.078633 / 0.075469 (0.003164) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.733677 / 1.841788 (-0.108111) | 24.650616 / 8.074308 (16.576308) | 22.033745 / 10.191392 (11.842353) | 0.211055 / 0.680424 (-0.469369) | 0.031658 / 0.534201 (-0.502543) | 0.467190 / 0.579283 (-0.112094) | 0.598303 / 0.434364 (0.163939) | 0.569318 / 0.540337 (0.028981) | 0.825984 / 1.386936 (-0.560952) |

</details>
</details>

![](https://cml.dev/watermark.png#7eef7749031232d0b29f7ca10e3fa9f997b19ef7 "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006873 / 0.011353 (-0.004479) | 0.004174 / 0.011008 (-0.006835) | 0.085874 / 0.038508 (0.047366) | 0.074207 / 0.023109 (0.051098) | 0.307342 / 0.275898 (0.031444) | 0.339972 / 0.323480 (0.016493) | 0.005522 / 0.007986 (-0.002463) | 0.003576 / 0.004328 (-0.000753) | 0.065680 / 0.004250 (0.061430) | 0.056274 / 0.037052 (0.019222) | 0.313121 / 0.258489 (0.054632) | 0.364699 / 0.293841 (0.070858) | 0.031297 / 0.128546 (-0.097249) | 0.008652 / 0.075646 (-0.066994) | 0.288431 / 0.419271 (-0.130840) | 0.053081 / 0.043533 (0.009548) | 0.309076 / 0.255139 (0.053937) | 0.329251 / 0.283200 (0.046052) | 0.024840 / 0.141683 (-0.116843) | 1.484155 / 1.452155 (0.032001) | 1.598665 / 1.492716 (0.105949) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.270933 / 0.018006 (0.252927) | 0.565867 / 0.000490 (0.565377) | 0.006964 / 0.000200 (0.006764) | 0.000298 / 0.000054 (0.000244) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.028393 / 0.037411 (-0.009018) | 0.081756 / 0.014526 (0.067230) | 0.095733 / 0.176557 (-0.080823) | 0.152426 / 0.737135 (-0.584710) | 0.096655 / 0.296338 (-0.199683) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.403992 / 0.215209 (0.188783) | 4.027230 / 2.077655 (1.949576) | 2.031102 / 1.504120 (0.526982) | 1.843727 / 1.541195 (0.302532) | 1.898342 / 1.468490 (0.429852) | 0.479186 / 4.584777 (-4.105591) | 3.488153 / 3.745712 (-0.257559) | 3.523953 / 5.269862 (-1.745909) | 2.078392 / 4.565676 (-2.487284) | 0.056104 / 0.424275 (-0.368171) | 0.007368 / 0.007607 (-0.000239) | 0.479630 / 0.226044 (0.253585) | 4.787400 / 2.268929 (2.518471) | 2.488268 / 55.444624 (-52.956356) | 2.229955 / 6.876477 (-4.646522) | 2.260468 / 2.142072 (0.118396) | 0.587934 / 4.805227 (-4.217294) | 0.147124 / 6.500664 (-6.353540) | 0.059954 / 0.075469 (-0.015515) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.283155 / 1.841788 (-0.558632) | 19.013574 / 8.074308 (10.939266) | 13.915188 / 10.191392 (3.723796) | 0.174101 / 0.680424 (-0.506323) | 0.018172 / 0.534201 (-0.516029) | 0.390322 / 0.579283 (-0.188961) | 0.405493 / 0.434364 (-0.028871) | 0.456914 / 0.540337 (-0.083424) | 0.635213 / 1.386936 (-0.751723) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006622 / 0.011353 (-0.004731) | 0.003997 / 0.011008 (-0.007011) | 0.064542 / 0.038508 (0.026034) | 0.074165 / 0.023109 (0.051056) | 0.392285 / 0.275898 (0.116387) | 0.423522 / 0.323480 (0.100042) | 0.006361 / 0.007986 (-0.001625) | 0.003463 / 0.004328 (-0.000866) | 0.064891 / 0.004250 (0.060641) | 0.058485 / 0.037052 (0.021433) | 0.425217 / 0.258489 (0.166728) | 0.435907 / 0.293841 (0.142066) | 0.031501 / 0.128546 (-0.097045) | 0.008575 / 0.075646 (-0.067071) | 0.072094 / 0.419271 (-0.347178) | 0.047904 / 0.043533 (0.004371) | 0.397174 / 0.255139 (0.142035) | 0.417940 / 0.283200 (0.134741) | 0.023324 / 0.141683 (-0.118358) | 1.517245 / 1.452155 (0.065090) | 1.586497 / 1.492716 (0.093781) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.268311 / 0.018006 (0.250305) | 0.561118 / 0.000490 (0.560628) | 0.004352 / 0.000200 (0.004152) | 0.000095 / 0.000054 (0.000041) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.033039 / 0.037411 (-0.004373) | 0.091596 / 0.014526 (0.077071) | 0.111520 / 0.176557 (-0.065036) | 0.161755 / 0.737135 (-0.575381) | 0.107681 / 0.296338 (-0.188657) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.427170 / 0.215209 (0.211961) | 4.252648 / 2.077655 (2.174994) | 2.257623 / 1.504120 (0.753503) | 2.098446 / 1.541195 (0.557251) | 2.128544 / 1.468490 (0.660054) | 0.496639 / 4.584777 (-4.088138) | 3.593385 / 3.745712 (-0.152328) | 3.396367 / 5.269862 (-1.873494) | 2.073369 / 4.565676 (-2.492308) | 0.058386 / 0.424275 (-0.365889) | 0.007515 / 0.007607 (-0.000093) | 0.502358 / 0.226044 (0.276313) | 5.015224 / 2.268929 (2.746296) | 2.735740 / 55.444624 (-52.708885) | 2.388368 / 6.876477 (-4.488109) | 2.682857 / 2.142072 (0.540785) | 0.595003 / 4.805227 (-4.210225) | 0.135419 / 6.500664 (-6.365245) | 0.062824 / 0.075469 (-0.012645) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.367507 / 1.841788 (-0.474281) | 19.569288 / 8.074308 (11.494979) | 14.748693 / 10.191392 (4.557301) | 0.198659 / 0.680424 (-0.481765) | 0.020954 / 0.534201 (-0.513247) | 0.414858 / 0.579283 (-0.164426) | 0.421226 / 0.434364 (-0.013138) | 0.477774 / 0.540337 (-0.062563) | 0.676173 / 1.386936 (-0.710763) |

</details>
</details>

![](https://cml.dev/watermark.png#87ed467af77d87ad7e38279cf9a24c341545cbac "CML watermark")