unpin fsspec
State: closed | Author: lhoestq (MEMBER)

close https://github.com/huggingface/datasets/issues/5618

Comments: _The documentation is not available anymore as the PR was closed or merged._

<details>
<summary>Show benchmarks</summary>

PyArrow==6.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.009954 / 0.011353 (-0.001398) | 0.005468 / 0.011008 (-0.005541) | 0.101228 / 0.038508 (0.062720) | 0.037878 / 0.023109 (0.014769) | 0.305635 / 0.275898 (0.029737) | 0.391672 / 0.323480 (0.068192) | 0.008893 / 0.007986 (0.000908) | 0.005861 / 0.004328 (0.001533) | 0.076940 / 0.004250 (0.072689) | 0.046242 / 0.037052 (0.009190) | 0.324033 / 0.258489 (0.065544) | 0.383306 / 0.293841 (0.089465) | 0.039298 / 0.128546 (-0.089249) | 0.012187 / 0.075646 (-0.063459) | 0.336774 / 0.419271 (-0.082498) | 0.053493 / 0.043533 (0.009960) | 0.303381 / 0.255139 (0.048242) | 0.323494 / 0.283200 (0.040295) | 0.118613 / 0.141683 (-0.023070) | 1.463430 / 1.452155 (0.011275) | 1.549856 / 1.492716 (0.057139) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.289264 / 0.018006 (0.271258) | 0.520348 / 0.000490 (0.519858) | 0.004543 / 0.000200 (0.004343) | 0.000090 / 0.000054 (0.000036) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.028183 / 0.037411 (-0.009229) | 0.107869 / 0.014526 (0.093343) | 0.124019 / 0.176557 (-0.052537) | 0.167769 / 0.737135 (-0.569367) | 0.130304 / 0.296338 (-0.166034) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.402296 / 0.215209 (0.187087) | 4.018884 / 2.077655 (1.941229) | 1.834050 / 1.504120 (0.329930) | 1.649974 / 1.541195 (0.108779) | 1.741697 / 1.468490 (0.273207) | 0.684354 / 4.584777 (-3.900423) | 3.778213 / 3.745712 (0.032501) | 2.158086 / 5.269862 (-3.111775) | 1.472671 / 4.565676 (-3.093006) | 0.083912 / 0.424275 (-0.340363) | 0.012285 / 0.007607 (0.004678) | 0.501689 / 0.226044 (0.275645) | 5.014722 / 2.268929 (2.745794) | 2.310722 / 55.444624 (-53.133902) | 1.983214 / 6.876477 (-4.893262) | 2.154518 / 2.142072 (0.012446) | 0.821277 / 4.805227 (-3.983950) | 0.164434 / 6.500664 (-6.336231) | 0.062568 / 0.075469 (-0.012901) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.224338 / 1.841788 (-0.617450) | 14.981623 / 8.074308 (6.907315) | 14.296356 / 10.191392 (4.104964) | 0.193554 / 0.680424 (-0.486870) | 0.028511 / 0.534201 (-0.505690) | 0.437649 / 0.579283 (-0.141634) | 0.448934 / 0.434364 (0.014570) | 0.552624 / 0.540337 (0.012287) | 0.654268 / 1.386936 (-0.732668) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007772 / 0.011353 (-0.003581) | 0.005534 / 0.011008 (-0.005474) | 0.074347 / 0.038508 (0.035839) | 0.034486 / 0.023109 (0.011376) | 0.343430 / 0.275898 (0.067532) | 0.385778 / 0.323480 (0.062298) | 0.006424 / 0.007986 (-0.001562) | 0.004241 / 0.004328 (-0.000087) | 0.072839 / 0.004250 (0.068589) | 0.055523 / 0.037052 (0.018471) | 0.342778 / 0.258489 (0.084289) | 0.389961 / 0.293841 (0.096120) | 0.037238 / 0.128546 (-0.091308) | 0.012450 / 0.075646 (-0.063197) | 0.085282 / 0.419271 (-0.333990) | 0.049678 / 0.043533 (0.006146) | 0.345300 / 0.255139 (0.090161) | 0.365220 / 0.283200 (0.082020) | 0.109257 / 0.141683 (-0.032426) | 1.480284 / 1.452155 (0.028129) | 1.627881 / 1.492716 (0.135165) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.323330 / 0.018006 (0.305324) | 0.530824 / 0.000490 (0.530334) | 0.000463 / 0.000200 (0.000263) | 0.000063 / 0.000054 (0.000009) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.032398 / 0.037411 (-0.005013) | 0.115889 / 0.014526 (0.101363) | 0.131093 / 0.176557 (-0.045464) | 0.180757 / 0.737135 (-0.556379) | 0.134395 / 0.296338 (-0.161943) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.423931 / 0.215209 (0.208722) | 4.238207 / 2.077655 (2.160553) | 2.075721 / 1.504120 (0.571602) | 1.887752 / 1.541195 (0.346557) | 2.055054 / 1.468490 (0.586564) | 0.703145 / 4.584777 (-3.881632) | 3.937120 / 3.745712 (0.191408) | 3.748550 / 5.269862 (-1.521311) | 1.562849 / 4.565676 (-3.002827) | 0.087695 / 0.424275 (-0.336580) | 0.012614 / 0.007607 (0.005007) | 0.523901 / 0.226044 (0.297856) | 5.230210 / 2.268929 (2.961282) | 2.592667 / 55.444624 (-52.851958) | 2.345662 / 6.876477 (-4.530815) | 2.475388 / 2.142072 (0.333316) | 0.836443 / 4.805227 (-3.968784) | 0.170304 / 6.500664 (-6.330360) | 0.067741 / 0.075469 (-0.007729) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.255171 / 1.841788 (-0.586617) | 16.312856 / 8.074308 (8.238548) | 13.184770 / 10.191392 (2.993378) | 0.145557 / 0.680424 (-0.534867) | 0.017723 / 0.534201 (-0.516478) | 0.423447 / 0.579283 (-0.155836) | 0.423063 / 0.434364 (-0.011301) | 0.494159 / 0.540337 (-0.046179) | 0.589590 / 1.386936 (-0.797346) |

</details>
</details>

![](https://cml.dev/watermark.png#4ea6f1db3f80eb3bb7ac6f252c2cd5bd97537c01 "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==6.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.012068 / 0.011353 (0.000715) | 0.006127 / 0.011008 (-0.004881) | 0.112550 / 0.038508 (0.074042) | 0.043201 / 0.023109 (0.020092) | 0.346666 / 0.275898 (0.070768) | 0.413852 / 0.323480 (0.090372) | 0.009342 / 0.007986 (0.001356) | 0.006302 / 0.004328 (0.001974) | 0.086901 / 0.004250 (0.082650) | 0.053992 / 0.037052 (0.016940) | 0.362192 / 0.258489 (0.103703) | 0.409867 / 0.293841 (0.116026) | 0.046124 / 0.128546 (-0.082422) | 0.014139 / 0.075646 (-0.061507) | 0.386386 / 0.419271 (-0.032886) | 0.058465 / 0.043533 (0.014932) | 0.344832 / 0.255139 (0.089693) | 0.370684 / 0.283200 (0.087485) | 0.122886 / 0.141683 (-0.018796) | 1.724013 / 1.452155 (0.271858) | 1.775756 / 1.492716 (0.283039) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.220289 / 0.018006 (0.202283) | 0.493585 / 0.000490 (0.493096) | 0.001970 / 0.000200 (0.001770) | 0.000099 / 0.000054 (0.000044) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.030763 / 0.037411 (-0.006649) | 0.128237 / 0.014526 (0.113711) | 0.138364 / 0.176557 (-0.038192) | 0.188115 / 0.737135 (-0.549021) | 0.145367 / 0.296338 (-0.150972) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.452487 / 0.215209 (0.237277) | 4.592728 / 2.077655 (2.515074) | 2.075712 / 1.504120 (0.571592) | 1.845424 / 1.541195 (0.304229) | 1.956400 / 1.468490 (0.487910) | 0.808387 / 4.584777 (-3.776390) | 4.483678 / 3.745712 (0.737966) | 3.870287 / 5.269862 (-1.399574) | 2.151205 / 4.565676 (-2.414471) | 0.098123 / 0.424275 (-0.326152) | 0.014139 / 0.007607 (0.006531) | 0.577775 / 0.226044 (0.351730) | 5.785545 / 2.268929 (3.516616) | 2.614418 / 55.444624 (-52.830206) | 2.312136 / 6.876477 (-4.564341) | 2.364189 / 2.142072 (0.222117) | 0.970028 / 4.805227 (-3.835199) | 0.189592 / 6.500664 (-6.311072) | 0.072883 / 0.075469 (-0.002586) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.414252 / 1.841788 (-0.427535) | 17.518307 / 8.074308 (9.443999) | 16.053748 / 10.191392 (5.862356) | 0.215297 / 0.680424 (-0.465127) | 0.033947 / 0.534201 (-0.500253) | 0.525794 / 0.579283 (-0.053489) | 0.514676 / 0.434364 (0.080312) | 0.595066 / 0.540337 (0.054728) | 0.689404 / 1.386936 (-0.697532) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008185 / 0.011353 (-0.003168) | 0.005776 / 0.011008 (-0.005232) | 0.084919 / 0.038508 (0.046411) | 0.037575 / 0.023109 (0.014466) | 0.401192 / 0.275898 (0.125294) | 0.443920 / 0.323480 (0.120440) | 0.006446 / 0.007986 (-0.001540) | 0.004428 / 0.004328 (0.000099) | 0.084013 / 0.004250 (0.079763) | 0.052013 / 0.037052 (0.014961) | 0.398429 / 0.258489 (0.139940) | 0.455676 / 0.293841 (0.161836) | 0.041568 / 0.128546 (-0.086978) | 0.013631 / 0.075646 (-0.062015) | 0.098709 / 0.419271 (-0.320563) | 0.055889 / 0.043533 (0.012356) | 0.402002 / 0.255139 (0.146863) | 0.424248 / 0.283200 (0.141049) | 0.113288 / 0.141683 (-0.028395) | 1.672214 / 1.452155 (0.220059) | 1.792940 / 1.492716 (0.300223) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.211847 / 0.018006 (0.193841) | 0.486711 / 0.000490 (0.486221) | 0.002907 / 0.000200 (0.002707) | 0.000118 / 0.000054 (0.000063) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.032931 / 0.037411 (-0.004480) | 0.142073 / 0.014526 (0.127547) | 0.142872 / 0.176557 (-0.033685) | 0.202612 / 0.737135 (-0.534523) | 0.154390 / 0.296338 (-0.141949) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.488682 / 0.215209 (0.273473) | 4.755805 / 2.077655 (2.678150) | 2.348778 / 1.504120 (0.844658) | 2.144992 / 1.541195 (0.603797) | 2.245654 / 1.468490 (0.777164) | 0.792690 / 4.584777 (-3.792087) | 4.569190 / 3.745712 (0.823478) | 3.919317 / 5.269862 (-1.350545) | 2.140302 / 4.565676 (-2.425374) | 0.096430 / 0.424275 (-0.327845) | 0.014551 / 0.007607 (0.006944) | 0.605138 / 0.226044 (0.379094) | 5.989470 / 2.268929 (3.720542) | 2.915525 / 55.444624 (-52.529099) | 2.516243 / 6.876477 (-4.360234) | 2.673114 / 2.142072 (0.531041) | 0.932330 / 4.805227 (-3.872897) | 0.191456 / 6.500664 (-6.309209) | 0.073887 / 0.075469 (-0.001582) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.455552 / 1.841788 (-0.386236) | 17.824864 / 8.074308 (9.750556) | 15.764150 / 10.191392 (5.572758) | 0.184935 / 0.680424 (-0.495489) | 0.020552 / 0.534201 (-0.513649) | 0.486816 / 0.579283 (-0.092467) | 0.489006 / 0.434364 (0.054642) | 0.609826 / 0.540337 (0.069488) | 0.721313 / 1.386936 (-0.665623) |

</details>
</details>

![](https://cml.dev/watermark.png#a0a35c5fa84a8a7df656c1f5b0a7266126fa9b75 "CML watermark")