Imagefolder docs: mention support of CSV and ZIP
State: closed | Author: lhoestq (MEMBER)


Comments: _The documentation is not available anymore as the PR was closed or merged._

<details>
<summary>Show benchmarks</summary>

PyArrow==6.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.009559 / 0.011353 (-0.001794) | 0.006425 / 0.011008 (-0.004583) | 0.112951 / 0.038508 (0.074443) | 0.030835 / 0.023109 (0.007725) | 0.313846 / 0.275898 (0.037948) | 0.352780 / 0.323480 (0.029301) | 0.007740 / 0.007986 (-0.000246) | 0.006843 / 0.004328 (0.002515) | 0.082632 / 0.004250 (0.078382) | 0.039704 / 0.037052 (0.002652) | 0.328526 / 0.258489 (0.070037) | 0.369162 / 0.293841 (0.075321) | 0.047603 / 0.128546 (-0.080943) | 0.015834 / 0.075646 (-0.059812) | 0.385912 / 0.419271 (-0.033360) | 0.053838 / 0.043533 (0.010306) | 0.325778 / 0.255139 (0.070639) | 0.361863 / 0.283200 (0.078663) | 0.097388 / 0.141683 (-0.044295) | 1.510132 / 1.452155 (0.057978) | 1.555980 / 1.492716 (0.063264) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.210792 / 0.018006 (0.192786) | 0.507270 / 0.000490 (0.506780) | 0.002383 / 0.000200 (0.002183) | 0.000095 / 0.000054 (0.000041) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.023057 / 0.037411 (-0.014355) | 0.103471 / 0.014526 (0.088945) | 0.111671 / 0.176557 (-0.064885) | 0.145665 / 0.737135 (-0.591470) | 0.131447 / 0.296338 (-0.164891) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.502979 / 0.215209 (0.287770) | 5.111471 / 2.077655 (3.033816) | 2.093604 / 1.504120 (0.589484) | 1.761342 / 1.541195 (0.220148) | 1.919485 / 1.468490 (0.450995) | 1.065672 / 4.584777 (-3.519105) | 5.109746 / 3.745712 (1.364034) | 4.694027 / 5.269862 (-0.575835) | 2.438401 / 4.565676 (-2.127275) | 0.133579 / 0.424275 (-0.290696) | 0.012355 / 0.007607 (0.004748) | 0.669077 / 0.226044 (0.443033) | 6.533905 / 2.268929 (4.264976) | 2.698832 / 55.444624 (-52.745792) | 2.146377 / 6.876477 (-4.730100) | 2.220563 / 2.142072 (0.078491) | 1.287855 / 4.805227 (-3.517372) | 0.238221 / 6.500664 (-6.262443) | 0.071426 / 0.075469 (-0.004043) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.332659 / 1.841788 (-0.509129) | 15.610100 / 8.074308 (7.535791) | 16.691117 / 10.191392 (6.499725) | 0.226338 / 0.680424 (-0.454086) | 0.039964 / 0.534201 (-0.494237) | 0.462911 / 0.579283 (-0.116372) | 0.575923 / 0.434364 (0.141560) | 0.592583 / 0.540337 (0.052245) | 0.658552 / 1.386936 (-0.728384) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008388 / 0.011353 (-0.002965) | 0.005360 / 0.011008 (-0.005648) | 0.104574 / 0.038508 (0.066066) | 0.030109 / 0.023109 (0.007000) | 0.389294 / 0.275898 (0.113396) | 0.424813 / 0.323480 (0.101333) | 0.006629 / 0.007986 (-0.001356) | 0.005222 / 0.004328 (0.000893) | 0.080157 / 0.004250 (0.075907) | 0.045811 / 0.037052 (0.008759) | 0.398708 / 0.258489 (0.140219) | 0.429449 / 0.293841 (0.135608) | 0.052242 / 0.128546 (-0.076304) | 0.017439 / 0.075646 (-0.058207) | 0.362678 / 0.419271 (-0.056593) | 0.054151 / 0.043533 (0.010618) | 0.387932 / 0.255139 (0.132793) | 0.410544 / 0.283200 (0.127344) | 0.101210 / 0.141683 (-0.040473) | 1.486496 / 1.452155 (0.034341) | 1.576404 / 1.492716 (0.083687) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.259468 / 0.018006 (0.241461) | 0.521661 / 0.000490 (0.521172) | 0.000456 / 0.000200 (0.000256) | 0.000078 / 0.000054 (0.000024) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.027045 / 0.037411 (-0.010366) | 0.107615 / 0.014526 (0.093089) | 0.133228 / 0.176557 (-0.043329) | 0.156807 / 0.737135 (-0.580328) | 0.125226 / 0.296338 (-0.171113) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.528804 / 0.215209 (0.313595) | 5.516402 / 2.077655 (3.438748) | 2.387531 / 1.504120 (0.883412) | 2.084734 / 1.541195 (0.543539) | 2.091894 / 1.468490 (0.623404) | 1.089761 / 4.584777 (-3.495016) | 5.093067 / 3.745712 (1.347355) | 2.670349 / 5.269862 (-2.599512) | 1.784723 / 4.565676 (-2.780953) | 0.125528 / 0.424275 (-0.298747) | 0.013702 / 0.007607 (0.006095) | 0.667755 / 0.226044 (0.441710) | 6.653900 / 2.268929 (4.384972) | 3.006058 / 55.444624 (-52.438567) | 2.512919 / 6.876477 (-4.363558) | 2.546824 / 2.142072 (0.404751) | 1.269008 / 4.805227 (-3.536219) | 0.234388 / 6.500664 (-6.266276) | 0.065675 / 0.075469 (-0.009795) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.372222 / 1.841788 (-0.469566) | 15.565156 / 8.074308 (7.490848) | 16.800666 / 10.191392 (6.609274) | 0.220656 / 0.680424 (-0.459768) | 0.023690 / 0.534201 (-0.510511) | 0.450049 / 0.579283 (-0.129234) | 0.580433 / 0.434364 (0.146069) | 0.558899 / 0.540337 (0.018561) | 0.676799 / 1.386936 (-0.710137) |

</details>
</details>

![](https://cml.dev/watermark.png#6cc5dcacecf41efc566385b323a3ca72ab44db36 "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==6.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.009440 / 0.011353 (-0.001913) | 0.005159 / 0.011008 (-0.005849) | 0.099152 / 0.038508 (0.060644) | 0.035939 / 0.023109 (0.012830) | 0.300968 / 0.275898 (0.025070) | 0.365676 / 0.323480 (0.042196) | 0.008220 / 0.007986 (0.000235) | 0.004071 / 0.004328 (-0.000257) | 0.075216 / 0.004250 (0.070965) | 0.042173 / 0.037052 (0.005121) | 0.315055 / 0.258489 (0.056566) | 0.338287 / 0.293841 (0.044446) | 0.037789 / 0.128546 (-0.090758) | 0.011856 / 0.075646 (-0.063791) | 0.332975 / 0.419271 (-0.086297) | 0.047087 / 0.043533 (0.003554) | 0.295107 / 0.255139 (0.039968) | 0.315416 / 0.283200 (0.032217) | 0.102273 / 0.141683 (-0.039410) | 1.464908 / 1.452155 (0.012754) | 1.500281 / 1.492716 (0.007565) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.208522 / 0.018006 (0.190516) | 0.446576 / 0.000490 (0.446086) | 0.005766 / 0.000200 (0.005566) | 0.000084 / 0.000054 (0.000029) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.027924 / 0.037411 (-0.009487) | 0.111296 / 0.014526 (0.096771) | 0.119055 / 0.176557 (-0.057502) | 0.157755 / 0.737135 (-0.579381) | 0.125539 / 0.296338 (-0.170799) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.395683 / 0.215209 (0.180474) | 3.962696 / 2.077655 (1.885042) | 1.789511 / 1.504120 (0.285391) | 1.591541 / 1.541195 (0.050346) | 1.661276 / 1.468490 (0.192786) | 0.693524 / 4.584777 (-3.891253) | 3.836526 / 3.745712 (0.090813) | 2.187284 / 5.269862 (-3.082578) | 1.521420 / 4.565676 (-3.044257) | 0.084370 / 0.424275 (-0.339905) | 0.012083 / 0.007607 (0.004476) | 0.498017 / 0.226044 (0.271972) | 4.982356 / 2.268929 (2.713428) | 2.235881 / 55.444624 (-53.208743) | 1.912067 / 6.876477 (-4.964410) | 2.052172 / 2.142072 (-0.089900) | 0.836232 / 4.805227 (-3.968995) | 0.165234 / 6.500664 (-6.335431) | 0.062933 / 0.075469 (-0.012536) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.197785 / 1.841788 (-0.644003) | 15.233655 / 8.074308 (7.159347) | 14.254450 / 10.191392 (4.063058) | 0.169149 / 0.680424 (-0.511274) | 0.028794 / 0.534201 (-0.505407) | 0.437214 / 0.579283 (-0.142069) | 0.434836 / 0.434364 (0.000472) | 0.531594 / 0.540337 (-0.008744) | 0.626266 / 1.386936 (-0.760670) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007394 / 0.011353 (-0.003959) | 0.005305 / 0.011008 (-0.005703) | 0.098888 / 0.038508 (0.060380) | 0.033069 / 0.023109 (0.009959) | 0.388427 / 0.275898 (0.112529) | 0.415216 / 0.323480 (0.091736) | 0.005610 / 0.007986 (-0.002375) | 0.004922 / 0.004328 (0.000593) | 0.073694 / 0.004250 (0.069443) | 0.047368 / 0.037052 (0.010315) | 0.379604 / 0.258489 (0.121115) | 0.424876 / 0.293841 (0.131035) | 0.039471 / 0.128546 (-0.089075) | 0.012219 / 0.075646 (-0.063427) | 0.345925 / 0.419271 (-0.073346) | 0.048981 / 0.043533 (0.005448) | 0.379303 / 0.255139 (0.124164) | 0.404682 / 0.283200 (0.121483) | 0.103932 / 0.141683 (-0.037751) | 1.490852 / 1.452155 (0.038697) | 1.578900 / 1.492716 (0.086183) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.201393 / 0.018006 (0.183387) | 0.452484 / 0.000490 (0.451994) | 0.005627 / 0.000200 (0.005428) | 0.000129 / 0.000054 (0.000075) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.029317 / 0.037411 (-0.008094) | 0.114904 / 0.014526 (0.100378) | 0.126678 / 0.176557 (-0.049878) | 0.178315 / 0.737135 (-0.558820) | 0.131603 / 0.296338 (-0.164736) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.459830 / 0.215209 (0.244621) | 4.595358 / 2.077655 (2.517703) | 2.383582 / 1.504120 (0.879462) | 2.181945 / 1.541195 (0.640750) | 2.309517 / 1.468490 (0.841027) | 0.704803 / 4.584777 (-3.879974) | 3.820411 / 3.745712 (0.074698) | 4.872173 / 5.269862 (-0.397689) | 2.266090 / 4.565676 (-2.299586) | 0.085805 / 0.424275 (-0.338470) | 0.012488 / 0.007607 (0.004881) | 0.557500 / 0.226044 (0.331456) | 5.570830 / 2.268929 (3.301901) | 2.836202 / 55.444624 (-52.608422) | 2.530534 / 6.876477 (-4.345943) | 2.599792 / 2.142072 (0.457720) | 0.843852 / 4.805227 (-3.961376) | 0.169427 / 6.500664 (-6.331237) | 0.065521 / 0.075469 (-0.009948) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.246014 / 1.841788 (-0.595774) | 15.455336 / 8.074308 (7.381028) | 13.559111 / 10.191392 (3.367719) | 0.169131 / 0.680424 (-0.511293) | 0.017812 / 0.534201 (-0.516389) | 0.421161 / 0.579283 (-0.158122) | 0.458286 / 0.434364 (0.023922) | 0.534692 / 0.540337 (-0.005645) | 0.639299 / 1.386936 (-0.747637) |

</details>
</details>

![](https://cml.dev/watermark.png#2b7558953b5a071194356bbe4c596a2890a3b847 "CML watermark")