Document BUILDER_CONFIG_CLASS
Opened by lhoestq (closed)

Related to https://github.com/huggingface/datasets/issues/6130

Comments:
_The documentation is not available anymore as the PR was closed or merged._

<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.009036 / 0.011353 (-0.002317) | 0.004564 / 0.011008 (-0.006444) | 0.114958 / 0.038508 (0.076449) | 0.087329 / 0.023109 (0.064220) | 0.440111 / 0.275898 (0.164213) | 0.486056 / 0.323480 (0.162576) | 0.006580 / 0.007986 (-0.001406) | 0.004257 / 0.004328 (-0.000072) | 0.093458 / 0.004250 (0.089208) | 0.063380 / 0.037052 (0.026328) | 0.469455 / 0.258489 (0.210966) | 0.521630 / 0.293841 (0.227790) | 0.053496 / 0.128546 (-0.075050) | 0.013466 / 0.075646 (-0.062181) | 0.361629 / 0.419271 (-0.057642) | 0.068095 / 0.043533 (0.024562) | 0.472440 / 0.255139 (0.217301) | 0.508682 / 0.283200 (0.225483) | 0.034648 / 0.141683 (-0.107035) | 1.820117 / 1.452155 (0.367962) | 1.933448 / 1.492716 (0.440732) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.276543 / 0.018006 (0.258537) | 0.563380 / 0.000490 (0.562890) | 0.005345 / 0.000200 (0.005146) | 0.000107 / 0.000054 (0.000053) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.029230 / 0.037411 (-0.008181) | 0.095613 / 0.014526 (0.081087) | 0.106178 / 0.176557 (-0.070378) | 0.181095 / 0.737135 (-0.556040) | 0.107789 / 0.296338 (-0.188550) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.612051 / 0.215209 (0.396842) | 6.065008 / 2.077655 (3.987353) | 2.720911 / 1.504120 (1.216791) | 2.495218 / 1.541195 (0.954023) | 2.423351 / 1.468490 (0.954860) | 0.835571 / 4.584777 (-3.749205) | 5.438230 / 3.745712 (1.692518) | 4.550301 / 5.269862 (-0.719561) | 2.919889 / 4.565676 (-1.645788) | 0.097748 / 0.424275 (-0.326527) | 0.009285 / 0.007607 (0.001678) | 0.741968 / 0.226044 (0.515923) | 7.285394 / 2.268929 (5.016466) | 3.433634 / 55.444624 (-52.010991) | 2.680823 / 6.876477 (-4.195654) | 2.931149 / 2.142072 (0.789076) | 1.012852 / 4.805227 (-3.792375) | 0.224899 / 6.500664 (-6.275765) | 0.089411 / 0.075469 (0.013942) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.622759 / 1.841788 (-0.219029) | 23.690030 / 8.074308 (15.615721) | 21.034451 / 10.191392 (10.843059) | 0.241504 / 0.680424 (-0.438920) | 0.030109 / 0.534201 (-0.504092) | 0.472536 / 0.579283 (-0.106747) | 0.631396 / 0.434364 (0.197032) | 0.598997 / 0.540337 (0.058659) | 0.798680 / 1.386936 (-0.588256) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008696 / 0.011353 (-0.002657) | 0.005032 / 0.011008 (-0.005977) | 0.087369 / 0.038508 (0.048861) | 0.078105 / 0.023109 (0.054996) | 0.464861 / 0.275898 (0.188963) | 0.509620 / 0.323480 (0.186140) | 0.006399 / 0.007986 (-0.001587) | 0.004276 / 0.004328 (-0.000052) | 0.081643 / 0.004250 (0.077393) | 0.062560 / 0.037052 (0.025508) | 0.495377 / 0.258489 (0.236888) | 0.484885 / 0.293841 (0.191044) | 0.054354 / 0.128546 (-0.074193) | 0.013851 / 0.075646 (-0.061795) | 0.089531 / 0.419271 (-0.329740) | 0.068732 / 0.043533 (0.025199) | 0.455842 / 0.255139 (0.200703) | 0.528775 / 0.283200 (0.245575) | 0.039646 / 0.141683 (-0.102037) | 1.733600 / 1.452155 (0.281445) | 1.879074 / 1.492716 (0.386358) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.369616 / 0.018006 (0.351610) | 0.607426 / 0.000490 (0.606936) | 0.055540 / 0.000200 (0.055341) | 0.000543 / 0.000054 (0.000488) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.036026 / 0.037411 (-0.001385) | 0.103968 / 0.014526 (0.089442) | 0.114852 / 0.176557 (-0.061705) | 0.187313 / 0.737135 (-0.549822) | 0.116839 / 0.296338 (-0.179500) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.614018 / 0.215209 (0.398809) | 6.139914 / 2.077655 (4.062259) | 2.826246 / 1.504120 (1.322126) | 2.524133 / 1.541195 (0.982938) | 2.606981 / 1.468490 (1.138491) | 0.844604 / 4.584777 (-3.740173) | 5.537178 / 3.745712 (1.791465) | 4.594624 / 5.269862 (-0.675237) | 3.032145 / 4.565676 (-1.533532) | 0.094771 / 0.424275 (-0.329504) | 0.008132 / 0.007607 (0.000525) | 0.714287 / 0.226044 (0.488242) | 7.296733 / 2.268929 (5.027804) | 3.698066 / 55.444624 (-51.746558) | 2.862781 / 6.876477 (-4.013696) | 3.114502 / 2.142072 (0.972429) | 0.986612 / 4.805227 (-3.818616) | 0.214438 / 6.500664 (-6.286226) | 0.076201 / 0.075469 (0.000732) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.747728 / 1.841788 (-0.094060) | 24.159845 / 8.074308 (16.085537) | 23.553485 / 10.191392 (13.362093) | 0.248387 / 0.680424 (-0.432037) | 0.029850 / 0.534201 (-0.504351) | 0.526416 / 0.579283 (-0.052867) | 0.625681 / 0.434364 (0.191317) | 0.619690 / 0.540337 (0.079352) | 0.827485 / 1.386936 (-0.559451) |

</details>
</details>

![](https://cml.dev/watermark.png#75639f9064dab9549add79fd5ee7de2a4429992c "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006728 / 0.011353 (-0.004625) | 0.003960 / 0.011008 (-0.007048) | 0.085569 / 0.038508 (0.047061) | 0.077463 / 0.023109 (0.054354) | 0.343112 / 0.275898 (0.067214) | 0.379128 / 0.323480 (0.055648) | 0.004087 / 0.007986 (-0.003899) | 0.003357 / 0.004328 (-0.000972) | 0.065570 / 0.004250 (0.061320) | 0.056259 / 0.037052 (0.019207) | 0.368595 / 0.258489 (0.110106) | 0.402672 / 0.293841 (0.108831) | 0.030946 / 0.128546 (-0.097600) | 0.008509 / 0.075646 (-0.067137) | 0.288552 / 0.419271 (-0.130719) | 0.052134 / 0.043533 (0.008601) | 0.344653 / 0.255139 (0.089514) | 0.374199 / 0.283200 (0.090999) | 0.026251 / 0.141683 (-0.115432) | 1.488258 / 1.452155 (0.036103) | 1.567119 / 1.492716 (0.074402) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.218740 / 0.018006 (0.200734) | 0.465483 / 0.000490 (0.464994) | 0.003959 / 0.000200 (0.003759) | 0.000083 / 0.000054 (0.000029) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.029860 / 0.037411 (-0.007551) | 0.087968 / 0.014526 (0.073442) | 0.098257 / 0.176557 (-0.078299) | 0.155478 / 0.737135 (-0.581657) | 0.100696 / 0.296338 (-0.195642) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.384642 / 0.215209 (0.169432) | 3.821692 / 2.077655 (1.744038) | 1.838012 / 1.504120 (0.333892) | 1.677554 / 1.541195 (0.136360) | 1.764284 / 1.468490 (0.295794) | 0.487512 / 4.584777 (-4.097265) | 3.614572 / 3.745712 (-0.131141) | 3.300740 / 5.269862 (-1.969122) | 2.079044 / 4.565676 (-2.486632) | 0.057392 / 0.424275 (-0.366883) | 0.007642 / 0.007607 (0.000035) | 0.456161 / 0.226044 (0.230117) | 4.554124 / 2.268929 (2.285196) | 2.319288 / 55.444624 (-53.125336) | 1.972024 / 6.876477 (-4.904452) | 2.210598 / 2.142072 (0.068526) | 0.588442 / 4.805227 (-4.216785) | 0.134474 / 6.500664 (-6.366191) | 0.062682 / 0.075469 (-0.012787) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.243548 / 1.841788 (-0.598239) | 20.267230 / 8.074308 (12.192922) | 14.872096 / 10.191392 (4.680704) | 0.165164 / 0.680424 (-0.515260) | 0.018985 / 0.534201 (-0.515216) | 0.394526 / 0.579283 (-0.184757) | 0.413918 / 0.434364 (-0.020446) | 0.467130 / 0.540337 (-0.073208) | 0.627055 / 1.386936 (-0.759881) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006940 / 0.011353 (-0.004412) | 0.004203 / 0.011008 (-0.006805) | 0.065828 / 0.038508 (0.027320) | 0.076604 / 0.023109 (0.053495) | 0.401781 / 0.275898 (0.125883) | 0.434838 / 0.323480 (0.111358) | 0.005626 / 0.007986 (-0.002359) | 0.003409 / 0.004328 (-0.000920) | 0.064702 / 0.004250 (0.060452) | 0.057525 / 0.037052 (0.020473) | 0.405032 / 0.258489 (0.146543) | 0.440906 / 0.293841 (0.147065) | 0.032713 / 0.128546 (-0.095833) | 0.008723 / 0.075646 (-0.066923) | 0.071448 / 0.419271 (-0.347823) | 0.048186 / 0.043533 (0.004653) | 0.403950 / 0.255139 (0.148811) | 0.419506 / 0.283200 (0.136307) | 0.023532 / 0.141683 (-0.118150) | 1.496435 / 1.452155 (0.044280) | 1.567236 / 1.492716 (0.074519) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.229194 / 0.018006 (0.211188) | 0.451363 / 0.000490 (0.450873) | 0.003651 / 0.000200 (0.003451) | 0.000108 / 0.000054 (0.000053) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.033674 / 0.037411 (-0.003737) | 0.097521 / 0.014526 (0.082995) | 0.108806 / 0.176557 (-0.067751) | 0.161002 / 0.737135 (-0.576133) | 0.108594 / 0.296338 (-0.187745) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.436638 / 0.215209 (0.221429) | 4.348844 / 2.077655 (2.271189) | 2.341737 / 1.504120 (0.837617) | 2.195850 / 1.541195 (0.654656) | 2.332147 / 1.468490 (0.863657) | 0.496180 / 4.584777 (-4.088597) | 3.680987 / 3.745712 (-0.064725) | 3.332203 / 5.269862 (-1.937659) | 2.099541 / 4.565676 (-2.466136) | 0.058629 / 0.424275 (-0.365646) | 0.007363 / 0.007607 (-0.000245) | 0.517658 / 0.226044 (0.291614) | 5.175321 / 2.268929 (2.906392) | 2.858660 / 55.444624 (-52.585964) | 2.540557 / 6.876477 (-4.335920) | 2.755360 / 2.142072 (0.613288) | 0.595488 / 4.805227 (-4.209739) | 0.134265 / 6.500664 (-6.366399) | 0.062033 / 0.075469 (-0.013436) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.389950 / 1.841788 (-0.451838) | 20.800274 / 8.074308 (12.725966) | 15.314531 / 10.191392 (5.123139) | 0.166822 / 0.680424 (-0.513602) | 0.021099 / 0.534201 (-0.513102) | 0.400388 / 0.579283 (-0.178895) | 0.419981 / 0.434364 (-0.014383) | 0.474259 / 0.540337 (-0.066078) | 0.731678 / 1.386936 (-0.655258) |

</details>
</details>

![](https://cml.dev/watermark.png#4566827557acbeba0d4cb66449bb70367e341b05 "CML watermark")