Ignore parallel warning in map_nested
State: closed | Author: lhoestq (MEMBER)

This warning message was shown every time you pass num_proc to `load_dataset` because of `map_nested`

```
parallel_map is experimental and might be subject to breaking changes in the future
```

This PR removes it for `map_nested`. If someone uses another parallel backend they're already warned when `parallel_backend` is called anyway

Comments: _The documentation is not available anymore as the PR was closed or merged._

<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006818 / 0.011353 (-0.004534) | 0.004166 / 0.011008 (-0.006842) | 0.086756 / 0.038508 (0.048248) | 0.084444 / 0.023109 (0.061335) | 0.319249 / 0.275898 (0.043351) | 0.358689 / 0.323480 (0.035209) | 0.004344 / 0.007986 (-0.003641) | 0.003564 / 0.004328 (-0.000765) | 0.065021 / 0.004250 (0.060771) | 0.055991 / 0.037052 (0.018939) | 0.319573 / 0.258489 (0.061084) | 0.373239 / 0.293841 (0.079398) | 0.031431 / 0.128546 (-0.097115) | 0.008671 / 0.075646 (-0.066975) | 0.288484 / 0.419271 (-0.130788) | 0.053501 / 0.043533 (0.009968) | 0.316934 / 0.255139 (0.061795) | 0.354233 / 0.283200 (0.071034) | 0.028088 / 0.141683 (-0.113595) | 1.510905 / 1.452155 (0.058750) | 1.568614 / 1.492716 (0.075898) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.292343 / 0.018006 (0.274337) | 0.592309 / 0.000490 (0.591819) | 0.003850 / 0.000200 (0.003650) | 0.000084 / 0.000054 (0.000030) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.033510 / 0.037411 (-0.003901) | 0.089546 / 0.014526 (0.075020) | 0.104909 / 0.176557 (-0.071648) | 0.162219 / 0.737135 (-0.574916) | 0.104137 / 0.296338 (-0.192202) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.407993 / 0.215209 (0.192784) | 4.063423 / 2.077655 (1.985768) | 2.050237 / 1.504120 (0.546117) | 1.888939 / 1.541195 (0.347744) | 2.015195 / 1.468490 (0.546704) | 0.492617 / 4.584777 (-4.092160) | 3.595871 / 3.745712 (-0.149841) | 3.320467 / 5.269862 (-1.949395) | 2.099987 / 4.565676 (-2.465690) | 0.058513 / 0.424275 (-0.365762) | 0.007709 / 0.007607 (0.000102) | 0.479277 / 0.226044 (0.253233) | 4.790712 / 2.268929 (2.521783) | 2.517292 / 55.444624 (-52.927332) | 2.167461 / 6.876477 (-4.709016) | 2.432011 / 2.142072 (0.289939) | 0.600537 / 4.805227 (-4.204690) | 0.133538 / 6.500664 (-6.367126) | 0.059621 / 0.075469 (-0.015848) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.280375 / 1.841788 (-0.561413) | 20.777971 / 8.074308 (12.703663) | 14.869539 / 10.191392 (4.678147) | 0.159372 / 0.680424 (-0.521052) | 0.018096 / 0.534201 (-0.516105) | 0.393945 / 0.579283 (-0.185338) | 0.409598 / 0.434364 (-0.024766) | 0.459202 / 0.540337 (-0.081136) | 0.632298 / 1.386936 (-0.754638) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006694 / 0.011353 (-0.004659) | 0.004299 / 0.011008 (-0.006709) | 0.064880 / 0.038508 (0.026372) | 0.083233 / 0.023109 (0.060124) | 0.366488 / 0.275898 (0.090590) | 0.405049 / 0.323480 (0.081569) | 0.005602 / 0.007986 (-0.002384) | 0.003623 / 0.004328 (-0.000705) | 0.064410 / 0.004250 (0.060160) | 0.057962 / 0.037052 (0.020910) | 0.365318 / 0.258489 (0.106829) | 0.403151 / 0.293841 (0.109310) | 0.031285 / 0.128546 (-0.097261) | 0.008867 / 0.075646 (-0.066780) | 0.071137 / 0.419271 (-0.348135) | 0.048398 / 0.043533 (0.004865) | 0.360187 / 0.255139 (0.105048) | 0.383872 / 0.283200 (0.100673) | 0.023232 / 0.141683 (-0.118451) | 1.526980 / 1.452155 (0.074826) | 1.587265 / 1.492716 (0.094549) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.362603 / 0.018006 (0.344596) | 0.557034 / 0.000490 (0.556544) | 0.025303 / 0.000200 (0.025103) | 0.000562 / 0.000054 (0.000508) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.030636 / 0.037411 (-0.006775) | 0.088085 / 0.014526 (0.073559) | 0.103238 / 0.176557 (-0.073318) | 0.155208 / 0.737135 (-0.581928) | 0.106661 / 0.296338 (-0.189678) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.413660 / 0.215209 (0.198451) | 4.122717 / 2.077655 (2.045063) | 2.097656 / 1.504120 (0.593536) | 1.931995 / 1.541195 (0.390801) | 2.071497 / 1.468490 (0.603007) | 0.490257 / 4.584777 (-4.094520) | 3.588076 / 3.745712 (-0.157636) | 3.423087 / 5.269862 (-1.846774) | 2.147974 / 4.565676 (-2.417703) | 0.058783 / 0.424275 (-0.365492) | 0.007456 / 0.007607 (-0.000151) | 0.492350 / 0.226044 (0.266305) | 4.935935 / 2.268929 (2.667006) | 2.604217 / 55.444624 (-52.840407) | 2.333723 / 6.876477 (-4.542754) | 2.585293 / 2.142072 (0.443220) | 0.608800 / 4.805227 (-4.196427) | 0.135806 / 6.500664 (-6.364858) | 0.062716 / 0.075469 (-0.012753) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.347359 / 1.841788 (-0.494429) | 21.420505 / 8.074308 (13.346197) | 14.325914 / 10.191392 (4.134522) | 0.159617 / 0.680424 (-0.520806) | 0.018769 / 0.534201 (-0.515432) | 0.399677 / 0.579283 (-0.179606) | 0.402992 / 0.434364 (-0.031372) | 0.484629 / 0.540337 (-0.055709) | 0.656007 / 1.386936 (-0.730929) |

</details>
</details>

![](https://cml.dev/watermark.png#ac94bb10d5c00ce8fdaf461eb1ff4b8572cfe956 "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007291 / 0.011353 (-0.004062) | 0.004501 / 0.011008 (-0.006508) | 0.097529 / 0.038508 (0.059021) | 0.079257 / 0.023109 (0.056147) | 0.356390 / 0.275898 (0.080492) | 0.390065 / 0.323480 (0.066585) | 0.006071 / 0.007986 (-0.001914) | 0.003783 / 0.004328 (-0.000546) | 0.074598 / 0.004250 (0.070348) | 0.059626 / 0.037052 (0.022574) | 0.395344 / 0.258489 (0.136855) | 0.418564 / 0.293841 (0.124723) | 0.041843 / 0.128546 (-0.086704) | 0.009293 / 0.075646 (-0.066354) | 0.332668 / 0.419271 (-0.086604) | 0.065753 / 0.043533 (0.022220) | 0.357285 / 0.255139 (0.102146) | 0.402974 / 0.283200 (0.119775) | 0.028714 / 0.141683 (-0.112968) | 1.733913 / 1.452155 (0.281759) | 1.802574 / 1.492716 (0.309858) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.253114 / 0.018006 (0.235108) | 0.606338 / 0.000490 (0.605848) | 0.006871 / 0.000200 (0.006671) | 0.000126 / 0.000054 (0.000072) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.031850 / 0.037411 (-0.005562) | 0.095148 / 0.014526 (0.080622) | 0.111499 / 0.176557 (-0.065057) | 0.174653 / 0.737135 (-0.562483) | 0.109396 / 0.296338 (-0.186943) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.440442 / 0.215209 (0.225233) | 4.408792 / 2.077655 (2.331137) | 2.149778 / 1.504120 (0.645658) | 1.922430 / 1.541195 (0.381235) | 2.029281 / 1.468490 (0.560791) | 0.611586 / 4.584777 (-3.973191) | 4.204571 / 3.745712 (0.458859) | 3.638194 / 5.269862 (-1.631668) | 2.336146 / 4.565676 (-2.229531) | 0.065383 / 0.424275 (-0.358892) | 0.008441 / 0.007607 (0.000834) | 0.527357 / 0.226044 (0.301313) | 5.247892 / 2.268929 (2.978963) | 2.654005 / 55.444624 (-52.790620) | 2.256596 / 6.876477 (-4.619881) | 2.432191 / 2.142072 (0.290119) | 0.672759 / 4.805227 (-4.132469) | 0.148494 / 6.500664 (-6.352170) | 0.068248 / 0.075469 (-0.007221) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.544250 / 1.841788 (-0.297538) | 21.882016 / 8.074308 (13.807708) | 16.470182 / 10.191392 (6.278790) | 0.166107 / 0.680424 (-0.514317) | 0.021305 / 0.534201 (-0.512896) | 0.445069 / 0.579283 (-0.134214) | 0.500631 / 0.434364 (0.066267) | 0.525801 / 0.540337 (-0.014536) | 0.806534 / 1.386936 (-0.580402) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007322 / 0.011353 (-0.004030) | 0.004206 / 0.011008 (-0.006802) | 0.074827 / 0.038508 (0.036319) | 0.084759 / 0.023109 (0.061650) | 0.421204 / 0.275898 (0.145306) | 0.464442 / 0.323480 (0.140962) | 0.006523 / 0.007986 (-0.001463) | 0.003613 / 0.004328 (-0.000716) | 0.073796 / 0.004250 (0.069545) | 0.066609 / 0.037052 (0.029557) | 0.430108 / 0.258489 (0.171619) | 0.463165 / 0.293841 (0.169324) | 0.036015 / 0.128546 (-0.092532) | 0.009696 / 0.075646 (-0.065951) | 0.083326 / 0.419271 (-0.335946) | 0.056804 / 0.043533 (0.013271) | 0.423333 / 0.255139 (0.168194) | 0.450538 / 0.283200 (0.167338) | 0.027067 / 0.141683 (-0.114616) | 1.700563 / 1.452155 (0.248408) | 1.748738 / 1.492716 (0.256021) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.395682 / 0.018006 (0.377675) | 0.540192 / 0.000490 (0.539702) | 0.140049 / 0.000200 (0.139849) | 0.000694 / 0.000054 (0.000639) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.036643 / 0.037411 (-0.000769) | 0.104422 / 0.014526 (0.089896) | 0.113072 / 0.176557 (-0.063484) | 0.179561 / 0.737135 (-0.557575) | 0.118620 / 0.296338 (-0.177718) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.476547 / 0.215209 (0.261338) | 4.716009 / 2.077655 (2.638354) | 2.412111 / 1.504120 (0.907991) | 2.246389 / 1.541195 (0.705194) | 2.307058 / 1.468490 (0.838568) | 0.552759 / 4.584777 (-4.032018) | 4.172484 / 3.745712 (0.426771) | 3.848419 / 5.269862 (-1.421443) | 2.310338 / 4.565676 (-2.255339) | 0.071757 / 0.424275 (-0.352518) | 0.011206 / 0.007607 (0.003599) | 0.609526 / 0.226044 (0.383482) | 5.583065 / 2.268929 (3.314136) | 3.081227 / 55.444624 (-52.363397) | 2.637782 / 6.876477 (-4.238695) | 2.887561 / 2.142072 (0.745489) | 0.667227 / 4.805227 (-4.138000) | 0.154421 / 6.500664 (-6.346243) | 0.070772 / 0.075469 (-0.004697) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.605500 / 1.841788 (-0.236288) | 22.872717 / 8.074308 (14.798409) | 15.865333 / 10.191392 (5.673941) | 0.170353 / 0.680424 (-0.510071) | 0.021854 / 0.534201 (-0.512347) | 0.461467 / 0.579283 (-0.117816) | 0.477743 / 0.434364 (0.043379) | 0.597234 / 0.540337 (0.056896) | 0.800416 / 1.386936 (-0.586520) |

</details>
</details>

![](https://cml.dev/watermark.png#a7f8d9019e7cb104eac4106bdc6ec0292f0dc61a "CML watermark")