Don't alter input in Features.from_dict
Opened by lhoestq (closed)


Comments:
_The documentation is not available anymore as the PR was closed or merged._

<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006166 / 0.011353 (-0.005187) | 0.003643 / 0.011008 (-0.007365) | 0.080966 / 0.038508 (0.042458) | 0.060538 / 0.023109 (0.037429) | 0.309205 / 0.275898 (0.033307) | 0.351007 / 0.323480 (0.027527) | 0.003592 / 0.007986 (-0.004393) | 0.002880 / 0.004328 (-0.001448) | 0.062957 / 0.004250 (0.058707) | 0.049015 / 0.037052 (0.011963) | 0.309436 / 0.258489 (0.050947) | 0.362695 / 0.293841 (0.068854) | 0.027818 / 0.128546 (-0.100728) | 0.008030 / 0.075646 (-0.067616) | 0.262678 / 0.419271 (-0.156594) | 0.046024 / 0.043533 (0.002491) | 0.316246 / 0.255139 (0.061107) | 0.337454 / 0.283200 (0.054254) | 0.022529 / 0.141683 (-0.119154) | 1.432492 / 1.452155 (-0.019662) | 1.499646 / 1.492716 (0.006929) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.190931 / 0.018006 (0.172925) | 0.428053 / 0.000490 (0.427564) | 0.002839 / 0.000200 (0.002639) | 0.000069 / 0.000054 (0.000014) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.024042 / 0.037411 (-0.013370) | 0.073952 / 0.014526 (0.059426) | 0.905973 / 0.176557 (0.729417) | 0.177767 / 0.737135 (-0.559368) | 0.125779 / 0.296338 (-0.170559) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.398997 / 0.215209 (0.183788) | 3.959575 / 2.077655 (1.881920) | 1.907038 / 1.504120 (0.402918) | 1.732908 / 1.541195 (0.191713) | 1.757038 / 1.468490 (0.288548) | 0.495917 / 4.584777 (-4.088860) | 3.021437 / 3.745712 (-0.724275) | 2.793960 / 5.269862 (-2.475901) | 1.827753 / 4.565676 (-2.737923) | 0.057143 / 0.424275 (-0.367132) | 0.006583 / 0.007607 (-0.001024) | 0.469402 / 0.226044 (0.243357) | 4.685623 / 2.268929 (2.416695) | 2.325200 / 55.444624 (-53.119424) | 1.985559 / 6.876477 (-4.890918) | 2.151208 / 2.142072 (0.009136) | 0.589498 / 4.805227 (-4.215730) | 0.125433 / 6.500664 (-6.375231) | 0.060834 / 0.075469 (-0.014636) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.228217 / 1.841788 (-0.613571) | 18.076089 / 8.074308 (10.001780) | 13.814460 / 10.191392 (3.623068) | 0.144674 / 0.680424 (-0.535750) | 0.016749 / 0.534201 (-0.517452) | 0.332839 / 0.579283 (-0.246444) | 0.357211 / 0.434364 (-0.077153) | 0.380367 / 0.540337 (-0.159971) | 0.531177 / 1.386936 (-0.855759) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006006 / 0.011353 (-0.005347) | 0.003552 / 0.011008 (-0.007456) | 0.061822 / 0.038508 (0.023313) | 0.057724 / 0.023109 (0.034615) | 0.462326 / 0.275898 (0.186428) | 0.492842 / 0.323480 (0.169362) | 0.004833 / 0.007986 (-0.003152) | 0.002847 / 0.004328 (-0.001481) | 0.062278 / 0.004250 (0.058028) | 0.046754 / 0.037052 (0.009702) | 0.464185 / 0.258489 (0.205696) | 0.496416 / 0.293841 (0.202576) | 0.028949 / 0.128546 (-0.099597) | 0.008038 / 0.075646 (-0.067608) | 0.067572 / 0.419271 (-0.351700) | 0.041176 / 0.043533 (-0.002356) | 0.460047 / 0.255139 (0.204908) | 0.482728 / 0.283200 (0.199528) | 0.020047 / 0.141683 (-0.121635) | 1.455958 / 1.452155 (0.003804) | 1.525730 / 1.492716 (0.033014) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.283643 / 0.018006 (0.265637) | 0.443046 / 0.000490 (0.442556) | 0.041019 / 0.000200 (0.040819) | 0.000340 / 0.000054 (0.000286) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.026229 / 0.037411 (-0.011182) | 0.081498 / 0.014526 (0.066972) | 0.091412 / 0.176557 (-0.085145) | 0.146621 / 0.737135 (-0.590514) | 0.092113 / 0.296338 (-0.204225) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.463525 / 0.215209 (0.248315) | 4.629852 / 2.077655 (2.552198) | 2.564831 / 1.504120 (1.060711) | 2.386976 / 1.541195 (0.845781) | 2.457757 / 1.468490 (0.989266) | 0.507317 / 4.584777 (-4.077460) | 3.142418 / 3.745712 (-0.603294) | 2.851642 / 5.269862 (-2.418219) | 1.894444 / 4.565676 (-2.671233) | 0.058495 / 0.424275 (-0.365780) | 0.006453 / 0.007607 (-0.001154) | 0.545363 / 0.226044 (0.319319) | 5.448092 / 2.268929 (3.179164) | 2.996328 / 55.444624 (-52.448296) | 2.664666 / 6.876477 (-4.211811) | 2.832247 / 2.142072 (0.690174) | 0.597631 / 4.805227 (-4.207596) | 0.126101 / 6.500664 (-6.374563) | 0.062573 / 0.075469 (-0.012896) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.366502 / 1.841788 (-0.475286) | 18.872990 / 8.074308 (10.798682) | 14.892114 / 10.191392 (4.700722) | 0.146668 / 0.680424 (-0.533756) | 0.017876 / 0.534201 (-0.516325) | 0.338490 / 0.579283 (-0.240793) | 0.357471 / 0.434364 (-0.076893) | 0.398730 / 0.540337 (-0.141608) | 0.542464 / 1.386936 (-0.844472) |

</details>
</details>

![](https://cml.dev/watermark.png#a6ff3e846d86814fa6962326e9346a4f1f1e8a80 "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.009132 / 0.011353 (-0.002221) | 0.005796 / 0.011008 (-0.005212) | 0.119495 / 0.038508 (0.080987) | 0.081708 / 0.023109 (0.058599) | 0.432940 / 0.275898 (0.157042) | 0.466793 / 0.323480 (0.143313) | 0.006464 / 0.007986 (-0.001521) | 0.004308 / 0.004328 (-0.000021) | 0.086344 / 0.004250 (0.082093) | 0.065987 / 0.037052 (0.028935) | 0.445213 / 0.258489 (0.186724) | 0.482405 / 0.293841 (0.188564) | 0.053553 / 0.128546 (-0.074993) | 0.015320 / 0.075646 (-0.060326) | 0.455669 / 0.419271 (0.036397) | 0.071619 / 0.043533 (0.028086) | 0.434843 / 0.255139 (0.179704) | 0.503224 / 0.283200 (0.220025) | 0.038280 / 0.141683 (-0.103403) | 1.901877 / 1.452155 (0.449722) | 2.040406 / 1.492716 (0.547690) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.268275 / 0.018006 (0.250269) | 0.622795 / 0.000490 (0.622305) | 0.004572 / 0.000200 (0.004372) | 0.000107 / 0.000054 (0.000053) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.032514 / 0.037411 (-0.004898) | 0.100619 / 0.014526 (0.086093) | 0.118407 / 0.176557 (-0.058149) | 0.190311 / 0.737135 (-0.546824) | 0.117160 / 0.296338 (-0.179178) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.629836 / 0.215209 (0.414627) | 6.236124 / 2.077655 (4.158470) | 2.750775 / 1.504120 (1.246655) | 2.380111 / 1.541195 (0.838916) | 2.487279 / 1.468490 (1.018789) | 0.849568 / 4.584777 (-3.735209) | 5.571308 / 3.745712 (1.825596) | 4.934114 / 5.269862 (-0.335747) | 3.205478 / 4.565676 (-1.360198) | 0.104804 / 0.424275 (-0.319471) | 0.009856 / 0.007607 (0.002248) | 0.753352 / 0.226044 (0.527308) | 7.523482 / 2.268929 (5.254554) | 3.660088 / 55.444624 (-51.784537) | 2.726493 / 6.876477 (-4.149984) | 3.011344 / 2.142072 (0.869271) | 1.093410 / 4.805227 (-3.711817) | 0.229758 / 6.500664 (-6.270906) | 0.081516 / 0.075469 (0.006047) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.700199 / 1.841788 (-0.141588) | 25.238736 / 8.074308 (17.164428) | 23.188131 / 10.191392 (12.996739) | 0.257862 / 0.680424 (-0.422562) | 0.028885 / 0.534201 (-0.505316) | 0.510693 / 0.579283 (-0.068590) | 0.648474 / 0.434364 (0.214110) | 0.576314 / 0.540337 (0.035976) | 0.800606 / 1.386936 (-0.586330) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.009426 / 0.011353 (-0.001927) | 0.006205 / 0.011008 (-0.004803) | 0.083947 / 0.038508 (0.045438) | 0.089164 / 0.023109 (0.066055) | 0.540500 / 0.275898 (0.264602) | 0.578825 / 0.323480 (0.255345) | 0.006792 / 0.007986 (-0.001194) | 0.005125 / 0.004328 (0.000797) | 0.083284 / 0.004250 (0.079034) | 0.067539 / 0.037052 (0.030487) | 0.544330 / 0.258489 (0.285841) | 0.593836 / 0.293841 (0.299995) | 0.050647 / 0.128546 (-0.077899) | 0.014688 / 0.075646 (-0.060959) | 0.095977 / 0.419271 (-0.323295) | 0.062326 / 0.043533 (0.018793) | 0.536096 / 0.255139 (0.280957) | 0.578691 / 0.283200 (0.295492) | 0.035488 / 0.141683 (-0.106194) | 1.911145 / 1.452155 (0.458990) | 1.977647 / 1.492716 (0.484931) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.368365 / 0.018006 (0.350359) | 0.609836 / 0.000490 (0.609346) | 0.054720 / 0.000200 (0.054520) | 0.000465 / 0.000054 (0.000411) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.036057 / 0.037411 (-0.001355) | 0.126434 / 0.014526 (0.111908) | 0.124740 / 0.176557 (-0.051817) | 0.198907 / 0.737135 (-0.538228) | 0.138201 / 0.296338 (-0.158137) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.684814 / 0.215209 (0.469605) | 6.738182 / 2.077655 (4.660527) | 3.231054 / 1.504120 (1.726934) | 2.889550 / 1.541195 (1.348355) | 2.933985 / 1.468490 (1.465495) | 0.867176 / 4.584777 (-3.717601) | 5.465475 / 3.745712 (1.719763) | 4.928370 / 5.269862 (-0.341492) | 3.126382 / 4.565676 (-1.439294) | 0.129673 / 0.424275 (-0.294603) | 0.009755 / 0.007607 (0.002148) | 0.797860 / 0.226044 (0.571816) | 8.003178 / 2.268929 (5.734250) | 4.081658 / 55.444624 (-51.362966) | 3.303837 / 6.876477 (-3.572640) | 3.574577 / 2.142072 (1.432505) | 1.064674 / 4.805227 (-3.740554) | 0.232894 / 6.500664 (-6.267770) | 0.082298 / 0.075469 (0.006829) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.858701 / 1.841788 (0.016913) | 25.839794 / 8.074308 (17.765485) | 24.291425 / 10.191392 (14.100033) | 0.250181 / 0.680424 (-0.430243) | 0.034479 / 0.534201 (-0.499722) | 0.540754 / 0.579283 (-0.038529) | 0.615996 / 0.434364 (0.181632) | 0.631499 / 0.540337 (0.091161) | 0.838719 / 1.386936 (-0.548217) |

</details>
</details>

![](https://cml.dev/watermark.png#0b6bb2f0e7a460d4ed04855eafe1184a7ce7c09c "CML watermark")