Allow self as key in `Features`
Opened by mariosasko (closed)

Fix #5641 

Comments:
_The documentation is not available anymore as the PR was closed or merged._

<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.009980 / 0.011353 (-0.001373) | 0.006643 / 0.011008 (-0.004366) | 0.140722 / 0.038508 (0.102214) | 0.036693 / 0.023109 (0.013584) | 0.430019 / 0.275898 (0.154121) | 0.463218 / 0.323480 (0.139738) | 0.006977 / 0.007986 (-0.001008) | 0.006488 / 0.004328 (0.002160) | 0.099385 / 0.004250 (0.095134) | 0.047160 / 0.037052 (0.010108) | 0.431440 / 0.258489 (0.172951) | 0.500232 / 0.293841 (0.206391) | 0.057968 / 0.128546 (-0.070578) | 0.020197 / 0.075646 (-0.055449) | 0.438269 / 0.419271 (0.018998) | 0.071149 / 0.043533 (0.027617) | 0.428502 / 0.255139 (0.173363) | 0.486861 / 0.283200 (0.203661) | 0.119855 / 0.141683 (-0.021828) | 1.875372 / 1.452155 (0.423218) | 1.955055 / 1.492716 (0.462339) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.243468 / 0.018006 (0.225462) | 0.547842 / 0.000490 (0.547352) | 0.004885 / 0.000200 (0.004685) | 0.000144 / 0.000054 (0.000089) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.031555 / 0.037411 (-0.005856) | 0.125869 / 0.014526 (0.111343) | 0.137816 / 0.176557 (-0.038741) | 0.206581 / 0.737135 (-0.530555) | 0.142976 / 0.296338 (-0.153362) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.624773 / 0.215209 (0.409564) | 6.154861 / 2.077655 (4.077206) | 2.504586 / 1.504120 (1.000466) | 1.989118 / 1.541195 (0.447923) | 2.092280 / 1.468490 (0.623790) | 1.240108 / 4.584777 (-3.344669) | 5.584893 / 3.745712 (1.839181) | 3.075369 / 5.269862 (-2.194492) | 2.174285 / 4.565676 (-2.391391) | 0.141555 / 0.424275 (-0.282720) | 0.016099 / 0.007607 (0.008492) | 0.720543 / 0.226044 (0.494498) | 7.489000 / 2.268929 (5.220071) | 3.239189 / 55.444624 (-52.205435) | 2.525772 / 6.876477 (-4.350704) | 2.773514 / 2.142072 (0.631441) | 1.410084 / 4.805227 (-3.395143) | 0.259252 / 6.500664 (-6.241412) | 0.082573 / 0.075469 (0.007104) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.458186 / 1.841788 (-0.383602) | 17.503738 / 8.074308 (9.429430) | 20.817682 / 10.191392 (10.626290) | 0.231221 / 0.680424 (-0.449203) | 0.032550 / 0.534201 (-0.501651) | 0.559020 / 0.579283 (-0.020263) | 0.592987 / 0.434364 (0.158623) | 0.602661 / 0.540337 (0.062324) | 0.731912 / 1.386936 (-0.655024) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.009543 / 0.011353 (-0.001810) | 0.006953 / 0.011008 (-0.004055) | 0.087651 / 0.038508 (0.049143) | 0.031717 / 0.023109 (0.008608) | 0.437813 / 0.275898 (0.161915) | 0.468448 / 0.323480 (0.144968) | 0.007378 / 0.007986 (-0.000607) | 0.005170 / 0.004328 (0.000842) | 0.102286 / 0.004250 (0.098035) | 0.043643 / 0.037052 (0.006591) | 0.458788 / 0.258489 (0.200299) | 0.519891 / 0.293841 (0.226050) | 0.052875 / 0.128546 (-0.075671) | 0.020518 / 0.075646 (-0.055128) | 0.112675 / 0.419271 (-0.306597) | 0.066390 / 0.043533 (0.022858) | 0.423037 / 0.255139 (0.167898) | 0.420345 / 0.283200 (0.137146) | 0.119221 / 0.141683 (-0.022462) | 1.632244 / 1.452155 (0.180090) | 1.829585 / 1.492716 (0.336869) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.242312 / 0.018006 (0.224305) | 0.547592 / 0.000490 (0.547102) | 0.006520 / 0.000200 (0.006320) | 0.000185 / 0.000054 (0.000131) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.032204 / 0.037411 (-0.005207) | 0.113320 / 0.014526 (0.098794) | 0.135667 / 0.176557 (-0.040889) | 0.194360 / 0.737135 (-0.542775) | 0.127934 / 0.296338 (-0.168404) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.648134 / 0.215209 (0.432925) | 6.470574 / 2.077655 (4.392920) | 2.799121 / 1.504120 (1.295001) | 2.160450 / 1.541195 (0.619255) | 2.261648 / 1.468490 (0.793158) | 1.244660 / 4.584777 (-3.340117) | 5.694636 / 3.745712 (1.948923) | 5.316191 / 5.269862 (0.046329) | 2.764551 / 4.565676 (-1.801126) | 0.152225 / 0.424275 (-0.272051) | 0.015959 / 0.007607 (0.008351) | 0.833606 / 0.226044 (0.607562) | 8.099765 / 2.268929 (5.830836) | 3.523005 / 55.444624 (-51.921620) | 2.855126 / 6.876477 (-4.021351) | 2.730849 / 2.142072 (0.588776) | 1.434351 / 4.805227 (-3.370876) | 0.251963 / 6.500664 (-6.248701) | 0.085718 / 0.075469 (0.010249) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.722466 / 1.841788 (-0.119322) | 17.846981 / 8.074308 (9.772673) | 21.578684 / 10.191392 (11.387292) | 0.239987 / 0.680424 (-0.440437) | 0.029189 / 0.534201 (-0.505012) | 0.543181 / 0.579283 (-0.036102) | 0.626527 / 0.434364 (0.192163) | 0.614334 / 0.540337 (0.073997) | 0.745934 / 1.386936 (-0.641002) |

</details>
</details>

![](https://cml.dev/watermark.png#4c506ad7cd22668f37ec51ff01b7c7f7235b9212 "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007395 / 0.011353 (-0.003958) | 0.004965 / 0.011008 (-0.006043) | 0.096376 / 0.038508 (0.057868) | 0.033243 / 0.023109 (0.010134) | 0.299990 / 0.275898 (0.024092) | 0.336287 / 0.323480 (0.012807) | 0.005528 / 0.007986 (-0.002458) | 0.004003 / 0.004328 (-0.000326) | 0.072820 / 0.004250 (0.068569) | 0.042867 / 0.037052 (0.005815) | 0.296719 / 0.258489 (0.038230) | 0.337313 / 0.293841 (0.043472) | 0.036809 / 0.128546 (-0.091738) | 0.012239 / 0.075646 (-0.063407) | 0.332351 / 0.419271 (-0.086921) | 0.050449 / 0.043533 (0.006916) | 0.301483 / 0.255139 (0.046344) | 0.316673 / 0.283200 (0.033474) | 0.102526 / 0.141683 (-0.039157) | 1.415429 / 1.452155 (-0.036726) | 1.544381 / 1.492716 (0.051665) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.211158 / 0.018006 (0.193152) | 0.434718 / 0.000490 (0.434228) | 0.003386 / 0.000200 (0.003186) | 0.000078 / 0.000054 (0.000024) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.027945 / 0.037411 (-0.009466) | 0.108743 / 0.014526 (0.094217) | 0.119771 / 0.176557 (-0.056785) | 0.178667 / 0.737135 (-0.558468) | 0.123718 / 0.296338 (-0.172620) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.413908 / 0.215209 (0.198699) | 4.136828 / 2.077655 (2.059174) | 1.932547 / 1.504120 (0.428427) | 1.715389 / 1.541195 (0.174194) | 1.791679 / 1.468490 (0.323189) | 0.692715 / 4.584777 (-3.892062) | 3.741807 / 3.745712 (-0.003905) | 2.066274 / 5.269862 (-3.203587) | 1.314106 / 4.565676 (-3.251570) | 0.087191 / 0.424275 (-0.337084) | 0.012866 / 0.007607 (0.005259) | 0.510012 / 0.226044 (0.283968) | 5.116419 / 2.268929 (2.847490) | 2.408562 / 55.444624 (-53.036063) | 2.002044 / 6.876477 (-4.874433) | 2.121868 / 2.142072 (-0.020204) | 0.837141 / 4.805227 (-3.968086) | 0.166596 / 6.500664 (-6.334068) | 0.063190 / 0.075469 (-0.012279) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.204152 / 1.841788 (-0.637636) | 14.739793 / 8.074308 (6.665485) | 14.403469 / 10.191392 (4.212077) | 0.165781 / 0.680424 (-0.514642) | 0.017826 / 0.534201 (-0.516375) | 0.423527 / 0.579283 (-0.155756) | 0.431410 / 0.434364 (-0.002954) | 0.499422 / 0.540337 (-0.040915) | 0.596116 / 1.386936 (-0.790820) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007365 / 0.011353 (-0.003988) | 0.005165 / 0.011008 (-0.005844) | 0.073403 / 0.038508 (0.034895) | 0.032542 / 0.023109 (0.009433) | 0.339304 / 0.275898 (0.063406) | 0.371892 / 0.323480 (0.048412) | 0.005544 / 0.007986 (-0.002442) | 0.004108 / 0.004328 (-0.000221) | 0.073750 / 0.004250 (0.069500) | 0.045613 / 0.037052 (0.008561) | 0.366159 / 0.258489 (0.107670) | 0.389864 / 0.293841 (0.096023) | 0.036006 / 0.128546 (-0.092540) | 0.012402 / 0.075646 (-0.063244) | 0.085137 / 0.419271 (-0.334135) | 0.048485 / 0.043533 (0.004952) | 0.334172 / 0.255139 (0.079033) | 0.353168 / 0.283200 (0.069969) | 0.099393 / 0.141683 (-0.042290) | 1.460584 / 1.452155 (0.008429) | 1.518601 / 1.492716 (0.025885) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.227352 / 0.018006 (0.209346) | 0.444211 / 0.000490 (0.443721) | 0.000410 / 0.000200 (0.000210) | 0.000060 / 0.000054 (0.000005) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.029517 / 0.037411 (-0.007894) | 0.115557 / 0.014526 (0.101031) | 0.125855 / 0.176557 (-0.050701) | 0.175214 / 0.737135 (-0.561922) | 0.129324 / 0.296338 (-0.167014) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.429783 / 0.215209 (0.214574) | 4.301159 / 2.077655 (2.223504) | 2.084939 / 1.504120 (0.580819) | 1.887781 / 1.541195 (0.346586) | 2.045712 / 1.468490 (0.577222) | 0.693319 / 4.584777 (-3.891458) | 3.788595 / 3.745712 (0.042883) | 2.087080 / 5.269862 (-3.182781) | 1.325247 / 4.565676 (-3.240429) | 0.085919 / 0.424275 (-0.338356) | 0.012710 / 0.007607 (0.005103) | 0.533432 / 0.226044 (0.307387) | 5.339468 / 2.268929 (3.070540) | 2.578351 / 55.444624 (-52.866273) | 2.224905 / 6.876477 (-4.651572) | 2.301064 / 2.142072 (0.158992) | 0.839622 / 4.805227 (-3.965605) | 0.166523 / 6.500664 (-6.334141) | 0.065254 / 0.075469 (-0.010215) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.262223 / 1.841788 (-0.579565) | 15.042523 / 8.074308 (6.968215) | 14.542719 / 10.191392 (4.351327) | 0.142230 / 0.680424 (-0.538194) | 0.017610 / 0.534201 (-0.516591) | 0.422357 / 0.579283 (-0.156926) | 0.417785 / 0.434364 (-0.016579) | 0.491990 / 0.540337 (-0.048348) | 0.585835 / 1.386936 (-0.801101) |

</details>
</details>

![](https://cml.dev/watermark.png#c2fcedd2a561fe6f5b6972ad18bfef722e1d2c77 "CML watermark")