Pin sqlalchemy
State: closed | Author: lhoestq (MEMBER)

since sqlalchemy update to 2.0.0 the CI started to fail: https://github.com/huggingface/datasets/actions/runs/4023742457/jobs/6914976514

the error comes from pandas: https://github.com/pandas-dev/pandas/issues/51015

Comments: _The documentation is not available anymore as the PR was closed or merged._

<details>
<summary>Show benchmarks</summary>

PyArrow==6.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.012442 / 0.011353 (0.001089) | 0.006274 / 0.011008 (-0.004734) | 0.128249 / 0.038508 (0.089741) | 0.040117 / 0.023109 (0.017008) | 0.383725 / 0.275898 (0.107827) | 0.510494 / 0.323480 (0.187014) | 0.009037 / 0.007986 (0.001051) | 0.008256 / 0.004328 (0.003927) | 0.105329 / 0.004250 (0.101079) | 0.046909 / 0.037052 (0.009857) | 0.401980 / 0.258489 (0.143491) | 0.461332 / 0.293841 (0.167491) | 0.065629 / 0.128546 (-0.062917) | 0.020043 / 0.075646 (-0.055604) | 0.453773 / 0.419271 (0.034501) | 0.063456 / 0.043533 (0.019923) | 0.384458 / 0.255139 (0.129319) | 0.449699 / 0.283200 (0.166499) | 0.118197 / 0.141683 (-0.023486) | 1.915080 / 1.452155 (0.462925) | 1.957132 / 1.492716 (0.464416) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.209657 / 0.018006 (0.191651) | 0.592478 / 0.000490 (0.591988) | 0.004137 / 0.000200 (0.003937) | 0.000124 / 0.000054 (0.000069) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.029607 / 0.037411 (-0.007804) | 0.129559 / 0.014526 (0.115033) | 0.148326 / 0.176557 (-0.028231) | 0.190506 / 0.737135 (-0.546629) | 0.143177 / 0.296338 (-0.153162) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.626166 / 0.215209 (0.410957) | 6.612680 / 2.077655 (4.535026) | 2.432354 / 1.504120 (0.928234) | 2.051482 / 1.541195 (0.510287) | 2.055822 / 1.468490 (0.587332) | 1.210099 / 4.584777 (-3.374678) | 5.498117 / 3.745712 (1.752405) | 3.054838 / 5.269862 (-2.215024) | 2.182875 / 4.565676 (-2.382802) | 0.144518 / 0.424275 (-0.279757) | 0.014132 / 0.007607 (0.006525) | 0.801805 / 0.226044 (0.575761) | 7.911235 / 2.268929 (5.642307) | 3.372762 / 55.444624 (-52.071862) | 2.517266 / 6.876477 (-4.359210) | 2.515329 / 2.142072 (0.373256) | 1.501731 / 4.805227 (-3.303497) | 0.252569 / 6.500664 (-6.248096) | 0.080987 / 0.075469 (0.005518) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.709880 / 1.841788 (-0.131907) | 18.640340 / 8.074308 (10.566032) | 23.560908 / 10.191392 (13.369516) | 0.265680 / 0.680424 (-0.414744) | 0.046438 / 0.534201 (-0.487763) | 0.571973 / 0.579283 (-0.007310) | 0.642425 / 0.434364 (0.208061) | 0.698167 / 0.540337 (0.157830) | 0.842132 / 1.386936 (-0.544804) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.009268 / 0.011353 (-0.002085) | 0.006052 / 0.011008 (-0.004956) | 0.133448 / 0.038508 (0.094939) | 0.034417 / 0.023109 (0.011308) | 0.435573 / 0.275898 (0.159675) | 0.479642 / 0.323480 (0.156162) | 0.008016 / 0.007986 (0.000030) | 0.006616 / 0.004328 (0.002288) | 0.106256 / 0.004250 (0.102005) | 0.048995 / 0.037052 (0.011942) | 0.450056 / 0.258489 (0.191567) | 0.511027 / 0.293841 (0.217187) | 0.052928 / 0.128546 (-0.075618) | 0.020824 / 0.075646 (-0.054822) | 0.450105 / 0.419271 (0.030834) | 0.062729 / 0.043533 (0.019196) | 0.438887 / 0.255139 (0.183748) | 0.468732 / 0.283200 (0.185532) | 0.116101 / 0.141683 (-0.025582) | 1.909689 / 1.452155 (0.457534) | 2.042007 / 1.492716 (0.549291) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.198265 / 0.018006 (0.180259) | 0.541799 / 0.000490 (0.541309) | 0.003938 / 0.000200 (0.003738) | 0.000116 / 0.000054 (0.000062) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.035933 / 0.037411 (-0.001478) | 0.130754 / 0.014526 (0.116229) | 0.146143 / 0.176557 (-0.030414) | 0.202042 / 0.737135 (-0.535094) | 0.155648 / 0.296338 (-0.140691) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.691123 / 0.215209 (0.475914) | 6.708370 / 2.077655 (4.630715) | 2.957120 / 1.504120 (1.453000) | 2.558350 / 1.541195 (1.017155) | 2.611271 / 1.468490 (1.142781) | 1.327355 / 4.584777 (-3.257422) | 5.755975 / 3.745712 (2.010263) | 3.295556 / 5.269862 (-1.974305) | 2.159831 / 4.565676 (-2.405845) | 0.161409 / 0.424275 (-0.262866) | 0.015470 / 0.007607 (0.007863) | 0.840611 / 0.226044 (0.614567) | 8.550064 / 2.268929 (6.281136) | 3.832013 / 55.444624 (-51.612612) | 3.032909 / 6.876477 (-3.843568) | 3.155651 / 2.142072 (1.013578) | 1.612486 / 4.805227 (-3.192741) | 0.273789 / 6.500664 (-6.226875) | 0.085618 / 0.075469 (0.010149) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.808376 / 1.841788 (-0.033412) | 18.267614 / 8.074308 (10.193306) | 21.047679 / 10.191392 (10.856286) | 0.259089 / 0.680424 (-0.421335) | 0.029211 / 0.534201 (-0.504990) | 0.556303 / 0.579283 (-0.022980) | 0.625264 / 0.434364 (0.190900) | 0.680814 / 0.540337 (0.140476) | 0.810146 / 1.386936 (-0.576790) |

</details>
</details>

![](https://cml.dev/watermark.png#20ea76c80e07acad78cf67198a4046a982feda21 "CML watermark")


<details>
<summary>Show benchmarks</summary>

PyArrow==6.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.008779 / 0.011353 (-0.002574) | 0.004644 / 0.011008 (-0.006364) | 0.099814 / 0.038508 (0.061306) | 0.029830 / 0.023109 (0.006721) | 0.299159 / 0.275898 (0.023261) | 0.354815 / 0.323480 (0.031335) | 0.006968 / 0.007986 (-0.001018) | 0.003521 / 0.004328 (-0.000808) | 0.077687 / 0.004250 (0.073437) | 0.035019 / 0.037052 (-0.002034) | 0.309548 / 0.258489 (0.051059) | 0.345228 / 0.293841 (0.051387) | 0.033644 / 0.128546 (-0.094902) | 0.011564 / 0.075646 (-0.064083) | 0.321835 / 0.419271 (-0.097437) | 0.041798 / 0.043533 (-0.001735) | 0.298190 / 0.255139 (0.043051) | 0.328874 / 0.283200 (0.045674) | 0.088175 / 0.141683 (-0.053508) | 1.481755 / 1.452155 (0.029600) | 1.503085 / 1.492716 (0.010369) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.170930 / 0.018006 (0.152924) | 0.422155 / 0.000490 (0.421666) | 0.001708 / 0.000200 (0.001509) | 0.000083 / 0.000054 (0.000028) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.022588 / 0.037411 (-0.014824) | 0.095775 / 0.014526 (0.081249) | 0.103939 / 0.176557 (-0.072618) | 0.138441 / 0.737135 (-0.598694) | 0.107896 / 0.296338 (-0.188442) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.418243 / 0.215209 (0.203034) | 4.171432 / 2.077655 (2.093777) | 1.906029 / 1.504120 (0.401909) | 1.698174 / 1.541195 (0.156979) | 1.748339 / 1.468490 (0.279849) | 0.691026 / 4.584777 (-3.893751) | 3.393354 / 3.745712 (-0.352358) | 2.722412 / 5.269862 (-2.547450) | 1.462439 / 4.565676 (-3.103238) | 0.084713 / 0.424275 (-0.339562) | 0.012131 / 0.007607 (0.004524) | 0.522153 / 0.226044 (0.296109) | 5.197916 / 2.268929 (2.928988) | 2.314270 / 55.444624 (-53.130354) | 1.986599 / 6.876477 (-4.889878) | 2.012757 / 2.142072 (-0.129315) | 0.802540 / 4.805227 (-4.002687) | 0.148673 / 6.500664 (-6.351991) | 0.065924 / 0.075469 (-0.009545) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.263790 / 1.841788 (-0.577998) | 13.874784 / 8.074308 (5.800476) | 13.842276 / 10.191392 (3.650884) | 0.149002 / 0.680424 (-0.531422) | 0.028550 / 0.534201 (-0.505651) | 0.396913 / 0.579283 (-0.182370) | 0.401543 / 0.434364 (-0.032821) | 0.473754 / 0.540337 (-0.066583) | 0.560455 / 1.386936 (-0.826481) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006724 / 0.011353 (-0.004629) | 0.004507 / 0.011008 (-0.006502) | 0.098447 / 0.038508 (0.059939) | 0.027888 / 0.023109 (0.004779) | 0.428956 / 0.275898 (0.153058) | 0.451557 / 0.323480 (0.128077) | 0.005056 / 0.007986 (-0.002929) | 0.003363 / 0.004328 (-0.000965) | 0.075990 / 0.004250 (0.071740) | 0.038688 / 0.037052 (0.001635) | 0.421550 / 0.258489 (0.163061) | 0.459480 / 0.293841 (0.165639) | 0.031408 / 0.128546 (-0.097138) | 0.011559 / 0.075646 (-0.064088) | 0.320054 / 0.419271 (-0.099217) | 0.041917 / 0.043533 (-0.001616) | 0.420878 / 0.255139 (0.165739) | 0.444813 / 0.283200 (0.161613) | 0.090409 / 0.141683 (-0.051274) | 1.490058 / 1.452155 (0.037904) | 1.645206 / 1.492716 (0.152489) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.221105 / 0.018006 (0.203099) | 0.407537 / 0.000490 (0.407047) | 0.000410 / 0.000200 (0.000210) | 0.000059 / 0.000054 (0.000004) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.024658 / 0.037411 (-0.012754) | 0.099230 / 0.014526 (0.084705) | 0.107788 / 0.176557 (-0.068769) | 0.143040 / 0.737135 (-0.594096) | 0.109440 / 0.296338 (-0.186899) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.453303 / 0.215209 (0.238094) | 4.520376 / 2.077655 (2.442722) | 2.133909 / 1.504120 (0.629789) | 1.926996 / 1.541195 (0.385801) | 2.019870 / 1.468490 (0.551380) | 0.707423 / 4.584777 (-3.877354) | 3.391903 / 3.745712 (-0.353809) | 1.860661 / 5.269862 (-3.409201) | 1.159940 / 4.565676 (-3.405736) | 0.083773 / 0.424275 (-0.340502) | 0.012228 / 0.007607 (0.004621) | 0.554666 / 0.226044 (0.328622) | 5.567564 / 2.268929 (3.298636) | 2.636718 / 55.444624 (-52.807907) | 2.240215 / 6.876477 (-4.636262) | 2.218951 / 2.142072 (0.076879) | 0.817167 / 4.805227 (-3.988060) | 0.151633 / 6.500664 (-6.349032) | 0.066515 / 0.075469 (-0.008954) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.296665 / 1.841788 (-0.545123) | 13.997898 / 8.074308 (5.923590) | 13.286607 / 10.191392 (3.095215) | 0.148906 / 0.680424 (-0.531518) | 0.016600 / 0.534201 (-0.517601) | 0.377459 / 0.579283 (-0.201824) | 0.379938 / 0.434364 (-0.054426) | 0.461628 / 0.540337 (-0.078709) | 0.550592 / 1.386936 (-0.836344) |

</details>
</details>

![](https://cml.dev/watermark.png#053f51a3e2adb762236eb29dd02791307f45f02f "CML watermark")