Do not use Parquet exports if revision is passed
State: closed | Author: albertvillanova (MEMBER)

Fix #6554.

Comments: The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6555). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

As shared on slack, `HubDatasetModuleFactoryWithParquetExport` raises a `DatasetsServerError` already if the user tries to load another revision that the one from the parquet export. And therefore it fall backs on using `HubDatasetModuleFactoryWithScript`

@lhoestq I would say that although current implementation finally returns `HubDatasetModuleFactoryWithScript` as expected, with this PR we avoid the useless call to `HubDatasetModuleFactoryWithParquetExport.get_module`, so this is more optimal.

<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005596 / 0.011353 (-0.005757) | 0.004022 / 0.011008 (-0.006986) | 0.064041 / 0.038508 (0.025533) | 0.030683 / 0.023109 (0.007574) | 0.245236 / 0.275898 (-0.030662) | 0.269657 / 0.323480 (-0.053823) | 0.003142 / 0.007986 (-0.004844) | 0.002821 / 0.004328 (-0.001507) | 0.048774 / 0.004250 (0.044523) | 0.043771 / 0.037052 (0.006719) | 0.258202 / 0.258489 (-0.000287) | 0.288381 / 0.293841 (-0.005460) | 0.028154 / 0.128546 (-0.100392) | 0.011071 / 0.075646 (-0.064576) | 0.209836 / 0.419271 (-0.209436) | 0.035923 / 0.043533 (-0.007609) | 0.248361 / 0.255139 (-0.006777) | 0.268728 / 0.283200 (-0.014472) | 0.019982 / 0.141683 (-0.121701) | 1.172330 / 1.452155 (-0.279824) | 1.192262 / 1.492716 (-0.300455) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.089231 / 0.018006 (0.071225) | 0.299192 / 0.000490 (0.298702) | 0.000214 / 0.000200 (0.000014) | 0.000054 / 0.000054 (-0.000001) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.018358 / 0.037411 (-0.019053) | 0.062633 / 0.014526 (0.048107) | 0.076276 / 0.176557 (-0.100280) | 0.120862 / 0.737135 (-0.616274) | 0.075958 / 0.296338 (-0.220380) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.291575 / 0.215209 (0.076366) | 2.855908 / 2.077655 (0.778253) | 1.459891 / 1.504120 (-0.044229) | 1.374945 / 1.541195 (-0.166250) | 1.333759 / 1.468490 (-0.134731) | 0.575428 / 4.584777 (-4.009348) | 2.414253 / 3.745712 (-1.331459) | 2.768222 / 5.269862 (-2.501639) | 1.705005 / 4.565676 (-2.860672) | 0.063406 / 0.424275 (-0.360869) | 0.004981 / 0.007607 (-0.002626) | 0.343826 / 0.226044 (0.117781) | 3.418143 / 2.268929 (1.149215) | 1.856571 / 55.444624 (-53.588053) | 1.571318 / 6.876477 (-5.305159) | 1.609897 / 2.142072 (-0.532175) | 0.646779 / 4.805227 (-4.158448) | 0.118143 / 6.500664 (-6.382521) | 0.042408 / 0.075469 (-0.033061) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.965091 / 1.841788 (-0.876697) | 11.569655 / 8.074308 (3.495347) | 10.587818 / 10.191392 (0.396426) | 0.128518 / 0.680424 (-0.551905) | 0.013954 / 0.534201 (-0.520247) | 0.287244 / 0.579283 (-0.292039) | 0.263755 / 0.434364 (-0.170609) | 0.321661 / 0.540337 (-0.218676) | 0.428753 / 1.386936 (-0.958183) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.005568 / 0.011353 (-0.005785) | 0.003755 / 0.011008 (-0.007253) | 0.049134 / 0.038508 (0.010626) | 0.032113 / 0.023109 (0.009004) | 0.276645 / 0.275898 (0.000747) | 0.299240 / 0.323480 (-0.024240) | 0.004297 / 0.007986 (-0.003689) | 0.002727 / 0.004328 (-0.001602) | 0.048420 / 0.004250 (0.044170) | 0.045070 / 0.037052 (0.008017) | 0.288597 / 0.258489 (0.030108) | 0.320824 / 0.293841 (0.026983) | 0.053293 / 0.128546 (-0.075253) | 0.011002 / 0.075646 (-0.064644) | 0.057747 / 0.419271 (-0.361524) | 0.034389 / 0.043533 (-0.009143) | 0.277914 / 0.255139 (0.022775) | 0.292919 / 0.283200 (0.009719) | 0.018252 / 0.141683 (-0.123431) | 1.187245 / 1.452155 (-0.264910) | 1.199823 / 1.492716 (-0.292893) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.088338 / 0.018006 (0.070332) | 0.297498 / 0.000490 (0.297008) | 0.000206 / 0.000200 (0.000006) | 0.000048 / 0.000054 (-0.000007) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.021445 / 0.037411 (-0.015966) | 0.075522 / 0.014526 (0.060996) | 0.086010 / 0.176557 (-0.090546) | 0.124938 / 0.737135 (-0.612197) | 0.087542 / 0.296338 (-0.208796) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.292460 / 0.215209 (0.077251) | 2.841290 / 2.077655 (0.763635) | 1.537941 / 1.504120 (0.033821) | 1.409903 / 1.541195 (-0.131291) | 1.435339 / 1.468490 (-0.033151) | 0.578967 / 4.584777 (-4.005810) | 2.398588 / 3.745712 (-1.347125) | 2.662342 / 5.269862 (-2.607520) | 1.743055 / 4.565676 (-2.822622) | 0.064043 / 0.424275 (-0.360232) | 0.005030 / 0.007607 (-0.002577) | 0.348542 / 0.226044 (0.122498) | 3.395854 / 2.268929 (1.126926) | 1.918935 / 55.444624 (-53.525689) | 1.639320 / 6.876477 (-5.237157) | 1.740406 / 2.142072 (-0.401666) | 0.653346 / 4.805227 (-4.151881) | 0.117298 / 6.500664 (-6.383366) | 0.040635 / 0.075469 (-0.034834) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.008277 / 1.841788 (-0.833510) | 12.069369 / 8.074308 (3.995061) | 10.967322 / 10.191392 (0.775930) | 0.131938 / 0.680424 (-0.548486) | 0.015418 / 0.534201 (-0.518783) | 0.297257 / 0.579283 (-0.282026) | 0.270742 / 0.434364 (-0.163622) | 0.332296 / 0.540337 (-0.208042) | 0.421606 / 1.386936 (-0.965330) |

</details>
</details>

![](https://cml.dev/watermark.png#8f22ec79a1ce4fbf0a1728d53f0338d5fdf664d8 "CML watermark")