Fix checking patterns to infer packaged builder 
Opened by polinaeterna (closed)

Don't ignore results of pattern resolving if `self.data_files` is not None. Otherwise lines 854 and 1037 make no sense.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

oh wow good catch

<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.006681 / 0.011353 (-0.004672) | 0.003967 / 0.011008 (-0.007041) | 0.085590 / 0.038508 (0.047082) | 0.079285 / 0.023109 (0.056176) | 0.311583 / 0.275898 (0.035685) | 0.345578 / 0.323480 (0.022098) | 0.004115 / 0.007986 (-0.003871) | 0.004286 / 0.004328 (-0.000043) | 0.064405 / 0.004250 (0.060155) | 0.055084 / 0.037052 (0.018032) | 0.316117 / 0.258489 (0.057628) | 0.354737 / 0.293841 (0.060896) | 0.031280 / 0.128546 (-0.097266) | 0.008395 / 0.075646 (-0.067251) | 0.288910 / 0.419271 (-0.130362) | 0.051291 / 0.043533 (0.007759) | 0.309125 / 0.255139 (0.053986) | 0.349673 / 0.283200 (0.066473) | 0.025016 / 0.141683 (-0.116667) | 1.475577 / 1.452155 (0.023422) | 1.558967 / 1.492716 (0.066251) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.208504 / 0.018006 (0.190498) | 0.462270 / 0.000490 (0.461780) | 0.003476 / 0.000200 (0.003276) | 0.000073 / 0.000054 (0.000018) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.030371 / 0.037411 (-0.007041) | 0.086157 / 0.014526 (0.071631) | 0.098162 / 0.176557 (-0.078395) | 0.154649 / 0.737135 (-0.582486) | 0.098697 / 0.296338 (-0.197642) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.405883 / 0.215209 (0.190674) | 4.049614 / 2.077655 (1.971959) | 2.075047 / 1.504120 (0.570927) | 1.917782 / 1.541195 (0.376587) | 2.030268 / 1.468490 (0.561778) | 0.483974 / 4.584777 (-4.100803) | 3.542147 / 3.745712 (-0.203566) | 3.305999 / 5.269862 (-1.963863) | 2.052287 / 4.565676 (-2.513390) | 0.057246 / 0.424275 (-0.367029) | 0.007631 / 0.007607 (0.000024) | 0.488189 / 0.226044 (0.262144) | 4.884784 / 2.268929 (2.615856) | 2.576304 / 55.444624 (-52.868320) | 2.241249 / 6.876477 (-4.635228) | 2.490512 / 2.142072 (0.348440) | 0.584495 / 4.805227 (-4.220733) | 0.134741 / 6.500664 (-6.365923) | 0.061639 / 0.075469 (-0.013830) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.317717 / 1.841788 (-0.524071) | 20.098594 / 8.074308 (12.024286) | 14.641051 / 10.191392 (4.449659) | 0.165291 / 0.680424 (-0.515133) | 0.019179 / 0.534201 (-0.515022) | 0.399506 / 0.579283 (-0.179777) | 0.407662 / 0.434364 (-0.026701) | 0.457965 / 0.540337 (-0.082372) | 0.626401 / 1.386936 (-0.760536) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007076 / 0.011353 (-0.004277) | 0.004125 / 0.011008 (-0.006884) | 0.064861 / 0.038508 (0.026353) | 0.082390 / 0.023109 (0.059281) | 0.423227 / 0.275898 (0.147329) | 0.452229 / 0.323480 (0.128750) | 0.005594 / 0.007986 (-0.002392) | 0.003465 / 0.004328 (-0.000863) | 0.064661 / 0.004250 (0.060411) | 0.057945 / 0.037052 (0.020892) | 0.424572 / 0.258489 (0.166083) | 0.465349 / 0.293841 (0.171509) | 0.032687 / 0.128546 (-0.095859) | 0.008573 / 0.075646 (-0.067074) | 0.073020 / 0.419271 (-0.346251) | 0.048423 / 0.043533 (0.004891) | 0.413425 / 0.255139 (0.158286) | 0.433778 / 0.283200 (0.150578) | 0.023942 / 0.141683 (-0.117741) | 1.495190 / 1.452155 (0.043036) | 1.586526 / 1.492716 (0.093810) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.271805 / 0.018006 (0.253799) | 0.454922 / 0.000490 (0.454432) | 0.015386 / 0.000200 (0.015186) | 0.000129 / 0.000054 (0.000074) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.033804 / 0.037411 (-0.003607) | 0.099317 / 0.014526 (0.084791) | 0.107207 / 0.176557 (-0.069349) | 0.160926 / 0.737135 (-0.576210) | 0.108669 / 0.296338 (-0.187670) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.430776 / 0.215209 (0.215567) | 4.297622 / 2.077655 (2.219967) | 2.285918 / 1.504120 (0.781798) | 2.109608 / 1.541195 (0.568413) | 2.208326 / 1.468490 (0.739836) | 0.490016 / 4.584777 (-4.094761) | 3.570609 / 3.745712 (-0.175103) | 3.406335 / 5.269862 (-1.863526) | 2.070664 / 4.565676 (-2.495012) | 0.058089 / 0.424275 (-0.366186) | 0.007425 / 0.007607 (-0.000182) | 0.506972 / 0.226044 (0.280927) | 5.078643 / 2.268929 (2.809714) | 2.858973 / 55.444624 (-52.585651) | 2.457344 / 6.876477 (-4.419132) | 2.687727 / 2.142072 (0.545654) | 0.592134 / 4.805227 (-4.213093) | 0.133966 / 6.500664 (-6.366698) | 0.061800 / 0.075469 (-0.013669) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.337167 / 1.841788 (-0.504620) | 20.743951 / 8.074308 (12.669643) | 15.402686 / 10.191392 (5.211294) | 0.164548 / 0.680424 (-0.515876) | 0.020244 / 0.534201 (-0.513957) | 0.399044 / 0.579283 (-0.180239) | 0.414036 / 0.434364 (-0.020328) | 0.474141 / 0.540337 (-0.066197) | 0.654455 / 1.386936 (-0.732482) |

</details>
</details>

![](https://cml.dev/watermark.png#4de930c45a81a6dff1805bf45f59170e9f953eeb "CML watermark")