Title: Make get_from_cache use custom temp filename that is locked
State: closed

This PR ensures that the temporary filename created is the same as the one that is locked, while writing to the cache.

This PR stops using `tempfile` to generate the temporary filename.

Additionally, the behavior now is aligned for both `resume_download` `True` and `False`.

Refactor temp_file_manager so that it uses the filename that is locked: 
- Use: `cache_path + ".incomplete"`, when the locked one is `cache_path + ".lock"`

Before it was using `tempfile` inside `cache_dir`, which was not locked: although very improbable name collision (8 random characters), this was not impossible when huge number of multiple processes.

Maybe related to "Stale file handle" issues caused by `tempfile`: 
- [ ] https://huggingface.co/datasets/tapaco/discussions/4
- [ ] https://huggingface.co/datasets/xcsr/discussions/1
- [ ] https://huggingface.co/datasets/covost2/discussions/3
```
Error code:   ConfigNamesError
Exception:    OSError
Message:      [Errno 116] Stale file handle
Traceback:    Traceback (most recent call last):
                File "/src/services/worker/src/worker/job_runners/dataset/config_names.py", line 61, in compute_config_names_response
                  for config in sorted(get_dataset_config_names(path=dataset, use_auth_token=use_auth_token))
                File "/src/services/worker/.venv/lib/python3.9/site-packages/datasets/inspect.py", line 323, in get_dataset_config_names
                  dataset_module = dataset_module_factory(
                File "/src/services/worker/.venv/lib/python3.9/site-packages/datasets/load.py", line 1219, in dataset_module_factory
                  raise e1 from None
                File "/src/services/worker/.venv/lib/python3.9/site-packages/datasets/load.py", line 1188, in dataset_module_factory
                  return HubDatasetModuleFactoryWithScript(
                File "/src/services/worker/.venv/lib/python3.9/site-packages/datasets/load.py", line 907, in get_module
                  dataset_readme_path = self.download_dataset_readme_file()
                File "/src/services/worker/.venv/lib/python3.9/site-packages/datasets/load.py", line 896, in download_dataset_readme_file
                  return cached_path(
                File "/src/services/worker/.venv/lib/python3.9/site-packages/datasets/utils/file_utils.py", line 183, in cached_path
                  output_path = get_from_cache(
                File "/src/services/worker/.venv/lib/python3.9/site-packages/datasets/utils/file_utils.py", line 611, in get_from_cache
                  http_get(
                File "/usr/local/lib/python3.9/tempfile.py", line 496, in __exit__
                  result = self.file.__exit__(exc, value, tb)
              OSError: [Errno 116] Stale file handle
```
- the stale file handle error can be raised when `tempfile` tries to close (when exiting its context manager) a filename that has been already closed by other process
  - note that `tempfile` filenames are randomly generated but not locked in our code

CC: @severo 

Comments:
_The documentation is not available anymore as the PR was closed or merged._

<details>
<summary>Show benchmarks</summary>

PyArrow==8.0.0

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007241 / 0.011353 (-0.004112) | 0.004574 / 0.011008 (-0.006434) | 0.120481 / 0.038508 (0.081973) | 0.040492 / 0.023109 (0.017383) | 0.391399 / 0.275898 (0.115501) | 0.422844 / 0.323480 (0.099365) | 0.004441 / 0.007986 (-0.003545) | 0.004544 / 0.004328 (0.000216) | 0.089482 / 0.004250 (0.085231) | 0.052939 / 0.037052 (0.015887) | 0.393649 / 0.258489 (0.135160) | 0.433852 / 0.293841 (0.140011) | 0.035882 / 0.128546 (-0.092664) | 0.010172 / 0.075646 (-0.065474) | 0.410331 / 0.419271 (-0.008940) | 0.061481 / 0.043533 (0.017948) | 0.405066 / 0.255139 (0.149927) | 0.417732 / 0.283200 (0.134532) | 0.121647 / 0.141683 (-0.020035) | 1.790624 / 1.452155 (0.338469) | 1.863398 / 1.492716 (0.370681) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.250650 / 0.018006 (0.232644) | 0.489044 / 0.000490 (0.488554) | 0.010421 / 0.000200 (0.010222) | 0.000106 / 0.000054 (0.000051) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.030340 / 0.037411 (-0.007071) | 0.128318 / 0.014526 (0.113792) | 0.140463 / 0.176557 (-0.036093) | 0.205762 / 0.737135 (-0.531373) | 0.147996 / 0.296338 (-0.148342) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.493158 / 0.215209 (0.277949) | 4.858346 / 2.077655 (2.780691) | 2.242942 / 1.504120 (0.738822) | 2.010092 / 1.541195 (0.468897) | 2.076765 / 1.468490 (0.608275) | 0.636669 / 4.584777 (-3.948108) | 4.478027 / 3.745712 (0.732314) | 2.157843 / 5.269862 (-3.112019) | 1.305133 / 4.565676 (-3.260543) | 0.079220 / 0.424275 (-0.345055) | 0.013858 / 0.007607 (0.006251) | 0.604501 / 0.226044 (0.378457) | 5.950071 / 2.268929 (3.681143) | 2.738373 / 55.444624 (-52.706251) | 2.380275 / 6.876477 (-4.496201) | 2.517108 / 2.142072 (0.375035) | 0.772249 / 4.805227 (-4.032979) | 0.169874 / 6.500664 (-6.330790) | 0.078026 / 0.075469 (0.002557) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.450200 / 1.841788 (-0.391588) | 17.810965 / 8.074308 (9.736657) | 15.518998 / 10.191392 (5.327606) | 0.200469 / 0.680424 (-0.479954) | 0.020777 / 0.534201 (-0.513424) | 0.504556 / 0.579283 (-0.074727) | 0.518493 / 0.434364 (0.084129) | 0.615335 / 0.540337 (0.074998) | 0.754065 / 1.386936 (-0.632871) |

</details>
PyArrow==latest

<details>
<summary>Show updated benchmarks!</summary>

### Benchmark: benchmark_array_xd.json

| metric | read_batch_formatted_as_numpy after write_array2d | read_batch_formatted_as_numpy after write_flattened_sequence | read_batch_formatted_as_numpy after write_nested_sequence | read_batch_unformated after write_array2d | read_batch_unformated after write_flattened_sequence | read_batch_unformated after write_nested_sequence | read_col_formatted_as_numpy after write_array2d | read_col_formatted_as_numpy after write_flattened_sequence | read_col_formatted_as_numpy after write_nested_sequence | read_col_unformated after write_array2d | read_col_unformated after write_flattened_sequence | read_col_unformated after write_nested_sequence | read_formatted_as_numpy after write_array2d | read_formatted_as_numpy after write_flattened_sequence | read_formatted_as_numpy after write_nested_sequence | read_unformated after write_array2d | read_unformated after write_flattened_sequence | read_unformated after write_nested_sequence | write_array2d | write_flattened_sequence | write_nested_sequence |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.007224 / 0.011353 (-0.004129) | 0.004663 / 0.011008 (-0.006345) | 0.092151 / 0.038508 (0.053643) | 0.038359 / 0.023109 (0.015250) | 0.486413 / 0.275898 (0.210515) | 0.521596 / 0.323480 (0.198116) | 0.004207 / 0.007986 (-0.003778) | 0.003745 / 0.004328 (-0.000583) | 0.089840 / 0.004250 (0.085589) | 0.050996 / 0.037052 (0.013943) | 0.498090 / 0.258489 (0.239601) | 0.533647 / 0.293841 (0.239806) | 0.035151 / 0.128546 (-0.093395) | 0.010293 / 0.075646 (-0.065354) | 0.099056 / 0.419271 (-0.320215) | 0.057365 / 0.043533 (0.013833) | 0.470652 / 0.255139 (0.215513) | 0.509801 / 0.283200 (0.226602) | 0.115650 / 0.141683 (-0.026033) | 1.810860 / 1.452155 (0.358705) | 1.896775 / 1.492716 (0.404059) |

### Benchmark: benchmark_getitem\_100B.json

| metric | get_batch_of\_1024\_random_rows | get_batch_of\_1024\_rows | get_first_row | get_last_row |
|--------|---|---|---|---|
| new / old (diff) | 0.261887 / 0.018006 (0.243880) | 0.489919 / 0.000490 (0.489430) | 0.006117 / 0.000200 (0.005917) | 0.000134 / 0.000054 (0.000079) |

### Benchmark: benchmark_indices_mapping.json

| metric | select | shard | shuffle | sort | train_test_split |
|--------|---|---|---|---|---|
| new / old (diff) | 0.035033 / 0.037411 (-0.002378) | 0.141093 / 0.014526 (0.126567) | 0.152613 / 0.176557 (-0.023943) | 0.218351 / 0.737135 (-0.518785) | 0.158366 / 0.296338 (-0.137972) |

### Benchmark: benchmark_iterating.json

| metric | read 5000 | read 50000 | read_batch 50000 10 | read_batch 50000 100 | read_batch 50000 1000 | read_formatted numpy 5000 | read_formatted pandas 5000 | read_formatted tensorflow 5000 | read_formatted torch 5000 | read_formatted_batch numpy 5000 10 | read_formatted_batch numpy 5000 1000 | shuffled read 5000 | shuffled read 50000 | shuffled read_batch 50000 10 | shuffled read_batch 50000 100 | shuffled read_batch 50000 1000 | shuffled read_formatted numpy 5000 | shuffled read_formatted_batch numpy 5000 10 | shuffled read_formatted_batch numpy 5000 1000 |
|--------|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 0.542219 / 0.215209 (0.327010) | 5.479358 / 2.077655 (3.401703) | 2.749586 / 1.504120 (1.245466) | 2.537686 / 1.541195 (0.996491) | 2.582351 / 1.468490 (1.113861) | 0.636750 / 4.584777 (-3.948027) | 4.537501 / 3.745712 (0.791789) | 2.141392 / 5.269862 (-3.128469) | 1.279711 / 4.565676 (-3.285965) | 0.079227 / 0.424275 (-0.345048) | 0.014141 / 0.007607 (0.006534) | 0.662070 / 0.226044 (0.436025) | 6.572144 / 2.268929 (4.303215) | 3.321349 / 55.444624 (-52.123275) | 2.928219 / 6.876477 (-3.948258) | 3.002732 / 2.142072 (0.860659) | 0.773808 / 4.805227 (-4.031419) | 0.166017 / 6.500664 (-6.334647) | 0.076424 / 0.075469 (0.000955) |

### Benchmark: benchmark_map_filter.json

| metric | filter | map fast-tokenizer batched | map identity | map identity batched | map no-op batched | map no-op batched numpy | map no-op batched pandas | map no-op batched pytorch | map no-op batched tensorflow |
|--------|---|---|---|---|---|---|---|---|---|
| new / old (diff) | 1.584325 / 1.841788 (-0.257463) | 18.359247 / 8.074308 (10.284938) | 16.977875 / 10.191392 (6.786483) | 0.195381 / 0.680424 (-0.485043) | 0.021048 / 0.534201 (-0.513153) | 0.512237 / 0.579283 (-0.067047) | 0.511435 / 0.434364 (0.077071) | 0.592856 / 0.540337 (0.052518) | 0.711905 / 1.386936 (-0.675031) |

</details>
</details>

![](https://cml.dev/watermark.png#d536e37b21a6dd5c122b6d8113994ec50846c5b5 "CML watermark")