Repository: huggingface/datasets
Pull request: Fix cache path to snakecase for `CachedDatasetModuleFactory` and `Cache`
Fix https://github.com/huggingface/datasets/issues/6750#issuecomment-2016678729

I didn't find a guideline on how to run the tests, so i just run the following steps to make sure that this bug is fixed.
1. `python test.py`,
2. then `HF_DATASETS_OFFLINE=1 python test.py`

The `test.py` is
```
  import datasets

  datasets.utils.logging.set_verbosity_info()

  ds = datasets.load_dataset('izhx/STS17-debug')
  print(ds)

  ds = datasets.load_dataset('C-MTEB/AFQMC', revision='b44c3b011063adb25877c13823db83bb193913c4')
  print(ds)
```




Problem statement:
`load_dataset` requires a network connection for local download?
### Describe the bug

Hi all - I see that in the past a network dependency has been mistakenly introduced into `load_dataset` even for local loads. Is it possible this has happened again?


### Steps to reproduce the bug

```
>>> import datasets
>>> datasets.load_dataset("hh-rlhf")
Repo card metadata block was not found. Setting CardData to empty.
*hangs bc i'm firewalled*
````
stack trace from ctrl-c:
```
^CTraceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/jobuser/.local/lib/python3.10/site-packages/datasets/load.py", line 2582, in load_dataset
    builder_instance.download_and_prepare(
    output_path = get_from_cache(                                                                                                                              [0/122]
  File "/home/jobuser/.local/lib/python3.10/site-packages/datasets/utils/file_utils.py", line 532, in get_from_cache
    response = http_head(
  File "/home/jobuser/.local/lib/python3.10/site-packages/datasets/utils/file_utils.py", line 419, in http_head
    response = _request_with_retry(
  File "/home/jobuser/.local/lib/python3.10/site-packages/datasets/utils/file_utils.py", line 304, in _request_with_retry
    response = requests.request(method=method.upper(), url=url, timeout=timeout, **params)
  File "/home/jobuser/build/lipy-flytekit-image/environments/satellites/python/lib/python3.10/site-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
  File "/home/jobuser/build/lipy-flytekit-image/environments/satellites/python/lib/python3.10/site-packages/requests/sessions.py", line 587, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/jobuser/build/lipy-flytekit-image/environments/satellites/python/lib/python3.10/site-packages/requests/sessions.py", line 701, in send
    r = adapter.send(request, **kwargs)
  File "/home/jobuser/build/lipy-flytekit-image/environments/satellites/python/lib/python3.10/site-packages/requests/adapters.py", line 487, in send
    resp = conn.urlopen(
  File "/home/jobuser/build/lipy-flytekit-image/environments/satellites/python/lib/python3.10/site-packages/urllib3/connectionpool.py", line 703, in urlopen
    httplib_response = self._make_request(
  File "/home/jobuser/build/lipy-flytekit-image/environments/satellites/python/lib/python3.10/site-packages/urllib3/connectionpool.py", line 386, in _make_request
    self._validate_conn(conn)
  File "/home/jobuser/build/lipy-flytekit-image/environments/satellites/python/lib/python3.10/site-packages/urllib3/connectionpool.py", line 1042, in _validate_conn
    conn.connect()
  File "/home/jobuser/build/lipy-flytekit-image/environments/satellites/python/lib/python3.10/site-packages/urllib3/connection.py", line 363, in connect
    self.sock = conn = self._new_conn()
  File "/home/jobuser/build/lipy-flytekit-image/environments/satellites/python/lib/python3.10/site-packages/urllib3/connection.py", line 174, in _new_conn
    conn = connection.create_connection(
  File "/home/jobuser/build/lipy-flytekit-image/environments/satellites/python/lib/python3.10/site-packages/urllib3/util/connection.py", line 85, in create_connection
    sock.connect(sa)
KeyboardInterrupt
```

### Expected behavior

loads the dataset

### Environment info

```
> pip show datasets
Name: datasets
Version: 2.18.0
```

Python 3.10.2

Hints from issue discussion:
Are you using `HF_DATASETS_OFFLINE=1` ?
> Are you using `HF_DATASETS_OFFLINE=1` ?

This doesn't work for me. `datasets=2.18.0`

`test.py`:
```
import datasets

datasets.utils.logging.set_verbosity_info()

ds = datasets.load_dataset('C-MTEB/AFQMC', revision='b44c3b011063adb25877c13823db83bb193913c4')

print(ds)
```

run `python test.py`
```
Generating dataset afqmc (/home/data/.cache/huggingface/datasets/C-MTEB___afqmc/default/0.0.0/b44c3b011063adb25877c13823db83bb193913c4)
Downloading and preparing dataset afqmc/default to /home/data/.cache/huggingface/datasets/C-MTEB___afqmc/default/0.0.0/b44c3b011063adb25877c13823db83bb193913c4...
Dataset not on Hf google storage. Downloading and preparing it from source
hf://datasets/C-MTEB/AFQMC@b44c3b011063adb25877c13823db83bb193913c4/data/validation-00000-of-00001-b8fc393b5ddedac7.parquet not found in cache or force_download set to True, downloading to /home/data/.cache/huggingface/datasets/downloads/78949f93104662359f4f3d5a2f7ec1ae37af5a5af44420a51212ea08c0be966b.incomplete
Downloading data: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 240k/240k [00:01<00:00, 178kB/s]
storing hf://datasets/C-MTEB/AFQMC@b44c3b011063adb25877c13823db83bb193913c4/data/validation-00000-of-00001-b8fc393b5ddedac7.parquet in cache at /home/data/.cache/huggingface/datasets/downloads/78949f93104662359f4f3d5a2f7ec1ae37af5a5af44420a51212ea08c0be966b
creating metadata file for /home/data/.cache/huggingface/datasets/downloads/78949f93104662359f4f3d5a2f7ec1ae37af5a5af44420a51212ea08c0be966b
Downloading took 0.0 min
Checksum Computation took 0.0 min
Generating test split
Generating test split: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3861/3861 [00:00<00:00, 3972.00 examples/s]
Generating train split
Generating train split: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 34334/34334 [00:00<00:00, 34355.50 examples/s]
Generating validation split
Generating validation split: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4316/4316 [00:00<00:00, 4477.00 examples/s]
All the splits matched successfully.
Dataset afqmc downloaded and prepared to /home/data/.cache/huggingface/datasets/C-MTEB___afqmc/default/0.0.0/b44c3b011063adb25877c13823db83bb193913c4. Subsequent calls will reuse this data.
DatasetDict({
    test: Dataset({
        features: ['sentence1', 'sentence2', 'score', 'idx'],
        num_rows: 3861
    })
    train: Dataset({
        features: ['sentence1', 'sentence2', 'score', 'idx'],
        num_rows: 34334
    })
    validation: Dataset({
        features: ['sentence1', 'sentence2', 'score', 'idx'],
        num_rows: 4316
    })
})
```

Then run `HF_DATASETS_OFFLINE=1 python test.py`
```
Traceback (most recent call last):
  File "test.py", line 9, in <module>
    ds = datasets.load_dataset('C-MTEB/AFQMC', revision='b44c3b011063adb25877c13823db83bb193913c4')
  File "/dev/shm/tmp_env/lib/python3.10/site-packages/datasets/load.py", line 2556, in load_dataset
    builder_instance = load_dataset_builder(
  File "/dev/shm/tmp_env/lib/python3.10/site-packages/datasets/load.py", line 2228, in load_dataset_builder
    dataset_module = dataset_module_factory(
  File "/dev/shm/tmp_env/lib/python3.10/site-packages/datasets/load.py", line 1871, in dataset_module_factory
    raise ConnectionError(f"Couldn't reach the Hugging Face Hub for dataset '{path}': {e1}") from None
ConnectionError: Couldn't reach the Hugging Face Hub for dataset 'C-MTEB/AFQMC': Offline mode is enabled.
```

Commit message: fix cache path snakecase

Patch to review:
diff --git a/src/datasets/load.py b/src/datasets/load.py
index c08ad274acf..6e4849322b8 100644
--- a/src/datasets/load.py
+++ b/src/datasets/load.py
@@ -1608,7 +1608,10 @@ def _get_modification_time(module_hash):
             }
             return DatasetModule(module_path, hash, builder_kwargs, importable_file_path=importable_file_path)
         cache_dir = os.path.expanduser(str(self.cache_dir or config.HF_DATASETS_CACHE))
-        cached_datasets_directory_path_root = os.path.join(cache_dir, self.name.replace("/", "___"))
+        dataset_name = Path(self.name).name
+        dataset_name_snakecase = camelcase_to_snakecase(dataset_name)
+        cached_relative_path = self.name.replace("/", "___").replace(dataset_name, dataset_name_snakecase)
+        cached_datasets_directory_path_root = os.path.join(cache_dir, cached_relative_path)
         cached_directory_paths = [
             cached_directory_path
             for cached_directory_path in glob.glob(os.path.join(cached_datasets_directory_path_root, "*", "*", "*"))
diff --git a/src/datasets/packaged_modules/cache/cache.py b/src/datasets/packaged_modules/cache/cache.py
index 54c4737be11..1473b28d7cc 100644
--- a/src/datasets/packaged_modules/cache/cache.py
+++ b/src/datasets/packaged_modules/cache/cache.py
@@ -11,7 +11,7 @@
 import datasets
 import datasets.config
 import datasets.data_files
-from datasets.naming import filenames_for_dataset_split
+from datasets.naming import camelcase_to_snakecase, filenames_for_dataset_split
 
 
 logger = datasets.utils.logging.get_logger(__name__)
@@ -25,7 +25,10 @@ def _find_hash_in_cache(
     dataset_name: str, config_name: Optional[str], cache_dir: Optional[str]
 ) -> Tuple[str, str, str]:
     cache_dir = os.path.expanduser(str(cache_dir or datasets.config.HF_DATASETS_CACHE))
-    cached_datasets_directory_path_root = os.path.join(cache_dir, dataset_name.replace("/", "___"))
+    dataset_basename = Path(dataset_name).name
+    dataset_basename_snakecase = camelcase_to_snakecase(dataset_basename)
+    cached_relative_path = dataset_name.replace("/", "___").replace(dataset_basename, dataset_basename_snakecase)
+    cached_datasets_directory_path_root = os.path.join(cache_dir, cached_relative_path)
     cached_directory_paths = [
         cached_directory_path
         for cached_directory_path in glob.glob(


Code review comments:
diff_hunk: @@ -1608,7 +1608,10 @@ def _get_modification_time(module_hash):
             }
             return DatasetModule(module_path, hash, builder_kwargs, importable_file_path=importable_file_path)
         cache_dir = os.path.expanduser(str(self.cache_dir or config.HF_DATASETS_CACHE))
-        cached_datasets_directory_path_root = os.path.join(cache_dir, self.name.replace("/", "___"))
+        dataset_name = Path(self.name).name
+        dataset_name_snakecase = camelcase_to_snakecase(dataset_name)
+        cached_relative_path = self.name.replace("/", "___").replace(dataset_name, dataset_name_snakecase)
path: src/datasets/load.py
text: @user1:
This can lead to issues if the dataset name is also in the namespace, here is a suggestion:

```suggestion
        namespace_and_dataset_name = self.name.split("/")
        namespace_and_dataset_name[-1] = camelcase_to_snakecase(namespace_and_dataset_name[-1])
        cached_relative_path = "___".join(dataset_name_parts)
```

diff_hunk: @@ -25,7 +25,10 @@ def _find_hash_in_cache(
     dataset_name: str, config_name: Optional[str], cache_dir: Optional[str]
 ) -> Tuple[str, str, str]:
     cache_dir = os.path.expanduser(str(cache_dir or datasets.config.HF_DATASETS_CACHE))
-    cached_datasets_directory_path_root = os.path.join(cache_dir, dataset_name.replace("/", "___"))
+    dataset_basename = Path(dataset_name).name
+    dataset_basename_snakecase = camelcase_to_snakecase(dataset_basename)
+    cached_relative_path = dataset_name.replace("/", "___").replace(dataset_basename, dataset_basename_snakecase)
path: src/datasets/packaged_modules/cache/cache.py
text: @user1:
```suggestion
        namespace_and_dataset_name = self.name.split("/")
        namespace_and_dataset_name[-1] = camelcase_to_snakecase(namespace_and_dataset_name[-1])
        cached_relative_path = "___".join(dataset_name_parts)
```