Configurable max filename length in file locks
Opened by lmmx (closed)

Resolve #2924 (https://github.com/huggingface/datasets/issues/2924#issuecomment-952330956) wherein the assumption of file lock maximum filename length to be 255 raises an OSError on encrypted drives (ecryptFS on Linux uses part of the lower filename, reducing the maximum filename size to 143). Allowing this limit to be set in the config module allows this to be modified by users. Will not affect Windows users, as their class passes 255 on init explicitly.

Reproduced with the following example ([the first few lines of a script from Lightning Flash](https://lightning-flash.readthedocs.io/en/latest/reference/speech_recognition.html), fine-tuning a HF model):

```py
import torch

import flash
from flash.audio import SpeechRecognition, SpeechRecognitionData
from flash.core.data.utils import download_data

# 1. Create the DataModule
download_data("https://pl-flash-data.s3.amazonaws.com/timit_data.zip", "./data")

datamodule = SpeechRecognitionData.from_json(
    input_fields="file",
    target_fields="text",
    train_file="data/timit/train.json",
    test_file="data/timit/test.json",
)
```

Which gave this traceback:

```py
Traceback (most recent call last):
  File "lf_ft.py", line 10, in <module>
    datamodule = SpeechRecognitionData.from_json(
  File "/home/louis/miniconda3/envs/w2vlf/lib/python3.8/site-packages/flash/core/data/data_module.py", line 1005, in from_json
    return cls.from_data_source(
  File "/home/louis/miniconda3/envs/w2vlf/lib/python3.8/site-packages/flash/core/data/data_module.py", line 571, in from_data_source
    train_dataset, val_dataset, test_dataset, predict_dataset = data_source.to_datasets(
  File "/home/louis/miniconda3/envs/w2vlf/lib/python3.8/site-packages/flash/core/data/data_source.py", line 307, in to_datasets
    train_dataset = self.generate_dataset(train_data, RunningStage.TRAINING)
  File "/home/louis/miniconda3/envs/w2vlf/lib/python3.8/site-packages/flash/core/data/data_source.py", line 344, in generate_dataset
    data = load_data(data, mock_dataset)
  File "/home/louis/miniconda3/envs/w2vlf/lib/python3.8/site-packages/flash/audio/speech_recognition/data.py", line 103, in load_data
    dataset_dict = load_dataset(self.filetype, data_files={stage: str(file)})
  File "/home/louis/miniconda3/envs/w2vlf/lib/python3.8/site-packages/datasets/load.py", line 1599, in load_dataset
    builder_instance = load_dataset_builder(
  File "/home/louis/miniconda3/envs/w2vlf/lib/python3.8/site-packages/datasets/load.py", line 1457, in load_dataset_builder
    builder_instance: DatasetBuilder = builder_cls(
  File "/home/louis/miniconda3/envs/w2vlf/lib/python3.8/site-packages/datasets/builder.py", line 285, in __init__
    with FileLock(lock_path):
  File "/home/louis/miniconda3/envs/w2vlf/lib/python3.8/site-packages/datasets/utils/filelock.py", line 323, in __enter__
    self.acquire()
  File "/home/louis/miniconda3/envs/w2vlf/lib/python3.8/site-packages/datasets/utils/filelock.py", line 272, in acquire
    self._acquire()
  File "/home/louis/miniconda3/envs/w2vlf/lib/python3.8/site-packages/datasets/utils/filelock.py", line 403, in _acquire
    fd = os.open(self._lock_file, open_mode)
OSError: [Errno 36] File name too long: '/home/louis/.cache/huggingface/datasets/_home_louis_.cache_huggingface_datasets_json_default-98e6813a547f72fa_0.0.0_c2d554c3377ea79c7664b93dc65d0803b45e3279000f993c7bfd18937fd7f426.lock'
```

Note the filename is 145 chars long:

```
>>> len("_home_louis_.cache_huggingface_datasets_json_default-98e6813a547f72fa_0.0.0_c2d554c3377ea79c7664b93dc65d0803b45e3279000f993c7bfd18937fd7f426.lock")
145
```

After installing datasets as an editable local package and modifying the script I was running to first include:

```py
import datasets
datasets.config.MAX_DATASET_CONFIG_ID_READABLE_LENGTH = 143
```

The error goes away.

If I instead deliberately set the value incorrectly as 144, the OSError returns:

```
Traceback (most recent call last):
  File "lf_ft.py", line 14, in <module>
    datamodule = SpeechRecognitionData.from_json(
  File "/home/louis/miniconda3/envs/w2vlf/lib/python3.8/site-packages/flash/core/data/data_module.py", line 1005, in from_json
    return cls.from_data_source(
  File "/home/louis/miniconda3/envs/w2vlf/lib/python3.8/site-packages/flash/core/data/data_module.py", line 571, in from_data_source
    train_dataset, val_dataset, test_dataset, predict_dataset = data_source.to_datasets(
  File "/home/louis/miniconda3/envs/w2vlf/lib/python3.8/site-packages/flash/core/data/data_source.py", line 307, in to_datasets
    train_dataset = self.generate_dataset(train_data, RunningStage.TRAINING)
  File "/home/louis/miniconda3/envs/w2vlf/lib/python3.8/site-packages/flash/core/data/data_source.py", line 344, in generate_dataset
    data = load_data(data, mock_dataset)
  File "/home/louis/miniconda3/envs/w2vlf/lib/python3.8/site-packages/flash/audio/speech_recognition/data.py", line 103, in load_data
    dataset_dict = load_dataset(self.filetype, data_files={stage: str(file)})
  File "/home/louis/dev/hf_datasets/src/datasets/load.py", line 1605, in load_dataset
    builder_instance = load_dataset_builder(
  File "/home/louis/dev/hf_datasets/src/datasets/load.py", line 1463, in load_dataset_builder
    builder_instance: DatasetBuilder = builder_cls(
  File "/home/louis/dev/hf_datasets/src/datasets/builder.py", line 285, in __init__
    with FileLock(lock_path):
  File "/home/louis/dev/hf_datasets/src/datasets/utils/filelock.py", line 326, in __enter__
    self.acquire()
  File "/home/louis/dev/hf_datasets/src/datasets/utils/filelock.py", line 275, in acquire
    self._acquire()
  File "/home/louis/dev/hf_datasets/src/datasets/utils/filelock.py", line 406, in _acquire
    fd = os.open(self._lock_file, open_mode)
OSError: [Errno 36] File name too long: '/home/louis/.cache/huggingface/datasets/_home_louis_.cache_huggingface_datasets_json_default-32c812b5c1272d64_0.0.0_c2d554c3377ea79c7664b93dc65d0803b45e3279...-5794079643713042223.lock'
```

Comments:
I've also added environment variable configuration so that this can be configured once per machine (e.g. in a `.bashrc` file), as is already done for a few other config variables here.

Cancelling PR in favour of @mariosasko's in #3173