wav2vec2 multi-node training problems in a shared file system
Author: gullabi
Pull request: no
Labels: name: bug

### System Info

```shell
- `transformers` version: 4.18.0
- Platform: Linux-4.18.0-147.8.1.el8_1.x86_64-x86_64-with-centos-8.1.1911-Core
- Python version: 3.7.4
- Huggingface_hub version: 0.1.2
- PyTorch version (GPU?): 1.9.0+rocm4.0.1 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: distributed
```


### Who can help?

@patrickvonplaten, @anton-l, @lhoestq

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Steps to produce the behavior:

1. clone [this huggingface model](https://huggingface.co/PereLluis13/wav2vec2-xls-r-300m-ca-lm) in order to run the custom `run_speech_recognition_ctc.py` script
2. Setup the `venv` according to the requirements of the model file plus `datasets==2.0.0`, `transformers==4.18.0` and `torch==1.9.0`
3. Launch the runner in a distributed environment which has a shared file system for two nodes, preferably with SLURM. Example [here](https://gist.github.com/gullabi/3f66094caa8db1c1e615dd35bd67ec71) 

The processes fail with the error:

```
Traceback (most recent call last):
  File "/gpfs/projects/bsc88/speech/asr/wav2vec2-xls-r-300m-ca-lm/run_speech_recognition_ctc.py", line 816, in <module>
    main()
  File "/gpfs/projects/bsc88/speech/asr/wav2vec2-xls-r-300m-ca-lm/run_speech_recognition_ctc.py", line 560, in main
    os.remove(vocab_file)
FileNotFoundError: [Errno 2] No such file or directory: 'wav2vec2-xls-r-300m-ca_dist/vocab.json'
```
as both nodes are seeing the `vocab_file` and trying to delete it at the same time, but since nodes are on a shared file system, the training fails. 

As further information, when the `os.remove` is escaped via
```
        with training_args.main_process_first():
            if training_args.overwrite_output_dir and os.path.isfile(vocab_file):
                try:
                    os.remove(vocab_file)
                except Exception as e:
                    logger.info(e)
```
the runner trains the model successfully until the first checkpoint. However, during the evaluation just before saving the checkpoint to the file system this error occurs:

```
File "/gpfs/projects/bsc88/speech/asr/wav2vec2-xls-r-300m-ca-lm/
run_speech_recognition_ctc.py", line 819, in <module>
    main()
  File "/gpfs/projects/bsc88/speech/asr/wav2vec2-xls-r-300m-ca-lm/
run_speech_recognition_ctc.py", line 770, in main
    train_result = trainer.train(resume_from_checkpoint=checkpoint)
  File "/gpfs/projects/bsc88/projects/speech-tech-resources/venv_amd_speech/
lib/python3.7/site-packages/transformers/trainer.py", line 1497, in train
    self._maybe_log_save_evaluate(tr_loss, model, trial, epoch,
ignore_keys_for_eval)
  File "/gpfs/projects/bsc88/projects/speech-tech-resources/venv_amd_speech/
lib/python3.7/site-packages/transformers/trainer.py", line 1624, in
_maybe_log_save_evaluate
    metrics = self.evaluate(ignore_keys=ignore_keys_for_eval)
  File "/gpfs/projects/bsc88/projects/speech-tech-resources/venv_amd_speech/
lib/python3.7/site-packages/transformers/trainer.py", line 2291, in evaluate
    metric_key_prefix=metric_key_prefix,
  File "/gpfs/projects/bsc88/projects/speech-tech-resources/venv_amd_speech/
lib/python3.7/site-packages/transformers/trainer.py", line 2535, in
evaluation_loop
    metrics = self.compute_metrics(EvalPrediction(predictions=all_preds,
label_ids=all_labels))
  File "/gpfs/projects/bsc88/speech/asr/wav2vec2-xls-r-300m-ca-lm/
run_speech_recognition_ctc.py", line 720, in compute_metrics
    metrics = {k: v.compute(predictions=pred_str, references=label_str) for k,
v in eval_metrics.items()}
  File "/gpfs/projects/bsc88/speech/asr/wav2vec2-xls-r-300m-ca-lm/
run_speech_recognition_ctc.py", line 720, in <dictcomp>
    metrics = {k: v.compute(predictions=pred_str, references=label_str) for k,
v in eval_metrics.items()}
  File "/gpfs/projects/bsc88/projects/speech-tech-resources/venv_amd_speech/
lib/python3.7/site-packages/datasets/metric.py", line 444, in compute
    os.remove(file_path)
FileNotFoundError: [Errno 2] No such file or directory: '/home/bsc88/bsc88474
/.cache/huggingface/metrics/wer/default/default_experiment-1-0.arrow'
```
This is presumably the metric evaluation is done on all nodes, but since they are in a shared file system removal of the cached evaluation files creates a conflict.

In principle, transformers library has a context `main_process_first` which in the case `local=False` is passed only the main node of the multi-node executes the tasks. The metric calculation is not within this context and we are not sure whether (apart from the `os.remove(vocab.json)` problem) the solution is to add the context [here](https://github.com/huggingface/transformers/blob/71e602725b90f63f404109bae9f72cbdf755477b/examples/pytorch/speech-recognition/run_speech_recognition_ctc.py#L659). 

Since this issue is also related to the filelock processes within the `datasets` library we also included `@lhoestq` as a person who can help.

### Expected behavior

```shell
The training process runs successfully without producing any errors concerning the write/delete process conflicts or any other error related to the file locks.
```


Comments:
When instantiating a metric in a distributed setup, you need to specify it:
```python
load_metric(..., process_id=rank, num_process=total_world_size)
```
this way there will be no collision on the files used to store the predictions and references used to compute the metric.

(and this makes me think one should rename `num_process` to `num_processes` or something like that)

Thank you for the quick responses and fixes.

Although you have closed the issue, I want to document the results. With the changes I did in the `load_metric` the processes errored out with the message:

```
ValueError: Error in _init_writer: another metric instance is already using the local cache file at /home/bsc88/bsc88474/.cache/huggingface/metrics/wer/default/default_experiment-8-rdv.lock. Please specify an experiment_id (currently: default_experiment) to avoid collision between distributed metric instances.
```
I understand that `experiment_id` is a string and it can be anything. I just made the change but I won't get the results for a while since I am in the job queue.

Should we maybe move this (still open) issue to datasets @gullabi @lhoestq ?

It might be a good idea but let me check the results of the experiment. If it is working as intended there is no reason to move it I think. But if we notice any problem, I will let you know and we can move the discussion to datasets. thanks!

So I am back, with more problems. Now might be a good idea to move the issue to datasets. I am still running into problems. I am continuing here but you will let me know if I need to do something.

Here are the changes I did to the `run_speech_recognition_ctc.py`
```
    process_id=int(os.environ["RANK"])
    num_process=int(os.environ["WORLD_SIZE"])
    eval_metrics = {metric: load_metric(metric,
                                        process_id=process_id,
                                        num_process=num_process,
                                        experiment_id="slurm")
                    for metric in data_args.eval_metrics}
```
For the test I am executing the world size is 4, with 2 GPUs in 2 nodes. However the process is not finding the necessary lock files 
```
  File "/gpfs/projects/bsc88/speech/asr/wav2vec2-xls-r-300m-ca-lm/run_speech_recognition_ctc.py", line 841, in <module>
    main()
  File "/gpfs/projects/bsc88/speech/asr/wav2vec2-xls-r-300m-ca-lm/run_speech_recognition_ctc.py", line 792, in main
    train_result = trainer.train(resume_from_checkpoint=checkpoint)
  File "/gpfs/projects/bsc88/projects/speech-tech-resources/venv_amd_speech/lib/python3.7/site-packages/transformers/trainer.py", line 1497, in train
    self._maybe_log_save_evaluate(tr_loss, model, trial, epoch, ignore_keys_for_eval)
  File "/gpfs/projects/bsc88/projects/speech-tech-resources/venv_amd_speech/lib/python3.7/site-packages/transformers/trainer.py", line 1624, in _maybe_log_save_evaluate
    metrics = self.evaluate(ignore_keys=ignore_keys_for_eval)
  File "/gpfs/projects/bsc88/projects/speech-tech-resources/venv_amd_speech/lib/python3.7/site-packages/transformers/trainer.py", line 2291, in evaluate
    metric_key_prefix=metric_key_prefix,
  File "/gpfs/projects/bsc88/projects/speech-tech-resources/venv_amd_speech/lib/python3.7/site-packages/transformers/trainer.py", line 2535, in evaluation_loop
    metrics = self.compute_metrics(EvalPrediction(predictions=all_preds, label_ids=all_labels))
  File "/gpfs/projects/bsc88/speech/asr/wav2vec2-xls-r-300m-ca-lm/run_speech_recognition_ctc.py", line 742, in compute_metrics
    metrics = {k: v.compute(predictions=pred_str, references=label_str) for k, v in eval_metrics.items()}
  File "/gpfs/projects/bsc88/speech/asr/wav2vec2-xls-r-300m-ca-lm/run_speech_recognition_ctc.py", line 742, in <dictcomp>
    metrics = {k: v.compute(predictions=pred_str, references=label_str) for k, v in eval_metrics.items()}
  File "/gpfs/projects/bsc88/projects/speech-tech-resources/venv_amd_speech/lib/python3.7/site-packages/datasets/metric.py", line 419, in compute
    self.add_batch(**inputs)
  File "/gpfs/projects/bsc88/projects/speech-tech-resources/venv_amd_speech/lib/python3.7/site-packages/datasets/metric.py", line 465, in add_batch
    self._init_writer()
  File "/gpfs/projects/bsc88/projects/speech-tech-resources/venv_amd_speech/lib/python3.7/site-packages/datasets/metric.py", line 552, in _init_writer
    self._check_rendez_vous()  # wait for master to be ready and to let everyone go
  File "/gpfs/projects/bsc88/projects/speech-tech-resources/venv_amd_speech/lib/python3.7/site-packages/datasets/metric.py", line 342, in _check_rendez_vous
    ) from None
ValueError: Expected to find locked file /home/bsc88/bsc88474/.cache/huggingface/metrics/wer/default/slurm-4-0.arrow.lock from process 3 but it doesn't exist.
```

When I look at the cache directory, I can see all the lock files in principle:
```
/home/bsc88/bsc88474/.cache/huggingface/metrics/wer/default/slurm-4-0.arrow
/home/bsc88/bsc88474/.cache/huggingface/metrics/wer/default/slurm-4-0.arrow.lock
/home/bsc88/bsc88474/.cache/huggingface/metrics/wer/default/slurm-4-1.arrow
/home/bsc88/bsc88474/.cache/huggingface/metrics/wer/default/slurm-4-1.arrow.lock
/home/bsc88/bsc88474/.cache/huggingface/metrics/wer/default/slurm-4-2.arrow
/home/bsc88/bsc88474/.cache/huggingface/metrics/wer/default/slurm-4-2.arrow.lock
/home/bsc88/bsc88474/.cache/huggingface/metrics/wer/default/slurm-4-3.arrow
/home/bsc88/bsc88474/.cache/huggingface/metrics/wer/default/slurm-4-3.arrow.lock
/home/bsc88/bsc88474/.cache/huggingface/metrics/wer/default/slurm-4-rdv.lock
```
I appreciate any help regarding this, thanks! @lhoestq 

After fixing the parts related to `datasets` we ran into another problem with the `run_speech_recognition_ctc.py` script. Maybe a question for @patrickvonplaten sorry to bother you with a mention. 

When we increase the number of nodes, we are getting a `JSONDecodeError` for `preprocessor_config.json`. Checking the file, we see that it is fine. We suspect that the nodes are trying to read a file that is currently being written. In order to solve the problem we put `local=False` to the `main_process_first` contexts, but it didn't help. We are putting the snippet which we think is causing the problem, plus the logs in the end

```
    with training_args.main_process_first(local=False, desc="dataset map preprocessing"):
        vectorized_datasets = raw_datasets.map(
            prepare_dataset,
            remove_columns=next(iter(raw_datasets.values())).column_names,
            num_proc=num_workers,
            desc="preprocess datasets",
        )

        def is_audio_in_length_range(length):
            return length > min_input_length and length < max_input_length

        # filter data that is shorter than min_input_length
        vectorized_datasets = vectorized_datasets.filter(
            is_audio_in_length_range,
            num_proc=num_workers,
            input_columns=["input_length"],
        )
```
and the log  (sorry it's a jumbled mess since many nodes are writing at the same time):
```
^Mpreprocess datasets #11:   0%|          | 0/12020 [00:00<?, ?ex/s]ESC[AESC[AESC[AESC[AESC[AESC[AESC[AESC[AESC[AESC[AESC[AFeature extractor saved in wav2vec2-xls-r-300m-ca_new/preprocessor_con
fig.json
Special tokens file saved in wav2vec2-xls-r-300m-ca_new/special_tokens_map.json
added tokens file saved in wav2vec2-xls-r-300m-ca_new/added_tokens.json
tokenizer config file saved in wav2vec2-xls-r-300m-ca_new/tokenizer_config.json
Special tokens file saved in wav2vec2-xls-r-300m-ca_new/special_tokens_map.json
Configuration saved in wav2vec2-xls-r-300m-ca_new/config.json
/gpfs/projects/bsc88/speech/asr/wav2vec2-xls-r-300m-ca-lm/run_speech_recognition_ctc.py:761: FutureWarning: Loading a processor from a feature extractor config that does not include a `processo
r_class` attribute is deprecated and will be removed in v5. Please add the following  attribute to your `preprocessor_config.json` file to suppress this warning:  `'processor_class': 'Wav2Vec2P
rocessor'`
  FutureWarning,
/gpfs/projects/bsc88/speech/asr/wav2vec2-xls-r-300m-ca-lm/run_speech_recognition_ctc.py:761: FutureWarning: Loading a processor from a feature extractor config that does not include a `processo
r_class` attribute is deprecated and will be removed in v5. Please add the following  attribute to your `preprocessor_config.json` file to suppress this warning:  `'processor_class': 'Wav2Vec2P
rocessor'`
  FutureWarning,
/gpfs/projects/bsc88/projects/speech-tech-resources/venv_amd_speech/lib/python3.7/site-packages/transformers/models/wav2vec2/processing_wav2vec2.py:58: FutureWarning: Loading a tokenizer inside
 Wav2Vec2Processor from a config that does not include a `tokenizer_class` attribute is deprecated and will be removed in v5. Please add `'tokenizer_class': 'Wav2Vec2CTCTokenizer'` attribute to
 either your `config.json` or `tokenizer_config.json` file to suppress this warning: 
  FutureWarning,
/gpfs/projects/bsc88/projects/speech-tech-resources/venv_amd_speech/lib/python3.7/site-packages/transformers/models/wav2vec2/processing_wav2vec2.py:58: FutureWarning: Loading a tokenizer inside
 Wav2Vec2Processor from a config that does not include a `tokenizer_class` attribute is deprecated and will be removed in v5. Please add `'tokenizer_class': 'Wav2Vec2CTCTokenizer'` attribute to
 either your `config.json` or `tokenizer_config.json` file to suppress this warning: 
  FutureWarning,
Traceback (most recent call last):
Traceback (most recent call last):
  File "/gpfs/projects/bsc88/projects/speech-tech-resources/venv_amd_speech/lib/python3.7/site-packages/transformers/feature_extraction_utils.py", line 454, in get_feature_extractor_dict
  File "/gpfs/projects/bsc88/projects/speech-tech-resources/venv_amd_speech/lib/python3.7/site-packages/transformers/feature_extraction_utils.py", line 454, in get_feature_extractor_dict
Feature extractor saved in wav2vec2-xls-r-300m-ca_new/preprocessor_config.json
    feature_extractor_dict = json.loads(text)    
feature_extractor_dict = json.loads(text)  File "/apps/PYTHON/3.7.4/INTEL/lib/python3.7/json/__init__.py", line 348, in loads

  File "/apps/PYTHON/3.7.4/INTEL/lib/python3.7/json/__init__.py", line 348, in loads
    return _default_decoder.decode(s)
  File "/apps/PYTHON/3.7.4/INTEL/lib/python3.7/json/decoder.py", line 337, in decode
    return _default_decoder.decode(s)
  File "/apps/PYTHON/3.7.4/INTEL/lib/python3.7/json/decoder.py", line 337, in decode
added tokens file saved in wav2vec2-xls-r-300m-ca_new/added_tokens.json
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/apps/PYTHON/3.7.4/INTEL/lib/python3.7/json/decoder.py", line 355, in raw_decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/apps/PYTHON/3.7.4/INTEL/lib/python3.7/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/gpfs/projects/bsc88/speech/asr/wav2vec2-xls-r-300m-ca-lm/run_speech_recognition_ctc.py", line 841, in <module>
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/gpfs/projects/bsc88/speech/asr/wav2vec2-xls-r-300m-ca-lm/run_speech_recognition_ctc.py", line 841, in <module>
^Mpreprocess datasets #5:   3%|▎         | 414/12021 [00:01<00:23, 495.45ex/s]ESC[AESC[AESC[AESC[AESC[A

    main()
  File "/gpfs/projects/bsc88/speech/asr/wav2vec2-xls-r-300m-ca-lm/run_speech_recognition_ctc.py", line 763, in main
    main()
  File "/gpfs/projects/bsc88/speech/asr/wav2vec2-xls-r-300m-ca-lm/run_speech_recognition_ctc.py", line 763, in main
    processor = Wav2Vec2Processor.from_pretrained(training_args.output_dir)
  File "/gpfs/projects/bsc88/projects/speech-tech-resources/venv_amd_speech/lib/python3.7/site-packages/transformers/models/wav2vec2/processing_wav2vec2.py", line 61, in from_pretrained
    processor = Wav2Vec2Processor.from_pretrained(training_args.output_dir)
  File "/gpfs/projects/bsc88/projects/speech-tech-resources/venv_amd_speech/lib/python3.7/site-packages/transformers/models/wav2vec2/processing_wav2vec2.py", line 61, in from_pretrained
    feature_extractor = Wav2Vec2FeatureExtractor.from_pretrained(pretrained_model_name_or_path, **kwargs)
  File "/gpfs/projects/bsc88/projects/speech-tech-resources/venv_amd_speech/lib/python3.7/site-packages/transformers/feature_extraction_utils.py", line 308, in from_pretrained
    feature_extractor = Wav2Vec2FeatureExtractor.from_pretrained(pretrained_model_name_or_path, **kwargs)
  File "/gpfs/projects/bsc88/projects/speech-tech-resources/venv_amd_speech/lib/python3.7/site-packages/transformers/feature_extraction_utils.py", line 308, in from_pretrained
    feature_extractor_dict, kwargs = cls.get_feature_extractor_dict(pretrained_model_name_or_path, **kwargs)
  File "/gpfs/projects/bsc88/projects/speech-tech-resources/venv_amd_speech/lib/python3.7/site-packages/transformers/feature_extraction_utils.py", line 458, in get_feature_extractor_dict
    feature_extractor_dict, kwargs = cls.get_feature_extractor_dict(pretrained_model_name_or_path, **kwargs)
  File "/gpfs/projects/bsc88/projects/speech-tech-resources/venv_amd_speech/lib/python3.7/site-packages/transformers/feature_extraction_utils.py", line 458, in get_feature_extractor_dict
    f"It looks like the config file at '{resolved_feature_extractor_file}' is not a valid JSON file."
OSError: It looks like the config file at 'wav2vec2-xls-r-300m-ca_new/preprocessor_config.json' is not a valid JSON file.
tokenizer config file saved in wav2vec2-xls-r-300m-ca_new/tokenizer_config.json
    f"It looks like the config file at '{resolved_feature_extractor_file}' is not a valid JSON file."
OSError: It looks like the config file at 'wav2vec2-xls-r-300m-ca_new/preprocessor_config.json' is not a valid JSON file.
Feature extractor saved in wav2vec2-xls-r-300m-ca_new/preprocessor_config.json
Configuration saved in wav2vec2-xls-r-300m-ca_new/config.json
Special tokens file saved in wav2vec2-xls-r-300m-ca_new/special_tokens_map.json
tokenizer config file saved in wav2vec2-xls-r-300m-ca_new/tokenizer_config.json
added tokens file saved in wav2vec2-xls-r-300m-ca_new/added_tokens.json
loading feature extractor configuration file wav2vec2-xls-r-300m-ca_new/preprocessor_config.json
loading configuration file wav2vec2-xls-r-300m-ca_new/config.json
```

Hmm, the file should not be written - I guess what might happen here is that one node is much much faster then another node and already wants read a file that has not been created in the previous step yet. 

Could you do the following:
- Simply run the script on one node to correctly write the tokenizer and feature processor config jsons
- Then pass all those files to the script on multi-node so that the in multi-node no config is being written at all? 

Thanks Patrick. For the second step, do I need to skip the data preprocessing also? If so is there a way to load the preprocessed files directly from the cache?

A bit ashamed to say but, I have been looking at the script to try to skip the data preprocessing phase, and did not manage to do it. In the output I see warnings saying that the preprocessed data is being loaded from the cache (for various steps), but I see the tdqm process task bar and the process takes a long time, so I am assuming at least some nodes are doing the preprocessing. I don"t know if this is the usual behavior. 

In principle I have avoided this error by putting a sleep, after the preprocessing and before loading the feature processing config json, not the most elegant solution.

Afterwards we have ran into other unrelated problems but in principle for now the script seems to be working. I am giving feedback just in case it is useful for someone else. 

Hey @gullabi,

Exactly in the second step, you should be able to fully skip the dataset processing. In short, I'd strongly advise to create the whole tokenizer file before doing the multi-node run and then this `if-branch`: https://github.com/huggingface/transformers/blob/da27c4b398e607c161451f335367ad666de08497/examples/pytorch/speech-recognition/run_speech_recognition_ctc.py#L500 should never be run which means that no files sholud be written **and** read at the same time. 

However if `sleep` works for you, I think that's also totally fine. So far we've never had a case where multi-node + shared file-system was used for the examples, so this issue here serves as a great readme guide for future such use cases :-)

Thanks for the suggestion and I am glad that it the issue is useful for something. For the sake of completeness, I would like to give feedback. Although your suggestion sped up the preprocessing  (I used the `--tokenizer_name_or_path` cli parameter), it does not directly solve the problem we were facing. 

The problem was at the step where the feature extractor, config and the tokenizer files are written, and just before the training starts. The errors were pointing out specifically to loading the feature extractor config right before the training. Realizing that the problem is simultaneous reading and writing going by multiple nodes to the same file systems, the problematic write was here:

https://github.com/huggingface/transformers/blob/da27c4b398e607c161451f335367ad666de08497/examples/pytorch/speech-recognition/run_speech_recognition_ctc.py#L675

So I put a sleep right after writing this `if is_main_process` block.