Pytorch T5 pre-training script request
Author: cyk1337
Pull request: no

# 🚀 Feature request

Any plan to release a pytorch pre-training script for T5 mlm? @patrickvonplaten


Comments:
Hi! For T5 MLM training the most important thing is the Data collator used for T5 span-masked language modeling, which is available here, https://github.com/huggingface/transformers/blob/master/examples/flax/language-modeling/run_t5_mlm_flax.py#L279

It's written with numpy so should be easy to port it to PyTorch. Then the rest of the script would be pretty similar to the translation or summarization example. Would you be interested in opening a PR to add this example? Happy to help with it :) 

Sure I am glad to add it. Thank you for the clarification ;)

Hi @patil-suraj ,  the summarization example [run_summarization_no_trainer.py](https://github.com/huggingface/transformers/blob/master/examples/pytorch/summarization/run_summarization_no_trainer.py) throws an error when loading datasets.

The running command was
```bash
python run_summarization_no_trainer.py \
    --model_name_or_path t5-small \
    --dataset_name cnn_dailymail \
    --dataset_config "3.0.0" \
    --source_prefix "summarize: " \
    --output_dir ./tst-summarization
```

The error output was:
```bash
---------------------------------------------------------------------------
NotADirectoryError                        Traceback (most recent call last)
<ipython-input-16-b3897e385817> in <module>()
    617 
    618 if __name__ == "__main__":
--> 619     main()

<ipython-input-16-b3897e385817> in main()
    343     if args.dataset_name is not None:
    344         # Downloading and loading a dataset from the hub.
--> 345         raw_datasets = load_dataset(args.dataset_name, args.dataset_config_name)
    346     else:
    347         data_files = {}

/usr/local/lib/python3.7/dist-packages/datasets/load.py in load_dataset(path, name, data_dir, data_files, split, cache_dir, features, download_config, download_mode, ignore_verifications, keep_in_memory, save_infos, revision, use_auth_token, task, streaming, script_version, **config_kwargs)
   1697         ignore_verifications=ignore_verifications,
   1698         try_from_hf_gcs=try_from_hf_gcs,
-> 1699         use_auth_token=use_auth_token,
   1700     )
   1701 

/usr/local/lib/python3.7/dist-packages/datasets/builder.py in download_and_prepare(self, download_config, download_mode, ignore_verifications, try_from_hf_gcs, dl_manager, base_path, use_auth_token, **download_and_prepare_kwargs)
    594                     if not downloaded_from_gcs:
    595                         self._download_and_prepare(
--> 596                             dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
    597                         )
    598                     # Sync info

/usr/local/lib/python3.7/dist-packages/datasets/builder.py in _download_and_prepare(self, dl_manager, verify_infos, **prepare_split_kwargs)
    659         split_dict = SplitDict(dataset_name=self.name)
    660         split_generators_kwargs = self._make_split_generators_kwargs(prepare_split_kwargs)
--> 661         split_generators = self._split_generators(dl_manager, **split_generators_kwargs)
    662 
    663         # Checksums verification

/root/.cache/huggingface/modules/datasets_modules/datasets/cnn_dailymail/3cb851bf7cf5826e45d49db2863f627cba583cbc32342df7349dfe6c38060234/cnn_dailymail.py in _split_generators(self, dl_manager)
    253     def _split_generators(self, dl_manager):
    254         dl_paths = dl_manager.download_and_extract(_DL_URLS)
--> 255         train_files = _subset_filenames(dl_paths, datasets.Split.TRAIN)
    256         # Generate shared vocabulary
    257 

/root/.cache/huggingface/modules/datasets_modules/datasets/cnn_dailymail/3cb851bf7cf5826e45d49db2863f627cba583cbc32342df7349dfe6c38060234/cnn_dailymail.py in _subset_filenames(dl_paths, split)
    154     else:
    155         logger.fatal("Unsupported split: %s", split)
--> 156     cnn = _find_files(dl_paths, "cnn", urls)
    157     dm = _find_files(dl_paths, "dm", urls)
    158     return cnn + dm

/root/.cache/huggingface/modules/datasets_modules/datasets/cnn_dailymail/3cb851bf7cf5826e45d49db2863f627cba583cbc32342df7349dfe6c38060234/cnn_dailymail.py in _find_files(dl_paths, publisher, url_dict)
    133     else:
    134         logger.fatal("Unsupported publisher: %s", publisher)
--> 135     files = sorted(os.listdir(top_dir))
    136 
    137     ret_files = []

NotADirectoryError: [Errno 20] Not a directory: '/root/.cache/huggingface/datasets/downloads/1bc05d24fa6dda2468e83a73cf6dc207226e01e3c48a507ea716dc0421da583b/cnn/stories'
```

Any solution to it?


This seems to be some issue with the cache directory. Could you try inspecting the content of that directory to see if the dataset is downloaded correctly?

Also, if the error persists feel free to create a new issue.

> This seems to be some issue with the cache directory. Could you try inspecting the content of that directory to see if the dataset is downloaded correctly?

The downloaded dataset file was not a directory but a web page file. Please refer to [#15130](https://github.com/huggingface/transformers/issues/15130) for the details.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

Gently pinging @cyk1337 :) 

Hi @patil-suraj, I've implemented a brief version. And it may take some time to update due to the time schedule ;)

Hey @cyk1337 Thanks a lot! And no worries, feel free to take your time.

Is there any recommended library devised for loading large-scale data for multi-node multi-gpu training in huggingface/PyTorch? Conditioned on that the pre-trained data can have trouble in entirely loading into the memory, like 1TB data. I've met this problem and have not found suitable solutions yet.

The `datasets` [library](https://github.com/huggingface/datasets)! You could look at the docs to know more, but this enables loading very large datasets without blowing up the memory by providing memory mapping. 

Is there an example of this? Have checked the docs, and not found a distributed solution. Maybe we need to implement one for assigning different data blocks to multi-node multi-gpu settings by [loading a specific subset of the files](https://huggingface.co/docs/datasets/loading.html?highlight=distributed#:~:text=load%20a%20specific%20subset%20of%20the%20files) ?

Could you post this question on the [forum](https://discuss.huggingface.co/) someone must have shared something there already. Forum is the best place for general discussion.  Also for the transformers example, a simple script should be enough. 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.