Repository: huggingface/transformers

Issue:
Dataloader number of workers in Trainer
https://github.com/huggingface/transformers/blob/175cd45e13b2e33d1efec9e2ac217cba99f6ae58/src/transformers/trainer.py#L252


If you want to use the Trainer from trainer.py, you only have the option to use only 0 number of workers for your dataloader.

However, even if I change the source code to have 10 number or workers for the data loader, the model still uses the same thread.




Discussion:
+1.

I suppose it should not be difficult to add additional argument to ```self.args``` specifying number of workers to use and to use it as:
```
return DataLoader(
            self.train_dataset,
            batch_size=self.args.train_batch_size,
            sampler=train_sampler,
            collate_fn=self.data_collator,
            drop_last=self.args.dataloader_drop_last,
            num_workers=self.args.num_workers
        )
```

Is there any particular reason why this was not yet implemented? 
+1
+1
No particular reason why it was not implemented, would welcome a PR!

Resolving patch:
diff --git a/src/transformers/trainer.py b/src/transformers/trainer.py
--- a/src/transformers/trainer.py
+++ b/src/transformers/trainer.py
@@ -352,6 +352,7 @@ def get_train_dataloader(self) -> DataLoader:
             sampler=train_sampler,
             collate_fn=self.data_collator,
             drop_last=self.args.dataloader_drop_last,
+            num_workers=self.args.dataloader_num_workers,
         )
 
     def _get_eval_sampler(self, eval_dataset: Dataset) -> Optional[torch.utils.data.sampler.Sampler]:
@@ -391,6 +392,7 @@ def get_eval_dataloader(self, eval_dataset: Optional[Dataset] = None) -> DataLoa
             batch_size=self.args.eval_batch_size,
             collate_fn=self.data_collator,
             drop_last=self.args.dataloader_drop_last,
+            num_workers=self.args.dataloader_num_workers,
         )
 
     def get_test_dataloader(self, test_dataset: Dataset) -> DataLoader:
diff --git a/src/transformers/training_args.py b/src/transformers/training_args.py
--- a/src/transformers/training_args.py
+++ b/src/transformers/training_args.py
@@ -122,6 +122,8 @@ class TrainingArguments:
         eval_steps (:obj:`int`, `optional`):
             Number of update steps between two evaluations if :obj:`evaluation_strategy="steps"`. Will default to the
             same value as :obj:`logging_steps` if not set.
+        dataloader_num_workers (:obj:`int`, `optional`, defaults to 0):
+            Number of subprocesses to use for data loading (PyTorch only). 0 means that the data will be loaded in the main process.
         past_index (:obj:`int`, `optional`, defaults to -1):
             Some models like :doc:`TransformerXL <../model_doc/transformerxl>` or :doc`XLNet <../model_doc/xlnet>` can
             make use of the past hidden states for their predictions. If this argument is set to a positive int, the
@@ -259,6 +261,10 @@ class TrainingArguments:
         default=False, metadata={"help": "Drop the last incomplete batch if it is not divisible by the batch size."}
     )
     eval_steps: int = field(default=None, metadata={"help": "Run an evaluation every X steps."})
+    dataloader_num_workers: int = field(
+        default=0,
+        metadata={"help": "Number of subprocesses to use for data loading (PyTorch only). 0 means that the data will be loaded in the main process."}
+    )
 
     past_index: int = field(
         default=-1,


Test patch: