Train Bart model only use one cpu core, Any solutions to use more cores?
Author: BeanSprouts
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version:
- Platform:  linux on arm
- Python version: 3.7
- PyTorch version (GPU?):CPU-1.9.0
- Tensorflow version (GPU?):None
- Using GPU in script?: No
- Using distributed or parallel set-up in script?:No


I am training a Bart Model(BartForConditionalGeneration)
The problem arises when using:
when I run my train script, only use one core of my cpu. I doesn't have any GPUs, but I got a cpu with 96 cores. How can I make it to use more cores?

my own scripts:
`from transformers import BartForConditionalGeneration,BartConfig,BartTokenizerFast,LineByLineTextDataset,DataCollatorForLanguageModeling,Trainer,TrainingArguments
from tokenizers import models, normalizers, pre_tokenizers,Tokenizer
from tokenizers.trainers import BpeTrainer

`_special_tokens = ["
","
","<unk>","<pad>","<mask>"]`

def trainBartTokenizer(files, vocab_size, tokenize_save_floder):
    tokenizer = Tokenizer(models.BPE(unk_token='<unk>'))
    tokenizer.normalizer = normalizers.Sequence(
        [normalizers.NFD(), normalizers.Lowercase(), normalizers.Strip()]
    )
    tokenizer.pre_tokenizer = pre_tokenizers.CharDelimiterSplit(" ")
    print(tokenizer.pre_tokenizer.pre_tokenize_str("This is an example!\r\n"))
    trainer = BpeTrainer(vocab_size=vocab_size, show_progress=True, special_tokens=_special_tokens)
    tokenizer.train(files=files,trainer=trainer)
    tokenizer.model.save(tokenize_save_floder)
    print("Tokenizer Trainning Completed! Vocab size {}".format(tokenizer.get_vocab_size()))

def train_bart_model(config:BartConfig,tokenizer:BartTokenizerFast,corpus_file_path,model_save_path):
    model = BartForConditionalGeneration(config=config)
    print('model size:{}'.format(model.num_parameters()))
    dataset = LineByLineTextDataset(tokenizer=tokenizer,file_path=corpus_file_path,block_size=128)
    data_collator = DataCollatorForLanguageModeling(
        tokenizer=tokenizer, mlm=False
    )
    training_args = TrainingArguments(
        output_dir=model_save_path,
        overwrite_output_dir=True,
        num_train_epochs=1,
        per_device_train_batch_size=32,
        save_steps=10000,
        save_total_limit=2
    )
    trainer = Trainer(
        model=model,
        args=training_args,
        data_collator=data_collator,
        train_dataset=dataset
    )
    trainer.train()
    trainer.save_model(model_save_path)


`data_file_path = '/Users/beansprouts/Documents/corpus/small.txt'
trainBartTokenizer([data_file_path],5000,'./bart_tokenize')
tokenizer = BartTokenizerFast.from_pretrained('./bart_tokenize',max_len=512)
tokenizer.normalizer = normalizers.Sequence(
        [normalizers.NFD(), normalizers.Lowercase(), normalizers.Strip()]
    )
tokenizer.pre_tokenizer = pre_tokenizers.CharDelimiterSplit(" ")
config = BartConfig(vocab_size=tokenizer.vocab_size,
                    max_position_embeddings=514)

train_bart_model(config,tokenizer,data_file_path,'./bart_model')`
@patrickvonplaten, @patil-suraj

Comments:
The HuggingFace Trainer does not support multi-CPU training. From the [docs](https://huggingface.co/transformers/main_classes/trainer.html): 
> The API supports distributed training on multiple GPUs/TPUs, mixed precision through NVIDIA Apex and Native AMP for PyTorch and tf.keras.mixed_precision for TensorFlow.

You can perhaps use [HuggingFace Accelerate](https://github.com/huggingface/accelerate) for this, as it supports multi-CPU both on a single machine as well as multiple machines. From their README:

> Supported integrations
CPU only
multi-CPU on one node (machine)
multi-CPU on several nodes (machines)
single GPU
multi-GPU on one node (machine)
multi-GPU on several nodes (machines)
TPU
FP16 with native AMP (apex on the roadmap)
DeepSpeed support (experimental)

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.