The efficiency of transformers running in thread
Author: SeaSpring17
Pull request: no

### System Info

transformers version: 4.29.1
platform: arm64
python version:  3.7.5

When using the model to get the generation in the Thread, the speed is only 1/4 of the main thread, while  the CPU usage is the same.
```python
model = AutoModelWithLMHead.from_pretrained(os.path.join(model_path, "opus-mt-en-zh"))
tokenizer = AutoTokenizer.from_pretrained(os.path.join(model_path, "opus-mt-en-zh"))

batch = self.toenizer.prepare_seq2seq_batch(src_texts=[text])
batch["input_ids"] = torch.Tensor(np.array(batch["input_ids"])[:, :512]).long()
batch["attention_mask"] = torch.Tensor(np.array(batch["attention_mask"])[:, :512]).long()

tt = time.time()
ans = self.base.generate(**batch)
print(f'generate:{time.time() - tt}')
```
In main:
![image](https://github.com/huggingface/transformers/assets/66456697/aed9e837-3efc-4027-9d0d-ab00510f74be)

In thread:
![image](https://github.com/huggingface/transformers/assets/66456697/d50bacd3-7f00-4da1-a7be-ca31b8775ea4)
@ArthurZucker and @younesbelkada

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction
```python
model = AutoModelWithLMHead.from_pretrained(os.path.join(model_path, "opus-mt-en-zh"))
tokenizer = AutoTokenizer.from_pretrained(os.path.join(model_path, "opus-mt-en-zh"))

text = "It's a challenge. We have to face it. God is challenging you. He's calling you a chump. This is the sentence to be translated"

class MyModel_token(nn.Module):
    # 初始化定义
    def __init__(self, m, t):
        super(MyModel_token, self).__init__()
        self.base = m
        self.toenizer = t
        print("init success")
    # 正向传播
    def forward(self, text):
        print(f'input text: {text}')
        tt = time.time()
        batch = self.toenizer.prepare_seq2seq_batch(src_texts=[text])
        print(f'toenizer.prepare_seq2seq_batch:{time.time()-tt}')
        tt = time.time()
        batch["input_ids"] = torch.Tensor(np.array(batch["input_ids"])[:, :512]).long()
        batch["attention_mask"] = torch.Tensor(np.array(batch["attention_mask"])[:, :512]).long()
        print(f'Tensor:{time.time()-tt}')
        tt = time.time()

        ans = self.base.generate(**batch)
        print(f'generate:{time.time() - tt}')
        tt = time.time()
        ans1 = self.toenizer.batch_decode(ans, skip_special_tokens=True)
        print(f'batch_decode:{time.time() - tt}')
        # print(f'ans: {ans}')
        # print(f'ans1: {ans1}')
        return ans1

myModel_token = MyModel_token(model, tokenizer)

def get_translation(raw_s):
    start = time.time()
    ans = myModel_token(raw_s)
    print(f'ans:{ans}, all time:{time.time()-start}')
    return ans

##
start = time.time()
print("\nwith tokenizer")
get_translation(text)
print(f"all time:{time.time()-start:.4f}")

print('\n'*3)
p = Thread(target=get_translation, args=(text,))
p.start()  
```

### Expected behavior

Can a thread run at the same speed as a main function with the same CPU usage

Comments:
Hey! The `transformer` (and the `torch/tf/jax`) library was made to leverage CPU operation in a builtin-manner. It does not support threading. If you want acceleration, you should look into the [optimum library](https://github.com/huggingface/optimum), or torch compilation of model. 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.