[`Llama2`] replace `self.pretraining_tp` with `self.config.pretraining_tp`
Author: younesbelkada
Pull request: yes

# What does this PR do?

Llama-2 (and also in the past Bloom) has introduced a new attribute in the config file `pretraining_tp` to mimic the behaviour of the original model at inference. Therefore, inside some layers the TP paradigm is "reproduced" by manually simulating the TP paradigm, see for example: https://github.com/huggingface/transformers/blob/476be08c4aa96f8c1cae4200d2677bbe8f12cf80/src/transformers/models/llama/modeling_llama.py#L291 

![Screenshot from 2022-04-22 15-55-48](https://user-images.githubusercontent.com/49240599/164728838-3f78585b-1018-4366-a499-95133fdeaa89.png)

In fact this can lead to unexpected behaviour to users, especially for peft library: (related: https://github.com/huggingface/peft/issues/726) (currently it is not possible to finetune llama-2 models with `pretraining_tp > 1`) 

```bash
  File "/home/younes_huggingface_co/code/transformers/src/transformers/models/llama/modeling_llama.py", line 209, in forward
    gate_proj = torch.cat([F.linear(x, gate_proj_slices[i]) for i in range(self.pretraining_tp)], dim=-1)
  File "/home/younes_huggingface_co/code/transformers/src/transformers/models/llama/modeling_llama.py", line 209, in <listcomp>
    gate_proj = torch.cat([F.linear(x, gate_proj_slices[i]) for i in range(self.pretraining_tp)], dim=-1)
RuntimeError: mat1 and mat2 shapes cannot be multiplied (2048x5120 and 1x6912)
```

I would argue that these slight numerical differences are ok to have in the context of training, thus we should let users the possibility to disable this behaviour, at their own risk. This PR fixes this and proposes to add a new method `disable_pretraining_tp` to disable that behavior. I also added a method to revert the TP behavior in case users want to revert that after training.

On par with: https://github.com/huggingface/peft/pull/728 

cc @ArthurZucker @sgugger @pacman100 @BenjaminBossan

Comments:
_The documentation is not available anymore as the PR was closed or merged._

I'm curious, if we just set `pretraining_tp` to 1 in `config.json`, does it mean we disable it? Because when I see the model, there is some `if-else condition` checking the value of `config.pretraining_tp > 1`.

@fahadh4ilyas the variable is stored as a private attribute so even if you modify the config it won't work sadly, hence the PR

EDT: yes if you manually modify the config file locally on the Hub or locally it will work yes, but this might break existing inference setups (if we modify the official repo)

> @fahadh4ilyas the variable is stored as a private attribute so even if you modify the config it won't work sadly, hence the PR
> 
> EDT: yes if you manually modify the config file locally on the Hub or locally it will work yes, but this might break existing inference setups (if we modify the official repo)

Well at least if you want to fine-tune model based on llama 2, you could just change the config and use that config value right?

yes this is definitely possible and correct, but in terms of UX, I personally would prefer to call a single line `model.disable_tp()` rather than manually going over the config, change it and use that version instead. For end-users and people that are not familiar with TP that want to run training of Llama2 out of the box (for example using PEFT library as linked in the PEFT PR) might get very confused by looking at the error, and this PR combined with the PEFT PR solves the issue. Let's see what arthur and sylvain will say, happy to close the PR if they think it is not relevant !

> yes this is definitely possible and correct, but in terms of UX, I personally would prefer to call a single line `model.disable_tp()` rather than manually going over the config, change it and use that version instead. For end-users and people that are not familiar with TP that want to run training of Llama2 out of the box (for example using PEFT library as linked in the PEFT PR) might get very confused by looking at the error, and this PR combined with the PEFT PR solves the issue. Let's see what arthur and sylvain will say, happy to close the PR if they think it is not relevant !

That's fair enough. I actually also didn't understand at first what is `pretraining_tp` until I saw this pull request. Changing config manually might be possible for people who understand the architecture. But, for people who just want to use the model wont bother changing it.

I'm wondering does having `pretaining_tp` make any difference? If the use case is for parallelism, what I saw in the script is that the script only for-looping each part of weight. I don't see any parallel method there.