How to untie input and output word embeddings (make embeddings independent) for Bart?
Author: baekg
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.5.1
- Platform: Linux-5.8.0-40-generic-x86_64-with-glibc2.17
- Python version: 3.8.10
- PyTorch version (GPU?): 1.8.1 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help
@patrickvonplaten, @patil-suraj


## Information

Model I am using (Bert, XLNet ...): BartForConditionalGeneration
I want to make "shared embeddings" non-trainabe but "lm-head" trainable but couldn't because of the way "lm-head" is initialised.
In bart’s modeling_utils.py there is the following piece of code. Now what this does is that BartForConditionalGeneration’s lm_head is tied to the token embeddings used by Bart’s enc/dec. This is wrong right? We just these two parameters to be initialised the same but not have the same value throughout.
Use of embeddings matrix-> this to convert sparse tokens at enc/dec input to dense embeddings
Use of lm_head -> this to convert dense embeddings at the decoder output side to a logits vector across tokens in the vocabulary.
Is there a reason they should be tied even when fine-tuning bart? Why is then this not the case when the "touch script" mode is on (as evident from the line ```output_embeddings.weight = nn.Parameter(input_embeddings.weight.clone())``` ) ?
```
def _tie_or_clone_weights(self, output_embeddings, input_embeddings):
    """Tie or clone module weights depending of whether we are using TorchScript or not"""
    if self.config.torchscript:
        output_embeddings.weight = nn.Parameter(input_embeddings.weight.clone())
    else:
         ################################
         ### NOTE THE FOLLOWING LINE ###
         ################################
        output_embeddings.weight = input_embeddings.weight

    if getattr(output_embeddings, "bias", None) is not None:
        output_embeddings.bias.data = torch.nn.functional.pad(
            output_embeddings.bias.data,
            (
                0,
                output_embeddings.weight.shape[0] - output_embeddings.bias.shape[0],
            ),
            "constant",
            0,
        )
    if hasattr(output_embeddings, "out_features") and hasattr(input_embeddings, "num_embeddings"):
        output_embeddings.out_features = input_embeddings.num_embeddings
```

## Expected behavior
The behaviour in torchscript mode should be kept default. This is because only the initialised weights of lm-head and embeddings should be the same not throughout the training.

Comments:
@patil-suraj any comment?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

Hey @baekg,

Sorry for replying so late here. By default BART's input and output word embeddings are tied (like it is the case for almost all models: BERT, RoBERTa, T5, ...). This means that the input and output word embeddings weights are identical and will stay identical during training. Each backward pass therefore does two gradient updates for one weight matrix. You can verify that easily by doing the following:

```python
from transformers import BartForConditionalGeneration

model = BartForConditionalGeneration.from_pretrained("facebook/bart-large")

# change value in input word embedding
model.model.shared.weight.data[0, 0] = -10.0

# see how this has also changed the value in `lm_head`
model.lm_head.weight.data[0, 0]
```

Now by passing `tie_word_embeddings=False` to the model at initialization, you can "untie" the input word embeddings and `lm_head`. See:

```python
from transformers import BartForConditionalGeneration

model = BartForConditionalGeneration.from_pretrained("facebook/bart-large", tie_word_embeddings=False)

# change value in input word embedding
model.model.shared.weight.data[0, 0] = -10.0

# see how this has also changed the value in `lm_head`
model.lm_head.weight.data[0, 0]
```


This way you can make the input embeddings non-trainable (set `.requires_grad=False) without the `lm_head` being affected.

For `torchscript` the word embeddings are untied by default since the compilation doesn't word otherwise.


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.