[PyTorch] Refactor Resize Token Embeddings
Author: patrickvonplaten
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

This PR extends the `resize_embeddings` function in PyTorch to models that have input/output embeddings that are **not** tied.
In PyTorch all models that have tied input/output embeddings by default can also untie those embeddings by setting `config.tie_word_embeddings=False`. This however requires the `_resize_token_embeddings` to be extended to also resize the `lm_head`. This PR does this extension by adding a `_get_resized_lm_head` method. Also, all models that have a `get_output_embedding()` function, now need a `set_output_embedding()` function. A test is added to make sure the new functionality works as expected. The Bart-like models currently skip this test because there is a rather weird `lm_head` behavior that I want to refactor in another PR.

In addition this PR:

- Fixes #8706: With MT5 and T5v1_1, T5 now has a configuration where input and output embeddings are not tied anymore. This PR fixes this.
- Refactors MobileBert




## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, XLM: @LysandreJik
 GPT2: @LysandreJik, @patrickvonplaten
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten, @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @patrickvonplaten
 T5: @patrickvonplaten
 Rag: @patrickvonplaten, @lhoestq
 EncoderDecoder: @patrickvonplaten
 Longformer, Reformer: @patrickvonplaten
 TransfoXL, XLNet: @TevenLeScao, @patrickvonplaten
 examples/seq2seq: @patil-suraj
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 FSMT: @stas00
 -->


Comments:
MobileBERT does this in the `tie_weights` function. Should we do the same here?

ALBERT also does it in the `_resize_token_embeddings`:

https://github.com/huggingface/transformers/blob/a7d46a060930242cd1de7ead8821f6eeebb0cd06/src/transformers/models/albert/modeling_albert.py#L635-L639

It probably should have been done in that method for MobileBERT as well

Fine by me :-) Should we do the mobileBERT change in this PR?

> Fine by me :-) Should we do the mobileBERT change in this PR?

will do!

@patrickvonplaten Not 100% sure that this fix works, forked and implemented the fixes as shown and receive the following error upon training:
```
 Expected tensor for argument #1 'indices' to have scalar type Long; but got torch.cuda.FloatTensor instead (while checking arguments for embedding). 
```
If I revert back to transformers-4.0.0 and run exactly the same script, with the same data then T5 trains fine. Could be user error, although I hope not! Did look back through the script and try and diagnose, but no luck as of yet.

> @patrickvonplaten Not 100% sure that this fix works, forked and implemented the fixes as shown and receive the following error upon training:
> 
> ```
>  Expected tensor for argument #1 'indices' to have scalar type Long; but got torch.cuda.FloatTensor instead (while checking arguments for embedding). 
> ```
> 
> If I revert back to transformers-4.0.0 and run exactly the same script, with the same data then T5 trains fine. Could be user error, although I hope not! Did look back through the script and try and diagnose, but no luck as of yet.

Can you attach a simple code snippet showing what code produces your error? It's for T5 no? 

> 
> 
> > @patrickvonplaten Not 100% sure that this fix works, forked and implemented the fixes as shown and receive the following error upon training:
> > ```
> >  Expected tensor for argument #1 'indices' to have scalar type Long; but got torch.cuda.FloatTensor instead (while checking arguments for embedding). 
> > ```
> > 
> > 
> > If I revert back to transformers-4.0.0 and run exactly the same script, with the same data then T5 trains fine. Could be user error, although I hope not! Did look back through the script and try and diagnose, but no luck as of yet.
> 
> Can you attach a simple code snippet showing what code produces your error? It's for T5 no?

Sure:

```python
from transformers import T5TokenizerFast, T5ForConditionalGeneration
dev = "cuda"

MODEL_NAME = 'google/t5-v1_1-base'
tokenizer = T5TokenizerFast.from_pretrained('t5-base')
special_tokens_dict = {'additional_special_tokens': ['<ORG>','<PERSON>']}
num_added_tokens = tokenizer.add_special_tokens(special_tokens_dict)
print(f'ADDED TOKENS: {num_added_tokens}')
model = T5ForConditionalGeneration.from_pretrained(MODEL_NAME)
model.resize_token_embeddings(len(tokenizer))
model.to(dev)
BATCH_SIZE = 8
```

```python
#Sets the module in training mode
from IPython.display import HTML, display
def progress(loss,value, max=100):
 return HTML(""" Batch loss :{loss}      <progress    
value='{value}'max='{max}',style='width: 100%'>{value}
      </progress>             """.format(loss=loss,value=value, max=max))

model.train()
num_of_batches= int(len(train_df) / BATCH_SIZE)
print(num_of_batches)
NUM_EPOCHS = 1
loss_per_10_steps=[]
loss_values = []
for epoch in range(1,NUM_EPOCHS+1):
  print('Running epoch: {}'.format(epoch))
  
  running_loss=0

  out = display(progress(1, num_of_batches+1), display_id=True)
  for i in range(num_of_batches):
    inputbatch=[]
    labelbatch=[]
    new_df=train_df[i*BATCH_SIZE:i*BATCH_SIZE+BATCH_SIZE]
    for indx,row in new_df.iterrows():
      input = 'Product: '+row['product_name']
      labels = row['product_description']
      inputbatch.append(input)
      labelbatch.append(labels)
    inputbatch=tokenizer.batch_encode_plus(inputbatch,padding=True, max_length=512,return_tensors='pt')["input_ids"]
    labelbatch=tokenizer.batch_encode_plus(labelbatch,padding=True, max_length=512,return_tensors="pt") ["input_ids"]
    inputbatch=inputbatch.to(dev)
    labelbatch=labelbatch.to(dev)

    # clear out the gradients of all Variables 
    optimizer.zero_grad()

    # Forward propogation
    outputs = model(input_ids=inputbatch, labels=labelbatch)
    loss = outputs.loss
    loss_num=loss.item()
    logits = outputs.logits
    running_loss+=loss_num
    if i%10 ==0:      
      loss_per_10_steps.append(loss_num)
    out.update(progress(loss_num,i, num_of_batches+1))

    # calculating the gradients
    loss.backward()

    #updating the params
    optimizer.step()
  
    loss_values.append(loss_num)
  running_loss=running_loss/int(num_of_batches)
```



> > > @patrickvonplaten Not 100% sure that this fix works, forked and implemented the fixes as shown and receive the following error upon training:
> > > ```
> > >  Expected tensor for argument #1 'indices' to have scalar type Long; but got torch.cuda.FloatTensor instead (while checking arguments for embedding). 
> > > ```
> > > 
> > > 
> > > If I revert back to transformers-4.0.0 and run exactly the same script, with the same data then T5 trains fine. Could be user error, although I hope not! Did look back through the script and try and diagnose, but no luck as of yet.
> > 
> > 
> > Can you attach a simple code snippet showing what code produces your error? It's for T5 no?
> 
> Sure:
> 
> ```
> MODEL_NAME = 'google/t5-v1_1-base'
> tokenizer = T5TokenizerFast.from_pretrained('t5-base')
> special_tokens_dict = {'additional_special_tokens': ['<ORG>','<PERSON>']}
> num_added_tokens = tokenizer.add_special_tokens(special_tokens_dict)
> print(f'ADDED TOKENS: {num_added_tokens}')
> model = T5ForConditionalGeneration.from_pretrained(MODEL_NAME, return_dict=True)
> model.resize_token_embeddings(len(tokenizer))
> model.to(dev)
> BATCH_SIZE = 8
> ```
> 
> ```
> #Sets the module in training mode
> from IPython.display import HTML, display
> def progress(loss,value, max=100):
>  return HTML(""" Batch loss :{loss}      <progress    
> value='{value}'max='{max}',style='width: 100%'>{value}
>       </progress>             """.format(loss=loss,value=value, max=max))
> 
> model.train()
> num_of_batches= int(len(train_df) / BATCH_SIZE)
> print(num_of_batches)
> NUM_EPOCHS = 1
> loss_per_10_steps=[]
> loss_values = []
> for epoch in range(1,NUM_EPOCHS+1):
>   print('Running epoch: {}'.format(epoch))
>   
>   running_loss=0
> 
>   out = display(progress(1, num_of_batches+1), display_id=True)
>   for i in range(num_of_batches):
>     inputbatch=[]
>     labelbatch=[]
>     new_df=train_df[i*BATCH_SIZE:i*BATCH_SIZE+BATCH_SIZE]
>     for indx,row in new_df.iterrows():
>       input = 'Product: '+row['product_name']
>       labels = row['product_description']
>       inputbatch.append(input)
>       labelbatch.append(labels)
>     inputbatch=tokenizer.batch_encode_plus(inputbatch,padding=True, max_length=512,return_tensors='pt')["input_ids"]
>     labelbatch=tokenizer.batch_encode_plus(labelbatch,padding=True, max_length=512,return_tensors="pt") ["input_ids"]
>     inputbatch=inputbatch.to(dev)
>     labelbatch=labelbatch.to(dev)
> 
>     # clear out the gradients of all Variables 
>     optimizer.zero_grad()
> 
>     # Forward propogation
>     outputs = model(input_ids=inputbatch, labels=labelbatch)
>     loss = outputs.loss
>     loss_num=loss.item()
>     logits = outputs.logits
>     running_loss+=loss_num
>     if i%10 ==0:      
>       loss_per_10_steps.append(loss_num)
>     out.update(progress(loss_num,i, num_of_batches+1))
> 
>     # calculating the gradients
>     loss.backward()
> 
>     #updating the params
>     optimizer.step()
>   
>     loss_values.append(loss_num)
>   running_loss=running_loss/int(num_of_batches)
> ```

thanks! `dev` would be equal to `"cuda"` I suppose? 

> 
> 
> > > > @patrickvonplaten Not 100% sure that this fix works, forked and implemented the fixes as shown and receive the following error upon training:
> > > > ```
> > > >  Expected tensor for argument #1 'indices' to have scalar type Long; but got torch.cuda.FloatTensor instead (while checking arguments for embedding). 
> > > > ```
> > > > 
> > > > 
> > > > If I revert back to transformers-4.0.0 and run exactly the same script, with the same data then T5 trains fine. Could be user error, although I hope not! Did look back through the script and try and diagnose, but no luck as of yet.
> > > 
> > > 
> > > Can you attach a simple code snippet showing what code produces your error? It's for T5 no?
> > 
> > 
> > Sure:
> > ```
> > MODEL_NAME = 'google/t5-v1_1-base'
> > tokenizer = T5TokenizerFast.from_pretrained('t5-base')
> > special_tokens_dict = {'additional_special_tokens': ['<ORG>','<PERSON>']}
> > num_added_tokens = tokenizer.add_special_tokens(special_tokens_dict)
> > print(f'ADDED TOKENS: {num_added_tokens}')
> > model = T5ForConditionalGeneration.from_pretrained(MODEL_NAME, return_dict=True)
> > model.resize_token_embeddings(len(tokenizer))
> > model.to(dev)
> > BATCH_SIZE = 8
> > ```
> > 
> > 
> > ```
> > #Sets the module in training mode
> > from IPython.display import HTML, display
> > def progress(loss,value, max=100):
> >  return HTML(""" Batch loss :{loss}      <progress    
> > value='{value}'max='{max}',style='width: 100%'>{value}
> >       </progress>             """.format(loss=loss,value=value, max=max))
> > 
> > model.train()
> > num_of_batches= int(len(train_df) / BATCH_SIZE)
> > print(num_of_batches)
> > NUM_EPOCHS = 1
> > loss_per_10_steps=[]
> > loss_values = []
> > for epoch in range(1,NUM_EPOCHS+1):
> >   print('Running epoch: {}'.format(epoch))
> >   
> >   running_loss=0
> > 
> >   out = display(progress(1, num_of_batches+1), display_id=True)
> >   for i in range(num_of_batches):
> >     inputbatch=[]
> >     labelbatch=[]
> >     new_df=train_df[i*BATCH_SIZE:i*BATCH_SIZE+BATCH_SIZE]
> >     for indx,row in new_df.iterrows():
> >       input = 'Product: '+row['product_name']
> >       labels = row['product_description']
> >       inputbatch.append(input)
> >       labelbatch.append(labels)
> >     inputbatch=tokenizer.batch_encode_plus(inputbatch,padding=True, max_length=512,return_tensors='pt')["input_ids"]
> >     labelbatch=tokenizer.batch_encode_plus(labelbatch,padding=True, max_length=512,return_tensors="pt") ["input_ids"]
> >     inputbatch=inputbatch.to(dev)
> >     labelbatch=labelbatch.to(dev)
> > 
> >     # clear out the gradients of all Variables 
> >     optimizer.zero_grad()
> > 
> >     # Forward propogation
> >     outputs = model(input_ids=inputbatch, labels=labelbatch)
> >     loss = outputs.loss
> >     loss_num=loss.item()
> >     logits = outputs.logits
> >     running_loss+=loss_num
> >     if i%10 ==0:      
> >       loss_per_10_steps.append(loss_num)
> >     out.update(progress(loss_num,i, num_of_batches+1))
> > 
> >     # calculating the gradients
> >     loss.backward()
> > 
> >     #updating the params
> >     optimizer.step()
> >   
> >     loss_values.append(loss_num)
> >   running_loss=running_loss/int(num_of_batches)
> > ```
> 
> thanks! `dev` would be equal to `"cuda"` I suppose?

Yeah "cuda" sorry.

Could you also attach some code for `train_df` and `optimizer`? So that I can fully reproduce :-) 

> 
> 
> Could you also attach some code for `train_df` and `optimizer`? So that I can fully reproduce :-)

Sure!
```
optimizer = Adafactor(model.parameters(),lr=1e-3,
                      eps=(1e-30, 1e-3),
                      clip_threshold=1.0,
                      decay_rate=-0.8,
                      beta1=None,
                      weight_decay=0.0,
                      relative_step=False,
                      scale_parameter=False,
                      warmup_init=False)
```

train_df is just a dataframe containing something like the following:
                product_name	product_description
37245	Test Product 1	Test Description 1
23451	Test Product 2  Test Description 2 

Not sure how to attach a file via GitHub my apologies.


> @patrickvonplaten Not 100% sure that this fix works, forked and implemented the fixes as shown and receive the following error upon training:
> 
> ```
>  Expected tensor for argument #1 'indices' to have scalar type Long; but got torch.cuda.FloatTensor instead (while checking arguments for embedding). 
> ```
> 
> If I revert back to transformers-4.0.0 and run exactly the same script, with the same data then T5 trains fine. Could be user error, although I hope not! Did look back through the script and try and diagnose, but no luck as of yet.

I tried with some dummy training data and it works for me...not sure what the problem is. Also the error message hints at a wrong `dtype` of either `input_ids` or `labels`...

Could you try to do the following:

```python
    inputbatch=inputbatch.to(dev).to(torch.long)
    labelbatch=labelbatch.to(dev).to(torch.long)
```

and see if the error persists?

> ```python
> .to(torch.long)
> ```

never mind, I can reproduce! Thanks for the message! Will see how to fix it -> weird error

> 
> 
> > ```python
> > .to(torch.long)
> > ```
> 
> never mind, I can reproduce! Thanks for the message! Will see how to fix it -> weird error

Great thank you :) 

Previous issue describing the same error #7026 . Gave me some guidance but couldn't quite work it out.

> > > ```python
> > > .to(torch.long)
> > > ```
> > 
> > 
> > never mind, I can reproduce! Thanks for the message! Will see how to fix it -> weird error
> 
> Great thank you :)
> 
> Previous issue describing the same error #7026 . Gave me some guidance but couldn't quite work it out.

Should be good now - was 100% introduces by this PR -> thanks a lot for spotting it!

> 
> 
> > > > ```python
> > > > .to(torch.long)
> > > > ```
> > > 
> > > 
> > > never mind, I can reproduce! Thanks for the message! Will see how to fix it -> weird error
> > 
> > 
> > Great thank you :)
> > Previous issue describing the same error #7026 . Gave me some guidance but couldn't quite work it out.
> 
> Should be good now - was 100% introduces by this PR -> thanks a lot for spotting it!

Amazing thank you will run through a test this evening. 

@sgugger @LysandreJik - I updated the PR description. It's good to merge for me. Let me know what you think.