Getting `raise NotImplementedError` for base_model.get_input_embeddings() when upgrading from pytorch-transformers
Author: gsrivas4
Pull request: no
Labels: name: Migration

# 📚 Migration

## Information

<!-- Important information -->
Getting `raise NotImplementedError` at 
 the lines - https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_utils.py#L474-L477 when I am trying to upgrade my code from pytorch-transformers to transformers

Model I am using (Bert, XLNet ...): Bert

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [ ] the official example scripts: (give details below): Not sure
* [ ] my own modified scripts: (give details below): Yes

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name) No 
* [ ] my own task or dataset: (give details below): No

## Details

<!-- A clear and concise description of the migration issue.
    If you have code snippets, please provide it here as well.
    Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
    Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.
    -->
I am using Oscar repo (https://github.com/microsoft/Oscar), which uses an older version of Huggingface pytorch-transformers (https://github.com/huggingface/transformers/tree/067923d3267325f525f4e46f357360c191ba562e). I am trying to upgrade the repo to use latest version of transformers (https://github.com/huggingface/transformers). However, I am getting below error : 
```
Traceback (most recent call last):
  File "oscar/run_captioning_airsplay.py", line 1019, in <module>
    main()
  File "oscar/run_captioning_airsplay.py", line 966, in main
    from_tf=bool('.ckpt' in args.model_name_or_path), config=config)
  File "/home/default/ephemeral_drive/work/image_captioning/Oscar_edited/transformers/src/transformers/modeling_utils.py", line 1188, in from_pretrained
    model.tie_weights()
  File "/home/default/ephemeral_drive/work/image_captioning/Oscar_edited/transformers/src/transformers/modeling_utils.py", line 504, in tie_weights
    self._tie_or_clone_weights(output_embeddings, self.get_input_embeddings())
  File "/home/default/ephemeral_drive/work/image_captioning/Oscar_edited/transformers/src/transformers/modeling_utils.py", line 469, in get_input_embeddings
    return base_model.get_input_embeddings()
  File "/home/default/ephemeral_drive/work/image_captioning/Oscar_edited/transformers/src/transformers/modeling_utils.py", line 471, in get_input_embeddings
    raise NotImplementedError
NotImplementedError
```
The error occurs at this block in the transformers code - https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_utils.py#L474-L477. My code runs fine when I use an older version of hugging face transformers - https://github.com/huggingface/transformers/tree/067923d3267325f525f4e46f357360c191ba562e, possibly, because pytorch-transformers did not have a requirement that `set_input_embeddings()` should be defined for base_model. The base model that I am using is a custom defined model `BertForImageCaptioning` (https://github.com/microsoft/Oscar/blob/df79152b708c3c46f2dc93324776a27406ccc634/oscar/modeling/modeling_bert.py#L604), which has a custom defined parent class ` CaptionPreTrainedModel` (https://github.com/microsoft/Oscar/blob/df79152b708c3c46f2dc93324776a27406ccc634/oscar/modeling/modeling_utils.py#L21), which has a parent class `BertPreTrainedModel`.

I have not seen any mention of how to deal with this issue in the migration documents from Pytorch-transformers or from transformers 3.x. (https://huggingface.co/transformers/migration.html#migrating-from-transformers-v3-x-to-v4-x). 

I have looked into examples to check how to define the function, but this did not give enough details to define the function at my side - https://github.com/huggingface/transformers/blob/master/examples/research_projects/movement-pruning/emmental/modeling_bert_masked.py#L487-L488. 

How should I define the function `get_input_embeddings()` for my use case and what are the guidelines for doing the same. Are there any examples explaining the process of defining the function.  

## Environment info
<!-- You can run the command `python transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
 
- `transformers` version: https://github.com/huggingface/transformers
- Platform: x86_64 GNU/Linux
- Python version: 3.6.8
- PyTorch version (GPU?):  1.7.0+cu101 (GPU)
- Tensorflow version (GPU?): 2.3.0 (GPU)
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: No

<!-- IMPORTANT: which version of the former library do you use? -->
* `pytorch-transformers` or `pytorch-pretrained-bert` version (or branch): https://github.com/huggingface/transformers/tree/067923d3267325f525f4e46f357360c191ba562e


## Checklist

- [ Yes] I have read the migration guide in the readme.
 ([pytorch-transformers](https://github.com/huggingface/transformers#migrating-from-pytorch-transformers-to-transformers);
  [pytorch-pretrained-bert](https://github.com/huggingface/transformers#migrating-from-pytorch-pretrained-bert-to-transformers))
- [ yes] I checked if a related official extension example runs on my machine.


Comments:
Hello! Do you have a reproducible code example so that we can try to understand what's happening here? Thank you!

I have generated a simplified version of the original Oscar (https://github.com/microsoft/Oscar) codebase here - https://github.com/gsrivas4/Oscar_latest. The branch `old_transformers` -https://github.com/gsrivas4/Oscar_latest/tree/old_transformers uses an old version of hugging face without an issue. However, the branch `latest_transformers` - https://github.com/gsrivas4/Oscar_latest/tree/latest_transformer gets below error when I run the command `oscar/run_captioning.py --model_name_or_path pretrained_models/base-vg-labels/ep_67_588997 --do_train --do_lower_case --evaluate_during_training --add_od_labels --learning_rate 0.00003 --per_gpu_train_batch_size 64 --num_train_epochs 30 --save_steps 5000 --output_dir output/`:
```
Traceback (most recent call last):
  File "oscar/run_captioning.py", line 1010, in <module>
    main()
  File "oscar/run_captioning.py", line 966, in main
    from_tf=bool('.ckpt' in args.model_name_or_path), config=config)
  File "/usr/local/lib/python3.6/site-packages/transformers/modeling_utils.py", line 1185, in from_pretrained
    model.tie_weights()
  File "/usr/local/lib/python3.6/site-packages/transformers/modeling_utils.py", line 497, in tie_weights
    self._tie_or_clone_weights(output_embeddings, self.get_input_embeddings())
  File "/usr/local/lib/python3.6/site-packages/transformers/modeling_utils.py", line 462, in get_input_embeddings
    return base_model.get_input_embeddings()
  File "/usr/local/lib/python3.6/site-packages/transformers/modeling_utils.py", line 464, in get_input_embeddings
    raise NotImplementedError
NotImplementedError
``` 

To replicate the experiment, follow the [README.md](https://github.com/gsrivas4/Oscar_latest/blob/old_transformers/README.md) file to use old version of transformers - https://github.com/huggingface/transformers/tree/067923d3267325f525f4e46f357360c191ba562e. Follow the [README.md](https://github.com/gsrivas4/Oscar_latest/blob/latest_transformer/README.md) to run the code with latest transformers. 

The platform information is below:
Platform: x86_64 GNU/Linux
Python version: 3.6.8
PyTorch version (GPU?): 1.7.0+cu101 (GPU)
Tensorflow version (GPU?): 2.3.0 (GPU)
Using GPU in script?: yes
Using distributed or parallel set-up in script?: No

Let me know if you have any issues generating the setup.


It seems your `BertForImageCaptioning` doesn't have a `get_input_embeddings()` method, and neither does your `CaptionPreTrainedModel`. 

You should implement that method on either of those in order to be able to resize them, like it is done in the `BertModel` for example:

https://github.com/huggingface/transformers/blob/6c25f5228e7fb48a520f63ee82dd9ce25b27d6df/src/transformers/models/bert/modeling_bert.py#L853-L854

Sorry for the inconvenience!


@LysandreJik I understand that I have to define the function `get_input_embeddings()` and I have also looked at the sample example where this function is defined - https://github.com/huggingface/transformers/blob/master/examples/research_projects/movement-pruning/emmental/modeling_bert_masked.py#L487-L488. It would be great if some description is given about the inputs and outputs of this function in a bit more detailed way. It would be also beneficial if the details about this function are documented in the migration document.
  

@LysandreJik I could resolve the issue by adding definition for the function at following lines in my code - https://github.com/gsrivas4/Oscar_latest/blob/latest_transformer/oscar/modeling/modeling_bert.py#L190-L191. Thanks for the help.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.