Finetune Hubert model : Adding new vocabulary
Author: harrypotter90
Pull request: no

Environment info
```
transformers version: 4.12.2
Platform: Mac
Python version: 3.7
PyTorch version (GPU?): 1.9
Tensorflow version (GPU?): No
Using GPU in script?: No
Using distributed or parallel setup in script?: No
```
I just run simple code to load Hubert pretrained base model

```
from transformers import Wav2Vec2Processor, HubertForCTC
import torch
import librosa
PROCESSOR = Wav2Vec2Processor.from_pretrained('facebook/hubert-large-ls960-ft')
model = HubertForCTC.from_pretrained('facebook/hubert-large-ls960-ft')
tokenizer = PROCESSOR.tokenizer
```

On a smaller dataset, i am able to get good WER around 0.0

But if I add new tokens/vocabulary to it by using the below code:

```
from transformers import Wav2Vec2Processor, HubertForCTC
import torch
import librosa
PROCESSOR = Wav2Vec2Processor.from_pretrained('facebook/hubert-large-ls960-ft')
model = HubertForCTC.from_pretrained('facebook/hubert-large-ls960-ft')
tokenizer = PROCESSOR.tokenizer
tokenizer.add_tokens(new_tokens=[' ','Ä','Ö','Ü'])
```


The loss and WER go bad and then worse (clearly), and later loss is NAN.

Is it the correct way to add new alphabets? 




dataset is same in both trainings

Comments:
cc @anton-l @patrickvonplaten

Hi @harrypotter90! Here you're adding a space (`' '`) as a separate token, while the tokenizer already has a special separator token `'|'`, that is a replacement for all whitespaces. Try adding just the new letters and fine-tune again.

Also I would recommend resizing the `lm_head` of Hubert to include your newly added tokens. Otherwise the model will never predict those

Thanks for the prompt reply. 

Error on this line :  `model.resize_token_embeddings(len(tokenizer))`

```
Traceback (most recent call last):
  File "train-main.py", line 47, in <module>
    model.resize_token_embeddings(len(tokenizer))
  File "/opt/anaconda/envs/opence/lib/python3.8/site-packages/transformers/modeling_utils.py", line 724, in resize_token_embeddings
    model_embeds = self._resize_token_embeddings(new_num_tokens)
  File "/opt/anaconda/envs/opence/lib/python3.8/site-packages/transformers/modeling_utils.py", line 738, in _resize_token_embeddings
    old_embeddings = self.get_input_embeddings()
  File "/opt/anaconda/envs/opence/lib/python3.8/site-packages/transformers/modeling_utils.py", line 563, in get_input_embeddings
    return base_model.get_input_embeddings()
  File "/opt/anaconda/envs/opence/lib/python3.8/site-packages/transformers/modeling_utils.py", line 565, in get_input_embeddings
    raise NotImplementedError
NotImplementedError
```


Using latest transformers: 
Name: transformers
Version: 4.12.3

Yeah this won't work because there are no input embeddings. Can you try:

```
tokenizer = PROCESSOR.tokenizer
tokenizer.add_tokens(new_tokens=['Ä','Ö','Ü'])
model = HubertForCTC.from_pretrained('facebook/hubert-large-ls960-ft', vocab_size=len(tokenizer))
```

After : 
`model = HubertForCTC.from_pretrained('facebook/hubert-large-ls960-ft', vocab_size=len(tokenizer))
`
Got error:
```
Traceback (most recent call last):
  File "train-main.py", line 49, in <module>
    model = HubertForCTC.from_pretrained(MODEL_NAME, gradient_checkpointing=True, ctc_loss_reduction="mean", vocab_size=len(tokenizer))
  File "/opt/anaconda/envs/opence/lib/python3.8/site-packages/transformers/modeling_utils.py", line 1441, in from_pretrained
    model, missing_keys, unexpected_keys, mismatched_keys, error_msgs = cls._load_state_dict_into_model(
  File "/opt/anaconda/envs/opence/lib/python3.8/site-packages/transformers/modeling_utils.py", line 1595, in _load_state_dict_into_model
    raise RuntimeError(f"Error(s) in loading state_dict for {model.__class__.__name__}:\n\t{error_msg}")
RuntimeError: Error(s) in loading state_dict for HubertForCTC:
        size mismatch for lm_head.weight: copying a param with shape torch.Size([32, 1024]) from checkpoint, the shape in current model is torch.Size([35, 1024]).
```
        size mismatch for lm_head.bias: copying a param with shape torch.Size([32]) from checkpoint, the shape in current model is torch.Size([35]).

How about `ignore_mismatched_sizes=True,` , wild guess, if it is just a warning ?


At least this started the training.. I will update if WER goes down or not .

Hey @harrypotter90,

exactly sorry I forgot to mention this parameter. To summarize, I would recommend to add new tokens and load your model as follows:

```python
from transformers import Wav2Vec2Processor, AutoModelForCTC

# load tokenizer & feature extractor
processor = Wav2Vec2Processor.from_pretrained('facebook/hubert-large-ls960-ft')

# add new tokens
tokenizer = processor.tokenizer
tokenizer.add_tokens(new_tokens=['Ä','Ö','Ü'])

# load pretrained model and replace fine-tuned head with resized randomly initialized head
model = AutoModelForCTC.from_pretrained("facebook/hubert-large-ls960-ft", vocab_size=len(tokenizer), ignore_mismatched_sizes=True)

# now use model for training
```

Cool, it worked. Thank you