[modeling_utils] Keys are still reported as "missing" during model loading even though specified in `_keys_to_ignore_on_load_missing`
Author: hui-wan
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version:  4.18.0.dev0
- Platform:  Ubuntu 18.04.6 LTS   Linux 4.15.0-167-generic
- Python version:  3.7.11
- PyTorch version (GPU?):  1.11.0 (True)
- Tensorflow version (GPU?):  N/A
- Using GPU in script?:  No
- Using distributed or parallel set-up in script?:  No

### Who can help
@patrickvonplaten @sgugger 

<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @

Models:

- ALBERT, BERT, XLM, DeBERTa, DeBERTa-v2, ELECTRA, MobileBert, SqueezeBert: @LysandreJik
- T5, Pegasus, EncoderDecoder: @patrickvonplaten
- Blenderbot, MBART, BART, Marian, Pegasus: @patil-suraj
- Longformer, Reformer, TransfoXL, XLNet, FNet, BigBird: @patrickvonplaten
- FSMT: @stas00
- Funnel: @sgugger
- GPT-2, GPT: @patil-suraj, @patrickvonplaten, @LysandreJik
- RAG, DPR: @patrickvonplaten, @lhoestq
- TensorFlow: @Rocketknight1
- JAX/Flax: @patil-suraj
- TAPAS, LayoutLM, LayoutLMv2, LUKE, ViT, BEiT, DEiT, DETR, CANINE: @NielsRogge
- GPT-Neo, GPT-J, CLIP: @patil-suraj
- Wav2Vec2, HuBERT, SpeechEncoderDecoder, UniSpeech, UniSpeechSAT, SEW, SEW-D, Speech2Text: @patrickvonplaten, @anton-l

If the model isn't in the list, ping @LysandreJik who will redirect you to the correct contributor.

Library:

- Benchmarks: @patrickvonplaten
- Deepspeed: @stas00
- Ray/raytune: @richardliaw, @amogkam
- Text generation: @patrickvonplaten @narsil
- Tokenizers: @SaulLu
- Trainer: @sgugger
- Pipelines: @Narsil
- Speech: @patrickvonplaten, @anton-l
- Vision: @NielsRogge, @sgugger

Documentation: @sgugger

Model hub:

- for issues with a model, report at https://discuss.huggingface.co/ and tag the model's creator.

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj

For research projetcs, please ping the contributor directly. For example, on the following projects:

- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->

## Information

Model I am using (Bert, XLNet ...):  BERT, but the issue should apply to any model

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Define a model class extending PreTrainedModel, with `base_model_prefix`, e.g.`aabbcc`, and a module with name starting with `base_model_prefix`, e.g. `aabbcc_encoder`;
2. Specify `_keys_to_ignore_on_load_missing` to include the module name, e.g. `aabbcc_encoder`;
3. When loading from pretrained, if the keys starting with `aabbcc_encoder` are missing, they are still reported as missing in warning although we specified to ignore them in step 2.

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

Here is a much simplified code snippet that reproduces the issue.
```py
from transformers import (
    AutoModel,
    PreTrainedModel,
    BertConfig,
)


class TestClass(PreTrainedModel):
    base_model_prefix = "aabbcc"
    _keys_to_ignore_on_load_missing = [r"position_ids", r"aabbcc_encoder"]

    def __init__(self, config):
        super(TestClass, self).__init__(config)
        self.aabbcc_encoder = AutoModel.from_config(config)

    def _init_weights(self, module):
        return


model_name = "bert-base-uncased"
config = BertConfig.from_pretrained(model_name)
model, loading_info = TestClass.from_pretrained(
    pretrained_model_name_or_path=model_name,
    config=config,
    output_loading_info=True,
)
print("Missing keys: {}".format(loading_info['missing_keys']))
```

This is the output. 
```
Missing keys: ['encoder.layer.10.attention.self.key.bias', 'encoder.layer.1.attention.self.query.weight', 'encoder.layer.8.attention.output.dense.weight', ....
```
And the logging includes:
```
Some weights of TestClass were not initialized from the model checkpoint at bert-base-uncased and are newly initialized: ['encoder.layer.10.attention.self.key.bias', 'encoder.layer.1.attention.self.query.weight', 'encoder.layer.8.attention.output.dense.weight', ...
```

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->
Expected output:
```
Missing keys: []
```
(And the logging should not include the warning for the missing keys.)

Comments:
Hey @hui-wan,

Thanks a lot for opening this issue. I'm having some problems following the exact error though here:

If `base_model_prefix = "aabbcc"`, then why is the line: `self.aabbcc_encoder = AutoModel.from_config(config)` and not `self.aabbcc = AutoModel.from_config(config)`?

If the `base_model_prefix` is set to `aabbcc` then the name of the encoder should also be exactly this

Thank you @patrickvonplaten.

If the name of the encoder is set to `aabbcc` in the above code snippet, the same output and warning of missing keys are produced:
```
Missing keys: ['encoder.layer.10.attention.self.key.bias', 'encoder.layer.1.attention.self.query.weight', 'encoder.layer.8.attention.output.dense.weight', ....
```
and
```
Some weights of TestClass were not initialized from the model checkpoint at bert-base-uncased and are newly initialized: ['encoder.layer.10.attention.self.key.bias', 'encoder.layer.1.attention.self.query.weight', 'encoder.layer.8.attention.output.dense.weight', ...
```

The issue goes away if the encoder name is set to `bert` or sth like `encoder_aabbcc` (also set in `_keys_to_ignore_on_load_missing`). 
 * In the former case, there would not be missing keys in the first place. However, in my actual code (not the simplified snippet), there is a real need not to set the encoder to just bert.
 * In the latter case, `startswith(prefix)` in `modeling_utils.py` would not be triggered, so the missing keys would successfully match `_keys_to_ignore_on_load_missing`.  

 The latter is my current get-around which works well. Nonetheless, I wanted to report the issue and the easy fix, just in case someone else got into the same situation and got puzzled by the warning. :-)

Thanks!

Ok I think I see. Yes I think in this case it would be advised to not try to overwrite the `base_model_prefix` (think this doesn't work) and instead one should the official prefix of the `AutoModel` being `bert`.

So I think this code snippet would work: 

```py
from transformers import (
    AutoModel,
    PreTrainedModel,
    BertConfig,
)


class TestClass(PreTrainedModel):
    _keys_to_ignore_on_load_missing = [r"position_ids", r"aabbcc_encoder"]

    def __init__(self, config):
        super(TestClass, self).__init__(config)
        self.bert = AutoModel.from_config(config)

    def _init_weights(self, module):
        return


model_name = "bert-base-uncased"
config = BertConfig.from_pretrained(model_name)
model, loading_info = TestClass.from_pretrained(
    pretrained_model_name_or_path=model_name,
    config=config,
    output_loading_info=True,
)
print("Missing keys: {}".format(loading_info['missing_keys']))
```

I see. Thank you @patrickvonplaten!  By removing `base_model_prefix`, indeed the issue is not triggered in the code snippet.

IMHO, the small change in #16318 still would be useful in preventing something similar from happening when `base_model_prefix` happens to be the beginning (but not whole module name) of missing keys or unexpected keys. Would appreciate your opinion on this. Thanks!


Glad that I could be a bit helpful here :-) 

I'm a bit nervous about the PR because it changes one of the core functions that is literally used by **everybody** using Transformers for a rather specific use case. If I understand correctly the PR aims to solve a problem when official code `BertModel` is wrapped into custom code (a higher level class such as `TestClass`) for which we cannot guarantee that things work correctly. So if possible, I'd really like to avoid having to make core changes to `from_pretrained(...)` for this edge case :sweat_smile: 

The PR changes the judging of prefix matching from part-of-word matching to whole-word matching of the first part of model/key, which I believe is the correct logic here. And it passed all the tests. With that being said, I understand your concern about making changes to the core function. :-) Thanks a lot for your help and prompt replies! Also many thanks for your great work!!!