Error with load_tf_weights_in_albert when transforming tf checkpoint to pytorch model
Author: Ala-Na
Pull request: no

### System Info

- `transformers` version: 4.24.0
- Platform: Linux-4.19.128-microsoft-standard-x86_64-with-glibc2.31
- Python version: 3.10.9
- Huggingface_hub version: 0.10.1
- PyTorch version (GPU?): 1.13.1 (False)
- Tensorflow version (GPU?): 2.10.0 (False)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

#### The error happens when :
- Using transformer-cli convert script for ALBERT model 
```
transformers-cli convert --model_type albert \
  --tf_checkpoint $ALBERT_BASE_DIR/model.ckpt-best \
  --config $ALBERT_BASE_DIR/albert_config.json \
  --pytorch_dump_output $ALBERT_BASE_DIR/pytorch_model.bin
```
- Or directly using convert_albert_original_tf_checkpoint_to_pytorch.py script
```
python3 compatibility.py --tf_checkpoint_path $ALBERT_BASE_DIR/model.ckpt-best --pytorch_dump_path $ALBERT_BASE_DIR/pytorch_model.bin --albert_config_file $ALBERT_BASE_DIR/albert_config.json
```

#### The error message :
```
Traceback (most recent call last):
  File "/path/transformers-cli", line 11, in <module>
    sys.exit(main())
  File "/path/transformers/commands/transformers_cli.py", line 55, in main
    service.run()
  File "/path/transformers/commands/convert.py", line 94, in run
    convert_tf_checkpoint_to_pytorch(self._tf_checkpoint, self._config, self._pytorch_dump_output)
  File "/path/transformers/models/albert/convert_albert_original_tf_checkpoint_to_pytorch.py", line 36, in convert_tf_checkpoint_to_pytorch
    load_tf_weights_in_albert(model, config, tf_checkpoint_path)
  File "/path/transformers/models/albert/modeling_albert.py", line 164, in load_tf_weights_in_albert
    pointer = getattr(pointer, "bias")
  File "/path/torch/nn/modules/module.py", line 1207, in __getattr__
    raise AttributeError("'{}' object has no attribute '{}'".format(
AttributeError: 'AlbertEmbeddings' object has no attribute 'bias'
```

#### Code causing this error
Inside load_tf_weights_in_albert function from modeling_albert.py file.
More precisely it's the part with ```scope_names[0] == gamma```` and ```beta``` :
```
        pointer = model
        for m_name in name:
            if re.fullmatch(r"[A-Za-z]+_\d+", m_name):
                scope_names = re.split(r"_(\d+)", m_name)
            else:
                scope_names = [m_name]

            if scope_names[0] == "kernel" or scope_names[0] == "gamma":
                pointer = getattr(pointer, "weight")
            elif scope_names[0] == "output_bias" or scope_names[0] == "beta":
                pointer = getattr(pointer, "bias")
            elif scope_names[0] == "output_weights":
                pointer = getattr(pointer, "weight")
            elif scope_names[0] == "squad":
                pointer = getattr(pointer, "classifier")
            else:
                try:
                    pointer = getattr(pointer, scope_names[0])
                except AttributeError:
                    logger.info(f"Skipping {'/'.join(name)}")
                    continue
            if len(scope_names) >= 2:
                num = int(scope_names[1])
                pointer = pointer[num]
```


#### What I suspect is happening :

In this part of the code, a newly instantiated pytorch ```AlbertForPreTraining``` model (instantiated inside convert_tf_checkpoint_to_pytorch.py) is being filled with tensorflow variables' arrays. In order to achieve this, tensorflow variables are red, their names modified and corresponding arrays are copied to similar pytorch variables. In order to fill the correct pytorch variable/attribute, a pointer is moved to the corresponding element according to the variable name.
This errors occurs when a tensorflow variable either contains a ```beta``` or ```gamma``` in its name (example of variable name : albert/embeddings/layer_normalization/beta). Because, in those cases, the class/object that the pointer is representing doesn't contains any ```bias``` or ```weight```, resulting in an error when the ```getattr``` function is trying to retrieve them.
This seems to happen with every variable name corresponding to a normalization layer.


#### Example of my reasoning :

The current variable name is ```albert/embeddings/layer_normalization/beta```. It was split on ```/``` and we're now on the ```beta``` substring. ```pointer``` is currently pointing to ```AlbertEmbeddings``` object. We reach the condition ```if scope_names[0] == "output_bias" or scope_names[0] == "beta```. ```getattr``` function is trying to retrieve ```bias``` from ```AlbertEmbeddings``` but there is no corresponding attribute, resulting in the displayed error.

#### What should ```pointer``` retrieve inside ```AlbertForPreTraining``` architecture when meeting a ```gamma``` or ```beta``` ? Normalization layer's weight and bias ?

### Expected behavior

Obtaining a pytorch bin file from a tensorflow checkpoint, without errors occurring in the process.

#### Note :
I couldn't find any recent or opened issues on this subject, but similar closed ones are #2006 and #3779

Comments:
Note that this command only works for the original TensorFlow checkpoints of Albert and we do not maintain it as those checkpoints have long been converted and are available on the Hub. To convert Hugging Face models from TensorFlow to PyTorch or vice versa, use [this guide](https://huggingface.co/docs/transformers/model_sharing#convert-a-model-for-all-frameworks).

Thanks @sgugger for your quick answer ! I tried the guide with :
```
pytorch_model = AlbertForPreTraining.from_pretrained("tf_checkpoint_folder", from_tf=True)
pt_model.save_pretrained("generated_pytorch_model")
```
But the same error is occurring.

What do you mean by "original Tensorflow checkpoints" ? 

Where does your TensorFlow checkpoint come from?

From a pretraining with google-resarch official github code

We do not maintain a generic conversion command that works with all repos outside of Hugging Face. The command you are using is the one we used to convert the original ALBERT checkpoints three years ago, but we don't guarantee it will work with more recent ones.

You will need to adapt the code a bit yourself to solve this error, I'm afraid.

I'm answering after trying to adapt a little bit the script. I managed to copy tensorflow variables to seemingly corresponding pytorch tensors. Only optimizers and the layer norm of AlbertAttention module keeps their originally instantiated values (as there is no need to copy optimizers variables and there is not equivalent for the layer norm of attention module). But for some reason, the model I obtained doesn't seems "right" as it doesn't perform learning when fine-tuned on a simple task.
Maybe some others manipulations are needed, such as the transpose operation performed in line 181 of modeling_albert.py ?

Sorry to bother you again with that, but is there someone which could have a slight idea of what could be done and give me some tips ?

Thanks again for your attention


Hi @Ala-Na, thanks for raising an issue! 

For custom situations like this, the question is best placed in the [forums](https://discuss.huggingface.co/). We try to reserve issues for feature requests and bug reports specific to the transformers library. 

Thank you @amyeroberts for the suggestion.

I just created a post about it on the forum : https://discuss.huggingface.co/t/help-appreciated-modifying-load-tf-weights-in-albert-for-transforming-albert-tensorflow-checkpoint-to-pytorch-model/34415

For anyone who may have an idea of what need to be done, please don't hesitate to respond there.
Thanks !

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.