Where I can find official pretrained weights of SOP in Albert and NSP in Bert?
Author: s4sarath
Pull request: no

Hi Guys,

I was checking for the pre-trained weights ( 2 layer classifier ) of ```SOP``` in ```Albert``` and ```NSP``` in ```Bert```. 
Seems like it is initializing randomly every time. 

Can we have the official weights loaded here or is it not available from official models?
Can anyone clarify please.

Thanks.

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

Is there any update?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

@LysandreJik - Can you help me here :)?

Hi @s4sarath, sorry for the delayed response. If the checkpoints on the hub do not satisfy you (I see the SOP/NSP layers are indeed lacking), conversion scripts are available for each model:

- [BERT](https://github.com/huggingface/transformers/tree/master/src/transformers/models/bert), see the `convert_*` scripts
- [ALBERT](https://github.com/huggingface/transformers/blob/master/src/transformers/models/albert/convert_albert_original_tf_checkpoint_to_pytorch.py)

I confirm this successfully exports the full model, including the NSP/SOP weights:

#### ALBERT

```bash
wget https://storage.googleapis.com/albert_models/albert_base_v2.tar.gz 
tar -xzf albert_base_v2.tar.gz 
python convert_albert_original_tf_checkpoint_to_pytorch.py --tf_checkpoint_path=albert_base/model.ckpt-best --albert_config_file=albert_base/albert_config.json --pytorch_dump_path=albert_base/pytorch_model.bin
cp albert_base/albert_config.json albert_base/config.json   
```
```python
>>> from transformers import TFAlbertForPreTraining
>>> model = TFAlbertForPreTraining.from_pretrained("albert_base", from_pt=True)
[...]
All the weights of TFAlbertForPreTraining were initialized from the PyTorch model.
If your task is similar to the task the model of the checkpoint was trained on, you can already use TFAlbertForPreTraining for predictions without further training.
```

#### BERT

Same for BERT:

```bash
wget https://storage.googleapis.com/bert_models/2018_10_18/uncased_L-12_H-768_A-12.zip
unzip uncased_L-12_H-768_A-12.zip  
python convert_bert_original_tf_checkpoint_to_pytorch.py --tf_checkpoint_path=uncased_L-12_H-768_A-12/bert_model.ckpt --bert_config_file=uncased_L-12_H-768_A-12/bert_config.json --pytorch_dump_path=uncased_L-12_H-768_A-12/pytorch_model.bin
```
```python
>>> from transformers import TFBertForPreTraining
>>> bert = TFBertForPreTraining.from_pretrained("uncased_L-12_H-768_A-12", from_pt=True)
[...[
All the weights of TFBertForPreTraining were initialized from the PyTorch model.
If your task is similar to the task the model of the checkpoint was trained on, you can already use TFBertForPreTraining for predictions without further training.
```

Hope that helps.





Thanks Lysandre. No worries.

I kind of did same hack. But was wondering, isit something that is supposed
to be a part of official model loading.

Thanks
Sarath

On Wed, 11 Aug, 2021, 8:12 pm Lysandre Debut, ***@***.***>
wrote:

> Hi @s4sarath <https://github.com/s4sarath>, sorry for the delayed
> response. If the checkpoints on the hub do not satisfy you (I see the
> SOP/NSP layers are indeed lacking), conversion scripts are available for
> each model:
>
>    - BERT
>    <https://github.com/huggingface/transformers/tree/master/src/transformers/models/bert>,
>    see the convert_* scripts
>    - ALBERT
>    <https://github.com/huggingface/transformers/blob/master/src/transformers/models/albert/convert_albert_original_tf_checkpoint_to_pytorch.py>
>
> I confirm this successfully exports the full model, including the NSP/SOP
> weights:
> ALBERT
>
> wget https://storage.googleapis.com/albert_models/albert_base_v2.tar.gz
> tar -xzf albert_base_v2.tar.gz
> python convert_albert_original_tf_checkpoint_to_pytorch.py --tf_checkpoint_path=albert_base/model.ckpt-best --albert_config_file=albert_base/albert_config.json --pytorch_dump_path=albert_base/pytorch_model.bin
> cp albert_base/albert_config.json albert_base/config.json
>
> >>> from transformers import TFAlbertForPreTraining>>> model = TFAlbertForPreTraining.from_pretrained("albert_base", from_pt=True)
> [...]All the weights of TFAlbertForPreTraining were initialized from the PyTorch model.If your task is similar to the task the model of the checkpoint was trained on, you can already use TFAlbertForPreTraining for predictions without further training.
>
> BERT
>
> Same for BERT:
>
> wget https://storage.googleapis.com/bert_models/2018_10_18/uncased_L-12_H-768_A-12.zip
> unzip uncased_L-12_H-768_A-12.zip
> python convert_bert_original_tf_checkpoint_to_pytorch.py --tf_checkpoint_path=uncased_L-12_H-768_A-12/bert_model.ckpt --bert_config_file=uncased_L-12_H-768_A-12/bert_config.json --pytorch_dump_path=uncased_L-12_H-768_A-12/pytorch_model.bin
>
> >>> from transformers import TFBertForPreTraining>>> bert = TFBertForPreTraining.from_pretrained("uncased_L-12_H-768_A-12", from_pt=True)
> [...[All the weights of TFBertForPreTraining were initialized from the PyTorch model.If your task is similar to the task the model of the checkpoint was trained on, you can already use TFBertForPreTraining for predictions without further training.
>
> Hope that helps.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/12196#issuecomment-896887301>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ACRE6KCCQLTEL7JVTE5YAKDT4KD4ZANCNFSM46ZJP3RA>
> .
> Triage notifications on the go with GitHub Mobile for iOS
> <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
> or Android
> <https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email>
> .
>


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.