Minor correction in Roberta Model docs, Roberta doesn't use NSP
Author: Santosh-Gupta
Pull request: no
Labels: name: wontfix

name: Documentation


In the roberta model docs

https://huggingface.co/transformers/model_doc/roberta.html

>pooler_output (tf.Tensor of shape (batch_size, hidden_size)):
Last layer hidden-state of the first token of the sequence (classification token) further processed by a Linear layer and a Tanh activation function. The Linear layer weights are trained from the next sentence prediction (classification) objective during Bert pretraining. This output is usually not a good summary of the semantic content of the input, you’re often better with averaging or pooling the sequence of hidden-states for the whole input sequence.

Roberta uses something else though

>FULL-SENTENCES: Each input is packed with
full sentences sampled contiguously from one
or more documents, such that the total length is
at most 512 tokens. Inputs may cross document
boundaries. When we reach the end of one document, we begin sampling sentences from the
next document and add an extra separator token
between documents. We remove the NSP loss.



Comments:
You're absolutely right. Feel free to submit a PR to rectify this!

> You're absolutely right. RoBERTa swaps NSP for sentence order prediction (SOP). 

I think SOP was introduced into Albert 

>To further improve the performance of ALBERT, we also introduce a self-supervised loss for
sentence-order prediction (SOP). SOP primary focuses on inter-sentence coherence and is designed
to address the ineffectiveness (Yang et al., 2019; Liu et al., 2019) of the next sentence prediction
(NSP) loss proposed in the original BERT.

https://arxiv.org/pdf/1909.11942.pdf

So it looks like the Albert document needs to be changed as well

https://huggingface.co/transformers/model_doc/albert.html#transformers.AlbertModel.forward

Roberta uses something called FULL-SENTENCES 

>FULL-SENTENCES: Each input is packed with
full sentences sampled contiguously from one
or more documents, such that the total length is
at most 512 tokens. Inputs may cross document
boundaries. When we reach the end of one document, we begin sampling sentences from the
next document and add an extra separator token
between documents. We remove the NSP loss.

https://arxiv.org/pdf/1907.11692.pdf

It sound like NSP isn't replaced, the task/training objective is removed altogether. So would that mean that that the linear layer which processes the CLS token outputs is untrained? It sounds like it, but I am not 100% sure. This is the linear layer I am talking about

>Last layer hidden-state of the first token of the sequence (classification token) further processed by a Linear layer and a Tanh activation function. The Linear layer weights are trained from the next sentence prediction (classification) objective during pre-training.

- 

>Feel free to submit a PR to rectify this!

Sure would love to. Would need to 100%  figure out if the aforementioned roberta tanh ff layer is trained, or if it's just random initialization. 

Are the docs on Github? I tried looking around, and found these

https://github.com/huggingface/transformers/blob/master/docs/source/model_doc/roberta.rst

https://github.com/huggingface/transformers/blob/master/docs/source/model_doc/albert.rst

But it doesn't seem to be the full documentation of the models.

I also tried looking up "next sentence prediction" in the repo but only found comments for the model code, which I can also update in the PR. 




I sent one of the authors an email asking about the layer, just wanted to be 100% sure before I make a PR. 

> I think SOP was introduced into Albert

Oof, sorry for the slip up. I've been working with different models these days so I sometimes mix 'em up.

The docs are generated from docstrings in the code. So you seem to be looking for this:

https://github.com/huggingface/transformers/blob/a34a9896ac2a4a33ff9cd805c76eed914c8d8965/src/transformers/modeling_tf_roberta.py#L194-L201

and for Albert:

https://github.com/huggingface/transformers/blob/a34a9896ac2a4a33ff9cd805c76eed914c8d8965/src/transformers/modeling_tf_albert.py#L686-L692

You can check which weights are (not) loaded by setting the logger to level INFO. In such a case, you'll see a message of the layers whose layers were not loaded.

```python
from transformers import BertForNextSentencePrediction

import logging

if __name__ == '__main__':
    logging.basicConfig(level=logging.INFO)
    model = BertForNextSentencePrediction.from_pretrained('bert-base-cased')
```

As the last line in the log, you'll see:

> Weights from pretrained model not used in BertForNextSentencePrediction: ['cls.predictions.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.decoder.weight', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.LayerNorm.bias']

Thanks, will use this info. For roberta it looks like all the weights are loaded; I didn't see a message about any weights not being loaded. I was expecting this since the architecture is the same, just the training is different. 

Just waiting to hear back if the tanh layer is untrained in roberta. 

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


Ah, I messaged one of the authors but didn't hear anything back. But I'm pretty sure by now that there is no training of the pooler layer, so I'll start on an update. 

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


Whoops, sort of fell off this. Will start looking into this soon. 

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.