is SOP(sentence order prediction) implemented?
Author: jinkilee
Pull request: no

# ❓ Questions & Help
I am reviewing huggingface's version of Albert.

However, I cannot find any code or comment about SOP.

I can find NSP(Next Sentence Prediction) implementation from modeling_from src/transformers/modeling_bert.py.

Is SOP inherited from here with SOP-style labeling? or Is there anything I am missing?

## Details
https://stackoverflow.com/questions/59961023/is-sopsentence-order-prediction-implemented

Comments:
Hi, the layer that was used for SOP is the pooler layer, which is available in the base `AlbertModel`. When doing a forward pass, the model returns the `pooled_output` as a second value in the returned tuple. You can use this for doing a SOP task.

Oh I see! Thank you so much :)

Sorry for reopening this issue.

As you suggested, I have checked the `pooled_output` which is the second value in the returned tuple at `AlbertModel`

```python
import torch
from transformers import AlbertTokenizer, AlbertModel
model_nm = 'albert-large-v1'

# Load pre-trained model tokenizer (vocabulary)
tokenizer = AlbertTokenizer.from_pretrained(model_nm)

# SOP label should be 1
sent_1 = 'I want to eat'
sent_2 = 'because I am hungry'

# Tokenized input
text = ' '.join(['[CLS]', sent_1, '[SEP]', sent_2, '[SEP]'])
tokenized_text = tokenizer.tokenize(text)

# Convert token to vocabulary indices
indexed_tokens = tokenizer.convert_tokens_to_ids(tokenized_text)
segments_ids = (len(tokenizer.tokenize(sent_1))+2)*[0] + (len(tokenizer.tokenize(sent_2))+1)*[1]

# Convert inputs to PyTorch tensors
tokens_tensor = torch.tensor([indexed_tokens])
segments_tensors = torch.tensor([segments_ids])

# Load pre-trained model (weights)
model = AlbertModel.from_pretrained(model_nm)
model.eval()

output = model(tokens_tensor, segments_tensors)
output[0].shape, output[1].shape
```
When I check `output[1].shape` it is just a vector of `torch.Size([1, 1024]))`.
How can I do SOP with this? Unlike `BertModel` in `modeling_bert.py`, there is no code like

```python
self.classifier = nn.Linear(config.hidden_size, self.config.num_labels)
```

From `src/transformers/modeling_albert.py`
```python
# No ALBERT model currently handles the next sentence prediction task
if "seq_relationship" in name:
    continue
```
I think current ALBERT model does not handle SOP. Let me know if I am wrong :)

ALBERT doesn't do NSP but SOP - as you said. I think the following is a copy-paste error (@LysandreJik could you confirm?). It should refer to SOP and not NSP.

https://github.com/huggingface/transformers/blob/ddb6f9476b58ed9bf4433622ca9aa49932929bc0/src/transformers/modeling_albert.py#L496-L500

I am not sure about the seq_relationship line.

https://github.com/huggingface/transformers/blob/ddb6f9476b58ed9bf4433622ca9aa49932929bc0/src/transformers/modeling_albert.py#L113-L115

Perhaps the final relationship classification isn't implemented in transformers. Shouldn't be too hard to implement by yourself, though. You can use `AlbertForSequenceClassification` for that.

Yeah, implementing SOP by my self is not difficult one. As you suggested, I can just use `AlbertForSequenceClassification`. By the way, what I really want to know is ...

when I load AlbertModel by `model = AlbertModel.from_pretrained(model_nm)`, does pre-trained model has already learnt SOP? or not? I think it has not learnt yet, because `AlbertModel` has never used `AlbertForSequenceClassification`.

The model names are just abstractions of the weights and layers. They weren't trained with this library. That being said, I would expect AlbertModel to load only the weights and layers except the last classifying layers.

You can see for instance that

```python
from transformers import AlbertForSequenceClassification
import logging

logging.basicConfig(level=logging.INFO)
model = AlbertForSequenceClassification.from_pretrained('albert-base-v1')
```

will log:

```
INFO:transformers.modeling_utils:Weights of AlbertForSequenceClassification not initialized from pretrained model: ['classifier.weight', 'classifier.bias']
INFO:transformers.modeling_utils:Weights from pretrained model not used in AlbertForSequenceClassification: ['predictions.bias', 'predictions.LayerNorm.weight', 'predictions.LayerNorm.bias', 'predictions.dense.weight', 'predictions.dense.bias', 'predictions.decoder.weight', 'predictions.decoder.bias']
```

Indicating that the pretrained weights that you are loading haven't all been loaded (the prediction layer) because that layer doesn't exist in this architecture. On the other hand, the classification layer that is present in the XXXSequenceClassification model has not been pretrained, so its weights are not in the pretrained weights.

I would have expected to see a similar message indicating that not all weights could be loaded in AlbertModel because it doesn't contain the prediction layer, but I don't get any such message - which seems odd to me.

```python
from transformers import AlbertModel
import logging

logging.basicConfig(level=logging.INFO)
model = AlbertModel.from_pretrained('albert-base-v1')
# doesn't log any info messages
```


Oh I see thank you so much.

Your answer is clear to me :)

Thanks

Hi all! What situation with Albert's SOP now?
@jinkilee do you have worked approach for SOP?
Thank you!

@jinkilee  Hi, please

I went through the discussion and tried to use "AlbertForSequenceClassification" instead but I cannot understand what does logits exactly indicates! 

```
import torch
from transformers import AlbertTokenizer, AlbertModel
from transformers import AlbertForSequenceClassification
import logging
model_nm = 'albert-large-v1'

# Load pre-trained model tokenizer (vocabulary)
tokenizer = AlbertTokenizer.from_pretrained(model_nm)

# SOP label should be 1
sent_1 = 'I was having cough and headache'
sent_2 = 'so I went to the doctor'

# Tokenized input
text = ' '.join(['[CLS]', sent_2, '[SEP]', sent_1, '[SEP]'])
tokenized_text = tokenizer.tokenize(text)

# Convert token to vocabulary indices
indexed_tokens = tokenizer.convert_tokens_to_ids(tokenized_text)
segments_ids = (len(tokenizer.tokenize(sent_1))+2)*[0] + (len(tokenizer.tokenize(sent_2))+1)*[1]

# Convert inputs to PyTorch tensors
tokens_tensor = torch.tensor([indexed_tokens])
segments_tensors = torch.tensor([segments_ids])

# Load pre-trained model (weights)
logging.basicConfig(level=logging.INFO)
model = AlbertForSequenceClassification.from_pretrained('albert-base-v1')
model.eval()

output = model(tokens_tensor, segments_tensors)
output#[0].shape, output[1].shape
```

The output value is:
SequenceClassifierOutput(loss=None, logits=tensor([[ 1.0192, -0.3174]], grad_fn=<AddmmBackward>), hidden_states=None, attentions=None)

Thanks in advance.