facebook/bart-large-mnli input format
Author: chrisdoyleIE
Pull request: no
Labels: name: wontfix

Hi folks, 

First off, I've been using you guys since the early days and think the effort and time that you put in is just phenomenal. Thank you. All the postgrads I know at the Uni of Edinburgh love HuggingFace.

My question concerns the usage of the ```facebook/bart-large-mnli``` checkpoint - specifically the input formatting. The paper mentions that inputs are concatenated and appended with an EOS token, which is then passed to the classification head. 

Something like below perhaps? If this is the case, the probabilities do not seem right, seeing as the first two sentences are the exact same.

```python
from transformers import AutoModelForSequenceClassification, AutoTokenizer, AutoModel
import torch

t = AutoTokenizer.from_pretrained("facebook/bart-large-mnli")
mc = AutoModelForSequenceClassification.from_pretrained("facebook/bart-large-mnli")

s1 = torch.tensor(t("i am good. [EOS] i am good.", padding="max_length")["input_ids"])
s2 = torch.tensor(t("i am good. [EOS] i am NOT good.", padding="max_length")["input_ids"])
s3 = torch.tensor(t("i am good. [EOS] i am bad.", padding="max_length")["input_ids"])

with torch.no_grad():
  logits = mc(torch.stack((s1,s2,s3)), output_hidden_states=True)[0]

sm = torch.nn.Softmax()
print(sm(logits)) 
# tensor([[0.2071, 0.3143, 0.4786],      # these sentences are the exact same, so why just 0.47?
#             [0.6478, 0.1443, 0.2080],       # slightly better, but this checkpoint gets ~80% acc on MNLI
#             [0.3937, 0.2987, 0.3076]])      # This distribution is almost random, but the sentences are the exact opposite
```

I note that ```[EOS]``` is not registered with the tokenizer special tokens. When I use the registers ```
``` or ```
``` I get similar results



Comments:
Thanks for the kind words!

1) definitely needs to be `
` instead of `[EOS]`, but the tokenizer should do this for you.
2) I suspect that the tokenizer takes pairs of sentences. I know that @VictorSanh and I have used that model inside the run_glue.py script and gotten reasonable accuracy.
The logic seems to be here: https://github.com/huggingface/transformers/blob/fcf0652460753f8a81f7576e8abdaa6b3742f00e/src/transformers/data/processors/glue.py#L132

Also note that there is a different class to label mapping issue for Roberta, XLM and Bart that datasets/glue.py takes care of:
Before the fix, the classes are `dict(entailment=0, contradiction=1, neutral=2)`.

@VictorSanh please confirm that I am not spewing lies. I have not worked with this dataset very much.


Thanks for the quick response, as always!

Firstly, using ```
``` or ```
``` with the initial code (above) seems to make no noticeable difference to the distribution.

I have taken a look at the [permalinked](https://github.com/huggingface/transformers/blob/fcf0652460753f8a81f7576e8abdaa6b3742f00e/src/transformers/data/processors/glue.py#L132
) code and have attempted to replicate it below, to no avail. 

Whether the tokenizer is passed a list of two sentences, a tuple of two sentences, or a list of a tuple; it returns a list of two tokenizations - one for each sentence - rather than one overall tokenization with automatic insertion of sep_token.

```python
from transformers import AutoModelForSequenceClassification, AutoTokenizer, AutoModel
import torch

t = AutoTokenizer.from_pretrained("facebook/bart-large-mnli")

s1 = torch.tensor(t(["here we are passing ...","... a list to the tokenizer"], padding="max_length")["input_ids"]).to("cuda:0") # typical use of tokenizer
s2 = torch.tensor(t(("here we are passing ...","... a tuple to the tokenizer"), padding="max_length")["input_ids"]).to("cuda:0") # atypical use
s3 = torch.tensor(t([("here we are passing ...","... a list containing a tuple to the tokenizer")], padding="max_length")["input_ids"]).to("cuda:0") # as glue.py (I think!)
s1.size()  # torch.Size([2, 1024]) 
s2.size()  # torch.Size([2, 1024])
s3.size()  # torch.Size([2, 1024]) >> none are the expected torch.Size([1, 1024])
```

I know how to solve the tokenizer problem
```python
tok = AutoTokenizer.from_pretrained('facebook/bart-large')
pair1 = ("this is a sent", "another sent")
pair2 = ("this is a sent about same", "another sent")
assert tok(*pair1, return_tensors='pt').input_ids.shape == (1,10)
assert tok([pair1, pair2], return_tensors='pt',padding=True).input_ids.shape == (2,12)
```

Great this is good progress, thank you!

A remaining issue is that the distributions still look off:
 1. One pair is entailed, the other contradicting - and yet their classification class remains the same.
 2. The certainty of the model is possibly very low for identical sentences. 

```python
t = AutoTokenizer.from_pretrained("facebook/bart-large-mnli")
mc = AutoModelForSequenceClassification.from_pretrained("facebook/bart-large-mnli").to("cuda:0")

s1 = ("this is good", "this is good")
s2 = ("this is good", "this is bad")

inputs = torch.tensor([t(*s1, padding="max_length")["input_ids"],
                       t(*s2, padding="max_length")["input_ids"]]
                      ).to("cuda:0")

with torch.no_grad():
  logits = mc(inputs, output_hidden_states=True)[0]
sm = torch.nn.Softmax()
print(sm(logits)) # tensor([[0.0991, 0.2503, 0.6507], 
                  #         [0.1670, 0.2707, 0.5623]], device='cuda:0')

```

Interesting. Happy to look into it if there's a bug, but otherwise I think this is just a model issue. (Bug = the prediction is very different from the fairseq model for the same input).

> Great this is good progress, thank you!
> 
> A remaining issue is that the distributions still look off:
> 
> 1. One pair is entailed, the other contradicting - and yet their classification class remains the same.
> 2. The certainty of the model is possibly very low for identical sentences.
> 
> ```python
> t = AutoTokenizer.from_pretrained("facebook/bart-large-mnli")
> mc = AutoModelForSequenceClassification.from_pretrained("facebook/bart-large-mnli").to("cuda:0")
> 
> s1 = ("this is good", "this is good")
> s2 = ("this is good", "this is bad")
> 
> inputs = torch.tensor([t(*s1, padding="max_length")["input_ids"],
>                        t(*s2, padding="max_length")["input_ids"]]
>                       ).to("cuda:0")
> 
> with torch.no_grad():
>   logits = mc(inputs, output_hidden_states=True)[0]
> sm = torch.nn.Softmax()
> print(sm(logits)) # tensor([[0.0991, 0.2503, 0.6507], 
>                   #         [0.1670, 0.2707, 0.5623]], device='cuda:0')
> ```

Have you tried with longer sentences? MNLI has inputs that are on average longer.
While I agree that the second distribution is a bit off, the first one seems fairly spiked to me.
Agree with Sam, might just be the model's behavior (as opposed to a bug)

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.