Same logits value for different input
Author: ghost
Pull request: no
Labels: name: Need more information

name: wontfix

name: Ex: Sequence Classification

# ❓ Questions & Help
## Details
I use BartForSequenceClassification pre-trained model from the HuggingFace Transformers library. During training phase logits from classification head gets different values, but during validation phase all logits values are equal even for different input texts.

I use BartTokenizer.batch_encode_plus to encode the text before feeding into the model.

I fine-tuned the model for 1 epoch using the following code:

```
config = BartConfig.from_pretrained(model_name)
config.num_labels = 3
config.output_hidden_states = False
config.output_attentions = False

transformer_model = BartForSequenceClassification.from_pretrained(model_name, config=config)
transformer_model.cuda();
optimizer = AdamW(transformer_model.parameters())

NUM_TRAIN_EPOCHS = 1
print("Training:")
for i in range(1, 1+NUM_TRAIN_EPOCHS):
  transformer_model.train()
  for batch_index, (batch_input_ids, batch_attention_masks, batch_y) in enumerate(train_dataloader, start=1):
    batch_input_ids_cuda = batch_input_ids.to(device)
    batch_attention_masks_cuda = batch_attention_masks.to(device)
    batch_y_cuda = batch_y.to(device)
    loss, logits, _ =  transformer_model(batch_input_ids_cuda, attention_mask=batch_attention_masks_cuda, labels=batch_y_cuda)
    transformer_model.zero_grad()
    loss.backward()
    optimizer.step()
```

And for validation I use the following code:

```
transformer_model.eval()
with torch.no_grad():
 for batch_index, (batch_input_ids, batch_attention_masks, batch_y) in enumerate(test_dataloader, start=1):
    batch_input_ids_cuda = batch_input_ids.to(device)
    batch_attention_masks_cuda = batch_attention_masks.to(device)
    batch_y_cuda = batch_y.to(device)
    loss, logits, _ =  transformer_model(batch_input_ids_cuda, attention_mask=batch_attention_masks_cuda, labels=batch_y_cuda)
    print("Input ids:", batch_input_ids_cuda)
    print("Attention masks:", batch_attention_masks_cuda)
    print("Logits:", logits)
```

Output of validation phase is:

```
Input ids: tensor([[    0,  3655,     9,  ...,     1,     1,     1],
        [    0, 31524,   347,  ...,     1,     1,     1],
        [    0, 12806, 24220,  ...,     1,     1,     1],
        ...,
        [    0,  8518,  7432,  ...,     1,     1,     1],
        [    0, 15006, 23613,  ...,     1,     1,     1],
        [    0, 14729, 13178,  ...,     1,     1,     1]], device='cuda:0')
Attention masks: tensor([[1, 1, 1,  ..., 0, 0, 0],
        [1, 1, 1,  ..., 0, 0, 0],
        [1, 1, 1,  ..., 0, 0, 0],
        ...,
        [1, 1, 1,  ..., 0, 0, 0],
        [1, 1, 1,  ..., 0, 0, 0],
        [1, 1, 1,  ..., 0, 0, 0]], device='cuda:0')
Logits: tensor([[-1.3014, -0.7394,  0.7862],
        [-1.3014, -0.7394,  0.7862],
        [-1.3014, -0.7394,  0.7862],
        [-1.3014, -0.7394,  0.7862],
        [-1.3014, -0.7394,  0.7862],
        [-1.3014, -0.7394,  0.7862],
        [-1.3014, -0.7394,  0.7862],
        [-1.3014, -0.7394,  0.7862],
        [-1.3014, -0.7394,  0.7862],
        [-1.3014, -0.7394,  0.7862],
        [-1.3014, -0.7394,  0.7862],
        [-1.3014, -0.7394,  0.7862],
        [-1.3014, -0.7394,  0.7862],
        [-1.3014, -0.7394,  0.7862],
        [-1.3014, -0.7394,  0.7862],
        [-1.3014, -0.7394,  0.7862]], device='cuda:0')
```

Why all the logits have exact same value for every input?

<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on Stack Overflow**: https://stackoverflow.com/questions/62097267/same-logits-value-for-different-input-in-huggingfaces-transformer


Comments:
Can you post a minimal dataset that we can test this with? So for instance two sentences that give the same result for you.

```
[
  ["Nah I don't think he goes to usf, he lives around here though", 2],
  ["URGENT! You have won a 1 week FREE membership in our $100,000 Prize Jackpot!", 1]
]
```

**The model outputs same results only after fine-tuning phase.**

This is my Dataset subclass:

```
class SequenceClassificationDataset(Dataset):
  def __init__(self, df, tokenizer, max_length):
    encodings = tokenizer.batch_encode_plus(df.values[:, 0].tolist(), return_tensors="pt", max_length=max_length, pad_to_max_length=True)
    self.input_ids = encodings.input_ids
    self.attention_masks = encodings.attention_mask
    self.y = torch.LongTensor(df.values[:,1].tolist())

  def __getitem__(self, index):
    return self.input_ids[index], self.attention_masks[index], self.y[index]
  
  def __len__(self):
    return self.input_ids.shape[0]
```

Am I passing the "y" value incorrectly? It's not one-hot encoded matrix, but it's vector with size (batch_size, ) where each element represents the category for that text.

Yes, your labels are correct.

I can't reproduce your problem, though. This seems to work correctly.


```python
import torch
from torch.utils.data import Dataset, DataLoader
from transformers import AutoTokenizer, BartForSequenceClassification

import pandas as pd


class SequenceClassificationDataset(Dataset):
    def __init__(self, df, tokenizer, max_length):
        encodings = tokenizer.batch_encode_plus(df.values[:, 0].tolist(), return_tensors="pt",
                                                max_length=max_length, pad_to_max_length=True)
        self.input_ids = encodings.input_ids
        self.attention_masks = encodings.attention_mask
        self.y = torch.LongTensor(df.values[:, 1].tolist())

    def __getitem__(self, index):
        return self.input_ids[index], self.attention_masks[index], self.y[index]

    def __len__(self):
        return self.input_ids.shape[0]


def main():
    df = pd.DataFrame([
      ["Nah I don't think he goes to usf, he lives around here though", 2],
      ["URGENT! You have won a 1 week FREE membership in our $100,000 Prize Jackpot!", 1]
    ])

    tokenizer = AutoTokenizer.from_pretrained("bart-large")
    model = BartForSequenceClassification.from_pretrained("bart-large")
    ds = SequenceClassificationDataset(df, tokenizer, 32)
    dl = DataLoader(ds)

    with torch.no_grad():
        for batch_index, (batch_input_ids, batch_attention_masks, batch_y) in enumerate(dl, start=1):
            batch_input_ids_cuda = batch_input_ids
            batch_attention_masks_cuda = batch_attention_masks
            batch_y_cuda = batch_y
            loss, logits, _ = model(batch_input_ids_cuda, attention_mask=batch_attention_masks_cuda,
                                                labels=batch_y_cuda)
            print("Input ids:", batch_input_ids_cuda)
            print("Attention masks:", batch_attention_masks_cuda)
            print("Loss:", loss)
            print("Logits:", logits)
            print("y:", batch_y)

if __name__ == '__main__':
    main()
```

Are you running this in a notebook? If so, try restarting the notebook. Having unexpected results like this can be a sign of cached cells.

Everything works fine if I don't fine-tune the model, the problem occurs after fine-tuning. Can you please check my training subroutine?

The training code seems okay. Even if there was a bug in the loop, you would not expect that any input gives the same output. I can't debug this unfortunately since I can't reproduce your issue. Can you share the dataset that you use for fine-tuning?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


> # ❓ Questions & Help
> ## Details
> I use BartForSequenceClassification pre-trained model from the HuggingFace Transformers library. During training phase logits from classification head gets different values, but during validation phase all logits values are equal even for different input texts.
> 
> I use BartTokenizer.batch_encode_plus to encode the text before feeding into the model.
> 
> I fine-tuned the model for 1 epoch using the following code:
> 
> ```
> config = BartConfig.from_pretrained(model_name)
> config.num_labels = 3
> config.output_hidden_states = False
> config.output_attentions = False
> 
> transformer_model = BartForSequenceClassification.from_pretrained(model_name, config=config)
> transformer_model.cuda();
> optimizer = AdamW(transformer_model.parameters())
> 
> NUM_TRAIN_EPOCHS = 1
> print("Training:")
> for i in range(1, 1+NUM_TRAIN_EPOCHS):
>   transformer_model.train()
>   for batch_index, (batch_input_ids, batch_attention_masks, batch_y) in enumerate(train_dataloader, start=1):
>     batch_input_ids_cuda = batch_input_ids.to(device)
>     batch_attention_masks_cuda = batch_attention_masks.to(device)
>     batch_y_cuda = batch_y.to(device)
>     loss, logits, _ =  transformer_model(batch_input_ids_cuda, attention_mask=batch_attention_masks_cuda, labels=batch_y_cuda)
>     transformer_model.zero_grad()
>     loss.backward()
>     optimizer.step()
> ```
> 
> And for validation I use the following code:
> 
> ```
> transformer_model.eval()
> with torch.no_grad():
>  for batch_index, (batch_input_ids, batch_attention_masks, batch_y) in enumerate(test_dataloader, start=1):
>     batch_input_ids_cuda = batch_input_ids.to(device)
>     batch_attention_masks_cuda = batch_attention_masks.to(device)
>     batch_y_cuda = batch_y.to(device)
>     loss, logits, _ =  transformer_model(batch_input_ids_cuda, attention_mask=batch_attention_masks_cuda, labels=batch_y_cuda)
>     print("Input ids:", batch_input_ids_cuda)
>     print("Attention masks:", batch_attention_masks_cuda)
>     print("Logits:", logits)
> ```
> 
> Output of validation phase is:
> 
> ```
> Input ids: tensor([[    0,  3655,     9,  ...,     1,     1,     1],
>         [    0, 31524,   347,  ...,     1,     1,     1],
>         [    0, 12806, 24220,  ...,     1,     1,     1],
>         ...,
>         [    0,  8518,  7432,  ...,     1,     1,     1],
>         [    0, 15006, 23613,  ...,     1,     1,     1],
>         [    0, 14729, 13178,  ...,     1,     1,     1]], device='cuda:0')
> Attention masks: tensor([[1, 1, 1,  ..., 0, 0, 0],
>         [1, 1, 1,  ..., 0, 0, 0],
>         [1, 1, 1,  ..., 0, 0, 0],
>         ...,
>         [1, 1, 1,  ..., 0, 0, 0],
>         [1, 1, 1,  ..., 0, 0, 0],
>         [1, 1, 1,  ..., 0, 0, 0]], device='cuda:0')
> Logits: tensor([[-1.3014, -0.7394,  0.7862],
>         [-1.3014, -0.7394,  0.7862],
>         [-1.3014, -0.7394,  0.7862],
>         [-1.3014, -0.7394,  0.7862],
>         [-1.3014, -0.7394,  0.7862],
>         [-1.3014, -0.7394,  0.7862],
>         [-1.3014, -0.7394,  0.7862],
>         [-1.3014, -0.7394,  0.7862],
>         [-1.3014, -0.7394,  0.7862],
>         [-1.3014, -0.7394,  0.7862],
>         [-1.3014, -0.7394,  0.7862],
>         [-1.3014, -0.7394,  0.7862],
>         [-1.3014, -0.7394,  0.7862],
>         [-1.3014, -0.7394,  0.7862],
>         [-1.3014, -0.7394,  0.7862],
>         [-1.3014, -0.7394,  0.7862]], device='cuda:0')
> ```
> 
> Why all the logits have exact same value for every input?
> 
> **A link to original question on Stack Overflow**: https://stackoverflow.com/questions/62097267/same-logits-value-for-different-input-in-huggingfaces-transformer

same problem occurs to me

> Yes, your labels are correct.
> 
> I can't reproduce your problem, though. This seems to work correctly.
> 
> ```python
> import torch
> from torch.utils.data import Dataset, DataLoader
> from transformers import AutoTokenizer, BartForSequenceClassification
> 
> import pandas as pd
> 
> 
> class SequenceClassificationDataset(Dataset):
>     def __init__(self, df, tokenizer, max_length):
>         encodings = tokenizer.batch_encode_plus(df.values[:, 0].tolist(), return_tensors="pt",
>                                                 max_length=max_length, pad_to_max_length=True)
>         self.input_ids = encodings.input_ids
>         self.attention_masks = encodings.attention_mask
>         self.y = torch.LongTensor(df.values[:, 1].tolist())
> 
>     def __getitem__(self, index):
>         return self.input_ids[index], self.attention_masks[index], self.y[index]
> 
>     def __len__(self):
>         return self.input_ids.shape[0]
> 
> 
> def main():
>     df = pd.DataFrame([
>       ["Nah I don't think he goes to usf, he lives around here though", 2],
>       ["URGENT! You have won a 1 week FREE membership in our $100,000 Prize Jackpot!", 1]
>     ])
> 
>     tokenizer = AutoTokenizer.from_pretrained("bart-large")
>     model = BartForSequenceClassification.from_pretrained("bart-large")
>     ds = SequenceClassificationDataset(df, tokenizer, 32)
>     dl = DataLoader(ds)
> 
>     with torch.no_grad():
>         for batch_index, (batch_input_ids, batch_attention_masks, batch_y) in enumerate(dl, start=1):
>             batch_input_ids_cuda = batch_input_ids
>             batch_attention_masks_cuda = batch_attention_masks
>             batch_y_cuda = batch_y
>             loss, logits, _ = model(batch_input_ids_cuda, attention_mask=batch_attention_masks_cuda,
>                                                 labels=batch_y_cuda)
>             print("Input ids:", batch_input_ids_cuda)
>             print("Attention masks:", batch_attention_masks_cuda)
>             print("Loss:", loss)
>             print("Logits:", logits)
>             print("y:", batch_y)
> 
> if __name__ == '__main__':
>     main()
> ```
> 
> Are you running this in a notebook? If so, try restarting the notebook. Having unexpected results like this can be a sign of cached cells.

you should set the mode of the model to 'eval'.

https://huggingface.co/transformers/main_classes/model.html#transformers.PreTrainedModel.from_pretrained

`The model is set in evaluation mode by default using model.eval() (Dropout modules are deactivated). To train the model, you should first set it back in training mode with model.train().`

@xylcbd No. The _default_ is already eval, as the documentation writes. So you do not have to explicitly set it to eval() again. But if you want to train, you need to set it to train().