List index out of range 
Author: MinnTrit
Pull request: no

### System Info

Hi everyone, I'm currently having the issue with my code. This is my code: 

```python
import transformers 
from transformers import AutoTokenizer, RobertaForSequenceClassification
import pandas as pd 
import numpy as np 
import torch 

df = pd.read_excel(r'D:\Data Intense\Level 2\Day9\comments_df.xlsx')

model = RobertaForSequenceClassification.from_pretrained("wonrax/phobert-base-vietnamese-sentiment")
tokenizer = AutoTokenizer.from_pretrained("wonrax/phobert-base-vietnamese-sentiment", use_fast=False, vocab_size=64001)

#Test with one sentence 
#sentence = 'Tôi yêu Việt Nam'

def sentiment(sentence):
    max_sequence_length = model.config.max_position_embeddings
    encoding = tokenizer(sentence, 
                        add_special_tokens=True,
                        padding='max_length',
                        max_length = max_sequence_length,
                        return_attention_mask=True)
    #Turn words to token IDs 
    tokens_id = encoding['input_ids']
    attention_mask = encoding['attention_mask']

     #Check if the tokenized sentence length exceeds the model's maximum sequence length
    if len(tokens_id) > max_sequence_length:
        # Truncate the tokens and attention mask
        tokens_id = tokens_id[:max_sequence_length]
        attention_mask = attention_mask[:max_sequence_length]

    #Pad the tokens and attention mask if needed
    while len(tokens_id) < max_sequence_length:
        tokens_id.append(tokenizer.pad_token_id)
        attention_mask.append(0)  

    #Turn tokens Id to the tensor 
    input_ids = torch.tensor(tokens_id)
    attention_mask = torch.tensor(attention_mask)

    #Feed the inputs for the embedding layers 
    torch.no_grad() 
    output = model(input_ids.unsqueeze(0), attention_mask=attention_mask.unsqueeze(0))

    #Take the result
    predicted_class = torch.argmax(output.logits.softmax(dim=-1)).item()

    return predicted_class 

df['sentiment'] = df['content'].apply(sentiment)
```
I understand that the maximum length of the token sequence must equal to the maximum length provided by the model, in this case, I have checked it, and the length is 256, I also added 2 condition to truncate (in case the length of the token is longer than the maximum length) and add the padding tokens (in case the length of the token is shorter than the maximum length). I've also tried to set out the vocabulary size at the beginning to make sure the model can work consistently within this range, then, I have this error below: 

```---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
[d:\Hugging](file:///D:/Hugging) Face\Day2 - Transformer\Apply.ipynb Cell 2 line 5
     [47](vscode-notebook-cell:/d%3A/Hugging%20Face/Day2%20-%20Transformer/Apply.ipynb#W2sZmlsZQ%3D%3D?line=46)     predicted_class = torch.argmax(output.logits.softmax(dim=-1)).item()
     [49](vscode-notebook-cell:/d%3A/Hugging%20Face/Day2%20-%20Transformer/Apply.ipynb#W2sZmlsZQ%3D%3D?line=48)     return predicted_class 
---> [51](vscode-notebook-cell:/d%3A/Hugging%20Face/Day2%20-%20Transformer/Apply.ipynb#W2sZmlsZQ%3D%3D?line=50) df['sentiment'] = df['content'].apply(sentiment)

File [c:\Users\PC\anaconda3\Lib\site-packages\pandas\core\series.py:4630](file:///C:/Users/PC/anaconda3/Lib/site-packages/pandas/core/series.py:4630), in Series.apply(self, func, convert_dtype, args, **kwargs)
   [4520](file:///C:/Users/PC/anaconda3/Lib/site-packages/pandas/core/series.py:4520) def apply(
   [4521](file:///C:/Users/PC/anaconda3/Lib/site-packages/pandas/core/series.py:4521)     self,
   [4522](file:///C:/Users/PC/anaconda3/Lib/site-packages/pandas/core/series.py:4522)     func: AggFuncType,
   (...)
   [4525](file:///C:/Users/PC/anaconda3/Lib/site-packages/pandas/core/series.py:4525)     **kwargs,
   [4526](file:///C:/Users/PC/anaconda3/Lib/site-packages/pandas/core/series.py:4526) ) -> DataFrame | Series:
   [4527](file:///C:/Users/PC/anaconda3/Lib/site-packages/pandas/core/series.py:4527)     """
   [4528](file:///C:/Users/PC/anaconda3/Lib/site-packages/pandas/core/series.py:4528)     Invoke function on values of Series.
   [4529](file:///C:/Users/PC/anaconda3/Lib/site-packages/pandas/core/series.py:4529) 
   (...)
   [4628](file:///C:/Users/PC/anaconda3/Lib/site-packages/pandas/core/series.py:4628)     dtype: float64
   [4629](file:///C:/Users/PC/anaconda3/Lib/site-packages/pandas/core/series.py:4629)     """
-> [4630](file:///C:/Users/PC/anaconda3/Lib/site-packages/pandas/core/series.py:4630)     return SeriesApply(self, func, convert_dtype, args, kwargs).apply()

File [c:\Users\PC\anaconda3\Lib\site-packages\pandas\core\apply.py:1025](file:///C:/Users/PC/anaconda3/Lib/site-packages/pandas/core/apply.py:1025), in SeriesApply.apply(self)
   [1022](file:///C:/Users/PC/anaconda3/Lib/site-packages/pandas/core/apply.py:1022)     return self.apply_str()
   [1024](file:///C:/Users/PC/anaconda3/Lib/site-packages/pandas/core/apply.py:1024) # self.f is Callable
...
   [2231](file:///C:/Users/PC/anaconda3/Lib/site-packages/torch/nn/functional.py:2231)     # remove once script supports set_grad_enabled
   [2232](file:///C:/Users/PC/anaconda3/Lib/site-packages/torch/nn/functional.py:2232)     _no_grad_embedding_renorm_(weight, input, max_norm, norm_type)
-> [2233](file:///C:/Users/PC/anaconda3/Lib/site-packages/torch/nn/functional.py:2233) return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)

IndexError: index out of range in self
Output is truncated. View as a [scrollable element](command:cellOutput.enableScrolling?e54bc3e5-d73c-439c-88f2-3de3b6fcfba2) or open in a [text editor](command:workbench.action.openLargeOutput?e54bc3e5-d73c-439c-88f2-3de3b6fcfba2). Adjust cell output [settings](command:workbench.action.openSettings?%5B%22%40tag%3AnotebookOutputLayout%22%5D)...
```
Does anybody know how to resolve this one, I'm a new learner in machine learning as well :<

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
[d:\Hugging](file:///D:/Hugging) Face\Day2 - Transformer\Apply.ipynb Cell 2 line 5
     [47](vscode-notebook-cell:/d%3A/Hugging%20Face/Day2%20-%20Transformer/Apply.ipynb#W2sZmlsZQ%3D%3D?line=46)     predicted_class = torch.argmax(output.logits.softmax(dim=-1)).item()
     [49](vscode-notebook-cell:/d%3A/Hugging%20Face/Day2%20-%20Transformer/Apply.ipynb#W2sZmlsZQ%3D%3D?line=48)     return predicted_class 
---> [51](vscode-notebook-cell:/d%3A/Hugging%20Face/Day2%20-%20Transformer/Apply.ipynb#W2sZmlsZQ%3D%3D?line=50) df['sentiment'] = df['content'].apply(sentiment)

File [c:\Users\PC\anaconda3\Lib\site-packages\pandas\core\series.py:4630](file:///C:/Users/PC/anaconda3/Lib/site-packages/pandas/core/series.py:4630), in Series.apply(self, func, convert_dtype, args, **kwargs)
   [4520](file:///C:/Users/PC/anaconda3/Lib/site-packages/pandas/core/series.py:4520) def apply(
   [4521](file:///C:/Users/PC/anaconda3/Lib/site-packages/pandas/core/series.py:4521)     self,
   [4522](file:///C:/Users/PC/anaconda3/Lib/site-packages/pandas/core/series.py:4522)     func: AggFuncType,
   (...)
   [4525](file:///C:/Users/PC/anaconda3/Lib/site-packages/pandas/core/series.py:4525)     **kwargs,
   [4526](file:///C:/Users/PC/anaconda3/Lib/site-packages/pandas/core/series.py:4526) ) -> DataFrame | Series:
   [4527](file:///C:/Users/PC/anaconda3/Lib/site-packages/pandas/core/series.py:4527)     """
   [4528](file:///C:/Users/PC/anaconda3/Lib/site-packages/pandas/core/series.py:4528)     Invoke function on values of Series.
   [4529](file:///C:/Users/PC/anaconda3/Lib/site-packages/pandas/core/series.py:4529) 
   (...)
   [4628](file:///C:/Users/PC/anaconda3/Lib/site-packages/pandas/core/series.py:4628)     dtype: float64
   [4629](file:///C:/Users/PC/anaconda3/Lib/site-packages/pandas/core/series.py:4629)     """
-> [4630](file:///C:/Users/PC/anaconda3/Lib/site-packages/pandas/core/series.py:4630)     return SeriesApply(self, func, convert_dtype, args, kwargs).apply()

File [c:\Users\PC\anaconda3\Lib\site-packages\pandas\core\apply.py:1025](file:///C:/Users/PC/anaconda3/Lib/site-packages/pandas/core/apply.py:1025), in SeriesApply.apply(self)
   [1022](file:///C:/Users/PC/anaconda3/Lib/site-packages/pandas/core/apply.py:1022)     return self.apply_str()
   [1024](file:///C:/Users/PC/anaconda3/Lib/site-packages/pandas/core/apply.py:1024) # self.f is Callable
...
   [2231](file:///C:/Users/PC/anaconda3/Lib/site-packages/torch/nn/functional.py:2231)     # remove once script supports set_grad_enabled
   [2232](file:///C:/Users/PC/anaconda3/Lib/site-packages/torch/nn/functional.py:2232)     _no_grad_embedding_renorm_(weight, input, max_norm, norm_type)
-> [2233](file:///C:/Users/PC/anaconda3/Lib/site-packages/torch/nn/functional.py:2233) return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)

IndexError: index out of range in self
Output is truncated. View as a [scrollable element](command:cellOutput.enableScrolling?e54bc3e5-d73c-439c-88f2-3de3b6fcfba2) or open in a [text editor](command:workbench.action.openLargeOutput?e54bc3e5-d73c-439c-88f2-3de3b6fcfba2). Adjust cell output [settings](command:workbench.action.openSettings?%5B%22%40tag%3AnotebookOutputLayout%22%5D)...

### Expected behavior

I

Comments:
Hello 🤗  
I would recommend you to try putting some breakpoints here and there to check what you are feeding to the model. 
Also truncation and padding should be done with the ` truncation` and `padding` arguments of the ` __call__` method of the tokenizers. You should also set ` return_tensors = "pt" ` to directly have tensors (and batched tensors) 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.