ValueError: too many values to unpack (expected 2) when Fine-tuning an LLM
Author: 0920GX
Pull request: no

 ### System Info

- `transformers` version: 4.35.2
- Platform: Linux-6.1.58+-x86_64-with-glibc2.35
- Python version: 3.10.12
- Huggingface_hub version: 0.19.4
- Safetensors version: 0.4.1
- Accelerate version: not installed
- Accelerate config: not found
- PyTorch version (GPU?): 2.1.0+cu121 (False)
- Tensorflow version (GPU?): 2.15.0 (False)
- Flax version (CPU?/GPU?/TPU?): 0.7.5 (cpu)
- Jax version: 0.4.23
- JaxLib version: 0.4.23
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>


### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

I'm learning how to fine-tune LLMs. However, I've been encountering a consistent error when executing Trainer.train() in my fine-tuning process, regardless of how I modify my data processing approach. The error is as follows:
`---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-11-3435b262f1ae> in <cell line: 1>()
----> 1 trainer.train()

12 frames
/usr/local/lib/python3.10/dist-packages/transformers/modeling_attn_mask_utils.py in _expand_mask(mask, dtype, tgt_len)
    152         Expands attention_mask from `[bsz, seq_len]` to `[bsz, 1, tgt_seq_len, src_seq_len]`.
    153         """
--> 154         bsz, src_len = mask.size()
    155         tgt_len = tgt_len if tgt_len is not None else src_len
    156 

ValueError: too many values to unpack (expected 2)`

I've checked the dimensions of my attention mask, and it appears to be two-dimensional, which I believe is correct. Here's an example of one of my data entries:
`{'input_ids': tensor([[    1,  1788, 29901, 29871, 30417, 31915, 30313,    13, 13496,  7451,
         29901, 29871,   233,   133,   171, 31076,   233,   176,   164,   235,
           194,   145, 30867,   235,   138,   171, 30214, 31482, 30408, 31522,
           232,   153,   160,   236,   190,   161,   231,   190,   131,   236,
           189,   191,   232,   148,   165, 29973,    13, 15539, 29901, 29871,
         30672, 30698, 30287,   233,   160,   178,   232,   137,   186,   233,
           182,   150, 30986,    13, 13496,  7451, 29901, 29871,   233,   133,
           171, 31522, 30698,   234,   151,   157,   236,   189,   191,   234,
           151,   159, 30898,   233,   189,   174, 30898,   232,   148,   165,
         29973,    13, 15539, 29901, 29871,   232,   188,   174, 30672,   232,
           132,   157, 31209,   234,   182,   153, 31935,   232,   137,   179,
            13, 13496,  7451, 29901, 29871,   236,   131,   156,   233,   171,
           166, 30392, 29946, 29900, 30824, 30214, 31383, 30698,   235,   191,
           140,   232,   136,   186,  3542, 10472,   232,   154,   145, 29973,
            13, 15539, 29901, 29871, 30769, 30698,    13, 13496,  7451, 29901,
         29871, 31356,   235,   191,   140,   232,   136,   186,   235,   174,
           142,   232,   179,   144,   233,   189,   153,   236,   131,   156,
           232,   131,   142,   236,   146,   164, 31876, 30214,  3542, 10472,
           235,   174,   142,   233,   145,   134,   233,   146,   146, 31803,
         30557, 31432, 30210,   233,   165,   160,   234,   165,   191,    13,
          5205, 29901, 29871, 31689,   233,   175,   193, 30494, 31134,    13,
         13496,  7451, 29901, 29871, 31076, 30210,   236,   131,   156,   233,
           171,   166, 31238, 30682, 30651, 30743, 30214,   235,   174,   142,
           233,   154,   132,   236,   133,   141,   234,   171,   144, 31184,
         30287, 30557,     0,     0,     0,     0,     0,     0,     0,     0,
             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
             0,     0,     0,     0,     0,     0,     0,     0,     0,     0,
             0,     0]]), 'attention_mask': tensor([[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
         1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
         1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
         1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
         1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
         1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
         1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
         1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
         1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
         1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
         1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
         0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
         0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
         0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
         0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
         0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
         0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
         0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
         0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
         0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
         0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
         0, 0, 0, 0, 0, 0, 0, 0]])}`
I am working with a dataset formatted like this:

{
    "id": "chat1",
    "conversations": [
        {"role": "user", "content": "Hello, how are you?"},
        {"role": "assistant", "content": "I'm doing great. How can I help you today?"},
        {"role": "user", "content": "I'd like to show off how chat templating works!"}
    ]
}

I am executing this script in Google Colab. Here's my script:

``` import json
   from transformers import AutoTokenizer, AutoModelForCausalLM, Trainer, TrainingArguments
   from torch.utils.data import Dataset
   import torch

   file_path = '/content/gdrive/MyDrive/kirin.json'
   with open(file_path, 'r') as file:
       data = json.load(file)

   tokenizer = AutoTokenizer.from_pretrained("/content/gdrive/MyDrive/ColabNotebooks/Taiwan-LLM-7B-v2.0.1-chat")
   model = AutoModelForCausalLM.from_pretrained("/content/gdrive/MyDrive/ColabNotebooks/Taiwan-LLM-7B-v2.0.1- 
   chat")

   def format_conversation(conversation):
       formatted = ""
       for turn in conversation['conversations']:
           formatted += turn['role'] + ": " + turn['content'] + "\n"
       return formatted.strip()

   formatted_data = [format_conversation(conv) for conv in data]

    class ConversationDataset(Dataset):
         def __init__(self, convs, tokenizer, max_length=512):
             self.tokenizer = tokenizer
             self.inputs = [tokenizer(text, return_tensors="pt", max_length=max_length, truncation=True, padding="max_length") 
             for text in convs]

       def __len__(self):
           return len(self.inputs)

       def __getitem__(self, idx):
           return self.inputs[idx]

dataset = ConversationDataset(formatted_data, tokenizer)

training_args = TrainingArguments(
       output_dir="./llama2-finetuned",
       num_train_epochs=3,
       per_device_train_batch_size=1,
       save_steps=10_000,
       save_total_limit=2,
   )

trainer = Trainer(
       model=model,
       args=training_args,
       train_dataset=dataset
   )
   trainer.train()```








### Expected behavior

The script is expected to run the training process smoothly.



Comments:
Hi @0920GX, thanks for raising an issue! 

This is a question best placed in our [forums](https://discuss.huggingface.co/). We try to reserve the github issues for feature requests and bug reports.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.