AttributeError: 'NoneType' object has no attribute 'shape'
Author: andysingal
Pull request: no

### System Info

kaggle notebook

```
Libraries:
!pip install -Uqqq pip --progress-bar off
!pip install -qqq bitsandbytes==0.40.2 trl==0.4.7 --progress-bar off
!pip install -q torch==2.0.1+cu118 torchvision==0.15.2+cu118 torchaudio==2.0.2+cu118 torchtext==0.15.2 torchdata==0.6.1 --extra-index-url https://download.pytorch.org/whl/cu118 -U
!pip install -qqq -U git+https://github.com/huggingface/transformers.git@e03a9cc --progress-bar off
!pip install -qqq -U peft==0.4.0 --progress-bar off
!pip install -qqq -U git+https://github.com/huggingface/accelerate.git@c9fbb71 --progress-bar off
!pip install -qqq datasets==2.12.0 --progress-bar off
!pip install -qqq loralib==0.1.1 --progress-bar off
!pip install -qqq einops==0.6.1 --progress-bar off
!pip install accelerate==0.21.0 --progress-bar off
```

```
import json
import os
from pprint import pprint
import pandas as pd
import numpy as np

import bitsandbytes as bnb
import pandas as pd
import torch
import torch.nn as nn

import transformers
from trl import SFTTrainer
import peft
from datasets import load_dataset, DatasetDict,Dataset
from huggingface_hub import notebook_login
from peft import (
     get_peft_config, get_peft_model, PromptTuningInit, PromptTuningConfig, TaskType, PeftType,prepare_model_for_kbit_training
)
from transformers import (
    AutoConfig,
    AutoModelForCausalLM,
    AutoTokenizer,
    BitsAndBytesConfig,
    LlamaForCausalLM,
    LlamaTokenizer
)

# Check for CUDA availability
# device = torch.device("cuda" if not(torch.cuda.is_available()) else "cpu")
# print(f"The device type is {device.type}")
def bytes_to_gb(bytes_value):
    gb_value = bytes_value / (1024 ** 3)
    return gb_value

def get_device():
    if torch.cuda.is_available():
        # Get current GPU's VRAM (in bytes)
        vram_bytes = torch.cuda.get_device_properties(0).total_memory
        print(f"Cuda Found! You have {((round(bytes_to_gb(vram_bytes))))} GB VRAM\n")

        # Convert 24 GB to bytes
        min_vram_required_bytes = 24 * (1024 ** 3)

        if vram_bytes >= min_vram_required_bytes:
            return torch.device("cuda")
        if ((round(bytes_to_gb(vram_bytes))) >= 16):
            return torch.device("cuda")
    print("You didn't have at least 16GB of VRAM. Switching to CPU.")
    return torch.device("cpu")

device = get_device()

```

### Who can help?

@sgugger @youn

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```
MODEL_NAME = "TheBloke/stable-vicuna-13B-HF"

tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME)
 
model = AutoModelForCausalLM.from_pretrained(
    MODEL_NAME,
    torch_dtype=torch.float16,
    load_in_8bit=True,
    device_map="auto",
)

dataset = load_dataset("Dahoas/first-instruct-human-assistant-prompt")
transformed into:
tokenizer.pad_token = tokenizer.eos_token
tokenizer.padding_side = "right"
```
```
data = formatted_dataset.map(lambda samples: tokenizer(samples["message_tree_text"], padding=True, truncation=True,), batched=True)
Dataset({
    features: ['message_tree_id', 'message_tree_text', 'input_ids', 'token_type_ids', 'attention_mask'],
    num_rows: 33143
})
```
```
training_args = transformers.TrainingArguments(
    per_device_train_batch_size=1,
    gradient_accumulation_steps=4,
    gradient_checkpointing=True,
    num_train_epochs=1,
    learning_rate=2e-4,
    fp16=True,
    save_total_limit=3,
    logging_steps=1,
    output_dir=OUTPUT_DIR,
    max_steps=80,
    optim="paged_adamw_8bit",
    lr_scheduler_type="cosine",
    warmup_ratio=0.05,
    report_to="tensorboard",
)

trainer = SFTTrainer(
    model=model,
    train_dataset=data,
    peft_config=peft_config,
    dataset_text_field="message_tree_text",
    max_seq_length=max_seq_length,
    tokenizer=tokenizer,
    args=training_args,
    packing=packing,
)


model.config.use_cache = False  # silence the warnings. Please re-enable for inference!
trainer.train()

# Save trained model
trainer.model.save_pretrained(new_model)
```

ERROR:
```
You're using a LlamaTokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[17], line 31
     18 trainer = SFTTrainer(
     19     model=model,
     20     train_dataset=data,
   (...)
     26     packing=packing,
     27 )
     30 model.config.use_cache = False  # silence the warnings. Please re-enable for inference!
---> 31 trainer.train()
     33 # Save trained model
     34 trainer.model.save_pretrained(new_model)

File /opt/conda/lib/python3.10/site-packages/transformers/trainer.py:1661, in Trainer.train(self, resume_from_checkpoint, trial, ignore_keys_for_eval, **kwargs)
   1656     self.model_wrapped = self.model
   1658 inner_training_loop = find_executable_batch_size(
   1659     self._inner_training_loop, self._train_batch_size, args.auto_find_batch_size
   1660 )
-> 1661 return inner_training_loop(
   1662     args=args,
   1663     resume_from_checkpoint=resume_from_checkpoint,
   1664     trial=trial,
   1665     ignore_keys_for_eval=ignore_keys_for_eval,
   1666 )

File /opt/conda/lib/python3.10/site-packages/transformers/trainer.py:1946, in Trainer._inner_training_loop(self, batch_size, args, resume_from_checkpoint, trial, ignore_keys_for_eval)
   1943     self.control = self.callback_handler.on_step_begin(args, self.state, self.control)
   1945 with self.accelerator.accumulate(model):
-> 1946     tr_loss_step = self.training_step(model, inputs)
   1948 if (
   1949     args.logging_nan_inf_filter
   1950     and not is_torch_tpu_available()
   1951     and (torch.isnan(tr_loss_step) or torch.isinf(tr_loss_step))
   1952 ):
   1953     # if loss is nan or inf simply add the average of previous logged losses
   1954     tr_loss += tr_loss / (1 + self.state.global_step - self._globalstep_last_logged)

File /opt/conda/lib/python3.10/site-packages/transformers/trainer.py:2753, in Trainer.training_step(self, model, inputs)
   2750     return loss_mb.reduce_mean().detach().to(self.args.device)
   2752 with self.compute_loss_context_manager():
-> 2753     loss = self.compute_loss(model, inputs)
   2755 if self.args.n_gpu > 1:
   2756     loss = loss.mean()  # mean() to average on multi-gpu parallel training

File /opt/conda/lib/python3.10/site-packages/transformers/trainer.py:2778, in Trainer.compute_loss(self, model, inputs, return_outputs)
   2776 else:
   2777     labels = None
-> 2778 outputs = model(**inputs)
   2779 # Save past state if it exists
   2780 # TODO: this needs to be fixed and made cleaner later.
   2781 if self.args.past_index >= 0:

File /opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py:1501, in Module._call_impl(self, *args, **kwargs)
   1496 # If we don't have any hooks, we want to skip the rest of the logic in
   1497 # this function, and just call forward.
   1498 if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks or self._forward_pre_hooks
   1499         or _global_backward_pre_hooks or _global_backward_hooks
   1500         or _global_forward_hooks or _global_forward_pre_hooks):
-> 1501     return forward_call(*args, **kwargs)
   1502 # Do not call functions when jit is used
   1503 full_backward_hooks, non_full_backward_hooks = [], []

File /opt/conda/lib/python3.10/site-packages/accelerate/utils/operations.py:581, in convert_outputs_to_fp32.<locals>.forward(*args, **kwargs)
    580 def forward(*args, **kwargs):
--> 581     return model_forward(*args, **kwargs)

File /opt/conda/lib/python3.10/site-packages/accelerate/utils/operations.py:569, in ConvertOutputsToFp32.__call__(self, *args, **kwargs)
    568 def __call__(self, *args, **kwargs):
--> 569     return convert_to_fp32(self.model_forward(*args, **kwargs))

File /opt/conda/lib/python3.10/site-packages/torch/amp/autocast_mode.py:14, in autocast_decorator.<locals>.decorate_autocast(*args, **kwargs)
     11 @functools.wraps(func)
     12 def decorate_autocast(*args, **kwargs):
     13     with autocast_instance:
---> 14         return func(*args, **kwargs)

File /opt/conda/lib/python3.10/site-packages/peft/peft_model.py:968, in PeftModelForCausalLM.forward(self, input_ids, attention_mask, inputs_embeds, labels, output_attentions, output_hidden_states, return_dict, **kwargs)
    966 prompts = prompts.to(inputs_embeds.dtype)
    967 inputs_embeds = torch.cat((prompts, inputs_embeds), dim=1)
--> 968 return self.base_model(inputs_embeds=inputs_embeds, **kwargs)

File /opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py:1501, in Module._call_impl(self, *args, **kwargs)
   1496 # If we don't have any hooks, we want to skip the rest of the logic in
   1497 # this function, and just call forward.
   1498 if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks or self._forward_pre_hooks
   1499         or _global_backward_pre_hooks or _global_backward_hooks
   1500         or _global_forward_hooks or _global_forward_pre_hooks):
-> 1501     return forward_call(*args, **kwargs)
   1502 # Do not call functions when jit is used
   1503 full_backward_hooks, non_full_backward_hooks = [], []

File /opt/conda/lib/python3.10/site-packages/peft/peft_model.py:933, in PeftModelForCausalLM.forward(self, input_ids, attention_mask, inputs_embeds, labels, output_attentions, output_hidden_states, return_dict, **kwargs)
    912         return self.base_model(
    913             input_ids=input_ids,
    914             attention_mask=attention_mask,
   (...)
    919             **kwargs,
    920         )
    922     return self.base_model(
    923         input_ids=input_ids,
    924         attention_mask=attention_mask,
   (...)
    930         **kwargs,
    931     )
--> 933 batch_size = input_ids.shape[0]
    934 if attention_mask is not None:
    935     # concat prompt attention mask
    936     prefix_attention_mask = torch.ones(batch_size, peft_config.num_virtual_tokens).to(attention_mask.device)

AttributeError: 'NoneType' object has no attribute 'shape'
```

    


### Expected behavior

model needs to run 

Comments:
In order for your model to compute a loss, you need to give it `labels`. The dataset you are using does not contain any `labels` key, which is why you are getting this error.

Please also consider using the [forums](https://discuss.huggingface.co/) to help debug your code.

@sgugger Thanks for your response, I converted the dataset to have message Id and
message text . Messages text includes human and assistant conversation. I
am able to tokenize and get ( message_id, message_text, input_ids,
attention, token) . When I train it gives the error.

Any advise what changes I can make to the address the issue(Goal is to train the model to get chat based results during inference) . I also observed
autotokenizer does not have support for vicuña . Is that true?. Any help to
address the issue is appreciated.

Looking forward to hearing from you.

Thanks,
Andy

On Sun, Aug 13, 2023 at 22:59 Sylvain Gugger ***@***.***>
wrote:

> In order for your model to compute a loss, you need to give it labels.
> The dataset you are using does not contain any labels key, which is why
> you are getting this error.
>
> Please also consider using the forums <https://discuss.huggingface.co/>
> to help debug your code.
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/25479#issuecomment-1676419327>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AE4LJNPDRTXPSGVVXQRMW7DXVEFIDANCNFSM6AAAAAA3OMKZ6Y>
> .
> You are receiving this because you authored the thread.Message ID:
> ***@***.***>
>


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.