Whisper Generate with changed model
Author: dengchengxifrank
Pull request: no
Labels: name: bug

name: Audio

### System Info


- `transformers` version: 4.36.2
- Platform: Linux-3.10.0-1160.59.1.el7.x86_64-x86_64-with-glibc2.17
- Python version: 3.9.19
- Huggingface_hub version: 0.24.6
- Safetensors version: 0.4.5
- Accelerate version: 0.21.0
- Accelerate config:    not found
- PyTorch version (GPU?): 2.0.1+cu117 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>


### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

@sanchit-gandhi  @gante @ylacombe 
Hi, I tried to change the original model structure when training whisper. I passed in the aid value of 0 or 1 in DataCollatorSpeechSeq2SeqWithPadding. When aid is 0, the network adds an additional embedding after CNN (changing the network structure), and does nothing when aid is 1:

class DataCollatorSpeechSeq2SeqWithPadding:

    def __init__(self, processor,soft_prompt_len=4):
        self.processor = processor
        self.soft_prompt_len = soft_prompt_len
        if soft_prompt_len > 0:
            self.decoder_start_token_id = self.processor.tokenizer.convert_tokens_to_ids('
')
            self.prev_sot_token_id = self.decoder_start_token_id
        else:    
            self.decoder_start_token_id = self.processor.tokenizer.convert_tokens_to_ids('
')
    def __call__(self, features: List[Dict[str, Union[List[int], torch.Tensor]]]) -> Dict[str, torch.Tensor]:

        # split inputs and labels since they have to be of different lengths and need different padding methods
        # first treat the audio inputs by simply returning torch tensors
        input_features = [{"input_features": feature["input_features"]} for feature in features]
        batch = self.processor.feature_extractor.pad(input_features, return_tensors="pt")

        # get the tokenized label sequences
        label_features = [{"input_ids": feature["labels"]} for feature in features]
        
        aid = torch.tensor([feature["aid"] for feature in features])
        
        # pdb.set_trace()

        # pad the labels to max length
        labels_batch = self.processor.tokenizer.pad(label_features, return_tensors="pt")

        # replace padding with -100 to ignore loss correctly
        labels = labels_batch["input_ids"].masked_fill(labels_batch.attention_mask.ne(1), -100)

        language_token_for_labels = torch.ones((labels.shape[0], 1), dtype=torch.long) * 50259
        task_token_for_labels = torch.ones((labels.shape[0], 1), dtype=torch.long) * 50359
        
        labels = torch.cat((labels[:,:1], language_token_for_labels,task_token_for_labels, labels[:,2:]),dim=1)
        
        if self.soft_prompt_len > 0:
            soft_prompt = torch.ones((labels.shape[0], self.soft_prompt_len+1), dtype=torch.long) * self.prev_sot_token_id
            labels = torch.cat([soft_prompt, labels], dim=1)
        
        if (labels[:, 0] == self.decoder_start_token_id).all().cpu().item():
            labels = labels[:, 1:]
        
        batch["labels"] = labels
        
        batch["aid"] = aid
        
        return batch



My trainer settings are as follows:
``
training_args = Seq2SeqTrainingArguments(
    output_dir="../p_tuning_with_aid_training",
    per_device_train_batch_size=1, 
    gradient_accumulation_steps=1, 
    learning_rate=3e-5,
    warmup_steps=2000, # 预热步数
    weight_decay=0.005,
    num_train_epochs=20, # 训练轮数
    save_steps=1,
    eval_steps=1,
    logging_steps=50, 
    generation_max_length=50,
    do_eval=True,
    load_best_model_at_end=True,
    save_total_limit=1,
    evaluation_strategy="epoch", 
    save_strategy="epoch",
    report_to=["tensorboard"],
    bf16=True,
    per_device_eval_batch_size=1, 
    predict_with_generate=True,
    remove_unused_columns=False, 
    label_names=["labels"],
    metric_for_best_model='wer',
    greater_is_better=False,
    push_to_hub=False,
)``


I found that the 'aid'  cannot be passed in during trainer.evalute()  after I use predict_with_generate=True, because the generation process calls model.generate, which cannot accept the aid parameter. I can only pass 'aid' during trainer.train()  How to deal with this problem?
The program reports an error as follows：
AttributeError: 'NoneType' object has no attribute 'shape' （ I use aid.shape in model forward function)


### Expected behavior

I hope trainer.evaluate can accept the ’aid‘

Comments:
Hey @dengchengxifrank, thanks for opening this issue!

I might be missing something obvious, but what is `aid` here?
Also, do you think you could provide a simple snippet to reproduce your issue? Also could you provide more info on how you've changed the model structure here, maybe with a snippet as well? Thanks

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.