CLAP Finetuning not working
Author: LeoPerelli
Pull request: no

### System Info

Hello! I am trying to finetune a CLAP model, without much luck. I am getting the following error:
`ValueError: The model did not return a loss from the inputs, only the following keys: logits_per_audio,logits_per_text,text_embeds,audio_embeds,text_model_output,audio_model_output. For reference, the inputs it received are input_ids,attention_mask,input_features.`

Is this expected? Can you also expand on which checkpoint is this pre-trained CLAP model? In the Github CLAP repo of LAION there are multiple checkpoints.
Thanks for your time! @ArthurZucker @younesbelkada 


### Who can help?

_No response_

### Information

- [x] The official example scripts
- [x] My own modified scripts

### Tasks

- [x] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

```
def preprocess_function(data):
    audios = [audio['array'] for audio in data['audio']]
    text = [text for text in data['transcription']]
    inputs = processor(
        text=text,
        audios=audios,
        padding=True,
        return_tensors='pt'
    )
    return inputs

dataset = load_dataset("audiofolder", data_dir="clap_data")
    batch_size = 4

    model = ClapModel.from_pretrained("laion/clap-htsat-unfused")
    processor = AutoProcessor.from_pretrained("laion/clap-htsat-unfused")

    dataset_processed_test = dataset['test'].map(preprocess_function, remove_columns=["audio", "transcription"] , batched=True)
    dataset_processed_train = dataset['train'].map(preprocess_function,remove_columns=["audio", "transcription"] , batched=True)

    dataset_processed_test.set_format("torch", columns=["input_ids","attention_mask","input_features"], output_all_columns=True)
    dataset_processed_train.set_format("torch", columns=["input_ids","attention_mask","input_features"], output_all_columns=True)


    args = TrainingArguments(
        output_dir = "clap_trainings"
        "debug",
        evaluation_strategy = "epoch",
        save_strategy = "epoch",
        learning_rate=1e-5,
        per_device_train_batch_size=batch_size,
        per_device_eval_batch_size=batch_size,
        num_train_epochs=50,
        warmup_ratio=0.05,
        logging_steps=10
    )

    trainer = Trainer(
        model,
        args,
        train_dataset=dataset_processed_train,
        eval_dataset=dataset_processed_test
    )

    trainer.train()
```

### Expected behavior

The model should train normally.

Comments:
Hey!  I seems that there is a `return_loss` argument that can be set  when calling the forward of the model. I am not really sure how the trainer handle's it internally but you should be able to pass it as an argument I believe

Indeed, see CLIP's training script, which is very similar to CLAP: https://github.com/huggingface/transformers/blob/main/examples/pytorch/contrastive-image-text/run_clip.py#L237.

One needs to additionally pass the `return_loss` argument, which the script does in the `collate_fn`, which is then passed to the `Trainer`.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

I'm curious whether @LeoPerelli  managed to finetune it eventually and if so, how? I would like to do the same thing :)