Choose appropriate loss function (MSE if regression) during forward as the doc suggests so that audio models could do regression
Author: nevikw39
Pull request: yes

# What does this PR do?

The general documentation of `transformers` says that a classifier would specialize in regression if `num_labels` in config is set to 1. The doc-string comments of `WhisperForAudioClassification` class also suggest that it would do so. Nevertheless, the actual implementation of `forward()` method always computes cross entropy loss in regardless of the config.

The issue is found on other audio classification models as well, such as wav2vec2.0, HuBERT, WavLM, etc.

So this PR assigns appropriate loss function to `loss_fct` in `forward()` method of audio classification models class (mean square loss for regression) as the documentation suggests.

Fixes #27862 

## Before submitting

- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
  -  No, this PR fixed a small bug.
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests), Pull Request section?
  -  Yes of course!
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link  to it if that's the case.
  -  I submitted an issue (#27862) earlier.
- [x] Did you make sure to update the documentation with your changes? Here are the [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
  -  Not necessary.
- [x] Did you write any new necessary tests?
  -  No.

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag members/contributors who may be interested in your PR.

Hello @sanchit-gandhi, @patrickvonplaten, @anton-l, and @huggingface team, thanks for your efforts to bring Whisper to the awesome library, please help review this PR.


Comments:
## Updated

I also found other audio classification models, such as wav2vec 2.0, HuBERT, WavLM, etc., except Audio Spectrogram Transformer, have similar issues. Therefore, I mended them as well.

~~After applying the patch, `make fixup` failed for `data2vec_audio` despite the fact that `data2vec_text` and `data2vec_vision` support MSE for regression. `make fix-copies` seem to solve the problem by removing the fix of `Data2VecAudioForAudioFrameClassification`.~~

---

P.S. When rebasing the main branch from upstream, I made some mistakes and accindentally merge the main branch into my one, which included others' commits into my PR. I reset the merge and pushed to my branch forcibly. Sorry for the mess.

## Updated Again

I just found that the shape of logits used to compute MSE loss should be reshaped in different way rather than cross entropy.

For the copy issue reported by `make fixup`, I realized that in this case, it checked if `Data2VecAudioForAudioFrameClassification.forward() ` is identical to `Wav2Vec2ForAudioFrameClassification.forward()` since the former was marked as copied from the latter. The reason why the issue was reported was that I overlooked
 `Wav2Vec2ForAudioFrameClassification` previously.

This time, I was certainly sure that I executed `git pull --rebase` yet I included others' commits in my PR again... As a result, I had to push forcibly once more. I sincerely apologize to my naïve lack of practical git experience.

Thanks, I'll ping @ylacombe and @sanchit-gandhi for a first review 😉 

> Thanks for these fixes @nevikw39 and apologies for the late review! The code looks perfect - in the style of the Transformers loss functions 🙌 May I request that we implement a test for at least one of these models to confirm that we get the correct loss in the case that `config.num_labels>1`? Otherwise, this PR looks good to go!

Hi HuggingFace team,

Apologies for that I was busy in recent weeks.

I believe it's a great chance for me to learn to design Python unit testing. Could you please provide some kind instructions or examples for what the test should be? I checked existing testings and fount that only Whisper had tester for audio classification (with config `num_labels=2`), whereas Wav2Vec2, WavLM or HuBERT didn't.

Thank!

For testing I would recommend you to follow what is usually done in the code base so check that a loss is properly computed by simply passing dummy inputs and dummy labels! Make sure the CI is green as well an rebase on main as it hase been a while! 🤗 

Hi HuggingFace team,

I resolved the suggestion of code review and the CI tests, and synced with the master branch (by merging rather than rebasing though I don't know whether it is appropriate, yet hopefully the PR would be squashed into a single commit, right?).

As for the test, I tried to take a close look on current unit tests. Sanchit (6th Jan.) said that we need "a test for at least one of these models to confirm that we get the correct loss in the case that `config.num_labels>1`". I fount that inside `tests/models/whisper/test_modeling_whisper.py`, class `WhisperEncoderModelTest` and `WhisperEncoderModelTester` seems to test the `forward()` method of `WhisperForAudioClassification` with [config `num_labels=2`](../tree/main/tests/models/whisper/test_modeling_whisper.py#L2597). So do these existing tests suffice?

For the remaining models, I haven't figured out how the existing tests deal with `num_labels` config. It appears that these models aren't covered by tests of forward, are they? Could we just leave those models as is?

Thanks!

> As for the test, I tried to take a close look on current unit tests. Sanchit (6th Jan.) said that we need "a test for at least one of these models to confirm that we get the correct loss in the case that config.num_labels>1". I fount that inside tests/models/whisper/test_modeling_whisper.py, class WhisperEncoderModelTest and WhisperEncoderModelTester seems to test the forward() method of WhisperForAudioClassification with [config num_labels=2](https://github.com/huggingface/transformers/tree/main/tests/models/whisper/test_modeling_whisper.py#L2597). So do these existing tests suffice?

yes it suffice if the `WhisperForAudioClassification` passes in that test! I can have a look and merge if it's alright! 

Do you want me to add the test? 🤗 

Hi Arthur,

Apologies for that I indefinitely postponed and just couldn’t find time to work on this task. I would really appreciate if you could help add the test. Thanks!

I don't have the bandwidth either, will let you handle it and the red ci! 🤗 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_27863). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.