Introduce modular files for speech models
Author: nikosanto13
Pull request: yes
Labels: name: run-slow

# What does this PR do?



## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@ArthurZucker @Cyrilvallez 


## Additional details

* Added modular files for models that have heavy duplication with classes from `modeling_wav2vec2.py`: Hubert, WavLM, Data2VecAudio, Wav2Vec2Conformer, Wav2Vec2Bert, UniSpeech, UniSpeechSat
* Added some modifications on the modular converter script, from issues that came up during writing the above modular scripts (see inline comments for justification)

Comments:
cc @arthurzucker @qubvel 

@Cyrilvallez hey, could you take a look again?

Hey @nikosanto13! Super sorry about the delay, last week we were on an offsite with the whole transformers team, and this week was a bit crazy because of some big refactoring of core parts and releases! 🙂 This PR is still very much welcome, and I'll get a deeper look into all models asap! Please bear with me in the meantime 🙏 Be assured that this is definitely on my to-do list! 🤗 You can check out https://github.com/huggingface/transformers/pull/36688 as well, which propose similar changes about the assignments 😉

Actually, tagging @eustlb as this is mostly audio models, maybe you have some time to help review the modular parts?

In the meantime, @nikosanto13 I believe some change can be reverted since https://github.com/huggingface/transformers/pull/36279, the imports inside functions should not need to be moved 😉

@Cyrilvallez hey, no worries for the delay! thanks for giving me pointers to latest changes on the modular converter

#36279 fix was not enough in the lazy imports inside class definitions, as it only works for functions. But inspired by this I added a similar change to work on class definitions (see inline comment)

@ArthurZucker ty for your review 🤗 the deepspeed and peft lazy import statements should be ok now, I added a fix on modular converter that makes my previous changes redundant

let me know if there is anything else

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_35902). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

run-slow: wavlm

This comment contains run-slow, running the specified jobs: This comment contains run-slow, running the specified jobs:

models: ['models/wavlm']
quantizations: [] ...

run-slow: wav2vec2_bert, wav2vec2_conformer, unispeech, unispeech_sat, hubert, data2vec

This comment contains run-slow, running the specified jobs: This comment contains run-slow, running the specified jobs:

models: ['models/data2vec', 'models/hubert', 'models/unispeech', 'models/unispeech_sat', 'models/wav2vec2_bert', 'models/wav2vec2_conformer']
quantizations: [] ...

run-slow: wav2vec2_bert, wav2vec2_conformer, unispeech, unispeech_sat, hubert, data2vec

This comment contains run-slow, running the specified jobs: This comment contains run-slow, running the specified jobs:

models: ['models/data2vec', 'models/hubert', 'models/unispeech', 'models/unispeech_sat', 'models/wav2vec2_bert', 'models/wav2vec2_conformer']
quantizations: [] ...

run-slow: wav2vec2_bert, wav2vec2_conformer, unispeech, unispeech_sat, hubert, data2vec

This comment contains run-slow, running the specified jobs: This comment contains run-slow, running the specified jobs:

models: ['models/data2vec', 'models/hubert', 'models/unispeech', 'models/unispeech_sat', 'models/wav2vec2_bert', 'models/wav2vec2_conformer']
quantizations: [] ...

@eustlb thanks for taking care of this

> BTW subsequent work could focus on propagating to other speech models that rely partially on wav2vec modelling: seamless m4t, speecht5, sew, sew_d

yeah I skipped them by mistake, maybe I could open another PR now that this has been merged

@nikosanto13 thanks again for the work !!
I would love to review such a PR 🤗