Add Granite Speech Support
Author: alex-jw-brooks
Pull request: yes

# What does this PR do?

This PR adds support for (upcoming) Granite Speech models. In terms of model architecture, it uses a conformer-based encoder, with a blip2 qformer-based projector to encode the audio, and masks it into a granite llm. This model also uses an audio-specific lora adapter, which should _only_ be enabled when the model is processing audio inputs.

Currently this is handled by using the transformers/peft mixin to load the adapter from the same directory as the base model and overriding `.generate` to turn the lora on/off based on the presence of audio input features before forwarding to `.generate` on the superclass. Doing it this way, as opposed to encapsulating a Peft Model, makes this a lot cleaner in a lot of ways and keeps the lora out of the modeling code.

(This PR is a collaboration with @avihu111  and @gsaon)
## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker
- vision models: @amyeroberts, @qubvel
- speech models: @eustlb
- graph models: @clefourrier

Library:

- flax: @gante and @Rocketknight1
- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- pipelines: @Rocketknight1
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @muellerzr and @SunMarc
- chat templates: @Rocketknight1

Integrations:

- deepspeed: HF Trainer/Accelerate: @muellerzr
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc @MekkCyber

Documentation: @stevhliu

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @Rocketknight1
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
cc @eustlb

Awesome, thanks a lot for the quick feedback @ArthurZucker! We'll work on addressing the changes 🙂 

One question - some of the things you pointed out are copied from`blip2`'s qformer code, which was needed to avoid depending on `modeling.blip2`. The current blip2 qformer code does work without any issues though. I believe the reason we did this instead of abstracting it and loading it out of the config like other models, e.g., the visual encoder in llava next ([here](https://github.com/huggingface/transformers/blob/main/src/transformers/models/llava_next/modeling_llava_next.py#L362))  is that `Blip2QFormerModel` can't currently be created through `AutoModel.from_config`, since it's not an attribute on `transformers.models`, and it seemed like an anti pattern to add it there without also pulling it out into its own model dir 😓 


Do you have any thoughts on the best way to handle this in this PR? We can keep things as is and work on the changes in the copied code if that is what makes sense, but thought it would be best to check in case you had over thoughts! AFAIK we don't really have a need on our end to abstract the encoder at the moment, just wondered what would be easiest 

Hey @ArthurZucker! I think we have addressed all of the changes you have requested, except for the `modular_granite_speech` part. Could you please take a look when you get the chance?

Regarding adding a modular file - I've opened a PR to expose `Blip2QFormer` here (and also rebased this PR on top of it). With this change, we can delete the mountain of copied Blip2 code and generically load the projector through `AutoModel.from_config`, similar to the vision tower in most VLMs 🎉 There is no functional diff with the existing Blip2 QFormer code, and it was only originally copied because it wasn't registered as a special auto model yet.

Given this, the model code is pretty much independent from existing `transformers` modules (i.e., not copying or inheriting from anything), so the code in the modular file would be identical to the current modeling code. Any thoughts on how to proceed here would be super appreciated!

Awesome, thank you for all the guidance @ArthurZucker! I made the changes and have verified that the skipped tests still look good on our side 🙂

Will open a follow-up PR to add an example to the docs and enable the tests once the version of the model that is happy with transformers is up on the hub!

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36801). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.