[WIP] - Add Spotify LLark
Author: jonflynng
Pull request: yes

# What does this PR do?

Implementing Spotify's LLark model: https://research.atspotify.com/2023/10/llark-a-multimodal-foundation-model-for-music/

The methodology is the same as LLaVa but works on audio rather than images. It uses a large pre-trained Music generation model (like Jukebox or MusicGen) to generate and project embeddings into the language model space. These embeddings are then injected into the input sequence of the language model for inference.

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker
- vision models: @amyeroberts
- speech models: @sanchit-gandhi
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- pipelines: @Narsil
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @muellerzr and @SunMarc

Integrations:

- deepspeed: HF Trainer/Accelerate: @muellerzr
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc 

Documentation: @stevhliu and @MKhalusova

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
@jonflynng thanks for opening a PR! Let us know when it's ready for review. 

Pinging some of the audio team for reference: @sanchit-gandhi @ylacombe @kamilakesbi 

Hey @jonflynng, thanks for opening a PR!

Let us know when it's ready for a first review. In the meantime, I'm a bit curious about the model. From the github repository, I see that there's no pre-trained checkpoints available (cf [here](https://github.com/spotify-research/llark?tab=readme-ov-file#llark-a-multimodal-instruction-following-language-model-for-music)), so I'm not sure to see the interest of adding the model.
I'd be curious to hear your thoughts on this, and the reasons behind adding the model!
Thanks in advance

> Hey @jonflynng, thanks for opening a PR!
> 
> Let us know when it's ready for a first review. In the meantime, I'm a bit curious about the model. From the github repository, I see that there's no pre-trained checkpoints available (cf [here](https://github.com/spotify-research/llark?tab=readme-ov-file#llark-a-multimodal-instruction-following-language-model-for-music)), so I'm not sure to see the interest of adding the model. I'd be curious to hear your thoughts on this, and the reasons behind adding the model! Thanks in advance

Hey @ylacombe 

Yeah, it seems like Spotify didn't release weights for the model. They state this in the paper:

"In order to comply with the licenses specified by the artists who contributed to the training data, we are unable to provide the exact training data, instruction data, or trained model weights."

I'm not sure about this since many of the datasets used for training like Slakh and Maestro were created specifically for training ML models and many have been released trained on these datasets.

Regarding interest of adding the model, there’s a few reasons for this:

 One is that the model design leverages a pre-trained music and text model, so I assume this doesn’t warrant the need for a well fine-tuned base checkpoint so much as people can pick up the model and fine-tune it as they would a text model. The model in the paper was trained “on 4 80GB NVIDIA A100 GPUs. Training takes approximately 54 hours.” I’m hoping though that with better/bigger text models and PEFT less resource will be needed to achieve the same performance as the paper. I've replicated the model in a Colab notebook using MusicGen and PEFT, to try to train it exclusively on transcribing music with the aim of the language model outputting the correct transcription in ABC notation (results are actually ok so far, I'll try a bunch of different things and maybe if it performs well this could be used as a checkpoint, we’ll see)

  The main reason is just ease of use and having an implementation people can quickly pick and fine-tune in something like Colab on a single A100 40GB. Running the original code was tedious and it doesn't allow for easy replacement and plugging in different music models like MusicGen or better text models. And also it's just way more time consuming to get set up with a GPU on a Colab or other cloud instance than using the `transformers` library. 

  Thirdly, the implementation itself is pretty simple, the methodology is identical to the LLaVa model, requiring a bit of torch code to inject the audio tokens into the input sequence for the language model, I got it done in about 200 lines on Colab. Therefore, it could be quite straight forward and quick to add to the library. 

Let me know what you think.

@jonflynng, thanks for the thorough explanation, I see what's the interest now! I'm actually quite excited about your efforts and what could come out of it!

That said, I also believe that the model has no place in transformers **for now**. Here's why:
1. Each new model in `transformers` adds a maintenance burden to an already heavily loaded library. We usually seek to integrate models / codes that are going to be heavily used, and most models are used mainly for inference, rather than training. 
2. No model checkpoints means no way to test that the model has been properly integrated. Note that we have thorough integration tests, see for example [those of Whisper](https://github.com/huggingface/transformers/blob/main/tests/models/whisper/test_modeling_whisper.py).

However, I do believe that integrating this model has potential, the only limiting factor being having promising checkpoints that can be used. 
The way I'd go if I were you is that:
1. I would first create an independent repository that would rely on `transformers`, which could be used for both training and inference.  If I remember correctly, there are two ways to still use `transformers` without having a proper integration. The first way is having the model on the hub. The best example of this is DeepSeek v2, that you can use like [this](https://huggingface.co/deepseek-ai/DeepSeek-V2-Lite-Chat#text-completion), its modeling code being hosted directly on the hub [here](https://huggingface.co/deepseek-ai/DeepSeek-V2-Lite-Chat/blob/main/modeling_deepseek.py). The second way to do it is to create a git repository.  See for example [MobileLLM](https://github.com/facebookresearch/MobileLLM/blob/main/utils/modeling_llama.py) from Meta, or [Parler-TTS](https://github.com/huggingface/parler-tts).
2. Then, if usage and interest is high enough, you can open a proper PR that I'd be really happy to review.

This approach would bring the best of both worlds: checkpoints would be available, training would be easy thanks to your library and the integration would be justified by usage / interest.

What do you think of this?

Note that I can still take a look at your approach to give you a first review, that would simplify future integration, if you want!

cc @amyeroberts, if you want to add something !  

@ylacombe 

Sounds good, thanks for the advice :)

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.