Music Transformers
Author: asigalov61
Pull request: no

# 🚀 Feature request

Hello guys! Thanks for your amazing work on the transformers! This is very needed and appreciated :)

I wanted to ask if it is possible to add a section/transformers dedicated specifically to Music. I searched GitHub and your model's repo but I could not find even a single model/solution that would be suitable for music.

NLP models are most capable when it comes to Music AI and I think it would be a great feature/section/branch to investigate/cover.

## Motivation

OpenAI MuseNet and Google Music Transformer. Enough said I think. If you never tried either, you have been really missing out. AFAIK MuseNet is built on a custom GPT2-like model/architecture. And Google used XLNet I think.

## Your contribution

I was able to create a decent model/code/implementation of Music AI based on GPT2 model/architecture. You are welcome to check it out here: https://github.com/asigalov61/Intelligent-VIRTUOSO

I used minGPT implementation to do it and it turned out quite capable and nice :)

However, I do want to ask the following:

1) What is the best Hugginface model/architecture you can recommend for Music AI applications? Please be specific and please give me a simple example of how to try it. This will be very much appreciated. I want to use Hugginface Transformers, so whatever works, please let me know. I am attaching a sample text file so that you can see my encoding, but I can adjust easily to any specs/needs of Huggnface Transformers. I have heard that BERT would be best at something like this but I can be mistaken...

2) Is there a nice Google Colab to try? I would prefer a simple working example to Python repos...

3) What would be the most optimal settings/hyperparameters you can recommend for GPT2 (right now I follow minGPT guidelines) and also what can you recommend to try for the most suitable Huggnface Transformer?

I really hope to hear constructive suggestions/advice because I want to learn and improve my skills and knowledge. Plus I love music almost as much as I love computers so I am quite passionate about both and would love to connect with others who are into Music and AI, if you guys exist...

Thank you very much in advance for your time and responses.
[TMIDI-TXT-Composition (13).txt](https://github.com/huggingface/transformers/files/5745869/TMIDI-TXT-Composition.13.txt)


Comments:
Hey @asigalov61,

I think applying `Transformers` to Music is a super cool idea! Regarding the best model to use for music composition, IMO it depends strongly on:

- What is the input to the model? Do you input tokens or float vectors?
- How long is the input? *e.g.* how many float vectors or tokens? GPT2 is limited to 1024 tokens / float vectors -> is this too short? 
- For generation (composition), I think only our `autoregressive models` make sense: https://huggingface.co/transformers/model_summary.html#autoregressive-models so mostly GPT2. For "classification" it would mostly be BERT. If you need very long inputs, it would be interesting to check-out ReformerLM: https://huggingface.co/google/reformer-enwik8

I bet people would be very interested in Transformers + Music. We've created a new examples folder structure for such projects, so feel free to open a PR to add a dir "music_transformers" here: https://github.com/huggingface/transformers/tree/master/examples/research_projects

Hey  Patrick,

Thank you for your help/guidance and for the welcome 🙂

I think I have created the proper PR for the new dir as you have suggested. Please check it and let me know if it is ok. I am new to PRs so it's still kinda difficult to do it right sometimes 🙂

What can I add there? Can I add my GPT2 implementation there? I should put it in a separate dirs there? Right?

Regarding your questions for me:

1) I am sorta working with what is available so atm I just use existing implementations. So I usually use implementation's way of feeding the model. I.e. for my GPT2 implementation, I use the minGPT char-based approach which is painfully slow and inefficient. minGPT does not have BPE yet so I can't really improve it and do it properly as it is very complex for me and difficult. This is why I was very interested in your work cuz you guys provide a standardized and easy way to do it.

So basically in my GPT2 implementation I simply feed it the text char tokens. I have attached the example of input/output in my original post. Check it out if you can, please.

2) I figured that GPT2 is most capable (OpenAI did the same thing with MuseNet). So I was wondering if you guys have a nice GPT2 version that is tuned to the limit. This would really help.
Also I need better tokenizer but I do not know how to do it. So if you can help/give me specific pointers, I will really appreciate it.

3) I most certainly heard about the Reformer. And it would be super cool to try it. But again, I have no idea how to make it compatible with the text input/text tokens I use. So if you can help, this also will be much appreciated.

Again, thank you for your advice.

Most sincerely,

Alex


________________________________
From: Patrick von Platen <notifications@github.com>
Sent: Monday, December 28, 2020 4:35 AM
To: huggingface/transformers <transformers@noreply.github.com>
Cc: Alex <asigalov61@hotmail.com>; Mention <mention@noreply.github.com>
Subject: Re: [huggingface/transformers] Music Transformers (#9324)


Hey @asigalov61<https://github.com/asigalov61>,

I think applying Transformers to Music is a super cool idea! Regarding the best model to use for music composition, IMO it depends strongly on:

  *   What is the input to the model? Do you input tokens or float vectors?
  *   How long is the input? e.g. how many float vectors or tokens? GPT2 is limited to 1024 tokens / float vectors -> is this too short?
  *   For generation (composition), I think only our autoregressive models make sense: https://huggingface.co/transformers/model_summary.html#autoregressive-models so mostly GPT2. For "classification" it would mostly be BERT. If you need very long inputs, it would be interesting to check-out ReformerLM: https://huggingface.co/google/reformer-enwik8

I bet people would be very interested in Transformers + Music. We've created a new examples folder structure for such projects, so feel free to open a PR to add a dir "music_transformers" here: https://github.com/huggingface/transformers/tree/master/examples/research_projects

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub<https://github.com/huggingface/transformers/issues/9324#issuecomment-751698309>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ANNXLI3QSD2OFJT5JQ4FTGLSXB3PVANCNFSM4VLQPGAA>.


Hello! I’d be willing to contribute work in this space if anyone would like to collaborate. In my previous life I was a professional audio engineer, now I’m an enterprise AI systems architect. https://www.paulprae.com/

@praeducer Hey Paul!

Thank you for responding to this thread. I would love to collab and create something based on hugginface implementations so if you can help, I would really appreciate it.

Basically, huggingface docs are very convoluted and unclear to me atm so if you can create a working collab with GPT2 hugginface implementation, I can take it from there and add music parts to it.

I need something similar to my own GPT2 implementation but based on huggingface so that we can add it here and contribute to their repo/library.

This is what I have and this is what I need:
https://github.com/asigalov61/Optimus-VIRTUOSO

And my attempt to use huggingface implementation is posted above in the thread so check it out also.

Thanks a lot. Looking forward to working together with like-minded people.

Alex.



This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.