Google's Trillson Audio Classification
Author: patrickvonplaten
Pull request: no
Labels: name: New model

name: Good Second Issue

### Model description

The TRILLsson models are described in the publication TRILLsson: Distilling Universal Paralingistic Speech Representations. From audio, they generate generally-useful paralinguistic speech representations (paralinguistics are aspects of speech other than text, such as emotion, language identification, synthetic or real, etc). These representations are smaller, faster, and publicly available versions of the state-of-the-art CAP12 embeddings, which are described in [Universal Paralinguistic Speech Representations Using Self-Supervised Conformers](https://arxiv.org/abs/2110.04621) (ICASSP 2022).

### Open source status

- [X] The model implementation is available
- [X] The model weights are available

### Provide useful links for the implementation

Google recently has done some very nice work on better audio / speech representations and distilled audio / speech representations. See:
- https://arxiv.org/abs/2110.04621
- https://arxiv.org/abs/2203.00236

Some of the distilled models are open-sourced and could be made more available via an integration to HuggingFace's Transformer library.

E.g. the following notebook shows how the weights can be loaded and run with publicly accessible model code:
https://colab.research.google.com/drive/1-D6pyxFyquIO8pss_lngL_mncHa3kAAT?usp=sharing

The relevent models to add are:
- https://tfhub.dev/google/nonsemantic-speech-benchmark/trillsson3/1 and
- https://tfhub.dev/google/nonsemantic-speech-benchmark/trillsson2/1

and the relevant code is publicly available: https://github.com/google-research/google-research/tree/master/non_semantic_speech_benchmark

The google colab shows exacty how the model can be run and debugged in TF.



Comments:
Those models are quite small <100MB, so they would be very interesting for mobile applications.

Happy to help on the integration here :-) 

Hi @patrickvonplaten , I am interested in taking a look.  Do you by chance have a good example on top of your head, maybe a pulling request for adding a model I can take a peek?  Thanks!  I can also do some search on the pull request to see whether I can find a good example. Not sure I am ready for a good second issue yet and I am happy to give it a try :)

Hi @patrickvonplaten, I would like to try on it.

Hi @patrickvonplaten  I would like to contribute to this model implementation.

Hey guys,

Cool to see so much interest here.
Would you guys like to work together on the PR here? @Ruihua-Fang if you want feel free to open a PR and then you could maybe invite @vumichien and @nandwalritik as collaborators so that you can work on the PR together. It'll be good to have multiple eyes on the model integration as it's not the easiest model. 

To begin with, I'd recommend starting the PR with https://github.com/huggingface/transformers/tree/main/templates/adding_a_new_model#add-new-model-like-command from the `speech-to-text` model (https://github.com/huggingface/transformers/tree/main/src/transformers/models/speech_to_text). 
Having automatically created all the files, the next step will then be to create the feature extractor and to verify that it matches with Google's pipeline.

Also inviting you guys to a Slack channel in case you have more questions :-)  

@Ruihua-Fang, feel free to send me an email to patrick[at]huggingface.co if you'd like to be in the Slack channel

> Hey guys,
> 
> Cool to see so much interest here. Would you guys like to work together on the PR here? @Ruihua-Fang if you want feel free to open a PR and then you could maybe invite @vumichien and @nandwalritik as collaborators so that you can work on the PR together. It'll be good to have multiple eyes on the model integration as it's not the easiest model.
> 
> To begin with, I'd recommend starting the PR with https://github.com/huggingface/transformers/tree/main/templates/adding_a_new_model#add-new-model-like-command from the `speech-to-text` model (https://github.com/huggingface/transformers/tree/main/src/transformers/models/speech_to_text). Having automatically created all the files, the next step will then be to create the feature extractor and to verify that it matches with Google's pipeline.
> 
> Also inviting you guys to a Slack channel in case you have more questions :-)

Hi @patrickvonplaten , sounds good, will get a pr and get on the slack. Thanks!

> > Hey guys,
> > Cool to see so much interest here. Would you guys like to work together on the PR here? @Ruihua-Fang if you want feel free to open a PR and then you could maybe invite @vumichien and @nandwalritik as collaborators so that you can work on the PR together. It'll be good to have multiple eyes on the model integration as it's not the easiest model.
> > To begin with, I'd recommend starting the PR with https://github.com/huggingface/transformers/tree/main/templates/adding_a_new_model#add-new-model-like-command from the `speech-to-text` model (https://github.com/huggingface/transformers/tree/main/src/transformers/models/speech_to_text). Having automatically created all the files, the next step will then be to create the feature extractor and to verify that it matches with Google's pipeline.
> > Also inviting you guys to a Slack channel in case you have more questions :-)
> 
> Hi @patrickvonplaten , sounds good, will get a pr and get on the slack. Thanks!

sounds great, Hey, @vumichien , @nandwalritik   everyone can work on this together and those of us with more more experience feel free to make suggestions and lead :) look forward to the fun :)

Hey @vumichien , @nandwalritik , A quick heads up.  I've made a branch and after generating the template files according the instruction of adding model, will then make an PR and invite your guys.  

@patrickvonplaten , @vumichien , @nandwalritik , is there any naming convention we need to follow?  Please take a look at the questions and answers listed below,  which is the result after running transformers-cli add-new-model-like and please fill/change as you see fit when you get a chance.  Thanks!  Once I got feedback from you, I'll make PR and sent invite.  Thanks!

I ran transformers-cli add-new-model-like and it ask a series question:

q1. what is the model you would like to duplicate?
wav2vec2 ?

q2. what is the name for your new model.  
ssl_conformer -> changing to trillson_efficient

q3. What identifier would you like to use for the model type of this model?
ssl_conformern -> changing to trillson_efficient

q4. What name would you like to use for the module of this model?
ssl_conformer -> changing to trillson_efficient

q5. What prefix (camel-cased) would you like to use for the model classes of this model?
Ssl_conformer -> changing to Trillson_efficient

q6. What prefix (upper-cased) would you like to use for the constants relative to this model?  
SSL_CONFORMER -> changing to TRILLSON_EFFICIENT

q7. What will be the name of the config class for this model?  
Ssl_conformerConfig -> changing to Trillson_efficientConfig

q8. Please give a checkpoint identifier (on the model Hub) for this new model. 
?

q9. Will your new model use the same processing class as wav2vec2 (Wav2Vec2FeatureExtractor, Wav2Vec2CTCTokenizer, Wav2Vec2Processor)? 
yes ?

q10. Should we add # Copied from statements when creating the new modeling file? 
yes

q11. Should we add a version of your new model in all the frameworks implemented by wav2vec2 (['pt', 'tf', 'flax'])? 
yes 



Hey @Ruihua-Fang, 

I think the model is actually not based on a Conformer architecture, rather it's based on efficientnet. So maybe trillson-efficient as a name?

> Hey @Ruihua-Fang,
> 
> I think the model is actually not based on a Conformer architecture, rather it's based on efficientnet. So maybe trillson-efficient as a name?

Hi @patrickvonplaten, thanks for catching it.  trillson_efficient sounds nice :) and I'll make the changes


> Hey guys,
> 
> Cool to see so much interest here. Would you guys like to work together on the PR here? @Ruihua-Fang if you want feel free to open a PR and then you could maybe invite @vumichien and @nandwalritik as collaborators so that you can work on the PR together. It'll be good to have multiple eyes on the model integration as it's not the easiest model.
> 
> To begin with, I'd recommend starting the PR with https://github.com/huggingface/transformers/tree/main/templates/adding_a_new_model#add-new-model-like-command from the `speech-to-text` model (https://github.com/huggingface/transformers/tree/main/src/transformers/models/speech_to_text). Having automatically created all the files, the next step will then be to create the feature extractor and to verify that it matches with Google's pipeline.
> 
> Also inviting you guys to a Slack channel in case you have more questions :-)

Sounds good, I just sent an email from my personal mail for slack channel invite, as you sent invite on my work email.

the link to the TensorFlow hub in the original trillsson https://github.com/google-research/google-research/tree/master/non_semantic_speech_benchmark/trillsson is wrong and I have submitted an issue in the original repo: https://github.com/google-research/google-research/issues/1098

Hi @Ruihua-Fang you can directly download **trillsson3** model from [here](https://tfhub.dev/google/nonsemantic-speech-benchmark/trillsson3/1)  or use this [colab demo](https://colab.research.google.com/drive/1-D6pyxFyquIO8pss_lngL_mncHa3kAAT?usp=sharing#scrollTo=Qp4bsjq8OqjT)

> Hi @Ruihua-Fang you can directly download **trillsson3** model from [here](https://tfhub.dev/google/nonsemantic-speech-benchmark/trillsson3/1) or use this [colab demo](https://colab.research.google.com/drive/1-D6pyxFyquIO8pss_lngL_mncHa3kAAT?usp=sharing#scrollTo=Qp4bsjq8OqjT)

@vumichien , great, thanks!  missed it in Patrick's instruction :)

Hi @Ruihua-Fang and @patrickvonplaten, not sure if I am late to the party or if I can contribute in some way to the model implementation. Happy to contribute. 

Sure, I'll invite you to the Slack channel! 

Hi @patrickvonplaten, if this issue is still open. I would love to contribute here. I have send you the request for the slack invite.