AST: Audio Spectrogram Transformer
Author: patrickvonplaten
Pull request: no
Labels: name: New model

name: Good Second Issue

# 🌟 New model addition

## Model description

In the past decade, convolutional neural networks (CNNs)
have been widely adopted as the main building block for endto-end audio classification models, which aim to learn a direct
mapping from audio spectrograms to corresponding labels. To
better capture long-range global context, a recent trend is to
add a self-attention mechanism on top of the CNN, forming a
CNN-attention hybrid model. However, it is unclear whether
the reliance on a CNN is necessary, and if neural networks
purely based on attention are sufficient to obtain good performance in audio classification. In this paper, we answer the question by introducing the Audio Spectrogram Transformer (AST),
the first convolution-free, purely attention-based model for audio classification. We evaluate AST on various audio classification benchmarks, where it achieves new state-of-the-art results of 0.485 mAP on AudioSet, 95.6% accuracy on ESC-50,
and 98.1% accuracy on Speech Commands V2.
Index Terms: audio classification, self-attention

## Open source status

* [x] the model implementation is available: https://github.com/YuanGongND/ast
* [x] the model weights are available: https://github.com/YuanGongND/ast#Pretrained-Models
* [x] who are the authors: @YuanGongND


Happy to supervise anyone interested in porting the model :-)

Comments:
Hi Patrick,

Thanks so much to bring this up. I'd love to add AST to Huggingface. 

I am checking https://github.com/huggingface/transformers/tree/main/templates/adding_a_new_model for how to do that, is that the right tutorial I should read? 

I also have a quick question: the AST model is built on the `timm` package, is that OK with such dependency? 

Thanks!

-Yuan

Oh wow super cool to get such a quick answer from you :-) It would be amazing if you could give it a try with the https://github.com/huggingface/transformers/tree/main/templates/adding_a_new_model docs! 

If possible, it'd be great to avoid a `timm` dependency and instead more or less copy paste the relevant code of `timm` into `transformers`. @NielsRogge mentioned that the model is more or less a ViT, so maybe we can get some inspiration from the existing ViT model: https://github.com/huggingface/transformers/blob/main/src/transformers/models/vit/modeling_vit.py . It's also totally fine to add the `timm` dependency in a first step and we'll help you adjust the PR afterwards :-) 

cc @sgugger @LysandreJik @anton-l for visibility 

Sure. Thanks for the suggestion. I will have a try and let you know if I have a question.



@YuanGongND if I may pitch in -- without `timm` TF folks like me can easily port your model to TF 🧡 

@YuanGongND Did it work ?

Hi, I just need to find some time to do that, will be soon!

@YuanGongND did you manage to get this going or do you need some help? I'd be interested in contributing if possible! 😄 

Hi @thefirebanks, I apologize that I won't have time to do it in near future. It would be great if you could contribute! FYI, we have a repo at [here](https://github.com/YuanGongND/ast) (implemented in PyTorch and depends on `timm`) and a colab demo at [here](https://colab.research.google.com/github/YuanGongND/ast/blob/master/colab/AST_Inference_Demo.ipynb). Please let me know if you have any question!

Best,
Yuan

Hi,

I actually have a working implementation which I need to finish. Will open a PR soon, hopefully 

Wow @NielsRogge, that's great!

Gotcha Niels, thanks!