Audio Spectrogram Transformer

## [](#overview)Overview

The Audio Spectrogram Transformer model was proposed in [AST: Audio Spectrogram Transformer](https://arxiv.org/abs/2104.01778) by Yuan Gong, Yu-An Chung, James Glass. The Audio Spectrogram Transformer applies a [Vision Transformer](vit) to audio, by turning audio into an image (spectrogram). The model obtains state-of-the-art results for audio classification.

The abstract from the paper is the following:

_In the past decade, convolutional neural networks (CNNs) have been widely adopted as the main building block for end-to-end audio classification models, which aim to learn a direct mapping from audio spectrograms to corresponding labels. To better capture long-range global context, a recent trend is to add a self-attention mechanism on top of the CNN, forming a CNN-attention hybrid model. However, it is unclear whether the reliance on a CNN is necessary, and if neural networks purely based on attention are sufficient to obtain good performance in audio classification. In this paper, we answer the question by introducing the Audio Spectrogram Transformer (AST), the first convolution-free, purely attention-based model for audio classification. We evaluate AST on various audio classification benchmarks, where it achieves new state-of-the-art results of 0.485 mAP on AudioSet, 95.6% accuracy on ESC-50, and 98.1% accuracy on Speech Commands V2._

Tips:

-   When fine-tuning the Audio Spectrogram Transformer (AST) on your own dataset, it’s recommended to take care of the input normalization (to make sure the input has mean of 0 and std of 0.5). [ASTFeatureExtractor](/docs/transformers/v4.30.0/en/model_doc/audio-spectrogram-transformer#transformers.ASTFeatureExtractor) takes care of this. Note that it uses the AudioSet mean and std by default. You can check [`ast/src/get_norm_stats.py`](https://github.com/YuanGongND/ast/blob/master/src/get_norm_stats.py) to see how the authors compute the stats for a downstream dataset.
-   Note that the AST needs a low learning rate (the authors use a 10 times smaller learning rate compared to their CNN model proposed in the [PSLA paper](https://arxiv.org/abs/2102.01243)) and converges quickly, so please search for a suitable learning rate and learning rate scheduler for your task.

![drawing](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/audio_spectogram_transformer_architecture.png) Audio pectrogram Transformer architecture. Taken from the [original paper](https://arxiv.org/abs/2104.01778).

This model was contributed by [nielsr](https://huggingface.co/nielsr). The original code can be found [here](https://github.com/YuanGongND/ast).

## [](#resources)Resources

A list of official Hugging Face and community (indicated by 🌎) resources to help you get started with the Audio Spectrogram Transformer.

-   A notebook illustrating inference with AST for audio classification can be found [here](https://github.com/NielsRogge/Transformers-Tutorials/tree/master/AST).
-   [ASTForAudioClassification](/docs/transformers/v4.30.0/en/model_doc/audio-spectrogram-transformer#transformers.ASTForAudioClassification) is supported by this [example script](https://github.com/huggingface/transformers/tree/main/examples/pytorch/audio-classification) and [notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/audio_classification.ipynb).
-   See also: [Audio classification](../tasks/audio_classification).

If you’re interested in submitting a resource to be included here, please feel free to open a Pull Request and we’ll review it! The resource should ideally demonstrate something new instead of duplicating an existing resource.

## [](#transformers.ASTConfig)ASTConfig

### class transformers.ASTConfig

[](#transformers.ASTConfig)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/audio_spectrogram_transformer/configuration_audio_spectrogram_transformer.py#L31)

( hidden\_size = 768num\_hidden\_layers = 12num\_attention\_heads = 12intermediate\_size = 3072hidden\_act = 'gelu'hidden\_dropout\_prob = 0.0attention\_probs\_dropout\_prob = 0.0initializer\_range = 0.02layer\_norm\_eps = 1e-12patch\_size = 16qkv\_bias = Truefrequency\_stride = 10time\_stride = 10max\_length = 1024num\_mel\_bins = 128\*\*kwargs )

This is the configuration class to store the configuration of a [ASTModel](/docs/transformers/v4.30.0/en/model_doc/audio-spectrogram-transformer#transformers.ASTModel). It is used to instantiate an AST model according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the AST [MIT/ast-finetuned-audioset-10-10-0.4593](https://huggingface.co/MIT/ast-finetuned-audioset-10-10-0.4593) architecture.

Configuration objects inherit from [PretrainedConfig](/docs/transformers/v4.30.0/en/main_classes/configuration#transformers.PretrainedConfig) and can be used to control the model outputs. Read the documentation from [PretrainedConfig](/docs/transformers/v4.30.0/en/main_classes/configuration#transformers.PretrainedConfig) for more information.

Example:

```
>>> from transformers import ASTConfig, ASTModel

>>> 
>>> configuration = ASTConfig()

>>> 
>>> model = ASTModel(configuration)

>>> 
>>> configuration = model.config```

## [](#transformers.ASTFeatureExtractor)ASTFeatureExtractor

( feature\_size = 1sampling\_rate = 16000num\_mel\_bins = 128max\_length = 1024padding\_value = 0.0do\_normalize = Truemean = -4.2677393std = 4.5689974return\_attention\_mask = False\*\*kwargs )

Parameters

-   [](#transformers.ASTFeatureExtractor.feature_size)**feature\_size** (`int`, _optional_, defaults to 1) — The feature dimension of the extracted features.
-   [](#transformers.ASTFeatureExtractor.sampling_rate)**sampling\_rate** (`int`, _optional_, defaults to 16000) — The sampling rate at which the audio files should be digitalized expressed in hertz (Hz).
-   [](#transformers.ASTFeatureExtractor.num_mel_bins)**num\_mel\_bins** (`int`, _optional_, defaults to 128) — Number of Mel-frequency bins.
-   [](#transformers.ASTFeatureExtractor.max_length)**max\_length** (`int`, _optional_, defaults to 1024) — Maximum length to which to pad/truncate the extracted features.
-   [](#transformers.ASTFeatureExtractor.do_normalize)**do\_normalize** (`bool`, _optional_, defaults to `True`) — Whether or not to normalize the log-Mel features using `mean` and `std`.
-   [](#transformers.ASTFeatureExtractor.mean)**mean** (`float`, _optional_, defaults to -4.2677393) — The mean value used to normalize the log-Mel features. Uses the AudioSet mean by default.
-   [](#transformers.ASTFeatureExtractor.std)**std** (`float`, _optional_, defaults to 4.5689974) — The standard deviation value used to normalize the log-Mel features. Uses the AudioSet standard deviation by default.
-   [](#transformers.ASTFeatureExtractor.return_attention_mask)**return\_attention\_mask** (`bool`, _optional_, defaults to `False`) — Whether or not [**call**()](/docs/transformers/v4.30.0/en/model_doc/audio-spectrogram-transformer#transformers.ASTFeatureExtractor.__call__) should return `attention_mask`.

Constructs a Audio Spectrogram Transformer (AST) feature extractor.

This feature extractor inherits from [SequenceFeatureExtractor](/docs/transformers/v4.30.0/en/main_classes/feature_extractor#transformers.SequenceFeatureExtractor) which contains most of the main methods. Users should refer to this superclass for more information regarding those methods.

This class extracts mel-filter bank features from raw speech using TorchAudio, pads/truncates them to a fixed length and normalizes them using a mean and standard deviation.

( raw\_speech: typing.Union\[numpy.ndarray, typing.List\[float\], typing.List\[numpy.ndarray\], typing.List\[typing.List\[float\]\]\]sampling\_rate: typing.Optional\[int\] = Nonereturn\_tensors: typing.Union\[str, transformers.utils.generic.TensorType, NoneType\] = None\*\*kwargs )

Parameters

-   [](#transformers.ASTFeatureExtractor.__call__.raw_speech)**raw\_speech** (`np.ndarray`, `List[float]`, `List[np.ndarray]`, `List[List[float]]`) — The sequence or batch of sequences to be padded. Each sequence can be a numpy array, a list of float values, a list of numpy arrays or a list of list of float values. Must be mono channel audio, not stereo, i.e. single float per timestep.
-   [](#transformers.ASTFeatureExtractor.__call__.sampling_rate)**sampling\_rate** (`int`, _optional_) — The sampling rate at which the `raw_speech` input was sampled. It is strongly recommended to pass `sampling_rate` at the forward call to prevent silent errors.
-   [](#transformers.ASTFeatureExtractor.__call__.return_tensors)**return\_tensors** (`str` or [TensorType](/docs/transformers/v4.30.0/en/internal/file_utils#transformers.TensorType), _optional_) — If set, will return tensors instead of list of python integers. Acceptable values are:
    
    -   `'tf'`: Return TensorFlow `tf.constant` objects.
    -   `'pt'`: Return PyTorch `torch.Tensor` objects.
    -   `'np'`: Return Numpy `np.ndarray` objects.
    

Main method to featurize and prepare for the model one or several sequence(s).

## [](#transformers.ASTModel)ASTModel

### class transformers.ASTModel

[](#transformers.ASTModel)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/audio_spectrogram_transformer/modeling_audio_spectrogram_transformer.py#L445)

( config: ASTConfig )

Parameters

-   [](#transformers.ASTModel.config)**config** ([ASTConfig](/docs/transformers/v4.30.0/en/model_doc/audio-spectrogram-transformer#transformers.ASTConfig)) — Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [from\_pretrained()](/docs/transformers/v4.30.0/en/main_classes/model#transformers.PreTrainedModel.from_pretrained) method to load the model weights.

The bare AST Model transformer outputting raw hidden-states without any specific head on top. This model is a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior.

#### forward

[](#transformers.ASTModel.forward)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/audio_spectrogram_transformer/modeling_audio_spectrogram_transformer.py#L469)

( input\_values: typing.Optional\[torch.Tensor\] = Nonehead\_mask: typing.Optional\[torch.Tensor\] = Noneoutput\_attentions: typing.Optional\[bool\] = Noneoutput\_hidden\_states: typing.Optional\[bool\] = Nonereturn\_dict: typing.Optional\[bool\] = None ) → [transformers.modeling\_outputs.BaseModelOutputWithPooling](/docs/transformers/v4.30.0/en/main_classes/output#transformers.modeling_outputs.BaseModelOutputWithPooling) or `tuple(torch.FloatTensor)`

The [ASTModel](/docs/transformers/v4.30.0/en/model_doc/audio-spectrogram-transformer#transformers.ASTModel) forward method, overrides the `__call__` special method.

Although the recipe for forward pass needs to be defined within this function, one should call the `Module` instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.

Example:

```
>>> from transformers import AutoProcessor, ASTModel
>>> import torch
>>> from datasets import load_dataset

>>> dataset = load_dataset("hf-internal-testing/librispeech_asr_demo", "clean", split="validation")
>>> dataset = dataset.sort("id")
>>> sampling_rate = dataset.features["audio"].sampling_rate

>>> processor = AutoProcessor.from_pretrained("MIT/ast-finetuned-audioset-10-10-0.4593")
>>> model = ASTModel.from_pretrained("MIT/ast-finetuned-audioset-10-10-0.4593")

>>> 
>>> inputs = processor(dataset[0]["audio"]["array"], sampling_rate=sampling_rate, return_tensors="pt")
>>> with torch.no_grad():
...     outputs = model(**inputs)

>>> last_hidden_states = outputs.last_hidden_state
>>> list(last_hidden_states.shape)
[1, 1214, 768]```

## [](#transformers.ASTForAudioClassification)ASTForAudioClassification

### class transformers.ASTForAudioClassification

[](#transformers.ASTForAudioClassification)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/audio_spectrogram_transformer/modeling_audio_spectrogram_transformer.py#L545)

( config: ASTConfig )

Parameters

-   [](#transformers.ASTForAudioClassification.config)**config** ([ASTConfig](/docs/transformers/v4.30.0/en/model_doc/audio-spectrogram-transformer#transformers.ASTConfig)) — Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [from\_pretrained()](/docs/transformers/v4.30.0/en/main_classes/model#transformers.PreTrainedModel.from_pretrained) method to load the model weights.

Audio Spectrogram Transformer model with an audio classification head on top (a linear layer on top of the pooled output) e.g. for datasets like AudioSet, Speech Commands v2.

This model is a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior.

#### forward

[](#transformers.ASTForAudioClassification.forward)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/audio_spectrogram_transformer/modeling_audio_spectrogram_transformer.py#L558)

( input\_values: typing.Optional\[torch.Tensor\] = Nonehead\_mask: typing.Optional\[torch.Tensor\] = Nonelabels: typing.Optional\[torch.Tensor\] = Noneoutput\_attentions: typing.Optional\[bool\] = Noneoutput\_hidden\_states: typing.Optional\[bool\] = Nonereturn\_dict: typing.Optional\[bool\] = None ) → [transformers.modeling\_outputs.SequenceClassifierOutput](/docs/transformers/v4.30.0/en/main_classes/output#transformers.modeling_outputs.SequenceClassifierOutput) or `tuple(torch.FloatTensor)`

The [ASTForAudioClassification](/docs/transformers/v4.30.0/en/model_doc/audio-spectrogram-transformer#transformers.ASTForAudioClassification) forward method, overrides the `__call__` special method.

Although the recipe for forward pass needs to be defined within this function, one should call the `Module` instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.

Example:

```
>>> from transformers import AutoFeatureExtractor, ASTForAudioClassification
>>> from datasets import load_dataset
>>> import torch

>>> dataset = load_dataset("hf-internal-testing/librispeech_asr_demo", "clean", split="validation")
>>> dataset = dataset.sort("id")
>>> sampling_rate = dataset.features["audio"].sampling_rate

>>> feature_extractor = AutoFeatureExtractor.from_pretrained("MIT/ast-finetuned-audioset-10-10-0.4593")
>>> model = ASTForAudioClassification.from_pretrained("MIT/ast-finetuned-audioset-10-10-0.4593")

>>> 
>>> inputs = feature_extractor(dataset[0]["audio"]["array"], sampling_rate=sampling_rate, return_tensors="pt")

>>> with torch.no_grad():
...     logits = model(**inputs).logits

>>> predicted_class_ids = torch.argmax(logits, dim=-1).item()
>>> predicted_label = model.config.id2label[predicted_class_ids]
>>> predicted_label
'Speech'

>>> 
>>> target_label = model.config.id2label[0]
>>> inputs["labels"] = torch.tensor([model.config.label2id[target_label]])
>>> loss = model(**inputs).loss
>>> round(loss.item(), 2)
0.17```