CLAP

## [](#overview)Overview

The CLAP model was proposed in [Large Scale Contrastive Language-Audio pretraining with feature fusion and keyword-to-caption augmentation](https://arxiv.org/pdf/2211.06687.pdf) by Yusong Wu, Ke Chen, Tianyu Zhang, Yuchen Hui, Taylor Berg-Kirkpatrick, Shlomo Dubnov.

CLAP (Contrastive Language-Audio Pretraining) is a neural network trained on a variety of (audio, text) pairs. It can be instructed in to predict the most relevant text snippet, given an audio, without directly optimizing for the task. The CLAP model uses a SWINTransformer to get audio features from a log-Mel spectrogram input, and a RoBERTa model to get text features. Both the text and audio features are then projected to a latent space with identical dimension. The dot product between the projected audio and text features is then used as a similar score.

The abstract from the paper is the following:

_Contrastive learning has shown remarkable success in the field of multimodal representation learning. In this paper, we propose a pipeline of contrastive language-audio pretraining to develop an audio representation by combining audio data with natural language descriptions. To accomplish this target, we first release LAION-Audio-630K, a large collection of 633,526 audio-text pairs from different data sources. Second, we construct a contrastive language-audio pretraining model by considering different audio encoders and text encoders. We incorporate the feature fusion mechanism and keyword-to-caption augmentation into the model design to further enable the model to process audio inputs of variable lengths and enhance the performance. Third, we perform comprehensive experiments to evaluate our model across three tasks: text-to-audio retrieval, zero-shot audio classification, and supervised audio classification. The results demonstrate that our model achieves superior performance in text-to-audio retrieval task. In audio classification tasks, the model achieves state-of-the-art performance in the zeroshot setting and is able to obtain performance comparable to models’ results in the non-zero-shot setting. LAION-Audio-6_

This model was contributed by [Younes Belkada](https://huggingface.co/ybelkada) and [Arthur Zucker](https://huggingface.co/ArtZucker) . The original code can be found [here](https://github.com/LAION-AI/Clap).

## [](#transformers.ClapConfig)ClapConfig

### class transformers.ClapConfig

[](#transformers.ClapConfig)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/clap/configuration_clap.py#L338)

( text\_config = Noneaudio\_config = Nonelogit\_scale\_init\_value = 14.285714285714285projection\_dim = 512projection\_hidden\_act = 'relu'initializer\_factor = 1.0\*\*kwargs )

Parameters

-   [](#transformers.ClapConfig.text_config)**text\_config** (`dict`, _optional_) — Dictionary of configuration options used to initialize [ClapTextConfig](/docs/transformers/v4.30.0/en/model_doc/clap#transformers.ClapTextConfig).
-   [](#transformers.ClapConfig.audio_config)**audio\_config** (`dict`, _optional_) — Dictionary of configuration options used to initialize [ClapAudioConfig](/docs/transformers/v4.30.0/en/model_doc/clap#transformers.ClapAudioConfig).
-   [](#transformers.ClapConfig.projection_dim)**projection\_dim** (`int`, _optional_, defaults to 512) — Dimentionality of text and audio projection layers.
-   [](#transformers.ClapConfig.logit_scale_init_value)**logit\_scale\_init\_value** (`float`, _optional_, defaults to 2.6592) — The inital value of the _logit\_scale_ paramter. Default is used as per the original CLAP implementation.
-   [](#transformers.ClapConfig.projection_hidden_act)**projection\_hidden\_act** (`str`, _optional_, defaults to `"relu"`) — Activation function for the projection layers.
-   [](#transformers.ClapConfig.initializer_factor)**initializer\_factor** (`float`, _optional_, defaults to 1.0) — Factor to scale the initialization of the model weights.
-   [](#transformers.ClapConfig.kwargs)**kwargs** (_optional_) — Dictionary of keyword arguments.

[ClapConfig](/docs/transformers/v4.30.0/en/model_doc/clap#transformers.ClapConfig) is the configuration class to store the configuration of a [ClapModel](/docs/transformers/v4.30.0/en/model_doc/clap#transformers.ClapModel). It is used to instantiate a CLAP model according to the specified arguments, defining the text model and audio model configs. Instantiating a configuration with the defaults will yield a similar configuration to that of the CLAP [laion/clap-htsat-fused](https://huggingface.co/laion/clap-htsat-fused) architecture.

Configuration objects inherit from [PretrainedConfig](/docs/transformers/v4.30.0/en/main_classes/configuration#transformers.PretrainedConfig) and can be used to control the model outputs. Read the documentation from [PretrainedConfig](/docs/transformers/v4.30.0/en/main_classes/configuration#transformers.PretrainedConfig) for more information.

Example:

```
>>> from transformers import ClapConfig, ClapModel

>>> 
>>> configuration = ClapConfig()

>>> 
>>> model = ClapModel(configuration)

>>> 
>>> configuration = model.config

>>> 
>>> from transformers import ClapTextConfig, ClapAudioConfig

>>> 
>>> config_text = ClapTextConfig()
>>> config_audio = ClapAudioConfig()

>>> config = ClapConfig.from_text_audio_configs(config_text, config_audio)```

#### from\_text\_audio\_configs

[](#transformers.ClapConfig.from_text_audio_configs)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/clap/configuration_clap.py#L427)

( text\_config: ClapTextConfigaudio\_config: ClapAudioConfig\*\*kwargs ) → [ClapConfig](/docs/transformers/v4.30.0/en/model_doc/clap#transformers.ClapConfig)

An instance of a configuration object

Instantiate a [ClapConfig](/docs/transformers/v4.30.0/en/model_doc/clap#transformers.ClapConfig) (or a derived class) from clap text model configuration and clap audio model configuration.

## [](#transformers.ClapTextConfig)ClapTextConfig

### class transformers.ClapTextConfig

[](#transformers.ClapTextConfig)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/clap/configuration_clap.py#L33)

( vocab\_size = 50265hidden\_size = 768num\_hidden\_layers = 12num\_attention\_heads = 12intermediate\_size = 3072hidden\_act = 'gelu'hidden\_dropout\_prob = 0.1attention\_probs\_dropout\_prob = 0.1max\_position\_embeddings = 514type\_vocab\_size = 1initializer\_range = 0.02initializer\_factor = 1.0layer\_norm\_eps = 1e-12projection\_dim = 512pad\_token\_id = 1bos\_token\_id = 0eos\_token\_id = 2position\_embedding\_type = 'absolute'use\_cache = Trueclassifier\_dropout = Noneprojection\_hidden\_act = 'relu'\*\*kwargs )

This is the configuration class to store the configuration of a [ClapTextModel](/docs/transformers/v4.30.0/en/model_doc/clap#transformers.ClapTextModel). It is used to instantiate a CLAP model according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the CLAP [calp-hsat-fused](https://huggingface.co/laion/clap-hsat-fused) architecture.

Configuration objects inherit from [PretrainedConfig](/docs/transformers/v4.30.0/en/main_classes/configuration#transformers.PretrainedConfig) and can be used to control the model outputs. Read the documentation from [PretrainedConfig](/docs/transformers/v4.30.0/en/main_classes/configuration#transformers.PretrainedConfig) for more information.

Examples:

```
>>> from transformers import ClapTextConfig, ClapTextModel

>>> 
>>> configuration = ClapTextConfig()

>>> 
>>> model = ClapTextModel(configuration)

>>> 
>>> configuration = model.config```

## [](#transformers.ClapAudioConfig)ClapAudioConfig

### class transformers.ClapAudioConfig

[](#transformers.ClapAudioConfig)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/clap/configuration_clap.py#L170)

( window\_size = 8num\_mel\_bins = 64spec\_size = 256hidden\_act = 'gelu'patch\_size = 4patch\_stride = \[4, 4\]num\_classes = 527hidden\_size = 768projection\_dim = 512depths = \[2, 2, 6, 2\]num\_attention\_heads = \[4, 8, 16, 32\]enable\_fusion = Falsehidden\_dropout\_prob = 0.1fusion\_type = Nonepatch\_embed\_input\_channels = 1flatten\_patch\_embeds = Truepatch\_embeds\_hidden\_size = 96enable\_patch\_layer\_norm = Truedrop\_path\_rate = 0.0attention\_probs\_dropout\_prob = 0.0qkv\_bias = Truemlp\_ratio = 4.0aff\_block\_r = 4num\_hidden\_layers = 4projection\_hidden\_act = 'relu'layer\_norm\_eps = 1e-05initializer\_factor = 1.0\*\*kwargs )

This is the configuration class to store the configuration of a [ClapAudioModel](/docs/transformers/v4.30.0/en/model_doc/clap#transformers.ClapAudioModel). It is used to instantiate a CLAP audio encoder according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the audio encoder of the CLAP [laion/clap-htsat-fused](https://huggingface.co/laion/clap-htsat-fused) architecture.

Configuration objects inherit from [PretrainedConfig](/docs/transformers/v4.30.0/en/main_classes/configuration#transformers.PretrainedConfig) and can be used to control the model outputs. Read the documentation from [PretrainedConfig](/docs/transformers/v4.30.0/en/main_classes/configuration#transformers.PretrainedConfig) for more information.

Example:

```
>>> from transformers import ClapAudioConfig, ClapAudioModel

>>> 
>>> configuration = ClapAudioConfig()

>>> 
>>> model = ClapAudioModel(configuration)

>>> 
>>> configuration = model.config```

## [](#transformers.ClapFeatureExtractor)ClapFeatureExtractor

( feature\_size = 64sampling\_rate = 48000hop\_length = 480max\_length\_s = 10fft\_window\_size = 1024padding\_value = 0.0return\_attention\_mask = Falsefrequency\_min: float = 0frequency\_max: float = 14000top\_db: int = Nonetruncation: str = 'fusion'padding: str = 'repeatpad'\*\*kwargs )

Constructs a CLAP feature extractor.

This feature extractor inherits from [SequenceFeatureExtractor](/docs/transformers/v4.30.0/en/main_classes/feature_extractor#transformers.SequenceFeatureExtractor) which contains most of the main methods. Users should refer to this superclass for more information regarding those methods.

This class extracts mel-filter bank features from raw speech using a custom numpy implementation of the _Short Time Fourier Transform_ (STFT) which should match pytorch’s `torch.stft` equivalent.

( ) → `Dict[str, Any]`

Dictionary of all the attributes that make up this configuration instance, excpet for the mel filter banks, which do not need to be saved or printed as they are too long.

Serializes this instance to a Python dictionary.

## [](#transformers.ClapProcessor)ClapProcessor

Constructs a CLAP processor which wraps a CLAP feature extractor and a RoBerta tokenizer into a single processor.

[ClapProcessor](/docs/transformers/v4.30.0/en/model_doc/clap#transformers.ClapProcessor) offers all the functionalities of [ClapFeatureExtractor](/docs/transformers/v4.30.0/en/model_doc/clap#transformers.ClapFeatureExtractor) and [RobertaTokenizerFast](/docs/transformers/v4.30.0/en/model_doc/roberta#transformers.RobertaTokenizerFast). See the `__call__()` and [decode()](/docs/transformers/v4.30.0/en/model_doc/clap#transformers.ClapProcessor.decode) for more information.

This method forwards all its arguments to RobertaTokenizerFast’s [batch\_decode()](/docs/transformers/v4.30.0/en/model_doc/speecht5#transformers.SpeechT5Tokenizer.batch_decode). Please refer to the docstring of this method for more information.

This method forwards all its arguments to RobertaTokenizerFast’s [decode()](/docs/transformers/v4.30.0/en/model_doc/speecht5#transformers.SpeechT5Tokenizer.decode). Please refer to the docstring of this method for more information.

## [](#transformers.ClapModel)ClapModel

### class transformers.ClapModel

[](#transformers.ClapModel)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/clap/modeling_clap.py#L1937)

( config: ClapConfig )

Parameters

-   [](#transformers.ClapModel.config)**config** ([ClapConfig](/docs/transformers/v4.30.0/en/model_doc/clap#transformers.ClapConfig)) — Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [from\_pretrained()](/docs/transformers/v4.30.0/en/main_classes/model#transformers.PreTrainedModel.from_pretrained) method to load the model weights.

This model inherits from [PreTrainedModel](/docs/transformers/v4.30.0/en/main_classes/model#transformers.PreTrainedModel). Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.)

This model is also a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior.

#### forward

[](#transformers.ClapModel.forward)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/clap/modeling_clap.py#L2067)

( input\_ids: typing.Optional\[torch.LongTensor\] = Noneinput\_features: typing.Optional\[torch.FloatTensor\] = Noneis\_longer: typing.Optional\[torch.BoolTensor\] = Noneattention\_mask: typing.Optional\[torch.Tensor\] = Noneposition\_ids: typing.Optional\[torch.LongTensor\] = Nonereturn\_loss: typing.Optional\[bool\] = Noneoutput\_attentions: typing.Optional\[bool\] = Noneoutput\_hidden\_states: typing.Optional\[bool\] = Nonereturn\_dict: typing.Optional\[bool\] = None ) → `transformers.models.clap.modeling_clap.ClapOutput` or `tuple(torch.FloatTensor)`

The [ClapModel](/docs/transformers/v4.30.0/en/model_doc/clap#transformers.ClapModel) forward method, overrides the `__call__` special method.

Although the recipe for forward pass needs to be defined within this function, one should call the `Module` instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.

Examples:

```
>>> from datasets import load_dataset
>>> from transformers import AutoProcessor, ClapModel

>>> dataset = load_dataset("ashraq/esc50")
>>> audio_sample = dataset["train"]["audio"][0]["array"]

>>> model = ClapModel.from_pretrained("laion/clap-htsat-unfused")
>>> processor = AutoProcessor.from_pretrained("laion/clap-htsat-unfused")

>>> input_text = ["Sound of a dog", "Sound of vaccum cleaner"]

>>> inputs = processor(text=input_text, audios=audio_sample, return_tensors="pt", padding=True)

>>> outputs = model(**inputs)
>>> logits_per_audio = outputs.logits_per_audio  
>>> probs = logits_per_audio.softmax(dim=-1)  ```

#### get\_text\_features

[](#transformers.ClapModel.get_text_features)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/clap/modeling_clap.py#L1973)

( input\_ids: typing.Optional\[torch.Tensor\] = Noneattention\_mask: typing.Optional\[torch.Tensor\] = Noneposition\_ids: typing.Optional\[torch.Tensor\] = Noneoutput\_attentions: typing.Optional\[bool\] = Noneoutput\_hidden\_states: typing.Optional\[bool\] = Nonereturn\_dict: typing.Optional\[bool\] = None ) → text\_features (`torch.FloatTensor` of shape `(batch_size, output_dim`)

The [ClapModel](/docs/transformers/v4.30.0/en/model_doc/clap#transformers.ClapModel) forward method, overrides the `__call__` special method.

Although the recipe for forward pass needs to be defined within this function, one should call the `Module` instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.

Examples:

```
>>> from transformers import AutoTokenizer, ClapModel

>>> model = ClapModel.from_pretrained("laion/clap-htsat-unfused")
>>> tokenizer = AutoTokenizer.from_pretrained("laion/clap-htsat-unfused")

>>> inputs = tokenizer(["the sound of a cat", "the sound of a dog"], padding=True, return_tensors="pt")
>>> text_features = model.get_text_features(**inputs)```

#### get\_audio\_features

[](#transformers.ClapModel.get_audio_features)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/clap/modeling_clap.py#L2021)

( input\_features: typing.Optional\[torch.Tensor\] = Noneis\_longer: typing.Optional\[torch.Tensor\] = Noneattention\_mask: typing.Optional\[torch.Tensor\] = Noneoutput\_attentions: typing.Optional\[bool\] = Noneoutput\_hidden\_states: typing.Optional\[bool\] = Nonereturn\_dict: typing.Optional\[bool\] = None ) → audio\_features (`torch.FloatTensor` of shape `(batch_size, output_dim`)

The [ClapModel](/docs/transformers/v4.30.0/en/model_doc/clap#transformers.ClapModel) forward method, overrides the `__call__` special method.

Although the recipe for forward pass needs to be defined within this function, one should call the `Module` instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.

Examples:

```
>>> from transformers import AutoFeatureExtractor, ClapModel
>>> import torch

>>> model = ClapModel.from_pretrained("laion/clap-htsat-unfused")
>>> feature_extractor = AutoFeatureExtractor.from_pretrained("laion/clap-htsat-unfused")
>>> random_audio = torch.rand((16_000))
>>> inputs = feature_extractor(random_audio, return_tensors="pt")
>>> audio_features = model.get_audio_features(**inputs)```

## [](#transformers.ClapTextModel)ClapTextModel

### class transformers.ClapTextModel

[](#transformers.ClapTextModel)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/clap/modeling_clap.py#L1767)

( configadd\_pooling\_layer = True )

The model can behave as an encoder (with only self-attention) as well as a decoder, in which case a layer of cross-attention is added between the self-attention layers, following the architecture described in _Attention is all you need_\_ by Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser and Illia Polosukhin.

To behave as an decoder the model needs to be initialized with the `is_decoder` argument of the configuration set to `True`. To be used in a Seq2Seq model, the model needs to initialized with both `is_decoder` argument and `add_cross_attention` set to `True`; an `encoder_hidden_states` is then expected as an input to the forward pass.

.. \__Attention is all you need_: [https://arxiv.org/abs/1706.03762](https://arxiv.org/abs/1706.03762)

#### forward

[](#transformers.ClapTextModel.forward)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/clap/modeling_clap.py#L1806)

( input\_ids: typing.Optional\[torch.Tensor\] = Noneattention\_mask: typing.Optional\[torch.Tensor\] = Nonetoken\_type\_ids: typing.Optional\[torch.Tensor\] = Noneposition\_ids: typing.Optional\[torch.Tensor\] = Nonehead\_mask: typing.Optional\[torch.Tensor\] = Noneinputs\_embeds: typing.Optional\[torch.Tensor\] = Noneencoder\_hidden\_states: typing.Optional\[torch.Tensor\] = Noneencoder\_attention\_mask: typing.Optional\[torch.Tensor\] = Nonepast\_key\_values: typing.Optional\[typing.List\[torch.FloatTensor\]\] = Noneuse\_cache: typing.Optional\[bool\] = Noneoutput\_attentions: typing.Optional\[bool\] = Noneoutput\_hidden\_states: typing.Optional\[bool\] = Nonereturn\_dict: typing.Optional\[bool\] = None )

encoder\_hidden\_states (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, _optional_): Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention if the model is configured as a decoder. encoder\_attention\_mask (`torch.FloatTensor` of shape `(batch_size, sequence_length)`, _optional_): Mask to avoid performing attention on the padding token indices of the encoder input. This mask is used in the cross-attention if the model is configured as a decoder. Mask values selected in `[0, 1]`:

-   1 for tokens that are **not masked**,
-   0 for tokens that are **masked**. past\_key\_values (`tuple(tuple(torch.FloatTensor))` of length `config.n_layers` with each tuple having 4 tensors of shape `(batch_size, num_heads, sequence_length - 1, embed_size_per_head)`): Contains precomputed key and value hidden states of the attention blocks. Can be used to speed up decoding.

If `past_key_values` are used, the user can optionally input only the last `decoder_input_ids` (those that don’t have their past key value states given to this model) of shape `(batch_size, 1)` instead of all `decoder_input_ids` of shape `(batch_size, sequence_length)`. use\_cache (`bool`, _optional_): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`).

## [](#transformers.ClapTextModelWithProjection)ClapTextModelWithProjection

### class transformers.ClapTextModelWithProjection

[](#transformers.ClapTextModelWithProjection)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/clap/modeling_clap.py#L2171)

( config: ClapTextConfig )

Parameters

-   [](#transformers.ClapTextModelWithProjection.config)**config** ([ClapConfig](/docs/transformers/v4.30.0/en/model_doc/clap#transformers.ClapConfig)) — Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [from\_pretrained()](/docs/transformers/v4.30.0/en/main_classes/model#transformers.PreTrainedModel.from_pretrained) method to load the model weights.

CLAP Text Model with a projection layer on top (a linear layer on top of the pooled output).

This model inherits from [PreTrainedModel](/docs/transformers/v4.30.0/en/main_classes/model#transformers.PreTrainedModel). Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.)

This model is also a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior.

#### forward

[](#transformers.ClapTextModelWithProjection.forward)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/clap/modeling_clap.py#L2187)

( input\_ids: typing.Optional\[torch.Tensor\] = Noneattention\_mask: typing.Optional\[torch.Tensor\] = Noneposition\_ids: typing.Optional\[torch.Tensor\] = Noneoutput\_attentions: typing.Optional\[bool\] = Noneoutput\_hidden\_states: typing.Optional\[bool\] = Nonereturn\_dict: typing.Optional\[bool\] = None ) → `transformers.models.clap.modeling_clap.ClapTextModelOutput` or `tuple(torch.FloatTensor)`

The [ClapTextModelWithProjection](/docs/transformers/v4.30.0/en/model_doc/clap#transformers.ClapTextModelWithProjection) forward method, overrides the `__call__` special method.

Although the recipe for forward pass needs to be defined within this function, one should call the `Module` instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.

Examples:

```
>>> from transformers import AutoTokenizer, ClapTextModelWithProjection

>>> model = ClapTextModelWithProjection.from_pretrained("laion/clap-htsat-unfused")
>>> tokenizer = AutoTokenizer.from_pretrained("laion/clap-htsat-unfused")

>>> inputs = tokenizer(["a sound of a cat", "a sound of a dog"], padding=True, return_tensors="pt")

>>> outputs = model(**inputs)
>>> text_embeds = outputs.text_embeds```

## [](#transformers.ClapAudioModel)ClapAudioModel

### class transformers.ClapAudioModel

[](#transformers.ClapAudioModel)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/clap/modeling_clap.py#L1709)

( config: ClapAudioConfig )

#### forward

[](#transformers.ClapAudioModel.forward)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/clap/modeling_clap.py#L1722)

( input\_features: typing.Optional\[torch.FloatTensor\] = Noneis\_longer: typing.Optional\[torch.BoolTensor\] = Noneoutput\_attentions: typing.Optional\[bool\] = Noneoutput\_hidden\_states: typing.Optional\[bool\] = Nonereturn\_dict: typing.Optional\[bool\] = None ) → [transformers.modeling\_outputs.BaseModelOutputWithPooling](/docs/transformers/v4.30.0/en/main_classes/output#transformers.modeling_outputs.BaseModelOutputWithPooling) or `tuple(torch.FloatTensor)`

The [ClapAudioModel](/docs/transformers/v4.30.0/en/model_doc/clap#transformers.ClapAudioModel) forward method, overrides the `__call__` special method.

Although the recipe for forward pass needs to be defined within this function, one should call the `Module` instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.

Examples:

```
>>> from datasets import load_dataset
>>> from transformers import AutoProcessor, ClapAudioModel

>>> dataset = load_dataset("ashraq/esc50")
>>> audio_sample = dataset["train"]["audio"][0]["array"]

>>> model = ClapAudioModel.from_pretrained("laion/clap-htsat-fused")
>>> processor = AutoProcessor.from_pretrained("laion/clap-htsat-fused")

>>> inputs = processor(audios=audio_sample, return_tensors="pt")

>>> outputs = model(**inputs)
>>> last_hidden_state = outputs.last_hidden_state```

## [](#transformers.ClapAudioModelWithProjection)ClapAudioModelWithProjection

### class transformers.ClapAudioModelWithProjection

[](#transformers.ClapAudioModelWithProjection)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/clap/modeling_clap.py#L2247)

( config: ClapAudioConfig )

Parameters

-   [](#transformers.ClapAudioModelWithProjection.config)**config** ([ClapConfig](/docs/transformers/v4.30.0/en/model_doc/clap#transformers.ClapConfig)) — Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [from\_pretrained()](/docs/transformers/v4.30.0/en/main_classes/model#transformers.PreTrainedModel.from_pretrained) method to load the model weights.

CLAP Audio Model with a projection layer on top (a linear layer on top of the pooled output).

This model inherits from [PreTrainedModel](/docs/transformers/v4.30.0/en/main_classes/model#transformers.PreTrainedModel). Check the superclass documentation for the generic methods the library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads etc.)

This model is also a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) subclass. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior.

#### forward

[](#transformers.ClapAudioModelWithProjection.forward)[< source \>](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/clap/modeling_clap.py#L2261)

( input\_features: typing.Optional\[torch.FloatTensor\] = Noneis\_longer: typing.Optional\[torch.BoolTensor\] = Noneoutput\_attentions: typing.Optional\[bool\] = Noneoutput\_hidden\_states: typing.Optional\[bool\] = Nonereturn\_dict: typing.Optional\[bool\] = None ) → `transformers.models.clap.modeling_clap.ClapAudioModelOutput` or `tuple(torch.FloatTensor)`

The [ClapAudioModelWithProjection](/docs/transformers/v4.30.0/en/model_doc/clap#transformers.ClapAudioModelWithProjection) forward method, overrides the `__call__` special method.

Although the recipe for forward pass needs to be defined within this function, one should call the `Module` instance afterwards instead of this since the former takes care of running the pre and post processing steps while the latter silently ignores them.

Examples:

```
>>> from datasets import load_dataset
>>> from transformers import ClapAudioModelWithProjection, ClapProcessor

>>> model = ClapAudioModelWithProjection.from_pretrained("laion/clap-htsat-fused")
>>> processor = ClapProcessor.from_pretrained("laion/clap-htsat-fused")

>>> dataset = load_dataset("ashraq/esc50")
>>> audio_sample = dataset["train"]["audio"][0]["array"]

>>> inputs = processor(audios=audio_sample, return_tensors="pt")
>>> outputs = model(**inputs)
>>> audio_embeds = outputs.audio_embeds```