auriankelen/matpac_general_audio

MATPAC / MATPAC++

MATPAC (Masked latent Prediction And Classification) is a self-supervised audio and music encoder. This repository is a transformers-native, trust_remote_code port of the official inference code (aurianworld/matpac, folder inference_matpac).

• 📝 MATPAC: Masked Latent Prediction and Classification for Self-Supervised Audio Representation Learning
• 📝 MATPAC++: Enhanced Masked Latent Prediction for Self-Supervised Audio Representation Learning

The encoder is a ViT-Base transformer (12 layers, 768-dim) operating on log-mel spectrograms of raw, mono, 16kHz audio.

The MATPAC family

This repo ships MATPAC++ pre-trained on general audio. This is the default, general-purpose MATPAC model for audio representation. The other checkpoints from the MATPAC++ release each get their own repo, all sharing the exact same configuration_matpac.py / modeling_matpac.py, just with a different config.json:

Repo	Trained on	Output	Notes
matpac_general_audio (this repo)	General audio (AudioSet)	3840-d embeddings	Default general-purpose audio encoder
matpac_music	Music	3840-d embeddings	Music-specialized encoder
matpac_audioset_finetune_encoder	AudioSet (fine-tuned)	3840-d embeddings	Fine-tuned encoder, no classification head
matpac_audioset_finetune_classifier	AudioSet (fine-tuned)	527-way logits	Ready-to-use AudioSet tagger

To use the downstream-task linear probes, download the weights from the original repo (Weights_probes_matpac.zip), and attach them to this encoder with the conversion script. See Downstream task probes below for details.

Usage

from transformers import AutoModel, Wav2Vec2FeatureExtractor
import torch
import torchaudio

model = AutoModel.from_pretrained("auriankelen/matpac_general_audio", trust_remote_code=True)
processor = Wav2Vec2FeatureExtractor.from_pretrained("auriankelen/matpac_general_audio", trust_remote_code=True)

waveform, sr = torchaudio.load("my_file.wav")
waveform = waveform.mean(dim=0)  # mono

resample_rate = processor.sampling_rate
if resample_rate != sr:
    waveform = torchaudio.transforms.Resample(sr, resample_rate)(waveform)

inputs = processor(waveform, sampling_rate=resample_rate, return_tensors="pt")

with torch.no_grad():
    outputs = model(**inputs)

# outputs.last_hidden_state: (batch, 3840) -- mean-pooled over time by default
# outputs.hidden_states: tuple of 12 tensors, one per encoder layer, each (batch, 3840)
all_layer_hidden_states = torch.stack(outputs.hidden_states)
print(all_layer_hidden_states.shape)  # [12, batch, 3840]

Keeping the time dimension / faster inference

# Keep the time axis instead of mean-pooling it
outputs = model(**inputs, pull_time_dimension=False)
# outputs.last_hidden_state: (batch, time, 3840)

# Vectorized inference: faster on large batches, adds some padding so it is
# slightly less precise than the default ("precise", used for the paper's results)
outputs = model(**inputs, inference_type="fast")

Downstream task probes

Get the probe weights straight from here Probes_weights and merge them into a loadable HF folder with convert_matpac_checkpoint.py:

python convert_matpac_checkpoint.py \
    --checkpoint_path matpac_plus_6s_2048_enconly.pt \
    --probe_checkpoint_path matpac++_general_audio/fsd50k.pth \
    --class_mappings_dir matpac/inference_matpac/matpac/class_mappings \
    --output_dir ./matpac-general-audio-fsd50k

outputs.logits then holds the task's logits and model.config.id2label the corresponding labels, exactly as for the AudioSet head above.

Requirements

This model needs a couple of packages beyond transformers to run its trust_remote_code files: torchaudio, einops, and timm==0.4.12 (the exact timm version the checkpoints were exported with — the internal naming of the ViT block submodules must match for the weights to load correctly).

Citation

@inproceedings{quelennec2025matpac,
  title={Masked Latent Prediction and Classification for Self-Supervised Audio Representation Learning},
  author={Quelennec, Aurian and Chouteau, Pierre and Peeters, Geoffroy and Essid, Slim},
  booktitle={ICASSP 2025 - 2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)},
  year={2025},
  url={https://ieeexplore.ieee.org/document/10887666},
  doi={10.1109/ICASSP49660.2025.10887666}
}

@article{quelennec2025matpacenhancedmaskedlatent,
  title={MATPAC++: Enhanced Masked Latent Prediction for Self-Supervised Audio Representation Learning},
  author={Aurian Quelennec and Pierre Chouteau and Geoffroy Peeters and Slim Essid},
  journal={arXiv preprint arXiv:2508.12709},
  year={2025},
  url={https://arxiv.org/abs/2508.12709}
}

Credits

• aurianworld/matpac for the original training and inference code.
• Fairseq for the training framework.
• M2D for the base of the inference code.
• DINO for the classification head.