Unmute the Patch Tokens: Rethinking Probing in Multi-Label Audio Classification
Authors: Lukas Rauch
René Heinrich
Houtan Ghaffari
Lukas Miklautz
Ilyass Moummad
Bernhard Sick
Christoph Scholz
Venue: ICLR 2026 Poster (2026)
Keywords: audio self-supervised learning
probing
frozen embeddings
bioacoustics
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: This paper investigates the poor performance of probing in multi-label audio, attributing it to a pooling bottleneck rather than deficient features.

Abstract: Although probing frozen models has become a standard evaluation paradigm, self-supervised learning in audio defaults to fine-tuning {when pursuing state-of-the-art on AudioSet}. A key reason is that global pooling creates an information bottleneck causing linear probes to misrepresent the embedding quality: The $\texttt{cls}$-token discards crucial token information about dispersed, localized events in audio. This weakness is rooted in the mismatch between the pretraining objective (globally) and the downstream task (localized). Across a comprehensive benchmark of 13 datasets and 6 spectrogram-based encoders, we investigate the global pooling bottleneck. We introduce binarized prototypical probes: a lightweight and simple pooling method that learns prototypes to perform class-wise information aggregation. Despite its simplicity, our method notably outperforms linear and attentive probing. Our work establishes probing as a competitive and efficient paradigm for evaluating audio SSL models, challenging the reliance on costly fine-tuning.

Published as a conference paper at ICLR 2026
UNMUTE THE PATCH TOKENS: RETHINKING PROBING IN MULTI-LABEL AUDIO CLASSIFICATION
Lukas Rauch 11
René Heinrich Houtan Ghaffari" Lukas Miklautz Ilyass Moummad
Bernhard Sick Christoph Scholz *IrauchBuni -aasell.dd-dddedeee
University Kassel Fraunhofer IEE
'Ghent University ML. Systems Biology, MPI Biochemistry PINRIA Monipellier
ABSTRACT
Although probing frozen models has become a standard evaluation paradigm, self- supervised learning in audio defaults fine-tuning when pursuing state-of-the-art on AudioSet. A key reason is that global pooling creates an information bottle- neck causing linear probes to misrepresent the embedding quality: The cls-token discards crucial token information about dispersed, localized events in audio. This weakness rooted in the mismatch between the pretraining objective (globally) and the downstream task (localized). Across comprehensive benchmark of 13 datasets and 6 spectrogram- based encoders, we investigate the global pooling bot- tleneck. We introduce binarized prototypical probes: lightweight and simple pooling method that learns prototypes to perform class-wise information aggrega- tion. Despite its simplicity, our method notably outperforms linear and attentive probing. Our work establishes probing as a competitive and efficient paradigm for evaluating audio SSL models, challenging the reliance costly fine munitu.
(a) Pretrained Token Map
(b) Pretrnined [c]a]
Sinn
EXTT1
Anention
Aftection Map
(c) Probing prosabin descriptor
Probing protobin descriptor
Music
The pooling bottleneck. Visualizing embeddings from purely self-supervise model (EAT) and its supervised** adapted version (EAT*) for a spectrogram from urban. (a) A PCA of the token map shows that EAT embeddings are rich but entangled, result of the masked prediction ob- jective, while EAT* embeddings are localized and aligned with input events.
Figure 1:
(b) The [cls] -token's attention starts similarly for both models, but is diffuse for EAT in later layers, while EAT* becomes spatially selective, highlighting its limitation as a probe vector. (c) Our protobin disentangles these correlated EAT embeddings recover localized event information. (d) For the EAT* model, protobin further enhances the embeddings, providing a superior representation to the
INTRODUCTION
Self-supervised learning (SSL) promises general-purpose embeddings that transfer across down- stream tasks (Oquab al., 2024). key advantage their out-of-the-box utility: instead of compute- and label-intensive fine-tuning, one can freeze the pretrained backbone and train only a lightweight probe. As an evaluation paradigm, probing offers a faithful and efficient assessment of pretrained embeddings by minimizing the confounding factors of fine-tuning (Chen et al., 2020; Rauch al., 2025a). Fine-tuning often yields stronger downstream performance (Park et al., 2023), but can obscure the intrinsic quality of the frozen embeddings (Kumar et al., 2022). While probing an established evaluation paradigm and is also utilized in audio SSL (Nizuumi al., 2024; Yadav et al., 2024) on benchmarks such as HEAR (Turian et al., 2022), the pursuit of state-of-the-ar (SOTA) performance on AudioSet (Gem- meke et al., 2017) still defaults issensce-ntensive fine-tuning (Alex et al., 2025). This discrepancy
computer vision (Oquab et al., 2024: Darcet et al., 2025)

---

Published as a conference paper at ICLR 2026
motivates central question: why does this influential benchmark still lack lightweight probing method that reliably reflects a model's performance as an alternative to fine-tuning?
The performance of frozen probe depends on the interplay between the pretraining objective (i.e., the pretext task) and the pooling method (i.c., embedding extraction). Poor probing performance for masked image modeling -MIM))pretrriine models is direct result of the pooling method, as the [cls] -token distributes attention too uniformly instead of focusing on key information (Przewięž- likowski al., 2025; Alkin al., 2025). The superior performance of probes that utilize the full token map (Psomas et al., 2025) creates critical deficit for simpler methods, rendering them un- reliable proxies for an encoder's embedding quality. This motivates the need for probes that can efficiently leverage all available information to provide a faithful assessment, avoiding the cost and confounding factors of fine-tuning- Many spectrogram- based audio SSL. encoders that report SOTA performance on AudioSet via fine-tuning apply MIM-style objectives, often coupled with student- teacher distillation (Chen et al., 2024; Alex al., 2025; Ahmed et al., 2024; Chen et al., 2023a; Li et al., 2024). By design, this task induces bias toward contextualized token-level embeddings, posing any probe's 's limitations that collapse the tokens into a simple global summary. While attentive pooling, which learns weightee summary, has emerged potential solution in computer vision (Przewigžlikowski et al., 2025), its application to audio remains a research gap, particularly for representing complex polyphonic scenes.
In addition, the downstream task plays role in the performance of probes (Alex et al., 2025). Polyphonic soundscapes are multi-label, with sparse and localized evidence for sound events in the time-frequency domain. Forcing this information into a single descriptor, whether fixed or learnable during probing, creates single-vector bottleneck: Quieter but important events could be overshad- owed by more prevalent sounds, making it difficult for a linear classifier to disentangle the mixed signals (see Figure 1). Therefore, the limited adoption of probing and its failure approach fine- tuning SOTA performance on AudioSet likely reflects a pooling mismatch, not an absence of usable information. While the pretrained [cls] -token struggles to summarize these sparse events and can underperform in audio classification (Alex et al., 2025; Li et al., 2024), fine-tuning implicitly learns a superior, class-conditioned aggregation over the full token map (see Figure 1).
Hypothesis: Pooling Bottleneck
twning:
40
The limited usage of probing as an eval- uation tool for multi-label audio SSL stems from the pooling method. Stan-
8..5
mAP [%)
A14.09
31.67 67
222.91
26.11
dard single-vector probes, from the off- the-shelf [cls] -token to attentive pool-
20
protobin
17.29
ing, underutilize token embeddings. A more valuable and reliable probe re- quires a shift to per-class, multi-vector aggregation fully exploit the informa- tion in the patch tokens (Figure 2).
10
linear
[cla]
takens
Figure 2: Probing on as20k with EAT.
Contributions
1. Audio probing benchmark. We conduct an extensive benchmark to systematically inves- tigate the pooling bottleneck in audio SSL. Our analysis establishes probing hierarchy, demonstrates that the cls-token probe and fine-tuning can be unfaithful evaluators of au- dio SSL models, quantifies the impact of polyphony in probing, and shows that supervised adaption after pre-training alters cls-token's quality and model rankings. We empirically show that the bottleneck stems from the pooling method, not the embeddings, challenging the validity of current evaluation practices in achieving SOTA performance AudioSet.
2. Elevating probing in audio. Prototype methods notably outperform other pooling methods, including linear and attentive. This result challenges the reliance on costly fine-tuning and establishes probing competitive and efficient paradigm for evaluating audio SSL models.
3. Binarized prototypical probes. We introduce an efficient probe that addresses the pooling bottleneck by performing class-wise and multi-vector information aggregation on the tokens. We simplify prior prototypical approaches by decoupling prototypes from class constraints and eliminating an explicit orthogonality loss, while achieving competitive performance.
2

---

Published as a conference paper at ICLR 2026
2 PROBING FROZEN EMBEDDINGS IN MULTI-LABEL AUDIO
This section formally introduces the probing task for (multi-label) audio, provides a taxonomy of the pooling methods, and introduces our binarized prototypical probes.
2.1 PROBLEM FORMULATION AND NOTATION
We consider a multi-label classification task with training dataset D {( y:)) where each input 2; belongs set of spectrograms X' C TxF with T time frames and F frequency bins. Each corresponding one-hot-encoded target vector y; E {0, 1) indicates the presence or absence of C possible classes. Multiple classes may simultaneously occur for single input. Additionally, we assume access to a pretrained embedding encoder fe. parameterized by frozen weights 0. This model Jo functions feature extractor, mapping an input 2; token map:
fz(z.)
Fole]
24 fo(;;) E DxS.xS
$1
attenionn
where the embedding dimension, and S,, Sy index grid of time and frequency patch tokens. If the backbone exposes a [cls] -token, we denote it by sel E RD, For instance, 10-second log- spectrogram with F=128 Mel bins (from 16 kHz audio) is patched into non-overlapping 16 x 16 time-frequeet frequeccy tokens, yielding T Az 1024 frames and thus 5,=64 and Sy=8. With an embedding dimension of D=768, the resulting token map is z, IR 768x6xx Given the frozen token map z,, probe Is consumes a pooled descriptor 2, determining how information is extracted. The resulting vector then processed by the probe, typically linear classifier go (Z:)
Wz; + b.
2.2 A TAXONOMY OF GLOBAL POOLING METHODS
This section provides brief taxonomy of pooling methods to contextualize our investigation.
Fixed global pooling (single-vector, non-learnable). The default approach collapses the token map 2; from the frozen backbone fe into single descriptor 2, = A(z,) E TR. via a non-learnable aggregator xx + IR D followed by linear probe. the model exposes a last-layer [cls]-token se produced via self-attention, one can 2; = عا While mean pooling all tokens Z; is a viable alternative, all encoders in our benchmark provide a cls-token, making it our standard for fixed global pooling. A k-NN probe is also used in multi-class settings, but vanilla k-NN performs single-label majority voting and is ill-suited to multi-label.
Learnable global pooling (single-vector, learnable). Instead of fixed pretext-task descriptor, this pooling family uses a learnable module to aggregate the token map into a single deseriptor 2, while keeping fo frozen. Attentive variants assign data-dependent weights tokens and form a weighted summary. They typically outperform fixed global pooling for pretrained encoders in computer vision (El-Nouby et al., 2024; Darcet al., 2025).
2.3 LEARNABLE PROTOTYPICAL POOLING: A PER- CLASS POOLING METHOD
As an alternative to single-vector pooling, prototypical probes aggregate evidence per class via mul- tiple learnable exemplars (i.e., prototypes). Inspired by explainability methods (Chen et al., 2019; Donnelly et al., 2022), the idea is score the frozen token map by its similarity to learnable proto- types in the embedding space, which naturally accommodates dispersed events by allowing different classes localize information in distinct fime-frequency regions (Rauch et al., 2025a).
Binarized prototypical probes. We introduce binarized prototypical probes, a novel and efficient instance from the prototypical pooling family (Rauch et al., 2025a; Heinrich et al., 2025) that scores token map embeddings by matching them against a small set of prototypes that are binarized on-the- fly. We maintain set of C J total learnable, class-agnostic prototypes, with parameters P. RD
3

---

Published as a conference paper at ICLR 2026
prototype vectoos By
binarized protocypesP)
similarity activations
deseriptors, 8.
embedding map 24
boward
Sr
sinclla Byy
pooling
beckward
S (768, 64, 8)
STE
liner clessifier
(J, 768, I)
(J,64, 8)
Figure 3: Binarized prototypical pooling (schematic). Example shown for a base audio SSL back- bone with D-768-dim tokens and 64x8 token map. There are J learnable prototypes, which are binarized on-the-fly. Tokens are matched against these prototypes, max pooling aggregates spatial evidence, and final linear layer maps the resulting prototype scores to class logits.
for each prototype index (1,.... . CJ). At each forward pass, form the binary prototype
Pj = sign(p)) E (-1,+1)
This constraint helps encouraging large angular margins between distinct prototypes. The near- orthogonality is an emergent property, forcing prototypes to the corners of a high-dimensional hy- percube, creating strong structural bias for diversity. The optimization process seeks discriminative features to minimize classification loss and is incentivized to select orthogonal solutions. The non- differentiability of sign(-) is handled with the arriiht-thhouuhe www.ttttroosstteeee through-estimator (STE) (Bengio et 201.,011)) during propaaaopa,,,onn sign(a))
1, so the forward pass uses hard #1 while gradients flow to the real- valued p,. Given the frozen token map z, fe(z;) E DD let t.f RR denote the token at time-frree menefruuynncy index (t, f) E (I....,$i) S.} x (l..... Sy). We score each prototype against all tokens using cosine similarity and aggregate evidence via max-pooling:
DJ
f)(t,f) = 112l12
33 max
Stacking the pooled scores across all J prototypes yields the vector s; E R We use this vector as the clip-level descriptor, i.e., set 2, S;, and map it to class logits with the linear classifier gø-
Rationale. A prototype layer is parameter-effiiient, re- quiring only J D) parameters. The value for J is set by multiplying the number of classes C' by a small con- stant (e.g., 20 (Rauch et al., 2025a)), offering compact alternative to attentive pooling heads that can require over 2D parameters (El-Nouby et al., 2024). By binarizing the prototypes to Pj E (-1, +1} our method yields an additional 32x memory reduction relative to 32-bit floats, making it ideal for centare-onstrained and on-device ap- plications (c.g., bioacoustics). Cosine matching inherently keeps scores scale- and iimeension-ivariian across back- bones. The near-orthogonality observed between proto- types (cf. Figure 4) is not enforced by an explicit mech- anism but is an emergent property arising from the method. Through the binarization, we also constrain them to the corners of D-dimensional hypercube, creating structural bias for diver- sity. During training, the optimization process seeks a set of maximally discriminative prototypes effectively classify different audio events. The optimization process, seeking to minimize clas- sification loss, is incentivized to select distinct. non-redundant prototypes. In the high- nigh-dimensional embedding space, this is most effectively achieved when prototypes are nearly orthogonal. There- fore, we simplify the training objective by eliminating the need for an explicit orthogonality loss term used in prior work to enforce diversity (Rauch et al., 2025a; Heinrich et al., 2025). Finally, unlike prior work, we make the prototypes class-agnostic, allowing the prototypes to better collabo- rate in disentangling task information. The link between these diverse prototypes and their specific class contributions is learned entirely by the linear classifier. This layer learns to map the similarity scores from the prototypes to the C' class logits, effectively assigning semantic meaning to each prototype based on its utility for the classification task.
Weights per protoype
street music
jackhanmmr
stre
gunshot
enginic
driling 3
day bark
children playing
GB hort
air
50
Dequeency
0.4
similanity prototypes
Figure 4: Weights and similarities ample. Trained protobin on urban.
4

---

Published as a conference paper at ICLR 2026
3 RELATED WORK
SSL paradigms in audio. In vision, two families dominate modern SSL: student-teacher clus- tering/distillation (Caron et al., 2021; 2020) and MIM (He et al., 2022; Darcet et al., 2025). Hy- brids combining global invariance with masking considered the current best-performing mod- els (Oquab et al., 2024; Assran et al., 2023). Spectrogram- baasd audio SSL largely adapts these paradigms: vision transformer (ViT) backbones trained via masked-spectrogram prediction or stu- dent-teacher paradigms with audio-specific augmentations (see Table 1). Audio-MAB (Huang al.. 2022) and Dasheng (Dinkel al., 2024) are generative masked reconstruction models (He et al., 2022). BEATs (Chen et al., 2023a) follows BEiT-style masked token prediction with discrete acoustic tokenizers (Bao et al., 2022). ASIT (Ahmed et al., 2024), EAT (Chen et al., 2024), and SSLAM (Alex et al., 2025) use momentum-teanne teacherr distillation with masked/local-gloo ance- objectives (Caron al., 2021; Baevski et al., 2022). Except for Dasheng (which uses additional datasets), these models pretrain AudioSet's S as2m (Gemmeke et al., 2017), establishing an influential line of work where SOTA is measured mostly by fine-tuning performance.
or utter-
Table 1: Spectrogrammbased backbones used in our work. Mask: input masking during pretrain- ing. EMA: student-teacher with EMA teacher. Global [cls]: explicit global/token objective during pretraining. Supervised* have an additional fine-tuned checkpoint on as2m available.
Model
Year
Paradigm
Supervised*
Mask
EMA
Global [cis]
Pretrain data
A-MAE
2022
Masked spec reconstruction
x
X
882n
BEATs
2022
Musked token prediction
x
X
882n
ASIT
2024
Masked spec reconstruction latent distillation
X
as2n
EAT
2024
Masked latent distillation
/
V
as2n
Dusheng
2024
Masked spec reconstruction
x
X
"
SSLAM
2025
Masked latent distillation mixtures
882n
Evaluation in audio SSL. Simple linear probes are widely used in computer vision (Oquab al.,
2024) and utilized by numerous audio SSL works (Niizumi et al., 2022; 2024; 2025; Yadav et al., 2024: Li al., 2024; Pepino et al., 2025) on benchmarks such as HEAR (Turian al., 2022) with a simple probing toolkit. However, these evaluations in audio SSL have largely treated probing as a fixed protocol. With the notable exception of a token reshaping approach from Niizumi et al. (2022), the impact of the pooling method and the underlying performance bottleneck it creates, has remained largely unexplored. When pursuing SOTA performance on AudioSet, audio SSL still defaults to fine-tuning (Huang et al., 2022; Chen et al., 2023a; Ahmed et al., 2024; Chen et al., 2024; Alex et al., 2025). We attribute this reliance on fine-tuning, further justified by the sentiment that linear probes cannot fully reflect embedding quality (Li et al., 2024), match: pretraining learns token-level information, yet standard probes compress the tokens into a single global vector, discarding per-source cues critical for polyphony and localized events. A-MAE yields weak linear prohe utility in bioacoustics (Rauch et al., 2025b), consistent with findings that generative objectives disperse information across tokens (Park al., 2023; Alkin et al., 2025). This limitation becomes evident in the line of work pursuing SOTA on AudioSet: Although masked- distillation models (BEATs, EAT, ASiT) are designed to produce stronger global representation in the [cls] -tosen their performance with mrozen-backbor work. SSLAM includes linear probing results on selected datasets, yet cross-backbone compara- bility is limited (Alex et al., 2025). Dasheng reports frozen MI.P and k-NN results on multi-class tasks on HEAR but does not address multi-labe settings. This gap motivates a systematic study of probing methods for frozen audio embeddings.
pretext- peatexttpoooing mis-
probing is rarely reported in related
Advanced probe architectures. Replacing fixed global pooling with learned pooling over token maps during probing improves alignment with MIM (Darcet et al., 2025). Attentive pooling consis- tently outperforms fixed global pooling (Psomas et al., 2025; El-Nouby et al., 2024; Darcet et al., 2025). Complementary analyses show that [cls] attention of backbones tends to be diffuse under MIM, weakening it as a global descriptor (PrzewiçZlikowski et al., 2025). Some works in audio have explored structured, non-attentive pooling. For instance, Niizumi et al. (2022) utilize the token map by concatenating frequency features at each time step before temporal pooling (linpre). Attentive pooling methods compute token weights and values differently, ranging from single-query multiple- instance learning (abmilp) (IIse et al., 2018) and multi-head cross-attes cooss-tttetiinn (mhca) (El-Nouby et al., 2024; Chen et al., 2023b; Bardes al., 2024) data-dependent single-head (simpool) (Psomas
5

---

Published as a conference paper at ICLR 2026
et al., 2023) and efficient multi-query (ep) approaches (Psomas et al., 2025). Other work in audio explores learnable per-class prototypes for probing (Rauch et al., 2025a), matched to multi-label audio where classes localize in distinct time-frequency regions. Real-valued prototype probes show promising results in bioacoustics (Rauch et al., 2025a). Token-aware attention and prototype de- signs better align with MIM embeddings and polyphonic labels than single-vects vector summaries, yet evaluations in audio SSL remain sparse. This further motivates our comprehensive analysis.
Positioning of this work. Prototype layers originated in vision for interpretability (Chen et al., 2019; Donnelly et al., 2022) and were adapted to bioacoustics (Heinrich et al., 2025). Closest to our setting, Rauch et al. (2025a) apply prototypical probing over spectrogram tokens for a domain- specific MAE in bird sound classification. We extend this line of research with a binarized STE variant that constrains prototypes to the hypereube, yielding strong compression and margin-like regularization. Additionally, our work introduces two key architectural simplifications. First, we decouple prototypes from classes, allowing class-agnostic features to emerge automatically via the final linear layer. Second, we find that the supervised learning signal is sufficient for prototype diver- sity in this context, eliminating the need for an explicit orthogonality loss term (Rauch et al., 2025a). Our variant of these simplified prototypes remains highly competitive while offering 32x memory reduction, consistent with successes of discrete parameterizations (Courbariaux et al., 2015; Hubara al., 2016). Beyond these method-level contributions, our work establishes prototypical probing as a general evaluation paradigm for the audio SSL field and delivers the first extensive probing benchmark. This study adapts recent attentive methods from vision to serve as strong baselines and ultimately reveals clear hierarchy. While learned pooling is broadly advantageous, mirroring trends in vision (Psomas et al., 2025; Darcet et al., 2025), prototypical methods consistently set the SOTA results for probing in audio SSL, providing a competitive alternative to fine-tuning.
4 EXPERIMENTAL STUDY: A BENCHMARK ON PROBING IN AUDIO
This section first outlines experimental setup, including backbones, pooling methods, datasets and the evaluation protocol of the benchmark. is followed by our main results organized as focused questions with short rationales.
4.1 EXPERIMENTAL SETUP AND EVALUATION PROTOCOL
Backbones. We evaluate six state-of-the-as art frozen spectrogram based SSL encoders fa. summa- rized in Table 1. To ensure a fair comparison, we only use the ViT-base checkpoints with an em- bedding dimension of 768 and circa 86M parameters since this is the only configuration offered across all models. We also include to pretraining. Such variants exist for EAT, BEATs, and SSLAM. Reporting results for the purely self-supervised and the supervised* versions allows us quantify how supervised adaptation to the AudioSet label space affects the quality of frozen embeddings (see Figure 1).
that were fine-tuned on as2m in addition
Datasets. We organize the benchmark into three topical groups. The primary group, general multi- label audio, contains the smaller, balanced AudioSet subset as20k (Gemmeke et al., 2017) and fsd50k (Fonseca al., 2022), a curated dataset aligned with the AudioSet ontology. Following Alex et al. (2025), we also include the polyphonic datasets desed (domestic sound events with 10 la- bels) (Johnson et al., 2021), spass (urban soundscapes with 28 labels (Viveros-Muñoz al., 2023)), and urban (urban soundscapes with 10 labels (Salamon et al., 2017)). The second group focuses on fine-grained multi-label bioacoustics, for which we use seven subsets from the birdset bench- mark (Rauch et al., 2025b). These tasks test the models' generalization under domain shift and a data-efficient, 64-shot few-shot learning protocol. The third group provides multi-class datasets using the esc50 and sc-2 datasets. These single-label tasks serve as a control condition to isolate the impact of polyphony and determine whether the pooling bottleneck unique to the multi-label audio setting. Appendix D.I provides detailed description of each dataset.
Pooling methods. We compare eleven pooling methods (cf. Section 3) that operate on frozen en- coders. Each technique produces descriptor ž that is passed through linear classification layer. Linear and mlp consume only the fixed global [cls]-token -token as a compact summary of the input. Linearc, conv the token map without attention. The former concatenates all tokens form the descriptor. The latter applies a lightweight convolution for local aggregation. Linpre also utilizes the token map by concatenating frequency features at each time step before temporal pooling. Atten-

---

Published as a conference paper at ICLR 2026
tive pooling of the token map include abmilp, simpool. ep, and mhca (see Section 3). Prototypical pooling for class-conde conditiond descriptors includes the class-dependent proto (Rauch et al., 2025a) and our class-agnostic protobin (see Section 2.3). Refer to Appendix D.3 for a detailed overview.
Caching and probing. For each input z; in a dataset, we run an augmentation free forward pass through the encoder fo and cache embeddings from the final hidden block. It contains the full token map 2 E IR Dy and the fixed global descriptor Z E RD given by the last-layer's [cls]- token sçLo, This produces static on-disk embedding store per backbone that we use as input to all probes. Caching avoids repeated model inference and isolates embedding quality at the cost of a less diverse training distribution with on on-the-fly data augmentations. We accept this trade-off to preserve computational efficiency as one of the central advantages of probing.
Training setup. All probes are trained for 30 epochs with AdamW, a cosine-annealed learning rate scheduler (Loshchilov & Hutter, 2017), batch size of 128, and the asymmetric multi-label loss (Ridnik et al., 2021). This setup ensured convergence in preliminary studies across probing methods. We apply the default settings to all pooling methods. For prototypical pooling methods, the prototype learning rate equals the global learning rate, and the number of prototypes J is fixed at 20 per class across datasets (10 for as20k), following Rauch et al. (2025b). While this fixed value ensures a fair comparison across pooling methods without confounding factors, we provide a sensitivity analysis in Appendix B which confirms that J=20 prototypes is robust choice for our benchmark. For future work, we hypothesize that J could be tuned for specific applications based on factors such as the intra-class diversity and the degree of polyphony in given dataset.
Hyperparameter selection, To keep comparisons fair, we optimize only two scalars: learning rate and weight decay. For each dataset, we select hyperparameters on a validation split and report final results on the held-out test split. If a validation split is unavailable, we reserve 20% of the training set. For datasets with F-fold vross-aali,,tio,, we designate one fold a priori for hyperparameter search. We use a two-stage procedure per (backbone, dataset, probe) probe)-comblel >combinnation. First, we run 50 trials with a fixed seed for comparability, using Sobol (Sobol, 1998) exploration for the first 25 trials and TPE for the remainder (Bergstra et al., 201 1), under a successive-halving schedule. All other training details are held constant across probes and backbones. Second, we take the top- k configurations and re-evaluate each with five random seeds to estimate the mean and standard deviation of the validation set's mean average precision (mAP). We then choose the configuration with the highest mean mAP, retrain it with this setting, and evaluate the test set. Appendix D.4 provides more details on hyperparameters.
4.2 EXPERIMENTAL RESULTS
To investigate the pooling bottleneck hypothesis, we conduct an extensive benchmark with focused questions. Our primary analysis evaluates all ten pooling methods across five general multi-label datasets using six encoders and their three ses, we use a representative subset of the most informative probes to test our hypothesis on seven fine-grained, few-shot bioacoustic datasets and two multi-class control tasks. Throughout iments, we report mean average precision (mAP) for multi-label tasks and accuracy for multi-class tasks. The complete results, which form the basis for all visualizations, are detailed in Appendix A.
addppte versions. For more targeted analy-
exper-
(Q1) Takeaway, Across backbones and datasets, the pairwise win matrices in Figure 5, supported by the absolute results in Table 2, show a stable and strong hierarchy. Our protobin wins most often, with an average improvement of + +14.41 %, MAP over linear on general audio and + 1 %op mAP on few-shot bioacoustics. Attentive pooling, with mhca as the besf-performing attentive method, and the simple, reshaping- limpre improve over fixed global descriptors but still lag behind prototypes (-4.59 %p mAP) on general audio despite its complexity. Simple baselines are at the bottom, including [cls]-token probes and naive token concatenation (linearc). This clear ordering provides strong support for our pooling bottleneck hypothesis: global single-vector probes severely underutilize the rich information in the token map, and a reliable evaluation of current MIMM-asee audio SSL models benefits from shift to per- class, multi-vector aggregation. Finally, our results reveal a trade-off between the float-based, class- dependent proto and our class-agnostic protobin. The full precision and class dependency of
(Q1) Pooling hierarchy: Is there best- performing pooling method?
Rationale: clear hierarchy with proto- types outperforming single-vector methods would support our hypothesis that probing benefits from multi-vector aggregation.
7

---

Published as a conference paper at ICLR 2026
proto appears advantageous in specific cases where capturing fine- line-grained details is critical (e.g. polyphonic urban or with the ASIT backbone). Protobin's simplification makes it a more robust choice for ceneral-purpore evaluation. To help disentangle these architectural factors from the effect of binarization, we provide an ablation with a float-based, class-agnostic variant in Appendix B.
(a) General multi-label audio
(b) mulishhl multi-label bioacoustic
protabin
35
prosobin
170
289
25
50
xhca
196
Pooling Method
proto
154
25
Pooling Mellod
4p
conv
146
wwitss
ahca
110
simpool
100
62
10
sbmilp
74
xlp
linearc
56
linear
profobia
proto
alica
up
crary
Iapoos
elp
ab
linous
Lisour
protabin
proto
abca
w/pp
linear
Opponent
Pooling Metbod
Opponent Pooling Method
Figure 5: Pairwise win matrices for pooling methods. Each cell shows the number of configura- tions where method outperforms another (ties omitted, one sd above opponent), aggregated over all datasets and base (non-supervised") backbones. Extracted from Table and Table 5 (Appendix D.2).
Table 2: Probing benchmark results in general audio. All results are the mean with std reported in mAP, averaged over five seeds. Best and second best probe per (dataset, backbone) are highlighted.
Input
[cle]
Baseline
Kap
Map
Hap
Backbone
Linesr
nip
Lineare
Limpre
op
simpoo1
stailp
proto
prosobia
9.80
11.87
14.80.aa
21.61a2a
22.3246a
18.35.4.
18.16.0.3
18.72-a.2
18:01.40
16.10
20.08
21.09103
18.57
BEATs
24.71
26
15.70
12.80
18.50asa
21.86
20.81a
14.99-m
12.52a
30.54
31
EAT
17.20
20.504wa
21.4
19.50.ms
28. 28.40asa
26.11-a4
18.83ame
25.15
19.91as.4
31.0
31.67
SSLAM
17.04-4a
19.99
20.51.9.1
17.45am5
24.81454
24.05-a.2
25.49460
22.59-a.1
18.91-44
30.54
30.94463
A-MAE
19.71
21.34
25.
20.
38.08
45.17-ma
34
32,73
49.65
49
ASIT
39.57
41 40
9.87
25
39.57
42.28
16.6
37.78-9.1
39.
46.70-6.
Dasheng
38.08
39.56
57.71
08.88ins
45.11
52.95
43.91
43.79 3.5
55.20
57.31
BEATs
46.89
19.58.0.
36.35
17.191116
39.00
00.95..1 i.s
48.51
06. 16.1
40.20.oa
60.52
57.17
58 27.2
EAT
36.39
$1.8 41.821oa
38 38..66...3
66.6Lin5
48.
18.21.15.
51.06....
211.291m
19.38
45, L5.95.4.
35.64..4
SSLAM
36.06.9.
01, 11.264n.
37.
30..
46.
16.11
51,48.9.5
50 83
19 19.86..2
46 38.3881444
56
56.97110
A-MAE
57.0 22.46.9.
60. 12.
60.88 .0.1
81. 10.03
71.26.14
83.57
50 13
72.05
76. 19..9
84.1
85 85.57
ASIT
72.02
79.19
57. 37.48.01
81 61.581o3
74.91
79 00.50
76 661n
73.57
76 58.88
$1.74
Beaed
Dasheng
68 06.39um0
68. 76am
72-48
82.52211
74.4
84.53 53
82 74an
75-10lan
76 48448
85
86
BEATs
77.56
80 6
72 22
85.83
76.97
86
88
81 48
81. 87.77a1
SUM
89 49.222m
ENT
26.15
N0. m
77. 10
81 1..00
06.00
84 13am
83 K3
78 70
88 N2
SSLAM
72.19.00
77. 17.96113
76.82 ...2
50.55103
80.51 13.
85.41...1
83.77
83.59..m
$1.69
67.00
88.33110
A-MAR
38.34ge
60. D
00.01
No 014an
77. 08
29.24
71 a1
60 84ga
68 7ham
78 2X
ASIT
08 68.80.4.
70.22 27 10.5
46. 46.40.44
73.26 26:11
73.88 aa.
75 70.
88. 11.0
60.01.aa
68.36
73
73.06:22
Dasheng
06.89 .8.
61.07 10.
76..55
75.051n7
72.07
80.71 a.3
73.621no
76.16.an
72.02 cc
76.64:22
BEATs
74.22-a.
70. 76.97
79 01.01.98
81 81.81:15
82 12.12
85 83.98
76 61
76.58
60 38.38
85.77
EAT
65.06
71 11.55an
84. $4.40lan
79 15inc
81 8las
85 85.9
TT 15.55
76 76.55
64 64.4
83.00 $3.08
SSLAM
68
75.05ane
83.06
79-4321
80.74
83 83
76 88gn
76
72.42a14
85.5 $5.90
86.01
A-MAB
58. 38.72
58 98.97ana
40 40.53
85, 28
70.01
82.48
78 K3an
76.21aa1
73.07as
85.17
ASIT
77.53
77.65an
44 5dax
82 12
70.32204
70.90
78 48am
77 77.2
76.76214 4
82 150
82.22 22.28
Dusheng
20.
68.07am
86.76an
77.00ga4
84.50
22.31
70004
77.28a-
85.97
86. Shann
BEATs
82.54
S0.76ame
75.90
85-57
81.61
86.20
84.31
82 74
77.80 a
80 044g
88. 88.74
EAT
27.76
$1.58am1
78 48
86.35e11
S4.04.a
86.40
40
83 38.
79.50
89.11
ND 24
SSLAM
25.86
50.64ama
77.97
86.25
79.01
86.45_9.4
$4.87
83.21
80.12s.a
88.50
89 000.06a
linear
protobin
(2)) [cls] quality, Is the Vinear probe a faithful evalua- tor?
70.71
702
66.71
54
1122
644
55.43
$5
$2.79
Rationale. We test if the off-the- shelf linear probe is a reliable and faithful proxy for embedding quality in audio SSL. A flawed proxy both underestimates the ab- solute potential the embeddings and distorts the relative ranking of different backbones.
50
40.64
40
BEATs 2023
ASIT
EAT
SSLAM
A-MAE 2022
2024
Backbone averages. Mean performance across general audio datasets for linear and protobin. Publica- tion years highlight how probing re-ranks models
Figure 6:
8

---

Published as a conference paper at ICLR 2026
(Q2) Takeaway. Probing the [cls] -token with linear is not just performance bottleneck, it is also an unreliable proxy for pretrained embedding quality in audio SSL. First, Figure 6 shows that the backbone ranking under linear is completely reshuffled when using protobin. For instance, the backhone ranking is completely inverted: ASIT' (from 2024), which appears to he the second- strongest model under linear, drops last place when evaluated with protobin. Conversely, the supposedly mediocre SSLAM (current fine-tuning SOTA from 2025), mid-tier performer with linear, revealed to be top-tier model, jumping second place. This demonstrates that the [cls] ttkeen is a poor indicator of the model's true token-level embedding quality. Figure 7 con- firms this is a systemic issue: linear/mlp act a performance ceiling, and the gains unlocked by token-aware pooling methods vary by backbone. Second, the [cls] underestimates the true potential of the embeddings. On as20k, protobin closes 63% of the performance gap fine- tuning (see Figure 2), demonstrating how much information standard probes discard. TThis trend holds across all encoders (Table 3), establishing that better pooling provides a more faithful measure of embeddings.
[clo] Baselines
Mop
Token Map (ailention)
Token Map (prolulypes)
- Thear x3p
linearc
Veippol
shailp
protobin
A(linerr) %op
20
10
0
-10
A-MAE ASTT
BEATs
Dasheng EAT
SSLAM
Figure 7:
Performance differences of probes across backhones. For each backbone, the plot displays the mean and standard deviation of each pooling method as absolute percentage points [7)p] compared the baseline performance of linear, All results are extracted from Table
(Qs) Takeaway. On single-label control tasks (sc-2, esc50), substantial performance gap persists be- tween the [cls]-token probe and token-aware meth- ods, indicating the bottleneck a general issue of the encoders (Table 3). However, the [cls]-probe's per- formance degrades more sharply than other methods when moving the multi-label as20k. In this single- label setting, mhca is often competitive with, or even superior to, our protobin probe. This suggests that well-learned single-vector descriptor can be as effective as our multi-vector approach for single- source audio. This dynamic changes in the presence of multiple sound sources, confirming our core hypothesis. The constant superiority of protobin the fundamental limitation of single-vector methods in polyphonic scenes. Methods like mh ca must compress localized evidence for multiple distinct events into single vector. In contrast, vector prototypical approach can activate different specialized prototypes for different sound events within the same audio clip. The discriminative nature of the prototypes is particularly effective disentangling these overlapping audio events.
(Qa) Multi- vs. single-label. Is the pool- ing bottleneck specific to multi-label?
Rationale. If fixed global pooling de- grades from single- to multi-label while token-aware methods remain stable, would implicate polyphony-inducee bottleneck.
the multi-label as20k task highlights
multi-
Table 3: Multi- vs. single-label pooling and fine-tuning, Accuracy on sc-2 and esc50 (single- label) and mAP on as20k (multi-label). denotes the reported fine-tuning performance in the respective backbone paper, bold marks the best probe per backbone and dataset.
sc sc-2 (single-label)
esc50 (single-label)
as20k (multi-label)
Backbone
linear
mbca
protobin
linear
nhca
protobin
linear
nhea
protobin
A-MAE
12.4
84,9
79.5
98.3
22.1
86.3
83.7
94.1
8.4
17.1
22.3
37.1
ASIT
62.2
86.3
89.5
76.1
78.3
21.0
38.6
BEATs
87.0
95.0
96.5
98.3
78.9
83.2
84.1
95.6
24.7
21.9
31.5
38.9
EAT
69.1
93.2
90.4
98.3
75.3
89.8
86.8
95.9
17.3
26.1
31.7
40.2
SSLAM
64.8
93.8
74.2
89.0
24.4
30.9
40.9

---

Published as a conference paper at ICLR 2026
(Q+) Takeaway. On in-domain, general audio tasks (Figure 8a), the probe rankings change notably. The [clss-token-busel methods see the largest gains, with linear jumping from rank #10 to #6 and mlp from #7 to #3. This confirms that supervised fine-tuning injects class-specific information into the global token. Mean- while, attentive pooling methods are stable, and the prototypical methods retain their top-ranked positions. In contrast, on out-of-domain bioacoustics tasks (Fig- ure 8b), the complete hierarchy remains stable. De- spite a minor performance uplift across the board (see Appendix D.2), the overall ranking is preserved: linear remains at the bottom while protobin stays at the top. This divergence demonstrates that supervised" primarily strengthens the single- vector [cls] descriptor for in in-domain tasks but fails to add transferable, token-level information for out-of-domain tasks. The consistent superiority of prototypical methods in both settings further aggregation.
(Q+) Supervised" weights. Does extra fine-tuning enrich the loken map?
Rationale. Supervised adaptation in- jects class information into the [cls]- token. This lets us test if the model has learned richer token-level information just stronger global descriptor. A lo- calized improvement, where gains are specific to [cls] probes and in-domain data, would suggest the latter.
highlights the robustness of per-class,
(a) General multi-lahel audio
(b) Few-shot multi-label bioacoustic
protobis
protobiz
proto
#1
#5
proto
up
#70+
#4
comT
45
+3
nlick
simpool
nlp
+9
0.7
Model Pype
op
+5
Model base espervical"
lipears
#8
#6
base rupervisod*
m1p
45
Stmilye100
1imoar
#10
-006
linear
86
20
80
100
120
winss
wins
Figure 8: Pairwise win ranking changes from base to supervised models. We display the num- ber of pairwise wins averaged over the backbones with fine-tuned variants (BEAT', EAT, SSLAM) and datasets for each pooling method. Extracted from Table 2 and Table 5 (Appendix D.2).
Pooling bottleneck. Our findings confirm our hypothesis and its implications for probing as a re- liable evaluation tool. The [cls] -token is a performance bottleneck, underutilizing the token map and leading an unreliable evaluation (Q2). While attentive pooling offers improvements, our re- sults show multi-vector, per-class aggregation is more robust strategy, particularly in polyphonic scenes where single-vector methods are limiting (Q1, Q3). This conclusion holds even when the (Q+). Thus, the primary obstacle to using probing as
[cls] -token is enhanced by supervised* evaluation tool is not the quality of the embeddings, but the limitation of the pooling method.
5 CONCLUSION AND FUTURE WORK
Conclusion. We demonstrated that the underperformance of probing in (multi-label) audio stems not from the frozen embeddings themselves, but from information bottleneck in pooling meth- ods. Single-vector representations, whether from a fixed ill-suited for polyphonic audio, they compress sparse, localized events into single descriptor. Addressing this, we introduced binarized prototypical probes, a lightweight method that performs per-class aggregation directly on the token map. Our comprehensive benchmark shows this approach consistently outperforms single-vector probes and notably narrows the gap to fine-tuning. By en- abling cladscconiiiional vectors with a minimal memory footprint, this work establishes prototypical probing as a viable, efficient, and faithful evaluation paradigm for audio SSL. This challenges the default reliance on costly and confounding fine-tuning when pursuing SOTA on AudioSet.
or learned via attention, are
Future Work. A next step is to move beyond the final encoder layer and explore multi-layer feature aggregation, which could unlock even richer embeddings. Furthermore, our coken-aware probing framework could be extended from clip-level classification to more granular tasks such as event detection and localization, where
benefits of multi-vector aggregation may be even stronger. While our study focused on audio, the insights into pooling bottlenecks likely apply to other domains as well. Future work could also explore integrating on-the-fly -lyy data augmentations with a frozen backbone to push the performance ceiling of the probing paradigm even higher.
10

---

Published as a conference paper at ICLR 2026
AUTHOR CONTRIBUTIONS
Lukas Rauch: Conceptualization (Lead), Methodology (Lead), Software (Lead), Investigation
(Lead), Writing Original Draft (Lead), Validation (Lead), Visualization (Lead).
Rene Heinrich: Conceptualization (Supporting), Validation (Support), Writing- - Fntting-Review & Editing.
Houtan Ghaffari: Validation (Support), Visualization (Support), Writing Review & Editing.
Lukas Miklautz: Conceptualization (Supporting), Writing Review Editing.
Ilyass Moummad: Conceptualization (Supporting), Writing Review & Editing.
Bernhard Sick: Funding Acquisition, Resources.
Christoph Scholz: Conceptualization (Supporting), Funding Acquisition, Writing Review & Edit- ing.
ETHICS STATEMENT
Our research is conducted exclusively on established, publicly available datasets intended for aca- demic audio and bioacoustics research. Our focus probing evaluation method promotes computational efficiency, significantly reducing the energy consumption and environmental impact compared to full model fine-tuning. The methods developed are for the purpose of model analysis and present no foreseeable societal risks or ethical concerns.
REPRODUCIBILITY STATEMENT
To ensure full reproducibility, we make our source code, including the implementation of our pro- posed prototypical probe and all evaluation scripts, publicly available on GitHuu. To further aid reproducibility and standardize access, we have also uploaded any datasets used this study that were not not previously available the Hugging Face Hub to the platform.
http:://gtaab.oo///rraaaaa//mmep
Our experimental setup, including the specific datasets Section D.I, pretrained backbones, and pool- ing methods Section D.3, is detailed in Section 4.1 and in Appendix D. Appendix D.4 also provides a complete breakdown of our hyperparameter selection protocol with the respective ranges.
USE OF LARGE LANGUAGE MODELS
LLM was utilized as writing and coding assistant during the preparation of this paper. The model was used to aid in literature discovery by summarizing concepts and identifying potentially relevant papers for the authors' review. Additionally, the LLM served as writing aid to refine gram- mar, improve sentence structure, and enhance the overall clarity and readability of the paper (c.g., shorten paragraph). was also used for streamlining code, debugging, and generating shell scripts to help manage the experimental workflow. All research ideas, including experimental design, code implementations, and analysis of results stem from the authors without LLM involvement. The au- thors directed all queries, critically reviewed and carefully edited all model-generee text, and take full responsibility for the final content of this paper.
ACKNOWLEDGEMENTS
This research was conducted under the DeepBirdDetect project (FKZ 67KI3, funded by the German Federal Ministry for the Environment, Nature Conservation, Nuclear Safety and Consumer
11

---

Published as a conference paper at ICLR 2026
Protection (BMUV), and the BioDroneAl project (FKZ 02WDG1758D), funded by the German Federal Ministry of Research, Technology and Space (BMFTR)
REFERENCES
Sara Atito Ali Ahmed, Muhammad Awais, Wenwu Wang, Mark D. Plumbley, and Josef Kittler. ASIT: Local-Glotte Audio Spectrogram Vision Transformer for Event Classification. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2024.
Tony Alex, Sara Ahmed, Armin Mustafa, Muhammad Awais, and Philip JB Jackson. SSLAM: Enhancing Self-Supervised Models With Audio Mixtures For Polyphonic Soundscapes. In Inter- national Conference on Learning Representations (ICLR), 2025.
Benedikt Alkin, Lukas Miklautz, Sepp Hochreiter, and Johannes Brandstetter. Mim-refiner: A con- trastive learning boost from intermediate pre-trained representations. In International Conference on Learning Representations (ICLR), 2025.
Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas. Self-supervised learning from images with joint-embedding predictive architecture. In IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2023.
Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, and Michael Auli. data2vec: A general framework for self-supervised learning in speech, vision and language. In International Conference on Machine Learning (ICML), 2022.
Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. Beit: Bert pre-training of image transformers. 0rXiv:2106.08254 2022. URL https: //axivv..iiii////aa org//oo/////i/ae/ abs///0b/2 http//11..0000 08254..
Adrien Bardes, Quentin Garrido, Jean Ponce, Michael Rabbat, Yann LeCun, Mahmoud Assran, and Nicolas Ballas. Revisiting feature prediction for learning visual representations from video. In arXiv preprint arXiv:2404.08477 2024.
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation. arXiv:1308.3432, 2013.
James Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl. Algorithms for typer-parameter optimization. In Advances in Neural Information Processing Systems (NeurIPS), 2011.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Un- supervised learning of visual features by contrasting cluster assignments. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In IEEE/CVF Inter- national Conference on Computer Vision (ICCV), 2021.
Chaofan Chen, Oscar Li, Daniel Tao, Alina Barnett, Cynthia Rudin, and Jonathan K Su. This looks like that: deep learning for interpretable image recognition. Advances Neural Information Processing Systems (NeurIPS), 2019.
Sanyuan Chen, Yu Wu. Chengyi Wang, Shujie Liu, Daniel Tompkins. Zhuo Chen, Wanxiang Che, Xiangzhan Yu, and Furu Wei. Beats: Audio pre-training with acoustic tokenizers. In International Conference on Machine Learning (ICML), 2023a.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning (ICML). 2020.
Wenxi Chen, Yuzhe Liang, Ziyang Ma, Zhisheng Zheng, and Xie Chen. Eat: Self-supervised pre- training with efficient audio transformer. In International Joint Conference on Artificial Intelli- gence (ICCAI), 2024.
12

---

Published as a conference paper at ICLR 2026
Xiaokang Chen, Mingyu Ding, Xiaodi Wang, Ying Xin, Shentong Mo, Yunhao Wang, Shumin Han, Ping Wang, Gang Zeng, and Jingdong Wang. Context autoencoder for self-supervised represen- tation learning. International Journal of Computer Vision, 2023b.
Matthieu Courbariaux, Yoshua Bengio, and ral Networks with binary weights during propagations. In Advances in Neural Information Pro- cessing Systems (NeurIPS), 2015.
David, BinaryConnech Training Deep Neu-
Timothée Darcet, Federico Baldassarre, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Clus- ter and Predict Latent Patches for Improved Masked Image Modeling. TMLR, 2025.
Heinrich Dinkel, Zhiyong Yan, Yongqing Wang, Junbo Zhang, Yujun Wang, and Bin Wang. Scaling up masked audio encoder learning for general audio classification. arXiv:2406.06992, 2024.
Jon Donnelly, Alina Jade Barnett, and Chaofan Chen. Deformable protopnet: An interpretable image classifier using deformable prototypes. In IEEF/CVE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
Alaaeldin El-Nouby, Michal Klein, Shuangfei Zhai, Miguel Angel Bautista, Vaishaal Shankar, Alexander Toshev, Joshua M Susskind, and Armand Joulin. Scalable pre-training of large au- toregressive image models. In International Conference on Machine Learning (ICML), 2024.
Eduardo Fonseca, Xavier Favory, Jordi Pons, Frederic Font, and Xavier Serra. FSDS0K: An Open Dataset of Human-Labeled Sound Events. arXiv:2010.00475, 2022. doi: 10.48550/arXi.20110. 00475.
Jort F. Gemmeke, Daniel P. W. Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R. Channing Moore, Manoj Plakal, and Marvin Ritter. Audio Set: An ontology and human-labeled dataset for audio events. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2017.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollar, and Ross Girshick. Masked autoencoders are scalable vision learners. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR),
René Heinrich, Lukas Rauch, Bernhard Sick, and Christoph Scholz. Audioprotopnet: inter- pretable deep learning model for bird sound classification. Ecological Informatics, 2025.
Po- Po-Yao Huang, Hu Xu, Juncheng Li, Alexei Baevski, Michael Auli, Wojeiech Galuba, Florian Metze, and Christoph Feichtenhofer. Masked autoencoders that listen. Advances in Neural Infor- mation Processing Systems (NeurIPS), 2022.
Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio. Binarized Neural Networks. In Advances Neural Information Processing Systems(NeurlPS), 2016.
Maximilian Ilse, Jakub M. Tomezak, and Max Welling, Attention-based deep multiple instance learning. International Conference on Machine Learning (ICML), 2018.
David S. Johnson, Wolfgang Lorenz, Michael Taenzer, Stylianos Mimilakis, Sascha Grollmisch, Jakob Abeßer, and Hanna Lukashevich. DESED-FL and URBAN-FL: Federated Learning Datasets for Sound Event Detection. arXiv:2102.08883. 2021.
Ananya Kumar, Aditi Raghunathan, Robbie Jones, Tengyu Ma, and Percy Liang, Fine-tuning can distort pretrained features and underperform out-of-distrrbuute arXiv:2202.10054, 2022. URL https: /ttpss///axxv..eee wrg. abs/ 2202.2222.1110 10054.
Xian Li. Nian Shao, and Xiaofei Li. Self-supervised audio teacher-student transformer for both clip-level and frame-level tasks. IEEE/ACM Transactions on Audio, Speech, and Language Pro- cessing, 2024.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. International Confer- on Learning Representations (ICLR), 2017.
13

---

Published as a conference paper at ICLR 2026
Daisuke Nizuumi, Daiki Takeuchi, Yasunori Ohishi, Noboru Harada, and Kunio Kashino. Masked spectrogram modeling using masked autoencoders for learning general general-purpose purpose audio represen- tation, 2022.
Daisuke Niizumi, Daiki Takeuchi, Yasunori Ohishi, Noboru Harada, and Kunio Kashino. Byol for Exploring pre-trained Audio, Speech, and Language Processing, 2024.
audio:
audio representations. IEEE/ACM Transactions on
Daisuke Nizuumi, Daiki Takeuchi, Masahiro Yasuda, Binh Thien Nguyen, Yasunori Ohishi, and Noboru Harada. M2D-CLAP: Exploring general-purpose audio-language representations beyond CLAP. IEEE Access, 2025.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. TMLR, 2024.
Namuk Park, Wonjae Kim, Byeongho Heo, Taekyung Kim, and Sangdoo Yun. What do self- supervised vision transformers learn? In International Conference on Learning Representations (ICLR), 2023. URL https:://penrree
Leonardo Pepino, Pablo Riera, and Luciana Ferrer. EnCodecMAE:: leveraging neural codecs for universal audio representation learning. In Interspeech 2025, 2025.
Karol J. Piczak. Esc: Dataset for environmental sound classification. In ACM International Confer- ence on Multimedia (MM), 2015.
Marcin Przewięšlikowski, Randall Balestriero, Wojeiech Jasinski, Marek Smieja, and Bartosz Zieliński. Beyond [CLS]: Exploring the true potential of masked image modeling representa- tions. arXiv preprint arXhv:2412.03215, 2025. URL htthstt::/ //aaaiv//////a/aa/aaa www.v.ogg/abba http:/bb//22...22/222
Bill Psomas, Ioannis Kakogeorgiou, Konstantinos Karantzalos, and Yannis Avrithis. Keep it sim- pool: Who said supervised transformers suffer from attention deficit? IEEE/CVF International Conference Computer Vision (ICCV), 2023.
Bill Psomas, Dionysis Christopoulos, Eirini Baltzi, Ioannis Kakogeorgiou, Tilemachos Aravanis, Nikos Komodakis, Konstantinos Karantzalos, Yannis Avrithis, and Giorgos Tolias. Attention, please! revisiting attentive probing for masked image modeling. arXiv:2506. 10178, 2025.
Lukas Rauch, René Heinrich, Ilyass Moummad, Alexis Joly, Bernhard Sick, and Christoph Scholz. Can masked autoencoders also listen to birds? 2504v:2504. 12880, 2025a.
Lukas Rauch, Raphael Schwinger, Moritz Wirth, René Heinrich, Denis Huseljic, Marek Herde, Jonas Lange, Stefan Kahl, Bernhard Sick, Sven Tomforde. and Christoph Scholz. BirdSet: A Large-Scale Dataset for Audio Classification Avian Bioacoustics. In International Conference Learning Representations (ICLR), 2025b.
Tal Ridnik, Emanuel Ben-Baruch, Nadav Zamir, Asaf Noy. Itamar Friedman, Matan Protter, and Lihi Zelnik-Manor. Asymmetric loss for multi-label classification. In IEEE/CVF International Conference on Computer Vision (ICCV), 2021.
Justin Salamon, Duncan MacConnell, Mark Cartwright, Peter Li, and Juan Pablo Bello. Scaper: A library for soundscape synthesis and augmentation. In 2017 IEEE Workshop Applications of Signal Processing to Audio and Acoustics (WASPAA), 2017.
Ilya M Sobol. On quasi-Monte Carlo integrations. Math. Comput. Simul., 1998.
Joseph Turian, Jordie Shier, Humair Raj Khan, Bhiksha Raj. Björn W. Schuller, Christian J. Stein- metz, Colin Malloy, George Tzanetakis, Gissel Velarde, Kirk McNally, Max Henry, Nicolas Pinto, Camille Noufi, Christian Clough, Dorien Herremans, Eduardo Fonseca, Jesse Engel, Justin Sala- mon, Philippe Esling, Pranay Manocha, Shinji Watanabe, Zeyu Jin, and Yonatan Bisk. Hear: Holistic evaluation of audio representations. arXiv:2203.03002. 2022.
14

---

Published as a conference paper at ICLR 2026
Rhoddy Viveros Munos,, Pablo Huijse, Victor Vargas, Diego Espejo. Victor Poblete, Jorge P. Arenas, Matthieu Vernier, Diego Vergara, and Enrique Suárez. The SPASS dataset: A new synthetic polyphonic dataset with spatiotemporal labels of sound sources. Applied Acoustics, 2023.
Pete Warden. Speech Commands: A Dataset for Limited-Vocabulle Speech Recognition.
arXiv:1804.03209, 2018.
Sarthak Yadav, Sergios Theodoridis, Lars Kai Hansen, and Zheng-Hua Tan. Masked autoencoders with window local-global attention are better audio learners. In International Conference on Learning Representations (ICLR), 2024.
15

---

Published as a conference paper at ICLR 2026
A DETAILED BENCHMARK RESULTS
This appendix provides supplementary material to the benchmark evaluation presented in the main paper. Our full benchmark spans general multi-label datasets, 7 few-shot bioacoustic datasets, and 2 multi-class control tasks across 6 backbones (plus 3 supervised* checkpoints) and 10 pooling methods. The following tables present the complete results, with all performance metries reported as mean average precision (mAP) averaged for multi-label and accuracy for multi-class tasks over 5 random seeds.
Table 4: Complete benchmark probing results for general multi-label audio. This table presents the full benchmark results, extending those in the main paper with the inclusion of Supervised* fine-tuned checkpoints for BEATs,, EAT, and SSLAM. All results are the mean mAP with standard deviation, averaged over seeds. The [best] and second-best performing probes for each configura- tion are highlighted.
Inpur
[cla]
Baseline
Token
Kap
Token
Map (Att.))
Token Map (Proto.)
Backbone
Tiear
nlp
linearc
conv
sp
simpoo1
apmil
proto
protobia
A-MAE
8.36e0.01
8.77c0.29
9.66=0:22
11.87+1.1
17 17.09
17.0340.00
14.60
14.24a0,sa
21.61-0.20
ASIT
18. 18.35e001
19.16
13.3620.2
13.80sc.
18. 18.727717
18.9540.0r
18.01a0.00
16. 10
800
20 20.96-00.0 02
Dasheng
20 06
21.08a0.0
07
18 11
18.57s1.0 os
27. 292000777
23.5330.00
20.80
22, 1.09
27 77.59.0000
29 09.31.5
a220k
BEATs
26.29 24.712001
15.70
12.80 os
21 16
20.81 as
14,90.0.0
12.5221.00
30.54:0.00
[3 1 05
BEATS
40.301002
40.77
15.3330015
34.201s.17
17
37.57
37.2310.3 3o
27.38
30.49...m6
[2 12.73
41 41.96 0.00
EAT EAT+
17. 21
20.30+0.10
21.91a0.01
19.50sc.44 84
26.11=m.10
26 26.8340.04
25.15+0.04
19.91=3.40
31.06-0.00
17.00
44.320002
+0.30 us
37.44amm.0
41.851c.15
41.87=02s
42.53lac.
41.666.0.0
39.14:3.42
43.36
44.64-u.02
SSLAM
17.042001
19.90
17.4636.54 20.51
24.452001
25.49.a..1
22.50
18:91:4.4
20.84-0000
30
SSLAM+
15. 15.7220002
10.00000000
10
37.61an.22
47.77
13. 10:000
11.11=0.0r
13.37e0.00
11.31=0.70
11.0-=0000
13,
33.70=0.0
A-MAE
19.71
21.341x..4
25.17
40.591c.7s
45.17
43.23110.44
34.80.0.a5
32.73....1
49.65 .0.7
69 Lo.as
ASIT
19.57 21.80.0..s
9.87
78.20....7
42.28
41.761c.:1
37.7810.6
39.59.:1.00
I5 25 L.our
46.70 0.38
Dasheng
38 17
39.56a0.1
(37.7420.51
48.88 38.8886.9 79
52.95e0.19
52, 62.4442..01 04
43.94a0. 04
43.70.a..0
55.23=0.00 oe
57.8 11.00000
trd50x
BEATs
46 80
40.58
36.35
37 19 as
48.51:2009
46.1044..r or
40.20
40.32:9.m2
67.17=0.14
68
27
BEATS+
60:721000
61.87.0.11
50.17.0.4
56.321c-4s
60:01.0000
m0 55.97.11.0
48.30..0.
53.88,93.7
65.30 .0.08
EAT
36. 1000
14.82,10.
38.3610.20
48.6411..4
51.06Lne0
51.291c..0
49.3810..7
45.90
56.077 1o0
55.64 0.37
EAT+
66.11s
[67
56.50
67.0110:21
64.37an26
65.0116.07
63.632..04
61.45:00.47
67.15
66.45
SSLAM
36, 066000
24.26
37,21an.4
43.
51.48:0a1
301883ac.00
49.881a0.21
46.38-22.44
56 56.93=uu.5
56.90 pouu
SSLAM+
06
55.12s0.31
65
61.28=0.66
61.5844.00
63.31=0.07
61.88=a0.88
66.55=0022
66.02-0.29
A-MAE
07. 1G
60.52e0.13
60.88en.14
81. 10 81
83.57=020
80.13s0.00
72.05en.0a
76.69...2r
$4.11=0.09
55.57 Ho..0
ASIT
72.92
24.19.0.a0
57.49.0.30
11.50cc.s
79.50.u44
76.66.c.u
73.57a9.uu
76.58..0.4
81.74
Dasheng
68. 00
68.76.0.14
22.48
85.32
84.53:031
82,74ac.uu
75.40..0.0
78.48:44.4
56
Breed
deced
BEATs
77. 77.56snus
80.56.20.1
72.23amm1
86.01:m0s
81.88ac.04
81.0820.0
81.77au.0
89.04-w.os
59
souss
BEATS+
87.20
87.92=0.02
86.93en.22
90.31ss. 14
90,22-005
87.91s0.2 28
85.52=0.54
86.33
22.17.0
10025
EAT
76. 15
80.92+0.02
77.90sn.04
86 68
86.06=0.19
81,13+0.0s
83.13=0.01
78, 78.80=56.6
88,70
88.70=0.00
15% 87 11
EAT+
89. 49
89.8220.0 ne
89.0320.004
90-49:005
80. 20.226a0.o on
80.03.0.0
no
88 88.9720.18
50 1e 16
91.69=0..x
SSLAM
72. 40
77.9620.14
76.8220.17
86 20.56a6.mm
85.44:0.10
83.77a6.0
83.50.0..0
81.0020.74
87.6 -0.0
[88
SSLAM+
80.30
80.6010.07
88.04_0.ca
91.1016.77
90.141mms
80.111600
88.88100.8
88.43..0.61
Looas
91.45 w.2
A-MAB
58. 0s
60.56.e.11
69.01
79.21=0.14
71.01+0.8
69.8110.02
68, 75.0.30
28.9220..4
ASIT
68, 80
20
16. 474.4477
73 10
70)e0.4
69.41s0.0
02
69.01s0.0
68, 3..366e0..6
73. 73.06:0.00
74.69=0.s
Dashang
66, 00
64.07
76.7Gso.0
76.05as.02
80.71
73.62a0.0
74.1620.0l
72.02a0.00
76.64-0.22
BEATs REATS
71.2220011
75.97
79.91
84.811149 49
83.982016
76.6140.00
0s
75.58
00 00.38:0..4
[
85.77-0.18
08.461002
80.2410.15
80.3010.ce
85.5211.9
84.8410m1
79.3916.m
76.64100
74.28.0..4
150115
07.85_0020
EAT EAT+
65. 05.961m01
11.55.0..1
84.49
15.15116.3
83.951n42
77.3516.0
76.55
64.44_..04
85.64 $3.00_
79.20sm01
80.85
87.08
88.31:1.22
86.052mm11
80.43ac.u
79.76
79.91.0.20
55 $0000
58.48
SSLAM
68. 28
73.05a0.00
83.06an.20
79.43as..2
83.45
76.58ac.0z
78.09
72.42:1.31
85.90em41
56 co.11
SSLAM+
78. 78.8220000
80.67
80.83en.02
87.
37.5660.03
85,97=000
79.63e0.02
79.01s0.2
79.00ie0.06
88.17=0.00
A-MAI
88.721008
58.97.10.19
40.53
85
28
82.49
79.8
76.21
73.07.2.46
83.63
85.17 0.32
ASTT
17.53g001
77.55
44.53...92
82.1218.53
79.93
78.4816.44
77.2510.0
78.76.1..M
10:21
82.28
Dasheng
60.61ggn:0
69.07.0.37
75.80am.3
85.76ac.59
84.59.u.s
82.31as.0
79.0420.00
77.28.0.m
55,97
10.31
56.55
pours
urban
BEATs BEATs+
82 04
83, 33.7622.00
75.0020.ce
86.0740 4s
86-23=004
84.3140.1
82.7420.01
77.80.1..07
88,
-8.74=011
87.70e0:01
87.79
83.81e4.1
89.15eec.1
89.022e0.0
87.7240.21
80.51=0.50
81.21=1.11
91.12-0.0
EAT EAT+
77.761001
81.58 1o.00
78.45.0.c8
86.3511.11
96.431u03
85.401c.::
83.5810.0
79.90_j2.0
19.11_n.32
89 24
88. 43e001
88.56
87.25
90.6410.35
89.23::015
88.33asau
87.80lao.s
87.28:20.a
91.31 3137
SSLAM
75. 865n02
80.642...0
77.97
86, 23
86,45:000
84,87ac.0
83,2120.44
80.12:1..a
88.82
50 p0.00
SSLAM+
88. 10
88.21=0.02
80.52en.c2
90.3846.05
88.84=020
87,8
87.19+0.18
86.07=0.34
[
2 1e
07
16

---

Published as a conference paper at ICLR 2026
Table 5: Benchmark probing results for few-shot multi-label bioacoustics. This table presents the benchmark results where the figures in the main text extracted from. All results are the mean mAP with standard deviation, averaged over seeds. The best and second- -bett performing probes for each configuration highlighted.
[cla] Basal ima
Token Map (Att.)
Taken (Proto.)
Backhone
11near
nlp
nhca
ap
proto
protobin
A-MAF
4.83_002
28.32
15.62 15.62
34.55
ASIT
5.51=1.45
9.47:10.0
6.11:0.36
12.24
BEATs
10.29:0.04
10.52=1.0r
21.86ie2.00
52.01
BEAT
12.70=0.18
12.922=0.77 72
20. 07:1444
12 2.0
28.40
han
Dasheng
8.03=0.19
6. 6.65
201. 78
17. 17.56i4..4
22.02=n.00
EAT
10.42=0.c0
EAT+
18.4320.40
18.80:20:0 55
22. 40 ss
17. a.0.1
12 216.422.:::.
25.97=1.11 1.11
SSLAM
8.14_001
8.42 Lo07
22. 78
14.98.0.7
21.85
25
SSLAM+
19.31am.48
21.05
22. 2.00
19.86-ac.1a
[50.00
29.85
A-MAE
11.01:n.06
9.56=m.44
25.
22.35iss..s
50,08=1..2
31.70
ASIT
10.74=0.00
10.00
13. 16:e0.02
10.52+4.19
14.1
BEATs
16.9620.06
16.32::0
22. 1:l.
17.43si..0
30.48 18.8%.1.7
15.67.10.0
21.98.0.
15.68
pox
Dasheng
13.3120.06
12.06 20
17. 20
15. 15.02a0.00
19.69
19.42=0.4
EAT
14,6010.0
14.04-10.4
21.02
18.39
24.84.1.31
2N. 22N.77
EAT+
17.28
19.60
29. 2..24
18.7 18.77
33 $1.98 2.031
30.26 11.36
SSLAM
10.63en.00
11.35:0044
22. 01100
16. 10.99es..1
22. 0:799
26.59=1.. 11.33
SSLAM+
16.15.00az
17.30
16.63
11.53
A-MAB
4.01=0001
3. 8.78
9. 0.06ee0.5
9.50=0.00
16 4..e
15.00
2.01
5.76
4.94
7.08.0.17
7.57
BEATs
6.00_0003
m.74
9.58
7.72.0.24
L5.16
14.40)
REATs+
6.50
6.01
10.6420.45
7.00
pez
Dasheng
5.61_.00
4.80
8.35
1..
10.20 10.00
Lo.37
EAT
4.92=001
1.98
8.67:0.44
8.34
12. 12.88
EAT+
6.61=0000
7.06=a.s
10.6224078
8.39 0..4
1.88.0022
SSLAM
4.68=0002
4 4.54 10
10:31
8.57
11 1.97
8 1.18
SSLAM+
6.69=0000
7.41=0.01
+0334
14:20:-0.50
A-MAE
3.05_0000
3.25
18.5210.46
16.641c.11
25.80.00.0
25.88 10.15
9,25
110.22
REATs
9.00_006
9.52
16.38.10.1
11.2 11.22
25,07 10.35
BEATS
11.43a0.6
11.85
16.91
12.07
24.47 10.35
Dasheng
5.64=0004
1.21=017
12. 19
22.08
17.22=0.12
700010
EAT
7,88=0000
8.78
16. 77:00.0.00
13. 13.799
21. 18
013
13.49.2.ce
14.06.0
17.71
13:67 os
23.45
SSLAM
5.66=001
6. 6.73=0:26
17. 8.822...0
14 11.9840022
21.63e0.4
SSLAM+
12.8220.1
14.08
18.
13.65..0.0
A-MAE
6.09=004
5.90=n.12
17. 17.56:00.04
13. 48.....
20.23=0.91
21 3133e110
ASIT
6.41_013
6.92
7.12.10.41
6.79..0m1
0.63 Lo.32
1
L0.35
BEATs
9.61=005
9.95
19. 18.91.0.a
12. 12.16:20.2
20.20
19.46=1.50
BEATs+
11.18a0.06
12.05
12.31
10.6140.52
16.83=0.71
256
Dasheng
8.72-000
7:30
12. 6000000..0
11.58a0.1
15.68.0.35 27.07
EAT
10.2910.00
10.67 .o.or
14. 14.58.1.7
11.89
16.48 10.30
16.70 L.-53
EAT+
9.63=001
10.04:2.44
12.
9.68=0017
15.40=0..4
SSLAM
9.52.005
9.96
13.42.0.5e
11.0410.00
16.25
16.40:00
SSLAM+
10.23:0.0
10.34.10.4
14.02.0.44
10.9110.34
17.64]
16,87
A-MAE
4.88=1000
4 8.
10. 70 15
8.42=0.44 24
12.17-0.3 77
22.67
ASIT
5.44m002
6.65em.e
6.67
6.01=0.37
6.12s0.30
BEATs
9.74=000
10.74.00.3
10.21
7.93 72
Teo
12.03
REATS
8.95
10.10.:
13.47.0.0
9.71
15.02.9.07
ubh
Dasheng
7.78
6.57.u.57
EAT
9.32=007
9.0G
10. 10.98
9.55
8.07 2.0770.77
1.344
EAT+
10.12.0.ca
10.32
10.68.0.75
9.95.0.54
10.00
33007 L.
SSLAM
7.36=002
10.600
8.61=0.25
11:45 [14430000 Boods
20.50
SSLAM+
10.30e0.18
11.20
9.78
9.07=0.06
13337 T6
12.46=0.41
A-MAE
9.82=0.1
8 20
32. 32.02..00
26 09.8994..14
10.07=0.00
47
ASiT'
12.44:0.29
19
16. 16.333ao.0
14.90a0.3
19.77
20.57 8:008
17.7210.36
28. 1m.1mm
17.4416.s5
BEATs+
21.4120.43
22.60
22.660:20
32. 22.58.
22. 22.92
12 8..844
42, 20:1..0
abp
Dashong
33.933e.00
32.12=0050
EAT
14.59.0.11
16.95.u.51
26. 18..8111.7
22. 14.4....
34.30
34. a.
EAT+
23.91s0.20
25.16=1.00
32.51=2.22
24.4140.5:
[10.7
39, 40.99
SSLAM
10.41=0.07
14.58
27.31
20.6240.15
34.30=1.17
SSLAM
20.84:0.37
21.00
30.32.10.0
21.7911011
120 N0..000
37.44 41.4411.0
17

---

Published as a conference paper at ICLR 2026
Table 6: Benchmark probing results for general multi-class audio. All results are the mean accuracy with standard deviation, averaged over 5 seeds. The best and second-best performing probes for each configuration are highlighted.
Backbone
linear
nhca
protobin
A-MAB
22.08:10.14
4:0.00
83.67 1.42
ASIT
76.08-00.76
78.25 4-0.0
79
+1.84
BEATS
78.92400229
83.17++9.76
84.08 11
e5c50
BEATs
94.03+0.14
94.25+10.48
94. 00.20
Dasheng
1..7511.75
990.17 1:0.2
86.3310.63
EAT
75.33110.95
89. 00.833:.0
19.83:111.44
1:1.44
86.83:10.35
EAT+
96.67
40.29
96.674-0.29
96.67
SSLAM
74.17:0.14
69.00
84.67+0.38
SSLAM+
[97.17|40.14
97.17440.14
96.75-40.50
A-MAB
12.4411.67
84.87 1:1.1
79.47:11.00
ASIT
62.23:10:22
86. 36.26:1-0113
89
0.558
BEATS
87.00+0.118
94. 440.95
96. 633.53.
+.111
BEATs+
85.7940.27
93. 60-40.77
94.72
00.53
Dusheng
78.57
18.1334-0.28
[08.40| +.
69.1440.14
0.113
90.44+2.00
EAT+
83.27:0.02
94.4240.200
95.76
SSLAM
64.75::0.20
93 76 10.62
91.86:10776
SSLAM+
83.57:10.03
91.21
0.333
B ABLATION STUDY
We conduct ablation study investigate two key aspects of our prototypical probes. First, analyze the sensitivity to the number of prototypes (J) per class to justify our choice in the main benchmark. Second, we aim disentangle the performance effects of our two main contributions: the architectural simplifications aclass-a design, no orthogonality loss) and the binarization itself. To achieve this, compare three methods:
1. proto: The baseline from Rauch et al. (2025a) using float-based, class-dependent proto- types with orthogonality loss.
2. protobin: Our proposed method using binarized, class-agnostic prototypes without an orthogonality loss.
3. protofloat: A new ablation variant that uses protobin's simplified, class-agnostic archi- tecture but with float-based prototypes. This allows us to isolate the impact of binarization.
The results across three diverse datasets (multi-label, high number of classes: as20k, multi-label low number of classes: urban, multi-class: esc50) are presented in Table 7.
Sensitivity to number of prototypes. Our results show clear trend across backbones and pro- totypical probes: performance is highly sensitive to J at lower values and begins to saturate as J increases. The jump in performance from J though the impact varies on the task (multi-class vs. multi-label) and the dataset's 's structure (c.g.. number classes). For instance, on urban with EAT, protobin increases from 80.05 mAP at J=1 to 89.01 with nearly 9 percentage points (%,). The effect is even more pronounced on esc50, which sees 14 %p increase in the same setting. The subsequent gain from to 20 only 0.23 %, on urban. The saturation suggests that while multiple prototypes are crucial, there are only diminish- ing returns after enough prototypes are added. In contrast, the performance difference on as20k is much less pronounced. Using the same EAT model, the gain from J 1 to J 10 is only circa 2.6 %p. This suggests that the multi-label as20k dataset, with its high number of classes (527), does not require many prototypes per class, and that using single prototype is detrimental
5 and 10 is notable on all datasets,
18

---

Published as a conference paper at ICLR 2026
Table Comparison of probe methods across J/ number of prototypes. The methods include the 1 linear baseline, protobin, proto and the ablation to the binarization protofloat. We addition- ally add linear as the baseline performance. We report the mean mAP for as20k and urban, and mean accuracy for esc50 across 3 seeds after our hyperparameter selection. The linear baseline static across
Bold marks the number prototypes used in our main benchmark results.
as20k (mAP)
urban (mAP)
eac50 (Accuracy)
Backbone Probe
J=l
J=5
J=10
J=20
J=l
J=5
J=10
J=20
J=1
J=5
J=10
J=20
protobin
20.14
21.91
22.32
22.40
73.64
83.80
84.67
85.17
55.92
78.92
81.33
83.70
protofloat
20.87
22.55
23.01
23.07
77.46
84.69
85.37
86,01
67.67
80.00
81.75
82.10
A-MAB
proto
19.08
21.05
21.61
21.95
64.03
83.12
83.02
83.63
49.75
73.92
77.25
82.59
linear
8.36
58.72
22.08
protobin
26.70
27.68
31.54
31.93
78.23
87.25
88. 12
88.74
69.50
82.08
83.25
84.10
protofleat
27.64
30.52
30.89
31.68
79.90
87.92
88,60
88 63
75.75
84,25
84.58
84.70
BEATs
proto
27.63
30.47
30.54
30.66
77.41
87.55
88.64
89.04
77.25
83.67
84.67
85.08
linear
24.71
82.54
78.92
protobin
20.74
20.21
20.96
21.71
78.59
8 81.35
81.87
82.28 28
75.08
79.25
80.17
80.30
protofleat
21.18
21.57
21.21
21.30
79.21
82.09
82.48
82.19
76.00
79.25
79.75
80.12
ASIT
proto
21.31
21.94
21.89
20.73
68.89
81.20
81.89
82.35
73.50
78.75
81.95
82.44
linear
18.35
77.53
76.08
protobin
29.08
31.61
31.67
12
80.05
88.60
89.01
89 24.24
71.00
84,25
85.00
86.81
EAT
protofloat
29.11
31.04
31.19
31.81
82.00
90
89.08
89.14
82.58
87.58
88,25
89.14
proto
28.64
30.65
31.06
31.30
72.34
87.65
88.84
89.11 1
55.33
78.08
82.58
85,91
linear
17.29
77.76
75.33
protobin
28.69
29.77
30.94
10
81.08
86.82
88.92 92
89.05
65.75
80.67
83.83
84.70
protofloat
29.08
30.50
30.55
31.26
81.63
87.16
89.05
89.05 05
79.75
86.00
86.17
86.69
SSLAM
proto
29.08
30.53
30.84
30.99
81.60
88.58
88.45
88.82
62.08
80.67
82.17
85.18
linear
17.04
75.86
74.17
as it is for the multi-class tasks. Regardless of the task, this analysis confirms that our choice of J 20 (and J = 10 for as20k) for the central benchmark is robust, capturing the vast majority of the method's 's potential performance without adding excessive parameters.
Binarization and architectural simplification. This ablation reveals that our architectural simpli- fications are the primary driver of performance gains, while binarization offers a highly effective tra between a very minor precision cost in certain cases and major efficiency benefits.
Impact of binarization (protobin protofloat): Comparing proposed method protobin to its float-based counterpart protofloat reveals the direct impact of bina- rization. On some configurations in lower and higher number of prototypes, protofloat performs slightly better than protobin. The performance differences are expected and highlights an inherent trade-off: the full precision of 32-bit floats can capture finer- finergraiieed details. However, protobin remains highly competitive, demonstrating that binarization achieves a 32x memory reduction at the cost of only very low drop in performance in certain cases.
Impact of simplification (protofloat vs. proto): This comparison provides the clean- est evidence for the impact of our architectural changes. Our simplified, class-agnostic protoflosd loat consistently and significantly outperforms the class-dependent proto base- line across nearly all settings. This confirms that decoupling prototypes from classes leads to better performance.
Overall (protobin vs. proto vs. linear): We observe that protobin frequently out- performs the original proto baseline and linear, especially the complex multi-label datasets as20k and urban. It shows that the benefits of our architectural simplifications (the class-agnostic design) are powerful enough often outweigh the minor precision loss from binarization, resulting in a performance gain with a simpler and more efficient model.
19

---

Published as a conference paper at ICLR 2026
Task and dataset characteristics. The ablation results also underscore the task-dependent nature of the different prototypical architectures.
Multi-label (as20k, urban): On these complex datasets, the results underscore the advan- tage of our simplified and class-agnostic architecture. The presence of polyphony requires flexible design where prototypes can collaborate to disentangle overlapping sound events, strength of protobin and protofloat. This is particularly evident for models with highly entangled embeddings (e.g.. A-MAE) and on datasets with many classes (as20k), where the scalability of reusable prototypes for different classes is beneficial.
Muti-class (esc50): Conversely, on this single-label task, the advantage of our class- agnostic design diminishes. With only a single dominant sound source, the simpler and more direct supervisory signal of a class-dependent mapping can be more effective. In cases with less discriminative embeddings (e.g., ASiT), the full float precision of proto may also be necessary to capture fine-grained acoustic details, making it more competitive than our regularized protobin.
On the complex, multi-label datasets (as20k, urban), the architectural flexibility of our class- agnostic protofloat and protobin provides an advantage over the more class-dependent proto. This supports our core hypothesis that disentangled design is valuable for polyphonic scenes. Conversely, on the single-label esc50 task, this advantage diminishes. Here, the baseline proto is highly competitive, as the simpler challenge of learning direct class-to-rroootppe mapping seems to be sufficient for single-source audio.
C COMPUTATIONAL RESOURCES
To motivate the upper bound calculation, our benchmark combined 14 datasets, backbones, and 10 pooling methods. Each these combinations involved up to 50 hyperparameter trials plus final evaluation runs, establishing the basis for total run count. The computational cost of our benchmark can be divided into two stages. The first was a one- timetimmepee pre-computation of embeddings for each of the 9 backbone checkpoints across all 14 datasets. For the general audio and control datasets, we generated embeddings once for each the 9 backbones. For the bioacoustic datasets, this process repeated times per backbone to create distinct augmented variants for training. This initial stage resulted in:
5
= 378 pre-computation runs
(4)
backhomes
dataaeet
runidata
bio data
rums/data/
The second stage was the training and evaluation of the probing methods, where the hyperparameter optimization involved 50 initial trials managed by successive-halving scheduler, followed by final evaluation runs.
The number of pooling methods evaluated varied by dataset category. For the 5 general multi-label audio datasets, where all 10 pooling methods were evaluated, the upper bound on training runs was:
10
50
= 24, 750
datasets
backbones
probes
HPS
final seeds
For the 7 few-shot bioacoustic datasets, used reduced set of 6 relevant pooling methods, result- ing in:
6 probes
50
20, 790
(6)
datasets
backbones
HPS
final seeds
Finally, for the 2 multi-class control datasets, we evaluated core set of 3 representative probes:
3
50
5 final seods
= 2, 970
datasets
backbones
probes
HPS
20

---

Published as a conference paper at ICLR 2026
Summing these values gives the total upper bound on individual training runs for the entire bench- mark:
24, 750 20, 790 + 2, 270 = 48,510 total runs
(8)
We executed all benchmark tasks on a high-performmncc compute cluster equipped with NVIDIA A100 GPUs. This includes the initial augmentation-free forward pass required to pre-compute and cache the embeddings for all backbones, as well as the subsequent training and evaluation of all probing methods. The resulting on-disk embedding store for all cached features occupied approxi- mately 3.6 TB of storage. Code development and preliminary tests were performed on a workstation using NVIDIA RTX4090 GPU and an AMD Ryzen 7950X CPU.
D BENCHMARK IMPLEMENTATION DETAILS
This Appendix provides further details on the components of our benchmark's experimental setup.
D.1 BENCHMARK DATASETS
Table 8 presents overview of all 14 downstream datasets used in our benchmark, categorized into three thematic groups along with their respective sizes.
Table 8: Overview of the benchmark datasets. The datasets are organized into three groups: general multi-label, few-shot bioacoustic multi-label, and general multi-class. For each dataset, report the size of the train, validation, and test splits, the number of classes, and the audio clip length. Note that all bioacoustic tasks follow 64-shot training protocol.
Dataset
Train
Validation
Test
#Classes
Clip Length [s]
Multi-labell Datasets
a=20k (Gemmeeke el al., 2017)
685
17. 142
527
dased (Johnson et al., 2021)
20, 000
2. 0.00
10
#ad55k (Fonseca et al., 2022)
40, 966
10, 231
200
10
(Viveros-Mulbos
al., 2023)
77,500
3. 750
3, 750
28
10
urban (Salamon al., 2017)
000
0.000
2. 000
10
Mult-lobel: Bisacoustic BindSer (fid (fid-shott
(Rauch et 2025b)
344
12, 000
21
pow (Rauch ct 2025h)
072
360
per
(Rauch et
aes
(Rauch ct 2025b)
696
24. 180
ene (Ranch ct 2025b)
584
23, 766
56
uhh (Rauch et al., 2025b)
600
36, 367
27
(Rauch et al., 2025b)
3. 264
563
51
Multi-closs: General Datasers
esc50 (Piczak, 2015)*
600
400
sc2 (Wanlem, 2018)
84, 848
9. 982
4, 890
AudioSet (Gemmeke et al., 2017). as2m is large-scale dataset used to pretrain general-purpose audio models and built from vast collection YouTube videos. features a comprehensive ontology of over 500 sound classes, making it standard benchmark for general-purpose audio event detection and classification. The as20k dataset represents commonly used subset with 20,000 samples.
Domestic Environment Sound Event Detection (Johnson et al., 2021). desed is designed for eval- uating sound event detection in domestic settings, featuring 10-second audio clips. These recordings are annotated with temporal labels for 10 common sound classes like dishes, speech, and vacuum cleaners. It was specifically created to facilitate research in both centralized and federated learning scenarios.
Free Sound Dataset 50k (Fonseca et al., 2022). fsd50k is large, open dataset for sound event research, containing over 51,000 audio clips from the Freesound platform. It covers 200 diverse sound classes drawn from the AudioSet Ontology, with focus label quality through multi- step human verification process. The dataset is widely used for multi-label sound classification and detection tasks.
21

---

Published as a conference paper at ICLR 2026
Synthetic Polyphonic Dataset with Spatiotemporal Labels of Sound Sources WVieeos-Muude et al., 2023). spass is a synthetic collection of polyphonic soundscapes created for sound source lo- calization and separation tasks for 28 urban sounds. It provides detailed spatiotemporal annotations, specifying the precise time, location, and class of each sound event within the clips. It contains a set of five distinct acoustic background scenes. This makes it particularly valuable for developing and testing models that can understand complex acoustic scenes.
Urban-SED (Salamon et al., 2017). urban is collection for urban sound classification, containing 10-second audio clips of 10 common urban sound classes. These classes include events such as car horns, sirens, and street music, recorded from real-world city environments. The dataset serves as a popular benchmark for models tasked with environmental sound analysis.
BirdSet (Rauch al., 2025b). BirdSet is a comprehensive, large-scale collection of datasets for avian bioacoustics research. It aggregates recordings from various global locations, with each loca- tion forming distinct subset (hsn, pow, per, nes, sne, uhh, nbp). The collection specifically tailored to benchmark audio classification models, reflecting realistic bioacoustic monitoring chal- lenges.
Environmental Sound Classification (Piczak, 2015). esc50 is a benchmark collection for Environ- mental Sound Classification, consisting of 2,000 five-second audio clips. It is uniformly organized into 50 distinct semantic classes, including animal sounds, natural soundscapes, and human non- speech sounds. The dataset is standardized with a pre-defined 5-fold cross-validation setup, making it standard for evaluating audio SSL models.
Speech Commands V2 (Warden, 2018). sc2 is designed for keyword spotting and limited- vocabulary speech recognition. It contains thousands of one-second utterances of short command words (e.g.. "up." "down," " "stop") spoken by many different individuals. It contains 35 commands in the vocabulary, providing a robust benchmark for testing general-pun peneral-purpose models in audio.
D.2 FEWSHOT BIRDSET DETAILS
For our few-shot learning evaluation on the seven BirdSet downstream tasks, we constructed 64-shot training subsets. The creation of these subsets follows the pipeline detailed in (Rauch et al., 2025a), which involves a selection of audio clips to mitigate label noise from weakly- leableel recordings. Full details of the subset creation process and dataset characteristics can be found in the original BirdSet publications (Rauch et al., 2025b;a). Given the challenging nature of these wshch are multi- class during training but multi-label during testing, we introduced a light data augmentation strategy. For each of the seven 64-shot datasets, we pre- rre-generated and saved five distinct augmented variants using only the mixup augmentation with = 0.9, which is highly effective for bird sounds (Rauch al., 2025a). During each experimental run, we randomly selected sample of one of these five variants for training, providing diversity to the learning process without on-the-fj computational overhead.
D.3 POOLING METHODS
Table 9 summarizes the ten distinct pooling methods evaluated in our study. It details their architec- tural family, whether they operate on the [cls] -token or the full token map, and their computational complexity.
Table 9: Pooling methods overview. Methods are grouped by architectural family. The #params row lists symbolic counts, and the urban row instantiates them for our EAT-B/768 setup on ur ban. Symbols: N tokens (= S. Sy), D embed dim, C classes, H/ MLP hidden, k conv kernel, D' conv hidden, F number of frequency patches, Q queries, J prototypes.
[ctol Baseline
Tokon Map
Taken Kap (Acs.)
Tokes Map (Pruso.)
Linase
#1p
2000
Stagee
2228
stspoot
xtellp
proto
grocabia
DC
DM MC
NDC 3.9M
DD D.C
FDC 81.4
DC 1.2
D² ('1 DC
D
20" (QD DC
J1JC 158k
J1 198k
7.7k
3964
2.7M
8224
508k
3.0M
22

---

Published as a conference paper at ICLR 2026
D.4 HYPERPARAMETER SETTINGS
For each unique combination of a backbone, dataset, and pooling method, we conducted systematic hyperparameter search find the optimal learning and weight decay. This process ensures that each method is evaluated under its best-performing configuration, providing a fair comparison. Our search strategy consists of 50 trials for each combination, managed by successive-halving pruner to improve efficiency. The search is structured in two stages. First, we explore with 25 trials, using a Sobol sequence to perform quasi-random search, ensuring broad and uniform coverage of the hyperparameter space. Second, we exploit with 25 trials using tree- rre-sstrutturd parzen estimator (TPE) to focus the search on promising regions identified during the exploration phase. The configuration yielding the highest mean Average Precision (mAP) on the validation set then selected for the final evaluation, where is re- re-trained and tested using five different random seeds.
The search spaces were kept compact. For all baseline, convolutional, and attentive pooling methods, the search space was:
Learning Rate (Ir): A log-uniform distribution between x 10 and 7 x 10-3 Weight Decay (wd): A log-uniform distribution between x 10-6 and 5 x 10-4.
Based on preliminary experiments showing that prototypical methods benefit from higher learning rate, their search space for the learning rate was adjusted, while the weight decay remained the same:
Learning Rate (Ir): A log log-uniform distribution between 10 and 8 10-2.
All other hyperparameters were held constant across all experiments to isolate the effects of the pooling method. These fixed settings are summarized in Table 10.
Table 10: Fixed hyperparameters used for training all probing heads.
Optimizer
Epochs
Batch Size
LR Scheduler
Loss Function
Prototypes/Class
Prototype LR
Adami
128
Cosine
Asymmetric
Global LR
23


Reviews:
[{"review_id": "3WfoSNlZGW", "reviewer": "Reviewer_gW4z", "summary": "This paper primarily investigates the limitations of the existing linear evaluation paradigm for audio encoders.  \n\nThe authors show that the use of [cls] tokens  for linear probing does not take into account the time-space localised representations contained in patch tokens, and this hurts downstream linear probing performance in audio encoders, especially given that most audio encoders employ reconstruction objectives where information is rich in the patch tokens.  \n\nThe paper runs an evaluation benchmark on a set of linear probing methods on multiple downstream tasks, evaluated over multiple pre-trained audio encoders, and show that [cls] based methods perform poorly as opposed to learned pooling methods. \n\nMotivated by this, the authors propose a pooling strategy (protobin) which adds to the existing method of prototype pooling by introducing binary constraints.", "questions": "Control of prototype distributions: \nHow are the cosine similarities of prototypes mostly zero after training (Figure 4) if there is no explicit method to control the distribution?\nWhat is preventing two prototypes from learning the same thing? More specifically, why does the \\(\\pm{1}\\) constraint force near-0 similarity? It is not clear in the paper, and how this prototype-to-class contributions emerge is not explained in the results.  \n\nWhy exactly does the binarization work better in some tasks and not in others?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "Clarity: The paper is well written and can be easily understood.  \n\nMotivation: The paper is well motivated, and the motivation is justified by the results in Appendix A, where [cls] tokens alone perform much worse than learned pooling methods. \n\nContribution 1: The paper presents a benchmark and evaluation of recent SSL methods under multiple pooling/aggregation strategies. Using this benchmark, SSL methods on audio data can be fairly evaluated without a bias towards the [cls] token, thereby nullifying the apparent advantage seen by discriminative-only based methods on linear probing performance. \n\nContribution 2: The results by using the binarized prototype tensor (protobin and proto combined) yield SoTA  linear probing performance across almost all tasks and pre-trained models. Therefore, prototype-based aggregation can act as a standard method for performing linear probing for multi-label audio classification.", "weaknesses": "Significance: The fact that [cls] tokens alone perform worse than learned pooling mechanisms is already mentioned in literature [1]. Therefore, the same message being repeated for audio needs a stronger justification, with possible reasons as to why the message in [1] might or might not hold for audio data, thereby justifying the motivation for evaluation.  \n\nOriginality: The protobin aggregator is simply a binarization of the prototype tensor derived from Bird-MAE. Looking at Table 4 and 5, proto and protbin share similar overall performance, with protobin performing marginally better in more tasks than protobin (approximately 2/3), but not significantly enough to justify the addition of binarization as a contribution. The actual performance gain over other methods seems to be gained by the use of prototypes, as in Bird-MAE [2]. \n\nThe paper does not clearly explain why the cosine similarities between prototypes are mostly zero after training, despite the absence of any explicit mechanism to control the prototype distribution. It is unclear how the \\(\\pm{1}\\) constraint alone enforces this near-orthogonality or prevents different prototypes from converging to similar representations. \n\nMulti-label classification as a standalone downstream task: Although the extension towards other tasks is mentioned in future works, having multi-label classification alone as the evaluation framework does not provide a full picture on leveraging information from frozen features.  \n\nMinor: Some of the formatting seems to be too cluttered, for example the contributions and Q sections in results. It takes away from the clarity of the paper rather than adding to it. However, I acknowledge that this is subjective. \n\n \n\n[1] Przewięźlikowski, Marcin, et al. \"Beyond [cls]: Exploring the true potential of Masked Image Modeling representations.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2025. \n\n[2] Rauch, Lukas, et al. \"Can Masked Autoencoders Also Listen to Birds?.\" arXiv preprint arXiv:2504.12880 (2025).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3fWXvIiKQl", "reviewer": "Reviewer_FqWa", "summary": "## Rethinking probing for Audio SSL models\n\n1. The paper proposes prototypical probing strategies for probing frozen Audio SSL models.\n2. Several pooling approaches have been evaluated across several audio SSL models with different pretext tasks for a holistic evaluation of the proposed pooling approaches.\n3. The empirical evaluation conducted in the paper is solid, and showcases that the proposed pooling approach works better than alternatives for pooling.", "questions": "No specific questions. Please address the weaknesses stated above.", "limitations": "", "rating": 2, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. For the most part, the paper is well written and well presented. Diagrams and illustrations aid in understanding the paper.\n2. The proposed binarised prototypical probing methods are well presented. \n3. The evaluation protocol is thorough, with a good selection of audio SSL models and several probing strategies.\n4. Results show that protobin probing works well, especially on 3 of the 5 evaluated tasks.", "weaknesses": "## Weakness 1\n\na. The core argument/motivating principle, repeated several times in the paper, is that \"audio SSL works default to fine-tuning instead of probing\", but only a handful of papers that support the argument are cited throughout the paper. E.g.\n\nLines 52-53\n> Given probing’s apparent practicality and widespread adoption as an evaluation paradigm in computer vision (Oquab et al., 2024; Przewi˛e´zlikowski et al., 2025; Darcet et al., 2025), why does the audio SSL community still default to resource-intensive fine-tuning (Alex et al., 2025; Chen et al., 2024)?  \n\nLines 72-73\n> Therefore, the limited adoption of probing in audio SSL ....\n\nLines 214-215\n> Evaluation in audio SSL. While probes are widely used in computer vision (Oquab et al., 2024), audio SSL defaults to fine-tuning (Rauch et al., 2025b).\n\nStatements such as the above make it seem like the audio SSL domain is not doing probing-based evaluation on frozen encoders at all.  However, the paper only cites a handful of papers that support this notion, while ignoring the plethora of audio SSL works that evaluate frozen encoders using probing [1-15] (to mention a few). This includes Dasheng [8], the model you do cite in the paper and conduct experiments on.\n\n---\n\n## Weakness 2\n\n1. Further, the core motivation/hypothesis/guiding principle of the paper is that there is a \"pooling bottleneck\", and that standard single vector probes and [cls]-token only pooling under utilize token embeddings. However, several missing references listed in W1 are already aware of the drawbacks of conducting probing experiments only on [cls]-token or single vector representations obtained by pooling of information across all the patches.\n\n2. In MIM style SSL from spectrograms, every patch is a fragment in time and frequency. Thus, aggregation strategies that merge information across patches will inherently decimate frequency information. Niizumi et al. [02] were aware of this and proposed a simple strategy to aggregate temporal features while preserving frequency information.\n\n3. Thus, the <[cls]-linear probe> strategy is not representative of the entire audio SSL community's approach to evaluate audio SSL models, unlike what the paper in its current form makes it seem like.\n\n---\n\n## Weakness 3\n\nFrom lines 135-136\n> If the model exposes a last-layer....... If not, we can mean pool $\\tilde{z}_i$ by averaging all token positions.\n\nAs mentioned in Weakness 2, Point 2. This pooling strategy across token positions is suboptimal and puts plain linear probe at a disadvantage for the models that do not expose cls tokens. \n\n---\n\n## Weakness 4 \n\nIn several instances, the paper makes unjustified statements.\n\n> All current spectrogram-based audio SSL encoders apply MIM-style objectives, often coupled with student-teacher distillation (Chen et al., 2024; Alex et al., 2025).\n\nThese two papers are not representative of the entire audio SSL field. For e.g. [9, 13, 15] do not use a straightforward MIM style objective for learning audio representations through SSL.\n\n> While probes are widely used in computer vision (Oquab et al., 2024), audio SSL defaults to fine-tuning (Rauch et al., 2025b).\n\nHow does the paper on the BIRDNET dataset (Rauch et al., 2025b) justify that audio SSL defaults to using fine-tuning?\n\n---\n\n## REFERENCES\n\n01. Koutini et al., \"Learning General Audio Representations With Large-Scale Training of Patchout Audio Transformers\", 2022.  \n02. Niizumi et al., \"Masked spectrogram modeling using masked autoencoders for learning general-purpose audio representation\",   2022.  \n03. Anton et al., \"AUDIO BARLOW TWINS: SELF-SUPERVISED AUDIO REPRESENTATION LEARNING\", 2023.  \n04. Niizumi et al., \"BYOL for Audio: Exploring Pre-Trained General-Purpose Audio Representations\", 2023.  \n05. Niizumi et al., \"Masked Modeling Duo: Towards a Universal Audio Pre-Training Framework\", 2024.  \n06. Yadav et al., \"Masked Autoencoders with Multi-Window Local-Global Attention Are Better Audio Learners\", 2024.  \n07. Yadav et al., \"Audio Mamba: Selective State Spaces for Self-Supervised Audio Representations\", 2024.  \n08. Dinkel et al., \"Scaling up masked audio encoder learning for general audio classification\", 2024.  \n09. Li et al., \"Self-Supervised Audio Teacher-Student Transformer for Both Clip-Level and Frame-Level Tasks\", 2024.  \n10. Yadav et al., \"AxLSTMs: learning self-supervised audio representations with xLSTMs\", 2025.  \n11. Yuksel et al., \"GRAM: Spatial general-purpose audio representation models for real-world applications\", 2025.  \n12. Schmid et al., \"Effective Pre-Training of Audio Transformers for Sound Event Detection\", 2025.  \n13. Pepino et al., \"EnCodecMAE: leveraging neural codecs for universal audio representation learning\", 2025.  \n14. Niizumi et al., \"M2D-CLAP: Exploring General-Purpose Audio-Language Representations Beyond CLAP\", 2025.  \n15. Chang et al., \"USAD: Universal Speech and Audio Representation via Distillation\", 2025", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iVzThKMese", "reviewer": "Reviewer_HEBB", "summary": "This paper investigates the underperformance and underutilization of probing as an evaluation paradigm for self-supervised audio models, particularly in multi-label classification tasks. The authors hypothesize that the standard approach of using a linear probe on a globally pooled [cls] token creates an information bottleneck, discarding crucial information about localized sound events contained within the patch tokens. To address this, they introduce \"binarized prototypical probes\" (protobin), a lightweight and efficient pooling method that learns a set of per-class prototypes to aggregate information directly from the full token map. Through a comprehensive benchmark, the authors demonstrate that their proposed method significantly outperforms standard linear and attentive probing. The work makes a strong case for establishing prototypical probing as a more faithful and efficient evaluation standard in the audio SSL community.", "questions": "Please refer to the weaknesses above for the questions.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- Systematic and Insightful Problem Analysis: The paper's greatest strength is its methodical approach. Rather than just presenting a new method, it first provides a deep and convincing analysis of the existing problem—the \"pooling bottleneck\"—and validates this hypothesis with extensive empirical evidence.\n- Methodological Elegance and Efficiency: The proposed protobin method is simple to understand and implement, yet highly effective. It is parameter-efficient and, through binarization, memory-efficient, making it a practical tool for a wide range of applications, including on-device scenarios.\n- Comprehensive and Rigorous Benchmarking: The scale of the experimental study is a major strength. The evaluation across 13 datasets, 6 encoders (and their supervised variants), and 10 pooling methods provides a robust and generalizable foundation for the paper's claims. The structured presentation of results in response to explicit research questions is exemplary.\n- Clarity of Presentation: The writing is clear, and the high-quality visualizations (especially Figures 1, 2, and 5) are instrumental in conveying the paper's core ideas and findings effectively.", "weaknesses": "- Fixed Number of Prototypes: The number of prototypes is set to 20 per class across all datasets, following a heuristic from a prior work. While this seems to work well, the paper would be slightly stronger with a brief sensitivity analysis or discussion on how this hyperparameter might be optimally chosen for datasets with different characteristics (e.g., number of classes, intra-class variance).\n- Trade-off of Binarization: The protobin method is compared to a float-based proto version, and while it performs competitively and often wins, it is not uniformly superior. A brief discussion on the specific trade-offs of binarization (e.g., in which scenarios might the precision of float-based prototypes be advantageous?) could add more nuance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)