Token Pruning Meets Audio: Investigating Unique Behaviors in Vision Transformer-Based Audio Classification

Authors: TaeHan Lee, Woojin Lee, Hyukjun Lee
Primary area: other topics in machine learning (i.e., none of the above)
Keywords: audio spectrogram transformer, token pruning

Abstract:
This submission is withdrawn due to errors in our token pruning analysis, which led to inaccurate claims regarding the retention of signal versus background tokens in AudioMAE-TopK. We have withdrawn our paper to prevent researchers from directly downloading it via the permanent hyperlink on OpenReview and relying on flawed information for their experiments. Please refer to the withdrawal statement for details. 

We are truly sorry for any inconvenience our error has caused.

Decision: withdrawn
Reviewer scores: 6, 6, 5, 5, 6

Reviews and discussion:
<thread>
creation date:
1730377141456

thread_type:
official_review

<thread_object>
creator: reviewer_psh5

<content>
summary:
The authors propose the MP-EViT model, which is the first time the token pruning technique is applied to the Vision Transformer-based audio classification task. A comprehensive experimental analysis reveals the unique token pruning patterns in audio classification, proves the importance of background tokens, and significantly reduces the amount of computation while maintaining high accuracy.

strengths:
1. For the first time, token pruning is applied to the audio classification task, and its unique behavioral patterns are deeply analyzed. This opens up a new research direction for the efficient application of ViT in the audio domain.

2. This study provides an important efficiency optimization scheme for the application of ViT in the audio domain, and reveals the unique characteristics of audio data processing, which is of great significance in promoting the development of audio AI.

3. The authors demonstrate through a large number of ablation experiments that what token pruning focuses on in the speech domain is dramatically different from that in the image domain, and analyze the reasons for this.

weaknesses:
1. The experimental dataset and model scale are relatively limited. The study utilized only the SPC-2 and ESC-50 datasets, and the ViT-B configuration was exclusively employed. Expanding the experimental scope could provide more comprehensive insights. It is recommended to conduct experiments on additional audio classification tasks, such as emotion recognition and speaker identification, to validate the generalizability of the proposed method.

2. There is a lack of comparison with other audio classification methods. The manuscript primarily compares the proposed method with baseline models, without assessing performance against other state-of-the-art audio classification techniques. This omission makes it difficult to comprehensively evaluate the advantages of the proposed approach. Has consideration been given to comparing token pruning with other model compression techniques, such as knowledge distillation and quantization?

3. The explanation of the mechanism for retaining background tokens is insufficiently detailed. While experiments demonstrate the significance of background tokens, the analysis of the underlying mechanisms that lead the model to preferentially retain these tokens is not adequately explored.

4. The manuscript mentions that due to resource constraints, only the ViT-B configuration was employed. Could supplementary small-scale experiments be conducted to investigate the impact of model capacity on the pruning strategy?

5. Is it possible to extend the use of mel spectrograms to magnitude spectrograms? This is particularly relevant, as magnitude spectrograms are more widely utilized in the field of speech processing.

questions:
The contents are the same as Weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730606324098

thread_type:
official_review

<thread_object>
creator: reviewer_8rry

<content>
summary:
The paper propose token pruning in Vision transformers fed with spectrograms for audio based classification tasks. Authors prune tokens which are least attended by the other tokens. The paper shows that token pruning can lead to less computation with very minimal drop in accuracy. Authors further show that unlike vision, for audio tasks background tokens are also being retained thereby showing their importance for such tasks.

strengths:
The paper presents a novel application of token pruning techniques by applying them in audio classifications tasks. The presentation of this paper is clear and it is easy to read. Authors also provide various analysis providing insights on the nature of token pruning for such tasks showing that background tokens can also be important for audio tasks. Moreover, on the two datasets considered in the paper, authors show almost no drop in accuracy while reducing MAC operations significantly.

weaknesses:
1. ViTs are very data hungry. The size of datasets considered is not that huge. This combined with high accuracy values in Table 1 could indicate model may be over-fitting to tasks or datasets at hand. Have authors tried to train the model on a bigger dataset e.g. Audioset and then evaluate its performance on other audio classification tasks such as the two mentioned in paper (SPC-v2 and ESC-50). Authors should show better generalizability of their proposed approach to make their claims stronger.

 2. The proposed approach reduces tokens from one layer to next similar to downsampling. Have the authors compared their proposed approach with other operations such as averaging blocks of n-tokens, uniform downsampling, downsampling by picking tokens randomly to name a few.

 3. Showing similar experiments with AST instead of audio-MAE can help give us an idea of generalizability of this method across different architectures. Also, since audio MAE uses mean pooling, it is kind of intuitive that CLS-EViT will work worse than MP-EViT. A better comparison would be against CLS-EViT of AST (since AST uses CLS token for classification) and MP-EViT of Audi-MAE

questions:
In the experiments, authors have used token pruning at layers 4, 7 and 10. Why were these three layers chosen? What is the effect of token pruning at every layer on the tasks at hand i.e. can too much pruning be harmful for the tasks at hand?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730620483418

thread_type:
official_review

<thread_object>
creator: reviewer_nryl

<content>
summary:
The authors applied a token pruning method originally used in vision tasks to audio classification, successfully reducing MAC operations and improving throughput with minimal accuracy loss. While the experiments were limited to only two datasets (SPC-2 and ESC-50), making the scope somewhat narrow, the detailed analysis provided valuable insights.

strengths:
I think the strength of the paper lies in its discussion and ablation studies.

Until Section 3.4, I thought it was merely applying a pruning method in ViT directly to audio classification, which, in my opinion, makes the contribution fall below the ICLR bar. However, the discussion and ablation study that follow are much more interesting than the proposed pruning method. I am a bit surprised that "background" patches are attended. So it brings some new knowledge to me.

weaknesses:
There are a few major flaws:

1- Audio MAE is a special Transformer model that is trained with Masked Data Modeling, i.e., it learns to attend to minimal patches by design. Thus, the conclusion of this paper is conditioned on a special MAE model, or the general Transformer model is questionable. This is a fatal issue and needs to be either clarified or experiments added on Transformers trained without MAE (e.g., the original AST). I cannot recommend acceptance without fixing this.

2- The pruning method is almost the same as the vision version, which is not novel enough.

3- The discussion is interesting, but for non-trivial conclusions, we also need to be careful. Why -0.65 (is that dB?) is a good measure of signal and background? If I remember correctly, speech commands are mostly clean speech, so the low intensity might be just some consonants / vowel transitions. Also, it is just 10-32 keywords classification; the model does not need to attend to everything (in contrast, doing large vocab ASR needs). So, more fine-tuning datasets are also needed.

In general, the discussions in the paper are interesting and can be the basis of a good paper, but a lot of further work needs to be done before publication.

questions:
See in weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730693738590

thread_type:
official_review

<thread_object>
creator: reviewer_vgml

<content>
summary:
This paper investigates the application of token pruning in Vision Transformer (ViT)-based audio classification, a novel approach given that token pruning has predominantly been explored in image classification. The authors propose the Mean-Pooling EViT (MP-EViT) model, which applies token pruning to audio data represented as Mel-spectrograms. Their model successfully reduces computational costs—specifically, the multiply-accumulate operations (MACs)—by approximately 50%, while maintaining a high level of accuracy with only a minor loss (<1%) in performance. The authors demonstrate that, unlike in vision tasks where background tokens are frequently pruned, audio classification tasks require a more nuanced approach as both signal (high-intensity) and background (low-intensity) regions contribute significantly to classification accuracy. The paper’s contribution is the adaptation of token pruning techniques for audio tasks, providing insights into the behavior of ViT models in audio classification and highlighting the importance of background regions in this domain.

strengths:
1.Originality:
This paper presents an innovative application of token pruning within Vision Transformer (ViT)-based audio classification, specifically utilizing Mel-spectrograms as input data. While token pruning has been extensively explored in vision tasks, its application to audio tasks is relatively novel, providing new insights into how ViT models can handle non-visual data.
2.Quality:
The research is methodologically robust, supported by thorough experimental setups and well-designed ablation studies. 
3.Clarity:
The paper is clearly structured, with each section building upon the previous one to ensure that the readers can follow the motivations, methods, and analyses without ambiguity. 
4.Significance:
By extending token pruning to audio classification, this paper contributes to the broader research community by highlighting the potential of ViT models in diverse non-visual domains. The findings regarding the significant role of background tokens in audio classification provide a valuable perspective that may influence future research on feature importance within various data types.

weaknesses:
1.Limited Generalization Across Audio Tasks:
The experiments focus primarily on speech command recognition and environmental sound classification, which may limit the generalizability of the MP-EViT model to other types of audio classification tasks, such as speaker identification, music genre classification, or emotion recognition. 
2.Lack of Comparative Baselines:
While the paper compares the proposed MP-EViT model against the baseline without token pruning, it lacks comparisons with other state-of-the-art models or alternative token pruning methods adapted for audio tasks. For example, methods like DynamicViT and SPViT, which use prediction-based token pruning, could serve as useful comparisons to understand the relative benefits and limitations of the mean-pooling approach used here. This would clarify the advantages of MP-EViT relative to existing pruning techniques.

questions:
1.Generalizability of MP-EViT:
Could the authors elaborate on the potential limitations of applying MP-EViT to other audio classification tasks beyond speech command recognition and environmental sound classification? Have they considered tasks like speaker identification or music genre classification, which may involve different types of audio features? Expanding the evaluation to these tasks would provide a clearer picture of MP-EViT’s adaptability and robustness.
2.Comparative Baselines:
MP-EViT is evaluated mainly against a non-pruned baseline. Could the authors include comparisons with other token pruning methods adapted for audio, to give readers a broader context of the model’s performance? This would highlight MP-EViT’s unique advantages and any areas where other pruning methods may outperform it.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730817295427

thread_type:
official_review

<thread_object>
creator: reviewer_qbkc

<content>
summary:
In this paper authors propose to prune blocks from the ViT when processing audio mel-spec data thus improving the computational efficiency without too much effect on the performance.

strengths:
Potentially useful way to process large audio corpora with this setup. ViT is useful in when first considering mel-spec as an image and then progressing as it would be an image. So the complete ViT pipeline for images would be applicable. Removing unnecessary blocks is reasonable as not every pixel in mel-spec is useful. 

Extensive ablation tests on the proposed method are clearly a positive side of this work.

weaknesses:
- Empirical comparisons to other published works is the main problem in this work. Authors do explain multiple different ways to improve computational efficiency (for images) and also for audio all using ViT framework as far as I can decode. 

- Reading the title I was expecting to see something where peculiarities of spectrogram (or mel-spec as it is compressed spectrogram) would be utilized to improve the pruning. But I really do not see anything of the sort, I hope authors can clarify this in the rebuttal stage. it is noteworthy that even though spectrogram is an image it is a very structured image, y-axis refers to frequency and x-axis refers to time, so for example in speech processing only a slice of the spectrogram would contain useful content. And in detail, speech spectrograms have a peculiar stucture of with harmonic and non-harmonic portions (such as 'a' has harmonics, whereas 's' not so much).

questions:
-
</content>
</thread_object>

</thread>
<thread>
creation date:
1732865166339

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Manuscript Update and Response

comment:
We sincerely thank Reviewers Qbkc, vgML, nrYL, 8rrY and pSH5 for their detailed and thoughtful reviews, as well as their valuable and insightful feedback. We incorporated the feedback to strengthen our revision. Major changes are indicated as blue in the revised paper.

* Wn: Weakness #n, Qn: Question #n


1. **We renamed our models’ name from MP-EViT to AudioMAE-TopK to correctly reflect the type of backbone model (AudioMAE) and a pruning method (TopK) at the beginning of Section 3.1**

    *(W1 by reviewer Qbkc, W2 & Q2 by reviewer VgML, Q2 by reviewer pSH5)*

2. **Since we primarily evaluated AudioMAE-TopK, we reordered the attention-score equations at Section 3.2.**

    We began our study with EViT, which uses pruning and fusion,  but later we removed the fusion portion to enable analysis of the origin of each token. Following **[1]**, we renamed our model to AudioMAE-TopK for better clarity. AudioMAE-TopK uses mean pooling as AudioMAE does, and AudioMAE-TopK-CLS is fine-tuned to use a [CLS] token for the prediction.

3. **We added additional datasets for AudioMAE-TopK to test the capability of the token pruning method in various audio classification tasks: VoxCeleb-1 and AudioSet-20K. Also, we applied different threshold values for each dataset; at Section 3 and Section 4.**

    *(Q1 & W1 by reviewer VgML, W1 by reviewer 8rrY, W1 by reviewer pSH5)*  
    After observing their histograms, we found that single threshold values could not be used to categorize signal and background tokens. Therefore, we applied different threshold values, denoted as \tau. For VoxCeleb-1 and AudioSet-20K, \tau is -1.25 and -1.0, respectively. Signal areas account for 63% in SPC-2, 81% in ESC-50, 65% in VoxCeleb-1, and 90% in AS-20K. For VoxCeleb-1, the accuracy dropped by 0.4% at a keep-rate of 0.7. For AudioSet-20K, the mean Average Precision (mAP) decreased by approximately 0.005 for every 0.1 reduction in the keep-rate, until it reached a keep-rate of 0.6.

4. **The reason for choosing 4th, 7th and 10th location is inserted as a footnote at Page 3.**  
    *(Q1 by reviewer 8rrY)*  
    In previous work, DynamicViT conducted intensive ablation studies on different token pruning locations and its numbers. They showed many pruning stages can hurt the model performance and simply using 4th, 7th, 10th layer can lead to best performance. EViT reported moving the reorganization modules into shallower layers deteriorates the accuracy and moving modules in deeper layers has only marginal influence.

5. **We added token pruning experiments with AST in Section 5.3.**  
        *(W1 by reviewer nrYL, W2 by reviewer 8rrY, W3 by reviewer pSH5, W1 by reviewer Qbkc)*  
    Many reviewers suggested testing other audio models, so we conducted additional experiments using AST with the TopK pruning method. Our revision paper covers all possible combinations of two models(AudioMAE, AST) and two attention score calculation methods(CLS, mean pooling).

    * For AST-TopK-CLS, we downloaded a pre-trained checkpoint.
    * For AST-TopK-MP (mean pooling as used in AudioMAE-TopK), we pretrained the AST model on AudioSet-2M following the same recipe **[2]**.
    By comparing AST and AudioMAE, our experiment shows,
    * AST is known for poor performance in speaker identification task (VoxCeleb-1) which is the reason why we did not choose AST as our baseline model in the original paper **[3]**, **[4]**.
    * However, AST-TopK-CLS maintains better accuracy than AudioMAE with a higher keep-rate(0.5), especially for the ESC-50 dataset. The accuracy drops by 0.2% on the SPC-2 dataset and 0.5% on the ESC-50 dataset when using a [CLS] token, and by 0.2% on SPC-2 and 1.0% on ESC-50 when using mean pooling for classification.
    * AST retains more signal tokens than AudioMAE, regardless of using CLS or mean-pooling to make predictions.
    * In addition, AST gives higher scores (high confidence) to signal tokens than to background tokens than AudioMAE. We added our analysis about why the pruning behaviors differ in the discussion Section 5.


6. **We added two more audio transformer models (HTS-AT [5] and DFT-AT [6]) to the related work section, as they are mentioned in the rebuttal, at Section 2.**  
    *(W2 & Q2 by reviewer VgML, W2 by reviewer pSH5)*  
    While we considered applying token pruning to these models, their use of windowed attention disrupted the direct mapping between tokens and their corresponding regions in the Mel spectrogram, making them unsuitable for our ablation studies.


7. **We believe using audio task-specific features (harmonic or non-harmonic portions) for token pruning can be useful, added at the end of the discussion section.**
    *(W2 by reviewer Qbkc)*
    We are currently exploring the importance of harmonic/non-harmonic portions in token pruning. We look forward to uploading additional findings during the rebuttal session.

(continued below)
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper investigates the application of token pruning to Masked Auto-Encoder based audio classification, a slightly novel investigation given that token pruning has predominantly been explored in image classification. The authors propose the AudioMAE-TopK (called Mean-Pooling EViT, MP-EViT, in the first version of the paper) model, which applies token pruning to audio data represented as Mel-spectrograms. The model successfully reduces computational costs—specifically, the multiply-accumulate operations (MACs)—by approximately 50%, while maintaining a high level of accuracy with only a minor loss (<1%) in performance. The authors demonstrate that, unlike in vision tasks where "background tokens" are frequently pruned in ViT, audio classification tasks (where ViT is called Audio MAE) require a more nuanced approach as there are no real "signal" (high-intensity) and "background" (low-intensity) regions, which can be pruned without affecting downstream performance. 

Strengths
- the paper presents a large number of experimental results with detailed analysis - supporting our understanding of how audio data is different from image data while all the downstream processing (ViT/ AudioMAE) is the same
- authors engaged in a good discussion with reviewers which led some to increase their score, authors increased the strengths of the results by adding additional datasets (VoxCeleb-1 and AudioSet-20K) and analyzing a related model (AST)
- the paper is well written and experiments are sound

Weaknesses
- Novelty appears somewhat limited, the paper is mostly analytical and does not present a significant new technique that works better for audio (the number of tokens that can be pruned with the TopK method before performance degrades is small)

Overall, this is a borderline paper. I recommend acceptance since the analysis is insightful and well executed, and I think we should understand the data we work with, even if we always apply mostly the same models to the data now. However, if the paper cannot be accepted, publication at a venue such as ICASSP is a viable alternative.