Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper extends earlier work on masked autoencoders (MAE) with an application on self-supervised learning paradigm for audio processing. The main subject that is subjected to is mel spectrogram. Both encoder and decoder acquires transformer-based architecture and local attention is deployed at decoder side, out-performing solely global attention. Such network takes advantages of the nature of spectrogram, which encodes correlation on local time and frequency bands.
strengths_and_weaknesses: ## Strengths
1. The improvement from the conventional MAE encoder is clearly explained and simple to grasp and re-produced, with detailed ablation analysis on receiptive field.
2. (Personal preference) The idea itself is closely related to the nature of spectrogram, instead of from the purely statistical perspective. Although the idea of regarding spectrogram as image patches is not new, this work put more emphasis on its relation with audio nature.

## Weaknesses
1. There are minor typos and grammatical mistakes in the paper. Please do proof-reading for the camera-ready version.
2. Mel spectrogram has its own limitations in information sparsity (especially for higher frequency regions). Therefore, raw or linear spectrogram might be better choices for this, at least as an additional ablation study.
3. VoxCeleb experiments. I am not 100% sure about how general ML community regard VoxCeleb, but officially it was proposed [1] as a verification task. Identification, in that case, can be regarded as an open-set verification.
4.  Probably some comparison with supervised models can be useful. But this is trivial.
5. There are some explanation in section I and II lacks justification or pointers to earlier studies, especially on explanining the nature of spectrogram's relation to speech cues - all things there makes sense to me, but needs some references.
questions: 1. The dataset covered in the work and related tasks are not that common for speech processing research community. Could you please address why you used AudioSet to train the model?

2. For random masking, is ther any specific technique involved for generating the masks and define the default mask ratios? From earlier works? Or just pretty random thing we cannot tell?
limitations: There are two main limitations in this work which are not addressed:
1. Acquisition of mel spectrogram only. It would be good if we try on various types of spectrograms.
2. Since VoxCeleb is involved - a scoring framework might be beneficial for verification experiments.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: The author apply Masked Autoencoders from the vision domain to learn representations from audio spectrograms in a self-supervised fashion. The learned representaions can be fine-tuned to achieve competetive performance on different tasks. The learned representations can be fine-tuned for competitive performance on various tasks. The authors propose using local attention windows.
strengths_and_weaknesses: Strengths:
- Clear and well-written presentation.
- Achieve SOTA on Audioset in a semi-supervised setup
- Doesn't relay on Imagenet and knowledge transfer from the vision domain.
- Extensive experiments on Audioset.

Weaknesses:
1. Low originality: the authors use MAE on spectrograms. I propose emphasizing the contributions of this work more clearly in the introduction (for example, adapting the local attention windows). Similar concurrent work  [42] [38] (as the authors point out clearly) exists. In particular, MaskSpec[42] is very close to the proposed work.  In light of this, what are the contibutions of your work(question to the authors)?

2. The authors argue repeatedly against knowledge transfer from Imagenet. However, the claims against are not quite justified, it's not clear what the drawbacks of imagenet are (question to the authors)?. For example, in line 32, what do you mean by label bias in this situation?

3. The huge computational complexity, even for fine tuning, especially compared to models pre-trained on Imagenet.
    - For finetuning the supervised methods on Audioset: AST[10] can be trained on 4  (titan RTX) GPUS in 1 week. PaST[28]  (also uses masking with Patchout) in 25-50 hours on a single 2080ti.
    - In SSL: MaskSpec[42] takes 4 days on 8x  V100 for the pretraining phase. I think this can be close in compute requirements to what the authors report?
    -  How do you justify the huge computation complexity with the limited improvement (within the error bars) on Audioset? (question to the authors)  
4.  The large computation complexity limits the usability of the models to extract representations (in my opinion).  (I suggest a large scale evaluation of the learned representaions on the tasks of the NeurIPS21 HEAR challange for example, https://hearbenchmark.com/ )
questions: I've listed the questions in the weaknesses section.
limitations: - The authors were up front about the similarity of their work to [42] [38]
- The authors do not justify the claims about ImageNet transfer learning.
- The large computational requirements compared to ImageNet transfer learning, with little improvments.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposes a method for self-supervised learning for audio classification. It is based on the prior work MAE which does self-supervised learning on images. The paper under review adapted the MAE model to audio by uses the spectrogram as an image. The decoder was changed to use the local self-attention as the audio tens to have more local correlations.

Then, the paper investigates the proposed architecture on several audio classification datasets. The paper conducts ablation studies on several aspects of the model: the masking strategies (finding that the unstructured masking works best for pre-training, and time+frequency masking works best for fine-tuning), the patch size and stride (finding that the overlap is not necessary), the encoder size, the decoder local or global attention, the length of the pre-training, and using the out-of-domain data for pre-training.

Finally, the paper compares the results to the previously published works. The proposed model outperforms all the previous in-domain models. Then, the in-domain proposed model matches or outperforms some of the out-of-domain methods on some datasets, while it is worse on other datasets.
strengths_and_weaknesses: # Strengths

The proposed model is sound. Applying and testing the CV models on audio is important and poses its own challenges which were overcome in this publication.

This paper conducts a thorough experimentation on multiple datasets with many ablations. The experimental section contains a lot of information and very interesting to read.

In general, the paper is very well written and easy to follow. I was able to all the logical steps and transitions. The amount of details is not overwhelming, but provides all the necessary information.

# Weaknesses

While I find it logical that pre-training on image data should not help much for audio training, the previous publications find it useful. This paper concludes that the out-of-domain data is not beneficial. This looks like an important area to address (see the question below).


questions: The paper argues in 4.4 that the out-of-domain training is not beneficial or even harmful. Nevertheless, the Table 2 shows that other methods benefit from the out-of-domain training. This needs some clarification.
limitations: N/A
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper presents the application of Masked Autoencoders (MAE) to audio spectrograms in a self-supervised framework. The overall idea is to rely on masking to widely corrupt the input data and train a Transformer AE to fill the blanked part of the spectrogram, leading to a form of inpainting self-supervised task. The authors study several different masking strategies in the context of spectrogram, and also analyse the impact of local or global attention in the decoder. The quality of the proposed method is assessed in a downstream speech and classification task.
strengths_and_weaknesses: Overall, the use of MAE for audio is an interesting avenue of research, as they perfectly fit to the self-supervised approaches. The experiments appear sound and are numerous, which is one of the major strength of the paper.

However, the two main weaknesses of this paper is that the originality and amount of contributions is quite low (as the author state themselves it is a "conceptually simple extension of MAE to audio"). Although the authors study several properties of the model and input data, it feels more like an engineering paper and I would recommend to resubmit this paper to a more applied conference. Furthermore, the paper is sometimes not very well written with several long and hard to follow sentences. The same applies to the related work section, which is a laundry list of citations, without clearly delineating the advantages or flaws of past methods.
questions: - My major question is on what you call a "unstructured" strategy, if we refer to Figure 2, I do not see any conceptual difference between (b) and (d), given that the "unstructured" is simply also a selection of time-frequency patches with a given size ? I think a truly unstructured approach would rather look like a Bernoulli dropout over the whole spectrogram ?
- I quite disagree with the overall premise (l.57) that "unlike image patches, spectrogram patches are mostly locally correlated" ... and I even think that it is exactly the opposite ! The stationarity property is way stronger in images and local correlation are largely more present than in spectrograms. You take the example of formants, which is actually the strongest local correlation that you could find, and even then vocal formants still follow an harmonic structure (hence global and cyclical correlation across frequencies). Any harmonic or noisy sound is a counter-example to this local correlation property.
- You talk about the high redundancy in the spectrogram (l. 118), and I think that analysing this aspect would make this paper stronger. How much it impacts the masked learning depending on the sound nature ? 
limitations: -
review_title: null

Meta-review: metareview: The paper has two strong accepts and two borderline reject reviews. However, as one of the reviewers did not engage with the authors post-rebuttal, I had to interpret the authors' response to the reviewer's concerns, and they seem to properly address them (even including a new experiment into the paper). The work seems to have been executed concurrently with other similar approaches, and while not entirely novel, the paper seems to include in-depth experiments and a discussion that can be beneficial to the research community. 
confidence: Less certain
Decision: Accept