Components Beat Patches: Eigenvector Removal for Robust Masked Image Modelling

Authors: Alice Bizeul, Thomas M. Sutter, Alain Ryser, Julius von Kügelgen, Julia E Vogt
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: Self-supervised Representation Learning; Unsupervised Representation Learning; Visual Representation Learning

Abstract:
Masked Image Modeling has gained prominence as a powerful self-supervised learning approach for visual representation learning by reconstructing masked-out patches of images. However, the use of random spatial masking can lead to failure cases in which the learned features are not predictive of downstream labels. In this work, we introduce a novel masking strategy that targets principal components instead of image patches. The learning task then amounts to reconstructing the information of masked-out principal components. The principal components of a dataset contain more global information than patches, such that the information shared between the masked input and the reconstruction target should involve more high-level variables of interest. This property allows principal components to offer a more meaningful masking space, which manifests in improved quality of the learned representations. We provide empirical evidence across natural and medical datasets and demonstrate substantial improvements in image classification tasks. Our method thus offers a simple and robust data-driven alternative to traditional Masked Image Modelling approaches.

Decision: reject
Reviewer scores: 8, 3, 8, 3

Reviews and discussion:
<thread>
creation date:
1729500392719

thread_type:
official_review

<thread_object>
creator: reviewer_1xjx

<content>
summary:
The authors present a variant of the masked autoencoder where instead of masking a subset of image patches, they mask a subset of principal components. They assume that the masked and unmasked principal components are likely to be correlated in a way that is pertinent to the class, which improves downstream classification performance.

strengths:
The paper is well-written, and provides a clear and agreeable motivation for masking principal components instead of image patches.

weaknesses:
The main purpose of self-supervised learning is to learn a representation that can be fine-tuned to achieve **strong** downstream performance. This is not demonstrated in this paper. Putting it bluntly, it is not compelling to achieve ~60% accuracy on CIFAR10 and ~20% accuracy on TinyImageNet. Instead of using significant resources to train a vision transformer with this approach, it's more practical to train a small convolutional net to achieve higher performance.

In the original masked autoencoder paper, the authors demonstrate very strong downstream performance on ImageNet1k (not the tiny version), which makes their approach compelling. In light of this, the fact that MAE allegedly performs so poorly on CIFAR10 and TinyImageNet is a bit suspicious. If it is indeed the case that MAE performs poorly on smaller images, we still need to see PMAE do similarly well on ImageNet1k, but these experiments are not in the paper.

questions:
Can you explain why it makes sense for MAE to achieve near-SOTA performance on ImageNet1k in the original paper, but performs so poorly on the simpler datasets CIFAR10 and TinyImageNet?

As stated in "weaknesses," I think we need to see PMAE's performance on ImageNet1k to reach a conclusion about it's superiority over MAE.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730460016660

thread_type:
official_review

<thread_object>
creator: reviewer_5bnu

<content>
summary:
The paper proposes a new approach to pre-training neural networks in the context of masked image modelling. Instead of masking out random patches in the image, the authors propose a masking strategy in which the principal components of the image are masked out and the networks are then trained to recover these principal components. The paper empirically demonstrates the advantage of this approach on natural and medical images in the context of image classification.

strengths:
- The idea for the masked image modelling strategy is simple and intuitive. It is presented in an easy-to-follow fashion.
- The paper is generally well-written and clear (except for the points below).
- The evaluation strategy is solid.
- The use of SOTA baselines is good.

weaknesses:
- The idea of using PCA as an invertible transformation into a latent space is sensible, however there many alternative choices for such a transformation (e.g. Fourier or wavelet transforms). 
- The evaluation scenario focuses on classification as a downstream task. This is typically a task that relies on global information. However, other downstream tasks, such as object detection or semantic segmentation, require local information, and here, the proposed may perform less well. However, this is not tested.
- The datasets used for evaluation contain very small images (e.g. CIFAR10, MedMNIST)

questions:
- The method uses a lossless scenario for the PCA. The rationale for this is understandable, but I am wondering if this leads to practical problems (e.g. for very small eigenvalues, their ordering becomes random, etc.)?
- Why not evaluate other similar transforms (e.g. Fourier or wavelet transforms)? 
- How would the pre-training perform on large, high-resolution images?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730701998495

thread_type:
official_review

<thread_object>
creator: reviewer_tnex

<content>
summary:
This paper proposed a new masked strategy based on principal component analysis, in order to capture more global information. The auto-encoder is employed to reconstruct the masked-out components. Experiments are conducted on various datasets.

strengths:
This paper is well organized and written. The perspective of masking principle components is interesting.

weaknesses:
It does not make sense to me that we can recover removed principal components from other components. This is because unlike patches, these principal components are independent to each other. It is unreasonable to say we can recover a random variable X from another independent random variable Y. This is my main concern for me to give rejection, but I am open to change my score if I am convinced. 

Besides, reporting experimental results on ImageNet would be more convincing.

questions:
None.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730733379775

thread_type:
official_review

<thread_object>
creator: reviewer_wjuv

<content>
summary:
This paper addresses the problem of Masked Image Modeling, which classically entails masking out patches of pixels within input images and training a model to reconstruct these missing values based on the visible pixels in a self-supervised fashion. The authors propose a novel alternative to the masking of pixel patches, instead suggesting that the data first be transformed into a latent subspace i.e. projected onto its principal components, and masking operations be done on the component level instead of pixel level. The idea is that because the principal components can represent global correlations, masking individual components still retains information at some level of all pixel locations, as opposed to the masking of entire patches of pixels, which could remove e.g. an entire object. In essence, this allows the model to still be exposed to information from all pixel locations during training, leading to robust representations that are more likely to contain meaningful information needed for downstream tasks, e.g. classification. The approach, termed PMAE, is evaluated against the vanilla Masked Autoencoder in an extensive set of experiments for image classification on multiple natural and medical image sets, showing clear and often significant improvement in nearly all settings.

strengths:
- Originality

The proposed method is a relatively straightforward combination of previously established approaches, but integrated in a novel, clever, and elegant way. 


- Quality

The authors provide a thorough review of previous research leading to and motivating this work, as well as a nice discussion of the broader context of related research. 

Results support claims made throughout the paper.



- Clarity

The paper is very well written and easy to follow; the foundations are solid and motivation is clear.


- Significance

The proposed method avoids extensive hyperparameter tuning, known to be challenging in established MIM/MAE regimes, as ratio / size of image patches masked strongly influences performance on different  downstream tasks.  Experiments presented using MAEs parameterized with range of masking ratios highlight this -- training models with the standard convention of masking $75$% of image patches often results in poorer classification accuracy, suggesting the necessity of tuning beyond this accepted norm. 
In contrast, PMAE doesn't require much hyperparameter tuning and seems to perform quite well straight off the shelf.


The idea is simple yet solid, even drawing intuitions from early research in image processing / Eigenfaces, which I can imagine inspiring future/new directions in representation learning.

weaknesses:
I wonder about the scalability of PMAE when applied to larger images, with the potentially prohibitive costs of the eigendecomposition, e.g. $o(pixels^3)$. Could the method still perform well with low-rank approximations?

Some of the presented results are a bit unclear to me, see 'questions'.

questions:
In the experiments exploring effects of masking ratios, as I understand it in the $PMAE_{rd}$ case, a random percentage of variance between $[10,90]$ is masked. I'm a bit confused as to why in Figure 5, which presents the impact of masking percentage, the classification accuracy results only shown for percentages between $[10, 50]$. What happens when a higher percentage of variance is masked?

How do you anticipate PMAE would perform with nonlinear transformations, e.g. kernelized PCA with an RBF kernel?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732272450145

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Rebuttal: General Answer to Reviewers

comment:
Dear reviewers,

We thank you for the thorough reviews of our work and the comprehensive and valuable feedback. **We particularly value the acknowledgment that our work is impactful (_reviewer WJuv_), well-presented (_reviewers WJuv, tnEx,5bnU,1XJX_), well-motivated (_reviewer WJuv,1XJX_), that our idea is novel, intuitive, solid and interesting (_reviewers WJuv, tnEx,5bnU_), and that the proposed approach is supported by a solid experimental validation (_reviewer 5bnU_).**

Your detailed reviews have helped us identify areas for improvement and provided valuable insights that enhance the impact of our research. We have thoughtfully discussed your comments regarding the scalability of our approach (PMAE), and potential extensions to PMAE and have clarified some misunderstandings regarding PCA. We also used your feedback to extend the discussion section in our manuscript regarding our approach’s scalability. We have also included additional interesting results using kernel PCA as suggested by reviewer WJuv and provided a larger set of ablations over the masking ratio. **Please find these changes in blue font in our updated manuscript in section 8 and the appendix.**
</content>
</thread_object>

</thread>
<thread>
creation date:
1732551151543

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
Thank you once again for your constructive review of our work. 

We have revised our submission to include additional results, integrated your feedback, and provided detailed responses to your questions below.

We are happy to engage in further discussion or address any additional questions you may have. 
 If your questions have been addressed, we kindly hope you will consider raising your score.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732676487487

thread_type:
official_comment

<thread_object>
creator: reviewer_wjuv

<content>
comment:
I thank the authors for thoroughly addressing my questions and comments, as well as those of the other reviewers. I'm impressed by the improvement we see with kPCA, which further supports my positive evaluation that this angle, i.e. masking components in a latent space, is solid and can benefit the wider ICLR community.  I acknowledge the concerns raised by other reviewers, but I remain positive about this work and its potential and look forward to seeing where it goes next.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes a new idea for self-supervised visual learning. The conventional approach relies on training a model to predict the pixel values of randomly masked patches in training images. This paper proposes a radically different idea: instead of masking random patches, it masks one of the principal components (PCs) of the image and has the model predict the masked component from the remaining ones. The proposed method is evaluated by training a vision transformer architecture on TinyImageNet, CIFAR-10, and Medical MNIST, showing gains compared to patch-based masking.

While the idea is radically quite novel and interesting, and led to an engaging discussion between the authors and reviewers, these discussions reveal that the current draft needs improvement to become solid and convincing:

Reviewer tnEx finds the idea of predicting one PC from the rest somewhat problematic, as these bases capture orthogonal aspects of the image, making it difficult to predict data projected onto one PC from its projection onto the others. While the authors clarify that the PC bases are indeed orthogonal and that projecting data onto them decorrelates the data (making them linearly independent), they argue that the projected data can still contain nonlinear dependencies, which a nonlinear model like a neural network could potentially pick up. While true, it is still concerning that the approach is completely blind to first-order information in the data (linear relationships), which often contains significant information about the image and is necessary for its proper reconstruction.

Reviewer 5bnU questions what makes PCs a distinct and favored choice if one is already considering projecting data onto orthogonal bases.  There are infinitely many choices for such bases, with some having specific names and properties, such as Fourier bases. The reviewer wonders why PCs are the sensible choice for self-supervised visual learning. The authors respond that their framework can easily be extended to handle PCs in a latent space via the kernel trick, broadening the set of transformations for masked component learning. While true, this raises further questions about how to choose the best transform for learning masked components in self-supervised vision tasks. While a complete answer may be impossible, the question warrants further exploration of the choice of bases (including alternatives like Fourier), compared to what the paper currently offers.

Reviewer 1XJX questions the evaluation setting and why simple datasets are used to train a ViT architecture, resulting in significantly poor test accuracies. The authors point out they had difficulty applying their approach to larger datasets like ImageNet. This raised further concerns about the scalability of the approach. Computing PCs requires eigendecomposition or SVD, which is computationally expensive. While the authors emphasize that their work is merely a proof of concept and their goal is not large-scale experiments, there is no clear path to resolving the scalability issue. This raises the question of whether using PCs (which require SVD) is necessary, as opposed to alternative orthogonal bases like Fourier, which are much cheaper to compute.

In sum, while the paper considers a very interesting and novel approach to masking for self-supervised learning, there are several aspects that it can improve on to be considered a strong submission. I encourage the authors to continue pursuing this approach and resubmit their work after addressing the raised issues.