MICM: Rethinking Unsupervised Pretraining for Enhanced Few-shot Learning

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-31 00:03:13

Abstract: Humans exhibit a remarkable ability to learn quickly from a limited number of labeled samples, a capability that starkly contrasts with that of current machine learning systems. Unsupervised Few-Shot Learning (U-FSL) seeks to bridge this divide by reducing reliance on annotated datasets during initial training phases. In this work, we first quantitatively assess the impacts of Masked Image Modeling (MIM) and Contrastive Learning (CL) on few-shot learning tasks. Our findings highlight the respective limitations of MIM and CL in terms of discriminative and generalization abilities, which contribute to their underperformance in U-FSL contexts. To address these trade-offs between generalization and discriminability in unsupervised pretraining, we introduce a novel paradigm named Masked Image Contrastive Modeling (MICM). MICM creatively combines the targeted object learning strength of CL with the generalized visual feature learning capability of MIM, significantly enhancing its efficacy in downstream few-shot learning inference. Extensive experimental analyses confirm the advantages of MICM, demonstrating significant improvements in both generalization and discrimination capabilities for few-shot learning. Our comprehensive quantitative evaluations further substantiate the superiority of MICM, showing that our two-stage U-FSL framework based on MICM markedly outperforms existing leading baselines. We provide the source code in the supplementary materials for reproducibility.

Main paper: 
 introduction: 
 Achieving high-level performance in deep representation learning typically requires large datasets, detailed labeling processes, and significant supervisory involvement. This requirement becomes even more daunting as the complexity of downstream tasks increases, challenging the scalability of supervised representation learning methods. In contrast, human learning is remarkably efficient, managing to acquire new skills from minimal examples with little supervision. Few-shot learning (FSL) [33, 40] aims to narrow this gap between human and machine learning capabilities.Although FSL has shown promising results in supervised settings, its dependence on extensive supervision remains a significant limitation. To address this, unsupervised FSL (U-FSL) [10, 32, 33] has been developed, mirroring the structure of supervised approaches.It involves pretraining on a wide dataset of base classes and then quickly adapting to novel, few-shot tasks. The primary goal of U-FSL pretraining is to develop a feature extractor that understands the global structure of unlabeled data, and subsequently tailors the encoder for new tasks. The increasing interest in U-FSL reflects its practicality and alignment with self-supervised learning methods, emphasizing its potential to significantly enhance machine learning processes.Recent advancements in state-of-the-art (SOTA) methods have largely employed Contrastive Learning (CL) [10, 32, 33] , particularly in transfer-learning scenarios. These methods have achieved impressive results across various benchmarks. As illustrated in Figure 2 (a), the principle of contrastive representation learning [9] involves drawing 'positive' samples closer and pushing 'negative' ones away in the representation space. This technique focuses on specific objects within datasets, thus improving representational learning for image classification tasks, as depicted in Figure 1(b) . Conversely, Masked Image Modeling (MIM) [12, 22] (Figure 2(b) ) trains models to predict the original content of intentionally obscured image portions. This approach facilitates comprehensive learning of features across all image patches, including peripheral ones, as shown in Figure 1(c) . In our research, we quantitatively assessed the impacts of MIM and CL on FSL tasks, revealing that while CL prepares models to prioritize features typical in training datasets, potentially compromising reliability in novel categories, and MIM fosters a broad and generalized understanding of image features but struggles to develop discriminative features crucial for accurately categorizing new classes in few-shot scenarios.To explore the trade-offs between generalization and discriminability in unsupervised pretraining, we introduce Masked Image Contrastive Modeling (MICM), a novel method that ingeniously combines essential aspects of both CL and MIM to boost downstream inference performance. Illustrated in Figure 2 (c), MICM utilizes an encoder-decoder architecture akin to MIM but integrates Modeling implements a decoder that simultaneously enhances features and reconstructs the original content of images. This method synergistically merges the principles of contrastive representation learning with effective pretext task design, thereby integrating the strengths of both approaches to achieve more nuanced and effective learning.a decoder that simultaneously enhances features and reconstructs images. This method not only merges contrastive learning with effective pretext task designs but also adapts efficiently to downstream task data. We further introduce a U-FSL framework with two phases: Unsupervised Pretraining and Few-Shot Learning. During Unsupervised Pretraining, MICM blends CL and MIM objectives in a hybrid training strategy. In the Few-Shot Learning phase, MICM adapts to various few-shot strategies, both inductive and transductive. Extensive experimental analysis confirms the benefits of MICM, showing it significantly enhances both the generalization and discrimination capabilities of pre-trained models, achieving top-tier performance on multiple U-FSL datasets.To summarize, the main contributions of our paper are as follows:• We reveal the limitations of MIM and CL in terms of discriminative and generalization abilities, respectively, to their consequent underperformance in U-FSL contexts.  
related work 2.1 few-shot learning: 
 Few-shot learning (FSL) in visual classification contends with the challenge of recognizing objects from very limited samples. Primarily, this task is approached through two principal methodologies: transfer learning and meta-learning. Transfer learning, as discussed by Tian et al. [40] , leverages knowledge from models pre-trained on large datasets to adapt to new, less-represented tasks. Metalearning, alternatively, encompasses several strategies: model-based [6] , metric-based [39] , and optimization-based [1] . Model-based approaches focus on adapting the model parameters rapidly for new tasks. Metric-based methods compute distances between samples to facilitate class differentiation, while optimization-based strategies aim to maximize learning efficiency with scarce examples. Building upon inductive FSL, transductive FSL seeks to enhance real-world application by incorporating unlabeled data into the learning process for pre-classification tuning. Among the various techniques employed, graph-based methods such as protoLP [52] utilize graph structures to strengthen the information flow and relationships between support and query samples. Clustering-based approaches, exemplified by EASY [4] , Transductive CNAPS [3] , and BAVARDAGE [26] , refine feature representations using advanced clustering techniques. Additionally, applications of the Optimal Transport Algorithm, like BECLR [33] , are used to align feature distributions more effectively during testing phases. A novel approach, TRIDENT [38] , integrates a unique variational inference network to enhance image representation in FSL scenarios.U-FSL broadens the scope of unsupervised learning by requiring models to not only learn data representations without supervision but also to rapidly adapt to new few-shot tasks. This challenging dual requirement has led to the exploration of various innovative methods. Knowledge distillation and contrastive learning are notably effective in U-FSL, enhancing model adaptability through robust feature representations [27] . Furthermore, clustering-based approaches have demonstrated considerable success by optimizing data groupings to better model task-specific nuances [31] . Despite these advancements, many traditional unsupervised methods are geared towards batch data processing, which may not seamlessly translate to the dynamic requirements of few-shot scenarios. To mitigate this, some strategies integrate meta-learning principles to generate synthetic training scenarios that improve data efficiency and model responsiveness [2] . However, such approaches can sometimes lead to suboptimal data usage [18] . Recent innovations in U-FSL also include the use of graph-based structures to map relationships within data [8] , the application of Structural Causal Models (SCM) in the context-aware multi-variational autoencoder (CMVAE) [34] , and the deployment of variational autoencoders (VAE) in frameworks like CMVAE and Meta-GMVAE [29] . Additionally, the exploration of rotation invariance in self-supervised learning enriches the robustness of models against geometric variations in few-shot learning tasks [45] . Another notable approach is MlSo, which leverages multi-level visual abstraction features combined with power-normalized second-order base learner streams to enhance the discriminative capability of models in U-FSL [49] . 
qualitative study on unsupervised pretraining methods for u-fsl: 
 This section delves into Unsupervised Few-Shot Learning (U-FSL), elucidating the task and assessing the impact of different unsupervised pretraining methodologies on model performance.U-FSL operates under a widely recognized protocol delineated in recent work [10, 27, 32, 33] . Initially, models undergo an unsupervised pretraining phase using a vast unlabeled dataset 𝐷 base = {𝒙 𝒊 }, which encompasses a variety of base classes. Subsequently, the models are tested in a few-shot inference phase using a smaller, labeled test dataset 𝐷 novel = {(𝒙 𝒊 , 𝒚 𝒊 )} comprising novel classes, ensuring no overlap exists between the base and novel classes. Each few-shot task T 𝑖 is structured around a support set S = {(𝒙 S 𝒊 , 𝒚 S 𝒊 )} 𝑁 𝐾 𝑖=1 , adhering to the (𝑁 -way, 𝐾-shot) scheme, where 𝐾 labeled examples from 𝑁 distinct classes are chosen. The query setQ = {𝒙 Q 𝒋 } 𝑁 𝑄𝑗=1 , typically unlabeled, comprises 𝑁𝑄 samples (where 𝑄 > 𝐾) and serves to evaluate the model's adaptation to novel classes.The foundation of U-FSL is significantly influenced by the capabilities of unsupervised pretraining methods to discern intricate patterns within unlabeled datasets. This segment explores the effects of two principal unsupervised pretraining strategies: Contrastive Learning (CL) and Masked Image Modeling (MIM). Key exemplars for these methods-SimCLR [9] for CL and MAE [22] for MIM-were selected due to their prominence and efficacy. Both methodologies were implemented using the Vision Transformer Small (ViT-S) architecture on the MiniImageNet dataset. Our analytical focus is directed towards understanding how these pretraining approaches modify the model's capability to transition effectively to novel tasks. We postulate that the intrinsic nature of the pretraining method-contrastive, which underscores learning distinctive features that delineate classes, versus masked, which centers on reconstructing absent segments of the input-will manifest differing strengths within the context of few-shot learning. Analysis of Contrastive Learning. Recent research [12] has demonstrated that CL models tend to focus predominantly on the primary objects within images during pretraining. This concentration frequently targets small, distinct features that distinctly characterize the primary categories. Such specificity, while beneficial for initial categorization tasks, adversely affects the generalizability of the learned features to new, unseen contexts, as evidenced in Figure 1(b) . We propose that this limitation arises from the inherent design of CL methodologies, which predispose the model to emphasize features that are salient in the training dataset but may be less relevant or even misleading in novel categories. To explore this proposition, we conducted empirical analyses comparing the feature representations of base category prototypes with those of novel category prototypes, both before and after applying SimCLR and its subsequent fine-tuning. The results, depicted in Figure 3 , indicate that the features extracted for novel categories by SimCLR closely mirror those associated with the base categories. This similarity persists when models trained via CL are applied to novel categories, leading to a continued reliance on the same features identified during the training phase. Consequently, there is a notable deficiency in the model's focus on principal objects in new categories, which hampers its adaptability. Building on established protocols [9, 15] , we fine-tuned the SimCLR-trained models on downstream few-shot classification tasks. Our findings, illustrated in Figure 3 , show that fine-tuning with a minimal set of labeled examples (e.g., in one-shot learning scenarios) fails to adequately rectify these issues of transferability. In some instances, this approach may even degrade the model's performance on few-shot classification tasks. It becomes apparent that only with an ample number of labeled samples for fine-tuning do these challenges begin to mitigate, consequently improving accuracy in few-shot classification scenarios. This highlights the intrinsic difficulties of directly applying CL models to few-shot learning tasks, especially considering the heightened risk of overfitting when training data are scarce. Hence, we assert the following conclusion concerning the impact of contrastive learning: Analysis of Masked Image Modeling. Recent advancements in MIM underscore a shift towards a more comprehensive feature extraction methodology, distinct from techniques that prioritize prominent image features. As detailed in [12] , MIM methods like the MAE engage systematically with every image patch to reconstruct absent segments, fostering a broad spatial activation across the entire image. This approach is vividly illustrated in Figure 1(c) , where feature maps from MAE reveal a widespread distribution of activation, suggesting a more holistic grasp of image features. Despite these strengths, the extensive focus on patch reconstruction in MIM can obscure class-specific feature delineation. Since MIM models are optimized for predicting missing image parts rather than distinguishing class features, they frequently lack the sharp, discriminative capabilities essential for class-specific recognition tasks. This deficiency is apparent in Figure 4 , where the first row demonstrates that features extracted by MAE are markedly less discriminative than those derived from our proposed method or the contrastive learning approach, SimCLR. The adaptability of MIM techniques to FSL scenarios is also challenged when these models are fine-tuned with limited labeled data. The second row of Figure 4 indicates that, even after fine-tuning with a modest sample size, such as in a 5-shot scenario, the discriminability of the features shows minimal enhancement. This observation implies that while MIM effectively encodes a rich array of generic visual features, it struggles to capture the subtleties required for distinguishing novel classes in few-shot configurations.. While MIM techniques cultivate a broad and generalized understanding of image features, they encounter significant obstacles in acquiring discriminative features crucial for accurately categorizing novel classes in few-shot learning scenarios. 
masked image contrastive modeling: 
 The preceding comparative analysis in Section 3 elucidates that while CL prioritizes refining focus on distinct objects within datasets, enhancing representational efficacy for image classification, MIM extends its reach to a comprehensive understanding across all image patches, thus facilitating a broader scope of feature extraction. This delineation underscores a pivotal trade-off in unsupervised pretraining between generalization and discriminability. To bridge this gap, we introduce a novel approach, Masked Image Contrastive Modeling (MICM), which amalgamates the strengths of both methodologies to foster robust representation learning coupled with an effective FSL task. Unsupervised Pretraining. Given an input image 𝒙 uniformly sampled from the training set 𝐷 base , we apply random data augmentations to create two distinct views 𝒙 1 and 𝒙 2 . These views are subsequently processed by the teacher and student networks, parameterized by 𝜽 𝒕 and 𝜽 𝒔 , respectively. Notably, the teacher network is updated via an Exponentially Moving Average (EMA) of the student network, facilitating knowledge transfer by minimizing the cross-entropy loss between the output categorical distributions of their augmented token representations, as expressed in the following equation:EQUATIONwhere H (𝑥, 𝑦) = -𝑥 log 𝑦, and Ẑ[cls] denotes the output class token. For MIM, we implement self-distillation as proposed in [50] .A random mask sequence 𝑚 ∈ {0, 1} 𝑀 is applied over an image with 𝑁 [patch] tokens 𝒙 = {𝒙 𝑖 } 𝑀 𝑖=1 . The masked patches, denoted by 𝑚 𝑖 = 1, are replaced by a learnable token embedding Z 𝑚 , resulting in a corrupted image 𝒙. The student and teacher networks receive the corrupted and original uncorrupted images, respectively, to recover the masked tokens. This is quantified by minimizing the crossentropy loss between their categorical distributions on masked patches:EQUATIONMoreover, we aim for the decoder-generated tokens to predict the RGB information of the image, incorporating an image reconstruction loss using Mean Squared Error (MSE) for the reconstruction targets Ȳ:EQUATIONFew-shot Learning. During the few-shot learning phase, the MICM approach is adeptly configured to adapt to diverse few-shot learning strategies, encompassing both inductive and transductive methods. The MICM methodology enhances feature learning by emphasizing generality across base classes and discriminative power.This dual capability significantly boosts the transferability of the learned features to few-shot tasks, thereby enabling superior adaptation to scenarios with limited labeled data. Further exploration of this adaptability is discussed in subsequent sections. 
experiments 5.1 datasets: 
 Unsupervised few-shot recognition experiments are conducted on three benchmark datasets widely recognized in the field: MiniIma-geNet [41] , TieredImageNet [35] , and CIFAR-FS [5] . MiniImageNet, derived from the larger ILSVRC-12 dataset [36] , consists of 100 categories, each represented by 600 images. It is divided into metatraining, meta-validation, and meta-testing segments, containing 64, 16, and 20 categories, respectively. TieredImageNet, also a subset of ILSVRC-12, includes 608 categories segmented into 351, 97, and 160 categories for training, validation, and testing. CIFAR-FS, a subset of CIFAR100 [28] , follows a similar structure to MiniImageNet, with 60,000 images spread across 100 categories. These datasets provide a robust framework for evaluating few-shot learning algorithms. Additionally, cross-domain experiments use MiniImageNet as the pretraining (source) dataset and ISIC [16] , EuroSAT [24] ,and CropDiseases [43] as inference (target) datasets, enhancing the generalizability assessment of the models.Self-Supervised Learning: The Vision Transformer (ViT) backbone and its associated projection head are pre-trained following the iBOT [50] framework, retaining most original hyper-parameters.The training employs a batch size of 640 and a learning rate of 0.0005 on a cosine decay schedule. The MiniImageNet and TieredImageNet datasets are pre-trained for 1200 epochs, while CIFAR-FS is pretrained for 950 epochs. Additional training details are provided in the appendix.Few-shot Evaluation: The pre-trained ViT backbone functions as the feature extractor. We utilize various few-shot learning (FSL) methods, including the prototypical networks approach [39] , for evaluation. In each N-way K-shot task, class prototypes are calculated as the mean of the features from K support samples per class. Query images are then classified based on the highest cosine similarity to these prototypes. The feature set for evaluation combines the [cls] token with the weighted average [patch] token, using selfattention values from the last transformer layer. Test accuracies are reported over 2000 episodes, with each episode featuring 15 query shots per class, consistent with standard practices in the literature [10, 32, 33] , and presented with 95% confidence intervals for all datasets.Discriminative and Generalization Capabilities. We investigate the critical balance between generalization and discriminability in unsupervised pretraining through our proposed MICM model. As depicted in Figure 6 , MICM surpasses other unsupervised pretraining methods in capturing comprehensive object information from novel classes, exhibiting superior overall perception. This is further corroborated by the distance distribution between the prototypes of novel and base classes in Figure 3 , where MICM's prototypes [12] ; (e) iBOT [51] ; (f) MAE [22] ; and (g) MICM (ours).Table 1 : Accuracies (in % ± standard deviation) on miniIma-geNet, comparing our model with various unsupervised pretraining methods (all models use VIT-S as backbone). CTB denotes the strategy of inserting a classification cls token before the processing by the encoder.Method Setting Inductive (ProtoNet) [39] Transductive (OpTA) [33] 5-way 1-shot 5-way 5-shot 5-way 1-shot distinctly differ more from the base class, affirming its enhanced generalization for novel classes. Additionally, Figure 4 demonstrates MICM's superior discriminative capabilities compared to models like MAE, with a clearer distinction in feature distributions between MICM and SimCLR. Notably, MICM maintains leading performance in small-sample scenarios, both pre-and post-fine-tuning. The experimental validations highlight MICM's adept integration of the strengths of CL and MIM, achieving remarkable discriminability and generalization. Improving FSL. Our exploration focuses on the synergy between MIM and CL, designed to overcome the limitations inherent to each Table 2 : Accuracies (in % ± std) on miniImageNet → CUB., comparing our model with various unsupervised pretraining methods (all models use VIT-S as backbone). CTB denotes the strategy of inserting a classification cls token before the processing by the encoder.Method Setting Inductive (ProtoNet) [39] Transductive (OpTA) [33] 5-way 1-shot 5-way 5-shot 5-way 1-shot approach individually. The MICM model we introduce effectively integrates the strengths of these methodologies, emphasizing the extraction of relevant feature scales within images. This integration not only enhances category discrimination but also bolsters robustness in subsequent FSL tasks. The effectiveness of MICM is demonstrated through its superior performance in both inductive and transductive few-shot classification settings, detailed in Table 1 .Enhancing Cross-Domain FSL. In cross-domain scenarios, MICM also significantly excels, notably on the CUB dataset (Table 2 ). Unlike traditional CL models, which often overfit to base classes, MICM maintains generalization across varied domains without the need Table 3 : Accuracies (in % ± standard deviation) on miniIm-ageNet, comparing our model with various unsupervised pretraining methods, which are adapted to several FSL methods [11, 33, 39, 44, 46] .Pretrained Model FSL method 5-way 1-shot 5-way 5-shot MAE [22] ProtoNet [39] 28.88 ± 0.43 37.19 ± 0.51 SimCLR [9] 54.42 ± 0.66 75.03 ± 0.35 iBOT [50] 61.26 ± 0.66 80.64 ± 0.45 MICM 61.37 ± 0.62 81.68 ± 0.43MAE [22]Fine-tuning [11] 28.50 ± 0.32 38.29 ± 0.50 SimCLR [9] 54.47 ± 0.59 75.01 ± 0.36 iBOT [50] 61. OpTA [33] 25.36 ± 0.48 35.21 ± 0.42 SimCLR [9] 65.83 ± 0.64 78.09 ± 0.40 iBOT [50] 74.58 ± 0.66 83.95 ± 0.34 MICM 78.40 ± 0.61 86.90 ± 0.33for fine-tuning, as illustrated in Figures 3 and 4 . This capability underscores MICM's effectiveness in capturing discernible features within an optimal range, thus boosting its adaptability and classification performance in few-shot learning across different domains. Broad Adaptation to FSL Methods. As a versatile pre-training model, MICM adapts seamlessly across a spectrum of FSL strategies. Comprehensive evaluations show that MICM invariably boosts performance, with notable improvements such as a nearly 4% enhancement over the iBOT model when utilizing the transductive method OpTA. These results affirm the robust generalization ability of MICM across a range of FSL approaches. cls token Variation. In exploring variations, we introduced a cls token as an input to the Encoder, with performance outcomes detailed in Tables 1 and 2 . Although this variant achieves commendable results in the inductive setting, it does not outperform the configuration where the cls token is input into the Decoder, especially in transductive scenarios. This suggests that introducing the cls token early in the encoder may impede the encoder's ability to learn comprehensive visual features effectively. Conversely, positioning the cls token in the decoder helps alleviate potential negative impacts by CL on learning holistic visual features.To assess the efficacy of MICM in FSL scenarios, particularly under the U-FSL framework, we developed and evaluated a novel methodology that combines unsupervised pre-training with pseudo-label training techniques. We integrated pseudo-label learning [33] with the transductive OpTA FSL method [33] , forming a hybrid approach designed to leverage the combined strengths of these methods to boost performance in scenarios with scarce labeled data. Our method's performance was benchmarked against SOTA models across various datasets, with detailed methodological descriptions provided in the Appendix.In-Domain Setting. Our model was evaluated against a broad range of baselines including established SSL baselines [7, 9, 14, 19, 20, 48] , prominent U-FSL methods [8, 10, 25, 27, 32, 37] , leading supervised FSL approaches [3, 30] , and a recent transductive U-FSL model [33] . Our model demonstrates superior performance, outperforming both inductive and transductive U-FSL methods as evidenced in Tables 6 and 4 , showing a notable accuracy improvement on the CIFAR-FS dataset.Our implementation utilizes the ViT architecture, which contrasts with the commonly used ResNet in U-FSL studies. To facilitate comprehensive evaluation, we compared results from models using both ResNet18 and ResNet50 architectures, and additionally, we benchmarked against a ViT-S model trained using the MIM method for transductive classification (MIM+OpTA), providing a baseline for ViT-based transductive U-FSL models. Regarding CIFAR-FS performance comparisons (Table 4 ), sourced from [33] , we note that the BECLR model reported results using ResNet18. Hence, our model's reported performance is achieved with a scaled-down version of ViT-S, comprising 6 layers (4 encoder layers and 2 decoder layers) as opposed to the full 12 layers in standard ViT-S. Codebase is attached to the supplementary material and will be made publicly available upon acceptance. 
 
 Appendix:  


Decision: The paper presents a novel method that integrates aspects of both contrastive learning and masked image modeling to address the limitations of unsupervised few-shot learning.

This paper received consistent positive ratings, and all reviewers confirmed that their concerns had been addressed during the rebuttal period. After reading the paper, reviews, and rebuttal document, I decided to recommend this paper for an oral presentation.

Reviews: review: paper_summary: paper_summary: This paper explores the difference between human and machine learning, particularly in learning from few labeled samples. Unsupervised Few-Shot Learning (U-FSL) aims to reduce reliance on annotated datasets. The study evaluates Masked Image Modeling (MIM) and Contrastive Learning (CL) for few-shot learning, finding limitations in their discriminative and generalization abilities. To address these issues, the authors introduce Masked Image Contrastive Modeling (MICM), combining the strengths of CL and MIM. MICM improves performance in few-shot learning by enhancing generalization and discrimination. Experimental results confirm the effectiveness of MICM in these tasks.
strength_weakness: strength_weakness: This paper identifies the limitations of Masked Image Modeling (MIM) and Contrastive Learning (CL) in terms of their discriminative and generalization abilities, which lead to their underperformance in Unsupervised Few-Shot Learning (U-FSL). To address these shortcomings, the authors propose a novel approach called Masked Image Contrastive Modeling (MICM). MICM combines the object learning strengths of CL with the generalized feature learning capabilities of MIM, aiming to enhance performance in U-FSL tasks.
limitations: limitations: 1. MICM seems to be a general self-supervised learning method. Why is it special for Few-shot Learning?
2. MICM is not novel. A lot of works[1] combine MAE and contrastive learning.

[1] Huang, Z., Jin, X., Lu, C., Hou, Q., Cheng, M. M., Fu, D., ... & Feng, J. (2023). Contrastive masked autoencoders are stronger vision learners. IEEE Transactions on Pattern Analysis and Machine Intelligence.
paper_summary: paper_summary: The paper presents a novel approach called Masked Image Contrastive Modeling (MICM) which integrates aspects of both Contrastive Learning and Masked Image Modeling to address the limitations in unsupervised Few-Shot Learning. The primary goal is to enhance the discriminative power and generalizability of models in few-shot scenarios by rethinking unsupervised pretraining. MICM uses a unique encoder-decoder architecture where the encoder focuses on visible patches of images to generate latent representations, and the decoder reconstructs masked patches while ensuring similarity in the representation space. This method significantly improves generalization and discrimination capabilities in downstream few-shot learning tasks, as demonstrated through extensive experimental analyses on various datasets.
strength_weakness: strength_weakness: 1. The paper is generally well-written, with clear motivation and discussion.
2. The proposed approach is simple and effective.
3. The analysis in Section 3.2 is interesting.
limitations: limitations: 1. The proposed approach can serve as a general unsupervised pre-training objective. It would be more interesting to see its performance on additional downstream tasks (other than unsupervised FSL).
2. It would be helpful to include a comparison with [1], which also aims to combine contrastive learning and MIM.

[1] Layer Grafted Pre-training: Bridging Contrastive Learning And Masked Image Modeling For Label-Efficient Representations. ICLR 2023.
paper_summary: paper_summary: This paper investigates the limitations of masked image modeling (MIM) and contrastive learning (CL) in terms of discriminative and generalization abilities in a few-shot scenario and proposes a novel unsupervised pretraining framework named MICM, which comprises the advantages of both MIM and CL. The approach leads to improvements in several FSL datasets.
strength_weakness: strength_weakness: 1. The paper identifies a crucial shortcoming of MIM and CL to few-shot learning; the features they extract need to be more balanced in discriminative and generalization. The analysis is intuitive and convincing.

2. The proposed method mitigates the above problem, providing a novel solution to capture better features for few-shot tasks.
limitations: limitations: 1. It is necessary to explain the method of generating attention maps in Figures 1 & 6. Are they generating from techniques like GradCAM? Also, which attention layers of competing methods are being visualized? 

2. Conducting experiments similar to those in Figure 3 for MAE would be beneficial. Will MAE without fine-tuning also provide features with high similarity to base categories due to the generaliztion ability?

3. Sec 4.2 illustrates the workflow using a positive image pair x1 and x2 as input. How can MICM develop discriminative ability (like SimCLR) without the use of negative image pairs?
paper_summary: paper_summary: This paper propose Masked Image Contrastive Modeling (MICM), a novel approach that effectively integrates the strengths of MIM and CL. Quantitative and  qualitative results demonstrate that MICM adeptly balances discriminative power with generalizability, particularly in few-shot learning scenarios characterized by limited sample sizes.
strength_weakness: strength_weakness: + The paper is well-written and easy to understand
+ The figures are well-designed and clearly express the ideas.
+ Both qualitative and quantitative results are provided.
+ The proposed method is orthogonal existing transductive and inductive FSL methods.
limitations: limitations: +  The computational complexity and well as the limitations of the proposed method should be further discussed.
+ Leveraging the idea of contrastive learning to establish more valuable representations has been explored in FSL [1][2]. Therefore, the differences w.r.t. those methods should be further discussed.

To the best of my knowledge, it is hard to find other major limitations in this paper.

[1] Free-lunch for Cross-domain Few-shot learning: Style-aware Episodic Training with Robust Contrastive Learning, ACM MM'22 \
[2] DETA: Denoised Task Adaptation for Few-Shot Learning, ICCV'23