One-shot In-context Part Segmentation

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-04 15:03:22

Abstract: In this paper, we present the One-shot In-context Part Segmentation (OIParts) framework, designed to tackle the challenges of part segmentation by leveraging visual foundation models (VFMs). Existing training-based one-shot part segmentation methods that utilize VFMs encounter difficulties when faced with scenarios where the one-shot image and test image exhibit significant variance in appearance and perspective, or when the object in the test image is partially visible. We argue that training on the one-shot example often leads to overfitting, thereby compromising the model's generalization capability. Our framework offers a novel approach to part segmentation that is training-free, flexible, and data-efficient, requiring only a single in-context example for precise segmentation with superior generalization ability. By thoroughly exploring the complementary strengths of VFMs, specifically DINOv2 and Stable Diffusion, we introduce an adaptive channel selection approach by minimizing the intra-class distance for better exploiting these two features, thereby enhancing the discriminatory power of the extracted features for the fine-grained parts. We have achieved remarkable segmentation performance across diverse object categories. The OIParts framework not only eliminates the need for extensive labeled data but also demonstrates superior generalization ability. Through comprehensive experimentation on three benchmark datasets, we have demonstrated the superiority of our proposed method over existing part segmentation approaches in one-shot settings.

Main paper: 
 related work 2.1 visual foundation models: 
 Visual Foundation Models (VFMs) are trained on broad data that can be adapted to a wide range of downstream tasks. One of the key strengths of VFMs is their adaptability and versatility. Unlike traditional models that are often tailored to specific tasks, VFMs exhibit a remarkable ability to generalize. This versatility allows them to be fine-tuned or adapted for different downstream tasks without the need for extensive retraining or modification. The existing Visual Foundation Models (VFMs) primarily fall into two categories: (1) General Visual Foundation Models: These models learn comprehensive visual representations, forming the foundation for a diverse array of downstream computer vision tasks. They often leverage techniques like self-supervised learning to extract valuable features without heavy reliance on labeled data. Prominent examples include CLIP [29] , which excels in zero-shot image recognition by learning from a vast corpus of image-text pairs. Subsequently, several works like [48] have exploited it for fewshot learning. DINO [4] , DINOv2 [27] , SimCLR [7] , MAE [11] and MoCo [12] are other notable models in this category, focusing on contrastive learning to derive robust representations. These models demonstrate remarkable versatility, adapting seamlessly to various vision tasks such as classification, detection, and segmentation. (2) Specialized Vision Foundation Models: These models are tailored to address specific sets of vision problems or tasks. They often exhibit exceptional performance in their respective domains due to their targeted design. For instance, DALL-E [31] , DALL-E 2 [30] ,Stable Diffusion [32] , and Imagen [34] are renowned for their proficiency in generating realistic and high-fidelity visual content from textual descriptions. In addition, models like GLIP [19] are specifically crafted for open-set object detection, excelling in identifying objects beyond predefined categories. The recently introduced "segment anything" model (SAM) [15] has garnered significant attention for its remarkable ability to segment objects based on diverse input prompts. Matcher [23] and PerSAM [45] have explored dataefficient semantic segmentation based on SAM. Hummingbird [1] is developed for in-context scene understanding. In this paper, we focus on exploring the pre-trained stable diffusion model and DI-NOv2 to develop a training-free framework for accurate one-shot part segmentation.Part segmentation, a fundamental task in computer vision, involves the delineation of objects into their constituent parts, thereby providing a more detailed understanding of their intricate structure. As a fine-grained variety of semantic segmentation, part segmentation has experienced notable advancements parallel to the rapid expansion of semantic segmentation [5, 6, 9, 13, 21, 24] . Previous efforts were predominantly centered around the design and refinement of network architectures. These efforts [10, 20, 25, 39, 40] involved enhancing existing semantic segmentation networks through the integration of novel modules aimed at enhancing contextual information or fine-grained details. Furthermore, some methods [22, 26, 33, 47] explored multi-task joint learning, like edge detection, for utilizing supplementary information from complementary tasks. In addition, Pan et al. [28] explored a new open-set part segmentation framework, achieving category-agnostic part segmentation by disregarding part category labels during training. To achieve data-efficient part segmentation, several approaches have explored the use of generative models [2, 37, 46] . Some approaches for universal semantic segmentation often encompass part segmentation as well. These methods, such as SEEM [49] , SegGPT [42] , Semantic-SAM [18] , and HIPIE [41]  
methods: 
 We present a novel One-shot In-context Part Segmentation (OIParts) framework designed to achieve part segmentation with just one labeled image as an in-context example, leveraging existing visual foundation models without requiring any training or fine-tuning. Given an in-context example composed of an image 𝐼 𝑟 ∈ R 𝐻 ×𝑊 ×3 and a corresponding binary mask 𝑀 𝑟 ∈ R 𝐻 ×𝑊 ×𝐶 , OIParts can segment the object in query image 𝐼 𝑞 into desired parts as defined in 𝑀 𝑟 , where 𝑀 𝑟 denotes binary mask of 𝐶 object parts. The overview of the whole framework is illustrated in Figure . 2. Specifically, we employ the pre-trained stable diffusion model (SD) and DINOv2 to extract complementary semantic features for images 𝐼 𝑟 and 𝐼 𝑞 . Further, we fuse these two types of features with an adaptive channel selection mechanism to obtain more distinctive representations for each part. Subsequently, the selectively fused features are used to calculate the pixel-wise semantic similarity between 𝐼 𝑟 and 𝐼 𝑞 , we can obtain segmentation masks by transferring the pixel-wise label in the in-context example 𝑀 𝑟 to the novel query images guided by the computed semantic similarity. In the following subsections, we will delve into the details of the overall process.Previous works have demonstrated DINOv2's capability to provide explicit information crucial for semantic segmentation tasks. Additionally, Stable Diffusion (SD) exhibits a robust internal representation of objects, effectively capturing both their content and layout. Leveraging these strengths, we employ DINOv2 to extract dense visual descriptors for object parts and utilize SD to derive complementary global structure information, thereby enhancing the overall part representation. Specifically, we extract the token features from layer 11 of DINOv2 for each image, denoted as 𝐹 𝑑𝑖𝑛𝑜 , and extract the SD features 𝐹 𝑠𝑑 from the denoising U-Net.To align the scales and distributions of these two types of features, we first normalize the SD feature and DINOv2 feature by L2 normalization respectively following [44] . Then concatenate them along the channel dimension to get the feature 𝐹 ∈ R 𝐻 ′ ×𝑊 ′ ×𝐷 :EQUATIONConsidering that not every channel of the feature contributes meaningful information for each object part, we perform feature selection for the concatenated features. Some channels may be corrupted by noise or may capture irrelevant variations, thereby obscuring the distinctiveness of the representation for the specific part. Therefore, it becomes crucial to eliminate these noisy channels for specific object parts, ensuring a more discriminative and focused feature representation. Although it may seem intuitive to adopt a learningbased method for further fusing the two types of features, relying solely on a one-shot in-context example could undermine their generalization capabilities, as explored in the experiments section. Instead, we present an innovative approach leveraging channel selection to generate more discriminative representations using the two complementary features without additional training.To achieve this, two primary concerns need to be addressed: (1) Identifying the specific channels that should be chosen to achieve the desired discriminative power; and (2) Determining the optimal number of channels to be selected. In this paper, we introduce a novel adaptive channel selection mechanism by formulating an optimization problem that minimizes the intra-class distance. By solving this optimization problem, we aim to select discriminative channels that effectively capture the distinguishing characteristics of each part. This selection process ensures that the new features formed by the selected channels minimize the distance within the same class, allowing for the effective separation of different classes.To identify representative feature channels for each object part, we select channels that generate new features, thereby enhancing the compactness of data points associated with the same object part in the new feature space. This approach can implicitly improve the separability of data points from different object parts, as illustrated 𝑟 represents the binary mask corresponding to part 𝑐, and |𝑀 𝑐 𝑟 | is the number of pixels belonging to the part 𝑐 at the resolution 𝐻 ′ × 𝑊 ′ . Conversely, 𝑥 𝑐 represents pixels that do not belong to part 𝑐. We use a binary matrix 𝐵 ∈ {0, 1} 𝐶 ×𝐷 to indicate the selected channels for each category. Therefore, 𝐵 𝑐 ∈ {0, 1} 𝐷 is a binary vector that denotes whether each channel is chosen for part category 𝑐. Then, to select 𝐾 representative channels, we define the channel selection as an optimization problem. The objective of this optimization is to minimize the intra-class distance computed using the new feature vectors composed of 𝐾 selected channels:EQUATIONwhere 𝑥 𝑐 ⊙𝐵 𝑐 denotes only selecting the feature channels according to 𝐵 𝑐 , D (•) is used to measure the distance of the feature set, 𝐾 is the number of channels to be selected. Hence, our goal is to identify a subset of K channels. When these channels represent a pixel, they minimize the distance between pixels belonging to the same part. Here, we utilize a straightforward metric, variance, for D (•) to identify the optimal subset among the various subsets of K channels. Although we have explored several metrics, like Kullback-Leibler divergence and Jensen-Shannon divergence, as evaluated in the experimental section, we have found that adopting variance is the simple yet most effective approach. We denote the feature set 𝑥 𝑐 ⊙𝐵 𝑐 as 𝑦 𝑐 = {𝑦 𝑐 𝑖 ∈ R 𝐾 , 𝑖 = 1, . . . , |𝑀 𝑐 𝑟 |}, and 𝑦 𝑐 𝑗 denotes the mean of 𝑗-th channel. Thus, D (•) can be formulated as:EQUATIONGiven that the variance of a set of feature vectors is calculated separately for each channel, we can efficiently solve this optimization problem of Equation 2 to obtain 𝐵 𝑐 by ranking the variances of all 𝐷 channels and subsequently selecting the top 𝐾 channels with the lowest variances. Those 𝐾 channels form a subset with the minimum variance among all subsets of 𝐾 channels. This approach is also intuitively reasonable, as a channel with low variance indicates that it represents a common characteristic among pixels belonging to that part, thereby making it suitable as the representative channel for that part.We further evaluate the segmentation accuracy on the given in-context example to find the optimal value of 𝐾 for each part category 𝑐. Specifically, given a particular value of 𝐾, we utilize the aforementioned method to decide which channels should be selected for the part category. Subsequently, we compute two class centers corresponding to two feature vector sets, one belonging to part 𝑐 and the other not. These two class centers are then utilized to re-assign labels for the in-context example to obtain a clustered mask M𝑐 𝑟 . By varying the value of 𝐾, we can obtain different clustered masks M𝑐 𝑟 . Finally, we evaluate the accuracy of the clustered mask M𝑐 𝑟 with the corresponding ground-truth mask 𝑀 𝑐 𝑟 , choosing the value of 𝐾 that yields the highest accuracy for our approach. The overall pipeline of performing this channel selection is illustrated in Figure 3 . Note that this channel selection process only needs to be calculated once using the in-context example and can subsequently be applied during inference.To perform part segmentation for a query image 𝐼 𝑞 using the incontext example, we first extract the complementary feature and utilize 𝐵 to obtain the selectively fused feature 𝐹 𝑞 ∈ R 𝐻 ′ ×𝑊 ′ ×𝐾 for each part category. Subsequently, we use cosine similarity to measure the pixel-wise semantic similarity. For each pixel in 𝐹 𝑖 𝑞 , we can calculate the similarity score with each pixel in 𝐹 𝑗 𝑟 by:EQUATIONwhere 𝑆 𝑖,𝑗 denotes the cosine similarity between 𝐹 𝑖 𝑟 and 𝐹 𝑖 𝑞 , 𝑏𝑒𝑡𝑎 serves as a hyper-parameter for scaling value before applying a softmax operation to obtain the score values, and 𝑎 𝑖,𝑗 stands for the similarity score that have been normalized using the softmax function. We then utilize this similarity score to combine the corresponding labels from 𝑀 𝑟 , thereby generating a prediction for each pixel in the query image and acquiring the predicted part segmentation mask 𝑀 𝑐 𝑞 :EQUATIONFinally, by concatenating the predictions of all the parts, we obtain the final part segmentation prediction 𝑀 𝑞 . This prediction is further upsampled to the original image size using bilinear interpolation. In addition, the resolution of the extracted features used for part segmentation is relatively low, resulting in a loss of object details and coarse segmentation results around the boundary regions. To mitigate this issue and enhance the quality of segmentation, we incorporate an edge smoothing technique called the Fast Bilateral Solver (FBS) [3] . This technique effectively refines the coarse segmentation masks, providing more precise boundaries, thereby improving the overall accuracy and visual quality of the segmentation results. 
experiments: 
 In this section, we comprehensively evaluate our approach both qualitatively and quantitatively.Implementation Details In our experiments, we employ the DI-NOv2 [27] and Stable Diffusion v1-5 model [32] for feature extraction following [44] . For the Stable Diffusion model, we set the timestep to 100 and use a generic text prompt template like "a photo of c", where c is the corresponding category name. The feature maps extracted from both the Stable Diffusion and DINOv2 models are at a consistent resolution of 60 × 60 with dimensions of 768 and 1024, respectively. All experiments are conducted on NVIDIA RTX3090 GPU.We conduct experiments on two datasets of three distinct object categories, PASCAL-Part [8] and CelebAMask-HQ [17] , following the same dataset setting as SLiMe [14] . We evaluate our results using the mean Intersection over Union (mIoU) metric. PASCAL-Part provides comprehensive annotations of various object parts across images, encompassing 20 distinct object categories. We focus on the object categories of car and horse. In the car category, the object is annotated into six parts: body, light, plate, wheel, window and background. In the horse category, the object is annotated into five parts: head, neck+torso, legs, tail, and background. CelebAMask-HQ is a large-scale face image dataset created for facial segmentation tasks. We report results on the parts used in ReGAN and SLiMe for comparison, which divide the face into ten parts: cloth, ear, eye, eyebrow, skin, hair, mouth, neck, nose and background. We conduct experiments on the same test sets of the mentioned datasets employed in previous methods to ensure fair comparisons. We mainly compare our method with three existing methods across three distinct categories of datasets. In the 1-shot setting, our primary comparisons are with SegGPT, SegDDPM [2] and SLiMe.SegGPT explored an in-context learning framework for semantic segmentation training on a vast amount of annotated data, enabling inference with just one in-context example. SegDDPM explored denoising diffusion probabilistic models as an effective source of image representation for semantic segmentation. SLiMe was designed for part segmentation, exploiting the SD model capable of learning a part prompt using just one labeled example. Notably, our method even surpasses other methods in the 10-shot settings and even some fully supervised part segmentation methods [38, 39] . For a more comprehensive comparison, we also include ReGAN in the 10-shot settings. ReGAN leverages pretrained GAN models, specifically trained on the FFHQ and LSUN-Horse datasets for face and horse part segmentation. Additionally, for car part segmentation, ReGAN employs a pre-trained GAN from the LSUN-Car dataset.Comparison on the Face Dataset The results presented in Table 1 demonstrate the effectiveness of our method compared to other 1-shot and 10-shot approaches on the CelebAMask-HQ10 dataset. Overall, our method surpasses SLiMe, SegDDPM and Seg-GPT in terms of mIoU performance and for the majority of facial parts in the 1-shot setting, achieving a mIoU of 72.0% compared to 69.6% of SLiMe, 60.1% of SegDDPM and 57.1% of SegGPT. Notably, our method achieves these results without any training or fine-tuning, whereas SegGPT requires a large annotated dataset for supervision, and SLiMe necessitates specific fine-tuning. Additionally, despite the inherent disadvantage of comparing against a 10-shot method like ReGAN, our approach still outperforms Re-GAN on mIoU, achieving 1.3% improvements compared to 70.7% of ReGAN. It's worth noting that the comparisons made here highlight the robustness and effectiveness of our method, particularly in scenarios where annotated data is limited or fine-tuning is impractical.Comparison on the Car Dataset Car images in PASCAL-Part present distinct challenges compared to well-aligned face images in CelebA-HQ10, as they exhibit larger variations in perspective and appearance. Table 2 presents the results for the car class. In the 1-shot setting, our method outperforms SegGPT and SLiMe in terms of mIoU, yielding improvements of 11.9% (53.3% vs. 65.2%) and 3.6% (61.6% vs. 65.2%) , respectively. Qualitative results are illustrated in Figure 2 , demonstrating the superior performance of our method compared to SegGPT and SLiMe. In the 10-shot setting, our method outperforms ReGAN in terms of mIoU. Additionally, our method performs better than fully supervised baselines like CNN [38] and CNN+CRF [16, 38] .Comparison on the Horse Dataset The part segmentation of horse images in PASCAL-Part is more challenging than the other two object categories because of the ambiguity in distinguishing between different parts and the horse object in this dataset is usually partially visible. Table 3 shows our results on the horse class. Our method exhibits superior performance. For the 1-shot setting, our method has a large improvement over SegGPT, SegDDPM and SLiMe in all parts as well as on mIoU, gaining improvements of 26.6% (40.3% vs. 66.9%), 23.9% (43.0% vs. 66.9%) and 10.5% (56.4% vs. 66.9%), respectively. Furthermore, our method outperforms ReGAN on mIoU in the 10-shot setting. For fully supervised baselines like shape+Appereance [39] and CNN+CRF [16, 38] our method also performs much better in the report results.To gain a deeper understanding of our method's performance, we conduct a qualitative comparison with SLiMe and SegGPT. As depicted in Figure 4 , in the face dataset example, SLiMe often produces erroneous segmentations that do not align with the original image, particularly near the hair area. SegGPT tends to produce noisy segmentations, especially around the nose and mouth regions. In contrast, our method excels in accurately capturing fine-grained object details and delineating object parts, surpassing existing methods in this aspect. From the example of the car dataset, when the in-context example and the query image are captured from different perspectives, the performance of SLiMe and SegGPT methods can be significantly affected. As demonstrated in Figure 4 , when the query image is taken from an aerial perspective, while the provided example is from a distinctly different perspective, both SegGPT and SLiMe struggle to achieve accurate part segmentation. From the example of the horse dataset, we observe that when the object in the query image is partially visible, segmentation performance suffers, indicating a challenge for SegGPT and SLiMe in handling occluded or partially visible objects. Additionally, given that both SLiMe and our proposed method rely solely on one labeled example, we conducted further experiments to comprehensively compare with SLiMe. As illustrated in Figure 5 , we conduct additional experiments to evaluate the segmentation of a single query image using various in-context examples. Our observations reveal that SLiMe's performance on the same query image fluctuates considerably depending on the provided examples. Specifically, SLiMe tends to perform well when the in-context example closely resembles the query image, but its performance deteriorates rapidly when significant differences exist between them. This pattern can be traced back to SLiMe's reliance on training specifically with the in-context example, which may result in overfitting to that specific example. Although our method also exhibits fluctuations, we demonstrate better stability compared to SLiMe across various in-context examples.In this section, to comprehensively evaluate the contributions of different components in our approach, we conducted an ablation study on the face and car dataset as presented in Table 4 . Each row in improvements across nearly all parts, indicating the capability of our adaptive channel selection approach to generate more representative features for each part. This enhancement facilitates one-shot part segmentation. We further provide a visualized example with and without the channel selection in Figure 6 . We can observe that without the channel selection, the hand area is misclassified as face skin and neck, whereas with our channel selection, it is correctly classified as background, well demonstrating the effectiveness of our approach. 2to identify the channels to be selected. In addition to variance, we further investigate cosine distance, Kullback-Leibler (KL) divergence, and Jensen-Shannon (JS) divergence. The results are presented in Table 5 . Both of these distance metrics are capable of selecting representative features, thereby enhancing the mIoU. However, the straightforward application of variance enables us to identify the channels that achieve optimal performance, resulting in the highest mIoU of 67.9%. Channels with low variance tend to have more consistent feature values within the part, indicating that they may capture more relevant information for distinguishing between different parts of objects. Therefore, we adopt variance as the preferred distance metric for channel selection in our method.Selection-based Fusion vs. Learning-based Fusion To further evaluate the effectiveness of our proposed selection-based feature fusion, we train a linear classifier with two linear projection layers on top of the extracted complementary features of the in-context example. We conduct two additional experiments: first, we exploit the classifier's ability to perform part segmentation directly, and the results are detailed in Table 6 under the column labeled "Classifier". Second, we utilize the intermediate linear projection layer in the classifier to fuse the two features and perform segmentation like our approach as described in Equations 4 and 5, results are reported in Table 6 under the column labeled "Classifier Feature". Our analysis demonstrates that direct part segmentation using the linear classifier achieved a notably low accuracy of 45.4%. When fusing the SD and DINOv2 features using the linear projection layer, the performance improves to 64.3%, yet remains lower than directly concatenating the two features, which yields 66.9%. Notably, our selective fusion approach achieved the highest accuracy of 67.9%. These findings underscore the effectiveness of our selection-based feature fusion method, surpassing learning-based fusion techniques. Additionally, our part segmentation method outperforms directly training a classifier based on the extracted features.Post-processing To address the loss of spatial details stemming from the relatively small size of the extracted features, we employ an edge-aware smoothing algorithm FBS, which aids in restoring finer details around boundaries. Results in Table 4 showcase the effectiveness. Nevertheless, it is worth mentioning that these improvements are primarily attributed to the detailed cues provided by the higher-resolution inputs. Generalization Capability The proposed OIParts is a trainingfree framework, thereby preserving the generalization capabilities of the VFMs. This remarkable feature ensures that our method can be seamlessly applied to various object categories, needing only a single in-context example. As illustrated in Figure 1 , with just one labeled example, OIParts precisely segments the parts of novel objects and exhibits robust performance when dealing with objects in different poses or perspectives. 
conclusion: 
 In conclusion, our One-shot In-context Part Segmentation (OIParts) framework presents a pioneering solution to the challenges of part segmentation by harnessing visual foundation models (VFMs). We address the limitations of existing training-based methods, which struggle with variance in appearance, perspective, and partial visibility between one-shot and test images, often leading to overfitting and reduced generalization. Our framework introduces a novel, training-free approach that requires only a single in-context example for precise segmentation with superior generalization ability. Through a comprehensive exploration of VFMs' strengths, particularly DINOv2 and Stable Diffusion, we integrate an adaptive channel selection approach by minimizing intra-class distance, enhancing feature extraction and discriminatory power for fine-grained part segmentation. Our method achieves remarkable performance across diverse object categories, showcasing its effectiveness in one-shot settings.  
: 
 increasing the challenge. Therefore, it is crucial to investigate a generalized and data-efficient approach for part segmentation that can flexibly adapt to various objects.Recent advancements in visual foundation models (VFMs) have revolutionized several computer vision tasks, demonstrating remarkable capabilities across a range of tasks. These models exhibit a remarkable generalization capacity for in-context learning, making them well-suited for adapting to downstream tasks with just a few examples. Efforts such as SegGPT [42] focused on developing generalized in-context learning frameworks for semantic segmentation, enabling inference for novel objects with one labeled example. However, these methods still heavily rely on labeled data for training. Some previous works have primarily focused on exclusively exploring the capabilities of specific VFMs for part segmentation. For instance, SLiMe [14] leveraged the Stable Diffusion [32] model to localize part regions by learning a prompt embedding for each part from only one or a few annotated examples. Nonetheless, these methods still rely on training from a single labeled example, which can result in overfitting and undermine the generalization capabilities of models. Consequently, they encounter challenges in dealing with significant appearance and perspective differences between the test and training examples, as well as difficulties with when the object in the test image is partially visible. These limitations have motivated us to explore a training-free paradigm that relies on a single in-context example for precise part segmentation with superior generalization ability, as the examples shown in Figure 1 .In this paper, we introduce a training-free One-shot In-context Part Segmentation (OIParts) framework designed to unleash the full potential of VFMs in part segmentation, which is achieved by establishing correspondence between an in-context example and the test image leveraging the representations extracted from VFMs. To enhance the representation for fine-grained object parts, we leverage the complementary strengths of two distinct VFMs: DINOv2 and Stable Diffusion. DINOv2 effectively captures dense visual descriptors critical for precise part correspondence, while Stable Diffusion is perceptual to global object structural information. The integration of these two types of features gives rise to two key considerations: (1) Distinguishability: The exploration of effective feature fusion techniques to extract discriminative information for fine-grained part segmentation, relying solely on a single example. (2) Generalization: It involves developing a harmonious fusion approach for these inherently different-dimensional andscale features, ensuring that the resulting representation maintains its generalization ability across diverse scenarios.To address the challenges, we introduce a novel adaptive channel selection approach that minimizes the intra-class distance. We employ this approach to create a distinctive representation for each object part category by selecting channels that improve intra-class compactness, leveraging the information provided by the in-context example. This approach allows us to selectively fuse features from both DINOv2 and Stable Diffusion, yielding a unique representation for each part category that enhances discriminatory power and maintains the generalization capabilities of the extracted features. By leveraging this selectively fused feature, we can accurately segment fine-grained object parts by computing the pixel-wise similarity between the provided in-context example and the query image, without requiring extensive labeled data for training. Furthermore, our approach enables flexible and effective selection of the most relevant features for object parts from different in-context examples, offering adaptation for various objects. Hence, our framework exhibits characteristics of generalization ability, data efficiency, and adaptation. The primary contributions of this work are summarized as follows:• We comprehensively explore the complementary features of DINOv2 and Stable Diffusion to enhance part segmentation effectively, resulting in a training-free framework for one-shot in-context part segmentation by synergizing the complementary strengths of the two models. • We propose a novel adaptive channel selection approach by improving intra-class compactness to effectively fuse the two features, yielding more discriminative fine-grained part representations without compromising the generalization ability. • Through comprehensive experimentation, we demonstrate that the segmentation performance of our proposed method surpasses that of existing part segmentation methods utilizing only one in-context example. This superiority is especially notable in datasets with significant pose and perspective variations, such as the horse and car datasets. 
 
 Appendix:  


Decision: This paper introduces a part segmentation method by leveraging visual foundation models. The original reviews are one accept, two borderline accept, and one borderline reject. The main concerns center around the mechanism of channel selection and in-context mask guidance. The rebuttal successfully addressed some of the concerns and one reviewer raised the final rating to weak accept.

Reviews: review: paper_summary: paper_summary: This paper introduces a training-free, flexible, and data-efficient one-shot in-context part segmentation method, which leverages the complementary strengths of DINOv2 and Stable Diffusion. Extensive experiments conducted on three benchmark datasets showcase the superiority of the proposed approach.
strength_weakness: strength_weakness: The concept of exploring the complementary strengths of DINOv2 and Stable Diffusion for part segmentation is interesting.
limitations: limitations: 1、Does the channel selection require the use of a mask? If a mask is required, how is the channel selection performed on the query image? 
2、Figure 2 is unclear regarding how the In-Context Mask guides the segmentation of the query image and should be clarified.
paper_summary: paper_summary: The paper introduces a novel framework named One-shot In-context Part Segmentation (OIParts) that addresses the challenges of part segmentation using visual foundation models (VFMs) without requiring extensive labeled data. This training-free approach leverages DINOv2 and Stable Diffusion to enhance generalization and segmentation accuracy. The framework employs an adaptive channel selection mechanism by formulating an optimization problem that minimizes the intra-class distance, which can learn more discriminative features of each part. The proposed method achieves superior segmentation performance across various datasets compared to existing methods utilizing only one in-context example.
strength_weakness: strength_weakness: The paper presents a novel training-free approach for part segmentation by proposing an adaptive channel selection method and utilizing the pretraining features of Visual Foundation Models. The paper is well-organized, with clear explanations of the methodology and thorough experimental analysis. Figures and tables effectively illustrate the results and comparisons with baseline methods. The proposed method is technically robust, utilizing state-of-the-art VFMs and demonstrating a clear understanding of the challenges in part segmentation. The experimental results are well-documented and show consistent improvements over existing methods.
limitations: limitations: 1、While the combination of DINOv2 and Stable Diffusion is effective, the main innovation lies in the adaptive channel selection. Why not consider different combinations of other Visual Foundation Models？ More combinations of experiments can demonstrate the effectiveness and generality of the proposed method.
2、The proposed method involves complex processes like feature extraction from two different VFMs and an adaptive channel selection mechanism. This could pose challenges for practitioners in terms of implementation and computational resources. Does the author have any plans for open source code？
3、The reviewer is concerned about the scalability of the method. The method's reliance on high-resolution feature extraction and adaptive selection might limit its scalability for very high-resolution images due to computational overhead.
4、If there are several components in the in context image that are outside the screen or are obstructed, how should the proposed method handle this situation?
paper_summary: paper_summary: This paper presents a novel framework for part segmentation tasks using visual foundation models (VFMs). It introduces a training-free, flexible, and data-efficient approach that leverages VFMs to perform part segmentation using only a single in-context example, which is designed to improve generalization and eliminate the need for extensive labeled data. And it demonstrates significant improvements in segmentation accuracy and generalization capabilities across various datasets.
strength_weakness: strength_weakness: 1.	The introduction of an adaptive channel selection mechanism to improve intra-class compactness and enhance feature discrimination is novel. By selectively fusing features from both models, the framework achieves more discriminative part representations, which is a significant improvement over existing methods that do not employ such adaptive techniques.
2.	The theoretical underpinning of a training-free framework is a significant strength. By avoiding extensive training and fine-tuning, the approach minimizes overfitting and enhances generalization capabilities. This makes the method highly adaptable and efficient, requiring only a single in-context example for accurate segmentation.
3.	Detailed ablation studies are conducted to validate the contributions of different components of the framework, such as the use of DINOv2 and Stable Diffusion features, adaptive channel selection, and post-processing with Fast Bilateral Solver (FBS). These studies provide a clear understanding of the importance and impact of each component on the overall performance.
limitations: limitations: 1.	The paper builds upon existing VFMs such as DINOv2 and Stable Diffusion, and while it combines them in a novel way, the core techniques and models are not entirely new. Previous works like SLiMe and SegGPT have already explored the use of VFMs for segmentation tasks. For example, SLiMe also leverages the Stable Diffusion model for part segmentation, highlighting similar complementary strengths. This could suggest that the paper's novelty primarily lies in the integration and slight modification of existing methods rather than introducing fundamentally new concepts.
2.	While this paper aims to mitigate overfitting by avoiding training on one-shot examples, it does not ensure and prove that adaptive channel selection does not implicitly cause model overfitting through its optimization process. It is crucial, especially when fine-tuning the number of channels or the selection mechanism based on a minimum number of examples.
3.	The experiments are conducted on three datasets, but the diversity of the datasets could be questioned. PASCAL-Part and CelebAMask-HQ are commonly used datasets, but additional evaluation on more varied and challenging datasets could provide a better assessment of the generalization capabilities of the proposed method. Furthermore, the paper primarily compares its method to a limited number of existing approaches.
paper_summary: paper_summary: This paper targets on the task of part segmentation via visual foundation models (VFMs). To relieve the overfitting of existing methods that train on the one-shot example, OIParts, a training-free framework is proposed. With the concatenated feature from DINOv2 and SD, an adaptive channel selection approach is proposed. Through comprehensive experimentation on three benchmark datasets, author have demonstrated the superiority of OIParts over existing part segmentation approaches in one-shot settings.
strength_weakness: strength_weakness: 1. The writing is easy to understand. 
2. The proposed method sounds reasonable and novel. 
3. A bunch of experiments are provided.
limitations: limitations: 1. The organization can be improved. For example, L66-88 is too long. I suggest the authors to be more concise in the Intro and leave more details to the related work. Besides, using a single long paragraph for each related work is not a good idea as in L168-274. 
2. Does in-context segmentation means one-shot? if yes, whether previous one-shot VOS tasks like [1] belongs to in-context learning?
3. Fig. 3 can be further polished. 

Ref:
1. In-N-Out Generative Learning for Dense Unsupervised Video Segmentation.