The benefits of Incorporating Shape Priors in Contrastive Learning

Authors: junru  zhao
tianqin  li
tai-sing  lee
Venue: ICLR_cc_2024_Workshop_Re-Align
Published: 2024-02-03 05:26:01

Abstract: Human babies develop the ability of figure-ground segregation based on motion, luminance and color cues early on during infancy. The availability of the global form or shape of the objects is known to facilitate  rapid learning of lexical categories in babies. Here, we explored the use of shape prototypes, computed by momentum clustering the global forms of objects, to bootstrap a form of self-supervised learning, called contrastive learning, to mimic human learning. We found that shape prototypes can play a positive role in speeding up representation learning by highlighting the importance of object boundaries and forms at the initial learning phase but might hinder learning detailed features for object recognition. Thus, a hybrid of "coarse-to-fine" or "shape-to-texture" training regimes that foster learning global shapes and local features produces high-performance object recognition systems with global shape sensitivity.

Main paper: 
 introduction: 
 The global shapes of objects play a pivotal role in human object recognition and representational learning. Many studies have revealed that human object recognition uses shape as a principal discriminative cue with few costs (Wagemans et al. (2008) ; Elder & Velisavljević (2009) ). Children, for example, have been found to learn new lexical categories much faster when the categories are organized by shapes (Gershkoff-Stowe & Smith (2004) ). How can global shape representation be learned? How can global shape representations facilitate the development of categorical recognition? Neural networks learned to perform object recognition have been found to rely heavily on local features and texture cues rather than global shapes of objects (Islam et al. (2021) ; Geirhos et al. (2022; 2020) ). Currently, there is active research in the deep learning community on the strategies and training paradigms that will endow deep networks with stronger "shape bias" (Geirhos et al. (2020; 2022) ; Wen et al. (2023) ). The presumption is that having shape representations that are aligned with human perception will make deep networks more flexible and generalizable, resulting in better object recognition performance. Most of those studies, however, focus on the supervised learning and the progress of enhancing shape bias through self-supervised learning is limited.In this paper, we investigated the impact of incorporating global shape priors in contrastive representational learning in deep neural networks. We introduced a novel contrastive learning framework called Shape Prototype Contrastive Learning (S-PCL ). In this framework, global shape in the form of the silhouette of an object, extracted using existing methods on figure-ground segregation, is used as to augment the input training image in contrastive learning. Shape prototypes, the cluster centroids of silhouette shapes of objects in the data set in the embedding space, are learned with momentum clustering and then used to organize the embedding space of the original input images. We found that deep networks learned with these shape prototype priors exhibit stronger shape representations that are more aligned with human perception. Furthermore, we found that S-PCL accelerates the learning process, particularly in the early stage of development, reminiscent of the impact of global shapes in children's lexical categorical learning. This also alleviate the problem that most of current contrastive learning frameworks require a long training period to achieve a a satisfying representation space (He et al. (2020) ; Grill et al. (2020) ; Li et al. (2021) ).  
related work: 
 Our work is closely related to two main topics: contrastive learning and shape bias of CNN.The contrastive learning is a training framework of self-supervised learning where a model is trained to distinguish between similar and dissimilar embedding pairs of instances . the instance-wise contrastive learning (van den Oord et al. (2019) ; Wu et al. (2018) ; He et al. (2020) ; Grill et al. (2020) ; Chen et al. (2020) ) aims to learn a embedding space where instances with similarities are pulled closer while dissimilar instances are pushed far away. This goal is always accomplished by optimizing the contrastive loss function. Another type of contrastive learning is combined with clustering (Caron et al. (2021) ; Li et al. (2021) ). In this kind of framework, the concept of prototype is introduced to guide the formation process of the embedding space. The prototype contrastive learning framework produces more separate clusters in the embedding space.Shape bias Shape bias is one significant difference between CNN-based models and humans where CNNs has texture bias while human has shape bias (Geirhos et al. (2022; 2021) ). In supervised learning, many studies show that enhancing shape bias of CNN through shape-related data augmentation (Wen et al. (2023) ) can result in improvement of prediction robustness and domain generalization (Geirhos et al. (2022) ; Li et al. (2023) ). For self-supervised learning, although the existence of texture bias of CNN is also identified (Geirhos et al. (2020) ), the approaches and benefits of enhancing shape bias in self-supervised learning is still unclear, which motivates our work. 
preliminaries: 
 Instance-wise contrastive learning In instance-wise contrastive learning, e.g. MoCoV2, given training images X = {x 1 , x 2 , ..., x n } and an encoder f (•) encoding X to embedding V = {v 1 , v 2 , ..., v n }, i.e. f (x) = v, the optimal encoder f is found by optimizing the InfoNCE (van den Oord et al. ( 2019)) loss function:L InfoNCE = n i=1 -log exp(v i • v ′ i )/τ ) r j=0 exp(v i • v ′ j /τ ) (1)where v ′ i is one positive sample of v i , v ′ j , j ̸ = i are negative samples of v i , and τ is the temperature. In MoCoV2, positive and negative samples v ′ are obtained by inputting train data to the momentum encoder f ′ (•), i.e. f ′ (x) = v ′ .Prototype contrastive learning In PCL, the embedding of training images obtained by momentum encoder are further clustered. The centroids of clusters are called prototypes. The optimal f (•) is found by optimizing the ProtoNCE (Li et al. (2021) ) loss function:EQUATIONwhere c p is the prototype of the cluster p that v i belongs to, c j , j ̸ = p are centroids of other clusters, and ϕ j is the concentration estimation of cluster j. The clustering is performed for multiple times with different number of clusters, therefore the loss for optimization is the mean of multiple runs.This section introduces our proposed contrastive learning framework, Shape Prototype Contrastive Learning (S-PCL), designed to incorporate shape silhouette of the object in the training phase so as to explicitly enhance the shape bias of the embedding space.Preprocessed shape silhouette In S-PCL, we choose to use the silhouette to represent the object shape which segregates the salient object and background, as shown in Figure .1. The silhouettes are generated offline as a part of data preprocessing. Studies show that human can instantly capture the shape of an object at minimal cost (Elder & Velisavljević (2009) ) and figure-ground representation is readily available to the human visual system (Lamme (1995); Zipser et al. (1996) ; Lee et al. (1998) ). This provides the ground for preprocessing to obtain the shape silhouette and input it directly into the network.Shape Prototype The term "shape prototype" refers to the representative embedding for a group of shape silhouettes. To find the shape prototype, we perform K-Means clustering. The set of shape silhouette masks, M = {m 1 , m 2 , ..., m n }, is encoded by the momentum encoder to the shape embedding set, U = {u 1 , u 2 , ..., u n }, i.e. u = f ′ (m). K-Means clustering is employed on the set U , producing K clusters and a set of centroids, S = {s 1 , s 2 , ..., s K }. The centroid of the cluster is representative for all shape embeddings in the cluster, thus is considered as the shape prototype. Subsequently, the shape bias is introduced through optimization using ShapeProtoNCE, which contrasts shape prototypes and the original image embeddings.ShapeProtoNCE Given the set of shape embedding U , the set of shape prototypes S, and the set of embeddings of original training images V , the ShapeProtoNCE is defined:EQUATIONwhere s p is the prototype of the cluster p to which u i belongs, s j , j ̸ = p are shape prototypes representing other shape clusters, and ϕ j is the concentration estimation of cluster j. Similar to the previous work (Li et al. (2021) ), we perform clustering for N times with different K i , i = 1, 2, ..., N and average the losses. Also, we find that combining InfoNCE and ShapeProtoNCE is beneficial. Therefore, the loss function we use is:EQUATIONS-PCL+X hybrid approach Taking the inspiration from the fact that humans learn new nouns based on the shape of objects during the infancy, and noticing S-PCL significantly speeds up the learning process but limiting the network later on, we decided to use S-PCL in the initial training stage to organize the embedding space based on global shape priors and then switch to the other state-of-the art instance-wise and prototype contrastive learning framework for further fine-tuning in the later stage. We find this sequential hybrid training strategy of "coarse-to-fine" and "shapeto-texture" to be beneficial. We denote the S-PCL warmed-up method as S-PCL+X, for example, MoCoV2 warmed-up by S-PCL is denoted by S-PCL+MoCoV2.The experiments are conducted on three different datasets: Imagenette, ImageNet-1001 , the subset of ImageNet-1K (Deng et al. (2009) ), and STL10 dataset (Coates et al. (2011) ). The commonly-used instance-wise MoCoV2 and PCL are served as baselines.The silhouette mask of the object is generated offline using TRACER (Lee et al. ( 2022)). The ResNet18 (He et al. (2016) ) is adopted as the encoder and the output dimension of the last fully-connected layer is 256-D. We follow the data augmentation methods in previous works (He et al. (2020) ; Li et al. (2021) ). We train the ResNet18 for a set of numbers of epochs {50, 100, 150, 200, 300, 400}. We use SGD as the optimizer, with a weight decay of 1e-4, a momentum of 0.9, and a batch size of 128. The initial learning rate is 0.3 for ImageNet-100 and STL10, 0.03 for imagenette, and they all follow the cosine learning rate schedule.In S-PCL training, the model is warmed-up in the first 20 epochs in the same way as stated in (Li et al. (2021) ). We set τ = 0.1, α = 10, and the number of clusters K = {2000, 4000, 6000}. In order to have fair comparison, we use the same hyperparameters as S-PCL for the PCL training. MoCoV2 uses the same τ = 0.1, α = 10. For all three frameworks, we set the number of negative samples r = 1024 when training on ImageNet-100 and STL10, and r = 128 when training on imagenette. When S-PCL serving as a warm-up approach, the first 100 epochs is trained by S-PCL framework. Then it will be trained using PCL or MoCoV2 following the same learning rate schedule in the subsequent epochs. Hyper-parameters are the same as above. 
linear classification: 
 We evaluate the learned representation on image classification tasks. During the inference phase, the shape silhouette is bot involved in order to ensure a fair comparison with other contrastive frameworks. A linear classifier is trained based on the frozen image representations using labeled data.For Imagenette and ImageNet-100, entire labeled training set is used and accuracy on the validation set is reported. For STL10, we use labeled training set and test on the test set. Detailed numerical results can be found in appendices. From Figure .2, it can be observed that the MoCoV2 and the PCL achieve a high accuracy, outperforming S-PCL on ImageNet-100 after 400 epochs of training, however the improvement rate is slow during the training. In contrast, S-PCL attains a relatively high accuracy within the first 100 epochs but plateaus and even performs worse than others in the later stage, suggesting that the shape, as a coarse visual cue, is beneficial in the early training while it is the finer cue such as local textures that pushes the performance to the next level in the later training. We can observe the similar trend that S-PCL based models converge faster on STL10 and imagenette as well.Importantly, the results of S-PCL+MoCoV2 and S-PCL+PCL, trained using the "shape-to-texture" sequential regime, are encouraging. They achieve high accuracy levels comparable to MoCoV2 and PCL but with only half the training epochs, indicating that shape priors accelerate early learning. Notably, after 400 epochs of training, our proposed regime ends up having the highest accuracy. Overall, our experimental results demonstrate that learning shape priors in the early stages can effectively bootstrap the training of other contrastive learning frameworks, such as MoCoV2 and PCL.We find it necessary to confirm our approach introduces shape bias to the model and it is the shape bias that leads to such a result. To this end, we quantitatively evaluate linear classifiers on styletransferred images and present qualitative evidence in the form of sensitivity maps computed by SmoothGrad (Smilkov et al. (2017) ). .1 and 2 report results, note that results of the style-transferred set are the mean of 3 different styles. On both datasets, the models trained or warmed-up by S-PCL end up having higher top-1 and top-5 accuracies on style-transferred validation even though the accuracies on the original sets are much lower, indicating that S-PCL produces models that are more robust against the drastic style and texture changes, i.e. exhibiting stronger shape bias. The sensitivity map reveals the image regions that contribute significantly to the prediction. We apply SmoothGrad to the linear classifier learned in the inference phase by each contrastive learning framework for 400 epochs on ImageNet-100 dataset and present the resulting sensitivity maps.Sensitivity on original images First, we evaluate sensitivity maps on original images, as depicted in Figure .4. Example (a) features a spider with a complex shape as well as rich textures, while the other three examples showcase objects with relatively simple shapes but rich textures. Specifically, for example (a), only S-PCL appears sensitive to the entire body of the spider, indicating that shape plays a crucial role in the predictions made by ResNet18 pretrained with S-PCL. In the case of the other three examples, MoCoV2 and PCL exhibit a strong preference for specific textures, such as folds on the leather bag in (b), decorative patterns on the coffee cup in (c), and facial features of the animal in (d). In contrast, the sensitivity maps of S-PCL reveal distinct object silhouette boundaries, suggesting much greater attention to the contour of the object rather than its content. This implies that ResNet18 pretrained with S-PCL makes predictions primarily based on object shape, while MoCoV2 and PCL favor local features.Sensitivity on style-transferred images In addition to visualizing sensitivity maps on original images, we also apply SmoothGrad to style-transferred images to further evaluate the shape bias. The style-transferred images preserve the shape while exhibiting different textures. It's important to note that style transferring not only alters the object's texture but also introduces new textures in the background, leading to intense responses in the background within the sensitivity maps of MoCoV2 and PCL, as shown in Figure .5. This further emphasizes that MoCoV2 and PCL exhibit a very  
discussion: 
 In this paper, we proposed a new contrastive learning framework called Shape Prototype Contrastive Learning. Shape prototypes, learned across images over time, provided a "memory mechanism" to influence the embedding space in contrastive learning. During the learning phase, shape silhouettes of objects are provided with the training images as augmented input to the systems. During the inference phase, the shape silhouettes are not provided. This ensures that the proposed framework can be compared on equal ground against other contrastive learning paradigms in object recognition performance. We found that having global shape prototype priors during training greatly accelerates the learning process in the early stage but hinders learning in the later stage (Figure .2). This suggests that the stereotypical shapes of objects can help guide the deep networks quickly into a better embedding space for distinguishing semantic categories more separable, but then later on, might actually hinder the network from learning the finer and subtler discriminating features useful for better recognition. Hence, we proposed a hybrid approach, using the shape prototype priors at the beginning but relinquishing their influence in the later stage. Using S-PCL as a warm up training procedure can effectively bootstrap the training of other contrastive learning frameworks, such as MoCoV2 and PCL. Our finding on the enhancement of categorical learning by auxiliary input of global shapes is reminiscent of the facilitation of global shapes in infant learning of lexical categories. Since self-supervised contrastive learning has been considered relevant for understanding human visual learning, our framework might thus share some underlying mechanisms on the role of global shape perception in representation learning.We found our framework of providing the global shape input and the modulation learning the embedding space with remembered shape prototypes increase the global shape sensitivity of the deep networks, as evident in the increased "shape bias" or robustness to style transfer in object recognition as shown in Table .1 and 2. Feature attribution sensitivity analysis, as shown in Figure.4 and Figure.5 , revealed that the S-PCL-trained networks indeed pay more attention to the contours of objects, even during inference when the silhouette is not provided as input to the network. This observation suggests that the network has learned to pay more attention to the global shapes or the boundaries of the objects because of this training with shape prototypes.Figure-ground segregation is known to be computed in one of the earlier stages in the visual system, and figure-ground representation resulting from motion, luminance, color, and stereo are readily available to the visual systems even for infants. Therefore, figure-ground representation can be and probably should be used in representational learning for object recognition. In this paper, for evaluating the benefits of S-PCL training, we did not use figure-ground representation during inference. However, figure-ground representation is likely used together with other signals in object recognition in animals. Thus, it would be interesting in future study to investigate whether recognition performance can be enhanced when figure-ground is provided as auxiliary input during inference as well. It is also important to investigate the nature of the shape prototypes that have been learned and how and when they can continue to help object recognition. To better relate S-PCL to compositional learning, it will be useful to investigate the impact of having shape clustering at multiple layers of the networks rather than just at the highest level. 
acknowledgement: 
 This work was supported by an NSF grant CISE RI 1816568, NIH R01 EY030226-01A1 and a graduate student fellowship from CMU Computer Science Department. We also thank NSF AC-CESS Allocations (Project Number CIS230221) for computational resource support and Ziqi Wen for helpful discussion. We carry out ablation studies on different combinations of loss functions on the STL10 dataset.Results are reported in Table .7. We find that using only ShapeProtoNCE results in lower accuracy on the original test set than combining InfoNCE and ShapeProtoNCE. However, it yields much higher accuracy on the style-transferred test set. This further suggests that using ShapeProtoNCE makes the model less prone to texture changes, while using InfoNCE makes the model prone to texture changes. We also explore the effect of the order of different training frameworks. The experiments are conducted on STL10 dataset. We find that using S-PCL first ends up with higher accuracy on the original data. Results on style-transferred data suggest that incorporating shape in the train does increase the shape bias regardless of the order yet we cannot tell which order is better from the perspective of improving shape bias given the current result. Li et al. (2021) ) between the clustering assignments and ground-truth labels for ImageNet-100. As the Shape-PCL performs clustering on the shape mask and the input data of clustering is not consistent with PCL, so the AMI of Shape-PCL is not comparable with that of PCL. Comparing the AMI curves of PCL and S-PCL+PCL, we can also conclude that incorporating shape can accelerate learning. Interestingly, we observe the AMI of Shape-PCL would drop when trained for longer timer, this is probably the key to understand why Shape-PCL plateaus and worth studying in the future work.The dataset used here is reorganized by adjusting the train-validation split to a 7:3 ratio to intensify the challenge 
 
 Appendix:  



Reviews: review: title: title: Interesting method to introduce shape bias, with some scalability limitations.
main_review: main_review: The paper describes an application of contrastive learning to instill shape bias in neural networks. It utilizes clustering to create a shape prototype, which is then employed by a contrastive mechanism to enforce shape bias. This shape bias serves as a means to counteract the networks' natural tendency to focus on texture.

The paper provides an intriguing mechanism that is loosely based on how the human brain develops shape bias for object classification. I believe the paper offers a clear explanation of the method. The experiments seem to demonstrate that the method effectively imposes shape bias on the evaluated neural networks.

The main weakness is that scalability appears challenging; for larger datasets, generating such extensive bias masks may prove problematic. Perhaps a minor comment is that some of the tables would have been more effectively presented as graphs. Since they describe performance over epochs, a graphical representation might have been more illustrative.
strength_weakness: strength_weakness: I think the paper has a good demonstration of why shape bias is beneficial in training deep neural networks. It also provide some experiments that demonstrate the claims in different aspects.
title: title: A not entirely convincing hack
main_review: main_review: This manuscript aims to bias contrastive learning DNNs towards using shape features by using an explicit estimate of the figure ground segmentation. Concretely, the authors extended prototype contrastive learning to use prototypes that are not based on the original images but based on shape masks that are passed through the momentum encoder corresponding to the network. The shape masks are generated by TRACER, another existing DNN. By itself the proposed method decreases performance on linear probe evaluations of the learned representations. It does seem to lead to faster training in the beginning though and does improve performance on style transferred images by a few %. 

Overall, this manuscripts is around the border for me. The topic seems reasonably close to realigning human and machine perception as the authors are trying to improve shape bias, a well known point where humans and DNNs do not align currently. On the other hand, the shape masks passing through (moment encoder of) the same network seems decidedly not humanlike to me and more like a hack and the results for improving the shape bias are somewhat underwhelming. Especially, different contrastive learning objectives are needed to achieve good clean image performance. Additionally, the concrete results are a little thin, some of the details are lacking, most of the references are brocken, etc. Overall the presentation quality is not great.
strength_weakness: strength_weakness: it is at least the right topic and there is some positive results here in terms of speed of training.