Unicom: Universal and Compact Representation Learning for Image Retrieval
Authors: Xiang An
Jiankang Deng
Kaicheng Yang
Jaiwei Li
Ziyong Feng
Jia Guo
Jing Yang
Tongliang Liu
Venue: ICLR 2023 poster (2023)
Keywords: Cluster Discrimination
Image Retrieval

Abstract: Modern image retrieval methods typically rely on fine-tuning pre-trained encoders to extract image-level descriptors.
However, the most widely used models are pre-trained on ImageNet-1K with limited classes. The pre-trained feature representation is therefore not universal enough to generalize well to the diverse open-world classes. 
In this paper, we first cluster the large-scale \laion{} into one million pseudo classes based on the joint textual and visual features extracted by the CLIP model. Due to the confusion of label granularity, the automatically clustered dataset inevitably contains heavy inter-class conflict. To alleviate such conflict, we randomly select partial inter-class prototypes to construct the margin-based softmax loss. To further enhance the low-dimensional feature representation, we randomly select partial feature dimensions when calculating the similarities between embeddings and class-wise prototypes. The dual random partial selections are with respect to the class dimension and the feature dimension of the prototype matrix, making the classification conflict-robust and the feature embedding compact. Our method significantly outperforms state-of-the-art unsupervised and supervised image retrieval approaches on multiple benchmarks. The code and pre-trained models are released to facilitate future research \url{https://github.com/deepglint/unicom}. 

Published as a conference paper at ICLR 2023
UNICOM: UNIVERSAL AND COMPACT REPRESENTATION LEARNING FOR IMAGE RETRIEVAL
Xiang An'. Jiankang Deng² Kaicheng Yang', Jiawei Li", Ziyong Feng', Jia Guo Jing Yang', Tongliang Liu
DeepGlintt HHaawei, InsightFace, *University of Cambridge, University of Sydney xi @nganndeepglind com, j iiankngdengg gma com
ABSTRACT
Modern image retrieval methods typically rely on fine-tuning pre-trained encoders extract image-level descriptors. However, the most widely used models are pre-trained on ImageNet-IK with limited classes The pre-trained feature rep- resentation is therefore not universal enough to generalize well to the diverse open-world classes. In this paper, we first cluster the large-scale LAION 400M dataset into one million pseudo classes based on the joint textual and visual fea- tures extracted by the CL.IP model. Due to the confusion of label granularity, the automatically clustered dataset inevitably contains heavy inter-class conflict. To alleviate such conflict, randomly select partial inter-class prototypes to con- struct the margin- based softmax loss. To further enhance the low-dimensional randomly select partial feature dimensions when cal-
feature representation, culating the similarities between embeddings and class-wise prototypes. The dual random partial selections with respect to the class dimension and the feature dimension of the prototype matrix, making the classification conflict-robust and the feature embedding compact. Our method significantly outperforms state- of-the-art art unsupervised and supervised image retrieval approaches on multiple benehmarks. The code and pre-trained models are released to facilitate future research /ttps::/gittiii hig.bb.hhbb...o com deep//eepppooo/////// 1 int /unt//niii inom
INTRODUCTION
Modern image retrieval methods (Lim al. 2022; Roth et al. 2022 Kim al. 2022 Ermolov et al.
2022. Patel et al.) 2022) can be roughly decomposed into two major components: (1) the encoder
(e.g., Convolutional Neural Networks (Szegedy et al. 2015) He al. 2016) or Vision Transformer
(Tourron et al. 2021: Dosovitskiy et al. 2021)) mapping the image to its compact representation and
(2) the loss function MMusgrave et al. 2020) grouping the representations of similar objects while pushing away representations of dissimilar objects in the embedding space. To train the encoder, networks pre-trained on crowd-labeled datasets (c.g., ImageNet (Deng et al. 2009)) are widely used for fine-tuning (Wang al.) 2019; Kim et al. 202T). However, ImageNet only contains 1,000 pre-defined object classes. The feature representation learned from ImageNet is not universal enough to generalize to diverse open-world objects.
Even though fully supervised pre-training can benefit from a strong semantic learning signal for scalable because manual annotation of large-scale training data is time-consuming, costly, and even infeasible. By contrast, self-supervised pre-training
each training example. supervised learning
methods (He et al. 2020 2022, Radford et al. 2021 Jia al. (021) can be easily scaled to billions of unlabeled examples by designing an appropriate pretext task, such as solving jigsaw puzzles (Noroozi & Favaro 2016), invariant mapping (Chen He 202T), and image-text matching (Radford)
et 202 Jia et al. 202T). Among them, CLIP (Radford et al. 202T) has recently demonstrated
success across various downstream tasks (e.g., image retrieval and classification) due to superior feature representation empowered by image-text contrastive learning. Specifically, CLIP aligns the visual and textual signals of each instance into a unified semantic space by cross-modal instance
denotes corresponding author.

---

Published as a conference paper at ICLR 2023
Margin-based softmax loss
90
Es W3
W2 =
[w,* Ws
80
Rocalig c1
70
es
Ws
Ws
60-
Random Fenture
550
Kunkom 336
40-
Unaom Vil Bi22
STML
pick positive
W
W
UNMML
30
SAN
Random Class
OcopC uster
128
256
512
W
Embeddinn Dimension
(a) Performance comparison on CUB
(b) The proposed Unicom
Figure 1: (a) Accuracy in Recall@ versus embedding dimension on the CUB dataset. The proposed Unicom is only trained on the LAION 400M dataset without any manual annotation. (b) The proposed Unicom employs two random selections along the class dimension and the feature dimension to alleviate inter-class conflict and achieve compact representation, respectively.
discrimination. Nevertheless, the instance discrimination method used by CLIP can hardly encode the semantic structure of training data, because instance-wise contrastive learning always treats two samples as a negative pair they are from different instances, regardless of their semantic similarity. When thousands of instances selected into the training batch to form the contrastive loss, negative pairs that share similar semantics will be undesirably pushed apart in the embedding space.
To handle the limitations of instance discrimination, cluster discrimination has been proposed for deep unsupervised learning through jointly learning image embeddings and cluster assignments. Learning representations with clusters will pull similar instances together, which is beneficial for capturing semantic structures in data. DeepCluster (Caron et al. 2018) performs iterative clustering by k-means and classification by cross-entropy loss, while SeLa (Asano al. 2020) proposes to solve an optimal transport problem for balanced assignment. However, both DeepCluster and SeLa need labels be assigned offline in a batch mode with representations of all instances. To reduce the cost of batch mode clustering, ODC (Zhan et al.] 2020), SwAV (Caron et al., (2020), and CoKe (Qian al. 2022) apply online clustering to avoid the multiple iterations over the entire dataset. Despite improved efficiency, the online clustering method still suffers from the collapsing problem (i.e., dominant cluster includes instances from multiple classes or most of the instances).
In this paper, we aim at boosting the semantic embedding power of the CLIP model by introducing novel cluster discrimination approach. We first conduct one step of off-line clustering by using the image and text features from a pre-trained CLIP model (Radford et al.) 202I). Due to the limited discrimination power of the CLIP model, the pseudo clusters contain heavy inter-class conflict. Instead of optimizing the cluster assignment (Qian al. 2022 we focus on how to robustly train classifier on the automatically clustered large-scale data. More specifically, we explore two random selections on the prototype matrix W E Raxk when preparing the classification loss (as illustrated in Fig. l(b)). The first one is with respect to the class dimension (k), and only part of negative prototypes are selected for inter-class comparisons. which helps alleviate inter-class conflict. The second one is with respect to the feature dimension (d), and only part of features are randomly selected to construct the classification loss, enhancing the representation power of each neuron and making feature representation compact for the efficient image retrieval task. Concurrent with our work, partial selection mechanisms along class and feature are separately proposed in (An et al., 2021 2022) and (Xu et al. (022) to accelerate model training and enhance locally distinguishable features. Both of their experiments are conducted on cleaned face recognition datasets. By contrast, target at universal and compact representation learning from automatically clustered large-scale data. The main contributions of our paper are the following:
We propose novel cluster discrimination method for universal and compact representation learning. In the clustering step. we employ both image and text features from the pre-trained CLIP model. In the discrimination step, explore two random selections along class and feature, which can potentially alleviate inter-class conflict and improve the feature compactness, respectively.
2

---

Published as a conference paper at ICLR 2023
For both zero-shot learning tasks (e.g., linear probe and unsupervised image retrieval) and transfer learning tasks (e.g., ImageNet-13 IK classification and supervised image retrieval), the proposed random negative prototype selection for conffict-rbbust cluster discrimination can significantly boost the representation power compared to the instance discrimination based model (e.g., CLIP).
RELATED WORK
Visual Model Pre-training. Model pre-training for visual recognition be categorized into three main groups: (1) supervised pre- mre-trining on datasets with manually annotated class labels (e.g..
ImageNet-IK/-21K (Deng al.
2009) and JFT-300M/-3B (Dosovitskiy et al. 2021 Zhai al.
). (2) weakly-supervised pre-training by using hashtag (Mahajan et al. 2018 Singh et al. 2022)
or text descriptions (Radford et al. 202T Jía et al. 202TD, and (3) unsupervised pre-training Chen
et al., 2020: He et 20.0 Caron et al. 2018). Since supervised pre-training relies on expensive manual annotations, we focus on annotation-Iree pre-training which has the advantages of being easily scaled to billions of training images and being able to learn universal feature representations for downstream tasks.
Instance and Cluster Discrimination. Instance discrimination (Chen et al. 2020: He et al. 2020 Radford et al. 202T) is realized with a contrastive loss which targets at pulling closer samples from the same instance while pushing away samples from different instances. Despite the impressive performance, instance- wise contrastive lcarning can not capture the semantic information from the training data because it is trained to ignore the similarity between different instances. Cluster discrimination (Caron et al. 2018: Zhan et al., 2020 Li et al., 2020a) is processed with iterative steps: the clustering step to assign pseudo class labels for each sample, and then the classification step to map each sample to its assigned label. Since one cluster has more than one instance, learning representations with clusters will gather similar instances together, which can explore potential semantic structures in data. As representative work, DeepCluster (Caron et al. (018) adopts a standard k-means for clustering, but it contains degenerate solutions. To this end, recent research work (Asano et al., 2020; Caron et al.) 2020; Qian et al.) (202)) focuses on improving the label assignment during clustering but employs a standard cross-entropy loss during discrimination. In this paper, we only employ one step of off-line line clustering but design a robust classifier to achieve good feature representation when training on the automatically clustered large-scale data.
Image Retrieval, Image retrieval task typically relies on fine-tuning pre- ire-trained visual models (Seggdy et al. 20155 He et al. 20166; Dosovitskiy et al., (022]) and can be divided into two learning categories: supervised and unsupervised metric learning. For supervised metric learning, pair-wise loss (Hadsell et al. 2006 Schroff et al. 2015 Sohn 2016) and cross-entropy loss (Zhai & Wu 2019)
Deng et al. [2019] Sun et al.] [020 Qian et al. 2019) extensively studied and recent bench- marking
results (Musgrave et al. (2020) indicate that the margin-based softmax loss (e.g., ArcFace (Deng et al.
2019h) can achieve state-of-the-ar performance. For unsupervised metric learning, pseudo labeling methods are employed to discover pseudo classes by applying k-means clustering (Kan et al. 2021
Li et al., (2220)), hierarchical clustering (Yan al.) 2)22]. random walk (Iscen et al. [2018). and class-equivalence relations (Kim et al., (2022) to unlabeled training data. In this paper, we focus on universal and compact feature embedding for both unsupervised and supervised image retrieval task.
METHODOLOGY
3.1
PRELIMINARIES OF INSTANCE AND CLUSTER DISCRIMINATION
Given training set X {-1, Z2, ..., Fn including n images, feature representation learning aims at learning function f that maps images X to embeddings E fer, c,, en with e; f(x),, such that embeddings can describe the similarities between different images.
Instance discrimination achieves this objective by optimizing a contrastive loss function defined as:
exp(e!Tee (..)))))))))))
Linstance
log
Σ5= exple Tee)
3

---

Published as a conference paper at ICLR 2023
fentures
k classes
Image Encoder
Clustering
Random Class Selection
aquarium wodding
Random
nobding couple while pigenas
Text Encoder
Feature
Embedding
Prototype Matrix
copple of bridee with flowess
Margin-based Softmax Loss
(a) Multi-modal Clustering
(b) Discrimination
Figure 2: Illustration of the proposed method. (a) The multi-modal clustering includes one off-line step of k-means on features from image and text produced by pre-trained CLIIP model (Radford al., (021). (b) Using the assigned clusters pseudo-labels, we propose conflict-robust and representation-coppar classification method through random class and feature selection along the two dimensions of the prototype matrix.
where e; and et are positive embeddings of the instance i, and e' consists of one positive embedding of and its negative embeddings from other instances.
By contrast, cluster discrimination for representation learning consists of two main phases: clustering and discrimination. The clustering step assigns each instance pseudo class label that will be subsequently used as supervision to train classifier in the discrimination step. Following this, automatic clustering on the features e; = f(x;) (zz is first performed to obtain & clusters and the centroid w; is viewed as the prototype of i-th cluster. Then, the training data {z;] are partitioned into k classes represented by prototypes W [w: J/ With pseudo labels and centroids obtained from the clustering step, cluster discrimination can be implemented by optimizing a standard softmax classification loss as:
exp(w e.)
Leluster
log
2j=1 exp(wfee)
i=1
where e is the embedding of the image 2; and F; belongs the class represented by w; By comparing Eq. and Eq. [2 we can observe the difference that instance discrimination employs an augmented feature eç calculate the similarities while cluster discrimination uses prototype wi.
3.2 MULTIMODAL CLUSTERING
In this paper, focus the standard clustering algorithm, k-means, which takes set of vectors input and clusters them into & distinct groups based on the nearest neighbor criterion. To seek a better representation.
combined the image and text features produced by the pre-trained CLIP model (Radford et al. 2021) due to their mutual complementary nature. The clustering step jointly learns d x k centroid matrix W and the cluster assignments y, of each image v; by solving the following problem:
min 1 Σ min [ W WeR (00,11
s.t.
y: 1k 1,
where f(x) is the image feature embedding by the image encoder f and f'(x.() is the text feature embedding by the text encoder f', + is feature ensemble function, W E Raik is the centroid matrix, y: (0, }}* is single label assignment constrained by y. 1 = 1, and 1 is 1-vector of size k. In this work, employ the simplest feature ensemble function, that is, averaging the image and text features, as the CLIP model provides an aligned visual-textual representation.
Considering that iterative clustering and discrimination are time-consuming. step of off- line clustering in this work. Aided by the efficient feature quantization (Johnson et al. 2019), the large-scale LAION 400M dataset can be clustered within 10 minutes using the embedded image and text features. The only appee-eer we consider here is the cluster number Even though the clustering step is straightforward, the automatically clustered large-scale dataset inevitably
only employ one
4

---

Published as a conference paper at ICLR 2023
(a) Wedding+Flower
(b) Wedding+Pair
(c) Wedding+ Place
Inter-class conflict between the automatically- classes The class name is given based on the observation of images and texts. Inter-class conflict exists due to specific granularity definitions and multi-label signals in image.
Figure 3:
confronts inter-class conflict due to multi-label signals in one image and specific definition of class gramularity (as illustrated in Fig. 3). For instance, the bird pair in Fig. 3(b is clustered into wedding pair, which will be confficted with another specific category of bird. In addition, the close-up capture of wedding flowers in Fig. 3(a) also exists in the class of wedding place, where flowers are the most popular decoration.
3.3 CONFLICT-ROBUS AND REPRESENTATION--OMMEN DISCRIMINATION
After the clustering step. we can employ vanilla classification loss (e.g.. softmax) to learn the feature representation. For the softmax loss given in Eq. 2 the derivatives to a class- wise prototype wy E I and sample embedding feature e; Rdd are:
ac = E(p 18 == Me,. Ow. 1=1 where 6 is the batch size, k is the class number, and Pij probability of the sample embedding belonging to the prototype wy, 1(.) is the indicator function which is 1 when the statement true and otherwise. In Eq. (] the derivative of the prototype is a "weighted sum" over sample features from the mini-batch, and the derivative of the sample feature is a "weighted sum" over the prototypes of all classes. If conflicted classes exist, wrong gradient signals from these confticted negative prototypes will affect the update of model parameters.
Oc
k
-- (pij (Pij j=1 =
1 18: = j})
(4)
de,
e es the normalized
To this end, we propose a random negative prototype selection to efficiently construct a negative prototype subset from the entire negative prototypes. Therefore, the derivative sample embedding feature is:
Oc de,
((1 - ---
yur
Σ Dj w) JEB
where pt and w denote the probability and prototype of the positive class, P, and wy, refer to negative probabilities and prototypes, S is a subset of all negative classes and one positive class, [S] k T1, and r1 E [0, 1] the sampling ratio. Even though all class-wise prototypes are still maintained throughout the whole training process, only positive prototypes and subset of negative prototypes are selected and updated each iteration. Therefore, the inter-class conflict will be reduced as the possibility of sampling conflict negative prototype is directly decreased by rı-
To achieve a compact representation for efficient image retrieval, previous methods (Babenko & Lempitsky 2015 Tolias et al. 2016) adopt Principal Component Analysis (PCA) on independent set for dimension reduction. To reduce the descriptor dimension to d', only cigenvectors corresponding retained. However, the representation produced by PCA is sub-optimal because it is a post- prot-processing step detached from the target task. To this end, we propose a feature approximation strategy by randomly selecting subspace features construct the classification loss: exp((T, @ w.) ( ee) Ejes (T, @ wy (TT,(e)))))))
to d' largest eigenvalues
Lunicom
log
where I, E {0, 13d is a random binary vector at the iteration step of t, the non-zero element ratio of T is r2 [0, 1), @ denotes element-wise product. Different from the well-known regularization
5

---

Published as a conference paper at ICLR 2023
technique, Dropout (Srivastava et al. 2014), our random feature selection I is same for all training samples within the mami-batch. I' is applied to both feature e, and prototypes w/, thus the dimension of derivatives in Eq. 4 decreases to d'. By contrast, Dropout is independently applied to each individual feature e; within the mini-batch by setting specific ratio r3 E [0. 1] of neurons to 0 and enlarging the rest neurons by 1/(1 rz)- The dimension of derivatives in Eq. 4 is still d. Therefore, the sub-feature optimization in the proposed random feature selection can not be completed by directly calling the Dropout function. Since the binary vector varies at different iterations, different sub-features are trained and sub sub-gradients are calculated. This leads to a solution that each embedding neuron contains the similarity representation power.
The schematic of the proposed method is in Fig. ((b) As shown, the prototype matrix W is maintained during the whole training process, but only part of the classes (k' = k r1) and features (d' = d * r2) are randomly selected to construct the softmax loss. The first random selection along the class dimension for conflictrobuust learning to achieve universal representation and the second random selection along the feature dimension is for feature compression required by efficient retrieval. Therefore, we name our method UNIversal and COMpact (UNICOM) representation learning.
in the memory at the dimension of
4 EXPERIMENTS
4.1 IMPLEMENTATION DETAILS
Unless otherwise specified, all VIT models in experiments follow the same architecture designs in CLIP, and are trained from scratch for 32 epochs on the automatically clustered LAION 400M dataset (Section 3.2) with cluster number &- = 1MM During training, we randomly crop and horizontally flip each image to get the input image with 224 x 224 resolution. We set the random class sampling ratio T1 as 0.1 in the pre-training step. The training conducted on 128 NVIDIA V100 GPUs across 16 nodes. To save memory and scale up the batch size, mixed- precision and gradient checkpoint are used. We use AdamW (Loshchilov & Hutter 2018) as the optimizer with an initial learning rate of 0.001, and a weight decay of 0.05. We employ margin-based softmax loss, ArcFace (Deng et al. 2019] 2020), for both pre-training and image retrieval tasks. The margin value is set to 0.3 and the feature scale is set to 64. For supervised retrieval, we follow the data-split settings of the baseline methods (Patel et al. 2022 Ermolov et al. 2022) to fine-tune models.
4.2 COMPARISONS ON FEATURE REPRESENTATION LEARNING
In this section, we first compare the performance of the proposed method and other baseline models (i.e., CLIP and OPEN-CLIP) on the linear probe and unsupervised image retrieval. Specifically, after the training the automatically clustered IM classes, we fix the backbones of our models. For the linear probe task, we learn an additional FC layer for classification on each test set. For unsupervised image retrieval, we directly use the embedding features for testing. Then, fine-tune the pre-trained models for supervised image retrieval on each image retrieval dataset.
Linear Probe. Following the same evaluation setting CLIP (Radford al. 2021 we freeze the pre-trained models on LATON 400M dataset and only fine-tune the last linear classification layer. We report the linear probing performance over 13 datasets in Tab. [ The proposed conffict-robust cluster discrimination method significantly outperforms the CLIP and OPEN-CLIP (Ilharco et al. 2021 models. Notably, our ViT B/32, VIT B/16, and ViT L/14 models surpass counterparts of OPEN-CLIP by 3.6%, 2.7% and 1.4% on average with the same training data, indicating that the proposed cluster discrimination can enhance the representation power over instance discrimination.
Unsupervised Image Retrieval In Tab. 2 we compare the performance of unsupervised image retrieval by directly using the pre-trained models for feature embedding. The GL.Dv (Weyand et al. 2020) employs mean Average Precision @ 100 (mAP@ 100) as the evaluation metric, while other datasets use Recall@11 Our ViT L/14 model achieves 69.9% average result across image retrieval datasets, surpassing the OPEN-CLIP counterpart by 7.5% and even outperforming the larger OPEN-CLIP model ViT H/14 by 5.4%. The reason behind this significant improvement is that the proposed cluster discrimination can capture the semantic structure in data, which is crucial for the image retrieval task. In Fig. [[a]. we compare our method with the ofate-of-the-ar unsupervised image retrieval approach, STML. (Kim al. 2022), under different dimension constraints on the

---

Published as a conference paper at ICLR 2023
Table I: Top-1 accuracy(%) of linear probe 13 image classification datasets. The proposed cluster discrimination significantly outperforms OPEN-CLIP (Ilbarco et al. 2021 on average by using the same training data (i.e., LAION 400M). "CLIP-R" denotes testing the public CLIP-ViT models in our code base. "-336" refers to one additional epoch of pre-training at higher 336 x 336 resolution boost performance.
CIFARIO
CIFAR100
Caltech101
Flwwers
Food101
Birdsnup
SUN397
Aireralt
EuroSAT
ImageNet
Cars
DTD
Pets
VIT B/32
95.1
80.5
93.0
81.8
96.9
88.8
58.5
76.6
76.5
52.0
90.0
97.0
76.1
81,8
VIT B/16
96.2
83.1
94.7
86.7
98.1
92.8
67.8
78.4
79.2
59.5
93.1
97.1
80.2
85.1
L/14
98.0
87.5
90.9
99.2
95.2
77.0
82.1
69.4
95.1
98.2
83.9
88.8
VIT L/14-336
97.9
87.4
91.5
95.9
79.9
82.2
83.0
95.1
98.1
85.4
89.5
VIT B/32
96.0
82.5
86.0
97.8
92.7
61.1
78.4
58.9
93.0
95.3
75.3
83.9
VIT B/16
96.0
82.5
94.1
86.0
97.8
92.7
69.5
79.1
78.4
58.9
93.0
95.3
79.6
84,8
VIT L/14
98.1
87.2
96.0
90.7
99.2
95.3
77.8
81.5
80.9
68.0
94.9
96.7
84.1
88.5
L/14-336
97.8
87.1
91.4
99.2
95.9
80.9
82.2
82.4
71.2
95.1
96.8
89.3
B/32
95.3
82.2
93.3
87.5
96.5
86.2
61.4
75.3
78.8
52.4
88.0
96.5
73.8
82.1
VIT B/16
84.0
91.8
98.1
81.6
90.0
96.2
78.5
85.4
VIT L/14
97.9
87.9
95.5
93.6
98.8
93.3
78.0
81.0
83.0
64.4
93.3
97.1
81.5
88.1
VIT B/32
96.8
86.6
94.6
93.3
98.5
85.8
70.2
74.6
78.0
70.7
93.1
96.8
75.0
85.7
VIT B/16
97.3
87.7
95.1
94.3
98.9
91.2
79.3
77.
81.2
73.4
93.9
97.0
79.1
88.1
L/14
98.5
90.8
95.7
94.6
99.3
93.4
82.4
80.0
82.2
74.5
94.2
96.7
81.8
89.5
IJ14-336
98.5
90.7
95.1
85.1
82.0
78.1
94.5
97.2
82.7
90.2
Table 2: Performance of unsupervised image retrieval on 7 image retrieval datasets. The proposed conflict-robust cluster discrimination significantly outperforms OPEN-CLIP on average by using the same training data.
CUB
Cars
SOP
In-Shop
INaturalist
VehicleID Medium
GLDv2
Average
Small
Large
Private
Public
B/32
56.7
79.0
60.5
45.4
53,0
54.8
52.2
44.6
7.5
7.5
46.1
B/16
66.1
85.2
63.2
56.1
63.1
55.1
50.9
43.8
8.4
10.6
50.3
76.0
90.3
65.6
62.9
72.9
62.4
58.9
51.8
12.1
13.6
56.7
I/14-336
77.3
67.8
66.3
76.8
64.1
60.3
53.8
17.0
15.6
59.0
B/32
62.3
89.2
65.9
64.6
54.9
71.0
67.1
9.17
8.4
55.2
OPEN
B/16
71.4
92.9
68.7
74.2
64.1
73.3
70.1
63.7
12.1
11.0
60.2
L/14
79.4
94.9
70.6
77.1
71.0
72.0
69.1
62.0
14.5
13.8
62.4
H/14
83.1
95.7
72.7
78.8
77.0
72.7
69.7
61.9
17.7
15.3
64.5
B/32
83.7
95.9
70.0
72.8
64.6
74.9
72.0
65.4
15.1
13.3
62.8
Ouss
86.5
96.8
70.4
74.6
73.6
74.5
70.6
58.7
18.7
17.2
64.2
L/14
88.5
96.9
72.7
83.6
83.7
80.2
74.6
21.1
20.1
69.9
L/14-336
89.2
97.3
74.5
86.7
81.0
84.1
81.4
75.6
23.2
22.0
71.5
CUB dataset. We set the random feature selection ratio r2 as 0.5 for additional training epoch the LAION 400M dataset. Then, we select the first 256-D, 128-D, 64-D, and 32-D features for testing. STML employs ImageNet-IK pre-trained GoogleNet (Szegedy al. 2015) and then explores unsupervised training on the CUB dataset. Even though our ViT-based model is only trained on the automatically clustered LAION 400M dataset without any further training on the image retrieval dataset, our method outperforms STML (Kim et al.) 2022 by large margin across different test dimensions, indicating the superiority of the proposed random feature selection for compact feature representation learning.
Fine-tune for ImageNet-1K Classification. Tab. we compare our method with state-of-the-an supervised and weakly supervised pre-training (Dosovitskiy et al. 2021 Zhai et al.) 2022) Singh et al.) 2022) in transfer-learning experiments on ImageNet-Ik. Our models consistently outperform OPEN- CLIP models in the Top-1 accuracy, showing the superiority of the proposed cluster discrimination. For ViT B/16, our pre- pre-training achieves 85.9%, surpassing both the supervised pre-training on IN-21K (84.0%) and the weakly supervised pre-training obtains 88.3%, outperforming ViT L/16 pre-trained on JFT 300M (87.8%) and ViT L/16 pre-trained trained on IG 3.6B (88.1 1.). The overall results in ImageNet-1K IK classification task show that models
IG 3,6B (85.3%). In addition, our ViT L/14
7

---

Published as a conference paper at ICLR 2023
Table 3: Transfer-leaming accuracy of models pre-trained the specified dataset followed by fine-tuning and testing on ImageNet. Model
Pre-train Pre-training Dataset
Resolution Pretrain
IN-IK
FLOPs
Finetune
Top-1 Accuracy
(B)
Supervised pre-training
WTL/33
Dosovitskiy
2021
IN-21k
224
384
81.3
54.4
VIT B/16
Dosovitskay
IN-21K
224
384
84.0
55.6
L/16
Dosoviiskyy
2021
IN-2IK
224
384
85,2
191.5
L/16
Dosoviskiy
2021
JFT 300M
224
512
87.8
362.9
VIT L/16
Zha el al.
2022
JFT 3B
224
384
88.5
191.5
Weakly supervised pre-fraining
VIT B/16 (Singh
2022
3.6B
224
384
85.3
55.6
VIT L/16 (Singh a
IG 3.6B
224
512
88.1
362.9
VIT B/32 OPEN-CLI CLIP OPEN-CLIP
LAION 400M
224
384
VIT
LAION
224
384
VIT L/14 OPEN-CLIP
LAION 400M
224
518
87.7
507.8
VIT B/32 Ours
LAION 400M
224
384
83.6
15.5
VIT B/16 Ours
LAION 400M
224
384
85,9
55.6
VIT L/14 Ours
LAION 400M
224
518
88.3
507.8
Table Performance of supervised image retrieval on 7 image retrieval datasets.
VIT-B/32
VIT-B/16
ViT-L/14
VIT-L/14-336
Previous SOTA
CUB
85.8
88.8
89.7
90.1
85.6
(Emolovv al.
2022
Cars
97.3
97.7
97.9
98.2
94.8
SOP
87.1
88.8
89.9
91.2
88.0
Paicl 2122
In-Shop
94.8
95.5
96.0
96.7
92.7
Ermolov 2022
INaturalist
72.8
82.5
85.4
88.9
83.9
VehiclelD-Sman
95.4
96.4
96.5
97.0
96.2
Patci 2022
PhiicelD-MMdium
94.1
95.1
95.7
96.1
95.2
Patci
VehicleID-LLage
93.6
95.0
95.4
96.0
94.7
Patci al.
Gl.Dv2-Private
32.6
35.7
36.1
36.4
32.5
ReaNetoo
GI.Dv2-Public
29.7
32.4
33.0
33.1
24.6
2021
very competitive they can achieve better or comparable accuracy even though the training data used by the competitors are much larger (e.g., JFT 3B and IG 3.6B).
Fine-tune for Supervised Image Retrieval. In Tab. we compare the proposed approach with the latest image retrieval methods (Patel et al. 2022 Ermolov et al. 2022) trained with vision transformer. our models, the random negative class selection ratio r11 set 1.0 as the training data is clean. Under different computing regimes, the proposed method consistently surpasses RA@ @A (Patel et al. (002) on the SOP, iNaturalist, and VehicleID datasets and outperforms Hyp-ViT (Ermolov al. 2022) on the CUB and In-shop datasets.
During fine-tuning
4.3 ABLATION STUDY
Encoder for Clustering, In Tab. we compare the results of linear probe and unsupervised image retrieval under image-based clustering and text-based clustering by using the pre-trained CLIP and OPEN-CLI models. As we can see, the text encoder is more powerful than the image encoder, and image and text signals are complementary as the joint clustering significantly outperforms each individual. By referring to Tab. and Tab. 2 the OPEN-CLIP ViT B/32 model achieves 82. % and 55.2% average results on the linear probe and unsupervised image retrieval tasks, while the proposed cluster discrimination method (r, = 0.1) significantly boosts the performance 84.1% and 61.1% by using the OPEN-CLIP image and text models for clustering. By using the CLIP image and text models for clustering, the performance and unsupervised image retrieval tasks. Therefore, we choose the CLIP models for clustering.
further increase 85.7% and 62.8% on the linear probe
Cluster Number. In Tab. 5 also compare the performance under different cluster numbers, i.e., 100K, 1M, and 10M, by using the CLIP image and text models. As can be seen, the best results can be achieved when the cluster number is set million, with the average image number per class being
8

---

Published as a conference paper at ICLR 2023
Table 5: Ablation study multi-modal clustering. ViT B/32 is used here for model training on the LAION 400M dataset, which is automatically clustered by different pre-trained models, We report the average performance of linear probe and unsupervised image retrieval.
Tasks
CLIP
OPEN-CLIP
Cluster Number by CLIP
Image
Text
Joint
Image
Text
Joint
100K
1M
10M
Linear Probe
84.4
85.3
85.7
83.9
84.0
84.1
75.9
85.7
83.6
Unsup. Retr.
61.8
62.3
62.8
58.9
60.1
61.1
53.2
62.8
60.7
Table 6: Ablation study on random negative class selection and random feature selection. ViT-B/32 used here and we report the average performance of linear probe and unsupervised image retrieval.
Tasks
Random Class Ratio (r))
Random Feature Ratio (rz)
Dropout Ratio (ra)
0.05
0.1
0.3
1.0
1.0
0.5
0.25
0.25
0.5
Linear Probe
85.1
85.7
84.9
77.9
85.7
85.5
84.2
85.4
85.1
Unsup. Retr.
62.3
62.8
62.1
55.9
62.8
62.7
62.0
62.5
62.3
Unsup. Retr. 258
61.4
61.8
61.0
60.7
60.1
around 400. The cluster number needs to be balanced between the intra-class noises and inter-class noises. Too small cluster numbers will incur heavy intra-class noise, which dramatically decreases the performance of the pre-trained classification model. Besides, too many clusters will increase the computation and storage burden the FC layer. Most important, the over-decomposition will increase the inter-class noise ratio and undermine the discriminative power the pre-trained model.
Random Class Selection. In Tab. 6 we train ViT B/32 models under different inter-class sampling ratios. The basic margin- based softmax loss (r₃ 1.0) only achieves 77.9% on the linear probe task as it hardly adapt the heavy inter-class conffict in the automatically clustered dataset. When the sampling ratio is decreased from 1.0 to 0.3 and 0.1, our method exhibits consistently better performance than the baseline, indicating random inter-class sampling is beneficial for the model's 's robustness. When 71 set to 0.05, there a slight performance drop because the inter-class interaction is insufficient during training. Therefore, we choose the random negative class selection ratio as 0.1, obtaining 85.7% and 62.8% on the linear probe and unsupervised image retrieval tasks.
Random Feature Selection. In Tab. 6 we compare the performance of the proposed Unicom under different random feature selection ratios (r2) on the task of
unsupervised image retrieval. Here, we also include Dropout different drop ratios (r3) for comparison. From the results, we can have the following observations: (1) both random feature selection and Dropout can not improve linear probe and unsupervised image retrieval at a full dimension of 512 as the LAION 400M dataset is large enough and regularization is not necessary for the final classification layer, (2) there is slight performance drop when the random feature selection ratio is decreasing, and (3) the proposed random feature selection (r22 0.5) can improve 0.4% for 256-D unsupervised unsupervised image retrieval.
image retrieval, while Dropout can not improve Even though Dropout enforces partial features for classification, the global randomization within the mini-batch makes the optimization involve all feature dimensions. By contrast, the proposed random feature selection is fixed within the mini-batch thus it can benefit from optimization in sub-feature space.
5 CONCLUSIONS
This paper introduces Unicom, a simple yet effective framework for universal and compact feature embedding. Given the automatically clustered large-scale data, we employ one random negative class selection to improve the robustness under the heavy inter-class conflict and another random feature selection to improve the compactness of the feature representation. For both unsupervised and supervised image retrieval different datasets, the proposed Unicom achieves state-of-the the-art performance, confirming that cluster discrimination is beneficial to explore the semantic structure within the large-scale training data.

---

Published as a conference paper at ICLR 2023
REFERENCES
Xiang An, Xuhan Zhu, Yuan Gao, Yang Xiao, Yongle Zhao, Ziyong Feng, Lan Wu, Bin Qin, Ming Zhang, Debing Zhang, and Ying Fu. Partial fe: Training 10 million identities on single machine. In ICCV Workshops, 2021.
Xiang An, Jiankang Deng, Jia Guo, Ziyong Feng, XuHan Zhu, Jing Yang, and Tongliang Liu. Killing two birds with one stone: Efficient and robust training of face recognition cnns by partial fc. In CVPR, 2022.
Yuki Markus Asano, Christian Rupprecht, and Andrea Vedaldi. Self-labelling via simultancous clustering and representation learning. In /CLR, 2020.
Artem Babenko and Victor Lempitsky. Aggregating deep convolutional features for image retrieval. In ICCV, 2015.
Thomas Berg, Jiongxin Liu, Seung Woo Lee, Michelle L Alexander, David W Jacobs, and Peter N Belhumeur. Birdsnap: Large-scale fine-grained visual categorization of birds. In CVPR, 2014.
Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool, Food-100 101-mining discriminative compo- nents with random forests. In ECCV, 2014.
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsuper- vised learning of visual features. In ECCV, 2018.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. In NeurIPS, 2020.
Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey E Hinton. Big self-supervised models are strong cemi-supervised learners. In NeurlPS, 2020.
Xinlci Chen and Kaiming He. Exploring simple siamese representation learning. In CVPR, 2021.
Mircea Cimpoi. Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. De- scribing textures in the wild. In CVPR, 2014.
Jia Deng, Wei Dong. Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou. Areface: Additive angular margin loss for deep face recognition. In CVPR, 2019.
Jiankang Deng, Jia Guo, Tongliang Liu, Mingming Gong, and Stefanos Zafeiriou. Sub-center arcface: Boosting face recognition by large-scale noisy web faces. In ECCV, 2020.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn. Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.
Aleksandr Ermolov, Leyla Mirvakhabova, Valentin Khrulkov, Nicu Sebe, and Ivan Oseledets. Hyper- bolic vision transformers: Combining improvements in metric learning. In CVPR, 2022.
Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In CVPR Workshops, 2004.
Raia Hadsell, Sumit Chopra, and Yann LeCun. Dimensionality reduction by learning an invariant mapping. In CVPR, 2006.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In ICCV, 2017.
10

---

Published as a conference paper at ICLR 2023
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. CVPR, 2022.
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2019.
Gabriel Iharco,, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openclip, 2021.
Ahmet Iscen, Giorgos Tolias, Yannis Avrithis, and Ondiej Chum. Mining on manifolds: Metric learning without labels. In CVPR, 2018.
Chao Jia, Yinfei Yang. Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-lane -aiiu-la representation learning with noisy text supervision. In ICML, 2021.
Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion-scale similarity search with gpus. IEEE Transactions Big Data, 2019.
HeeJae Jun, Byungsoo Ko, Youngjoon Kim, Insik Kim, and Jongtack Kim. Combination of multiple global descriptors for image retrieval. arXiv: 10663, 2019.
Shichao Kan, Yigang Cen, Yang Li, Vladimir Mladenovic, and Zhihai He. Relative order analysis and optimization for unsupervised deep metric learning. In CVPR, 2021.
Sungyeon Kim, Dongwon Kim, Minsu Cho, and Suha Kwak. Embedding transfer with label relaxation for improved metric learning. In CVPR, 2021.
Sungyeon Kim, Dongwon Kim, Minsu Cho, and Suha Kwak. Self-taught metric learning without labels. In CVPR, 2022.
Jonathan Krause, Jia Deng, Michael Stark, and Li Fei-Fei. Collecting large-scale dataset of fine-grained cars. 2013a.
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In ICCV workshops, 2013b.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. 2009.
Seongwon Lee, Hongje Seong, Suhyeon Lee, and Euntai Kim. Correlation verification for image retrieval. CVPR, 2022.
Junnan Li, Pan Zhou, Caiming Xiong, and Steven Hoi. Prototypical contrastive learning of unsuper- vised representations. In ICLR, 2020a.
Yang Li, Shichao Kan, and Zhihai He. Unsupervised deep metric learning with transformed attention consistency and contrastive clustering loss. In ECCV, 2020b.
Yanghao Li, Hanzi Mao, Ross Girshick, and Kaiming He. Exploring plain vision transformer
backbones for object detection. In ECCV, 2022.
Jongin Lim, Sangdoo Yun, Seulki Park, and Jin Young Choi. Hypergraph-induuced semantic tuplet loss for deep metric learning. In CVPR, 2022.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona. Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. ECCV, 2014.
Hongye Liu, Yonghong Tian, Yaowei Yang, Lu Pang, and Tiejun Huang. Deep relative distance learning: Tell the difference between similar vehicles. In CVPR, 2016a.
11

---

Published as a conference paper at ICLR 2023
Ziwei Liu, Ping Luo, Shi Qiu, Xiaogang Wang, and Xiaoou Tang. Deepfashion: Powering robust clothes recognition and retrieval with rich annotations. In CVPR, 2016b.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2018.
Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens Van Der Maaten. Exploring the limits of weakly supervised pretraining. In ECCV, 2018.
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-gran tine-grained visual classification of aircraft. arXiv: 1306.5151, 2013.
Kevin Musgrave, Serge Belongie, and Ser-Nam Lim. metric learning reality check. In ECCV, 2020.
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In Indian Conference on Computer Vision, Graphics & Image Processing, 2008
Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. In ECCV, 2016.
Hyun Oh Song, Yu Xiang, Stefanie Jegelka, and Silvio Savarese. Deep metric learning via lifted structured feature embedding. In CVPR, 2016.
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In CVPR, 2012.
Yash Patel, Giorgos Tolias, and Jiff Matas. Recall surrogate loss with large batches and similarity mixup. In CVPR, 2022.
Qi Qian, Lei Shang, Baigui Sun, Juhua Hu, Hao Li, and Rong Jin. Softtriple loss: Deep metric learning without triplet sampling. In ICCV, 2019.
Qi Qian, Yuanhong Xu, Juhua Hu, Hao Li, and Rong Jin. Unsupervised visual representation learning by online constrained k-means. In CVPR, 2022.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021.
Karsten Roth, Oriol Vinyals, and Zeynep Akata. Non-isotropy regularization for proxy-based deep metric learning. CVPR, 2022.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma. Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. Imagenet large scale visual recognition challenge. IJCV, 2015.
Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clustering. In CVPR, 2015.
Mannat Singh, Laura Gustafson, Aaron Adcock, Vinicius de Freitas Reis, Bugra Gedik, Raj Prateek Kosaraju, Dhruv Mahajan, Ross Girshick. Piotr Dollár, and Laurens van der Maaten. Revisiting weakly supervised pre-training of visual perception models. In CVPR, 2022.
Kihyuk Sohn. Improved deep metric learning with multi-class n-pair loss objective. NeurIPS, 2016.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way prevent neural networks from overfitting. JMLR, 2014.
Yifan Sun, Changmao Cheng, Yuhan Zhang, Chi Zhang, Liang Zheng, Zhongdao Wang, and Yichen Wei. Circle loss: A unified perspective of pair similarity optimization. In CVPR, 2020.
12

---

Published as a conference paper at ICLR 2023
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In CVPR, 2015.
Fuwen Tan, Jiangbo Yuan, and Vicente Ordonez. Instance-level image retrieval using reranking transformers. In ICCV, 2021.
Giorgos Tolias, Ronan Sicre, and Hervé Jégou. Particular object retrieval with integral max-pooling of cnn activations. ICLR, 2016.
Hugo Touyron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In ICMI., 2021.
Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The inaturalist species classification and detection dataset. In CVPR, 2018
Xun Wang. Xintong Han, Weilin Huang, Dengke Dong, and Matthew R Scott. Multi-similarity loss with general pair weighting for deep metric learning. In CVPR, 2019.
Peter Welinder, Steve Branson, Takeshi Mita, Catherine Wah, Florian Schroff, Serge Belongie, and Pietro Perona. Caltech-ucsd birds 200. 2010.
Tobias Weyand, Andre Araujo, Bingyi Cao, and Jack Sim. Google landmarks dataset v2-a large-scale benchmark for instance-level recognition and retrieval. In CVPR, 2020.
Jianxiong Xiao, James Hays, Krista Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey zoo. CVPR, 2010.
Hongwei Xu, Suncheng Xiang, and Dahong Qian. Subface: Learning with softmax approximation for face recognition. arXiv:2208. 11483, 2022.
Jiexi Yan, Lei Luo, Cheng Deng, and Heng Huang. Unsupervised hyperbolic metric learning. In CVPR, 2021.
Andrew Zhai and Hao-Yu Wu. Classification is a strong baseline for deep metric learning. In BMVC, 2019.
Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers. In CVPR, 2022.
Xiaohang Zhan, Jiahao Xie, Ziwei Liu, Yew-Soon Ong, and Chen Change Loy. Online deep clustering for unsupervised representation learning. In CVPR, 2020.
A APPENDIX
A.1 MODEL. ARCHITECTURES
We follow the same architecture design as CLIP. Tab. 7 describes the details of architectures.
A.2 VISUALIZATION OF PSEUDO CLUSTERS AND DATA DISTRIBUTION
In Fig. we show the data distribution under different settings of class number &. In Fig. 5 we show some exemplar classes from the proposed automatic clustering. As we can see, there are some fine-grain mne-grained classes, such the top with love icons and the top with cartoons. Even though such clustering is reasonable and explainable, there is class confusion if we classify these samples from other views, such as color and targeting customer age.
13

---

Published as a conference paper at ICLR 2023
Table 7: The architecture parameters for ViT models.
Model
Batch Size
FLOPs
Embedding dimension
Input resolution
Vision Transformer
(128 V100)
G
layers
width
heads
ViT-B/32
256*128
4.3
512
224
12
768
12
ViT-B/16
256*128 128
17.6
768
224
12
768
12
ViT-L/14
64 128
80.9
768
224
24
1024
16
ViT-L/14-336
48 128
191.3
768
336
24
1024
16
x10
3 1x10
8000
Class Numbsr
Clacs Number
Clars Number
4000
2000
0.5
Number Class
Image
Image Number in Class
(a) k 100,000
(b) k 1000,000
(c) k= 10,000,000
Figure 4: Data distribution under different settings of cluster number k.
900
(a) Bag+Monkes
(b) Bag+ Wing
(c) Fantaantasy
(d) Lov+Loove
Grey Top+Pink Sleeve
(f) Top+Cartoon
Figure 5: Pseudo classes clustered by the image encoder and text encoder of the pre-trained CLIP model. The class name is given based on the manual observation of images and texts.
Train From Scratch
25
Loss
15
5
10
15
20
25
30
35
Epoch
Figure 6: Training loss curves fine-tuning from the CLIP model and training from scratch.
14

---

Published as a conference paper at ICLR 2023
Table 8: Comparison between fine-tuning and training from scratch. Top-1 accuracy(%) of linear
probe is reported on 13 image classification datasets. ViT B/32 is used here.
CIFARIO
CIFAR100
Catech10)
Flowers
Food101
Hirdsnap
SUN397
Aincrai
EuroSAT
ImageNet
Average
Cars
DTD
Pets
VIT B/32 (Scratch)
96.8
86.6
94.6
93.3
98.5
85.8
70.2
74.6
78.0
70.7
93.1
96.8
75.0
85.7
(Fine-tune)
95.8
94.1
98.5
87.3
67.3
75.3
79.8
96.1
75.1
Table 9: Object detection and instance segmentation on COCO. We evaluate bounding-box AP (APb)) and mask AP (APmk) on val2017.
Method
Pre-training Data
Apbb
APh 50
AP 75
Apmik
AP 50
Apm 75
VIT B/16 (Li et al. 2022)
IN-1K
47.6
42.4
ViT B/16 et al. 2022
IN-21K
47.8
42.6
VIT B/16 OPEN-CLIP
LAION 400M
48.1
69.1
51.7
42.6
66.4
44.3
VIT B/16 Ours
LAION 400M
48.5
69.8
52.4
42.9
66.9
45.9
A.3 TRAINING FROM SCRATCH VS. FINE-TUNING FROM THE CLIP MODEL
In this paper, the CLIP model is only used for the clustering step and our models are trained from scratch. In Fig. 6 compare the training loss curves between fine-tuning and training from scratch. For fine-tuning, the backbone is initialized from the CLIP model (ViT-B/12), and the classifier (FC layer) randomly initialized. The fine-tuning strategy can converge faster than training from scratch, but the final loss value is higher. In Tab. g we also find that training from scratch outperforms fine-tuning from the CLIP model by 0. 0.7% on the task of linear probe.
A.4 COMPARISONS ON COCO DETECTION AND SEGMENTATION
Following the experiment setting in ct al. 2022 use Mask R-CNN (He et al., 2017) for bounding-boo box object detection and instance segmentation. We fine-tune models on the COCO (Lin et al. 2014) train2017 split and evaluate the val2017 split. In Tab. 9 method outperforms both OPEN-CLIP and supervised pre-training in all metrics, demonstrating the effectiveness of the proposed cluster discrimination.
A.5 LINEAR PROBE DATASETS
We use 13 image classification datasets to prove the effectiveness of our method. These datasets
include CiTARl0 & Hinton] 22099, CIFAR (OiKrihyessky Hinton 2009). Cal-
tech101 et al.) 2004). Stanford Cars(Krause
al. 2013a). Oxford Flowers(Nilsback &
Zisserman 2008), 101d-101(Bossaar)) et al.
al,] 2014), Birdsnap et al. 20014), SUN397(Xiao et al.
2010), Describable Textures(Cimpar
al. [2044. FGVC Mire al. 2013). Oxford-IIV
Pets Parkhi et al. 2012), FuroSAT et al. 2019), I Russsskkoysky et al. 2015).
Details on each dataset and the corresponding evaluation metrics provided in Tab. 10)
A.6 IMAGE RETRIEVAL DATASETS
The training and evaluation of image retrieval experiments seven widely used datasets, namely
CUB-200-201 (CUB) (Welinder al. 2010), Stanford Cars(Cars 196) (Krause et al. (2013)), Stanford Online Products(SOP) (Oh Song et al. 2016). In-shop Clothes Retrieval(In-Sho) (Liu al. 2016b). iNaturalist (Van Horn et al. 2018), VehicleID (Liu et al.) (0166)), and Google Landmarks dataset (GLDv2) (Weyand al. 2020). The number of examples and classes can be found in Tab. I1
15

---

Published as a conference paper at ICLR 2023
Table 10: List linear probe datasets with the data distribution and evaluation metrics.
Dataset
Classes
Train size
Test size
Evaluation metric
CIFAR-10
10
50,000
10,000
accuracy
CIFAR-100
100
50,000
10,000
accuracy
Caltech- 101
102
3,060
6,085
mean-per-class
Stanford Cars
196
8,144
8,041
accuracy
Oxford Flowers
102
2,040
6,149
mean per class
Food-101
102
75,750
25,250
accuracy accuracy accuracy accuracy mean per class
Birdsnap
500
42.283
2,149
SUN397
397
19,850
19,850
Describable Textures
3,760
1,880
FGVC Aircraft
100
6,667
3,333
Oxford-IIIT Pets
37
3,680
3,669
mean per class
EuroSAT
10
10,000
5,000
accuracy
ImageNet
1000
1,281,167
50,000
accuracy
Table 11: Dataset composition for training and evaluation in the image retrieval task.
Dataset
Images
Classes
CUB Train (Welinder al.
2010
5,864
100
CUB Test [Welnder et al. 2010)
5,924
100
Cars 196 Trai (Kausse al.
2013b
8,054
98
Cars 196 Test (Kause et al. 20133)
8,131
98
SOP Train (Oh Song et 2016)
59,551
11,318
SOP Test (Oh Song al.
2016)
60,502
11,316
In-Shop (Liu cl al.
2016b
25,882
3,997
In-Shop (Lu et al.
2016b
26,830
3,985
iNaturalist Irain Van Horn et al.
2018
325,846
5,690
iNaturalist Test Van Horn et al.
2018
136,093
2,452
VehiclelDD Train (Liu et al.
2016a)
110,178
13,134
VehicleID Test (L.u et al.
2016aa
40,365
4,800
GLDv2 Train [Weand ei al.
2020
1,580,470
81,314
GLDv2 Test
(Weyand
2020
762,884
1,129
16


Reviews:
[{"review_id": "xHweo5_mMNI", "reviewer": "Reviewer_HyPg", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a method to learn representations for image retrieval with two main ideas: 1) during pre-training, automatically clustering a large image-text dataset and randomly selecting inter-class prototypes to avoid noisy assignments, and 2) to reduce embedding dimensionality, randomly select feature dimensions to be deactivated. Experiments are performed on standard datasets in the area of image retrieval, besides on standard image classification datasets, comparing against unsupervised and supervised methods.", "summary_of_the_review": "This is a paper with interesting ideas, some of which are simple and shown to work well, for an important problem in computer vision. Experiments are comprehensive. However, I also see several issues at this point, as described in the weaknesses above.", "strength_and_weaknesses": "Strengths:\n\nS1) A well-motivated idea to improve unsupervised learning via cluster discrimination. Cluster discrimination may suffer from inter-class conflicts of the prototypes, and the authors propose a very simple and efficient method to deal with it: randomly sample the negative prototypes. Ablation experiments show large improvements when using this idea.\n\nS2) The proposed pre-training method outperforms other CLIP training methods on image classification linear probe experiments and on unsupervised image retrieval experiments. The comparison is strict against the OPEN-CLIP work, where the exact same dataset is used.\n\nS3) The results on the supervised image retrieval tasks outperform other recent competitor methods.\n\nS4) Overall, the experiments are comprehensive, including ablations and even extra comparisons on GLDv2 and the recent Google Universal Embedding Challenge.\n\nWeaknesses:\n\nW1) Random selection of class prototypes seems very simple and effective, but I am wondering whether something smarter can be even better. For example, discarding “hard negative prototypes” directly instead of only sampling randomly. The paper does not provide such comparisons, which would make the proposal stronger.\n\nW2) I wonder if it is fair to call the proposed method “unsupervised”. The technique works on a large dataset of images and associated text, which could be considered already some form of weak supervision (it has more details than the Instagram hashtag dataset, for example). The CLIP text embeddings are even directly used to perform clustering. Generally, “unsupervised” methods work directly on images only and nothing else.\n\nW3) I don’t see a direct connection between equations (1) and (2) in section 3.1. How is equation (2) actually maximizing the log-likelihood function? Do you mean that p(x_i;\\theta) would be the probability of drawing the augmented image? Please explain. Similar comment for equations (3) and (4).\n\nW4) Figure (3)-a is confusing. By “class number”, do the authors mean the ID of the class? By “photo number”, do you mean the number of photos in that class?\n\nW5) There is a very straightforward connection between the random feature subsampling and the concept of Dropout. This connection is not made, but it should be.\n\nW6) The random feature subsampling method works comparably to PCA or the addition of an FC layer to reduce dimensionality in the network. I don’t see a concrete benefit of using it, compared to a simple FC layer to reduce dimensionality, which is more principled.\n\nW7) The story of the paper is inconsistent. There are many experiments on image classification datasets (section 4.2), but the paper is directed at the image retrieval problem.\n\nMinor details:\n\nM1) In the caption of Fig 1: the word “dataset” is repeated on the second line.\n\nM2) Page 3: “cluttering” -> “clustering”\n\nM3) Section 3.1: the set X  is repeated twice in the same first sentence, please remove one of them to make it more concise.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "U_CrR6OrINI", "reviewer": "Reviewer_FUwi", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a self-supervised image representation approach that leverages using pre-existing CLIP image-text embedding. The approach first clusters images using CLIP embedding, and then learns image feature with softmax loss using cluster ids as class ids. In addition, random dimension sampling is employed during learning to achieve low dimensionality for retrieval tasks. The authors conducted extensive results on benchmark datasets to demonstrate the effectiveness of proposed approach in various settings.", "summary_of_the_review": "While approach is appealing and practically useful, the novelty seems slightly below the bar of ICLR.  In additional, the dimension sampling part seems unnecessary to me. I feel directly learning a feature projection would be more intuitive and work better.", "strength_and_weaknesses": "Strength \n+ Intuitive idea and easy to reproduce\n+ Extensive experimental results + ablation studies\n\nWeakness\n- Novelty is a bit low. Sampled softmax is a commonly used technique. \n- Dimension sampling is a bit strange. A more natural approach to me would be to directly learn a projection to low dimensional feature space. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "GcpPINHdnGj", "reviewer": "Reviewer_htKd", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper addresses the problem of representation learning for image retrieval and follow-ups on recent self-supervised methods that use cluster discrimination to handle the limitations of instance discrimination. These methods however either require to perform iterative clustering, or online clustering to avoid multiple iterations over the entire dataset but that still suffer from the collapsing problem. The authors propose a method that performs random selections along class and feature to alleviate the inter-class conflict and to improve feature compactness that only requires a single pre-step of off-line clustering.", "summary_of_the_review": "The paper is well written and the idea and motivation behind the method is presented very well. This idea, although simple (this also makes it widely applicable) it is novel in this context. I also think that the methods has some limitations, like the fact that one assumes access to a large pre-trained model.", "strength_and_weaknesses": "Pros\n- The idea and motivation behind the method is presented very well by pointing out the limitations of currents methods regarding their clustering approach and proposing a possible solution. \n- This idea is also very simple, which makes it widely applicable, not only to this problem/setting.\n- Experiments are also extensive and thorough.\n\nCons:\n- The fact that you need access to a large pre-trained model it makes its applicability more limiting and not directly comparable to OPEN-CLIP. I see the approach as a way of further improving the results of a given base model, CLIP in this case.\n- Related to my previous point, and given that we have assume that we have access to CLIP and that we are going to train the same backbone architecture (is this true?), would it make sense to initialize the backbone using CLIP's weights? Would this speed up training without impacting performance? This might address the problem of needing to have access to a large cluster of GPUs.\n- It would have been interesting to see this applied to other large pre-trained models used as base, eg DINO or MAE.\n- In table 8, I did not understand what are the different columns below Unicom. Could the authors elaborate please?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "DI6goi1rc-W", "reviewer": "Reviewer_1Pq2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes to use random sampling for samples in clustering classes to avoid inter-class conflict and a random dropout mechanism for features to generate compact features.", "summary_of_the_review": "As I mentioned above, although the method is easy to implement, the novelty of this paper is not significant. Especially for feature random selection part. For class selection, although it is easy to implement, it lacks further detailed analysis and improvement. For instance, we can use another clustering method for each cluster to exclude the potential inter-class conflict. ", "strength_and_weaknesses": "The method is easy to understand and implement. The experiments about using VL models for zero-shot/transfer learning tasks are exhaustive.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster