MetricFormer: A Unified Perspective of Correlation Exploring in Similarity Learning
Authors: Jiexi Yan
Erkun Yang
Cheng Deng
Heng Huang
Venue: NeurIPS 2022 Accept (2022)

Abstract: Similarity learning can be significantly advanced by informative relationships among different samples and features. The current methods try to excavate the multiple correlations in different aspects, but cannot integrate them into a unified framework. In this paper, we provide to consider the multiple correlations from a unified perspective and propose a new method called MetricFormer, which can effectively capture and model the multiple correlations with an elaborate metric transformer. In MetricFormer, the feature decoupling block is adopted to learn an ensemble of distinct and diverse features with different discriminative characteristics. After that, we apply the batch-wise correlation block into the batch dimension of each mini-batch to implicitly explore sample relationships. Finally, the feature-wise correlation block is performed to discover the intrinsic structural pattern of the ensemble of features and obtain the aggregated feature embedding for similarity measuring. With three kinds of transformer blocks, we can learn more representative features through the proposed MetricFormer. Moreover, our proposed method can be flexibly integrated with any metric learning framework.  Extensive experiments on three widely-used datasets demonstrate the superiority of our proposed method over state-of-the-art methods.

FetricFormer: A Unified Perspective of Correlation Exploring in Similarity Learning
Jiexi Yan 1.2 Erkun Yang² Cheng Deng"* Heng Huang
School of Computer Science and Technology, Xidian University, Xi an 710071, China 2 School of Electronic Engineering, Xidian University, Xi'an 710071, China Electrical and Computer Engineering, University of Pittshurgh, PA 15261, USA (jxyan1995, erkunyang, chdeng. .d, henghuanghh @agghannghh)@mmaill. @gmail com
Abstract
Similarity learning be significantly advanced by informative relationships among different samples and features. The current methods try to excavate the multiple correlations in different aspects, but cannot integrate them into unified framework. In this paper, we provide to consider the multiple correlations from unified perspective and propose can effectively capture and model the multiple correlations with an elaborate metric transformer. In MetricFormer, the feature decoupling block is adopted learn an ensemble of distinct and diverse features with different discriminative characteristics. After that, apply the batch-wise correlation block into the batch dimension of each mini-batch to implicitly explore sample relationships. Finally, the feature-wis wiie correlation block is performed to discover the intrinsic structural pattern of the ensemble of features and obtain the aggregated feature embedding for similarity measuring. With three kinds of transformer blocks, we can learn more representative features through the proposed MetricFormer. Moreover, our proposed method can flexibly integrated with any metric learning framework. Extensive experiments on three widely- used datasets demonstrate the superiority our proposed method over state-of-the-an methods.
method called MetricFormer, which
Introduction
Exploring how effectively learn semantic similarities among examples an important problem in the field of machine learning and computer vision. Metric learning aims to transform the original data into an embedding space and learn a distance metric this space to promote intra-class compactness and inter-class separability. In recent years, benefiting from the emerging advance of deep learning technique [20 [7]. deep metric learning (DML) can capture more representative embeddings, and has been widely applied to various downstream tasks, including image retrieval [29] 28). face recognition and person re-identification [57].
19 36,,
The core idea of DML is to excavate the intrinsic structural relations of data, and utilize them to learn discriminative embeddings. As shown in Figure [. such relations can be divided into two main categories: sample relationship and feature relationship. The former, i.e. sample relationship. includes pairwise correlation and batch-wise correlation. Most of existing supervised DMI. methods utilize pairwise correlations (i.e., positive and negative pairs) to construct and utilize valid tuples of pairs through well-designed pair-based losses, e.g. triplet loss (36] and multi-similarity loss [46]. Due to training with random sampling, such methods suffer from heavily low convergence and model degradation. To discover the implicit global structure the dataset, recent studies started to explore
"Corresponding Author.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Pusitive Pairs
Nogative Pairs
(a) Pairwise Correlation
(b) Batch-wise Correlation
(c) Feature-wise Correlation
Figure I: An illustration of multiple intrinsic correlations among different samples and features. (a) According the labels, used in DML. (b) The relationships between different samples are more informative, and not limited the samples from the same class. Samples in the same class have
naturally derive series of positive and negative pairs, which is widely
invariant features, while similar classes share similar parts. (c) The ensemble of features also has inner structural relations between different sub-features, which helps the feature ensemble comprehensively represent samples.
the batch-wise correlation shown in Figure (b) in the embedding space 18 example, IBC [12] adopt the message passing strategy capture the overall structure of data in each mini-batch.
12
11
49
37]. For
Considering that using only single vector for similarity measurement cannot fully capture the semantics of inputs and exacerbates the unexplainability caused by the black-box nature of deep neural networks, recent studies 222 29 56 learn ensemble of embeddings to jointly represent the input instance. Such ensemble-
method generally outperforms those that use a single feature vector since the ensemble encourages the learned embeddings to capture
comprehensive characteristics. However, simply concatenating the ensemble of sub-fcatures ignores the structural relations among them, i.e. feature--ise correlation shown in Figure (c), which is crucial to ensure each sub-feature modeling different and discriminative characteristics. For better performance and generalization ability, the feature-level correlation should also be considered during training.
this paper, we provide unified perspective to learn the multiple correlations and propose a metric transformer framework, named MetricFormer, which enables the information propagation along features of different samples within each mini-batch. Specifically, we introduce correlation-awee transformer into the internal structure of deep metric networks, which contains a feature decoupling block, a feature-wiss wise correlation block, and batch-wise correlation block. We adopt the feature decoupling block with several trainable query vectors to Iearn series of sub-features with different discriminative characteristics. Meanwhile, the feature- wise correlation block is performed to excavate the intrinsic relations among sub-features, which ensure the learned ensemble of features is representative and sufficient for similarity measurement. Furthermore, to implicitly capture and model the sample relationships, we apply the batch-wise correlation block into the batch dimension of each mini-batch with the learned ensemble of features. Moreover, local embedding module is proposed to focus more informative neighborhood relationships without redundancy. Guided by the sparse graph, we can encode the local neighborhood relationships of each sample into the transformer feature.
Our work delivers the following contributions:
We propose to simultancously explore the multiple correlations among data from unified learning perspective.
We provide a simple yet effective method termed as MetricFormer to implicitly capture three levels of correlations in end-to-end model.

---

We achieve significant performance gains against the state-of tfate-of methods on three usde benchmark datasets.
Preliminaries
2.1 Problem Formulation of DML
Given the training dataset D = {(x;, 1=) where y; is the corresponding label of the sample X;, DML methods aims to learn a feature mapping X + Z E R with deep neural networks, which maps input samples to D-dimensional embedding space 2. With the learned projection f: z; = fxx), we can effectively measure the similarity between any two samples and x, using Euclidean distance
di.j 1|: 2 1|
or cosine similarity
z 2
8i.j
To improve the ability of similarity measurement, majority of DML studies develop various metric loss functions LM, which can impose discriminative constraint on the feature embeddings.
2.2 Revisiting Transformer Encoder
Transformer architecture has achieved great success in the field of natural language processing and computer vision and brings novel paradigms for several fundamental tasks (e.g. translation and segmentation). The overall transformer model usually consists encoder blocks. The transformer encoder contains multi-head self-attention layer (MHSA) and a feed-forward network (FFN), which are followed by a LayerNorm (LN), respectively, Given sequence of input features X E RR xD with the length of the sequence M and the dimension of the input features D, we can derive the output of the transformer encoder as follows,
stack of multiple transformer
Y, = LN (MHSA(Z,-) + Y
Y7 = LN (FFN + Zr),
where 2 1 is the index of layers in the transformer encoder, and Zo= = X.
The self-attention mechanism is widely-used to capture and model the relationships from the channel and spatial dimensions. With Q, K, V E I M as the query, key, and value learned from the same input, can calculate the output Z for the self-attention module:
QK
Y softmax
V.
vD
And then, the representations from different heads are concatenated.
3
The Proposed Method
In this section, we first introduce the overall motivation and framework of the proposed Metric Transformer, and then describe the three transformer blocks, respectively.
3.1 Metric Transformer
Since the relationships among samples and features are both various and complex, how to effectively discover the intrinsic structure of data and excavate more representative information from the training data is an essential problem in DML. Most of existing methods can only utilize the pair- based similarity, which neglects the batch-wise correlation and feature-wise correlation. Recently, some studies are proposed to mine such rich relations. However, they can only learn the batch-wise or feature-wist wise correlations individually, and the multiple correlations cannot be automatically learned in unified end-to-end DML model.
3

---

For Training
Input Image
Batch-wise Correlation Black
Festure-aisse
Feanare Decoupling
Correlation Block
Share Weights
Share Weights
Backbone
Local Embodding
Graph
Feature Maps
Festure-aiis
Featurr Decoupling
Correlation Block
Testing
Figure 2: An illustration of the proposed MetricFormer framework. Our MetricFormer contains one feature decoupling block, one fcature-wise correlation block, and T batch-wist correlation blocks. After obtaining the feature maps generated by a backbone network, we can derive the ensemble of sub-features to represent the input samples from different aspects in the feature decoupling block. And then, the batch-wise correlation block and feature-wis wise correlation block with the same architecture employ the cross-attention mechanism from different dimensions explore the relationships among samples and features. To tackle the inconsistency issue, we utilize the consistency preserving strategy during training.
We introduce metric transformer (MetricFormer) to enable DML model itself with the ability to learn the sample relationships in each mini-batch and the structural relationships among the ensemble of sub-features. As shown in Figure 2 our MetricFormer consists of one feature decoupling block, one feature-wise correlation block, and batch- wise correlation block. Specifically, we adopt backbone network extract the feature maps from input samples. After that, feature decoupling block performed to split the feature maps into several sub-features with different discriminative characteristics. And then, the batch-wise correlation block is employed to model the relationships among different samples by utilizing the cross-attention mechanism the transformer. Finally, we the feature-wise correlation block to discover the structural pattern in the feature ensembles and obtain the aggregated features guided by the learned teature-wise relations.
Consistency Preserving Strategy Considering the inconsistency of input data structure between training and testing as the mini-batch of data is not available testing, we apply a consistency preserving strategy, which forces the features learned with the batch-wise block be consistent with the ones learned without the batch-wise block. By doing this, we can remove the batch-wise correlation block during the test, while still benefiting from the sample relationship learning.
3.2 Feature Decoupling
Compared with lcarning the single embedding vector, employing ensemble of embeddings with sufficient diversity can provide different characteristics of the input samples and improve the performance of DML. To achieve this, exploit the self-attention mechanism with several trainable queries to automatically split the feature map into an ensemble of several distinct and diverse sub- features. The learned ensemble of features comprehensively characterize the input sample from different aspects.
Specifically, we apply backbone network to extract original feature maps J E XC where H. W, C. denotes the height, width, and the number channels, respectively. After that, we use two different lincar layers g1 (-) and g2(-) to process the feature map tensor J and derive the corresponding

---

key and value matrices in the feature decoupling block as follows:
K 91 (J) E HWW
(5)
V (2( E Hwxd
where Ck denotes the numbers of feature maps in the key matrix, and d the embedding dimension of the learned sub-features. Here, we set d [D/k] according the number of sub-features &s and the final embedding dimension D.
To automatically learn the representative ensemble of features with the self-attention mechanism,
introduce a set trainable query vectors Q1 = qq: q2: ... q& E R where k is the number of sub-features. These query vectors are randomly initialized and adaptively updated along with the other parameters in our model. To do so, we can calculate the attention score matrix as
QK
As softmax
E BR
VCK K
which measures the importance of spatial positions to guide the learned sub-features to identify different spatial positions. To this end,
obtain the ensemble of features
As R
Different rows of Y represent the learned sub-features with different characteristics. Moreover, we employ diversity loss to encourage the diversity between every two sub-features as follows:
Ldiv log (1 exp(a((;,) m))))),
where $t,j is the cosine similarity between two sub-features Z and Z of the same sample. And the metric loss is applied to each sub-feature of training samples separately to ensure that the learned ensemble of features can effectively represent the inputs from different aspects.
3.3 Multiple Correlations Exploration
The feature decoupling block enables the spatial information propagation and learns the ensemble of sub-features with discriminative characteristies from differ- ent aspects. With the learned ensembles of features, we can further explore the information propagation among different samples within each training mini-batch and different sub-features within each en- semble. Both of them can significantly improve the performance of DML.
B (a) Batch-wise
B
To effectively capture the multiple corre- lations, we design two transformer en- coder blocks with the same architec- ture, i.e. batch-wise correlation block and feature-wise correlation block. As shown in Figure 3. if we treat each sample in the mini-batch as a node of a sequence, the batch-wise wise correlation block can implicitly mine the relation- ships among samples in the mini-batch.
(b) Feature-wise
Figure 3: An illustration of the attention mechanisms batch and feature dimensions. The batch-wist wish correlation block treats mini-batch of samples sequence, while the feature-wiss wise correlation block regards an ensemble of sub-features as sequence.
While if we regard each sub-feature of the sample as node of sequence, the structural relations among the sub-features through the feature- wise block can be explored. Considering a set of features calculate the corresponding query, key, and value tensors Q. K. V Bxkxd After that, we apply the attention mechanism from different dimensions.
F R Bxxxd in mini-
To explore the sample relationships in each mini-batch, we general- ize the attention mechanism into the batch dimension as follows:
Batch-we hatch-wise Correlation Block
Ag = softmax
E
R
BxB
(10)
Vd
5

---

where : :, V.a: E R Bxdd The attention score matrix Ag can measure the sample relationship in this mini-batch. To further enhance the local manifold embedding, we introduce a local embedding graph G defined as
G - $ij if x; EX;,, otherwise,
{
(x;, m) and 81,j 2c,
(11)
where (x,, n) denotes the set of top-n samples of x; according to the rank list { (t.r)i=1. and is the threshold. And then, the attention score matrix can be updated as follows:
A o AB,
(12)
where @ denotes the element- wise product operation. To this end, we can derive the output embed- dings:
ABV... y = concat .V....
y
:i:
(13)
To excavate the structural relations among different sub-features, we apply the attention mechanism from the feature dimension. For each sample x; (i = 1,2, ... B), sequence, i.e., have B sequences with the length of k. Specifically, the output embeddings are learned according to the query, key and value matrices
Feature-wise Correlation Block
treat its ensemble of sub-features
Qin, Kin, Vi: kxx in the position of the i-th sample:
V s:: softmax
Ve: BR
(14)
Vd
4 Related Work
Deep Metric Learning Recently, deep metric learning has been well studied and achieved great
advances 6 36
27
47
3
4
54
53
The majority of works focus designing discriminative metric
loss functions 4
28
46
39
21
38 and hard mining strategies [1]
15
48
52 50 9 for
training more effectively. Recently, some methods consider learn ensemble of embeddings to capture more characteristies of inputs from different aspects [22 52
35 26
56 Nevertheless, the
rich relationships among samples and embeddings are usually ignored in the existing methods.
Transformer In recent years, based on multi-head self-attention mechanism [42], transformers, represented by BERT 7 and ViT [8]. develop rapidly and have been widely used in the field of natural language processing and computer vision. Visual transformer [8 25 has gradually become a
new backbone for computer vision tasks, including classification 8 25 detection (2). segmentation
55
and representation learning [5 [6
5
Experiments
In this section, we evaluate the effectiveness of our method on three widely-used datasets on image clustering and retrieval tasks.
5.1
Experimental Settings
Datasets To evaluate the performance of the proposed MetricFormer, we conduct the experiments on three widely- used benchmarks 2UB-200-2011
Table 1: Statistics of three datasets.
Training
Testing
Datasets
classes
Samples
# classes
Samples
CUB
100
58,64
100
5,924
(CUB) 43). Cars196 (24 and Stanford Online Products (SOP)
1ars196
98
8,054
98
8,131
SOP
11,318
59,551
11,316
60,502
28
Following the standard
protocol [38 28 we split the training and testing sets without intersection. As shown in Table [
CUB-200-2011 contains 200 bird species of 11, 788 images. The first 100 categories of 5,864 images
used as the training set and the remaining 5,924 images from the other 100 categories used for

---

testing. Cars196 includes 16,185 images from 196 car categories. The first 98 classes (8,054 images) are used for training and the rest 98 classes (8,131 images) for testing. Standard Online Products contains I 120,035 images from 22,634 products. We use 59,551 images of the first 11,318 products for training and 60,502 images of the remaining 11,316 categories for testing.
we conduct experiments on both image retrieval and clustering tasks. For the retrieval task, calculate Recall @ K. to evaluate the retrieval performance. Recall K measures the percentage of well sepl-seerrtted samples acknowledged if we can find at least one corrected retrieved sample in the nearest neighbors. For the clustering task, we adopt the Normalized Mutual Information (NMI) based on the K-means algorithm as the evaluation metric.
Evaluate Metrics Following the existing works [8] 28
Baselines We apply the proposed MetricFormer the margin loss 48 and the ProxyAnchhr loss [2I] for comparison. We also compare our method with other state-of-the-art methods including
pair-based methods, N-Pair [38), HTL [13], Angular 444, MS [46. Circle [39] proxy-based meth-
ods SoftTriple [3I] ProxyGML (88, and ensemble methods HDC [22] A-BIER [29 ABE [2]
DiVA 26
DRML [56]
Implementation Details We implement all the following experiments using Py PyTorch 30 on an NVIDIA RTX 1080ti GPU, and the ADAM optimizer [23] with learning rate of 0.0001 is adopted to train the model. We use ResNet-50 [77 as backbone model pre-trained on ImageNet dataset [34 for fair comparisons. For data augmentation, and the embedding size of each sub-feature fixed 170. For
use random cropping and horizontal mirroring. We
set the number of sub-features each iteration, we fix the batch size to 112. For the margin loss [48), we set the margin factors and B to 1.2 and 0.2, respectively. For the ProxyAnchor loss [21], we set the temperature a 16, positive margin Tpos 1.8, and negative margin "eeg 2.2.
5.2 Experimental Results
We evaluate the performance of our method on three benchmarks CUB-200- 2011 1 43 Cars [24] and Stanford Online Products (SOP) 28 The experimental results are shown Table Note that MetricFormer-PP and MetricFormer- represent our method combined with ProxyAnchor and Margin loss, respectively. The best results are marked in bold red. We highlight our superior results over the associated methods without MetricFormer in bold black.
Quantitative Results
We can easily observe that our MetricFormer can improve the performance of the original DML. methods by significant margin. Compared with the original methods, method mine more informative relations from different aspects, which useful for similarity learning. Moreover, our method outperforms the state-of-the-an methods on all three datasets. These empirical results support our proposal that our MetricFormer can discover the relationships among samples and features, which ignored by most of the existing DML methods. Within the multiple relations, can capture the characteristics of samples more comprehensively.
To further demonstrate the performance of our method, we give the visualization results of our Met- ricFormer on the CUB-200-2011 dataset. As shown in Fig- ure [ decoupled sub-features focus on the body, neck, and head of birds respectively. which demonstrates that the learned ensemble of features can capture and model different characteristics. The trainable queries in the feature decoupling block can regarded as pattern detectors and large weights generated once a particular pattern is detected.
Qualitative Results
Moreover, we visualize the sub-features for all the samples on the CUB-200-200 dataset using t-SNE [4] in Figure We use different colors to represent the different sub-features in a 2-dimensional space. We observe that the learned sub-features are very diverse in the feature space. These visualization results support proposal that MetricFormer can learn
Figure 4: The t-SNE visualization results of the learned feature ensem- ble on the CUB-200-22 dataset.

---

the ensemble of distinct and diverse features with different discriminative characteristics.
Table Experimental results (%) of MetricFormer in comparison with the state-of-the-art methods on the testing sets of CUB-200-2011 20000 Cars 196 and Stanford Online Products. n-G/BN/R indicates
the model setting where is the embedding size, and G.BN.R denote GoogleNet 40 Inception with Batch Normalization 20 and ResNet-50 [7]. respectively.
Datasets
CUB
Cars
SOP
Methods
Setting
R@l
R@2
NMI
R@1
R@2
NMI
R@1
R@10
NMI
N-Pair
3
512G
50.1
63.3
60.4
71.1
79.7
64.0
67.7
83.8
88.1
Angular
44
512G
53.1
65.0
71.3
80.7
62.4
67.9
83.2
87.8
RLL-H 45
512BN
57.4
69.7
63.6
74.0
83.6
65.4
76.1
89.1
89.7
HTL 13
512BN
57.1
68.8
81.4
88.0
74.8
88.3
ProxyGML 68
512BN
66.6
77.6
69.8
85.5
91.8
72.4
78.0
90.6
90.2
SoftTriple 81
S12BN
65.4
76.4
69.3
84.5
90.7
70.1
78.3
90.3
92.0
MS
512BN
65.7
77.0
84.1
90.4
78.2
90.5
GroupLoss
512BN
65.5
77.0
69.0
85.6
91.2
72.7
75.1
87.5
90.8
A- A-BIER 29
512R
57.5
68.7
82.0
89.0
74.2
86.9
ABE
512R
60.6
71.5
85.2
90.5
76,3
88.4
MIC 32
128R
66.1
76.8
69.7
82.6
89.1
68.4
77.2
89.4
90.0
TML 5
512R
62.5
73.9
86.3
92.3
78.0
91.2
DiVA 26
512R
69.2
87.6
92.9
72.2
79.6
90.6
PADS
128R
67.3
78.0
69.9
83.5
89.7
68.8
76.5
89.0
89.9
CircleLoss LoSS
39
512R
66.7
77.4
83.4
89.8
78.3
90.5
IBC 37
512R
70.3
80.3
74.0
88.1
93.3
74.8
81.4
91.4
92.6
DIML 54
512R
68.2
87.0
79.3
DRML 56
512R
68.7
78.6
69.3
86.9
92.1
72.1
79.9
90.7
90.1
AVSL
512R
71.9
81.7
91.5
95.0
79.6
91.4
48
512R
63.1
74.4
66.7
79.9
87.5
64.1
78.3
82.7
90.2
MetricFormer- M
512R
69.2
79.6
71.0
89.5
93.8
73.1
79.8
83.6
92.0
ProxyAnchor 21
512R
69.7
80.0
72.3
87.7
93.0
75.7
78.4
90.5
91.0
MetricFormer PA
512R
74.4
83.7
75.4
91.8
95.4
76.2
82.2
92.6
92.7
5.3 Ablation Study
Ablation Study of Different Compo- nents We conduct experiments with the margin loss to analyze the effec- tiveness of different blocks our Met- ricFormer. Margin denotes the base- line method of using the margin loss. +FDB, +BCB, +FCB short for 'fea- ture decoupling block', "batch-wis wise cor- relation block', and 'feature-wiss correla- tion block', respectively. Table 3 shows experimental results on the CUB-200- 2011 1 dataset. It is clear that our Metric- Former achieves better performance than all the compared counterparts and of the proposed blocks contribute to the overall improvement, which demonstrates that exploring and capturing informative
Table Ablation study with different model settings.
Method
R@D
R @
NMI
Margin
63.1
74.4
66.7
Margin FDB
66.8
76.5
67.7
Margin FDB + BCB
68.7
79.3
70.8
Margin FDB + FCB
68.4
79.1
70.6
Margin MetricFormer
69.2
79.6
71.0

---

Sub-feature
Sub-feature
Sub-feature
Figure 5: Attention map visualization results on the CUB-200-2011 dataset. The highlighted regions indicate the spatial locations that different sub-features are detecting. The three sub-features learned by our FetricFormer capture different characteristics of the bird images regardless of the spatial locations of these characteristics. The first sub-featur focuses the body of birds, the second sub-feature focuses on the neck of birds, and the third sub-feature focuses on the head of birds.
relations is crucial for similarity learning. The relationships between samples and sub-features are both useful for similarity measurement.
Influence of Different Number of Sub-features We employ i individual features describe each input sample from different aspects. We conduct experiments with the margin loss (i.e. MetricFormer- M) analyze the influence of the number of individual features. Table shows the experimental results on the CUB-200 CUB-200-2011 dataset. The value of k ranges from 2, 3 to 4, rendering the size of cach sub-fcature 256, 170, and 128, respectively. We can observe that our method achieves the best performance when & = 3, and using a larger slightly harms the performance since each sub-fcature captures less information with a larger k.
Influence of Different Number of Batch-wise Correlation Blocks T We conduct experiments with the margin loss on the CUB-200-2011 dataset to analyze the effect of the number batch-wise correlation blocks. According to the experimental results shown in Table more batch-wise blocks can improve the model performance. Considering the amount of model parameters, we apply three batch-wise block in our MetricFormer, i.e. 33
that exploiting
Table Experimental results (%) of our MetricFormer with different number of the sub-features and different number of the batch-wise blocks T.
k = 2
h A= 3
k = 4
==1
T 2
T 3
R
R@2 NMI
R@1 R@2 NMI R@1 R@2 NMI R@ RØ 10 NMI R@1 R@2 R@ R@2 NMI
65.8
76.2
68.9
69.2
79.6 71.0
68.5 78.9 70.9
67.7
78.1
70.2
68.4
79.0 70.8 69.2 79.6 71.0
6
Conclusion
In this paper, we propose a metric transformer (MetricFormer) framework to explore multiple correlations in a unified perspective. In our MetricFormer, we take the pairwise relationship. the sample relationship among each mini-batch, and the structural relationship among the ensemble of features into consideration through the feature decoupling block, the batch-wise correlation block. and the feature-wise correlation block respectively. To this end, we can the ensemble of distinct and diverse features with different and discriminative characteristies. Moreover, our proposed method can be flexibly integrated with any metric learning method. We have conducted extensive experiments on three commonly used datasets which demonstrated that our method can effectively boost the performance of the existing DML methods.

---

Acknowledgment
Our work was supported part by the National Natural Science Foundation of China under Grant 62132016, Grant 62171343, and Grant 62071361, in part by Key Research and Development Program of Shaanxi under Grant 20212DLGY01--5 and in part by the Fundamental Research Funds for the Central Universities ZDRC2102. Erkun Yang was supported in part by the National Natural Science Foundation of China under Grant 62202365, Guangdong Basic and Applied Basic Research Foundation No.2022IQ-608).
and Natural Science Basic Research Program of Shaanxi (program
Broader Impact
Limitation Due the usage of transformer, the parameters of the model are increased, which leads to more computation cost and training times.
Negative Impact The proposed method predicts content based learned statisties of the training dataset and as such will reflect biases in those data, including ones with negative societal impacts. This issue warrants further research and consideration.
References
[1] Fatih Cakir, Kun He, Xide Xia, Brian Kulis, and Stan Sclaroff. Deep metric learning rank. In CVPR, pages 1861-1870, 2019.
[2] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. ECCV, pages 213-229. Springer, 2020.
[3] Shuo Chen, Lei Luo, Jian Yang, Chen Gong, Jun Li, and Heng Huang. Curvilinear distance metric learning. NeurIPS, 32, 2019.
[4] Shuo Chen, Gang Niu, Chen Gong, Jun Li. Jian Yang, and Masashi Sugiyama, Large-margin contrastive learning with distance polarization regularizer. In ICML, pages 1673-1683, PMLR, 2021.
[5] Xinlei Chen, Saining Xie, and Kaiming Hc. An empirical study of training self-supervised vision transformers. ICCV, pages 9640-9649, 2021.
[6] Sumit Chopra, Raia Hadsell, Yann LeCun, et al. Learning a similarity metric discrininatively, with application to face verification. In CVPR, pages 539-546. 2005.
[7] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: urXiv:1810004805. 2018.
[8] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x 16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.1 1929, 2020.
[9] Jason Xiaotian Dou, Alvin Qingkai Pan, Runxue Bao, Haiyi Harry Mao, and Lei Luo. Sampling through the lens of sequential decision making. arXiv preprint arXiv:2208.08 080556 2022.
[10] Yueqi Duan, Lei Chen, Jiwen Lu, and Jie Zhou. Deep embedding learning with discriminative sampling policy. In CVPR, pages 4964 4973, 2019.
[11] Yueqi Duan, Wenzhao Zheng, Xudong Lin, Jiwen Lu, and Jie Zhou. Deep adversarial metric learning. In CVPR, pages 2780-2789, 2018.
[12] Ismail Flezi, Sehastiano Vascon, Alessandro Torcinovich, Marcello Pelillo, and Laura Leal- Taixé. The group loss for deep metric learning. In ECCV, pages 277-294. Springer, 2020.
10

---

[13] Weifeng Ge. Deep metric learning with hierarchical triplet loss. In ECCV, pages 269-285, 2018.
[14] Raia Hadsell, Sumit Chopra, and Yann LeCun. Dimensionality reduction by learning an invariant mapping, In CVPR, volume 2, pages 1735-1742. IEEE, 2006.
[15] Ben Harwood, Vijay Kumar BG, Gustavo Carneiro, Ian Reid, and Tom Drummond. Smart mining for deep metric learning. In ICCV, pages 2821-2829, 2017.
[16] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. arXiv preprint 1rXhv:211.00377 2021.
[17] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, pages 770-778, 2016.
[18] Xinwei He, Yang Zhou, Zhichao Zhou, Song Bai, and Xiang Bai. Triplet-center loss for multi-view 3d object retrieval. In CVPR, pages 1945-1954, 2018.
[19] Junlin Hu, Jiwen Lu, and Yap-Peng Tan. Discriminative deep metric learning for face verification in the wild. In CVPR, pages 1875-1882, 2014.
[20] Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167 2015.
[21] Sungyeon Kim, Dongwon Kim, Minsu Cho, and Suha Kwak. Proxy anchor loss for deep metric learning. In CVPR, pages 3238-3247, 2020.
[22] Wonsik Kim, Bhavya Goyal, Kunal Chawla, Jungmin Lee, and Keunjoo Kwon. Attention-basee ensemble for deep metric learning. In ECCV, pages 736-751, 2018.
[23] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 141.v91412.6980. 2014.
[24] Jonathan Krause, Michael Stark, Jia Deng, and L. Fei-Fei. object representations for fine-grained categorization. In ICCV workshop, pages 954-561,2015
[25] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, pages 10012-10022, 2021.
[26] Timo Milbich, Karsten Roth, Homanga Bharadhwaj. Samarth Sinha, Yoshua Bengio, Björn Ommer, and Joseph Paul Cohen. Diva: Diverse visual feature aggregation for deep metric learning. In ECCV, pages 590-607. Springer, 2020.
[27] Hyun Oh Song, Stefanie Jegelka, Vivek Rathod, and Kevin Murphy. Deep metric learning via facility location. In CVPR, pages 1392--339, 2017.
[28] Hyun Oh Song, Yu Xiang, Stefanie Jegelka, and Silvio Savarese. Deep metric learning via lifted structured feature embedding. ICCV, pages 4004 4012,401,,40 2016.
[29] Michael Opitz, Georg Waltner, Horst Possegger, and Horst Bischof. Deep metric learning with bier: Boosting independent embeddings robustly. IEEE PAMI, 2018.
[30] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. In NIPS, pages 8026-8037, 2019.
[31] Qi Qian, Lei Shang, Baigui Sun, Juhua Hu, Hao Li, and Rong Jin. Softtriple loss: Deep metric learning without triplet sampling. In CVPR, pages 6450-6458, 2019.
[32] Karsten Roth, Biagio Brattoli, and Bjorn Ommer. Mic: Mining interclass characteristics for improved metric learning. In ICCV, pages 8000 8009, 2019.
[33] Karsten Roth, Timo Milbich, and Bjorn Ommer. Pads: Policy-adapted sampling for visual similarity learning. In CVPR, pages 3568-6577, 2020.

---

[34] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma. Zhiheng Huang. Andrej Karpathy, Aditya Khosla, Michael Bersstein, al. Imagenet large scale visual recognition challenge. Int. J. Comput. Vis., 115(3):211-252, 2015.
[35] Artsiom Sanakoyeu, Vadim Tschernzzki, Uta Buchler, and Bjorn Ommer. Divide and conquer the embedding space for metric learning. In CVPR, pages 471 171-480, 2019.
[36] Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: unified embedding for face recognition and clustering. In CVPR, pages 815-823, 2015.
[37] Jenny Denise Seidenschwarz, Ismail Elezi, and Laura Leal-Taixé. Learning intra-batch connec- tions for deep metric learning. In ICMI, pages 9410-9421. PMIR, 2021.
[38] Kihyuk Sohn. Improved deep metric learning with multi-class n-pair loss objective. In NIPS, pages 1857-1865, 2016.
[39] Yifan Sun, Changmao Cheng, Yuhan Zhang, Chi Zhang, Liang Zheng, Zhongdao Wang, and Yichen Wei. Circle loss: A unified perspective of pair similarity optimization. In CVPR, pages 6398-6407, 2020.
[40] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In CVPR. pages 1-9, 2015.
[41] Laurens Van Der Maaten. Accelerating t-sne using tree-based algorithms. The Journal of Machine Learning Research, 15(1):3221-3245 2014.
[42] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. NIPS, 30, 2017.
[43] Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-uesd birds-200-2011 dataset. 2011.
[44] Jian Wang, Feng Zhou, Shilei Wen, Xiao Liu, and Yuanqing Lin. Deep metric learning with angular loss. In ICCV, pages 2593-2601, 2017.
[45] Xinshao Wang, Yang Hua, Elyor Kodirov, Guosheng Hu, Romain Garnier, and Neil M Robertson. Ranked list loss for deep metric learning. In CVPR, pages 5207-5216, 2019.
[46] Xun Wang, Xintong Han, Weilin Huang, Dengke Dong, and Matthew R Scott. Multi-similarity loss with general pair weighting for deep metric learning. In CVPR, pages 5022-5030, 2019.
[47] Xun Wang, Haozhi Zhang. Weilin Huang, and Matthew R Scott. Cross-batch memory for embedding learning. In CVPR, pages 6388-6397, 2020.
[48] Chao- Yuaa Wu, R Manmatha, Alexander J Smola, and Philipp Krahenbuhl. Sampling matters in deep embedding learning. In CVPR, pages 2840-2848, 2017.
[49] Jianlong Wu, Keyu Long, Fei Wang, Chen Qian, Cheng Li, Zhouchen Lin, and Hongbin Zha. Deep comprehensive correlation mining for image clustering. In ICCV, pages 8150-8153 2019.
[50] Xinyi Xu, Yanhua Yang, Cheng Deng, and Feng Zheng. Deep asymmetrie metric learning via rich relationship mining. In CVPR, pages 4076-4085, 2019.
[51] Baosheng Yu and Dacheng Tao. Deep metric learning with tuplet margin loss. In ICCV, pages 6499, 2019.
[52] Yuhui Yuan, Kuiyuan Yang, and Chao Zhang. Hard-aware deeply cascaded embedding. In ICCV, pages 814-823, 2017.
[53] Borui Zhang, Wenzhao Zheng, Jie Zhou, and Jiwen Lu. Attributable visual similarity learning. 2022.
arXiv preprint arXiv:2203.
[54] Wenliang Zhao, Yongming Rao, Ziyi Wang, Jiwen Lu, and Jie Zhou. Towards interpretable deep metric learning with structural matching. Proceedings of the IEFF/CVF International Conference Computer Vision, pages 9887-9896, 2021.
12

---

[55] Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu. Zekun Luo, Yabiao Wang, Yanwei
Fu, Jianfeng Feng. Tao Xiang. Philip HS Torr, al. Rethinking semantic segmentation from sequence-to-sequenee perspective with transformers. In CVPR, pages 6881-6890, 2021.
[56] Wenzhao Zheng, Borui Zhang, Jiwen Lu, and Jie Zhou. Deep relational metric learning. In CVPR, pages 12065 12074, 2021.
[57] Jiahuan Zhou, Pei Yu, Wei Tang, and Ying Wu. Efficient online local metric adaptation via negative samples for person re-identification. In ICCV, pages 1120-2428, 2017.
[58] Yuehua Zhu, Muli Yang, Cheng Deng, and Wei Liu, Fewer is more: A deep graph metric learning perspective using fewer proxies. arXiv preprint 2rXiv:20100136666 2020.
Checklist
The checklist follows the references. Please read the checklist guidelines carefully for information on how to answer these questions. For each question, change the default [TODO] to [Yes], [No], or [N/A] You are strongly encouraged to include a justification your answer, either by referencing the appropriate section of your paper or providing brief inline description. For example:
Did you include the license to the code and datasets? [Yes] See Section ??.
Did you include the license to the code and datasets? [No] The code and the data are proprietary.
Did you include the license to the code and datasets? [N/A]
Please do not modify the questions and only use the provided macros for your answers. Note that Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material URL)? [No]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [No]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes]
4. If you using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [No]
13

---

(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [No]
5. If you used crowdsoureing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
A
Appendix
Optionally include extra information (complete proofs, additional experiments and plots) in the appendix. This section will often be part of the supplemental material.
14


Reviews:
[{"review_id": "rWAppUKtmpn", "reviewer": "Reviewer_3kwp", "summary": "This paper proposes a Transformer-based method named MetricFormer for metric learning. The proposed MetricFormer includes three blocks: feature decoupling block, feature-wise correlation block, and batch-wise correlation block. Given an image, feature decoupling block uses several trainable queries and cross-attention to extract different sub-features to focus on different spatial regions. Feature-wise correlation block uses self-attention to model the relations among different sub-features of a sample, while batch-wise correlation block uses self-attention to model the relations among different samples in a mini-batch.", "questions": "As pointed out in the \"Weaknesses\" part, what is the most novel point in the proposed method?", "limitations": "The authors have pointed out the limitation and negative impact at the end of the paper, but not provided strategies to address them. I have no suggestions for improvement to the authors either.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The authors conduct sufficient experiments to validate each component of their method and the results on three benchmarks achieve competitive results compared with state-of-the-art methods.\n\nWeaknesses:\n- The biggest weakness lies in the novelty. Specifically, feature decoupling block and diversity loss are consistent with [1]; feature-wise correlation block is similar to [1,2,3,4]; batch-wise correlation block is similar to [5];  Consistency Preserving Strategy is similar to [6]. All these methods have been verified in person re-id. The authors just combine them in a framework, so the novelty is limited.\n- The presentation should be improved greatly. For example, it is preferred to use consistent format to list the references.\n\n[1] Diverse Part Discovery: Occluded Person Re-Identification With Part-Aware Transformer. In CVPR, 2021.\n\n[2] Spatial-temporal graph convolutional network for video-based person re-identification. In CVPR, 2020.\n\n[3] Learning multi-granular hypergraphs for video-based person re-identification. In CVPR, 2020.\n\n[4] Multi-granularity reference-aided attentive feature aggregation for video-based person reidentification. In CVPR, 2020.\n\n[5] Spectral Feature Transformation for Person Re-Identification. In ICCV, 2019.\n\n[6] Masked Graph Attention Network for Person Re-identification. In CVPR workshop, 2019.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "dyQFIiXaph1", "reviewer": "Reviewer_3PRC", "summary": "The authors improved DML by exploring multiple informative relationships among different samples and features from a unified perspective. Current DML methods can only excavate the multiple correlations in different aspects. Considering this problem, they proposed an elaborate metric transformer named MetricFormer to effectively capture and model the multiple correlations. The proposed is generic, effective, and can be easily incorporated into many existing DML methods.", "questions": "- More descriptions should be added to the figures in this paper. For example, The number N of the used Batch-wise Correlation Block should be highlighted in Figure 2.", "limitations": "Yes, limitations have been discussed. I do not find the potential negative societal impact of this work.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "[Strengths]\n- This is a well-written paper. The motivation is clear and the method is easy to follow. By introducing the well-designed transformer blocks, the proposed method can effectively learn more representative features.\n\n- The authors studied DML from a new and unified perspective of correlation exploration. Previous works excavate the multiple correlations in different aspects, while the authors consider the multiple correlations from a unified perspective by the proposed MetricFormer.  The proposed method is generic. It can be widely applied to strengthen existing DML approaches since it can excavate multiple correlations in a unified perspective.\n\n- Comprehensive experiments are provided to understand and evaluate the proposed methods.\n\n[Weaknesses]\n- Although the authors mentioned in their limitations, I think they still should report and analyze their model size and inference cost in their experimental results.\n\n- The effectiveness of the Local Embedding Graph should be discussed in the ablation study.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "VyySTMqOEW", "reviewer": "Reviewer_TNVU", "summary": "This paper proposes a method called MetricFormer, which can extract an ensemble of distinct features, model the relationship among samples in a minibatch, and discover the structural pattern in the ensemble of features. The experimental results on three public datasets are better compared with other approaches. ", "questions": "Please see my questions in the weakness section. ", "limitations": "The equations and variables need to be well defined and add more explanations. ", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The Metric transformer includes three modules: (1) feature decoupling module:  extracting an ensemble of district features; (2) batch-wise correlation block: modeling the sample relationship in each minibatch; (3) feature-wise correlation block: discovering the structural relations among the ensemble of features.\n2. The proposed method results are promising in three public datasets, compared with other approaches. An ablation study is performed to show the effectiveness of each module.\n\nWeaknesses:\n\nThe presentation of this paper needs to be improved. In equation (3), there is no relation between the two equations. Y^{-}_ should be included in the second equation.   Is the dimension or variable D at line 128 the same as the variable C at line 122? What do \\alpha and \\beta mean in equation (9)? What do MetricFormer-M and MetricFormer-PA mean in Table 2?  ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "OzEfiUT2cJ", "reviewer": "Reviewer_dg3A", "summary": "Towards the issue that existing methods try to excavate the multiple correlations in different aspects, this paper provides a unified perspective to learn the multiple correlations, which enables the information propagation along features of different samples within each mini-batch. The proposed MetricFormer introduces a correlation-aware transformer into the internal structure of deep metric networks. It can be ﬂexibly integrated with the existing metric learning framework. Experiments on three widely-used datasets demonstrate its superiority.", "questions": "Please refer to my detailed comments above.", "limitations": "Yes", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\n\n(1) The idea of transformer-based multiple correlations exploration is interesting.\n\n(2) The method is simple and effective. Experimental results are good.\n\nCons:\n\n(1) The writing could be improved. There are several grammar mistakes and typos. For example, \"a elaborate\" should be \"an elaborate\" in line 6. The order of \"Samples\" and \"Classes\" are reversed in Table 1. The abbreviation of \"PA\" and \"M\" should be explained in Table 2.\n\n(2) The novelty might be insufficient. The contribution of existing correlations combination  is not significant.\n\n(3) Please analyze the computational complexity of the proposed transformer based correlation computation and compare it with existing methods.\n\n(4) The reference is inadaquate. [1] also investigates multiple correlations for different tasks. Please also discuss the difference.\n\n(5) Will the codes be released?\n\n[1] Deep Comprehensive Correlation Mining for Image Clustering, ICCV 2019", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept