TransMatcher: Deep Image Matching Through Transformers for Generalizable Person Re-identification
Authors: Shengcai Liao
Ling Shao
Venue: NeurIPS 2021 Poster (2021)
Keywords: Transformer
Deep Image Matching
Deep Metric Learning
Person Re-Identification
Generalizable Person Re-Identification
Domain Generalization

Abstract: Transformers have recently gained increasing attention in computer vision. However, existing studies mostly use Transformers for feature representation learning, e.g. for image classification and dense predictions, and the generalizability of Transformers is unknown. In this work, we further investigate the possibility of applying Transformers for image matching and metric learning given pairs of images. We find that the Vision Transformer (ViT) and the vanilla Transformer with decoders are not adequate for image matching due to their lack of image-to-image attention. Thus, we further design two naive solutions, i.e. query-gallery concatenation in ViT, and query-gallery cross-attention in the vanilla Transformer. The latter improves the performance, but it is still limited. This implies that the attention mechanism in Transformers is primarily designed for global feature aggregation, which is not naturally suitable for image matching. Accordingly, we propose a new simplified decoder, which drops the full attention implementation with the softmax weighting, keeping only the query-key similarity computation. Additionally, global max pooling and a multilayer perceptron (MLP) head are applied to decode the matching result. This way, the simplified decoder is computationally more efficient, while at the same time more effective for image matching. The proposed method, called TransMatcher, achieves state-of-the-art performance in generalizable person re-identification, with up to 6.1% and 5.7% performance gains in Rank-1 and mAP, respectively, on several popular datasets. Code is available at https://github.com/ShengcaiLiao/QAConv.

TransMatcher: Deep Image Matching Through Transformers for Generalizable Person Re-identification
Shengcai Liao" and Ling Shao Inception Institute of Artificial Intelligence (IIAT), Abu Dhabi, UAE
wordpress.com/
Abstract
Transformers have recently gained increasing attention in computer vision. How- ever, existing studies mostly use Transformers for feature representation learning, e.g. for image classification and dense predictions, and the generalizability of Transformers unknown. In this work, we further investigate the possibility of applying Transformers for image matching and metric learning given pairs of im- ages. We find that the Vision Transformer (ViT) and the vanilla Transformer with decoders are not adequate for image matching due to their lack of image-to-image attention. Thus, we further design two naive solutions, i.e. query-gallery concatena- tion in VIT, and query-gale qulry-gallery cross-attention in the vanilla Transformer. The latter improves the performance, but still limited. This implies that the attention mechanism in Transformers is primarily designed for global feature aggregation, which is not naturally suitable for image matching. Accordingly, we propose a simplified decoder, which drops the full attention implementation with the softmax weighting, keeping only the query-key similarity computation. Addition- ally. global max pooling and multilayer perceptron (MLP) head are applied decode the matching result. This way, the simplified decoder is computationally more efficient, while at the same time more effective for image matching. The proposed method, called Trtnhh,,cher, achieves state-of-the-ae generalizable person re- e--ieetification, with up 6. 1.11% % and 5.7% performance gains in Rank-1 and mAP, respectively, on several popular datasets. Code is available at hittp:://gitee http://eeengggaaaaaa//oooooo
performance
Introduction
The Transformer [24] is a neural network based on attention mechanisms. It has shown great success in the field of natural language processing. Recently, it has also shown promising performance for computer vision tasks, including image classification [7] 14], object detection 2 37 14] 26).
and image segmentation [14 26). thus gaining increasing attention in this field. However, existing studies mostly use Transformers for feature representation learning, e.g. for image classification or dense predictions, and the generalizability of Transformers is unknown. similarities are computed by dot products in the attention mechanisms of Transformers. Therefore, these models could potentially be useful for image matching. this work, we further investigate the possibility of applying Transformers for image matching and metric learning given pairs of images, with applications generalizable person re-identification.
glance, query-key
Attention mechanisms are used to gather global information from different locations according query-key similarities. The vanilla Transformer (24 is composed of encoder that employs
Shengeai Liao is the corresponding author.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Enooder
MIP
Head Ti
BNA
GMP
FCi
Iea
Ine-ing
Norm
Product
Attenion
Taard
FC
FCs
Fwed
farreing,s
Frie
Froediigg
CNN
Query
The structure of the proposed Matchhr for image matching. A standard Transformer encoder without positional encoding is used for feature encoding. Then, query and gallery encodings are matched by a dot product. Global max pooling (GMP) is applied find the optimal matching scores and locations, and an MLP head is appended to produce the final matching scores, Note that the batch dimension is ignored in this figure for simplicity.
Figure I:
self-attention, and decoder that further incorporates cross-attention module. The difference is that the query and key are the same in the seff-attention, while they are different in the cross-attention. The Vision Transformer (VIT) 7 applies pure Transformer encoder for feature learning and image classification. While the Transformer encoder facilitates feature interaction among different locations of the same image, it cannot address the image matching problem being studied in this paper, because it does not enable interaction between different images. In the decoder, however, the cross-attention module does have the ability for cross interaction between query and the encoded memory. For example, in the decoder of the detection Transformer (DETR) 221. learnable query embeddings are designed decode useful information in the encoded image memory for object localization. However, the query embeddings are independent from the image inputs, and so there is still no interaction between pairs of input images. Motivated by this, how about using actual image queries instead of learnable query embeddings as input to decoders?
Person re-identification is typical image matching and metric learning problem. In recent study called QAConv [10], was shown that explicitly performing image matching between pairs of deep feature maps helps the generalization of the learned model. This inspires us investigate the capability and generalizability of Transformers for image matching and metric learning between pairs of images. Since training through classification is also popular strategy for metric learning, start from a direct application of ViT and the vanilla Transformer with a powerful ResNet (3) backbone for person re-identification. However, this results in poor generalizann generalization different datasets. Then, we consider formulating explicit interactions between query and gallery images in Transformers. Two naive solutions are thus designed. The first one uses a pure Transformer encoder, as in ViT, but concatenates the query and gallery features together as inputs, so as to enable the self-attention module read both query and gallery content and apply the attention between them. The second design employs the vanilla Transformer, but replaces the learnable query embedding in the decoder by the ready-to-use query feature maps. This way, the query input acts as a real query from the actual retrieval inputs, rather than a learnable query which is more like prior template. Accordingly, the cross-attention module in the decoder is able to gather information across query-key pairs, where the key comes from the encoded memory of gallery images.
While the first solution does not lead to improvement, the second one is successful with notable performance gain. However, compared to the state of the art in generalizable person re- re-identification.
?Query/gallery in person re ideetificatiion and query/key target/memory in Transformers have very similar concepts originated from information retrieval. We use the same word query here different contexts.

---

the performance of the second variant is still not satisfactory. We further consider that the attention mechanism in Transformers might be primarily for global feature aggregation, which not naturally suitable for image matching, though the two naive solutions already enable feature interactions between query and gallery images. Therefore, to improve the effectiveness of image matching, we propose new simplified decoder, which drops the full attention implementation with the softmax weighting, keeping only the query-key similarity computation. Additionally, inspired from QAConv [10) global max pooling (GMP) is applied, which acts as a hard attention to gather similarity values, instead of soft attention to weight feature values. This is because, in image matching, we are more interested in matching scores than feature values. Finally, multilayer perceptron (MLP) head maps the matching result to a similarity score for each query- gallery pair. This way, the simplified decoder is computationally more efficient, while at the same time more effective for image matching.
We call the above design TransMatcher (see Fig. D. which targets at efficient image matching and metric learning in particular. The contributions of this paper are summarized as follows.
We investigate the possibility and generalizability of applying Transformers for image match- ing and metric learning, including direct applications of ViT and the vanilla Transformer, and two solutions adapted specifically for matching images through attention. This furthers our understanding of the capability and limitation of Transformers for image matching.
According the above, new simplified decoder is proposed for efficient image matching, with a focus on similarity computation and mapping.
With generalizable person re-identification experiments, the proposed Trache is shown to achieve state-of-the-ee performance several popular datasets, with up to 6. % and 5.7% performance gains in Rank- and mAP, respectively.
2
Related Work
Given pairs of input images, deep feature matching has been shown to be effective for person re-identification. Li et al. [8] proposed novel filter pairing neural network (FPNN) to handle mis- aligmment and occlusions in person re-identification. Ahmed et al. []] proposed local neighborhood matching layer to match deep feature maps of query and gallery images. Suh et al. [21] proposed deep neural network to learn part-aligned bilinear representations for person re-identification Shen et al. (18] proposed Kronecker-prodee -roouuct matching (KPM) module for matching person images in softly aligned way. Liao and Shao [10] proposed the query adaptive convolution (QAConv) for ex- plicit deep feature matching, which is proved to be effective for generalizable person re- identification. They further proposed a graph sampler (GS) for efficient deep metric learning
Generalizable person re-identification has gained increasing attention in recent years. Zhou et al. (36 proposed the OSNet, and showed that this new backbone network has advantages in generalization. Jia et al. (5] applied IBN-Net-b [15] together with a feature normalization to alleviate both style and content variance across datasets to improve generalizability. Song et al. (20] proposed a domain- invariant mapping network (DIMN) and further introduced a meta-learning pipeline for effective training and generalization. Qian et al. 17 proposed a deep architecture with leader-based multi- scale attention (MuDeep), with improved generalization of the learned models. Yuan et al. [31 proposed an adversarial comainn-nvarrian feature learning network (ADIN) to separate identity-related features from challenging variations. Jin et al. [6] proposed a style normalization and restitution module, which shows good generalizability for person re-identification. Zhuang et al. (38] proposed camera-based batch normalization (CBN) method for domain-invariant representation learning, which utilizes unlabeled target data to adapt the BN layer in a quick and unsupervised way. Wang et al. (27) created a large-scale synthetic person dataset called RandPerson, and showed that models learned from synthesized data generalize well to real-world datasets. However, current methods are still far from satisfactory in generalization for practical person re-identification.
There are a number of attentional networks 12
proposed for person re- ieiidiifiiiaain, but focusing on representation learning. More recently, Zhao et al. [33 proposed cross-attention network for person re-identificaiton. However, it is still applied for feature refinement, instead of explicit image matching between gallery and probe images studied in this paper.
16
13
30
32
Transformers have recently received increasing attention for computer vision tasks, including image classification 7 [4] object detection 2 37 [4] 26), image segmentation [4. 26) and on. For
3

---

example, VIT was proposed in [7], showing that a pure Transformer-bise based architecture is capable of effective image classification. DETR was proposed in [2]. providing a successful end-to-end Transformer solution for object detection. Later, several studies, such as the Deformable DETR [7 Swin [4), and PVT (26), improved the computation of Visual Transformers and further boosted their performance. However, existing studies mostly use Transformers for feature representation learning, e.g. for image classification and dense predictions. There lacks a comprehensive study on whether Transformers are effective for image matching and metric learning and how its capability is in generalizing to unknown domains.
3 Transformers
For the vanilla Transformer [24], the core module is the multi-head attention (MHA). First, a scaled dot-product attention is defined as follows:
Attention(Q,K, softmax QK Vax VV,
where Q Txxd is the query (or target) matrix, K E R. Mxdx the key (or memory) matrix, V E IR M xd. is the value matrix, T and M are the sequence lengths of the query and key, respectively, de the feature dimension of the query and key, and d, the feature dimension of V. In visual tasks, Q and K are usually reshaped query and key feature maps, with T= M = haw, where h and w are the height and width of the query and key feature maps, respectively. Then, the MHA is defined as:
head;- = Attention KW,, VW.),
(2)
MultiHead (Q, K, V) = Concat(head,,...
headh))
(3)
where W.O. E Bddxds, W.K E gdxds, W.V E Rdxd, and W E Rgd xd are parameter matrices, and H is the number of heads. Then, Q K V in the multi-head self-attention (MHSA) in the encoders, while they are defined separately in the multi-head cross-attention (MHCA) in the decoders.
The structure of the Transformer encoder without positional encoding is shown on the left of Fig. I. Beyond MHSA, it further appends ferd-forward layer to first increase the feature dimension from d to D, and then recover it back from D to d. Besides, the encoder can be self-stacked N times, where N is the total number of encoder layers. In ViT 77, only Transformer encoders are used, and positional encoding is further applied. In the vanilla Transformer (24), decoders with MHCA are further applicd, with the query being learnable query embeddings initially, and the output of the previous decoder layer later on, and the key and value being the output of the encoder layer. The decoder can also be self-stacked N times.
4 Image Matching with Transformers: Naive Solutions
While the above ViT and vanilla Transformer are able to perform image matching through black-box feature extraction and distance learning, they are not optimal for this task because they lack image-to- image interaction in their designs. Though cross-attention is employed in the Transformer decoders, in its original form the query either comes from learnable query embeddings, or from the output of the previous decoder layer.
Therefore, adapt Transformers with two naive solutions for image matching and metric learning- Building upon a powerful ResNet 3 backbone, the first solution appends ViT, but not simply for feature extraction. Instead, a pair of query and gallery feature maps are concatenated to double the sequence length, forming a single sample for the input of ViT. Thus, both the query and key for the self-attention layer contain query image information in one half and gallery image information the other half. Therefore, the attention computation in Eq. (I) is able to interact query and gallery inputs for image matching. This variant is denoted as Transformer-Cat
The second solution appends the vanilla Transformer, but instead of learnable query embeddings, ResNet query features are directly input into the first decoder. This way, the cross-attention layer in the decoders is able to interact the query and gallery samples being matched. This variant is denoted Transformer-Cross.
The structure these two variants can be found in the Appendix. Note that these two solutions have high computational and memory costs, especially for large d, D, and N (c.f. Section 6.4).

---

5 The Proposed TransMatcher
Though the above two solutions enable query-gallery interaction in the attention mechanism for image matching, they are not adequate for distance metric learning. This is because, taking a deeper look at Eq. (T) for the attention, it can be observed that, though similarity values between Q and K are computed, they are only used for softmax-base weighting to aggregate features from V. Therefore, the output of the attention is always weighted version of V (or K), and thus cross- cross-matching between a pair of inputs is not directly formulated.
To address this, we propose a simplified decoder, which is explicitly formulated towards similarity computation. The structure of this decoder is shown in the middle of Fig. T First, both gallery and query images are independently encoded by N sequential Transformer encoders after backbone network, as shown on the left of Fig. [ This encoding helps aggregating global information from similar body parts for the subsequent matching step. The resulting feature encodings are denoted by Qn E ghwud and Kn E Rawxdd n = 1..... N, for the query and gallery, respectively. Then, as in Eq. both the gallery and query encodings are transformed by fully connected (FC) layer FC1: Ø Wn K₁, = K,, Wn.
(4)
where Wi, E Rddd is the parameter matrix for encoder-decoder layer n. Different from Eq. 2 we use shared FC parameters for both query and gallery, because they are exchangeable the image matching task, and the similarity metric needs to be symmetrically defined. Then, the dot product is computed between the transformed features, as in Eq. T
=
(5)
where Sn, E Rouxkw are the similarity scores. In addition. learnable prior score embedding R E R hhwxku is designed, which defines prior matching scores between different locations of query and gallery images. Then, it is used weight the similarity values:
S% = Sn * o (R),
(6)
where * denotes deement-wise multiplication, and o is the sigmoid function to map the prior score embedding into weights in [0, 1].
After that, a GMP layer is applied along the last dimension of ha elements:
S" max dim=
This way, the optimal local matching over all key locations is obtained, as in QAConv 10 Compared to Eq. (T). the GMP here can be considered as a hard attention, but it is used for similarity matching rather than softmax-based feature weighting like in the soft attention. Note that multi-head design in MHA is not considered here (c.f. Section 6.6).
Then, after a batch normalization layer BN1, an MLP head is further appended, similar to the feed- forward layer of Transformers. is composed of MLPHeud,=(FC, BN2. ReLU) map the hw similarity values to dimension D, and MLPHcad,=((C3,, BN₃) to map dimension D to as a single output score S""
Finally, decoder n outputs a similarity score by fusing the output of the previous decoder:
S = S"" n + S"" n-1, where S"" is defined as O. With N stacked encoder-decoder blocks, as shown in Fig. 1 this can be considered as residual similarity learning. Note that the stack of encoder-decoder blocks in TransMatcher is different from that in the vanilla Transformer. In TransMatcher, the encoder and decoder are connected before being stacked, while in the vanilla Transformer they are stacked independently before connection. This way, the decoder of Mrtcher is able to perform cross matching with different levels of encoded features for residual similarity learning.
(8)
However, the GMP operation in Eq. is not symmetric. To make TrahsM symmetric for the query and gallery, the GMP operation in Eq. 77) can also be applied along dim=0; that is, conduct an inverse search of best matches over all query locations. Keeping other operations the same, this will result in another set of similarity scores, which are summed with the original ones after the FC3 layer. Further details can be found in the Appendix. Note that this is not reflected in Fig. for simplicity of illustration.
Finally, the outputs of Mather scores for all query-gallery pairs in batch are collected for pairwise metric learning following the same pipeline QAConv-GS 11, and the same binary cross entropy loss is used as in the QAConv-GS.
5

---

6 Experiments
6.1 Datasets
Four large-scale person re-identification datasets, CUHK03 8 Market-1501
34 MSMT17 28
and RandPerson [27]. which are publicly available for research purpose, are used our experiments. The CUHK03 dataset includes 1,360 persons and 164 images, with 767 and 700 subjects used for training and testing, respectively, as the CUHK03-NP protocol [35]. Besides, the "detected" subset is used, which is more challenging than the "labeled" subset. The Market-1501 dataset contains 32,668 images of I 1,501 identities captured from six cameras, with 12,936 images from 751 identities for training, and 19,732 images from 750 identities for testing. MMMM..7 includes 4,101 identities and 126,441 images captured from 15 cameras, with 32,621 images from 1,041 identities for training, and the remaining images from 3,010 identities for testing. RandPerson is recently released synthetic person re- re-identification dataset for large-scale training towards generalization testing. It is with 8,000 persons and 1,801,8 886,816 images. A subset with 132,145 images of the 8,000 IDs is used for training.
Cross-dataset evaluation is performed on these datasets by training on the training subset of one dataset, and evaluating on the test subsets of other datasets. Except that for MSMT17 we further use an additional setting with all images for training. regardless of the subset splits. This is denoted by MSMT17 All evaluations follow the single-query evaluation protocol. The Rank-1 (Topl) accuracy and mean average precision (mAP) are used as the performance evaluation metries.
6.2 Implementation Details
The implementation of TransMatcher is built upon the official PyTorch project of QAConv-Gr
[T. as the graph sampler (GS) proposed in this project is efficient for metric learning and quite
suitable for the learning of TransMatcher. We keep most of the settings the same as QAConv-Gr
Specifically, ResNet-50 [3] is used as the backbone network, with three instance normalization (IN)
[33 layers further appended as in IBN-Net-b [[5), following several recent studies [5] 36 6 38 I.I.
The backbone network is pre-trained on ImageNet, with the states of the BN layers being fixed. The
layer3 feature map is used, with a 3 neck convolution appended produce the final feature map.
The input image is resized to 384 x 128. The batch size is set to 64, with K=4 for the GS sampler. The network is trained with the SGD optimizer, with learning rate of 0.0005 for the backbone network, and 0.005 for newly added layers. They are decayed by 0.1 after 10 epochs, and 15 epochs are trained in total. Except that for RandPerson (27) the total number of epochs is 4, and the learning rate step size is 2, according to the experiences in [T) Gradient clipping is applied with 7 = 4 [. Several commonly used data augmentation methods are applied, including random flipping. cropping, occlusion, and color jittering. All experiments are run on single NVIDIA V100 GPU.
For the proposed Matcher, unless otherwise indicated, d=512 and D=2048 by default as in the original Transformer (24), and H=1 and N=3 for higher efficiency. Please refer to Section 6.6 for further parameter analysis. Besides, in practice,
find that when N decoders used, using N 1 encoders together with the ResNet feature map directly pairing the first decoder slightly improves the results while being more efficient, which is preferred in the implementation (c.f. Appendix).
6.3 Comparison to the State of the Art
A comparison to the state of the art (SOTA) in generalizable person re-identification is shown in
Table D Several methods published very recently for generalizable person re-identification compared, including OSNet [36), MuDeep [7]. ADIN (31], SNR (6. CBN (38), QAConv [10], and QAConv-GS (.. From Table 1 can be observed that TrahhM significantly improves the previous SOTA. For example, with Market-1501 for training, the Rank-1 and mAP are improved by 5.8% and 5.7% on CUHK03-NP, respectively, and they are improved by 6.1% 11 and 3.4% on MSMT17, respectively. With MSMT17 - Market- 1501, the improvements are 5.0% for Rank- 1 and 5.3% for mAP. With the synthetic dataset RandPerson for training. the improvements on Market-1500 are 3.3% for Rank-1 and 5.3% for mAP, and the gains on MSMT17 are 5.9% for Rank-1 and 3.3% for mAP.
Compared to the second best method QAConv-GS, since it shares the same code base and training setting with the proposed Mathhrr it indicates that Mathhrr is a superior image matching
QQAConv-GS GS project under MIT License: httpp////hhhhhbbbcm//////eeeeeal

---

Table 1:
Comparison of the state-of-the- direct cross-dataset evaluation results (%). MSMT means all images are used for training, regardless of the subset splits.
CUHK03-NP
Market-1501
MSMT17
Method
Venue
Train Set
Rank-1
mAP
Rank-1
mAP
Rank-1
MAP
MGN [25
MM 18
Market
8.5
7.4
MuDeep 17
PA
Market
10.3
9.1
CBN B8
2CCV'20
Market
25.3
9.5
QAConv 10
2C
Market
9.9
8.6
22.6
7.0
QAConv- GS [
arXiv'21
Market
16.4
15.7
41.2
15.0
Machher
Ours
Market
22.2
21.4
47.3
18.4
PCB [22
ECCV' 18
MSMT
52.7
26.7
MGN 31
MM' 18
MSMT
48.7
25.1
ADIN
31
WACV'20
MSMT
59.1
30.3
SNR
6
CVPR 20
MSMT
70.1
41.4
CBN 38
ECCV'20
MSMT
73.7
45.0
QAConv-Gr
arXiv'21
MSMT
20.0
19.2
75.1
46.7
Transhatcher
Ours
MSMT
23.7
22.5
80.1
52.0
OSNet [36
CVPRT19
MSMT all
66.5
37.2
QAConv [10
ECCV'20
MSMT all
25.3
22.6
72.6
43.1
QAConv-GS
arXiv'21
MSMT alt
27.2
27.1
80.6
55.6
TrahsMatcher
Ours
MSMT all
31.9
30.7
82.6
58.4
RP Baseline 27
MM' 20
RandPerson
13.4
10.8
55.6
28.8
20.1
6.3
QAConv-GS [
arXiv '21
RandPerson
14.8
13.4
74.0
43.8
42.4
14.4
TransMatcher
Ours
RandPerson 17.1
16.0
77.3
49.1
48.3
17.7
Table 2: Comparison of different Transformers trained MSMT17 for direct cross-dataset evaluation (%). m/Acc (%) is the average of all Rank-1 and mAP results on both CUHK03-NP and Market-1501 over four random runs.
CUHK03-NV
Market-
Method VIT
d
D
N
Time (h)
mAce
Rank-1
mAP
Rank-1
mAP
512
2048
0.99
12.0
12.4
57.7
29.8
27.42
Transformer
512
2048
1.16
13.2
13.3
54.3
29.0
27.01
TransMatcher
512
2048
1.44
23.7
22.5
80.1
52.0
44.29
Transformer Cat
128
512
4.89
13.1
13.2
53.9
27.4
25.34
Transformer-Co Cross
128
512
3.48
18.9
19.8
66.2
40.1
36.70
TransMatcher
128
512
1.11
22.5
21.4
77.4
49.3
42.12
and metric learning method for generalizable person re-identification, thanks to the effective cross- matching design the new decoders.
6.4 Comparison of Transformers
comparison of different Transformers trained on MSMT17 for direct cross-dataset evaluation shown in Table For fair comparison, they are all trained with the same settings described in Section 6.2 Besides, H=1 for all models. ViT, the vanilla Transformer, and TransMatcher all have the same parameter settings. Though Transformer- Cat and Transformer-Cros still encounter the memory overflow problem under the same parameter settings as Trahsr. Therefore, we have to set d=128, D=512, and N=2 for them to run, and accordingly, a smaller version TrahsMatcher with the same set of parameters is also provided for comparison.
an NVIDIA V100 GPU with 32GB of memory,
From the results shown Table 2 it can be observed that VIT and the vanilla Transformer perform poor in generalizing to other datasets. In contrast, the proposed TransMatcher significantly improves the performance. This confirms that simply applying Transformers for the image matching task not effective, because they lack cross-image interaction in their designs.

---

Table 3: Ablation of different components in TransMatcher. Training is performed MSMT17.
mAccc (%) is the average of all Rank-1 and mAP results on all test sets over four random runs. PriorEmbed is the prior score embedding. while PosEmbed is the positional embedding.
FC1
BN1
MLPHead, MLPHead2 PriorEmbed PosEmbed
mAcc
41.44
42.82
43.66
43.70
44.04
44.29
42.39
42.56
Besides, we find that Transformer-Cat does not lead to improvement compared to VIT and the vanilla Transformer. It is smaller model, though. However, Transformer-Cross does lead to notable improvements, indicating that the cross-matching of gallery and query images in Transformer decoders is potentially more effective. However, it is still not good as the smaller version of TransMatcher. For example, on Market-1501, TransMatcher improves the Rank-1 by 11.2% and the mAP by 9.2% over the Transformer-Cross Therefore, the cross-attention design in the original Transformers is not efficient enough for image matching, due to its focus on feature aggregation but not similarity matching. More variants and experiments of Transformers can be found in Appendix.
As for the running speed, the training times of these methods are also listed Table 2 As can be seen, without cross-matching, VIT is the most efficient, followed by the vanilla Transformer. TransMatcher is not as efficient as ViT due to the explicit cross-matching between query and gallery images. However, it is still acceptable, thanks to the new simplified decoder. In contrast, with small set of parameters, Transformer-Can and Transformer-Cross are still quite heavy to compute.
6.5 Ablation Study
The structure of the proposed Trahher shown in Fig. D is carefully ablated, with results listed in Table The training is performed on MSMT17. For ease and reliable comparison, we report the average of all Rank-1 I and mAP results on all test over four random runs. This is denoted by mAcc. We start with Dot Product + GMP + MLPHeadz (the input dimension to FCa needs to be adapted to hw accordingly), which is the simplest and most necessary configuration. Then, by adding MLPHead,, the performance is improved by 1.38%, indicating that increasing the dimension D, in Transformers, is useful. Then, by including FC1 BN, independently. the performance gain is 0.84% / 0.88%, and by including them together, the performance can be further improved. Finally, when the prior score embedding appended, the best performance achieved. Interestingly, when we include learnable positional embedding in the encoders, as in VIT, either independently or together with the prior score embedding, the performance degraded. This indicates that mixing the position information with visual features for image matching is not useful in our design. In contrast, learning spatial-aware prior matching scores separately for score weighting is more effective. More ablation study and analysis can be found in the Appendix.
6.6 Parameter Analysis
To understand the parameter selection of the proposed TransMatcher, we train it on MSMT17 with different parameter configurations the defaults, with the mAcc results as well as the training time shown in Fig. 2 First, the performance is gradually improved by increasing the model dimension d. However, the training time is also increased quadratically. Therefore, to provide balance between accuracy and running speed, d=512 is selected, which is the same as in the vanilla Transformer (24]
For the feed value. However, the training time is less affected, because the cerw-forward operation is only applied after the dot product and GMP, where the dimension of d and one spatial dimension ha are already contracted. Nevertheless, large D will increase the memory usage. Therefore, D=2048 selected, which is also the same in the vanilla Transformer (24
D, the performance is also gradually improved when increasing the

---

Tr
Tire
1000
2000
42.27
42.45
41:2
44:33
1000
7000
40.5
4000
4723
1993
900
4075
6396
5332
3000
5000
64
120
226
532
1004
3045
64
112
330
512
3224
2046
Tru
PAAcE
Time
45.0
66.7
5000
44.14
4336
42.65
2000
1000
200
0065
55.26
5090
X00
0454
1000
3000
Figure 2: Parameter analysis of the proposed TransMatcher.
As for the number of layers N, the performance is also gradually improved with increasing N. However, after N=3 the performance tends to saturate, and the training time grows linearly with the increasing number of layers. Therefore, N=3 is reasonable balance for choice. In addition, with N = there is no encoder used (for details please see Appendix), and from Fig. 2 it is clear that this is inferior, indicating that including encoder is important. On the other hand, from the poor performance of ViT where there are only encoders, it is clear that the decoder is also important.
Finally, for the number of heads H in the encoders, it appears that larger H does not lead to improved results. Since the training time is also not affected, we simply select H=l the encoders, and do not implement the multi-head mechanism in the decoders.
6.7 Qualitative Analysis
With the help of the GMP layer, inspired from QAConv 110, the proposed Trahhe is able to find the best local correspondence matches each decoder layer. Some qualitative matching results shown in Fig. for better understanding of TransMatcher. More examples can be found in the Appendix. The model used here is trained on the MSMT17 dataset (28). and the evaluations are done
on the query subset of the Market-1501 dataset [34]. Results of both positive pairs and hard negative pairs are shown. For clear illustration, only reliable correspondences with matching scores over a certain threshold shown, where the threshold is determined by false acceptance rate of 1% over all matches negative pairs. Note that the local positions are coarse due to the 24 x 8 size of the feature map.
As can be observed from Fig. 3 the proposed method is able to find correct local correspondences for positive pairs of images, even if there are notable misalignments in both scales and positions, pose, viewpoint, and illumination variations, occlusions, and low resolution blur. Besides, for hard negative pairs, the matching of TransMatcher still appears to be mostly reasonable, by linking visually similar parts or even the same person who might incorrectly labeled.
This indicates that the proposed TrahsMatcher is effective in local correspondence matching, and note that it learns do this with the only supervision of identity information. Besides, the matching capability is generalizable to other datasets beyond the training set. From the illustration it can also be seen that, generally, matching results of the first decoder layer are not as successful as the next two layers, and the matching with the last decoder layer appears to be the best. This indicates that both Transformer encoders and decoders helps the model to match better by aggregating global similarity information.

---

0.84
0.76
0.82
0.80
0.77 (a) Positive pairs
0.75
0.93
0.79
0.79
0.68
0.70 (b) Negative pairs
0.75
Figure 3: Examples of qualitative matching results the Market-1501 dataset, by the proposed Mrthhrr trained on the MSMT17 dataset. For each pair of images, local correspondence matches found on the three layers of the TransMatcher are shown. Numbers represent similarity scores.
Conclusion
With the study conducted in this paper, conclude that: (1) direct applications of VIT and the vanilla Transformer are not effective for image matching and metric learning, because they lack cross-image interaction in their designs; (2) designing query-gallery concatenation in VIT does help, while introducing query-gallery cross-attention in the vanilla Transformer leads to notable but not adequate improvements, probably because the attention mechanism Transformers might be primarily designed for global feature aggregation, which is not naturally suitable for image matching; and (3) new simplified decoder thus developed, which employs hard attention to mraching similarity scores, is more efficient and effective for image matching and metric learning. With generalizable person re-identification experiments, the proposed Mrthhhr is shown achieve state-of of-the-art performance on several popular datasets with large improvements. Therefore, this study proves that Transformers can be effectively adapted for the image matching and metric learning tasks, and other potentially useful variants will be of future interest.
Acknowledgements
The authors would like to thank Yanan Wang who helped producing Fig. [ in this paper, and Anna Hennig who helped proofreading the paper, and all the anonymous reviewers for the valuable feedbacks in improving the paper.
10

---

References
[1] Ejaz Ahmed, Michael Jones, and Tim K Marks. An improved deep learning architecture for person re-identificatio. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3908-3916, 2015.
[2 Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usumier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV, 2020.
[3] K. He, X. zhang, S. Ren, and Sun. Deep residual learning for image recognition. In Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pages 770-778, 2016.
[4] Ruibing Hou, Bingpeng Ma, Hong Chang, Xinqian Gu, Shiguang Shan, and Xilin Chen. Interaction- and-aggregation network for person re-identification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9317-9326, 2019.
[5] Jieru Jia, Qiuqi Ruan, and Timothy M Hospedales. Frustratingly easy person re- re-identification: Generalizing person re-id in practice. In British Machine Vision Conference, 2019.
[6] Xin Jin, Cuiling Lan, Wenjun Zeng, Zhibo Chen, and Li Zhang. Style Normalization and Restitution for Generalizable Person Re-identification. In CVPR, feb 2020.
[7] lexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.
[8] Wei Li, Rui Zhao, Tong Xiao, and Xiaogang Wang. DeepRelD: Deep filter pairing neural network for person re-identification. In IEEE Conference on Computer Vision and Pattern Recognition, 2014.
[9] Wei Li. Xiatian Zhu, and Shaogang Gong, Harmonious attention network for person re-identification. In Proceedings the IEEE Conference on Computer Vision and Pattern Recognition, pages 2285-2294, 2018.
[10] Shengeai Liao and Ling Shao. Interpretable and Generalizable Person Re- Re-Identification with Query- Adaptive Convolution and Temporal Lifting. In European Conference on Computer Vision (ECCV). 2020.
[11] Shengeai Liao and Ling Shao. Graph Sampling Based Deep Metric Learning for Generalizable Person Re- Re-Identification. CoRR, abs/2104.01 0156,, April 4, 2021.
[12] Hao Liu, Jiashi Feng. Meibin Qi, Jianguo Jiang, and Shuicheng Yan. End-to-end comparative attention networks for person re-identification. IEEE Transactions on Image Processing, 26(7):3992-3566 2017.
[13] Xihui Liu, Haiyu Zhao, Maoqing Tian, Lu Sheng. Jing Shao, Shuai Yi, Junjie Yan, and Xiaogang Wang. Hydraplus-ner net: Attentive deep features for pedestrian analysis. In Proceedings of the IEEE international conference computer vision, pages 350-359, 2017.
[14] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin Transforme: Hicrarchical Vision Transformer using Shifted Windows. CoRR, 10332103.111 14030, 2021.
[15] Xingang Pan, Ping Luo, Jianping Shi, and Xiaoou Tang. Two at once: Enhancing learning and general- ization capacities via ien-net. In Proceedings of the European Conference on Computer Vision (ECCV). pages 464-479, 2018
[16] Xuelin Qian, Yanwei Fu, Yu-Gang Jiang, Tao Xiang, and Xiangyang Xue. Multi-scale deep learning architectures for person re-identification. In Proceedings of the IEEE International Conference on Computer Vision, pages 5399-5408, 2017.
[17] Xuelin Qian, Yanwei Fu, Tao Xiang. Yu-Gang Jiang, and Xiangyang Xue. Leader-based Multi-Scale Attention Deep Architccture for Person Re-identification. TPAMI, 2020.
[18] Yantao Shen, Tong Xiao, Hongsheng Li, Shuai Yi. and Xiaogang Wang. End-to-end deep kronecker- product matching for person re-identification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6886-6895, 2018
[19] Jianlou Si, Honggang Zhang, Chun-Guang Li, Jason Kuen, Xiangfei Kong, Alex C Kot, and Gang Wang. Dual attention matching network for context-awan aware feature sequence based person re-identification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2018

---

[20] Jifei Song, Yongxin Yang, Yi-Zhe Song, Tao Xiang, and Timothy M Hospedales. Generalizable person re- identification by domain-invariant mapping network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 719-728, 2019.
[21] Yumin Suh, Jingdong Wang, Siyu Tang, Tao Mci, and Kyoung Mu Lec. Part-aligned bilincar representations for person re-identification. Proceedings of the European Conference on Computer Vision (ECCV), pages 402 402-419, 2018.
[22] Yifan Sun, Liang Zheng, Yi Yang, Qi Tian, and Shengjin Wang. Beyond part models: Person retrieval with refined part pooling (and a strong convolutional baseline). In Proceedings of the European Conference on Computer Vision (ECCV), 2018.
[23] Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky, Improved texture networks: Maximizing quality and diversity in Computer Vision and Pattern Recognition, CVPR 2017, 2017.
stylization and texture synthesis. In Proceedings - 30th IEEE Conference on
[24] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Eukasz Kaiser, and Illia Polosukhin. Altention is all you need. In Advances in Neural Information Processing Systems, June 2017.
[25] Guanshuo Wang, Yufeng Yuan, Xiong Chen, Jiwei Li, and Xi Zhou. Learning discriminative features with multiple granularitics person re-identification. 2018 ACM Multimedia Conference on Multimedia Conference, pages 274 282. ACM, 2018.
[26] Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song. Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions. CoRR, abs/2102.12112. 2021.
[27] Yanan Wang, Shengeai Liao, and Ling Shao. Surpassing Weal-World Source Training Data: Random 3D Characters for Generalizable Person Re-Identification In 28th ACM International Conference on Multimedia (ACMMM), 2020.
[28] Longhui Wei, Shiliang Zhang, Wen Gao, and Qi Tian. Person transfer gan to bridge domain gap for person re-identification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 79-88, 2018
[29] Jing Xu, Rui Zhao, Feng Zhu, Huaming Wang, and Wanli Ouyang, Attention -warre compositional network for person re- re-identification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2119 21282 2018
[30] Shuangjie Xu, Yu Cheng, Kang Gu, Yang Yang. Shiyu Chang, and Pan Zhou. Jointly attentive spatial- temporal pooling networks for video-based person re-identification. In Proceedings of the IEEE Interna- tional Conference on Computer Vision, pages 4733-4742, 2017.
[31] Ye Yuan, Wuyang Chen, Tianlong Chen, Yang Yang. Zhou Ren, Zhangyang Wang, and Gang Hua. Calibrated Invariat Learning for Highly Generalizable Large Scale Re- Identification. WACV, pages 3578 3587, nov 2020.
[32] Zhizheng Zhang, Cuiling Lan, Wenjun Zeng. Xin Jin, and Zhibo Chen. Relation- glohal attention for person re-identification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 31 3186-3195, 2020.
[33] Shizhen Zhao, Changxin Gao, Jun Zhang, Hao Cheng, Chuchu Han, Xinyang Jiang, Xiaowei Guo, Wei-Shi Zheng, Nong Sang, and Xing Sun. Do not disturb me: Person re-identification under the interference of other pedestrians. In European Conference on Computer Vision, pages 647-663. Springer, 2020.
[34] I.. Zheng, I. Shen, L. Tian, S. Wang, J. Wang, and Q. Tian. Scalable person re-identification: A benchmark. In Proceedings of IEFE International Conference Computer Vision, 2015.
[35] Zhun Zhong, Liang Zheng, Donglin Cao, and Shaozi Li. Re-ranking person re-identification with k- reciprocal encoding. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pages 1318-1327. 2017.
[36] Kaiyang Zhou, Yongxin Yang, Andrea Cavallaro, and Tao Xiang, Omni-scale feature learning for person re-identification. In Proceedings of the IEFE International Conference on Computer Vision, 2019.
[37] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable DETR: Deformable Transformers for End-to-Fnd Object Detection. CoRR, abw/2010.04119:91-- 2020.
[38] Zijie Zhuang, Longhui Wei, Lingxi Xie, Tianyu Zhang, Hengheng Zhang. Haozhe Wu, Haizhou Ai, and Qi Tian. Rethinking the Distribution Gap of Person Re-identification with Camera based Batch Normalization. In ECCV, pages 140-157, 157. 2020.
12


Reviews:
[{"review_id": "rD0pom_ApdT", "reviewer": "Reviewer_SVbe", "summary": "This paper studies person re-identification using visual transformers with query and gallery interactions. The paper presents first simple query and gallery image concatenation and cross attention layer for interaction query and gallery interactions; and later presents the proposed TransMatcher architecture, which replaces the decoder self-attention by only keeping query-key matching followed by global max pooling and multi-layer perceptron head maps composed of fully connected and batch norm layers.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The analysis on the standard transformer for detection is interesting. I further appreciated that it presents the naive extensions that do not lead to improvement along with the proposed model that leads to improvement.\n\nThe visual transformers is an active area of research and most of the very recent (a few months) papers are also cited in the related work. \nThe experimental setup contains comparisons between the proposed approach and several earlier studies on the re-identification task. However, it would be further interesting to see how the proposed model performs compared to DeTR which is known for object detection on the re-identification task.\n\nIt is not very clear how the global max pooling is symmetrized, could you please provide the equations of this part and the concatenation afterwards, what is the dimension of the output? \n\nHow are the prior score embeddings obtained? It is not clear why it is used, what is the rationale behind it?  \n\nSharing the code will increase the reproducibility of results. Could you please share your implementation?\n\n\n---\nLine 170, 171, 173, 177 $\\mathbb{R}$ instead of $R$ for the dimensions\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "It mentions the limitations and possible societal impact.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4-5 hours"}, "extra_text": {}}, {"review_id": "dWElXnwiH1", "reviewer": "Reviewer_Gj64", "summary": "This paper investigates how to apply the transformer architecture to solve the image match problem of generalised person Re-ID. The features of query/gallery images are first extracted with a ResNet model, then a transformer encoder without positional embedding is applied to query/gallery features, respectively. A customised decoder is designed to obtain the final match score for the following pairwise metric learning. The experiments are conducted on several common Re-ID datasets and the proposed method have shown better performance than the baseline.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "STRENGTH\n+ The performance on CUHK03/Market-1501/MSMT17 is superior to the baseline methods.\n+ The idea of applying the transformer model to person Re-ID seems to be new.\n+ The paper writing is easy to follow.\n\nWEAKNESS\n- There are other attention modules, e.g. no-local or channel-wise attention, why particularly use the transformer if no positional embedding is required?\n- The proposed pipeline is still a naive solution. TThe transformer encoder is simply used to refine the ResNet features, while the query /gallery features are combined together through a simple dot product in decoder. I don't think the cross attention between query/gallery can be learned in this way - it is more likely the performance improvement is gained by the transformer encoder instead of the proposed decoder.\n- The comparison with ViT and transformer in Table 2 is not quite fair, since the authors use a smaller transformer model (line 263-267) due to memory overflow problem. The original settings may actually work better\n- Since both person-ID and self-attention have the same concept \"query\" but having quite different meanings, the authors should make clear which query is referred to in the paper to avoid confusion.\n- Intuitive visualizations can be helpful to reveal the reason why the transformer can work better for person-ReID problems.\n\nOverall, although this paper has reported higher performance than the baseline, the motivation of using transformer is not very clear, the pipeline design is naive and lack novelties, and some experimental results are still questionable. Considering the superior performance, I vote for a marginally below.\n\n------After rebuttal------\nThe authors solved parts of my concerns and therefore I increase the rating to marginally above (6).", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, I think so.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "6638JBPdhb8", "reviewer": "Reviewer_MpiZ", "summary": "This paper applies Transformer to the matching problem and metric learning. The direct applications of ViT and the vanilla Transformer, and two solutions adapted specifically for matching images through attention. A new simplified decoder is proposed for efficient image matching, focusing on similarity computation and mapping. ", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nOriginality: Middle \n\n(+) This paper is to compare and extend the prevalent ViT[6] and DeTR[2], which contain both self-attention and cross (gallery-probe)-attention, in the person re-id. \n\n(-) There are many works that use self-attention for person re-id. Such existing works are not mentioned at all. For example, \n\n[A] Z.Zhang et al., Relation-Aware Global Attention for Person Re-identification, CVPR2020.\n[B] R.Hou et al., Interaction-and-Aggregation Network for Person Re-identification, CVP2019. \n\n(-) There are several cross-attention works on person re-id; approach which maximizes the similarity of local features of gallery and probe pair. Such existing works are not explained. For example, \n\n[C] S.Zhao et al., Do Not Disturb Me: Person Re-identification Under the Interference of Other Pedestrians, ECCV2020 \n\n\nQuality: Middle\n\n(+) Direct application of Transformer and naïve application of Transformer to image matching problem are compared. \n\n(+) The accuracies are better, and the training time is shorter than the compared naïve application of Transformer. \n\n(+) The proposed method shows better rank-1 rates and mAP than the results reported on state-of-the-art domain generalized person re-id works. \n\n(-)  Details of baseline architecture are not clear.  \n- ViT is an image classification model. Is the same binary cross-entropy loss in GS sampler applied for all methods? Is the same MLP head applied as the proposed model?   \n- For Transformer-Cross, what is the value $V$ in the decoder in Eq.(1) ? Why are the Resnet query features directly used (line155) without encoders, unlike the proposed model?\n\n(-) The performance of Transformer-Cat in Table 2 is lower than vanilla Transformer. How about appending the results of the Transformer in the lower block?\n\n(-) The proposed method applies many improvements to the Transformer. For example, \n- The proposed method uses the shared FC parameter to query and key (line-174-175). \n- Multi-scale similarity scores are summed in the proposed model as in Eq.(8). \n\nHowever, Table 2 compares only with the final model. Thus, we can not see how much each component improved the naive application of the Transformer. We can see the ablation study, but improvement to the naïve application of Transformer is more understandable by adding the new component to Transformer-Cross. \n\n(+) This paper shows that replacing the first encoder directly with the ResNet feature map improves the results while being more efficient (line 241-242 and A.2). \n\n(-) Why is only the first encoder removed? Does the replaced place  and the number of replaced encoders affect performance? \n\n\nClarity: Middle\n\n(+)  The paper is easy to understand. \n\n(-)  What loss function is used should be made clear. GS sampler seems to be the negative sampling method and not loss function. \n\n(-)  It would be better to clarify MLPHead_1 and MLPHead_2 in Figure1. \n\nLine 289-290, “We start with Dot Product + GMP + MLPHead_1”. MLPHead_1 seems to be MLPHead_2.  \n\nWhen MLPHead_2 is not used, the output dimension seems to be D. How can we output the score?  \nWhen MLPHead_1 is not used, the input dimension to FC3 seems to be changed to hw.  \n\n\n(-) For the ablation study, the author used their own measure mACC, which is averages of rank-1 rates and mAP on different datasets and runs. It is not clear if the average of different measures is possible. Also, we cannot compare the results of Table2 and Table3 because it is not used in Table2. \n\n--\nSignificance: Middle\n\n(+) The Transformer is a timely topic. ViT[6] and DeTR[2] are prevalent in computer vision. \n\n(+) This paper clearly outperforms the state-of-the-art performance on domain generalized person re-id. \n\n(-) The theoretical contribution is limited. This paper has made only engineering to improve the rank-1 rates and mAP and computational time.\n  \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors discussed the social concerns of person re-id.\n \n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "10hours"}, "extra_text": {}}, {"review_id": "2iKvYoqGC4B", "reviewer": "Reviewer_XZ51", "summary": "This paper proposes a Transformer-Based Deep Image Matching approach for generalizable person re-identification. It adopts two simple solutions based on the transformer to achieve pair image matching. The idea is simple and easy to follow. ", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper proposes a Transformer-based image matching to achieve generalizable person re-identification. \n\nThis idea is based on the assumption that local feature matching helps to improve the generalization ability of models. I agree with this conclusion. The cross image attention proposed in this paper is indeed making sense for image-to-image matching. The ablation studies and experiments are convincing and demonstrate the improvement of the proposed TansMatcher. \n\n\nHowever, the authors only use the transformer to achieve the local matching, therefore, the contribution is limited. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "There are four limitations:\n1.\tIn this experiment, single dataset training and single dataset testing cannot verify the generalizable ability of models, it should conduct experiments on large-scale datasets.\n2.\tThe efficiency of such pairwise matching is very low, making it difficult to be used in practical application systems. \n3.\tI hope to see that you can compare your model with ResNet-IBN / ResNet of FastReID, which is practical work in the person Reid task.\n4.\tI think the authors only use the transformer to achieve the local matching, therefore, the contribution is limited.\n\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper copes with the interesting problem of using transformer-based models for image matching. This work shows that such a mechanism can successfully be used for person re-identification. The reviewers have recognised the simplicity of the proposed approach and good simple baselines described along with the approach. The experimental results show improvement over recent work on domain generalized person re-id. Despite the lack of important theoretical novelty, the tackled topic is timely and the proposed model simple. I suggest the acceptance of this work.
Decision: Accept (Poster)