TransMatcher: Deep Image Matching Through Transformers for Generalizable Person Re-identification

1 Introduction . The Transformer [ 24 ] is a neural network based on attention mechanisms . It has shown great success in the field of natural language processing . Recently , it has also shown promising performance for computer vision tasks , including image classification [ 7 , 14 ] , object detection [ 2 , 37 , 14 , 26 ] , and image segmentation [ 14 , 26 ] , thus gaining increasing attention in this field . However , existing studies mostly use Transformers for feature representation learning , e.g . for image classification or dense predictions , and the generalizability of Transformers is unknown . At a glance , query-key similarities are computed by dot products in the attention mechanisms of Transformers . Therefore , these models could potentially be useful for image matching . In this work , we further investigate the possibility of applying Transformers for image matching and metric learning given pairs of images , with applications in generalizable person re-identification . Attention mechanisms are used to gather global information from different locations according to query-key similarities . The vanilla Transformer [ 24 ] is composed of an encoder that employs ∗Shengcai Liao is the corresponding author . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . self-attention , and a decoder that further incorporates a cross-attention module . The difference is that the query and key are the same in the self-attention , while they are different in the cross-attention . The Vision Transformer ( ViT ) [ 7 ] applies a pure Transformer encoder for feature learning and image classification . While the Transformer encoder facilitates feature interaction among different locations of the same image , it can not address the image matching problem being studied in this paper , because it does not enable interaction between different images . In the decoder , however , the cross-attention module does have the ability for cross interaction between query and the encoded memory . For example , in the decoder of the detection Transformer ( DETR ) [ 2 ] , learnable query embeddings are designed to decode useful information in the encoded image memory for object localization . However , the query embeddings are independent from the image inputs , and so there is still no interaction between pairs of input images . Motivated by this , how about using actual image queries instead of learnable query embeddings as input to decoders ? Person re-identification is a typical image matching and metric learning problem . In a recent study called QAConv [ 10 ] , it was shown that explicitly performing image matching between pairs of deep feature maps helps the generalization of the learned model . This inspires us to investigate the capability and generalizability of Transformers for image matching and metric learning between pairs of images . Since training through classification is also a popular strategy for metric learning , we start from a direct application of ViT and the vanilla Transformer with a powerful ResNet [ 3 ] backbone for person re-identification . However , this results in poor generalization to different datasets . Then , we consider formulating explicit interactions between query2 and gallery images in Transformers . Two naive solutions are thus designed . The first one uses a pure Transformer encoder , as in ViT , but concatenates the query and gallery features together as inputs , so as to enable the self-attention module to read both query and gallery content and apply the attention between them . The second design employs the vanilla Transformer , but replaces the learnable query embedding in the decoder by the ready-to-use query feature maps . This way , the query input acts as a real query from the actual retrieval inputs , rather than a learnable query which is more like a prior or a template . Accordingly , the cross-attention module in the decoder is able to gather information across query-key pairs , where the key comes from the encoded memory of gallery images . While the first solution does not lead to improvement , the second one is successful with notable performance gain . However , compared to the state of the art in generalizable person re-identification , 2Query/gallery in person re-identification and query/key or target/memory in Transformers have very similar concepts originated from information retrieval . We use the same word query here in different contexts . the performance of the second variant is still not satisfactory . We further consider that the attention mechanism in Transformers might be primarily for global feature aggregation , which is not naturally suitable for image matching , though the two naive solutions already enable feature interactions between query and gallery images . Therefore , to improve the effectiveness of image matching , we propose a new simplified decoder , which drops the full attention implementation with the softmax weighting , keeping only the query-key similarity computation . Additionally , inspired from QAConv [ 10 ] , global max pooling ( GMP ) is applied , which acts as a hard attention to gather similarity values , instead of a soft attention to weight feature values . This is because , in image matching , we are more interested in matching scores than feature values . Finally , a multilayer perceptron ( MLP ) head maps the matching result to a similarity score for each query-gallery pair . This way , the simplified decoder is computationally more efficient , while at the same time more effective for image matching . We call the above design TransMatcher ( see Fig . 1 ) , which targets at efficient image matching and metric learning in particular . The contributions of this paper are summarized as follows . • We investigate the possibility and generalizability of applying Transformers for image matching and metric learning , including direct applications of ViT and the vanilla Transformer , and two solutions adapted specifically for matching images through attention . This furthers our understanding of the capability and limitation of Transformers for image matching . • According to the above , a new simplified decoder is proposed for efficient image matching , with a focus on similarity computation and mapping . • With generalizable person re-identification experiments , the proposed TransMatcher is shown to achieve state-of-the-art performance on several popular datasets , with up to 6.1 % and 5.7 % performance gains in Rank-1 and mAP , respectively . 2 Related Work . Given pairs of input images , deep feature matching has been shown to be effective for person re-identification . Li et al . [ 8 ] proposed a novel filter pairing neural network ( FPNN ) to handle misalignment and occlusions in person re-identification . Ahmed et al . [ 1 ] proposed a local neighborhood matching layer to match deep feature maps of query and gallery images . Suh et al . [ 21 ] proposed a deep neural network to learn part-aligned bilinear representations for person re-identification . Shen et al . [ 18 ] proposed a Kronecker-product matching ( KPM ) module for matching person images in a softly aligned way . Liao and Shao [ 10 ] proposed the query adaptive convolution ( QAConv ) for explicit deep feature matching , which is proved to be effective for generalizable person re-identification . They further proposed a graph sampler ( GS ) for efficient deep metric learning [ 11 ] . Generalizable person re-identification has gained increasing attention in recent years . Zhou et al . [ 36 ] proposed the OSNet , and showed that this new backbone network has advantages in generalization . Jia et al . [ 5 ] applied IBN-Net-b [ 15 ] together with a feature normalization to alleviate both style and content variance across datasets to improve generalizability . Song et al . [ 20 ] proposed a domaininvariant mapping network ( DIMN ) and further introduced a meta-learning pipeline for effective training and generalization . Qian et al . [ 17 ] proposed a deep architecture with leader-based multiscale attention ( MuDeep ) , with improved generalization of the learned models . Yuan et al . [ 31 ] proposed an adversarial domain-invariant feature learning network ( ADIN ) to separate identity-related features from challenging variations . Jin et al . [ 6 ] proposed a style normalization and restitution module , which shows good generalizability for person re-identification . Zhuang et al . [ 38 ] proposed a camera-based batch normalization ( CBN ) method for domain-invariant representation learning , which utilizes unlabeled target data to adapt the BN layer in a quick and unsupervised way . Wang et al . [ 27 ] created a large-scale synthetic person dataset called RandPerson , and showed that models learned from synthesized data generalize well to real-world datasets . However , current methods are still far from satisfactory in generalization for practical person re-identification . There are a number of attentional networks [ 12 , 16 , 13 , 30 , 19 , 9 , 29 , 32 , 4 ] proposed for person re-identification , but focusing on representation learning . More recently , Zhao et al . [ 33 ] proposed a cross-attention network for person re-identificaiton . However , it is still applied for feature refinement , instead of explicit image matching between gallery and probe images studied in this paper . Transformers have recently received increasing attention for computer vision tasks , including image classification [ 7 , 14 ] , object detection [ 2 , 37 , 14 , 26 ] , image segmentation [ 14 , 26 ] , and so on . For example , ViT was proposed in [ 7 ] , showing that a pure Transformer-based architecture is capable of effective image classification . DETR was proposed in [ 2 ] , providing a successful end-to-end Transformer solution for object detection . Later , several studies , such as the Deformable DETR [ 37 ] , Swin [ 14 ] , and PVT [ 26 ] , improved the computation of Visual Transformers and further boosted their performance . However , existing studies mostly use Transformers for feature representation learning , e.g . for image classification and dense predictions . There lacks a comprehensive study on whether Transformers are effective for image matching and metric learning and how its capability is in generalizing to unknown domains . 3 Transformers . For the vanilla Transformer [ 24 ] , the core module is the multi-head attention ( MHA ) . First , a scaled dot-product attention is defined as follows : Attention ( Q , K , V ) = softmax ( QKT√ dk ) V , ( 1 ) where Q ∈ RT×dk is the query ( or target ) matrix , K ∈ RM×dk is the key ( or memory ) matrix , V ∈ RM×dv is the value matrix , T and M are the sequence lengths of the query and key , respectively , dk is the feature dimension of the query and key , and dv is the feature dimension of V . In visual tasks , Q and K are usually reshaped query and key feature maps , with T =M = hw , where h and w are the height and width of the query and key feature maps , respectively . Then , the MHA is defined as : headi = Attention ( QW Q i , KW K i , V W V i ) , ( 2 ) MultiHead ( Q , K , V ) = Concat ( head1 , . . . , headH ) WO , ( 3 ) where WQi ∈ Rd×dk , WKi ∈ Rd×dk , WVi ∈ Rd×dv , and WO ∈ Rhdv×d are parameter matrices , and H is the number of heads . Then , Q = K = V in the multi-head self-attention ( MHSA ) in the encoders , while they are defined separately in the multi-head cross-attention ( MHCA ) in the decoders . The structure of the Transformer encoder without positional encoding is shown on the left of Fig . 1 . Beyond MHSA , it further appends a feed-forward layer to first increase the feature dimension from d to D , and then recover it back from D to d. Besides , the encoder can be self-stacked N times , where N is the total number of encoder layers . In ViT [ 7 ] , only Transformer encoders are used , and positional encoding is further applied . In the vanilla Transformer [ 24 ] , decoders with MHCA are further applied , with the query being learnable query embeddings initially , and the output of the previous decoder layer later on , and the key and value being the output of the encoder layer . The decoder can also be self-stacked N times .

Summary: This paper applies Transformer to the matching problem and metric learning. The direct applications of ViT and the vanilla Transformer, and two solutions adapted specifically for matching images through attention. A new simplified decoder is proposed for efficient image matching, focusing on similarity computation and mapping.