Cross‐modality person re‐identification using hybrid mutual learning

IET Computer VisionVolume 17, Issue 1 p. 1-12 ORIGINAL RESEARCHOpen Access Cross-modality person re-identification using hybrid mutual learning Zhong Zhang, Zhong Zhang Tianjin Key Laboratory of Wireless Mobile Communications and Power Transmission, Tianjin Normal University, Tianjin, ChinaSearch for more papers by this authorQing Dong, Qing Dong Tianjin Key Laboratory of Wireless Mobile Communications and Power Transmission, Tianjin Normal University, Tianjin, ChinaSearch for more papers by this authorSen Wang, Sen Wang Tianjin Key Laboratory of Wireless Mobile Communications and Power Transmission, Tianjin Normal University, Tianjin, ChinaSearch for more papers by this authorShuang Liu, Corresponding Author Shuang Liu shuangliu.tjnu@gmail.com orcid.org/0000-0002-9027-0690 Tianjin Key Laboratory of Wireless Mobile Communications and Power Transmission, Tianjin Normal University, Tianjin, China Correspondence Shuang Liu, Tianjin Key Laboratory of Wireless Mobile Communications and Power Transmission, Tianjin Normal University, Tianjin, China. Email: shuangliu.tjnu@gmail.comSearch for more papers by this authorBaihua Xiao, Baihua Xiao The State Key Laboratory for Management and Control of Complex Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, ChinaSearch for more papers by this authorTariq S. Durrani, Tariq S. Durrani Electronic and Electrical Engineering, University of Strathclyde, Glasgow, UKSearch for more papers by this author Zhong Zhang, Zhong Zhang Tianjin Key Laboratory of Wireless Mobile Communications and Power Transmission, Tianjin Normal University, Tianjin, ChinaSearch for more papers by this authorQing Dong, Qing Dong Tianjin Key Laboratory of Wireless Mobile Communications and Power Transmission, Tianjin Normal University, Tianjin, ChinaSearch for more papers by this authorSen Wang, Sen Wang Tianjin Key Laboratory of Wireless Mobile Communications and Power Transmission, Tianjin Normal University, Tianjin, ChinaSearch for more papers by this authorShuang Liu, Corresponding Author Shuang Liu shuangliu.tjnu@gmail.com orcid.org/0000-0002-9027-0690 Tianjin Key Laboratory of Wireless Mobile Communications and Power Transmission, Tianjin Normal University, Tianjin, China Correspondence Shuang Liu, Tianjin Key Laboratory of Wireless Mobile Communications and Power Transmission, Tianjin Normal University, Tianjin, China. Email: shuangliu.tjnu@gmail.comSearch for more papers by this authorBaihua Xiao, Baihua Xiao The State Key Laboratory for Management and Control of Complex Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, ChinaSearch for more papers by this authorTariq S. Durrani, Tariq S. Durrani Electronic and Electrical Engineering, University of Strathclyde, Glasgow, UKSearch for more papers by this author First published: 10 July 2022 https://doi.org/10.1049/cvi2.12123AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinkedInRedditWechat Abstract Cross-modality person re-identification (Re-ID) aims to retrieve a query identity from red, green, blue (RGB) images or infrared (IR) images. Many approaches have been proposed to reduce the distribution gap between RGB modality and IR modality. However, they ignore the valuable collaborative relationship between RGB modality and IR modality. Hybrid Mutual Learning (HML) for cross-modality person Re-ID is proposed, which builds the collaborative relationship by using mutual learning from the aspects of local features and triplet relation. Specifically, HML contains local-mean mutual learning and triplet mutual learning where they focus on transferring local representational knowledge and structural geometry knowledge so as to reduce the gap between RGB modality and IR modality. Furthermore, Hierarchical Attention Aggregation is proposed to fuse local feature maps and local feature vectors to enrich the information of the classifier input. Extensive experiments on two commonly used data sets, that is, SYSU-MM01 and RegDB verify the effectiveness of the proposed method. 1 INTRODUCTION Person re-identification (Cross-modality person re-identification (Re-ID)) aims to retrieve a query identity from a gallery under non-overlapping camera views [1-5]. It has many practical applications, such as missing person search, forensic and so on [6, 7]. Many person Re-ID studies mainly focus on the single-modality retrieval task where both query and gallery pedestrian images are from the red, green, blue (RGB) modality [8-10]. However, the single-modality person Re-ID could not fully capture the pedestrian appearance in the case of insufficient illumination, which limits the applicability in the night-time surveillance scenario. Hence, the cross-modality person Re-ID is proposed to meet the practical applications [11]. The challenge of cross-modality person Re-ID lies in significant discrepancies of intra-modality and cross-modality [12-15]. The intra-modality discrepancy arises from the appearance difference of pedestrians caused by the changes in illumination, postures and viewpoints [16, 17]. The cross-modality discrepancy results from different imaging principles of RGB and infrared (IR) cameras, in which exists the physical difference between the reflectivity of RGB spectrum and the emissivity of IR spectrum [18, 20]. Many approaches have been proposed to overcome the above-mentioned challenges. Some of them design different network structures to align the distributions of global or local features between two modalities [7, 11, 19, 20]. Another line of work utilises the Generative Adversarial Network (GAN) as a modality transformer to generate auxiliary images or features and preserve the identity information between RGB modality and IR modality [12, 14, 20-22]. Despite their success in mitigating cross-modality discrepancies, there are still two main issues. Firstly, as shown in Figure 1a,b, most existing methods learn features using specific and shared modules, and they utilise metric learning and feature transfer to reduce the modality gap between RGB modality and IR modality. However, they ignore the guidance from the two modalities. Secondly, the feature maps contain abundant information of pedestrian images, but they are usually pooled into feature vectors directly as the input of the classifier, which results in information loss. Hence, the input of the classifier is expected to contain more information. FIGURE 1Open in figure viewerPowerPoint The common framework of cross-modality person and Cross-modality person re-identification (Re-ID) methods, and they usually apply the identity loss and metric learning to optimise the deep model. (a) The two-stream framework of cross-modality person Re-ID methods not only learn the specific features from different modalities in shallow level of network but also learn the shared features between two modalities. (b) The two-stream framework with the shared-specific feature transfer to explore the potential modality information about shared and specific features. (c) The proposed Hybrid Mutual Learning (HML) builds a cross-modality communication bridge between RGB images and IR images through the mutual learning To overcome the above-mentioned drawbacks, we propose a novel method named Hybrid Mutual Learning (HML) for cross-modality person Re-ID, as shown in Figure 1c. The proposed HML builds the collaborative relationship via mutually learning between RGB images and IR images, which contains local-mean mutual learning and triplet mutual learning. Hence, it could mutually transfer information from different aspects. The local information has demonstrated the effectiveness for person Re-ID because it possesses the property of invariance to poses, viewpoints etc. [9, 16, 23]. Inspired by this, we propose the local-mean mutual learning to encourage to transfer the local representational knowledge between RGB images and IR images. The local-mean refers to the average operation on the local parts of RGB images and IR images, which could increase the amount of information compared with a single local part. The local-mean mutual learning not only inherits the invariance property of local features but also learns the invariant modality-shared information from the local aspect. The triplet constraint is usually applied in person Re-ID, which pulls the positive pairs together and pushes the negative pairs away [24-27]. It could learn the structural geometry knowledge in the feature embedding space. In order to take full advantage of the structural geometry knowledge from RGB images and IR images, we propose the triplet mutual learning. Specifically, we first select triplets from RGB modality and IR modality, where the anchors from the two modalities possess the same identity. We measure the triplet relation by computing the ratio between the distance of positive pairs and all pairs. Then, we transfer the triplet relation between the two modalities. As a result, the triplet mutual learning could enhance the representation consistency of the two modalities, because the triplet relation of the two modalities supervises each other and provides the auxiliary guidance for discriminative feature learning. Furthermore, we propose the Hierarchical Attention Aggregation (HAA) to fuse local feature maps and local feature vectors so as to enrich the information of the classifier input. The feature maps are first divided into several local feature maps and then conducted by the average pooling to obtain local feature vectors. In order to fuse the local feature maps and the local feature vectors, the proposed HAA learns attention weights for each element of inputs. It is designed as two paths to learn the attention weights from different aspects. Furthermore, it possesses the hierarchical structure to repeat the process of fusion and attention weight learning in order to enhance the fusion effect. The main contributions of this paper are summarised as follows: We propose HML for mutually learning the local representational knowledge and the structural geometry knowledge, which explores the collaborative relationship between RGB modality and IR modality. We propose HAA to enrich the information of the classifier input by fusing local feature maps and local feature vectors using attention mechanism and hierarchical structure, which can improve the discriminability of features. Moreover, HAA is designed as a dual path structure that could learn the complementary information from different paths. Extensive experimental results verify that the proposed method outperforms the state-of-the-art methods on two cross-modality data sets. 2 RELATED WORK 2.1 Single-modality person Re-ID Single-modality person Re-ID searches the specific pedestrian in RGB modality, which is suitable for a sufficiently illuminated environment [1, 5, 28]. Existing deep learning methods for single-modality person Re-ID are roughly divided into two categories: feature representation learning and deep metric learning. The former [29-32] is interested in learning robust and discriminative feature representations. For example, Zhou et al. [32] introduce the Omni-Scale Network to learn the combination of feature representations from homogeneous and heterogeneous scales. The deep metric learning methods [29, 33, 34] automatically learn distance metrics from RGB pedestrian images by designing various losses. Zhao et al. [34] propose an improved triplet loss to closely cluster the pedestrian images with the same identity, resulting in discriminative and generalised feature representations for person Re-ID. However, these methods are designed for RGB modality, and the performance dramatically drops when directly applied to the cross-modality person Re-ID due to a large cross-modality discrepancy. Hence, it limits the applicability in all-weather surveillance scenarios. 2.2 Cross-modality person Re-ID Cross-modality person Re-ID focuses on matching the query images and the gallery images from different modalities [11, 16, 22, 27, 35].The cross-modality learning is common in computer vision. For example, Lin et al. [59] propose the multi-grained Matching and Integration to obtain the multi-grained features in order to represent the appearance and spatial relations of group images for human-centric video analysis. Furthermore, Lin et al. [60] propose a large-scale benchmark for Human-centric Video Analysis in Complex Events (HiEve) and present a robust framework for pose estimation on the HiEve benchmark, which aims to analyse complex events about people in real scenarios. Some works design the framework of deep learning networks to extract modality invariant features [11, 19, 35, 36]. For example, Ye et al. [36] propose the Middle-level Sharable Two-stream Network to handle the modality difference in the feature level and the classifier level for cross-modality person Re-ID. Afterwards, Lu et al. [57] propose the Cross-Modality Shared-Specific Feature Transfer, which aims to explore the potential feature information about modality-specific features and modality-shared features. On the other hand, some loss functions are presented to reduce intra-class variations or enlarge inter-class differences for cross-modality person Re-ID [5, 16, 37, 38]. Zhu et al. [16] introduce the hetero-centre loss to reduce the distance between intra-identity centres of two modalities in order to overcome the modality gap. Park et al. [37] propose the dense triplet loss, which aggregates similar features for positive and negative images, respectively, and design the co-attention map to highlight the person regions. Some researchers adopt GAN to learn modality-invariant feature representations [12, 18, 20, 39]. Wang et al. [12] propose AlignGAN to implement pixel-level and feature-level alignment for alleviating the cross-modality discrepancy and preserving the identity information. Choi et al. [18] design the Hierarchical Cross-Modality Disentanglement (Hi-CMD) to eliminate the influence of ID-excluded factors and extract ID-discriminative factors for cross-modality person Re-ID. Zhao et al. [39] propose the Colour Irrelevant Consistency Learning (CICL) to apply random clothing colour transformation on RGB images to generate the clothing colour transformed image set for colour irrelevant feature learning. Different from most existing approaches, the proposed method overcomes intra-modality and cross-modality discrepancies by learning the collaborative relationship between RGB images and IR images. 2.3 Mutual learning Mutual Learning allows to transfer useful knowledge between networks with the same complexity [40-45], and it is used in many research fields, such as Li et al. [58] exploit the modality-shared information to promote the target-modality segmentation by using mutual learning for cross-modality medical image segmentation. Zhang et al. [40] propose the deep mutual learning to extend the knowledge distillation of one-way transmission from the teacher network to student network to mutual learning between two student networks. Wang et al. [41] propose the Human Skeleton Mutual Learning Person Re-identification (HSMLP-Reid) for single-modality person Re-ID, which applies local matching and global skeleton matching for mutual learning in order to handle the impact of excessive background and pose changes. Ge et al. [42] propose the Mutual Mean Teaching (MMT) for unsupervised domain adaptive person Re-ID, which designs two symmetrical networks to collaboratively refine pseudo labels. Then, Zhai et al. [55] propose the Multiple Expert Brainstorming (MEB) algorithm for domain adaptive person Re-ID, which aims to make different models learn from the predictions of other temporally average models in the mutual learning manner. Liu et al. [56] propose the Graph Consistency Based Mean-Teaching (GCMT) framework, which builds the graph between features of teacher networks and leverages the similarity relationship between features to train the student network. Tian et al. [43] present the Variational Self-Distillation (VSD) and extend to multi-view learning, which expects to preserve task-relevant information and discard task-irrelevant information. In this paper, we introduce mutual learning into the cross-modality person Re-ID task and propose HML to transfer the local representational knowledge and the structural geometry knowledge, simultaneously. 3 APPROACH 3.1 Overview The overall framework of the proposed method is shown in Figure 2, which is designed as a two-stream network based on ResNet-50 [5, 46]. The first convolutional layers of the two-stream network have different weights, which is beneficial to extract the modality-specific information of RGB modality and IR modality. To bridge the gap between two modalities, the remaining layers of the two-stream network share the same weights, and its architecture is the same as ResNet-50. After obtaining the feature maps, we divide them into P horizontal local feature maps and then utilise the average pooling to obtain P local feature vectors. FIGURE 2Open in figure viewerPowerPoint The overall framework of the proposed method. We apply a two-stream network based on ResNet-50 to extract local feature maps of red, green, blue (RGB) images and IR images and conduct the average pooling on the local feature maps to obtain local feature vectors. Then, we propose Hierarchical Attention Aggregation (HAA) to fuse the local feature maps and the local feature vectors to enrich the information of the classifier input. Afterwards, we propose Hybrid Mutual Learning (HML), which contains the local-mean mutual learning and triplet mutual learning to exploit the collaborative relationship between RGB modality and IR modality. Meanwhile, we apply the cross entropy (CE) loss and the hetero-centre loss to optimise the network Rather than feeding the local feature vectors into the classifier directly, we design HAA to fuse the local feature maps and the local feature vectors so as to enrich the information of the classifier input. Meanwhile, HML is proposed to effectively learn the collaborative relation between RGB modality and IR modality by using the local-mean mutual learning and the triplet mutual learning. 3.2 Hybrid Mutual Learning The proposed HML consists of two parts, local-mean mutual learning and triplet mutual learning, which aims to build the collaborative relationship by transferring the local representational knowledge and the structural geometry knowledge, simultaneously. The local information possesses the property of invariance to poses, viewpoints etc. To make full use of the local information and increase the amount of information, we propose to transfer the mean of local features between RGB images and IR images. Specifically, we conduct the average pooling on the local feature maps that are the outputs of HAA to obtain the local feature vectors. The proposed HAA will be introduced in the next subsection. Afterwards, the local feature vectors in the same position with the same identity are averaged to obtain the local-mean features: g ¯ i p = 1 M i ∑ j = 1 M i g i j p ${\bar{g}}_{i}^{p}=\frac{1}{{M}_{i}}\sum\limits _{j=1}^{{M}_{i}}{g}_{ij}^{p}$ (1)where g i j p ${g}_{ij}^{p}$ is the pth local feature of the jth pedestrian image with the ith identity for RGB modality, and Mi is the number of pedestrian images with the ith identity for RGB modality. Similarly, the local-mean features of IR modality are denoted as r ¯ i p ${\bar{r}}_{i}^{p}$ . There are two advantages to exchanging information between local-mean features of RGB images and IR images. Firstly, the local-mean features are more representative than individual local features, and they could prevent the influence of outliers to some extent. Secondly, the local-mean features increase the amount of information in the mutual learning process because they contain all the individual local features. In order to build the collaborative relationship between the local-mean features of RGB images and IR images, we apply the Kullback–Leibler (KL) divergence: K L g ¯ ‖ r ¯ = − ∑ i = 1 N ∑ p = 1 P σ g ¯ i p log σ r ¯ i p σ g ¯ i p $KL\left(\bar{g}\Vert \bar{r}\right)=-\sum\limits _{i=1}^{N}\sum\limits _{p=1}^{P}\sigma \left({\bar{g}}_{i}^{p}\right)\mathrm{log}\frac{\sigma \left({\bar{r}}_{i}^{p}\right)}{\sigma \left({\bar{g}}_{i}^{p}\right)}$ (2)where g ¯ i p ${\bar{g}}_{i}^{p}$ and r ¯ i p ${\bar{r}}_{i}^{p}$ are the pth local-mean features of the ith identity for RGB images and IR images, respectively, N is the number of identities, and σ(⋅) indicates the softmax activation. Equation (2) denotes the KL divergence from RGB images to IR images, and the KL divergence is asymmetric. We also calculate the KL divergence from IR images to RGB images, denoted as K L r ¯ ‖ g ¯ $KL\left(\bar{r}\Vert \bar{g}\right)$ . Thus, the objective function of local-mean mutual learning is formulated as L m = K L g ¯ ‖ r ¯ + K L r ¯ ‖ g ¯ ${\mathcal{L}}_{m}=KL\left(\bar{g}\Vert \bar{r}\right)+KL\left(\bar{r}\Vert \bar{g}\right)$ (3) The local-mean mutual learning transfers local representational knowledge between RGB modality and IR modality, and it reduces the difference between the two modalities by minimising the KL divergence. As a result, the distributions of feature vectors of the two modalities are close to each other. To further learn the cooperative relationship between RGB images and IR images, we propose the triplet mutual learning to transfer knowledge based on the triplet relation. The triplet relation could capture the structural geometry knowledge, which is important to person Re-ID. Specifically, we first concatenate the local feature vectors of the RGB image: g i j = C A T g i j 1 , g i j 2 , … , g i j P ${g}_{ij}=\mathcal{C}\mathcal{A}\mathcal{T}\left[{g}_{ij}^{1},{g}_{ij}^{2},\text{\ldots },{g}_{ij}^{P}\right]$ (4)where C A T $\mathcal{C}\mathcal{A}\mathcal{T}$ indicates the concatenation of the local features to form the global feature. Similarly, the global feature of the IR image is denoted as rij. Afterwards, we generate the triplet (gij, gik, gin) for RGB modality, where gij is the anchor, and gik and gin are the positive and negative samples for gij. Similarly, the IR triplet is denoted as (rij, rik, rin). The triplet relation of the ith identity for RGB modality is formulated as G i = ∑ j , k , n exp − d j k i w j k i exp − d j k i w j k i + exp − d j n i w j n i ${\mathcal{G}}_{i}=\sum\limits _{j,k,n}\frac{\mathrm{exp}\left(-{d}_{jk}^{i}{w}_{jk}^{i}\right)}{\mathrm{exp}\left(-{d}_{jk}^{i}{w}_{jk}^{i}\right)+\mathrm{exp}\left(-{d}_{jn}^{i}{w}_{jn}^{i}\right)}$ (5) s . t . d j k i < d j n i < d j k i + α $s.t.\hspace*{.5em}\hspace*{.5em}\hspace*{.5em}{d}_{jk}^{i}< {d}_{jn}^{i}< {d}_{jk}^{i}+\alpha $ (6)where w j k i = exp d j k i ∑ d j k i ∈ P o exp d j k i ${w}_{jk}^{i}=\frac{\mathrm{exp}\left({d}_{jk}^{i}\right)}{{\sum }_{{d}_{jk}^{i}\in {P}_{o}}\mathrm{exp}\left({d}_{jk}^{i}\right)}$ (7) w j n i = exp d j n i ∑ d j n i ∈ N e exp d j n i ${w}_{jn}^{i}=\frac{\mathrm{exp}\left({d}_{jn}^{i}\right)}{{\sum }_{{d}_{jn}^{i}\in {N}_{e}}\mathrm{exp}\left({d}_{jn}^{i}\right)}$ (8)where d j k i ${d}_{jk}^{i}$ and d j n i ${d}_{jn}^{i}$ are the Euclidean distance of the positive and negative pairs of the jth pedestrian image with the ith identity. w j k i ${w}_{jk}^{i}$ and w j n i ${w}_{jn}^{i}$ are the weighted regularisation terms to adaptively temper the distances in Equation (5), where Po is the positive pair set and Ne is the negative pair set. In order to obtain appropriate statistical information, we select the triplets using Equation (6), where α is a predefined margin. The denominator terms in Equation (7) and Equation (8) are the normalisation coefficients. Similarly, the triplet relation for IR modality is denoted as R i ${\mathcal{R}}_{i}$ . We utilise the KL divergence to transfer the triplet relation from RGB modality to IR modality: K L ( G ‖ R ) = − ∑ i = 1 N σ G i log σ R i σ G i $KL(\mathcal{G}\Vert \mathcal{R})=-\sum\limits _{i=1}^{N}\sigma \left({\mathcal{G}}_{i}\right)\mathrm{log}\frac{\sigma \left({\mathcal{R}}_{i}\right)}{\sigma \left({\mathcal{G}}_{i}\right)}$ (9) Based on the asymmetry of KL divergence, the triplet relation from IR modality to RGB modality is denoted as K L ( R ‖ G ) $KL(\mathcal{R}\Vert \mathcal{G})$ . Thus, the triplet mutual learning is defined as L t = K L ( G ‖ R ) + K L ( R ‖ G ) ${\mathcal{L}}_{t}=KL(\mathcal{G}\Vert \mathcal{R})+KL(\mathcal{R}\Vert \mathcal{G})$ (10) The triplet mutual learning transfers structural geometry knowledge between RGB modality and IR modality, and it aligns this kind of knowledge by minimising the KL divergence to encourage the network to generate consistent features for the two modalities. 3.3 Hierarchical Attention Aggregation Most person Re-ID methods [16, 47] directly pool the local feature maps to obtain the local feature vectors and then feed them into the classifier. However, this results in information loss. Hence, we propose HAA to overcome this limitation, and the flowchart of HAA is shown in Figure 3a. Specifically, the inputs of HAA are the local feature maps z ∈ R C × H × W $z\in {\mathbb{R}}^{C\times H\times W}$ and the corresponding local feature vectors G ( z ) ∈ R C × 1 × 1 $G(z)\in {\mathbb{R}}^{C\times 1\times 1}$ where H and W are the height and width of the feature map, C is the number of feature maps and G indicates the average pooling. FIGURE 3Open in figure viewerPowerPoint The flowchart of (a) Hierarchical Attention Aggregation (HAA) and (b) the attention module. ⊙: pixel-wise multiplication. ⊕: broadcasting addition. ⊗: broadcasting multiplication. +: pixel-wise addition The proposed HAA contains two attention modules, and the input of the attention module is formulated as Z = z ⊕ G ( z ) $\mathcal{Z}=z\oplus G(z)$ (11)where ⊕ denotes the broadcasting addition, that is, this operation first duplicates G(z) into the same size as z and then adds them. The attention module consists of two branches, as shown in Figure 3b, and the output of the attention module is A = σ ( G ( B ( Z ) ) ⊕ B ( Z ) ) $A=\sigma (G(B(\mathcal{Z}))\oplus B(\mathcal{Z}))$ (12)where B is implemented by two fully connected (FC) layers and ReLU, and σ is the Sigmoid activation. Here, the FC layers in B include a dimensionality-reduction FC layer R C × C 2 ${\mathbb{R}}^{C\times \tfrac{C}{2}}$ and a dimensionality-increasing FC layer R C 2 × C ${\mathbb{R}}^{\tfrac{C}{2}\times C}$ , and each FC layer is followed by a batch normalisation (BN) layer. With the output of the attention module, the aggregated feature maps are formulated as Z ′ = ( A ⊙ z ) + ( ( 1 − A ) ⊗ G ( z ) ) ${\mathcal{Z}}^{\prime }=(A\odot z)+((1-A)\otimes G(z))$ (13)where ⊙ denotes the pixel-wise multiplication, ⊗ refers to the broadcasting multiplication, and + indicates the pixel-wise addition. In this paper, we repeat the above-mentioned operation twice to aggregate the local feature maps and the local feature vectors. The proposed HAA not only reduces the loss of modality information but also promotes the model to extract more discriminative features. Furthermore, pixel-level operations such as pixel-wise multiplication, broadcasting multiplication and pixel-wise multiplication could lead to the full integration of information from different paths. 3.4 Optimisation In this paper, we also utilise the cross entropy (CE) loss and the hetero-centre loss [16] to optimise the network. The total loss of the proposed method is defined as L total = L c e + L h c + α L m + β L t ${\mathcal{L}}_{\mathit{total}}={\mathcal{L}}_{ce}+{\mathcal{L}}_{hc}+\alpha {\mathcal{L}}_{m}+\beta {\mathcal{L}}_{t}$ (14)where α and β are the balancing parameters. 4 EXPERIMENTS 4.1 Experimental settings Data sets. The proposed method is evaluated on two publicly available cross-modality person Re-ID data sets, that is, SYSU-MM01 [11] and RegDB [48]. SYSU-MM01 [11] is separated into the training set and the test set, where there is no overlap between the two sets. The training set contains 22,258 RGB images and 11,909 IR images of 395 identities. The test set contains 96 identities with two test modes: All-Search mode and Indoor-Search mode. There are 3803 IR images in the query set. For the two test modes, there are two kinds of settings: Single-Shot setting and Multi-Shot setting, according to whether one or 10 RGB images are selected from the same identity to form a gallery set. Generally, the Single-Shot setting under All-Search mode is the most challenging test setting. We follow the evaluation protocol [11, 27, 37] to perform 10 trials on this data