# Learning Identity-Consistent Feature for Cross-Modality Person Re-Identification via Pixel and Feature Alignment

Authors: Sixian Chan; Feng Du; Yanjing Lei; Zhounian Lai; Jiafa Mao; Chao Li
Journal: Mobile Information Systems
 (2022)

---

## Abstract

RGB-IR cross-modality person re-identification (ReID) can be seen as a multicamera retrieval problem that aims to match pedestrian images captured by visible and infrared cameras. Most of the existing methods focus on reducing modality differences through feature representation learning. However, they ignore the huge difference in pixel space between the two modalities. Unlike these methods, we utilize the pixel and feature alignment network (PFANet) to reduce modal differences in pixel space while aligning features in feature space in this paper. Our model contains three components, including a feature extractor, a generator, and a joint discriminator. Like previous methods, the generator and the joint discriminator are used to generate high-quality cross-modality images; however, we make substantial improvements to the feature extraction module. Firstly, we fuse batch normalization and global attention (BNG) which can pay attention to channel information while conducting information interaction between channels and spaces. Secondly, to alleviate the modal difference in feature space, we propose the modal mitigation module (MMM). Then, by jointly training the entire model, our model is able to not only mitigate the cross-modality and intramodality variations but also learn identity-consistent features. Finally, extensive experimental results show that our model outperforms other methods. On the SYSU-MM01 dataset, our model achieves a rank-1 accuracy of40.83% and an mAP of 39.84%.


---

## Body

## 1. Introduction

Person ReID can be viewed as a cross-camera image retrieval problem, which aims at matching individual pedestrian images in a query set to ones in a gallery set captured by different cameras. Its main challenge lies in the interclass and intraclass variations caused by different lighting, poses, occlusions, and views. Most existing methods [1–5] mainly focus on matching RGB images captured by visible cameras, which can be formulated as an image matching problem under a single modality. However, these methods cannot be applied to images taken in poor lighting conditions, because the visible camera cannot capture pictures with discriminative features. However, in practical application scenarios, the camera should ensure all-weather operation.Since the visible camera has limited effect on the security work at night, the camera that can switch the infrared mode is being widely used in the intelligent monitoring system. In visible mode and infrared mode, RGB images and infrared images are collected, respectively, which belong to two different modalities. RGB images have three channels but IR images have only one channel, so the ReID problem in a cross-modality setting becomes extremely challenging, which is essentially a cross-channel retrieval problem. First, infrared images of different identities are difficult to distinguish but are easy to distinguish in visible images. In addition, the same person varies greatly in different modalities. It is known as modality discrepancy.To address visible-infrared person ReID, several approaches [6–10] have been proposed, aiming to mitigate modal differences by aligning features or pixel distributions. Feature alignment methods [6, 8, 10] mainly focus on bridging the gap between RGB and IR images through features. It is difficult to match RGB and IR images in a shared space due to large cross-modality differences between the two modalities. Different from existing methods that directly match RGB and IR images, we use generative adversarial networks to generate fake IR images based on real RGB images and then match the generated images through a feature alignment network. The generated fake IR images are used to reduce the modality difference between the RGB and IR images. Although the generated fake IR images are very similar to real images, there are still intraclass differences due to pose variations, viewpoint changes, and occlusions.Inspired by the above discussion, in this paper, we propose a pixel and feature alignment network (PFANet) that simultaneously mitigates cross-modality differences in pixel space and intramodality variation in feature space. As shown in Figure1, to reduce the modal difference, we apply a generator GI to generate fake IR images. Then, to alleviate the intramodality variation, a feature extraction module (F) is designed to encode fake and real IR images into a shared feature space by exploiting identity-based classification and triplet loss. The batch normalization and global (BNG) attention is added to the feature extraction network (F), which can make the network learn which channel is more important as well as can interact between channels and spaces. Furthermore, to mitigate the modal difference in the feature space, a modal mitigation module (MMM) is proposed, which can significantly mitigate the difference between the two modalities. Finally, to learn identity-consistent recognition, a joint discriminator D is utilized. Its input is an image-feature pair.Figure 1
Framework of the proposed model. It consists of an image generation module (G), a joint discriminator module (D), and a feature extraction module (F). The G can generate fake IR imagesXir′ to mitigate the cross-modality variation, and the F can alleviate the intramodality variation. The F module contains ResNet-50 and BNG attention and MMM module. The BNG module can focus on channel and spatial information, and the MMM module can reduce modality differences.The major contributions of this work can be summarized as follows:(i)
We propose a generative adversarial network to generate cross-modality images that alleviated modal differences in pixel space. This model consists of a generator and a joint discriminator, by playing a max-min game, our model is able to not only reduce the cross-modality and intramodality variations but also learn identity-consistent features.(ii)
We design a batch normalization and global (BNG) attention, which consists of channel attention and global attention. In the channel attention, we measure the importance of each channel by applying the scale factor of BN to the channel dimension and suppressing insignificant features. As for the global attention module, it can reduce information attenuation and amplify the features of global dimension interaction.(iii)
We apply a modal mitigation module (MMM) to mitigate the modal distribution. The instance normalization (IN) is utilized to mitigate modal differences on a single instance. Moreover, the channel attention is used to guide the learning of IN, which can mitigate modal differences while preserving identity information.


## 2. Related Works

### 2.1. RGB-IR Person ReID

RGB-IR cross-modality person ReID can be seen as a multicamera retrieval problem that aims to match pedestrian images captured by visible and infrared cameras, which are widely used in video surveillance, public security, and smart cities. Compared with RGB-RGB single-modality person ReID which only deals with RGB images, the key challenge in this work is to mitigate the large differences between the two modalities. To address the challenge caused by differences in modality distributions, a variety of approaches to cross-modality person re-identification have been proposed. Some early work focused on solving the channel mismatch between RGB images and IR images, due to RGB images having three channels. In contrast, IR images have only one channel. Wu et al. [10] proposed a deep zero-padding network and contributed a new ReID dataset SYSU-MM01. In [11], a dual-path network with a bi-directional dual-constrained top-ranking loss was introduced to learn modality alignment feature representations for RGB-IR ReID. Feng et al. [12] proposed a framework for solving heterogeneous matching problems using modality-specific networks. Ye et al. [13] proposed a dual-stream network with feature learning and metric learning to convert two heterogeneous modalities into a consistent space where the modalities share a metric. Dai et al. [6] introduced a cross-modality generative adversarial network (cmGAN) to reduce the distribution differences between RGB and IR features. Most of the above approaches mostly focus on reducing intermodality differences by feature alignment, while ignoring the large cross-modality differences in pixel space.Unlike these approaches, the proposed model in this paper is able to combine feature alignment and pixel alignment, effectively reducing intramodality and cross-modality variations. By training the model, the model is able to learn identity consistency features.


### 2.2. GAN in Person ReID

A generative adversarial network (GAN) consists of a generator and a discriminator, using the idea of game theory, where the generator tries to generate an image to deceive the discriminator, and the discriminator tries to discriminate whether the image is real or generated. Through multiple adversarial training, generative adversarial networks are able to learn deep representations of data in a self-supervised manner. GAN can generate high-quality images, perform image enhancement, generate images from text, and convert images from one domain to another [14, 15]. GAN was first proposed in 2014’s [16]. After that, researchers have proposed a variety of task-specific GAN structures, such as CycleGAN [14], Pix2Pix [17], and StarGAN [15]. There are many works in the field of pedestrian re-identification that also apply GAN to improve accuracy. Li et al. [18] proposed a network that allows querying images of different resolutions to process cross-resolution person ReID. Wang et al. [19] designed an end-to-end alignment generative adversarial network (AlignGAN) for the RGB-IR ReID task. JSIA-ReID [20] implemented a two-layer alignment of pixels and features in a unified GAN framework.In our work, we apply GAN to generate cross-modality images that mitigate modal differences between RGB-IR image data in pixel space.


### 2.3. Attention Mechanisms

There is an important feature in the human visual system that allows people to selectively focus on things of interest in order to capture valuable information. Inspired by the human visual system, many works have attempted to employ attention mechanisms to improve the performance of CNNs.Attention mechanisms enable the network to focus on areas of interest to the human body and better extract useful information. SENet [21]integrated spatial information into the channel-level feature responses and computed the corresponding attention with two MLP layers. Later, bottleneck attention module (BAM) [22] built independent space and channel submodules in parallel and embedded them into each bottleneck block. Considering the relationship between any two positions of the feature map, nonlocal feature attention [23] was proposed to capture the relationship between them. The convolution block attention module (CBAM) [24] sequentially cascaded channel attention and spatial attention. However, these works ignored the information about the weights adjusted from the training; therefore, we wanted to highlight the significant features by using the variance of the trained model weights, which also was able to amplify cross-dimensional interactions and captured important features of all three dimensions. We propose new attention (BNG) to solve the above problem. A modal mitigation module (MMM) is designed to mitigate the modal distribution, using channel attention to guide the learning of instance normalization (IN) for mitigating modal differences while preserving identity information.


## 2.1. RGB-IR Person ReID

RGB-IR cross-modality person ReID can be seen as a multicamera retrieval problem that aims to match pedestrian images captured by visible and infrared cameras, which are widely used in video surveillance, public security, and smart cities. Compared with RGB-RGB single-modality person ReID which only deals with RGB images, the key challenge in this work is to mitigate the large differences between the two modalities. To address the challenge caused by differences in modality distributions, a variety of approaches to cross-modality person re-identification have been proposed. Some early work focused on solving the channel mismatch between RGB images and IR images, due to RGB images having three channels. In contrast, IR images have only one channel. Wu et al. [10] proposed a deep zero-padding network and contributed a new ReID dataset SYSU-MM01. In [11], a dual-path network with a bi-directional dual-constrained top-ranking loss was introduced to learn modality alignment feature representations for RGB-IR ReID. Feng et al. [12] proposed a framework for solving heterogeneous matching problems using modality-specific networks. Ye et al. [13] proposed a dual-stream network with feature learning and metric learning to convert two heterogeneous modalities into a consistent space where the modalities share a metric. Dai et al. [6] introduced a cross-modality generative adversarial network (cmGAN) to reduce the distribution differences between RGB and IR features. Most of the above approaches mostly focus on reducing intermodality differences by feature alignment, while ignoring the large cross-modality differences in pixel space.Unlike these approaches, the proposed model in this paper is able to combine feature alignment and pixel alignment, effectively reducing intramodality and cross-modality variations. By training the model, the model is able to learn identity consistency features.


## 2.2. GAN in Person ReID

A generative adversarial network (GAN) consists of a generator and a discriminator, using the idea of game theory, where the generator tries to generate an image to deceive the discriminator, and the discriminator tries to discriminate whether the image is real or generated. Through multiple adversarial training, generative adversarial networks are able to learn deep representations of data in a self-supervised manner. GAN can generate high-quality images, perform image enhancement, generate images from text, and convert images from one domain to another [14, 15]. GAN was first proposed in 2014’s [16]. After that, researchers have proposed a variety of task-specific GAN structures, such as CycleGAN [14], Pix2Pix [17], and StarGAN [15]. There are many works in the field of pedestrian re-identification that also apply GAN to improve accuracy. Li et al. [18] proposed a network that allows querying images of different resolutions to process cross-resolution person ReID. Wang et al. [19] designed an end-to-end alignment generative adversarial network (AlignGAN) for the RGB-IR ReID task. JSIA-ReID [20] implemented a two-layer alignment of pixels and features in a unified GAN framework.In our work, we apply GAN to generate cross-modality images that mitigate modal differences between RGB-IR image data in pixel space.


## 2.3. Attention Mechanisms

There is an important feature in the human visual system that allows people to selectively focus on things of interest in order to capture valuable information. Inspired by the human visual system, many works have attempted to employ attention mechanisms to improve the performance of CNNs.Attention mechanisms enable the network to focus on areas of interest to the human body and better extract useful information. SENet [21]integrated spatial information into the channel-level feature responses and computed the corresponding attention with two MLP layers. Later, bottleneck attention module (BAM) [22] built independent space and channel submodules in parallel and embedded them into each bottleneck block. Considering the relationship between any two positions of the feature map, nonlocal feature attention [23] was proposed to capture the relationship between them. The convolution block attention module (CBAM) [24] sequentially cascaded channel attention and spatial attention. However, these works ignored the information about the weights adjusted from the training; therefore, we wanted to highlight the significant features by using the variance of the trained model weights, which also was able to amplify cross-dimensional interactions and captured important features of all three dimensions. We propose new attention (BNG) to solve the above problem. A modal mitigation module (MMM) is designed to mitigate the modal distribution, using channel attention to guide the learning of instance normalization (IN) for mitigating modal differences while preserving identity information.


## 3. The Proposed Method

In this part, we introduce the proposed PFANet in detail. Our network will be presented in the following three parts, including (1) RGB-IR images generation module, (2) BNG attention module, and (3) modal mitigation module. To reduce cross-modality variation, we apply generative adversarial networks to convert RGB images to fake IR images, which have IR style while maintaining their original identity.Then, the features of the two modalities are extracted for feature alignment. The BNG attention is designed to make the network focus on channel and spatial information. In addition, the modal mitigation module (MMM) is proposed to mitigate the differences between the two modalities. The main output of the PFAnet during testing is the feature for person ReID.


### 3.1. RGB-IR Images Generation Module

There is a large cross-modality difference between RGB and IR images, which significantly increases the difficulty of the task of cross-modality pedestrian re-identification. To reduce cross-modality variation, we apply generative adversarial networks to convert RGB imagesXrgb to fake IR images Xir′, which has IR style while maintaining their original identities. The generated fake IR image Xir′ can mitigate the modality differences between RGB and IR images. The module consists of a generator GI that generates a fake IR image from an RGB image and a joint discriminator DI that discriminates whether the image is a real image or a generated image. The input of the generator is the real images Xrgb, and its output is the fake IR images Xir′=GIXrgb. The input of the discriminator is the generated fake IR image Xir′; if the image is real, its output is one, and if the image is the generated image, the output is zero. The goal of the generator is to make the generated image as similar as possible to the real image, and the goal of the discriminator is to discriminate as much as possible whether the input image is real or generated. Unlike ordinary discriminators, the input to our discriminator is a pair of IR images and ReID feature maps. The generator and discriminator play the min-max game as [16], and the modal can make the fake IR image Xir′ as realistic as possible.The adversarial loss for generating IR images is defined as follows:(1)LGI=ElogDIXir′,fmapXir′,(2)LDI=LDIreal+LDIfake,where(3)LDIreal=Ex,f∈Xir,fmap,RXirlogDIx,f,(4)LDIfake=Ex,f∈Mlog1−DIx,f,(5)M=Xir′,fmap,RXir′∪Xir′,fmap,RXir∪Xir,fmap,RXir′.Among them,fmap,RXir is the extracted image feature of Xir and fmap,RXir′ is the extracted image feature of generated image Xir′. Equation (1) is used to train the generator model; after the constraint of the loss function, the generator will generate a more realistic IR image. Equations (3) and (4) are used to train the discriminator, which differs from traditional discriminators in that the input is a pair of image features. It has two advantages, firstly, the fake IR image Xir′ will be closer to the real IR image Xir through the max-min game [16], and the distribution of the features fmap,RXir′ of the fake IR image will be more similar to the real image features fmap,RXir. Secondly, fmap,RXir′ is able to maintain the identity-consistency by the corresponding image Xir′ constraint. Although LGI loss can ensure that the fake IR image Xir′ resembles the real IR image Xir, there is no guarantee that the generated fake IR images retain the structure and content of the original RGB images Xrgb. To deal with this problem, we introduce a generator GR for generating IR images into RGB images and the corresponding discriminator DR. Also we introduce cycle-consistency loss which is defined as follows:(6)Lcyc=EGRGIXrgb−Xrgb1+EGIGRXir−Xir1.Lcyc loss enables the GI generated IR image to be consistent with the input real RGB image. We use the L1 norm instead of the L2 norm because the L1 norm allows the generator to generate better image edges. Specifically, we input the real RGB image Xrgb into the generator GI to generate the fake IR image Xir′ and then use the generator GR to generate the reconstructed RGB image from the fake IR image. We do something similar with IR images.Now, the loss of the generator can be defined as follows:(7)LG=LGI+ω∗Lcyc,where ω is the weight of cycle loss and ω is set to 10 as in [14]. By using this loss during adversarial training, we can generate high-quality IR images.


### 3.2. The BNG Attention Module

Our proposed BNG attention is an efficient and lightweight attention mechanism. The BNG attention can be embedded at the end of any convolutional neural network, for the residual network ResNet-50; the end of the residual structure can be embedded. The structure of BNG is shown in Figure2.Figure 2
BNG attention.BNG attention consists of two submodules, as shown in Figure2(a); the channel attention submodule can use the weight information of the trained model to highlight salient features. We obtain its scale factor from batch normalized (BN [25]) as shown in(8)Bout=BNBin=γBin−μBσB2+ϵ+β,where μB and σB are the mean and standard deviation of mini batch B and γ and β are the trainable parameters used to fit the data distribution.The formula for channel attention can be expressed as follows:(9)F1=sigmoidWγBNF,where γ is the scale factor for each channel, and the weights are obtained as Wγ=γi/∑j=0γj. We measure the importance of each channel by applying the scale factor of BN to the channel dimension and suppressing insignificant features. Since channel attention only focuses on channel information, there is no global space-channel information interaction; to solve this problem, we design a global attention module. It can reduce information attenuation and amplify the features of global dimension interaction. Inspired by CBAM [24], the channel attention and spatial attention are connected in turn. The main structure is shown in Figure 2(b). Given the input feature map F1∈RC×H×W, the intermediate state F2 and output F3 are defined as follows:(10)F2=McF1⊗F1,F3=MsF2⊗F2,where Mc and Ms are the channel and spatial attention maps, respectively. ⊗ denotes element-wise multiplication.The channel attention submodule uses a 3D arrangement to preserve information across three dimensions and then uses a two-layer MLP layer that amplifies the channel spatial dependencies across dimensions. The channel attention submodule is illustrated in Figure3.Figure 3
Channel attention submodule.In the spatial attention submodule, to focus on the spatial information, two convolutional layers are used to fuse the spatial information. The size of the convolution kernel is set to7∗7. Since max-pooling reduces information and has a negative influence, we remove the max-pooling operation to retain more features. The same reduction ratio γ is adopted from the channel attention submodule, same as BAM. The spatial attention submodule without group convolution is shown in Figure 4.Figure 4
Spatial attention submodule.


### 3.3. Modal Mitigation Module (MMM)

To mitigate the modal distribution, a modal mitigation module (MMM) is designed. For the input imageX, we denote the features extracted in the convolution block as M∈Rh×w×c and input it into the MMM, where h,w,andc represent the height, width, and a number of channels of the feature map M, respectively. The instance normalization (IN) is used to mitigate modal differences on a single instance [27]. Instance normalization (IN) computes the mean and variance in a single instance and reduces the difference between the two data distributions. However, using IN directly may has a negative impact on the ReID task. Because the distribution of image data has changed significantly, some identifying information may be lost.To overcome these shortcomings, we use channel attention to guide the learning of IN, which mitigates modal differences while preserving identity information. Specifically, we input the feature into a two-layer MLP to downsample the channels and then upsample to the original number of channels and use the activate function to activate the feature as a mask to supervise the IN operation:(11)F=mC⊙M+1−mC⊙M^,where mC is the channel mask, representing the identity-related channels, and M^ is the instance-normalized result of the input M.Similar to SENet [21], the method of generating a mask with channel dimension can be expressed as follows:(12)mC=σW2δW1gM,where W1∈Rc/r×c and W2∈Rc×c/r are learnable parameters in the two bias-free fully connected (FC) layers, which are followed by ReLU activation function δ· and sigmoid activation function σ·. g· denotes global average pooling of features. In order to balance performance and reduce the number of parameters, the downsampling ratio is set to r=16.The formula for instance normalization is defined as follows:(13)M^j=INMj=Mj−EMjVarMj+ϵ,where E· is to calculate the mean of each dimension and Var· is to calculate the standard deviation of each dimension. To avoid dividing by zero, we add ϵ to the denominator, and Mj∈Rh×w is the j-th dimension of the feature map M.


### 3.4. Loss Function

In this section, we will introduce the loss we used when training the generator to generate a fake IR imageXir′. On the one hand, Xir′ should be classified to the same identity class as the corresponding Xrgb; on the other hand, Xir′ should satisfy the triplet loss [28] of the corresponding Xrgb identity constraint. We define these two losses as Lclsgan and Ltrigan and denote them in(14)Lclsgan=LclsXir′=Ex∈Xir′−logpx,Ltrigan=12LtriXir′,Xir,Xir+LtriXir,Xir′,Xir′,where p· is the predicted probability of belonging to the ground-truth identity; the ground-truth identity of the fake IR image Xir′ should be the same as that of the original RGB image Xrgb.Although the generated imageXir can reduce cross-modality differences, there are still large intramodality differences caused by lighting, human pose, and view. We minimize the fake IR image Xir′ and the real IR image Xir in a shared space via identity-based classification and triplet loss. We define these two losses as Lclsfeat and Ltrifeat and denote them in(15)Lclsfeat=LclsXir∪Xir′=Ex∈Xir∪Xir′−logpx,Ltrifeat=LtriXir,Xir′,Xir′+LtriXir′,XirXir,where p· represents the predicted probability that the input belongs to the ground-truth identity, and ∪ means the union sets. In summary, the overall loss of our module is shown in(16)LReID=λ1LG+λ2LDI+λ3Lclsgan+λ4Ltrigan+λ5Lclsfeat+λ6Ltrifeat,whereLG and LDI are calculated by equations (1) and (2). Lclsgan, Ltrigan, Lclsfeat, and Ltrifeat are calculated by equations (14) and (15), respectively. Among them, λ1=1.0,λ2=1.0, λ3=0.1, λ4=0.1, λ5=1.0, and λ6=1.0 .


## 3.1. RGB-IR Images Generation Module

There is a large cross-modality difference between RGB and IR images, which significantly increases the difficulty of the task of cross-modality pedestrian re-identification. To reduce cross-modality variation, we apply generative adversarial networks to convert RGB imagesXrgb to fake IR images Xir′, which has IR style while maintaining their original identities. The generated fake IR image Xir′ can mitigate the modality differences between RGB and IR images. The module consists of a generator GI that generates a fake IR image from an RGB image and a joint discriminator DI that discriminates whether the image is a real image or a generated image. The input of the generator is the real images Xrgb, and its output is the fake IR images Xir′=GIXrgb. The input of the discriminator is the generated fake IR image Xir′; if the image is real, its output is one, and if the image is the generated image, the output is zero. The goal of the generator is to make the generated image as similar as possible to the real image, and the goal of the discriminator is to discriminate as much as possible whether the input image is real or generated. Unlike ordinary discriminators, the input to our discriminator is a pair of IR images and ReID feature maps. The generator and discriminator play the min-max game as [16], and the modal can make the fake IR image Xir′ as realistic as possible.The adversarial loss for generating IR images is defined as follows:(1)LGI=ElogDIXir′,fmapXir′,(2)LDI=LDIreal+LDIfake,where(3)LDIreal=Ex,f∈Xir,fmap,RXirlogDIx,f,(4)LDIfake=Ex,f∈Mlog1−DIx,f,(5)M=Xir′,fmap,RXir′∪Xir′,fmap,RXir∪Xir,fmap,RXir′.Among them,fmap,RXir is the extracted image feature of Xir and fmap,RXir′ is the extracted image feature of generated image Xir′. Equation (1) is used to train the generator model; after the constraint of the loss function, the generator will generate a more realistic IR image. Equations (3) and (4) are used to train the discriminator, which differs from traditional discriminators in that the input is a pair of image features. It has two advantages, firstly, the fake IR image Xir′ will be closer to the real IR image Xir through the max-min game [16], and the distribution of the features fmap,RXir′ of the fake IR image will be more similar to the real image features fmap,RXir. Secondly, fmap,RXir′ is able to maintain the identity-consistency by the corresponding image Xir′ constraint. Although LGI loss can ensure that the fake IR image Xir′ resembles the real IR image Xir, there is no guarantee that the generated fake IR images retain the structure and content of the original RGB images Xrgb. To deal with this problem, we introduce a generator GR for generating IR images into RGB images and the corresponding discriminator DR. Also we introduce cycle-consistency loss which is defined as follows:(6)Lcyc=EGRGIXrgb−Xrgb1+EGIGRXir−Xir1.Lcyc loss enables the GI generated IR image to be consistent with the input real RGB image. We use the L1 norm instead of the L2 norm because the L1 norm allows the generator to generate better image edges. Specifically, we input the real RGB image Xrgb into the generator GI to generate the fake IR image Xir′ and then use the generator GR to generate the reconstructed RGB image from the fake IR image. We do something similar with IR images.Now, the loss of the generator can be defined as follows:(7)LG=LGI+ω∗Lcyc,where ω is the weight of cycle loss and ω is set to 10 as in [14]. By using this loss during adversarial training, we can generate high-quality IR images.


## 3.2. The BNG Attention Module

Our proposed BNG attention is an efficient and lightweight attention mechanism. The BNG attention can be embedded at the end of any convolutional neural network, for the residual network ResNet-50; the end of the residual structure can be embedded. The structure of BNG is shown in Figure2.Figure 2
BNG attention.BNG attention consists of two submodules, as shown in Figure2(a); the channel attention submodule can use the weight information of the trained model to highlight salient features. We obtain its scale factor from batch normalized (BN [25]) as shown in(8)Bout=BNBin=γBin−μBσB2+ϵ+β,where μB and σB are the mean and standard deviation of mini batch B and γ and β are the trainable parameters used to fit the data distribution.The formula for channel attention can be expressed as follows:(9)F1=sigmoidWγBNF,where γ is the scale factor for each channel, and the weights are obtained as Wγ=γi/∑j=0γj. We measure the importance of each channel by applying the scale factor of BN to the channel dimension and suppressing insignificant features. Since channel attention only focuses on channel information, there is no global space-channel information interaction; to solve this problem, we design a global attention module. It can reduce information attenuation and amplify the features of global dimension interaction. Inspired by CBAM [24], the channel attention and spatial attention are connected in turn. The main structure is shown in Figure 2(b). Given the input feature map F1∈RC×H×W, the intermediate state F2 and output F3 are defined as follows:(10)F2=McF1⊗F1,F3=MsF2⊗F2,where Mc and Ms are the channel and spatial attention maps, respectively. ⊗ denotes element-wise multiplication.The channel attention submodule uses a 3D arrangement to preserve information across three dimensions and then uses a two-layer MLP layer that amplifies the channel spatial dependencies across dimensions. The channel attention submodule is illustrated in Figure3.Figure 3
Channel attention submodule.In the spatial attention submodule, to focus on the spatial information, two convolutional layers are used to fuse the spatial information. The size of the convolution kernel is set to7∗7. Since max-pooling reduces information and has a negative influence, we remove the max-pooling operation to retain more features. The same reduction ratio γ is adopted from the channel attention submodule, same as BAM. The spatial attention submodule without group convolution is shown in Figure 4.Figure 4
Spatial attention submodule.


## 3.3. Modal Mitigation Module (MMM)

To mitigate the modal distribution, a modal mitigation module (MMM) is designed. For the input imageX, we denote the features extracted in the convolution block as M∈Rh×w×c and input it into the MMM, where h,w,andc represent the height, width, and a number of channels of the feature map M, respectively. The instance normalization (IN) is used to mitigate modal differences on a single instance [27]. Instance normalization (IN) computes the mean and variance in a single instance and reduces the difference between the two data distributions. However, using IN directly may has a negative impact on the ReID task. Because the distribution of image data has changed significantly, some identifying information may be lost.To overcome these shortcomings, we use channel attention to guide the learning of IN, which mitigates modal differences while preserving identity information. Specifically, we input the feature into a two-layer MLP to downsample the channels and then upsample to the original number of channels and use the activate function to activate the feature as a mask to supervise the IN operation:(11)F=mC⊙M+1−mC⊙M^,where mC is the channel mask, representing the identity-related channels, and M^ is the instance-normalized result of the input M.Similar to SENet [21], the method of generating a mask with channel dimension can be expressed as follows:(12)mC=σW2δW1gM,where W1∈Rc/r×c and W2∈Rc×c/r are learnable parameters in the two bias-free fully connected (FC) layers, which are followed by ReLU activation function δ· and sigmoid activation function σ·. g· denotes global average pooling of features. In order to balance performance and reduce the number of parameters, the downsampling ratio is set to r=16.The formula for instance normalization is defined as follows:(13)M^j=INMj=Mj−EMjVarMj+ϵ,where E· is to calculate the mean of each dimension and Var· is to calculate the standard deviation of each dimension. To avoid dividing by zero, we add ϵ to the denominator, and Mj∈Rh×w is the j-th dimension of the feature map M.


## 3.4. Loss Function

In this section, we will introduce the loss we used when training the generator to generate a fake IR imageXir′. On the one hand, Xir′ should be classified to the same identity class as the corresponding Xrgb; on the other hand, Xir′ should satisfy the triplet loss [28] of the corresponding Xrgb identity constraint. We define these two losses as Lclsgan and Ltrigan and denote them in(14)Lclsgan=LclsXir′=Ex∈Xir′−logpx,Ltrigan=12LtriXir′,Xir,Xir+LtriXir,Xir′,Xir′,where p· is the predicted probability of belonging to the ground-truth identity; the ground-truth identity of the fake IR image Xir′ should be the same as that of the original RGB image Xrgb.Although the generated imageXir can reduce cross-modality differences, there are still large intramodality differences caused by lighting, human pose, and view. We minimize the fake IR image Xir′ and the real IR image Xir in a shared space via identity-based classification and triplet loss. We define these two losses as Lclsfeat and Ltrifeat and denote them in(15)Lclsfeat=LclsXir∪Xir′=Ex∈Xir∪Xir′−logpx,Ltrifeat=LtriXir,Xir′,Xir′+LtriXir′,XirXir,where p· represents the predicted probability that the input belongs to the ground-truth identity, and ∪ means the union sets. In summary, the overall loss of our module is shown in(16)LReID=λ1LG+λ2LDI+λ3Lclsgan+λ4Ltrigan+λ5Lclsfeat+λ6Ltrifeat,whereLG and LDI are calculated by equations (1) and (2). Lclsgan, Ltrigan, Lclsfeat, and Ltrifeat are calculated by equations (14) and (15), respectively. Among them, λ1=1.0,λ2=1.0, λ3=0.1, λ4=0.1, λ5=1.0, and λ6=1.0 .


## 4. Experiments

### 4.1. Datasets and Settings

We evaluate our model on SYSU-MM01 [10]. SYSU-MM01 is a very popular RGB-IR ReID dataset; it contains pedestrian images captured by six cameras, including two infrared cameras (camera3 and camera6), and four natural light cameras (camera1, camera2, camera4, and camera5). For each pedestrian, there are at least 400 RGB images and IR images with different poses and viewpoints. Among them, 296 IDs are used for training, 99 IDs are used for verification, and 96 IDs are used for testing. Following [29], there are two test modes, i.e., all-search mode and indoor-search mode. For the all-search mode, all images are used. For the indoor-search mode, only use indoor images from 1st, 2nd, 3rd, and 6th cameras. Both modes employ single-shot and multishot settings, in which 1 or 10 images of a person are randomly selected to form a gallery setting. Both modes use IR images as probe sets and RGB images as gallery sets.Evaluation protocols: we use cumulative matching features (CMC) and mean average precision (mAP) as evaluation metrics. Following [29], the results of SYSU-MM01 are evaluated using the official code based on the mean of 10 repeated random splits of the gallery and probe set.Implementation details: we use the ResNet-50 [30] pretrained on ImageNet as the CNN backbone, use the output of its pool5 layer as the feature map M, and use the average pooling to obtain the feature vector V. We add BNG-attention to each layer of residual blocks in ResNet-50 and MMM module after the third and fourth layers. For triplet loss, we use the FC layer to map the feature vector V into a 256-dimensional embedding vector. For classification loss, the classifier takes the feature vector V as input and includes a 256-dim fully connected (FC) layer, followed by batch normalization [25], dropout, and RELU as the middle layer, and an FC layer with the identity number logit as the output layer. The dropout rate is set at 0.5. We use PyTorch to implement the model, the images are data augmented by horizontal flipping, and the batch size is set to 72 (9 people, each of which has 4 RGB images and 4 IR images). For the learning rate, the learning rate of the generation module and discriminator module is set to 0.0002 and optimized using the Adam optimizer. We set the classifier and the embedder to 0.2 and the CNN backbone to 0.02 and optimize them by SGD.


### 4.2. Comparison with the Other Methods

In this section, we compare our method with several different cross-modality person ReID methods including the following methods: (1) with different structures and loss functions, two-stream [10], one-stream [10], zero-padding [10], BCTR [13], BDTR [13], D-HSME [26], and DGD + MSR [12] learned modality-invariant features and align them in feature space and (2) cmGAN [6] and JSIA [20] use the generative adversarial networks (GANs) to generate cross-modality IR images; they mitigate modal differences in pixel space. The experimental results are shown in Table 1.Table 1
Comparison of CMC (%) and mAP (%) performances with other methods on SYSU-MM01.
MethodsAll-searchIndoor-searchSingle-shot multishotSingle-shot multishotR1R10R20mAPR1R10R20mAPR1R10R20mAPR1R10R20mAPTwo-stream [10]11.6547.9965.5012.8516.3358.3574.468.0315.6061.1881.2021.4922.4972.2288.6113.92One-stream [10]12.0449.6866.7413.6716.2658.1475.058.5916.9463.5582.1022.9522.6271.7487.8215.04Zero-padding [10]14.8054.1271.3315.9519.1361.4078.4110.8920.5868.3885.7926.9224.4375.8691.3218.86BCTR [13]16.2054.9071.519.2————————————BDTR [13]17.155.572.019.7————————————D-HSME [26]20.762.878.023.2————————————cmGAN [6]27.067.580.627.831.572.785.022.331.777.289.242.237.080.992.332.8DGD + MSR [12]37.3583.4093.3438.1143.8686.9495.6830.4839.6489.9297.6650.8846.5693.5798.840.08JSIA-ReID [20]38.1080.7089.9036.9045.1085.7093.8029.5043.8086.2094.2052.9052.7091.1096.4042.70Ours40.8383.4092.3839.8448.1386.093.6732.5444.086.894.8752.9653.4090.5295.7044.35In Table1, we can find that there are various evaluation protocols, i.e., all-search/indoor-search and single-shot/multishot; firstly, for the same method, indoor-search performs better than all-search, because the images have less background variation in indoor mode, and matching is easier. Secondly, the rank scores of single-shot are lower than ones of multi-shot, but mAP scores of single-shot are higher than ones of multishot. This is because, in multishot mode, there are ten images in the gallery setting, while in single-shot, there is only one image. As a consequence, under the multishot mode, it is much easier to hit an image but difficult to hit all images. This situation is inverse under the single-shot mode.The R1, R10, and R20 denote Rank-1, Rank-10, and Rank-20 accuracy%. The mAP denotes the mean average precision score %, and our model shows good performance. Compared with JSIA, our model achieves over 2.7% on Rank-1 and 2.49% on mAP in the single-shot setting of all-search mode. In the single-shot setting of indoor-search mode, our model achieves a rank-1 accuracy of 44.0% and an mAP of 52.96%. In the multishot setting of indoor search, our model achieves a rank-1 accuracy of 53.40%, and an mAP of 44.35%, which is higher than JSIA by 0.7% and 1.65%, respectively.


### 4.3. Ablation Study

In this section, we design ablation experiments to test the effectiveness of the BNG module and MMM module. Our ablation experiments are performed on the dataset SYSU-MM01 and use the single-shot setting of all-search mode.Influence of BNG module: the results of ablation experiments for BNG attention are shown in Table2. Compared with the baseline model (B), by adding BNG attention, the rank-1 accuracy and mAP are improved by 5.57% and 4.39%, proving the effectiveness of BNG attention.Table 2
Ablation study in terms of CMC (%) and mAP (%) SYSU-MM01.
MethodSYSU-MM01Single-shot all-searchR1R10R20mAPBaseline34.1378.8690.0733.54B + BNG39.6081.9591.6037.93B + MMM39.9782.3892.5439.52B + BNG + MMM40.8383.4092.3839.84Influence of MMM module: as shown in Table2, the model with MMM (B + MMM) achieves a rank-1 accuracy of 39.97% and an mAP of 39.52%, which are higher than those of the baseline (B) by 5.84% and 5.98%, respectively. It is proved that our proposed MMM module has good performance.


### 4.4. Visualization of Generated Images

For a more intuitive understanding of the generator model, we show the learned fake IR images in Figure5. As shown in Figure 5, the first row is the real RGB image, the middle is the fake IR image generated by the generator, and the last row is the real IR image. We can observe that fake IR images have similar content (e.g., pose and view) and maintain the identity of the corresponding real RGB images while having an IR style. Therefore, the generated fake IR images can bridge the gap between RGB and IR images and can reduce cross-modality variation in pixel space.Figure 5
Fake IR images generated by our module. The fake IR images can maintain identities and contents with original real RGB ones and have IR style.


## 4.1. Datasets and Settings

We evaluate our model on SYSU-MM01 [10]. SYSU-MM01 is a very popular RGB-IR ReID dataset; it contains pedestrian images captured by six cameras, including two infrared cameras (camera3 and camera6), and four natural light cameras (camera1, camera2, camera4, and camera5). For each pedestrian, there are at least 400 RGB images and IR images with different poses and viewpoints. Among them, 296 IDs are used for training, 99 IDs are used for verification, and 96 IDs are used for testing. Following [29], there are two test modes, i.e., all-search mode and indoor-search mode. For the all-search mode, all images are used. For the indoor-search mode, only use indoor images from 1st, 2nd, 3rd, and 6th cameras. Both modes employ single-shot and multishot settings, in which 1 or 10 images of a person are randomly selected to form a gallery setting. Both modes use IR images as probe sets and RGB images as gallery sets.Evaluation protocols: we use cumulative matching features (CMC) and mean average precision (mAP) as evaluation metrics. Following [29], the results of SYSU-MM01 are evaluated using the official code based on the mean of 10 repeated random splits of the gallery and probe set.Implementation details: we use the ResNet-50 [30] pretrained on ImageNet as the CNN backbone, use the output of its pool5 layer as the feature map M, and use the average pooling to obtain the feature vector V. We add BNG-attention to each layer of residual blocks in ResNet-50 and MMM module after the third and fourth layers. For triplet loss, we use the FC layer to map the feature vector V into a 256-dimensional embedding vector. For classification loss, the classifier takes the feature vector V as input and includes a 256-dim fully connected (FC) layer, followed by batch normalization [25], dropout, and RELU as the middle layer, and an FC layer with the identity number logit as the output layer. The dropout rate is set at 0.5. We use PyTorch to implement the model, the images are data augmented by horizontal flipping, and the batch size is set to 72 (9 people, each of which has 4 RGB images and 4 IR images). For the learning rate, the learning rate of the generation module and discriminator module is set to 0.0002 and optimized using the Adam optimizer. We set the classifier and the embedder to 0.2 and the CNN backbone to 0.02 and optimize them by SGD.


## 4.2. Comparison with the Other Methods

In this section, we compare our method with several different cross-modality person ReID methods including the following methods: (1) with different structures and loss functions, two-stream [10], one-stream [10], zero-padding [10], BCTR [13], BDTR [13], D-HSME [26], and DGD + MSR [12] learned modality-invariant features and align them in feature space and (2) cmGAN [6] and JSIA [20] use the generative adversarial networks (GANs) to generate cross-modality IR images; they mitigate modal differences in pixel space. The experimental results are shown in Table 1.Table 1
Comparison of CMC (%) and mAP (%) performances with other methods on SYSU-MM01.
MethodsAll-searchIndoor-searchSingle-shot multishotSingle-shot multishotR1R10R20mAPR1R10R20mAPR1R10R20mAPR1R10R20mAPTwo-stream [10]11.6547.9965.5012.8516.3358.3574.468.0315.6061.1881.2021.4922.4972.2288.6113.92One-stream [10]12.0449.6866.7413.6716.2658.1475.058.5916.9463.5582.1022.9522.6271.7487.8215.04Zero-padding [10]14.8054.1271.3315.9519.1361.4078.4110.8920.5868.3885.7926.9224.4375.8691.3218.86BCTR [13]16.2054.9071.519.2————————————BDTR [13]17.155.572.019.7————————————D-HSME [26]20.762.878.023.2————————————cmGAN [6]27.067.580.627.831.572.785.022.331.777.289.242.237.080.992.332.8DGD + MSR [12]37.3583.4093.3438.1143.8686.9495.6830.4839.6489.9297.6650.8846.5693.5798.840.08JSIA-ReID [20]38.1080.7089.9036.9045.1085.7093.8029.5043.8086.2094.2052.9052.7091.1096.4042.70Ours40.8383.4092.3839.8448.1386.093.6732.5444.086.894.8752.9653.4090.5295.7044.35In Table1, we can find that there are various evaluation protocols, i.e., all-search/indoor-search and single-shot/multishot; firstly, for the same method, indoor-search performs better than all-search, because the images have less background variation in indoor mode, and matching is easier. Secondly, the rank scores of single-shot are lower than ones of multi-shot, but mAP scores of single-shot are higher than ones of multishot. This is because, in multishot mode, there are ten images in the gallery setting, while in single-shot, there is only one image. As a consequence, under the multishot mode, it is much easier to hit an image but difficult to hit all images. This situation is inverse under the single-shot mode.The R1, R10, and R20 denote Rank-1, Rank-10, and Rank-20 accuracy%. The mAP denotes the mean average precision score %, and our model shows good performance. Compared with JSIA, our model achieves over 2.7% on Rank-1 and 2.49% on mAP in the single-shot setting of all-search mode. In the single-shot setting of indoor-search mode, our model achieves a rank-1 accuracy of 44.0% and an mAP of 52.96%. In the multishot setting of indoor search, our model achieves a rank-1 accuracy of 53.40%, and an mAP of 44.35%, which is higher than JSIA by 0.7% and 1.65%, respectively.


## 4.3. Ablation Study

In this section, we design ablation experiments to test the effectiveness of the BNG module and MMM module. Our ablation experiments are performed on the dataset SYSU-MM01 and use the single-shot setting of all-search mode.Influence of BNG module: the results of ablation experiments for BNG attention are shown in Table2. Compared with the baseline model (B), by adding BNG attention, the rank-1 accuracy and mAP are improved by 5.57% and 4.39%, proving the effectiveness of BNG attention.Table 2
Ablation study in terms of CMC (%) and mAP (%) SYSU-MM01.
MethodSYSU-MM01Single-shot all-searchR1R10R20mAPBaseline34.1378.8690.0733.54B + BNG39.6081.9591.6037.93B + MMM39.9782.3892.5439.52B + BNG + MMM40.8383.4092.3839.84Influence of MMM module: as shown in Table2, the model with MMM (B + MMM) achieves a rank-1 accuracy of 39.97% and an mAP of 39.52%, which are higher than those of the baseline (B) by 5.84% and 5.98%, respectively. It is proved that our proposed MMM module has good performance.


## 4.4. Visualization of Generated Images

For a more intuitive understanding of the generator model, we show the learned fake IR images in Figure5. As shown in Figure 5, the first row is the real RGB image, the middle is the fake IR image generated by the generator, and the last row is the real IR image. We can observe that fake IR images have similar content (e.g., pose and view) and maintain the identity of the corresponding real RGB images while having an IR style. Therefore, the generated fake IR images can bridge the gap between RGB and IR images and can reduce cross-modality variation in pixel space.Figure 5
Fake IR images generated by our module. The fake IR images can maintain identities and contents with original real RGB ones and have IR style.


## 5. Conclusion

In this paper, we proposed a new pixel and feature alignment network (PFANet) for the RGB-IR ReID task. The model consisted of a feature extractor, a generator, and a joint discriminator. The BNG attention and the MMM module were designed in the feature extraction module. Through these two modules, the model not only mitigated modality differences but also paid attention to channel and global information. The cross-modality IR images were generated by the generator, which could bridge the gap between RGB and IR images and reduce cross-modality variation. Ablation experiments verified the effectiveness of each module. Extensive experiments on the SYSU-MM01 dataset illustrated that our model achieved state-of-the-art performance.



---

*Source: 4131322-2022-10-11.xml*