A NEW PARADIGM FOR CROSS-MODALITY PERSON RE-IDENTIFICATION
Authors: Yumeng Wang
Feng Yang
Tongkai Xu
Yanze Zhu
Venue: Submitted to ICLR 2023 (2023)
Keywords: People Re-identification，Cross-modality

Abstract: Visible and infrared Person Re-identification(ReID) is still very challenging on account of few cross-modality dataset and large inter-modality variation. Most existing cross-modality ReID methods have trouble eliminating cross-modality discrepancy resulting from the heterogeneous images. In this paper, we present an effective framework and build a large benchmark, named NPU-ReID. To this end, we propose a dual-path fusion network and taking transformer as the smallest feature extraction unit. To expand cross-modality sample diversity, we propose a modality augmentation strategy to generate semi-modality pedestrian images by exchanging certain patch and the main innovation is that the cross-modality gap can be indirectly minimized by reducing the variance of semi-modality and infrared or visible modality. Moreover, in order to make the traditional triplet loss more suitable for cross-modal matching tasks, multi-masking triplet loss is a targeted design for optimizing the relative distance between anchor and positive/negative samples pairs from cross-modality, especially constraining the distance between simple and hard positive samples. Experimental results demonstrate that our proposed method achieves superior performance than other methods on SYSU-MM01, RegDB and our proposed NPU-ReID dataset, especially on the RegDB dataset with significant improvement of 6.81$\%$ in rank1 and 9.65$\%$ in mAP.

Under review as a conference paper at ICLR 2023
A NEW PARADIGM FOR CROSS-MODALITY PERSON RE-IDENTIFICATION
Anonymous authors Paper under double-blind review
ABSTRACT
Visible and infrared Person Re-identifcation(ReDD) is still very challenging on account of few cross-modality dataset and large inter-modality variation. Most existing cross-modality ReID methods have trouble eliminating cross-modalit, discrepancy resulting from the heterogeneous images. In this paper, we present an effective framework and build a large benchmark, named NPU-ReID. To this end, we propose dual-path fusion network and taking transformer as the smallest feature extraction unit. To expand cross-modality sample diversity, we propose modality augmentation strategy to generate semi-modality pedestrian images by exchanging certain patch and the main innovation that the cross- modalityy gap can be indirectly minimized by reducing the variance of semi-modality and infrared or visible modality. Moreover, in order to make the traditional triplet loss more suitable for cross-modal matching tasks, multi-masking triplet loss is a targeted design for optimizing the relative distance between anchor and posi- tive/negative samples pairs from cross-man modss-to,,,,t,, especially constraining the dis- tance between simple and hard positive samples. Experimental results demon- strate that our proposed method achieves superior performance than other meth- ods on SYSU-MM01, RegDB and our proposed NPU-RelD dataset, especially on the RegDB dataset with significant improvement of 6.81% in rankl and 9.65% mAP.
INTRODUCTION
Person re- re-identification (ReID) is a challenging task in computer vision, which is widely used in au- tonomous driving, intelligent video surveillance and compnrr interaction systems Ye et al. (2021) /Zeng et al. (2019): Miao et al. (201)). Person RelD aims to search target pedestrian across multiple non-overlapping surveillance cameras or from different video clips. At present, most re- searches performed on single-modality visible images captured in daytime has achieved good per- formance, such as TransRelD He et al. (202T). AGW Ye et al. (202T), MMT [e et al.) (2020), HOReID Wang al. (2020), PAT et al. (2021) and Ihu al. (2020). However, in night-time time surveillance and low liw-light environments, visible cameras fail to capture person images with rich appearance information. The light limitation determines that modlliym RelD framework fails to satisfy all-weather practical application scenarios.
MMaao
With cameras which can be switched infrared mode being widely used in intelligent surveillance systems, cross-modality infrared-visible ReID has been key but challenging technology. Visible images and infrared images heterogeneous images pairs with very different visual features. Intu- itively, pedestrians in visible images have clearer texture features and valid appearance information than infrared images under good illumination environment, and infrared images can provide more distinct pedestrian outward appearance and integrated contour information. Naturally, robust fea- tures representation can be generated by sufficiently incorporating cross-modality complementary information. However, single modality person RelD method is difficult to be directly used for cross- modality tasks because of large inter-modality variations. The differences of images belonging to the same identity from mross-modality will be even greater than that of images belonging to the dif- ferent identity from the same modality. The large modality gap between visible images and infrared images and unknown environmental factors arises vitally challenging cross- moos-modalltyy problem. As shown in Figure [ the same identity from the same modality suffers from large intra intra-modaad modality variations arising from different human poses as well diverse camera viewpoints. Meanwhile, the

---

Under review as a conference paper at ICLR 2023
Ranked Results
Intte peoddlllgg Pasiaace
Single-cnodaiity Rell
query
Cross-modality variance
Inta. uoodaliy varianee
Palee Poutrie
Crus--mddalty ReID
Tue Positive
verbence
Figure I: Illustration of the Person Re-Identification. When the left visible image is used as a query. the images list above is ranked results in single-modality and the images list below is ranked results in cross-modality. The 1st to 3rd cols present True Positive samples, which means these gallery images and the query images all belong the same person. The last col presents False Positive samples.
heterogeneous imaging processes of different spectrum cameras result in large cross-modality vari- ations. These variations may lead to larger intra-identification difference than cross-identification difference, and then cause wrong matching results. Therefore, it is demanded prompt solution to reduce the cross-modality discrepancy and intra-modality variations.
Researchers have proposed many methods address the aforementioned challenges in cross- modality ReID. Several methods map persons images from different modality into a common feature space minimize modality gap Ye al. (2018a b) 2020 ?p. To alleviate the color discrepancy, gen- erative adversarial networks (GANs) is used to synthesize fake RGB/IR images while preserving the identity information much as possible in many works Wang et al. (2019) 2020); Wang et al. (2019); ZZang et al. (2019). However, there is still the challenge of appearance variations including background clutter and viewpoint variations. Furthermore, these methods continue to use triplet loss ranking loss in single-modality metric learning to supervise the network to mining identity- related cues, rather than designing modality-related loss function to learn discriminative features in the cross modsllty setting.
The quality of dataset directly affect the representation ability of embedding feature, which deter- mines the accuracy and efficiency of identification modality dataset called NPU-RelI,, which makes up for the deficiency of small-scale and uneven modality distribution. We collect images with multi-view camera system consisting of 4 visible cameras and infrared cameras, which ensures that each identity has several infrared and visible images under each camera. Aiming at tackling the concurrent challenge in intra- and cross-modaal modality variations, we present a novel modality augmentation to to eliminate the modality discrepancy. The straightforward operation to generate semi-modality images by exchanging certain regions with patch from images of the same identity from another modality, which can deepen the information communication between infrared images and visible images. Simply, the augmented image con- tains two types of information from different modality, which can effectively reduce the difficulty of cross-modal matching.
extent. Consequently, build cross-
In addition, the ReID network always trained with cross entropy loss and triplet loss to improve the discrepancy between inter-category and intra-category. We propose the musti-masking triplet loss neutralize advantages and disadvantages of traditional triplet loss and triplet loss with hard sample mining and design a cross cross-modality modality positive sample distance compress function to reduce intra-category difference.
2

---

Under review as a conference paper at ICLR 2023
The main contributions of this paper are as follows:
We build new paradigm for mross-modality person re-identification, which outperforms state-of-the-arts on SYSU-MM01, RegDB and our proposed NPU-RelD dataset, espe- cially the RegDB dataset with significant improvement of 6.81 65% on rank1/mAP 91 91.84%.
We build NPU-ReID, comprehensive visible-infrares dataset for person re-identification tasks, giving in total 34621 visible images and 38578 infrared images of 600 identities.
We present modality augmentation strategy fully exploit heterogeneous image prop- erties, which is plug-and-play and can be easily applied to the most existing methods for cross-modality tasks. To guide network to learn more powerful and generic features, We design a multi-mask triplet loss for cross-modality recognition to compress the distance be- tween anchor from one modality and corresponding sample pairs from the other modality.
2 RELATED WORKS
2.1 CROSS-MODALITY PERSON RE-IDENTIFICATION
(2020); Wang et al. (2019); Li et al.] (2020); Wang et al. (2019); Chen et al. (202I), which has achieved higher and higher accuracy in RegDB Nguyen et (2017) and SYSU-MMO1 Wu et al. (2017). Generally, in neural networks, two-stream structure Ye et al. 2020); Ye et al. (2018a) and FC layer structure are used to extract the features with different modalities separately and calculate loss through the distance metric. et al. (2020) introduced X- X-modality to the shared feature space, which was generated by lightweight network including two convolution layers and one ReLUU layer on visible images. Ye al. Ye et al. (202T) proposed channel augmentation strategy. The primary idea is to evenly generate color independent images by randomly exchanging channels of RGB. Thanks to the widespread application of GANs in recent years, several method leverage GANs to apply cross-modality style transfer, feature disentanglement or intermediate modality generation Wang et al. 2020 Choi et al. 2020) Wang et al. (2019).
Cross-modalis Person re-identification has developed rapidly Ye et al.
The comparison among existing commonly used Re-ID datasets is shown in Table Single modality person re- re-identification has made great achievements supported by constantly emerging visible ReID datasets, such as Market 1 1501 Zheng et al. (2015). DukeMTMC /Zheng et al. (2017), CUHK03 Li al.) (2014) and MSMT17 Wei et al. (2018). There are two datasets for visible and thermal Person re-identification tasks, RegDB Nguyen et al. (017)) and SYSU-MM01 [Wu et al. (2017). The images in SYSU-MM01 dataset are collected by cameras including two infrared cameras and four visible cameras.The scale of RegDB dataset is smaller than that of SYSU-MM01 dataset, including 4120 infrared images and 4120 visible images, corresponding 412 identities.
Recent years have witnessed remarkable progress in Face Recognition and RGB ReID task, with variety of approaches proposed in the literatures and applied real applications. Liu et Liu & Tan (22T)) proposed hetero-center triplet loss on the basis of HC loss Zhu et al. (2020). Hetero- center triplet loss improve HC loss in the form of triplet, which constrain the distance of different class centers from both intra-modality and cross-modality and reduce the number of calculations. Lv et al. [Lv et al. (002)) proposed the whole constraint and partial triplet-center loss (WCPTL) to deal with the discrepancy between different modalities in local features. But center loss with triplet is so keen to coincidence centers of the same ID from cross modality that decrease the inter-category discrepancy.
3 THE NPU-REID DATASET
3.1 DATASET DESCRIPTION
We propose cross-modality multi-view Re-ID dataset named NPU-ReID. In this section, will talk about how we collect and annotate images, and then analyze the advantages, disadvantages and application scenarios of the dataset.
3

---

Under review as a conference paper at ICLR 2023
RegDB
SYSU-MM01
NWPU-ReID
Examples of RGB images and thermal images in SYSU-MM01 dataset, RegDB dataset and our NPU-RelD. Each columns are of the different person. In each image pairs, the images on the top is infrared and the images the bottom is visible.
Figure 2:
Since deep learning performance benefits from the dataset that is trained on, it is inevitable to build large-scale and comprehensive dataset. For this purpose, we present new dataset named NPU- ReID for visible-thermal Person re-identification, covering 600 identities respectively. NPU-ReID dataset contains images captured by 8 cameras, including four visible cameras and four thermal cameras. To ensure that the number of pedestrian images of every identity is as equal possible and there are peoples with different poses and viewpoints images, visible cameras and thermal cameras are mutually crosswise installed. Moreover, there are also equal numbers of cameras in- stalled indoor or outdoor. Since the thermal images have only one channel, they are expanded to three channels at the same value that feed them into the network with 3-channel visible images.
There are 600 valid pedestrians identities in NPU-ReID dataset. We have a fixed split using 480 identities for training, 120 for validation and 120 for testing. During training, all images of the 480 persons in training dataset in 8 cameras can be sampled. During testing, there are two test modes, Visible to Thermal Thermal thermal modality were used as the gallery set while those from visible modality (default is visible) were used the query set. In both modes, we adopt three forms of the scope of search and ways to select gallery dataset, which motivated by SYSU-MMOL. For the scope of search, indoor-search, outdoor-search and all-search all can be selected and details indoor-search mode, the gallery set only contains the visible images captured by four indoor visible cameras. Generally, the all-search mode more challenging than the indoor-search mode. The single-shot and multi-shot setting are main two ways to select gallery dataset, where 1 or 10 images of person are randomly selected to form the gallery set. Significantly, since each of the eight cameras has a different view, only the images under the camera in which query is sampled need to be skipped during the search.
Visible. Visible to Thermal mode means that the images from
Table 1: Comparison of NPU-ReID and exsiting RelD datasets. There datasets for single modal- ity and cross-modality ReID. IP denotes the number of images per identity.
Dataset
ID
Visible Thermal cameras
IP
Single modality datasets
Market-1550
1501
32,668
0
6
21.76
CUHK03
1467
13164
0
6
8.97
MSMT17
4101
126441
0
15
30.83
DukeMTMC-reID
1404
36441
25.95 8
Cross-modalit datasets
RegDB
412
4120
4120
2
20.00
SYSU-MMOI
491
30071
15792
6
93.40
NPU-ReID
600
34621
38578
122.00
4

---

Under review as a conference paper at ICLR 2023
3.2 SPECIALITY.
Some of the visualizations results of NPU-ReID, RegDB and SYSU-MM01 dataset are shown in Figure _ The images from two modalities in RegDB dataset are captured by visible and thermal cameras installed in the same location, there the same identity, which fails to satisfy the task of matching pedestrian images across camera views. In SYSU-MM01 dataset, the number of visible images is several times that of infrared images, that the problem of modality imbalance, which fails to support Visible to Thermal mode. However, in NPU-ReID, the images captured by different cameras differs greatly with different viewpoints and poses. The view angle of the dataset differs greatly and the number of images in each modality balanced compared others. As can be seen from Figure 2] images in our dataset has higher resolution and contains more characteristic information of pedestrians.
viewpoint change in the pedestrian images of
4 OUR PROPOSED METHOD
In this section, introduce the overview framework of our proposed cross- coos-modalityy ReID method. Our framework mainly consists three components: modality augmentation, the dual-path rep- resentation learning network based on transformer and metric learning with identity loss and our proposed multi-mask triplet loss.
4.1 OVERALL ARCHITECTURE
Input stem
Stage
Stage2
Stage3
Stage4
ID Loss
MAG
Multi-nsk Triplet Lows
8"
Figure 3: Our proposed framework for cross- ReID.
The transformer-based methods springing up like mushrooms, but there is specific model designed for cross-modally moss-modality person re-identification. We propose a carefully-designed dual-path transformer network for sifrared-visible person re-identification, is shown Figure We explore how many parameters a two-stream network based on transformer should share, which is still not well investigated in the existing literature. The network consists of input stem module and four stages. The first two stages form the modality-specific branch and the following two stages form the modality-shared branch.
Given two image of size Hx W. we first feed them into modality augmentation(MAAGI to operate data transforms. Then the augmented images are resized to 224x224 and split into non-overlapping patches of size PxP to reduce computational complexity. We have verified that taking the image of size 224 224 as input can achieve better performance than the original image through several experiments, which demonstrates that square images more helpful for window partition, window merging and attention calculation.
In stage and stage 2, the transformer blocks used capture infrared features and visible fea- tures in parallel simultancously. The design of the transformer block is motivated by the Swin Transformer's success Liu et al. (002)) which built with encoder, bottleneck and decoder. The in-
5

---

Under review as a conference paper at ICLR 2023
put patches are fed into the transformer block following patch merging layer generate hierarchical features with multi-scale and decrease the computational complexity with respect to image size.
In order to avoid the lost of large amount of specific-modality feature information due to concen- trating on learning cross-modality shared features, ppecific-modalit features are learned the first two stages respectively, and shared-modality features are learned in the last two stages. The advan- tages of this framework have been proved in [A.3 Finally, all features are used to calculate identity loss and multi-masking triplet loss optimize the identity relationship among different person from cross-modality.
In this section, first analyze the impact of triplet loss functions with different variants concerning of samples pairs selection and weight contribution of hard samples on model performance.
Commonly, the sample strategy random sample identities within each batch, of which K visible and K infrared images are random selected. In triplet loss, all selected simple pairs are considered equally for the overall loss calculation, which limits network's s ability to identify more informative pairs among the selected ones. Whereas triplet loss with hard example mining only consider hard samples for each triplet, it is the other extreme form compared triplet loss. This method abandons a majority of samples and fails to take full advantage of all train samples, resulting at the training is inefficient.
4.2 THE MULTI-MASKING TRIPLET LOss
Through abundant experiments, we find that the posi- tive samples are all from another modality and the neg- ative samples are almost from the same modality for anchor sample from one modality. The former is un- doubted because the variance in heterogeneous images larger than that in homogeneous images of the same identity. The latter makes cross-modality retrieval de- generate into single-modality retrieval, raising the prob- lem of mis- mis-alignment in task. In training parse, anchor and negative samples are all from the same modality, while query and gallery are from two different modality reference. Therefore, the network should attach more importance to learning discriminative features from dif- ferent modalities in cross-modality ReID.
Illustration of the hard triplet pairs. The left circle contains visible im- ages of category and infrared images of category II, and the right circle contains infrared images of category and visi- ble images of category II. The dotted line points to the corresponding hard positive samples.
Figure 4:
Moreover, we also find that the hard positive sam- ples of all anchors from one identities are always the same sample from another modality, which indicates that hard examples are probably outlicrs. These out- liers may magnify intra-category variance and exist sta- bly even when the model is converged.
Therefore, solve the problem mentioned above, in addition to the baseline identity loss with cross entropy loss, we adopt moderate cross-modality triplet loss function, multi-masking triplet(MMT) loss. The loss, consisting of MMAP and MMAN loss, is used to optimize the distance of cross-modality positive pairs and negative pairs. The loss in- herits the strength of relative distance optimization from common triplet loss and meanwhile relieves the problem of mis-alignn task and outlier sample, as is shown in Figure
Specifically, for visible anchor sample, we select negative samples only in thermal images. In calcu- lation. The number of negative sample pairs to be selected is reduced from 2(P- 1)K to (P (P-1) 1) Then we reduce the inter-mon modaltyyy diserepancy by constraining the distance between the hard sam- ple and the easy sample from thermal modality as shown in Figure Anchor and negative sample from different modalities are pushed far away and positive sample from different modalities are pulled closer, which is consistent to analysis. As Visible to Thermal mode example, the loss function is defined as:
[ mmxx min da,n + +m + max max - min da.p - m2 ) pev nET
Temtri
Σ aEbatch
da.p NEV
(1)
NeV

---

Under review as a conference paper at ICLR 2023
where dapp/n denotes the Euclidean distance of an- chor samples and positive/negtive samples. V and T denote visible modality and thermal modality.
Our design has two benefits: 1) It is likely to con- sider all the possible triplet relations in the aug- mented image set. We can take two different sam- ples from the same modality into account the same time, which is beneficial to reduce the intra-category difference and improve the recogmition accuracy. 2) Compared with the method which constrains the dis- tance between one sample and the center of its cate- gory by center loss, our method avoids that the out- lier sample destroy other pairwise distance.
Illustration of the proposed MMT Loss. Graphics with different colors denote features belonging to different identities and the dot and the triangle means features from visible modality and infrared modality re- spectively.
Figure 5:
4.3 THE MODALITY AUGMENTATION
For cross-modality visible-thermal person re-identification, one of the challenge is to learn fused feature embedding of the same identity from different modalities. The cross- modality matching is usually formulated by learning modality shared or invariant features. Our image augmentation strategy is similar to cutmix, while the critical difference that the removed regions filled with patches from the image with the same iden- tity from another modality rather than random training images. Some visualization results of our augmentation method are shown in Figure 6
We describe the MAG operation in detail. The visi- ble training set and the thermal training set of the iden- tity are denoted as Vi (vi. vs.... v and T (4, U3. ... (... The goal of MAG is to generate new training images Vyg and 4. by exchanging part region of vi and ti. A pair of images is exchanged partly by the probability of p. which can be used to simulate the image of obscured pedestrian. The sample strategy of exchang- ing region (æ1, y1 22, 8/2 is as follows. Assume the size of input image is H x W, then the aera of origin image
S = H x W. We constrain the area ratio of the ex- changing region the original image to [Ai, As] and the aspect ratio of exchanging region is randomly initialized to [r2,7%]. Then,
infer that the height and width of
the exchanging region is h VS A X r, w V 8XA by calculation. Finally, we can determine the size, shape and position of the exchanging region by randomly initial- izing center point c (Ie, Mc).
Original Region Exclanged Augnemattio
Original Imnges
The basic motivation behind the Modality Augmentation (MAG) is that the cross-modality gap can be indirectly minimized by reducing the variance of semi-modality and infrared or visible modality. Since the labels of the orig- inal image are the same as those of the exchanging area, the labels of the augmented image remain unchanged. It can let model pay attention to the efficient discriminative information buried in the shared features simultaneously increase training efficiency. Moreover, the MAG method also can be regarded as special regional pixel-level image fusion. The fused image not only con- tains the visible feature which represents the color information of the appearance, but also contains the thermal feature which represents the contour information. More importantly, the augmented
Figure 6: Illustration of the area ex- changeable augmentation in visible- infrared person re-identification, where the images in first column and last col- umn original images from RegDB and NPU-ReID, The middle two columns display augmented images.

---

Under review as a conference paper at ICLR 2023
Table 2: Evaluation of MAG and MMT on three cross-modality RelD datasets, where the 'B' refers to the most two-path IV-RelD network training the network only with cross-entropy loss.
Method
NeU-RelD
RegDB
SYSU-MMUI
MAG
MMIN
MMTP
RT0
R20
mAP
RI
R10
R20
mAP
RI
RTO
R20
MAP
K8.66
97.26
95.15
79.27
56.89
95.73
92.12
83.91
52.49
91.18
96. 14
54.99
90.1
98.14
99.37
82.11
89.05
96.74
98.37
85.68
58.91
92.27
96.71
56.33
99.79
91.19
86.43
61.76
93.35
98.88
99.63
83.61
91.74
97.07
98.71
87.94
65.35
93.02
96.73
58.92
84.92
98.01
98.93
88.79
64.16
60.17
images from two different modalities corresponding to the same ID label, which can guarantee that the intra-class distance of the feature mapped to the shared space is shorter than the inter modality distance. Therefore, the intra-ID inter-modality variant can be effectively reduced. We observe that the MAG successfully lets the model less focus on discriminative but modality more focus on common but modality-shared features. Our design has two benefits: 1) The area exchanging strategy can replaced random region in original images with patch in the image from the other modality by chance, which improves generalization and localization by letting a model pay attention to both the most discriminative parts of objects and the entire object region rather than excessive depending on certain region. 2) It can be seamlessly integrated with other common image augmentation operations and introduces minimal computation overhead.
features and
EXPERIMENTAL RESULTS
In this section, we analyze the effectiveness of each component our our network and compare our results with existing state-of-the-ard (SOTA) algorithms.
5.1 MODULE ABLATION
We evaluate the effect of the cross-modality augmentation strategy and the multi-masking triplet loss on all three Rell) datasets and the results are shown in Table We observe the performance on the three datasets improved significantly when applying our designed cross- augmentation operation. Our proposed MAG evenly improves the baseline rank-1 and mAP by about 4% and 3% three datasets. When incorporating with the multi-masking triplet loss, the performance is further dramatically reinforced. Our proposed MMTN and MMTP work together and are mutually reinforcing, improving the baseline mAp by average four percent. It can be noted that the overall performance reaches the best when utilizing the mixed loss function of MMTN and MMTP.
Table Comparison with state-of- -hetterrrt methods on RegDB dataset.
Visible to Thermal
Thermal to Visible
Method
Venue
R1
R10
R20
mAP
R1
R10
R20
mAP
BITRR
LICA 18
33.56
58.61
67.43
32.76
32.92
58.46
68.43
31.96
Deep Zero-padding
ICCV 17
17.75
34.21
44.35
18.90
16.63
34.68
44.25
17.82
MAC
MM 19
36.43
62.36
71.63
37.03
36.20
61.68
70.99
36.63
MSR
TIP 19
48.43
70.32
79.95
48.67
D2RL
CVPR 19
43.40
66.10
76.30
44.10
AlignGAN
ICCV 19
57.90
53.60
56.30
53.40
cm-SSFT
CVPR 20
72.30
0.00
72.90
X-modality
AAAI 20
62.21
83.13
91.72
60.18
DDAG
ECCV 20
69.34
86.19
91.49
63.46
68.06
85.15
90.31
61.80
HAT
TIFS20
71.83
87.16
92.16
67.56
Hi-CMD
CVPR 20
70.93
86.39
66.04
JSIA-ReID
AAAI 20
48.50
49.30
48.10
48,90
AGW
TPAMI 21
70.05
66.37
NFS
CVPR 21
80.54
91.96
95.07
72.10
77.95
90.45
93.62
69.79
CAJ
ICCV21
85.03
95.49
97.54
79.14
70.02
86.45
91.61
66.
Our method
91.84
98.01
98.93
88.79
89.54
97.29
98.60
86.22
This subsection presents a comparison with the current state-of-the-art methods on RegDB, NPU- ReID and SYSU-MM01 datasets, as shown in Table 3 Table (nd TabpeeSJrepecctively.
Comparisons on RegDB dataset. The experiments on the RegDB dataset demonstrate that the proposed method obtains the best performance under visible infrared mode and infrared to visible
8

---

Under review as a conference paper at ICLR 2023
mode. Specially, achieve rank-1 accuracy of 91.84% and mAP of 88.79% in Visible to Thermal mode, and rank-1 accuracy of 89 99.54% and mAP of 86.22% in thermal to visible mode and the mAP on RegDB climbed above 85 for the first time.
Table 4: Comparison with state-of-the-art on NPU-ReID dataset.
All Search
Indoor Search
Methods
Venue
Viaall
Thermal
Themmal
Visable
Visible
Thernal
Thermal
Visible
R.0
R1
R10
R1
mAP
mAP
94.37
47.21
47.03
54.38
49.15
DDAG
RCCV 20
90.80
99.25
99.72
79.99
89.0
99.30
99.78
78.35
36.29
96.22
99,05
80.89
82.89
98.13
98.58
80.54
AGW
TPAMI 21
90.97
99.13
99.00
80.96
90.30
99.36
99.84
79.67
53.87
97.13
99.11
71.76
73.96
97.56
99102
70194
TSLEN
2C
90.27
99.13
99.20
78.84
90.68
79.99
90.19
79.10
89.86
80.21
ICCV2
33.97
50.83
52.19
Oum
94.41
99.41
09.90
84.92
93.46
99.88
99.99
83.37
85.91
98.76
99.75
85,90
83.37
98.10
99.67
84.33
Table 5: Comparison with the state-of-the arts on SYSU-MM01.
Tralnce-Srachh
Motheds
Single-Shet
MuhtSllo
R10
R20
mAP
R20
HOO
CVPROS
2.74
LOMO
41.8
DCAL 1X
35.45
19.66
Teep Zero poro-podding
10CV
14.8
54:12
71:33
15:95
19:13
61.4
76.41
10.89
20.58
68.38
85.79
26.92
24.43
75.86
91.32
18.64
coKGAN
LICAL 18
26.97
67.51
30.56
27.8
$3.49
72.74
95001
22.27
31.63
77.25
89.18
42.19
37
30.94
92.11
32.76
D2RR1
CVPR 19
25.9
20.6
82.4
20.5
AliiGGAN
19
42.4
83
93.7
40.7
51.5
19..
95.7
33.9
45.9
57.6
94.8
54.3
57.1
92.7
97.8
45.3
MAC
MM
33.26
79.94
90.00
36.22
46.56
93.57
98.8
40008
MISK
19
27.35
33.40
12.
43.90
16.34
95.64
10.30
39.66
89.25
97.60
30.88
5308
93.71
98.03
47.45
TSFF
MM 19
46.11
88.36
10.20
48.00
54.83
9182
9633
42.1
13.25
80.86
95.8
19.68
IPPIEN
19
11.35
38.78
91.3
13.55
47.55
BR13
3632
SSFT
36.
98.10
X-modality
BCCV
54.55
67.8
HI-CMD
CNIPR
38.95
17.58
ISIA-RelD
26
80
89.9
36.9
45.1
85.
29.5
438
94.2
52.9
52.7
91.1
96.4
42.7
ISLFN
NESEOL
36.95
91.
50.32
$4.33
12.19
93.44
91.3
48.00
39.74
91.07
96.22
6631
09.76
55.3
98.9
37.81
ACWW
21
47.30
64.30
92.14
47.65
$4.17
91.14
95.8
6297
TTAT
TI
55.29
92.64
97.36
$3.89
62
95.75
99.2
69.77
69.76
95.85
96.9
5781
NFS
CVPX
36.91
31
90.3
33.05
6331
91
49.6
02.99
9907
69.99
7000
973
1931
61.5
Our michud
64.16
9342
993
60.17
67.61
96.28
98.6
36.72
64.33
97.66
99.37
70.33
73.28
97.12
98.74
6344
Comparisons on NPU-ReID dataset. To keep things fair, we evaluate the state-of-the-an VI-ReID methods published in the last years on our NPU-RelD) dataset. We apply Visible to Thermal mode and Thermal to Visible mode in the case of single-shot in all-search and indoor-search testing mode respectively. It is worth mentioning that our proposed method achieves the best performance under various settings. In Visible to Thermal all-search mode, our method outperforms achieves 94.41 rank- 1 and 84.92% mAP.
Comparisons on SYSU-MM01 dataset. The results on SYSU-MM01 are shown in Table 5 our method has outperformed the of-e-of-thhear methods. In single-shot all-search mode, method achieves 64. 64% Rank-1 and 60.17% mAP and it surpasses the performance of the state-of-thh--ar method.
Above results all demonstrate that our proposed model outperforms existing method. We have achieved much better performance under all-search and indoor-seard settings, suggesting that our method can learn better intra-modality specific features and cross-modality sharing features by well designing modality augmentation strategy and multi-masking triplet loss.
CONCLUSION
In this paper. we propose NPU-ReID, visible-infrared dataset for cross-modality person identifi- cation and discuss the differences among these datasets and ours. We also propose novel modality augmentation method for cross-modality vision tasks. It can effectively reduce modality variance and integrated with other common data augmentation operations. In addition, we propose a novel multi-masking triplet loss to improve the traditional triplet, which ensure that hard positive sample is closer to easy positive sample and hard negative sample is far from anchor. Experimental results on three datasets demonstrate that our method achieves remarkable improvements compared to the cur- rent state-of-ther thet-ar methods. In addition, we verify the effectiveness of our augmentation strategy and loss function in other visible and thermal Rell) and person detection method.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Yehansen Chen, Lin Wan, Zhihang Li, Qianyan Jing, and Zongyuan Sun. Neural feature search for rgb-infrared person re-identification. In CVPR, pp. 587-597, 2021.
S Choi, S Lee, Y Kim, T Kim, and C Hi-CMD Kim. Hierarchical cross-modality disentanglement for visible-infrared person re-identification. In Proceedings of the IEEE Conference on Computer Vision and Paltern Recognition, Seatlle, WA, USA, pp. 13-19, 2020.
Yixiao Ge, Dapeng Chen, and Hongsheng Li. Mutual mean-teaching: Pseudo label refinery for unsupervised domain adaptation on person re- ideniifficaaiin. In ICLR, 2020.
Shuting He, Hao Luo, Pichao Wang. Fan Wang, Hao Li, and Wei Jiang. Transreid: Transformer-
based object re-identification. In ICCV, pp. 15013-15022, 2021.
Diangang Li, Xing Wei, Xiaopeng Hong, and Yihong Gong. Infrared-visible cross-modal person re-identification with an x modality. In AAAI, volume 34, pp. 4610-4617, 2020.
Wei Li, Rui Zhao, Tong Xiao, and Xiaogang Wang. Deepreid: Deep filter pairing neural network for person re- identification. 2014 IEEE Conference on Computer Vision and Pattern Recognition, pp. 152-159, 2014.
Yulin Li, Jianfeng He, Tianzhu Zhang, Xiang Liu, Yongdong Zhang, and Feng Wu. Diverse part discovery: Occluded person re-identification with part-aware transformer. In CVPR, pp. 2898- 2907, 2021.
Haijun Liu and Xiaoheng Tan. Parameter sharing exploration and hetero-center triplet loss for visible-thermal person re-identification. IEEE Transactions on Multimedia, 23:4414-4420 2021.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. 2021 IEEE/CVF In- ternational Conference on Computer Vision (ICCV), pp. 9992-10002, 2021.
Zhihan Lv, Songhao Zhu, Dongsheng Wang, and Zhiwei Liang. Whole constraint and partial triplet- center loss for infrared-visible re-identification. Neural Computing and Applications, pp. 1-14, 2022.
Jiaxu Miao, Yu Wu, Ping Liu, Yuhang Ding, and Yi Yang. Pose-guided feature alignment for occluded person re-identification. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 542-551, 2019.
Dat Tien Nguyen, Hyung Gil Hong, Ki Wan Kim, and Kang Ryoung Park. Person recognition system based combination of body images from visible light and thermal cameras. Sensors, 17(3):605, 2017.
Guan-An Wang, Tianzhu Zhang, Yang Yang. Jian Cheng, Jianlong Chang, Xu Liang, and Zeng- Guang Hou. Cross-modality paired-images generation for rgb-infrared person re-identification. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 12144-12151, 2020.
Guan' an Wang, Tianzhu Zhang, Jian Cheng, Si Liu, Yang Yang, and Zengguang Hou. Rgb-infrared cross-modality person re-identification via joint pixel and feature alignment. In ICCV, pp. 3623- $632,2019)
Guan'a an Wang, Shuo Yang, Huanyu Liu, Zhicheng Wang, Yang Yang. Shuliang Wang, Gang Yu, Erjin Zhou, and Jian Sun. High-order information matters: Learning relation and topology for occluded person re-identification. In CVPR, pp. 6449-6458, 2020.
Zhixiang Wang, Zheng Wang. Yinqiang Zheng, Yung- Yu Chuang, and Shin' ich Satoh. Learning to reduce dual-level discrepancy for infrared-vsiille person re-identification. In CVPR, pp. 616-626, 2019.
Longhui Wei, Shiliang Zhang, Wen Gao, and Qi Tian. Person transfer gan to bridge domain gap for person re-identification. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recogni- fion, pp. 79-88, 2018.
10

---

Under review as a conference paper at ICLR 2023
Ancong Wu, Wei-Shi Zheng, Hong-X Xing Yu, Shaogang Gong, and Jianhuang Lai. Rgb-infrared modsstmy person re-identification. In ICCV, pp. 5390-5399, 2017.
Mang Ye. Xiangyuan Lan, Jiawei Li, and Pong Chi Yuen. Hierarchical discriminative learning for visible thermal person re-identification. In AAAI, 2018a.
Mang Ye, Zheng Wang, Xiangyuan Lan, and Pong Chi Yuen. Visible thermal person re-identification via dual-constrained top-ranking. In IJCAI, 2018b.
Mang Ye, Xiangyuan Lan, and Qingming Leng. Cross-modality person re-identification via modality-aware collaborative ensemble learning. IEEE Transactions on Image Processing, 29: 9387-9399, 2020.
Mang Ye, Xiangyuan Lan, Zheng Wang, and Pong C. Yuen. Bi-directional center-constrained top- ranking for visible thermal person re- re-identification. IEEE Transactions on Information Forensics and Security, 15:407-419, 2020.
Mang Ye, Jianbing Shen, David J. Crandall, Ling Shao, and Jiebo Luo. Dynamic dual-attentive aggregation learning for visible-infrared person re-identification. ArXiv, abbs/2007.09314, 2020.
Mang Ye, Weijian Ruan, Bo Du, and Mike Zheng Shou. Channel augmented joint learning for visible-infrared recognition. 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 13547-13556, 2021.
Mang Ye, Jianbing Shen, Gaojie Lin, Tao Xiang, Ling Shao, and Steven C.H. Hoi. Deep learning for person re-identification: A survey and outlook. PAMI, (1):1-1,
He Zhang, Benjamin S Riggan, Shuowen Hu, Nathaniel J Short, and Vishal M Patel. Synthesis of high-quality visible faces from polarimetric thermal faces using generative adversarial networks. International Journal of Computer Vision, 127(6):845 8766882858 2019.
Feng Zheng, Cheng Deng, Xing Sun, Xinyang Jiang, Xiaowei Guo, Zongqiao Yu, Feiyue Huang, and Rongrong Ji. Pyramidal person re- re-identification via multi-loss dynamic training. In Proceed- ings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8522, 2019.
Liang Zheng, Liyue Shen, Lu Tian, Shengjin Wang, Jingdong Wang, and Qi Tian. Scalable person re-identification: A benchmark. In ICCV, pp. 1116-1124, 2015.
Zhedong Zheng, Liang Zheng, and Yi Yang. Unlabeled samples generated by gan improve the person re-identification baseline in vitro. In ICCV, pp. 3774-3782, 2017.
Kuan Zhu, Haiyun Guo, Zhiwei Liu, Ming Tang, and Jinqiao Wang. Identity- guided human semantic parsing for person re- re-identification. In ECCV, pp. 346-363, 2020.
Yuanxin Zhu, Zhao Yang, Li Wang, Sai Zhao, Xiao Hu, and Dapeng Tao. Hetero-center loss for croas-my person re- re-identification. Neurocomputing. 186:97-109, 2020.
A APPENDIX
A.1 DATASETS AND EXPERIMENTAL SETTINGS
Datasets. All of the experiments are performed on benchmark datasets SYSU-MM01 Wu et al. 2017) and RcgDB N et al. (2017). The detailed introduction of dataset has been described in Sec. this paper, we evaluate our method on RegDB visible-to-thermal and toemmal-to-viiblle modes and SYSU-MM01 by all search and indoor search single search. On proposed NPU- ReID, we carried out extensive testings and adopting an integrated evaluation mode on the other two datasets.
Evaluation metrics. All experiments follow widely used standard evaluation protocols for Re-ID task. Cumulative Matching Characteristics (CMC) and mean Average Precision (mAP) adopted
11

---

Under review as a conference paper at ICLR 2023
the evaluation metrics. CMC(rank-r accuracy) is more concerned with the top-R positive sample in ranking results, and mAP determined by the ranking results of all positive samples in the gallery.
Experimental settings. The proposed method is implemented in PyTorch. The improved dual-path Swin Transformer is adopted as the backbone. All models are trained on NVIDIA 3090Ti GPU and the pre-trained weights on ImageNet used initialize the model. A batch size of 8, initial learning rate of 0.01, and the SGD optimizer with momentum 0.9 and weight decay 5e-4 are used.
F
F
F.
F
Thrral
Figure 7: Illustration of network architecture with different fusion way.
A.2 THE EVALUATION ON OTHER CROSS-MODALIH METHOD
Applicability with Other RelD Methods To assess whether MAG and MMT loss benefit other network, we incorporate our method into several popular representative network, such TSLFN, AGW and CAJ. THe experimental results on two datasets are shown in Table G It is obviously that the network performance is all improved when utilizing the MAG augmentation and MMT loss function, which strongly demonstrates the effectiveness of the proposed method for cross-modality ReID tasks.
Pedestrian Detection. To demonstrate the generalizability of Modality augmen- tation, we also evaluate our proposed augmentations incorporated on cross-modality pedestrian de- tection task. We conduct the baseline method and the CFT method using CSPDarknet53 and CFB as backbone on LLVIP dataset and use the default settings with the authors' released code. The results be significantly improved under various metries.
in Table 7 demonstrate that the performance Across these methods, our approach consistently improves the origin performance. Equipped with strong augmentation, the performance further improves 66.6%. It would be further improved by fine-tuning the syper-parameters, such as the number and the scale limitation of exchanging area.
Table Applicability of our proposed channel augmentation with state-of-the-an methods NPU- ReID and RegDB dataset.
NPU-RelD
RegDB
Method MAG+MMT
R10
R20
mAP
RI
R10
R20
mAP
TSLEN
83.78
98.99
99.74
69.10
86.65
96.94
98.50
72.73
TSLFN
85.26
99.16
99.78
72.35
88.40
96.75
98.35
79.56
AGW
79.77
98.37
99.52
68.36
70.05
66.37
AGW
/
81.95
98.58
99.64
71.49
79,82
96.29
98.17
71.49
CAJ
60.49
91.43
97.16
44.59
85.03
95.49
97.54
79.14
CAJ
V
80.18
98.37
99.46
67.45
88.47
96.59
98.21
83,39
12

---

Under review as a conference paper at ICLR 2023
Table 7: Evaluation of proposed modality augmentation incorporated cross-modality pedes- trian detection method.
Model
Backbone
mAP50
mAP75
MAP
YOLOv5 CSPDarknet53
95.8
71.4
62.3
YOLOv5+MAG
CSPDarkne153
96.4
73.7
63.5
CFT CFT+MAG
CFB
97.5
72.9
63.6
CFB
97.8
73.6
66.5
A.3 THE EXPLORATION OF NETWORK ARCHITECTURE.
In this subsection, we explore how many parameters a dual-path transformer network should share, which is still not well investigated in the existing works.
We first explore which the most reasonable structure of feature extractor. Several ablation ex- periments are conducted to explore the impact of share parameters on the performance of network. We build the following baseline network with Swin Transformer model. As is shown in Figure 7 The network is divided into input stem and four stages, and it is set to learn cross- cross-modality features from stages 1, 2, 3 or 4 in sequence. We evaluate the performance of different backbone networks with different shared mechanisms
NPU-RelD datasets and RegDB datasets, is listed in Ta- ble 8 We find that it is not effective to learn cross-modality features from beginning or learn only intra-modality features from beginning end, such Fl and F5. As can see from the result of F3 shared mechanism, only after the network has fully learned the modality-speche modality-shun features can represent not only infrared information but also visible information, which is more beneficial for the subsequent tasks. The experiment results has verified the network of learning modality-speciff features the first two stages and learning modality-sharee features in the last two stages can achieve the best performance.
features,
Table 8: The results of different fusion architecture RegDB and NPU-ReID.
NPU-ReID
RegDB
Structure
R1
R10
R20
mAP
R1
R10
R20
mAP
F1
79.93
96.24
98.38
70.04
83.5
94.81
97.67
78.73
F2
81.96
98.44
99.12
72.29
83.54
95.19
98.01
78.23
F3
82.36
98.52
99.85
73.57
87.89
95.73
98.12
84.91
F4
81.6
98.47
98.89
72.8
83.11
92.3
95.58
81.34
F5
80.23
97.63
98.32
71.16
85.8
94.11
97
82.75
13


Reviews:
[{"review_id": "oCmFxiJq3kF", "reviewer": "Reviewer_heJg", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies the cross-modality person re-identification problem and tries to build a new paradigm. Particularly, this paper proposes a large scale dataset called NPU-ReID that includes more identities and identities with even modality distribution. What's more, this paper presents 1) a modality augmentation method by synthesizing semi-modality samples and 2) a multi-mask triplet loss  by constraining both cross-modality triplet and intra-modality simple/hard positive pairs. Experiments show the effectiveness of the proposed method", "summary_of_the_review": "From the discussion above, though the paper contributes a new dataset and raises some techniques to build a baseline, the novelty is incremental. What is more, the writing quality is quite low. It is far from a decent ICLR paper.\n", "strength_and_weaknesses": "Strengths:\n\n\t1. This paper presents a large-scale cross-modality Re-ID dataset .\n\t2. This paper presents an augmentation method that generates semi-modality sample by randomly exchanging patches between modalities. The experiments show the effectiveness of such augmentation for cross-modality person Re-ID.\n\t3. A multi-task triplet loss is designed for cross-modality person Re-ID.\n\nWeaknesses:\n\n\t1. The paper is poorly organized. It is hard to quickly get the motivations and main ideas of the proposed methods.\n\t2. The thermal sensor and environment setting for data collection is not described in details. From Figure 2, why is the quality of thermal image significantly higher than RegDB and SYSU-MM01 ? Does the thermal sensor or capturing time cause it? \n\t3. The paper presents a transformer-based network as backbone,  what is the benefits over the CNN-based backbones in traditional methods? The reason using such a transformer-based method is not clearly discussed.\n\t4. The proposed multi-task triplet loss is not clearly clarified. It is strongly to re-organize the part and make a proof-reading. In addition, It seems there is mistake in Eq. (1). I suppose a max( x,0) is lost for both terms in $L_{mtri}$. \n\t5. The sensitivity of hyper-parameters such as $m_1$, $m_2$, $\\lambda$ is not discussed. In particular, their values are not specified in the paper.\n\t6. There are lots of grammar mistakes, typos and description blurs that makes the paper hard to follow. It is strongly to find some experts to make a proofreading.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "ba0Nt1R6bM", "reviewer": "Reviewer_Wz3t", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors focus on cross-modality person re-identification tasks. They build a new benchmark named NPU-ReID, and propose a dual-path fusion network. Besides, they also propose a modality data augmentation strategy and a cross modality triplet loss for optimizations. The experiments are conducted on NPU-ReID, RegDB and SYSU-MM01 datasets.", "summary_of_the_review": "See weaknesses.\n\nIf the authors address all my concerns or point out that I misunderstand some parts of the submission, I am glad to improve the final rating. ", "strength_and_weaknesses": "Strength:\n1. The authors build a new benchmark named NPU-ReID.\n2. The authors propose a dual-path fusion network for cross-modal person re-identification.\n3. The authors propose a modality data augmentation strategy and a cross modality triplet loss for optimization\n4. The experiments are conducted on NPU ReID, RegDB and SYSU-MM01 datasets.\n\nWeakness:\n1. The novelty of the submission is limited for community. The authors only combine some existing techniques such as swin transformer, two stream feature fusion, modality triplet loss.\n2. As stated in Table 1, there are no obvious advantages compared with SYSU-MM01 and the proposed NPU-ReID. Both of them contain hundred-level identities and similar numbers of images. The authors should clarifiy the advantages of NPU-ReID for community, compared with the previous datasets.\n3. Do all the identities in NPU-ReID assign the agreement of privacy?  If the authors claim collecting and releasing a new dataset as a contribution of the submission, the dataset must be released for community after acceptance.\n4. As shown in Figure 3, the authors use a two-stream network to fuse visible and infrared modalities. Are there any feature fusions between stage3 and stage4? If so, how to use the two-stream network for evalution? If not, the authors should clarify the detailed operations of stage 3 and stage 4 for training phase in Figure 3. \n5. The authors use swin transformer and 224x224 input for experiments. It is not a fair comparison for previous work. The swin transformer is a stronger pretrained models than ResNet, which leads to better performance. Besides, previous methods usually use 256x128 input size. \n6. The proposed data augmentation method is also limited. It is only used for the identities which contain both visible and infrared modality images. If there is no paired images for some identities, the proposed augmentation may not work.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "TaK6YUEyic_", "reviewer": "Reviewer_2Ybv", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a dual-path fusion network based on transformer blocks, and builds a visible-infrared dataset for cross-modality person re-identification.", "summary_of_the_review": "This paper introduces a new dataset for visible-infrared person re-identification. However, this dataset does not provide additional insights for the community. It is just an another dataset. For the technical part, the novelty is too limited for ICLR. Seems that the presented details are already known in this field. The writing should also be polished. Therefore, the quality of this paper might not be suitable for ICLR.", "strength_and_weaknesses": "Pros:\n+ This paper explores the effectiveness of transformer model for cross-modality person re-identification, and presents a new cross-modal feature learning paradigm.\n\n+ This paper builds a comprehensive visible-infrared dataset, which could assist the development community\n\nCons:\n- The motivation and the idea of the MAG are nearly the same with [1] and [2] which reduces the modality gap by introducing semi-modality . The authors should clarify the difference and the advantage.\n- The transformer model has also been explored in visible-infrared person re-identification by methods such as [3] and [4]. It is necessary to discuss and compare these methods.\n- I feel confused why authors write the paper into a complicated style. It deeply makes it hard to understand paper. For example, the details of the loss function which is similar to [5] are not presented clearly enough. The logic of the introduction of the methods section is confusing.\n- Some terms are confusingly defined and inconsistently expressed. For example, MMAP and MMAN in page 6; MMTP and MMTN in Table 2.\n- The writing could be improved. For example, In calculation. The number of negative…\n\n[1] Sfanet: A spectrum-aware feature augmentation network for visible-infrared person reidentification[J]. IEEE Transactions on Neural Networks and Learning Systems, 2021.\n\n[2] Channel Augmented Joint Learning for Visible-Infrared Recognition. In ICCV 2021.\n\n[3] Structure-Aware Positional Transformer for Visible-Infrared Person Re-Identification. In TIP 2021.\n\n[4] CMTR: Cross-modality Transformer for Visible-infrared Person Re-identification[J]. arXiv preprint arXiv:2110.08994, 2021.\n\n[5] Visible-Infrared Person Re-Identification via Homogeneous Augmented Tri-Modal Learning, in IEEE Transactions on Information Forensics and Security, vol. 16, pp. 728-739, 2021.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "NraI6QkEEIP", "reviewer": "Reviewer_hWin", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a new visible-infrared dataset, NPU-ReID, for cross-modality person identification. Compared to existing datasets, this dataset contains more subjects, and the amount of visible and infrared data is well-balanced.\nThis paper also proposes a dual-path framework for cross-modality person re-identification, with Swin transformer as the backbone, combined with a modality augmentation strategy, and a multi-mask triplet loss to compress distances for sample pairs from different modalities. Experiments show the effectiveness of the proposed method.\n", "summary_of_the_review": "The newly constructed visible-infrared dataset has advantages over existing ones, and may be beneficial for future cross-modality person re-id research. However, considering the limited theoretical novelty of the proposed method, the lack of experiments and references to related works, as well as the writing quality, I’m inclined to reject.", "strength_and_weaknesses": "Pros:\n(1) A new larger-scale visible-infrared dataset is proposed, which is collected by more cameras and with a well-balanced amount of data for each modality.\n(2) The proposed method achieves state-of-the-art performance in most experiments.\n\nCons:\n(1) Eq. 1 appears to be inconsistent with the description in Sec. 4.2 and Fig. 5. From the description, both positive and negative samples seem to come from different modality from the anchor. Is there a labeling error in Eq. 1? If this is true, the proposed multi-mask triplet loss looks like a modified version of batch-hard triplet loss by changing the sampling strategy (i.e., restricting positive and negative samples to only one other modality). Furthermore, additional constraints on positive samples to make them closer are also often considered in other retrieval or matching tasks. In this case, the incremental modification somewhat limits the theoretical novelty of the proposed method. \nOn the other hand, this positive and negative sampling strategy is specialized for cross-modality matching. In real application scenarios, visible-infrared, visible-visible, and infrared-infrared matching may all be needed, while such a sampling strategy seems to only work in the visible-infrared case.\n\n(2) More ablation experiments are needed to demonstrate the effectiveness of the proposed loss function and augmentation strategy. Specifically, the proposed multi-mask triplet loss should be compared with the general batch-hard and batch-all triplet loss, and the proposed augmentation, i.e., exchanging parts of the image with a positive sample from another modality, should be compared with the mentioned baseline strategy, i.e., changing image regions using random samples. In addition, there are several hyper-parameters in the proposed augmentation, such as the exchanging probability, area, and location. It is better to analyze their sensitivities.\n\n(3) Lack of some related works on cross-modality person re-identification using two-stream frameworks. Some of the latest works have also achieved competitive or even better performance. More relevant papers need to be mentioned and differences need to be explained.\nFor examples:\n- 1. Jiang et al. Cross-Modality Transformer for Visible-Infrared Person Re-Identification, ECCV 2022.\n- 2. Zhang et al. FMCNet: Feature-Level Modality Compensation for Visible-Infrared Person Re-Identification, CVPR 2022.\n- 3. Liu et al. Learning Memory-Augmented Unidirectional Metrics for Cross-modality Person Re-identification, CVPR 2022.\n- 4. Zhang et al. Dual Mutual Learning for Cross-Modality Person Re-Identification, IEEE transactions on circuits and systems for video technology, Vol. 32, No. 8, 2022.\n- 5. Chen et al. Structure-Aware Positional Transformer for Visible-Infrared Person Re-Identification, IEEE transactions on image processing, Vol. 31, 2022.\n- 6. Zhang et al. Attend to the Difference: Cross-Modality Person Re-Identification via Contrastive Correlation, IEEE transactions on image processing, Vol. 30, 2021.\n- 7. Lu et al. Cross-modality Person re-identification with Shared-Specific Feature Transfer, CVPR 2020.\n\n(4) Minor:\n- To avoid misleading, the results of the proposed method that are not the best in the tables should not be bolded.\n- For easier understanding, it is better not to define too many variables that are never used in the following content (e.g., in Sec. 4.3). Useful variables should be defined in advance (e.g., $r$ and $A$).\n- Some abbreviations need to be defined, e.g., MMAP and MMAN in Sec. 4.3.\n- Some typos and grammatical errors exist, for example,\nthe sample strategy random sample -> the sampling strategy randomly samples (p. 6),\nrandom selected -> randomly selected (p. 6),\ncompared -> compared to (p. 6),\nresulting at the training is inefficient -> resulting in the training being inefficient (p. 6),\none identities -> one identity (p. 6).\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}]

Author rebuttal:

Decision: Reject