Learning to Reduce Dual-level Discrepancy for Infrared-Visible Person
Re-identification
Zhixiang Wang1 Zheng Wang2∗ Yinqiang Zheng2 Yung-Yu Chuang1 Shin’ichi Satoh2,31National Taiwan University 2 National Institute of Informatics 3 The University of Tokyo
1.0

Abstract𝛿𝑚

1. Introduction
Person re-identification (re-ID) has recently received increasing attention in the computer vision community [18,17, 6, 7, 20, 28] because of its great importance in video
surveillance. Most current re-ID methods rely on person’s
appearance under good visible light conditions. Under poor
illumination conditions, due to poor appearance, conventional re-ID models could become “blind in dark”. In prac∗ Corresponding Author

Left Gap

0.8

Initial Gap

0.6

Initial Gap

𝛿𝑎

mAP

Infrared-Visible person RE-IDentification (IV-REID) is
a rising task. Compared to conventional person reidentification (re-ID), IV-REID concerns the additional
modality discrepancy originated from the different imaging
processes of spectrum cameras, in addition to the person’s
appearance discrepancy caused by viewpoint changes, posevariations and deformations presented in the conventional
re-ID task. The co-existed discrepancies make IV-REID
more difficult to solve. Previous methods attempt to reduce
the appearance and modality discrepancies simultaneously
using feature-level constraints. It is however difficult toeliminate the mixed discrepancies using only feature-level
constraints. To address the problem, this paper introduces anovel Dual-level Discrepancy Reduction Learning (D2 RL)
scheme which handles the two discrepancies separately.
For reducing the modality discrepancy, an image-level subnetwork is trained to translate an infrared image into its
visible counterpart and a visible image to its infrared version. With the image-level sub-network, we can unify the
representations for images with different modalities. With
the help of the unified multi-spectral images, a feature-level
sub-network is trained to reduce the remaining appearance
discrepancy through feature embedding. By cascading thetwo sub-networks and training them jointly, the dual-level
reductions take their responsibilities cooperatively and attentively. Extensive experiments demonstrate the proposed
approach outperforms the state-of-the-art methods.

Left Gap

0.4

0.2
0.0
1

Visible domain

Intra person

Infrared domain

Inter person

V–V

10
2500
Training steps

5000

IR–IR

IV-REID

Figure 1. Illustration of the difficulty in the IV-REID task. (Left)
The modality discrepancy δm is much larger than δa , the appearance discrepancy. Thus, the modality discrepancy could result in the intra-person distance D( , ) being larger than theinter-person distance D( , ). (Right) The performance obtainedby [23] on the RegDB dataset using only the feature-level constraints. “V-V” and “IR-IR” respectively denote the performanceof visible-visible and infrared-infrared re-ID. The red gap indicates the performance gap between visible-visible and infraredinfrared single-modality re-ID. The black gap denotes the performance gap between cross-modality re-ID and single-modality reID. It is clear that the re-ID problem across modalities is much
more difficult than the one with the same modality.

tice, for dealing with poor illumination, most surveillancecameras automatically switch from the visible mode to theinfrared mode in the dark [21]. Consequently, this raises
a new task in which, given a visible (or infrared) image ofa specific person, the goal is to find the corresponding infrared (or visible) images of the person captured by otherspectrum cameras [21, 22, 23, 2]. This cross-modality image matching task is named Infrared-Visible person REIDentification (IV-REID).
Compared to the conventional re-ID task, IV-REID encounters additional modality discrepancy resulting from thedifferences between imaging processes of different spectrum cameras, in addition to the person’s appearance discrepancy caused by viewpoint changes, pose variations,scale changes and deformations. The modality discrepancy
is often more significant than the appearance discrepancy.

Imagespace

Large GapxyUnifiedspace

Featurespace
LegendImage-level
reduction
Feature-level
reduction
Mixed
discrepency
Feature-level
discrepency
State

Figure 2. A high-level overview of our approach. Previous methods reduce the total discrepancy by converting images to the feature space and using only the feature-level constraint (the blue
dashed line). Our dual-level solution first converts images intoa unified space (the green arrows) and then embeds them into thefeature space (the orange arrow). After the image-level modality reduction by unifying image representations, the gap becomesmuch smaller than the one in the original image space. Thus, thefeature-level embedding can be effective in reducing the remaining
appearance discrepancy.

The intra-person (images from the same person) distanceacross visible and infrared cameras is often larger than theinter-person (images from different persons) distance of thesame type of cameras. While the main goal of IV-REID isstill to maximize the inter-person distance and meanwhile tominimize the intra-person distance. The co-existed modality and appearance discrepancies make IV-REID difficult(as Figure 1 shows). As far as we know, all previous methods [21, 22, 23, 2] regard the modality discrepancy δm as apart of the appearance discrepancy δa and attempt to reducethe mixed discrepancy δm+δa using feature-level constraints
employed by most conventional re-ID methods. Note thatthe performance gap between single-modality re-ID (“V-V”
or “IR-IR”) and cross-modality IV-REID is extremely large.
It indicates that the modality discrepancy cannot be effectively eliminated using only feature-level constraints.
Figure 2 gives the main idea of the proposed method.
Because they are taken with different modalities, infrared
images and visible images have quite different appearances.
Thus, direct mappings of them into the feature space can not
be effective. For alleviating the problem, we propose to reduce the modality discrepancy first by unifying the image
representations using image-level conversion. More specifically, we form a multi-spectral image by augmenting an infrared image with its visible counterpart or a visible image
with its infrared version. In the unified space, the appearance discrepancy is greatly alleviated. Thus, conventional
re-ID methods can be effective in reducing the remaining
appearance discrepancy by feature-level constraints.
With the idea in mind, we propose a novel dual-level
solution, named Dual-level Discrepancy Reduction Learn-ing (D2 RL). We separate the modality discrepancy apart,and alleviate it through the image-level discrepancy reduction sub-network TI which unifies image representations bysynthesizing multi-spectral images from the given visible orinfrared images. The appearance discrepancy is then handled by the feature-level discrepancy reduction sub-network
TF , where feature embedding is more effective with the unified representation. These two sub-networks are cascaded
and jointly optimized in an end-to-end manner. To this end,
TF benefits TI to generate spectral images more discriminatively, and meanwhile TI provides TF with more translated
samples. The contributions of this paper are summarized
below:• A novel dual-level discrepancy reduction learning
scheme is introduced. We are the first to decompose themixed modality and appearance discrepancies and handle
them separately.
• Our end-to-end scheme enforces these two sub-networks
benefit each other. The balance between them affects theperformance.
• Extensive experiments on two datasets demonstrate thesuperior performance of our proposed approach compared to the state-of-the-art methods.

2. Related Work
Single-modality re-ID. The conventional re-ID researches
mainly focus on the challenges of appearance changes inthe single visible modality, such as image misalignment [6],viewpoint variations [11] and scale changes [16, 19]. Li etal. [6] formulated a harmonious attention CNN model forjoint learning of pixel and regional attention to optimize reID performance with misaligned images. Liu et al. [11] proposed a pose transferrable framework for generating samples with rich pose variations. Wang et al. [16] combined
effective embedding schemes built on multiple layers fromhigh- and low-level details. Wang et al. [19] cascaded
multiple super-resolution networks to overcome the resolution misalignment problem. Existing state-of-the-art singlemodality re-ID methods are very effective in reducing theappearance discrepancy as their retrieval accuracy has already surpassed the accuracy of human [24].
Infrared-visible re-ID. For the IV-REID problem, in addition to the appearance discrepancy, the modality discrepancy needs to be addressed. Existing methods attempt to reduce the mixed appearance and modality discrepancies using feature embedding frameworks similar to conventional
re-ID methods. Wu et al. [21] proposed a deep zero-padding
framework for shared feature learning under two different
modalities. Ye et al. [22] introduced a two-step frameworkfor feature learning and metric learning. They [23] also

proposed an end-to-end dual-path network to learn common representations. Dai et al. [2] designed a network tolearn discriminative representations from different modalities. Due to the modality discrepancy brought in by different spectrum with dramatic image-level unbalance, all thesefeature-level methods cannot obtain satisfactory results.
Image generation meets re-ID. Recently developed GANs
provide a powerful tool for image translation [29, 12, 1].
A lot of researches attempted to utilize GANs to generate
more training samples and then facilitate solving the conventional re-ID problem. Ma et al. [13] manipulated theforeground, background and pose information, and generated images based on manipulated information. Li et al. [7]used the GAN to generate target-like images. There is another category of researches trying to deal with the problem of the domain gap using GANs. For the conventional
re-ID task, the domain gap mainly lies in the camera styleor illumination differences. Zhong et al. [28, 27] utilized
the CycleGAN with label smooth regularization to generate person images with different camera styles. Deng etal. [4] also exploited the CycleGAN with self-similarity anddomain-dissimilarity constraints. Li et al. [7] exploited the
CycleGAN to generate images under different illumination
conditions. With the similar idea, Wei et al. [20] proposeda person transfer GAN to bridge the domain gap. However,working towards the problem of different poses, illuminations, and camera styles, all these methods focus on generating visible images based on visible images.

3. Proposed Method
Let X = {x | x ∈ RH×W×3 } and Y = {y | y ∈ RH×W×1 }denote the visible image set and the infrared image set respectively, where H and W are the height and the width ofimages. Each image x ∈ X or y ∈ Y corresponds to a labell ∈ {1, 2, . . . , Np }, and Np is the number of person identities. Given an infrared (or visible) query image y (or x) anda visible (or infrared) gallery set X (or Y ), the goal of the
IV-REID task is to propose a ranking list R of the gallery
set, in which the images with the same identity as the query
image should be ranked to the top. A common strategy is
projecting x and y to a feature space through feature embedding, fx = hx (x) and fy = hy (y), where fx ∈ Rd andfy ∈ Rd , and then generating the ranking list R using the
distance between them, fxT fy .
We propose a new strategy to replace the direct mapping function hx and hy . Figure 3 shows the frameworkof our proposed method. It consists of two sub-networks:(1) the image-level discrepancy reduction sub-network TI
for reducing the modality discrepancy, and (2) the featurelevel discrepancy reduction sub-network TF for reducing
the appearance discrepancy. These two sub-networks arecascaded and jointly optimized in an end-to-end manner. In
the following, we describe their details.

3.1. Image-level discrepancy reduction — TI
To reduce the modality discrepancy, TI exploits twovariational autoencoders (VAEs) for style disentanglement
followed by two GANs for domain specific image generation. TI translates the visible (infrared) image x (y) to its
infrared (visible) counterpart x̂ (ŷ). Together, they form themulti-spectral image [x, x̂] (or [ŷ, y]) to provide a unified
representation for reducing the modality discrepancy.
Style disentanglement. It consists of two encoder-decoder
pairs: VAEv = {Ev , Gv } and VAEi = {Ei , Gi }, that are responsible for visible and infrared modality disentanglement
respectively. For VAEv , given a visible input x ∈ X, theencoder Ev first maps x to the latent vector z, and then the
decoder Gv reconstructs the input from the latent vector z.
The reconstructed image is x̂v→v = Gv (zv ∼ qv (zv |x)),where qv (zv |x) is the distribution of latent information zv .
The loss for VAEv is defined as
LVAEv (Ev , Gv ) = λ0 KL(qv (zv |x)||pη (z))−λ1 Ezv ∼qv (zv |x) [kx − Gv (zv )k1 ] , (1)where the hyper-parameters λ0 and λ1 control the weightsof the objective terms, and the Kullback-Leibler divergenceterm (KL) penalizes deviation between the distribution of
the latent information and the prior pη (z) which is a zeromean Gaussian distribution. The `1 loss penalizes dissimilarity between the image and the reconstructed image, andalso encourages sharp output images.
Domain specific image generation. Two generative adversarial networks GANv = {Gv , Dv } and GANi = {Gi , Di }are employed to generate domain specific images from thestyle-free latent vector z. In GANv , the generator Gv is
expected to generate realistic visible images from the latentvector z that can fool the discriminator Dv , while the discriminator Dv is expected to discriminate real and synthetic
visible images. Adversarial losses are utilized to play theminimax game, which can be expressed as
LGANv (Ei , Gv , Dv ) = λ2 Ex∼PX [log Dv (x)]+λ2 Ezi ∼qi (zi |y) [log(1−Dv (Gv (zi )))] ,(2)

where the hyper-parameter λ2 controls the impact of GAN.
The discriminator is trained to maximize Equation (2) while
the generator tries to minimize it. The loss is used to ensure the translated images resemble images in the visible
domain.
Cycle-consistency. The cycle consistency is used to further regularize the ill-posed unsupervised image-to-imagetranslation problem. Similar to CycleGAN [29], our cycleconsistency loss is defined as
LCCv (Ev , Gv , Ei , Gi ) =λ3 Ezi ∼qi (zi |xv→i ) [kx − Gv (zi )k1 ] , (3)Feature Space

Unified Space

Image Spaceftx

Deep feature extractor

ŷ

Ev

N/Y

x

F

f

Cross

⁞

z
y
y
Visibleimage

x̂Infraredimage

TripletN/YMulti-spectral image[x, xˆ] or [y, yˆ]fc
Feature vector

E

Encoder

G Generator D Discriminator

Figure 3. The framework of our proposed method. The image-level discrepancy reduction sub-network TI first projects inputs from theimage space (visible or infrared modality) to the unified space, where the modality discrepancy is alleviated. Then, the feature-level
discrepancy reduction sub-network TF is utilized to eliminate the remaining appearance discrepancy. The two sub-networks are cascaded
and jointly optimized in an end-to-end manner.

where the negative log-likelihood objective term ensuresa twice translated image resembles the input. The hyperparameter λ3 controls the weight of this objective term.
The losses LVAEi (Ei , Gi ), LGANi (Ev , Gi , Di ) andLCCi (Ei , Gi , Ev , Gv ) can be similarly defined. Morespecifically, they are defined by substituting the subscript
i for v, v for i, and y for x in Equation (1), Equation (2)and Equation (3).
Modality unification. There are three possible options formodality unification, i.e., unifying images to the infrared
modality, the visible modality or the multi-spectral modality. We choose to generate the multi-spectral images formodality unification for two reasons. First, the infrared andvisible images are two representations of the same reflected
light of the same person due to different imaging processes.
They are potentially related and it is likely for them to reconstruct each other. Second, if we unify the images to thevisible or infrared modality, some distinctive information in
the visible or infrared modality may be lost.
Objective for training TI . The total loss is a combinationof the VAE loss, the GAN loss and the CC loss:
LI = LVAEv +LVAEi +LGANv +LGANi +LCCv +LCCi . (4)By optimizing the above loss, we obtain a network TI which
is able to translate a visible image x into its infrared counterpart x̂ and translate an infrared image y into its visible
counter part ŷ. Thus, we can form a training set S by constructing multi-spectral images, uv = [x, x̂] and ui = [ŷ, y],as the unified representations. This way, all images includ-ing both query and gallery images are represented in thesame way and the modality discrepancy is greatly reduced.

3.2. Feature-level discrepancy reduction — TF
Since TI has unified all the images to the same modality,
a feature embedding network could be sufficient for reducing the appearance discrepancy. For each batch, we use TI
to generate a sample set S. The feature-level discrepancy
reduction network TF plays a role of feature learning on
the unified multi-spectral images generated by TI . Given amulti-spectral image u, sampled from S, the deep feature
extractor F : u → f maps it to the person descriptor f . In
particular, we use ResNet-50 as the backbone network of
F and follow the training strategy in [27]. The last 1000d fully connected (FC) layer is replaced with a new layer
named as “FC-1024”. The person descriptor f ∈ R1024 uses
the output feature vector of “FC-1024” followed by Batch
Normalization, ReLU and Dropout. The output f of the
FC-1024 layer is then fed to two independent FC layers Ht
and Hc for generating two feature vectors ft ∈ R128 andfc ∈ RNp . Two types of loss functions are utilized to supervise the training of TF . One is the triplet loss, employed
for identity information learning, and the other is the crossentropy loss, used for similarity learning. The triplet loss iscoupled with ft while the cross-entropy loss is tied with fc .
Triplet loss. It is used for similarity learning. It tries to reduce the feature distances between images of the same person and expand the distances between images of different

people. The triplet loss can be formulated as following:X[D(fta , ftp ) − D(fta , ftn ) + ξ]+ , (5)LTF =fta ,ftp ,ftn∈S

where fta is the anchor point; ftp is a positive sample withthe same identity with fta ; and ftn is a negative sample withthe different identity from fta . Note that fta 6= ftp . ξ is amargin parameter. D(·) calculates the Euclidean distance,and [d]+ = max(d, 0) truncates negative numbers to zero
while keeping positive numbers the same.
Cross-entropy loss. It is employed for identity learning andis written as
N

LC
F =−

b
1 X
log pj ,Nb j=1(6)(7)

3.3. End-to-end joint training
We optimize our network in an end-to-end manner, bycascading TI and TF and minimizing the combined loss:
arg min(1 − γ)LI + γLF ,θTI ,θTF

Evaluation metrics. The standard Cumulative Matching
Characteristics (CMC) curve and mean Average Precision(mAP) are adopted to evaluate the performance. Note that
there is a slight difference with the conventional re-ID problem [25]. Images from one modality are used as the gallery
set while the ones from the other modality as the probe set
during testing.

4.2. Implementation details

where Nb = |S| is the number of images in the training minibatch; p is the predicted probability of the input belonging
to the ground-truth class with p = softmax(Wf + b),
where W and b are the trainable weight and bias of Hc .
Objective for training TF . The loss is a combination of
the cross-entropy and triplet losses as follows:
T
LF = λ4 LC
F + λ5 LF .

by at least two different cameras. Following [21], we
adopt the most challenging single-shot all-search mode
evaluation protocol. The training set contains 395 persons, with 22,258 visible images and 11,909 infrared images. The testing set contains 96 persons, with 3,803 infrared images for query and 301 randomly selected visible images as the gallery set.

(8)

where 0 < γ < 1 and it is a trade-off parameter for balancing
the contributions of two sub-networks TI and TF .

4. Experiments
This section reports the experiment settings, implementation details, comparisons with other methods, the ablation
study and analysis of our method.

4.1. Experiment settings
Datasets. We evaluated our method on two publicly available datasets: RegDB [15] and SYSU-MM01 [21].
• RegDB [15]. It was collected from two aligned cameras (one visible and one far-infared). It contains totally 412 persons. Each person has 10 visible images and
10 far-infrared images. We follow the evaluation protocol in [23] to randomly split the dataset into two halves,
which are used for training and testing respectively.
• SYSU-MM01 [21]. It is a large-scale dataset collectedby six cameras (four visible and two near-infared), including both indoor and outdoor environments. It contains in total 491 persons, and each person was captured

Network architecture. The architecture of our proposed
method is shown in Figure 3. The sub-network TI is basedon UNIT1 . The size of inputs and outputs is resized to
228×228×3, for both visible and infrared images. For infrared images, the three channels are the same. TF is basedon Open-reid2 with the difference that our inputs have fourchannels.
Training strategy. In order to avoid mode collapse andover-fitting, we pretrained the sub-network TI and TF respectively with the Market-1501 dataset [25], where weused the original images as the visible input, and the decomposed illuminations as the infrared input. Then, we jointly
trained them in an end-to-end manner. Note that the SYSUMM01 [21] dataset includes outdoor and indoor scenes. We
trained them separately. We set weight parameters of thelosses in TI as λ0 = 0.1, λ1 = 100, λ2 = 10, λ3 = 100 by following [12]. For the sub-network TF , we set λ4 = λ5 = 10.
The pre-defined margin for the triplet loss is set as ξ = 0.8.
The model is optimized using Adam [5] with a learning rateof 0.0002 and the momentum terms β1 = 0.5, β2 = 0.999.

4.3. Comparison with the state-of-the-art methods
To demonstrate the effectiveness of our method, we
compare our method with most of the related methods
for IV-REID. These methods include Zero-Padding [21],TONE [22], HCML [22], BDTR [23] and cmGAN [2]. In
addition, several other learning-based methods are also included for comparisons. The additional competing methods
contain some feature learning methods including HOG [3],LOMO [8], one-stream and two-stream networks [21]. The
one-stream and two-stream networks are modifications of
the IDE method [26] under IV-REID settings. Their detailed
descriptions can be found in [21]. In addition, two matching model learning methods, MLAPG [9] and GSM [10],1 UNIT code: https://github.com/mingyuliutw/UNIT
2 Open-reid code: https://github.com/Cysu/open-reidTable 1. Comparison with the state-of-the-art IV-REID methods on two different datasets, RegDB and SYSU-MM01.
Constraints
Feature-level
Image-level

Approach

7
7
7
7
3
3
3
3
3
3
3
3

LOMO [8]MLBP [9]HOG [3]
GSM [10]One-stream [21]Two-stream [21]Zero-Padding [21]TONE [22]HCML [22]BDTR [23]cmGAN [2]Proposed D2 RL

7
7
7
7
7
7
7
7
7
7
7
3

CMC-1

RegDB
CMC-10
CMC-20mAP

CMC-1

SYSU-MM01
CMC-10
CMC-20

mAP

0.85
2.02
13.49
17.28
13.11
12.43
17.75
16.87
24.44
33.47
–
43.4

2.47
7.33
33.22
34.47
32.98
30.36
34.21
34.03
47.53
58.42
–
66.1

2.28
6.77
10.31
15.06
14.02
13.42
18.90
14.92
20.80
31.83
–
44.1

1.75
2.12
2.76
5.29
12.04
11.65
14.80
12.52
14.32
17.01
26.97
28.9

14.14
16.23
18.25
33.71
49.68
47.99
54.12
50.72
53.16
55.43
67.51
70.6

3.48
3.86
4.24
8.00
13.67
12.85
15.95
14.42
16.16
19.66
27.80
29.2

Table 2. Ablation study on the RegDB dataset.
MethodComponents
VAE CC CE triplet

Baseline
D2 RL (no VAE)
D2 RL (no CC)
D2 RL (no CE)
D2 RL (no triplet)
D2 RL

3
7
3
3
3
3

3
3
7
3
3
3

7
3
3
7
3
3

7
3
3
3
7
3

26.63
28.32
31.91
52.95
66.74
65.50
71.33
68.60
69.17
71.96
80.56
82.4

4.5. Discussions

RegDB
CMC-1 (%) mAP (%)
28.5
34.8
33.7
41.7
39.5
43.4

4.10
10.90
43.66
45.26
42.51
40.96
44.35
44.10
56.78
67.52
–
76.3

23.8
31.3
29.9
40.6
37.4
44.1

are also included for comparisons. Table 1 presents the results of all the methods. The methods specially designed
for IV-REID generally perform much better than the ones
that are not designed for IV-REID. Our method significantly
outperforms the state-of-the-art IV-REID methods on both
the RegDB and SYSU-MM01 datasets.

4.4. Ablation study
Our method consists of two sub-networks, the imagelevel discrepancy reduction sub-network TI and the featurelevel discrepancy reduction sub-network TF , respectively
taking GAN and ResNet-50 as their backbones. TI is
mainly configured with the VAE and cycle-consistency(CC) losses while TF is optimized with the cross-entropy(CE) and triplet losses. For the ablation study, Table 2 reports the resultant CMC-1 and mAP values on the RegDB
dataset by removing one loss at a time. Note that thebaseline is obtained by only using TF without image-level
modality unification.
Note that the first two losses, VAE (for modality disentanglement) and cycle-consistency (for modality transfer),
are responsible for the image-level modality unification.
Removing either of them affects the image generation, thus
degrading the performance more significantly. When removing both of them (the baseline), the performance drops
dramatically to 28.5% in CMC-1 as it can only rely on feature embedding across modalities. The triplet loss is slightly
more effective than the cross-entropy loss.

Why reducing discrepancy separately? For IV-REID,
previous methods try to reduce the appearance and modality discrepancies together from the feature-level view. Our
method aims at reducing the appearance and modality discrepancies separately. We compare a feature-level method
BDTR [23] with our proposed dual-level discrepancy reduction method D2 RL for investigating which strategy ismore effective. We evaluate both BDTR and our methodon the RegDB dataset. First, we plot the 1024-d person descriptor in the 2D feature space for visualization using thet-SNE method [14]. Testing samples of 20 persons wererandomly selected from the RegDB dataset. Figure 4(a)and Figure 4(b) stand for visualizations of the initial andbest results of the BDTR model respectively. We can observe that single-modality intra-person samples get closer
to each other after training, but cross-modality intra-person
samples relatively do not change too much. Figure 4(c) and
Figure 4(d) are visualizations of the initial and best resultsof our proposed D2 RL network respectively. From thesefigures, we can find that not only the single-modality intraperson samples get closer to each other after training, butalso some cross-modality intra-person samples move closer(as indicated by the red cycles in Figure 4(d)).
For further validating the effectiveness of reducing discrepancy separately, we conduct experiments to see how
effective the embedded features are by again comparing
BDTR and our method on the RegDB dataset. Figure 4(e)shows the initial distribution of distances of the inter-person
and intra-person pairs for BDTR. Figure 4(f) shows the distribution after 4,800 training steps. Figure 4(g) and (h) show
the distributions before and after training for our method.
It is clear that, after training, our method can separate theinter-person and intra-person pairs more further apart than
BDTR. It indicates that feature embedding in the unified
space is more effective than the one in the image space.
Which modality to unify? We evaluate three options formodality unification, i.e., the visible modality, the infrared

(a) BDTR initial

700

600

(c) Ours initial

700

intra-person
inter-person

600

400

400

400

300

200

200

200

100

100

100

0
0.0

0.2

0.4

0.6

0.8

dist.

1.0

1.2

1.4

1.6

0
0.0

0.2

(e) BDTR initial

0.4

0.6

0.8

dist.

1.0

1.2

1.4

(f) BDTR best

1.6

#pairs500

400#pairs

500

300

0
0.0

intra-person
inter-person

600

500

300

(d) Ours best700

intra-person
inter-person

500#pairs#pairs600(b) BDTR best700

intra-person
inter-person

300
200
100

0.2

0.4

0.6

0.8

dist.

1.0

1.2

1.4

1.6

0
0.0

0.2

(g) Ours initial

0.4

0.6

0.8

dist.

1.0

1.2

1.4

1.6

(h) Ours best

Figure 4. (Top) Visualization of the feature space. A total of 20 persons are randomly selected from the testing set. Samples with the samecolor indicate they are of the same person. The markers “dot” and “cross” denote images from the visible and infrared domain respectively.
(a-b) are obtained using BDTR [23] on the RegDB datasetm which uses only feature-level constraints; (c-d) are obtained by our method.
(Bottom) Histogram of the feature distances. (e-f) are obtained by BDTR; (g-h) are obtained by our method.
Table 3. Comparison of different modality unification options.
Metrics (%)

RegDB
mAP
CMC-1

SYSU-MM01
mAP
CMC-1

D2 RL(v)
D2 RL(i)
D2 RL

36.4
43.6
44.1

28.4
27.8
29.2

39.1
42.9
43.4

28.1
27.4
28.9

modality and the multi-spectral modality. We respectively
use D2 RL(v), D2 RL(i) and D2 RL to denote these three options. Table 3 shows the results and there are several observations. First, unification to the multi-spectral modality, D2 RL, performs the best. Second, compared with othermethods in Table 1, modality unification helps no matter which modality is chosen for unification. Finally, we
find that D2 RL(i) performs better than D2 RL(v) on the
RegDB dataset, while performing worse on the SYSUMM01 dataset. We attribute this phenomenon to the setting for dataset evaluation. For the RegDB dataset, thegallery consists of infrared images, implying that the infrared modality plays an important role. The majority oforiginal infrared images makes D2 RL(i) more effective on
the dataset. As for the SYSU-MM01 dataset, the gallery
consists of visible images, the results go the other way
round. Our unification to the multi-spectral modality takes
advantages of both domains and is thus more robust.
Why joint training? The whole framework consists ofan image-level discrepancy reduction sub-network and afeature-level discrepancy reduction sub-network. They playdifferent roles. They can be trained separately or jointly.
Table 4 compares these two options. First, joint training provides significant performance boost as the two sub-Table 4. Comparison between joint and separate training.
Metrics (%)

RegDB
mAP
CMC-1

SYSU-MM01
mAP
CMC-1

Separate
Joint

40.7
44.1

25.7
29.2

39.9
43.4

26.1
28.9

networks benefits each other. Second, when comparing withother methods in Table 1, even with the separate training,
our method outperforms the state-of-the-art methods.
How to balance sub-networks TI and TF ? In the total
loss of the proposed method defined in Equation (8), we
use the weight γ to balance the contributions of TI and TF .
As TI focuses on the modality discrepancy reduction and
TF pays attention to the appearance discrepancy reduction,
the larger γ is, the more contribution will attribute to theappearance reduction, in other words, the feature-level discrepancy reduction sub-network.
Figure 5 shows the results of the mAP and CMC-1 values on RegDB dataset by varying the weight γ. We can findthat the re-identification accuracy varies when the weight γchanges, and there exists a suitable value to balance the contributions of TI and TF . Although TI alleviates the modality discrepancy, it could also brings in noisy information.
Thus, the balance between TI and TF is important.

4.6. Visualization of results
The ability of modality unification. To demonstrate theeffectiveness of our image-level discrepancy reduction subnetwork TI , we show some visual results of image translation in Figure 6. For each of the RegDB and SYSU-MM01
datasets, we show six groups of images. Each group has

50

50

44.1

45
40.7

41.6

41.2
39.7

36.6
33.9

35

Baseline
Ours

30

25

20

23.8

0

43.4

45
41

41.7

40
CMC top-1 (%)

mAP (%)

40

43.443

36.4
35

30

GT
42.9
41.8
40.239.4

Top-10 Ranking

Query

ŷ

x̂

y

x

x̂

y

x

ŷ

Baseline
Ours

32.8
28.5

25

0.2

0.4
0.6
Parameter

0.8

1

20

0

0.2

0.4
0.6
Parameter

0.8

1

Figure 5. Curves of mAP (left) and CMC-1 (right) with respect tothe hyper-parameter γ on RegDB dataset.

Figure 7. Visualization of failure cases. (Top) The query set is[ŷ; y] and the gallery set is [x; x̂]; (Bottom) The query set is [x; x̂]and the gallery set is [ŷ; y].

MM01 dataset is more colorful, and the person images are
not aligned well with different postures and scales. It may
lead to difficulties for the image-level discrepancy reduction sub-network to be trained well. However, note that the
final goal is not to generate images with good visual appearances, but to have good retrieval results. From the results inTable 1, the translated images do help a lot on IV-REID.
Failure cases. We select the two worst query results (inwhich none of the top-10 results is correct) for illustrating
the failure cases. For each query, the two rows respectively
show the ranking list of generated images and the list of corresponding original images. It shows that, for some cases,
the generated images could be bad and image-level modality unification can not work well on these queries.

5. Conclusions(a) RegDB(b) SYSU-MM01

Figure 6. Examples of translated images generated by TI on (a)RegDB and (b) SYSU-MM01. For each dataset, from left to right,
the four images of a row are the original visible image, the generated infrared image, the original infrared image, and the generated
visible image respectively. The original visible and infrared images of the same row have the same identity.

four images: the original visible image, the generated infrared image, the original infrared image and the generated
visible images. From the visual examples, we can observethat the sub-network TI is good at translating visible images to infrared ones, and the effectiveness of translating
infrared images to visible ones is acceptable. However,some generated images could have color distortion, such as
the sixth person of Figure 6(a). We can also find that thetranslation results of the RegDB dataset looks better thanthose of the SYSU-MM01 dataset. It is because the SYSU-In this paper, we present Dual-level Discrepancy Reduction Learning network (D2 RL) for the IV-REID task thatexhibits both modality discrepancy and appearance discrepancy. Unlike previous IV-REID methods, instead of handling the mixed discrepancy with feature embedding, we
propose to handle the discrepancies separately. We propose
an image-level sub-network for modality unification, which
generates a unified multi-spectral representation by imagetranslation. With the unified representations, a feature-level
sub-network can better reduce the appearance discrepancy
by feature embedding. The proposed method shows significant improvement against the state-of-the-art methods.
Acknowledgements. This work was finished when Zhixiang Wang visited the Optical Sensing and Camera System Laboratory, National Institute of Informatics, Japan,
through the NII International Internship Program. This
work was supported by JST CREST (JPMJCR1686), Grantin-Aid for JSPS Fellows (18F18378), NSFC (61801335),
MOST (107-2221-E-002-147-MY3), and Microsoft Research Asia.

References[1] Yu-Sheng Chen, Yu-Ching Wang, Man-Hsin Kao, and YungYu Chuang. Deep photo enhancer: Unpaired learning for image enhancement from photographs with GANs. In Proceedings of Conference on Computer Vision and Pattern Recognition, 2018.
[2] Pingyang Dai, Rongrong Ji, Haibin Wang, Qiong Wu, andYuyu Huang. Cross-modality person re-identification withgenerative adversarial training. In Proceedings of International Joint Conferences on Artificial Intelligence, 2018.
[3] Navneet Dalal and Bill Triggs. Histograms of oriented gradients for human detection. In Proceedings of Conference on
Computer Vision and Pattern Recognition, 2005.
[4] Weijian Deng, Liang Zheng, and Jianbin Jiao. Imageimage domain adaptation with preserved self-similarity anddomain-dissimilarity for person reidentification. In Proceedings of Conference on Computer Vision and Pattern Recognition, 2018.
[5] Diederik P. Kingma and Jimmy Ba. Adam: A method forstochastic optimization. arXiv:1412.6980, 2014.
[6] Wei Li, Xiatian Zhu, and Shaogang Gong. Harmonious attention network for person re-identification. In Proceedingsof Conference on Computer Vision and Pattern Recognition,
2018.
[7] Xiang Li, Ancong Wu, and Wei-Shi Zheng. Adversarial
open-world person re-identification. In Proceedings of European Conference on Computer Vision, 2018.
[8] Shengcai Liao, Yang Hu, Xiangyu Zhu, and Stan Z. Li. Person re-identification by local maximal occurrence representation and metric learning. In Proceedings of Conference on
Computer Vision and Pattern Recognition, 2015.
[9] Shengcai Liao and Stan Z. Li. Efficient PSD constrained
asymmetric metric learning for person re-identification. In
Proceedings of International Conference on Computer Vision, 2015.
[10] Liang Lin, Guangrun Wang, Wangmeng Zuo, Xiangchu
Feng, and Lei Zhang. Cross-domain visual matching via
generalized similarity measure and feature learning. IEEE
Transactions on Pattern Analysis and Machine Intelligence,39(6):1089–1102, 2017.
[11] Jinxian Liu, Bingbing Ni, Yichao Yan, Peng Zhou, Shuo
Cheng, and Jianguo Hu. Pose transferrable person reidentification. In Proceedings of Conference on Computer
Vision and Pattern Recognition, 2018.
[12] Ming-Yu Liu, Thomas Breuel, and Jan Kautz. Unsupervised image-to-image translation networks. In Proceedingsof Neural Information Processing Systems Conference, 2017.
[13] Liqian Ma, Qianru Sun, Stamatios Georgoulis, Luc
Van Gool, Bernt Schiele, and Mario Fritz. Disentangled
person image generation. In Proceedings of Conference on
Computer Vision and Pattern Recognition, 2018.
[14] Laurens van der Maaten and Geoffrey Hinton. Visualizing
data using t-SNE. Journal of Machine Learning Research,
9(Nov):2579–2605, 2008.
[15] Dat Tien Nguyen, Hyung Gil Hong, Ki Wan Kim, andKang Ryoung Park. Person recognition system based on a
combination of body images from visible light and thermal
cameras. Sensors, 17(3):605, 2017.

[16] Yan Wang, Lequn Wang, Yurong You, Xu Zou, Vincent
Chen, Serena Li, Gao Huang, Bharath Hariharan, and Kilian Q. Weinberger. Resource aware person re-identification
across multiple resolutions. In Proceedings of Conference on
Computer Vision and Pattern Recognition, 2018.
[17] Zheng Wang, Ruimin Hu, Chen Chen, Yi Yu, Junjun Jiang,
Chao Liang, and Shin’ichi Satoh. Person reidentification via
discrepancy matrix and matrix metric. IEEE Transactions on
Cybernetics, 2018.
[18] Zheng Wang, Ruimin Hu, Chao Liang, Yi Yu, Junjun Jiang,
Mang Ye, Jun Chen, and Qingming Leng. Zero-shot person
re-identification via cross-view consistency. IEEE Transactions on Multimedia, 18(2):260–272, 2016.
[19] Zheng Wang, Mang Ye, Fan Yang, Xiang Bai, and Shin’ichi
Satoh. Cascaded SR-GAN for scale-adaptive low resolution person re-identification. In Proceedings of International
Joint Conferences on Artificial Intelligence, 2018.
[20] Longhui Wei, Shiliang Zhang, Wen Gao, and Qi Tian. Person transfer GAN to bridge domain gap for person reidentification. In Proceedings of Conference on Computer
Vision and Pattern Recognition, 2018.
[21] Ancong Wu, Wei-Shi Zheng, Hong-Xing Yu, Shaogang
Gong, and Jianhuang Lai. RGB-Infrared cross-modality person re-identification. In Proceedings of International Conference on Computer Vision, 2017.
[22] Mang Ye, Xiangyuan Lan, Jiawei Li, and Pong C. Yuen. Hierarchical discriminative learning for visible thermal person
re-identification. In Proceedings of AAAI Conference on Artificial Intelligence, 2018.
[23] Mang Ye, Zheng Wang, Xiangyuan Lan, and Pong C. Yuen.
Visible thermal person re-identification via dual-constrained
top-ranking. In Proceedings of International Joint Conferences on Artificial Intelligence, 2018.
[24] Xuan Zhang, Hao Luo, Xing Fan, Weilai Xiang, Yixiao Sun,
Qiqi Xiao, Wei Jiang, Chi Zhang, and Jian Sun. Alignedreid: Surpassing human-level performance in person reidentification. arXiv:1711.08184, 2017.
[25] Liang Zheng, Liyue Shen, Lu Tian, Shengjin Wang, Jingdong Wang, and Qi Tian. Scalable person re-identification:
A benchmark. In Proceedings of International Conference
on Computer Vision, 2015.
[26] Liang Zheng, Yi Yang, and Alexander G Hauptmann. Person re-identification: Past, present and future.
arXiv:1610.02984, 2016.
[27] Zhun Zhong, Liang Zheng, Shaozi Li, and Yi Yang. Generalizing a person retrieval model hetero- and homogeneously.
In Proceedings of European Conference on Computer Vision,
2018.
[28] Zhun Zhong, Liang Zheng, Zhedong Zheng, Shaozi Li,
and Yi Yang. Camera style adaptation for person reidentification. In Proceedings of Conference on Computer
Vision and Pattern Recognition, 2018.
[29] Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A
Efros. Unpaired image-to-image translation using cycleconsistent adversarial networks. In Proceedings of International Conference on Computer Vision, 2017.