New Trends of Person re-ID System
Zheng Wang
National Institute of Informatics, Japan

ACM MM 2020 Tutorial on “Effective and Efficient: Toward Open-world Instance Re-identification”

Outline• General Person Re-identification• Person Re-identification New Trends• Cross-modality• Long-term• Group• Discussion

Person Re-identification
XXX Case in Nanjing,
China

……
Search for XXX

1500 Police, One Month

329 shots

Person Re-identification
XXX Case in Nanjing,
China

……
Search for XXX

329 shots

1500 Police, One Month

Person Re-identification

Same person?

Camera aCamera b

Person Re-identification• Image Retrieval / Instance Search• Target: Certain Person

Camera A Camera CCamera ECamera DCamera BDottUnknown

pub pub street

pub2Jim

LR1 LR1 LR1BradUnknown
actor

Surveillance Camera [1]Timeline

LR1

pub2

LR1streetFirst Person Vision [3]TV Media [2][1] Wang, et al., Incremental Re-identification by Cross-Direction and Cross-Ranking Adaption, TMM, 2019[2] Fischer, et al., Person re-identification in tv series using robust face recognition and user feedback, MTAP, 2011[3] Chakraborty, et al., Person re-identification using multiple first-person-views on wearable devices, WACV, 2016

Robot Vision

General Person Re-identification• Challenges• Appearance changes / No Clothes change• Intra-Modality

Viewpoint [6]Occlusion [4]Illumination [5][4] Luo, et al., STNReID: Deep Convolutional Networks with Pairwise Spatial Transformer Networks for Partial Person Re-identification, TMM, 2020
[5] Zeng, et al., Illumination-Adaptive Person Re-identification, TMM, 2020
[6] Wu, et al., Viewpoint Invariant Human Re-Identification in Camera Networks Using Pose Priors and Subject-Discriminative Features, TPAMI, 2014General Person Re-identification• Challenges• Appearance changes / No Clothes change• Intra-Modality

Diagramgallery from camera
view Bprobe from camera
view A…

Personrepresentation

Personrepresentation
Similarity
measureViewpoint [6]Occlusion [4]Illumination [5][4] Luo, et al., STNReID: Deep Convolutional Networks with Pairwise Spatial Transformer Networks for Partial Person Re-identification, TMM, 2020
[5] Zeng, et al., Illumination-Adaptive Person Re-identification, TMM, 2020
[6] Wu, et al., Viewpoint Invariant Human Re-Identification in Camera Networks Using Pose Priors and Subject-Discriminative Features, TPAMI, 2014

ranking result…

…

General Person Re-identification

Market-1501 [7]DukeMTMC-reID [8]• Rank-1 accuracy surpass the human performance [9]• Intra-modality discrepancy has been well addressed• daytime, visible spectrum, sufficient details[7] https://paperswithcode.com/sota/person-re-identification-on-market-1501[8] https://paperswithcode.com/sota/person-re-identification-on-dukemtmc-reid[9] Zhang, et al., AlignedReID: Surpassing Human-Level Performance in Person Re-Identification, arXiv, 2018

Outline• General Person Re-identification• Person Re-identification New Trends• Cross-modality• Long-term• Group• Discussion

Person Re-identification New Trends
simulated scene

BAC

DLong term

A

7:00 PMCD
1:00 PM
4:00 PMcamera space

CCross modality

camera spacecamera space

ADA

GroupCD

Cross-modality
different camera specifications and settings(low- vs. high resolution data)

different sensory devices(infrared vs. visible light devices)

reproduction of human memory and direct recording by a camera(sketch/text description vs. digital images)

Cross-modality
Main Focus

Feature[10]

Gait sequencesa special and different focus[11]

Appearance

a multi-dimensional overview[12]

Appearancea systematic evaluation with different features and metrics[13]

Appearance

a limited summary of current efforts or problems present in different modalities[10] Nambiar, et al., Gait-based person re-identification: A survey. ACM Computing Surveys, 2019[11] Vezzani, et al., People reidentification in surveillance and forensics: A survey. ACM Computing Surveys, 2013.
[12] Gou, et al. A systematic evaluation and benchmark for person re-identification: Features, metrics, and datasets. TPAMI, 2018[13] Leng, et al., A survey of open-world person re-identification. TCSVT, 2019.

Cross-modality
Main Focus

Feature[10]

Gait sequencesa special and different focus[11]

Appearance

a multi-dimensional overview[12]

Appearancea systematic evaluation with different features and metrics[13]

Appearance

a limited summary of current efforts or problems present in different modalities

There is also a big performance gap
between Homo-ReID and Hetero-ReID.

[10] Nambiar, et al., Gait-based person re-identification: A survey. ACM Computing Surveys, 2019[11] Vezzani, et al., People reidentification in surveillance and forensics: A survey. ACM Computing Surveys, 2013.
[12] Gou, et al. A systematic evaluation and benchmark for person re-identification: Features, metrics, and datasets. TPAMI, 2018[13] Leng, et al., A survey of open-world person re-identification. TCSVT, 2019.

The Diagram

Datasets

Market-1501

MLR-VIPeRSYSU-MM01

CUHK-PEDESPKU-Sketch

LR-HRLR-1-JUDEA [14]• Contributions• The first work focusing on the LRReID• a multi-scale discriminantdistance metric learning model

Heterogeneous class mean discrepancy (HCMD)[14] Li, et al., Multi-scale Learning for Low-resolution Person Re-identification, ICCV, 2015

• Existing ReID models have a clear
performance drop at the LR task,
but the proposed method does
not.

Metric Learning

LR-2-SDF [15]• Contributions• a new issue - Scale-adaptive Lowresolution Person Re-identification• the discriminating power of thefeasible and infeasible SDFs
respectively generated by positiveand negative image pairs[15] Wang, et al., Scale-Adaptive Low-Resolution Person Re-Identification via Learning a Discriminating Surface, IJCAI, 2016

Representation LearningLR-3-SING [16]• Contributions• image SR and ReID techniques in anovel unified formulation• a joint loss function on optimsing ahybrid CNN architecture• a multi-resolution adaptive fusion
mechanism by aggregating a set ofanchor SING CNN models[16] Jiao, et al., Deep Low-Resolution Person Re-Identification, AAAI, 2018

Modality Unification

LR-4-CSR-GAN [17]• Contributions• cascade multiple SRGANs in series,capable of super-resolving LR
images with multi-scale upscaling• the integration compatibility
between scale-adaptive superresolution and re-identification• a common-human loss to makethe super-resolved image look
more like human[17] Wang, et al., Cascaded SR-GAN for Scale-Adaptive Low Resolution Person Re-identification, IJCAI, 2018

Modality Unification

LR-5-CAD [18]• Contributions• advances adversarial learning
strategies• learns resolution-invariant
representations while recovering
the missing details in LR input
imagesModality Unification + Representation Learning[18] Li, et al., Recover and Identify: A Generative Dual Model for Cross-Resolution Person Re-Identification, ICCV, 2019

LR-6-INTACT [19]• Contributions• an idea of leveraging theassociation between image SR andperson re-id tasks• a regularisation method
implements the proposed intertask association

Modality Unification + Representation Learning[19] Cheng, et al., Inter-Task Association Critic for Cross-Resolution Person Re-Identification, CVPR, 2020

IR-RGBIR-1-Zero-padding [20]• Contributions• study the RGB-IR Re-ID for the firsttime and raise a standard benchmark• analyse three different network
structures (one-stream, two-stream
and asymmetric FC layer)• deep zero-padding[20] Wu, et al., RGB-Infrared Cross-Modality Person Re-Identification, ICCV, 2017

Representation LearningIR-2-BDTR [21]• Contributions• an end-to-end dual-path feature andmetric learning framework• a bi-directional dual-constrained
top-ranking loss to simultaneously
consider the cross-modality andintra-modality variations[21] Ye, et al., Visible thermal person re-identification via dual constrained top-ranking, IJCAI, 2018

Representation LearningIR-3-cmGAN [22]• Contributions• a loss function for cross-modality
generative adversarial network• identification loss and crossmodality triplet loss together forgenerator• a modality classifier as
discriminator[22] Dai, et al., Cross-Modality Person Re-Identification with Generative Adversarial Training, IJCAI, 2018

Representation Learning

2IR-4-D RL [23]• Contributions• A dual-level discrepancy reduction
learning scheme. the first to
decompose the mixed modality andappearance discrepancies.
• An end-to-end scheme enforcesthese two sub-networks benefit
each other.

[23] Wang, et al., Learning to Reduce Dual-level Discrepancy for Infrared-Visible Person Re-identification, CVPR, 2019

Modality UnificationIR-5-XIV [24]• Contributions• an adjoint and auxiliary X modality.
• an extra lightweight network togenerate the X modality throughself-supervised learning• a modality gap constraint to direct
the learning and knowledge
communication across modalities

cross modality gap (CMG) and the modality respective gap (MRG)[24] Li, et al., Infrared-Visible Cross-Modal Person Re-Identification with an X Modality, AAAI, 2020

Representation LearningIR-6-Hi-CMD [25]• Contributions• A Hierarchical Cross-Modality
Disentanglement (Hi-CMD) method
extracts pose- and illuminationinvariant features for cross-modality
matching.
• The proposed ID-preserving Person
Image Generation (ID-PIG) networkchanges the pose and illumination
attributes while maintaining theidentity characteristic of a specificperson.

[25] Choi, et al., Hi-CMD: Hierarchical Cross-Modality Disentanglement for Visible-Infrared Person Re-Identification, CVPR, 2020

Modality Unification

Text-Image

Text-1-GNA-RNN [26]• Contributions• study the problem of searching persons
with natural language• a novel Recurrent Neural Network withGated Neural Attention (GNA-RNN) for
person search[26] Li, et al., Person search with natural language description, CVPR, 2017

Affinity/Metric LearningText-2-CMPM+CMPC [27]• Contributions• a cross-modal projection matching (CMPM)
loss attempts to minimize the KL
divergence between projection
compatibility distributions and thenormalized matching distributions• a cross-modal projection classification(CMPC) loss attempts to classify the vector
projection of the features from one
modality onto the matched features fromanother modality[27] Zhang, et al., Deep cross modal projection learning for image-text matching, ECCV, 2018

Representation Learning

Sketch-PhotoSketch-1-CDAFL [28]• Contributions• A deep adversarial learning architecture to
jointly learn identity features and domaininvariable features• filtering low-level features and remaining
high-level semantic features.
• A sketch Re-ID dataset containing 200
persons, in which each person has one
sketch and two photos[28] Pang, et al., Cross-domain adversarial feature learning for sketch re-identification, ACM MM, 2018

Our Sketch-Photo Focus

Large-scaleMulti-styleSemi-professional

From the perspective of application scenario• Most of the methods selected a deep learning framework.

From the perspective of application scenario• Most of the methods selected a deep learning framework.

• Different methods have different focuses.

From the perspective of application scenario• Most of the methods selected a deep learning framework.

• Different methods have different focuses.
• The existing researches in each application scenario still have
many limitations.

From the Perspective of Learning Pipeline

Method

MethodStrategy

Focus

CMC-1

CMC-5CMC-10

LR MLR-VIPeR

JUDEA

Multi-scale Metrics

ML

26.0

55.1

69.2

SLD2L

Dictionary Learning

RL

20.3

44.0

62.0

SDF

Resolution-Distance Variation

RL

9.3

38.1

52.4

SING

Super Resolution

MU

33.5

57.0

66.6

CSR-GAN

Cascaded SR and ReID

MU

37.2

62.3

71.6

FFSR+RIFE

Foreground Focus SR

MU

41.6

64.9

--

CAD

Adversarial Learning

MU

43.1

68.2

77.5

INTACTInter-task Association

MU

46.2

73.1

81.6

Zero-padding

One-stream and Zero-padding

RL

14.80

--

54.12

71.33

15.95

HCML

Feature & Metric Learning

ML

14.32

--

53.16

69.1716.16

BCTR

End-to-End

RL

17.01

--

55.43

71.96

19.66

cmGAN

Adversarial Learning

RL

26.97

--

67.51

80.56

27.80

D2RL

Dual-level Reduction

MU

28.90

--

70.60

82.40

29.20

XIV

X Modality

RL

49.92

--

89.79

95.96

50.73

Hi-CMD

Disentanglement

MU

34.94

--

77.58

--

35.94

cm-SSFT

Affinity Modeling

RL

61.6

--

89.2

93.9

63.2

GNA-RNN

Affinity Learning

ML

19.05

--

53.64

CNN-LSTM

Two-Stage Matching

ML

25.94

--

60.48

CMPM+CMPC

Cross-modal Projection

RL

49.37

--

79.27

GDA+LRA

Local and Global Association

RL

43.58

66.93

76.26CDAFL

Adversarial Learning

RL

34.0

56.3

72.5

IR SYSU-MM01

Text CUHK-PEDES

Sketch PKU-Sketch

CMC-20mAP

Multi-task Learning
Modality Unification

84.7

Adversarial Learning

Focus on Person Details

ML: Metric Learning
RL: Representation Learning
MU: Modality Unification

Outline• General Person Re-identification• Person Re-identification New Trends• Cross-modality• Long-term• Group• Discussion

Challenge1-Illumination[29] Z Zeng, Z Wang, Z Wang*, Y Zheng, YY Chuang, S Satoh, Illumination-Adaptive Person Re-identification, IEEE T-MM, 2020

Illumination AdaptiveIllumination AdaptiveChallenge2-Cloth Changing
The same identity with different clothes

PRCC Dataset
LTCC Dataset•••

33698 images from 221 identities
each person in Cameras A and B is wearing the same clothes
for Camera C, the person wears different clothes

https://www.isee-ai.cn/~yangqize/clothing.html••

one cloth-change set where 91 persons appearing with 416
different sets of outfits in 14,783 images
one cloth-consistent subset containing the remaining 61
identities with 2,336 images without outfit changes

https://naiq.github.io/LTCC_Perosn_ReID.html[30] Qize Yang, Ancong Wu, Wei-Shi Zheng. “Person Re-identification by Contour Sketch under Moderate Clothing Change.” IEEE TPMI, 2020[31] Xuelin Qian, Wenxuan Wang, et al., Long-Term Cloth-Changing Person Re-identification, ECCV, 2020

Outline• General Person Re-identification• Person Re-identification New Trends• Cross-modality• Long-term• Group• Discussion

Group: Challenge[32] Group Re-Identification: Leveraging and Integrating Multi-Grain Information, ACM MM, 2018

DoT-GNN[33] Z Huang, Z Wang*, W Hu, CW Lin, S Satoh, DoT-GNN: Domain-Transferred Graph Neural Network for Group Re-identification, ACM MM, 2019

DoT-GNN

DoT-GNN: Domain-Transferred Graph Neural Network

DoT-GNN

Outline• General Person Re-identification• Person Re-identification New Trends• Cross-modality• Long-term• Group• Discussion

Conclusion and Future Directions• Dataset Construction [5][5] Zeng, et al., Illumination-Adaptive Person Re-identification, TMM, 2020[34] Yang,, et al., Mining on heterogeneous manifolds for zeroshot cross-modal image retrieval. AAAI, 2020.
[35] Mirjalili , et al., Soft biometric privacy: Retaining biometric utility of face images while perturbing gender, IJCB, 2017.

Conclusion and Future Directions• Dataset Construction [5]• Taking Advantages of general ReID Datasets and Methods [34][5] Zeng, et al., Illumination-Adaptive Person Re-identification, TMM, 2020[34] Yang,, et al., Mining on heterogeneous manifolds for zeroshot cross-modal image retrieval. AAAI, 2020.
[35] Mirjalili , et al., Soft biometric privacy: Retaining biometric utility of face images while perturbing gender, IJCB, 2017.

Conclusion and Future Directions• Dataset Construction [5]• Taking Advantages of general ReID Datasets and Methods [34]• Human Interaction and Crowd-sourcing[5] Zeng, et al., Illumination-Adaptive Person Re-identification, TMM, 2020[34] Yang,, et al., Mining on heterogeneous manifolds for zeroshot cross-modal image retrieval. AAAI, 2020.
[35] Mirjalili , et al., Soft biometric privacy: Retaining biometric utility of face images while perturbing gender, IJCB, 2017.

Conclusion and Future Directions• Dataset Construction [5]• Taking Advantages of general ReID Datasets and Methods [34]• Human Interaction and Crowd-sourcing• Investigation on Unifying the Modality[5] Zeng, et al., Illumination-Adaptive Person Re-identification, TMM, 2020[34] Yang,, et al., Mining on heterogeneous manifolds for zeroshot cross-modal image retrieval. AAAI, 2020.
[35] Mirjalili , et al., Soft biometric privacy: Retaining biometric utility of face images while perturbing gender, IJCB, 2017.

Conclusion and Future Directions• Dataset Construction [5]• Taking Advantages of general ReID Datasets and Methods [34]• Human Interaction and Crowd-sourcing• Investigation on Unifying the Modality• Integrating Multiple cross-modality ReID Application Scenarios[5] Zeng, et al., Illumination-Adaptive Person Re-identification, TMM, 2020[34] Yang,, et al., Mining on heterogeneous manifolds for zeroshot cross-modal image retrieval. AAAI, 2020.
[35] Mirjalili , et al., Soft biometric privacy: Retaining biometric utility of face images while perturbing gender, IJCB, 2017.

Conclusion and Future Directions• Dataset Construction [5]• Taking Advantages of general ReID Datasets and Methods [34]• Human Interaction and Crowd-sourcing• Investigation on Unifying the Modality• Integrating Multiple cross-modality ReID Application Scenarios• Considering the Privacy Issue [35]Conclusion and Future Directions• Dataset Construction [5]• Taking Advantages of general ReID Datasets and Methods [34]• Human Interaction and Crowd-sourcing• Investigation on Unifying the Modality• Integrating Multiple cross-modality ReID Application Scenarios• Considering the Privacy Issue [35]• Common Model for General ReID and New TrendsCollaborators

Shin’ichi
NII/UTokyo

WenjunMicrosoft

Chia-wen
NTHU

Zhixiang
NTUFanUTokyo

ZelongUtokyoMang
IIAIZiling
NTHU

Yinqiang
NII

Kajal
IIIT Delhi

Yang
KyotoUThanks!