ACM MM 2020 Tutorial on “Effective and Efficient: Toward Open-world Instance Re-identification”

Vehicle Re-identification:
Past, Present and Future
Wu Liuliuwu1@jd.com
Computer Vision and Multimedia Lab
JD AI Research1What is Vehicle Re-Identification?

VehicleRe-Identification

Probe vehicle
image

Quickly discover, locate, and track the target vehicles in large
scale urban surveillance videos!
2

2020-10-02

Pervasive Applications

Crime Investigation

Park Management

Intelligent Transportation

Urban Computing
3

2020-10-02

Challenges
Vehicle variety• Large number of types & models• Trivial inter-class differences• Large intra-class differences

Environmental factors• Various viewpoints• Uncertain illuminations• Clutter and occlusion
42020-10-02

1Datasets

23Methods

Discussion
5

1Datasets

23

Methods

Discussion
6

Datasets• VeRi Dataset [Liu, ICME16&ECCV16]

Dataset construction
vehicle recurrences in 20 cameras
50K+ Boxes, 9000+ tracks, 776 vehicles
Attributes

10 colors, 9 types and 30+ brands
Spatio-temporal + License PlateXinchen Liu, Wu Liu, Huadong Ma, Huiyuan Fu: Large-scale vehicle re-identification in urban
surveillance videos. ICME 2016: 1-6
Xinchen Liu, Wu Liu, Tao Mei, Huadong Ma: A Deep Learning-Based Approach to Progressive Vehicle
Re-identification for Urban Surveillance. ECCV (2) 2016: 869-884
72020-10-02

Datasets• PKU VehicleID [Yan, ICCV17]

Hongye Liu, Yonghong Tian, Yaowei Wang, Lu Pang, Tiejun Huang: Deep Relative Distance Learning: Tell the Difference between
Similar Vehicles. CVPR 2016: 2167-2175
Ke Yan, Yonghong Tian, Yaowei Wang, Wei Zeng, Tiejun Huang: Exploiting Multi-grain Ranking Constraints for Precisely Searching
Visually-similar Vehicles. ICCV 2017: 562-570
82020-10-02

Datasets• Vehicle-1M [Guo, AAAI 2018]• Day and night, head or rear

• Multiple surveillance cameras
installed in several cities• 936,051 images from 55,527 vehicles
and 400 vehicle models• Labeled with attributes (make, modeland year)

Haiyun Guo, Chaoyang Zhao, Zhiwei Liu, Jinqiao Wang, Hanqing Lu: Learning Coarse-to-Fine Structured Feature
Embedding for Vehicle Re-Identification. AAAI 2018: 6853-6860
92020-10-02

Yihang Lou, Yan Bai, Jun Liu

Datasets• VERI-Wild [Lou, CVPR19]

Yihang Lou, Yan Bai, Jun Liu, Shiqi Wang, Lingyu Duan: VERI-Wild: A Large Dataset and a New Method for Vehicle Re10
Identification in the Wild. CVPR 2019: 3235-3243
2020-10-02

Datasets• CityFlow [Tang, CVPR19]
CamerasBoxes

40229,680

Boxes/ID Identities
344.9666

Challenge Track 1: City-Scale Multi-Camera Vehicle Tracking
Challenge Track 2: City-Scale Multi-Camera Vehicle Re-Identification
Challenge Track 3: Traffic Anomaly DetectionZheng Tang, Milind Naphade, Ming-Yu Liu, Xiaodong Yang, Stan Birchfield, Shuo Wang, Ratnesh Kumar, David C. Anastasiu, Jenq-Neng Hwang:
CityFlow: A City-Scale Benchmark for Multi-Target Multi-Camera Vehicle Tracking and Re-Identification. CVPR 2019: 8797-8806
11
2020-10-02

Datasets
VideoGeom.

Multiview

LicensePlate776

X

√√

√

6.0

141,756

X

X

X

X

936,051

16.8

55,527

X

X

X

X

174

406,3141040,671

X

√

√

X

40

229,680

344.9666√√√

X

BenchmarkYearCameras

Boxes

Boxes/ID IdentitiesVeRi-776

2016

20

49,357

63.6

PKU-VD1&2

2017-846,358

Vehicle-1M

2018-VeRi-Wild

2019

CityFlow2019

12
2020-10-02

Datasets• MVP Dataset [Liu, ACM MM 20]

2020-10-02

Xinchen Liu, Wu Liu, Jinkai Zheng, Chenggang Yan, Tao Mei13: Beyond the Parts: Learning Multi-view Cross-part Correlation for Vehicle. Reidentification.. ACM MM 2020

1Datasets

2
3

Methods

Discussion
14

Methods[Bai, TMM2018]

1. Attribute Recognition + Metric Learning

3. GAN-based

2. Detail-aware Attentive Multi-view Inference15

4. Multi-modality data-based methods

Methods
1. Attribute Recognition + Metric Learning• Xinchen Liu, Wu Liu, Huadong Ma, Huiyuan Fu, "Large-Scale Vehicle Re-Identification in Urban
Surveillance Videos", IEEE ICME, 2016, pp.1-6• Hongye Liu, Yonghong Tian, Yaowei Wang, Lu Pang, Tiejun Huang: Deep Relative Distance
Learning: Tell the Difference between Similar Vehicles. CVPR 2016: 2167-2175• Ke Yan, Yonghong Tian, Yaowei Wang, Wei Zeng, Tiejun Huang: Exploiting Multi-grain Ranking
Constraints for Precisely Searching Visually-similar Vehicles. ICCV 2017: 562-570• Haiyun Guo, Chaoyang Zhao, Zhiwei Liu, Jinqiao Wang, Hanqing Lu: Learning Coarse-to-Fine
Structured Feature Embedding for Vehicle Re-Identification. AAAI 2018: 6853-6860• Yan Bai, Yihang Lou, Feng Gao, Shiqi Wang, Yuwei Wu, Ling-Yu Duan: Group-Sensitive Triplet
Embedding for Vehicle Reidentification. IEEE Trans. Multimedia 20(9): 2385-2399 (2018)• Jianqing Zhu, Huanqiang Zeng, Jingchang Huang, Shengcai Liao, Zhen Lei, Canhui Cai, Lixin
Zheng: Vehicle Re-identification Using Quadruple Directional Deep Learning Features. IEEE TITS
(2020)[Bai, TMM2018]16
2020-10-02

Attribute Recognition + Metric Learning•Hongye Liu, Yonghong Tian, Yaowei Wang, Lu Pang, Tiejun Huang: Deep Relative Distance Learning: Tell
the Difference between Similar Vehicles. CVPR 2016: 2167-2175

Vehicle Model

Vehicle ID

Multi-task learning:

Attribute Recognition17

2020-10-02

Metric Learning
Dataset: VehicleIDAttribute Recognition + Metric Learning•Ke Yan, Yonghong Tian, Yaowei Wang, Wei Zeng, Tiejun Huang: Exploiting Multi-grain Ranking Constraints for
Precisely Searching Visually-similar Vehicles. ICCV 2017: 562-570

Pairwise Ranking

List Ranking
Vehicle ID, model and color

18

Dataset: PKU-VD1&2

Attribute Recognition + Metric Learning•Haiyun Guo, Chaoyang Zhao, Zhiwei Liu, Jinqiao Wang, Hanqing Lu: Learning Coarse-to-Fine Structured Feature
Embedding for Vehicle Re-Identification. AAAI 2018: 6853-6860

Dataset: Vehicle-1M

Vehicle ModelClassification loss term

Vehicle Model
Coarse-grained ranking loss term

models19Vehicle ID
Fine-grained ranking loss term

IDVehicle ID
Pairwise loss term

Attribute Recognition + Metric Learning•Feng Zheng, Xin Miao, Heng Huang: Fast Vehicle Identification in Surveillance via Ranked Semantic Sampling
Based Embedding. IJCAI 2018: 3697-3703

20

Attribute Recognition + Metric Learning•

Yan Bai, Yihang Lou, Feng Gao, Shiqi Wang, Yuwei Wu, Ling-Yu Duan: Group-Sensitive Triplet Embedding for Vehicle
Reidentification. IEEE Trans. Multimedia 20(9): 2385-2399 (2018)21

Methods
2. Detail-aware Attentive Multi-view Inference• Zhongdao Wang, Luming Tang, Xihui Liu, Zhuliang Yao, Shuai Yi, Jing Shao, Junjie Yan, Shengjin Wang, Hongsheng Li,Xiaogang Wang: Orientation Invariant Feature Embedding and Spatial Temporal Regularization for Vehicle Reidentification. ICCV 2017: 379-387• Zhou, Yi, Li Liu, and Ling Shao. "Vehicle re-identification by deep hidden multi-view inference." IEEE TIP 27.7 (2018):
3275-3287.
• Bing He, Jia Li, Yifan Zhao, Yonghong Tian: Part-Regularized Near-Duplicate Vehicle Re-Identification. CVPR 2019:3997-4005• Pirazh Khorramshahi, Amit Kumar, Neehar Peri, Sai Saketh Rambhatla, Jun-Cheng Chen, Rama Chellappa: A Dual-PathModel With Adaptive Attention for Vehicle Re-Identification. ICCV 2019: 6131-6140• Zheng Tang, Milind Naphade, Stan Birchfield, Jonathan Tremblay, William Hodge, Ratnesh Kumar, Shuo Wang,
Xiaodong Yang: PAMTRI: Pose-Aware Multi-Task Learning for Vehicle Re-Identification Using Highly Randomized
Synthetic Data. ICCV 2019: 211-220[CVPR 2019]• Ruihang Chu, Yifan Sun, Yadong Li, Zheng Liu, Chi Zhang, Yichen Wei: Vehicle Re-Identification With Viewpoint-Aware
Metric Learning. ICCV 2019: 8281-8290• Xiaobin Liu, Shiliang Zhang, Xiaoyu Wang, Richang Hong, Qi Tian: Group-Group Loss-Based Global-Regional Feature
Learning for Vehicle Re-Identification. IEEE Trans. Image Process. 29: 2638-2652 (2020)• Dechao Meng, Liang Li, Xuejing Liu, Yadong Li, Shijie Yang, Zheng-Jun Zha, Xingyu Gao, Shuhui Wang, Qingming
Huang: Parsing-Based View-Aware Embedding Network for Vehicle Re-Identification. CVPR 2020: 7101-7110• Tsai-Shien Chen, Chih-Ting Liu, Chih-Wei Wu, Shao-Yi Chien: Orientation-aware Vehicle Re-identification withSemantics-guided Part Attention Network. ECCV 2020
• Pirazh Khorramshahi, Neehar Peri, Jun-Cheng Chen, Rama Chellappa: The Devil is in the Details: Self-Supervised
Attention for Vehicle Re-Identification. ECCV 2020
• Xinchen Liu, Wu Liu, Jinkai Zheng, Chenggang Yan, Tao Mei: Beyond the Parts: Learning Multi-view Cross-part
Correlation for Vehicle Re-identification, ACM MM 2020
22

[ACMMM 2020]

Multi-view Inference•Zhou, Yi, Li Liu, and Ling Shao. "Vehicle re-identification by deep hidden multi-view inference." IEEE TIP 27.7 (2018):
3275-3287.

23

Multi-view Inference•

Zhou, Yi, Li Liu, and Ling Shao. "Vehicle re-identification by deep hidden multi-view inference." IEEE TIP 27.7 (2018):
3275-3287.

24

Global and Local Paths• Bing He, Jia Li, Yifan Zhao, Yonghong Tian: Part-Regularized Near-Duplicate Vehicle Re-Identification. CVPR 2019: 3997-4005

front and back light;
front and back window;
vehicle brand.

This framework consist of two modules, a local module which focuses on the part features todistinguish the subtle discrepancy in visual features and a global module, which is regularized by thepart attentions in the local module.
25

Global and Local Paths•Pirazh Khorramshahi, Amit Kumar, Neehar Peri, Sai Saketh Rambhatla, Jun-Cheng Chen, Rama Chellappa: A Dual-PathModel With Adaptive Attention for Vehicle Re-Identification. ICCV 2019: 6131-6140

Orientation conditioned key-point selection and localized feature extraction modules is used in parallel
to supplement the features from the global path.
26

Global and Local Paths•Ruihang Chu, Yifan Sun, Yadong Li, Zheng Liu, Chi Zhang, Yichen Wei: Vehicle Re-Identification With Viewpoint-Aware
Metric Learning. ICCV 2019: 8281-8290

27

Keypoints-based•Zhongdao Wang, Luming Tang, Xihui Liu, Zhuliang Yao, Shuai Yi, Jing Shao, Junjie Yan, Shengjin Wang, Hongsheng Li, Xiaogang
Wang: Orientation Invariant Feature Embedding and Spatial Temporal Regularization for Vehicle Re-identification. ICCV 2017: 379-387

20 selected vehicle key points,
clustered into four set,
Each set shares the same visibility

28

Keypoints-based•Zheng Tang, Milind Naphade, Stan Birchfield, Jonathan Tremblay, William Hodge, Ratnesh Kumar, Shuo Wang,Xiaodong Yang: PAMTRI: Pose-Aware Multi-Task Learning for Vehicle Re-Identification Using Highly Randomized
Synthetic Data. ICCV 2019: 211-220

29

Keypoints-based•Peng Wang, Bingliang Jiao, Lu Yang, Yifei Yang, Shizhou Zhang, Wei Wei, Yanning Zhang: Vehicle Re-Identification in Aerial
Imagery: Dataset and Approach. ICCV 2019: 460-469

skylight, window, LOGO,
30

Parsing-based•Dechao Meng, Liang Li, Xuejing Liu, Yadong Li, Shijie Yang, Zheng-Jun Zha, Xingyu Gao, Shuhui Wang, Qingming Huang:Parsing-Based View-Aware Embedding Network for Vehicle Re-Identification. CVPR 2020: 7101-7110

31

Parsing-based•

Xinchen Liu, Wu Liu, Jinkai Zheng, Chenggang Yan, Tao Mei: Beyond the Parts: Learning Multi-view Cross-part
Correlation for Vehicle Re-identification, ACM MM 2020

Parsing-guided Cross-part Reasoning network (PCRNet)
32

Segmentation-based•

Tsai-Shien Chen, Chih-Ting Liu, Chih-Wei Wu, Shao-Yi Chien: Orientation-aware Vehicle Re-identification withSemantics-guided Part Attention Network. ECCV 202033

Methods
3. GAN• Yi Zhou, Ling Shao: Viewpoint-Aware Attentive Multi-View Inference for Vehicle Re-Identification.
CVPR 2018: 6489-6498• Yihang Lou, Yan Bai, Jun Liu, Shiqi Wang, Lingyu Duan: VERI-Wild: A Large Dataset and a New
Method for Vehicle Re-Identification in the Wild. CVPR 2019: 3235-3243• Yihang Lou, Yan Bai, Jun Liu, Shiqi Wang, Ling-Yu Duan: Embedding Adversarial Learning forVehicle Re-Identification. IEEE Trans. Image Process. 28(8): 3794-3807 (2019)• Pirazh Khorramshahi, Neehar Peri, Jun-Cheng Chen, Rama Chellappa: The Devil is in the Details:
Self-Supervised Attention for Vehicle Re-Identification. ECCV 2020

34

IEEE TIP (2019)

GAN•

Yi Zhou, Ling Shao: Viewpoint-Aware Attentive Multi-View Inference for Vehicle Re-Identification. CVPR 2018: 64896498

35

GAN
•
•

Yihang Lou, Yan Bai, Jun Liu, Shiqi Wang, Lingyu Duan: VERI-Wild: A Large Dataset and a New Method for Vehicle
Re-Identification in the Wild. CVPR 2019: 3235-3243
Yihang Lou, Yan Bai, Jun Liu, Shiqi Wang, Ling-Yu Duan: Embedding Adversarial Learning for Vehicle ReIdentification. IEEE Trans. Image Process. 28(8): 3794-3807 (2019)

36

GAN
•

Pirazh Khorramshahi, Neehar Peri, Jun-Cheng Chen, Rama Chellappa: The Devil is in the Details: Self-Supervised
Attention for Vehicle Re-Identification. ECCV 202037

Methods
Multimodal data-based methods
1. Spatial-temporal•Zhongdao Wang, Luming Tang, Xihui Liu, Zhuliang Yao, Shuai Yi, Jing Shao, Junjie Yan, Shengjin Wang, Hongsheng Li,Xiaogang Wang: Orientation Invariant Feature Embedding and Spatial Temporal Regularization for Vehicle Re-identification.
ICCV 2017: 379-387

2. License Plate + Spatial-temporal•Wu Liu, Xinchen Liu, Huadong Ma, Peng Cheng: Beyond Human-level License Plate Super-resolution with Progressive Vehicle
Search and Domain Priori GAN. ACM Multimedia 2017: 1618-1626•Xinchen Liu, Wu Liu, Tao Mei, Huadong Ma: PROVID: Progressive and Multimodal Vehicle Reidentification for Large-Scale
Urban Surveillance. IEEE Trans. Multimedia 20(3): 645-658 (2018)

382020-10-02

Spatial-temporal•

Zhongdao Wang, Luming Tang, Xihui Liu, Zhuliang Yao, Shuai Yi, Jing Shao, Junjie Yan, Shengjin Wang, Hongsheng
Li, Xiaogang Wang: Orientation Invariant Feature Embedding and Spatial Temporal Regularization for Vehicle Reidentification. ICCV 2017: 379-387

39

License Plate + Spatial-temporal•Xinchen Liu, Wu Liu, Tao Mei, Huadong Ma: PROVID: Progressive and Multimodal Vehicle Reidentification for LargeScale Urban Surveillance. IEEE Trans. Multimedia 20(3): 645-658 (2018)40

License Plate + Spatial-temporal
Wu Liu, Xinchen Liu, Huadong Ma, Peng Cheng: Beyond Human-level License Plate Super-resolution with
Progressive Vehicle Search and Domain Priori GAN. ACM Multimedia 2017: 1618-1626

Domain Priori Gan based Super-resolution
OutputGenerator

Deconvolution

Summation

BNConvolution

Convolution

Label

Residual Block...

Deconvolution

Data

Color,Texture,
Semantic
Attributes

ReLU

N Residual Blocks

ReLU

Appearance based Coarse
Filtering in Null Space

ReLU

Input

Search Results

BN

Progressive Vehicle Search

Convolution•License Plate based Fine Search

Discriminator×(c)(d)41

Sigmoid...

SpatialSplit

AvgPoolingBN

LeakyReLU

Convolution

LeakyReLU

Convolution

LabelCamera: 16Time:
16:17:30(b)

License Plate
Recognition

Data

Spatio-temporal Context based
Re-ranking(a)

M Convolutional Blocks

Output: 0
Fake image

APP855
Output: 1
Real image

(e)

1

Datasets

2
3

Methods

Discussion
42

Discussion
1. Cross-domain/Generalizable Vehicle Re-ID
2. Un-supervised/Self-supervised Vehicle Re-ID
3. Synthetic Data Generation for Vehicle Re-ID
4. Vehicle Re-ID with 3D models
5. Real-world Applications

43

Synthetic Data Generation for Vehicle Re-ID•Yue Yao, Liang Zheng, Xiaodong Yang, Milind Naphade, Tom Gedeon: Simulating Content Consistent Vehicle
Datasets with Attribute Descent. ECCV (2020)

44

3D Vehicle Analysis•

Hou-Ning Hu, Qi-Zhi Cai, Dequan Wang, Ji Lin, Min Sun, Philipp Krähenbühl, Trevor Darrell, Fisher Yu: Joint
Monocular 3D Vehicle Detection and Tracking. ICCV 2019: 5389-5398

45

Discussion
1. Cross-domain/Generalizable Vehicle Re-ID
2. Un-supervised/Self-supervised Vehicle Re-ID
3. Synthetic Data Generation for Vehicle Re-ID
4. Vehicle Re-ID with 3D models5. Real-world Applications

46

Real-world Applications
FastReID: A Pytorch Toolbox for General Instance Re-identification✓ Modular and extensible
design✓ Manageable system
configuration✓ Richer evaluation system✓ Richer evaluation system✓ State-of-the-art pre-trainedmodels•Lingxiao He, Xingyu Liao, Wu Liu, Xinchen Liu, Peng Cheng, Tao Mei: FastReID: A Pytorch Toolbox for General
47
Instance Re-identification. CoRR abs/2006.02631 (2020) https://github.com/JDAI-CV/fast-reid

Real-world Applications
FastReID: A Pytorch Toolbox for General Instance Re-identification

1300+Star•Lingxiao He, Xingyu Liao, Wu Liu, Xinchen Liu, Peng Cheng, Tao Mei: FastReID: A Pytorch Toolbox for General
48
Instance Re-identification. CoRR abs/2006.02631 (2020) https://github.com/JDAI-CV/fast-reid

Real-world Applications
Results on VeRi

Results on VehicleID

Results on VeRi-Wild•

Lingxiao He, Xingyu Liao, Wu Liu, Xinchen Liu, Peng Cheng, Tao Mei: FastReID: A Pytorch Toolbox for General
49
Instance Re-identification. CoRR abs/2006.02631 (2020) https://github.com/JDAI-CV/fast-reidFastReID

Thanks!
liuwu1@jd.comComputer Vision and Multimedia Lab
JD AI Research50