从⾏⼈重识别到⽆⼈机定位
郑哲东
http://zdzheng.xyz

About MePresent-Final-year PhD student
Advised by Prof. Yi Yang and Dr. Liang Zheng
Published 4 first-author conference papers and 5 first-author journal papers
GoogleScholar 3000+ citations; Github 5000+ stars
First-place winner in AICity Challenge CVPR 2020
Looking for the post-doc position

Research Interests
Image Retrieval, Person Re-identification, Image-text Understanding
Image Generation, Domain Adaptation, Adversarial Samples
Zhedong.Zheng@student.uts.edu.au2

（考古）2017年 19 期-》
2020年 71 期⽐较早的开源2016 Matlab https://github.com/layumi/2016_person_re-ID
2017Python https://github.com/layumi/Person_reID_baseline_pytorch

Outlines
1. ⾏⼈重识别的⼀些实践 (⾏⼈动，camera不动)
2. ⻋辆重识别 CVPR2020 智慧城市⽐赛冠军 (⻋动，camera不动)
3. ⽆⼈机与重识别的机遇与挑战 ACM Multimedia2020 (camera动，建筑不动)Outlines
1. ⾏⼈重识别的⼀些实践 (⾏⼈动，camera不动)
2. ⻋辆重识别 CVPR2020 智慧城市⽐赛冠军
3. ⽆⼈机与重识别的机遇与挑战 ACM Multimedia2020

Background• Person re-identification (re-id) is to find the person of interest from different
camera views.
• The emergence of this task can be attributed to 1) the increasing demand ofpublic safety and 2) the widespread large camera networks in public space.

Person Detection

Cam 2, 3, 4, 5, 6, 7Background• Person re-identification is challenging in the learning one discriminative androbust visual representation against the viewpoint changes.

All people love yellow shirt
and short pants?

Evolution in State-of-the-art Performance

2016年
39.6%

Gated Siamese

2017年
62.1% - 66.1%

SVDNetGAN

2018年
74.3% - 81.6%

MLFN

PCB

2019年
85.1% - 86.0%

Auto-reID DG-Net

Market-1501, ResNet-50, mAP accuracy without re-ranking
1. R. R. Varior, M. Haloi, and G. Wang. Gated Siamese convolutional neural network architecture for human reidentification. In ECCV, 2016.
2. Z. Zheng, L. Zheng, and Y. Yang. Unlabeled samples generated by gan improve the person re-identification baseline in vitro. In ICCV, 2017.
3. Y. Sun, L. Zheng, W. Deng, and S.Wang. SVDNet for pedestrian retrieval. In ICCV, 2017.
4. Suh et al., Part-Aligned Bilinear Representations for Person Re-identification, In ECCV 2018.
5. Sun et al., Beyond Part Models: Person Retrieval with Refined Part Pooling (and a Strong Convolutional Baseline). In ECCV 2018
6. Quan et al., Auto-ReID: Searching for a Part-Aware ConvNet for Person Re-Identification. In ICCV 2019.
7. Zheng et al. Joint discriminative and generative learning for person re-identification. In CVPR 2019

Datasets• Image-based Datasets:••••

Market-1501 (ICCV 2015)
CUHK03 (CVPR 2014)
DukeMTMC-reID (ICCV 2017)
DG-Market (CVPR 2019)• MSMT-17 （CVPR 2019）• …

• Tracklet-based Datasets:••••

iLIDS (BMVC 2009)MARS (ECCV 2016)
DukeMTMC-video (CVPR 2018)…

• Scene-based Datasets:• PRW (CVPR 2017)• CUHK-SYSU (CVPR 2017)• …

https://github.com/NEU-Gou/awesome-reid-dataset

Large-scale Datasets are needed.

One Million Training
Images for ImageNet

Dataset

CUHK01

VIPeR

PRID

CAVIAR

MarketDukeMTMC

MSMT-17

BBoxes

3,884

1,264

1,134

610

32,66836,411

126,441

Identities

971

632

934

72

1,501

1,812

4,104

Cameras

102

2

2

6

815

Detector

hand

hand

hand

hand

DPM

hand

FasterRCNN

Scene

indooroutdooroutdoorindooroutdooroutdoorindoor/outdoor• Due to the annotation cost and privacy concerns, large-scale datasets are not easy to obtain.
• Although recent re-id datasets contain more images, they are still far from the real-world application.

Current Challenges

Potential Solutions

1. Limited Training Data 数据

1. Synthetic Pedestrian Images
2. Parts/Losses
3. Auto-ML / Pruning
4. Domain Adaptation
5. Alignment / 3D Model

2. Effectiveness 性能
3. Efficiency 效率
4. Domain Gap 实⽤
5. Unconstrained Environment(e.g., Occlusion) 实⽤12

Limited Training Data四两拨千⽄3D Game Engine❑ 最直接的⽅法，我们可以通过3D游戏引擎来⽣成不同视⻆下的⼈体数据。但是，缺点是
和实际的数据分布有⼀些差异。相⽐实际⾮限制场景来说，相对简单。
SOMAset and SOMAnet(Barbosa et al., CVIU 2018)

PersonX (Sun et al., CVPR 2019)

Sun X, Zheng L. Dissecting person re-identification from the viewpoint of viewpoint In CVPR 2019.

Generative Adversarial Network (GAN)❑ 我们也可以使⽤GAN去模拟训练数据分布。借助GAN⽣成图像，我们缓解了数据集数据有限的问题。让模型看到更多样本，提升⼀定的鲁棒性。我们从ICCV17到CVPR19的两个⼯作，改进了⽣成质量，让模型可以端到端的⽣成和学习reID特征。

DGNet (Zheng et al., CVPR 2019)

LSRO (Zheng et al., ICCV 2017)

Zheng, Z., Yang, X., Yu, Z., Zheng, L., Yang, Y., & Kautz, J. (2019). Joint discriminative and generative learning for person re-identification. In CVPR 2019.

Generative Adversarial Network (GAN)❑ 通过⻛格迁移，GAN也被⼴泛⽤于不同⻛格（光照）的⾏⼈图像⽣成。模型基于这些数
据训练的，可以更容易迁移到不同应⽤场景。
SPGAN (Deng et al., CVPR 2018)

PNGAN (Qian et al., ECCV 2018)
FDGAN (Ge et al., NeurIPS 2018)

PTGAN (Wei et al., CVPR 2018)

16

EffectivenessPart Alignment in the Pixel Level❑ 基于局部部件特征学习，最直接的⽅式就是在图像的像素层⾯先进⾏基于部件的像素对⻬。
Pose-driven CNN
Dense semantic alignment(Su et al., CVPR2017)(Zhang et al., CVPR2019)

Pedestrian alignment network(Zheng et al., TCSVT2018)

Part Alignment in the Feature Level❑基于局部部件特征学习，以及特征对⻬，我们在ECCV上达到了当时最佳重识别效果，形成了解释性更强
特征表示的⽅法。

PCB: Part-based Convolutional Baseline (Sun et al., ECCV2018)
Horizontal Pyramid Features(Yang et al., AAAI 2019)

Stripe partition
Simple and effective
93.8% R-1 on Market-1501

Multiple Granularities(Wang et al., ACM MM2019)

Yifan Sun, Liang Zheng, Yi Yang, Qi Tian, Shengjin Wang. Beyond Part Models: Person Retrieval with Refined Part Pooling (and A Strong Convolutional Baseline). In ECCV, 2018.

Discriminative Losses❑ 除了基于局部，也可采⽤不同的损失函数，从⾼维冗余特征中甄别⾏⼈身份，形成了鉴别
能⼒更强特征表示。
Verification + Identification(Zheng et al., TOMM 2017)

In Defence of Triplet(Alexndar et al., arXiv)

Circle Loss(Sun et al., CVPR 2020)

OIM Loss(Xiao et al., CVPR 2017)
Sphere ReID
(Fan et al., arXiv)
Quadruplet Loss
(Chen et al., CVPR 2017)
Zhedong Zheng, Liang Zheng, and Yi Yang. A discriminatively learned CNN embedding for person reidentification. ACM TOMM, 14(1):13, 2017.

20

Efficiency

Model Pruning for Image Retrieval❑ 最直接的想法就是模型剪枝。今年我们提出了利⽤预训练⽹络的通道局部相关性的通道衰减“软”剪
枝模型。针对最常⽤的ResNet-50主⽹络，可在减少88.85% FLOPs情况下，超越最新软剪枝⽅法8%mAP，原模型⼤⼩由27.1M，压缩为 6.97M。

传统⽅法
传统⽅法⼀般按照模⻓来删除filter我们⽽我们根据局部聚类的冗余剪枝，可有效保持原始⽹络通道分布

Wang, X., Zheng, Z., He, Y., Yan, F., Zeng, Z., & Yang, Y. (2020). Progressive local filter pruning for image retrieval acceleration. in arXiv.

Auto-ML to Learn a Network for Re-id❑ 使⽤⾃动机器学习⽅法⾃动化地设计针对⾏⼈再识别问题的神经⽹络结构，最终设计出⽐经典模型ResNet参数量少⼀半，运⾏开销也减少了53%，性能也达到最领先⽔平的⾏⼈再识别神经⽹
Auto Re-id
络。

(Quan et al., ICCV 2019)ReID Search Space:part-aware module•
3×3 max pooling•
3×3 average pooling••••

3×3 depth-wise separable convolution
3×3 dilated convolution
zero operation, and identity mapping

⾃动化设计⾏⼈再识别神经⽹络结构的可视化。如此复杂的结构很难被⼈为设计，其中设计的⼈
体部位感知元件(part_aware)被⾃动化地安置在⽹络结构中。

Quan, R., Dong, X., Wu, Y., Zhu, L., & Yang, Y. (2019). Auto-ReID: Searching for a Part-Aware ConvNet for Person Re-Identification. In ICCV 2019.

23

Domain Gap

24

Domain Gap❑实际场景中，往往由于摄像头⻆度、穿着、光照以及天⽓的不同，导致模型的准度下降，考验模型的泛化能⼒。

• 通常情况下，将⼀个已经训练好的模型直接应⽤到新场景、新数据集上，识别的准确率普遍较低。
• 右图展示了不同数据集⾥
⾏⼈穿着、背景以及光照的差异。这些差异⼤⼤影响了模型的泛化能⼒。
• 如若对每⼀新场景都重新进⾏⼈⼯标记数据、训练模型，这将花费⼤量⼈⼒财⼒。

25

Domain Adaptation❑ 对新场景的数据不进⾏⼈⼯标注，⽽是通过聚类的⽅式赋予新数据伪标签，最后对已有模型进⾏
fine-tune。

PUL(Fan et al., TOMM 2018)

SSG(Fu et al., ICCV 2019)

Adaptive Exploration(Ding et al., TOMM 2020)

26

Domain Adaptation❑ 利⽤先验知识和soft label 在⽬标域进⼀步挖掘信息。

HHL (Zhong et al., CVPR 2018)

Tracklet Association(Li et al., ECCV 2018)➢ 同构学习：不同相机⻛格的同⼀样本组成正样本，学习相机不变性➢ 异构学习：不同域的样本组成负样本对，学习域在特征空间中的潜在关系，提升域连通性Generalizing A Person Retrieval Model Hetero- and Homogeneously. ECCV 2018

27

Domain Adaptation❑ 我们在更通⽤的分类和语义分割问题上也取得了⼀些进展。
CLAN (Luo et al., CVPR 2019)
Deep Adversarial Attention Alignment(Kang et al., ECCV 2018)

CAN (Kang et al., CVPR 2019)

Memory Regularization (Zheng et al., IJCAI 2020)

28

Unconstrained Environment

29

Occlusion❑实际场景中，遮挡也常常⽆法避免，导致模型的准度下降，考验模型的泛化能⼒。

Leverage Person Structure（1）⾏⼈重识别中的遮挡问题。解决思路：使⽤⾏⼈关键点检测，区分⾏⼈图⽚的遮挡部分和⾮遮挡部分。
提取⾮遮挡部分的表征进⾏相似性匹配，忽略遮挡部分表征。
（2）构建⼤型遮挡⾏⼈识别数据库， Occluded-DukeMTMC。
训练：使⽤关键点⽣成的注意⼒图构建全局表征；类似PCB，构建局部部件表征。

测试：使⽤关键点区分遮挡与⾮遮挡局部表征，使⽤⾮遮挡表征进⾏相似性匹配，忽略遮挡部分。

Miao J, Wu Y, Liu P, Ding Y, Yang Y . Pose-Guided Feature Alignment for Occluded Person Re-Identification. ICCV, 2019.

Person Re-identification in the 3D Space❑ ⼈们⽣活在3D 的世界中。我们在2020年新的⼯作中，将⼈体映射到三维空间，然后再通过点云，融⼊⼏何结构(geometry structure)来学习⼈体表达，得到鲁棒特征，处理遮挡问题。同时，OG-Net只有9MB，推理速度也⽐常⽤的ResNet-50快。

OG-Net(Zheng et al., arXiv 2020)

Zhedong Zheng, Yi Yang . Person Re-identification in the 3D Space. ArXiv, 2020.

Multi-modal Inputs❑ 最后，多模态的输⼊也可以提⾼模型在⾮限制场景下的鲁棒性，让模型更接近实⽤。⽐如通过语⾔描述来搜⼈。

CUHK-PEDES (Li et al., ICCV 2017)

Improving reid via Pedestrian Attribute(Lin et al., Pattern Recognition 2019)

Instance Loss (Zheng et al., TOMM 2020)

TJ-AIDL(Wang et al., CVPR 2018)

Future Research for Person Re-id
1. Millions of Distractor Images (Is the model robust? )2. Efficient Training from Millions of Data (Which data is important?
Long-tail )3. Fast Domain Adaptation / Online Learning (Update model itself)
4. Unconstrained Environment (e.g., Occlusion and Illumination)

Outlines
1. ⾏⼈重识别的⼀些实践
2. ⻋辆重识别 CVPR2020 智慧城市⽐赛冠军 (⻋动，camera不动)
3. ⽆⼈机与重识别的机遇与挑战 ACM Multimedia2020

Going Beyond Real Data:
A Robust Visual Representation
for Vehicle Re-identification
AI City Challenge 2020
Zhedong Zheng1,2*, Minyue Jiang1*, Zhigang Wang1,Jian Wang1, Zechen Bai1, Xuanmeng Zhang1,3,Xin Yu2, Xiao Tan1, Yi Yang2 , Shilei Wen1 , Errui Ding1
1 Baidu Inc.
2 University of Technology Sydney
3 Zhejiang University

What is Vehicle Re-identification?
TrackingRe-identification

Main Challenge?

What should we care about?

What should we care about?
• Limited Training Data.
We explore three different data generation approaches.
• Representation Learning.
We adopt two widely-used baseline.
• How to use Meta Data/ Attributes?
Meta-data also plays an important role in post processing.

What should we care about?
• More Training Data
We explore three different data generation approaches.
• A Strong Baseline
We adopt two widely-used baseline.
• Post-processing Methods
Meta-data also plays an important role.

Style Transfer
Real Data

Synthetic Data

Style Transfer

Content Manipulation

Copy & Paste

What should we care about?
• More Training Data
We explore three different data generation approaches.
• A Strong Baseline
We adopt two widely-used baseline.
• Post-processing Methods
Meta-data also plays an important role.

Cross-entropy Loss Definition

Ranking Loss Definition

Re-ID Baseline

New Classifier Module- SVDNet for Pedestrian Retrieval- In Defense of the Triplet Loss for Person Re-Identification

Re-ID BaselineWhat should we care about?
• More Training Data
We explore three different data generation approaches.
• A Strong Baseline
We adopt two widely-used baseline.
• Post-processing Methods
Meta-data also plays an important role.

Camera Verification

Camera Verification

Experiment

AICity Competition
We achieve the 1st place in vehicle reID track of AICity Challenge, CVPR 2020.

•mAP Accuracy ⾼出
第⼆名 6%

Code is available at https://github.com/layumi/AICIty-reID-2020

Ablation Study: With/without Synthetic Data

Ablation Study: Post-processing Methods

ExplainableFuture Works for Vehicle Re-idPossible Approaches
1) Investigate the feasibility of high-fidelity generated samples fortraining. The generated samples could largely enrich the training set.
2) Mixture of Unsupervised Learning/ Semi-supervised Learning
3) Domain Adaptation

One last commentNeural Networks are lazyThe models could easily overfit the datasets. Sometimes adding prior knowledge
and data augmentation are important to obtain a robust re-id system.
Training Neural Networks sometime is tricky, and models will find the short way to
overfit the objective. Therefore, generated data with proper pseudo labels
helps.

The code is available atOutlines
1. ⾏⼈重识别的⼀些实践
2. ⻋辆重识别 CVPR2020 智慧城市⽐赛冠军
3. ⽆⼈机与重识别的机遇与挑战 ACM Multimedia2020 (camera动，建筑不动)Drone is coming!
University-1652: A Multi-view Multi-source Benchmark
for Drone-based Geo-localization

Zhedong Zheng, Yunchao Wei, Yi Yang
University of Technology Sydney

It is cheaper.

Use Cases: What can Drones do? Why we study?
Drone is a new aerial platform.
• Accurate Delivery (e.g., send mask)• Agriculture (e.g., pesticide)• Event Detection (e.g. traﬃc jam)• ….

• Task (Visual Gap)
• Dataset• Baseline & ExperimentUniversity-1652• We consider one conventional task: cross-view Geo-localization.
Ground-view Images

Limited RoofGap

Satellite-view Images (GPS tag)

Whole RoofWe notice that the drone can be a bridge.

Ground-view

Drone-view

No dataset to verify it.

Satellite-view (GPS tag)• Task• Dataset (Missing in existing works)• Baseline & ExperimentUniversity-1652• We collect the data from three platforms of 1652 buildings.
• More training images per class (instead of image pairs).
• More viewpoints -> More intra-class variants• Me: I want to build one dataset.
• Supervisor: No! Too much cost.
• Me: We use free data from Internet.
• Supervisor: Do it!

Building names from Wikipedia

Get latitude/longitude from GoogleMap

1. Drone-view Data
• Due to the privacy concerns and the cost, we deploy the simulated data via
Google Earth. We write scripts to drive the engine as drone camera.

2. Ground-view Data from GoogleMap

3. Satellite-view Data from GoogleMap

4. Noisy Ground-view Data from GoogleImage• We search the building

name and download
images from GoogleImage• We then remove theindoor images andduplicate images.

Krause J, Sapp B, Howard A, et al. The unreasonable eﬀectiveness of noisy data for fine-grained
recognition[C]//European Conference on Computer Vision. Springer, Cham, 2016: 301-320.

• Task• Dataset (Now we have data.)• Baseline & ExperimentBaseline
Flexible and Strong Baseline- Objective: Instance Loss (Share Classifier)
Structure: Generally, the backbone- network do not share low-level patterns

New data ->add one branch!
Zheng Z, Zheng L, Garrett M, et al. Dual-Path Convolutional Image-Text Embeddings with Instance Loss[J]. ACM Transactions on Multimedia Computing,Communications, and Applications (TOMM), 2020, 16(2): 1-23.

Baseline
CVUSA

ICCV 2015
CVPR 2017
ECCV 2016
CVPR 2018
CVPR 2019

Oxford and Paris

Ground-view query vs. drone-view query.

Apply the model trained on University-1652 to real drone videos.

The model haven’t seen any data of UIUC.

Apply the model trained on University-1652 to real drone videos.

The model haven’t seen any data of UIUC.

Scalability

Ablation Studies
Whether Share Weights

Diﬀerent Loss Functions

Diﬀerent Input Sizes

Future Works - Boost Performance
We run a leaderboard.
You are welcomed to push the state-of-the-art performance.

Future Works - LBP

Local binary patterns (LBP) descriptor

LPN

Tingyu W, Zhedong Z, Chenggang Y, and Yi, Y. Each Part Matters: Local Patterns Facilitate Cross-view Geo-localization. arXiv 2020.

Future Works - Keypoint Matching
SIFT does not work very well. Deeply-learned Methods are needed.

Thanks a lot!
University-1652: A Multi-view Multi-source Benchmark
for Drone-based Geo-localization

Zhedong Zheng, Yunchao Wei, Yi Yang
University of Technology Sydney

Dataset & CodeHave been downloaded
By 300+ times.

Data License• We carefully check the data license from Google. There are two main points.
• First, the data of Google Map and Google Earth could be used based on fairusage. We follow the guideline on this oﬃcial website 3 .

• Second, several existing datasets have utilized the Google data. In practice,

we adopt a similar policy of existing datasets 4, 5 to release the dataset basedon the academic request.

3. https://www.google.com/permissions/geoguidelines/
4. http://www.ok.ctrl.titech.ac.jp/~torii/project/247/
5. http://mvrl.cs.uky.edu/datasets/cvusa/