Effectiveness of Function Matchingin Driving Scene Recognition
2022/10/23

The 3rd AVVision Workshop

Shingo Yashima (Denso IT Laboratory, inc.)Learning Compact DNNs for Driving Scene Recognition
PurposeObtaining a lightweight and well-preforming recognition model
for autonomous driving (e.g., segmentation, detection)

Issue

The trained model sometimes does not reach the desired performanceCause

1. Not enough model capacity → Need to change the architecture

Effectiveness of Function Matching in Driving Scene Recognition / 2022/10/23 / Shingo Yashima© DENSO IT LABORATORY,INC. All Rights Reserved.

2 /11

Learning Compact DNNs for Driving Scene Recognition
PurposeObtaining a lightweight and well-preforming recognition model
for autonomous driving (e.g., segmentation, detection)

Issue

The trained model sometimes does not reach the desired performanceCause

1. Not enough model capacity → Need to change the architecture
2. Enough model capacity, but not optimized sufficiently

Compact models are often harder to generalize than large models,even though they may have the capacity to represent solutions of the large models
Effectiveness of Function Matching in Driving Scene Recognition / 2022/10/23 / Shingo Yashima© DENSO IT LABORATORY,INC. All Rights Reserved.

3 /11

Learning Compact DNNs for Driving Scene Recognition
PurposeObtaining a lightweight and well-preforming recognition modelfor autonomous driving (e.g., segmentation, detection)

Issue

The trained model sometimes does not reach the desired performanceCause

1. Not enough model capacity → Need to change the architecture
2. Enough model capacity, but not optimized sufficiently
Method to exploit full generalization power of compact models︖→ Knowledge Distillaion
Effectiveness of Function Matching in Driving Scene Recognition / 2022/10/23 / Shingo Yashima© DENSO IT LABORATORY,INC. All Rights Reserved.

4 /11

Knowledge Distillation as Function Matching
Recent study showed critical components of distillation to improve performance [1]︓
1. Consistency of teacherʼs and studentʼs inputs
2. Distilling on a wide range of data points for large number of epochs→ Imitating teacher as a function is critical（function matching）[1] L Beyer et al., “Knowledge Distillation: A Good Teacher Is Patient and Consistent.” CVPR2022.
Effectiveness of Function Matching in Driving Scene Recognition / 2022/10/23 / Shingo Yashima© DENSO IT LABORATORY,INC. All Rights Reserved.

5 /11

Function Matching on Driving Scene Recognition
Learning directly from labels

Function Matching

Large model
Compact model: Labeled data

Compact models are hard to generalize
when directly learned from labels: Unlabeled data
Imitating the large model on a large
number of points yields generalization

Generally, we can access an almost infinite amount of (unlabeled) driving scene data
How effective is using large amounts of unlabeled scene data for distillation?
Effectiveness of Function Matching in Driving Scene Recognition / 2022/10/23 / Shingo Yashima© DENSO IT LABORATORY,INC. All Rights Reserved.

6 /11

Experiments
Evaluation on semantic segmentation (SS) and object detection (OD) in BDD100K
Datasets

SS Labeled:

10,000 images

OD Labeled:

100,000 images

Unlabeled:

16,000,000 images (clipped from original videos)Learning Methods① Train the compact model directly from the labeled dataset (Supervised)② Train the large model from the labeled dataset→ Distill it to the compact model with the labeled dataset (Distill)③ Train the large model from the labeled dataset→ Distill it to the compact model with the unlabeled dataset (FunMatch)
Effectiveness of Function Matching in Driving Scene Recognition / 2022/10/23 / Shingo Yashima© DENSO IT LABORATORY,INC. All Rights Reserved.

7 /11

Results（Semantic Segmentation）
Model

Large:

PSPNet with ResNet-101 backbone

Compact:

PSPNet with ResNet-18 backbone

Results
MethodTraining DatamIoU

Large (teacher)Supervised

Labeled (70K)

64.83

Compact ①

Supervised

Labeled (70K)

61.48

Compact ②

DistillLabeled (70K枚)

61.70

Compact ③

FunMatchLabeled (70K) + Unlabeled (16M)

64.74(picked the best performing model among all training schedules)

The peformance of the compact model is improved dramaticallyby function matching with a large number of unlabeled data
Effectiveness of Function Matching in Driving Scene Recognition / 2022/10/23 / Shingo Yashima© DENSO IT LABORATORY,INC. All Rights Reserved.

8 /11

Results（Object Detection）
Model

Large:FCOS with ResNet-101 + FPN backbone

Compact:

FCOS with ResNet-18 + FPN backbone

Results
MethodTraining DatamAP

Large (teacher)Supervised

Labeled (70K)

31.4

Compact ①

Supervised

Labeled (70K)

29.5

Compact ②

DistillLabeled (70K枚)

30.4

Compact ③

FunMatchLabeled (70K) + Unlabeled (16M)

31.2(picked the best performing model among all training schedules)

The peformance of the compact model is improved dramaticallyby function matching with a large number of unlabeled data
Effectiveness of Function Matching in Driving Scene Recognition / 2022/10/23 / Shingo Yashima© DENSO IT LABORATORY,INC. All Rights Reserved.

9 /11

Results（Agreement with teacher）
Performance w.r.t. ground truth

Agreement with teacher
FunMatch

FunMatch

Use of unlabeled data and longer training yields better agreement with teacher,resulting in better performance
Effectiveness of Function Matching in Driving Scene Recognition / 2022/10/23 / Shingo Yashima© DENSO IT LABORATORY,INC. All Rights Reserved.

10 /11