Electronic Imaging 2021, Image Processing: Algorithms and Systems

Does end-to-end trained deep model
always perform better thannon-end-to-end counterpart?
Ikuro Sato1,2
Guoqing Liu1
Kohta Ishikawa1
Teppei Suzuki1
Masayuki Tanaka2

1Denso IT Laboratory, Inc., Japan

2Tokyo Institute of Technology, Japan

Outline• Introduction• Overview: FOCA

FOCA: Feature-extractor Optimization through Classifier Anonymization
I. Sato, et al., ICML2019.

• Experiment• Improvement over Sato et al.
• Comparison with end-to-end training methods• Effect of network fine-tuning after FOCA

• SummaryDoes end-to-end trained deep model always perform better than non-end-to-end counterpart?
DENSO IT LABORATORY,INC. / Tokyo Institute of Technology

Ikuro Sato, et al.

EI2021

2/21

Outline• Introduction• Overview: FOCA

FOCA: Feature-extractor Optimization through Classifier Anonymization
I. Sato, et al., ICML2019.

• Experiment• Improvement over Sato et al.
• Comparison with end-to-end training methods• Effect of network fine-tuning after FOCA

• Summary

Does end-to-end trained deep model always perform better than non-end-to-end counterpart?
DENSO IT LABORATORY,INC. / Tokyo Institute of Technology

Ikuro Sato, et al.

EI2021

3/21

Flourishing E2E network optimization
Successful by replacing intermediate tasks with learnable layerseg)• CNN  feature extractor (eg. SIFT) + pooling (eg. Fisher Vector) + classif.
• Spatial Transformer  coordinate preprocessing + classif.

D. Lowe, 2004. F. Perronnin+, 2007.

• Faster RCNN  region proposal + classif.

M. Jaderberg+, 2015.
S. Ren+, 2015.

• Monocular depth  optical flow + epipolar geometry estimation• PointNet  voxelization + classif.

C. Godard+, 2016.
C. Qi+, 2017.

deep network
InputFeature Ext.

Classifier𝑥𝐹 ⋅𝐶 ⋅Does end-to-end trained deep model always perform better than non-end-to-end counterpart?
DENSO IT LABORATORY,INC. / Tokyo Institute of Technology

Loss w/ target 𝑡

𝐿 ⋅, 𝑡Ikuro Sato, et al.

EI2021

4/21

Is E2E optimization always good?
Co-adaptation between feature extractor and classifier can occur.

G. Hinton+, 2012.

• Feature distribution is only good at a particular decision boundary.
• Vice versa.
E2E optimization𝜙⋆, 𝜃 ⋆1arg min,𝒟𝐿 𝐶𝐹 𝑥,𝑡, ∈𝒟co-adaptation
Input

Feature Ext.

Classifier𝑥𝐹 ⋅𝐶 ⋅Does end-to-end trained deep model always perform better than non-end-to-end counterpart?
DENSO IT LABORATORY,INC. / Tokyo Institute of Technology

Loss w/ target 𝑡

𝐿 ⋅, 𝑡Ikuro Sato, et al.

EI2021

5/21

Is E2E optimization always good?

Toy ex.)2-class regression

Feature dim-2

Worst cases: excessively complex feature distribution‘+1’

color: 𝐶 value‘-1’• Disjointed• SplitFeature dim-1

 Vulnerable to a small change in the feature distribution, i.e., bad transferability.
Input

Feature Ext.

Classifier𝑥𝐹 ⋅𝐶 ⋅Does end-to-end trained deep model always perform better than non-end-to-end counterpart?
DENSO IT LABORATORY,INC. / Tokyo Institute of Technology

J. Yosinski+, 2014.

Loss w/ target 𝑡𝐿 ⋅, 𝑡
Ikuro Sato, et al.

EI2021

6/21

Question we try to answer
Q.

Does end-to-end (E2E) trained deep model always
perform better than non-end-to-end counterpart?

A.

Not always. We show empirical evidenceswhere a non-E2E training method known as FOCA
outperforms strong E2E counterparts in image classification tasks.
FOCA: Feature-extractor Optimization through Classifier Anonymization

E2E training scheme

𝑥joint training

𝐹 ⋅𝐶 ⋅

Does end-to-end trained deep model always perform better than non-end-to-end counterpart?
DENSO IT LABORATORY,INC. / Tokyo Institute of Technology

𝐿 ⋅, 𝑡Ikuro Sato, et al.

EI2021

7/21

Our answer
Q.

Does end-to-end (E2E) trained deep model always
perform better than non-end-to-end counterpart?

A.

Not always. We show empirical evidenceswhere a non-E2E training method known as FOCAoutperforms strong E2E counterparts in image classification tasks.
FOCA: Feature-extractor Optimization through Classifier Anonymization

FOCA’s training scheme𝑥

Step-1Feature ext.
training

𝐹 ⋅Step-2
Classifier training
with frozen features

𝐶 ⋅Does end-to-end trained deep model always perform better than non-end-to-end counterpart?
DENSO IT LABORATORY,INC. / Tokyo Institute of Technology

𝐿 ⋅, 𝑡Ikuro Sato, et al.

EI2021

8/21

Outline• Introduction• Overview: FOCA

FOCA: Feature-extractor Optimization through Classifier Anonymization
I. Sato, et al., ICML2019.

• Experiment• Improvement over Sato et al.
• Comparison with end-to-end training methods• Effect of network fine-tuning after FOCA

• Summary

Does end-to-end trained deep model always perform better than non-end-to-end counterpart?
DENSO IT LABORATORY,INC. / Tokyo Institute of Technology

Ikuro Sato, et al.

EI2021

9/21

FOCA: Feature-extractor Optimization through Classifier Anonymization
FOCA𝜙

⋆

1arg min𝒟𝔼 ~𝐿 𝐶𝐹 𝑥,𝑡, ∈𝒟

Not arg min ⋯Random weak classifier: 𝜃~Θ, Feature extractor is optimized wrt an ensemble of weak classifiers, not a particular strong classifier.
Only this partis optimized.
Input

Feature Ext.

𝑥𝐹 ⋅Anonymized classifiers Loss w/ target 𝑡

𝐶𝐶𝐶 ⋅⋅⋅

Does end-to-end trained deep model always perform better than non-end-to-end counterpart?
DENSO IT LABORATORY,INC. / Tokyo Institute of Technology

𝐿 ⋅, 𝑡Ikuro Sato, et al.

EI2021

10/21

Why weak ??
strong classifier  Features do adapt…

a strong classifierfeature distribution  complex

a set of weak classifiersfeature distribution  simple

feature distribution
at some iteration

Does end-to-end trained deep model always perform better than non-end-to-end counterpart?
DENSO IT LABORATORY,INC. / Tokyo Institute of Technology

Ikuro Sato, et al.

EI2021

11/21

Why weak ??
many random weak classifiers  Features do not adapt to a particular one.

a strong classifierfeature distribution  complex

a set of weak classifiersfeature distribution  simple

feature distribution
at some iteration

Does end-to-end trained deep model always perform better than non-end-to-end counterpart?
DENSO IT LABORATORY,INC. / Tokyo Institute of Technology

Ikuro Sato, et al.

EI2021

12/21

Pseudocode

source code: https://github.com/DensoITLab/FOCA-v1

Optimizes 𝜃 with a small batch.
Works weakly tothe entire dataset.

Updates 𝜙 with 𝜃.

* Weak classifier 𝜃 is discarded after a single use.
Does end-to-end trained deep model always perform better than non-end-to-end counterpart?
DENSO IT LABORATORY,INC. / Tokyo Institute of Technology

Ikuro Sato, et al.

EI2021

13/21

Property: simple feature distribution
In words [I. Sato, et al., ICML2019],

If feature extractor has an enough representation ability,
all input data of the same class are projected toa single point in the feature space in a class-separable way
under certain conditions.

Feature dim-2

Features form simple
point-like distribution
per class (under someconditions).

E2E
Feature dim-2

FOCA

‘+1’
‘-1’

Feature dim-1

Feature dim-1
Does end-to-end trained deep model always perform better than non-end-to-end counterpart?
DENSO IT LABORATORY,INC. / Tokyo Institute of Technology

Ikuro Sato, et al.

EI2021

14/21

Outline• Introduction• Overview: FOCA

FOCA: Feature-extractor Optimization through Classifier Anonymization
I. Sato, et al., ICML2019.

• Experiment• Improvement over Sato et al.
• Comparison with end-to-end training methods• Effect of network fine-tuning after FOCA

• Summary

Does end-to-end trained deep model always perform better than non-end-to-end counterpart?
DENSO IT LABORATORY,INC. / Tokyo Institute of Technology

Ikuro Sato, et al.

EI2021

15/21

Improvement over Sato et al., ICML2019
Careful hyperparameter tuning with following techniques greatly improved FOCA’s generalization.

• global features (GF) with• Global Average Pooling (GAP) after convolution part• 2-layer perceptron (2-LP) after GAP• Batch Normalization
Sato et al., ICML2019.
Table 1. improvement over Satoet al., 2019. Wide ResNet (2810) base network used in thefeature extractor. CIFAR-10
dataset used.

Does end-to-end trained deep model always perform better than non-end-to-end counterpart?
DENSO IT LABORATORY,INC. / Tokyo Institute of Technology

this work

Ikuro Sato, et al.

EI2021

16/21

Comparison with E2E training methods
The non-E2E training method (FOCA) outperformed strong baselines that use E2E training under fair settings.

this work
this work

Does end-to-end trained deep model always perform better than non-end-to-end counterpart?
DENSO IT LABORATORY,INC. / Tokyo Institute of Technology

Ikuro Sato, et al.

EI2021

17/21

Comparison with E2E training methods
The non-E2E training 2D
methodhistograms(FOCA)ofoutperformed
normalized CIFAR-10
strong baselines
features thatprojected
use E2E
bytraining
PCA. under fair settings.
FOCA exhibits well-separated, point-like distribution.

this work
this work

Does end-to-end trained deep model always perform better than non-end-to-end counterpart?
DENSO IT LABORATORY,INC. / Tokyo Institute of Technology

Ikuro Sato, et al.

EI2021

18/21

Effect of network fine-tuning after FOCAAim

To see if E2E network fine-tuning
improve performance after FOCA.

opt. step process
so farhow about?

1

feature extractor optimization

2classifier optimization
with frozen features

3

E2E network fine-tuning

Result E2E network fine-tuning yields
no improvement orslightly worse performance.

Fig. 1 CIFAR-100 test error rate curve.
Epoch 0 means the start of fine-tuning.
Similar results obtained for CIFAR-10
and Tiny ImageNet.
Does end-to-end trained deep model always perform better than non-end-to-end counterpart?
DENSO IT LABORATORY,INC. / Tokyo Institute of Technology

Ikuro Sato, et al.

EI2021

19/21

Outline• Introduction• Overview: FOCA

FOCA: Feature-extractor Optimization through Classifier Anonymization
I. Sato, et al., ICML2019.

• Experiment• Improvement over Sato et al.
• Comparison with end-to-end training methods• Effect of network fine-tuning after FOCA

• Summary

Does end-to-end trained deep model always perform better than non-end-to-end counterpart?
DENSO IT LABORATORY,INC. / Tokyo Institute of Technology

Ikuro Sato, et al.

EI2021

20/21

SummaryQuestion we try to answer
Does end-to-end trained deep model always perform better than non-end-to-end counterpart?

Our answer
Not always, with supportive evidences:

FOCA’s training scheme• We found evidences in which a non-E2E
training method, FOCA, outperforms
strong E2E training counterparts
on CIFAR-10, 100, and Tiny ImageNet.
• E2E network fine-tuning after FOCA
yields no improvement orslightly worse performance.

𝑥Does end-to-end trained deep model always perform better than non-end-to-end counterpart?
DENSO IT LABORATORY,INC. / Tokyo Institute of Technology

Step-1FeatureExtractor
training

Step-2
Classifier
training withfrozen features

𝐹 ⋅𝐶 ⋅

Ikuro Sato, et al.

EI2021𝐿 ⋅, 𝑡21/21