Breaking Inter-Layer Co-Adaptation by Classifier Anonymization

Ikuro Sato 1 Kohta Ishikawa 1 Guoqing Liu 1 Masayuki Tanaka 2

Abstract

0.4This study addresses an issue of co-adaptation
between a feature extractor and a classifier ina neural network. A naı̈ve joint optimizationof a feature extractor and a classifier often
brings situations in which an excessively complex feature distribution adapted to a very specific classifier degrades the test performance.
We introduce a method called Feature-extractor
Optimization through Classifier Anonymization(FOCA), which is designed to avoid an explicit
co-adaptation between a feature extractor anda particular classifier by using many randomlygenerated, weak classifiers during optimization.
We put forth a mathematical proposition thatstates the FOCA features form a point-like distribution within the same class in a class-separable
fashion under special conditions. Real-data experiments under more general conditions provide
supportive evidences.

0.2

3

8

1.1

2

1. Introduction
When specific signal patterns are repeatedly delivered byhidden neurons in a neural network during training, thenetwork parameters are updated in a strongly tied way, or
co-adapted, so that the network becomes vulnerable against
small input perturbations (Hinton et al., 2012; Srivastava et al.,2014). To discourage co-adaptation, Hinton et al. proposeda method called Dropout that randomly deactivates neurons
during training. Properties of Dropout training have been
intensively studied (Helmbold & Long, 2015; Baldi & Sadowski,
2013; Gal & Ghahramani, 2016; Wager et al., 2013; Ren et al.,2016; Warde-Farley et al., 2013; Bengio et al., 2013); whereas

there is a critique saying it does not necessarily yield coadaptation prevention ability (Helmbold & Long, 2018).
Yosinski et al.

studied the degrees of inter-layer co-

1

Denso IT Laboratory, Inc., Japan 2 National Institute of Advanced Industrial Science and Technology, Japan. Correspondence
to: Ikuro Sato <isato@d-itlab.co.jp>.
Proceedings of the 36 th International Conference on Machine
Learning, Long Beach, California, PMLR 97, 2019. Copyright2019 by the author(s).

1
0

0.1

6

-0.2
4

-1.5

-1

-0.1

-2

2

-2.7

0

-3

-0.3

0.1

0.4

0.8

-3.3

-0.7

1.8

4.4

(a) Joint opt. Single- (left) and multi-layered classifiers (right).
0.4

2

-2.3

0.3

1

-3.2

0.1

0

1.5
1
0.5
0

-4

-0.5
-1

-0.1

-4.9

-1
-1.5

-2

-0.2

0.1

0.4

0.7

-6.4

-4.7

-3

-1.3

(b) FOCA. Single- (left) and multi-layered classifiers (right).
Figure 1. Visualization of typical 2D features of two-class training
data. (a) A naı̈ve joint optimization of a feature extractor and aclassifier; (b) FOCA (ours). Features in (b) form nearly point-like
distributions per class, whereas those in (a) form more complex
distributions. An L2 loss is minimized in each case. Black (white)
dots indicate +(−)1-class data points, and the colored maps indicate the classifiers’ outputs, where in (b) “averaged” outputs of
256 weak classifiers are shown.

adaptation by examining test performance that mid-layer
features can yield (Yosinski et al., 2014). In part of their experiments, they split an end-to-end trained network into two
blocks of layers, initialized the second-block parameterswith random numbers, and trained the second block fromscratch with the first-block parameters held fixed. They
found that there are often cases where the secondary optimization degrades the test performance compared to thepreceding primary joint optimization, despite that Dropout
is adopted. In these cases, inter-layer co-adaptation (orfragile co-adaptation, in their words) happens between two
blocks. Potentially, there is a chance that the secondary optimization finds the same minimum achieved by the primary
optimization; however, in reality the chance rate is usually
not high. Excessively complex feature distribution, like theones shown in Fig. 1 (a), would be a major factor that induces inter-layer co-adaptation. Yosinski et al. also showedthat inter-layer co-adaptation tends to cause negative effects

Breaking Inter-Layer Co-Adaptation by Classifier Anonymizationin the cross-domain transfer.
Is there a way to fundamentally avoid inter-layer coadaptation? Based on a thought that naı̈ve joint optimization of a feature extractor and a classifier would result inunwanted co-adaptation between them, we seek a more
fundamental approach to break the adhesion, rather thansearching best-performing feature layers empirically (Yosinski et al., 2014; Kobayashi, 2017). The questions we try toanswer in this work are: a) Is it possible to train a feature
extractor without inter-layer co-adaptation to a particularclassifier? b) After such training, what kind of characteristics along with robustness against unwanted inter-layer
co-adaptation does the feature extractor acquire?
Regarding the first problem, we introduce a particular feature-extractor optimization method called Featureextractor Optimization through Classifier Anonymization(FOCA) in Section 2. FOCA is designed so that the feature
extractor does not explicitly co-adapt to a particular classifier. Instead, it uses randomly generated, weak classifiers
during the feature-extractor training. FOCA belongs to a
family of network randomization methods (Srivastava et al.,2014; Wan et al., 2013; Zeiler & Fergus, 2013; Huang et al., 2016;
Singh et al., 2016), but is different from others in terms that

FOCA does not employ a joint optimization of a feature
extractor and a classifier. The classifier part is anonymized
by marginalizing independently generated, weak classifiers;in this way explicit co-adaptation to a particular classifier isavoided.

training dataset D contains nD such pairs. Feature extractor
Fφ : RdI → RdF transforms an input to a dF -dimensionalfeature with parameter set φ, and classifier Cθ : RdF →RdO transforms a feature to a dO -dimensional output vector
with parameter set θ. A joint optimization problem is given
as
1 X(φ? , θ? ) = arg min
L (Cθ (Fφ (x)), t) , (1)nDφ,θ(x,t)∈D

where L(·, t) : RdO → R defines the sample-wise loss
between the network output and the target.
When SGD training is naı̈vely applied, at each step the classifier is updated so as to become more discriminative for thepresented features, no matter how complex the feature distribution is. The feature extractor, on the other hand, is updated
so that the classifier at that moment becomes stronger, nomatter how complex the decision boundary is. The toy example in Fig. 1 (a) demonstrates such a case, where training
results in excessively complex feature distribution.
2.2. Feature-extractor Optimization through Classifier
Anonymization (FOCA)
Below, we introduce FOCA for optimizing a feature extractor without explicitly co-adapting to a particular classifier.
The optimization problem is defined as
1 X
φ? = arg minEθ∼Θφ L (Cθ (Fφ (x)), t) , (2)nD
φ(x,t)∈D

Regarding the second problem, we obtained intriguing mathematical proposition (Section 3) and experimental evidences
about simplicity of FOCA feature distributions. Let us
suppose class-c features form a point-like distribution in aclass-separable fashion. In that case a strong classifier for apartial dataset must be also strong for the entire dataset. This
characteristics is largely confirmed for the FOCA features(Section 4.1). The distance between large-dataset solution
and small-dataset solution in the classifier parameter spaceis indeed very small when FOCA is adopted (Section 4.2).
Low-dimensional analyses of the FOCA features exhibit
nearly point-like distributions (Section 4.3).

2. Optimization Method to Break Inter-Layer
Co-Adaptation
In this section, we introduce FOCA that aims at training
a feature extractor without inter-layer co-adaptation to aparticular classifier. We first go over the basic joint optimization method, then introduce FOCA.
2.1. Joint Optimization: a Review
Let (x, t) be a pair of a dI -dimensional input data and thecorresponding dO -dimensional target data, respectively. The

where Θφ represents a predefined distribution function of
weak classifiers for a given parameter set φ, and Eθ∼Θφrepresents the expectation value over θ ∼ Θφ . The feature
extractor is optimized with respect to a set of weak classifiers that are independently sampled from Θφ and thus is
not able to co-adapt to a particular classifier, as long as Θφgenerates distinct weak classifiers.
The weakness of the discriminative power of θ ∼ Θφ isessential in this formulation. If θ ∼ Θφ is designed to be
too strong for D, its decision boundary likely becomes fairly
complex during the training, and the feature extractor would
update itself to better fit the complex decision boundary,
resulting in a vicious cycle. On the other hand, if θ ∼ Θφis too weak or even adversarial, the optimization process
would not converge.
The marginalization over weak classifiers likely preventsthe feature distribution from becoming excessively complex.
Even at the end of the optimization, there is generally alarge number of distinct weak classifiers, and the feature
extractor is optimized with respect to the ensemble of theseweak classifiers. Although some of the weak classifiers may
have excessively complex decision boundaries, marginalization over the classifier ensemble likely smoothens thoseBreaking Inter-Layer Co-Adaptation by Classifier Anonymizationout. This likely yields a relatively simple decision boundaryand reasonably strong classification power, as is essential in
the classical Classifier Bagging algorithms (Breiman, 1996)
and other ensemble learning algorithms (Hara et al., 2017;
Zahavy et al., 2018). Therefore, the form of the feature distribution likely becomes simple as far as the feature-extractor’s
description ability allows.
There is some room in defining Θφ , and here we introduce
a particular definition. Let
Θφ = U({θφ,b ; b = b1 , b2 , · · · }),(3)

where U(s) is a discrete uniform distribution function for
all elements in set s, and θφ,b is a solution that minimizes abatch-wise loss function with a norm regularization,
θφ,b = arg minθ1 X
L (Cθ (Fφ (x)), t) + λkθk22 . (4)nb(x,t)∈b

Here, batch b comprises nb training samples that cover allclasses, and λ > 0. We further assume that the classifierparameters are initialized with random numbers prior to
optimizing; therefore, there is almost no chance of having
continuity between θφ,b and θφ+δφ,b for kδφk  1.
A solution θφ,b , a strong classifier for the batch b, is not
generally strong for the entire dataset D for given φ because
it does not “see” training samples other than the ones in b.
However, there is no guarantee that θφ,b is always a weakclassifier to D in a classical sense; that is, a weak classifier
performs only slightly better than random guesses. Indeed,
θφ,b can even work adversarially to D, meaning its accuracy
is below the chance rate. But, for brevity, we simply call
θφ,b a “weak classifier” in this work.
The norm regularization term in Eq. (4) helps to avoid
blowups during the feature-extractor training. The scaleof θφ,b can be very large without the regularizer when two
feature vectors in b stand close to each other. In such a caseinstability likely occurs.
After a feature extractor is obtained by Eq. (2), the following
secondary optimization using the entire dataset provides a
final, single classifier.
θ? = arg min
θ

1
nD

X

L (Cθ (Fφ? (x)), t) .

(5)(x,t)∈D

Here, the classifier is trained with fixed features. Note thatthe classifier architecture in this secondary optimization candiffer from the one used in the primary optimization.
Our method and meta-learning (Finn et al., 2017) share afollowing similarity, despite that the goals are different (coadaptation prevention vs. transferable multi-task learing).
Our feature extractor acts like task-generic base network,and our classifiers act like taskwise fine-tuned models.

Approximate minimization. Regarding the number of
weak classifiers used in a single update of φ, it is impossible
to prepare a complete set of possible weak classifiers due tothe huge number of distinct batches of the same size. We
must adopt approximation instead. Algorithm 1 1 gives anapproximate solution of φ? in two senses: 1) a single weakclassifier is sampled from Θφ per φ-update instead of taking
a marginalization over Θφ , and 2) Θφ is held fixed in thecomputation of gradients with respect to φ.
Algorithm 1 Approximate minimization in Eq. (2)Input: total number of iterations T ; number of classes C; numberof class-c samples nc (c = 1, · · · , C); number of samples perclass for θ-update k; total number of samples nD ; minibatch
size for φ-update m; learning rate η
1: Begin2:
Initialize φ by random variables.
3:
for t = 1 : T do
4:
Ic = [randi(nP
1 , k), · · · , randi(nC , k)]5:θ = arg min i∈Ic L (Cθ0 (Fφ (xi )), ti ) + λkθ0 k22
θ06:
If = randi(nPD , m)η7:
φ←φ− mi∈If ∂L (Cθ (Fφ (xi )), ti ) /∂φ
8:
end for
9: EndOutput: feature-extractor parameters φ? = φ

It is worth mentioning that there is another way of generating a reasonably weak classifier: to take a batch (which
can be large) and then to optimize the batch-wise loss in
an incomplete fashion by stopping after a relatively small
number of iterations, say 20 times. This works fine, though
the definition of θφ,b becomes mathematically less clear.

3. Mathematical Property
We now show a proposition about the simplicity of FOCA
feature distributions. It will be proven that under somespecial conditions any two samples have exactly the samefeatures when target classes are the same, but have different features when target classes are different. Let us first
introduce a lemma about implicit optimality for individualfeatures, and then put forth the proposition.
Lemma 3.1. Suppose that a multi-layered feature extractor
with two restrictions is used:
1) The activation function a satisfies
a : R → R+ ,

∂a(z)
6= 0.
∂z

(6)

2) The last layer is fully-connected.
If φ? simultaneously minimizes sample-wise losses
1
In the pseudocode, randi(i, j) returns a j-dimensional vector
with each element being a random variable ∼ U({1, 2, · · · , i}).

Breaking Inter-Layer Co-Adaptation by Classifier Anonymization

L(Cθ (Fφ (x)), t) for all (x, t) ∈ D, then,∂Cθ ∂L(Cθ (Fφ? (x)), t)= 0, ∀(x, t) ∈ D.
∂Fφ?
∂Cθ
∂Cθ
θ (f )( ∂F
is a short-hand notation for ∂C∂f
φ?

(7)
. A

f =Fφ∗ (x)summation symbol over Cθ indices is ignored in Eq. (7).)
Proof. Let φ` be the parameter set of the last weight layer
in the feature extractor, and let x` be its input. Then, the i-th
element of the feature layer, which is fully-connected
fromP
the previous layer, is given as Fφ (i) = a( j φ` (i, j)x` (j)).
P∂F (i)
Let z = j φ` (i, j)x` (j). Then, ∂φ`φ(i,j) = ∂a(z)∂z x` (j) 6=∂F (i)

φ0, since ∂a(z)∂z 6= 0 and x` > 0. The inequality ∂φ` (i,j) 6= 0,

the supposition ∂φ∂L
` (i,j)φ=φ?

= 0, ∀(x, t) ∈ D, and the

The singular-value decomposition of the matrix consistingof column vectors θ̄b sampled from Θφ? yields
θ̄b1 , θ̄b2 , · · · = U SV > , where diagonal elements ofthe positive diagonal matrix S consists of the singularvalues aligned in decreasing order. Then, U > θ̄b =[θ̄bn> , 0, · · · , 0]> , where θ̄bn is the non-singular componentsof θ̄b . Taking the non-singular part in Eq. (9),Eθ θ̄n θ̄n> Fφn? (x) = Eθ θ̄n (t − θ0 ), ∀(x, t) ∈ D. (10)hi>s>Here, U > Fφ? (x) = Fφn>, where Fφn? (x)? (x), Fφ? (x)is the corresponding non-singular part, meaning Fφn? (x) andθ̄n sharing the same dimension. The matrix Eθ θ̄n θ̄n> is
obviously invertible; therefore, Eq. (10) can be solved forFφn? (x). It then tells us that (a): Fφn? (x) depends only on tand θb ’s. On the other hand, the minimum-norm solution θ̄bsatisfies,

chain rule immediately leads Eq. (7).
[(f1 − f2 )(f1 − f2 )> + λI]θ̄b = (t1 − t2 )(f1 − f2 ), (11)In the following discussion, we assume the conditions statedbelow hold.
(C1) A multi-layered feature extractor with two restriction
is used: 1) The activation function satisfies Eq. (6); 2) The
last layer is fully-connected.
(C2) The target values are t ∈ {t1 , t2 } for all samples.
(C3) A sample-wise loss function of the form L̃φ,θ (x, t) =(Cθ (Fφ (x)) − t)2 is adopted.

where I is the identity matrix and f1(2) = Fφ? with thetarget t = t1 (t2 ) as a short-hand notation. Taking the nonsingular part in Eq. (11), the minimum-norm solution θ̄bn
satisfies (b):[(f1n −f2n )(f1n −f2n )> +λI]θ̄bn = (t1 −t2 )(f1n −f2n ), (12)where superscript n denotes the non-singular part. Given
the definition that [θ̄bn1 , θ̄bn2 , · · · ] is full-rank, statements (a)and (b) do not contradict only if(C4) A linear classifier Cθ (Fφ (x)) = θ̄> Fφ (x)+θ0 is used.
(C5) Θφ = U({θφ,b ; b = b1 , b2 , · · · }), where θb =P
arg min (x,t)∈b L̃φ,θ (x, t) + 12 λkθ̄k22 , and b1 , b2 , · · · areθ

distinct batches, each of which comprises one sample fromt1 class and one sample from t2 class.
Proposition 3.2. Suppose that φ? simultaneously minimizes the classifier-anonymized, sample-wise lossesEθ∼Θφ L̃φ,θ (x, t) in a class-separable fashion for all(x, t) ∈ D. Then, samples from the same class sharethe same features; i.e., Fφ? (x) = Fφ? (x0 ), ∀x, x0 ∈ Xc ,but samples from different classes do not; i.e., Fφ? (x) 6=
Fφ? (x0 ), ∀x ∈ Xc , ∀x0 ∈ Xc0 6=c .
Proof. Lemma 3.1 about the implicit optimality of individual features with respect to sample-wise losses yields,Eθ∂Cθ ∂ L̃φ? ,θ (x, t)= 0, ∀(x, t) ∈ D,
∂Fφ?
∂Cθ

(8)

where Eθ is a short-hand notation for Eθ∼Θφ . By taking thepartial derivatives in Eq. (8), one obtains,Eθ θ̄θ̄> Fφ? (x) = Eθ θ̄(t − θ0 ), ∀(x, t) ∈ D.
(9)∃v ∈ R, θ̄bn = v, ∀b.

(13)

It is, θ̄bn is one-dimensional and constant for all b. Then,statement (a) yields, Fφn? (x) = Fφn? (x0 ), ∀x, x0 ∈ Xc .
Note that Fφn? (x) 6= Fφn? (x0 ), ∀x ∈ Xc , ∀x0 ∈ Xc0 6=c ;otherwiseP
φ? is not a class-separable solution. Because0θb = 1/2 (x,t)∈b (t − θ̄n> Fφn? ), θb0 must be the same forall b. Since θ̄b = U [θ̄bn> , 0, · · · , 0]> , θ̄b must be the samefor all b also. The fact that the minimum-norm solutions
are the same for all combinations of t = t1 and t = t2
data points tells that Fφ? (x) = Fφ? (x0 ), ∀x, x0 ∈ Xc andFφ? (x) 6= Fφ? (x0 ), ∀x ∈ Xc , ∀x0 ∈ Xc0 6=c .
According to this proposition, if the feature extractor has anenough representation ability under certain conditions, all
the input data of class c are projected to a single point in
the feature space in a class-separable way. The left side ofFig. 1 (b) visualizes 2D features of the toy data optimized
by FOCA under the conditions (C1)-(C5)2 . Features of thesame class are confined in a vicinity, the size of which ismuch smaller than the distance between the class centroids.
It is intriguing to observe such a “point-like” distribution
2

Here θφ,b is the analytical solution.

Breaking Inter-Layer Co-Adaptation by Classifier Anonymization

property, even though we do not explicitly impose a thing
like maximization of the between-class scatter with respect
to the sum of within-class scatters. Although we have notsucceeded in proving for multi-layered classifier cases, thetoy experiment still exhibits the point-like distribution property as well; see the right side of Fig. 1 (b).

4. Experiment
Let us first state our motivations for a series of experiments.
We saw in Section 3 that the FOCA features obey a pointlike distribution per class under the special conditions. The
question we try to answer in this section is, Do the FOCA
features form a point-like distribution or some similar distribution under more realistic conditions? Let us supposethat features form a point-like distribution. Then, following
secondary classifier optimization with the feature extractor held fixed should yield a similar decision boundary nomatter what subset of the entire dataset it learns, as long as
all classes are covered. We indeed confirmed that high test
performances are achieved by FOCA, even when the secondary optimization uses smallest possible partial datasets;
namely, only one data from each class (see Section 4.1).
When FOCA is used, the secondary optimization leads theclassifier parameter vector to almost the same point regardless of the size of the partial dataset it uses (see Section 4.2).
Lastly, low-dimensional analyses revealed that FOCA features projected onto a hypersphere form a nearly a point-like
distribution in a class-separable fashion (see Section 4.3).
Datasets. We use the CIFAR-10 dataset, a 10-class image
classification dataset having 5 × 104 training samples, andthe CIFAR-100 dataset, a 100-class image classification
dataset having the same number of samples (Krizhevsky &Hinton, 2009). Both datasets have similar properties except
for the number of classes (10:100) and the number of samples per class (5000:500). The idea is to see how thesedifferences affect feature distribution properties.
Methods. In each experiment, FOCA is compared withother methods below. Plain: a vanilla mini-batch SGD
training. Noisy (Graves, 2011): the same training rule as in
Plain, except that a zero-mean random Gaussian noise isadded to each of the classifier parameters during training.
Dropout (Hinton et al., 2012): adopted only to the classifier
part. Batch Normalization (Ioffe & Szegedy, 2015): adopted
to the entire architecture.
Dropout is claimed to reduce co-adaptation (Hinton et al.,2012), though there is a counterpoint to this view (Helmbold& Long, 2018). FOCA, Dropout and Noisy share the samecharacteristics in a sense that the classifier’s descriminative
power is weakened and the classifier ensemble is implicitly taken during training. Apart from FOCA, Dropout andNoisy employ joint optimization, and we are interested tosee how this affects the robustness against inter-layer coadaptation. Batch Normalization is included in comparison
based on a thought that the way it propagates signals fromone layer to the other may have some functionality mitigating inter-layer co-adaptation.
Architecture. The architecture that we use for the primary
optimization in each CIFAR-10 experiment is the one introduced in (Lee et al., 2016), except that we replaced the last
two layers by three fully-connected (FC) layers of the form:
4096(feature dim.)-β-β-10, where β = 1024 for Dropout
and β = 128 otherwise. The architecture for the secondary
optimization is 4096-128-128-10 for all methods. The architecture for the CIFAR-100 primary optimizations is VGG-16(Simonyan & Zisserman, 2015), except that the last three FC
layers are replaced by 512(feature dim.)-β-β-100, whereβ = 512 for Dropout and β = 128 otherwise. The architecture for the secondary optimization is 512-128-128-100 for
all methods.
Training details. SGD with momentum is used in eachbaseline experiment. In each FOCA experiment, the featureextractor part uses SGD with momentum, and the classifierpart uses gradient descent with momentum. In each training,
we tested a couple of different initial learning rates andchose the best-performing one in the validation. A manual
learning rate scheduling is adopted; the learning rate is
dropped by a fixed factor 1-3 times. The weak classifiers
are randomly initialized each time by zero-mean Gaussian
distribution with standard deviation 0.1 for both CIFAR-10
and -100. Cross entropy loss with softmax normalization
and ReLU activation (Nair & Hinton, 2010) are used in everycase. No data augmentation is adopted. The batch size bused in the weak-classifier training is 100 for the CIFAR10 and 1000 for the CIFAR-100 experiments. The numberof updates to generate θ is 32 for the CIFAR-10 and 64
for the CIFAR-100 experiments. Max-norm regularization(Srivastava et al., 2014) is used for the FOCA training, tostabilize the training. We found that the FOCA training
can be made even more stable when updating the featureextractor parameters u times for a given weak classifierparameters. We used this trick with u = 8 in the CIFAR100 experiments.
4.1. Test Performances of Classifiers Trained on Partial
Datasets
Motivation. We are interested in two aspects. 1) Is thetest performance produced by the primary joint optimization reproducible by the secondary classifier optimization?
(FOCA is excluded here because it is not a joint optimization method.) 2) How does the test performance degrade
when smaller datasets are learned in the secondary optimization? Remember that a point-like distributed features are
expected to demonstrate little degradation.

Breaking Inter-Layer Co-Adaptation by Classifier Anonymization
end- secondary
to0.6
end @5x104
11.06
0.5FOCA
14.15 13.51
Plain
0.4Noisy
12.66 11.87
14.95 13.43
Dropout
0.3
Batch Norm 10.32 9.65

end- secondary
toend @5x104
11.06
FOCA
13.51 14.15
Plain
11.87 12.66
Noisy
13.43 14.95
Dropout
Batch Norm 9.65 10.32

0.7

0.2
0.1
5x104 10 4

10 3

10 2

10 1

(a) CIFAR-10
end- secondary
toend @5x104
43.33
FOCA
44.04 51.81
Plain
43.77 50.57
Noisy
43.24 51.14
Dropout
Batch Norm 39.28 47.91

0.7
0.65
0.6
0.55
0.5
0.45
0.4
5x104

10 4

10 3

10 2

(b) CIFAR-100
Figure 2. Test error rates of classifiers trained on partial datasets.
For each method, partial datasets D0 are constructed r(nD0 ) times,
where r(5 × 104 ) = 1, r(104 ) = 5, r(103 ) = 15, r(102 ) = 50,
and r(10) = 150. The solid lines indicate the mean values andthe error bars indicate ±1 standard deviations of test error rates.
The numbers in the right side indicate the test error rates (%) of
the end-to-end optimizations and the secondary optimizations for
nD0 = 5 × 104 (also shown by the bullet points).

Experimental procedure. We trained feature extractors
on the entire training dataset using FOCA and the othermethods. All methods except for FOCA employ the end-toend learning scheme. Then, we detach the feature extractors and the classifiers after learning. Next, for all methods, we fix the feature-extractor parameters and train classifiers from scratch by the orthodox backpropagation on
reduced datasets D0 of size nD0 . For CIFAR-10 experiments, nD0 = 5 × 104 , 104 , 103 , 102 , 10 (10 is the smallest possible dataset size). For CIFAR-100 experiments,
nD0 = 5 × 104 , 104 , 103 , 102 (102 is the smallest possible).
Figure 2 now shows the test error rates vs. nD0 .
Cases of nD0 = nD . For all end-to-end optimization methods tested here, the test performances after the secondary
optimizations at nD0 = 5 × 104 happen to be worse thancorresponding end-to-end optimization results (see the rightside of Fig. 2). This is one of the indications that inter-layer
co-adaptation occurs to some degree in each of these joint
optimization methods.
3

Cases of nD0 < nD . For CIFAR-10, when nD0 ≤ 10 ,FOCA outperforms the other methods. To our surprise, theaverage test error rate of FOCA at nD0 = 10 is 14.45%,
which is indeed better than the test error rates of Dropout(14.95%) at nD0 = 5 × 104 Plain (14.50%) at nD0 = 104 ,Noisy (14.75%) at nD0 = 103 , and Bach Normalization(16.93%) at nD0 = 102 . For CIFAR-100, FOCA outperforms the other methods at any nD0 . The average error
rate of FOCA at nD0 = 102 is 46.03%, which is indeed
better than the test error rate of all the other methods at
nD0 = nD = 5 × 104 . We think that a high test performance of FOCA for nD0  nD is one indication ofrelatively simple form of feature distribution.
4.2. Approximate Geodesic Distances between
Solutions
Motivation. We just saw that classifiers trained with thelargest possible dataset (nD0 = nD ) and classifiers trainedwith the smallest possible partial dataset (nD0 = 10 for
CIFAR-10, nD0 = 100 for CIFAR-100) exhibit similar
test performances when the FOCA features are used. Let
us call the former the large-dataset solution θLD and thelatter the small-dataset solution θSD . The above observation
drove us to investigate distances between θLD and θSD .
The distance should be small when the features form apoint-like distribution per class, and we expect FOCA has
this characteristics. We employ the approximate geodesic
distance here to take changes in the loss landscape into
account. If θLD and θSD are virtually the “same” point with
FOCA, we further expect that the test error rate is almost
unchanged at any intermediate point between θLD and θSD .
Since two neural networks having the same architectureand different parameters can produce the same output for
an arbitrary input (Watanabe, 2009), we initialized networks
with the same set of random numbers in the experiments
conducted in this subsection.
Experimental procedure. After optimizing a feature extractor on the full training dataset, θLD is optimized withfeatures of the full dataset of size nD0 = nD = 5 × 104 ,and θSD is optimized with features of a smallest possible
partial dataset; nD0 = 10 for CIFAR-10 and nD0 = 100
for CIFAR-100. To quantify the separation between θLDand θSD , we partition the straight line connecting θLD andθSD into P line segments of equal lengths in the parameter
space; i.e.,θα =αθSD + (P − α)θLD, α = 0, 1, · · · , P.
P

(14)

We define in this article an approximate geodesic distanced(θLD , θSD ) between θLD and θSD asd(θLD , θSD ) =

"P −1
X

# 12
d(θα , θα+1 )2

.

(15)

α=0

Here, d(θα , θα+1 ) is the distance between θα and θα+1 with

Breaking Inter-Layer Co-Adaptation by Classifier Anonymization
10 -2
10

10 -3

0.6

10 -3

10

-4

10 -4

0.4

10 -4
10 -5

0.5

1

(a) CIFAR-10, 1st layer
10

0

10 -5

0.5

1

(b) CIFAR-10, 2nd layer

-2

10

0

10

10

-3

10

-4

0

0.2
0.5

1

(c) CIFAR-10, 3rd layer

0.1

0

0.5

-0.5

1

(d) CIFAR-10, error-1rate
-1

0.65

-2

10
-3

0.5

0.3

10 -5
0

1

0.5

-3

0

-2

0.6
0.55

10 -3

0.5
10 -4

0

0.5

1

(e) CIFAR-100, 1st layer

10
0

0.5

-4

1

(f) CIFAR-100, 2nd layer

0

0.45
0.5

1

(g) CIFAR-100, 3rd layer

0.4

0

0.5

1

(h) CIFAR-100, error rate

Figure 3. The approximate geodesic distances (a-c, e-g) between θLD and θSD and test error rates at θα (d, h). In (a-c, e-g), the solidlines indicate segment-wise distances d(θα , θα+1 ), α ∈ {0, · · · , P − 1}, and the dashed lines indicate total distances d(θLD , θSD ).

respect to the Fisher information metric I α evaluated at θα ,d(θα , θα+1 )2 = (θα+1 − θα )> I α (θα+1 − θα ),>∂Lφ? ,θ (x, t)∂Lφ? ,θ (x, t)
I α = E(x,t)∈D̃
∂θ
∂θ

(16)
,
θ=θ α

where D̃ is either D or a subset of D for ease ofcomputation, and Lφ? ,θ (x, t) is a short-hand notation ofL(Cθ (Fφ? (x)), t). To compute a genuine geodesic distance,
one needs to compute the sum of Fisher-metric distances
between pairs of infinitesimally separated points along the
curve that minimizes the squared sum. This is computationally infeasible; we instead approximate the curve by thestraight line, as explained. In the experiment, we let D̃ be arandomly chosen subset consisting of 5% of the entire training samples. We set P = 15. We evaluated d(θLD , θSD )layer by layer.
Figure 3 now shows the approximated geodesic distances
and test error rates at θα .
Approximate geodesic distances. For CIFAR-10, FOCA
exhibits some orders-of-magnitude, say 40-180 times,
smaller distances d(θLD , θSD ) than the other methods. For
CIFAR-100, FOCA exhibits 3-9 times smaller distances
than the other methods. We guess the reason why thedifferences are moderate for the CIFAR-100 cases is thatpoint-like distribution is harder to obtain for CIFAR-100.
Nevertheless, FOCA exhibit smallest approximate geodesic
distances in all cases, and we think this is an implicit evidence that the distribution of the FOCA features is simple

enough so that the discriminative function generated with
nD0 = nD is virtually reproducible when nD0  nD .
Test error rates at θα . For FOCA, test error rates
are almost constant at all θα . Together with the smalld(θLD , θSD ), two points θLD and θSD could be viewed
as virtually the same point, when FOCA is used. For
other methods, test error rates increase more rapidly towardθSD = θ15 .
4.3. Low-Dimensional Properties
Motivation. We now use classical component analyses toclarify the low-dimensional structure of the FOCA features.
In this subsection we only show the CIFAR-10 results, because the CIFAR-100 results are qualitatively similar.
Principal Component Analysis (PCA). Figure 4 (a) shows
scatter plots of training-data features projected by 2D basesof the PCA that is applied to all features. For a given class,the projected FOCA features look nearly one-dimensional,
not a point-like, per class. This one-dimensional characteristics is probably due to the use of softmax normalization at
the last layer, though we have no proof so far. In contrast,
the projected features of the other methods clearly span two
dimensions, roughly confined in an ellipse-like region, for agiven class.
Linear Discriminant Analyses (LDA) with normalization. Next, we examine the LDA on features normalized
to unit lengths. Normalization is taken based on the observation that the 2D features in Fig. 4 (a) are distributed1

Breaking Inter-Layer Co-Adaptation by Classifier Anonymization
20

50
0

0

f2(x)

f2(x)

f2(x)

f2(x)

-40
-60

-50
-2000

-2000 -1000

-100
-200

0

-100

f1(x)

-100

0

-20
-40

-100

f1(x)

-50

-80
-100

0

20
0

-60

-80
-3000

40

-20

0
-1000

60

0

f2(x)

1000

-50

f1(x)

-20
-40

0

-20

f1(x)

0

20

0.04

0.06

f1(x)

(a) Training-data features projected by the PCA bases.
0.15

0.04

0.02

0.02

0.02

-0.02

-0.05
0

0.1

-0.04

0.2

0
-0.02

0

f1(x)

0.02

-0.04

0.04

f2(x)

0

f2(x)

0

f2(x)

0.01

0.05

f2(x)

f2(x)

0.04

0.02

0.1

-0.1

0.04

0

0

f1(x)

0.02

0.04

-0.04

0.06

0
-0.01

-0.02

-0.02
0

f1(x)

0.02

0

0.04

f1(x)

0.02

f1(x)

(b) Training-data features first normalized to unit lengths then projected by the LDA bases,
constructed on the class-1, normalized features vs. the rest of the normalized features.
0.04

0.1

0.04

0.02
0.02
0.01

0.02

0
-0.02

f2(x)

f2(x)

f2(x)

f2(x)
-0.1

0

0

f2(x)

0.02
0

-0.02

-0.02

0
-0.01
-0.02

-0.2

0

0.1

f1(x)

0.2

-0.04

0

0.02

0.04

-0.04

0

f1(x)

0.02

0.04

0.06

-0.04

0

f1(x)

0.02

f1(x)

0.04

0

0.02

0.04

f1(x)

(c) Test-data features first normalized to unit lengths then projected by the same LDA bases used in (b).
Figure 4. Two-dimensional visualization of 4096-dimensional, CIFAR-10 features. Methods are (from the left): FOCA (ours), Plain,
Noisy, Dropout, and Bach Normalization. Colors indicate true classes.

Table 1. The results of the class-1-vs-rest LDA with normalization.
MethodEigenvalue Test error rate
FOCA (ours)
247.28
2.01%
Plain
5.74
2.71%
Noisy
7.49
2.86%
Dropout
5.81
2.78%
Bach Norm
7.28
2.43%

mostly along radial direction about a point close to the origin. Figure 4 (b) shows the 2D features that are normalized
and projected by the LDA bases described above. Only
class-1-vs-rest results are shown because no significant differences are observed when replacing class-1 by anotherclass. Here, we can observe a remarkable differences; theprojected FOCA features are linearly separable with a fairly
large margin, compared to the characteristic scales of theclass-1 distribution or of the rest-of-the-class distribution.
The form of the feature distribution is close to point-like perclass, somewhat similar to the observation in the toy experiment shown in Fig. 1 (b). In contrast, the other methods
exhibit linearly non-separable feature distributions.
Linear separability by the LDA with normalization.
The generalized eigenvalue computed in the LDA discussed
above are given in Table 1. The values are the largest ra-

tios of the between-class scatters to the within-class scattersafter linear projections. FOCA exhibits orders of magnitude larger generalized eigenvalue than other methods. This
supports the high level of linear separability of the FOCA
features. Figure 4 (c) shows the normalized features of test
data, projected by the same LDA bases, to see the generalizability. Table 1 also shows the smallest possible test
error rates (class-1 vs. rest) by setting a threshold along theprincipal axis. FOCA yields the lowest test error rate.

5. Conclusion
A naı̈ve joint optimization of a feature extractor and a classifier in a neural network often brings cases where both setsof parameters are tied in a so complex way that the classifier
is irreplaceable without degrading the test performance. We
introduced a method called Feature-extractor Optimization
through Classifier Anonymization (FOCA), that is designed
to break unwanted inter-layer co-adaptation. FOCA produces a feature extractor that does not explicitly adapt to aparticular classifier. We gave a mathematical proposition
that guarantees a simple form of feature distribution under special conditions; indeed, features form a point-like
distribution in a class-separable way. Different kinds of realdataset experiments under more general conditions provide
supportive evidences.

Breaking Inter-Layer Co-Adaptation by Classifier Anonymization

References
Baldi, P. and Sadowski, P. J. Understanding dropout. In
Neural Information Processing Systems (NIPS), pp. 2814–
2822, 2013.
Bengio, Y., Léonard, N., and Courville, A. C. Estimatingor propagating gradients through stochastic neurons forconditional computation. arXiv:1308.3432, 2013.
Breiman, L. Bagging predictors. Machine Learning, 24:
123–140, 1996.
Finn, C., Abbeel, P., and Levine, S. Model-agnostic metalearning for fast adaptation of deep networks. In International Conference on Machine Learning (ICML), 2017.
Gal, Y. and Ghahramani, Z. Dropout as a Bayesian approximation: Representing model uncertainty in deep learning. In International Conference on Machine Learning(ICML), 2016.
Graves, A. Practical variational inference for neural networks. In Neural Information Processing Systems (NIPS),
2011.
Hara, K., Saitoh, D., and Shouno, H.
Analysisof dropout learning regarded as ensemble learning.
arXiv:1706.06859, 2017.
Helmbold, D. and Long, P. Surprising properties of dropout
in deep networks. Journal of Machine Learning Research,
18:1–28, 04 2018.
Helmbold, D. P. and Long, P. M. On the inductive biasof dropout. Journal of Machine Learning Research, 16:3403–3454, 2015.
Hinton, G. E., Srivastava, N., Krizhevsky, A., Sutskever,I., and Salakhutdinov, R.
Improving neural networks by preventing co-adaptation of feature detectors.
arXiv:1207.0580, 2012.
Huang, G., Sun, Y., Liu, Z., Sedra, D., and Weinberger,
K. Q. Deep networks with stochastic depth. In European
Conference on Computer Vision (ECCV), 2016.
Ioffe, S. and Szegedy, C. Batch normalization: Accelerating deep network training by reducing internal covariate
shift. In International Conference on Machine Learning(ICML), 2015.
Kobayashi, T. Sharing convnet across heterogeneous tasks.
In International Conference on Neural Information Processing (ICONIP), 2017.
Krizhevsky, A. and Hinton, G. Learning multiple layers offeatures from tiny images. Technical Report 4, 2009.

Lee, C.-Y., Gallagher, P., and Tu, Z. Generalizing pooling functions in convolutional neural networks: Mixed,
gated, and tree. In International Conference on Artificial
Intelligence and Statistics (AISTATS), 2016.
Nair, V. and Hinton, G. E. Rectified linear units improve restricted boltzmann machines. In International Conference
on Machine Learning (ICML), 2010.
Ren, Y., Zhang, L., and Suganthan, P. N. Ensemble classification and regression-recent developments, applications
and future directions [review article]. IEEE Computational Intelligence Magazine, 11(1):41–53, 2016.
Simonyan, K. and Zisserman, A. Very deep convolutional
networks for large-scale image recognition. In International Conference on Learning Representations (ICLR),
2015.
Singh, S., Hoiem, D., and Forsyth, D. Swapout: Learning
an ensemble of deep architectures. In Neural Information
Processing Systems (NIPS), 2016.
Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I.,
and Salakhutdinov, R. Dropout: A simple way to preventneural networks from overfitting. Journal of Machine
Learning Research, 15:1929–1958, 2014.
Wager, S., Wang, S., and Liang, P. S. Dropout training as
adaptive regularization. In Neural Information Processing
Systems (NIPS), 2013.
Wan, L., Zeiler, M., Zhang, S., LeCun, Y., and Fergus, R.
Regularization of neural network using dropconnect. In
International Conference on Machine Learning (ICML),
2013.
Warde-Farley, D., Goodfellow, I. J., Courville, A. C., andBengio, Y. An empirical analysis of dropout in piecewise
linear networks. arXiv:1312.6197, 2013.
Watanabe, S. Algebraic geometry and statistical learningtheory. Cambridge University Press, 2009.
Yosinski, J., Clune, J., Bengio, Y., and Lipson, H. Howtransferable are features in deep neural networks? In
Neural Information Processing Systems (NIPS), 2014.
Zahavy, T., Sivak, A., Kang, B., Feng, J., and Mannor, H.
X. S. Ensemble robustness and generalization of stochastic learning algorithms. In Workshop of International
Conference on Learning Representations (ICLR), 2018.
Zeiler, M. and Fergus, R. Stochastic pooling for regularization of deep convolutional neural networks. In International Conference on Learning Representation (ICLR),
2013.