1

Optimization on Product Submanifolds of
Convolution Kernels
Mete Ozay, Takayuki Okatani
Graduate School of Information Sciences, Tohoku University, Sendai, Miyagi, Japan.

arXiv:1701.06123v2 [] 27 Nov 2017

{mozay,okatani}@vision.is.tohoku.ac.jp

✦

Abstract—Recent advances in optimization methods used for training
convolutional neural networks (CNNs) with kernels, which are normalized according to particular constraints, have shown remarkable
success. This work introduces an approach for training CNNs using
ensembles of joint spaces of kernels constructed using different constraints. For this purpose, we address a problem of optimization on
ensembles of products of submanifolds (PEMs) of convolution kernels.
To this end, we first propose three strategies to construct ensembles of
PEMs in CNNs. Next, we expound their geometric properties (metric and
curvature properties) in CNNs. We make use of our theoretical results by
developing a geometry-aware SGD algorithm (G-SGD) for optimization
on ensembles of PEMs to train CNNs. Moreover, we analyze convergence properties of G-SGD considering geometric properties of PEMs.
In the experimental analyses, we employ G-SGD to train CNNs on Cifar10, Cifar-100 and Imagenet datasets. The results show that geometric
adaptive step size computation methods of G-SGD can improve training
loss and convergence properties of CNNs. Moreover, we observe that
classification performance of baseline CNNs can be boosted using GSGD on ensembles of PEMs identified by multiple constraints.

1

I NTRODUCTION

In the recent works [4], [5], [8], [10], [13], [16], [17], [19], [22],
[23], several methods have been suggested to train deep neural
networks using kernels (weights) with various normalization constraints to boost their performance. Spaces of normalized kernels
have been explored using Riemannian manifolds (e.g. the Stiefel),
and stochastic optimization algorithms have been employed to
train CNNs using kernel manifolds in [7], [14], [15], [21].
In this work, we suggest an approach for training CNNs using
multiple constraints on kernels in order to learn a richer set of
features compared to the features learned using single constraints.
We address this problem by optimization on ensembles of products
of different kernel submanifolds (PEMs) that are identified by
different constraints of kernels. However, if we employ the aforementioned Riemannian SGD algorithms [6], [7], [21] on PEMs
to train CNNs, then we observe early divergence, vanishing and
exploding gradients problems. Therefore, we elucidate geometric
properties of PEMs to assure convergence to local minima while
training CNNs using our proposed geometry-aware stochastic
gradient descent (G-SGD). Our contributions are summarized as
follows:
1) We explicate the geometry of space of convolution kernels
defined by multiple constraints. For this purpose, we explore
the relationship between geometric properties of PEMs, such

as sectional curvature, geodesic distance, and gradients computed at PEMs, and those of component submanifolds of
convolution kernels in CNNs (see Lemma 3.2 in Section 3).
2) We propose an SGD algorithm (G-SGD) for optimization on
different ensembles of PEMs (Section 3) by generalizing the
SGD methods employed on kernel submanifolds [14], [15],
[21]. Next, we explore the effect of geometric properties
of the PEMs on the convergence of the G-SGD using our
theoretical results. Then, we employ the results for adaptive
computation of step size of the SGD (see Theorem 3.3
and Corollary 3.4). Moreover, we provide an example for
computation of a step size function for optimization on
PEMs identified by the sphere (Corollary 3.4). In addition,
we propose three strategies in order to construct ensembles
of identical and non-identical kernel spaces according to
their employment on input and output channels in CNNs
in Section 2. To the best of our knowledge, our proposed
G-SGD is the first algorithm which performs optimization on
different ensembles of PEMs to train CNNs with convergence
properties.
3) We experimentally analyze convergence properties and classification performance of CNNs on benchmark image classification datasets such as Cifar 10/100 and Imagenet, using various manifold ensemble schemes (Section 4). In the results,
we observe that G-SGD employed on ensembles of PEMs
can boost baseline state-of-the-art performance of CNNs.
Proofs of the theorems, additional results, and implementation
details of the algorithms and datasets are given in the supplemental
material.

2

C ONSTRUCTION OF E NSEMBLES OF PEM S

Suppose that we are given a set of training samples
S = {si = (Ii , yi )}N
i=1 of a random variable s drawn from a
distribution P on a measurable space S, where yi is a class
label of the ith image Ii . An L-layer CNN consists of a set of
Al ×Bl ×Cl Dl
tensors W = {Wl }L
}d=1 , and
l=1 , where Wl = {Wd,l ∈ R
Al ×Bl Cl
1
Wd,l = [Wc,d,l ∈ R
]c=1 is a tensor composed of kernels
(weight matrices) Wc,d,l constructed at each layer l = 1, 2, . . . , L,
for each cth channel c = 1, 2, . . . , Cl and each dth kernel
d = 1, 2, . . . , Dl . At each lth convolution layer, we compute a
1. We use shorthand notation for matrix concatenation such that
C
[Wc,d,l ]c=l1 ≜ [W1,d,l , W2,d,l , ⋯, WCl ,d,l ].

2

feature representation fl (Xl ; Wl ) by compositionally employing
non-linear functions, and convolving an image I with kernels by

fl (Xl ; Wl ) = fl (⋅; Wl ) ○ ⋯ ○ f1 (X1 ; W1 ),

(1)

l
where X1 ∶= I is an image for l = 1, and Xl = [Xc,l ]C
c=1 . The
th
c channel of the data matrix Xc,l is convolved with the kernel
Wc,d,l to obtain the dth feature map Xc,l+1 ∶= X̂d,l by X̂d,l =
Wc,d,l ∗ Xc,l , ∀c, d, l 2 . Given a batch of samples s ⊆ S , we denote
a value of a classification loss function for a kernel ω ≜ Wc,d,l by
L(ω, s), and the loss function of kernels W utilized in the CNN by
L(W , s). Assuming that s contains a single sample, an expected
loss or cost function of the CNN is computed by

L(W) ≜ EP {L(W , s)} = ∫ L(W , s)dP .

(2)

The expected loss L(ω ) for ω is computed by

L(ω ) ≜ EP {L(ω, s)} = ∫ L(ω, s)dP .

W

(3)

(4)

using an SGD algorithm. In the SGD algorithms employed on
kernel submanifolds [14], [15], [21], each kernel is assumed to
reside on an embedded kernel submanifold Mc,d,l at the lth layer
of a CNN, such that ω ∈ Mc,d,l , ∀c, d. In this work, we propose
a geometry-aware SGD algorithm (G-SGD), by generalizing the
SGD algorithms [14], [15], [21] for optimization on ensembles of
different products of the kernel submanifolds, which are defined
next.
Definition 2.1 (Products of embedded kernel submanifolds of
convolution kernels (PEMs) and their ensemble). Suppose that
Gl = {Mι ∶ ι ∈ IGl } is an ensemble of Riemannian kernel submanifolds Mι of dimension nι , which is identified by a set
of indices IGl , ∀l = 1, 2, . . . , L. More concretely, IGl contains
indices each of which represents an identity number (ι) of a kernel
that resides on a manifold Mι at the lth layer. In addition, a
m
subset IG
⊆ IGl , m = 1, 2, . . . , M , is used to determine a subset
l
m
Gl ⊆ Gl of kernel submanifolds which will be aggregated to
construct a PEM, and satisfies the following properties:
● Each subset of indices contains at least one kernel such that
m
IG
≠ ∅, for each m = 1, 2, . . . , M .
l
m
● The set of indices IGl is covered by the subsets IG such that
l
M

m
IGl = ⋃ IG
.
l
m=1
● If kernels are not shared among PEMs such that ensembles
m
m̄
are constructed using non-overlapping sets, then IG
∩ IG
=
l
l
∅ for m ≠ m̄.
● If kernels are shared among PEMs such that ensembles are
m
m̄
constructed using overlapping sets, then IG
∩ IG
≠ ∅ for
l
l
m ≠ m̄.
m
A Gm
l product manifold of convolution kernels (Gl -PEM) conth
structed at the l layer of an L-layer CNN, denoted by MGm
,
l
is a product of embedded kernel submanifolds belonging to Gm
l
which is computed by

MGm
= ⨉ Mι ,
l
m
ι∈IG

l

2. We ignore the bias terms in the notation for simplicity.

We compute a PEM MGm
using component submanifolds
l
m
Mι in (5) utilizing IG
⊆ IGl , m = 1, 2, . . . , M , and construct
l
ensembles of PEMs Gl using IGl . Recall that, at each lth layer of
an L-layer CNN, we compute a convolution kernel ωι ≜ Wc,d,l ,
c ∈ Λl , Λl = {1, 2, . . . , Cl }, d ∈ Ol , Ol = {1, 2, . . . , Dl }. We
first choose A subsets of indices of input channels Λa ⊆ Λl , a =
1, 2, . . . , A and B subsets of indices of output channels Ob ⊆
A

For a finite set of samples S , L(W) is approximated by an
∣S ∣
empirical loss ∣S1 ∣ ∑i=1 L(W , si ), where ∣S ∣ is the size of S
(similarly, L(ω ) is approximated by the empirical loss for ω ).
Then, feature representations are learned by solving

min L(W)

where ⨉ is the topological Cartesian product, and therefore MGm
l
is a product topology. Each Mι ∈ Gm
l is called a component
submanifold of MGm
. A kernel ωGm
∈ MGm
is then obtained
l
l
l
m
by concatenating kernels belonging to Mι , ∀ι ∈ IG
, using
l
m
m
m ∣ ), where ∣I
ωGm
=
(
ω
,
ω
,
⋯
,
ω
∣
is
the
cardinality
of
IG
.
1
2
∣I
Gl
l
l
Gl
A Gl -PEM is called an ensemble of PEMs constructed using (5)
for m = 1, 2, . . . , M .
∎

(5)

B

Ol , b = 1, 2, . . . , B, such that Λl = ⋃ Λa and Ol = ⋃ Ob . Then,
a=1
b=1
we propose three strategies for determination of index sets (see
Figure 1);
1) PEMs for input channels (PI): For each cth input channel,
c
c
we construct IGl = ⋃ IG
, where IG
= Ob × {c} and the
l
l
Cl

Cartesian product Ob × {c} preserves the input channel index,
∀b, c.
2) PEMs for output channels (PO): For each dth output channel,
c=1

d
d
we construct IGl = ⋃ IG
, where IG
= Λa × {d} and the
l
l
Dl

Cartesian product Λa ×{d} preserves the output channel index,
∀a, d.
a,b
=
3) PEMs for input and output channels (PIO): We construct IG
l
b
a
b
a
IGl ∪ IGl , where IGl = {Λa × a} and IGl = {Ob × b} such
d=1

that IGl =

A,B

⋃

a=1,b=1

a,b
IG
.
l

Example 2.2. An illustration of employment of PI, PO and PIO
at the lth layer of a CNN is given in Figure 1. Suppose that we
have a kernel tensor of size 3 × 3 × 4 × 6 where the number of input
and output channels is 4 and 6. In total, we have 4 ∗ 6 = 24 kernel
matrices of size 3 × 3. An example of construction of an ensemble
of PEMs is as follows.
1) PI: For each of 4 input channels, we split a set of 6 kernels
associated with 6 output channels into two subsets of 3 kernels.
Choosing the sphere (Sp) for the first subset, we construct
a PEM as a product of 3 Sp using (5). That is, each of 3
component manifolds Mι , ι = 1, 2, 3, of the PEM is a sphere.
Similarly, choosing the Stiefel (St) for the second subset,
we construct another PEM as a product of 3 St (each of 3
component manifolds Mι , ι = 1, 2, 3, of the second PEM is a
Stiefel manifold.). Thus, at this layer, we construct an ensemble
of 4 PEMs of 3 St and 4 PEMs of 3 Sp.
2) PO: For each of 6 output channels, we split a set of 4 kernels
corresponding to the input channels into two subsets of 2
kernels. We choose the Sp for the first subset, and we construct
a PEM as a product of 2 Sp using (5). We choose the St for the
second subset, and we construct a PEM as a product of 2 St.
Thereby, we have an ensemble consisting of 6 PEMs of St and
6 PEMs of Sp.
3) PIO: We split the set of 24 kernels into 10 subsets. For each of 6
output channels, we split the set of kernels corresponding to the
input channels into 3 subsets. We choose the Sp for 2 subsets
each containing 3 kernels, and 3 subsets each containing 2

3

Fig. 1: An illustration for employment of the proposed PI, PO and PIO strategies at the lth layer of a CNN.

kernels. We choose the St similarly for the remaining subsets.
Then, our ensemble contains 5 PEMs of St and 5 PEMs of Sp.
Our framework can be used to model both overlapping and
non-overlapping sets. If ensembles are constructed using overlapping sets, then kernels having different constraints can be
applied to the same input or output channels. For example, kernels
belonging to a PEM of 3 St and kernels belonging to a PEM of
3 Sp can be applied to the same output (input) channel for PI
(PO) in the previous example (see Figure 1). More complicated
configurations can be obtained using PIO. In the experiments,
we selected non-overlapping sets for simplicity. We consider
theoretical and experimental analyses of overlapping sets as a
future work.

3 O PTIMIZATION ON E NSEMBLES OF PEM S USING
G EOMETRY - AWARE SGD IN CNN S
If an SGD is employed on non-linear kernel submanifolds, then
the gradient descent is generally performed by three steps; i)
projection of gradients on tangent spaces of the submanifolds,
ii) movement of kernels on the tangent spaces in the gradient
descent direction, and iii) projection of the moved kernels onto
the submanifolds [21]. These steps are determined according to
the geometric properties of the submanifolds, such as sectional
curvature and metric properties. For example, the Euclidean space
has zero sectional curvature, i.e. it is not curved (flat). Thereby,
these steps can be performed using a single step if an SGD
employs kernels residing on the Euclidean space. However, if
kernels belong to the unit sphere, then the kernel space is curved
by constant positive curvature. Moreover, a different tangent space
is computed at each kernel located on the sphere. Therefore,
nonlinearity of operations and transformations applied on kernels
implied by curvature and metric of kernel spaces are used for
gradient descent in the aforementioned three steps. In addition,
martingale properties of stochastic processes defined by kernels
are determined by geodesics, metrics, gradients projected at tangent spaces and injectivity radius of kernel spaces (see proofs of
Theorem 3.3 and Corollary 3.4 in the supp. mat. for details).
Geometric properties of PEMs can be different from that of the
component submanifolds of PEMs, even if they are constructed
using identical submanifolds. For example, we observe locally
varying curvatures when we construct PEMs of spheres (see Figure 2). Kernel spaces with more complicated geometric properties
can be obtained using the proposed strategies (PI, PO, PIO),
especially by constructing ensembles of PEMs of non-identical
submanifolds (see Section 4 for details and examples). Thus, as

the complexity of geometry of kernel spaces increases, their effect
on performance and convergence of SGD gradually increases.
In order to address these problems and consider geometric
properties of kernel submanifolds for training of CNNs, we
propose a geometry aware SGD (G-SGD). We employ metric
properties of PEMs to perform gradient descent steps of GSGD, and use curvature properties PEMs to explore convergence
properties of G-SGD. We explore metric and curvature properties
of PEMs in the next theorem.
Definition 3.1 (Sectional curvature of component submanifolds).
Let X(Mι ) denote the set of smooth vector fields on Mι . The
sectional curvature of Mι associated with a two dimensional
subspace T ⊂ Tωι Mι is defined by
cι =

⟨Cι (Xωι , Yωι )Yωι , Xωι ⟩

⟨Xωι , Xωι ⟩ ⟨Yωι , Yωι ⟩ − ⟨Xωι , Yωι ⟩2

(6)

where Cι (Xωι , Yωι )Yωι is the Riemannian curvature tensor3 , ⟨⋅, ⋅⟩
is an inner product, Xωι ∈ X(Mι ) and Yωι ∈ X(Mι ) form a basis
of T.
∎
Lemma 3.2 (Metric and curvature properties of PEMs). Suppose that uι ∈ Tωι Mι and vι ∈ Tωι Mι are tangent vectors belonging to the tangent space Tωι Mι computed at
m
ωι ∈ Mι , ∀ι ∈ IG
, m = 1, 2, . . . , M . Then, tangent vecl
m
tors uGm
∈
T
M
∈ TωGm MGm
are computed at
ωGm Gl and vGm
l
l
l
l
l
m
m
m ∣)
ωGm
∈
M
by
concatenation
as
u
=
(
u
1 , u2 , ⋯, u∣IG
Gl
Gl
l
l
m ∣ ). If each kernel submanifold Mι
and vGm
= (v1 , v2 , ⋯, v∣IG
l
l
is endowed with a Riemannian metric dι , then a Gm
l -PEM is
endowed with the metric dGm
computed
by
l
dGm
(uGm
, vGm
) = ∑ dι (uι , vι ).
l
l
l

(7)

m
ι∈IG

l

In addition, suppose that C̄ι is the Riemannian curvature tensor
m
field (endomorphism) [20] of Mι , xι , yι ∈ Tωι Mι , ∀ι ∈ IG
l
defined by

C̄ι (uι , vι , xι , yι ) = ⟨Cι (U, V )X, Y ⟩ωι ,

(8)

where U, V, X, Y are vector fields such that Uωι = uι , Vωι = vι ,
Xωι = xι , and Yωι = yι . Then, the Riemannian curvature tensor
field C̄Gl of MGl is computed by

C̄Gm
(uGm
, vGm
, xGm
, yGm
) = ∑ C̄ι (uι , vι , xι , yι ),
l
l
l
l
l
m
ι∈IG

l

3. Additional definitions are given in the supp. mat.

(9)

4

(a) S2 .

(b) T2 = S1 × S1 .

(c) S1 × R.

Fig. 2: (a) An orthonormalized convolution kernel ω ∈ R3×1
(ω ∈ RA×B ) resides on a two-sphere S2 (SAB −1 ) which has
constant positive sectional curvature, 1. (b) A kernel ω = (ω1 , ω2 ),
where each ωι ∈ R2×1 , ι = 1, 2, belongs to a circle S1 , resides on
a two-torus T2 with varying curvature. (c) If ω1 ∈ S1 (ω1 ∈ Sp )
and ω2 ∈ R (ω2 ∈ Rq−p ), then ω resides on a cylinder S1 × R
with varying curvature (q -cylinder Sp × Rq−p ). In general, if
a convolution kernel ω ∈ M resides on a PEM M, then M
does not admit a metric with negative sectional curvature (see
Lemma 3.2). Therefore, geometric properties of PEMs, which
can be composed of non-identical component submanifolds, may
crucially affect convergence of SGD methods for training of CNNs
(see Theorem 3.3 and Corollary 3.4).
m ∣ ) and yGm = (y1 , y2 , ⋯, y∣I m ∣ ).
where xGm
= (x1 , x2 , ⋯, x∣IG
l
l
Gl
l
Moreover, MGm
has never strictly positive sectional curvature
l
cG m
in the metric (7). In addition, if MGm
is compact, then MGm
l
l
l
does not admit a metric with negative sectional curvature cGm
.∎
l

We compute the metric of a Gm
using the metrics
l -PEM MGm
l
identified on the component manifolds Mι employing (7) given
in Lemma 3.2. In addition, we use the Riemannian curvature and
sectional curvature of the MGm
given in Lemma 3.2 to analyze
l
convergence of our proposed G-SGD, and to compute adaptive
step size.
Note that some sectional curvatures vanish on the MGm
by
l
the lemma. For instance, suppose that each Mι is a unit twosphere S2 , ∀ι ∈ IGl (see Figure 2.a). Then, MGm
computed
l
by (5) has unit curvature along two-dimensional subspaces of its
tangent spaces, called two-planes. On the other hand, MGm
has
l
zero curvature along all two-planes spanning exactly two distinct
spheres. Therefore, learning rates need to be computed adaptively
according to sectional curvatures at each layer of the CNN and
at each epoch of the G-SGD for each kernel ω on each manifold
MGm
.
l
3.1 Optimization using G-SGD in CNNs
An algorithmic description of our proposed geometry-aware SGD
(G-SGD) is given in Algorithm 1. At the initialization of the GSGD, we identify the component embedded kernel submanifolds
Mι according to the constraints that will be applied on the
kernels ωι ∈ Mι . For instance, we employ an orthonormalization
constraint ∥ωι ∥F = 1 for kernels ωι residing on nι dimensional
unit sphere Mι ≡ Snι , where ∥ ⋅ ∥F is the Frobenius norm [2]4 .
, each
When we employ a G-SGD on a Gm
l -PEM MGm
l
t
m
kernel ωG
m ∈ MGm is moved on the Gl -PEM in the descent
l
l
direction of gradient of loss at each tth step of the G-SGD. More
t
precisely, direction and amount of movement of a kernel ωG
m are
l

4. In the experimental analyses, we use the oblique and the Stiefel manifolds
as well as the sphere and the Euclidean space to identify subcomponent
manifolds Mι .

Algorithm 1 Optimization using G-SGD on an ensemble of
PEMs.
1: Input: T (number of iterations), S (training set),
Θ (set of hyperparameters), L (a loss function),
m
IG
⊆ IGl , ∀m = 1, 2, . . . , M, l = 1, 2, . . . , L.
l
2: Initialization: Construct an ensemble of products of kernel
t
submanifolds Gl , and initialize kernels ωG
m ∈ MGm using (5)
l
l
m
with IGl ⊆ IGl , ∀m, l.
3: for each iteration t = 1, 2, . . . , T do
4:
for each layer l = 1, 2, . . . , L do
t
m
5:
Compute the gradient gradE L(ωG
m ), ∀Gl .
l

6:
7:
8:
9:
10:
11:
12:

t
m
t
gradL(ωG
m ) ∶= Πω t (gradE L(ωGm ), Θ), ∀Gl .
l
l
l
t
m
vt ∶= h(gradL(ωG
m ), g (t, Θ)), ∀Gl .
l
t
m
t+1
ωGm ∶= φωt m (vt ), ∀ωGm , ∀Gl .
l
l
G
l
t ∶= t + 1.
end for
end for
T
L
m
Output: A set of estimated kernels {ωG
m }l=1 , ∀Gl ⊆ Gl .
l

determined at the tth step and the lth layer by the following steps
of Algorithm 1:
t
1) Line 6: Using Lemma 3.2, the gradient gradE L(ωG
m ), which
l
is obtained using back-propagation from the upper layer, is
t
projected onto the tangent space Tωt m MGm
at gradL(ωG
m ),
l
=
where Tωt m MGm
l
G
l

2) Line

7:

G
l

l

t Mι .
⨉ Tωι,l

ι∈IGm
l

Movement

of

t
h(gradL(ωG
m ), g (t, Θ))
l

t
ωG
m
l

on

T ω t m MGm
l
G
l

using

computed by

t
m ), g (t, Θ)) = −
h(gradL(ωG
l

g (t, Θ)
t
m
t ) gradL(ωGl ), (10)
g(ωG
m
l

where g (t, Θ) is the learning rate that satisfies
∞

∞

t=0

t=0

2
∑ g (t, Θ) = +∞ and ∑ g (t, Θ) < ∞,

(11)

t
t
2 t
g(ωG
=
max{1, Γt1 } 2 , Γt1
=
(RG
m)
m ) Γ2 ,
l
l
t
2
t
t
Γt2 = max{(2ρtGm + RG
m ) , (1 + cGm (ρGm + RGm ))},
l
l
l
l
l
t
t
t
ρGm ≜ ρ(ωGm , ω̂Gm
) is the geodesic distance between ωG
m
l
l
l
l
m , cGm is the sectional
and a local minima ω̂Gm
on
M
G
l
l
l
t
t
curvature of MGm
, RG
m ≜ ∥gradL(ωGm )∥2 which can be
l
l
l
computed using Lemma 3.2 by
1

1
2

t 2
t
m )∥2 = (
∥gradL(ωG
∑ gradL(ωl,ι ) ) .
l

(12)

ι∈IGm
l

3) Line 8: Projection of the moved kernel at vt onto the manifold
t+1
(vt ) is
MGm
using φωt m (vt ) to compute ωG
m , where φω t
m
l
G
l

l

G
l

an exponential map, or a retraction which is an approximation
of the exponential map [3].
t
The denominator g(ωG
m ) used for computation of the step
l
size in (10) is employed as a regularizer to control the change
t
of gradient gradL(ωGm ) at each step of G-SGD. This property
l
is examined in the experimental analyses for PEMs of different
t
manifolds. For computation of g(ωG
m ), we use (12) utilizing
l
Lemma 3.2. Unlike related works, kernels residing on each PEM
are moved and projected jointly on the PEMs in G-SGD, by which
we can employ their interaction using the corresponding gradients

5

considering nonlinear geometry of manifolds. G-SGD can perform
optimization on PEMs and their ensemble according to sets
Gm
l , ∀m, recursively. Thereby, G-SGD can consider interactions
between component manifolds as well as those between PEMs in
an ensemble. SGD methods studied in the literature do not have
assurance of convergence when it is applied to optimization on
ensembles of PEMs. Employment of (10) and (11) at line 7, and
retractions at line 8 are essential for assurance of convergence as
explained next.
3.2 Convergence Properties of G-SGD
In some machine learning tasks, such as clustering [6], [24], the
geodesic distance ρtGm can be computed in closed form. However,
l
a closed form solution may not be computed using CNNs due
to the challenge of computation of local minima. Therefore, we
provide an asymptotic convergence property for Algorithm 1 in
the next theorem.

TABLE 1: Results for Resnet-44 on Cifar-10 with DA.
Model

Class. Error(%)

Euc. [9]
Euc. [21]
Euc. †
Sp/Ob/St [21]
Sp/Ob/St †
PEMs of Sp/Ob/St
PI for PEMs of Sp/Ob/St
PI (Euc.+Sp/Euc.+St/Euc.+Ob)
PI (Sp+Ob/Sp+St/Ob+St)
PI (Sp+Ob+St/Sp+Ob+St+Euc.)
PO for PEMs of Sp/Ob/St
PO (Euc.+Sp/Euc.+St/Euc.+Ob)
PO (Sp+Ob/Sp+St/Ob+St)
PO (Sp+Ob+St/Sp+Ob+St+Euc.)
PIO for PEMs of Sp/Ob/St
PIO (Euc.+Sp/Euc.+St/Euc.+Ob)
PIO (Sp+Ob/Sp+St/Ob+St)
PIO (Sp+Ob+St/Sp+Ob+St+Euc.)

7.17
7.16
7.05
6.99/6.89/6.81
6.84/6.87/ 6.73
6.81/6.85/ 6.70
6.82/6.81/ 6.70
6.89/6.84/6.88
6.75/6.67/6.59
6.31/6.34
6.77/6.83/ 6.65
6.85/6.78/6.90
6.62/6.59/6.51
6.35/6.22
6.71/6.73/ 6.61
6.95/6.77/6.82
6.21/6.19/6.25
5.95/5.92

Theorem 3.3. Suppose that there exists a local minimum
ω̂Gm
∈ MGm
, ∀Gm
⊆ Gl , ∀l, and ∃ǫ > 0 such that
l
l
l

inf

1
ρtGm >ǫ 2
l

t
⟨φωt m (ω̂Gm
)−1 , ∇L(ωG
m )⟩ < 0, where φ is an exl
l

G
l

ponential map or a twice continuously differentiable retraction,
and ⟨⋅, ⋅⟩ is the inner product. The loss function and the gradient
a.s.
t
converges almost surely (a.s.) by L(ωG
ÐÐ→ L(ω̂Gm
), and
m) Ð
l

t
∇L(ωG
m) Ð
ÐÐ→ 0, for each MGm
, ∀l.
l
a.s.

l

l

t→∞

∎

t→∞

Theorem 3.3 assures convergence of the G-SGD (Algorithm 1)
to minima. For implementation of G-SGD, we use the result
given in Lemma 3.2 for PEMs to employ sectional curvatures.
Although sectional curvatures of non-identical embedded kernel
submanifolds can be different [21], Lemma 3.2 assures existence
of zero sectional curvature in PEMs along their tangent spaces. In
the next theorem, we provide an example for computation of a step
size function g(⋅) for component embedded kernel submanifolds
determined by the sphere using the result given in Lemma 3.2, and
explore its convergence property using Theorem 3.3.
Corollary 3.4. Suppose that Mι are identified by nι ≥ 2 dimensional unit sphere Snι , and ρtGm ≤ ĉ−1 , where ĉ is an upper bound
l
t
on the sectional curvatures of MGm
, ∀l at ωG
m ∈ MGm , ∀t. If
l
l
l
step size is computed using (10) with
2
2
t
t
t
m ) = (max{1, (RGm ) (2 + RGm ) }) 2 ,
g(ωG
l
l
l
1

(13)

t
t
), and ∇L(ωG
ÐÐ→ 0, for each
then L(ωG
ÐÐ→ L(ω̂Gm
m) Ð
m) Ð
l
l
l
t→∞
t→∞
MGm
, ∀l.
∎
l
a.s.

a.s.

In the experimental analyses, we use different step size functions and analyze convergence properties and performance of
CNNs trained using G-SGD by relaxing assumptions of Theorem 3.3 and Corollary 3.4 for different CNN architectures and
benchmark image classification datasets.

4

and 100 classes, respectively. The Imagenet dataset consists of 103
classes (12 × 104 training and 5 × 104 validation images).
We construct ensembles of PEMs using the sphere (Sp), the
oblique (Ob) and the Stiefel (St) manifolds. We also use the
kernels residing on the ambient Euclidean space of embedded
kernel submanifolds (Euc.). In order to preserve the task structure
(classification of RGB images), we employed PI for the layers
l = 2, 3, . . . , L considering the RGB space of images, PO for
l = 1, 2, . . . , L − 1 considering the number of classes learned
at the top Lth layer of a CNN, and PIO for l = 2, . . . , L − 1.
Suppose that we have a set of Nl kernels Nl with ∣Nl ∣ = Nl
and ∣IGl ∣ = Nl at the lth layer of a CNN. In the construction
of ensembles, we employ PI, PO and PIO using a kernel set
splitting (KSS) scheme. In KSS, we split the kernel set Nl into
m
M subsets Nm
l ⊂ Nl , ∀m = 1, 2, . . . , M , where kernels ω ∈ Nl
m
th
belonging to Nl reside on the m PEM MGm
identified by
l
I Gm
which
is
determined
according
to
PI,
PO
and
PIO,
∀m. For
l
the sake of simplicity of the analyses, we split the kernel set into
l
subsets with size N
M in KSS, while the proposed schemes enable
us to construct new kernel sets with varying size. Implementation
details of G-SGD for different ensembles and Resnets, data preprocessing details of the benchmark datasets and additional results
are given in the supp. mat.

E XPERIMENTAL A NALYSES

We examine the proposed G-SGD method for training of state-ofthe-art CNNs, called Residual Networks (Resnets) [9], equipped
with different number of layers and kernels. We use three benchmark RGB image classification datasets, namely Cifar-10, Cifar100 and Imagenet [18]. The Cifar-10 and Cifar-100 datasets
consist of 5 × 104 training and 104 test images belonging to 10

4.1 Analysis of Classification Performance on Benchmark Datasets
We analyze classification performance of CNNs trained using GSGD on benchmark Cifar-10, Cifar-100 and Imagenet datasets.
In order to construct ensembles of kernels belonging to Euc., Sp,
St and Ob using KSS, we increase the number of kernels used
in CNNs to 24 and its multiples (see the supp. mat.). We use
other hyperparameters of CNNs as suggested in [9], [12], [21]. We
depict performance of our implementation of CNNs for baseline
geometries (Euc., Sp, St and Ob) by † marker in the tables. For
t
computation of g(ωG
m ), we used
l

t
g(ωGm
)
l

) (2 + RGm
) }) 2 , ∀m, l
= (max{1, (RGm
l
l
t

2

t

2

1

(14)

as suggested in Corollary 3.4. Implementation details are given in
the supp. mat.

6

TABLE 2: Results for Resnet-18 which are trained using Imagenet
for single crop validation error rate (%).
Model

Top-1 Error (%)

Euc. [21]
Euc. †
Sp/Ob/St[21]
Sp/Ob/St †
PEMs of Sp/Ob/St
PI for PEMs of Sp/Ob/St
PI (Euc.+Sp/Euc.+St/Euc.+Ob)
PI (Sp+Ob/Sp+St/Ob+St)
PI (Sp+Ob+St/Sp+Ob+St+Euc.)
PO for PEMs of Sp/Ob/St
PO (Euc.+Sp/Euc.+St/Euc.+Ob)
PO (Sp+Ob/Sp+St/Ob+St)
PO (Sp+Ob+St/Sp+Ob+St+Euc.)
PIO for PEMs of Sp/Ob/St
PIO (Euc.+Sp/Euc.+St/Euc.+Ob)
PIO (Sp+Ob/Sp+St/Ob+St)
PIO (Sp+Ob+St/Sp+Ob+St+Euc.)

30.59
30.31
29.13/28.97/28.14
28.71/28.83/28.02
28.70/28.77/28.00
28.69/28.75/27.91
30.05/29.81/29.88
28.61/28.64/28.49
27.63/27.45
28.67/28.81/27.86
29.58/29.51/29.90
28.23/28.01/28.17
27.81/27.51
28.64/28.72/27.83
29.19/28.25/28.53
28.14/27.66/27.90
27.11/27.07

We examine classification performance of Resnets with 44
layers (Resnet-44) and 18 layers (Resnet-18) on Cifar-10 with
data augmentation (DA) and Imagenet in Table 1 and Table 2,
respectively. The results show that performance of CNNs are
boosted by employing ensembles of PEMs (denoted by PI, PO
and PIO for PEMs) using G-SGD compared to the employment of
baseline Euc. We observe that PEMs of component submanifolds
of identical geometry (denoted by PEMs of Sp/St/Ob), and their
ensembles (denoted by PI, PO, PIO for PEMs of Sp/St/Ob) provide better performance compared to employment of component
submanifolds (denoted by Sp/Ob/St) [21]. For instance, we obtain
28.64%, 28.72% and 27.83% error using PIO for PEMs of Sp, Ob
and St in Table 2, respectively. However, the error obtained using
Sp, Ob and St is 28.71%, 28.83% and 28.02%, respectively.
In addition, we obtain 0.28% and 2.06% boost of the performance by ensemble of the St with Euc. (6.77% and 28.25% using
PIO for Euc.+St, respectively) for the experiments on the Cifar10 and Imagenet datasets using the PIO scheme in Table 1 and
Table 2, respectively. Moreover, we observe that construction of
ensembles using Ob performs better for PI compared to PO. For
instance, we observe that PI for PEMs of Ob provides 6.81%
and 28.75% while PO for PEMS of Ob provides 6.83% and
28.81% in Table 1 and Table 2, respectively. We may associate
this result with the observation that kernels belonging to Ob are
used for feature selection and modeling of texture patterns with
high performance [1], [21]. However, ensembles of St and Sp
perform better for PO (6.59% and 28.01% in Table 1 and Table 2)
compared to PI (6.67% and 28.64% in Table 1 and Table 2) on
kernels employed on output channels.
It is also observed that PIO performs better than PI and PO
in all the experiments. We observe 1.13% and 3.24% boost by
construction of an ensemble of four manifolds (Sp+Ob+St+Euc.)
using the PIO scheme in Table 1 (5.92%) and Table 2 (27.07%),
respectively. In other words, ensemble methods boost the performance of large-scale CNNs more for large-scale datasets (e.g.
Imagenet) consisting of larger number of samples and classes
compared to the performance of smaller CNNs employed on
smaller datasets (e.g. Cifar-10). This result can be attributed to
enhancement of sets of features learned using multiple constraints
on kernels.

We analyze this observation by examining the performance
of larger CNNs consisting of 110 layers on Cifar-10 and Cifar100 datasets with and without using DA in Table 3. The results
show that employment of PEMs can boost the performance of
CNNs that use component submanifolds (e.g. PEMs of Sp, Ob
and St) more for larger networks (Table 3) compared to smaller
networks (Table 1 and Table 2). Moreover, employment of PIO for
PEMs of Sp+Ob+St+Euc. boosts the performance of CNNs that
use Euc. more for Cifar-100 (3.55% boost in average) compared to
the performance obtained for Cifar-10 (1.58% boost in average).
In addition, we observe that ensembles boost the performance of
CNNs that use DA methods more compared to the performance of
CNNs without using DA.
Our method fundamentally differs from network ensembles.
In order to analyze the results for network ensembles of CNNs,
we employed an ensemble method [9] by voting of decisions of
Resnet 44 on Cifar 10. When CNNs trained on individual Euc,
Sp, Ob, and St are ensembled using voting, we obtained 7.02%
(Euc+Sp+Ob+St) and 6.85% (Sp+Ob+St) errors (see Table 1 for
comparison). In our analyses of ensembles (PI, PO and PIO), each
l
PEM contains N
M kernels, where Nl is the number of kernels used
th
at the l layer, and M is the number of PEMs. When each CNN
in the ensemble was trained using an individual manifold which
contains 14 of kernels (using M = 4 as utilized in our experiments),
then we obtained 11.02% (Euc), 7.76% (Sp), 7.30% (Ob), 7.18%
(St), 9.44% (Euc+Sp+Ob+St) and 7.05% (Sp+Ob+St) errors.
Thus, our proposed methods outperform ensembles constructed by
voting. Additional results are given in the supplemental material.

5

C ONCLUSION AND D ISCUSSION

We introduced and elucidated a problem of training CNNs using
multiple constraints employed on convolution kernels with convergence properties. Following our theoretical results, we proposed
the G-SGD algorithm and adaptive step size estimation methods
for optimization on ensembles of PEMs that are identified by
the constraints. The experimental results show that our proposed
methods can improve convergence properties and classification
performance of CNNs. Overall, the results show that employment
of ensembles of PEMs using G-SGD can boost the performance
of larger CNNs (e.g. RCD and RSD) on large scale datasets
(e.g. Imagenet) more compared to the performance of small and
medium scale networks (e.g. Resnets with 16 and 44 layers)
employed on smaller datasets (e.g. Cifar-10).
In future work, we plan to extend the proposed framework
by development of new ensemble schemes to perform various
tasks such as machine translation and video recognition using
CNNs and Recurrent Neural Networks (RNNs). In addition, the
proposed methods can be applied to other stochastic optimization
methods such as Adam and trust region methods. We believe that
our proposed framework will be useful for researchers to study
geometric properties of parameter spaces of deep networks, and to
improve our understanding of deep feature representations.

R EFERENCES
[1]

[2]

P. A. Absil and K. A. Gallivan. Joint diagonalization on the oblique
manifold for independent component analysis. In Proc. 31st IEEE
Int. Conf. Acoust., Speech Signal Process, volume 5, pages 945–948,
Toulouse, France, May 2006.
P.-A. Absil, R. Mahony, and R. Sepulchre. Optimization Algorithms on
Matrix Manifolds. PUP, Princeton, NJ, USA, 2007.

7

TABLE 3: Classification error (%) for training 110-layer Resnets with constant depth (RCD) and Resnets with stochastic depth (RSD)
using the PIO scheme on Cifar-10 and Cifar-100, with and without using DA.
Model
RCD [11]
(Euc.) †
Sp/Ob/St ([21])
Sp/Ob/St †
PEMs of Sp/Ob/St
PIO for PEMs of Sp/Ob/St
PIO (Euc.+Sp/Euc.+St/Euc.+Ob)
PIO (Sp+Ob/Sp+St/Ob+St)
PIO (Sp+Ob+St/Sp+Ob+St+Euc.)
RSD [11]
Euc. †
Sp/Ob/St [21]
Sp/Ob/St †
PEMs of Sp/Ob/St
PIO for PEMs of Sp/Ob/St
PIO (Euc.+Sp/Euc.+St/Euc.+Ob)
PIO (Sp+Ob/Sp+St/Ob+St)
PIO (Sp+Ob+St/Sp+Ob+St+Euc.)

[3]
[4]
[5]

[6]
[7]
[8]

[9]
[10]
[11]
[12]
[13]
[14]
[15]
[16]

[17]
[18]
[19]
[20]

Cifar-10 w. DA
6.41
6.30
6.22/6.07/5.93
6.05/6.03/5.91
6.00/6.01/5.86
5.95/5.91/5.83
6.03/5.99/6.01
5.97/5.86/5.46
5.25/5.17
5.23
5.17
5.20/5.14/4.79
5.08/5.11/4.73
5.05/5.08/4.69
4.95/5.03/4.62
5.00/5.08/5.14
4.70/4.58/4.90
4.29/4.31

Cifar-100 w. DA
27.22
27.01
26.44/25.99/25.41
26.19/25.87/25.39
25.93/25.74/25.18
25.89/25.71/25.12
25.57/25.49/25.64
24.71/24.96/24.76
23.96/23.79
24.58
24.39
23.77/23.81/23.16
23.69/23.75/23.09
23.51/23.60/23.85
23.47/23.51/23.77
23.69/23.25/23.32
22.84/22.91/22.80
22.19/22.03

P. A. Absil and J. Malick. Projection-like retractions on matrix manifolds.
SIAM Journal on Optimization, 22(1):135–158, 2012.
M. Arjovsky, A. Shah, and Y. Bengio. Unitary evolution recurrent neural
networks. In Proc. of the 33rd Int. Conf. on Mach. Learn. ICML, pages
1120–1128, New York City, NY, USA, June 2016.
D. Arpit, Y. Zhou, B. U. Kota, and V. Govindaraju. Normalization
propagation: A parametric technique for removing internal covariate shift
in deep networks. In Proc. of the 33rd Int. Conf. on Mach. Learn. ICML,
pages 1168–1176, New York City, NY, USA, June 2016.
S. Bonnabel. Stochastic gradient descent on riemannian manifolds. IEEE
Trans. Autom. Control, 58(9):2217–2229, Sept 2013.
M. Cho and J. Lee. Riemannian approach to batch normalization. In
Advances in Neural Information Processing Systems (NIPS), 2017.
M. Cisse, P. Bojanowski, E. Grave, Y. Dauphin, and N. Usunier. Parseval
networks: Improving robustness to adversarial examples. In D. Precup
and Y. W. Teh, editors, Proceedings of the 34th International Conference
on Machine Learning, volume 70 of Proceedings of Machine Learning
Research, pages 854–863, International Convention Centre, Sydney,
Australia, 06–11 Aug 2017. PMLR.
K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image
recognition. In IEEE Int. Conf. on Comp. Vis. Patt. Recog. (CVPR), 2016.
M. Henaff, A. Szlam, and Y. LeCun. Recurrent orthogonal networks
and long-memory tasks. In Proc. of the 33rd Int. Conf. Mach. Learn.,
ICML’16, pages 2034–2042, 2016.
G. Huang, Z. Liu, and K. Q. Weinberger. Densely connected convolutional networks. In Proc. IEEE Conf. Comp. Vis. Patt. Recog. (CVPR),
2017.
G. Huang, Y. Sun, Z. Liu, D. Sedra, and K. Q. Weinberger. Deep
networks with stochastic depth. In Proc. of the 14th European Conf.
Comp. Vis., pages 646–661, Amsterdam, The Netherlands, 2016.
L. Huang, X. Liu, Y. Liu, B. Lang, and D. Tao. Centered weight
normalization in accelerating training of deep neural networks. In The
IEEE International Conference on Computer Vision (ICCV), Oct 2017.
Z. Huang and L. V. Gool. A riemannian network for spd matrix learning.
In Assoc. for the Adv. of Artificial Intelligence (AAAI), Feb 2017.
Z. Huang, C. Wan, T. Probst, and L. V. Gool. Deep learning on lie groups
for skeleton-based action recognition. In Proc. IEEE Conf. Comp. Vis.
Patt. Recog. (CVPR), 2017.
Z. Huang, R. Wang, S. Shan, X. Li, and X. Chen. Log-euclidean metric
learning on symmetric positive definite manifold with application to
image set classification. In Proc. of the 32nd Int. Conf. Mach. Learn.
(ICML-15), pages 720–729, 2015.
G. Klambauer, T. Unterthiner, A. Mayr, and S. Hochreiter. Selfnormalizing neural networks. In Advances in Neural Information Processing Systems (NIPS), 2017.
A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification
with deep convolutional neural networks. In Advances in Neural Information Processing Systems (NIPS), pages 1097–1105, 2012.
A. Kumar, P. Sattigeri, and T. Fletcher. Improved semi-supervised
learning with gans using manifold invariances. In Advances in Neural
Information Processing Systems (NIPS), 2017.
J. Lee. Manifolds and Differential Geometry. Graduate studies in

Cifar-10 w/o DA
13.63
13.57
13.11/12.94/12.88
12.96/12.85/12.79
12.74/12.77/12.74
12.71/12.72/12.69
12.77/12.21/12.92
11.47/11.65/ 11.51
11.29/11.15
11.66
11.40
10.91/10.93/10.46
10.52/10.66/10.33
10.41/10.54/10.25
10.37/10.51/10.19
10.74/10.25/10.93
10.13/10.24/10.06
9.52/9.56

Cifar-100 w/o DA
44.74
44.65
42.51/42.30/40.11
42.13/42.00/39.94
42.02/42.88/39.90
41.68/42.75/39.83
41.90/41.37/41.85
41.49/40.53/40.34
39.53/ 39.35
37.80
37.55
36.90/36.47/35.92
36.71/36.38/35.85
36.40/36.11/35.53
36.33/36.02/35.41
35.76/35.55/35.81
35.66/35.01/35.35
34.49/34.25

mathematics. American Mathematical Society, 2009.
[21] M. Ozay and T. Okatani. Optimization on submanifolds of convolution
kernels in cnns. CoRR, abs/1610.07008, 2016.
[22] T. Salimans and D. P. Kingma. Weight normalization: A simple reparameterization to accelerate training of deep neural networks. In Advances
in Neural Information Processing Systems (NIPS), 2016.
[23] W. Shang, J. Chiu, and K. Sohn. Exploring normalization in deep residual
networks with concatenated rectified linear units. In AAAI Conference
on Artificial Intelligence, 2017.
[24] H. Zhang and S. Sra. First-order methods for geodesically convex
optimization. In Conference on Learning Theory (COLT), Jun. 2016.

