船井情報科学振興財団

留学報告書 2016 年 6 月

On Deep Learning
川口賢司
今回の留学報告書では、ディープラーニング（深層学習）の論文を簡単に紹介してみようと思います。ディープラーニングは実用が進んでいる分野ですが、一方で、理論的な説明ができない現象が多い分野でもあります。今回紹介する論文は、ディープラーニングの理論に関するものです。

This memo informally introduces some context and implications of the following technical report at MIT:Deep Learning without Poor Local Minima [1](1) A very rough, informal summary of the context

I

n deep learning literature, poor local minima have been considered to be not a big problem by many
practitioners since a long time ago. But, there is not yet a complete picture of why this is the case. Thispaper [1] provides a theory to explain the reason. It also partially provides a justification of why weshould care about saddle points in a sense.
(2) Abstract [1]this paper,
we aprovea conjecture
publishedin 1989
and alsoaddress
partiallyan openn thisnpaper,
we prove
conjecture
publishedin 1989 andalso partiallyaddress an open problem announced at the Conference on Learning Theory
problemannounced
the Conferenceon Learning
Theoryadopted(COLT)from2015.
Under the(COLT)
2015. atUnderthe independence
assumption
recentindependence assumption adopted from recent work, we prove the following statements
work, we prove the following statements for an expected loss function of afor an expected loss function of a deep nonlinear neural network: 1) the function is nondeep nonlinear neural network: 1) the function is non-convex and nonconvex and non-concave, 2) every local minimum is a global minimum, 3) every criticalminimum
is a global3) everycriticalsaddle
point points
point concave,that is not2)a every
globallocal
minimum
is a saddle
point,minimum,and 4) there
exist “bad'”thatis nota global(withminimum
is a saddle
point, and4) there
exist
for thedeep
networks
more than
three layers)
whereasthereis “bad'”no badsaddle
saddle point
pointsforthedeep
networks(withmorethanthreelayers)
whereasthere
no four
for the shallow networks (with three layers). Moreover, we prove that
theissamebad saddle
for theshallow
networks(withthreestatements
hold point
for deep
linearneuralnetworks
withanylayers).
depth, Moreover,any widthsweand nounrealistic
we present
instance,which
we canwith
answer toproveassumptions.
that the same As
foura result,statements
hold forandeep
linear forneuralnetworks
the following
question:
howdifficult
todirectly
train
adeep
modelintheory?
It is more
any depth, any widths and no unrealistic assumptions. As a result, we present
difficultthan the classicallearningtomodels(becauseof the non-convexity),
an instance,
for whichmachine
we can answer
the following
question:
how difficult but
not toodifficult(becauseofthenonexistence
ofpoorlocal
minima).
to directly train a deep model in theory? It is more difficult than theFurthermore,classical theexistence of bad saddle points would suggest an open problem. We note that even thoughmachine learning models (because of the non-convexity), but not too difficult
we have advanced the theoretical foundations of deep learning, there is still a gap between(because of the nonexistence of poor local minima). Furthermore, thetheory and practice.
existence of bad saddle points would suggest an open problem. We note thateven though we have advanced the theoretical foundations of deep learning,
there is still a gap between theory and practice.

On Deep Learning

I I1

(3) “No poor local minima” does not mean “training is easy”

Even if there is no poor local minimum, we can have bad saddle points, plateaus, and the illconditioning problem. Indeed, these problems can create training behaviors in practice that can be missdiagnosed as being trapped around poor local minimum even if there is no poor local minimum.
(4) Beyond over-parameterization assumption and linearity assumptionM

any great new theoretical papers are around now on the topic of local minima. These theoretical
papers often make some simplification assumptions. Major simplification assumptions are overparameterization assumption and linearity assumption.
Some people would claim that the linearity assumption is more reasonable than over-parameterization(because over-parameterization results in a simple model that would fail on the unseen new data
especially in high-dimension) and others would claim the opposite (because linearity assumption results
in a simple model). But, both of these assumptions are meant to be unreasonable in different ways andsuch claims seem to be not constructive.
This new paper, Deep Learning without Poor Local Minima [1], uses a weaker assumption than theprevious papers and provides several theoretical results about local minima and saddle points.

References

On Deep Learning[1] Kawaguchi, Kenji. Deep Learning without Poor Local Minima. Massachusetts Institute ofTechnology, Technical Report, MIT-CSAIL-TR-2016-005.

2