2014/11/12	

第8回全脳アーキテクチャ勉強会
2014年11月10日，グラントウキョウサウスタワー	
 深層学習によるロボットの感覚運動ダイナミクスの学習尾形哲也
早稲田大学	 理工学術院

本発表の内容に関する論文• Kuniaki Noda, Hiroaki Arie, Yuki Suga, and Tetsuya Ogata:
Multimodal Integration Learning of Robot Behavior using Deep
Neural Networks, Robotics and Autonomous Systems, Vol. 62, No.
6, pp. 721-736, 2014.
• Wataru Hinoshita, Horiaki Arie, Jun Tani, Hiroshi G. Okuno,
Tetsuya Ogata: Emergence of Hierarchical Structure mirroring
Linguistic Composition in a Recurrent Neural Network, Neural
Networks, Vol.24, No.4, pp.311-320, 2011.
• Shingo Murata, Yuichi Yamashita, Hiroaki Arie, Tetsuya Ogata, Jun
Tani, and Shigeki Sugano: Generation of Sensory Reflex Behavior
versus Intentional Proactive Behavior in Robot Learning of
Cooperative Interactions with Others, The 4th Joint IEEE
International Conference on Development and Learning and on
Epigenetic Robotics (ICDL-EpiRob 2014), pp. 234-240, 2014.	
 1	

2014/11/12	

自己紹介
尾形哲也（おがたてつや）
1993
早稲田大学理工学部機械工学科卒業
1999～2001 早稲田大学理工学部 助手
ヒューマノイドロボット	 2001～2003 理化学研究所
脳科学総合研究センター

研究員

ロボットとニューラルネットワーク	 2003～2012 京都大学大学院情報学研究科
知能情報学専攻 講師&准教授
認知ロボティクス

2012〜現在 早稲田大学理工学術院
基幹理工学部表現工学科

教授

E-mail: ogata@waseda.jp

背景• オープンエンドな実環境下におけるロボットの活動▫ 未知のノイズに頑健な環境認識，行動生成▫ 環境変化に適応的な行動選択

大規模な感覚運動情報の効率的な処理が課題	

• Deep learning（深層学習）▫ 深い階層を持ったニューラルネットワークの学習技術
 多次元，大量のデータを効率よく学習可能 精度が高く，高次の特徴量の自己組織化	
 ▫ 画像認識，音声認識の分野にブレークスルー(Google official blog,
2012)	

 Large Scale Visual Recognition Challenge 2012 (ILSVRC2012)
 Meet The Guy Who Helped Google Beat Apple's Siri (Forbes, 2013)

ロボティクスへの応用はこれから

2	2014/11/12	

Most Downloaded Robotics andAutonomous Systems Articles

階層を深くする問題と克服• 深い階層のニューラルネットDeep Neural Network (DNN)▫ 高い表現能力▫ 学習が困難• 従来の誤差逆伝搬の問題	
 [Bangio et al., 1994]▫ 入力側の層にいくほど誤差がうまく伝わらない(gradientvanishment)▫ 局所解や過学習に陥りやすい
精度が上がらない	 Error	
 Output	• 過学習の克服	 ▫ 学習法の改善（計算能力の向上）	
 ▫ 構造の改良	 ▫ 学習データの増加

Input	

3	

2014/11/12	

REPORTSREPORTScalled weights
an Bautoencoder[
is depicted
an Bautoencoder[inand is depicted inStarting with random
in thewith
two random
in the twoand called
to transform the high-dimensional
to transform
data into thea high-dimensional
data intoweights
a Starting
Fig.be1. trained together by Fig. 1.
networks,
they canbe trained
networks,
togetherthey
by canlow-dimensional code and a similar
low-dimensional
Bdecoder[ code
and a similar
Bdecoder[It is difficult
is difficult
in to optimize the weights inminimizing
the discrepancy
minimizing
the origthe discrepancy
betweentotheoptimize
orig- the Itweights
network to recover the data fromnetwork
the tocode.
recoverthe data fromthe code. between
nonlinear autoencoders
that nonlinear
have multiple
autoencoders that have multiple
inal data and its reconstruction.
inal
The
data
requiredand its reconstruction.
The required
layers(2–4).theWithlargehidden
initial weights,
layers (2–4). With large initial weights,
gradients
are easily
using the arechaineasilyhidden
obtainedby using
chain
Department of Computer Science, University
Departmentof Toronto,of Computer
6
Science, Universityof obtained
Toronto, 6 bygradientstypically findlocal minima;
typically find poor local minima;
rule tobackpropagate
error derivatives
rule to backpropagate
first autoencoders
error derivatives
firstpoorautoencoders
King’s College Road, Toronto, Ontario M5S
King’s
3G4,
College
Canada.
Road, Toronto,
Ontario
M5S 3G4, Canada.
withsmall andinitialweights,
with smallininitial
the weights, the gradients in thethroughthroughthen throughthe decoder
networkthenthroughthe gradients*To whom correspondence should be *To
addressed;
whom correspondence
E-mail:shouldthe
be decoder
addressed;network
E-mail: andare tiny,
makingit infeasible
layers are
to tiny, making it infeasible tothe encoder
systemnetwork
is early(1). layers
The wholesystemis early
hinton@cs.toronto.edu
hinton@cs.toronto.edu the encoder network (1). The wholetrain autoencoders with many hidden
train autoencoders
layers. If with many hidden layers. If
the initial weights are close to athegoodinitialsolution,weights are close to a good solution,
Decoder
Decoder
gradient descent works well, but
gradient
findingdescentsuch works well, but finding such
30
30
initial weights requires a very different
initial weights
type ofrequires a very different type of
W4
W4
algorithm
thatlearnsone
layer
algorithmoffeaturesthatat alearns one layer of features at a
Top
Top
500
500
time. We introduce this Bpretraining[time. We
procedure
introduce this Bpretraining[ procedure
RBM
RBM
TT
T
T
for binary data, W
generalize
it to real-valued
for binary data,data,generalize it to real-valued data,
W1W
W11+ε 8
1 +ε 8and show that
anda show
variety
thatof it works well for a variety of
2000
2000
2000
2000it works well forTTTTdata sets.
data sets.
W2
W
W22+ε 7
W 2 +ε 7500
500
An ensemble
An(e.g.,ensemble
im- of binary vectors (e.g., im1000
1000 of binary vectors
1000
W3
W 3 1000TTTTages) can be modeledusing aages)
two-layer
can benet-modeled using a two-layer netW3
W
W33+ε 6
W 3 +ε 6
1000
1000
RBM
RBM
work called 500
a Brestricted Boltzmann
work machine[called a Brestricted Boltzmann machine[
500
500
500
T
T
(RBM) (5, 6) inWwhich
stochastic,(RBM)
binary(5,pixels
6) in which stochastic, binary pixels
T
W4
WT4+ε
W
4
5
4 +ε 5are connected
to stochastic, arebinaryconnected
feature to stochastic, binary feature
30 Code layer
30 Code layer
30
30
1000
1000
detectors using symmetrically detectors
weighted using
con- symmetrically weighted conW4
W
W44+ε 4W 4 +ε 4W2
W2
nections. The pixels
correspondnections.
to Bvisible[
The pixels correspond to Bvisible[
500
500
500
500
2000
2000
units of the RBM because theirunits statesof theareRBM because their states are
RBM
RBM
W3
W
W33+ε 3W 3 +ε 3
observed; the featuredetectorsobserved;
correspond
thetofeature detectors correspond to
1000
1000
1000
1000
Bhidden[ units.
A joint configuration
Bhidden[(v, units.
h) of A joint configuration (v, h) of
W2W
W22+ε 2W+ε2
2the visible and hidden units hasthe
anvisible
energyand(7) hidden units has an energy (7)20002000 2000
2000
2000
2000
given bygiven by
W1
W1
W1
W
W11+ε 1W 1 +ε 1X
X
X
X
Eðv, hÞ 0 j
Eðv,bj hÞbi vi jhj 0 jbi vi jbj hj

Deep learning▫ Deep learning の	 epoch making 的な論文
▫ RBM によって2階層のネットワークを	 pre-training
した後，階層を連結し	
 fine-tuning を行う• J. Martens, “Deep learning via Hessian-free
optimization,” ICML, 2010.

iZpixelsj

RBM

Pretraining

Encoder

RBM

Pretraining
UnrollingX

jZfeaturesvi hj wiji, j

EncoderFine-tuning
Unrolling

iZpixels

Downloaded from www.sciencemag.org on February 27, 2013

• G. E. Hinton and R. R. Salakhutdinov, “Reducing
the dimensionality of data with neural networks,”
Science, 2006.

X
ð1Þ
jvi hj wiji, jjZfeaturesð1Þ

where vFine-tuning
whereofvi pixeland hij are the binary states of pixel ii and hj are the binary statesand featurej, b(RBMs),
featureand b each
are theirandbiases,andj,wb and b are their biases, and w▫ ネットワークの学習を2次計画問題として実現
Fig. 1. Pretraining consists of Fig.
learning
1. Pretraining
a stack of consists
restrictedofBoltzmann
learning a machines
stack of restricted(RBMs), each
Boltzmannmachinesis the weight
betweenthem. Theis thenetwork
weightas-between them. The network as(Hinton,2006)	
having only one layer of featurehaving
detectors.
only The
one learned
layer of featurefeature activations
detectors.
Theof onelearnedRBM
featureare usedactivationsof one RBM
are useda probability
to everysigns image
a probability
via
to every possible image via
as the ‘‘data’’ for training theasnext
the RBM‘‘data’’in for
the training
stack. Afterthe next
the pretraining,
RBM in the thestack.
RBMs
After
arethe signs
pretraining,
the RBMs
are possible▫ Pre-training の必要がない
thisenergyfunction,asexplained
thisinenergy(8).
Thefunction, as explained in (8). The‘‘unrolled’’ to create a deep autoencoder,‘‘unrolled’’ towhich
createis athendeepfine-tuned
autoencoder,usingwhichbackpropagationis then fine-tunedofusing backpropagation ofDeeplearning
via Hessian-free optimization
error
derivatives.
error derivatives.
probability of a training image canprobability
be raisedofby
a training image can be raised by▫ ニュートン法をベース（近似）とし収束が早い
Fig. 2. (A) Top to bottom: Fig. 2. (A) Top to bottom:
Table 2. Results (training and test errors)
i

j

iji

j

PT+NCG

4

Random samples of curves from Random samples of curves from
RAND+HF
PT+HF
HFset; reconstructions
thePT
test +data
the test data set; reconstructions
3
producedby the six-dimensional produced by the six-dimensional
0.10, 0.21
deep
autoencoder;reconstruc- deep autoencoder; reconstruc1.63,
2.46
2
tions
by ‘‘logistic
1.60,
2.28 PCA’’ (8) using tions by ‘‘logistic PCA’’ (8) using
six components;
six components; reconstructions-,- reconstructions
1
by logistic -,PCA and standard by logistic PCA and standard
PCA using 18 components. The PCA using 18 components. The0
3 squared error per im4
average squared error per imaverage
10
10
age forwhere
the last
is (log−scale)
overfitting, and so we ran additional experiments
wefour rows is age for the last fourtime inrows
seconds
1.44, 7.64, 2.45, 5.90. (B) Top 1.44, 7.64, 2.45, 5.90. (B) Top
introduced an l2 prior on the connection weights.
to bottom: A random testFigure
image 2.
to Errorbottom:(train
A random
test image
and test)
vs. computation time on CURVES
from each class;reconstructions from each class; reconstructions
is theTable 2 summarizes our results, where PT+NCG
by the 30-dimensional
autoen- by the 30-dimensional autoenpre-training + non-linear CG fine-tuning approachof H&S,
coder; reconstructions by
30- oncoder;
reconstructions
by 30Basedthese
results we canhypothesize that the way preRAND+HF is our Hessian-free method initialized
ran- PCA and dimensionalst logistic PCA anddimensional logistic
training helps 1 -order optimization algorithms overcome
domly, and PT+HF is our approach initialized
with
pre-The average standard PCA. The average
standard
PCA.
the
under-fitting
problemis threeby placing the parameters in asquarederrors
for
the
last
three
squarederrorsforthelast
trained parameters. The numbers given in each entry of thelessareaffectedby andissuesof pathological curvature inrows are 3.00,
8.01, and region
13.87. rows
3.00, 8.01,
13.87.
table are the average sum of squared reconstruction
errorsthe objective,
as those
discussed in section 2.2. This(C) Top to bottom: Random(C) Top suchto bottom:
Randomon the training-set and the test-set. The *’sindicatesamplesfrom thatthe test data
set; samples
from thewhy
test ourdata HF
set; approach optimizes fasterwould
also explain
an l2 prior was used, with strength 10−4 on
MNIST andby the 30- reconstructions by the 30reconstructions
from pre-trained parameters, as more favorable local curautoencoder; reconstructions
dimensionalby 30-dimensional
autoencoder; reconstructions
PCA. The averageby 30-dimensionalsquared errors PCA.
are 126
The andaverage
135.squared errors are 126 and 135.
10−2 on FACES. Error numbers for FACESdimensional
which involve

PT + NCG
0.74, 0.82
2.31, 2.72
2.07, 2.61-, 124-,-

RAND+HF
0.11, 0.20
1.64, 2.78
1.75, 2.55
55.4, 139
60.6, 122

error

CURVES
MNIST
MNIST*FACES
FACES*

ij

本研究では，最適化学習に
Hessian-free optimization を採用	(Martens, 2010)	

pre-training are missing due to our failure to reproduce theresults of H&S on that dataset (instead we just give the testerror number they reported).
Figure 2 demonstrates the performance of our implementations on the CURVES dataset. Pre-training time is included
where applicable. This plot is not meant to be a definitive
performance analysis, but merely a demonstration that our
method is indeed quite efficient.

vature conditions allow the CG runs to make more rapid
progress when optimizing qθ (p).
www.sciencemag.org

SCIENCEwww.sciencemag.org
VOL 313 28 JULYSCIENCE2006

Finally, while these early results are very encouraging,clearly further research is warranted in order to address
the many interesting questions that arise from them, suchas how much more powerful are deep nets than shallow
ones, and is this power fully exploited by current pretraining/fine-tuning schemes?

VOL 31328 JULY 2006

505

505

Acknowledgments

8. Discussion of results and implications
The most important implication of our results is that learning in deep models can be achieved effectively and efficiently by a completely general optimizer without any needfor pre-training. This opens the door to examining a diverse
range of deep or otherwise difficult-to-optimize architectures for which there are no effective pre-training methods,
such as asymmetric auto-encoders, or recurrent neural nets.

The author would like to thank Geoffrey Hinton, Richard
Zemel, Ilya Sutskever and Hugo Larochelle for their helpful suggestions. This work was supported by NSERC andthe University of Toronto.
R EFERENCES

ディープラーニングによる学習
A clear theme which emerges from our results is that the
HF optimized nets have much lower training error, implying that our HF approach does well because it is more effective than pre-training + fine-tuning approaches at solving the under-fitting problem. Because both the MNISTand FACES experiments used early-stopping, the training
error numbers reported in Table 2 are actually much higher
than can otherwise be achieved. When we initialized randomly and didn’t use early-stopping we obtained a training
error of 1.40 on MNIST and 12.9 on FACES.

高次元データ入力	
 高次元データ入力	 A pre-trained initialization benefits our approach in termsof optimization speed and generalization error. However,there doesn’t seem to be any significant benefit in regards
特徴量抽出（設計）	
 to under-fitting, as our HF approach seems to solve thisproblem almost completely by itself. This is in stark contrast to the situation with 1st -order optimization algorithms,
where the main hurdle overcome by pre-training is thatof under-fitting, at least in the setting of auto-encoders.

特徴量	
 Amari, S., Park, H., and Fukumizu, K. Adaptive method of realizing natural gradient learning for multilayer perceptrons. NeuralComputation, 2000.
Bengio, Y., Lamblin, P., Popovici, D., and Larochelle, H. Greedy
layer-wise training of deep networks. In NIPS, 2007.
Erhan, D., Bengio, Y., Courville, A., Manzagol, P., Vincent, P.,
and Bengio, S. Why does unsupervised pre-training help deeplearning? Journal of Machine Learning Research, 2010.
Hinton, G. E. and Salakhutdinov, R. R. Reducing the dimensionality of data with neural networks. Science, July 2006.
LeCun, Y., Bottou, L., Orr, G., and Muller, K. Efficient backprop.
In Orr, G. and K., Muller (eds.), Neural Networks: Tricks ofthe trade. Springer, 1998.
Mizutani, E. and Dreyfus, S. E. Second-order stagewise backpropagation for hessian-matrix analyses and investigation ofnegative curvature. Neural Networks, 21(2-3):193 – 203, 2008.
Nocedal, J. and Wright, S. J. Numerical Optimization. Springer,
1999.
Pearlmutter, B. A. Fast exact multiplication by the hessian. Neural Computation, 1994.
Schraudolph, N. N. Fast curvature matrix-vector products forsecond-order gradient descent. Neural Computation, 2002.

Deep
Learning	 分類（機械学習）	
 クラス識別	
 クラス識別	
 画像認識・音声認識などのデータで有効性が報告[A. Krizhevsky et al., 2012], [G. E. Dahlet al., 2012]4	

2014/11/12	

ディープラーニングによるロボット
行動学習
高次元感覚（＋運動）入力	
 高次元感覚	 （＋運動）入力	
 特徴量抽出（設計）	
 

特徴量	
 

Deep
Learning	 分類（機械学習）	
 クラス識別	 運動（＋感覚）
出力	
 プラニング	 運動（＋感覚）出力	
 マルチモーダル学習

Ball lift	

Ball rolling	

Bell ring R	

Bell ring L	

Ball rolling on a plate	

Ropeway	5	

2014/11/12	

評価実験• 物体操作行動の感覚運動統合学習▫ 6種類の物体操作行動を直接教示• 感覚運動データ▫ 20x15のRGB画像：900次元▫ 左右の腕関節角：10自由度▫ 時間ウィンドウの幅：30ステップ
学習データ	
 

テストデー
タ	
 

入出力次元	
 ネットワーク構造	 画像特徴学習	
 

8444	
 

948	
 

900	
 

1000-500-250-150-80-30-80-150-250-500-1000	
 時系列学習	
 

6848	
 

776	
 

1200	
 

1000-500-250-150-80-30-80-150-250-500-1000	
 • GPGPU実装（CUBLAS）による最適化計算▫ 特徴抽出，時系列学習：各約30分	
 多層型から再帰結合型へ• Wikipediaの学習▫ [I. Sutskever et al.,2011 ]• プログラミング言語の自動解釈（LSTM）▫ [A. Graves, 2014]• 大量データの処理	
 6	

2014/11/12	

コンテキストを利用したロボット行動学習
高次元感覚	 （＋運動）入力	 高次元感覚	 （＋運動）入力	 DeepNeural Net	 Recurrent
Neural Net	 運動（＋感覚）出力	
 運動（＋感覚）
出力	
 MTRNN (Y. Yamashita et. al, 2008)
Multiple Timescale Recurrent Neural Network	 低次のネットワークがプリミティブを表現し，
高次ネットワークはその組み合わせを表現n 多様なシーケンスを回路の初期状態に埋め込める
Actualsensory input	

Lower level:
Primitive	

Higher level:
Combination / Intention	

Error	

Variance	Predicted sensory inputà Motor command	

Fast
Context	

SlowContext	
Initial state (IS) space	7	2014/11/12	

階層間のバランス：トップダウンとボ
トムアップ
ロボットNAOが相手のロボットの手が左右のどちらにいくかをあてる課題
学習するロボット
S-MTRNNで制御実験者のロボット
固定プログラムで制御Home	

5つの分岐を含む
32種類のシーケンス

Right	

Home	

Left	
……	

S-CTRNN

Namikawa et. al., 2011	

BPTTで学習パラメータを変更▫ 尤度関数：
予測可分散予測
V(t+1)	

RNN
C(t)	
S(t+1)	

不可

予測可

不可

S*(t+1)	
t	

S*(t)	
S*(t+1)	

時系列データの予測しにくさ（予測精度）を予測値の分散という形で内部で表現	 大E(t+1)	
小

大小t	V(t+1)を予測することで予測可能・
不可能部分のE(t+1)を同等に扱える	 大V(t+1)	

小

大
小

t	

8	

2014/11/12	

S-CTRNNの２つの振る舞い
Deterministic modeling（トップダウン型）▫ High variance at branch point▫ Low variance at branch pointab	

Stochastic modeling（ボトムアップ）

a	

a

b	

a	

time	

time	

ボトムアップ型行動
知覚→行動のサイクルが繰り返される
Perception-action cycle
Noise /
Actual sensory	

Variance	

L	

L	

R	

Perception	

Action	

Perception	

Action	

Perception	

Action	

Perception	

Action	

R	

L	

R	

time	

9	

2014/11/12	

トップダウン型行動
行動（意図）→知覚（の修正）のサイクルが繰り返される
Intention (action)-perception cycle
Initial state	

Intention	

Perception	

Bottom-up regression	

Variance	
Intention	

L	

Intention	

L	

R	

L	

Perception	

Bottom-up regression	

R	

Perception	

Bottom-up regression	

R	

Intention	

Perception	

time	

まとめ
• ディープラーニングのロボットへの応用▫ 実ロボットへの応用は萌芽段階▫ クラス認識より生成，プラニングへの応用が重要 “認識”の為ではなく”行動生成”の為の特徴量の自己組織化• 再帰結合型神経回路のロボット認知への応用▫ 音声，映像等のリカレントネットの拡張
 スケーラビリティの拡大▫ 階層構造の機能バランス
 トップダウン，ボトムアップ，• ロボット応用への課題▫ 大量の”行動”学習データ
 マルチモーダルな身体経験としてのデータ	 10	

2014/11/12	

研究メンバー
菅佑樹

有江浩明

早稲田大学
HRI
招聘研究員	

早稲田大学	
理工学術院
助教

谷淳
KAIST教授

野田邦昭

早稲田大学	
理工学術院
客員教授	

早稲田大学	
日本学術振興会特別研究員

おわり
ogata@waseda.jp	
 11