2019年度高柳健次郎賞 受賞記念講演（2020.1.20）

ニューラルネットワークの先駆的研究とパターン認識への応用

NHK時代の研究関係者 （敬称略）
上司樋渡涓二
共同研究者
山口 幸也
三宅 誠
伊藤 崇之
藤井 真人

福島 邦彦
Kunihiko Fukushima
fukushima@m.ieice.org
http://personalpage.flsi.or.jp/fukushima/

1

他 大勢の方々2

Receptive fields of simple cells
単純型細胞の受容野

ネオコグニトロンの着想

on-center cells(LGN)
simple cell(visual cortex)(modified from Hubel, Wiesel, 1962)

34

Complex cell

Hypercomplex cell （超複雑型細胞）（複雑型細胞）

complexcells
simple cells（単純型細胞）

hypercomplexcell

complex cell
（複雑型細胞）(modified from Hubel, Wiesel, 1962)

5(modified from Hubel, Wiesel, 1965)

61

Hubel-Wiesel の階層仮説
網膜 → LGN → 単純型 → 複雑型 → 超複雑型
（外側膝状体）大脳 視覚野(Fukushima 1971)

78

Perceptron (Rosenblatt)(3層直列結合型)

fixedconnections

variable:supervised
learning(Fukushima 1971)

910

cognitron
競合学習則 ----- 多層回路の教師なし学習を可能に

cognitron:
教師なし学習でパターン認識能力を獲得

次の目標
名前の由来

入力パターンの 変形・位置ずれ・拡大縮小 に対応

cognitron= cognition + tron

CNN（畳み込み神経回路）の学習(perceptron= perception + tron)(Fukushima, 1975)11122

網膜（retina）視蓋（optic tectum）間の結合の再生

seed cell の導入
shared connections の条件を保ちつつ CNN を自己組織化（共有結合）（畳み込み神経回路）

seed cell（結晶成長の核のような働き）
他の細胞もseed cellにならって入力結合を変化

normalhalf retina

half tectum

Shared connections
between cell-planes
retinatectumretinatectumretinatectum13

14

featurepooling
extraction

featureextraction

featureextraction

pooling

neocognitron

Spatial spread of input connections of a celland the size of its receptive field

pooling(Fukushima 1979, 1980)

15

16

neocognitron = deep convolutional neural networkfor visual pattern recognition

頑強な 視覚パターン認識 の原理
Principles of Robust Visual Pattern Recognitionshared connections= spatial filtering= convolution

input
patternfeature
extraction(S-cells)

17

pooling(C-cells)

recognition(classification)

18

3

simple cells

Hierarchical information processingin the layered network

19

extracting featurescomplex cell

tolerating shift(spatial pooling)

C-cells accept shift in local features
20

extracting features

tolerating shift

extracting features

tolerating shift(spatial pooling)(spatial pooling)

C-cells accept shift in local features

C-cells accept shift in local features

21

22

extracting features

tolerating shift(spatial pooling)

C-cells accept shift in local features23

24

4

tolerance area

S-celltolerance area

learneddeformed

Robust recognition of deformed patterns
by tolerating positional error of local features

Robust recognition of deformed patterns
by tolerating positional error of local featuresunlearned

25

26

tolerance area

S-celltolerance area

S-celldeformed

deformed

Robust recognition of deformed patterns
by tolerating positional error of local features

Robust recognition of deformed patterns
by tolerating positional error of local featuresunlearned

27

unlearned

28

S-cells(simple)C-cells(complex)

S-cells(simple)C-cells(complex)

S-cells(simple)

C-cells(complex)

shift tolerance

shift tolerance

spatial blurtolerating positional error

tolerating positional error

blurring responses

Connections from S- to C-cells --- pooling

29

S-cellConnections from S- to C-cells --- pooling30

5Without blur:

Without blur:

31

32

Without blur:

Without blur:

After blurring operation:

After blurring operation:

33

34

intermediate layers of S-cells

中間層の S細胞の学習
Training S-cells--- intermediate layers ---AiS則による 教師なし学習
unsupervised learning

AiS (add-if-silent) rule(Fukushima 2013)

35

36

6

The progress of the learningby the AiS (add-if-silent) rule

Generation of a new S-cell:AiS (add-if-silent rule)training stimulus

t=0

training stimulus

t=1

t=2
no more modification
after generation

AiS (add-if-silent) rule

A new cell (cell-plane) is generated
if all post-synaptic cells are silent.

t=3(Fukushima, 2013)37

38

内挿ベクトル法
IntVec (Interpolating-Vector)
for the highest stage(= the deepest layer)(Fukushima 2007)

39

40

Largest similarity (nearest neighbor)in a multi-dimensional feature space

Training vectors are represented bya smaller number of labeled reference vectors.
41--- old neocognitron --WTA (winner-take-all)42

7

XiXjξ = piXiXj+ pjkX ikkX j k(pi + pj = 1)

Interpolating vectors （内挿ベクトル）
between two reference vectorsof the same label

43

Find the nearest reference vector (conventional WTA)

44

Find the nearest referencevector(conventional WTA)
interpolating
linevector

Find the nearest interpolating vector

45

46

Recognition by IntVec (interpolating-vector)

47

Recognition by IntVec (interpolating-vector)

48

8

Examples of patterns that were recognized,
erroneously by the WTA,
but correctly by the IntVec （interpolating-vector, Int-2).

Recognition by IntVec (interpolating-vector)

49

50

Recognition by Int-Vec (the interpolating-vector)

内挿ベクトル法（IntVec）はなぜ強力か？
CNN （畳込み神経回路）の認識率は，学習パターン数を増やすことによって向上する．
十分多数の学習パターンが得られないときには，人工的に作った学習パターンで不足を補っても，認識率を向上させることができる．
----- data augmentation ----内挿ベクトル法（IntVec）では この状態を，学習時ではなく，認識時に作り出している．
比較すべき特徴の数を，学習パターン数を増やさずに，実質的に増やしている．

This operation is applied,
not directly to input images,
but to extracted features.

5152

Interpolating-vector from three reference vectors

Int-2

Find the nearest line.

Find the nearest plane.

Int-2 (interpolating-vector

Int-3 (interpolating-vector

from 2 reference vectors)

Int-1 (WTA)
Int-2
Int-3
Int-4

from 3 reference vectors)

Int-3

number of hyperplanes
K
K(K1)/2 =: K2/2
K(K1)(K2)/6 =: K3/6K(K1)(K2)(K3)/24 =: K4/24
K: number of reference vectors of a class

5354

9

IntVec （内挿ベクトル法）の演算量削減

ネオコグニトロンの細胞の反応例
Examples of the responseof cells in neocognitron
WTA で 完全敗者

WTA で 一人勝ち

IntVec の演算省略（class C 及び D）

全class に対して
IntVec の演算不要(class A に対しても)(Fukushima 2019)

55

56

57

58

59

60

10

一部が隠されたパターンの認識
Recognition of Partly Occluded Patterns(Fukushima 2004)

61

62

63

64

Why occluded patterns are easier to recognize
when occluding objects are visible?

Invisible occluding objects:
Difficult to distinguishwhich features are relevant to the original pattern,
and which are newly generated by occlusion.

responds only tooccluding object

Add “masker layer” to the neocognitron
to inhibit irrelevant featureslocated near the contourof the occluding objects

Visible occluding objects:
Easy to distinguish between
relevant and irrelevant features.

65

66

11

edge extraction

higher-order features

Proposed Model = Neocognitron + Masker layer

67

68

69

70

An identical pattern is perceived differently
by the placement of gray objects.

71

An identical pattern is perceived differently
by the placement of gray objects.

72

12

73

74

選択的注意
Selective Attention— トップダウン信号の導入 —--- introducing backward signal paths ---Bottom-up and Top-down signals in the network(Fukushima 1986)

75

76

Selective Attention （選択的注意）
Interaction between forward and backward signals

U-net

Selective Att. Model(Fukushima, 1986)

77(Fukushima, 1986)

78

13

(Fukushima, 1986)

79(Fukushima, 1986)

80(Fukushima, 1986)

81(Fukushima, 1986)

82(Fukushima, 1986)

83(Fukushima, 1986)

84

14

Segmentation of connected characters

tt

85

86

人工神経回路の歴史
neuron model (McCulloch & Pitts 1943)

1950
1960

perceptron (Rosenblatt 1957)
1st boom

1970
1980
1990
2000

winter age
2nd boom

theoretical limit of perceptron (Minsky & Papert 1969)
neocognitron (Fukushima 1979)
backpropagation (Rumelhart, Hinton & Williams 1986)

winter age

2010
2020

3rd boom

deep learning?

脳に学べ ーー 再び冬の時代にならないために ーー
脳を制御している基本原理を見つけ出そう

8715