Brain-like Unsupervised Feature Learning
for Convolutional Neural Networks
Takashi Shinozaki
CiNet, National Institute of Information and Communications Technology (NICT)
Graduate School of Information Science and Technology, Osaka University
3.2 Results

Summary• Calculated by NVIDIA Tesla P100 with Chainer v4 [Tokui+2012]• Integrates competitive learning to CNNs

– The code is available at https://github.com/t-shinozaki/convcp– Unsupervised pre-training by competitive learning– Supervised fine-tuning by error based learning without BP signals• Validate the method with MNIST, CIFAR-10 and ImageNet– State-of-the-art performance as a biologically-motivated NN• Validate the proposed method with three datasets– MNIST dataset [LeCun 1998]– CIFAR-10 dataset [Krizhevsky+2009]– ImageNet dataset [Russakovsky+2015]• State-of-the-art performance as a biologically-motivated NN• Could apply for various types of dataMethod– Video data, 1D-temporal sequence data, medical data, etc.
1. Introduction

1.1 Deep Neural Networks• Key technologies of the recent revolutionary advance of AI– Convolutional Neural Network (CNN) [LeCun+1989]– Long Short Term Memory (LSTM) [Hochreiter & Schmidhuber 1997]

MNIST CIFAR-10 ImageNet Top-1 Top-5

Baseline [Bartunov+2018]

0.90

37.74

63.93 40.17

Ours

1.79

39.31

87.72 73.84

DTP, Parallel, LC [Bartunov+2018]

1.52

39.47

98.34 94.56

SDTP, Parallel, LC [Bartunov+2018]

1.98

46.63

99.28 97.15

FA, LC [Bartunov+2018]

1.85

37.44

93.08 82.54

Table 1: Test errors of image discrimination tasks.

• Both of them use Back Propagation (BP) learning [Rumelhart+1986]

3.3 CIFAR-10– Tremendously good for fine-tuning– Extracts features for the discrimination → Limits the generalization

(a)
0.8

1.2 Competitive Learning

(b)

(c)

(e)

(f)

BaselineOurs

Test Error Rate

0.7

• Traditional unsupervised learning method for NNs• Used for a couple of classical neural networks

0.6
0.5
0.4

– Self Organizing Map (SOM) [Kohonen 1982]– Neocognitron [Fukushima 1980]

01000

2000
3000
Iteration #4000

5000

(d)
Neocognitron [Fukushima 1980]• Good for pre-training, but not good for fine-tuning

Combine competitive learning with error based learning!!

Test Error Rate

• Extracts bases of input data like as ICA

0.8
0.7
0.60.5r Ratio

Winne

1.0
0.8
0.6
0.4
0.2
0.0v2of Con

2. Competitive Learning in Convolutional Layers

1.0

0.2
0.8 0.6Ratio0.4
of Conv1
Winner0.0• Employ the simplest competitive learning– Just use winner-takes-all (WTA) algorithm– Do not use any spatial information over the filter space• (a) Transitions of test error rates during fine-tuning– The competitive learning accelerated the initial speed of fine-tuning∗ Though the competitive layers do not perform BP learning at all• Basically behave alike conventional convolutional layers• Weight gradient is calculated with the feedforward propagationρzl−1, if i = argmaxk(ul,k + bk)∆wl,i = 0,otherwise,• (b,c) Obtained learning representations for conv1 & conv2(1)• ul: output vector of l-th layer described as ul = Wl zl−1
• Wl: connection matrix, zl−1: output vector of the previous layer• ρ: learning coefficient of competitive learning, 0.01

• (d) Comparison over several winner ratios

bi = C(1/N − pi).

(2)

– Stronger filters dominated and obstructed weaker filters– Some filters couldn’t get clear spatial patterns (especially in conv2)– Prevents for some filters to dominate over the layer– C : conscience coefficient, 5.0, N : the number of filters at the layer– pi: the probability of winning for the i-th unit in the mini-batch

4. Discussion• Update weights with the gradient by conventional method: e.g. SGD, Adam

• Integrates competitive learning into conventional CNNs

– The weight vector is normalized by L2-norm at every update– Apply for unsupervised pre-training• WTA activation function• Powerful representation learning– ReLU must required well controlled threshold learned by BP3. Experiments(a)• LeNet5 based networks– Two or three conv layers∗ w/inception-like structure– WTA activation function∗ Instead of ReLU
– Max-pooling– One fc layer w/softmax(b)Softmax– Winners Share All (WSA) activation function∗ Only the upper units in the value order has output (controlled by winner ratio)– 0.0 is the best, meaning WTA condition• (e,f) Obtained learning representations without the conscience factor• bi: conscience factor [Desieno 1988]

3.1 Network Structure– Competitive learning robustly extracted bases of images– WTA activation function enables hierarchical competitive learning(c)Softmax

Softmax

FC(10)

FC(10)

FC(1000)

4x4 MaxPool(stride 2)

4x4 MaxPool(stride 2)

4x4 MaxPool(stride 2)

Filter concat

Filter concat

Filter concat

WTA activation

WTA activation

WTA activation

WTA activation

WTA activation

WTA activation

5x5 Conv(64)

3x3 Conv(64)

5x5 Conv(1024)

3x3 Conv(1024)

5x5 Conv(1024)

3x3 Conv(1024)

4x4 MaxPool(stride 2)

4x4 MaxPool(stride 2)

4x4 MaxPool(stride 2)

WTA activation

WTA activation

Filter concat

3x3 Conv(16)

3x3 Conv(128)

WTA activation

WTA activation

Input 28x28

Input 32x32

5x5 Conv(1024)

3x3 Conv(1024)• Pre-training with competitive learning– Unsupervised learning with a huge amount of unlabeled data

• Evaluation in fine-tuning by BP learning– Supervised learning with labeled data
– Just for the last FC layer → No BP signal is required!!

4x4 MaxPool(stride 2)
WTA activation
3x3 Conv(256)
Input 235x235Filters
extracted by
BP learning

Filters extracted by– Acquires task-independent filterscompetitive learning∗ Much more generalized filters
All signal space– Requires much more filters∗ Most filters are redundant for a specific task
∗ Maybe tolerance for adversarial attacks?
– Fundamentally, equivalent to BP learning (∆wl,i = δl,i zl−1)?
– Utilize a huge amount of unlabeled data
[Athalye+2018]∗ Application for many kind of time-series signals• More effective when the number of filters was sufficiently large• Enables seamless switching between unsupervised and supervised learning– Applicable for semi-supervised learning?

• Future works– Apply for Various types of data: Video data, 1D-temporal sequence data, etc.
– Utilize SOM like organization over the filter space∗ The interpolation may enrich expressions of the filters∗ Must require an explosive amount of memory and calculation power• Pruning method for a specific task

International Symposium on Artificial Intelligence and Brain Science, Oct 10-12, 2020, Tokyo, JAPAN Online. This work was supported by JST ERATO Grant Number JPMJER1801, Japan.