TuePM1-3
Pattern Recognition 1
Chair: Tong Zhiqiang
Room: International Conference Hall III

13:20

Weighted Discriminant Analysis and Kernel Ridge Regression
Face Verification- 13:40 Metric Learning for
Siew-Chin Chong1 , Andrew Jin Teoh2 , Thian-Song Ong1
1
Multimedia University 2 Yonsei University

13:40

An Incremental One Class Learning Framework for Large
Scale Data
Qilin Deng1 Yi Yang1 Furao Shen1 Chaomin Luo2 Jinxi Zhao1- 14:00 1 Department of Computer Science and Technology, Nanjing
University 2 Department of Electrical and Computer Engineering, University of Detroit Mercy

14:00

Gesture Spotting by Using Vector Distance of Self-Organizing
Map- 14:20 Yuta Ichikawa1 , Shuji Tashiro1 , Hidetaka Ito1 , Hiroomi
Hikawa1
1Kansai University

14:20

Cross-Database Facial Expression Recognition via UnsuperAdaptive Dictionary Learning- 14:40 vised Domain
Keyu Yan1 , Wenming Zheng1 , Zhen Cui1 , Yuan Zong1
1Southeast University

14:40

Adaptive Multi-View Semi-Supervised Nonnegative Matrix
Factorization
1
Wang2 , Feng Tian1 , Chang Hong Liu1 ,- 15:00 Jing Wang , Xiao
1Hongchuan Yu , Yanbei Liu3
1Bournemouth University 2 Tsinghua University 3 TianjinUniversity78

TuePM2-3
Pattern Recognition 2
Chair: Hiroomi Hikawa, Tong Zhiqiang
Room: International Conference Hall III

15:20

Robust Soft Semi-Supervised Discriminant Projection for Fea- 15:40 ture Learning1
Xiaoyu Wang , Zhao Zhang1 , Yan Zhang1
1
Soochow University

15:40

A Hybrid Pooling Method for Convolutional Neural Networks- 16:00 Tong Zhiqiang1 Kazuyuki Aihara1 Gouhei Tanaka1
1
The University of Tokyo

16:00

Multi-nation and multi-norm License plates detection in real
- 16:20 traﬃc surveillance environment using Deep LearningAmira Naimi1 Yousri Kessentini1 Mohamed Hammami1
1MIRACL

16:20

A study on cluster size sensitivity of fuzzy c-means algorithmvariants- 16:40 Laszlo Szilagyi1 , Sandor Miklos Szilagyi2 , Calin Enachescu2
1
Sapientia University of Transylvania, Tirgu Mures, Romania
2Petru Maior University

16:40

Invited talkStatistical mechanics of pre-training and fine tuning in deep
- 17:20 learning
Masayuki Ohzeki1
1Graduate School of Informatics, Kyoto University.

82

Invited talkStatistical mechanics of pre-training and fine tuning in deep learning
Masayuki Ohzeki1
1Graduate School of Informatics, Kyoto University
Abstract
Lack of analytical study on the architecture of deep learning hampers understanding of its origin of the outstanding performance. A recent study has formulated atheoretical basis for the relationship between the recursive manipulation of variational renormalization groups and the multi-layer neural network in deep learning[2]. Indeed, it is confirmed that the renormalization group indeed can mitigate thecomputational cost in the learning without any significant degradation [3]. The
statistical mechanical approach is a rare successful approach to pave a way to thenature of the deep learning. We present a statistical-mechanical analysis on a partof architecture in the deep learning. We first elucidate some of the essential components of deep learning ―pre-training by unsupervised learning and fine tuning
by supervised learning. We formulate the extraction of features from the training
data as a margin criterion in a high-dimensional feature-vector space. The selforganized classifier is then supplied with small amounts of labelled data, as in deeplearning. For simplicity, we employ a simple single-layer perceptron model, rather
than directly analyzing a multi-layer neural network. The surprising performance ofthe deep learning does not necessarily come from deep neural network, but rather
stem from the potential of the neural network itself. We find a nontrivial phasetransition that is dependent on the number of unlabelled data in the generalization
error of the resultant classifier. The resultant phenomena exhibits the eﬃcacy of theunsupervised learning in deep learning. Increasing the number of unlabelled data
again leads to an improvement in the generalization error. A gradual increase in thenumber of labelled data allows us to escape from a metastable solution in multiple
solutions. In this sense, fine tuning by supervised learning is necessary to achieve
the lower-error state and mitigate the diﬃculties in reaching the desired solution.
We should emphasize that the emergence of the metastable state does not comefrom the multi-layer neural networks, but from the combination of unsupervisedand supervised learning. The analysis is performed by the replica method, which isa sophisticated tool in statistical mechanics. We validate our result in the mannerof deep learning, using a simple iterative algorithm to learn the weight vector on
the basis of belief propagation.
Reference:[1] M. Ohzeki: Journal of the Physical Society of Japan 84,034003 (2015)[2] P. Mehta and D. J. Schwab: arXiv:1410.3831.
[3] K. Tanaka, S. Kataoka, M. Yasuda, and M. Ohzeki: Journal of thePhysical Society of Japan 84, 045001 (2015)

118