Modularized Morphing Of Neural Networks

# About this Publication

- Authors: Tao Wei, Changhu Wang, Chang Wen Chen (Microsoft)
- Conference: ICLR
- Years: 2017
- Paper Link: https://arxiv.org/abs/1701.03281

---
# Motivation

1. inherite entire knowledge from a well-trained network to a new one
2. booster performance
3. explore new model and architecture for constitutional neural network

<img src="https://qiita-image-store.s3.amazonaws.com/0/142884/11a09d73-d7f6-c1f0-5fb2-e80e783167ef.png" width="200" height="150" align="right"/>

---
# Previous works

- not practical use
 - limits the change of network parameters including width, depth and kernel size
 - allow to apply a limited layers in a CNN

---
# Goal

- design the architecture allowing to apply hundreds of layers with primitive operations

---
# What you would learn from this presentation

1. an idea of the architecture for knowledge inheritance in neural networks

2. some technical terminology about neural networks

---
# Proposition - Network Morphism
explore a morphing space and apply changes of the basic modules to modularized networks

<img src="https://qiita-image-store.s3.amazonaws.com/0/142884/4544a598-8465-211a-a371-81fec0da51ad.png" align="right" width=300px >

advantages 
1. simple
2. scalability

---
## [for beginners] What is CNN (Convolutional Neural Network)?
![cnn.png](https://qiita-image-store.s3.amazonaws.com/0/142884/bc3a2351-403e-d8eb-810a-da2afe787662.png)

---
## [for beginners] What is modularized network?

a network that is divided into a set of functional units (modules)

![image](https://qiita-image-store.s3.amazonaws.com/0/142884/910499df-832d-4101-21d5-1fe8e9bd2495.png)

---
# Goal of Network Morphism

When a CNN $B_i$ grows to a new CNN $B_j$, $B_i$ applies convolutional layers $G_i$.
To be specific, a convolutional layer $G_i$ is morphed into two convolutional layers $F_l$ and $F_{l+1}$.
<img src="https://qiita-image-store.s3.amazonaws.com/0/142884/20c9eb8c-9722-cb9c-5a5d-8af520da7d3b.png" align="right">

---
# core algorithm

1. create irreducible networks according to the types of morphable networks
2. create a new convolution layers only if an irreducible network impacts to all fixed layers (if needed)

---
# Types of Morphable Networks

1. Simple morphable networks
 -  morphed with atomic morphing operations
     - TYPE-I: a convolutional layer is morphed into two convolutional layers
     - TYPE-II: a convolutional layer is morphed into two-way convolutional layers
     
2. Complex morphable networks
<img src="https://qiita-image-store.s3.amazonaws.com/0/142884/64a83d88-3dd6-b86f-26a9-08a6dfe5298e.png" align="right">


--- 
# Quiz
Classify 4 networks into simple and complex morphable networks

<img src="https://qiita-image-store.s3.amazonaws.com/0/142884/c384aa2b-afda-dc8c-cebf-e3b2a1332db5.png" width=350px align="right">

---
# Answer [1/2]
- Simple morphable network = (A) (B) (C)
- Complex morphable network = (D)

<img src="https://qiita-image-store.s3.amazonaws.com/0/142884/c384aa2b-afda-dc8c-cebf-e3b2a1332db5.png" width=350px align="right">

---
# Answer [2/2]
- Simple morphable network
<img src="https://qiita-image-store.s3.amazonaws.com/0/142884/701fc0e7-0e9f-2199-b3ca-4f0ed6eee671.png" width=500px>

- Complex morphable module
<img src="https://qiita-image-store.s3.amazonaws.com/0/142884/fa6344bc-d952-ec9f-30a4-00e27f6d69b5.png" width=500px>

---
# How to solve Complex Morphable Networks

1. create a irreducible network from a complex morphable network
2. apply atomic morphing operations and create a new convolutional layer for the network

---
# Implementation 
The authors extend the proposed morphable algorithm in ResNet

![fig3.png](https://qiita-image-store.s3.amazonaws.com/0/142884/bf64a2a4-e4bd-f81e-6a00-4eb9dfe8353b.png)

---
# [for beginners] ResNet
<img src="https://qiita-image-store.s3.amazonaws.com/0/142884/5042479c-fba7-22dc-03da-23df3cf77e0e.png" align="right" height="400px" width=200px>

- a winner @ ILSVRC 2015
- Residual Learning
   - conquer the proportional error rate to the number of convolutional layers

---
# List of Evaluated Morphable Network

![fig4.png](https://qiita-image-store.s3.amazonaws.com/0/142884/9e547250-935e-79cc-c35f-f4718fab9b48.png)

---
# Dataset

- [x] CIFAR10 
 - consists of 32×32 color images in 10 categories, with 50,000 training images and 10,000 testing images.

- [ ] CIFAR100
 - consists of 500 training images in 100 categories and 100 testing images per category
- [ ] ImageNet
 - consists of 1.28 million training images in 1,000 object categories and 50K validation images

---
# Experimental Results (CIFAR10)
![table1.png](https://qiita-image-store.s3.amazonaws.com/0/142884/748e2fbd-896e-87bf-0cfa-da673be5f3f4.png)

---
# Conclusion
- Network morphism is an effective learning scheme to morph a well-trained neural network to a new one with the network function completely preserved

- In benchmark, CNNs with the network morphism shows smaller error rate compared to CNNs by compromising the higher calculation cost 

---
# Discussions @ coffee with science meetup

---
# Reference
https://www.microsoft.com/en-us/research/project/network-morphism-2/

https://adeshpande3.github.io/adeshpande3.github.io/A-Beginner%27s-Guide-To-Understanding-Convolutional-Neural-Networks/
http://cs231n.github.io/convolutional-networks/

https://deepage.net/deep_learning/2016/11/30/resnet.html