[Implementation] CNN in Python with Numpy

## Preface
This is the supplemental article, which I focus on the implementation of CNN in python.
***PYTHON=3.6***

## Inspirations
http://cs231n.github.io/convolutional-networks/
http://cs231n.stanford.edu/slides/2017/cs231n_2017_lecture5.pdf

Thank you, Dr.Li FeiFei

## warming-up
I have followed this link to remember the methods for image-processing.
http://www.turbare.net/transl/scipy-lecture-notes/advanced/image_processing/index.html

```python
import numpy as np
from scipy import misc, ndimage
import matplotlib.pyplot as plt

f = misc.face()
plt.imshow(f)
plt.show()
print(f.shape)
```
<img width="336" alt="Screen Shot 2018-04-24 at 18.38.08.png" src="https://qiita-image-store.s3.amazonaws.com/0/240801/af270cf4-3eee-399d-77b5-8dd96b42bba2.png">

```python
f = misc.face(gray=True)

plt.figure(figsize=(10, 3.6))

plt.subplot(131)
plt.imshow(f, cmap=plt.cm.gray)

plt.subplot(132)
plt.imshow(f, cmap=plt.cm.gray, vmin=30, vmax=200)
plt.axis('off')

plt.subplot(133)
plt.imshow(f, cmap=plt.cm.gray)
plt.contour(f, [50, 200])
plt.axis('off')

plt.subplots_adjust(wspace=0, hspace=0., top=0.99, bottom=0.01, left=0.05, right=0.99)
plt.show()
```
<img width="716" alt="Screen Shot 2018-04-24 at 18.38.40.png" src="https://qiita-image-store.s3.amazonaws.com/0/240801/865bd0fd-922d-7349-a423-545ef1d3f58a.png">

```python
f = misc.face(gray=True)
plt.figure(figsize=(8, 4))

plt.subplot(1, 2, 1)
plt.imshow(f[320:340, 510:530], cmap=plt.cm.gray)
plt.axis('off')

plt.subplot(1, 2, 2)
plt.imshow(f[320:340, 510:530], cmap=plt.cm.gray, interpolation='nearest')
plt.axis('off')

plt.subplots_adjust(wspace=0.02, hspace=0.02, top=1, bottom=0, left=0, right=1)
plt.show()
```
<img width="592" alt="Screen Shot 2018-04-24 at 18.39.07.png" src="https://qiita-image-store.s3.amazonaws.com/0/240801/c6c369fa-d143-bfa0-197d-81f38e2009a9.png">

```pyhton
plt.imshow(ndimage.gaussian_filter(f, sigma=3))
plt.show()

noisy = f + 0.4 * f.std() * np.random.random(f.shape)
plt.imshow(noisy)
plt.show()
```
<img width="327" alt="Screen Shot 2018-04-24 at 18.40.06.png" src="https://qiita-image-store.s3.amazonaws.com/0/240801/69b17e63-a292-4ab8-1fc4-1d4a8a7b82b6.png">

## Actual Implementation
From now, I will show the implementation of components of CNN.

### Convolution Layer

```python
# download some free image from google!
image_file = 'nine.png'
im = misc.imread(image_file)
plt.imshow(im)
plt.show()
print(im.shape)
```
<img width="208" alt="Screen Shot 2018-04-24 at 18.41.42.png" src="https://qiita-image-store.s3.amazonaws.com/0/240801/c4cfc6dc-d93f-5755-1773-68e7feb65087.png">

```python
def convolution(mat, fil, stride=1, verbose=False):
    mat_pad = np.pad(mat, 1, 'constant')
    results = np.zeros((len(mat)-stride, len(mat)-stride))
    for row in range(len(mat)-stride):
        for col in range(len(mat)-stride):
            if verbose:
                print("indices:", col, col+stride+1, "and", row, row+stride+1)
            spot = mat_pad[col:col+stride+1, row:row+stride+1]
            if verbose:
                print("spotted matrix: ", "\n",spot)
            filetered = spot.dot(fil)
            if verbose:
                print("After filtering", "\n", filetered)
            result = sum([sum(i) for i in filetered])
            if verbose:
                print("convoluted value: ", result)
            results[row, col] = result
    return results.T

# test 1
fil_22 = np.random.rand(2, 2)
results = convolution_original(np.random.rand(4,4), fil=fil_22)
print(results.shape)
plt.imshow(results)
plt.show()

# test 2
fil_33 = np.random.rand(4, 4)
results = convolution(im[:100,:100,1], fil=fil_33, stride=3)
print(results.shape)
plt.imshow(im[:100,:100,1])
plt.show()
plt.imshow(results)
plt.show()

# test 3
fil_33 = np.random.rand(3, 3)
results = convolution(np.random.rand(7,7), fil=fil_33, stride=2)
print(results.shape)
plt.imshow(results)
plt.show()
```
<img width="267" alt="Screen Shot 2018-04-24 at 18.46.35.png" src="https://qiita-image-store.s3.amazonaws.com/0/240801/d53bdcb8-1e91-1fe4-c87a-6b8fcf022914.png">
<img width="256" alt="Screen Shot 2018-04-24 at 18.46.49.png" src="https://qiita-image-store.s3.amazonaws.com/0/240801/22fa8a50-baf2-5d4e-9c67-776ca7b91d65.png">
<img width="253" alt="Screen Shot 2018-04-24 at 18.46.53.png" src="https://qiita-image-store.s3.amazonaws.com/0/240801/cb82bcd7-d6d8-f627-8f69-879c687c5b9d.png">

### pool-families
In this section, I will show my implementation of max_pooling in python.
And I have additionally made ReLU, then combined the convolution layer we made previously. So we could see the sequence of CNN at once.

```python
def ReLU(x):
    return x * (x > 0)

def max_pooling(mat, verbose=False):
    results = np.zeros((int(len(mat)/2), int(len(mat)/2)))
    for i, row in enumerate(range(0, len(mat), 2)):
        for j, col in enumerate(range(0, len(mat), 2)):
            if verbose:
                print("indices:", col, col+2, "and", row, row+2)
            spot = mat[col:col+2, row:row+2]
            if verbose:
                print("spotted matrix: ", "\n",spot)
            for vec in spot:
                result = np.max(vec)
            if verbose:
                print("convoluted value: ", result)
            results[i, j] = result
    return results.T

fil_22 = np.random.rand(2, 2)
results = ReLU(convolution(np.random.rand(5,5), fil=fil_22, stride=1))
print(results)
max_pooling(results)

# output
[[0.71590445 1.84275891 1.22907016 0.32425841]
 [1.06931596 2.4216673  1.63294066 1.41982642]
 [0.89558694 1.90399044 1.16696177 1.5738113 ]
 [0.87497548 2.07498316 1.67574505 1.7039923 ]]
array([[2.4216673 , 1.63294066],
       [2.07498316, 1.7039923 ]])
```

- Another way to implement max pooling

```python
import numpy as np

mat = np.random.randint(10, size=(4,4))

M, N = mat.shape
K = 2
L = 2

MK = M // K
NL = N // L

print(mat)
print(mat[:MK*K, :NL*L].reshape(MK, K, NL, L).max(axis=(1, 3)))

#output
[[3 0 0 0]
 [0 2 2 7]
 [0 5 0 1]
 [4 1 2 2]]
[[3 7]
 [5 2]]
```

## Discovery in Numpy

### numpy.pad
https://qiita.com/horitaku1124/items/6ae979b21ddc7256b872

```python
print(np.random.rand(3,3).shape)
np.pad(mat,1, 'constant')

#output
(3, 3)
array([[0, 0, 0, 0, 0, 0],
       [0, 8, 9, 3, 8, 0],
       [0, 8, 0, 5, 3, 0],
       [0, 9, 9, 5, 7, 0],
       [0, 6, 0, 4, 7, 0],
       [0, 0, 0, 0, 0, 0]])
```

### numpy.reshape for multi-dimensional usage
modifying the "axis" argument for numpy.reshape, we could convert the array into any dimensional matrix.  
In this case, I have formatted 32x1 matrix in 1x4x8x1x1, 2x2x2x2x2 or 1x1x1x1x32 dimensional matrix.

```python
np.random.seed(3)
a = np.random.rand(32)
print(a)
print(a.reshape(1,4,8,1,1))
print(a.reshape(2,2,2,2,2))
print(a.reshape(1,1,1,1,32))
```