# Improvement for Convolutional Neural Networks in Image Classification Using Long Skip Connection
## 2 Related Works  
Skip connection has become a standard in today's convolutional architectures. Short skip connections are used along with consecutive convolutional layers that do not change the dimension inside blocks of layers until a convolution with stride equal to two or pooling layers; while long skip connections usually exist in the architectures that are symmetrical.  
Highway networks used learned gating mechanisms (as a first form of skip connection) to regulate information flow, inspired by Long Short-Term Memory (LSTM) [14] recurrent neural networks. The gating mechanisms allowed neural networks to have paths for information to follow across different layers (\"information highways\"), using and spreading knowledge from previous layers. The idea was explained again in ResNet that: the deeper layers are identity mapping, and the other layers are learned from the shallower layers. By this idea, ResNet's authors created a deep residual learning architecture with 152 layers and won ILSVRC 2015 with an incredible error rate of 3.6%. Another popular model that utilized the effect of skip connection is DenseNet. This architecture heavily used feature concatenation so as to ensure maximum information flow between layers in the network. This was achieved by connecting via concatenation of all layers directly with each other, as opposed to ResNet. The DenseNet's authors claimed that their idea led to an enormous amount of feature channels on the last layers of the network, to more compact models, and extreme feature reusability.  
U-Net [15] performed long skip connections between layers in encoder and decoder; while the encoder did the convolution work, the decoder did the deconvolution (transpose convolution), which is opposed to the first path. By introducing skip connections in the encoder-decoder architecture, fine-grained details can be recovered in the prediction. Even though there was no theoretical justification, symmetrical long skip connections worked incredibly effectively in dense prediction tasks (medical image segmentation).  
For the task of improving a model's computational performance, Depthwise Separable Convolution was first introduced in Xception as an Inception [16] module and placed throughout the deep learning architecture. Depthwise Separable Convolution is made by a Depthwise Convolution (channel-wise \(D_{\text{K}}\ \times\ D_{\text{K}}\) spatial convolution) followed by a pointwise convolution (1 \(\times\) 1 convolution) to change the dimension. With \(D_{\text{K}}\ \times\ D_{\text{K}}\) equals 3 \(\times\) 3, Depthwise Separable Convolution lowers computation effort, with only a small reduction in accuracy. MobileNet adopts the idea and built network architecture that even outperforms SqueezeNet [17] and AlexNet while the multi-adds and parameters were much fewer. MobileNet v2 [18] introduced two new features to the base architecture of the first version, which were linear bottlenecks between the layers, and shortcut connections between the bottlenecks. The latest MobileNet (v3) [19] family member used h-swish instead of sigmoid function [20] and mobile-friendly Squeeze-and-Excitation blocks [21], together with model architecture auto searching. The two later versions of MobileNet family achieved about 75% accuracy and less than 5 ms latency (running on Pixel4 Edge TPU mobile).  
Recent model architectures have focused on balancing accuracy and latency; MnasNet [22] was the first to introduce automated neural architecture search approach for designing mobile models using reinforcement learning. EfficientNet [23] uniformly scaled each dimension (width, depth and resolution) with a fixed set of scaling coefficients, achieving much better accuracy and efficiency than previous CNNs.