ATTENTION INCORPORATE NETWORK: A NETWORK CAN ADAPT VARIOUS DATA SIZE

In traditional neural networks for image processing , the inputs of the neural networks should be the same size such as 224×224×3 . But how can we train the neural net model with different input size ? A common way to do is image deformation which accompany a problem of information loss ( e.g . image crop or wrap ) . In this paper we propose a new network structure called Attention Incorporate Network ( AIN ) . It solve the problem of different size of input images and extract the key features of the inputs by attention mechanism , pay different attention depends on the importance of the features not rely on the data size . Experimentally , AIN achieve a higher accuracy , better convergence comparing to the same size of other network structure . 1 INTRODUCTION . Human use the attention mechanism to recognize the world , pay different attention on different region of a image . The mixture of attention and neural network has been studied in the previous literature ( Zhao et al. , 2017 ; Itti & Koch , 2001 ; Shin et al. , 2016 ; Wang et al. , 2017 ) . Attentionbased network is widely used in sequence model to process text or audio ( Mnih et al. , 2014 ) . In this paper we propose Attention Incorporate Network ( AIN ) , AIN have a huge advantage in image classification problem especially for different size of input . Recent advances of image classification focus on a ” very deep ” network structure , from AlexNet ( Krizhevsky et al. , 2012 ) to DenseNet ( Huang et al. , 2016 ) the precision become higher and higher , but some problems are still remain to be solved , our model exhibits following appealing properties to solve these problem : ( 1 ) Many people comparing Neural Network to human neurons , human can recognize the object easily no matter what the image shape is . But all kinds of popular network like AlexNet ( Krizhevsky et al. , 2012 ) , VGG ( Simonyan & Zisserman , 2014 ) , Inception ( Szegedy et al. , 2016 ) and ResNet ( He et al. , 2016 ) couldn ’ t process the image in different size . An engineering solution is to scale the image into a same shape , which isn ’ t an aesthetic way due to human neurons won ’ t scale the object image while recognizing it . how does our network structure AIN training feed-forward is that : combine the attention matrix and image together and give a label after incorporate all the key information ( Fig . 1 ) . The input image can be arbitrary size due to the AIN feed-forward operation which gives a more natural solution than traditional image scaling ( Fig . 1 ) . ( 2 ) In modern convolutional network structure , image size will decrease while convolution layer get deeper . For example , the input image size of AlexNet ( Krizhevsky et al. , 2012 ) is 224×224×3 and the output of the last convolutional layer is 7×7×4096 . The spatial information of image will be transferred into the different channels . Pooling and Convolution with stride are the well known layers to reduces the image size . AlexNet ( Krizhevsky et al. , 2012 ) used maxpooling , Resnet ( He et al. , 2016 ) used Conv1×1 with stride of 2 . Information will remain only 1 s2 ( s=stride or pooling size ) during feed-forward the transition layer . In AIN we used attention incorporate layer ( AIL ) to extract the key features of the image without information loss . Experimentally , AIN achieved a lower error comparing to pooling or covolution with stride in the same network structure . 2 RELATED WORK . The field of attention is one of the oldest in psychology . Evidence from human perception process ( Mnih et al. , 2014 ) show the important of the attention mechanism which uses top information to guide bottom-up feed-forward process ( Wang et al. , 2017 ) . Attention was widely used in many applications : Recommendation system ( Vinh et al. , 2018 ) , Image classification ( Asada , 2014 ; Zhao et al. , 2017 ) , Image caption generation ( Xu et al. , 2015 ) , Machine translation ( Luong et al. , 2015 ; Firat et al. , 2016 ) , Speech recognition ( Chorowski et al. , 2015 ; Bahdanau et al. , 2016 ) and gave us some good result . In Image classification , people tried some method to solve the arbitrary input size . He , K. et . al ( He et al. , 2015 ) . used spatial pyramid pooling ( SPP ) to remove the fixed-size constraint of the network . Pooling size and stride depends on the input data to get a fixed output at the last convolutional layer . The main idea of their network is artificially divide image into fixed numbers of region . The disadvantage of SPP is that pooling layer lost lot of information while training feed-forward especially for those ” very deep ” networks . Pooling reduces the size of the hidden layers so quickly , stacks of back-to-back convolutional layers are needed to build really deep networks . Zeiler et . al ( Zeiler & Fergus , 2013 ) introduce Stochastic Pooling where the act of picking the maximum value in each pooling region is replaced by a form of size-biased sampling . Graham et . al ( Graham , 2014 ) introduce fractional max-pooling ( FMP ) . The idea of FMP is to reduce the spatial size of the image by a factor of α with 1 ≤ α ≤ 2 . Their solutions reduce the information loss of the pooling layer but still left the problem behind . 3 ATTENTION INCORPORATE NETWORK . Our Attention Incorporate Network constructed by stacking multiple local attention incorporate layer ( LAIL ) and a global attention incorporate layer ( GAIL ) before the FC-layer . The output of LAIN can be any size depend on the input , and the output of GAIN will be the fixed size design by the network structure to ensure that no size conflict between the GAIN and FC-layer . And for both GAIL and LAIL share exactly the same mathematical calculation . 3.1 ATTENTION INCORPORATE LAYER . Forward Pass . Traditional convolutional feed-forward networks connect the output of the ` th layer as input to the ( ` + 1 ) th layer ( Krizhevsky et al. , 2012 ) . The transition layer in AlexNet ( Krizhevsky et al. , 2012 ) , Resnet ( He et al. , 2016 ) fellow the equation ( 1 ) ( 2 ) : X ` +1 =MaxPooling ( X ` ) ( 1 ) X ` +1 = Cov ( X ` , stride = 2 ) ( 2 ) We propose AIL instead of those traditional sharpen transition layer . Suppose that Xin is the input of AIL with the size of ( M , N , c ) , Xout is the out of AIL with the size of ( Ms , N s , c ’ ) , and the mathematical principle of AIL follow these equations : X = Relu ( Cov ( Xin , 1× 1 ) ) ( 3 ) W = Sigmoid ( SeparableConv ( Xin , k × k ) ) ( 4 ) X in equation ( 3 ) extract the content matrix of the input with the operation RELU after Cov1×1 . W in equation ( 4 ) extract the attention matrix with the operation Sigmoid after SeparableConv k×k . Both of them have the size of ( M , N , c ’ ) . The motion of choosing this setting is that the interaction between channel is enough for transmit the information in content matrix X . We design the W matrix as the mixture of spatial information , k in equation ( 4 ) refer to the kernel size which is a hyper-parameter in AIN and it should be an integer greater than 1 . Experimentally we choose k=3 as the default value by fine tuning k in our three experiments . Separable Convolution which made up of Depthwise ( Chollet , 2016 ) and Conv1×1 . We choose it for the reason of model compression . Window shifting is one of the basic operation in convolutional networks , the operation is shown in Fig . 2 . For mathematically intuitive , we are going to do the calculation on one of the selected window . Suppose that X̃ , W̃ is one of the selected window from X , W . Both of X̃ , W̃ have the size of ( m , n , c ’ ) . Suppose that Xout is the output of AIL with the shape of ( 1,1 , c ’ ) . X̃out , k = ∑m i ∑n j ( W̃i , j , k · X̃i , j , k ) ∑m i ∑n j W̃i , j , k + ( 6 ) X̃out , k stands for the value of X̃out in kth channel . Xout equals to the stack by X̃out from all the sampling window in equation ( 5 ) . is a small number equals to 10−8 to make sure the fraction won ’ t divide by zero . Kernel size m , n can arbitrary value during the feed-forward calculation . If the kernel m=n=constant then the calculation present as Conv+Pooling with attention mechanism , we name it as local attention incorporate layer ( LAIL ) . If m=M and n=N , there is only one window produced during the sampling , the window represent the whole image , the calculation extract the global information of the input image . We name it as global attention incorporate layer ( GAIL ) . For more concretely , the forward operation shows the following ( Fig . 3 ) Backward Pass . X̃ , W̃ in equation ( 5 ) have different representation of content and weight . These matrix learnt by the backward equation ( 7 ) ( 8 ) : ∂X̃out , k ∂X̃x , y , k = W̃x , y , k∑m i ∑n j W̃i , j , k + ( 7 ) ∂X̃out , k ∂W̃x , y , k = ∑m i ∑n j W̃i , j , k · ( X̃x , y , k − X̃i , j , k ) ( ∑m i ∑n j W̃ 2 i , j , k + ) ( 8 ) ∂X̃out , k ∂X̃x , y , k stands for the gradient in the X̃x , y , k passed byX̃out , k , it ’ s pretty obvious that if the weight matrix W has a high value in point ( x , y , k ) , the gradient of content X in this point will also have a high value . In global vision that if there are something important in this image , AIL will give a high gradient on those important area . ∂X̃out , k ∂W̃x , y , k stands for the gradient in the W̃x , y , k passed byX̃out , k , from the equation ( 8 ) we can know that AIL will likely to give a higher gradient to the point ( x , y , k ) if constant X̃ in this point has a high variance to other points . 3.2 NETWORK STRUCTURE . DenseNet ( Huang et al. , 2016 ) achieve a high accuracy in image classification problem . We use the Dense Block as the basic block in AIN for the reason that DenseNet is widely used in image classification problem and require substantially fewer parameters and less computation to achieve a reasonable good performances . We implement AIN in the kaggle competition of Furniture-1281 which has various image size for training . The specific networks are shown in Table . 1 Local Attention Incorporate Layer . The size of input tensors will decrease by 1 s ( s=stride ) through LAIL , matrix Weights produce by Cov+Sigmoid which gives a value between 0-1 as an attention gate , matrix X produce by standard Cov+Relu which contains the key information of the image . The weighted average output is the sum of the matrix X each weighted by matrix Weights . A simple visualization example of LAIL is shown in Fig . 4 . Global Attention Incorporate Layer . GAIL was designed to solve the arbitrary inputs problem . The output of GAIL will be a fixed size such as 1 × 1 × 512 what ever the input size is . The computational logic of GAIL and LAIL are exactly the same . GAIL is the last layer of convolutional layers , the feed-forward feature will be highly abstract ( e.g . Whether it is a object or the unimportant background ) . The image information will be transferred to the fixed number of channels to make sure that where will be no conflict between GAIL and Fully-Connected Layer . A visualization example of GAIL is shown in Fig . 5 . Implement Details . All image data set can use the network structure in Table . 1 by tuning some model parameters ( e.g . number of AIL , number of class ) . Experimentally we used AIN-121 to train the model in kaggle competition of Furniture-128 . Image size in the data set have various size . AIN-121 include 4×LAIL , 4×DenseBlock and 1×GAIL before the Fully-Connected layer . All of the LAIL have the kernel size of 3×3 . Dense Block are exactly the same as DenseNet ( Huang et al. , 2016 ) . Part of the training data visualized in Fig . 6

Summary: This paper presents a strategy to overcome the limitation of fixed input image sizes in CNN classifiers. To this end, the authors incorporate some local and local attention modules, which fit inputs of arbitrary size to the fixed-size fully connected layer of a  CNN. The method is evaluated on three public classification benchmarks: CIFAR-10, ImageNet and Kaggle-Furniture128.