A Gradient-Based Approach to Neural Networks Structure Learning

1 INTRODUCTION . During the last few years , deep neural networks ( DNNs ) have played an incomparable role in addressing challenging problems of different areas , most impressively computer vision ( Simonyan & Zisserman ( 2014 ) ; He et al . ( 2016 ) ) . However , well-designing the architecture of a DNN requires a lot of expertise and is usually a cumbersome and time-consuming task , since many possible configurations are assessed before the best one is found . Additionally , the performance of different architectures heavily relies on the nature of the task and the data . There are several studies ( Zagoruyko & Komodakis ( 2016 ) ; Szegedy et al . ( 2016 ) ) providing insight on how to tune models with state of the art performance for different tasks . Moreover , many studies ( Zoph & Le ( 2016 ) ; Cai et al . ( 2018 ) ; Rohekar et al . ( 2018 ) ; Zhong et al . ( 2018 ) ) have been accomplished recently to avoid architecture engineering . These methods are based on network architecture search over the space of possible architectures ( Elsken et al . ( 2019 ) ) . However , these approaches control the structure externally , hence adding a computational overhead when training the model . On the other hand , some studies ( Li et al . ( 2016 ) ; Han et al . ( 2015 ) ) show that it is possible to reduce the number of network parameters and still achieve comparable performance . These smaller networks have the benefit of requiring less storage space and less computational power during inference . These findings also show that there is still much room for improvement in designing efficient neural network architectures . In this paper , we propose a new neural network model , called Farfalle Neural Network ( FNN ) in which the trainable parameters are not the weights on the connections between neurons . Instead , the network learns embedding vectors for neurons and uses these vectors to determine the weights of neural connections . More importantly , in the proposed method , instead of hand-crafting the network architecture , it is learned during the training process . The connections between neurons are indirectly specified according to neuron embeddings . Therefore , the proposed network configures its structure itself during the training process given solely the number of nodes and an upper bound on the network depth . We also establish the effectiveness of our models through various experiments . In particular , we show that our model is able to replace fully connected networks achieving higher performance with a 70 % reduction in the number of parameters . 2 RELATED WORK . Recent proposed methods for automatic design of neural networks are commonly focused on treating the architecture decisions as hyper-parameters . These methods use either a supervised or an unsupervised approach to optimize these hyper-parameters . There are several approaches ( Zoph & Le ( 2016 ) ; Baker et al . ( 2016 ) ; Zhong et al . ( 2018 ) ) that utilize reinforcement learning for effectively searching the design space . These approaches usually have a lot of computational overhead because of their need to compute the model ’ s accuracy during the search in the space of architectures Elsken et al . ( 2019 ) . To circumvent this issue , Smithson et al . ( 2016 ) uses another neural network to estimate the trained model ’ s accuracy . However , training the estimator network is itself a computationally expensive task . Rohekar et al . ( 2018 ) propose a lightweight unsupervised approach using Bayesian network structure learning . Using this approach , they replace fully connected layers at the end of known networks such as VGGNet with smaller models while still showing a comparable performance in accuracy . Note that while the reported results show the effectiveness of this method , it still optimizes the structure externally . Hence , it requires an additional environment setup and have an external overhead in the learning process , though the latter is reported to be reasonably small . One of the key ideas in our proposed method is to assign embedding vectors to neurons of the network and use the attention mechanism to relate them . Similar ideas appear in the Transformer network ( Vaswani et al . ( 2017 ) ) and CapsNet ( Sabour et al . ( 2017 ) ) . In Transformer networks ( Vaswani et al . ( 2017 ) ) input words and their positional information are embedded in a low-dimensional space . However , they utilized a specific case of attention called self-attention to relate different parts of the sequence . In addition , the embeddings used in that architecture are not trained for the purpose of structure learning . CapsNet ( Sabour et al . ( 2017 ) ) considers an output vector for each capsule and routes the outputs from one capsule to the next layer ’ s capsules according to its ability to predict the output vector of those capsules . However , CapsNets still use weight matrices between neurons and also are not able to self configure their structure . 3 FARFALLE NEURAL NETWORKS . In traditional neural networks , the number of neurons in each layer and the arrangement of the neurons is fixed . This rigid configuration prevents straightforward optimization of the network structure during the training process . Therefore , finding a network with proper structure requires testing a lot of configurations . In contrast , FNNs utilize a new type of neurons which can float and find the most suitable neurons to obtain information from them . The connections between these floating neurons are defined based on their relative similarities . Thus , during the training process , relevant neurons move toward each other to strengthen their connection . Figure 1 shows how these neurons float to obtain more relevant information . 3.1 FLOATING NEURONS . A floating neuron gathers information from relevant neurons at its input , transforms it with a trainable transformation , and emits the result at its output . In order to avoid confusion , we might refer to a neuron ’ s input as its head . Similarly , we sometimes refer to its output as the neuron ’ s tail . Inputs and outputs of these neurons are embedded in a d-dimensional space . These embedding vectors regulate the weights connecting relevant neurons and are updated during the training process . Figure 2 shows a schematic representation of a floating neuron and its connections . Specifically , a floating neuron v consists of three parts : • Input embedding : A trainable d-dimensional vector Iv which indicates the coordinates of the neuron ’ s head . The similarity between this vector and output embedding of other neurons determines the connection weights between this neuron and other neurons using the attention mechanism . • Transformation function : A trainable nonlinear function Fv that transforms the gathered information . The transformation used in this study is of the form Fv ( x ) = ReLU ( av ·x+bv ) where av and bv are neuron-specific trainable parameters . • Output embedding : A trainable d-dimensional vector Ov which indicates the coordinates of the neuron ’ s tail . In addition to normal floating neurons , there are two custom types of floating neurons : Input neurons , which receive the input of the whole network and output neurons , which provide the processed data to the outside . Consequently , input neurons do not have input embedding and output neurons do not have output embedding . 3.2 CONSTRUCTION OF MULTI-LAYER FLOATING NEURAL NETWORKS . Before introducing FNNs , we discuss how to employ floating neurons in a layered structure . In order to form such a network , floating neurons are grouped in layers . The neurons at each layer obtain their values from neurons at the previous layer . The connections between neurons of two consecutive layers are defined based on the attention mechanism . Formally , suppose v is a neuron in the layer i + 1 and u1 , u2 , . . . , uM are neurons of the previous layer , i.e . layer i . The weights connecting v to related neurons is defined by w1 , w2 , . . . , wM = N ( ITv Ou1 , ITv Ou2 , . . . , ITv OuM ) ( 1 ) where Iv is the input embedding of neuron v , Oui is the output embedding of neuron ui , and N is a normalization function . For normalization , one can choose softmax function to force each neuron ’ s input to be a convex combination of the outputs of the neurons in the previous layer . We found l2 normalization to work best in our experiments and thus the following function is used for normalization Nl2 ( x1 , x2 , . . . , xn ) = x1∑ i x 2 i , x2∑ i x 2 i , . . . , xn∑ i x 2 i . ( 2 ) Utilizing these weights , given y1 , y2 , . . . , yM as the values of neurons u1 , u2 , . . . , uM , the output value of neuron v will be Fv ( ∑ m wiyi ) where Fv is the transformation function of neuron v. To efficiently connect neurons v1 , v2 , . . . , vN of the layer i+ 1 to neurons u1 , u2 , . . . , uM of the layer i , let I = [ Iv1 |Iv2 | · · · |IvN ] be the concatenation of input embedding vectors of the layer i + 1 and O = [ Ou1 |Ou2 | . . . |OuM ] be the concatenation of output embedding vectors of the layer i . Then , the weights connecting neurons of these two layers are determined by W = Ñ ( ITO ) ( 3 ) where Ñ is a function normalizing each row of its input matrix according to the normalization function N . Finally , the output of layer i+ 1 will be Z = F̃ ( WY ) ( 4 ) where Y is the vector containing output values of layer i and F̃i = Fvi . Using this model , the number of parameters required to connect a layer of size M to a layer of size N is of the order O ( d ( M + N ) ) . In contrast , in dense models the number of parameters for connecting two such layers would be of the order O ( MN ) . Hence , by using this model one of the limitations in designing neural networks , that is the huge number of parameters in the weight matrices , is resolved . 3.3 CONSTRUCTION OF FARFALLE NEURAL NETWORKS . An FNN is a recurrent network of floating neurons . Hence neurons in this architecture are used in iterations . In each iteration , floating neurons receive the output of the previous iteration along with the input . Figure 3 represents the architecture of FNNs . The expanded architecture is available in Supplementary Figure 7 . This recurrent structure allows neurons to process high-level information along with low-level features . Also , the floating nature of these neurons allows the network to balance the number of neurons employed in different levels of abstraction . Supplementary Figure 8 illustrates how this recurrent structure allows the network to combine information from different levels of abstraction . This flexibility allows the architecture to evolve during the training . Before describing the data flow of FNNs , let ’ s define an auxiliary function . For two groups of neurons V and U , equation ( 3 ) defines the weights connecting neurons in V to neurons in U . In this equationO and I are the concatenation of output embeddings of neurons in U and the concatenation of input embeddings of neurons in V respectively . Given Y as the values of neurons in U , equation ( 4 ) describes the output values of neurons in V . Combining these equations results in Step ( V , U , Y ) = F̃ ( Ñ ( ITO ) Y ) ( 5 ) where Ñ is a function normalizing each row of its input matrix and F̃ transforms values of each floating neuron using its own transformation function . The function Step can be seen as the combination of attention and neuronal transformation . Constructing an FNN for the input size of R and output size of S needs R input floating neurons for feeding data to the network , N floating neurons for processing data in k iterations , and S output floating neurons for the final deduction from hidden neurons . Here , N and k are the hyperparameters of the network . Let ’ s call the input neurons I = i1 , i2 , . . . , iR , the hidden neurons V = v1 , v2 , . . . , vN , and the output neuronsO = o1 , o2 , . . . , oS . The following procedure describes the flow of the network : 1 . Each input neuron assigns an embedding vector to its input variable after applying its transformation function , i.e . given input x = ( x1 , x2 , . . . , xR ) , input neurons will provide values Y0 = Fi1 ( x1 ) , Fi2 ( x2 ) , . . . , FiR ( xR ) at locations Oi1 , Oi2 , . . . , OiR . 2 . In the first step of the iterative part , hidden neurons process the data provided by input neurons . Thus , the resulting values of this step is Y1 = Step ( V , I , Y0 ) . 3 . In iteration 1 < j ≤ k , each hidden neuron process Y0 along with the outputs of all hidden neurons in the previous iteration ( Yj−1 ) and produce Yj . Indeed , given Y0 and Yj−1 hidden floating neurons will provide Yj = Step ( V , [ I|V ] , [ Y0 , Yj−1 ] ) . 4 . Finally , utilizing output neurons the final output will be Step ( O , V , Yk ) . The following theorem shows how an FNN with its recurrent structure can model a multi-layer floating neural network . Theorem 1 . Every multi-layer floating neural network with l layers , total of N floating neurons , and embedding dimension d can be modeled by an FNN containing N + 1 floating neurons with embedding dimension of l · d which iterates for l iterations . Proof . Suppose a multi-layer floating neural networkN1 with layer sizes of n1 , n2 , . . . , nl is given . Let i1 , i2 , . . . , iR be the input floating neurons , v j 1 , v j 2 , . . . , v j nj be the floating neurons in layer j , and o1 , o2 , . . . , oS be the output floating neurons of the network . To construct an FNN N2 with the same functionality , define a network with input neurons i′1 , i ′ 2 , . . . , i ′ R , output neurons o ′ 1 , o ′ 2 , . . . , o ′ S , and hidden neurons v ′ 0 , v ′1 1 , v ′1 2 , . . . , v ′l nl such that v′ji correspond to hidden neuron vji of N1 . Let all floating neurons in N2 have the same transformation function as their corresponding neuron in N1 and v′0 be a neuron with Fv′0 = 0 · x and Iv′0 = Ov′0 = ( , . . . , ) where is a negligible values . Before defining neural embeddings of N2 , for 0 < j ≤ l define Ej ( x1 , x2 , . . . , xd ) = ( ( j−1 ) ·d︷ ︸︸ ︷ 0 , . . . , 0 , x1 , x2 , . . . , xd , ( l−j ) ·d︷ ︸︸ ︷ 0 , . . . , 0 ) . This implies that Ei ( x ) TEj ( y ) = { xT y i = j 0 i 6= j . ( 6 ) Now , define all remaining embedding vectors of N2 by Oi′r =E1 ( Oir ) Io′s =E1 ( Ios ) Ov′ji =E ( j mode l ) +1 ( Ovji ) Iv′ji =Ej ( Ivji ) . These definitions along Eq . ( 6 ) result that in the first iteration , the only neurons which have a non-zero connection with input neurons are v′11 , v ′1 2 , . . . , v ′1 n1 . In addition , the weights of these connections are the same as the connections in the first layer of N1 . Similarly , by mathematical induction over j we can state that floating neurons v′j1 , v ′j 2 , . . . , v ′j nj can only connect to v′j−11 , v ′j−1 2 , . . . , v ′j−1 nj−1 , the connection weights are the same as those in layer j of N1 , and iteration j is the first time v′j1 , v ′j 2 , . . . , v ′j nj can get nonzero values . This implies that the connections of hidden floating neurons of N2 in l iterations , form a chain similar to the architecture of N1 . Finally , Io′s is just connected to v ′l 1 , v ′l 2 , . . . , v ′l nl with the same weights as in N1 . This completes the proof . It is worth mentioning that neuron v′0 is defined to eliminate the division by zero occurred in Eq . ( 2 ) when a neuron does not have a non-zero dot product to any input neuron . Note that by choosing a right value for d , traditional neural networks can be replaced with multi-layer floating neural networks introduced in this paper . Additionally , Theorem 1 implies that the farfalle neural networks are more general than multi-layer floating neural networks . Thus , as a corollary we note that it is possible to replace traditional multi-layer neural networks with the recurrent structure of FNNs . Furthermore , there is a repeated observation that replacing weights of a neural network with their low-rank approximations gives a comparable ( or even improved ) performance ( Sainath et al . ( 2013 ) ; Denil et al . ( 2013 ) ; Denton et al . ( 2014 ) ) . Such approximation allows us to replace dense networks with FFNs with small values of d and thus reasonable number of parameters . Utilizing this model , more flexibility in specifying the layer sizes is provided accordingly . Furthermore , since we do not need to have weight parameters explicitly , we can consider the whole network as a fully connected structure in which each ( hidden ) neuron can be potentially connected to all other neurons and even itself .

Summary: This paper proposes a new architecture based on attention model to replace the fully-connected layers. In this architecture, each neuron is associated with an embedding vector, based on which the attention scores (between two consecutive layers) are calculated, and the computational flow through the layers are derived based on these attention scores. The experiments on MNIST and CIFAR demonstrate some degree of superiority over plan FC layers.