A Gradient-Based Approach to Neural Networks Structure Learning
Authors: Amir Ali Moinfar
Amirkeivan Mohtashami
Mahdieh Soleymani
Ali Sharifi-Zarchi
Venue:  (2020)

Abstract: Designing the architecture of deep neural networks (DNNs) requires human expertise and is a cumbersome task. One approach to automatize this task has been considering DNN architecture parameters such as the number of layers, the number of neurons per layer, or the activation function of each layer as  hyper-parameters, and using an external method for optimizing it. Here we propose a novel neural network model, called Farfalle Neural Network, in which important architecture features such as the number of neurons in each layer and the wiring among the neurons are automatically learned during the training process. We show that the proposed model can replace a stack of dense layers, which is used as a part of many DNN architectures. It can achieve higher accuracy using significantly fewer parameters.  

Under review as a conference paper at ICLR 2020
A GRADIENT-BASED SELF-ORGANIZING NEURAL NETWORK
Anonymous authors Paper under double-blind review
ABSTRACT
Designing the architecture of deep neural networks (DNNs) is a cumbersome task that requires significant human expertise. There have been attempts automatize this task, mainly by considering DNN architecture parameters such as the num- ber of layers, the number of neurons and the activation function of each layer as typer-parameters, and using external method for optimizing them. This frame- work is computationally too expensive since each myper-parameee optimization iteration required a whole model training. Here propose Farfalle Neural Net- works, a novel recurrent structure in which neurons connect each other through their trainable embeddings. Hence, important architecture features such the number of neurons in each layer and the wiring among the neurons are automati- cally learned during the training process. We theoretically prove that the proposed model can replace stack of dense layers, which is used as part of many DNN architectures. By comparing the performance of dense networks and FNNs we show FNNs achieve comparable or higher accuracy using significantly fewer pa- rameters.
INTRODUCTION
During the last few years, deep neural networks (DNNs) have played an incomparable role in ad- dressing challenging problems of different areas, most impressively computer vision (Simonyan & Zisserman (2014); He et al. (2016)). However, well-designing the architecture of DNN requires a lot of expertise and is usually cumbersome and time-consuming task, since many possible con- figurations are assessed before the best one found. Additionally, the performance of different architectures heavily relies on the nature of the task and the data.
There are several studies (Zagoruyko & Komodakis (2016); Szegedy et al. (2016)) providing insight on how to tune models with state of the art performance for different tasks. Moreover, many studies (Zoph & Le (2016); Cai et al. (2018); Rohekar et al. (2018); Zhong et al. (2018)) have been accom- plished recently to avoid architecture engineering. These methods are based network architecture search over the space of possible architectures (Elsken et al. (2019)). However, these approaches control the structure externally, hence adding computational overhead when training the model. On the other hand, some studies (Li et al. (2016); Han et al. (2015)) show that it is possible to re- duce the number of network parameters and still achieve comparable performance. These smaller networks have the benefit of requiring less storage space and less computational power during infer- ence. These findings also show that there still much room for improvement designing efficient neural network architectures.
In this paper, we propose new neural network model, called Farfalle Neural Network (FNN) in which the trainable parameters are not the weights on the connections between neurons. Instead, the network learns embedding vectors for neurons and uses these vectors to determine the weights of neural connections. More importantly, in the proposed method, instead of hand-crafting the network architecture, it is learned during the training process. The connections between neurons are indirectly specified according to neuron embeddings. Therefore, the proposed network configures its structure itself during the training process given solely the number of nodes and an upper bound on the network depth. We also establish the effectiveness of models through various experiments. In particular, we show that our model is able to replace fully connected networks achieving higher performance with 70% reduction in the number of parameters.

---

Under review as a conference paper at ICLR 2020
Neurons with relevant output
Neuron needing information
Neurons with irreevant output
Figure I:
The intuition behind floating neurons. The connections between floating neurons are defined based on their relative similarities. Thus, neurons needing an information and providers of it tend to approach each other.
2 RELATED WORK
Recent proposed methods for automatic design of neural networks are commonly focused on treat- ing the architecture decisions as hyper-parametens These methods use either a supervised or an unsupervised approach to optimize these Le (2016); Baker al. (2016); Zhong et al. (2018)) that utilize reinforcement learning for effec- tively searching the design space. These approaches usually have lot of computational overhead because of their need to compute the model's accuracy during the search in the space of architectures Elsken et al. (2019). To circumvent this issue, Smithson et al. (2016) to estimate the trained model's s accuracy. However, training the estimator network is itself a compu- tationally expensive task. Rohekar et al. (2018) propose a lightweight unsupervised approach using Bayesian network structure learning. Using this approach, they replace fully connected layers at the end of known networks such as VGGNet with smaller models while still showing a comparable per- formance in accuracy. Note that while the reported results show the effectiveness of this method, still optimizes the structure externally. Hence, requires an additional environment setup and have an external overhead in the learning process, though the latter is reported to be reasonably small.
There are several approaches (Zoph &
another neural network
One of the key ideas in our proposed method is to assign embedding vectors to neurons of the net- work and use the attention mechanism to relate them. Similar ideas appear in the Transformer net- work (Vaswani al. (2017)) and CapsNet (Sabour et al. (2017)). Transformer networks (Vaswani al. (2017)) input words and their positional information However, they utilized a specific case of attention called self-attention to relate different parts the sequence. In addition, the embeddings used in that architecture not trained for the purpose of structure learning. CapsNet (Sabour et al. (2017)) considers an output vector for each capsule and routes the outputs from capsule to the next layer's 's capsules according to its ability to predict the output vector of those capsules. However, CapsNets still use weight matrices between neurons and also not able self configure their structure.
embedded in low-dimensional space.
FARFALLE NEURAL NETWORKS
In traditional neural networks, the number of neurons in each layer and the arrangement of the neu- rons is fixed. This rigid configuration prevents straightforward optimization of the network structure during the training process. Therefore, finding network with proper structure requires testing lot of configurations.
In contrast, FNNs utilize a new type of neurons which can float and find the most suitable neurons obtain information from them. The connections between these floating neurons are defined based on their relative similarities. Thus, during the training process, relevant neurons move toward each other to strengthen their connection. Figure shows how these neurons float to obtain more relevant information.
3.1 FLOATING NEURONS
A floating neuron gathers information from relevant neurons at its input, transforms it with train- able transformation, and emits the result at its output. In order to avoid confusion, we might refer to neuron's input its head. Similarly, sometimes refer to its output the neuron's tail. Inputs
2

---

Under review as a conference paper at ICLR 2020
Trainable Input Embedding (l2)
Trainable Output
Embedding (02)
Trainable Transformation:
F.(2) ReLU(a,-x++bb)
Figure 2: schematic representation of a floating neuron and its connections. Each neuron has an input embedding, output embedding, and transformation function.
and outputs of these neurons are embedded d-dimensional space. These embedding vectors reg- ulate the weights connecting relevant neurons and are updated during the training process. Figure 2 shows schematic representation of a floating neuron and its connections. Specifically, a floating neuron v consists of three parts:
Input embedding: A trainable d-dimensional vector L, which indicates the coordinates of the neuron's is head. The similarity between this vector and output embedding of other neurons determines the connection weights between this neuron and other neurons using the attention mechanism.
A trainable nonlinear function F. that transforms the gathered information. The transformation used in this study is of the form (x (a. where a and b, are neuron-specifi trainable parameters.
Transformation function:
Output embedding: trainable d-dimensional vector O, which indicates the coordinates of the neuron's 's tail.
In addition to normal floating neurons, there are two custom types of floating neurons: Input neurons, which receive the input of the whole network and output neurons, which provide the processed data to the outside. Consequently, input neurons do not have input embedding and output neurons do not have output embedding.
3.2
CONSTRUCTION OF MULTI-LAYER FLOATING NEURAL NETWORKS
Before introducing FNNs, we discuss how to employ floating neurons in a layered structure. In order form such a network, floating neurons grouped in layers. The neurons each layer obtain their values from neurons at the previous layer. The connections between neurons of two consecutive layers are defined based the attention mechanism.
Formally, suppose v is neuron in the layer + 1 and u1 U2,..., are neurons of the previous layer, i.c. layer The weights connecting v to related neurons defined by
(1) where L is the input embedding of neuron Ou, is the output embedding of neuron uy and V normalization function. For normalization, one can choose softmax function to force each neuron's 's input to be convex combination of the outputs of the neurons in the previous layer. We found 12 normalization work best in our experiments and thus the following function is used for normalization
W1, .,, WM = NNT Ou, 1T" O"'Ou,..... Own))
NI2(I1, D2, Fa)
12
n
(2)
Σ,
Σ 2
Utilizing these weights, given y/, 1/2 ......... PM as the values of neurons W1 U2..... UM. the output value of neuron will be Fo(E.. ways) where F, is the transformation function of neuron v. To efficiently connect neurons U1, U2,..., UN of the layer + 1 to neurons U1, U2.... UM of the layer i, let / 4021 Log] ] be the concatenation of input embedding vectors of the layer i + 1 and O = [O., 102l ... |maz be the concatenation of output embedding vectors of the layer Then, the weights connecting neurons of these two layers are determined
W = N (1"
(3)
where N is a function normalizing each row of its input matrix according to the normalization function N. Finally, the output layer + will be
7 F(WY)
3

---

Under review as a conference paper at ICLR 2020
Floating Neurons
Input Neurons
Output Neurons
Figure 3: The architecture of Farfalle Neural Networks.
where Y the vector containing output values of layer and F, F...
Using this model, the number of parameters required to connect a layer of size M to a layer of size N of the order ((d(M + N)). In contrast, in dense models the number of parameters for connecting two such layers would be of the order (MMN). Hence, by using this model one of the limitations designing neural networks, that is the huge number of parameters the weight matrices, is resolved.
3.3 CONSTRUCTION OF FARFALLE NEURAL NETWORKS
An FNN is a recurrent network of floating neurons. Hence neurons in this architecture are used in iterations. In each iteration. floating neurons receive the output of the previous iteration along with the input. Figure represents the architecture of FNNs. The expanded architecture available in Supplementary Figure 7. This recurrent structure allows neurons process high-level information along with low-level features. Also, the floating nature of these neurons allows the network to balance the number of neurons employed in different levels of abstraction. Supplementary Figure 8 illustrates how this recurrent structure allows the network to combine information from different levels of abstraction. This flexibility allows the architecture to evolve during the training.
Before describing the data flow of FNNs, let's define an auxiliary function. For two groups of neurons V and U, equation (3) defines the weights connecting neurons in to neurons in U. this equation O and / are the concatenation of output embeddings of neurons in U7 and the concatenation of input embeddings of neurons V respectively. Given Y the values of neurons in U, equation (4) describes the output values of neurons in V. Combining these equations results in
Step U. Y) = F( (T"
(5)
where N function normalizing each row of its input matrix and transforms values of each floating neuron using its own transformation function. The function Step can be seen as the combi- nation of attention and neuronal transformation.
Constructing an FNN for the input size of R and output size of S needs R input floating neu- rons for feeding data to the network, floating neurons for processing data in k iterations, and output floating neurons for the final deduction from hidden neurons. Here, N and & are the hyper- parameters of the network. Let's call the input neurons I 11, .2,..., ,,g the hidden neurons .2, V U1 . . UN, and the output neurons O 01, 02 ... Os. The following procedure describes the flow of the network:
1. Each input neuron assigns embedding vector its input variable after applying its trans- formation function, i.c. given input = (I1, T2..... æR), input neurons will provide val- ues Yo F., (æ1), (1,)))).... Fix (œR) at locations O,, O,,...., Otn.
2. In the first step of the iterative part, hidden neurons process the data provided by input neurons. Thus, the resulting values of this step is Y; = Step(V, Yo).
3. In iteration 1 < j S k each hidden neuron process Y, along with the outputs of all hidden neurons in the previous iteration (Y and produce Yj. Indeed, given Yo and Y5-1 hidden floating neurons will provide Y, Step(V, [Σ])], (Yo, Y,-1)).
4. Finally, utilizing output neurons the final output will be Step(O, Ye).
4

---

Under review as a conference paper at ICLR 2020
The following theorem shows how FNN with its recurrent structure model multi-layer floating neural network.
Theorem 1. Every multi-layer floating neural network with I layers, total of N floating neurons, and embedding dimension d can be modeled by an FNN containing N + floating neurons with embedding dimension of I. d which iterates for iterations.
Proof. Suppose multi-layer floating neural network N1 with layer sizes of 01, 722. .., ny is given. Let i1, ,., in be the input floating neurons, ví, u2...., ohy be the floating neurons in layer j. and 01, .2, os be the output floating neurons of the network.
To construct an FNN N2 with the same functionality, define a network with input neurons 14, RR. output neurons of, o2...., dg. and hidden neurons v/o, v1', n2 ....... on such that of correspond to hidden neuron oy of N1. Let all floating neurons in N2 have the same transfor- mation function as their corresponding neuron in N1 and to be a neuron with Fog 0 and
To = Ogg = (c....,() where is negligible values. Before defining neural embeddings of N2, for 0 < j S define
(j-1)dd
(I .dd
,.a) = (O.... O, r1, 22
Ld
0,
This implies that
fIty i=
E,(:)
(6)
(0
itj
Now, define all remaining embedding vectors of N2 by
O₂ =Ei (O;,) 0gf = Eij made t)+1 (0
To; =1 (Io.) L0p E, (Lx))
These definitions along Eq. (6) result that in the first iteration, the only neurons which have a non-zero connection with input neurons are oft 122 ,..., 12g In addition, the weights of these connections are the same as the connections in the first layer of N1. Similarly, by mathemat- ical induction over j we can state that floating neurons of 12 ........ "..,,,,,,,,, can only connect to vf -/2-1, , n2
-/ the connection weights are the same as those layer j of N1, and iter- ation j is the first time vf² off. :....,""" v// can get nonzero values. This implies that the connections of hidden floating neurons of N2 in iterations, form chain similar to the architecture of N1. Finally, lo is just connected to v/", v// ...., of with the same weights as in N1. This completes the proof. It is worth mentioning that neuron vn is defined to eliminate the division by zero occurred in Eq. (2) when a neuron does not have non-zero dot product to any input neuron.
Note that by choosing right value for d, traditional neural networks can be replaced with multi-layer floating neural networks introduced in this paper. Additionally, Theorem implies that the farfalle neural networks are more general than multi-layer floating neural networks. Thus, as a corollary we note that is possible to replace traditional multi-layer neural networks with the recurrent structure of FNNs.
Furthermore, there is a repeated observation that replacing weights of neural network with their low-rank approximations gives comparable (or even improved) performance (Sainath et al. (2013); Denil et al. (2013); Denton et al. (2014)). Such approximation allows us to replace dense networks with FENs with small values of and thus reasonable number of parameters.
Utilizing this model, more flexibility in specifying the layer sizes is provided accordingly. Further- more, since do not need to have weight parameters explicitly, as fully connected structure which each (hidden) neuron can be potentially connected to all other neurons and even itself.
consider the whole network
5

---

Under review as a conference paper at ICLR 2020
3.4 SCALABILITY & PRODUCTION
During training, the weight matrix W needs to be computed to apply the normalization function. Since the number of elements in this matrix is quadratic the number of neurons, it is possible that this matrix becomes quite big. However, after the training, it is possible to workaround the normalization step by updating the matrix I. Specifically, using the same notation as the last section, enough to replace I, with o,, This simplification significantly reduces the required memory space during inference since the dimension of the embedding space is usually much smaller than the number neurons.
Furthermore, although the need to compute the weight matrix during the training imposes practical limit on the maximum number of neurons in FNN, the upper bound is still very large. Additionally, is possible to stack FNNs similar to normal layers to employ more floating neurons. Such structure does not allow the use of information from all deeper layers but is still much more flexible than commonly used dense layers.
4 RESULTS & DISCUSSION
In the following subsections, we present comparison results our model with other DNN archi- tectures CIFAR (Krizhevsky et al. (2009)), widely-used image classification dataset. First, compare FNNs with a dense model and show that our model can outperform them. Then we discuss some characteristics of FNNs, such their ability to learn locality. In the final section, discuss how our model can be integrated with existing convolutional neural networks.
4.1 COMPARISON WITH DENSE MODELS
Although dense models are not among the state the art models for neither datasets, the goal of this section is establish the effectiveness of our model in comparison with dense models. We do not claim that our model, in any way, can directly outperform highly specialized models such as convolutional neural networks. Instead, we demonstrate how model may be used in conjunction with CNNs in subsequent sections.
In order to obtain a proper baseline for CIFAR, we used HyperOpt Bergstra et al. (2013) to search among fully connected layers with up to 4 hidden layers. Number of neurons in each layer was chosen from (500, 1000, 1500, 2000). The top performing model was a fully connected network with 2 hidden layers of 2000 and 500 neurons, in order from input to output.
We trained a multi-layer floating network with the same structure and embedding size 256. We also trained a FNN the same number of neurons and embedding size. The number of iterations was set to Furthermore, we trained a smaller fully connected network with the same number of parameters as our models by reducing the number of neurons in the first hidden layer of baseline to 850. We trained all four models using Adam (Kingma Ba (2015)) algorithm for 200 epochs. In all models applied 0.1 dropout (Srivastava et al. (2014)) rate of the input and used ReLU (Glorot et al. (2011)) for activation function.
Summarized results are shown in Table 1. It is evident that our models significantly outperform the fully connected networks. This is especially significant since there is more than 70% reduction in number of parameters in multi-layer floating network with respect to the fully connected network of the same architecture. The performance gap is worsened when comparing with the smaller base-line.
The results show that FNNs able to obtain comparable results. The slightly better performance of multi-layer floating network perform with respect to FNN is not unexpected. The structure for the multi-layer floating network is the same the baseline which was chosen by searching through the architecture space. Moreover, the FNN was trained with almost no hyper-parameter tuning.
4.2 ANALYSIS OF LEARNED EMBEDDINGS
We analyzed the embeddings of an FNN with 1024 neurons, 256 embedding size, and 1 iteration trained on MNIST (LeCun al. (1998a)). The training setting the same as the previous section.

---

Under review as a conference paper at ICLR 2020
Model
Top-1 Accuracy(%)
#f Parameters
Baseline
53.69
7151510
Small-Baseline
52.29
2077850
Floating Network
63.18
2080156
FNN
63.01
2080156
Table 1: The performance of FFNs in comparison with baseline method.
Figure The 12-norm of each input neuron's Figure The input embeddings are projected embedding is calculated and plotted at its corre- to 2D space using UMAP. Only neurons in the sponding cell.
inner 20 20 box are included. Neurons corre- sponding to adjacent cells are connected with line.
In Figure 4, each cell is colored according to the (2-norm of its corresponding neuron's output embeddings. It can be seen that marginal neurons have much lower 12-norms. This means that the embeddings are much smaller for these neurons, and so they have little effect on the model's output. Note that this expected since the marginal pixels in MNIST images seldom provide useful information.
Figure depicts the projection of the learned embeddings 2D space. The projection is performed using Uniform Manifold Approximation and Projection (UMAP. McInnes et al. (2018)). Neurons corresponding to adjacent cells are connected with line. Marginal neurons are excluded in order to obtain a better projection of the embeddings. It is apparent in the figure that the learned embeddings respect the locality of pixels, so a pair of pixels close each other have similar embeddings. Hence, it can be seen that FNN is able to assign meaningful embeddings to the neurons.
4.3 INTEGRATION WITH CNNS
Convolutional neural networks (LeCun et al. (1998b)) are widely used image classification tasks and have been able to produce state of the art results. Commonly in such networks, convolutional layers are employed for feature extraction. The extracted features are then fed into several fully con- nected layers for classification. We propose that FNNs can be used to replace these fully connected layers.
To test this, we compared FNN with baseline model on CIFAR10 and CIFAR100 datasets. We use a slightly modified version VGG16 Simonyan & Zisserman (2014) which suitable for CIFAR as baseline. In this version, all layers after the last max-pooling layer are replaced with fully connected network with one hidden layer of 512 neurons. Batch normalization and a dropout rate of 0.5 is applied after the hidden layer. The same dropout rate is also applied before the hidden layer. We used ReLU as the activation function. A similar model has been used as a baseline in other studies (Rohekar et al. (2018); Li et al. (2016)).

---

Under review as a conference paper at ICLR 2020
Top-I Accuracy($e)
Model
CIFARIO
CIFAR100
VGGI6 + Dense
73.19 92.98
VGGI6 + FNN
93.51
73.33
Table 2: Top-1 accuracies when using normal dense layers FNN in VGG16.
We compare this baseline with alternative architecture consisting of FNN with 1024 neurons after the last max-pooling layer. The number of iterations was set to 3. We used Stochastic Gradi- Descent (SGD) with 0.9 momentum Rumelhart et al. (1988) for training and employed 0.0005 weight decay regularization.
The maximum test aceuracy of both models are presented in Table 2. It is evident that our model outperforms the dense layers on both datasets.
FUTURE WORK
We established that FNNs are able to replace and outperform fully connected layers.
FNNs are extreme case of sharing embeddings between layers of multi-layer floating neural network. We note that possible to use different patterns of sharing. Exploring this area might specially be interesting in the case of auto-encoders.
We conjecture that some of the neuron embeddings trained to solve one task can be used to solve similar tasks. The input neuron embeddings projection in Figure 5 supports this conjecture. How- ever, this task is not trivial and we leave it as a future work.
Another interesting direction is to assign different activation functions to different neurons in an FNN. This feature allows a combination of activation functions to be used together which is, to the best of our knowledge, something less explored in hand-crafted architectures.
CONCLUSION
In this paper, we introduced a method to learn the network structure internally during training. This was done mainly based the approach of assigning parameters the neurons instead of the connection between them. Using this approach, we introduced a novel neural network structure called Farfalle Neural Network. We established through experiments that this new structure can outperform dense layers in various scenarios while even sometimes using significantly (70%) lower number of parameters. We also discussed how this approach could significantly reduce the memory requirements during the inference process.
REFERENCES
Bowen Baker, Otkrist Gupta, Nikhil Naik, and Ramesh Raskar. Designing neural network architec- tures using reinforcement learning, arXiv preprint arXiv:1611.0216. 2016.
J. Bergstra, D. Yamins, and D. D. Cox. Making science of model search: Hy- perparameter optimization in hundreds of dimensions for vision architectures. In Pro- ceedings of the 30th International Conference on International Conference on Ma- chine Learning Volume 28, ICML'13, pp. 1-115-1-123. JMLR.org, 2013. URL http://dl. wcw. wwt.:///t...iiii..oo// wtt.ooommmmmmmmmm1..... 30428332.....
Han Cai, Tianyao Chen, Weinan Zhang, Yong Yu, and Jun Wang. Efficient architecture search by network transformation. In Thirty-Secone AAAI Conference on Artificial Intelligence, 2018.
Misha Denil, Babak Shakibi, Laurent Dinh, Mare' Aurelio Ranzato, and Nando De Freitas. Pre- dicting parameters in deep learning. Advances in neural information processing systems, pp- 2148-2156, 2013.
8

---

Under review as a conference paper at ICLR 2020
Emily L Denton, Wojeiech Zaremba, Joan Bruna, Yann LeCun, and Rob Fergus. Exploiting linear structure within convolutional networks for efficient evaluation. In Advances in neural informa- tion processing systems. pp. 1269-1277, 2014.
Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Neural architecture search: A survey.
Journal of Machine Learning Research, 20(55):1-21, 2019.
Xavier Glorot, Antoine Bordes, and Yoshua Bengio. Deep sparse rectifier neural networks. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pp- 315-323, 2011.
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. In Advances in neural information processing systems, pp. 1135-1143, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In 3rd Inter- national Conference Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Truck Proceedings, 2015. URL http: / httpa/aww...ggg//a/a.a/// abs /1422 6980.
Alex Krizhevsky et al. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11))2278-23) 2224,, November 1998a.
Yann LeCun, Léon Bottou, Yoshua Bengio, Patrick Haffner, al. Gradient-bart learning applied to document recognition. Proceedings of the IEEE, 86(1 1):2278-2324, 1998b.
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets, 2016.
Leland McInnes, John Hcaly, and James Melville. Umap: Uniform manifold approximation and projection for dimension reduction, 2018.
Raanan Y Rohekar, Shami Nisimov, Yaniy Gurwicz, Guy Koren, and Gal Novik. Constructing deep neural networks by bayesian network structure learning. In Advances Neural Information Processing Systems, pp. 3047-3058, 2018.
David Rumelhart, Geoffrey E Hinton, Ronald Williams, al. Learning representations by pack-propagating errors. Cognitive modeling, 5(3):1, 1988.
Sara Sabour, Nicholas Frosst, and Geoffrey E Hinton. Dynamic routing between capsules. In Advances in neural information processing systems, pp. 3856-3866, 2017.
Tara N Sainath, Brian Kingsbury. Vikas Sindhwani, Ebru Arisoy, and Bhuvana Ramabhadran. Low- rank matrix factorization for deep neural network training with high-dimensional output targets. In 2013 IEEE international conference acoustics, speech and signal processing, pp. 6655- 6659. IEEE, 2013.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1409. 1556, 2014.
Sean C Smithson, Guang Yang, Warren Gross, and Brett H Meyer. Neural networks designing neural networks: multi-objective hyper-parameteer optimization. In Proceedings of the 35th Inter- national Conference on Computer-Aided Design, pp. 104. ACM, 2016.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929-199 2014.

---

Under review as a conference paper at ICLR 2020
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethink- ing the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2818-28266 2016.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998- 1998-6008,22 2017.
Sergey Zagoruyko and Nikos Komodakis. arXiv:1605.07146, 2016.
Wide residual networks.
arXiv preprint
Guoqiang Zhong, Tao Li, Wenxue Liu, and Yang Chen. Structure learning of deep networks via dna computing algorithm. arXiv preprint arXiv: 10687, 2018.
Barret Zoph and Quoe V Le. Neural architecture search with reinforcement learning, arXiv preprint arXiv:1611.01578, 2016.
10

---

Under review as a conference paper at ICLR 2020
A ABLATION STUDY
In this section, we investigate the effect of three hyper-paramettes,, specifically number iterations, embedding size, and normalization function, the performance of FNNs. To assess the effect of each hyper-parametter we set all other hyper-parameters to a default value and evaluate model performance for various values of this hyper-parameter The default values are 3 for the number of iterations, 256 for the embedding size, and 12 for the normalization function. In all experiments, the FNN consists of 1024 neurons. The results are presented Figure 6.
61.-
60
55
Accuracy
Accuray
65-
35
60.0
no hidden
384 512
J. No Nortaaaaatt Safrus
Number of iterations
Embedding Size (d)
Normalization function
The effect of different parameters on the performance of FNNs. The default values for number of iterations, embedding size, number neurons, and normalization function 256, 1024, and 12 respectively. The default values are used if not specified.
Figure 6:
B INTEGRATION WITH OTHER CNNS
In this section present results of using FNNs instead of the fully connected layers the tail of other convolutional networks, specifically Residual Networks He et al. (2016). We compare performance of our models with baseline models on CIFARI00 dataset.
We train ResNet18 as our baseline and compare it with a variant of ResNet18 with an FNN instead of the tail fully connected layer. The FNN has 512 neurons with 64 embedding size. The number of iterations is set to so the network equivalent with multi-layer floating network with one hidden layer. For this task, we fixed the transformation function of the input neurons to identity. The results are presented in Table 3.
Model
Top-1 Accuracy(%)
ResNet18 + Dense
76.55
ResNet18 FNN
76.60
Table 3: The performance of ResNet18 and variant of it with FNN instead of the tail fully connected layer on CIFAR100.

---

Under review as a conference paper at ICLR 2020
SUPPLEMENTARY FIGURES
Floating Neurons
Floating Neurons
Floating Neurons
AA...
A
Output Neurons
wwy
444
Input Neurons
Input Neurons
Input Neurons
Paramaca
Params:8
Params
Paramsso
Paramssp
Paramsop
Paramsap
Figure 7: Unrolling Farfalle Neural Networks. FNNs have a recurrent block of floating neurons. Unrolling this structure results in layered structure of floating neurons, in which the parameters are shared between layers. In addition, input fed into each hidden layer.
Floating Neurons
Floating Neurons
Floating Neurons
Output Neurons
Input Neurons
Input Neurons'
Input Neurons
Params
Paramsxp
Figure 8: The recurrent structure allows the network to balance the number of neurons employed in different levels abstraction. This illustration employs unrolled structure to show how neurons combine information from different levels of abstraction. Red lines specify active connections (connections that significantly influence output) of each itcration.
12


Reviews:
[{"review_id": "rylhQa4CKB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a new architecture based on attention model to replace the fully-connected layers. In this architecture, each neuron is associated with an embedding vector, based on which the attention scores (between two consecutive layers) are calculated, and the computational flow through the layers are derived based on these attention scores. The experiments on MNIST and CIFAR demonstrate some degree of superiority over plan FC layers.\n\nPros:\n\n1. The idea is indeed interesting and AFAIK, there is no prior works trying to derive embedding for each neuron. The embedding based connection might encourage other follow-up works.\n\nCons:\n\n1. The writing sometimes seems unnecessarily complicated. For example, the “iteration” in section 3.3 is actually “layer”, right?  I furthermore see no motivation of listing the four items in this section, even the whole section 3.3: they are just re-stating the feedforward process of FNN. \n2. I donot believe FC is essential in modern computer vision (CV) tasks, so the better performance over a plain FFN on CV tasks are not that convincing (especially the two datasets are typically regarded as debugging dataset nowadays). I suggest the authors conduct more experiments on Transformer based tasks (e.g., machine translation), since in Transformer, the FFN is quite important. If the replace of FFN using the proposed FNN is successful for Transformer on some large scale task (e.g., WMT14 En-De Translation), this work will be much stronger in terms of empirical performance.\n\nQuestion:\n\n1. What is the embedding size d in the experiments? If d is large, the complexity comparison in the last paragraph of section 3.2 will not make too much sense.\n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "rJgEXQv6tS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces a new neural network architecture, in which all neurons (called \"floating neurons\") are essentially endowed with \"input\" and \"output\" embedding vectors, the product of which defines the weight of the connection between any two neurons. The authors discuss two network architectures employing floating neurons: (a) multi-layer floating neural networks and (b) farfalle neural network (FNN), in which there is one hidden layer, but additional recurrent connections are introduced between the hidden neurons.\n\nAs mentioned by the authors, the proposed architecture is similar to architectures employing low-rank weight matrix factorization. In my opinion, the main novelty lies in: (a) \"floating neuron\" interpretation, (b) additional weight matrix normalization, and (c) FNN architecture similar to that of a \"floating neuron\" RNN network with additional restrictions.\n\nI find the proposed idea to be promising and quite intriguing, but I think that the paper has some room for improvement and provided empirical evidence might be insufficient (including for understanding the importance of individual model components), which in turn makes the claims of potential practical attractiveness less justified. I will be happy to update the final score provided with more compelling arguments or empirical analysis of the proposed architecture.\n\nAddressing the following issues might greatly improve the quality of the paper:\n\n1. In Section 4.1, the authors compare FNN and DNN on MNIST and CIFAR10 datasets. My concern is that the authors pick a seemingly arbitrary DNN architecture (just a single one) and restrict comparison to it. One issue is that ~50% accuracy on CIFAR10 can be easily demonstrated by a variety of 5-layer DNN architectures including those much smaller, with just ~600k parameters (!) and possibly even lower. This makes the 90% parameter reduction claim not particularly meaningful. And why were models matched based on the total number of neurons, but not, say, the total number of parameters, or other measures? I believe that these questions require additional discussion and empirical evidence. Just as an example, if it was possible to sample (potentially randomly) different DNN architectures (with a reasonable parameter prior) and compare them with FNNs on a 2D accuracy-parameters plot (or using other important metrics), it would provide much more information to the reader.\n\n2. Another important point that I would like to make is that there is much more that can be done to explore the hyper-parameter space of FNN to isolate which particular factors play a decisive role in its superior accuracy. The authors present us with a specific choice of the normalization function, and values of k and d, but it would be very informative to study how results change when different choices are considered. FNNs differ from DNNs in at least three aspects: usage of low-rank factorization, weight normalization and recurrent structure. How important are these individual aspects? Are some of them redundant, or almost redundant, or do FNNs require all of these components to achieve their peak performance? In other words, I believe that a careful ablation study would greatly improve this publication.\n\n3. As a minor note, I think that the statement that FNNs \"are more general\" than floating neural networks is only partially correct. If I am not mistaken, FNN can also be \"unrolled\" and represented as a multi-layer floating neural network with additional parameter sharing. Also, the computational complexity of the constructed FNN (in Theorem 1) appears to be significantly higher than that of the floating neural network (especially for high l). This would imply that FNNs do not necessarily supersede multi-layer floating neural networks, at least when the computational complexity is of importance.\n\n4. There are a few minor misprints throughout the text. For example, in \"0<j<=j\" in the proof of Theorem 1, or in \"R output floating neurons for the final deduction from hidden neuron\" (output should be S). Also, I could not find information about the value of d used in the described experiments (which I estimated to be 256; is this correct?).\n\n5. In Section 4.3, the authors propose to use FNNs for the final layers of conventional CNN architectures. The issue is that the VGG16 network chosen for experiments was probably picked because it uses several large fully-connected (FC) layers in its tail whereas all more recent and efficient CNN architectures actually gravitate towards a smaller single FC layer. It is possible that FNNs could still be used in FC layers of these modern networks as well (especially with a large number of classes). But additional empirical results for these architectures would, in my opinion, be much more convincing.\n\nUpdated: The authors updated the text and addressed many of my questions. In my opinion, this improved the paper and made some of its claims much better justified. I change the rating to \"Weak Accept\".", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HklEgucTtH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Overall the paper is easy to read and I welcome that.\n\nI like the idea of using node-level embedding instead of pairwise weights to learn a low-rank weight representation. However, I am more skeptical about using this in a recurrent architecture and claiming that this is structure learning. The empirical results do not provide sufficient evidence that this performs structure learning.\n\n1. Theorem 1 seems rather straightforward because the FNN has much more representational power in the sense that its number of parameters is O(Nld) whereas the multi-layer version has O(Nd/l) parameters (in the uniform-width case). \n\n -- A more interesting question when it comes to structure learning is this: Suppose the best architecture for task A is shallow-and-wide while for task B is deep-and-narrow, each requiring roughly the same number of parameters. Can I use the proposed FNN with a similar number of parameters to learn the corresponding architecture for A and B respectively, without the need to figure out which is which? There is no evidence, analytical nor empirical, in this work, that suggests that this is the case.\n\n2. Section 4. It would be interesting to try baselines that have roughly the same number of parameters as the proposed FNN. Also, the choice of d (embedding size) and the number of iterations can be viewed as making architectural decisions. How were they chosen? Assuming that the same amount of computational resource is spent on searching through baseline architectures as well, could the results have been different from those in Table 1?\n\nThere are interesting ideas in this work but in its present form I cannot yet recommend acceptance.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a neural network architecture that represents each neuron with input and output embeddings. Experiments on CIFAR show that the proposed method outperforms baseline models with a fully connected layer.

I like the main idea of the paper. However, I agree with R1 and R2 that experiments presented in the paper are not enough to convince readers of the benefit of the proposed method. In particular, I would like to see a more comprehensive set of results across a suite of datasets. It would be even better, although not necessary, if the authors apply this method on top of different base architectures in multiple domains. At the very least, the authors should run an experiment to compare the proposed approach with a feed forward network on a simple/toy classification dataset. I understand that these experiments require a lot of computational resources. The authors do not need to reach SotA, but they do need to provide more empirical evidence that the method is useful in practice.

I also would like to see more discussions with regards to the computational cost of the proposed method. How much slower/faster is training/inference compared to a fully connected network?

The writing of the paper can also be improved. There are many a few typos throughout the paper, even in the abstract. 

I recommend rejecting this paper for ICLR, but would encourage the authors to polish it and run a few more suggested experiments to strengthen the paper.
Decision: Reject