Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper proposes a new architecture based on attention model to replace the fully-connected layers. In this architecture, each neuron is associated with an embedding vector, based on which the attention scores (between two consecutive layers) are calculated, and the computational flow through the layers are derived based on these attention scores. The experiments on MNIST and CIFAR demonstrate some degree of superiority over plan FC layers.

Pros:

1. The idea is indeed interesting and AFAIK, there is no prior works trying to derive embedding for each neuron. The embedding based connection might encourage other follow-up works.

Cons:

1. The writing sometimes seems unnecessarily complicated. For example, the “iteration” in section 3.3 is actually “layer”, right?  I furthermore see no motivation of listing the four items in this section, even the whole section 3.3: they are just re-stating the feedforward process of FNN. 
2. I donot believe FC is essential in modern computer vision (CV) tasks, so the better performance over a plain FFN on CV tasks are not that convincing (especially the two datasets are typically regarded as debugging dataset nowadays). I suggest the authors conduct more experiments on Transformer based tasks (e.g., machine translation), since in Transformer, the FFN is quite important. If the replace of FFN using the proposed FNN is successful for Transformer on some large scale task (e.g., WMT14 En-De Translation), this work will be much stronger in terms of empirical performance.

Question:

1. What is the embedding size d in the experiments? If d is large, the complexity comparison in the last paragraph of section 3.2 will not make too much sense.


review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: Overall the paper is easy to read and I welcome that.

I like the idea of using node-level embedding instead of pairwise weights to learn a low-rank weight representation. However, I am more skeptical about using this in a recurrent architecture and claiming that this is structure learning. The empirical results do not provide sufficient evidence that this performs structure learning.

1. Theorem 1 seems rather straightforward because the FNN has much more representational power in the sense that its number of parameters is O(Nld) whereas the multi-layer version has O(Nd/l) parameters (in the uniform-width case). 

 -- A more interesting question when it comes to structure learning is this: Suppose the best architecture for task A is shallow-and-wide while for task B is deep-and-narrow, each requiring roughly the same number of parameters. Can I use the proposed FNN with a similar number of parameters to learn the corresponding architecture for A and B respectively, without the need to figure out which is which? There is no evidence, analytical nor empirical, in this work, that suggests that this is the case.

2. Section 4. It would be interesting to try baselines that have roughly the same number of parameters as the proposed FNN. Also, the choice of d (embedding size) and the number of iterations can be viewed as making architectural decisions. How were they chosen? Assuming that the same amount of computational resource is spent on searching through baseline architectures as well, could the results have been different from those in Table 1?

There are interesting ideas in this work but in its present form I cannot yet recommend acceptance.

review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: This paper introduces a new neural network architecture, in which all neurons (called "floating neurons") are essentially endowed with "input" and "output" embedding vectors, the product of which defines the weight of the connection between any two neurons. The authors discuss two network architectures employing floating neurons: (a) multi-layer floating neural networks and (b) farfalle neural network (FNN), in which there is one hidden layer, but additional recurrent connections are introduced between the hidden neurons.

As mentioned by the authors, the proposed architecture is similar to architectures employing low-rank weight matrix factorization. In my opinion, the main novelty lies in: (a) "floating neuron" interpretation, (b) additional weight matrix normalization, and (c) FNN architecture similar to that of a "floating neuron" RNN network with additional restrictions.

I find the proposed idea to be promising and quite intriguing, but I think that the paper has some room for improvement and provided empirical evidence might be insufficient (including for understanding the importance of individual model components), which in turn makes the claims of potential practical attractiveness less justified. I will be happy to update the final score provided with more compelling arguments or empirical analysis of the proposed architecture.

Addressing the following issues might greatly improve the quality of the paper:

1. In Section 4.1, the authors compare FNN and DNN on MNIST and CIFAR10 datasets. My concern is that the authors pick a seemingly arbitrary DNN architecture (just a single one) and restrict comparison to it. One issue is that ~50% accuracy on CIFAR10 can be easily demonstrated by a variety of 5-layer DNN architectures including those much smaller, with just ~600k parameters (!) and possibly even lower. This makes the 90% parameter reduction claim not particularly meaningful. And why were models matched based on the total number of neurons, but not, say, the total number of parameters, or other measures? I believe that these questions require additional discussion and empirical evidence. Just as an example, if it was possible to sample (potentially randomly) different DNN architectures (with a reasonable parameter prior) and compare them with FNNs on a 2D accuracy-parameters plot (or using other important metrics), it would provide much more information to the reader.

2. Another important point that I would like to make is that there is much more that can be done to explore the hyper-parameter space of FNN to isolate which particular factors play a decisive role in its superior accuracy. The authors present us with a specific choice of the normalization function, and values of k and d, but it would be very informative to study how results change when different choices are considered. FNNs differ from DNNs in at least three aspects: usage of low-rank factorization, weight normalization and recurrent structure. How important are these individual aspects? Are some of them redundant, or almost redundant, or do FNNs require all of these components to achieve their peak performance? In other words, I believe that a careful ablation study would greatly improve this publication.

3. As a minor note, I think that the statement that FNNs "are more general" than floating neural networks is only partially correct. If I am not mistaken, FNN can also be "unrolled" and represented as a multi-layer floating neural network with additional parameter sharing. Also, the computational complexity of the constructed FNN (in Theorem 1) appears to be significantly higher than that of the floating neural network (especially for high l). This would imply that FNNs do not necessarily supersede multi-layer floating neural networks, at least when the computational complexity is of importance.

4. There are a few minor misprints throughout the text. For example, in "0<j<=j" in the proof of Theorem 1, or in "R output floating neurons for the final deduction from hidden neuron" (output should be S). Also, I could not find information about the value of d used in the described experiments (which I estimated to be 256; is this correct?).

5. In Section 4.3, the authors propose to use FNNs for the final layers of conventional CNN architectures. The issue is that the VGG16 network chosen for experiments was probably picked because it uses several large fully-connected (FC) layers in its tail whereas all more recent and efficient CNN architectures actually gravitate towards a smaller single FC layer. It is possible that FNNs could still be used in FC layers of these modern networks as well (especially with a large number of classes). But additional empirical results for these architectures would, in my opinion, be much more convincing.

Updated: The authors updated the text and addressed many of my questions. In my opinion, this improved the paper and made some of its claims much better justified. I change the rating to "Weak Accept".
review_title: Official Blind Review #3

Meta-review: This paper proposes a neural network architecture that represents each neuron with input and output embeddings. Experiments on CIFAR show that the proposed method outperforms baseline models with a fully connected layer.

I like the main idea of the paper. However, I agree with R1 and R2 that experiments presented in the paper are not enough to convince readers of the benefit of the proposed method. In particular, I would like to see a more comprehensive set of results across a suite of datasets. It would be even better, although not necessary, if the authors apply this method on top of different base architectures in multiple domains. At the very least, the authors should run an experiment to compare the proposed approach with a feed forward network on a simple/toy classification dataset. I understand that these experiments require a lot of computational resources. The authors do not need to reach SotA, but they do need to provide more empirical evidence that the method is useful in practice.

I also would like to see more discussions with regards to the computational cost of the proposed method. How much slower/faster is training/inference compared to a fully connected network?

The writing of the paper can also be improved. There are many a few typos throughout the paper, even in the abstract. 

I recommend rejecting this paper for ICLR, but would encourage the authors to polish it and run a few more suggested experiments to strengthen the paper.
Decision: Reject