Venue: ICLR 2018 conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper proposes a structural equivalence between the function realised by a convolutional arithmetic circuit (ConvAC) and a quantum many-body wave function, which facilitates the use of quantum entanglement measures as quantifiers of a deep network’s expressive ability to model correlations. 

The work is definitely worthwhile digging deeper, bridging some gap and discussions between physics and deep learning. The ultimate goal for this work, if I understands correctly, is a provide a theoretical explanation to the design of deep neural architectures. The paper is well-written (above most submissions, top 10%) with clear clarity. However, removing all the fancy stuff and looking into the picture further, I have several major concerns.

+ Potential good research direction to connect physical sciences (via TN) to deep learning theories (via ConvAC).

- [Novelty is limited and proof is vague] The paper uses physical concepts to establish a "LAYER WIDTHS EFFECT ON THE EXPRESSIVENESS OF A DEEP NETWORK", the core theory (proposed method) part is Section 5 alone and the rest (Section 2,3,4) is for introductory purposes. Putting Theorem 1 in simple, deep learning based English, it says for a dataset with features of a size D, there exists a partition of length scale \epsilon < D, which is guaranteed to separate between different parts of a feature. Based on this, they give a rule of thumb to design the width (i.e., channel numbers) of layers in a deep neural network: (a) layer l = logD is more important than those of deeper layers; (b) among these deeper layers, deeper ones need to be wider, which is derived from the min-cut in the ConvAC TN case. How (a) is derived or implied from theorem 1? 

It seems to me that the paper goes with a rigorous manner till the proof of theorem 1, with all the concepts and denotations well demonstrated. Suddenly when it comes to connecting the practical design of deep networks, the conclusion becomes qualitative without much explanation via figures or visualisation of the learned features to prove the effectiveness of the proposed scheme.

- [Experiments are super weak] The paper has a good motivation and a beautiful story and yet, the experiments are poor to verify them. The reason as to why authors use ConvAC is that it more resembles the tensor operations introduced in the paper. There is a sentence, "Importantly, through the concept of generalized tensor decompositions, a ConvAC can be transformed to a standard convolutional network with ReLU activation and average/max pooling", to tell the relation between ConvAC and traditional convolutions. The theory is based on the analysis of ConvAC, and all of a sudden the experiments are conducted on the traditional convolution. This is not rigorous and not professional for a "technically-sound" paper. How the generalized concepts of tensor decompositions can be applied from ConvAC to vanilla convolutions?

The experiments seem to extend the channel width of *all* layers in a hand-crafted manner (10, 4r, 4r, xxx). Based on the derived rule of thumb, the most important layer in MNIST should be layer 3 or 4 (log 10). Some simple ablative analysis should be:
(i) baseline: fix layer 3, add more layers thereafter in the network;
(ii) fix layer 3, reduce the channel numbers after layer 3.
The (ii) case should be at least comparable to (i) if theorem 1 is correct.

Moreover, to verify conclusion (b) which I mentioned earlier, does the current setting (10, 4r, 4r, xx) consider "deeper ones need to be wider"? What is the value of r? MNIST is a over-used dataset and quite small. I see the performance in Figure 4 (the only experiment result in the paper) just exceeds 90%. A simple trained NN (not CNN) could reach well 96% or so.

More ablative study (convAC or vanilla conv, other datasets, comparison components in width design, etc.) are seriously needed. Otherwise, it is just not convincing to me.

If the authors target on the network design in a more general manner (not just in layer width, but the design of number of filters, layers, etc.), there are already some neat work in this community and you should definitely compare them: e.g., Neural Architecture Search with Reinforcement Learning, ICLR 2017. I know the paper starts with building the connection from physics to deep learning and it is natural to solve the width design issue alone. This is not a major concern.

-------------
We see lots of fancy conceptions, trying to bind interdisciplinary subjects to help analyse deep learning theories over the last few years, especially in ICLR 2017. This year same thing happens. I am not degrading the motivation/intuition of the work; instead, I think it is pretty novel to explain the design of neural nets, by way of quantum physics. But the experiments and the conclusion derived from the analysis make the paper not solid to me and I am quite skeptical about its actual effectiveness.

review_title: Beautiful story, good writing, weak proof between TN and ConvNet, poor experiment

final_score: rating: 7: Good paper, accept
review_content: The paper makes a striking connection between two apparently unrelated problems: the problem of designing neural networks to handle a certain type of correlation and the problem of designing a structure to represent wave-function with quantum entanglement. In the wave-function context, the Schmidt decomposition of the wave function is an inner product of tensors. Thus, the mathematical glue connecting the neural networks and quantum entanglement is shown to be tensor networks, which can represent higher order tensors through inner product of lower-order tensors. 

The main technical contribution in the paper is to map convolutional networks with product pooling function (called ConvACs) to a tensor network. Given this mapping, the authors exploit results in tensor networks (in particular the quantum max-flow min-cut theorem) to calculate the rank of the matricized tensor between a pair of vertex sets using the (appropriately defined) min-cut. 

The connection has potential to yield fruitful new results, however, the potential is not manifested (yet) in the paper. The main application in deep convolutional networks proposed by the paper is to model how much correlation between certain partition of input variables can be captured by a given convolutional network design. However, it is unclear how to use Theorem 1 to design neural networks that capture a certain correlation. 

A simple example is given in the experiment where the wider layers can be either early in the the neural network or at the later stages; demonstrating that one does better than the other in a certain regime. It seems that there is an obvious intuition that explains this phenomenon: wider base networks with large filters are better suited to the global task and narrow base networks that have more parameters later down have more local early filters suited to the local task. The experiments do not quite reveal the power of the proposed approach, and it is unclear how, if at all, the proposed approach can be applied to more complicated networks. 

In summary, this paper is of high theoretical interest and has potential for future applications.
review_title: Interesting theoretical connections

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper draws an interesting connection between deep neural networks and theories of quantum entanglement. They leveraged the tool for analyzing quantum entanglement to deep neural networks, and proposed a graph theoretical analysis for neural networks. They demonstrated how their theory can help designing neural network architectures on the MNIST dataset.

I think the theoretical findings are novel and may contribute to the important problem on understanding neural networks theoretically. I am not familiar with the theory for quantum entanglement though.
review_title: Interesting theoretical connections and practical implications

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: The authors try to bring in two seemingly different areas and try
to leverage the results in one for another.
First authors show that the equivalence of the function realized(in
tensor form, given in earlier work) by a ConvAC and
the function used to model n-body quantum system. After establishing
the equivalence of two, the authors argue that
quantum entanglement measures used to measure correlations in n-body
quantum systems can be used as an expressive measure
(how much correlation in input they can handle) of the function
realized by a ConvAC. Separation Rank analysis, which was done
earlier, becomes a special case. As the functional equivalence is
established, authors adopt Tensor Network framework,
to analyze the properties of the ConvAC. The main result being able
to quantify the expressiveness to some extend to the min
cut of the underlying Tensor Network graph corresponding to ConvAC.
This is further used to argue about guide-lining the
width of various parts of ConvAC, if some prior correlation
structure is known about the input. This is also validated
experimentally.

Although I do not see major results at this moment, this work can be
of great significance. The attempt to bring in two areas
have to be appreciated. This work opens up a footing to do graph
theoretical analysis of deep learning architectures and from
the perspective of Quantum entanglement, this could lead to open up new directions. 
The paper is lucidly written, comprehensively covering the
preliminaries. I thoroughly enjoyed reading it, and I think the
paper and the work would be of great contribution to the community.

(There are some typos  (preform --> perform ))
review_title: Good contribution, accept

Meta-review: This paper seemingly joins a cohort of ICLR submissions which attempt to port mature concepts from physics to machine learning, make a complex and non-trivial theoretical contribution, and fall short on the empirical front. The one aspect that sets this apart from its peers is that the reviewers agree that the theoretical contribution of this work is clear, interesting, and highly non-trivial. While the experiment sections (MNIST!) is indubitably weak, when treating this as a primarily theoretical contribution, the reviewers (in particular 6 and 3) are happy to suggest that the paper is worth reading. Taking this into account, and discounting somewhat the short (and, by their own admission, uncertain) assessment of reviewer 5, I am leaning  towards pushing for the acceptance of this paper. At very least, it would be a shame not to accept it to the workshop track, as this is by far the strongest paper of this type submitted to this conference.
Decision: Accept (Poster)