Venue: ICLR 2018 conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper presents a new parameterization of linear maps for use in neural networks, based on the Multiscale Entanglement Renormalization Ansatz (MERA). The basic idea is to use a hierarchical factorization of the linear map, that greatly reduces the number of parameters while still allowing for relatively complex interactions between variables to be modelled. A limited number of experiments on CIFAR10 suggests that the method may work a bit better than related factorizations.

The paper contains interesting new ideas and is generally well written. However, a few things are not fully explained, and the experiments are too limited to be convincing.


Exposition
On a first reading, it is initially unclear why we are talking about higher order tensors at all. Usually, fully connected layers are written as matrix-vector multiplications. It is only on the bottom of page 3 that it is explained that we will reshape the input to a rank-k (k=12) tensor before applying the MERA factored map. It would be helpful to state this sooner. It would also be nice to state that (in the absense of any factorization of the weight tensor) a linear contraction of such a high-rank tensor is no less general than a matrix-vector multiplication.

Most ML researchers will not know Haar measure. It would be more reader friendly to say something like "uniform distribution over orthogonal matrices (i.e. Haar measure)" or something like that. Explaining how to sample orthogonal matrices / tensors (e.g. by SVD) would be helpful as well.

The article does not explain what "disentanglers" are. It is very important to explain this, because it will not be generally known by the machine learning audience, and is the main thing that distinguishes this work form earlier tree-based factorizations.

On page 5 it is explained that the computational complexity of the proposed method is N^{log_2 D}. For D=2, this is better than a fully connected layer. Although this theoretical speedup may not currently have been realized, it perhaps could be achieved by a custom GPU kernel. It would be nice to highlight this potential benefit in the introduction.


Theoretical motivation
Although I find the theoretical motivation for the method somewhat compelling, some questions remain that the authors may want to address. For one thing, the paper talks about exploiting "hierarchical / multiscale structure", but this does not refer to the spatial multi-scale structure that is naturally present in images. Instead, the dimensions of a hidden activation vector are arbitrarily ordered, partitioned into pairs, and reshaped into a (2, 2, ..., 2) shape tensor. The pairing of dimensions determines the kinds of interactions the MERA layer can express. Although the earlier layers could learn to produce a representation that can be effectively analyzed by the MERA layer, one is left to wonder if the method could be made to exploit the spatial multi-scale structure that we know is actually present in image data.

Another point is that although from a classical statistics perspective it would seem that reducing the number of parameters should be generally beneficial, it has been observed many times that in deep learning, highly overparameterized models are easier to optimize and do not necessarily overfit. Thus at this point it is not clear whether starting with a highly constrained parameterization would allow us to obtain state of the art accuracy levels, or whether it is better to start with an overparameterized model and gradually constrain it or perform a post-training compression step.


Experiments
In the introduction it is claimed that the method of Liu et al. cannot capture correlations on different length scales because it lacks disentanglers. Although this may be theoretically correct, the paper does not experimentally verify that the proposed factorization with disentanglers outperforms a similar approach without disentanglers. In my opinion this is a critical omission, because the addition of disentanglers seems to be the main or perhaps only difference to previous work.

The experiments show that MERA can drastically reduce the number of parameters of fully connected layers with only a modest drop in accuracy, for a particular ConvNet trained on CIFAR10. Unfortunately this ConvNet is far from state of the art, so it is not clear if the method would also work for better architectures. Furthermore, training deep nets can be tricky, and so the poor performance makes it impossible to tell if the baseline is (unintentionally) crippled.

Comparing MERA-2 to TT-3 or MERA-3 to TT-5 (which have an approximately equal number of parameters), the difference in accuracy appears to be less than 1 percentage point. Since only a handful of specific MERA / TT architectures were compared on a single dataset, it is not at all clear that we can expect MERA to outperform TT in many situations. In fact, it is not even clear that the small difference observed is stable under random retraining.


Summary
An interesting paper with novel theoretical ideas, but insufficient experimental validation. Some expository issues need to be fixed.
review_title: Interesting idea but more experimental validation required

final_score: rating: 5: Marginally below acceptance threshold
review_content: In the paper the authors suggest to use MERA tensorization technique for compressing neural networks. MERA itseld in a known framework in QM but not in ML. Although the idea seems to be fruitful and interesting I find the paper quite unclear. The most important part is section 2 which presents the methodology used. However there no equations or formal descriptions of what is MERA and how it works. Only figures which are difficult to understand. It is almost impossible to reproduce the results based on such iformal description of tensorization method. The authors should be more careful and provide more details when describing the algorithm. There was enough room for making the algorithm more clear. This is my main point for critisism.

Another issue is related with practical usefulness. MERA allows to get better compression than TT keeping the same accuracy. But the authors do compress only one layer. In this case the total compression of DNN is almost tha same so why do we need yet another tensorization technique? I think the authors should try tenzorizing several layers and explore whether they can do any better than TT compression. Currently I would say the results are comparable but not definitely better.

UPDATE: The revised version seems to be a bit more clear. Now the reader unfamiliar with MERA (with some efforts) can understand how the methods works. Although my second concern remains. Up to now it looks just yet another tensorization method with only slight advantages over TT framework. Tensorization of conv.layers could improve the paper a lot. I increased the score to 5 for making the presentation of MERA more readable.
review_title: Interesting method but the paper needs to be rewritten

final_score: rating: 4: Ok but not good enough - rejection
review_content: The authors study compressing feed forward layers using low rank tensor decompositions. For instance a feed forward layer of 4096 x 4096 would first be reshaped into a rank-12 tensor with each index having dimension 2, and then a tensor decomposition would be applied to reduce the number of parameters. 

Previous work used tensor trains which decompose the tensor as a chain. Here the authors explore a tree like decomposition. The authors only describe their model using pictures and do not provide any rigorous description of how their decomposition works.

The results are mediocre. While the author's approach does seem to reduce the feed forward net parameters by 30% compared to the tensor train decomposition for similar accuracy, the total number of parameters for both MERA (authors' approach) and Tensor Train is similar since in this regime the CNN parameters dominate (and the authors' approach does not work to compress those).


review_title: Not very rigorously written, results are mediocre

Meta-review: This paper proposes a tree-structured tensor factorisation method for parameter reduction. The reviewers felt the paper was somewhat interesting, but agreed that more detail was needed in the method description, and that the experiments were on the whole uninformative. This seems like a promising research direction which needs more empirical work, but is not ready for publication as is.
Decision: Reject