Venue: ICLR 2018 conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: While deep learning usually involves estimating a large number of variable, this paper suggests to reduce its number by assuming that these variable lie in a low-dimensional subspace. In practice, this subspace is chosen randomly. Simulations show the promise of the proposed method.  In particular, figure 2 shows that the number of parameters could be greatly reduced while keeping 90% of the performance; and figure 4 shows that this method outperforms the standard method. The method is clearly written and the idea looks original. 

A con that I have is about the comparison in figure 4. While the proposed subspace method might have the same number of parameters as the direct method, I wonder if it is a fair comparison since the subspace method could still be more computational expensive, due to larger number of latent variables.


review_title: An proposal that reduces the degree of freedom in deep learning 

final_score: rating: 7: Good paper, accept
review_content: [ =============================== REVISION =========================================================]
My questions are answered, paper undergone some revision to clarify the presentation. I still maintain that it is a good paper and argue for acceptance - it provides a witty way of checking whether the network is overparameterized. Mnist  with shuffled labels is a great example that demonstrates the value of the approach, I would though have moved the results of it into the main paper, instead of supplemental materials
[ ======================== END OF REVISION =========================================================]

Authors introduce ransom subspace training (random subspace neural nets) where for a fixed architecture, only a subset of the parameters is trained, and the update for all the parameters is derived via random projection which is fixed for the duration of the training. Using this type of a network, authors introduce a notion of intrinsic dimension of optimization problems - it is minimal dimension of a subset, for which random subset neural net already reaches best (or comparable) performance.
Authors mention that this can be used for compressing networks - one would need to store the seed for the random matrix and the # of params equal to the intrinsic dimension of the net. 
They then demonstrate that the intrinsic dimension for the same problem stays the same when different architectures are chosen. Finally they mention neural nets with comparable number of params to intrinsic dimension but that don’t use random subspace trick don’t achieve comparable performance. This does not always hold for CNNs
Model with smaller intrinsic dimension is suggested to be better . They also suggest that intrinsic dimension might be a good approximation to Minimum Description Length metric

My main concern is computational efficiency. They state that if used for compressing, their method  is different from post-train compression, and the authors state that they train once end-to-end. It is indeed the case that once they found model that performs well, it is easy to compress, however they do train a number of models (up to a number of intrinsic dimension) until they get to this admissible model, which i envision would be computationally very expensive.

Questions:
- Are covets always better on MNIST: didn’t understand when authors said that intrinsic dimension of FC on shuffled data stayed the same (why) and then say that it becomes 190K - which one is correct?
- MNIST - state the input dimension size, not clear how you got to that number of parameters overall

review_title: Good paper

final_score: rating: 7: Good paper, accept
review_content: This paper proposes an empirical measure of the intrinsic dimensionality of a neural network problem. Taking the full dimensionality to be the total number of parameters of the network model, the authors assess intrinsic dimensionality by randomly projecting the network to a domain with fewer parameters (corresponding to a low-dimensional subspace within the original parameter), and then training the original network while restricting the projections of its parameters to lie within this subspace. Performance on this subspace is then evaluated relative to that over the full parameter space (the baseline). As an empirical standard, the authors focus on the subspace dimension that achieves a performance of 90% of the baseline. The authors then test out their measure of intrinsic dimensionality for fully-connected networks and convolutional networks, for several well-known datasets, and draw some interesting conclusions.

Pros:

* This paper continues the recent research trend towards a better characterization of neural networks and their performance. The authors show a good awareness of the recent literature, and to the best of my knowledge, their empirical characterization of the number of latent parameters is original. 

* The characterization of the number of latent variables is an important one, and their measure does perform in a way that one would intuitively expect. For example, as reported by the authors, when training a fully-connected network on the MNIST image dataset, shuffling pixels does not result in a change in their intrinsic dimensionality. For a convolutional network the observed 3-fold rise in intrinsic dimension is explained by the authors as due to the need to accomplish the classification task while respecting the structural constraints of the convnet.

* The proposed measures seem very practical - training on random projections uses far fewer parameters than in the original space (the baseline), and presumably the cost of determining the intrinsic dimensionality would presumably be only a fraction of the cost of this baseline training.

* Except for the occasional typo or grammatical error, the paper is well-written and organized. The issues are clearly identified, for the most part (but see below...).

Cons:

* In the main paper, the authors perform experiments and draw conclusions without taking into account the variability of performance across different random projections. Variance should be taken into account explicitly, in presenting experimental results and in the definition and analysis of the empirical intrinsic dimension itself. How often does a random projection lead to a high-quality solution, and how often does it not?

* The authors are careful to point out that training in restricted subspaces cannot lead to an optimal solution for the full parameter domain unless the subspace intersects the optimal solution region (which in general cannot be guaranteed). In their experiments (FC networks of varying depths and layer widths for the MNIST dataset), between projected and original solutions achieving 90% of baseline performance, they find an order of magnitude gap in the number of parameters needed. This calls into question the validity of random projection as an empirical means of categorizing the intrinsic dimensionality of a neural network.

* The authors then go on to propose that compression of the network be achieved by random projection to a subspace of dimensionality greater than or equal to the intrinsic dimension. However, I don't think that they make a convincing case for this approach. Again, variation is the difficulty: two different projective subspaces of the same dimensionality can lead to solutions that are extremely different in character or quality. How then can we be sure that our compressed network can be reconstituted into a solution of reasonable quality, even when its dimensionality greatly exceeds the intrinsic dimension?

* The authors argue for a relationship between intrinsic dimensionality and the minimum description length (MDL) of their solution, in that the intrinsic dimensionality should serve as an upper bound on the MDL. However they don't formally acknowledge that there is no standard relationship between the number of parameters and the actual number of bits needed to represent the model - it varies from setting to setting, with some parameters potentially requiring many more bits than others. And given this uncertain connection, and given the lack of consideration given to variation in the proposed measure of intrinsic dimensionality, it is hard to accept that "there is some rigor behind" their conclusion that LeNet is better than FC networks for classification on MNIST because its empirical intrinsic dimensionality score is lower.

* The experimental validation of their measure of intrinsic dimension could be made more extensive. In the main paper, they use three image datasets - MNIST, CIFAR-10 and ImageNet. In the supplemental information, they report intrinsic dimensions for reinforcement learning and other training tasks on four other sets.

Overall, I think that this characterization does have the potential to give insights into the performance of neural networks, provided that variation across projections is properly taken into account. For now, more work is needed.

====================================================================================================
Addendum:

The authors have revised their paper to take into account the effect of variation across projections, with results that greatly strengthen their results and provide a much better justification of their approach. I'm satisfied too with their explanations, and how they incorporated them into their revised version. I've adjusted my rating of the paper accordingly.

One point, however: the revisions seem somewhat rushed, due to the many typos and grammatical errors in the updated sections. I would like to encourage the authors to check their manuscript once more, very carefully, before finalizing the paper.
====================================================================================================
review_title: ICLR 2018 official review (Reviewer 2)

Meta-review: The authors make an empirical study of the "dimension" of a neural net optimization problem, where the "dimension" is defined by the minimal random linear parameter subspace dimension where a (near) solution to the problem is likely to be found.   I agree with reviewers that in light of the authors' revisions, the results are interesting enough to be presented at the conference.
Decision: Accept (Poster)