Venue: ICLR 2018 conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: Authors propose a deep architecture learning algorithm in an unsupervised fashion. By finding conditional in-dependencies in input as a Bayesian network and using a stochastic inverse mechanism that preserves the conditional dependencies, they suggest an optimal structure of fully connected hidden layers (depth, number of groups and connectivity). Their algorithm can be applied recursively, resulting in multiple layers of connectivity. The width of each layer (determined by number of neurons in each group) is still tuned as a hyper-parameter.

Pros:
- Sound derivation for the method.
- Unsupervised and fast algorithm. 
Cons:
- Poor writing, close to a first draft. 
- Vague claims of the gain in replacing FC with these structures, lack of comparison with methods targeting that claim.
 - If the boldest claim is to have a smaller network, compare results with other compression methods.
 - If it is the gain in accuracy compare with other learn to learn methods and show that you achieve same or higher accuracy. The NAS algorithm achieves 3.65% test error. With a smaller network than the proposed learned structure (4.2M vs 6M) here they achieve slightly worse (5.5% vs 4.58%) but with a slightly larger (7.1M vs 6M) they achieve slightly better results (4.47% vs 4.58%). The winner will not be clear unless the experiments fixes one of variables or wins at both of them simultaneously.

Detailed comments:

- Results in Table 4 mainly shows that replacing fully connected layer with the learned structures leads to a much sparser connectivity (smaller number of parameters) without any loss of accuracy. Fewer number of parameters usually is appealing either because of better generalizability or less computation cost. In terms of generalizability, on most of the datasets the accuracy gain from the replacement is not statistically significant. Specially without reporting the standard deviation. Also the generalizability impact of this method on the state-of-the-art is not clear due to the fact that the vanilla networks used in the experiments are generally not the state-of-the-art networks. Therefore, it would be beneficial if the authors could show the speed impact of replacing FC layers with the learned structures. Are they faster to compute or slower?
- The purpose of section 5.1 is written as number of layers and number of parameters. But it compares with an FC network which has same number of neurons-per-layer. The rest of the paper is also about number of parameters. Therefore, the experiments in this section should be in terms of number of parameters as well. Also most of the numbers in table 1 are not significantly different. 

Suggestions for increasing the impact:

This method is easily adaptable for convolutional layers as well. Each convolutional kernel is a fully connected layer on top of a patch of an image. Therefore, the input data rather than being the whole image would be all patches of all images. This method could be used to learn a new structure to replace the KxK fully connected transformation in the convolutional layer. 

The fact that this is an unsupervised algorithm and it is suitable for replacing FC layers suggests experimentation on semi-supervised tasks or tasks that current state-of-the-art relies more on FC layers than image classification. However, the experiments in this paper are on fully-labeled image classification datasets which is possibly not a good candidate to verify the full potential of this algorithm.
review_title: Promising method, inconclusive results

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper tackles the important problem of structure learning by introducing an unsupervised algorithm, which encodes a hierarchy of independencies in the input distribution and allows introducing skip connections among neurons in different layers. The quality of the learnt structure is evaluated in the context of image classification, analyzing the impact of the number of parameters and layers on the performance.

The presentation of the paper could be improved. Moreover, the paper largely exceeds the recommended page limit (11 pages without references).

My main comments are related to the experimental section:

- Section 5 highlights that experiments were repeated 5 times; however, the standard deviation of the results is only reported for some cases. It would be beneficial to include the standard deviations of all experiments in the tables summarizing the obtained results.

- Are the differences among results presented in table 1 (MNIST) and table 2 (CIFAR10) statistically significant?

- It is not clear how the numbers of table 4 were computed (size replaced, size total, t-size, replaced-size). Would it be possible to provide the number of parameters of the vanilla model, the pre-trained feature extractor and the learned structure separately?

- In section 5.2., there is only one sentence mentioning comparisons to alternative approaches. It might be worth expanding this and including numerical comparisons.

- It seems that the main focus of the experiments is to highlight the parameter reduction achieved by the proposed algorithm. There is a vast literature on model compression, which might be worth reviewing, especially given that all the experiments are performed on standard image classification tasks.




review_title: Interesting unsupervised structure learning algorithm

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper proposes an unsupervised structure learning method for deep neural networks. It first constructs a fully visible DAG by learning from data, and decomposes variables into autonomous sets. Then latent variables are introduced and stochastic inverse is generated. Later a deep neural network structure is constructed based on the discriminative graph. Both the problem considered in the paper and the proposed method look interesting. The resulting structure seems nice.

However, the reviewer indeed finds a major technical flaw in the paper. The foundation of the proposed method is on preserving the conditional dependencies in graph G. And each step mentioned in the paper, as it claims, can preserve all the conditional dependencies. However, in section 2.2, it seems that the stochastic inverse cannot. In Fig. 3(b), A and B are no longer dependent conditioned on {C,D,E} due to the v-structure induced in node H_A and H_B. Also in Fig. 3(c), if the reviewer understands correctly, the bidirectional edge between H_A and H_B is equivalent to H_A <- h -> H_B, which also induces a v-structure, blocking the dependency between A and B. Therefore, the very foundation of the proposed method is shattered. And the reviewer requests an explicit explanation of this issue.

Besides that, the reviewer also finds unfair comparisons in the experiments.

1. In section 5.1, although the authors show that the learned structure achieves 99.04%-99.07% compared with 98.4%-98.75% for fully connected layers, the comparisons are made by keeping the number of parameters similar in both cases. The comparisons are reasonable but not very convincing. Observing that the learned structures would be much sparser than the fully connected ones, it means that the number of neurons in the fully connected network is significantly smaller. Did the authors compare with fully connected network with similar number of neurons? In such case, which one is better? (Having fewer parameters is a plus, but in terms of accuracy the number of neurons really matters for fair comparison. In practice, we definitely would not use that small number of neurons in fully connected layers.)

2. In section 5.2, it is interesting to observe that using features from conv10 is better than that from last dense layer. But it is not a fair comparison with vanilla network. In vanilla VGG-16-D, there are 3 more conv layers and 3 more fully connected layers. If you find that taking features from conv10 is good for the learned structure, then maybe it will also be good by taking features from conv10 and then apply 2-3 fully-connected layers directly (The proposed structure learning is not comparable to convolutional layers, and what it should really compare to is fully-connected layers.) In such case, which one is better? 
Secondly, VGG-16 is a large network designed for ImageNet data. For small dataset such as CIFAR10 and CIFAR100, it is really overkilled. That's maybe the reason why taking the output of shallow layers could achieve pretty good results.

3. In Fig. 6, again, comparing the learned structure with fully-connected network by keeping parameters to be similar and resulting in large difference of the number of neurons is unfair from my point of view.

Furthermore, all the comparisons are made with respect to fully-connected network or vanilla CNNs. No other structure learning methods are compared with. Reasonable baseline methods should be included.

In conclusion, due to the above issues both in method and experiments, the reviewer thinks that this paper is not ready for publication.

review_title: There is a major technical flaw in this paper. And some experiment settings are not convincing.

Meta-review: The updated draft has helped to address some of the issues that the reviewers had, however the reviewers believe there are still outstanding issues. With regard to the technical flaw, one reviewer has pointed out that the update changes the story of the paper by breaking the connection between the generative and discriminative model in terms of preserving or ignoring conditional dependencies.

In terms of the experiments, the paper has been improved by the reporting of standard deviation, and comparison to other works. However it is recommended that the authors compare to NAS by fixing the number of parameters and reporting the results to facilitate an apples-to-apples comparison. Another reviewer also recommends comparing to other architectures for a fixed number of neurons.
Decision: Reject