Venue: ICLR 2017 conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper provides a new perspective to understanding the ResNet and Highway net. The new perspective assumes that the blocks inside the networks with residual or skip-connection are groups of successive layers with the same hidden size, which performs to iteratively refine their estimates of the same feature instead of generate new representations. Under this perspective, some contradictories with the traditional representation view induced by ResNet and Highway network and other paper can be well explained.

The pros of the paper are:
1. A novel perspective to understand the recent progress of neural network is proposed.
2. The paper provides a quantitatively experimentals to compare ResNet and Highway net, and shows contradict results with several claims from previous work. The authors also give discussions and explanations about the contradictories, which provides a good insight of the disadvantages and advantages between these two kind of networks.

The main cons of the paper is that the experiments are not sufficient. For example, since the main contribution of the paper is to propose the “unrolled iterative estimation" and the stage 4 of Figure 3 seems not follow the assumption of "unrolled iterative estimation" and the authors says: "We note that stage four (with three blocks) appears to be underestimating the representation values, indicating a probable weak link in the architecture.". Thus, it would be much better to do experiments to show that under some condition, the performance of stage 4 can follow the assumption. 

Moreover, the paper should provide more experiments to show the evidence of "unrolled iterative estimation", not comparing ResNet with Highway Net. The lack of experiments on this point is the main concern from myself.


review_title: A New Perspective of ResNet and Highway Network

final_score: rating: 7: Good paper, accept
review_content: Thank you for an interesting angle on highway and residual networks. This paper shows a new angle to how and what kind of representations are learnt at each layer in the aforementioned models. Due to residual information being provided at a periodic number of steps, each of the layers preserve feature identity which prevents lesioning unlike convolutional neural nets.                                 
                                                                                                                                                                                                          
Pros                                                                                                                                                                                                      
- the iterative unrolling view was extremely simple and intuitive, which was supported by theoretical results and reasonable assumptions.                                                                 
- Figure 3 gave a clear visualization for the iterative unrolling view                                                                                                                                    
                                                                                                                                                                                                          
Cons                                                                                                                                                                                                      
- Even though, the perspective is interesting few empirical results were shown to support the argument. The major experiments are image classification and language models trained on mutations of character-aware neural language models.                                                                                                                                                                         
- Figure 4 and 5 could be combined and enlarged to show the effects of batch normalization.   
review_title: An interesting angle on resNets and Highway nets

final_score: rating: null
review_content: 1. What is the definition of A_i in the paper?

2. What is the x-axis of Figure 3?
review_title: several questions

final_score: rating: null
review_content: An interesting view on skip-connection based networks.

I had one comment though, it would be great to have significance test on the experimental results to verify claims made in Section 5.1
review_title: Batch Normalization

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: The paper describes an alternative view on hierarchical feature representations in deep neural networks. The viewpoint of refining representations is well motivated and is in agreement with the success of recent model structures like ResNets.

Pros:

- Good motivation for the effectiveness of ResNets and Highway networks
- Convincing analysis and evaluation

Cons:

- The effect of this finding of the interpretation of batch-normalization is only captured briefly but seems to be significant
- Explanation of findings in (Zeiler & Fergus (2014)) using UIE viewpoint missing

Remarks:

- Missing word in line 223: "that it *is* valid"
review_title: An interesting perspective on representations in DNNs

Meta-review: The paper provides interesting new interpretations of highway and residual networks, which should be of great interest to the community.
Decision: Accept (Poster)