Venue: ICLR 2018 conference

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: 

The paper is well written and easy to follow. The main contribution is to propose a variant of the RevNet architecture that has a built in pseudo-inverse, allowing for easy inversion. The results are very surprising in my view: the proposed architecture is nearly invertible and is able to achieve similar performance as highly competitive variants: ResNets and RevNets.

The main contribution is to use linear and invertible operators (pixel shuffle) for performing downsampling, instead of non-invertible variants like spatial pooling. While the change is small, conceptually is very important.

Could you please comment on the training time? Although this is not the point of the paper, it would be very informative to include learning curves. Maybe discarding information is not essential for learning (which is surprising), but the cost of not doing so is payed in learning time. Stating this trade-off would be informative. If I understand correctly, the training runs for about 150 epochs, which is maybe double of what the baseline ResNet would require?

The authors evaluate in Section 4.2 the show samples obtained by the pseudo inverse and study the properties of the representations learned by the model. I find this section really interesting. Further analysis will make the paper stronger.

Are the images used for the interpolation train or test images?

I assume that the network evaluated with the Basel Faces dataset, is the same one trained on Imagenet, is that the case?

In particular, it would be interesting (not required) to evaluate if the learned representation is able to linearize a variety of geometric image transformations in a controlled setting as done in:

Hénaff, O,, and Simoncelli, E. "Geodesics of learned representations." arXiv preprint arXiv:1511.06394 (2015).

Could you please clarify, what do you mean with fine tuning the last layer with dropout?

The authors should cite the work on learning invertible functions with tractable Jacobian determinant (and exact and tractable log-likelihood evaluation) for generative modeling. Clearly the goals are different, but nevertheless very related. Specifically,

Dinh, L. et al  "NICE: Non-linear independent components estimation." arXiv preprint arXiv:1410.8516 (2014).


Dinh, L. et al "Density estimation using Real NVP." arXiv preprint arXiv:1605.08803 (2016).

The authors mention that the forward pass of the network does not seem to suffer from significant instabilities. It would be very good to empirically evaluate this claim.

review_title: Official review

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: In this paper, the authors propose deep architecture that preserves mutual information between the input and the hidden representation and show that the loss of information can only occur at the final layer. They illustrate empirically that the loss of information can be avoided on large-scale classification such as ImageNet and propose to build an invertible deep network that is capable of retaining the information of the input signal through all the layers of the network until the last layer where the input could be reconstructed.

The authors demonstrate that progressive contraction and separation of the information can be obtained while at the same time allowing an exact reconstruction of the signal.

As it requires a special care to design an invertible architecture, the authors architecture is based on the recent reversible residual network (RevNet) introduced in (Gomez et al., 2017) and an invertible down-sampling operator introduced in (Shi et al., 2016). The inverse (classification) path of the network uses the same convolutions as the forward (reconstructing) one. It also uses subtraction operations instead of additions in the output computation in order to reconstruct intermediate and input layers.

To show the effectiveness of their approach on large-scale classification problem, the authors report top-1 error rates on the validation set of ILSVRC-2012. The obtained result is competitive with the original Resnet and the RevNet models. However, the proposed approach is expensive in terms of parameter budget as it requires almost 6.5 times more parameters than the RevNet and the Resnet architectures. Still, the classification and the reconstructing results are quite impressive as the work is the first empirical evidence that learning invertible representation that preserves information about the input is possible on large-scale classification tasks. Worth noting that recently, (Shwartz-Ziv and Tishby) demonstrated, not on large-scale datasets but on small ones, that an optimal representation for a classification task must reduce as much uninformative variability as possible while maximizing the mutual information between the desired output and its representation in order discriminate as much as possible between classes. This is called “information bottleneck principle”. The submitted paper shows that this principle is not a necessary condition large-scale classification.

The proposed approach is potentially of great benefit. It is also simple and easy to understand. The paper is well written and the authors position their work with respect to what has been done before. The spectral analysis of the differential operator in section 4.1 provide another motivation for the “hard-constrained” invertible architecture. Section 4.2 illustrates the ability of the network to reconstruct input signals. The visualization obtained suggests that network performs linear separation between complex learned factors. Section 5 shows that even when using either an SVM or a Nearest Neighbor classifier on n extracted features from a layer in the network, both classifiers progressively improve with deeper layers. When the d first principal components are used to summarize the n extracted features, the SVM and NN classifier performs better when d is bigger. This shows that the deeper the network gets, the more linearly separable and contracted the learned representations are.

In the conclusion, the authors state the following: “The absence of loss of information is surprising, given the wide believe, that discarding information is essential for learning representations that generalize well to unseen data”. Indeed, the authors have succeed in showing that this is not necessarily the case. However, the loss of information might be necessary to generalize well on unseen data and at the same time minimize the parameter budget for a given classification task.

review_title: The paper proposes an invertible architecture which allows to verify that the loss of input information is not needed in deep architectures in order to generalize on large scale supervised tasks.

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: ICLR I-Revnet


This paper build on top of ReVNets (Gomez et al., 2017)  and introduce a variant that is fully 
invertible. The model performs comparable to its variants without any loss of information.
They analyze the model and its learned representations from multiple perspectives in detail. 
 
It is indeed very interesting an thought provoking to see that contrary to popular belief in the community no information loss is necessary to learn good generalizable features. What is missing, is more motivation for why such a property is desirable. As the authors mentions the model size has almost doubled compared to comparable ResNet. And the study of the property of the learned futures might probably limited to this i-RevNet only. It would be good to see more motivation, beside the valuable insight of knowing it’s possible.

Generally the paper is well written and readable, but few minor comments:
1-Better formatting such as putting results in model sizes, etc in tables will make them easier to find.
2-Writing down more in detail 3.1, ideally in algorithm or equation than all in text as makes it hard to read in current format.
review_title: Gives an interesting insight that loss of information is not necessary for good generalizable features. 

Meta-review: This paper constructs a variant of deep CNNs which is provably invertible, by replacing spatial pooling with multiple shifted spatial downsampling, and capitalizing on residual layers to define a simple, invertible representation. The authors show that the resulting representation is equally effective at large-scale object classification, opening up a number of interesting questions.

Reviewers agreed this is an strong contribution, despite some comments about the significance of the result; ie, why is invertibility a "surprising" property for learnability, in the sense that F(x) = {x,  phi(x)}, where phi is a standard CNN satisfies both properties: invertible and linear measurements of F producing good classification. All in all, this will be a great contribution to the conference. 
Decision: Accept (Poster)