i-RevNet: Deep Invertible Networks

Abstract: It is widely believed that the success of deep convolutional networks is based on progressively discarding uninformative variability about the input with respect to the problem at hand. This is supported empirically by the difficulty of recovering images from their hidden representations, in most commonly used network architectures. In this paper we show via a one-to-one mapping that this loss of information is not a necessary condition to learn representations that generalize well on complicated problems, such as ImageNet. Via a cascade of homeomorphic layers, we build the i-RevNet, a network that can be fully inverted up to the final projection onto the classes, i.e. no information is discarded. Building an invertible architecture is difficult, for one, because the local inversion is ill-conditioned, we overcome this by providing an explicit inverse. 
      An analysis of i-RevNet’s learned representations suggests an alternative explanation for the success of deep networks by a progressive contraction and linear separation with depth. To shed light on the nature of the model learned by the i-RevNet we reconstruct linear interpolations between natural image representations.

Reviews and author responses:


1) As mentioned in section 3.1 and detailed in the 4th paragraph of section 3.2, $\tilde{S}$ splits the input into two tensors. In our case, we stick to the choice of Revnets and split the number of input channels in half. You will be able to check how this is done in detail in the code we will release alongside the de-anonymized version of the paper.

2) Thanks for this question. What is displayed in fig. 5, are not noisy images, but rather precise reconstructions of an interpolation in feature space. Images obtained by this interpolation have no particular reason to look like real images, as their representation suffers from the curse of dimensionality. However, as indicated in the paper, it indeed opens the question about the structure of the feature space.

I am interested in your paper. But I have two questions:

1) As mentioned, features at each layer are decomposed into two parts. So how to decompose the input images? (I don't find the corresponding descriptions in your paper.) 

2) The reconstructed sequences x_t in Fig.5 contain lots of noise and are with low visual qualities. Can you explain the reasons?

Thank you for the detail response and adding new sets of experiments for the questions that was raised and I updated the score to reflect the changes.

ICLR I-Revnet


This paper build on top of ReVNets (Gomez et al., 2017)  and introduce a variant that is fully 
invertible. The model performs comparable to its variants without any loss of information.
They analyze the model and its learned representations from multiple perspectives in detail. 
 
It is indeed very interesting an thought provoking to see that contrary to popular belief in the community no information loss is necessary to learn good generalizable features. What is missing, is more motivation for why such a property is desirable. As the authors mentions the model size has almost doubled compared to comparable ResNet. And the study of the property of the learned futures might probably limited to this i-RevNet only. It would be good to see more motivation, beside the valuable insight of knowing it’s possible.

Generally the paper is well written and readable, but few minor comments:
1-Better formatting such as putting results in model sizes, etc in tables will make them easier to find.
2-Writing down more in detail 3.1, ideally in algorithm or equation than all in text as makes it hard to read in current format.

In this paper, the authors propose deep architecture that preserves mutual information between the input and the hidden representation and show that the loss of information can only occur at the final layer. They illustrate empirically that the loss of information can be avoided on large-scale classification such as ImageNet and propose to build an invertible deep network that is capable of retaining the information of the input signal through all the layers of the network until the last layer where the input could be reconstructed.

The authors demonstrate that progressive contraction and separation of the information can be obtained while at the same time allowing an exact reconstruction of the signal.

As it requires a special care to design an invertible architecture, the authors architecture is based on the recent reversible residual network (RevNet) introduced in (Gomez et al., 2017) and an invertible down-sampling operator introduced in (Shi et al., 2016). The inverse (classification) path of the network uses the same convolutions as the forward (reconstructing) one. It also uses subtraction operations instead of additions in the output computation in order to reconstruct intermediate and input layers.

To show the effectiveness of their approach on large-scale classification problem, the authors report top-1 error rates on the validation set of ILSVRC-2012. The obtained result is competitive with the original Resnet and the RevNet models. However, the proposed approach is expensive in terms of parameter budget as it requires almost 6.5 times more parameters than the RevNet and the Resnet architectures. Still, the classification and the reconstructing results are quite impressive as the work is the first empirical evidence that learning invertible representation that preserves information about the input is possible on large-scale classification tasks. Worth noting that recently, (Shwartz-Ziv and Tishby) demonstrated, not on large-scale datasets but on small ones, that an optimal representation for a classification task must reduce as much uninformative variability as possible while maximizing the mutual information between the desired output and its representation in order discriminate as much as possible between classes. This is called “information bottleneck principle”. The submitted paper shows that this principle is not a necessary condition large-scale classification.

The proposed approach is potentially of great benefit. It is also simple and easy to understand. The paper is well written and the authors position their work with respect to what has been done before. The spectral analysis of the differential operator in section 4.1 provide another motivation for the “hard-constrained” invertible architecture. Section 4.2 illustrates the ability of the network to reconstruct input signals. The visualization obtained suggests that network performs linear separation between complex learned factors. Section 5 shows that even when using either an SVM or a Nearest Neighbor classifier on n extracted features from a layer in the network, both classifiers progressively improve with deeper layers. When the d first principal components are used to summarize the n extracted features, the SVM and NN classifier performs better when d is bigger. This shows that the deeper the network gets, the more linearly separable and contracted the learned representations are.

In the conclusion, the authors state the following: “The absence of loss of information is surprising, given the wide believe, that discarding information is essential for learning representations that generalize well to unseen data”. Indeed, the authors have succeed in showing that this is not necessarily the case. However, the loss of information might be necessary to generalize well on unseen data and at the same time minimize the parameter budget for a given classification task.




The paper is well written and easy to follow. The main contribution is to propose a variant of the RevNet architecture that has a built in pseudo-inverse, allowing for easy inversion. The results are very surprising in my view: the proposed architecture is nearly invertible and is able to achieve similar performance as highly competitive variants: ResNets and RevNets.

The main contribution is to use linear and invertible operators (pixel shuffle) for performing downsampling, instead of non-invertible variants like spatial pooling. While the change is small, conceptually is very important.

Could you please comment on the training time? Although this is not the point of the paper, it would be very informative to include learning curves. Maybe discarding information is not essential for learning (which is surprising), but the cost of not doing so is payed in learning time. Stating this trade-off would be informative. If I understand correctly, the training runs for about 150 epochs, which is maybe double of what the baseline ResNet would require?

The authors evaluate in Section 4.2 the show samples obtained by the pseudo inverse and study the properties of the representations learned by the model. I find this section really interesting. Further analysis will make the paper stronger.

Are the images used for the interpolation train or test images?

I assume that the network evaluated with the Basel Faces dataset, is the same one trained on Imagenet, is that the case?

In particular, it would be interesting (not required) to evaluate if the learned representation is able to linearize a variety of geometric image transformations in a controlled setting as done in:

Hénaff, O,, and Simoncelli, E. "Geodesics of learned representations." arXiv preprint arXiv:1511.06394 (2015).

Could you please clarify, what do you mean with fine tuning the last layer with dropout?

The authors should cite the work on learning invertible functions with tractable Jacobian determinant (and exact and tractable log-likelihood evaluation) for generative modeling. Clearly the goals are different, but nevertheless very related. Specifically,

Dinh, L. et al  "NICE: Non-linear independent components estimation." arXiv preprint arXiv:1410.8516 (2014).


Dinh, L. et al "Density estimation using Real NVP." arXiv preprint arXiv:1605.08803 (2016).

The authors mention that the forward pass of the network does not seem to suffer from significant instabilities. It would be very good to empirically evaluate this claim.



At a given block, the tensor $\tilde x_j$ can potentially have a different size from $x_j$ without losing in generality. Consequently, as the bottleneck layer  $\mathcal{F}_{j+1}$ is applied to $\tilde x_j$, its output must match the shape of $x_j$. This is done, for instance, by applying an intermediary stride via $\mathcal{F}_{j+1}$ (it is also possible to upsample its output), or increasing/reducing the channel size, accordingly to the size of $x_j$. 

In our implementation and for consistency, we followed the approach of the RevNet which down-samples two interlaced blocks at a given depth $j$ (e.g. ${x_j,\tilde x_j}). It means that two successive blocks (and thus also $S_j$ & $S_{j+1}$) of an $i$-RevNet work in concert, in order to have downsampled the signals ${x_{j+2},\tilde x_{j+2}}$ by a factor $2^2$ w.r.t. the depth $j$.

We will add clarification of this to the camera ready. As mentioned in the manuscript, we will also release our code so you can check how this is implemented in detail.

We thank you very much again for your comment!

I have a question regarding the description of S_j in section 3.2. It's mentioned that "at each layer of depth j = 3, 21, 69, 285, the spatial resolution is reduced by 2^2 via S_j". Consider the case when a tensor with N channels of spatial resolution M by M is passing through layer j. If S_j is a downsampling operation (i.e. j=3, 21, 69 or 285), then x_tilda_{j+1} will have 4N channels of size M/2 by M/2 while x_{j+1} will still have N channels of size M by M. This means that the addition operation in layer j+1 will be an addition between two tensors of unequal dimension. How is this dealt with for the bijective i-RevNet? Thank you!

We thank the reviewer very much for raising many interesting and important points. Furthermore, we thank the reviewer for acknowledging that the presented results are surprising and our technical contributions conceptually important. We are also pleased that the reviewer finds the analysis of the learned representation very interesting.  

To open up another dimension of the analysis, we have added a model which replaces the initial injective operator with a bijective operator as used in later layers. This model has almost the same number of parameters as the baseline and trains about a day faster, albeit performs worse by 1.5%. This is to show, that model size can be reduced substantially while the invertibility property improves.

 ==> Maybe discarding information is not essential for learning (which is surprising), but the cost of not doing so is paid in learning time.

Thank you for raising this interesting point.
We have added plots of the loss curves to the paper that show very similar training behaviour for an i-RevNet compared to a non-invertible ResNet baseline. Training hyperparameters (e.g. learning rate schedule, training iterations, regularization) are identical for all models we have analyzed in the paper.
Thus, there does not seem to be a cost to pay for not discarding information in terms of convergence behaviour.

==> Are the images used for the interpolation train or test images?    

The images used for interpolation are partially from datasets not seen during training (describable textures, Basel faces) and from the Imagenet training set. All interpolations have been obtained from the ILSVRC-2012 trained model.

==> Could you please clarify, what do you mean with fine tuning the last layer with dropout?    

We thank the reviewer for raising this question. For the sake of brevity, we have removed this fine-tuning in the current revision entirely. This change only affected Figure 4, we have updated the figure with interpolations from the newly added bijective model.

==> The authors mention that the forward pass of the network does not seem to suffer from significant instabilities. It would be very good to empirically evaluate this claim.  

Thank you for this important remark, we have empirically evaluated our claims by measuring the normalized l2 error between original and reconstruction on the whole validation set of ILSVRC-2012 and on randomly drawn uniform noise \in [-1,1], with the same number of draws as the size of the validation set. We report expectation of the error over all samples. The results show no significant instabilities and the error is visually imperceivable:

i-RevNet bijective:
ILSVRC-2012 validation set reconstruction error: 5.17e-06
50k uniform noise draws reconstruction error: 2.63e-06

i-RevNet injective:
ILSVRC-2012 validation set reconstruction error: 8.26e-7
50k uniform noise draws reconstruction error: 5.52e-07
  
We thank the reviewer for the additional references, we have added NICE and Real-NVP to the related work section and discussed their relationship to our work.

To add results on more controlled geometric transformations, we have added interpolations between small geometrical perturbations to the reconstruction experiment. 
 
We thank the reviewer once again for the very interesting and important remarks. We believe they have substantially improved the manuscript and helped to clarify many important points.


We thank the reviewer very much for the valuable comments and for acknowledging that our main claims are very interesting and thought-provoking. In the following, we will elaborate on the increased model size and usefulness of such an architecture in detail.  

To add another dimension to the model analysis and to shed light on the necessary model size, we have added a model which replaces the initial injective operator with a bijective operator as used in later layers. This model has almost the same number of parameters as the baselines and trains about a day faster, albeit performs worse by 1.5%. This is to show, that model size can be reduced substantially while the invertibility property improves.

==> The authors mention model size has almost doubled  

Thanks to this important remark, we have added another model that shows it is possible to avoid an excessive increase of model size in i-RevNets. The newly added model has 29M parameters as opposed to 28M in the RevNet baseline while having a top-1 accuracy of 73.3%, which is ~1.5% worse than the RevNet baseline.  

We thank the reviewer once again for raising this point and believe that the newly introduced model makes the paper even stronger, as it shows that the invertibility property can even be improved by decreasing model size.

==> Does the analysis apply to other models as well?  

We thank the reviewer for this question, section 5.1 shows that progressive properties that are known to hold for lossy AlexNet type models on limited datasets, are in fact also possible to obtain in an architecture that is not able to discard information about the input on a large-scale task like Imagenet.

To further strengthen the results, we have extended our analysis of the separation contraction to a ResNet baseline. Our results show, that the behaviour of the non-invertible ResNet is the same as the one observed in i-RevNets, substantiating the generality of our findings.

==> Why is such a model desirable?  

The core question we answer is if the success of deep convolutional networks is based on progressively discarding uninformative variability, which is a wide standing believe in the CV and ML community. We show this does not have to be the case, which has been acknowledged as "important", "interesting" and "thought-provoking" by all reviewers. Thus, the invertibility property is desirable for understanding the success of deep learning better and shed light on some of the necessities for it to work well.
From a practical point of view, invertible models are useful for feature visualization [1,2,3] and possibly useful to overcome difficulties in upsampling/decoding pixel-wise tasks that are still quite challenging [4]. Further, lossless models might be a good candidate for transfer learning. 

In summary, we do believe that besides the theoretical interest of our work, which has been acknowledged by all reviewers, there is also a potential impact in deep learning applications for invertible models.  

We thank the reviewer once again for the important questions and remarks, we believe that the added discussion and results of the new bijective i-RevNet and ResNet baseline substantially improve the paper. We have also incorporated suggested formatting improvements into the manuscript.

[1] Mahendran, Aravindh, and Andrea Vedaldi. "Understanding deep image representations by inverting them." Proceedings of the IEEE conference on computer vision and pattern recognition. 2015.
[2] Dosovitskiy, Alexey, and Thomas Brox. "Inverting visual representations with convolutional networks." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2016.
APA	
[3] Selvaraju, Ramprasaath R., et al. "Grad-cam: Why did you say that? visual explanations from deep networks via gradient-based localization." arXiv preprint arXiv:1610.02391 (2016).
[4] Wojna, Zbigniew, et al. "The Devil is in the Decoder." arXiv preprint arXiv:1707.05847 (2017).  


We thank the reviewer very much for this encouraging review and the comments on our paper.
We would also like to thank the reviewer for acknowledging the presented results being impressive and potentially of great benefit.

Inspired by the reviewer's remark on model size and the quest for an optimal parameter budget, we have added another model to the paper that has a similar number of parameters as the RevNet and ResNet baselines. This way we show that an increased number of parameters is not necessary to obtain the invertible architecture. This newly added i-RevNet replaces the initial injective mapping with a bijective mapping. In consequence, the new model is slightly different in architecture from the baselines, as it keeps the input dimensionality constant. We have replaced the analysis of the injective i-RevNet by an analysis of the bijective i-RevNet throughout the whole paper.

Furthermore, to show that the observed separation and contraction occur independently of invertibility, we have added a non-invertible ResNet baseline to the model analysis in section 5.1. We have also added training plots of ResNets compared to i-RevNets. The results show a progressive separation and contraction in invertible and non-invertible models and very similar training behaviour. 

Our main conclusions remain the same, while the new results substantiate their generality.

We thank the reviewer once again for the insightful comments thanks to which we were able to further improve the paper.

Dear Reviewers,

We sincerely thank you for your work and effort in reviewing the manuscript. Your comments and remarks have been very helpful to improve the quality of the paper. 

We have made two major additions to the manuscript that substantiate the generality of our claims.

1) A bijective i-RevNet with 6 times fewer parameters, that shows a reduction of parameters can even improve the invertibility property.
2) An Imagenet-trained ResNet to show our findings apply to non-invertible state-of-the-art models as well.

Please find detailed answers below, and thank you once again!

Meta-review: This paper constructs a variant of deep CNNs which is provably invertible, by replacing spatial pooling with multiple shifted spatial downsampling, and capitalizing on residual layers to define a simple, invertible representation. The authors show that the resulting representation is equally effective at large-scale object classification, opening up a number of interesting questions.

Reviewers agreed this is an strong contribution, despite some comments about the significance of the result; ie, why is invertibility a "surprising" property for learnability, in the sense that F(x) = {x,  phi(x)}, where phi is a standard CNN satisfies both properties: invertible and linear measurements of F producing good classification. All in all, this will be a great contribution to the conference. 

Decision: accepted-poster-papers