Title: Towards Understanding the Invertibility of Convolutional Neural Networks
Authors: Anna C. Gilbert
Yi Zhang
Kibok Lee
Yuting Zhang
Honglak Lee
Venue: ICLR 2017
Keywords: Deep learning
Theory

Abstract: Several recent works have empirically observed that Convolutional Neural Nets (CNNs) are (approximately) invertible. To understand this approximate invertibility phenomenon and how to leverage it more effectively, we focus on a theoretical explanation and develop a mathematical model of sparse signal recovery that is consistent with CNNs with random weights. We give an exact connection to a particular model of model-based compressive sensing (and its recovery algorithms) and random-weight CNNs. We show empirically that several learned networks are consistent with our mathematical analysis and then demonstrate that with such a simple theoretical framework, we can obtain reasonable reconstruction results on real images. We also discuss gaps between our model assumptions and the CNN trained for classification in practical scenarios.


Review discussion:
the paper presents theoretical results on inverting CNN in a random model .
There are several confusing notations in the paper, it is not clear at which level in the CNN the  invertibility is considered. Seems no pooling or non linearity is considered which is in fine a linear convolution invertibility in the sparse model.
- Section 2.2 , is not clear about the model, the activation h is a linear model, followed by RELU , then by max pooling. The final activation has therefore a dimension K.
Seems that invertibility is tackled only for the linear model (of dimension nK), no non linear activation considered and no pooling throughout the paper. Which is deceiving, since the purpose of the paper is the invertibility of a non linear model .
-- In Section 3, ReLU is turned to a linear model with some motivation from concatenated RELU. why not consider concatenated RELU in section 2.2 and do the analysis based on that.
-- what about the invertibility from the pooling operation as in showed in Bruna et al (2014)?
-- What is the motivation behind using M^2_{k} versus M_{k}?
Q. No pooling or non linearity?
A. Our paper considers nonlinear activation and max pooling.
Regarding nonlinear activation, we argue that if the filters come in positive and negative pairs, then one can disregard the ReLU or nonlinear operation as applying ReLU to the coefficients corresponding to both the positive and negative filters is equivalent to not applying ReLU to a vector of coefficients from the filters (not in pairs). See Section 3 for this argument.
About max pooling, it is denoted as {\mathbb M} (different from {\mathcal M}, which is the block-sparse model), but we agree that the definition is somewhat hidden. We revised sentences in Section 2.2 and 2.3 to define the max pooling operator. (updated on Dec. 16)
+: After applying the pooling operator {\mathbb M}, we lose the information of the location where the maximum value was, because it technically performs unpooling after pooling to keep the dimension. Our reconstruction bound calculation starts from c and h, which are pooled feature maps, so the information of the location where the maximum was is not important.
Q. Why not considering concatenated ReLU?
A. ReLU is more commonly used in CNNs. We believe ReLU-based networks are more realistic and more broadly interesting.
Moreover, the performance gain (coherence) with CReLU was not significant. We experimented with CReLU for conv(1,1) and conv(3,1): the coherence of ReLU-based network is 0.9427 and 0.5838, and that of CReLU-based network is 0.9476 and 0.5265.
Q. What about the invertibility of the pooling operation in Bruna et al. (2014)?
A. Our approach is different from Bruna et al.: we estimated || c - h ||_2 in Lemma B.4. (c is the true code and h is the reconstructed code from x.) "h" is the activation after the pooling operation. (h = {\mathbb M}(Wx; k))
Q. {\mathcal M}^2_{k} versus {\mathcal M}_{k}?
A. We need to look at differences in model-k sparse signals which are vectors with at most 2k non-zero coefficients.
Specifically, {\mathcal M}_{k} is a set of model-k sparse vectors, (a vector can be divided into k blocks, each of which contains at most 1 non-zero value), and {\mathcal M}^2_{k} is a set of the linear combinations of two elements in {\mathcal M}_{k}. {\mathcal M}^2_{k} is necessary in the middle of the proof, to represent the vector in the union of supports for c and h. See the proof of Lemma B.3 in detail.
- The explanation of how IHT is applied in Fig 4c-d, since it only inverts a single layer. From what I understand, IHT is applied to invert a single layer (conv5_1 to get pool4 activations), and then a trained decoder network is applied on pool4. Is this true ? If so, what happens if you apply IHT recursively on all layers ? Does this work ?
- The analysis seems to be about the invertibility of a single convolutional layer (+ non-linearity). In that sense, it's not very different from the standard sparse recovery setup.
But the surprising phenomenon is that deep neural "networks" are invertible, which consist of a cascade of such layers. To understand this invertibility, I feel one has to analyze whether the probability of failure blows up as you repeat the argument across multiple layers. Could you comment on how your analysis might extend to that ?
Q. What happens if you apply IHT recursively on all layers ?
A. To apply IHT on all layers, we need to define W matrices for each layer and run a separate version of IHT (perhaps with just one iteration or perhaps with more than one iteration, as I just described).
As long as the input to each layer remains sufficiently sparse and the number of filters, shifts, etc. are suitably balanced according to our Theorem 3.3, the entire process should, in theory, converge.
In addition, even on one layer (which is our current setup), we could apply IHT for more than one iteration to improve the reconstruction performance.
That is, we could repeatedly refine our estimate for the reconstruction of one layer by iterating several times.
In theory, this process will converge to an accurate reconstruction of the original image (with the error decreasing geometrically at each iteration).
However, in practice, the practical parameters may not strictly conform with theoretical expectations. See Table 2 and 4 for how the sparsity/coherence deteriorates from one layer to the next.
Q. What is the difference from the standard sparse recovery setup? Does the probability of failure blows up as you repeat the argument across multiple layers?
A. The main difference from the standard sparse recovery setup would be three folds: we considered pooling as well, we applied a positive-negative pair assumption in our analysis (which turns out to be true), and we showed that the feedforward CNN is equivalent to the one step IHT under the model-RIP condition.
If we used fully random filters at each layer and set up the parameters within our bounds, then running IHT at each layer would yield an accurate reconstruction.
Although, in this setting, fully random filters for the classification network do not give state-of-the-art classification performance, they are still reasonably good compared to learned filters (Table 1). For the learned filter case, the reconstruction can be less accurate by IHT, and the errors may be accumulated over layers.
However, if we focus IHT on a single layer and train a decoder to compensate the error for the rest layers, we can get decent reconstruction results as shown in Figure 4 and 5.
Summary of the paper
The paper studies the invertiblity of convolutional neural network in the random model. A reconstruction algorithm similar to IHT is proposed for layer-wise inversion of the network.
Clarity:
- The paper is confusing wrt to standard notations in deep learning.
Comments:
The paper makes two simplifications in the analysis of a CNN, that makes it map to a model based compressive sensing framework:
1-  The non linearity (RELU) is dropped. This is a big simplification, for random gaussian weights for instance we know by JL that we can preserve L_2 distance, when RELU is applied the metric changes (see for instance the kernel for n=1 in  http://cseweb.ucsd.edu/~saul/papers/nips09_kernel.pdf).
2- The pooling operation is modified to be shrinkage operator, that keeps the maximum value in a block  and sets to the zero other values, hence the dimensionality of the pooled representation is the same of the un-pooled representations. Note that in that cases the locations of where the maximum happens is known. This is not the case in the ’standard’ max pooling definition. IHT does not map to a forward of a CNN as described in the paper. (see next point)
3- It maybe that the notations  used in the paper are implying some confusions wrt to the standard notations in deep learning.
- Let z be the standard  pooled representation of dimension ‘k’.
- Define U as the unpooling  operation U(z, locations of maximum) :=  M(Wx,k).
Hence your model of inversion is assuming the knowledge of the ’standard’ pooling representation and the switches (max locations). Referring to M as a pooling operation is misleading and confusing, it is the ‘standard’ unpooling operation.
- Under this notations W^{\top} U(z, locations of maximum) is a sensible reconstruction algorithm , with the simplification of ignoring the RELU.
- Under these notations,  IHT is similar to a backward of the encoding neural network not  a forward. It is known if you take the derivative with respect to the input in a neural network, you get the transpose convolution also known as ‘deconvolution’ (which is not a correct naming) , hence this IHT iteration is nothing else then the well known transposed convolution.
4- I suggest the authors to rewrite the paper taking into account standard definitions and notation in deep learning as discussed in point 3, and to clearly state that the recovery is done under the knowledge of the switches.
After reading the authors rebuttal and the revisions , the reviewer maintains his main concerns with the paper. Many improvements are still needed for the paper to be ready to be published in ICLR, at the notation , presentation and the theory levels.
Thank you for your helpful comments! We have updated our responses as follows.
Re: The nonlinearity (ReLU) is dropped.
- Our analysis does not simply ignore the nonlinearity (ReLU). In the beginning of Section 3, we discussed that "the assumption that trained CNN filters come in positive-and-negative pair is validated by Shang et al. (2016)." Due to this fact, we can cut down half of the convolution weights and remove the ReLU unit. We updated the equations to make the reasoning more clear.
Re: The pooling operation is modified to be shrinkage operator. The locations of where the maximum happens (pooling switch) is known in the analysis.
- We would like to clarify that our theoretical analysis does not require the information about the pooling switches (i.e., locations of where the maximum happens). We emphasize this point in the end of Section 3 (right after Theorem 3.3).
We also revise our submission to define {\mathbb M} more explicitly in Eq. (2) (Section 2.3).
In particular, {\mathbb M} performs max-pooling and upsampling, The upsampling can place the non-zero value anywhere in the local region without affecting the validity of our reconstruction bound in Theorem 3.3.
In our experimental analysis, we particularly use the naive uniform upsampling so that the pooling switch has never been used for reconstruction with IHT.
Note that we use pooling switches in Algorithm 2 to recover the true underlying sparse code in order to study the model-RIP property.
However, Algorithm 2 is not for the IHT-based reconstruction, and the \mathbb M_known is also used in Algorithm 2 just for notation convenience.
Re: Notation
- Thank you very much for your suggestion. We have revised the paper to clarify the notation. We highlight the changes below:
c -> z: "true" underlying sparse code for generating the CNN input x
h: hidden activations before pooling
h -> \hat{z}: hidden activations after pooling = reconstruction of original activations
Please let us know if you have more suggestions on the notations. We will be grateful to revise our submission to eliminate any confusions.
Re: relation between IHT and deconvolution
- You are right in that IHT (when \Phi=W) involves convolution (line 4 of Alg. 1) and "transposed convolution" (or "deconvolution") for reconstruction (line 6 of Alg. 1).
In fact, this was the exactly the same intuition that motivated our work. Our main contribution is to provide a novel theoretical analysis by exploiting the connection between IHT and convolution/deconvolution (or encoder/decoder) in CNNs.
The authors propose a theoretical framework to analyze the recoverability of sparse activations in intermediate layers of deep networks, using theoretical tools from compressed sensing. They relate the computations that are performed by a CNN and a particular recovery algorithm (Iterative Hard Thresholding, IHT). They present proofs of necessary conditions for recoverability to hold, and also show detailed empirical evidence of how they hold in practice.
This is a well-written paper that presents a new angle on why the current CNN architectures work so well. The authors give a brief but sufficient review of the fundamentals of compressed sensing, present their main result relating feed-forward networks and IHT (a surprising result), and progress naturally to a detailed experimental section. The introductory analysis at the beginning of Section 3, in particular, delivers the gist of why the method should work with very approachable and simple math, which is not common in theoretical papers. The increasing complexity of the experiments, done in small steps, shows a nice progression from artificial distributions to a realistic experiment.
A few aspects should be improved.
First of all, although the treatment of ReLU non-linearities is sufficient, it is assumed with little discussion that max-pooling non-linearities shouldn't present a problem as well. A discussion of how this is inverted (e.g., with pooling switches) is needed.
The relationship between feed-forward nets and Algorithm 1 assumes tied weights. It might be worthwhile to mention that the result is stronger for the case of RNNs, where this is the case by design.
Although it might be obvious, it might help some readers to briefly note that the reconstruction algorithm is meant to be applied to each layer sequentially, basing the activations of each layer on the one above it (in back-propagation order).
Finally, the filter coherence measure must be defined either mathematically or with a proper reference.
Thank you for your helpful comments! We have updated our responses as follows.
Re. A discussion of how this is inverted (e.g., with pooling switches) is needed.
- The analysis of how CNN is inverted is separated in two sections. Our treatment on the ReLU nonlinearity is discussed in the beginning of the section 3. Pooling switches are discussed in the paragraph after Theorem 3.3 in the latest version.
In particular, we do not need to know the pooling switch for upsampling, and the reconstruction bound in Theorem 3.3 applies to any valid upsampling algorithm ("valid sampling algorithm" means that a non-zero value is placed at one location in the upsampled activation map. See Footnote 6).
Re. The relationship between feed-forward nets and Algorithm 1 assumes tied weights. It might be worthwhile to mention that the result is stronger for the case of RNNs, where this is the case by design.
- We added footnote 7 on page 5 in the latest revision:
"Multiple iterations of IHT can improve the quality of signal recovery.
However, it is rather equivalent to the recurrent version of CNNs and does not fit to the scope of this work."
Re. it might help some readers to briefly note that the reconstruction algorithm is meant to be applied to each layer sequentially.
- Footnote 3 on page 2 explicitly clarifies this:
"We can extend the equivalency on a single layer of CNNs to multiple layer CNNs simply by using the output on one layer as the input to another, still using the steps of the inner loop of IHT."
Re. The filter coherence measure must be defined either mathematically or with a proper reference.
- The filter coherence is defined in the end of Section 4.4 (where the table for coherence comparison is referred to). We added an equation for the definition in the latest version (footnote 12 at the bottom of the page 9).
The paper proposes to provide a theoretical explanation for why deep convolutional neural networks are invertible (at-least, when going back from certain intermediate layers to the image itself). It does so by considering the invertibility of a single layer, assuming the convolutional filters essentially correspond to incoherent measurements satisfying RIP.
In my opinion, while this is an interesting direction of research, the paper is not ready for publication. I feel the treatment does not go sufficiently towards explaining the phenomenon in deep neural networks. Even after reading the response from the authors, I feel the results are only a minor variation of the standard results from compressive sensing for sparse reconstruction with incoherent measurements.
A deep neural network is fundamentally different from a single layer---it is the "deep" part that makes the forward task work. As the authors note, there is significant deterioration when IHT is applied recursively----therefore, at best the theory explains the partial invertibility of a single layer. That a single layer is approximately invertible isn't surprising, that a cascade of layers *is*.
For any theoretical analysis of this phenomenon to be useful, I believe it must go beyond analyzing a single compressive measurement-type layer, and try to explain how much of the same theory holds for a cascade. I say this because it's entirely possible that the sparse recovery theory breaks down beyond a single layer, and invertibility ends up being a property caused by correlations between the weights of different layers.
In other words, there is no way to tell from the current results for individual layers whether they are in fact a step towards explaining the invertibility of whole networks.
Thank you for your helpful comments! We have updated our responses as follows.
We agree that it is more difficult to theoretically analyze on multiple layers due to the error aggregation. As Theorem 3.3 implies, the reconstruction error bound becomes larger if we stack multiple layers.
However, analyzing a single layer CNN is still important for understanding the invertibility of CNN. Even for a single layer CNN, although the invertibility may be intuitively plausible, there has been no nontrivial theoretical analysis in the literature.
In addition, our analysis is nontrivial as the pooling operator is involved. we believe our work provides an interesting and novel connection between neural networks and compressed sensing and hope that it will stimulate further work.

Meta-review:
role: PC
data: comment: While the reviewers found some interest in this work, I'm afraid I have to agree with the critique that the model studied is too simple that its relevance for deep learning is questionable.

Decision: Reject