Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper presents an approach to simultaneously train a prediction network along with a generative model to map the neural network activations back to inputs. The use cases of doing so include (1) interpreting the information encoded in an activation, (2) composing the network with itself and other pre-trained networks, which in turn provides a means to get uncertainties and to regularize the network, and (3) to assert that the network satisfies some constraints on what it can and cannot learn in the intermediate layers. The paper is evaluated on several image benchmarks and speech benchmarks and showcases all the above benefits. 

main_review: The paper is well written, proposes a very new technique for training and analyzing the neural networks, and performs extensive evaluation with promising results. I thoroughly enjoyed the paper and definitely recommend it for acceptance. 

The idea of learning a generative model to map the activations back to the inputs in-tandem is unique. In fact, I was expecting the performance to degrade, but surprisingly the performance is as good as a standard network. 

However, it is not clear to me how to evaluate the quality of the inversions (it can turn out that the generative modeling didn’t converge). Some of the images in figure 3 are promising and at the same time, some of them raise suspicions. For eg 3(b), in many of the intermediate images, it looks like the person has hair (instead of being bald). Can the authors comment on this?

Additionally, since ReDecNN performance is similar to DecNN, the expected “data augmentation” (line 162) is not really happening. 

Nevertheless, the indirect evaluation through the uncertainties, OOD detection, and calibration is interesting. 

Also, what is the rationale for using a deterministic generator model? What would happen with a non-deterministic generator model?

Overall, the technique presented in this paper is novel and promising and is definitely interesting to the community. 


limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes to jointly train a generative model that maps activations back to the input space. The work demonstrates several benefits of this approach.

* Probing: It provides a way to decode the activations which enables the users to make sense out of it since the decoded activations are in the input space. 
* Composing networks: The decoded activations allow to compose a DecNN model with another model with the same input type. 
* Data Augmentation: The DecNN model can be composed with itself. This can be viewed as data augmentation since the decoded activations are of the same "type" as the input and retain the features that are relevant to the prediction task.
* Ensemble network. The DecNN model can be recursively composed with itself to get the “Recursively Decodable Neural Network” (ReDecNN) model. This model can be viewed as an ensemble network. The paper shows the usefulness of ReDecNN for the tasks of measuring uncertainty, OOD detection, and calibration.
* Constrain the activations: The decodable activations makes it possible to specify the kind of information an activation is not supposed to capture. This provides assertion-like capability to the neural network.

main_review: **Originality**

The idea of invertible generative models has been explored in earlier works (invertible flows). However, this work is in a supervised setting where the activations are mapped to input space by training a generative model jointly with the classifier. Also in this work, no prior is imposed on the activations. Instead of relying on post-hoc computation, the proposed approach constrains the activations to be invertible. 

The proposed recursive network ReDecNN presents a natural way for defining an ensemble network by randomly selecting activations (from the activation tree) to be fed to the network. This is especially appealing as the decoded activations retain only the task specific information.

**Quality**
* The paper is technically sound and the claims are supported by appropriate experiments
* The analogy with composition in traditional software seems a bit far fetched. The proposed composition by using the decoded activation as conduit is very different from the conventional composition operation. For example, consider two hypothetical image to image networks f_flip and f_darken.   Now f_darken(f_flip(img)) will flip and darken the image. However, with the proposed approach f_darken(g_phi(h)) will only darken the image since the decoded activation will invert the flip. Admittedly, f_darken(f_flip(img)) is possible only when the input and output are of the same type. Comparing this conventional composition is confusing. Considering that “Compositionality” appears in the title, authors may want to clarify this aspect.
* The authors have gone to great lengths to demonstrate the usefulness of the recursive self-composition. All the claims are well supported by appropriate experiments. To demonstrate that the methods are applicable to other domains, authors have also presented results on the utterance and action recognition tasks. 

**Clarity**

* The paper is well written and easy to read. The experiment results are presented well.
* Section 5. Line 143 talks about f1(f2(x)) but line 149 gives an example for f2(g_phi(h)). It will be better if the outer function is the same at both the places. It will also make sense to make it explicit that g_phi is a generative model trained along with f1 (and not f2). 
* Apart from the computational cost, are there other reasons for parameter sharing across layers?
* Authors may want to emphasize the role of probing in error analysis.

**Significance**

* The idea of decoding the activations is novel. Since the decoded activations are in the input space, users can make sense out of them. The distinguishing feature of the approach is that no post-training computation is required to get the decoded activations.
* In addition to providing a regularizing effect, the decoded activations can also be used for data augmentation. This is especially useful since the decoded activations retain only the features that are important for prediction. 
* The functionality of “constraining decodable computation” has a potential for significant impact. One can specify the protected information in input space and the method will take care that the protected information is not abused for solving the task.


limitations_and_societal_impact: It remains to be seen if the regularization effect of constraining the activations to the decodable space prevents it from learning complex tasks.
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: The paper proposes decodable neural networks (NN) where the idea is to add an extra term in the objective that focuses on reconstructing the input from the hidden activations thereby allowing for probing the NN, composing them ( along with a pretrained model), and constraining certain decodable computations. They show experiments on simple image datasets ( MNIST, FashionMNIST, CelebA) and highlight the usefulness of their formulation for out-of-distribution detection, adversarial robustness, calibration, and protection attribute prediction.
main_review: **Originality**: The idea of reconstructing the input from hidden activations seems old and used extensively in autoencoders. I didn't quite get the analogy to software 1.5. What does it mean to obtain the decodable representation of the same "type" as input? I can see that the space of representations can be the same but there is nothing in the objective that forces the regenerated input to be the same "type" as the original input, especially if you take the software analogy where types have specific meanings ( like keywords, identifiers). I felt that the analogy to software was unnecessary because the main idea doesn't bear any connection or even takes motivation from the software domain. I felt the invocation of software terms like *probing our “variables” (i.e. activations) for meaning, constraining model logic with “assertions”, and composing neural network modules together* (Line 60-61) was just a fancy way of motivating the work and somehow showing that they are doing something different from previous work. 

**Technique**: I appreciate that the authors have mentioned this in limitations that their technique has a high computational cost. I would some analysis of the same though. I have some questions about some of the choices that the authors made in the formulation. For example, in Equation (4), what is the need for reconstructing the input from *every* hidden layer and then taking a mean of that and adding to the objective? Essentially, we are directing the NN here to decode the input from every hidden layer in the NN, which places an additional burden on the NN. Imagine a big NN with multiple layers ( which will be used for slightly complex datasets like Imagenet), it will be extremely hard computationally to perform training. I understand that this gives the ability to probe every layer in the NN but at a huge computational cost thereby limiting the scalability of the approach to bigger models (especially if the recursive decodable NN in Equation 5 is used - which is the model that performs better in experiments).

**Experiments and Results**: I like the use-cases where the paper has shown results. To me, Section 5.2 and Section 6 were quite interesting. In Section 5.2, though I fail to understand why using a ResNet-18 pretrained classifier was worse than both without using any pretrained model or using a linear classifier? Also, in Table 3, how were the attributes for OOD ( wearing a hat, long hair, bald ) chosen. Since in Table 2, MC is better than ReDecNN except in CelebA, choosing only CelebA for results in Table 3 seems unfair to other baselines. Similar question for attributes listed in Table 6. Were these results cherry-picked to show the superiority of ReDecNN over other techniques? As mentioned previously, I would also like to see an analysis of computational complexity of all the methods and the tradeoff with performance and interpretability. Anouther major concern I have related to the results is that the authors have chosen very simple datasets (MNIST, Fashion, CelebA). I feel like to evaluate the effectiveness of the proposed formulation, experiments on ImageNet or other datasets ( with higher image resolution and more complex) are needed.

**Clarity**: The paper is clear and well written. I felt that there are too many results though that may divert the attention from the main claims (like Section 7) that could have been placed in the Appendix.
limitations_and_societal_impact: I think the authors have done a fair job of listing the limitations and potential negative societal impact of their work. One suggestion in limitation can be to add that they have shown results on very simple datasets and model (only fully connected nets). So their technique might not be generalizable to other complicated datasets and models. The authors mention that extending to other models will be straightforward, but I think it is not fairly straightforward especially at the cost of severely increasing the computational complexity.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This work proposes DecNN, and the main goal is to allow people to interpret the intermediate neural network activations. The base model is an 8-layer MLP. The core idea is to train another generative model that takes the MLP activation as the input, and the generative model aims to decode an image in the model input space. During training, the loss of the generative model aims to reconstruct the input image given the activations of all intermediate layers. They further design ReDecNN, which recursively composes DecNN by taking the output of the generative model as the input of DecNN. They demonstrate that the generative model produces interestingly interpretable outputs to explain why the classifier makes correct or wrong predictions. They also show some promising results for out-of-distribution detection, calibration, and fairness.
main_review: Model interpretability is an important topic, and the difficulty of debugging is indeed one weakness comparing neural networks to traditional programs. Therefore, in general this work is well-motivated, and it demonstrates some pretty interesting results on interpreting neural network activations.

However, my main concern of this work is that simply applying this framework to an MLP model does not seem convincing. Due to the simplicity of the model architecture, the experiments are only conducted on relatively simple classification tasks, and the baseline performance is not great. What are the main difficulties of applying this framework to convolutional neural networks and/or recurrent neural networks? I wonder whether DecNN/ReDecNN still performs well when applied to more complicated neural network architectures.

Related to the fact that the implementation is based on a simplified model architecture, the authors did not empirically compare with recent OOD detection and calibration works, e.g., [20] [22] [23] [24]. Therefore, it is hard to tell whether ReDecNN indeed achieves a better trade-off between the classification accuracy and the OOD/calibration performance. The results in Table 5 show that even if the model simply utilizes a better model to improve its own performance, the OOD/calibration performance decreases notably, and in some cases the performance becomes worse than most or all baselines. This kind of indicates that the framework might not fit models that are able to better classification accuracies.

The results show that in general ReDecNN has lower classification accuracy than DecNN. I suppose the recursion depth D could be an important hyper-parameter for the performance. Do you have any ablation study that evaluates the performance of different D? Does increasing D generally improve the OOD detection/calibration results but decrease the classification accuracy?

While the discussion in Section 6 about fairness is interesting, I think the training objective is not directly related to the ReDecNN architecture. As we have seen, ReDecNN itself does not improve fairness. Have you tried applying the new term in Equation (8) to standard neural networks? I suppose you can simply achieve similar fairness results in this way.
limitations_and_societal_impact: The authors adequately addressed the limitations and potential negative societal impact.
review_title: null

Meta-review: UPDATE: The revision has been reviewed and the paper is officially accepted.

----

After extensive discussions among SACs, ACs, and the program chairs, we have decided to conditionally accept this paper. The primary concerns were around over-claiming, the interpretation of results, and the analogy to programming (which was viewed as a distraction from the main contributions of the paper).  There was some debate around whether these issues constitute a "fatal flaw" that should prevent the paper from being published at NeurIPS, but all agreed that these issues need to be addressed. We hope that the authors will take into account feedback from the reviewers and make the changes that were promised as part of the rebuttal.

Additionally, in order to be accepted, the following changes must be made in the revision:

Introduction:

L52: "This joint training optimizes the classifier not only to be highly performant, but constraints its activations to a manifold that the generative model is able to decode. Consequently, the networks entire computation is constrained to a 'decodable' vector space... " This is misleading and should be rephrased or removed; just because the activations at a given layer can decode the input doesn't mean that that information is being used by the downstream layers to actually make decisions.

L59: This paragraph should focus on what the paper actually demonstrates empirically; essentially remove the first sentence. 

Section 3: 

Highlight that equation (4) doesn't require that the reconstructed images $g_\phi(h_l)$ actually produce the same output y, so the reconstruction may not actually be highlighting the features that were important for f to make its decision in the first place.

Section 4: 

This section should be significantly rewritten to avoid making claims unsupported by the evidence. Specifically:

L116: "In practice, this theory leads us to expect 'useless' information..." This would be true if Equation (4) wasn't rewarding the network for keeping the useless information to help support the decoding task. 

L119: This paragraph is misleading and should be removed. While it is true that you can visualize the information at each layer, you cannot actually tell which of that information is actually being relied upon to make the decision. 

L130: This is the only paragraph in this section that states an empirical fact; the whole section should focus more on what this experiment objectively shows, but avoid making claims not supported by the observations made in this experiment.

L138: This paragraph again misrepresents what can really be inferred from the generated images. 

Section 6. 

The analogy with assertions is misleading as it implies a level of guarantees that is not supported by the evidence. The section should focus more closely on what the experiments actually demonstrate for this application. The attractiveness task should also be replaced with the "bald vs not bald", and "bearded vs not bearded" experiment presented in the rebuttal. 

Abstract: 

The abstract should be consistent with all the changes made to the rest of the paper. 

----

The reviewers found the idea of jointly training a generative model to additionally map neural network activations back to inputs quite interesting and novel. During discussions, a major concern about the approach that came up was regarding the computational cost of the approach and thereby its scalability for training more complex architectures and tasks. The additional results with CNN layers was helpful, but scalability still remains a challenge. However, there are enough interesting novel ideas and applications presented in the paper that could potentially open up many interesting future directions. It would be great if authors can incorporate the detailed feedback and clarifications from reviews and additional experiments in the final version of the paper.
Decision: Accept (Poster)