Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors present a novel 2-stage technique to improve the classification accuracy of feedforward ANNs.  In particular, after the feedforward pass, a so-called introspective stage occurs, the goal of which is to ascertain why the particular class label was provided rather than a different label.  This stage, which tends to improve the accuracy of the class predictions, is modular and can be added onto networks under varying task conditions.
main_review: Vanilla feedforward networks are single-stage, filtering the inputs as they pass forward through the network and predicting the class of the input in the final layer. Motivated by the human tendency to not only gather sensory data, but to reflect on those data in the face of uncertainty to arrive at a classification, the authors introduce a second, introspective or reflective, stage. 

This stage is defined as the change in the parameters in the network if a different label were assigned to the network's input, and can be measured as a gradient with respect to the network weights of a loss function.  After showing how the calculation can be simplified in space and time, they then demonstrate how introspection can improve the generalizability of classifications to distributions that are shifted from the original training sets and can improve the calibration of the network, the confidence of the classification minus the accuracy.  They finally demonstrate how the approach can be used in a number of different applications.

Overall, the introspective approach is, to the reviewer's knowledge, novel and potentially of interest as a general method for improving the performance of feed-forward networks. The authors provide the reader with intuition (e.g., Fig 2), rigor, and empirical data (Fig 3/4).  One potential issue that arises in this empirical data is that introspection does not improve the accuracy of ResNet-18 on CIFAR-10C, as they state, although the ECE decreases systematically.  

Unfortunately, it is difficult to judge by eye whether the differences in Fig 4 are statistically significant.  The authors should include error bars or some other visual cue that indicate the uncertainty of the plotted values.  Additionally, without a statistically significant result, it is difficult to convince the reader that the authors' approach will work in a meaningful manner empirically in cases of interest.  

If possible, I would suggest the authors find such an example, or, if they cannot, to perhaps indicate future work that might improve the technique in such a way that it could in fact do so. Beyond this, the paper is written and organized well and clearly.
summary_of_the_review: The authors present a new method, introspection, that can be added on top of feedforward networks to improve accuracy and generalization of the networks.  While they have written a well-structured paper that provides both intuition and rigor, the empirical results are somewhat lacking in that there are no statistically significant results that support the claims of accuracy improvements that introspection can obtain.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors propose a modification to standard neural networks used for object classification tasks to incorporate what they call “introspective learning”. This consists on training a multi-layer perceptron (MLP) on the neural network introspective features. These are obtained by calculating the gradients over the last layer weights of the model on a loss function corresponding to posing an introspective question: why is the correct label A instead of B? The authors apply this procedure to several neural networks of the ResNet model family and show that the introspective-networks have better generalization to distributional shifts and smaller calibration errors on datasets with the same classes and image sizes as CIFAR-10. Finally, they show further improvements in multiple applications.
main_review: # Strengths
* An interesting modification to standard neural networks for object recognition with improvements in generalization and  calibration
* Extensive experiments showing the usefulness of the method in multiple applications

# Weaknesses
* Confusing explanation of the proposed method lacking a visual representation of the full model architecture, its components, and what the introspective features represent
* Model is evaluated only on CIFAR-10 related datasets which are very low resolution images and few possible classes
* According to the authors, the proposed method does not scale well for larger datasets. This makes its applicability very limited as CIFAR-10 is only a toy dataset
* Some controls seem rather arbitrary and a better choice of parameters for the alternative baselines could have been done

# Detailed review
## Role of introspection for object recognition
Object recognition in humans is an extremely fast visual computation, taking place within ~200ms of stimulus presentation, and it is unlikely to depend on introspection. It is not clear the motivation of why including introspective learning in artificial neural networks would improve their performance. Furthermore, standard neural networks are typically optimized to classify an object as belonging to one class and not belonging to the others (all classes are used in the cost function).  In that sense, it could be argued that they also are trained to answer the questions illustrated in Figure 1.

## Confusing explanation of the proposed methodology
Unfortunately, the authors could have done a better job at explaining their method. In my opinion, some of the mathematical details could be moved to the appendix, and in their place, the authors should focus on clarifying the training and inference process of the introspective networks. A visual representation of the full model architecture, its components, what the introspective features represent, and how the model is trained would be extremely helpful to the reader. Because of this it was not clear where the computational complexity of the introspective networks come from. The MLP used is very standard and the introspective features are obtained from the model’s last layer gradients.

## Poor choice of controls
Since adding the 3 layers to the standard model made it considerably worse, the choice of using 3 additional layers on the control for the same number of parameters was a poor choice by the authors. The MLP in the introspective network has a higher number of features but only a single layer. A better comparison would be to make the last layer of the standard model with more features, keeping only 1 layer, or keeping the same penultimate layer and adding a new much wider layer to compensate the smaller number of parameters. 

## Minor comments
* Fonts in several figures are very small. Please improve their readability
* In Figure 4b and Figure 6, the x-axis should be corruption severity and go from 1 to 5
* Figure 2 does not add much and could go to supplementary
* Figure 1 is very high-level and in its place, the authors should provide a more concrete visualization of their method

summary_of_the_review: The authors propose a very interesting modification to neural networks for object recognition and do a convincing job in showing that it improves generalization and calibration. However, the proposed method is only tested in CIFAR-10 and its use in larger networks and datasets is questionable. Furthermore, the explanation of the method is unnecessarily confusing and better controls could have been chosen to make a more convincing case.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes introspective networks, a technique utilizing the gradients of a trained base model to make more robust predictions when faced with distribution shift.
main_review: The paper proposes an interesting and novel approach for improving robustness and calibration under distribution shift.  Results on CIFAR10-Corrupted are very promising, and the authors further show the method is able to improve in active learning and OOD detection experiments when inputs are corrupted with noise.

Lemma 1: assumptions should be stated precisely in the lemma statement itself. As it stands, the lemma statement makes it seem like the statement holds exactly, which seems very unreasonable at first glance.
Regarding the reasonableness of the assumptions, a  "well trained" network might have high confidence predictions on input data it was trained on, but not necessarily on test data. Empirically, it has been observed that test data (particularly OOD test data) tends to have much higher entropy in predictions. Regarding theorem 1, can you test how well this approximation holds? In CIFAR10, there are only 10 classes, so it's reasonable to actually compute and store the full gradient features (especially since only the last layer is used). It would be good to see how a classifier performs using the full set of introspective features instead of the gradient wrt to the all ones vector. I would also highly recommend experiments to see if introspection (and this approximation in particular) scale well to more complex tasks with more classes like CIFAR100 or Imagenet, as I believe the current set of experiments to be quite limited.

**Questions and Suggested Ablations:**

Can you elaborate on why we should expect the introspective features to provide benefits when the new classifier is trained on the exact same training data as the base network? It seems very plausible to me that the gradients on training points the base
network has already overfitted heavily are a very different distribution than the gradients on previously unseen points, so I wouldn't necessarily have expected the introspective classifier to generalize well. I'm also curious what would happen if the introspection network is trained on a separate validation set.

How does this compare to directly taking the activations of the network instead of gradients? The final activations should be very closely related to the gradients of the last layer, differing by multiplicative terms corresponding to derivatives of the loss wrt to the output logits. This experiment would help to examine whether the structure of the gradient information from each class over possibly some benefits of just retraining model components on top of a pretrained model features.

Did you examine introspective features from other layers?

In the OOD detection, are the same adversarial images used for the feedforward and introspective network, or is the adversarial image for the introspective method specifically targeted at the extra network using introspective features?

Some missing related work:

Gradients as features: [2] is a crucial piece of missing related work, which also considers using gradients of a network as features, and for example show some similar results in recovering the original classifier performance on the trianing set. It would also be good to discuss and reference [1], which similarly uses gradients of base models as features, but applies it towards finetuning for transfer learning.
 
Conditional/predictive normalized maximum likelihood: Conditional/predictive normalized maxmimum likelihood [3, 4] considers retraining models for each label of the test input and combining these models' predictions to output a final prediction, essentially introspecting for each possible label to get predictions. While this work differs by using introspection to provide features for a separate classifier, it would be good to mention these other works as applying similar ideas of utilizing information on how model's change for specific labels of the test input in order improve predictions in the face of distribution shift.

**Misc comments:**

There's a bit of a weird jump in the assumptions in the paper: lemma 1 assumes cross-entropy loss, but in sec 3, we're assuming network is trained with MSE for the derivation. 

In addition to showing ECE for CIFAR10-C, I would also recommend including uncertainty-aware metrics like NLL and Brier score in the appendix, as these are proper scoring rules.


**Citations:**

[1] Zinkevich, Martin A. et al. “Holographic Feature Representations of Deep Networks.” UAI (2017).

[2] Mu, Fangzhou, Yingyu Liang, and Yin Li. "Gradients as features for deep representation learning." ICLR (2020)

[3] Zhou, Aurick, and Sergey Levine. "Amortized Conditional Normalized Maximum Likelihood: Reliable Out of Distribution Uncertainty Estimation." International Conference on Machine Learning (2021)

[4] Bibas, Koby, Yaniv Fogel, and Meir Feder. "Deep pnml: Predictive normalized maximum likelihood for deep neural networks." arXiv preprint arXiv:1904.12286 (2019).

summary_of_the_review: Overall, the paper presents an interesting approach with promising results for improving robustness, both in terms of accuracy and uncertainty estimation. I believe with a more clear presentation, more thorough discussion of related work, more extensive experimental evaluations, and ablations/evaluations of specific assumptions and design choices, this will be a valuable contribution.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors present a new inference pipeline for neural networks $f$ trained to solve classification problems. They augment the features processed by $f$ with the loss gradients with respect to the weights of the last layer. Their pipeline requires two steps: the first one coincides with a standard evaluation of $f(x)$, while gradients are evaluated and processed in the second step to derive the final prediction. Experiments show slight improvements in classification accuracy and cuts of calibration errors. 
main_review: The reading of the paper does not flow properly. The methods are hard to understand and are not technically sound. Many core elements of the math discourse are introduced abruptly, such as $L$ and $\mathcal{H}$. The definition of introspection is vague and not formal. The gap between the intuition about introspection presented in the introduction and the practical realization described in the methods sections remains wide and leaves the readers without a proper foundation to build their understanding. For instance, the intuition of introspection the authors give in the introduction and in Figure 1 calls for a reflection stage applied only to some promising alternative classes, while the proposed method applies the reflection stage to every single class.
Moreover, I think that some less relevant sections should be shortened and others more relevant should be deepened. For instance, given the nature of the scientific contribution, this paper does not require in my opinion such an extensive Application section, while spending more words on making clear the methods in the middle sections and the general idea in the Introduction could provide more value to the reader.

Yet, to me, the idea of enriching neural networks features with gradient information through a two-stage inference process seems novel and sounds interesting. Results suggest that this research direction can be relevant, so I encourage the authors to pursue this work and focus on communicating it properly.

In the following, I list some comments about specific parts of the paper:

INTRODUCTION
- I believe that philosophical references can be of great value to the ML and AI community. In this sense I appreciate the reference to Locke, I wish the authors have expanded more this point.
- "For the application of recognition... ...EQUATION (1)" This part is not clear. I do not understand why the authors use the projection concept in this context. $L$ seems a tag to indicate the final layer but then becomes a natural number ($L-1$). The first equation about $y_{feat}$ has a reference to $y$ that does not appear in the equation.

INTROSPECTIVE FEATURES
- The authors use the term explanations without defining what is an explanation for them and justify that definition. It is a textual string? A heat map on an image? Why we can consider heat maps as *explanations*? 
- The definition of introspection is very vague. "...is the measurement of change induced in the network parameters...", explicit this measurement, state the quantities. The reader here is expecting a formal definition but this definition is not formal.

INTROSPECTIVE FEATURE VISUALIZATION
- "...Grad-CAM indicates that the pink and round body, and straight beak are the reasons for the decision. [...] the network highlights the neck of the spoonbill to indicate that since S-shaped neck is not observed, x cannot be a flamingo. ..." I believe that here authors are overinterpreting the heat-maps. How can we say that the heat-map is caused by shapes rather than patterns? The clean high-level concepts of neck, beak, straight, S-shaped are what the network sees or what the authors see?

INTROSPECTIVE FEATURE EXTRACTION
- The assumption that a forward and backward pass through f is O(1) does not sound familiar to me. There is a dependence with respect to the size of the input. 
- Probably this subsection could be shorter, and at least part of the computational cost consideration could go in the appendix.

INTROSPECTIVE NETWORK
- H appears suddenly and everything is built on it. This is one of the more obscure parts of the paper.
- The assumption of a mean squared error loss is not trivial, we usually do not train classifiers with the mean squared error loss but with cross-entropy. 
- "having networks that introspect consecutively, by creating a trade-off" is not clear.
-  let the reader grasp how the reflection stage can fix the first inference of the sensing stage, spend some words on this case or maybe provide an example.

EXPERIMENTS
- "Given a data distribution $x \in X$" why distribution?
- "10 separate bins based on their prediction confidence". Considering the outputs of a neural network as confidences is problematic, this point should be expanded. DNNs are known to have unreliable uncertainty estimates (MacKay, 1995, Szegedy et al., 2014, Nguyen et al., 2015)
summary_of_the_review: The core parts of the paper are not clear and do not allow a proper comprehension and evaluation of the scientific contribution of this work. 
The overall idea seems valuable and experimental results appear promising. Yet, with the current manuscript, I can only judge the potential of this work rather than the work itself, so at this stage my recommendation is a 3: reject, not good enough.
review_title: null

Meta-review: The reviewers all appreciated the novel concept behind the work. I agree with this, I think the principles behind the work are novel and interesting, and I would encourage the authors to improve the validation of this method and publish it in the future.

However, reviewers also raised a number of issues with the current paper: (1) the evaluation appears a bit preliminary, and could be improved significantly with additional datasets and more ablations/comparisons; (2) it's not clear if the improvements from the method are especially significant; (3) the writing could be improved (I do see that the authors made a significant number of changes and improved parts of the paper in response to reviewer concerns to a degree). Probably the writing issues could be fixed, but the skepticism about the experiment results seems harder to address, and while I recognize that the authors made an effort to point some existing ablations in the paper that do address parts of what the reviewers raised, I do think that in the balance the experimental results leave the validation of the work as somewhat borderline.

While less important for the decision, I found that the paper is somewhat overselling the contribution in the opening -- while the particular concept of using gradients as features in this way is interesting, similar ideas have been proposed in the past, and the paper would probably be better if it was more clearly positioned in the context of prior work rather than trying to present a new "framework" like this. It kind of feels like it's biting off too much in the opening, and then delivering a comparatively more modest (but novel and interesting!) technical component.
Decision: Reject