Introspective Learning : A Two-Stage approach for Inference in Neural Networks
Authors: Mohit Prabhushankar
Ghassan AlRegib
Venue: NeurIPS 2022 Accept (2022)
Keywords: Recognition
Active Learning
Out-of-distribution Detection
Model Calibration
Uncertainty

Abstract: In this paper, we advocate for two stages in a neural network's decision making process. The first is the existing feed-forward inference framework where patterns in given data are sensed and associated with previously learned patterns. The second stage is a slower reflection stage where we ask the network to reflect on its feed-forward decision by considering and evaluating all available choices. Together, we term the two stages as introspective learning. We use gradients of trained neural networks as a measurement of this reflection. A simple three-layered Multi Layer Perceptron is used as the second stage that predicts based on all extracted gradient features. We perceptually visualize the post-hoc explanations from both stages to provide a visual grounding to introspection. For the application of recognition, we show that an introspective network is 4% more robust and 42% less prone to calibration errors when generalizing to noisy data. We also illustrate the value of introspective networks in downstream tasks that require generalizability and calibration including active learning, out-of-distribution detection, and uncertainty estimation. Finally, we ground the proposed machine introspection to human introspection for the application of image quality assessment.

Introspective Learning: A Two-Stage Approach for Inference in Neural Networks
Mohit Prabhushankar
Ghassan AlRegib
Electrical and Computer Engineering Georgia Institute of Technology
Electrical and Computer Engineering
Georgia Institute of Technology
Atlanta, GA 30308 mohit. www.tet..gteecc.ccmm
Atlanta, GA 30308 al alregibågatech. edu
Abstract
In this paper, advocate for two stages in neural network's decision making process. The first is the existing feed-forward inference framework where patterns in given data sensed and associated with previously learned patterns. The second stage is a slower reflection stage where we ask the network to reflect on its feed-forward decision by considering and evaluating all available choices. Together, we term the two stages as introspective learning. We use gradients of trained neural networks as measurement of this reflection. A simple three-layered Multi Layer Perceptron is used as the second stage that predicts based on all extracted gradient features. We perceptually visualize the post-hoc explanations from both stages provide a visual grounding to introspection. For the application of recognition, we show that an introspective network 4% more robust and 42% less prone to calibration errors when generalizing to noisy data. We also illustrate the value of introspective networks in downstream tasks that require generalizability and calibration including active learning, nut-of-distribution detection, and uncertainty estimation. Finally, we ground the proposed machine introspection to human introspection for the application of image quality assessment.
Introduction
Introspection is the act of looking into one's own mind (L. Classical introspection has its roots in philosophy. Locke (2), the founder of empiricism, held that all human ideas come from experience. receives passive
This experience is result of both sensation and reflection. By sensation, information using the sensory systems of sight, sound, and touch. Reflection is the objective observation of our x and an objective of recognizing æ, we first sense some key features the bird. These features include the color and shape of the body, feathers and beak. The features chosen based existing notion of what is required to contrast between birds. We then associate these features with existing knowledge of birds and make coarse decision that & is spoonbill. This is the sensing stage. Reflection involves questioning the coarse decision and asking why x cannot be a Flamingo, Crane, Pig or any other class. If the answers are satisfactory, then an introspective decision that indeed spoonbill is made. The observation of this reflection is introspection.
mental operations. Consider the toy example in Fig. [ Given image
In this paper, adopt this differentiation between sensing and reflection to advocate for two- stage neural network architectures for perception- based applications. Specifically, we consider classification. The sensing stage is any existing feed-forward neural network including VGG (3), ResNet (4) and DenseNet (5) architectures among others. These networks sense patterns in x and make coarse feed-forward decision, ý. The second stage examines this decision by reflecting on the introspective question 'Why ÿ. rather than yr ?' where yr is any introspective class that the sensing network has learned. Note that there is no external intervention new information that informs
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Two Stages of Inference by Introspection
Sensing
Reflection
Sense pink feathers, straight beak
Why Spoonbill, rather than Flamingo? does not have S-shaped neck
X
Spoonbill
Spoonbill
Why Spoonbill, rather than Crane? does not have white feathers
y
Why Spoonbill, rather than Pig? x's leg and neck shapes are different
Feed-Forward Sensing
Figure 1: Example of the introspection process. The visual post-hoc explanations are from Grad- CAM (6) framework using the proposed features, Additional heatmaps along with the generation process is provided in Appendix A
The written text is for illustrative purpose only.
the reflection stage. Hence, the introspective features are post-hoc in the sense that they are not causal answers the introspective questions but rather, are the network's notion of the answers. This is inline with the original definition of introspection which is that introspection is the observation of existing knowledge reflected upon by the network. These post-hoc introspective features are visualized using Grad-CAM (6) in the reflection stage of Fig.
The challenge that address in this paper is in defining the reflection stage in terms of neural networks. The authors in (7. utilize the same intuition to construct reflective stage composed of explanations. However, when the train and distributions are dissimilar, the predictions and hence the explanations are incorrect. We overcome this by considering alternative questions for the explanation. Consider any dataset with N classes. A network trained on this dataset will have N possible introspective questions and answers, similar to the ones shown in Fig. I Our goal is to implicitly extract features that answer all N introspective questions without explicitly training on said features. This involves 1) implicitly creating introspective questions, 2) answering the posed questions obtain introspective features, and 3) predicting the output ý from the introspective features. We show that gradients w.r.t network parameters store notions about the difference between classes and can used as introspective features. We use an MLP termed H(()), our introspective network that combines all features to predict y. A limitation of the proposed method is the size of N which is discussed Sections 3
We show that the introspective prediction īj is robust to noise. An intuition for this robustness is that not only should the network sense the feed-forward patterns, it must also satisfy H(-)'s N notions of difference between classes. Hence, during inference, we extract N additional features that inform the introspective prediction j. The main contributions of this paper are the following:
1. We define implicit introspective questions that allow for reflection in neural network. This reflection measured using loss gradients w.r.t. network parameters across all possible introspective classes in Section
2. We provide methodology to efficiently extract introspective loss gradients and combine them using a second H(-) MLP network in Section
3. We illustrate H(-))s robust and calibrated nature Section We validate the effectiveness of this generalization in downstream applications like active learning, detection, uncertainty estimation, and Image Quality Assessment in Appendix D
2 Background
2.1 Introspective questions, features, and network
Introspective questions The choice of 'Why i, rather than yr'? not arbitrary. The authors in ( describe three questions that complete post-hoc explanations correlation, counterfactual, and contrastive questions. These questions together allow for an alternate form of reasoning within neural networks called abductive reasoning. The sensing network predicts based on correlations.

---

Since, we do not intervene within the data during the reflection stage, counterfactual questions cannot be answered. Hence, we use contrastive questions as our introspective questions. Further details regarding abductive reasoning and our choice of question is provided in Appendix A
Gradients as Introspective Features The gradients from a base network have been utilized in diverse applications including post-hoc visual explanations (6, 2 TCO, adversarial attacks (.D, uncertainty estimation
(12), anomaly detection (3 [2), and salieney detection (15) among others. Fisher Vectors use gradients of generative models to characterize the change that data creates within features (16). (i uses gradients of parameters to characterize the change in manifolds when new data is introduced to an already trained manifold. Our framework uses the intuition from (17) to characterize changes for datapoint that is perceived as new, due to it being assigned an introspective class yı. that is different from its predicted class ÿ. In (3), the authors view the network as a graph and intervene within it to obtain holographic features. Our introspective features are also holographic the sense that they characterize the change between ý and y/ without changing the network. However, our features do not require interventions that become expensive with scale.
Two-stage networks The usage of two-stage approaches to inference in neural networks is not new In 7 the authors extract Grad-CAM explanations from feed-forward networks to train a reflective stage. However, our framework involves reflecting on all contrastive questions rather than correlation questions. The authors in 19 propose SimCLR, a self-supervised framework where multiple data augmentation strategies are used to contrastively train an overhead MLP. The MLP provides features which are stored as a dictionary. This feature dictionary is used as a look-up up table for new test data. In this paper, we use gradients against all classes as features and an MLP (-)). to predict on these features. (20) and (21) consider all classes in a conditional maximum likelihood estimate on test data to retrain the model. These works differ from ours in our usage of the base sensing network. (22) uses gradients and activations together as features and note that the validity of gradients as features is in pretrained base networks rather than additional parameters from the two-stage networks. This adds
to our argument of using two-stage networks but with loss gradients against introspective classes.
2.2 Feed-forward Features
For the application of recognition, a sensing neural network f(-) is trained on a distribution A' to classify data into N classes. The network learns notions about data samples when classifying them. These notions are stored as network weights W. Given a data sample x, fx) is a projection on the network weights. Let Vfeat be the logits projected before the final fully connected layer. In the final fully connected layer [r)(-), the parameters Wr, can be considered as N filters each of dimensionality dr-1 x 1. The output of the network ğ is given by,
Ifoat fL- f2.1(1)) NN1
(1)
arg max y VWL E I 11) E Fdc-1x1
Here i is the feed-forwart inference and feat are the ford-forward features. In this paper, we compare our introspective features against feed-forward features. Since introspection occurs after f(.), all our results are plug-in -n on top of existing f()).
3 Introspective Features
In this section, we describe introspective features and implicitly extract them using the sensing network. We then analyze them for sparsity, efficiency, and robustness.
Definition 3.1 (Introspection), Given network f(-), datum x, and the network's prediction f()) G. introspection in f() the measurement of change induced in the network parameters when a label yr is introduced as the label for x. This measurement is the gradient induced by a loss function J 9i), w.r.t. the network parameters.
This definition for introspection is in accordance with the sensing and reflection stages in Fig. | The network's 's prediction ÿ is the output of the sensing stage and the change induced by an introspective label, yı, is the network reflecting on its decision ÿ as opposed to yı. Note that introspection can occur when ý is contrasted against any trained label yr, E [1, Nj. For instance, in Fig. D the network is asked to reflect on its decision of spoonbill by considering other yr that x can take flamingo, crane, or pig. These results are extracted from ImageNet (23) and hence, / < [1, 1000). Additional visual introspective saliency maps similar to Fig. are provided in Appendix A
3

---

Why 5, rather than
Why 5, rather than 1?
Why 5, rather than 2?
Why 5, rather than 4?
Input Image
Why 5, rather than
Why 5, rather than
Figure 2: For the input image on the left, the Vw, Jyr,, 5) are shown on the right. Each image is visualization of the 50 x 10 gradient matrix. All images are sparse except in the prediction row 5 and introspective question row i.
Reflection is the empirical risk that the network has predicted a as instead of yr- Given the network parameters, this risk is measured through some loss function J (yr, ) yr one-hot vector with at the 7th location. The change that is induced in the network is given by the gradient of Jyy1.) 9) w.r.t the network parameters. For an N-class classifier, there are N possible introspective classes and hence N possible gradients each given by, the introspective features. Since we introspect based on classes, we measure the change in network weights in the final fully connected layer. Therefore, the introspective features are given by,
J ú). E [1, Here, ry
V w. J(y 9), [1,N,,r RAI-IXN
where WL are the network weights for the final fully connected layer. Note that the final fully connected layer from Eq. [ has dimensionality of R 1xN, For every x, Eq. 2 is applied N times to obtain N separate rr. We first analyze these features.
3.1 Sparsity and Robustness of Introspective Features
Consider vy Eq. Each ry a dr.-1 N matrix. Expressing gradients rr separately w.r.. the different filters in Wr. we have row-wise concatenated set of gradients given by.
I 0): J D): Vw ((n. WE (y),
(3)
where each Wr.gj E Rdd--x1 and r1 E Rdd.-xN2 For all data x E X the following lemma holds:
Lemma 1. Given a unique ordered pair (z. ④) and a well-trained network f(.), the gradients for loss function J (31 8) w.r.t. classes pairwise orthogonal under the second-order Taylor series approximation, each class paired with the predicted class.
Proof. Provided Appendix B.
Sparsity Lemmal] S that backpropagating class yı does not provide any information W W.j... of and hence there no need to use V Vw J [uy, 9). A as features when considering yı- The proof Jur 9) for an introspective class reduces to,
provided in Appendix B.I
U/
Vw-J(yr 9) wwrr + Vwlog
(4)
where Vý is the logit associated with the predicted class. We demonstrate the sparsity of Eq. 4 in Fig. 2 A two-layer CNN is trained on MNIST (24) dataset with test accuracy exceeding 99%. This satisfies the condition for Lemma that f(() is well-trained. The final fully connected layer this network has a size of 50 10. We provide
input image of number 5 to trained network as shown in Fig. The network correctly identifies the image as 5. We then backpropagate the introspective class using (5, 0) with ÿ 5 and yr 0. This answers the question 'Why 5, rather

---

Introspective Features
Introspective Network
Gradients Weights, W,
aj y1)
y1 1"
047
MI
MLP 20()
y
Sensing
(wd(3,y))
Network f()
C67
007
CB/
621
WL
=
vectorized anc normalized gradients
f()-1)(x)
Figure 3: Introspective Learning process. Once for all images in the dataset are generated, an introspective network H(-) is trained. During testing, noisy image passed through sensing network f(.), extraction module to generate r, and finally through the introspective network H(--).
than 0?". The gradient features in the final fully connected layer are the same dimensions as the final fully connected layer. 50 x 10 more generally Odd,-1 N). This matrix is displayed normalized image in Fig. 2 Yellow scales to and blue is -1 while green is O. It can be seen that the only values present the matrix are negative WL,0, in blue, and positive in WL.3 L.s, in yellow. This validates Eq. 5 that for fully-trained network the only values, and hence the only information, required from Wr. i. for difference among all matrices is the location of the negative values that exist at V Ww2. for different values of T.
0 Vwz.o- We show the matrix V Vwz when
0, 1,
The
Robustness Eq. motivates the generalizable nature of introspective features. Consider some noise added x. To change the prediction ý. the noise must sufficiently decrease Vy from Eq. and increase the closest logit value, yr, to change the prediction. Hence, it needs to change one orthogonal relationship. However, by constraining our final prediction on N such features, the noise needs to change the orthogonal relationship between N pairwise logits. This motivates an introspective network H(-)) that conditioned on all N pairwise logits.
3.2 Efficient Extraction of Introspective Features
From Lemma 1 the introspective feature is only dependent the predicted class and the introspec- tive class making their span orthogonal to all other gradients. Hence,
Ww.., J(y. D). E [1,N),r idd--x1
(5)
Building on Lemma I we present the following theorem.
Theorem 1. Given unique ordered pair (x. 9) and a well-trained network f(),, the gradients for a loss function J (yr, ú), [1,N] w.r.t. classes when are N orthogonal one-hot vectors is equivalent to when yr vector of all ones, under the second-order Taylor series approximation.
Proof. Provided in Appendix B.2
The proof follows Lemma Theorem states that backpropagating a vector ones (1w) is equivalent backpropagating N one-hot vectors with ones at orthogonal positions. This reduces the time complexity from (NN) to constant (l1) since we only require a single pass to backpropagate Ly. Hence, our introspective feature is given by.
Ta = Vw, Jw (1N, 0), Ta E Pd 1y = Nx1
Note the LHS now Tz instead of r from Eq. 5 The final introspective feature is a matrix of the size Wz extracted in (11) time with space complexity of O(dz-1 N). rz is vectorized
and scaled between [-1, 1] before being used in Sections and as introspective features. This procedure is illustrated in Fig.
5

---

Introspective Network
Once In are extracted using Eq. 6 the introspective lahel ý from Fig. is given by ú = H(rr) where H()) is an MLP. In this section, we analyze H(-). From Fig. 3 (()) is any existing trained network used to obtain introspective features rz- It trained to predict the ground truth y given any
x. Based on the assumption that H(rr) = [(ylf(x)) and hence expectation of H(rrz) is 0, the
loss function can be decomposed as,
[[(f(x) - y) B[(f(x) -
y)
(7)
Note that since the goal is to predict y given 2, M(rx)) = (f)l(())) is a fair assumption to make. Substituting for f(x) in Eq. 7 and using variance decomposition of onto f(x), we have,
B(U -) Var(ū) + -
(8)
This decomposition is adopted from structured calibration techniques. A full derivation is presented in (25). The first term Var(y) is the the variance the prediction from f()). This term the precision of f()) and is low for well trained network. The third term is the MSE function between the
introspective network H(()) and the ground truth. minimized while training the H((-) network. Note that minimizing Eq. g can occur by maximizing Var(H((..).. We use fisher vector interpretation from (I7) to analyze Var(H(rzz). If HH()) is a linear layer with parameters WH, the Var(H((ra) term reduces to W Ww x Var(rz) x Tr(r-"--rr E-ir,) where Σ is the covariance matrix. a gaussian approximation for the shape of the manifold. Generalizing it to higher dimensional manifold and replacing E with F, we have,
The second term is the variance of the network H(-). given features
Var(H(r=)) Tr(r" rx),
(9)
N Var(H((zz)) = Σ F
(10)
j=1
The RHS of Eq. 10 is sum of fisher vectors taken across all possible labels.
When do introspective networks provide robustness? We use Eq. I0 to analyze introspective learning usage. Specifically, we consider two cases: When input is taken from the training distribution X, and when is taken from a noisy distribution X".
1. When a sample x E X' is provided to a network f(-) trained on X, all TJ. # Ø in Eq. 10 tend to 0. The RHS reduces to r F- rg is a function of (xx only and hence adds no new information to the framework. The results of H(-) remain the same as f()). In other words, given trained ResNet-18 on CIFAR-10, the results of feed-forward learning will be the same as introspective learning on CIFAR-10 testset.
2. When new sample x' & X' is provided to a network f(-) trained on X, fisher vector based projection across labels is more descriptive compared feed-forward approach. The N gradients in Eq. T0 add new information based on how the network needs to change the manifold shape F to accommodate the introspective gradients. Hence, given distorted version of CIFAR- 10 testset, our proposed introspective learning generalizes with a higher accuracy while providing calibrated outputs from Eq. 8) We empirically illustrate these claims in Section 6 We motivate other applications including active learning, Out-Of- Distribution (OOD) detection, uncertainty estimation, and Image Quality Assessment (IQA) that are dependent on generalizability and calibration Section
5 Related Works for Considered Applications
Augmentations and Robustness The considered Ta features from Eq. 6 can be considered as feature augmentations. Augmentations, including SimCLR (1), Augmix (6) adversarial augmen- tation (27). and noise augmentations (28) have shown to increase robustness of neural networks.
We use introspection on top of non-augmented and augmented (Section 5) networks and show that our proposed two-stage framework increases the robustness to create generalizable and calibrated inferences which aids active learning and out-of df-iiiiiibuttion (OOD) detection. The same framework that robustly recognizes images despite noise can also detect noise to make an out-of-distribution detection.

---

Confidence and Uncertainty The existence of adversarial images ① heuristically decouples the probability of neural network predictions from confidence and uncertainty. A number of works including (29) and (30) use bayesian formulation to provide uncertainty. However, in downstream tasks like active learning and Out- -fuist--Ditriittiion (OOD) detection applications, existing state-of-the- art methods utilize softmax probability as confidences. This is because of the simplicity and ease of numerical computation of softmax. In active learning, uncertainty is quantified by the entropy (.I) least confidence (3.. maximum margin (32) of predicted logits, or through extracted features in BADGE (33) and BALD H. In OOD detection, (35) proposes Maximum Softmax Probability (MSP) as a baseline method by creating a threshold function on the softmax output. (36) proposes ODIN and improves on MSP by calibrating the network's 's softmax probability using temperature scaling 37 In this paper, show that the proposed introspective features are better calibrated than their forward counterparts. Hence existing methods in active learning and OOD detection have a superior performance when using H(-) make predictions.
Human Introspection We are unaware of any direct application that tests visual human introspec- tion. In its absence, we choose the application of hull-Reference Image Quality Assessment (FR-IQA) to connect machine vision with human vision. The goal in FR-IQA is to objectively estimate the subjective quality of an image. Humans are shown a pristine image along with a distorted image and asked to score the quality of the distorted image (38) This requires reflection on the part of the observers. We take an existing algorithm (39) and show that introspecting on top of this IQA technique brings its assessed scores closer to human scores.
6 Experiments
Across all applications except in Ablation studies in Table y we use a 3-layyrre MLP with sigmoid activations as H()). The structure is presented in Appendix C.I We first define robustness and calibration in the context of this paper.
Robustness In this paper, without loss of consistency with related works, we say that the network trained on distribution X' is robust if it correctly prediets shifted distribution X The difference in data distributions can be because of data acquisition setups, environmental conditions, distortions among others. We use CIFAR-10 for X' and two distortion datasets CIFAR-IOC (27) and CIFAR- 10-CURE (40) as X". Generalization is measured through performance accuracy.
Calibration Given a data distribution & E X. belonging to any of E M. a neural network provides two outputs- the decision ý and the confidence associated with ÿ. given by p. Let p be the true probability empirically estimated as p = P.. Vi E M1]. Then calibration is given by (37).
P(y و p) pp
(11)
Calibration measures the difference between the confidence levels and the prediction accuracy. To showcase calibration we use the metric of Expected Calibration Error (ECE) as described in (37).
The network predictions are placed in 10 separate bins based on their prediction confidences. Ideally, the accuracy equals the mid-point of confidence bins. The difference between accuracy and mid-point of bins, across bins is measured by ECE. Lower the ECE, better calibrated is the network.
TEFAR-10C consists of 950, 000 images whose purpose is to evaluate the robustness of networks trained on original CIFAR-10 trainset. CIFAR- perturbs the CIFAR-10 10 testset using 19 distortions in 5 progressive levels. Hence, there are 95 separate X' distributions to test on with each 2' consisting of 10000 images. Note that we are not using any distortions or data from CIFAR- 10C that they used to benchmark real-world recognition applications including Amazon Rekognition and Microsoft Azure. We use these distortions perturb the test set of CIFAR-10. There are 6 distortions, each with 5 progressive levels. Of these 6 distortions Salt and Pepper, Over Exposure, and Under Exposure noises are new compared to CIFAR-10C, We train four ResNet architectures ResNet- 18, 34, 50, and 101 4 All four ResNets are evaluated as sensing networks f().. The training procedure and hyperparameters are presented in Appendix C.T
Datasets and networks
validation split during training. The authors in (40) provide realistic distortions
Testing on CIFAR- testset The trained networks are tested on CIFAR-10 10 testset with accuracies 91. 91.02%, 93.01 %, 93.09%. and 93. 13% respectively. Next we extract rs on all training and testing images in CIFAR-1O. H(-) is trained using r, from the trainset using the same procedure as ʃ(-). When tested on T. of the testset, the accuracy for ResNees-18.34.50, 101 is 90.93%, 92.92%, 93.17%.
7

---

CIFAR-10C
Performance Accuracy (%)
Performance Accuracy (%)
75
75
24
Introspéction
Introspection
73
22
22
20
69
70
0.08
0.10
012
Expected Calibration error
Expected Calibration error
Legend
Feed-Forward Networks
ResNet-18
ResNet- 34
ResNet-50
ResNet-100 101
After Introspection
ResNet-18
ResNet-34
ResNet-50
ResNet- 101
Figure 4: Scatter plot with performance accuracy expected calibration error. Ideally, networks are
top left. Introspectivity increases performance accuracy while decreasing calibration error.
ResNet-18
brightness
FeedForward
Intropepectie
introspactive Gain
Experted Cilbrration
68
Accuracy
Challenge Levels
saturate
04
Errr
Fessiorward
mtrapspective
Expeced
brgithaa
contaat
delure Mar
tay
gaussoon biir
grorson noles
Sove, phir
Inputee satting
mation Mlrr
citirana
showe
Watteer
Speate polen
200m Miler
Challenge Levels
Figure 5: (a) ResNet-18 on CIFAR-10C. (b) Expected calibration error across 5 challenge levels in brightness and saturate distortions. Note that both these distortions do not affect the performance of the network and their feed-forward accuracy is high. The improvement in accuracy statistically insignificant. However, introspection decreases the ECE across challenge levels.
and 93.03%. Note that this is similar to the feed-forward results. The average ECE of all feed- forward and introspective networks 0.04, Hence, when the distribution is the same as training distribution there is no change in performance.
Testing on CIFAR-10C and CIFAR-1O-CURE The results of all networks averaged across distortions in both the datasets are shown in Fig. Note that in each case, there shift leftward and upward indicating that the performance improves while the calibration error decreases. In the larger CIFAR- 10C dataset, the introspective ResNet-18 performs similar ResNets-34 and 50 in terms of accuracy while beating them both in calibration. A more fine-gr fine-grained analysis is shown in Fig. for ResNet-18. The blue bars in Fig. (5) represent the feed-forward accuracy. The red bars are the introspective accuracy gains over the feed-forward accuracy. Among of the 19 distortions, the accuracy gains are over 5%. Appendix [.2.1] we see that the gains are higher when the distortions are higher. Introspection performs well on blur-like distortions while struggling with distortions that disrupt the lower level characteristics of the image like brightness, contrast, saturate. This can be attributed to the fact that rx are derived from the last layer of f()) and are missing low-level

---

statistics that are filtered out by network in the initial layers. However, in Fig. Eb), we show ECE for brightness and saturate distortions across all distortion levels higher the level, more is the distortion affecting X". It can be seen that while the ECE for feed-forward networks increases across levels, the ECE for introspective networks decrease. Hence, even when there are no accuracy gains to be had, introspection helps in calibration.
Plug-in results of Introspection Note that there are number of techniques proposed to alleviate a neural network's robustness challenges against distortions.
The authors in (28) show that finetuning VGG-16 us- ing blurry training images increases the performance of classification under blurry conditions. 44 propose utiliziig distorted virtual images to boost performance accuracy. The authors in (27 use adversarial images to augment the training data. All these works require knowledge of distortion or large amounts of new data during training. Our proposed method can infer intro- spectively on top of any existing f() enhanced using existing methods. In Table we show results of intro- spection plug-in approach on top of existing tech- niques. In all methods, introspection outperforms its counterpart. While introspecting on top of Augmix, trained on WideResNet (26) provides insignif- icant recognition accuracy gains, introspection reduces
Table 1: Introspecting on top of existing robustness techniques.
METHODS
ACCURACY
RESNET-18
FEED- FORWARD
67, 89%
INTROSPICTIVE
71.4%
DENDISING
PREn-FORWARD
65.02%
IXTROSPECTIVE
68.86%
ADVERSARIAL TRAIN AM
PRED- PORWARD
68.02%
INTRONPICTIVE
70.86%
SIMCLR (T)
FEED- FORWARD
28%
INTROSPECTIVE
73.32%
AUGMENT NOISE
PEED- FORWARD
76.86%
INTROSPECTIVE
77.98%
AEGMIX AC1
Fren-FORWARE
80. 89.85%
INTROSPOCTIVE
89.89%
ECE of Augmix network by 43.33%. In Appendix C.2 we show performance on top of (2) and (27) of 6.8%
on Level 5 distortions. In Appendix C.3
we analyze SimCLR and show that introspecting on the
self supervised features increases its CIFAR-IOC performance by about 6% on ResNet-101, A number of ablation studies including analysis of structure of H(--), loss functions, distortion levels on performance accuracy and ECE are shown in Appendix C.4. Moreover, we examine introspection when X' is domain shifted data from Office 4p) dataset in Appendix C.6.1
Downstream Applications We consider four downstream applications: Active Learning, Out-of- Distribution detection, Uncertainty estimation, and Image Quality Assessment (IQA) to demonstrate the validity of introspection. Similar to recognition experiments, in all considered applications there is a distributional difference between train and testset. We show results that conform to Eq. 10] We show statistically significant introspective IQA results in Appendix D.3 and Table 13 and uncertainty estimation results in Appendix D.4
Active Learning The goal in active learning is to decrease the test error in a model by choosing the best samples from large pool of unlabeled data to annotate and train the model. A number of strategics are proposed to query the best samples. A full review of active learning and query strategies are given in 43.. Existing active learning strategies define best samples to annotate those samples that the model is either most uncertain about. This uncertainty is quantified by either entropy (1), least confidence (II,, maximum margin (32), or through extracted features in BADGE (R)), and BALD (34) We show the results of ResNet-18 18 and 34 architecture in Table 2 Implementations of all query strategies in Table are
results. Note that the query strategies act on f(-) to sample images at every round. Instead of sampling on
are taken from the codebase of (83) and reported as
f()), all query strategies sample using H.(-) in the Introspective results. The training, testing, and all strategies are the same as Feed-Forward from (33) Doing so we find similar results as recognition on the original testset the active learning results are the same while there a gain across strategies on Gaussian noise testset from CIFAR- Note that the results shown are averaged over 20 rounds with query batch size of 1000 and initial random choice- which were kept same for f()) and H(-) of 100. Further details, plots, and variances are shown in Appendix D.I
fiitr---dittibution Detection The goal of Out-Of out-Of-Distriie Of-Distributton (OOD) detection is to detect those samples that are drawn from a distribution X' # X' given a fully trained f(-). A number of techniques are proposed to detect out-of out-of- distribbution samples. The authors in (35) propose Maximum Softmax Probability (MSP) as a baseline method by creating a threshold function on the softmax output. The authors in (36) propose ODIN and improved on MSP by calibrating the network's softmax probability using temperature scaling (7)) In this paper, we illustrate that applying existing methods when applied on H()), their detection performance is greater than if they were applied on the feed- ferw-forward
9

---

Table 2: Recognition accuracy of Active Learn- ing strategies.
Table 3: dusto--dist Detection of exist- ing techniques compared between feed-forward and introspective networks.
Methods
Architecture
Original Testset
Caussian Noise
8-18
R-34
R-18
R-34
Metool
DOD
FPR (95% at TPR)
Detection Eroor
AUROC
Dutasets
Entropy
Forl-Furwadd
0.365
0.358
0.244
0.219
Introspective
0.365
0.258
0.255
Foad-Forword
0.371
0.399
0.252
0.25
Introspective
0.373
0.362
0.264
0.36
Textures
58.74/19.66
18,04/7.49
88.56/97.79
MSP
SVHN
61,41/91.27
16:92/15.67
Margin Ki
Feod- Forward
0.369
0.251
0.253
Places365
58.0055.43
17.01/15.07
89, 39,399913
Introspective
0.381
0.373
0.265
0.263
LSUN-C
27.95/27.5
9.42/10.29
96,07/95.72
BALD 88
Feed- Fard-Forwadd
Introspective
0,396
0.395
0,315
0.368
0,273
0.26
0.263
0.253
ODIN RM
SVIN
Textures
66.81/48.52
52.3/9331
25.51/15.86
17,1776.12
83.52/9107
84,91/91.9
Places365
42.21/51.87
16.21/15771
91.06/9095
BADXF L3
Feed Beed-Fune Furward Introspcctive
0
0.39
0.37
0.37
0.365
0.25
0.247
0.260
LSUN-C
6.5W23.66
5.54/10.2
98.74/ 95.87
f(.). The code for OOD detection techniques are taken from (44) along with all hyperparameters and the training regimen for their reported DenseNet (5) architecture. The temperature scaling coefficient for ODIN is set to 1000. Note that we do not use temperature scaling on H. to illustrate the effectiveness of our method. We use three established metrics to evaluate OOD detection False Positive Rate (FPR) 95% True Positive Rate (TPR), Detection error, and AUROC. Ideally, AUROC values for a given method is high while the other two metrics are low. We use CIFAR-10 as our in-distribution dataset and use four OOD datasets SVHN (45), Describable Textures Dataset (46), Places 365 47), and LSUN 4% The results are presented in Table 3 Note that among the four datasets, textures and SVHN are more out-of-distribution from CIFAR-IO than the natural image datasets of Places365 and LSUN. The results of the introspective network is highest on Textures DTD dataset and gets progressively worse among the natural image datasets. Further analysis on networks and methods, along with their training regimen is provided in Appendix D.2
7 Discussion and Conclusion
Limitations and future work The paper illustrates the benefits of utilizing the change in model parameters as a measure of model introspection. In Section 3.2 we accelerate the time complexity to O()), However, the space complexity is still dependent on N. The paper uses MLP for H-)) and constructs Tx by vectorizing extracted gradients. For datasets with large N, usage of T. as a vector is prohibitive. Hence, required future work is to provide a method of combining all N gradients without vectorization. Also, our implementation uses serial gradient extraction across images. This is non-ideal since the available GPU resources are not fully utilized. A parallel implementation with per-sample gradient extraction 449 is a pertinent acceleration technique for the future.
Broader and Societal Impact systems of thought and reasoning in humans fast and instinctive 'system 1' that heuristically associates sensed patterns followed by more deliberate and slower 'system 2' that examines and analyzes the data in context of intrinsic biases. Our framework derives its intuition based on these two systems of reasoning. The introspective explanations can serve to examine the intrinsic notions and biases that network uses to categorize data. Note that the network H(-)) obtains its introspective answers through f-)). Hence, similar to the 'system 2' reasoning in humans, any internal bias present in f()) only gets strengthened in H-) through confirmation bias. The framework will benefit from human intervention between f(-) and H(-) in sensitive applications. One way would be to ask counterfactual questions by providing an established counterfactual and asking the network to reflect based on that. While the introspective framework will remain the same, the features will change.
In his seminal book in 2011 50). Daniel Kahneman outlines two
Conclusion We introduce the concept of introspection in neural networks as two separate stages in network's 's decision process the first is making quick assessment based on sensed patterns in data and the second is reflecting on that assessment based on all possible decisions that could have been taken and making final decision based on this reflection. We show that doing so increases the generalization performance of neural networks as measured against distributionally shifted data while reducing the calibration error of neural networks. Existing state-of-the-arr methods in downstream tasks like active learning and -istt--iistiitttion detection perform better an introspective setting compared to a feed-forwart setting especially when the distributional difference is high.
10

---

References
[1] Edwin G Boring, "A history of introspection."" Psychological bulletin, vol. 50, no. 3, pp. 169, 1953.
[2] John Locke, An essay concerning human understanding, Kay & Troutman, 1847.
[3] Karen Simonyan and Andrew Zisserman, "Very deep convolutional networks for large-scale image recognition," " in International Conference on Learning Representations, 2015.
[4] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun, "Deep residual learning for image recognition," " in Proceedings of the IEEF conference on computer vision and pattern recognition, 2016, pp. 770-778.
[5] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Weinberger, "Densely connected convolutional networks," in Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 4700-4708.
[6] Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra, "Grad-cam: Visual explanations from deep networks via gradient- based localization,"" in Proceedings of the IEEE international conference on computer vision, 2017, pp. 618-626.
[7] Johannes Schneider and Michalis Vlachos, "Reflective-net: Learning from explanations," arXiv preprint arXiv:2011.13385 2020.
[8] Ghassan AlRegib and Mohit Prabhushankar, "Explanatory paradigms in neural networks: Towards relevant and contextual explanations," IEEE Signal Processing Magazine, vol. 39, no.
4, pp. 59-72, 2022.
[9] Mohit Prabhushankar, Gukyeong Kwon, Dogancan Temel, and Ghassan AlRegib. "Contrastive explanations in neural networks," in 2020 IEEE International Conference on Image Processing (ICIP). IEEE, 2020, pp. 3289-3293,
[10] Mohit Prabhushankar and Ghassan AlRegib, "Extracting causal visual features for limited label classification," in 2021 IEEE International Conference Image Processing (ICIP). IEEE, 2021, pp. 3697-3701.
[11] Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy, "Explaining and harnessing adver- sarial examples," arXiv preprint 1rXiv:1412.6572, 2014.
[12] Jinsol Lee and Ghassan AlRegib, "Gradients as measure of uncertainty in neural networks," in 2020 IEEE International Conference on Image Processing (ICIP). IEFE, 2020, pp. 2416-2420.
[13] Gukyeong Kwon, Mohit Prabhushankar, Dogancan Temel, and Ghassan AlRegib, "Backpropa- gated gradient representations for anomaly detection." in European Conference on Computer Vision. Springer, 2020, pp. 206-226.
[14] Gukyeong Kwon, Mohit Prabhushankar, Dogancan Temel, and Ghassan AlRegib, "Novelty detection through mosel-based characterization of neural networks." in 2020 IEEE International Conference on Image Processing (ICIP). IEEE, 2020, pp. 3179-3183.
[15] Yutong Sun, Mohit Prabhushankar, and Ghassan AlRegib, "Implicit saliency in deep neural networks." in 2020 IEEE International Conference on Image Processing (ICIP). IEEE, 2020, pp. 2915-2919.
[16] Tommi S Jaakkola, David Haussler, et al., "Exploiting generative models in discriminative classifiers," Advances in newral information processing systems, pp. 487-493, 1999.
[17] David A Cohn, "Neural network exploration using optimal experiment design," 1994.
[18] Martin A Zinkevich, Alex Davies, and Dale Schuurmans, "Holographic feature representations of deep networks.." in UAI, 2017.

---

[19] Ting Chen, Simon Kormblith, Mohammad Norouzi, and Geoffryy Hinton, "A simple framework for contrastive learning of visual representations," in International conference on machine learning. PMLR, 2020, pp- 1597-1607.
[20] Aurick Zhou and Sergey Levine, "Amortized conditional normalized maximum likelihood: Reliable out of distribution uncertainty estimation," in International Conference on Machine Learning. PMLR, 2021, pp. 12803-12812.
[21] Koby Bibas, Yaniv Fogel, and Meir Feder, "Deep poml: Predictive normalized maximum likelihood for deep neural networks," arXiv preprint arXiv: /904.. 12286, 2019.
[22] Fangzhou Mu, Yingyu Liang, and Yin Li, "Gradients as features for deep representation learning," arXiv preprint arXiv:2004.05552) 2020.
[23] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei, "Imagenet: A large- scale hierarchical image database," in 2009 IEEE conference on computer vision and pattern recognition. leee, 2009, pp. 248-255.
[24] Yann LeCun, Léon Bottou, Yoshua Bengio, Patrick Haffner, et al., "Gradient- learning applied to document recognition," Proceedings of the IEEE, vol. 86, no. 11, pp. 2278-2324 1998.
[25] Volodymyr Kuleshov and Percy S Liang, "Calibrated structured prediction." Advances in Neural Information Processing Systems, vol. 28, pp. 3474-3482, 2015.
[26] Dan Hendrycks, Norman Mu, Ekin D Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshmi- narayanan, "Augmix: A simple data processing method to improve robustness and uncertainty," arXiv preprini arXiv: www../...2...... 2019.
[27] Dan Hendrycks and Thomas Dietterich, "Benchmarking neural network robustness to common corruptions and perturbations," arXiv preprint 1rXiv:1903.12266 2019.
[28] Igor Vasiljevic, Ayan Chakrabarti, and Gregory Shakhnarovich, "Examining the impact of blur on recognition by convolutional networks," arXiv preprini arXiv:1611.05700 2016.
[29] Murat Sensoy, Lance Kaplan, and Melih Kandemir, "Evidential deep learning to quantify classification uncertainty." arXiv preprint arXiv:1806.01768, 2018.
[30] David JC MacKay, "Probable networks and plausible predictions-a review of practical bayesian methods for supervised neural networks," Network: computation in neural systems, vol. 6, no. pp. 469, 1995.
[31] Dan Wang and Yi Shang, "A new active labeling method for deep learning," in 2014 Interna- tional joint conference neural networks (IJCNN). IEEE, 2014, pp. 112-119.
[32] Dan Roth and Kevin Small, "Margin- active learning for structured output spaces," in European Conference on Machine Learning. Springer, 2006, pp. 413-424.
[33] Jordan T Ash, Chicheng Zhang, Akshay Krishnamurthy, John Langford, and Alekh Agarwal, "Deep batch active learning by diverse, uncertain gradient lower bounds," arXiv preprint arXiv:1906.036/7 036771 2019.
[34] Yarin Gal, Riashat Islam, and Zoubin Ghahramani, "Deep bayesian active learning with image data," in International Conference on Machine Learning. PMLR, 2017, pp. 1183-1192.
[35] Dan Hendrycks and Kevin Gimpel, "A baseline for detecting misclassified and out-of- distribution examples in neural networks," arXiv preprint arXiv:1610.02136. 2016.
[36] Shiyu Liang, Yixuan Li, and Rayadurgam Srikant, "Enhancing the reliability of out-of- distribution image detection in neural networks." arXiv preprint 17Xiv:/706.026900 2017.
[37] Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger, "On calibration of modern neural networks," in International Conference Machine Learning. PMLR, 2017, pp. 1321-1330.
12

---

[38] Hamid R Sheikh, Muhammad F Sabir, and Alan C (Bovik, "A statistical evaluation of recent full reference image quality assessment algorithms," IEEE Transactions on image processing. vol. 15, no. 11, pp. 1440-3451, 2006.
[39] Dogancan Temel, Mohit Prabhushankar, and Ghassan AlRegib, "Unique: Unsupervised image quality estimation," " IEEE signal processing letters, vol. 23, no. 10, pp. 1414-1418, 2016.
[40] Dogancan Temel, Jinsol Lee, and Ghassan AlRegib, "Cure-or: Challenging unreal and real environments for object recognition," in 2018 17th IEEE International Conference on Machine Learning and Applications (ICMLA). IEEE, 2018, pp. 137-144.
[41] Dogancan Temel, Gukyeong Kwon, Mohit Prabhuhankar, and Ghassan AlRegib, "CURE-TSR: Challenging unreal and real environments for traffic sign recognition," Advances in Neural Information Processing Systems (NIPS) Machine Learning for Intelligent Transportations Systems Workshop, 2017.
[42] Kate Saenko, Brian Kulis, Mario Fritz,, and Trevor Darrell, "Adapting visual category models to new domains," in European conference on computer vision. Springer, 2010, pp. 213-226.
[43] Burr Settles, "Active learning literature survey," 2009.
[44] Jiefeng Chen, Yixuan Li, Xi Wu, Yingyu Liang, and Somesh Jha, "Robust out-of-distribution detection for neural networks," arXiv preprint arXiv:2003.09711 2020.
[45] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. "Reading digits in natural images with unsupervised feature learning," 2011.
[46] M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, and A. Vedaldi, "Describing textures in the wild." in Proceedings of the IEEE Conf. on Computer Vision and Pattern Recognition (CVPR). 2014.
[47] Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba, "Places: A 10 million image database for scene recognition," " IEEE transactions on pattern analysis and machine intelligence, vol. 40, no. 6, pp. 1452-1464, 2017.
[48] Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao, "Lsun: Construction of large-scale image dataset using deep learning with humans in the loop."
arXiv preprint 1rx.vv:506.03350 2015.
[49] Ian Goodfellow, "Efficient per-example gradient computations,"
arXiv preprint
11X1v:1510001799. 2015.
[50] Daniel Kahneman, Thinking, fast and slow, Macmillan, 2011.
[51] Wilhelm Max Wundt, Grundzüge der physiologischen Psychologie, vol. 1, W. Engelman, 1874.
[52] Eric Schwitzgehel, "Introspection," in The Stanford Encyclopedia of Philosophy, Edward N. Zalta, Ed. Metaphysics Research Lab, Stanford University, winter 2019 edition, 2019.
[53] Charles Sanders Peirce, Collected papers of charles sanders peirce, Harvard University Press, 1931.
[54] Gabriele Paul, "Approaches to abductive reasoning: an overview," Artificial intelligence review, vol. 7, no. 2. pp. 109-152, 1993.
[55] Mohit Prabhushankar and Ghassan AlRegib, "Contrastive reasoning in neural networks," arXiv preprint arXiv:2103.123325 2021.
[56] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei, "3d object representations for fine-grained categorization," in 4th International IEEE Workshop on 3D) Representation and Recognition (3dRR- /3), Sydney, Australia, 2013.
[57] Yazeed Alaudah, Motaz Alfarraj, and Ghassan AlRegib, "Structure label prediction using similarity-based retrieval and weakly supervised label mapping." Geaphysics, vol. 84, no. 1, pp. V67-V79, 2018.
13

---

[58] Muhammad A Shafiq, Mohit Prabhushankar, Haibin Di, and Ghassan AlRegib, "Towards understanding common features between natural and seismic images." in SEG Technical Program Expanded Abstracts 2018, pp- 2076-2080. Society of Exploration Geophysicists, 2018.
[59] Xingchao Peng, Ben Usman, Neela Kaushik, Judy Hoffman, Dequan Wang, and Kate Saenko, "Visda: The visual domain adaptation challenge." arXiv preprint arXiv:1110.06924, 2017.
[60] N. Ponomarenko, O. leremeiev, V. Lukin, K. Egiazarian, and M. Carli, "Modified image visual quality metrics for contrast change and mean shift accounting," " in Proc. CADSM, 2011, 1, pp. 305-311.
[61] Z. Wang and Q. Li, "Information content weighting for perceptual image quality assessment," IEEE Trans. Image Proc., vol. 20, no. 5, pp. 1185-1198, 2011.
[62] L. Zhang and H. Li, "SR-SIM: A fast and high performance iqa index based on spectral residual," in IEEE Int. Conf. Image Proc., 2012, pp. 1473-1476.
[63] L. Zhang, L. Zhang, X. Mou, D. Zhang, et al., "Fsim: a feature similarity index for image quality assessment," IEEE Trans. Image Proc., vol. 20, no. 8, pp.
2011.
[64] D. Temel and G. AlRegib, "PerSIM: Multi-resolution image quality assessment in the perceptu- ally uniform color domain," in IEEE Int. Conf. Image Proc., 2015, pp. 1682-1686.
[65] D. Temel and AlRegib, "CSV: Image quality assessment based on color, structure, and visual system," Sig. Proc.: Image Comm., vol. 48, pp. 92 103, 2016.
[66] D. Temel and G. AlRegib, "Perceptual image quality assessment through spectral analysis of error representations," Sig. Proc.: Image Comm., 2019.
[67] Mohit Prabhushankar, Dogancan Temel, and Ghassan AlRegib, "Generating adaptive and robust filter sets using unsupervised learning framework," " in 2017 IEEE International Conference on Image Processing (ICIP). IEEE, 2017, pp. 3041-3045.
[68] Mohit Prabhushankar, Dogancan Temel, and Ghassan AlRegib, "Ms-unique: Multi-model and sharpness-weighted unsupervised image quality estimation," Electronic Imaging, vol. 2017, no. 12. pp. 30-35, 2017.
[69] Dinesh Jayaraman, Anish Mittal, Anush K Moorthy, and Alan C Bovik, "Objective quality assessment of multiply distorted images," " in 2012 Conference record of the forty sixth asilomar conference on signals, systems and computers (ASILOMAR). IEEF, 2012, pp. 1693-1697.
[70] Nikolay Ponomarenko, Lina Jin, Oleg leremeiev, Vladimir Lukin, Karen Egiazarian, Jaakko Astola, Benoit Vozel, Kacem Chehdi, Marco Carli, Federica Battisti, et al., "Image database tid2013: Peculiarities, results and perspectives," " Signal processing: Image communication, vol. 30, pp. 57-77, 2015.
[71] Moloud Abdar, Farhad Pourpanah, Sadiq Hussain, Dana Rezazadegan, Li Liu, Mohammad Ghavamzadeh, Paul Fieguth, Xiaochun Cao, Abbas Khosravi, U Rajendra Acharya, et al., "A review of uncertainty quantification in deep learning: Techniques, applications and challenges." Information Fusion, vol. 76, pp. 243-297, 2021.
[72] Yarin Gal and Zoubin Ghahramani, "Dropout as bayesian approximation: Representing model uncertainty in deep learning," " in international conference on machine learning. PMLR, 2016, pp. 1050-1059.
[73] Durk P Kingma, Tim Salimans, and Max Welling, "Variational dropout and the local reparame- terization trick," Advances in neural information processing systems, vol. 28, 2015.
[74] Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell, "Tent: Fully test-time adaptation by entropy minimization." arXiv preprint arXiv:2006. 10726, 2020.
14

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]. See Section 0
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See
Section /
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]. See Appendix B
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [No]. The code will be made public upon acceptance.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]. See Section 2 and Appendix C.
(c) Did you report error bars (e-g., with respect the random seed after running experi- ments multiple times)? [Yes]. See Appendix D
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?[Yes)] in Appendix +
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you eite the creators? [Yes] See Appendix
(b) Did you mention the license of the assets? [Yes]. See Appendix
(c) Did you include any new assets either the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [No]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [No]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
15


Reviews:
[{"review_id": "vs38SslVtDY", "reviewer": "Reviewer_aTYk", "summary": "The authors propose a two-stage formulation for neural network decision making inspired by human introspection. The first stage is the regular discriminative feed-forward stage, where a deep network processes an input and generates the logit features. The introspective features for class I are then given as the change in the network parameters (up until the last layer) when a label of I is introduced for a sample x. The introspective features are then used to train a second introspective network, which tries to match the class predictions obtained from the introspective features to the ground truth predictions as well as the predictions from the first network. The authors provide theoretical derivations for these features, their approximation, sparsity, as well as how to extract them efficiently. The extensive experiments performed by the authors demonstrate that introspective learning improve neural network performance in terms of robustness, calibration, domain adaptation and active learning.", "questions": "1. In L43, the authors say \"Consider any dataset with N classes. A network trained on this dataset will have N possible introspective questions and answers.\" Shouldn't such a network have N^2 possible introspective questions (Why class i, rather than j? where i, j ∈ {1, N})\n\n2. L245 Typo: Robustness", "limitations": "The authors have adequately discussed the limitations of their work as well as its broader societal impact in Section 7.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n1. While the idea of logit gradients encoding class similarity information is not new, using this information to improve robustness and calibration in neural networks is quite novel and very well motivated theoretically and empirically. The biological analogue drawn to Kahneman's Fast/Slow learning to motivate introspective learning is relevant, but somewhat tenuous.\n\n2. The experiments conducted on robustness and calibration show that introspection helps improve both the out-of-distribution test accuracy for distorted images as well as the calibration of the neural network. The introspective neural networks tend to have very low expected calibration error in general.  \n\n3. The results in Table 1 show consistent improvement in performance on existing robustness techniques when using introspection on top. This motivates the usage of introspection as a general purpose technique to train more robust neural network models.\n\nWeakness \n\n1. This is not an issue with the work itself but against the nomenclature of the approach. The authors use a narrow definition of introspection pertinent specifically to discriminative learning : Why class X over class Y? In general, I am against the practice of co-adapting terms for ideas established in other fields (cognitive science in this case) and using them in an extremely narrow scope in machine learning. This proposed method covers one abductive reasoning based post-hoc 'introspection' approach, not all of introspection as it pertains to humans.  \n\n2. Could the low calibration error of the introspective model be affected by the fact that the introspective model is an MLP, whereas the models used for comparison are CNNs? [1] showed that vector/matrix scaling of CNN logits leads to better calibration error, and I wonder if the introspective model ends up doing something similar to reduce the calibration error.\n\nReferences\n\n[1] Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger, “On calibration of modern neural networks,” in International Conference on Machine Learning. PMLR, 2017, pp. 1321–1330\n\nDisclaimer\n\nI have not formally verified the proofs provided by the authors in Appendix", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "iGvu6oM2Xrk", "reviewer": "Reviewer_Qogq", "summary": "The paper introduces a novel 2-stage classification network where the 1st stage consists of a normal feed-forward pass through a network and the 2nd pass(Introspection) creates a set of features using gradients of the penultimate layer of feed-forward network and passes it through another MLP to predict the correct class. The set of features for each sample is created by concatenating the class-conditioned cross-entropy gradients at the penultimate layer for each class in the dataset. In other words, they calculate cross-entropy gradients for each sample by exhaustively setting every target class as the sample label. These gradients are then concatenated and passed through the MLP to generate the class label. They show that this approach gives them SOTA for Cifar10C and Cifar10 Cure datasets.", "questions": "SEE Strengths and Weaknesses section", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "STRENGTHS:\n1. The paper proposes a novel approach to rectify predictions for a classification network.\n2. They show theoretically that the features calculated using gradients could be useful.\n3. They show results on Cifar10C and Cifar10 Cure datasets.\n\nWEAKNESSES:\n1. This applicability of this work is currently limited to classification tasks that have very few target classes and a penultimate layer of modest dimension.\n2. Results are shown only for Cifar10 related dataset and raises concerns if they work for other datasets like Flowers 102,Caltech 101, Cub200 etc.\n3. Introspection stage is memory and compute-intensive. The authors should find ways to address this and other scaling issues\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "WGZE5JfjLeE", "reviewer": "Reviewer_W6Yu", "summary": "This paper proposes a novel idea by using a simple three-layer MLP, instead of fully connected layers, as the reflection stage to ask the network to reflect on its feed-forward decision by considering and evaluating all available choices/classes.", "questions": "1. How does the intuition behind getting Introspective Features different from traditional approaches with fully connected layers + softmax + cross-entropy loss? And why does the proposed approach works better?\n\n2. Please clarify what does $\\mathcal{H}(\\cot)$ function stand for in equation 7.\n\n3. What is the increased computational complexity of changing the last layer of the network into an Introspective Network?", "limitations": "N/A", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The proposed Introspection Network is very inspiring in transforming $N$ classes into $N$ possible introspective questions and answers. However, the robustness provided by the introspective networks (in Section 4) is not clearly presented.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "RxwdP3QnXfS", "reviewer": "Reviewer_8XSC", "summary": "The paper advocates the use of a two stage process for classification to increase robustness of NN. The first net is standard. The second network leverages gradients of the first network as inputs, i.e., they compute gradients for all possible outcomes towards the last layer (at least this is what seems to be the case according to Fig. 3).\n", "questions": "None", "limitations": "Limitations are ok.", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- The basic idea is not novel against the authors claim. See [1]. This work published in 2020 uses also a 2-stage process (one fast and one slow stage), it speaks of reflection, uses the Kahnemann reference, etc. However, aside from these similarities, the technical details seem to be quite different. Also the cited work does not rely on explanations of all classes. However, the authors should discuss this work.\n- Theory is appreciated, but I have doubt on correctness. In the proof supplementary material from equation (15) to (16) the authors transform using log(a0*b0+a1*b1)= log(a0+a1) +log(b0+b1). But this equation is not true.\n- The paper does not improve in general, i.e., for regular CIFAR-10, but only for CIFAR-10C. Thus, it seems valuable only in a few cases (e.g. the work above improves on CIFAR-10 on the test set using reflection)\n- The evaluation is just on one dataset and one architecture. This is insufficient and makes the generalization very questionable.\n\nDetailed comments:\n- Robsutness  -> Robustness\n[1] \"Reflective-Net: Learning from Explanations\" from 2020 (https://arxiv.org/abs/2011.13986)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept