Introspective Learning : A Two-Stage approach for Inference in Neural Networks
Authors: Mohit Prabhushankar
Ghassan AlRegib
Venue: ICLR 2022 Submitted (2022)
Keywords: Reasoning
Knowledge Representation
Robustness
Recognition

Abstract: In this paper, we advocate for two stages in a neural network's decision making process. The first is the existing feed-forward inference framework where patterns in given data are sensed and associated with previously learned patterns. The second stage is a slower reflection stage where we ask the network to reflect on its feed-forward decision by considering and evaluating all available choices. Together, we term the two stages as introspective learning. We use gradients of trained neural networks as a measurement of this reflection. We perceptually visualize the explanations from both stages to provide a visual grounding to introspection. For the application of recognition, we show that an introspective network is $4\%$ more robust and $42\%$ less prone to calibration errors when generalizing to noisy data. We also illustrate the value of introspective networks in downstream tasks that require generalizability and calibration including active learning and out-of-distribution detection. Finally, we ground the proposed machine introspection to human introspection in the application of image quality assessment.

Under review as a conference paper at ICLR 2022
INTROSPECTIVE LEARNING : A TWO-STAGE APPROACH FOR INFERENCE IN NEURAL NETWORKS
Anonymous authors Paper under double-blind review
ABSTRACT
In this paper, we advocate for two stages in a neural network's decision making process. The first is the existing feed-forward inference framework where patterns in given data are sensed and associated with previously learned patterns. The second stage is slower reflection stage where we ask the network to reflect its feed-forward decision by considering and evaluating all available choices. Together, we term the two stages introspective learning. We use gradients of trained neural networks as a measurement of this reflection. We perceptually visualize the post- hoc explanations from both stages provide a visual grounding introspection. For the application of recognition, we show that an introspective network is 4% more robust and 42% less prone to calibration errors when generalizing noisy data. We also illustrate the value of introspective networks in downstream tasks that require generalizability and calibration including active learning and out-of- distribution detection. Finally, we ground the proposed machine introspection to human introspection the application of image quality assessment.
INTRODUCTION
Introspection is the act of looking into one's own mind (Boring 1953). Classical introspection has its roots in philosophy. Locke (1847. ) the founder of empiricism, held that all human ideas come from experience. This experience is a result of both sensation and reflection. By sensation, one receives passive information using the sensory systems of sight, sound, and touch. Reflection is the objective observation of our own mental operations. Consider the task differentiating a spoonbill from a flamingo and crane. This task requires prior knowledge of some differentiating features between the birds. These features include the color and shape of the body, and beak of all birds. We first associate these features with our existing knowledge of birds and make coarse decision that the given bird is a spoonbill. This is the sensing stage. Reflection involves questioning the coarse decision and asking why the bird cannot be flamingo or crane. If the answers are satisfactory, then an introspective decision that the bird is indeed spoonbill is made. The observation of this reflection is introspection.
In this paper, we adopt this differentiation between sensing and reflection to advocate for two-stage neural network architectures for perception- based applications. We first ground introspection based on existing neural networks. The above-mentioner task of differentiating a spoonbill from flamingo and crane is provided in Fig. for neural networks A network f(-), is trained on distribution X' to classify data into N classes. The network learns notions about data samples when classifying them. These notions are stored as network weights W. Let feat be the logits projected before the final fully connected layer. We denote the final fully connected layer as fL. where is the layer number. fL-1 is then the layer before the final fully connected layer. Using the weight parameters Wz, the output of the network ò is given by,
Ifeat = fL-1(I), Vy E NX1
arg max (WT yfeat), YW2 E IxN fx))(x)) E Rdd-1x1
Hence, ÿ is the class in which the sensed features maximally correlate with the stored notion. This is the feed-forward prediction in Fig. [ Existing recognition architectures including VGG (Simonyan & Zisserman 2015). ResNet (He et al. 2016), and DenseNet (Huang et al. 2017) among others all sense and predict using Eq[[] In Fig. we depict our proposed introspective learning framework. An additional reflection stage extracts the introspective features as Not Detect features from Let

---

Under review as a conference paper at ICLR 2022
Pretrained Sensing Network
Feed-Forward Inference vs Introspective Inference
Prediction
Spoonbill
Spoonbill:
Feed-Forward
f(x)
Feed-Forward Spoonbill, 9
Pirk round body, Straight
Network Buisuas
and round body,
Detect
Streight neck
Spoonbil
Flamingo
Flamingo Pink and round bocy, dhapped
Reflection
Why Hamingo?
Network. !I())
Introspective
Not 5 S-shaped neck
Input
H(r))
Introspective
Grane
Image,
Spoonbill, ý
White/Slate-gre
body, long beak
why not Crane?
Not Detect White Foathers
Figure I: Toy example of feed-forward and introspection process. The visual post-hoc explanations in the sensing from Grad-CAM (Selvaraju al. 2017) while the explanations in the reflection stage are our own. The written text is for illustrative purpose only.
rI and r2 be the two introspective features. this case, Ti is the absence of the S-shaped neck in the spoonbill. And r2 is the lack of white feathers in the given input image. We post-hoc visual explanation to depict these feature Note that there can be N such features for a sensing network f(-) trained to differentiate between N classes. These features the final introspective feature Ta- Tz is characteristic of the input image J and is passed through an introspective network, H(-)), to obtain the introspective prediction ý. We term the combination of both f(-) and (-) as introspective learning.
then combined obtain
Note that the proposed reflection stage does not require a predefined knowledge base of introspective features, Rather, Ti, i E [1, N] in the reflection stage, is extracted using the feed-forward prediction y and the sensing network parameters. Hence, 72 are Not Detect features based on f(-)'s 's notion of classes. H(rx) predicts ğ explicitly based on the differences between f-')'s notion of classes. Not only should the network sense the ferd-forward patterns, must also satisfy H(-)'s N notions of differences. In this paper, we show that the inference process is more generalizable due to these N additional inferential constraints. Specifically, the introspective network is more robust to noise and less prone to calibration errors. The challenge is implicitly extract features that answer introspective questions without explicitly training on said questions as is the norm in Visual Question Answering applications (Antol et al. 2013). We show that gradients w.r.t network parameters store notions about the difference between classes and can be used as introspective features. We first describe the methodology of introspective feature extraction in Section 2 We then analyze H(-)) in Section. (3 We show that H-() as a simple multi-layer perceptron that introspects on ý is more generalizable for the application of recognition in Section5) We then illustrate the benefits of our two-stage architecture in other downstream tasks including out-of nuttof-distribution detection, active learning and image quality assessment in Section G
INTROSPECTIVE FEATURES
In this section, we describe introspective features and implicitly extract them using the sensing network. We then analyze their extraction procedure and provide methodology to accelerate it. Definition 2.1 (Introspection). Given a network f(-), a datum x, and the network's prediction f(x) = I, introspection in f()) is the measurement of change induced in the network parameters when a label yı introduced as the label for x. This measurement is the gradient induced by a loss function (yr, 1), w.E.t. the network parameters.
This definition for introspection in accordance with the sensing and reflection stages in Fig. D The network's 's prediction is the output of the sensing stage and the change induced by introspective label, yr, is the network reflecting on its decision á as opposed to yı. Combination of the two is introspection. Note that introspection can occur when is contrasted against any trained label y1, I E 1, N]. For instance, in Fig. the network is asked to reflect on its decision of spoonbill by considering other y, that x can take flamingo and crane.
poott-ooc explanations are justifications made by neural network after a decision has been made. They require human interpretation. Further details provided in Appendix
2

---

Under review as a conference paper at ICLR 2022
Reflection is the empirical risk that the network has predicted parameters, this risk is measured through some loss function J [yr. : 9). yr is one-hot vector with a one at the 4 location. The change that is induced in the network is given by the gradient of J (yr, û) w.r.t. the network parameters. In this paper, we introspect based on reflecting on all possible classes. For an N-class classifier, there are N possible introspective classes and hence N possible gradients each given by, rı = V 0), E [J. Here, rJ are the introspective features, Since we introspect based on classes, we measure the change in network weights in the final fully connected layer. Hence the introspective features are given by,
instead of yı- Given the network
V w. J(r ú), E [1, NJ,rr E P
where Wt are the network weights for the final fully connected layer. Note that the final fully connected layer from Eq. has dimensionality of Rd 1xN, For every z, Eq. 2 applied N times to obtain N separate rr- We first analyze these features before accelerating their extraction.
2.1
INTROSPECTIVE FEATURE ANALYSIS
Each ry is a d x N matrix. Expressing gradients in r' separately w.r.t. the different filters in WL. have row-wise concatenated of gradients given by,
Consider the extraction process in Eq. 2
VVW., 0): (ur,, ú); V
J 9)]
where each WL.5 E Rd1-1x1 and Pı E R For all data x E X the following lemma holds:
Lemma 1. Given a unique ordered pair (x, 1) and trained network f(),, the gradients for loss function (yr, 9) w.r.t. classes are pairwise orthogonal under the second-order Taylor series approximation, each class paired with the predicted class.
Proof. Provided in Appendix B.I
Lemma T states that backpropagating class yr does not provide any information to WL5: * and hence there is no need to use Jwz,,J(y D). * i as features when considering yr. In Appendix B.I we provide the complete proof when (1s, û) is the cross entropy loss. Vw D) for introspective class reduces to,
U
V 9) Wwyr + Vwlog
where Vi the logit associated with the predicted class. In Fig. we use network trained on MNIST (LeCun al.) 1998) dataset to simulate a well-trained network and we visualize the gradients from the final fully connected layer to demonstrate Eq.
Eq. motivates the generalizable nature of our introspective features. Consider some noise added to
Jr To change the prediction y. the noise must sufficiently decrease M/ from Eq. 4q. increase the closest logit value, y/, to change the prediction. However, by constraining our final prediction from
Fig. on N such Eq. 4 the noise needs to change the orthogonal relationship between N pairwise
logits. This motivates function H(-)) that is conditioned on N such pairwise logits. In Section. we empirically show the robustness of our feature set.
2.2 INTROSPECTIVE FEATURE EXTRACTION
From Lemma the introspective feature is only dependent on the predicted class ş and the introspec- tive class yı making their span orthogonal to all other gradients. Hence
rI Ww.,, (yr 9). [],rr 3d
(5)
Compare Eq. 5 aggainst the introspective feature from 2 Assuming that forward and backward passes through the final layer (-) are each of (1) time complexity, the feed-forward prediction for given is ((1) time complex. Given that f(-) is trained to classify between N classes, extracting
3

---

Under review as a conference paper at ICLR 2022
N introspective features require N backpropagations and hence is ((N) complex. Each ry in Eq. has a dimenisonality d N. Hence for features, the space complexity is O(N2 x dz-1). From Lemma [ the introspective feature is only dependent the predicted class ÿ and the introspective class yr making their span orthogonal to all other gradients. For introspective features in Eq. [ the space complexity of ry reduces from O N2) to (d x N). Note that the bottleneck in time complexity for N gradient extractions are the serial N backpropagations in Eq. 3 Building on Lemma !
we present the following theorem.
Theorem 1. Given unique ordered pair (x. and trained network f()), the gradients for loss finctionn J [yr, f(r)), 1 E [] w.r.. classes when are N orthogonal one-hot vectors is equivalent to when yr a vector of all ones, under the second-order Taylor series approximation.
Proof. Provided in Appendix B.2
The proof follows Lemma| Theorem ! states that backpropagating a vector of all ones (1v) is equivalent to backpropagating N one-hot vectors with ones at orthogonal positions. This reduces the time complexity from ((N) to constant ((1) since we only require single pass to backpropagate 1y. Hence, our introspective feature is given by,
(6) Note the LHS is now ry instead of ry from Fq.5) The final introspective feature is a matrix of the same size as Wr, extracted in ((1) with space complexity of O(d,-1 N). ra is vectorized and scaled between [-1, 1] before being used in Sections 5 and a introspective features.
Ta Vwz J(N, 9), E R Iy = 1 Nx1
3 INTROSPECTIVE NETWORK
Once re are extracted using Eq. 6 the introspective label ÿ from Fig. is given by z (rx)). In this section, we analyze H((-). From Fig. [1 f()) is any existing trained network used to obtain introspective features rz- It is trained to predict the ground truth y given any x. Let f()) be trained using the mean squared error loss function. Based on the assumption that H(rx) = [[y]s(x)) and hence expectation of y - (rxx) is O, the loss function can be decomposed as,
[I( - R) B[(((x) - H(r+))))) ([(H((r)) - -
Note that since the goal is to predict y given x, H(rx) ) = [(yls())) is a fair assumption to make. Substituting for f(x) in Eq. 7 and using variance decomposition of onto fax), we have,
[ y) = Var(ū) Var(H((r.)) B((H(())) - y)
This decomposition adopted from structured calibration techniques. A full derivation is presented in Kuleshov & Liang 2015). The first term Var(g) is the the variance in the prediction from f()). This term is the precision of f-) and is low for well trained network. The third term is the MSE function between the introspective network H(-) and the ground truth. It is minimized while training the H((-) network. The second term is the variance of the network H(-) given features rN, Note that minimizing Eq. can occur by maximizing Var(H((r.). However, Var(H((rz) is also part the third term which minimized. This prevents perpetual
of the bias-variance decomposition introspection, ie having multiple gradient extraction- based H(.) networks bootstrapped together, by creating a trade-off. We use a fisher vector interpretation to analyze Var(H((r.)). If H(-) W.. layer with parameters WH. the Var(H(r=z) term reduces to WWWW x Var(r.) where is the covariance matrix. Σ is a gaussian approximation for the shape of the manifold. Generalizing it to higher dimensional manifold and replacing E with F, we have, ((rr))
()) is a linear Tr(
Var(H((z2) Tr(r F
(9)
N
Var(H((z))
Σ j=1
T
F rj-
(10)
The RHS of Eq. is a sum of fisher vectors taken across all possible labels. We analyze two cases of usage of H(-) through Fisher Vectors When input X' is same the training distribution and when X' is from noisy distribution.
4

---

Under review as a conference paper at ICLR 2022
Estimation of X using H When sample X provided to network f(.) trained on N', all T5.3 # y in Eq. 10 tend to 0. The RHS reduces to Frr.="rg. "g- rg is a function of f(z) only and hence adds no new information to the framework. The results of H(-) remain the same as f()). In other words, given a trained ResNet-18 on CIFAR-IO, the results of feed-forward learning will be the same as introspective learning on CIFAR-10 testset.
Generalization to X' using H When a new samples x' & X' is provided to a network f(-) trained on X. fisher vector based projection across labels is more descriptive compared to a ferd-forward approach. The N gradients in Eq. 10 add new information based on how the network needs to change the manifold shape F to accomodate the introspective gradients. Hence, given a distorted version of CIFAR-10 testset, our proposed introspective learning generalizes with higher accuracy while providing calibrated outputs from Eq We show these two claims in Section []] The benefits of calibrated generalizability are further explored in downstream tasks like active learning and out-of-distribution detection in Section 6
RELATED WORKS
Two-stage Networks The usage of two-stage approaches to inference in neural networks is not new. The authors in Chen et al. (2020b) propose SimCLR, a self-supervised framework where multiple data augmentation strategies are used to contrastively train an overhead MLP. The MLP provides features which are stored as a dictionary. This feature dictionary is used as a look-up table for new test data. The classical object detection technique of R-CNN (Girshick et al. 2014) uses separate feature extraction and detection stages for inference. In all these works, the extracted features are feed-forward activations. In this paper, use gradients against all classes as features. Zhou & Levine (202T) and Bibas et al.] (2019) consider all classes in a conditional maximum likelihood estimate on test data retrain the model. These works differ from ours in our usage of the pairwise orthogonality of logits. We make use of this by having H--) as a classifier that explicitly learns the introspected pairwise relationships between classes.
Gradients-as- Featurs The gradients from a base network have been utilized in diverse applica- tions including post-hoc visual explanations (Selvaraju et al.) 2017 Prabhushankar et al.) [2020) adversarial attacks (Goodfellow et al., 2014]. and anomaly detection (Kwon et al. 2020 among others. In explanations, gradients are used to highlight features while in adversarial attacks, gradients characterize the required alterations to the features. Fisher Vectors use gradients of generative models
to characterize the change that data creates within features (Jaakkola et al.) 1999). A formulation sim- used to characterize the change in manifolds when new data is introduced to an already trained manifold. Our framework uses the intuition from Cohn (994)) characterize changes for datapoint that is perceived as new, due to it being assigned an introspective class that is different from its predicted class. In Zinkevich al. (0077) the authors view the network graph and intervene within obtain holographic features. Our introspective features are also holographic in the sense that they are not true. However, our features are dependent on the notions from the network itself and do not require engineered interventions that can become expensive with scale. Mu et al.) (020)) use gradients and activations together as features and note that the validity of gradients as features is in pretrained base networks rather than additional parameters from the two-stage networks. We demonstrate this well in Appendix C.4.
ilar to that of Fisher Kernels is used in Cohn (1994). Gradients of parameters
Augmentations and Robustness The considered r. features from Eq. 6 can be considered as feature augmentations. Augmentations, including SimCLR, Augmix (Hendrycks et al.) 2019). adversarial augmentation (Hendrycks & Dietterich) 2019) and noise augmentations (Vasiljevic et al. 2016) have shown to increase robustness of neural networks. We use introspection on top of non-augmented (Section (5) and augmented (Appendix C.2) networks and show that our proposed two-stage framework increases the robustness to create generalizable and calibrated inferences which aids active learning and diit-iiiittiitiionn (OOD) detection. The same framework that robustly recognizes images despite noise can also detect noise to make an out-of- detection.
Confidence and Uncertainty The existence of adversarial images (Goodfellow et al.) 2014) heuris- tically decouples the probability of neural network predictions from confidence and uncertainty. A number of works including [eensyy et al. (2018) and MMccKy (1995) use bayesian formulation to provide uncertainty. However, in downstream tasks like active learning and Of-is--Distiiuuttio
5

---

Under review as a conference paper at ICLR 2022
(OOD) detection applications, existing state-of-the-ait methods utilize softmax probability confi-
dences. This is because of the simplicity and ease of numerical computation of softmax. In active
learning, uncertainty is quantified by the entropy (Wang & Shang 2014). least confidence (Wang &
Shang 2014), or maximum margin (Roth & Small (006) of predicted logits, or through extracted
Teatures in BADGE et al. (2019), and BALD (Gal al. 2017). In OOD detection, [Hendrycks &
Cimpel 6 propose Maximum Softmax Probability (MSP) as a baseline method by creating threshold function on the softmax output. Liang et al. (2017) propose ODIN and improved MSP by calibrating the network's 's softmax probability using temperature scaling (Guo et al. 2017). In this paper, we show that the proposed introspective features are better calibrated than their feed- forward counterparts. Hence existing methods in active learning and OOD detection have superior performance when using H--- to make predictions.
Human Introspection There is no direct application that tests visual human introspection. In its absence, we choose the application of Full- Reference Image Quality Assessment (FR-IQA) to connect machine vision with human vision. The goal in FR-IQA is to objectively estimate the subjective quality of an image. Humans are shown a pristine image along with a distorted image and asked to score the quality of the distorted image (Sheikh et al., 2006). This requires reflection the part of the observers. We take an existing algorithm (Temel et al. (016) and show that introspecting on top of this IQA technique brings its assessed scores closer to human scores.
5 EXPERIMENTS
Across Sections 5 and 6 we use a 3-layered MLP with sigmoid activations as H. The structure is We first define generalization and calibration in the context of this paper.
presented in Appendix C.I
Generalization In this paper, without loss of consistency with related works, we say that the network trained on distribution X' is generalizable if it predicts correctly X". The difference in data distributions can be because of data acquisition setups, environmental conditions, distortions among others. We use CIFAR-10 for A' and distortion datasets CIFAR- 10C (Hendrycks & Dietterich 2019) and CIFAR- (Temel et al. 2018) as X". Generalization measured through performance accuracy.
shifted distribution
Calibration Given a data distribution x E X, belonging to any of E N1. a neural network provides two outputs- the decision ý and the confidence associated with ÿ. given by p. Let p be the true probability empirically estimated as = p., Vi E [1, M). Then calibration is given by (Guo
al. 2017).
Ply 0 p) =
(11)
Calibration measures the difference between the confidence levels and the prediction accuracy. To showcase calibration we use the metric of Expected Calibration Error (ECE) as described in (Guo et al.) 2017). The network predictions are placed in 10 separate bins based on their prediction confidences. Ideally, the accuracy equals the mid-point of confidence bins. The difference between accuracy and mid-point of bins, across bins is measured by ECE. Lower the ECF, better calibrated is the network.
Datasets and networks CIFAR- consists of 950, 000 images whose purpose is to evaluate the robustness of networks trained on original CIFAR- 10 trainset. CIFAR-10C perturbs the CIFAR-10 10 testset using 19 distortions in 5 progressive levels. Hence, there are 95 separate X' distributions to test on with each X' consisting of 10000 images. Note that we are not using any distortions or data from CIFAR-10C as a validation split during training. The authors in Temel et al. (2018) provide realistic distortions that they used to benchmark real-world recognition applications including Amazon Rekognition and Microsoft Azure. We use these distortions to perturb the test set of CIFAR- 10. There are 6 distortions, each with progressive levels. Of these 6 distortions Salt and Pepper, Over Exposure, and Under Exposure noises are new compared to CIFAR-10C. We train four ResNet architectures ResNet-18, 34, 50, and 101 He et al. (2016). All four ResNets are evaluated as sensing networks f-)). The training procedure and hyperparameters are presented in Appendix C.I
Testing on CIFAR-10 testset The trained networks are tested on CIFAR-10 testset with accuracies 91.02%. 93.01%, 93.09%, and 93.11% respectively. Next extract rz all training and testing images in CIFAR-10. H((-) is trained using r. from the trainset using the same procedure as f(-). When tested r. of the testset, the accuracy for Res4ets-18,,4,50, 101 is 90.93%, 92.92% 93.17%,

---

Under review as a conference paper at ICLR 2022
CIFAR-10C
CIFAR-1O-CURE
Performance Accuracy (%)
Performance Accuracy (%)
"
14
Introspéction
Introspection
73
12
71
89
70
0.06
0.08
0.10
0.2
0.1
ni6
0:18
0.10
0.12
0.14
0.16
018
Expected Calibration error
Expected Calibration error
Legend
Feed-Forward Networks After Introspection
ResNet-18
ResNet-34
ResNet-50
ResNet-108
ResNet-18
ResNet-34
ResNet-50
ResNet-101
Figure 2: Scatter plot with performance accuracy expected calibration emror. Ideally, networks
in top left. Introspectivity increases performance accuracy while decreasing calibration error.
ResNet-18
brightness
Expected Calbbratoon Firer
Reshet 18
Intropepective
introspective Gain
6419
Accuracy
67
Challenge Levels saturate
Expectd Collbration Enror
05
Drigthuet
Costraut
gourksar Slar
Blunsann nolee
Impudtee... moigieg
Hlur
piotlato
tstorara
Drow
Stattor
noiee
delocco
metion
Mbst
Challenge (b)
5 challenge levels in brightness and saturate distortions. Note that both these distortions do not affect the performance of the network and their feed-forward accuracy is high. The improvement in accuracy statistically insignificant. However, introspection decreases the ECE across challenge levels.
Figure 3: (a) ResNet-18 CIFAR-10C. (b) Expected calibration
and 93.03%. Note that this is similar to the feed-forward results. The average ECE of all feed- forward and introspective networks 0.04. Hence, when the test distribution is the same training distribution there is no change in performance.
Testing on CIFAR-10C and CIFAR-1O-CURS tortions in both the datasets are shown in Fig. 2 Note that in each case, there is a shift leftward and upward indicating that the performance improves while the calibration error decreases. In the larger CIFAR-10G dataset, the introspective ResNet-18 performs similar to ResNets-34 and 50 in terms of accuracy while beating them both calibration. more fine-grained analysis is shown in Fig. 3 for ResNet-18. The blue bars in Fig. 3 represent the feed-forward accuracy. The red bars are the introspective accuracy gains over the feed-forward accuracy. Among of the 19 distortions, the accuracy gains are over 5%. In Appendix C.2.1 and Fig. 6 we see that the gains are higher when the distortions are higher. Introspection performs well on blur-like distortions while struggling with distortions that disrupt the lower level characteristics of the image like brightness, contrast, and
The results of all networks averaged across dis-

---

Under review as a conference paper at ICLR 2022
Table 1: Recognition accuracy of Active Learn- ing strategies.
Table 2: OOD techniques applied on feed- forward and introspective networks when the data is under adversarial attack.
Methods
Architecture
Original Testset
Canssiio Noise
8-18
R-54
Methods
Dutaeets
OOD
(95% TPR)
Detection Emor
Ferdl-oorwedd
0.365
0.358
0.344
(Anack)
Entropy
Introspective
0.365
0.359
Feal-Forrsstt Sral-Forwwrrfatrosppaa
Confidence
Peed-Forward
Introspective
0.371
0.375
0.359
0.362
0.364
0.25
0.26
MSP
Textures
ISUN
59, 99/2999
98.63/87.2
46.71/28.J8
45 $5.9/7.9
46,44/75.881
A9%4%
Margin
Feed- Forward
Introspective
0:381
0.38
0,573
0.369
0.365
0.251
0.253
0.363
Places365
LSUN
99.65/9766
100/83,59
47 63.66/66.46
43.38/26.31
45,47778.4
25.08/79
BALD
Feed: Forward Innospective
0.396
0.393
0.368
0,375
0.273
0.26
0.253
0,263
ODIN
Textures
ISUN
96.8/90.42
99,95/2.06
44.77/31.11
47,7/3.48
53.88/73.22
37.5/99111
Places 365
99 99,97/82.5
47.12/26.88
32,69/78.88
BADGE
Introspontive
Fead-Forwar
0.388
0.37
0.25
0.247
1.260
LSUN
98.6/88.25
40.31/27.88
56.71 77.25
saturate. This can be attributed to the fact that T2 are derived from the last layer of f()) and are missing low-level statistics that are filtered out by network in the initial layers. However, in Fig. (b), we show ECE for brightness and saturate distortions across all 5 distortion levels higher the level, more is the distortion affecting X'. It can be seen that while the ECE for feed-forward networks increases across levels, the ECE for introspective networks decrease. Hence, even when there no accuracy gains to be had, introspection helps in calibration.
Plug-in results of Introspection Note that there are number of techniques proposed alleviate neural network's 's robustness challenges against distortions. The authors in (Vasiljevic et al. (0166) show that finetuning VGG-16 16 using blurry training images increases the performance of classification under blurry conditions. (Temel et al. 2017) propose utilizing distorted virtual images to boost performance accuracy. The authors (Hendrycks Dietterich] 2019) use adversarial images to augment the training data. All these works require knowledge of distortion large amounts of new data during training. Our proposed method can infer introspectively on top of any existing f() enhanced using existing methods. In Appendix C.2 we show performance on top of (Vasiljevic et al. 2016) and (Hendrycks & Dietterich] 2019) of 6.8% on Level 5 distortions. In Appendix C.3] we analyze SimCLR and show that introspecting on the self supervised features increases its COF performance by about 6% on ResNet-101. We introspect on top of Augmix (Hendrycks et al. 2019) and show that while recognition accuracy is the same, introspection reduces ECE of Augmix network by 43.33 43.33%. A number of ablation studies including analysis of structure of H. loss function, distortion levels on performance accuracy and ECE are shown in Appendix C.4 Moreover, we examine introspection when X' is domain shifted data from Office (Saenko et al. 2010) dataset in Appendix C.6
APPLICATIONS
In this section, we illustrate the advantages of introspective networks in two applications that are a function of both generalization and calibration active learning and out-of out-of-distribution detection. In both these applications, we show that the existing state-of-the-art methods perform better in their respective tasks and metrics if they were applied on H((-) than on f()). We then conclude by grounding the proposed introspection in neural nets with introspection in humans through IQA.
Active Learning The goal in active learning is to decrease the error in a model by choosing the best samples from a large pool of unlabeled data to annotate and train the model. A number of strategies are proposed to query the best samples. A full review of active learning and query strategies are given in [Settles] (2009). Existing active learning strategies define best samples to annotate as those samples that the model is most uncertain about. We use the strategies given in Section 4 to showcase the effectiveness of H(()). We show the results of ResNet- 18 and 34 architecture in Table Implementations of all query strategies in Table Jrr taken from the codebase Af|Ash et al. (2019) and reported as feed-forward results. Note that the query strategies act on f(-) to sample images every round. In the introspective results, all query strategies sample using H(-). The training and testing procedures strategies are the same feed-forward from Ash et al. (2019). Doing so we find similar results as recognition on the original testset the active learning results are the same while there is a gain across strategies on Gaussian noise testset from CIFAR-106 Note that the results shown
8

---

Under review as a conference paper at ICLR 2022
averaged over 20 rounds with query batch size of 1000 and initial random choice- which were kept same for f()) and H(-)- - of 100. Further details and plots are shown in Appendix C.8
outtof-distribution Detection The goal of Out-Or-Diitributioon (OOD) detection is detect those samples that are drawn from a distribution X' * X' given fully trained f(.). As mentioned in Section 4 we use MSP (Hendrycks & Gimpel| 2016) and ODIN (Liang et al. 2017) to illustrate the effectiveness of existing OOD methods when applied on MV() than if they were applied on the feed-forward f(-). The code for OOD detection techniques are taken from Chen et al. (2020a) along with all hyperparameters and the training regimen. The temperature scaling coefficient for ODIN is set to 1000. Note that we do not use additional temperature scaling on H--) to illustrate the effectiveness of our method. We use three established metrics to evaluate OOD detection - False Positive Rate (FPR) at 95% True Positive Rate (TPR), Detection error, and AUROC. Ideally, AUROC values for given method is high while the other two metrics are low. We use CIFAR-10 as in-distribution dataset and use four OOD datasets- - iSUN (Xiao et al., 2010), Describable Textures Dataset (Cimpoi al., 2014), Places 365 (Zhou et al. 2017), and LSUN (Yu et al. 2015). As in recognition and active learning, we consider two difficulty settings. The first is the vanilla case with the above-listed -ut-t--diittibuttonn datasets. The results are presented in Appendix (.9 and Table [I] The harder setting is when the datasets are all attacked with adversarial noise on top of being OOD. This setting is formalized in Chen et al. (2020a). We show these results Table In all cases in Table introspection outperforms its feed- counterpart. In the vanilla setting. introspection is better than its feed-forwe forward network in 15 of considered 24 metrics. This is inline with results from recognition and active learning where introspection generalizes better on X' compared to X' testsets.
Image Quality Assessment (IQA) TID 2013 (Ponomarenko et al. 2015) and MULTI-LIVE (Ja) yaraman et al. 2012) are two IQA datasets with 3000 and 225 distorted images respectively in and 4 progressively increasing levels of distortions. This setup is similar to CIFAR-1OC. The goal is to objectively assess the subjective quality of the distorted images given the pristine image. Inline with existing techniques (Temel et al., 2016), we use five metrics to measure the similarity between the algorithmically predicted qualities and human qualities outlier ratio (consistency), root mean square error (RMSE, accuracy). Pearson correlation (PRCC, linearity). Spearman correlation (SRCC, rank), and Kendall correlation (KRCC, rank). To do so, we use an existing IQA technique called UNIQUE (Temel et al.] (0166) and introspect on top of it. The details of UNIQUE as well related works and compared methods are presented in Appendix IC.10. Since UNIQUE is an autoencoder architecture and since the pristine image is available, introspection can occur based on features and on earlier layers. The exact procedure is given in Appendix (C.10 and is termed as
The results are shown in Table [2] The proposed framework acts as a plug-in on top of UNIQUE. For instance, UNIQUE is the third best performing method in MULTI dataset in terms of RMSE, PLCC, SRCC, and KRCC. However, Introspective-UNiCe improves the performance for these metrics by 1.315, 0.036, 0.020, and 0.023, respectively and achieves the best performance on all metrics.
DISCUSSION AND CONCLUSION
Limitations and future work The paper illustrates the benefits of utilizing the change in model parameters measure of model introspection. In Section 2.2 we analyze the time complexity and accelerate it to (1). However, the space complexity is still dependent on N. The paper uses an MLP for H((-) and constructs r- by vectorizing extracted gradients. Hence, taking the dimensionality of the final feature layer from Eq. [ the space complexity is O(N x d=-1). For large datasets with large N, usage of Tz as vector of concatenated gradients is prohibitive. Hence, required future work is to provide a method of combining all N gradients without vectorization. Also, our implementation uses serial gradient extraction across images. This is non-ideal since the available GPU resources are not fully utilized. A parallel implementation with per-sample gradient extraction (Goodfellow 2015) a pertinent acceleration technique for the future.
Conclusion We introduce the concept of introspection in neural networks as two separate stages in network's 's decision process the first is making quick assessment based on sensed patterns in data and the second is reflecting on that assessment based on all possible decisions that could have been taken and making final decision based on this reflection. We show that doing so increases the generalization performance of neural networks as measured against distributionally shifted data while reducing the calibration error of neural networks. Existing state-of-the-att art methods in downstream tasks like active learning and out-of--dstrrbbue detection perform better in an introspective setting compared to a feed-forward setting especially when the distributional difference is high.
9

---

Under review as a conference paper at ICLR 2022
8 ETHICS STATEMENT
The introspective explanations can serve to examine the intrinsic notions and biases that a network uses categorize data since H(-) obtains its introspective answers through f(.). However, any internal bias present in f(-) only gets strengthened in H(-) through confirmation bias. The framework will benefit from a human intervention between f(-) and H(-) in sensitive applications. One way would be to ask counterfactual questions by providing an established counterfactual and asking the network to reflect based on that. While the introspective framework will remain the same, the features will change. Such confirmation bias is also present in humans. In his seminal book in 20 Kahneman (201I) outlines two systems of thought and reasoning in humans fast and instinctive "system I' that heuristically associates sensed patterns followed by more deliberate and slower 'system 2' that examines and analyzes the data in context of intrinsic notions. Our framework derives its intuition based on these two systems of reasoning.
REPRODUCIBILITY STATEMENT
The paper uses publicly available datasets to showcase the results. Our introspective learning framework is built on top of existing deep learning apparatus including ResNet architec- tures (He al., 2016) (inbuilt PyTorch architectures), CIFAR-IOC data (Hendrycks & Dietterich
2019) (source code at https: / htte///ecor//255596656 calibration ECE and MCE metrics (Guo et al.] 2017) (source code at Matarkkss//N_cceeo
11335677.YLPTE--- -LpTE-IKhhh),
out-of-distribution detec- tion metrics, and codes for existing methods were adapted from (Chen et al.] (020a) (source code at https: /tit::.///ihhbb... com//b.com//// /f htt////ouuttoooo pood-detections learning methods and their codes were adapted from (Ash et al. 2019) (source code at https: //tps://gis /i thub. wwt.:/odaaasa///aaaaoooo 2017) (code used is at / own codes will be released upon acceptance. The exact training hyperparameters for f(-) and H(-). and all considered H(-) architectures are shown in Appendix C.1 Extensive ablation studies on H(--) shown C.4
https://gáthu/
active
Grad-CAM was adapted from (Selvaraju et al.,
com/adizyaco 4/taac44////e Gradddrr_a.eer _CAM CAM_pius palss_ppuss.._____ Our
REFERENCES
Yazeed Alaudah, Motaz Alfarraj, and Ghassan AlRegib. Structure label prediction using similarity- based retrieval and weakly supervised label mapping. Geophysics, 84(1):V6--V79, 2018
Stanislaw Antol. Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pp. 2425-2433, 2015.
Jordan T Ash, Chicheng Zhang, Akshay Krishnamurthy, John Langford, and Alekh Agarwal. Deep batch active learning by diverse, uncertain gradient lower bounds. arXiv preprint 2019.
Koby Bibas, Yaniv Fogel, and Meir Feder. Deep pnml: Predictive normalized maximum likelihood for deep neural networks. arXiv preprint arXiv:1904.12288. 2019.
Edwin G Boring. A history of introspection. Psychological bulletin, 50(3):169, 1953.
Jiefeng Chen, Yixuan Li, Xi Wu, Yingyu Liang, and Somesh Jha. Robust out-of-distribution detection for neural networks. arXiv preprint arXiv:2003.09711, 2020a.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597-1607. PMLR, 2020b.
M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, and A. Vedaldi. Describing textures in the wild. In Proceedings of the IEEE Conf. Computer Vision and Pattern Recognition (CVPR), 2014.
David A Cohn. Neural network exploration using optimal experiment design. 1994.
10

---

Under review as a conference paper at ICLR 2022
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 JEEE conference on computer vision and pattern recognition, pp. 248-255. Icee, 2009.
Yarin Gal, Riashat Islam, and Zoubin Ghahramani. Deep bayesian active learning with image data. In International Conference on Machine Learning, pp. 1 1183-1192 PMLR, 2017.
Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 580-587, 2014.
Ian Goodfellow. Efficient per-example gradient computations. arXiv preprint arXiv:1510001999 2015.
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6571 2014.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International Conference on Machine Learning, pp. 1321-1330 PMLR, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. arXiv preprint
2019.
Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distribution examples in neural networks. arXiv preprint arXiv:1610.02136, 2016.
Dan Hendrycks, Norman Mu, Ekin D Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshmi-
narayanan. Augmix: A simple data processing method to improve robustness and uncertainty. arXiv preprint arXiv: 2019.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4700-4708, 2017.
Tommi S Jaakkola, David Haussler, et al. Exploiting generative models in discriminative classifiers. Advances in neural information processing systems, pp. 487-493, 1999.
Dinesh Jayaraman, Anish Mittal, Anush K Moorthy, and Alan Bovik. Objective quality assessment of multiply distorted images. In 2012 Conference record of the forty sixth asilomar conference on signals, systems and computers (ASILOMAR), pp. 1693-1697. IEEE, 2012.
Daniel Kahneman. Thinking, fast and slow. Macmillan, 2011.
Jonathan Krause, Michael Stark, Jia Deng, and Fei-Fei. object representations for fine- graingra categorization. In 4th International IEEE Workshop on 3D Representation and Recognition (3dRR-13), Sydney, Australia, 2013.
Volodymyr Kuleshov and Percy S Liang. Calibrated structured prediction. Advances in Neural Information Processing Systems, 28:3474-3482, 2015.
Gukyeong Kwon, Mohit Prabhushankar, Dogancan Temel, and Ghassan AlRegib. Backpropagated gradient representations for anomaly detection. In European Conference on Computer Vision, pp. 206-226 Springer, 2020.
Yann LeCun, Léon Bottou, Yoshua Bengio, Patrick Haffner, al. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278-22) 222224444 1998.
Shiyu Liang, Yixuan Li, and Rayadurgam Srikant. Enhancing the reliability of duitr---ii image detection in neural networks. arXiv preprint arXiv: /7x.v:/706.02690, 2017.
John Locke. An essay concerning human understanding. Kay & Troutman, 1847.
11

---

Under review as a conference paper at ICLR 2022
David JC MacKay. Probable networks and plausible predictions-a review of practical bayesian methods for supervised neural networks. Nenwork: computation in neural systems, 6(3):469, 1995.
Fangzhou Mu, Yingyu Liang, and Yin Li. Gradients as features for deep representation learning. arXiv preprint wxXv::200400552, 2020.
Gabriele Paul. Approaches abductive reasoning: overview. Artificial intelligence review, 7(2): 109-152, 1993.
Charles Sanders Peirce. Collected papers of charles sanders peirce. Harvard University Press, 1931.
Xingchao Peng, Ben Usman, Neela Kaushik, Judy Hoffman, Dequan Wang, and Kate Saenko. Visda: The visual domain adaptation challenge. arXiv preprint
2017.
N. Ponomarenko, O. leremeiev, V. Lukin, K. Egiazarian, and M. Carli. Modified image visual quality metrics for contrast change and mean shift accounting, In Proc. CADSM, pp. 305-311, 2011.
Nikolay Ponomarenko, Lina Jin, Oleg leremeiev, Vladimir Lukin, Karen Egiazarian, Jaakko Astola, Benoit Vozel, Kacem Chehdi, Marco Carli, Federica Battisti, al. Image database tid2013: Peculiarities, results and perspectives. Signal processing: Image communication, 30:57-77, 2015.
Mohit Prabhushankar, Gukyeong Kwon, Dogancan Temel, and Ghassan AlRegib. Contrastive explanations in neural networks. In 2020 IEEE International Conference on Image Processing (ICIP), pp. 3289-3293, IEEE, 2020.
Dan Roth and Kevin Small. Margin-based active learning for structured output spaces. In European Conference on Machine Learning, pp. 413-424. Springer, 2006.
Kate Saenko, Brian Kulis, Mario Fritz, and Trevor Darrell. Adapting visual category models to new domains. In European conference computer vision, pp. 213-226. Springer, 2010.
Eric Schwitzgebel. Introspection. In Edward N. Zalta (ed.), The Stanford Encyclopedia of Philosophy. Metaphysics Research Lab, Stanford University, winter 2019 edition, 2019.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based local- ization. In Proceedings of the IEEE international conference on computer vision, pp. 61 618-626, 2017.
Murat Sensoy, Lance Kaplan, and Melih Kandemir. Evidential deep learning quantify classification uncertainty. arXiv preprint arXiv: 01768, 2018.
Burr Settles, Active learning literature survey. 2009.
Muhammad A Shafiq, Mohit Prabhushankar, Haibin Di, and Ghassan AlRegib. Towards understand- ing common features between natural and seismic images. In SEG Technical Program Expanded Abstracts 2018, pp. 2076-2080. Society of Exploration Geophysicists, 2018.
Hamid R Sheikh, Muhammad F Sabir, and Alan C Bovik. A statistical evaluation of recent full reference image quality assessment algorithms. IEEE Transactions on image processing. 15(11): 3440-3451, 2006.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. International Conference on Learning Representations, 2015.
D. Temel and AlRegib. PerSIM: Multi-resolution image quality assessment in the perceptually uniform color domain. In IEEE Int. Conf. Image Proc., pp. 1682-1686, 2015.
D. Temel and G. AlRegib. CSV: Image quality assessment based color, structure, and visual
system. Sig. Proc.: Image Comm., 48:92 103, 2016. ISSN 0923-5965. doi: http://dx.do
com/ 200.000.60000...0 URL http:/ www. www.ecccneieeeee....c...... www.:/eeete.eteceeeeeeeeeeeeeeet somnnscience/// aat.ccle/p.1/S022555555 63011111611111111111111111111111
D. Temel and G. AlRegib. Perceptual image quality assessment through spectral analysis of error representations. Sig. Proc.: Image Comm., 2019.
12

---

Under review as a conference paper at ICLR 2022
Dogancan Temel, Mohit Prabhushankar, and Ghassan AlRegib. Unique: Unsupervised image quality estimation. IEEE signal processing letters, 23(10):1414-447 2016.
Dogancan Temel, Gukyeong Kwon, Mohit Prabhuhankar, and Ghassan AlRegib. CURE-TSR: Chal- lenging unreal and real environments for traffic sign recognition. Advances Neural Information Processing Systems (NIPS) Machine Learning for Intelligent Trunsportations Systems Workshop, 2017.
Dogancan Temel, Jinsol Lee, and Ghassan AlRegib. Cure-or: Challenging unreal and real environ- ments for object recognition. In 2018 17th IEEE International Conference on Machine Learning and Applications (ICMLA), pp. 137-144. IEEE, 2018.
Igor Vasiljevic, Ayan Chakrabarti, and Gregory Shakhnarovich. Examining the impact of blur on recognition by convolutional networks. arXiv preprint arXiv: 2016.
Dan Wang and Yi Shang. A new active labeling method for deep learning. In 2014 International joint conference on neural networks (IJCNV), pp. 119. IEEE, 2014.
Z. Wang and Q. Li. Information content weighting for perceptual image quality assessment. IEEE Trans. Image Proc., 20(5):1185-11933 2011.
Wilhelm Max Wundt. Grundzüge der physiologischen Psychologie, volume 1. W. Engelman, 1874.
Jianxiong Xiao, James Hays, Krista Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In 2010 IEEE computer society conference on computer vision and pattern recognition, pp. 3485-3492. IEBE, 2010.
Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of large-scale image dataset using deep learning with humans in the loop. arXiv 2015.
preprint
L. Zhang and H. Li. SR-SIM: A fast and high performance iqa index based on spectral residual. In IEEE Int. Conf. Image Proc., pp. 1473-1476, 2012.
L. Zhang, L. Zhang, X. Mou, D. Zhang, et al. Fsim: feature similarity index for image quality assessment. IEEE Trans. Image Proc., 20(8):2378- 2386, 2011.
Aurick Zhou and Sergey Levine. Amortized conditional normalized maximum likelihood: Reliable out of distribution uncertainty estimation. In International Conference on Machine Learning, pp. 12803-12812. PMLR, 2021.
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE transactions on pattern analysis and machine intelligence, 145)):4452-1444 2017.
Martin Zinkevich, Alex Davies, and Dale Schuurmans. Holographic feature representations of deep networks. In UAI, 2017.
A APPENDIX INTROSPECTION, REASONING, AND EXPLANATIONS
Introspection was formalized by Wundi 884) as field in psychology to understand the concepts of memory, feeling, and volition (Schwitzgebel] 2019). The primary focus of introspection in reflecting on oneself through directed questions. While the directed questions are an open field of study in psychology, we use reasoning as a means of questions in this paper. Specifically, abductive reasoning. Abductive reasoning was introduced by the philosopher Charles Sanders Peirce (Peirce) 1931), who saw abduction as a reasoning process from effect to cause (Paul 1993). An abductive reasoning framework creates a hypothesis and tests its validity without considering the cause. From the perspective of introspection, a hypothesis can be considered as an answer to one of the three following questions: causal 'Why P?' question, counterfactual 'What if?' question, and contrastive 'Why P. rather than Q?' question. Here P is the prediction and Q is any contrast class. Both the causal and counterfactual questions require active interventions for answers. These questions
13

---

Under review as a conference paper at ICLR 2022
Bugatti Convertible from CARS dataset
Representative Bugatti Coupe image
Why Convertible, rother than Coupe?
Representative 1993 Volvo image
Why Bugatti, rather than Volvo?
Grad-CAM
Fault from LANDMASS dataset
Representative Salt Dome image
Fault, rather than Salt Dome
Representative Horizon image
Why Fault, rather than Horizon
Grad-CAM
Figure 4: Introspective feature visualizations. The images in the leftmost column the input æ. The representative images are for illustrative purposes and are not used to extract features.
try to assess the causality of some endogenous or exogenous variable. However, introspection the assessment of ones own notions rather than external variable. Hence, a contrastive question of the form 'Why P, rather than Q?' lends itself as the directed question for introspection. Here Q is the introspective class. It has the additional advantage that the network f()) serves the knowledge base of notions. All reflection images from [ Fig. and Fig. [ are contrastive. We describe the generation process of these post-hoc explanations.
Introspective Feature Visualization We modify Grad- Grad-CAM (Selvaraju et al. 2017)) to visualize ry from [ Grad-CAM visually justifies the decision made by f(-) by highlighting features that lead to ÿ. It does so by backpropagating the logit associated with the prediction, ÿ. The resulting gradients at every feature map are global average pooled and used as importance scores. The importance scores multiply the activations of the final convolutional layer and the resultant map the Grad-CAM visualization. Hence, gradients highlight the activation areas that maximally lead to the prediction ÿ. In Fig. given a spoonbill image x and ImageNet-prene perperraned (Deng et al. 2009) VGG-16 network, the sensing visualization shown is Grad-CAM. Grad-CAM indicates that the pink and round body, and straight beak are the reasons for the decision. Instead of backpropagating the ý logit, we backpropagate (yr, 9)) in the Grad-CAM framework. The gradients represent introspective features and are used as importance scores. It can be seen that they visually highlight the explanations to Why rather than yı In Fig. [ the network highlights the neck of the spoonbill to indicate that since an S-shaped neck not observed, x cannot be a flamingo. Similarly, the body of the spoonbill is highlighted when asked why is not a crane since cranes have white feathers while spoonbills pink. Two more examples are shown in Fig. In the first row, a VGG-16 architecture is trained on Stanford Cars dataset (Krause et al.) 2013). Given Bugatti convertible image. Grad-CAM highlights the bonnet as the classifying factor. An introspective question of why it cannot be a bugatti coupe is answered by highlighting the open top of the convertible. The entire car is highlighted to differentiate the bugatti convertible from Volvo. In the second row, we explore visual explanations in computed seismic images using LANDMASS dataset (Alaudah et al. 2018). ResNet- 18 architecture using the procedure from (Shatiq et al. (2018) is trained. The dataset has four geological features as classes faults, salt domes, horizons, and chaotic regions. Given a fault image Fig. Grad-CAM highlights the regions where the faults are clearly visible as fractures between rocks. However, these regions resemble salt domes as shown in the representative image. The introspective answer of why æ is not predicted salt dome tracks fault instead of highlighting general region that also resembles salt dome. Note that no representative images are required to obtain introspective visualizations. The gradients introspect based notions of classes in network parameters.
14

---

Under review as a conference paper at ICLR 2022
B APPENDIX: PROOFS
B.I PROOF FOR LEMMA
I
We start by assuming J() is cross-entropy loss. Jyr. E [1 can also be written as,
Jyr 0) 1/y + log
where I = E E
(12)
1=1
This definition used in PyTorch to implement cross entropy. Here we assume that the predicted logit, i.e, the argument of the max value in the logits ÿ is y/g. While training, V/ is the true label. In this paper, we backpropagate any trained class 1, as an introspective class. Hence, Eq. T2 can be rewritten
N
Jur 0) = -ys
-/s + log Σ v
where ý = f(x), E RN
(13)
==1
Approximating the exponent within the summation with its second order Taylor series expansion, we have,
N
y5 2
(y1 8) = yu + log 1 1 y
(14)
j=1
Note that for a well trained network S0, the logits of all but the predicted class are high. As noted before, the predicted logit y/g- Hence Σ/=
2 2
Substituting,
N
Jyr, 0) y- + log log(N) log
Σ fΙ) + log j=1
y
(15)
The quantity in Eq. 15 is differentiated, hence nulling the effect of constant log(N). For a well trained
network f(),, small changes in W do not adversely affect the sum of all logits Σ}= Vj- Hence approximating its gradient to 0 and discarding it, we can obtain W 0) as function of two logits, yr and V/ given by,
y 2
V J 0) -wer + V Vo
(16)
yr is a one-hot vector dimensionality x 1 while Vw a dr,1 x N matrix. The product extracts only the tth filter in the W matrix in gradient calculations. Following the above logic for yg. we have,
Vw J(r, V Vw.,g,(ygg.
(17)
where g(-) is some function of Øg. Hence the gradient Tı = w J 9) lies in the span of the filter gradients of W, and Wg. making r, orthogonal to all other filter gradient pairs. Hence proven.
We demonstrate this sparsity in Fig. 5 A two-layer CNN is trained on MNIST (LeCun et al. 1998)
dataset. MNIST is handwritten digits dataset consisting of 50, 000 training images and 10, 000 testing images among 10 classes. Our two-layer CNN recognizes these digits with an averaged test accuracy exceeding 99%. The final fully connected layer in this network has a size of 50 10. Hence, the dimensionality of each filter Wt.,, i E [1, 10] that corresponds to a class is 50 x 1. We provide an input image of number 5 to trained network shown in Fig. The network correctly identifies the image as 5. We then backpropagate the introspective class 0 using the cross entropy loss J(5, 0) with P and Q = This answers the question 'Why 5, rather than 0?' The gradient features in the final fully connected layer are the same dimensions as the final fully connected layer 50 x 10. This matrix displayed as a normalized image Fig. Yellow scales and blue is -1 while green is O. It can be seen that the only values present in the matrix negative at WL,0. in blue, and positive in W. in yellow. This validates Eq. 5 that for fully-trained network the only values, and hence the only information, required from WL for Q 0 is Vwz.o- We show the matrix Vwz when Q = 1, 2, 4, 5, 6. The difference among all matrices is the location of the negative values that exist at V V for different values of Q.
Hence, for N introspective features in the space complexity of ru which is a concatenation of N separate r;, reduces from Odd,-1 22) ) to O(d,-1 x N).
15

---

Under review as a conference paper at ICLR 2022
Why 5, rather than
Why 5, rather than
Why 5, rather than 2?
Why 5, rather than 4?
Input Image
Why 5, rather than
Why 5, rather than
Figure 5: For the input image on the left, the Vw, Jyr, 5) are shown on the right. Each image is visualization of the 50 10 gradient matrix. All images are sparse except in the prediction row 5 and introspective question row i.
B.2 PROOF FOR THEOREM
The proof for Theorem T follows from Lemma For any given data 2, there N possible introspections and hence possible reflections. The LHS in Eq. 17 is summed across N losses. Since Vj. N1 one-hot vectors, they are orthogonal and the first term in RHS is addition across j. The second term in RHS is independent of j. Representing this in equation form, we have,
N
Σ VwJ(u
--
Vw.yi + N x VWagg(ys).
(18)
The first term is added N times for N orthogonal yı- Hence, the first term reduces to sum of all gradients of jth filters when backpropagating Uj- Removing the summation and replacing yj = 1 or vector of all ones in the LHS, we still have the same RHS given by.
Jw
Σ
Vw.yi N V W.,, 9Wy))
(19)
Equating the LHS from Eq. 18 and Eq.
19
we have the proof.
B.3 TRADEOFF IN EQ. 8
Eq, 8 suggests a trade-off between minimizing ([(H(r)) which is the cost function for training (- and the variance of the network H(-)). Ideally, optimal point exists that optimally minimizes the cost function of H(-)) while maximizing its variance. This also prevents decomposing H(-) into (-) and H-) that further introspect H() In this paper, create single introspective network H()). Hence, we do not comment further on the practical nature of the trade-off or perpetual introspection. is currently beyond the scope of this work. In all experiments, train H((-) any other network feed-forward network by minimizing an empirical loss function given the ground truth.
B.4 FISHEE VECTOR INTERPRETATION
We make two claims before Eq. 10 both of which are well established. These include
Variance linear function For linear function y = x x + b, the variance of y is given by Var(W æ + b) W²Var((z) if Var(W) =
Variance of a linear function when W is estimated by gradient descent Ignoring the bias b, and taking y Wr x E- 1 (xW), we have Var(W -²Tr( - -z).
Both these results lead to Eq. 9 Since r E 3d the trace of the matrix given by Tr(r of projections on individual weight gradients given by ΣØ= of F-lry in the Fisher sense.
16

---

Under review as a conference paper at ICLR 2022
Table 3: Structure of H((-) and accuracies on CIFAR-IOC as reported the paper.
f()
Part 1: Structure of H(-) - All layers separated by sigmoid
Accuracy (%)
R-18,34
640 x 300 300 x 100 100 x
71.4, 73.36
R-50, 101
2560 x 300 300 x 100 100 x 10
75.2, 75.47
C APPENDIX: ADDITIONAL RESULTS
C.I STRUCTURE OF H(-) AND TRAINING DETAILS
In this section, we provide the structure of the proposed H(-) architecture. Note that, from Eq. D the feat in feed-forward learning are processed through a linear layer. We process the introspective features Tg through an MLP H(()). whose parameter structure is given in Table 3 Hence. follow the same workflow as feed-forward networks in introspective learning. The feed-forward features (x) are passed through the last linear layer in f(-) to obtain the prediction ý. The introspective features are passed through an MLP to obtain the prediction ÿ. The exact training procedure for H-(-) presented below.
Training f(-) and Hyperparameters We train four ResNet architectures: ResNet- 18, 34, 50, and 101 He et al. (2016). Note that we are not using any known techniques that promote either gener- alization (training on noisy data (Vasiljevic et al., (0166) or calibration (Temperature scaling (Guo) jet al. (2017)) The networks are trained from scratch on CIFAR- 10 dataset which consists of 50000 training images with 10 classes. The networks
trained for 200 epochs using SGD optimizer with momentum = 0.9 and weight decay- = 5e 4. The learning rate starts at 0.1 and changed as 0.02, 0.004, 0.0008 after epochs 60, 120, and 160 respectively. PyTorch in-built Random Horizontal Flip and standard CIFAR-10 normalization is used as preprocessing transforms.
Training H(()) The structures of all MLPs are shown in Table ResNet- 18,34 trained on CIFAR-10
provide Px of dimensionality 640 x 1. This is fed into H((-) which is trained to produce a 10 x output.
Note that rz from ResNet-50,100 are of dimensionality 2560 x due larger dimension of f fr-1()-
All MLPs are trained similar to f(-)- for 200 epochs, SGD optimizer, momentum = 0.9, weight decay 33 learning rates of 0.1, 0.02, 0.004, 0.0008 in epochs 1 60, 120, 121 160, 161 200 respectively. For the larger 5-layered GesNNet-00,101 networks in Table S dropout with 0.1 is used and the weight decay reduced to 5e
-3
C.2 INTROSPECTIVE ACCURACY GAIN AND CALIBRATION ERROR STUDIES
In this section, we present additional recognition and calibration results. In Fig. Bh), showed distortion-wis wisee accuracy and the introspective gain for ResNet-18, 18. In this section, we present level- wise and network-wise accuracies for all four considered ResNet architectures. We show that an introspective ResNet- 18 matches Feed-Forward ResNet-50 in terms of recognition performance. We then compare the results ResNet- 18 against existing techniques that promote robustness. We show that introspection is a plug-in approach that acts on top of existing methods and provides gain. We do the same for calibration experiments on CIFAR- 10C where we provide level-wise distortion-wise graphs for Expected Calibration Error (ECE) similar to Fig. 3)).
C.2.1 LEVEL-WISE RECOGNITION ON CIFAR-10C
In 6p). the introspective performance gains for the four networks are categorized based on the distortion levels. All 19 categories of distortion on CIFAR-IOC are averaged for each level and their respective feed-forward accuracy and introspective gains are shown. Note that the levels are progressively more distorted. Hence, level distribution 2' similar to the training distribution X' when compared to level 5 distributions. As the distortion level increases, the introspective gains also increase. This is similar to the results from Section 6 In both active learning and OOD applications as X" deviates from X, introspection performs better. In Fig. 6a). we show the distortion- wise and
17

---

Under review as a conference paper at ICLR 2022
Comparing Introspective Gain across networks
Level Intrespective Accuracy gein acress networks
Average Accuracy
perspettvv
Introspactiva Gain
74-
Accuracy
Accuracy
Ressvvet
ResNett-4
ResVet-50
ResNet-100 101
ResNet-12
ResNet-34
ResNet-50
Res Nes 101
ResNet-18
ResNet-34
ResNerc50
Nes-Nt-101
ResNet-18
ResNet-34
ResNet-50
RessNet-10 101
ReaNer-
ReshNet-3
ResNert-5
RosNat-18
Rashet-344
ResNat-50 5D
Rashet-101
Distortion Levels
(a)
(b)
Figure 6: Introspective performance gains over Feed-Forward networks of a) ResNets- 18,3tss-18,,4,,,,11111 b) Level-wise averaged results across 18,35,tt11,,,,,,100
brightness
elastic
Chalneos Leats
Chalennr Level
Challeze Levic
gaussian blur
gai.seian noise
spatter
Chalenas Londs
glass blur
impulse noise
Chalana Lendb
jPSy compression
blur
Chalkeas Lowbl
200m blur
10
ChulkreeLodh shot_ nolse
Dillorocivees speckee_nee nolse
pixelate
Saturate
ACDISSY
Fead-Forward Introspective
Ohplange
Chollnce Leantt
Chelecge Leve
Figure Introspective performance gains over Feed-Forward Resnet-18 across distortions and levels
level-wise increase for each network. Note that, an Introspective ResNet-18 performs similarly to Feed-Forward ResNet-50.
C.2.2 DISTORTION- AND LEVEL-WISE RECOGNITION CIFAR-10C
In Fig. 7 the introspective accuracy performance for Resnet-18 18 across 19 distortions and distortion levels is shown. Note that CIFAR-106 consists of 950,000 test images. The 4% increase in performance translates to around 35,000 more images correctly classified over its fcrd-forward counterpart. These gains are especially visible among Level 5 distortions.
C.2.3 INTROSPECTION AS A PLUG-IN ON TOP OF EXISTING TECHNIQUES
Several techniques exist that boost the robustness of neural networks to distortions. These include training with noisy images (Vasiljevic et al. 2016). training with adversarial images (Hendrycks) & Dietterich/ 2019), and self-supervised methods like SimCLR (Chen et al. (020b) that train by augmenting distortions. Another commonly used technique is to pre-process the noisy images to denoise them. All these techniques can be used to train f()). Our proposed framework sits top of
18

---

Under review as a conference paper at ICLR 2022
Table 4: Introspecting top of existing robustness techniques.
Methods
Accuracy
ResNet-18
67.89%
Denoising
05.0
Adversurial Train [lendrvcks & Dietterich 2019
68.02%
SimCLR Chen al. 20/200
70.28%
Augment Noise Vesi of 2016
76. 86%
Augmix (Hendrycks al
4019
89.85%
ResNet-18 18 + Introspection
71.4%
Denoising Introspection
68.86%
Adversarial + Introspection
70.86%
SimCLR Introspection
73.32%
Augment Noise Introspection
77.98%
Augmix + Introspection
89.89% (ECC 43.33% 4)
any ʃ()). Hence, it be used plug-in network. These results are shown in Table Denoising 19 distortions is not a viable strategy assuming that the characteristics of the distortions are unknown. We use Non-Local Means denoising and the results obtained are lower than the feed-forward accuracy by almost 3%. However, introspecting on this model increases the results by 3.84%. We create untargeted adversarial images using I-FGSM attack with a = 0.01 and use them to train ResNet- 18 architecture. In our experiments this did not increase the feed-forwar this network provides a gain of 2.84%. SimCLR Chen et al. (2020b) and introspection on SimCLR is discussed in Section 2016), we augment the training data of CIFAR-10 with six distortions gaussian blur, salt and pepper, gaussian noise, overexposure, motion blur, and underexposure to train ResNet- 18 network f'(-). We use the noise characteristics provided by (Temel et al., (018)) to randomly distort 500 CIFAR-10 training images by each of the six distortions. The original training set is augmented with the noisy data and trained. The results of the feed-forward f'(-) show substantial increase in performance 76.86%. This is about 9% increase from the original architecture. We show that introspecting on (-) provides a further gain in accuracy of 1.12%. Note that to train H(-), we do not use the augmented data. We only use the original CIFAR- 10 undistorted training set. The gain obtained by introspecting on only the undistorted data, even though f-() contains knowledge of the distorted data. Hence, introspection is a plug-in approach that works on top of any network f()) or enhanced network (((). Augmix (Hendrycks et al. 2019) is currently the best performing technique on CIFAR-10C It creates multiple chains of augmentations to train the base WideResNet network. On CIFAR- 10C, f'()) obtains 89.85% recognition accuracy. We use f'(-) as base sensing model and train an introspective MLP on f'(.). Note that we do not use any augmentations for training H(--) Doing so, obtain statistically similar accuracy performance of 89.89% However, the expected calibration error of the feed-forme forwfrr f'(+) model decreases by 43.33% after introspection. Hence, when there is no accuracy gains be had, introspection provides calibrated models.
accuracy. Introspecting on
In the final experimental setup of augmenting noise (Vasiljevic et al.,
Table 5: Expected Calibration Error and Maximum Calibrated Error for Feed-Forward vs Introspective Networks.
Architoctures
ResNet-18
ReaNot-34
ReaNet-50
ReaNot- 101
f(-)
0.14
0.18
0.13
0.16
ECE (L)
R((-)
0.07
0.09
0.06
0.1
MCE (4)
f(-)
0.27
0.34
0.27
0.32
P6(-)
0.23
0.24
0.25
0.23
F())
0.046
0.041
0.042
Brier Loss (1)
H((-)
0.054
0.053
0.053
0.052
C.2.4 EXPECTED CALIBRATION ERROR (ECE)
In Fig. 3D), we show ECE for two distortion types brightness and saturation across 5 distortion levels. In Fig. & we show results across five distortion levels for the first 12 distortions. The blue plot is the Fred-Forwann ECE while the lower orange plot is its introspective counterpart. Apart from Level 5 contrast, intrsopective ResNet-18 is more calibrated than its feed-forward counterpart. This in addition to the performance gains. The trend remains the same in the remaining distortions and among all considered networks. We average out ECE across 19 distortions and 5 challenge
19

---

Under review as a conference paper at ICLR 2022
brightness
contrast
defocus blur
elastic transform
Sepicter Calb att Eror
Infrorppctive
Hispopecte
Propppertio
Challenee Lezees
Challenge Levels
Challange Levels
Challenge Levels
fag
frost
gaussian blur
gaussian noise
Donc Ertor
I9Po01podivv
Itrogpeths
iropppettic
Horospochhi
Ciltratin
glass_ blur
impulse impulse_noise
jpeg_ compression
motion blur
Dxpected albratton Eneer
Nee ti vanl
teei Haraare
Expreted Callrroo Chlibration Enter
Chalenge
Challengs Level
Challange Lavels
Chalenge Levels
Figure 8: ECE vs distortion levels across 12 separate distortions from CIFAR-10C for ResNet-18 18.
levels and provide ECE results for ResNets- 34, 50, 101 in Table Lower the error, better is the architecture. The proposed introspective framework decreases the ECE of its feed-forward backbone by approximately 42%. An additional metric called Maximum Calibration Error (MCE) is also used for comparison. While ECE averages out the calibration difference in all bins (From Section 6 MCE takes the maximum error among all bins Guo et al. (2017). The introspective networks outperform their forward backbones among all architectures when compared using ECE and MCE.
Table 6: SimCLR and its supervised and introspective variations tested on CIFAR- 10C.
Methods
RosNet-18
ResNet-34
ResNet-50
ResNet-101
SimCLR Chen al.
2020b
70.28%
69.3%
67.32% 70.37%
61.68%
SinCLR
72.79%
72.54%
70.89%
SimCI.A Introspcctive (Proposed)
73.06%
71.28%
71.76%
C.3 SIMCLR AND INTROSPECTION
SimCLR Chen et al. 2020b) is self-supervised contrastive learning framework that is robust to noise distortions. The algorithm involves creating augmentations of cxisting data including blur, noise, rotations, and jitters. The network is made to contrast between all the augmentations of the image and other images in the batch. A separate network head g(-) is placed on top of the network to extract features and inference is made by creating similarity matrix
feature bank. Note that 9(-) is a simple MLP. Our proposed framework is similar to SimCLR in that we extract features and use an MLP H-) to infer from these features. In Table 4 we show the results of Introspecting ResNets against SimCLR. However, this comparison is unfair since the features in SimCLR are trained in a self-supervised fashion. In this section, we train SimCLR for ResNets-188 18, 34. 50. 101 and train a new MLP g(-), not for extracting features, but to classify images. In other words, in Chen et al. (2020b). the authors create g()) to 512 x 128- 128 x 10 that is trained to classify images. We then introspect on this g(-) to obtain rz- Hence, our extracted features are a result of introspecting on self-supervision. Note that g(-) is now a fully supervised network. We pass CIFAR-IOC through g(-) and name it SimCLR-ML. in Table 6 is unsurprising that the sully-supervised SimCLR M-M beats the self-supervised SimCLR across all four ResNets. The introspective network is called SimCLR- Introsssrseeectiee in Table 6 Note that there is less than 1% recognition performance increase across networks compared to SimCLR-ME Hence, the performance gains for introspecting on SiMCLR-MMPP is not as high base ResNet
512 x 128 layer that extracts features. We train a network of the form
20

---

Under review as a conference paper at ICLR 2022
Table 7: Introspective Learning accuracies when 7z is extracted with different loss functions for 18 on CIFAR- 1OC.
Feal-Foresrd
MSE-M
CF
L11M
Smooth
Smooth f.1-M
Mongin
67.80% 71.4%
69.47%
70.76% 70.12%
70.72%
70. 70.42%
70.63%
70. 10.93%
70. 91%
architectures from Table 4 One hypothesis for this marginal increase is that the notions created within SimCLR-MLP are predominantly from the supervviee features in SimCLR. These may not be amenable for the current framework of introspection that learns to contrast between classes and not between features within-classes.
C.4 ABLATION STUDIES
The feature generation process Section 2 is dependent on the loss function ( y). In this section, we analyze the performance of our framework for commonly used loss functions and show that the introspective network outperforms its feed-forward counterpart under any choice of J(. y). We also ascertain the effect of the size of the parameter set in H((-) on performance accuracy.
C.4.1 EFFECT OF LOSS FUNCTIONS
We extract Tn using loss functions and report the final distortion-wist level-wise averaged results Table We do so for ResNet-18 18 and for the architecture of H-) shown in Table 3 The following loss functions are compared: CE is Cross Entropy, MSE is Mean Squared Error, distance, Smooth L1 is the leaky extension of Manhattan distance, BCE is Binary Cross Entropy, and NLL is Negative Log Likelihood. Notice that the performance of Ta extracted using all loss functions exceed that of the feed-forward performance. The shown results of MSE, LI-M and Smooth L1-M are obtained by backpropagating a 1y from Theorem I vector multiplied by the average of all maximum logits M, in the training dataset. We use M instead of because be as confidant of the introspective label z/r as it is with the prediction label ÿ. Note that the results in Table are for CIFAR-IOC. MSE-M outperforms NLL loss by 0.37% in average accuracy and is used in our experiments.
Manhattan
want the network to
C.4.2 EFFECT OF H(-))
We conduct ablation studies to empirically show the following 1) the design of H(--) does not significantly vary the introspective results, 2) the extra parameters in H(-) are not the cause of increased performance accuracy.
How does changing the structure of H()) change the performance? We vary the architecture of H(()) from a single linear layer to 4 layers in the first half of Table for ResNet-18. The results the first three cases similar. A four layered network performs worse than f()). However, changing the weight decay from 5e to 5e during training increases the results to above 70% but does not beat the smaller networks. For ResNet-18 architecture, the highest results are obtained when H(-) is a 2-layered architecture but for the sake of uniformity, we use the results from a 3-layered network across all ResNet architectures.
Are the extra parameters in H(-) the only cause for increase in performance accuracy? We show an ablations study of the effect of structure of H(-) and ʃ(-) on the introspective and feed- forward results in the second half of Table CIFAR- 10-C dataset. The results are divided into four sections. In the first section, we show the performance of the original feed-forward network f()), the performance when the final layer, (.).) is retrained using features Vfeat from Eq, [ and the introspective network when H(-)) is single layer. The second section shows the results when the features Vfeat are used to train a three layered network (t).). and the introspective network is also three layered. Finally, in section 3, we try to equate the number of parameters for f-)-) and H(-)) Note that in all cases, f.) and H1(-) are trained in the same manner as detailed in Section C. H(-)) beats the performance of f-) among all ablation studies. Finally, similar to SimCLR, we forego using an MLP and use 10-Nearest Neighbors on yeat (64 x 1) and ra (640 x 1) for predictions. Both results are worse-off than their MI. results but r. outperforms Ifeat-
21

---

Under review as a conference paper at ICLR 2022
Table 8: Ablation studies for H(-) on CIFAR-IOC.
Part 1: Varying the number of layers
Ferd-Forward 64 x
67.89%
640 10
71%
R-18
640
100 100
10
71.57 %
640 300
300 100 100 10
71.4%
640 400 400
100
100 x 10
66.1%
Ford-Forward 64 x 10
71.8%
R-50
2560 x 300 300 100 100 10
75.2% %
2560
1000
1000 500 500 300 300 x 100 100 10
73%
Part 2: the performance increase only because of large H((-)?
Feed-Forward
67.89%
f-) Layer: 64 x 10
67.86%
H(-)) Layer 640 10
71 %
(.).) Layers 64 30 30 x 20
20 x 10
63.61% 61%
fz(-) 3 Layers 64 512 512 x 256
256 x 10
64.78%
R-18
H(-) 3 Layers: 640 300
300
100
100 10
71.4 %
f-))), 6200 parameters 64 50 50 40 40
20 10
66.85%
H(.). 6400 parameters 640 x
%
Prediction on Myeat using 10-NN (No fx.())
66.31% 11%
Prediction on using 10-NN (No H(-))
68.76 %
Part 3 VGG-16
Feed-Forward
68.96%
VGG-16
f(-) 512 x 1024 1024 x 256 256 x 10
62,43%
H(--) 5120 x 1000 1000 x 100 100 x 10
73.79 %
CIFAR-00 Testing Accuracy across epochs
CIFRR-10CC Motion blur accuraty across epochs
17
CIFA 10 Geursten Noize accuracy across epochs
Accuracy
Accuracy
Accuracy
Epochs
Epochs
(a)
Figure Introspective Feed-Forward accuracy of ResNet- across training epochs (a) CIFAR- 10 original testset. (b) CIFAR- 10C Motion Blur Testset on all challenge levels, (c) CIFAR-10C Gaussian Noise Testset on all 5 challenge levels.
C.5 INTROSPECTIVE ACCURACY ACROSS TRAINING EPOCII
In Section we make the assumption that f(-) well trained to approximate rz using Theorem In Section the Fisher Vector analysis works when the gradients form distances across the manifold in f(.) which occurs f-) is well trained. In this section we show that, practically, introspection per- forms as well feed-forward accuracy across training epochs CIFAR- 10 testset and outperforms feed-forward accuracy on CIFAR- 10C distortions, We show results on original testset, gaussian noise and motion blur testsets in Fig.
22

---

Under review as a conference paper at ICLR 2022
CIFAR Testing Accuracy across epochs
CIFAR-10 Testing Accuracy across epochs
2.0
0.8
Accuracy
Accuracy
0.6
0.4
0.2
Feed- Forward
Feed-Forward
Introspective Gain
Introspective Gain
25
50
100
125
150
175
200
25
50
75
100
125
150
200
Epochs
Epochs
(a)
(b)
Figure 10: Introspective vs. Feed-Forward accuracy of ResNet-18 across training epochs when (a)
π(+) and H(-) are trained the same training set (b) H-)) is trained on a separate held-out validation set
Training, Testing, and Results in Fig. In this experimental setup, ResNet- 18 is trained for 200 epochs. The model states multiples of 3 epochs from 1 to 200 are stored. This provides 67 states of (() along its training process. Each f(.) is tested on CIFAR-10 10 testset and the recognition accuracy is plotted in blue in Fig. (h). The introspective features Tr for all 67 states are extracted for the 50, 000 training samples. These Ta used to train 67 separate H(--) of structure provided in Table with a similar training setup as in Section C.I The Ty from the 10, 000 testing samples extracted individually for each of the 67 f()) states and tested. The results are plotted in red in Fig. 9h). Note the sharp spikes at epochs 60 and 120 where there a change in the learning rate. similar, introspective and feed-forward learning provides statistically similar performance across varying states of f(-).
Hence, when training and testing distributions
Training, Testing, and Results in Fig- 9b, We now consider the case when network f(-) is trained on distribution X' and tested on X from CIFAR-IOC distortions. The 67 trained models of ResNet-18 are tested on two distortions from CIFAR-10C. From the results in Fig. introspective learning achieves one of its highest performance gains in Gaussian noise, and an average increase in motion blur after epoch 200. The results Fig. 9 indicate that after approximately 60 epochs, the feed-forward network has sufficiently sensed notions to reflect between classes. This is seen in the performance gains in both the motion blur and Gaussian noise experiments.
Training of H on a separate validation set Fig. TOp In experiments, the introspective network H(-) is trained on the same training set f-)). In Fig. 10 show the results when the introspective network is trained on a separate portion of the dataset. We use 40, 000 images to train f() and 10, 000 train H(-) both of which from before. The model states at multiples of 3 epochs from 1 to 200 are stored. This provides 67 states of f(-) along its training process. Each f(-) is tested on CIFAR-10 testset and the recognition accuracy is plotted in blue in Fig. 10b). The H(-) at each iteration on the other hand is trained with the 10, 000 images. However, it has access to the notions created from the remaining 40, 000 images and hence the results for introspection match Fig. (9h) which is reproduced in Fig. (10)). The feed-forward
randomly chosen. We follow the training procedure
results catch up to the introspective results around epoch 60. At Epoch 120, add back the 10. 000
held-out images into the training set of [() and the results match between Fig. 10a) and Fig. 10
C.6 DOMAIN ADAPTATION ON OFFICE DATASET
In Section we claim that introspection helps a network to better classify distributions that it has not while training. In Section we tested on 95 new distributions in CIFAR-10C and 30 new distributions in IIFAR-IO-CURE In this section, we evaluate the efficacy of introspection when there is domian shift between training and testing data under changes in background, and camera
23

---

Under review as a conference paper at ICLR 2022
Table Performance of Proposed Introspective H(-) vs Feed-Forward f() Learning under Domain Shift on Office dataset
DSLR
Amazon
Amazon
Webcam
Architectures
4 DSLR
Amazon
Webcam
DSLR
Wehcam
Amazon
ResNet 18
5()
39.1
78
62.9
59
89.8
42.2
(%)
M01-)
90.7
67.3
63,9
96
ResNot-34
((-
41.8
83.3
67.3
60.1
90.6
41.7
N(1))
89.8
67.3
63.9
97.8
ResNet-50 (%)
f()
67. :3
62
92.4
33.4
H(-))
78.1
68.4
97.8
30.8
ResNet- 101
5()
62.0
80
31.77
(%)
P0(-)
76.5
67.3
92.4
33,6
Table 10: Performance of Proposed Introspective H((-) vs Feed-Forward f(-) Learning under Domain Shift on VisDA Dataset
ResNet-18
Plane
Cycle
Bus
Carr
Horse
Knile
Bike
Person
Plant
Skote
Train
Truck
All
FC-) (%)
7.2
38.1
54.8
4.2
72.7
28.7
87.2
2.9
38.
*(-))0 (36)
39.9
27.6
19.6
79.9
73.5
2.7
46.6
6.5
43.8
73.6
4.3
43.58
acquisition setup among others. Specifically, the robust recognition performance of H(-)) is validated on Office SSannoo et al.] (2010) dataset using Top-1 accuracy. The Office dataset has 3 domains images taken from either Webcam or DSLR, and extracted from Amazon website. Images can belong to any of 31 classes and they are of varying sizes upto 1920 1080. Hence, results on Office shows the applicability of introspection on large resolution images. ImageNet pre-trained ResNet-18,34.1 10. He et al. (2016) architectures the source domain while the remaining two domains are for testing. The experimental setup, the same detailed in Section 5 is applied and the Top-1 accuracy is calculated. The results Table In every instance, the top domain is X' - the training distribution, and the bottom domain is X" the testing distribution. Note that ResNet-50 and 101 failed train on 498 images in DSLR source domain. The results of introspection exceed that of feed-forward learning in all but ResNet-50 when classifying between Webcam and Amazon domains.
used for f()). The final layer is retrained using
summarized in
C.7 DOMAIN ADAPTATION ON VIS-DA DATASET
Validation results on a synthetic-to-real re-rel domain shift dataset called VisDA Peng et al. (201)) are presented in Table TO VisDA has 12 classes with about 152, 000 synthetie training images, and 55, 000 real validation images. The validation images are cropped images from MS-COCO. ResNet- 18 architecture pretrained ImageNet finetuned on the synthetically generated training images from VisDA dataset fro 200 epochs. It is then tested on the validation images and the recognition performance shown in Table 10 as feed-forward f() results. Introspective H.(-) results are obtained and shown when f-) is ResNet-18 18. There is an overall improvement of 5.48% in terms of performance accuracy. However, the individual class accuracies leave room for improvement.
C.8 ACTIVE LEARNING
In Ta the mean recognition accuracies across the first 20 rounds of Active Learning experiments for commonly used query strategies are shown. We plot these recognition accuracies across for all five query strategies in Fig. IT The x x-axis isssii is the round at which the performance is calculated. The caleulated accuracy is plotted on the y-axis. The experiment starts with a random 100 images in round 1. Each strategy queries using either round-wise sample trained f(.) or round-wiss wise sample trained H-)). Note that at each round, the networks are retrained. This continues for rounds. Both BALD Gal et al. (2017) and BADGE Ash et al. (2019) applied on H(-) consistently beat its f(-) counterpart on every round. This because both these methods rely on extracting features from the network as compared the other three techniques that directly use the output logits from either H(-)) or f.). Since the network is not well-trained at the initial stages due to a dearth of training data the introspective network is not consistent as the feed-forward network among Entropy, Least
24

---

Under review as a conference paper at ICLR 2022
BADGE
BALD
Introspective
Introspective
0.35
Fead-Forward
Feed-Forward
0.30
0.30
Accuracy
0.20
0.15
0.15
0.0
2.5
3.0
7,5
10.0
12.5
15.0
17.5
0.0
2.5
50
7.5
10.0
12.5
15.0
17.5
Rounds
Rounds
Entropy
Leas:Confidence
MarginSampling
0.3
Introspective
introspective Feed Forward
introspective Feedi
...-
Feed Forward
Feedi-Forwadd Forward
Accuracy
0.2%
0.25
0.33
0.35
Roundss
Rounds
Rounds
Figure 11: Introspective Feed-Forward accuracy of ResNet- 18 across training rounds for state-of- the-art techniques an active learning setting. The query batch size per round is 1000. The trainset is CIFAR-10 and testset is Gaussian Noise from CIFAR-10C.
Table 11:
Detection of existing techniques compared between feed-forward and
introspective networks.
Methods
OOD Danasets
FPR
Detection
AUROC
Foed FeneFeeee////reeeechhe
MSP Hendryoks & Gimpel 2016
SVHN
61.4115127
16.92/15.67
39.39/91.2
Placest6d
58,04544.43
17.01/15.07
39,38/9133
LSUN-C
27.95/27.5
9.42/10.29
96.07/9.733
Textures
52.399.31
22.17/6.12
$4.919199
Linng
SYHN
Phoes: 365
42.21/51.87
16.23/15.71
91.0690.95
LSUN-C
6,5923.66
5.54/10.2
98.74/ 95.87
Confidence, and Margin strategies. Nonetheless, H(-)) outperforms f(-) on average across all rounds.
C.9 OOD
Adversarial setting in Table A datapoint is perturbed as + and the goal of the detector is to classify X' or z E X". This modality proposed by the authors in in/Chen et al. (2020a) and we use their setup. PGD attack with perturbation 0.0014 is used. The same MSP and ODIN detectors from Table 2 are utilized. On OOD datasets, both MSP and ODIN show a performance gain across all three metrics on H--) compared f-)). Note that the results in Table 11 is for ResNet- 18 architecture for the same f(.) and H(-) used in other experiments including Fig.
Vanilla setting in Table X' is CIFAR-10 and X' are the four considered datasets. Note that among the four datasets, textures and SVHN are more out-of-distibuttion from CIFAR-10 than the natural image datasets Places365 and LSUN. The results of the introspective network is highest Textures DTD dataset.
Table II
we show the results of -f-riitriittion detection when
25

---

Under review as a conference paper at ICLR 2022
Table 12: Performance of Contrastive Features against Feed-Forward Features and other Image
Quality Estimators. Top 2 results in each row are highlighted.
PSNR
IW
SR
FSTMe
Per
CSV
SUM
Fees-Forwward UNIQUE
Introspective UNIQUE
Database
HA
SSIM
SIM
SIM
MER
Outlier Ratio (OR, L)
MULTI
0.013
0.013
0.000
0.016
0.004
0.000
0.000
0.000
0.000
TID13
0.615
0.701
0.632
0.728
0.655
0.687
0.620
0.640
0.620
Root
Mean Square Error (RMSE, L)
MULTI
11,320
10.049
8.686
10.794
9.898
9.895
8.212
9.258
7.943
TID13
0.652
0.688
0.619
0.687
0.643
0.647
0.630
0.615
0.596
Pearson Linear Correlation Coefficient (PL.CC, 1)
0.801
0.847
0.888
0.821
0.852
0.852
0.901
0.872
0.908
MULTI
0
0.851 -1
0.832
0.866
0.832
0.855
0,853
0,861
0.869
0.877
TIDI3
0
-1
0
0
Spearman's 's Rank Correlation Coefficient (SRCC, 7)
0.715
0.884
0.867
0.867
0.818
0,849
0.884
0.867
0.887
MULTI
-1
0
0
-1
0.847
0.778
0.807
0.851
0.854
0.846
0.856
0.860
0.865
TID13
Kendall's Rank Correlation Coefficient (KRCC)
0.532
0.702
0.678
0.677
0.624
0.655
0.698
0.679
0.702
MULTI
-1
0
0
-1
0.666
0.598
0.641
0.667
0.678
0.654
0.667
0.667
0.677
TID13
0
-1
-1
0
0
C.10
IMAGE QUALITY ASSESSMENT
Related Works Multiple methods have been proposed predict the subjective quality of images
including PSNR-HA (Ponomarenko et al., 2011). IW-SSIM (Wang & Li (011T) SR-SIM (Zhang &
Li 2012), FSIMc (Zhang et 2011). PERSIM TTmel & AlRegib 2015). CSV (Temel & AlRegib
2016), SUMMER (Temel& & AlRegib 2019), and UNIQUE (Temel et al. 2016). All these methods extract structure related hand-crafted features from both reference and distorted images and compare them to predict the quality. Recently, machine learning models have been proposed to directly extract features from images (Temel et al., (2016). TTmel et al. (2016) propose UNIQUE that uses a sparse autoencoder trained ImageNet to extract features from both reference and distorted images. We use UNIQUE as our base network f()).
Feed-Forward UNIQUE Temel et al. (0166) train sparse autoencoder with a one layer encoder and decoder and sigmoid non-linearity on 100, 000 patches of size 8 x 8 x 3 extracted from ImageNet testset. The autoencoder is trained with MSE reconstruction loss. This network is our f(.). UNIQUE follows a full reference IQA workflow which assumes access to both reference and distorted images while estimating quality. The reference and distorted images are converted to YGCr color space and converted to 8 x 8 x 3 patches. These patches are mean subtracted and 7CA whitened before being passed through the trained encoder. The activations of all reference patches in the latent space are extracted and concatenated. Activations lesser than threshold of 0.025 are suppressed to 0. The choice of threshold 0.025 is made based on the sparsity coefficient used during training. Similar procedure is followed for distorted image patches. The suppressed and concatenated features of both the reference and distorted images are compared using Spearman correlation. The resultant is the feed-forward estimated quality of the distorted image.
We use the architecture and the workflow from Temel et al. (016)) which based on feed-forward learning to demonstrate the value of introspection. We replace the feed- forward features with the proposed introspective features. The loss in Eq. 6 for introspection is not between classes but between the image x and its reconstruction 7 from the sparse autoencoder from Temel et al. 2016). For a reference image Tx is derived using J(x .). Hence, gradients of To span the space reconstruction noise. Since the need in IQA is to characterize distortions, we obtain y, for reference images from the first layer and project both reference and distorted images onto rg. These projections are compared using Spearman correlation to assign a quality estimate. In this setting, H(() is the projection operator and Spearman correlation. Hence, ProospectiveUNNQQU broadens introspection in the following ways 1) defining introspection on generative models, 2) using gradients in the earlier layers of network.
Introspectiv-UNNQQU
26

---

Under review as a conference paper at ICLR 2022
Results We report the results of the proposed introspective model in comparison with com- monly cited methods Table 12] We utilize MULTI-LIVE (MULTT) (Jayaraman et al.) 2012) and
TID2013 (Ponomarenko et al. 2013) datasets for evaluation. The performance is validated using out- lier ratio (consistency), root mean square error (accuracy), Pearson correlation (linearity). Spearman correlation (rank), and Kendall correlation (rank). Arrows next each metric in Table T2 indicate the desirability of a higher number (1) or a lower number()). Two best performing methods for each metric are highlighted. The proposed framework is always the top two methods for both datasets in all evaluation metrics. In particular, it achieves the best performance for all the categories except in OR and KRCC TID2013 dataset. The feed-forward model does not achieve the best performance for any of the metrics in MULTI dataset. However, the same network using introspective features significantly improves the performance and achieves the best performance on all metrics. For instance, the feed-forward model is the third best performing method in MULTT dataset in terms of RMSE, PLCC, SRCC, and KRCC. However, the introspective features improve the performance for those metrics by 1.315, 0.036, 0.020, and 0.023, respectively and achieve the best performance for all metrics. This further reinforces the plug-in capability the proposed introspective inference.
27


Reviews:
[{"review_id": "pvT6MT4OOya", "reviewer": "Reviewer_8FMs", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The reading of the paper does not flow properly. The methods are hard to understand and are not technically sound. Many core elements of the math discourse are introduced abruptly, such as $L$ and $\\mathcal{H}$. The definition of introspection is vague and not formal. The gap between the intuition about introspection presented in the introduction and the practical realization described in the methods sections remains wide and leaves the readers without a proper foundation to build their understanding. For instance, the intuition of introspection the authors give in the introduction and in Figure 1 calls for a reflection stage applied only to some promising alternative classes, while the proposed method applies the reflection stage to every single class.\nMoreover, I think that some less relevant sections should be shortened and others more relevant should be deepened. For instance, given the nature of the scientific contribution, this paper does not require in my opinion such an extensive Application section, while spending more words on making clear the methods in the middle sections and the general idea in the Introduction could provide more value to the reader.\n\nYet, to me, the idea of enriching neural networks features with gradient information through a two-stage inference process seems novel and sounds interesting. Results suggest that this research direction can be relevant, so I encourage the authors to pursue this work and focus on communicating it properly.\n\nIn the following, I list some comments about specific parts of the paper:\n\nINTRODUCTION\n- I believe that philosophical references can be of great value to the ML and AI community. In this sense I appreciate the reference to Locke, I wish the authors have expanded more this point.\n- \"For the application of recognition... ...EQUATION (1)\" This part is not clear. I do not understand why the authors use the projection concept in this context. $L$ seems a tag to indicate the final layer but then becomes a natural number ($L-1$). The first equation about $y_{feat}$ has a reference to $y$ that does not appear in the equation.\n\nINTROSPECTIVE FEATURES\n- The authors use the term explanations without defining what is an explanation for them and justify that definition. It is a textual string? A heat map on an image? Why we can consider heat maps as *explanations*? \n- The definition of introspection is very vague. \"...is the measurement of change induced in the network parameters...\", explicit this measurement, state the quantities. The reader here is expecting a formal definition but this definition is not formal.\n\nINTROSPECTIVE FEATURE VISUALIZATION\n- \"...Grad-CAM indicates that the pink and round body, and straight beak are the reasons for the decision. [...] the network highlights the neck of the spoonbill to indicate that since S-shaped neck is not observed, x cannot be a flamingo. ...\" I believe that here authors are overinterpreting the heat-maps. How can we say that the heat-map is caused by shapes rather than patterns? The clean high-level concepts of neck, beak, straight, S-shaped are what the network sees or what the authors see?\n\nINTROSPECTIVE FEATURE EXTRACTION\n- The assumption that a forward and backward pass through f is O(1) does not sound familiar to me. There is a dependence with respect to the size of the input. \n- Probably this subsection could be shorter, and at least part of the computational cost consideration could go in the appendix.\n\nINTROSPECTIVE NETWORK\n- H appears suddenly and everything is built on it. This is one of the more obscure parts of the paper.\n- The assumption of a mean squared error loss is not trivial, we usually do not train classifiers with the mean squared error loss but with cross-entropy. \n- \"having networks that introspect consecutively, by creating a trade-off\" is not clear.\n-  let the reader grasp how the reflection stage can fix the first inference of the sensing stage, spend some words on this case or maybe provide an example.\n\nEXPERIMENTS\n- \"Given a data distribution $x \\in X$\" why distribution?\n- \"10 separate bins based on their prediction confidence\". Considering the outputs of a neural network as confidences is problematic, this point should be expanded. DNNs are known to have unreliable uncertainty estimates (MacKay, 1995, Szegedy et al., 2014, Nguyen et al., 2015)", "summary_of_the_paper": "The authors present a new inference pipeline for neural networks $f$ trained to solve classification problems. They augment the features processed by $f$ with the loss gradients with respect to the weights of the last layer. Their pipeline requires two steps: the first one coincides with a standard evaluation of $f(x)$, while gradients are evaluated and processed in the second step to derive the final prediction. Experiments show slight improvements in classification accuracy and cuts of calibration errors. ", "summary_of_the_review": "The core parts of the paper are not clear and do not allow a proper comprehension and evaluation of the scientific contribution of this work. \nThe overall idea seems valuable and experimental results appear promising. Yet, with the current manuscript, I can only judge the potential of this work rather than the work itself, so at this stage my recommendation is a 3: reject, not good enough.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "lYTaN7SHpqk", "reviewer": "Reviewer_Q4tj", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper proposes an interesting and novel approach for improving robustness and calibration under distribution shift.  Results on CIFAR10-Corrupted are very promising, and the authors further show the method is able to improve in active learning and OOD detection experiments when inputs are corrupted with noise.\n\nLemma 1: assumptions should be stated precisely in the lemma statement itself. As it stands, the lemma statement makes it seem like the statement holds exactly, which seems very unreasonable at first glance.\nRegarding the reasonableness of the assumptions, a  \"well trained\" network might have high confidence predictions on input data it was trained on, but not necessarily on test data. Empirically, it has been observed that test data (particularly OOD test data) tends to have much higher entropy in predictions. Regarding theorem 1, can you test how well this approximation holds? In CIFAR10, there are only 10 classes, so it's reasonable to actually compute and store the full gradient features (especially since only the last layer is used). It would be good to see how a classifier performs using the full set of introspective features instead of the gradient wrt to the all ones vector. I would also highly recommend experiments to see if introspection (and this approximation in particular) scale well to more complex tasks with more classes like CIFAR100 or Imagenet, as I believe the current set of experiments to be quite limited.\n\n**Questions and Suggested Ablations:**\n\nCan you elaborate on why we should expect the introspective features to provide benefits when the new classifier is trained on the exact same training data as the base network? It seems very plausible to me that the gradients on training points the base\nnetwork has already overfitted heavily are a very different distribution than the gradients on previously unseen points, so I wouldn't necessarily have expected the introspective classifier to generalize well. I'm also curious what would happen if the introspection network is trained on a separate validation set.\n\nHow does this compare to directly taking the activations of the network instead of gradients? The final activations should be very closely related to the gradients of the last layer, differing by multiplicative terms corresponding to derivatives of the loss wrt to the output logits. This experiment would help to examine whether the structure of the gradient information from each class over possibly some benefits of just retraining model components on top of a pretrained model features.\n\nDid you examine introspective features from other layers?\n\nIn the OOD detection, are the same adversarial images used for the feedforward and introspective network, or is the adversarial image for the introspective method specifically targeted at the extra network using introspective features?\n\nSome missing related work:\n\nGradients as features: [2] is a crucial piece of missing related work, which also considers using gradients of a network as features, and for example show some similar results in recovering the original classifier performance on the trianing set. It would also be good to discuss and reference [1], which similarly uses gradients of base models as features, but applies it towards finetuning for transfer learning.\n \nConditional/predictive normalized maximum likelihood: Conditional/predictive normalized maxmimum likelihood [3, 4] considers retraining models for each label of the test input and combining these models' predictions to output a final prediction, essentially introspecting for each possible label to get predictions. While this work differs by using introspection to provide features for a separate classifier, it would be good to mention these other works as applying similar ideas of utilizing information on how model's change for specific labels of the test input in order improve predictions in the face of distribution shift.\n\n**Misc comments:**\n\nThere's a bit of a weird jump in the assumptions in the paper: lemma 1 assumes cross-entropy loss, but in sec 3, we're assuming network is trained with MSE for the derivation. \n\nIn addition to showing ECE for CIFAR10-C, I would also recommend including uncertainty-aware metrics like NLL and Brier score in the appendix, as these are proper scoring rules.\n\n\n**Citations:**\n\n[1] Zinkevich, Martin A. et al. “Holographic Feature Representations of Deep Networks.” UAI (2017).\n\n[2] Mu, Fangzhou, Yingyu Liang, and Yin Li. \"Gradients as features for deep representation learning.\" ICLR (2020)\n\n[3] Zhou, Aurick, and Sergey Levine. \"Amortized Conditional Normalized Maximum Likelihood: Reliable Out of Distribution Uncertainty Estimation.\" International Conference on Machine Learning (2021)\n\n[4] Bibas, Koby, Yaniv Fogel, and Meir Feder. \"Deep pnml: Predictive normalized maximum likelihood for deep neural networks.\" arXiv preprint arXiv:1904.12286 (2019).\n", "summary_of_the_paper": "The paper proposes introspective networks, a technique utilizing the gradients of a trained base model to make more robust predictions when faced with distribution shift.", "summary_of_the_review": "Overall, the paper presents an interesting approach with promising results for improving robustness, both in terms of accuracy and uncertainty estimation. I believe with a more clear presentation, more thorough discussion of related work, more extensive experimental evaluations, and ablations/evaluations of specific assumptions and design choices, this will be a valuable contribution.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "ZJ08YDHdcht", "reviewer": "Reviewer_yYCU", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Vanilla feedforward networks are single-stage, filtering the inputs as they pass forward through the network and predicting the class of the input in the final layer. Motivated by the human tendency to not only gather sensory data, but to reflect on those data in the face of uncertainty to arrive at a classification, the authors introduce a second, introspective or reflective, stage. \n\nThis stage is defined as the change in the parameters in the network if a different label were assigned to the network's input, and can be measured as a gradient with respect to the network weights of a loss function.  After showing how the calculation can be simplified in space and time, they then demonstrate how introspection can improve the generalizability of classifications to distributions that are shifted from the original training sets and can improve the calibration of the network, the confidence of the classification minus the accuracy.  They finally demonstrate how the approach can be used in a number of different applications.\n\nOverall, the introspective approach is, to the reviewer's knowledge, novel and potentially of interest as a general method for improving the performance of feed-forward networks. The authors provide the reader with intuition (e.g., Fig 2), rigor, and empirical data (Fig 3/4).  One potential issue that arises in this empirical data is that introspection does not improve the accuracy of ResNet-18 on CIFAR-10C, as they state, although the ECE decreases systematically.  \n\nUnfortunately, it is difficult to judge by eye whether the differences in Fig 4 are statistically significant.  The authors should include error bars or some other visual cue that indicate the uncertainty of the plotted values.  Additionally, without a statistically significant result, it is difficult to convince the reader that the authors' approach will work in a meaningful manner empirically in cases of interest.  \n\nIf possible, I would suggest the authors find such an example, or, if they cannot, to perhaps indicate future work that might improve the technique in such a way that it could in fact do so. Beyond this, the paper is written and organized well and clearly.", "summary_of_the_paper": "The authors present a novel 2-stage technique to improve the classification accuracy of feedforward ANNs.  In particular, after the feedforward pass, a so-called introspective stage occurs, the goal of which is to ascertain why the particular class label was provided rather than a different label.  This stage, which tends to improve the accuracy of the class predictions, is modular and can be added onto networks under varying task conditions.", "summary_of_the_review": "The authors present a new method, introspection, that can be added on top of feedforward networks to improve accuracy and generalization of the networks.  While they have written a well-structured paper that provides both intuition and rigor, the empirical results are somewhat lacking in that there are no statistically significant results that support the claims of accuracy improvements that introspection can obtain.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "KY2lzbPs8dA", "reviewer": "Reviewer_DxqE", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# Strengths\n* An interesting modification to standard neural networks for object recognition with improvements in generalization and  calibration\n* Extensive experiments showing the usefulness of the method in multiple applications\n\n# Weaknesses\n* Confusing explanation of the proposed method lacking a visual representation of the full model architecture, its components, and what the introspective features represent\n* Model is evaluated only on CIFAR-10 related datasets which are very low resolution images and few possible classes\n* According to the authors, the proposed method does not scale well for larger datasets. This makes its applicability very limited as CIFAR-10 is only a toy dataset\n* Some controls seem rather arbitrary and a better choice of parameters for the alternative baselines could have been done\n\n# Detailed review\n## Role of introspection for object recognition\nObject recognition in humans is an extremely fast visual computation, taking place within ~200ms of stimulus presentation, and it is unlikely to depend on introspection. It is not clear the motivation of why including introspective learning in artificial neural networks would improve their performance. Furthermore, standard neural networks are typically optimized to classify an object as belonging to one class and not belonging to the others (all classes are used in the cost function).  In that sense, it could be argued that they also are trained to answer the questions illustrated in Figure 1.\n\n## Confusing explanation of the proposed methodology\nUnfortunately, the authors could have done a better job at explaining their method. In my opinion, some of the mathematical details could be moved to the appendix, and in their place, the authors should focus on clarifying the training and inference process of the introspective networks. A visual representation of the full model architecture, its components, what the introspective features represent, and how the model is trained would be extremely helpful to the reader. Because of this it was not clear where the computational complexity of the introspective networks come from. The MLP used is very standard and the introspective features are obtained from the model’s last layer gradients.\n\n## Poor choice of controls\nSince adding the 3 layers to the standard model made it considerably worse, the choice of using 3 additional layers on the control for the same number of parameters was a poor choice by the authors. The MLP in the introspective network has a higher number of features but only a single layer. A better comparison would be to make the last layer of the standard model with more features, keeping only 1 layer, or keeping the same penultimate layer and adding a new much wider layer to compensate the smaller number of parameters. \n\n## Minor comments\n* Fonts in several figures are very small. Please improve their readability\n* In Figure 4b and Figure 6, the x-axis should be corruption severity and go from 1 to 5\n* Figure 2 does not add much and could go to supplementary\n* Figure 1 is very high-level and in its place, the authors should provide a more concrete visualization of their method\n", "summary_of_the_paper": "In this paper, the authors propose a modification to standard neural networks used for object classification tasks to incorporate what they call “introspective learning”. This consists on training a multi-layer perceptron (MLP) on the neural network introspective features. These are obtained by calculating the gradients over the last layer weights of the model on a loss function corresponding to posing an introspective question: why is the correct label A instead of B? The authors apply this procedure to several neural networks of the ResNet model family and show that the introspective-networks have better generalization to distributional shifts and smaller calibration errors on datasets with the same classes and image sizes as CIFAR-10. Finally, they show further improvements in multiple applications.", "summary_of_the_review": "The authors propose a very interesting modification to neural networks for object recognition and do a convincing job in showing that it improves generalization and calibration. However, the proposed method is only tested in CIFAR-10 and its use in larger networks and datasets is questionable. Furthermore, the explanation of the method is unnecessarily confusing and better controls could have been chosen to make a more convincing case.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers all appreciated the novel concept behind the work. I agree with this, I think the principles behind the work are novel and interesting, and I would encourage the authors to improve the validation of this method and publish it in the future.

However, reviewers also raised a number of issues with the current paper: (1) the evaluation appears a bit preliminary, and could be improved significantly with additional datasets and more ablations/comparisons; (2) it's not clear if the improvements from the method are especially significant; (3) the writing could be improved (I do see that the authors made a significant number of changes and improved parts of the paper in response to reviewer concerns to a degree). Probably the writing issues could be fixed, but the skepticism about the experiment results seems harder to address, and while I recognize that the authors made an effort to point some existing ablations in the paper that do address parts of what the reviewers raised, I do think that in the balance the experimental results leave the validation of the work as somewhat borderline.

While less important for the decision, I found that the paper is somewhat overselling the contribution in the opening -- while the particular concept of using gradients as features in this way is interesting, similar ideas have been proposed in the past, and the paper would probably be better if it was more clearly positioned in the context of prior work rather than trying to present a new "framework" like this. It kind of feels like it's biting off too much in the opening, and then delivering a comparatively more modest (but novel and interesting!) technical component.
Decision: Reject