Year: 2021

Abstract:
In this paper, we advocate for two stages in a neural network’s decision making process. The first is the existing feed-forward inference framework where patterns in given data are sensed and associated with previously learned patterns. The second stage is a slower reflection stage where we ask the network to reflect on its feed-forward decision by considering and evaluating all available choices. Together, we term the two stages as introspective learning. We use gradients of trained neural networks as a measurement of this reflection. We perceptually visualize the posthoc explanations from both stages to provide a visual grounding to introspection. For the application of recognition, we show that an introspective network is 4% more robust and 42% less prone to calibration errors when generalizing to noisy data. We also illustrate the value of introspective networks in downstream tasks that require generalizability and calibration including active learning and out-ofdistribution detection. Finally, we ground the proposed machine introspection to human introspection in the application of image quality assessment.

heading: 1 INTRODUCTION
text: Introspection is the act of looking into one’s own mind (Boring, 1953). Classical introspection has its roots in philosophy. Locke (1847), the founder of empiricism, held that all human ideas come from experience. This experience is a result of both sensation and reflection. By sensation, one receives passive information using the sensory systems of sight, sound, and touch. Reflection is the objective observation of our own mental operations. Consider the task of differentiating a spoonbill from a flamingo and a crane. This task requires prior knowledge of some differentiating features between the birds. These features include the color and shape of the body, and beak of all birds. We first associate these features with our existing knowledge of birds and make a coarse decision that the given bird is a spoonbill. This is the sensing stage. Reflection involves questioning the coarse decision and asking why the bird cannot be a flamingo or crane. If the answers are satisfactory, then an introspective decision that the bird is indeed a spoonbill is made. The observation of this reflection is introspection.
In this paper, we adopt this differentiation between sensing and reflection to advocate for two-stage neural network architectures for perception-based applications. We first ground introspection based on existing neural networks. The above-mentioned task of differentiating a spoonbill from a flamingo and crane is provided in Fig. 1 for neural networks A network f(·), is trained on a distribution X to classify data into N classes. The network learns notions about data samples when classifying them. These notions are stored as network weights W . Let yfeat be the logits projected before the final fully connected layer. We denote the final fully connected layer as fL, where L is the layer number. fL−1 is then the layer before the final fully connected layer. Using the weight parameters WL, the output of the network ŷ is given by,
yfeat = fL−1(x),∀yfeat ∈ <N×1, ŷ = arg max(WTL yfeat),∀WL ∈ <dL−1×N , fL−1(x) ∈ <dL−1×1.
(1)
Hence, ŷ is the class in which the sensed features maximally correlate with the stored notion. This is the feed-forward prediction in Fig. 1. Existing recognition architectures including VGG (Simonyan & Zisserman, 2015), ResNet (He et al., 2016), and DenseNet (Huang et al., 2017) among others all sense and predict using Eq.1. In Fig. 1, we depict our proposed introspective learning framework. An additional reflection stage extracts the introspective features as Not Detect features from x. Let
r1 and r2 be the two introspective features. In this case, r1 is the absence of the S-shaped neck in the spoonbill. And r2 is the lack of white feathers in the given input image. We use a post-hoc visual explanation to depict these features1. Note that there can be N such features for a sensing network f(·) trained to differentiate between N classes. These features are then combined to obtain the final introspective feature rx. rx is characteristic of the input image x and is passed through an introspective network, H(·), to obtain the introspective prediction ỹ. We term the combination of both f(·) andH(·) as introspective learning. Note that the proposed reflection stage does not require a predefined knowledge base of introspective features. Rather, ri, i ∈ [1, N ] in the reflection stage, is extracted using the feed-forward prediction ŷ and the sensing network parameters. Hence, ri are Not Detect features based on f(·)’s notion of classes. H(rx) predicts ỹ explicitly based on the differences between f(·)’s notion of classes. Not only should the network sense the feed-forward patterns, it must also satisfy H(·)’s N notions of differences. In this paper, we show that the inference process is more generalizable due to these N additional inferential constraints. Specifically, the introspective network is more robust to noise and is less prone to calibration errors. The challenge is to implicitly extract features that answer introspective questions without explicitly training on said questions as is the norm in Visual Question Answering applications (Antol et al., 2015). We show that gradients w.r.t network parameters store notions about the difference between classes and can be used as introspective features. We first describe the methodology of introspective feature extraction in Section 2. We then analyzeH(·) in Section. 3. We show thatH(·) as a simple multi-layer perceptron that introspects on ŷ is more generalizable for the application of recognition in Section 5. We then illustrate the benefits of our two-stage architecture in other downstream tasks including out-of-distribution detection, active learning and image quality assessment in Section 6.

heading: 2 INTROSPECTIVE FEATURES
text: In this section, we describe introspective features and implicitly extract them using the sensing network. We then analyze their extraction procedure and provide a methodology to accelerate it. Definition 2.1 (Introspection). Given a network f(·), a datum x, and the network’s prediction f(x) = ŷ, introspection in f(·) is the measurement of change induced in the network parameters when a label yI is introduced as the label for x. This measurement is the gradient induced by a loss function J(yI , ŷ), w.r.t. the network parameters.
This definition for introspection is in accordance with the sensing and reflection stages in Fig. 1. The network’s prediction ŷ is the output of the sensing stage and the change induced by an introspective label, yI , is the network reflecting on its decision ŷ as opposed to yI . Combination of the two is introspection. Note that introspection can occur when ŷ is contrasted against any trained label yI , I ∈ [1, N ]. For instance, in Fig. 1, the network is asked to reflect on its decision of spoonbill by considering other yI that x can take - flamingo and crane.
1post-hoc explanations are justifications made by a neural network after a decision has been made. They require human interpretation. Further details are provided in Appendix A.
Reflection is the empirical risk that the network has predicted x as ŷ instead of yI . Given the network parameters, this risk is measured through some loss function J(yI , ŷ). yI is a one-hot vector with a one at the Ith location. The change that is induced in the network is given by the gradient of J(yI , ŷ) w.r.t. the network parameters. In this paper, we introspect based on reflecting on all possible classes. For an N -class classifier, there are N possible introspective classes and hence N possible gradients each given by, rI = ∇WJ(yI , ŷ), I ∈ [1, N ]. Here, rI are the introspective features. Since we introspect based on classes, we measure the change in network weights in the final fully connected layer. Hence the introspective features are given by,
rI = ∇WLJ(yI , ŷ), I ∈ [1, N ], rI ∈ <dL−1×N (2)
where WL are the network weights for the final fully connected layer. Note that the final fully connected layer from Eq. 1 has a dimensionality of <dL−1×N . For every x, Eq. 2 is applied N times to obtain N separate rI . We first analyze these features before accelerating their extraction.

heading: 2.1 INTROSPECTIVE FEATURE ANALYSIS
text: Consider the extraction process in Eq. 2. Each rI is a dL−1 ×N matrix. Expressing gradients in rI separately w.r.t. the different filters in WL, we have a row-wise concatenated set of gradients given by,
rI = [∇WL,1J(yI , ŷ);∇WL,2J(yI , ŷ);∇WL,3J(yI , ŷ) . . .∇WL,NJ(yI , ŷ)] (3)
where each WL,j ∈ <dL−1×1 and rI ∈ <dL−1×N 2
. For all data x ∈ X the following lemma holds: Lemma 1. Given a unique ordered pair (x, ŷ) and a trained network f(·), the gradients for a loss function J(yI , ŷ) w.r.t. classes are pairwise orthogonal under the second-order Taylor series approximation, each class paired with the predicted class.
Proof. Provided in Appendix B.1.
Lemma 1 states that backpropagating class yI does not provide any information to WL,j , j 6= I and hence there is no need to use ∇WL,jJ(yj , ŷ), j 6= i as features when considering yI . In Appendix B.1, we provide the complete proof when J(yi, ŷ) is the cross entropy loss. ∇WJ(yI , ŷ) for an introspective class reduces to,
∇WJ(yI , ŷ) = −∇W yI +∇W log ( y2ŷ 2 ) . (4)
where yŷ is the logit associated with the predicted class. In Fig. 5, we use a network trained on MNIST (LeCun et al., 1998) dataset to simulate a well-trained network and we visualize the gradients from the final fully connected layer to demonstrate Eq. 4.
Eq. 4 motivates the generalizable nature of our introspective features. Consider some noise added to x. To change the prediction ŷ, the noise must sufficiently decrease yŷ from Eq. 4 and increase the closest logit value, yI , to change the prediction. However, by constraining our final prediction ỹ from Fig. 1 on N such Eq. 4, the noise needs to change the orthogonal relationship between N pairwise logits. This motivates a functionH(·) that is conditioned on N such pairwise logits. In Section. 5, we empirically show the robustness of our feature set.

heading: 2.2 INTROSPECTIVE FEATURE EXTRACTION
text: From Lemma 1, the introspective feature is only dependent on the predicted class ŷ and the introspective class yI making their span orthogonal to all other gradients. Hence
rI = ∇WL,IJ(yI , ŷ), I ∈ [1, N ], rI ∈ <dL−1×1 (5)
Compare Eq. 5 against the introspective feature from Eq. 2. Assuming that forward and backward passes through the final layer fL(·) are each ofO(1) time complexity, the feed-forward prediction for a given x is O(1) time complex. Given that f(·) is trained to classify between N classes, extracting
N introspective features require N backpropagations and hence is O(N) complex. Each rI in Eq. 2 has a dimenisonality dL−1×N . Hence forN features, the space complexity isO(N2×dL−1). From Lemma 1, the introspective feature is only dependent on the predicted class ŷ and the introspective class yI making their span orthogonal to all other gradients. For N introspective features in Eq. 5, the space complexity of rI reduces from O(dL−1 ×N2) to O(dL−1 ×N). Note that the bottleneck in time complexity for N gradient extractions are the serial N backpropagations in Eq. 3. Building on Lemma 1, we present the following theorem.
Theorem 1. Given a unique ordered pair (x, ŷ) and a trained network f(·), the gradients for a loss function J(yI , f(x)), I ∈ [1, N ] w.r.t. classes when yI are N orthogonal one-hot vectors is equivalent to when yI is a vector of all ones, under the second-order Taylor series approximation.
Proof. Provided in Appendix B.2.
The proof follows Lemma 1. Theorem 1 states that backpropagating a vector of all ones (1N ) is equivalent to backpropagating N one-hot vectors with ones at orthogonal positions. This reduces the time complexity from O(N) to a constant O(1) since we only require a single pass to backpropagate 1N . Hence, our introspective feature is given by,
rx = ∇WLJ(1N , ŷ), rx ∈ <dL−1×N , 1N = 1N×1 (6)
Note the LHS is now rx instead of rI from Eq. 5. The final introspective feature is a matrix of the same size as WL extracted in O(1) with a space complexity of O(dL−1 ×N). rx is vectorized and scaled between [−1, 1] before being used in Sections 5 and 6 as introspective features.

heading: 3 INTROSPECTIVE NETWORK
text: Once rx are extracted using Eq. 6, the introspective label ỹ from Fig. 1 is given by ỹ = H(rx). In this section, we analyze H(·). From Fig. 1, f(·) is any existing trained network used to obtain introspective features rx. It is trained to predict the ground truth y given any x. Let f(·) be trained using the mean squared error loss function. Based on the assumption thatH(rx) = E(y|f(x)) and hence expectation of y −H(rx) is 0, the loss function can be decomposed as,
E[(f(x)− y)2] = E[(f(x)−H(rx))2)] + E[(H(rx)− y)2)]. (7)
Note that since the goal is to predict y given x, H(rx) = E(y|f(x)) is a fair assumption to make. Substituting for f(x) in Eq. 7, and using variance decomposition of y onto f(x), we have,
E[(ŷ − y)2] = Var(ŷ)− Var(H(rx)) + E[(H(rx)− y)2]. (8)
This decomposition is adopted from structured calibration techniques. A full derivation is presented in Kuleshov & Liang (2015). The first term Var(ŷ) is the the variance in the prediction from f(·). This term is the precision of f(·) and is low for a well trained network. The third term is the MSE function between the introspective networkH(·) and the ground truth. It is minimized while training the H(·) network. The second term is the variance of the network H(·) given features rx. Note that minimizing Eq. 8 can occur by maximizing Var(H(rx)). However, Var(H(rx)) is also part of the bias-variance decomposition in the third term which is minimized. This prevents perpetual introspection, i.e having multiple gradient extraction-basedH(·) networks bootstrapped together, by creating a trade-off. We use a fisher vector interpretation to analyze Var(H(rx)). IfH(·) is a linear layer with parameters WH, the Var(H(rx)) term reduces to WTHWH × Var(rx) ∝ Tr(rTx Σ−1rx) where Σ is the covariance matrix. Σ is a gaussian approximation for the shape of the manifold. Generalizing it to a higher dimensional manifold and replacing Σ with F , we have,
Var(H(rx)) = Tr(rTx F−1rx), (9)
Var(H(rx)) = N∑ j=1 rTj F −1rj . (10)
The RHS of Eq. 10 is a sum of fisher vectors taken across all possible labels. We analyze two cases of usage ofH(·) through Fisher Vectors : When input X is same as the training distribution and when X ′ is from a noisy distribution.
Estimation of X usingH When a sample x ∈ X is provided to a network f(·) trained on X , all rj , j 6= ŷ in Eq. 10 tend to 0. The RHS reduces to rTŷ F−1rŷ . rŷ is a function of f(x) only and hence adds no new information to the framework. The results of H(·) remain the same as f(·). In other words, given a trained ResNet-18 on CIFAR-10, the results of feed-forward learning will be the same as introspective learning on CIFAR-10 testset.
Generalization to X ′ usingH When a new samples x′ 6∈ X is provided to a network f(·) trained on X , a fisher vector based projection across labels is more descriptive compared to a feed-forward approach. The N gradients in Eq. 10 add new information based on how the network needs to change the manifold shape F to accomodate the introspective gradients. Hence, given a distorted version of CIFAR-10 testset, our proposed introspective learning generalizes with a higher accuracy while providing calibrated outputs from Eq. 8. We show these two claims in Section 5. The benefits of calibrated generalizability are further explored in downstream tasks like active learning and out-of-distribution detection in Section 6.

heading: 4 RELATED WORKS
text: Two-stage Networks The usage of two-stage approaches to inference in neural networks is not new. The authors in Chen et al. (2020b) propose SimCLR, a self-supervised framework where multiple data augmentation strategies are used to contrastively train an overhead MLP. The MLP provides features which are stored as a dictionary. This feature dictionary is used as a look-up table for new test data. The classical object detection technique of R-CNN (Girshick et al., 2014) uses separate feature extraction and detection stages for inference. In all these works, the extracted features are feed-forward activations. In this paper, we use gradients against all classes as features. Zhou & Levine (2021) and Bibas et al. (2019) consider all classes in a conditional maximum likelihood estimate on test data to retrain the model. These works differ from ours in our usage of the pairwise orthogonality of logits. We make use of this by havingH(·) as a classifier that explicitly learns the introspected pairwise relationships between classes.
Gradients-as-Features The gradients from a base network have been utilized in diverse applications including post-hoc visual explanations (Selvaraju et al., 2017; Prabhushankar et al., 2020), adversarial attacks (Goodfellow et al., 2014), and anomaly detection (Kwon et al., 2020) among others. In explanations, gradients are used to highlight features while in adversarial attacks, gradients characterize the required alterations to the features. Fisher Vectors use gradients of generative models to characterize the change that data creates within features (Jaakkola et al., 1999). A formulation similar to that of Fisher Kernels is used in Cohn (1994). Gradients of parameters are used to characterize the change in manifolds when new data is introduced to an already trained manifold. Our framework uses the intuition from Cohn (1994) to characterize changes for a datapoint that is perceived as new, due to it being assigned an introspective class that is different from its predicted class. In Zinkevich et al. (2017), the authors view the network as a graph and intervene within it to obtain holographic features. Our introspective features are also holographic in the sense that they are not true. However, our features are dependent on the notions from the network itself and do not require engineered interventions that can become expensive with scale. Mu et al. (2020) use gradients and activations together as features and note that the validity of gradients as features is in pretrained base networks rather than additional parameters from the two-stage networks. We demonstrate this as well in Appendix C.4.
Augmentations and Robustness The considered rx features from Eq. 6 can be considered as feature augmentations. Augmentations, including SimCLR, Augmix (Hendrycks et al., 2019), adversarial augmentation (Hendrycks & Dietterich, 2019), and noise augmentations (Vasiljevic et al., 2016) have shown to increase robustness of neural networks. We use introspection on top of non-augmented (Section 5) and augmented (Appendix C.2) networks and show that our proposed two-stage framework increases the robustness to create generalizable and calibrated inferences which aids active learning and out-of-distribution (OOD) detection. The same framework that robustly recognizes images despite noise can also detect noise to make an out-of-distribution detection.
Confidence and Uncertainty The existence of adversarial images (Goodfellow et al., 2014) heuristically decouples the probability of neural network predictions from confidence and uncertainty. A number of works including Sensoy et al. (2018) and MacKay (1995) use bayesian formulation to provide uncertainty. However, in downstream tasks like active learning and Out-Of-Distribution
(OOD) detection applications, existing state-of-the-art methods utilize softmax probability as confidences. This is because of the simplicity and ease of numerical computation of softmax. In active learning, uncertainty is quantified by the entropy (Wang & Shang, 2014), least confidence (Wang & Shang, 2014), or maximum margin (Roth & Small, 2006) of predicted logits, or through extracted features in BADGE Ash et al. (2019), and BALD (Gal et al., 2017). In OOD detection, Hendrycks & Gimpel (2016) propose Maximum Softmax Probability (MSP) as a baseline method by creating a threshold function on the softmax output. Liang et al. (2017) propose ODIN and improved on MSP by calibrating the network’s softmax probability using temperature scaling (Guo et al., 2017). In this paper, we show that the proposed introspective features are better calibrated than their feedforward counterparts. Hence existing methods in active learning and OOD detection have a superior performance when usingH(·) to make predictions. Human Introspection There is no direct application that tests visual human introspection. In its absence, we choose the application of Full-Reference Image Quality Assessment (FR-IQA) to connect machine vision with human vision. The goal in FR-IQA is to objectively estimate the subjective quality of an image. Humans are shown a pristine image along with a distorted image and asked to score the quality of the distorted image (Sheikh et al., 2006). This requires reflection on the part of the observers. We take an existing algorithm (Temel et al., 2016) and show that introspecting on top of this IQA technique brings its assessed scores closer to human scores.

heading: 5 EXPERIMENTS
text: Across Sections 5 and 6, we use a 3-layered MLP with sigmoid activations as H. The structure is presented in Appendix C.1. We first define generalization and calibration in the context of this paper.
Generalization In this paper, without loss of consistency with related works, we say that the network trained on distribution X is generalizable if it predicts correctly on a shifted distribution X ′. The difference in data distributions can be because of data acquisition setups, environmental conditions, distortions among others. We use CIFAR-10 for X and two distortion datasets - CIFAR10C (Hendrycks & Dietterich, 2019) and CIFAR-10-CURE (Temel et al., 2018) as X ′. Generalization is measured through performance accuracy.
Calibration Given a data distribution x ∈ X , belonging to any of y ∈ [1, N ], a neural network provides two outputs - the decision ŷ and the confidence associated with ŷ, given by p̂. Let p be the true probability empirically estimated as p = p̂i,∀i ∈ [1,M ]. Then calibration is given by (Guo et al., 2017),
P(y = ŷ|p = p̂) = p (11) Calibration measures the difference between the confidence levels and the prediction accuracy. To showcase calibration we use the metric of Expected Calibration Error (ECE) as described in (Guo et al., 2017). The network predictions are placed in 10 separate bins based on their prediction confidences. Ideally, the accuracy equals the mid-point of confidence bins. The difference between accuracy and mid-point of bins, across bins is measured by ECE. Lower the ECE, better calibrated is the network.
Datasets and networks CIFAR-10C consists of 950, 000 images whose purpose is to evaluate the robustness of networks trained on original CIFAR-10 trainset. CIFAR-10C perturbs the CIFAR-10 testset using 19 distortions in 5 progressive levels. Hence, there are 95 separate X ′ distributions to test on with each X ′ consisting of 10000 images. Note that we are not using any distortions or data from CIFAR-10C as a validation split during training. The authors in Temel et al. (2018) provide realistic distortions that they used to benchmark real-world recognition applications including Amazon Rekognition and Microsoft Azure. We use these distortions to perturb the test set of CIFAR10. There are 6 distortions, each with 5 progressive levels. Of these 6 distortions - Salt and Pepper, Over Exposure, and Under Exposure noises are new compared to CIFAR-10C. We train four ResNet architectures - ResNet-18, 34, 50, and 101 He et al. (2016). All four ResNets are evaluated as sensing networks f(·). The training procedure and hyperparameters are presented in Appendix C.1. Testing on CIFAR-10 testset The trained networks are tested on CIFAR-10 testset with accuracies 91.02%, 93.01%, 93.09%, and 93.11% respectively. Next we extract rx on all training and testing images in CIFAR-10. H(·) is trained using rx from the trainset using the same procedure as f(·). When tested on rx of the testset, the accuracy for ResNets-18,34,50,101 is 90.93%, 92.92%, 93.17%,
and 93.03%. Note that this is similar to the feed-forward results. The average ECE of all feedforward and introspective networks is 0.04. Hence, when the test distribution is the same as training distribution there is no change in performance.
Testing on CIFAR-10C and CIFAR-10-CURE The results of all networks averaged across distortions in both the datasets are shown in Fig. 2. Note that in each case, there is a shift leftward and upward indicating that the performance improves while the calibration error decreases. In the larger CIFAR-10C dataset, the introspective ResNet-18 performs similar to ResNets-34 and 50 in terms of accuracy while beating them both in calibration. A more fine-grained analysis is shown in Fig. 3 for ResNet-18. The blue bars in Fig. 3a) represent the feed-forward accuracy. The red bars are the introspective accuracy gains over the feed-forward accuracy. Among 7 of the 19 distortions, the accuracy gains are over 5%. In Appendix C.2.1 and Fig. 6, we see that the gains are higher when the distortions are higher. Introspection performs well on blur-like distortions while struggling with distortions that disrupt the lower level characteristics of the image like brightness, contrast, and
saturate. This can be attributed to the fact that rx are derived from the last layer of f(·) and are missing low-level statistics that are filtered out by network in the initial layers. However, in Fig. 3b), we show ECE for brightness and saturate distortions across all 5 distortion levels - higher the level, more is the distortion affecting X ′. It can be seen that while the ECE for feed-forward networks increases across levels, the ECE for introspective networks decrease. Hence, even when there are no accuracy gains to be had, introspection helps in calibration.
Plug-in results of Introspection Note that there are a number of techniques proposed to alleviate a neural network’s robustness challenges against distortions. The authors in (Vasiljevic et al., 2016) show that finetuning VGG-16 using blurry training images increases the performance of classification under blurry conditions. (Temel et al., 2017) propose utilizing distorted virtual images to boost performance accuracy. The authors in (Hendrycks & Dietterich, 2019) use adversarial images to augment the training data. All these works require knowledge of distortion or large amounts of new data during training. Our proposed method can infer introspectively on top of any existing f(·) enhanced using existing methods. In Appendix C.2, we show performance on top of (Vasiljevic et al., 2016) and (Hendrycks & Dietterich, 2019) of 6.8% on Level 5 distortions. In Appendix C.3, we analyze SimCLR and show that introspecting on the self supervised features increases its CIFAR-10C performance by about 6% on ResNet-101. We introspect on top of Augmix (Hendrycks et al., 2019) and show that while recognition accuracy is the same, introspection reduces ECE of Augmix network by 43.33%. A number of ablation studies including analysis of structure ofH, loss function, distortion levels on performance accuracy and ECE are shown in Appendix C.4. Moreover, we examine introspection when X ′ is domain shifted data from Office (Saenko et al., 2010) dataset in Appendix C.6.

heading: 6 APPLICATIONS
text: In this section, we illustrate the advantages of introspective networks in two applications that are a function of both generalization and calibration - active learning and out-of-distribution detection. In both these applications, we show that the existing state-of-the-art methods perform better in their respective tasks and metrics if they were applied on H(·) than on f(·). We then conclude by grounding the proposed introspection in neural nets with introspection in humans through IQA.
Active Learning The goal in active learning is to decrease the test error in a model by choosing the best samples from a large pool of unlabeled data to annotate and train the model. A number of strategies are proposed to query the best samples. A full review of active learning and query strategies are given in Settles (2009). Existing active learning strategies define best samples to annotate as those samples that the model is most uncertain about. We use the strategies given in Section 4 to showcase the effectiveness ofH(·). We show the results of ResNet-18 and 34 architecture in Table 1. Implementations of all query strategies in Table 1 are taken from the codebase of Ash et al. (2019) and reported as feed-forward results. Note that the query strategies act on f(·) to sample images at every round. In the introspective results, all query strategies sample usingH(·). The training and testing procedures strategies are the same as feed-forward from Ash et al. (2019). Doing so we find similar results as recognition - on the original testset the active learning results are the same while there is a gain across strategies on Gaussian noise testset from CIFAR-10C. Note that the results shown are
averaged over 20 rounds with a query batch size of a 1000 and initial random choice - which were kept same for f(·) andH(·) - of 100. Further details and plots are shown in Appendix C.8. Out-of-distribution Detection The goal of Out-Of-Distribution (OOD) detection is to detect those samples that are drawn from a distribution X ′ 6= X given a fully trained f(·). As mentioned in Section 4, we use MSP (Hendrycks & Gimpel, 2016) and ODIN (Liang et al., 2017) to illustrate the effectiveness of existing OOD methods when applied onH(·) than if they were applied on the feed-forward f(·). The code for OOD detection techniques are taken from Chen et al. (2020a) along with all hyperparameters and the training regimen. The temperature scaling coefficient for ODIN is set to 1000. Note that we do not use additional temperature scaling on H(·) to illustrate the effectiveness of our method. We use three established metrics to evaluate OOD detection - False Positive Rate (FPR) at 95% True Positive Rate (TPR), Detection error, and AUROC. Ideally, AUROC values for a given method is high while the other two metrics are low. We use CIFAR-10 as our in-distribution dataset and use four OOD datasets - iSUN (Xiao et al., 2010), Describable Textures Dataset (Cimpoi et al., 2014), Places 365 (Zhou et al., 2017), and LSUN (Yu et al., 2015). As in recognition and active learning, we consider two difficulty settings. The first is the vanilla case with the above-listed out-of-distribution datasets. The results are presented in Appendix C.9 and Table 11. The harder setting is when the datasets are all attacked with adversarial noise on top of being OOD. This setting is formalized in Chen et al. (2020a). We show these results in Table 2. In all cases in Table 2, introspection outperforms its feed-forward counterpart. In the vanilla setting, introspection is better than its feed-forward network in 15 of considered 24 metrics. This is inline with results from recognition and active learning where introspection generalizes better on X ′ compared to X testsets. Image Quality Assessment (IQA) TID 2013 (Ponomarenko et al., 2015) and MULTI-LIVE (Jayaraman et al., 2012) are two IQA datasets with 3000 and 225 distorted images respectively in 5 and 4 progressively increasing levels of distortions. This setup is similar to CIFAR-10C. The goal is to objectively assess the subjective quality of the distorted images given the pristine image. Inline with existing techniques (Temel et al., 2016), we use five metrics to measure the similarity between the algorithmically predicted qualities and human qualities - outlier ratio (consistency), root mean square error (RMSE, accuracy), Pearson correlation (PRCC, linearity), Spearman correlation (SRCC, rank), and Kendall correlation (KRCC, rank). To do so, we use an existing IQA technique called UNIQUE (Temel et al., 2016) and introspect on top of it. The details of UNIQUE as well as related works and compared methods are presented in Appendix C.10. Since UNIQUE is an autoencoder architecture and since the pristine image is available, introspection can occur based on features and on earlier layers. The exact procedure is given in Appendix C.10 and is termed as Introspective-UNIQUE. The results are shown in Table 12. The proposed framework acts as a plug-in on top of UNIQUE. For instance, UNIQUE is the third best performing method in MULTI dataset in terms of RMSE, PLCC, SRCC, and KRCC. However, Introspective-UNIQUE improves the performance for these metrics by 1.315, 0.036, 0.020, and 0.023, respectively and achieves the best performance on all metrics.

heading: 7 DISCUSSION AND CONCLUSION
text: Limitations and future work The paper illustrates the benefits of utilizing the change in model parameters as a measure of model introspection. In Section 2.2, we analyze the time complexity and accelerate it toO(1). However, the space complexity is still dependent on N . The paper uses an MLP forH(·) and constructs rx by vectorizing extracted gradients. Hence, taking the dimensionality of the final feature layer from Eq. 1, the space complexity is O(N × dL−1). For large datasets with large N , usage of rx as a vector of concatenated gradients is prohibitive. Hence, a required future work is to provide a method of combining all N gradients without vectorization. Also, our implementation uses serial gradient extraction across images. This is non-ideal since the available GPU resources are not fully utilized. A parallel implementation with per-sample gradient extraction (Goodfellow, 2015) is a pertinent acceleration technique for the future.
Conclusion We introduce the concept of introspection in neural networks as two separate stages in a network’s decision process - the first is making a quick assessment based on sensed patterns in data and the second is reflecting on that assessment based on all possible decisions that could have been taken and making a final decision based on this reflection. We show that doing so increases the generalization performance of neural networks as measured against distributionally shifted data while reducing the calibration error of neural networks. Existing state-of-the-art methods in downstream tasks like active learning and out-of-distribution detection perform better in an introspective setting compared to a feed-forward setting especially when the distributional difference is high.

heading: 8 ETHICS STATEMENT
text: The introspective explanations can serve to examine the intrinsic notions and biases that a network uses to categorize data since H(·) obtains its introspective answers through f(·). However, any internal bias present in f(·) only gets strengthened inH(·) through confirmation bias. The framework will benefit from a human intervention between f(·) and H(·) in sensitive applications. One way would be to ask counterfactual questions by providing an established counterfactual and asking the network to reflect based on that. While the introspective framework will remain the same, the features will change. Such a confirmation bias is also present in humans. In his seminal book in 2011, Kahneman (2011) outlines two systems of thought and reasoning in humans - a fast and instinctive ‘system 1’ that heuristically associates sensed patterns followed by a more deliberate and slower ‘system 2’ that examines and analyzes the data in context of intrinsic notions. Our framework derives its intuition based on these two systems of reasoning.

heading: 9 REPRODUCIBILITY STATEMENT
text: The paper uses publicly available datasets to showcase the results. Our introspective learning framework is built on top of existing deep learning apparatus - including ResNet architectures (He et al., 2016) (inbuilt PyTorch architectures), CIFAR-10C data (Hendrycks & Dietterich, 2019) (source code at https://zenodo.org/record/2535967#.YLpTF-1KhhE), calibration ECE and MCE metrics (Guo et al., 2017) (source code at https://github.com/markus93/NN_calibration), out-of-distribution detection metrics, and codes for existing methods were adapted from (Chen et al., 2020a) (source code at https://github.com/jfc43/robust-ood-detection), active learning methods and their codes were adapted from (Ash et al., 2019) (source code at https://github.com/JordanAsh/badge), Grad-CAM was adapted from (Selvaraju et al., 2017) (code used is at https://github.com/adityac94/Grad_CAM_plus_plus). Our own codes will be released upon acceptance. The exact training hyperparameters for f(·) andH(·), and all consideredH(·) architectures are shown in Appendix C.1. Extensive ablation studies onH(·) are shown in C.4.

heading: A APPENDIX : INTROSPECTION, REASONING, AND EXPLANATIONS
text: Introspection was formalized by Wundt (1874) as a field in psychology to understand the concepts of memory, feeling, and volition (Schwitzgebel, 2019). The primary focus of introspection is in reflecting on oneself through directed questions. While the directed questions are an open field of study in psychology, we use reasoning as a means of questions in this paper. Specifically, abductive reasoning. Abductive reasoning was introduced by the philosopher Charles Sanders Peirce (Peirce, 1931), who saw abduction as a reasoning process from effect to cause (Paul, 1993). An abductive reasoning framework creates a hypothesis and tests its validity without considering the cause. From the perspective of introspection, a hypothesis can be considered as an answer to one of the three following questions: a causal ‘Why P?’ question, a counterfactual ‘What if?’ question, and a contrastive ‘Why P, rather than Q?’ question. Here P is the prediction and Q is any contrast class. Both the causal and counterfactual questions require active interventions for answers. These questions
try to assess the causality of some endogenous or exogenous variable. However, introspection is the assessment of ones own notions rather than an external variable. Hence, a contrastive question of the form ‘Why P, rather than Q?’ lends itself as the directed question for introspection. Here Q is the introspective class. It has the additional advantage that the network f(·) serves as the knowledge base of notions. All reflection images from 1, Fig. 4, and Fig. 5 are contrastive. We describe the generation process of these post-hoc explanations.
Introspective Feature Visualization We modify Grad-CAM (Selvaraju et al., 2017) to visualize rj from Eq. 2. Grad-CAM visually justifies the decision made by f(·) by highlighting features that lead to ŷ. It does so by backpropagating the logit associated with the prediction, ŷ. The resulting gradients at every feature map are global average pooled and used as importance scores. The importance scores multiply the activations of the final convolutional layer and the resultant map is the Grad-CAM visualization. Hence, gradients highlight the activation areas that maximally lead to the prediction ŷ. In Fig. 1, given a spoonbill image x and a ImageNet-pretrained (Deng et al., 2009) VGG-16 network, the sensing visualization shown is Grad-CAM. Grad-CAM indicates that the pink and round body, and straight beak are the reasons for the decision. Instead of backpropagating the ŷ logit, we backpropagate J(yI , ŷ) in the Grad-CAM framework. The gradients represent introspective features and are used as importance scores. It can be seen that they visually highlight the explanations to ‘Why ŷ, rather than yI ’. In Fig. 1, the network highlights the neck of the spoonbill to indicate that since an S-shaped neck is not observed, x cannot be a flamingo. Similarly, the body of the spoonbill is highlighted when asked why x is not a crane since cranes have white feathers while spoonbills are pink. Two more examples are shown in Fig. 4. In the first row, a VGG-16 architecture is trained on Stanford Cars dataset (Krause et al., 2013). Given a Bugatti convertible image, Grad-CAM highlights the bonnet as the classifying factor. An introspective question of why it cannot be a bugatti coupe is answered by highlighting the open top of the convertible. The entire car is highlighted to differentiate the bugatti convertible from a Volvo. In the second row, we explore visual explanations in computed seismic images using LANDMASS dataset (Alaudah et al., 2018). A ResNet-18 architecture using the procedure from Shafiq et al. (2018) is trained. The dataset has four geological features as classes - faults, salt domes, horizons, and chaotic regions. Given a fault image in Fig. 4, Grad-CAM highlights the regions where the faults are clearly visible as fractures between rocks. However, these regions resemble salt domes as shown in the representative image. The introspective answer of why x is not predicted as a salt dome tracks a fault instead of highlighting a general region that also resembles a salt dome. Note that no representative images are required to obtain introspective visualizations. The gradients introspect based on notions of classes in network parameters.

heading: B APPENDIX : PROOFS

heading: B.1 PROOF FOR LEMMA 1
text: We start by assuming J(·) is a cross-entropy loss. J(yI , ŷ), I ∈ [1, N ] can also be written as,
J(yI , ŷ) = −yŷ + log N∑ j=1 eyj , where ŷ = f(x), ŷ ∈ <N×1. (12)
This definition is used in PyTorch to implement cross entropy. Here we assume that the predicted logit, i.e, the argument of the max value in the logits ŷ is yŷ. While training, yŷ is the true label. In this paper, we backpropagate any trained class I , as an introspective class. Hence, Eq. 12 can be rewritten as,
J(yI , ŷ) = −yI + log N∑ j=1 eyj , where ŷ = f(x), ŷ ∈ <N×1. (13)
Approximating the exponent within the summation with its second order Taylor series expansion, we have,
J(yI , ŷ) = −yI + log N∑ j=1 ( 1 + yj + y2j 2 ) . (14)
Note that for a well trained network f(), the logits of all but the predicted class are high. As noted before, the predicted logit is yŷ . Hence ∑N
j=1 y2j 2 = y2ŷ 2 . Substituting,
J(yI , ŷ) = −yI + log(N) + log( N∑ j=1 yj) + log ( y2ŷ 2 ) . (15)
The quantity in Eq. 15 is differentiated, hence nulling the effect of constant log(N ). For a well trained network f(·), small changes in W do not adversely affect the sum of all logits ∑N j=1 yj . Hence approximating its gradient to 0 and discarding it, we can obtain ∇WJ(yj , ŷ) as a function of two logits, yI and yŷ given by,
∇WJ(yI , ŷ) = −∇W yI +∇W log ( y2ŷ 2 ) . (16)
yI is a one-hot vector of dimensionality N ×1 while∇W is a dL−1×N matrix. The product extracts only the Ith filter in the W matrix in gradient calculations. Following the above logic for yŷ, we have, ∇WJ(yI , ŷ) = −∇W,IyI +∇W,yŷg(yŷ), (17) where g(·) is some function of yŷ . Hence the gradient rI = ∇WJ(yI , ŷ) lies in the span of the filter gradients of WI and Wŷ , making rI orthogonal to all other filter gradient pairs. Hence proven.
We demonstrate this sparsity in Fig. 5. A two-layer CNN is trained on MNIST (LeCun et al., 1998) dataset. MNIST is a handwritten digits dataset consisting of 50, 000 training images and 10, 000 testing images among 10 classes. Our two-layer CNN recognizes these digits with an averaged test accuracy exceeding 99%. The final fully connected layer in this network has a size of 50×10. Hence, the dimensionality of each filter WL,i, i ∈ [1, 10] that corresponds to a class is 50× 1. We provide an input image x of number 5 to a trained network as shown in Fig. 5. The network correctly identifies the image as a 5. We then backpropagate the introspective class 0 using the cross entropy loss J(5, 0) with P = 5 and Q = 0. This answers the question ‘Why 5, rather than 0?’. The gradient features in the final fully connected layer are the same dimensions as the final fully connected layer - 50× 10. This matrix is displayed as a normalized image in Fig. 5. Yellow scales to 1 and blue is −1 while green is 0. It can be seen that the only values present in the matrix are negative at WL,0, in blue, and positive in WL,5, in yellow. This validates Eq. 5 that for a fully-trained network the only values, and hence the only information, required from WL for Q = 0 is ∇WL,0 . We show the matrix∇WL when Q = 1, 2, 4, 5, 6. The difference among all matrices is the location of the negative values that exist at ∇WL,Q for different values of Q. Hence, for N introspective features in 5, the space complexity of rx which is a concatenation of N separate ri, reduces from O(dL−1 ×N2) to O(dL−1 ×N).

heading: B.2 PROOF FOR THEOREM 1
text: The proof for Theorem 1 follows from Lemma 1. For any given data x, there are N possible introspections and hence N possible reflections. The LHS in Eq. 17 is summed across N losses. Since yj , j ∈ [1, N ] are one-hot vectors, they are orthogonal and the first term in RHS is an addition across j. The second term in RHS is independent of j. Representing this in equation form, we have,
N∑ j=1 ∇WJ(yj , ŷ) = − N∑ j=1 ∇W,jyj +N ×∇W,yŷg(yŷ). (18)
The first term is added N times for N orthogonal yI . Hence, the first term reduces to a sum of all gradients of jth filters when backpropagating yj . Removing the summation and replacing yj = 1N or a vector of all ones in the LHS, we still have the same RHS given by,
∇WJ(1N , ŷ) = − N∑ j=1 ∇W,jyj +N ×∇W,yŷg(yŷ). (19)
Equating the LHS from Eq. 18 and Eq. 19, we have the proof.

heading: B.3 TRADEOFF IN EQ. 8
text: Eq. 8 suggests a trade-off between minimizing E[(H(rx)−y)2], which is the cost function for training H(·), and the variance of the networkH(·). Ideally, an optimal point exists that optimally minimizes the cost function of H(·) while maximizing its variance. This also prevents decomposing H(·) into H1(·) and H2(·) that further introspect on H()̇. In this paper, we create a single introspective networkH(·). Hence, we do not comment further on the practical nature of the trade-off or perpetual introspection. It is currently beyond the scope of this work. In all experiments, we trainH(·) as any other network feed-forward network - by minimizing an empirical loss function given the ground truth.

heading: B.4 FISHER VECTOR INTERPRETATION
text: We make two claims before Eq. 10 both of which are well established. These include :
• Variance of a linear function For a linear function y = W × x + b, the variance of y is given by Var(Wx+ b) = W 2Var(x) if Var(W ) = 0.
• Variance of a linear function when W is estimated by gradient descent Ignoring the bias b, and taking y = Wx = xT Σ−1xT (xW ), we have Var(Wx) = σ2Tr(xT Σ−1x).
Both these results lead to Eq. 9. Since rx ∈ <dL−1×N , the trace of the matrix given by Tr(rTx F−1rx), is a sum of projections on individual weight gradients given by ∑N j=1 r T j F −1rj in the Fisher sense.

heading: C APPENDIX : ADDITIONAL RESULTS

heading: C.1 STRUCTURE OF H(·) AND TRAINING DETAILS
text: In this section, we provide the structure of the proposedH(·) architecture. Note that, from Eq. 1, the yfeat in feed-forward learning are processed through a linear layer. We process the introspective features rx through an MLPH(·), whose parameter structure is given in Table 3. Hence, we follow the same workflow as feed-forward networks in introspective learning. The feed-forward features fL−1(x) are passed through the last linear layer in f(·) to obtain the prediction ŷ. The introspective features are passed through an MLP to obtain the prediction ỹ. The exact training procedure forH(·) is presented below.
Training f(·) and Hyperparameters We train four ResNet architectures - ResNet-18, 34, 50, and 101 He et al. (2016). Note that we are not using any known techniques that promote either generalization (training on noisy data (Vasiljevic et al., 2016)) or calibration (Temperature scaling (Guo et al., 2017)). The networks are trained from scratch on CIFAR-10 dataset which consists of 50000 training images with 10 classes. The networks are trained for 200 epochs using SGD optimizer with momentum = 0.9 and weight decay = 5e − 4. The learning rate starts at 0.1 and is changed as 0.02, 0.004, 0.0008 after epochs 60, 120, and 160 respectively. PyTorch in-built Random Horizontal Flip and standard CIFAR-10 normalization is used as preprocessing transforms.
TrainingH(·) The structures of all MLPs are shown in Table 3. ResNet-18,34 trained on CIFAR-10 provide rx of dimensionality 640×1. This is fed intoH(·) which is trained to produce a 10×1 output. Note that rx from ResNet-50,101 are of dimensionality 2560× 1 - due to larger dimension of fL−1(). All MLPs are trained similar to f(·) - for 200 epochs, SGD optimizer, momentum = 0.9, weight decay = 5e−3, learning rates of 0.1, 0.02, 0.004, 0.0008 in epochs 1− 60, 61− 120, 121− 160, 161− 200 respectively. For the larger 5-layered ResNet-50,101 networks in Table 8, dropout with 0.1 is used and the weight decay is reduced to 5e−4.
C.2 INTROSPECTIVE ACCURACY GAIN AND CALIBRATION ERROR STUDIES
In this section, we present additional recognition and calibration results. In Fig. 3a), we showed distortion-wise accuracy and the introspective gain for ResNet-18. In this section, we present levelwise and network-wise accuracies for all four considered ResNet architectures. We show that an introspective ResNet-18 matches a Feed-Forward ResNet-50 in terms of recognition performance. We then compare the results of ResNet-18 against existing techniques that promote robustness. We show that introspection is a plug-in approach that acts on top of existing methods and provides gain. We do the same for calibration experiments on CIFAR-10C where we provide level-wise distortion-wise graphs for Expected Calibration Error (ECE) similar to Fig. 3b).

heading: C.2.1 LEVEL-WISE RECOGNITION ON CIFAR-10C
text: In Fig. 6b), the introspective performance gains for the four networks are categorized based on the distortion levels. All 19 categories of distortion on CIFAR-10C are averaged for each level and their respective feed-forward accuracy and introspective gains are shown. Note that the levels are progressively more distorted. Hence, level 1 distribution X ′ is similar to the training distribution X when compared to level 5 distributions. As the distortion level increases, the introspective gains also increase. This is similar to the results from Section 6. In both active learning and OOD applications as X ′ deviates from X , introspection performs better. In Fig. 6a), we show the distortion-wise and
level-wise increase for each network. Note that, an Introspective ResNet-18 performs similarly to a Feed-Forward ResNet-50.

heading: C.2.2 DISTORTION-WISE AND LEVEL-WISE RECOGNITION ON CIFAR-10C
text: In Fig. 7, the introspective accuracy performance for Resnet-18 across 19 distortions and 5 distortion levels is shown. Note that CIFAR-10C consists of 950,000 test images. The 4% increase in performance translates to around 35,000 more images correctly classified over its feed-forward counterpart. These gains are especially visible among Level 5 distortions.
C.2.3 INTROSPECTION AS A PLUG-IN ON TOP OF EXISTING TECHNIQUES
Several techniques exist that boost the robustness of neural networks to distortions. These include training with noisy images (Vasiljevic et al., 2016), training with adversarial images (Hendrycks & Dietterich, 2019), and self-supervised methods like SimCLR (Chen et al., 2020b) that train by augmenting distortions. Another commonly used technique is to pre-process the noisy images to denoise them. All these techniques can be used to train f(·). Our proposed framework sits on top of
any f(·). Hence, it can be used as a plug-in network. These results are shown in Table 4. Denoising 19 distortions is not a viable strategy assuming that the characteristics of the distortions are unknown. We use Non-Local Means denoising and the results obtained are lower than the feed-forward accuracy by almost 3%. However, introspecting on this model increases the results by 3.84%. We create untargeted adversarial images using I-FGSM attack with α = 0.01 and use them to train a ResNet-18 architecture. In our experiments this did not increase the feed-forward accuracy. Introspecting on this network provides a gain of 2.84%. SimCLR Chen et al. (2020b) and introspection on SimCLR is discussed in Section C.3. In the final experimental setup of augmenting noise (Vasiljevic et al., 2016), we augment the training data of CIFAR-10 with six distortions - gaussian blur, salt and pepper, gaussian noise, overexposure, motion blur, and underexposure - to train a ResNet-18 network f ′(·). We use the noise characteristics provided by (Temel et al., 2018) to randomly distort 500 CIFAR-10 training images by each of the six distortions. The original training set is augmented with the noisy data and trained. The results of the feed-forward f ′(·) show a substantial increase in performance to 76.86%. This is about 9% increase from the original architecture. We show that introspecting on f ′(·) provides a further gain in accuracy of 1.12%. Note that to train H(·), we do not use the augmented data. We only use the original CIFAR-10 undistorted training set. The gain obtained is by introspecting on only the undistorted data, even though f ′(·) contains knowledge of the distorted data. Hence, introspection is a plug-in approach that works on top of any network f(·) or enhanced network f ′(·). Augmix (Hendrycks et al., 2019) is currently the best performing technique on CIFAR-10C. It creates multiple chains of augmentations to train the base WideResNet network. On CIFAR-10C, f ′(·) obtains 89.85% recognition accuracy. We use f ′(·) as our base sensing model and train an introspective MLP on f ′(·). Note that we do not use any augmentations for trainingH(·). Doing so, we obtain a statistically similar accuracy performance of 89.89%. However, the expected calibration error of the feed-forward f ′(·) model decreases by 43.33% after introspection. Hence, when there is no accuracy gains to be had, introspection provides calibrated models.

heading: C.2.4 EXPECTED CALIBRATION ERROR (ECE)
text: In Fig. 3b), we show ECE for two distortion types - brightness and saturation across 5 distortion levels. In Fig. 8, we show results across five distortion levels for the first 12 distortions. The blue plot is the Feed-Forward ECE while the lower orange plot is its introspective counterpart. Apart from Level 5 contrast, intrsopective ResNet-18 is more calibrated than its feed-forward counterpart. This is in addition to the performance gains. The trend remains the same in the remaining distortions and among all considered networks. We average out ECE across 19 distortions and 5 challenge
levels and provide ECE results for ResNets-18, 34, 50, 101 in Table 5. Lower the error, better is the architecture. The proposed introspective framework decreases the ECE of its feed-forward backbone by approximately 42%. An additional metric called Maximum Calibration Error (MCE) is also used for comparison. While ECE averages out the calibration difference in all bins (From Section 6), MCE takes the maximum error among all bins Guo et al. (2017). The introspective networks outperform their feed-forward backbones among all architectures when compared using ECE and MCE.

heading: C.3 SIMCLR AND INTROSPECTION
text: SimCLR Chen et al. (2020b) is a self-supervised contrastive learning framework that is robust to noise distortions. The algorithm involves creating augmentations of existing data including blur, noise, rotations, and jitters. The network is made to contrast between all the augmentations of the image and other images in the batch. A separate network head g(·) is placed on top of the network to extract features and inference is made by creating a similarity matrix to a feature bank. Note that g(·) is a simple MLP. Our proposed framework is similar to SimCLR in that we extract features and use an MLPH(·) to infer from these features. In Table 4, we show the results of Introspecting ResNets against SimCLR. However, this comparison is unfair since the features in SimCLR are trained in a self-supervised fashion. In this section, we train SimCLR for ResNets-18, 34, 50, 101 and train a new MLP g(·), not for extracting features, but to classify images. In other words, in Chen et al. (2020b), the authors create g(·) to be a 512× 128 layer that extracts features. We train a network of the form 512× 128− 128× 10 that is trained to classify images. We then introspect on this g(·) to obtain rx. Hence, our extracted features are a result of introspecting on self-supervision. Note that g(·) is now a fully supervised network. We pass CIFAR-10C through g(·) and name it SimCLR-MLP in Table 6. It is unsurprising that the fully-supervised SimCLR-MLP beats the self-supervised SimCLR across all four ResNets. The introspective network is called SimCLR-Introspective in Table 6. Note that there is less than 1% recognition performance increase across networks compared to SimCLR-MLP. Hence, the performance gains for introspecting on SimCLR-MLP is not as high as base ResNet
architectures from Table 4. One hypothesis for this marginal increase is that the notions created within SimCLR-MLP are predominantly from the self-supervised features in SimCLR. These may not be amenable for the current framework of introspection that learns to contrast between classes and not between features within-classes.

heading: C.4 ABLATION STUDIES
text: The feature generation process in Section 2 is dependent on the loss function J(ŷ, y). In this section, we analyze the performance of our framework for commonly used loss functions and show that the introspective network outperforms its feed-forward counterpart under any choice of J(ŷ, y). We also ascertain the effect of the size of the parameter set inH(·) on performance accuracy.

heading: C.4.1 EFFECT OF LOSS FUNCTIONS
text: We extract rx using 9 loss functions and report the final distortion-wise level-wise averaged results Table 7. We do so for ResNet-18 and for the architecture ofH(·) shown in Table 3. The following loss functions are compared : CE is Cross Entropy, MSE is Mean Squared Error, L1 is Manhattan distance, Smooth L1 is the leaky extension of Manhattan distance, BCE is Binary Cross Entropy, and NLL is Negative Log Likelihood. Notice that the performance of rx extracted using all loss functions exceed that of the feed-forward performance. The shown results of MSE, L1-M and Smooth L1-M are obtained by backpropagating a 1N from Theorem 1 vector multiplied by the average of all maximum logits M , in the training dataset. We use M instead of 1 because we want the network to be as confidant of the introspective label yI as it is with the prediction label ŷ. Note that the results in Table 7 are for CIFAR-10C. MSE-M outperforms NLL loss by 0.37% in average accuracy and is used in our experiments.
C.4.2 EFFECT OF H(·)
We conduct ablation studies to empirically show the following : 1) the design of H(·) does not significantly vary the introspective results, 2) the extra parameters in H(·) are not the cause of increased performance accuracy.
How does changing the structure ofH(·) change the performance? We vary the architecture of H(·) from a single linear layer to 4 layers in the first half of Table 8 for ResNet-18. The results in the first three cases are similar. A four layered network performs worse than f(·). However, changing the weight decay from 5e−3 to 5e−4 during training increases the results to above 70% but does not beat the smaller networks. For ResNet-18 architecture, the highest results are obtained whenH(·) is a 2-layered architecture but for the sake of uniformity, we use the results from a 3-layered network across all ResNet architectures.
Are the extra parameters in H(·) the only cause for increase in performance accuracy? We show an ablations study of the effect of structure of H(·) and f(·) on the introspective and feedforward results in the second half of Table 3 on CIFAR-10-C dataset. The results are divided into four sections. In the first section, we show the performance of the original feed-forward network f(·), the performance when the final layer, fL(·) is retrained using features yfeat from Eq. 1, and the introspective network when H(·) is a single layer. The second section shows the results when the features yfeat are used to train a three layered network fL(·), and the introspective network is also three layered. Finally, in section 3, we try to equate the number of parameters for fL(·) and H(·). Note that in all cases, fL(·) andH(·) are trained in the same manner as detailed in Section C.1. H(·) beats the performance of fL(·) among all ablation studies. Finally, similar to SimCLR, we forego using an MLP and use 10-Nearest Neighbors on yfeat (64 × 1) and rx (640 × 1) for predictions. Both results are worse-off than their MLP results but rx outperforms yfeat.
C.5 INTROSPECTIVE ACCURACY ACROSS TRAINING EPOCHS
In Section 2, we make the assumption that f(·) is well trained to approximate rx using Theorem 1. In Section 3, the Fisher Vector analysis works when the gradients form distances across the manifold in f(·) which occurs if f(·) is well trained. In this section we show that, practically, introspection performs as well as feed-forward accuracy across training epochs on CIFAR-10 testset and outperforms feed-forward accuracy on CIFAR-10C distortions. We show results on original testset, gaussian noise and motion blur testsets in Fig. 9.
Training, Testing, and Results in Fig. 9a In this experimental setup, ResNet-18 is trained for 200 epochs. The model states at multiples of 3 epochs from 1 to 200 are stored. This provides 67 states of f(·) along its training process. Each f(·) is tested on CIFAR-10 testset and the recognition accuracy is plotted in blue in Fig. 9a). The introspective features rx for all 67 states are extracted for the 50, 000 training samples. These rx are used to train 67 separate H(·) of structure provided in Table 3 with a similar training setup as in Section C.1. The rx from the 10, 000 testing samples are extracted individually for each of the 67 f(·) states and tested. The results are plotted in red in Fig. 9a). Note the sharp spikes at epochs 60 and 120 where there is a change in the learning rate. Hence, when training and testing distributions are similar, introspective and feed-forward learning provides statistically similar performance across varying states of f(·).
Training, Testing, and Results in Fig. 9b, c We now consider the case when a network f(·) is trained on distribution X and tested on X ′ from CIFAR-10C distortions. The 67 trained models of ResNet-18 are tested on two distortions from CIFAR-10C. From the results in Fig. 3, introspective learning achieves one of its highest performance gains in Gaussian noise, and an average increase in motion blur after epoch 200. The results in Fig. 9 indicate that after approximately 60 epochs, the feed-forward network has sufficiently sensed notions to reflect between classes. This is seen in the performance gains in both the motion blur and Gaussian noise experiments.
Training of H on a separate validation set in Fig. 10b In all experiments, the introspective network H(·) is trained on the same training set as f(·). In Fig. 10, we show the results when the introspective network is trained on a separate portion of the dataset. We use 40, 000 images to train f(·) and 10, 000 to trainH(·) both of which are randomly chosen. We follow the training procedure from before. The model states at multiples of 3 epochs from 1 to 200 are stored. This provides 67 states of f(·) along its training process. Each f(·) is tested on CIFAR-10 testset and the recognition accuracy is plotted in blue in Fig. 10b). TheH(·) at each iteration on the other hand is trained with the 10, 000 images. However, it has access to the notions created from the remaining 40, 000 images and hence the results for introspection match Fig. 9a) which is reproduced in Fig. 10a). The feed-forward results catch up to the introspective results around epoch 60. At Epoch 120, we add back the 10, 000 held-out images into the training set of f(·) and the results match between Fig. 10a) and Fig. 10b).

heading: C.6 DOMAIN ADAPTATION ON OFFICE DATASET
text: In Section 3, we claim that introspection helps a network to better classify distributions that it has not seen while training. In Section 5, we tested on 95 new distributions in CIFAR-10C and 30 new distributions in CIFAR-10-CURE. In this section, we evaluate the efficacy of introspection when there is a domian shift between training and testing data under changes in background, and camera
acquisition setup among others. Specifically, the robust recognition performance ofH(·) is validated on Office Saenko et al. (2010) dataset using Top-1 accuracy. The Office dataset has 3 domains - images taken from either Webcam or DSLR, and extracted from Amazon website. Images can belong to any of 31 classes and they are of varying sizes - upto 1920 × 1080. Hence, results on Office shows the applicability of introspection on large resolution images. ImageNet pre-trained ResNet-18,34,50,101 He et al. (2016) architectures are used for f(·). The final layer is retrained using the source domain while the remaining two domains are for testing. The experimental setup, the same detailed in Section 5, is applied and the Top-1 accuracy is calculated. The results are summarized in Table 9. In every instance, the top domain is X - the training distribution, and the bottom domain is X ′ - the testing distribution. Note that ResNet-50 and 101 failed to train on 498 images in DSLR source domain. The results of introspection exceed that of feed-forward learning in all but ResNet-50 when classifying between Webcam and Amazon domains.

heading: C.7 DOMAIN ADAPTATION ON VIS-DA DATASET
text: Validation results on a synthetic-to-real domain shift dataset called VisDA Peng et al. (2017) are presented in Table 10. VisDA has 12 classes with about 152, 000 synthetic training images, and 55, 000 real validation images. The validation images are cropped images from MS-COCO. ResNet18 architecture pretrained on ImageNet is finetuned on the synthetically generated training images from VisDA dataset fro 200 epochs. It is then tested on the validation images and the recognition performance is shown in Table 10 as feed-forward f(·) results. Introspective H(·) results are obtained and shown when f(·) is ResNet-18. There is an overall improvement of 5.48% in terms of performance accuracy. However, the individual class accuracies leave room for improvement.

heading: C.8 ACTIVE LEARNING
text: In Table 1, the mean recognition accuracies across the first 20 rounds of Active Learning experiments for commonly used query strategies are shown. We plot these recognition accuracies across for all five query strategies in Fig. 11. The x-axis is the round at which the performance is calculated. The calculated accuracy is plotted on the y-axis. The experiment starts with a random 100 images in round 1. Each strategy queries using either a round-wise sample trained f(·) or a round-wise sample trainedH(·). Note that at each round, the networks are retrained. This continues for 20 rounds. Both BALD Gal et al. (2017) and BADGE Ash et al. (2019) applied on H(·) consistently beat its f(·) counterpart on every round. This is because both these methods rely on extracting features from the network as compared to the other three techniques that directly use the output logits from eitherH(·) or f(·). Since the network is not well-trained at the initial stages - due to a dearth of training data - the introspective network is not as consistent as the feed-forward network among Entropy, Least
Confidence, and Margin strategies. Nonetheless,H(·) outperforms f(·) on average across all rounds.

heading: C.9 OOD
text: Adversarial setting in Table 2 A datapoint z, is perturbed as z+ and the goal of the detector is to classify z ∈ X or z ∈ X ′. This modality is proposed by the authors in Chen et al. (2020a) and we use their setup. PGD attack with perturbation 0.0014 is used. The same MSP and ODIN detectors from Table 2 are utilized. On 4 OOD datasets, both MSP and ODIN show a performance gain across all three metrics onH(·) compared to f(·). Note that the results in Table 11 is for ResNet-18 architecture for the same f(·) andH(·) used in other experiments including Fig. 2.
Vanilla setting in Table 11 In Table 11, we show the results of out-of-distribution detection when X is CIFAR-10 and X ′ are the four considered datasets. Note that among the four datasets, textures and SVHN are more out-of-distribution from CIFAR-10 than the natural image datasets of Places365 and LSUN. The results of the introspective network is highest on Textures DTD dataset.
C.10 IMAGE QUALITY ASSESSMENT
Related Works Multiple methods have been proposed to predict the subjective quality of images including PSNR-HA (Ponomarenko et al., 2011), IW-SSIM (Wang & Li, 2011), SR-SIM (Zhang & Li, 2012), FSIMc (Zhang et al., 2011), PERSIM (Temel & AlRegib, 2015), CSV (Temel & AlRegib, 2016), SUMMER (Temel & AlRegib, 2019), and UNIQUE (Temel et al., 2016). All these methods extract structure related hand-crafted features from both reference and distorted images and compare them to predict the quality. Recently, machine learning models have been proposed to directly extract features from images (Temel et al., 2016). Temel et al. (2016) propose UNIQUE that uses a sparse autoencoder trained on ImageNet to extract features from both reference and distorted images. We use UNIQUE as our base network f(·).
Feed-Forward UNIQUE Temel et al. (2016) train a sparse autoencoder with a one layer encoder and decoder and a sigmoid non-linearity on 100, 000 patches of size 8 × 8 × 3 extracted from ImageNet testset. The autoencoder is trained with MSE reconstruction loss. This network is our f(·). UNIQUE follows a full reference IQA workflow which assumes access to both reference and distorted images while estimating quality. The reference and distorted images are converted to YGCr color space and converted to 8×8×3 patches. These patches are mean subtracted and ZCA whitened before being passed through the trained encoder. The activations of all reference patches in the latent space are extracted and concatenated. Activations lesser than a threshold of 0.025 are suppressed to 0. The choice of threshold 0.025 is made based on the sparsity coefficient used during training. Similar procedure is followed for distorted image patches. The suppressed and concatenated features of both the reference and distorted images are compared using Spearman correlation. The resultant is the feed-forward estimated quality of the distorted image.
Introspective-UNIQUE We use the architecture and the workflow from Temel et al. (2016) which is based on feed-forward learning to demonstrate the value of introspection. We replace the feedforward features with the proposed introspective features. The loss in Eq. 6 for introspection is not between classes but between the image x and its reconstruction x̃ from the sparse autoencoder from Temel et al. (2016). For a reference image x, rx is derived using J(x, x̃). Hence, gradients of rx span the space of reconstruction noise. Since the need in IQA is to characterize distortions, we obtain rx for reference images from the first layer and project both reference and distorted images onto rx. These projections are compared using Spearman correlation to assign a quality estimate. In this setting,H(·) is the projection operator and Spearman correlation. Hence, Introspective-UNIQUE broadens introspection in the following ways - 1) defining introspection on generative models, 2) using gradients in the earlier layers of a network.
Results We report the results of the proposed introspective model in comparison with commonly cited methods Table 12. We utilize MULTI-LIVE (MULTI) (Jayaraman et al., 2012) and TID2013 (Ponomarenko et al., 2015) datasets for evaluation. The performance is validated using outlier ratio (consistency), root mean square error (accuracy), Pearson correlation (linearity), Spearman correlation (rank), and Kendall correlation (rank). Arrows next to each metric in Table 12 indicate the desirability of a higher number (↑) or a lower number(↓). Two best performing methods for each metric are highlighted. The proposed framework is always in the top two methods for both datasets in all evaluation metrics. In particular, it achieves the best performance for all the categories except in OR and KRCC in TID2013 dataset. The feed-forward model does not achieve the best performance for any of the metrics in MULTI dataset. However, the same network using introspective features significantly improves the performance and achieves the best performance on all metrics. For instance, the feed-forward model is the third best performing method in MULTI dataset in terms of RMSE, PLCC, SRCC, and KRCC. However, the introspective features improve the performance for those metrics by 1.315, 0.036, 0.020, and 0.023, respectively and achieve the best performance for all metrics. This further reinforces the plug-in capability of the proposed introspective inference.

Review summaries:
The authors present a novel 2-stage technique to improve the classification accuracy of feedforward ANNs.  In particular, after the feedforward pass, a so-called introspective stage occurs, the goal of which is to ascertain why the particular class label was provided rather than a different label.  This stage, which tends to improve the accuracy of the class predictions, is modular and can be added onto networks under varying task conditions.

In this paper, the authors propose a modification to standard neural networks used for object classification tasks to incorporate what they call “introspective learning”. This consists on training a multi-layer perceptron (MLP) on the neural network introspective features. These are obtained by calculating the gradients over the last layer weights of the model on a loss function corresponding to posing an introspective question: why is the correct label A instead of B? The authors apply this procedure to several neural networks of the ResNet model family and show that the introspective-networks have better generalization to distributional shifts and smaller calibration errors on datasets with the same classes and image sizes as CIFAR-10. Finally, they show further improvements in multiple applications.

The authors present a new inference pipeline for neural networks $f$ trained to solve classification problems. They augment the features processed by $f$ with the loss gradients with respect to the weights of the last layer. Their pipeline requires two steps: the first one coincides with a standard evaluation of $f(x)$, while gradients are evaluated and processed in the second step to derive the final prediction. Experiments show slight improvements in classification accuracy and cuts of calibration errors.