Enhancing Visual Representations for Efficient Object Recognition during Online Distillation
Authors: Shashanka Venkataramanan
Bruce W McIntosh
Abhijit Mahalanobis
Venue:  (2021)

Abstract: We propose ENVISE,  an online distillation framework that ENhances VISual representations for Efficient object recognition. We are motivated by the observation that in many real-world scenarios, the probability of occurrence of all classes is not the same and only a subset of classes occur frequently. Exploiting this fact, we aim to reduce the computations of our framework by employing a binary student network (BSN) to learn the frequently occurring classes using the pseudo-labels generated by the teacher network (TN) on an unlabeled image stream. To maintain overall accuracy, the BSN must also accurately determine when a rare (or unknown) class is present in the image stream so that the TN can be used in such cases. To achieve this, we propose an attention triplet loss which ensures that the BSN emphasizes the same semantically meaningful regions of the image as the TN. When the prior class probabilities in the image stream vary, we demonstrate that the BSN adapts to the TN faster than the real-valued student network. We also introduce Gain in Efficiency (GiE), a new metric which estimates the relative reduction in FLOPS based on the number of times the BSN and TN are used to process the image stream. We benchmark CIFAR-100 and tiny-imagenet datasets by creating meaningful inlier (frequent) and outlier (rare) class pairs that mimic real-world scenarios. We show that ENVISE outperforms state-of-the-art (SOTA) outlier detection methods in terms of GiE, and also achieves greater separation between inlier and outlier classes in the feature space.

Under review as a conference paper at ICLR 2021
ENHANCING VISUAL REPRESENTATIONS FOR EFFI- CIENT OBJECT RECOGNITION DURING ONLINE DISTIL- LATION
Anonymous authors Paper under double-blind review
ABSTRACT
We propose ENVISE, an online distillation framework that ENhances VISual rep- resentations for Efficient object recognition. We are motivated by the observation that in many real-world scenarios, the probability of occurrence of all classes is not the same and only subset of classes occur frequently. Exploiting this fact, we aim to reduce the computations of our framework by employing a binary student network (BSN) to learn the frequently occurring classes using the pseudo-labels generated by the teacher network (TN) on an unlabeled image stream. To main- tain overall accuracy, the BSN must also accurately determine when rare (or unknown) class is present in the image stream so that the TN can be used in such cases. To achieve this, we propose an attention triplet loss which ensures that the BSN emphasizes the same semantically meaningful regions of the image as the TN. When the prior class probabilities in the image stream vary, we demonstrate that the BSN adapts to the TN faster than the real-valued student network. We also introduce Gain Efficiency (GiE), new metric which estimates the relative reduction in FLOPS based on the number of times the BSN and TN are used to process the image stream. We benchmark CIFAR- 100 and tiny-imagend datasets by creating meaningful inlier (frequeen) and outlier (rare) class pairs that mimic real-world scenarios. We show that ENVISE outperforms state-of-the-ar (SOTA) outlier detection methods in terms of GiF, and also achieves greater separation between inlier and outlier classes in the feature space.
INTRODUCTION
Deep CNNs that are widely used for image classification (Huang al. (2017) often require large computing resources and process each image with high computational complexity (Livni et al. (2114). In real-world scenarios, the prior probability occurrence individual classes in an image stream is often unknown and varies with the deployed environment. For example, in zoo, the image stream input to the deep CNN will mostly consist of animals, while that of vehicles would be rare. Other object classes such as furniture and aircraft would be absent. Therefore, only subset of the many classes known to a deep CNN may be presented to it for classification during its deployment.
To adapt to the varying prior class probability in the deployed scenario with high efficiency, propose an online distillation framework ENVISE. Here, we employ high capacity general purpose image classifier network. For greater efficiency and faster convergence, we require the coefficients of the SN to be binary and refer to it as the binary student network (BSN). When the BSN first deployed, is trained on the unlabeled image stream using the predicted labels of the TN as pseudo-labels. Once the BSN converges to the performance the TN, used as the primary classifier rapidly classify the frequent classes faster than the TN. However, if rare class appears (i.e. a class absent during online training) in the image stream. the BSN must accurately detect it as a class it has not yet encountered, which is then processed by the TN. Since the BSN is trained only on the frequent classes, refer to these classes as inlier (IL) and the rare classes as outlier (OL). is important to note, that the OL classes are outliers with respect to the BSN only, but known to the TN. Detecting extremely rare classes which are unknown to both the BSN and TN (global unknowns) is beyond the scope this paper. Thus, assuming that the TN knows all possible classes from the deployed environment,
the leacher network (TN), while the student network (SN) is low capacity

---

Under review as a conference paper at ICLR 2021
aim to increase the overall efficiency of the system (without sacrificing performance) by exploiting the higher probability of occurrence of frequent classes in a given scenario.
Our approach for detecting OL classes is motivated from the observation by Ren et al. (2019), where networks incorrectly learn to emphasize the background rather than the semantically important regions of the image leading poor understanding of the IL. classes We know that attention maps highlight the regions of the image responsible for the classifier's prediction (Selvaraju et al. (2017)) We empirically observe that, the attention map of the BSN may focus on the background even when the attention map of the TN emphasizes the semantically meaningful regions of the image. In doing so, the BSN memorizes the labels of the TN, making it difficult to differentiate between the representations of IL and OL classes. To mitigate these issues, we propose an attention triplet loss that achieves two key objectives- a) guide the attention map of the correct prediction of the BSN to focus on the semantically meaningful regions, and b) simultaneously ensure that the attention map from the correct and incorrect predictions of the BSN are dissimilar. We show that by focusing on the semantically relevant regions of the image, the BSN will learn to distinguish between the representations of IL and OL classes, thereby improving its ability to detect OL classes
To assess the overall gain in efficiency of ENVISE, we propose new evaluation metric- GiE, based on the number of times the BSN and TN are used to process the image stream. Since the deployed scene is comprised mostly of IL classes with few OL classes, we expect the BSN to be employed most of the time for classifying II. classes. The TN is used rarely i.e. only when the BSN detects an OL class. We refer to efficiency the overall reduction in FLOPs in the online distillation framework to process the varying prior probability of classes in the image stream. This term differs from conventional model compression techniques (Frankle & Carbin (2019); Chen et al. (2020)) which process the image stream comprising of classes with equal probability using a single compressed model. To the best of our knowledge, we are the first to propose supervision on attention maps for OL detection, and new evaluation metric that measures the gain in computational efficiency of an online distillation framework. A summary of our main contributions is:
Faster convergence of BSN: We theoretically justify and empirically illustrate that the BSN adapts to the performance of the TN faster than the real-valued SN (RvSN). We also demonstrate the faster convergence of the BSN for different BSN architectures over its corresponding RvSN.
Attention triplet loss (Lac), which guides the BSN to focus on the semantically meaningful regions of the image, thereby improving OL detection.
New evaluation metric GiE to measure the overall gain in computational efficiency of the online distillation framework, and
We benchmark CIFAR-100 and tiny-imagenet datasets with SOTA OL detection meth- ods by creating meaningful IL and OL class pairs. ENVISE outperforms these baseline methods for OI. detection, improves separation of II. and OL classes in the feature space, and yields the highest gain in computational efficiency.
2 RELATED WORK
Distilling knowledge to train a low-capacity student network from a high- capacity teacher network has been proposed as part of model compression (Hinton et al. (2015)). Wang & Yoon (2020) provide a detailed review of different knowledge distillation methods. Mullapudi al. (2019) propose an online distillation framework for semantic segmentation in videos, while Abolghasemi et al. (019)) use knowledge distillation to augment a visuomotor policy from visual attention. Lin et al. (2019) propose an ensemble student network that recursively learns from the teacher network in closed loop manner while Kim et al. (2019) use feature fusion module to distill knowledge. [ao et al. (2019) propose online mutual learning and Cioppa et al. (2019) propose to periodically update weights to train an ensemble of student networks. These ensemble based methods require large computational resources and are expensive to train. However, ENVISE involves training compact model that mimics the performance of the TN with less computation.
Online distillation :
Outlier detection: Outlier detection or out-of-distribution detection (OOD) refers to detecting a sample from unknown class (Hendrycks & Gimpel (2016)). G2mpel(2016)) Existing SOTA OOD methods use outlier samples during training or validation. Yu & Aizawa (2019) increase the distance between II. and OL samples during training while Vyas et al. (2018) Lee al. (2018) add perturbations
2

---

Under review as a conference paper at ICLR 2021
onto test images and train their network to differentiate between IL and OL samples. However in ENVISE, the BSN is adaptively trained only on IL class images without any knowledge of OL class images, which is more realistic scenario. Without using outlier classes during training, [Hendrycks & Gimpel| (2016); Papadopoulos et al. (2019): [Bendale & Boult (016); [Hendrycks et al. (2019) 19) employ class probabilities or measure the similarity between II. and OI. features. Yoshihashi et al. (2019) generate discriminative feature space using reconstruction based approaches. However, the likelihood of correct classification by these networks based on background regions as described in Ren et al. 2019). Hence, these methods fail when applied to real-world IL and OL class pairs since the confidence of prediction is based on background information. We overcome this drawback by proposing supervision on attention maps where we guide the BSN to focus on the important semantic regions using the attention map of the teacher network as a reference. This enables ENVISE to classify II. images with high confidence while also improving OL detection.
Teacler network Joneuanee over classusi
aftertion &bbdss from Tracheer nelwork
Use teacher network
ontension maps
Inpur
Loss
ucd weightte
inutiims leomiing
(a((was))
L
Loga
learning
au
Ippu Imaege
Conwerged
Yos
Outlir
Binary weighs
bij (win)
Outler ?
Binary studiot oporation
Biewry CNN
Trairing
(a)
Inference
(b)
Figure 1: (a) The ENVISE architecture adaptively trains the BSN from predictions of the TN, (b) The process updates real valued weights that minimize the error produced by their binarized version.
3 ONLINE DISTILLATION FOR EFFICIENT OBJECT RECOGNITION
The framework of ENVISE is shown in Figure (a). Initially, the TN (indicated by "1" in the figure) classifies the images from the input stream with high accuracy, albeit slower rate. The BSN (red dotted box) comprises of an adaptation module ("22") and binary CNN ("3"). Given an image stream, the adaptation module learns the optimum binary weights to mimic the performance of the TN. Once its accuracy converges to that of the TN, the adaption of the real-valued weights stops, but the inference with binary weights continues. The OL, detector ("4"") uses the softmax output of the BSN confidence to either generate the final prediction (if the confidence is high), treats the image as an OL class and redirects it to the TN for classification.
Binarization algorithm for student network: During adaptive training, the BSN does not have access to the labeled data and mimics the behavior of the TN on the IL classes as shown in TigureTby For an input image x, let f(((w) weights of tt kernel in the 1th layer, and g(.) is binarizing function for these weights. During adaptive training, the error between the output of the BSN and TN is minimized by optimizing wh, = arg min ((ws,;) m
represent the output of the BSN where Way are real-values
[|Z.],, where L is the overall loss function used for training the BSN. While the intermediate weight Waj is computed during adaptive learning, the binary versions of these weights w g are used for fast computation. After convergence, the real valued weights are discarled, and only the binary weights are used. The question becomes what is good choice for the binarization function g-)? Following Rastegari et al. (2016). we find a binary vector by (whose 1), and a non-negative scalar oi, such that wi,; = .... is the best approximation of Wa,j in a minimum squared error sense, where as.j = 1 However, unlike Rastegari et al. (2016) where standalone network is binarized, our BSN learns the w during adaptive training in an online distillation framework. In doing so, the BSN compensates for the effects of binarization the overall loss function.
elements are either +1
N [W b = sign (w.,))
3

---

Under review as a conference paper at ICLR 2021
Our motivation for using the BSN is that it classifies the image stream with high accuracy and utilizes fewer FLOPs as compared to the RvSN. This results an increase in efficiency of the overall framework. Furthermore, the BSN converges to the performance of the TN faster than the RvSN, which we theoretically emphasize the Lemma below and experimentally validate Section provide the sketch proof here and the complete proof in Appendix [..]
We
Lemma 3.1. Lef RvSN and BSN represent the real-valued student network and binary student network respectively with the same network architecture. Let R(.) denote the rate of convergence in terms of accuracy when the student network is adaptively trained using the seeudo-labels from the teacher network. Then, R(BSN) > RiRvSN) for the same image stream and number of iterations.
Proof: We assume our image stream as xn) where n 2, 3...N, comprises of N samples from the deployed scenario. Since the weights of the BSN are derived from the RvSN, we prove this lemma first for RySN and then extend it to the BSN. Let be the optimal weight value which represents network convergence i.e. misclassification enror = 0.
The weight update rule using back propagation is given by:
w(n + 1) = w(n) + ()((m) Since 77 1] w(n + 1) w(n) + x(n)
Computing the bounds of the weights of the RvSN, we have
n 2 2
S [|
1)1l² < n²
(2)
I 1|
where a = ang w (n) and 3 argmax [=x())]] From eq. we can say that for this inequality to be satisfied, there exists nt which denotes the optimal number of samples for which the RvSN converges i.e. we obtain wt
nt. This is given follows:
n a²
n
[ ||
3 |w=1|
n
a
Thus from eq. we can say that the RvSN achieves convergence after being trained on number of samples. Since, the weights of the BSN are derived from the RvSN, we substitute the value of binary weight ú² from Rastegari et al. (2016) to obtain
@|
d²
7
n
N
Here, nī is the optimal number of samples for which the BSN converges. Comparing eq. and eq. 4 we observe that number of nig < n, i.e. the BSN takes fewer samples to converge to the performance of the TN than RvSN given the same network architecture.
Knowledge transfer from teacher to student network: Given an unlabeled image stream, the BSN is trained from the predictions of the TN as hard pseudo-labels using cross-entropy loss, as La = - N E; yslog(ss). Here, and are the pseudo-labels generated by the TN and the softmax output of the BSN respectively. N is the total images in the image stream. Once the BSN converges to the performance of the TN, we employ the BSN as the primary classifier during inference. When the deployed scenario does not change for a long duration, the BSN classifies the IL classes faster than the TN without relying on the latter. This improves the overall efficiency of the framework as the TN is now used to classify only the OL classes (number of OL images << number of II. images). However, maintain overall accuracy, the BSN must also accurately determine when the image from the input stream belongs to an OL class so that the TN can be used in such cases.
We know that attention maps highlight the regions of the image responsible for the classifier's prediction (Li et al. (2018)). Some examples of the attention maps from the correct predictions of the BSN and the TN shown in Figure (a). The first three columns in Figure (a) show that although the BSN (adaptively trained using Lạ) and the TN both correctly classify the images, their attention maps are significantly different. The attention map from the correct prediction of the BSN
4

---

Under review as a conference paper at ICLR 2021
Altention map IRE norr prodiction
Atterton map PEM from powdiatiinn
Altention BBN after us
Iipur image
Attertion map
La
0.93
G- 0.60
C- 0.21
C- 0.96
Lat
0.77
0.42
0.82
(a)
(b)
Figure 2: (a) The attention map visualization illustrating that after the BSN is trained using the
proposed attention triplet loss, learns to focus on the semantically meaningful regions in the image. The confidence of prediction (c) also increases. (b) Separation between IL and OL classes in feature
space improves with the proposed attention triplet loss. More visualizations in Appendix A.4
focuses the background while that of the TN lies the semantically meaningful regions of the image. Furthermore, the attention maps of the BSN's s correct and incorrect predictions are visually similar which causes the BSN learn incorrect representations of the II. images while correctly classifying them. To better differentiate between IL. and OL classes, the BSN should not memorize the predictions of the TN, but learn proper representations of the IL images.
To achieve this, we propose the attention triplet loss given in Equation 5 that causes the BSN's attention map to be similar to that of the TN while forcing the attention map from the BSN's correct and incorrect prediction to be different. We use Grad-CAI (Selvaraju et al. (2017)) to compute the attention map A from the one-hot vector of the predicted class. To generate the attention map triplets, we use the attention map from the TN's prediction A, as the anchor. The hard positive attention map A,, obtained from correct prediction of the BSN. When the BSN misclassifies an image (prediction of BSN and TN are different), by the findings Wang et al. 2019), we observe that the attention maps generated from the correct and incorrect predictions of the BSN are visually similar. Hence, we use the attention map from an incorrect prediction as the hard negative attention map As, to enforce its separability from As, To avoid bad local minima and collapsed model early during training, mining suitable hard negatives are important (Schroff et al. )2015)). Hence, we use the attention map from the second most probable class (incorrect prediction) (third and fourth column in Figure 2 (a)). Thus, we formulate the attention triplet loss Lat as:
use the label from the TN to compute As, Motivated
hard negative due its similarity with the hard positive
N
K
K Σ
Lot 11 NK
A
A
A
Agns +
(5)
Initially, the hard negative lies within the margin since its squared distance with the hard positive is small. Hence, Lot enforces separation between hard positive and hard negative by a distance greater than the value of margin 8 which empirically set as 1.5. N is the total number of samples and K the total number of pixels in A. Using Lat and Ld, we formulate our final objective function as:
L = L1 + AzLat
(6)
where A and A2 are the weights of La and Lat which we empirically set as and 0.2 respectively. The effect of Lat is shown in the last column of Figure 2 of the IL image due to its attention map being very similar to that of the TN (second column). Furthermore, the improvement in feature space separation between the II. and OL classes in Figure (b) shows that the BSN not only forms clusters of the different II. classes (in red), but is also well separated from the OL classes (in purple).
(a) where the BSN learns proper representations
EXPERIMENTAL DETAILS
Datasets and implementation: We evaluate ENVISE on CIFAR-100 (Krzhhvsky Hinton 2009)) and imagenttt (TI) (Yao & Miller (2015) datasets by creating meaningful IL and OL suwpe- class pairs that mimic real-world scenarios. On the test set CIFAR- 100 and validation set of TI dataset, create 12 and 10 super-classes respectively. We summarize our experimental settings in Table
5

---

Under review as a conference paper at ICLR 2021
We use DenseNet-201 (Huang et al. (017)) as the TN, pre-trained on the training set of all classes of CIFAR- 100 or the TI dataset individually. The BSN is a VGG16 (Simonyan & Zisserman (2014)) network whose weights, except for the first and last convolution layer, are binarized (Rastegari et al. (2016)). We also compare this with other choices for the BSN including AlexNet, ResNet- 18 and ResNet-50. In the Appendix A.3] we illustrate that ENVISE is insensitive to the specifie TN and BSN architecture used and achieves high performance gains even with different network architectures.
Table 1: Different pairs of IL and OL super-classes on the CIFAR-100 and TI datasets.
CIFAR-100 (Krizhevsky & Hinton 2009))
tiny-imagenet (Yao & Miller (013))
IL OL super-classes
# IL A OL
# IL
4 VOO # total classes
IL outlier super-clases
# IIL OOL
IL
FOL total classes
train
train
test
train
test
aquatic animals' food container (C2)
840
0
1200
500
17
animals garments (T)
875
0
1250
900
43
flora electrical
(C2)
630
900
500
reptiles edible items (72)
630
0
900
800
fruits furmiture (Ca)
350
0
500
500
10
aquatic animals birds (2s)
420
0
600
250
inseets manmade things (C4)
560
0
800
500
13
houschold items nature (2a)
1432
0
2050
700
55
animals people (Cs)
1750
0
2500
500
30
vehicles miscellannouus (Ts)
525
0
750
1800
51
outdoor places vehicles
1000
Throughout our experiments, we fix the TN and do not train it. We train the BSN using the pseudo-labels and attention map from the TN using the cost function in eq. onlyy on the IL classes of the super-class from Table 1 This is done using a learning rate of le for 10 epochs using the Adam optimizer (Kingma & Ba (2014)) with batch size of 10. Once the BSN converges to the performance of the TN, we employ it as our primary classifier during inference. To create a more realistic scenario during inference, we random center crop and randomly rotate the image stream between [-15", 15 We assume that the distribution of classes will not change rapidly in the deployed scenario, and that the BSN can be used for inference for long durations (e.g days, weeks months). Thus, the epochs for online distillation are expected to require small fraction of that time. For each image from the input stream during inference (comprising of IL and OL classes), following Heennryceerccsss & Gimpel (0066), compute the confidence of prediction from the softmax probability of the predicted class. If the confidence is low, we treat the image as an OL class and transfer it to the TN for classification.
Training and evaluation:
Retrained BSN Retrained RvSN Quantized BSN Standalone BSN
Maply
www
yul
Vandan
(a) BSN Binary Alexnet
(b) BSN Binary VGG-16 16
mmm
Ayaa
coutinn
Myste
m
(c) BSN Binary ResNet-18
BSN Binary ResNet-50
Figure 3: (a) : (d) The BSN converges to the performance of the TN faster than the RvSN and the variants of BSN. The red period illustrates adaptive training and white period illustrates inference. We observe a similar convergence pattern with different binary network architectures.
When the prior probabilities of the classes change in the deployed scenario, the BSN quickly re-trains to learn the new IL classes and regains efficiency on the new image stream. Figure 3 illustrates the learning behavior of the BSN when the IL and OL classes are changed. Initially, we assume that the image stream is comprised of only IL classes from C1. Once the BSN (in green) converges to the performance of the TN (in red), stop training the BSN and observe that it achieves an accuracy comparable to the TN's accuracy. The learning behaviour of the RvSN (in blue) is slower than the BSN and has poorer accuracy after 10 epochs.
Gradually changing the deployed scenario:

---

Under review as a conference paper at ICLR 2021
Directly binarizing the RvSN (purple line) is also worse than the BSN, and substantially differs from the binarization algorithm in Section which compensates for the quantization effect. The yellow line shows the performance of the standalone BSN which is not trained during the adaptive training.
In Figure 3 we keep the IL/OL superclasses the same for 40 epochs, and then switch the input stream to different scenario. The 10 epochs shaded in red indicate the adaptive training phase, while the unshaded (white) intervals indicate the 30 epochs for inference. We randomly choose our inference phase as 30 epochs. We show in Appendix A..3 that during inference, the combined accuracy of ENVISE is identical to that of the stand-alone TN, which illustrates that ENVISE maintains the overall accuracy of the system. When the scenario changes after 40 epochs (e.g from C1 to C2). we observe similar and consistent learning behaviour, and the BSN retrains quickly from the TN to regain efficiency. We also observe a similar convergence pattern for different BSN architectures when adaptively trained from the same TN in Figure [3] Here, we observe that the rate of convergence for Binary AlexNet is the fastest and that of Binary Resnet-50 is the slowest. This illustrates that smaller binary network would make ideal BSN in real-world scenarios, since it would utilize fewer epochs during adaptive training.
CE
CS
ODIN
OECC
MCD
CAL
ENVISE
p-
p 0.56
p 0.42
p 0.59
p 0.064
p- 0.077
p-
p 0.041
p p p 0.40
p- 0.28
p- p- p
p- 0.018
Figure 4: Feature space separation of the IL and OL classes for different baseline methods. p- vavulue denotes the overlap between the II. and OL features where, lower value indicates better separation.
Comparison with SOTA outlier detection methods: The ability to accurately distinguish between IL and OL classes is key for improving the efficiency of ENVISE. We benchmark ENVISE on the CIFAR-100 and TI datasets with SOTA OL detection methods like error detection (ED) (Hendrycks) & (0166) confidence estimation (CH) (DeVs & Taylor (2018), confidence scaling (CS)
(DeVsies & Taylor (2018), ). ODIN (Liang al. (2018)). outlier exposure with confidence control (OECC) (Papadopoulos et al. 2019)), maximum classifier discrepancy (MCD) (Yu & Aizawa (2019) and confidence aware learning (CAL) (Moon et al. (2000). We use the official code of these methods and adaptively train them using their proposed loss functions on our experimental settings. For fair comparisons in terms of network architecture, we use the TN as DenseNet-201 201 and the SN as our binary VGG-16. Following Hendrycks & Gimpel (2016), we use FPR at 95% True positive rate (TPR), detection error (DE), area under ROC curve (AuROC), and area under precision recall curve (AuPR) as evaluation metric. From Table 2 we observe that ENVISE outperforms the best performing bascline method by achieving the lowest FPR and detection error with high AuROC and AuPR. SOTA model compression techniques (Frankle & Carbin 2019) do not focus on processing images from the input stream with varying prior class probabilities. Hence, direct comparison with these methods is not meaningful since their objectives are different from those of ENVISE.
We visualize the separation of IL and OL classes in the feature space using UMAP (McInnes) let al. (2018)) across the last fully connected layer of the BSN. Figure 4 shows that ENVISE has the best separation between IL and OL classes which is consistent with the quantitative analysis shown in Table 2 To quantify the feature separation. we compute the p-value using Wilcoxon's 's rank sum test [Wilcoxon] (1992)) for the null hypothesis that the IL and OL feature distribution are the same i.e they overlap. Ideally for high separation, the p-value should be less than 0.05 (rejecting the hypothesis with 95% confidence). We observe from Figure 4 that the p-value of ENVISE has the smallest value as compared to the SOTA OL detection methods. This indicates that ENVISE achieves the least overlap between IL and OL classes, thereby outperforming the SOTA OL detection methods in its ability to better detect OL classes. A detailed comparison of ENVISE with each baseline method on different pairs of IL and OL classes and the corresponding feature space visualization is presented in the Appendix A.4 Furthermore, in Appendix A.2 we ablate the effect of the proposed Tett, margin (8) with the BSN
7

---

Under review as a conference paper at ICLR 2021
and also performance of OL detection with RySN. We present additional discussions in Appendix A.3
Table 2: Performance comparison of ENVISE with the best performing baseline method on different IL and OL class pairs on CIFAR- 100 and TI dataset. and indicate smaller and greater value is better respectively. The detailed comparison with each baseline is in the Appendix A.4
Dataet
Group
Method
Metric
(95% TPR)
Detection emur
AuROC
AuPR (IL)
AuPR (OL)
+
T
T
Best Raseline.
0.90
0.42
0.59
0.77
0.37
ENVISE
0.88
0.39
0.62
0.80
0.39
Besi Baseline ENVISE
0.69
0.28
0.79
0.86
0.71
C2
0.41
0.16
0.89
0.88
0.92
Best Baseline ENVISE
0.69
0.21
0.86
0.87
0.87
CIFAR-100
C3
0.57
0.18
0.89
0.91
0.87
0.65
0.85
ENVISE
0.61
0.82
0.77
Best Baseline
0.82
0.39
0.62
0,90
0.23
ENVISE
0.76
0.29
0.79
0.95
0.42
Best Baseline ENVISE
0.63
0.22
0.85
0.79
0.90
Cc
0.49
0.16
0.91
0.87
0.97
Best Buseline. ENVISE.
0.72
0.28
0.84
0,88
0.65
T1
0.69
0.23
0.89
0.91
0.71
Best Baseline ENVISE
0.29
0.79
0.73
Best Baseline ENVISE
0.78
0.32
0.68
0.76
0.61
-magenett
T3
0,74
0.20
0.81
0,93
0.63
Best Baseline ENVISE
0.83
0.20
0.69
0.78
0.51
T4
0.78
0.18
0.75
0.91
0.44
Best Baseline EN VIVE
0
0.90
0.85
0.79
23
0.78
0.26
0.91
0.85.
0.84
3.8
02
00:4
Eary VGG 25
C4
06
(a)
(b)
Figure 5: (a) Comparison of ENVISE with SOTA OL detection methods, and (b) Comparison of different BSN architectures in terms of GiE on different super-classes of the CIFAR-100 dataset.
Gain in Efficiency (GiE): The main focus of our work is to develop efficient system to classify the image stream with low computational cost and high accuracy. We propose a new evaluation metric GiE to measure the overall gain in efficiency from the number of times the BSN and TN are used individually to classify the image stream. During inference, we require the BSN to accurately classify the IL classes such that it does not rely the TN thereby reducing the overall computation cost. Furthermore, the BSN should detect an image as OL so that the TN can be used in such cases. To process single image on the CIFAR-I00 100 dataset, the TN uses 9048 MFLOPs and 28.5msec, while the RvSN uses 310 MFLOPs and 3.04 msec. However, the BSN achieves 6 MFLOPs and 0.57 msec: 50x reduction computation and 6x speed improvement over the RySN. Furthermore, the BSN occupies ~ 30x and ~ 100x less memory than the RvSN and TN respectively. During adaptive training. since each image is processed by both the BSN and TN, the total FLOPPs used is FL,, = (X + Y), where X and Y are the FLOPs used by the TN and BSN respectively. Once the the BSN converges to the performance
the TN, only the BSN is used to process the input image. Hence, for IL images the total FLOPs is F FL., = + (fp X X), where (fp x X) indicates the number of times the TN is used when the BSN misclassifies an IL class as an OL. Similarly for OL images, the FLOPs is FL, = Y + (od x X), where (od x X) denotes the OL classes correctly detected by the BSN which is then processed by the TN. Since the adaptive training and inference phase for given super-class occur for Ner and N., epochs respectively, the relative FLOPS of
8

---

Under review as a conference paper at ICLR 2021
ENVISE compared to the TN is given
Nor
FLer X
No (px FLin + q x FLo) NT
RFI.
NT
X
Here, NT= - Ner + Nin- For high computational efficiency, RFL should be small. If the BSN works with perfect accuracy (i.e. fp- = 0 and od 1.0), then the minimum value of RFL during inference R.LLmmn X We define GiE as RFL normalized with respect to its minimum value, i...
RF
Y + 2ggx X
GiE = RFL
(8)
NY. NT
FL
No NT
(p Flin q x FL.)
To numerically compute GiE, we use while p and q are calculated from Table [ However, as discussed previously, Nan can be very large value since the inference phase can occur for very long duration. Jillustates that ENVISE outperforms SOTA OL detection methods in terms of GiE for different scenarios CIFAR-100 dataset. Thus, the Lat ensures that the BSN can accurately distinguish between representations of II. and OL classes which enables ENVISE operate in efficient manner.
9048 (TN), Y 6 (BSN), N,, = 10 and Ni 30,
CONCLUSION
We propose ENVISE, an online distillation framework which uses a BSN to adaptively learn relevant knowledge from a TN to quickly classify the frequent classes in the deployed scenario. In doing so, it automatically allocates processing resources (i.e. either the BSN or the TN) to reduce overall computation. To learn proper representations of the IL classes, we propose an attention triplet loss that enables the BSN to learn the semantically relevant regions of the image that are emphasized by the TN. This enables the BSN accurately distinguish between representations of IL and OL classes which is key for maintaining overall accuracy. Our experiments show that the BSN i) quickly converges the performance of the TN, ii) classifies II. classes more accurately than RySN and other variants of the BSN, and iii) distinguishes between II. and OL classes with lower FPR and detection error than other SOTA OL detection methods on CIFAR-IO0 100 and tiny-imagenet datasets. We introduce a new metric GiE to assess the overall gain in efficiency, and experimentally show that the attention triplet loss enables ENVISE to achieve higher GiE than SOTA OL detection methods. We show that ENVISE is agnostic to the specific TN and BSN and achieves high gains with different BSN and TN architectures.
REFERENCES
Pooya Abolghasemi, Amir Mazaheri, Mubarak Shah, and Ladislau Boloni. Pay attention!-robustiiyingg deep visuomotor policy through task-focused visual attention. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4254 4262, 2019.
Abhijit Bendale and Terrance E Boult. Towards open set deep networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1563-1572 2016.
Yun-Chun Chen, Chen Gao, Esther Robb, and Jia-Bin Huang. Nas-dip: Learning deep image prior with neural architecture search. In European Conference on Computer Vision (ECCV), 2020.
Anthony Cioppa, Adrien Deliege, Maxime Istasse, Christophe De Vleeschouwer, and Marc Van Droogenbroeck. Arthus: Adaptive real-time human segmentation in sports through online the
distillation. In Proceedings the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 1-9, 2019.
Terrance VeVries and Graham W Taylor. Learning confidence for out-of-distribution detection in neural networks. arXiv preprint arXiv:1802.00 04865, 2018.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. International Conference on Learning Representations, 2019.
Liang Gao, Xu Lan, Haibo Mi, Dawei Feng, Kele Xu, and Yuxing Peng. Multistructure-basee collaborative online distillation. Entropy, 21 (4):357, 2019.

---

Under review as a conference paper at ICLR 2021
Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-diisrrbuttio examples in neural networks. arXiv preprint arXiv:1610.02116, 2016.
Dan Hendrycks, Mantas Mazeika, and Thomas Dietterich. Deep anomaly detection with outlier exposure, Proceedings of International Conference on Learning Representations, 2019.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge neural network. arXiv preprint arXiv:/503.022 2015.
Gao Huang. Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4708, 2017.
Jangho Kim, Minsung Hyun, Inseop Chung. and Nojun Kwak. Feature fusion for online mutual knowledge distillation. arXiv preprint 1rXiv:1904.09058 2019.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Alex Krizhevsky and Geoffreyx Hinton. Learning multiple layers features from tiny images. 2009.
Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting out-of-distribution samples and adversarial attacks. In Advances in Neural Information Processing Systems, pp. 7167-7177, 2018.
Kumpeng Li, Ziyan Wu, Kuan-Chuan Peng, Jan Ernst, and Yun Fu. Tell me where to look: Guided attention inference network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 9215-9223, 2018.
Shiyu Liang, Yixuan Li, and Rayadurgam Srikant. Enhancing the reliability of outtof-distriuution image detection in neural networks. International Conference on Learning Representations, 2018.
Rongcheng Lin, Jing Xiao, and Jianping Fan. Mod: A deep mixture model with online knowledge distillation for large scale video temporal concept localization. arXiv preprint arXiv: 12295, 2019.
Roi Livni, Shai Shalev-Shwartz, and Ohad Shamir. On the computational efficiency of training neural networks. In Advances in neural information processing systems, pp. 855-863, 2014.
Leland McInnes, John Healy, Nathaniel Saul, and Lukas Grossberger. Umap: Uniform manifold approximation and projection. The Journal of Open Source Software, 3(29):861, 2018.
Jooyoung Moon, Jihyo Kim, Younghak Shin, and Sangheum Hwang. Confidence-aware learning for deep neural networks. In International Conference Machine Learning, 2020.
Ravi Teja Mullapudi, Steven Chen, Keyi Zhang, Deva Ramanan, and Kayvon Fatahalian. On- line model distillation for efficient video inference. In Proceedings of the IEEE International Conference on Computer Vision, pp. 3573-3582, 2019.
Aristotelis-Angelos
Papadopoulos, Mohammad Reza Rajati, Nazim Shaikh, and Jiamian Wang. Outlier exposure with confidence control for out-of-distribution detection. arXiv preprint
arXiv:1906.03509 2019.
Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classification using binary convolutional neural networks. In European conference computer vision, pp. 525-542. Springer, 2016.
Jie Ren, Peter J Liu, Emily Fertig, Jasper Snoek, Ryan Poplin, Mark Depristo, Joshua Dillon, and Balaji Lakshminarayanan. Likelihood ratios for out-of-distribution detection. In Advances in Neural Information Processing Systems, pp. 14680-14691, 2019.
Florian Schroff. Dmitry Kalenichenko. and James Philbin. Facenet: A unified embedding for face recognition and clustering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 813,823, 2015.
10

---

Under review as a conference paper at ICLR 2021
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based local- ization. In Proceedings of the IEEE international conference computer vision, pp. 618-626, 2017.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXivv1409.1553 1556, 2014.
Apoorv Vyas, Nataraj Jammalamadaka, Xia Zhu, Dipankar Das, Bharat Kaul, and Theodore Willke. Out-of-distribution detection using an ensemble of self supervised lcave-out classifiers, In Proceedings of the European Conference on Computer Vision (ECCV), pp. 550-564, 2018.
Lezi Wang, Ziyan Wu, Srikrishna Karanam, Kuan-Chuan Peng, Rajat Vikram Singh, Bo Liu, and Dimitris N Metaxas. Sharpen focus: Learning with attention separability and consistency. In Proceedings of the IEEE International Conference on Computer Vision, pp. 512-521, 2019.
Lin Wang and Kuk-Jin Yoon. Knowledge distillation and student-teacher learning for visual intelli- gence: A review and new outlooks. arXiv preprint Sttv:2000.059337 2020.
Frank Wilcoxon. Individual comparisons by ranking methods. In Breakthroughs in statistics, pp. 192-202. Springer, 1992.
Leon Yao and John Miller. Tiny imagenet classification with convolutional neural networks. CS 231N, 2(5):8, 2015.
Ryota Yoshihashi, Wen Shao, Rei Kawakami, Shaodi You, Makoto lida, and Takeshi Naemura. ceaafficaaioorreoonsttttttti learning for open-set recognition. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 4016-4025, 2019.
Qing Yu and Kiyoharu Aizawa. Unsupervised out-of-distribune detection by maximum classifier discrepancy. In Proceedings of the IEEE International Conference Computer Vision, pp. 9518-9526 2019.
A APPENDIX
A.1 PROOF OF CONVERGENCE
In Section 3 of the main paper, we describe that, one of the motivations for using the BSN instead of the RvSN is the ability of the BSN to adapt to the performance of the TN faster than the RySN.
Lemma A.1. Let RvSN and BSN represent the real-valued student network and binary student network respectively with the same architecture. Let R denote the rate of convergence terms of accuracy when the student network is adaptively trained using the teacher network's 's predictions. Then, R(BSN) > R(RvSN) for the same input image stream and number of iterations.
Proof For the ease of understanding, we assume to have binary classification problem with 2 classes
ie.. C' = 2, the data points are lincarly separable, the weights are initialized to 0 i.c. w()) = 0 and learning rate 7 = 1. The initialization value of weights and learning rate do not affect the proof. Let us further assume that we have stream of input images x(n)) where n = 1, 2, 3..N, where N the total number of samples. Since the weights of the BSN are derived from the RvSN (From Section 3.1 of the main paper), we prove this lemma for RySN first and then extend it to the BSN. Since, the pretrained performance of the SN is worse than the TN (comparing starting point of blue line with the red line from Figure 3(a) of the main paper), images from the input stream iee. w())æ((n) < 0 where w(n) is the weight of the network. The misclassification generates error through which the network's weights updated and there exists an optimal weight value w² such that the network converges i.e. misclassification error = 0. The weight update rule using back propagation is given by:
assume that the SN misclassifies the
w(n + 1) w(n) + yp((n)
(9)
Since 1]
w(n + 1) = (n) ) + (n1)
11

---

Under review as a conference paper at ICLR 2021
Expanding eq (9), we have:
w(1) w(0) + +0)
((2) = w(1) +1(
(10)
= (2) ((0) + =(0) +z(1)
Since w(0) = 0] w(2) = z0 +
Similarly we have w(3) = s²2 + x(1) + (0) and this recursively continues untill w(n + 1). Thus, w(n + 1) (x1 + (2) + xx3) + (().
(11)
Ideally, when the network converges, we obtain optimal weight vector, which is represented as w". We define the margin a as the shortest distance of wx to the datapoint x E x(x). The concept of margin is similar to that of support vectors. Hence, the margin is denoted
argmin w" x(() =(n)EC
(12)
We ignore z0() for ease of caleulation. Pre-multiplying both the sides of eq. (11) by w we have
w" =*x(1)= + (2()
+....=x(((n xn()
(13)
From eq. (12))
w" w 2 na
From Cauchy Schwarz inequality, we have
|| 1| [ww( + 1)11 2 (w* w(( + 1))²
|| I 1|² I[w(n + 1)1l² 2 (na)²
(14)
n² a²
||w( + 1)1l 2
|
We highlight eq. (14) as we will be using this in the later stages of the proof. We re-write eq. (9) in
different way as w(k + 1) x<k) + w(k) A le e [0, 0,,1,233.17 3.. Squaring both sides of this equation and expanding it, we have
| + 1)]1 = || + w()]l
= l|w( + 1)11² = ||=( + [|)( + (x(k))(())
(15)
[Since in misclassification w(n)x(n) S ||uek + 1)ll < [|=( 2 + [()())]
+ |lw + 1)1l [1)( s [|)())]
Iterating eq (I5) for all values of & E [0, 2, we have
[Since when &s = 0, w
0]
||w + 1)11
|=(A)]1?
(16)
k=
Since ||=(k]] > 0, define | = argmax [
thus eq. (9) can be re-written
[uon + 1)11 n/
(17)
From eq. (14) and (17), we have
n 2 2
< | + 1 < n²
(18)
From eq. (18), we can say that for this inequality be satisfied, there exists n² which denotes the optimal number of samples for which the network converges i.e. we obtain w* at n = n"" This is given as follows:
n a
=n²; =
|
(19)
8
|w=
n
2
12

---

Under review as a conference paper at ICLR 2021
Thus from eq. (19), we can say that the network achieves convergence after being trained on number of samples. the case of the BSN, the binary weights o' = A D", where A x |000 and D sign(w") (w"), Let ni be the number of samples required for the BSN converge. Thus, eq. (19) for the BSN would be given as
@ 3/w²' a²
3 |100 |
ni
Q²
B 21 * D 1|
n2g
a²
3 1100 D 12
N²a²
(20)
3 ||lw*1
DDI 1>
ni
N2a²
n" N
From eq. (19) and Rastegari et al. (2016)
ng
N
n"
ni
N
From eq. (19) and (20), we see that ni < n²" i.c. the BSN takes fewer samples than the RySN to converge the TN's performance given the same network architecture. We also experimentally observe the higher rate of convergence of the BSN over the RvSN in Figure 3f the main paper.
A.2 ABLATION STUDY
All the ablation studies are performed on the CIFAR-I00 dataset. We illustrate the effectiveness of the attention triplet loss (Lat). the value of margin in the (Lut, significance of the hard negative term in Lat and the performance of the BSN over the RvSN for OL detection. The quantitative analysis is summarized in T
Effect of attention triplet loss (L,t): To test the effectiveness the attention triplet loss (Laz). ). we train the BSN without it. Specifically, use the TN DenseNet-201, the BSN Binary VGG-16 and train the BSN using only the distillation loss L,g- Comparing column ID and in Table 3 we observe that training the BSN using La + Lut improves the mean FPR by 35.4%, mean detection error by 41.6%, mean AuROC by 12.5%, mean AuPR of IL class by 7.4% and mean AuPR of OL class by 30.3%. From Figure G we observe that the IL (red) and OL features (purple) are barely separable when evaluated on pre-trdi BSN. When the BSN is adaptively trained using only La- the features begin to separate but still have some overlap. However training the BSN using La + Lac significantly enhances the feature separation thereby improving OL detection. Furthermore, the p-value (noted in T below each UMAP visualizations) of BSN when trained using La + Lat has the smallest value of 0.041. This validates assertion that Las learns the proper representations of the IL classes thereby improving its ability to differentiate between IL and OL classes
(a)
(c)
La
L
Lat
p-value 0.56
p-value = 0.18
p-value 0.041
Figure 6: (a):() Illustration of the effect of applying La and its combination with Tut on the separation between the IL (red) and OL (purple) classes in the feature space.
Effect of margin ò : We mention in Section 3 that, in order to avoid a collapsed model early on during training, we select the hard negative attention map such that its squared distance is closest
13

---

Under review as a conference paper at ICLR 2021
to the hard positive attention map. Without using any OL images during training validation, empirically observe that Lg norm of the hard negative attention map with respect to the hard positive attention map ranges between [0.33, 2.8]. Hence, choose as 1.5 to ensure separability with respect to the hard positive attention map.
To show that ENVISE is insensitive the specific value of margin, train the BSN using Lat with value of 5 as 0.5, 1.0, 1.5, 2.0 and 2.5. From Table 3 comparing column with 4,5,6,7 and 8, observe that ENVISE insensitive the specific value of margin and outperforms the best performing baseline method for each super-class, even with different values of the margin. Furthermore, from Fig. we observe that ENVISE has the best performance when the margin is set
to 1.5
0.5
1.0
6
0.14
0.046
p 0.021
0.038
0.041
Figure 7: Effect of different values of margin on the separation between the IL (red) and OL (purple) classes in the feature space.
Table Ablation study illustrate the effect of proposed Tutt loss and the effect different values of margin 8 on the performance of ENVISE for outlier detection on CIFAR- 100 dataset.
super-class
ENVISE La only
ENVISE Lg Las
Best Baseline
ENVISE 8=0.5
ENVISE 8=10
ENVISE
8=1.5
8=2.0
8=2.5
column ID
3
0.93
0.88
0.90
0.95
0.89
0.93
Ca
0.81
0.41
0.69
0.49
0.61
0.41
0.41
0.41
C3
(95% TPR)
C4
0.88
0.61
0.65
0.84
0.61
0.61
0.79
0.78
Cs
0.87
0.76
0.82
0.78
0.81
0.76
0.83
0.77
Ge
0.64
0.49
0.63
0.54
0.53
0.49
0.67
0.57
0.84
0.62
0.73
0.70
0.69
0.62
0.69
0.72
C2
0.33
0.16
0.28
0.17
0,21
0.16
0.16
0.16
Detection
Co
0.18
0.21
0.24
C4
0.35
0.26
0.27
0.26
0.26
0.3
0.26
0.3
Cs
0.39
0.29
0.39
0.29
0.33
0.29
0.31
0.3
0.21
0.16
0.22
0.16
0.17
0.16
0.25
0.16
meam
0.59
C2
0.83
0.89
0.79
0.9
0.87
0.89
0.95
0.91
AaROC
Ca
0.69
0.89
0.86
0.84
0.83
0.89
0.87
0.82
0.79
0.76
0.76
Cs
0.64
0.79
0.62
0.76
0.72
0.79
0.74
0.76
Cg
mean
0.72
0.81
0.75
0.78
0.80
0.81
0.79
C1
0.81
0.80
0.77
0.75
0.85
0.80
0.82
0.79
0.86
0.88
0.86
0.94
0.92
0.88
0.87
0.95
AuPR (inlicr)
Cs
0.73
0.91
0.87
0.87
0.84
0.91
0.89
0.85
0.84
Cc
0.88
0.95
0.90
0.94
0.91
0.95
0.93
0.94
Ce
0.87
0.79
0.76
0.86
mean
0.81
0.87
0.84
0.87
0.87
0.87
0.85
0.87
C1
0.41
0.39
0.37
0.32
0.45
0.39
0.43
0.35
AuPR (outlier)
Ca
0.64
0.93
0.87
0.78
0.81
0.93
0.85
0.77
C4
0.72
0.63
Cs
0.26
0.42
0.23
0.38
0.33
0.42
0.34
0.38
Co
0.88
0.97
0.90
0.92
0.92
0.97
0.88
0.94
mcan
0.56
0.73
0.59
0.63
0.65
0.73
0.68
0.65
14

---

Under review as a conference paper at ICLR 2021
Table 4: Ablation study illustrate the effect of employing the BSN over RvSN on the performance of ENVISE for outlier detection on CIFAR- 100 dataset.
Method
mean
FPR (4)
Detection error (L)
AuROC (1)
AuPR (IL) (t)
AuPR (OL) (1)
ENVISE (w/ RvSN)
0.65
0.30
0.73
0.73 0.87
0.63
ENVISE (w/ BSN)
0.62
0.24
0.81
0.73
Significance of using hard negative term in Lat: As mentioned in Section of the main paper, we observe that the attention maps from the correct and incorrect predictions are visually similar. This causes the BSN to learn improper representations of the II. classes during online distillation. Hence, we choose the attention map from the incorrect prediction (second most probable class) as hard negative ensure separability with the attention map from the correct prediction. To illustrate the significance of the hard negative attention map, we train ENVISE without the second term in equation along with La. The attention L2 loss is formulated as:
La
1 NK
Σ
2
Ae.
Alp
(21)
From Figure observe that the confidence of classifying image decreases when ENVISE is trained using L.. Furthermore, the attention map of ENVISE after training using La + La does not focus on the semantically meaningful regions of the image (fourth column) unlike training it with Lat (last column). Furthermore, we also observe training ENVISE with La + L. results in a poor OL detection with 0.98 FPR and 0.47 detection error respectively on C1 as compared 0.88 and 0.40 FPR and detection error respectively with L - Lat- This validates that the hard negative attention map ensures separability with the hard positive attention map and also improves the confidence of classification and OL detection.
Attention highest prediction
Attantion map BSN after using La
Attention map BSN after using L.t
Imput
map TN
Image
C 0.93
c 0.68
== 0.41
C 0.96
0.42
0.83
0.92
Figure 8: The attention map visualization representing the regions of the image that ENVISE focuses on when trained using Ln and Lat. The confidence of classification is reported below each image.
Effeet of using RySN instead of BSN To illustrate the effectiveness of using the BSN to improve the overall gain in efficiency of the system, we employ RvSN instead of the BSN our SN. We use the TN as DenseNet-22
and real-valued VGG-16 as our SN and adaptively train using La + Lot- We observe from Table that using BSN instead of RvSN 4.8%, mean detection error by 25%, mean AuROC by 11, mean AuPR of IL. class by 19.2% and mean AuPR of OL class by 15.9%. Furthermore. also observe that mean GiE for the BSN is 0.53 while that of the RySN is 0.78 which illustrates that ENVISE has 47,3 2.% gain in efficiency with the BSN compared to the RvSN.
SN improves the mean FPR by
15

---

Under review as a conference paper at ICLR 2021
0.0
Shannon Ereropy
0.6
infier classes
egochs
Figure 9: Shannon Entropy of IL and OL of the BSN during online distillation. The entropy of the IL. decreases, the entropy of OL is high. This shows that BSN does not learn OL class information.
A.3 DISCUSSIONS
The TN does not leak OL class information to the BSN during online distillation We mention in Section 3 that the BSN is trained using only the hard pseudo-labels from the TN on the II classes. We show that during adaptive training, the TN does not transfer OL class information to the BSN. Hence, the ability of the BSN to detect OL classes is due to its ability to accurately differentiate between IL and OL class representations. To illustrate this, we compute the Shannon entropy H (.) of OL and IL classes of the BSN during the online distillation process (when BSN is trained using La + L.t. Figure 9 illustrates that while the XIL classes decreases, the HOO) class is already high, where X is the image from the input stream. This illustrates that during the online distillation, the BSN learns the information the classes, while information about the OL class completely absent. Thus, the ability of the BSN to detect OL classes better than the SOTA outlier detection methods is not due to an OL information leak, but due to the BSN's s ability to accurately differentiate between the representations of the IL and OL classes.
Gain with smaller teacher network: We investigate the importance the architecture and size of the TN network on the performance of the BSN for efficient OL detection. Here, we ResNet-18 (11M parameters) for the originally used DenseNet-201 (18M parameters) as the TN, which is much smaller network as compared to the latter. We train ResNet- 18 using the same training procedure as discussed in Section 4 Once the BSN converges to the performance of the TN, we evaluate the BSN on the different CIFAR- 100 super-classes in Table [ OL detection. From Table 5) we observe that regardless of the TN used, ENVISE outperforms the best performing baseline method for OL detection by achieving lower FPR and detection error with higher AuROC and AuPR. Furthermore, we also observe that the BSN achieves similar gain in performance with ResNet- 18 as with DenseNet-2011 From Figure 33 and Table we show that ENVISE agnostic to the specific TN and BSN network architectures and achieves similar performance gains with different architectures.
Table 5: Performance of ENVISE in terms of OL detection, with smaller TN network (ResNet- 18) as compared larger TN (DenseNet-20) 201) using the same BSN (Binary VGG-16) on different super-classes of CIFAR-100 dataset. Notations R: ResNet- 18 as TN, D DenseNet-201 as TN.
Metric
Best Baseline
ENVISE (R)
Best Baseline
ENVISE
(R)
(D)
(D)
mean FPR (95% TPR)
0.83
0.69
0,76
0.62
mean Detection error
0.29
0.24
0.30
0.24
mean AuROC 1
0.74
0.80
0.74
0.81
mean AuPR (IL) 1
0.83
0.86
0.83
0.87
mean AuPR (OL) 1
0.61
0.81
0.63
0.73
GiE when probability of IL class >> probability of OL class: We evaluate GiE of ENVISE for the case when the II. class occurs with much higher probability than the OL class. Hence for Cs.
16

---

Under review as a conference paper at ICLR 2021
we consider all 2500 images from IL class and 300 images from OL class, making the probability of occurrence of IL class = 0.9 2500
2500+3300 = 0.89 ~ 0.9). In such setting, we obtain an FPR (f) of 0.26 and detection error (od) of 0.15. From the formulation of Fl, mentioned in Section 4 of the main paper, we use X 9048 x 1es and Y = 6 x Le to obtain GiE 0.79. Thus, in the case when the images from IL class occur more frequently than OL class (i.e = 0.9), ENVISE achieves a gain in efficiency of 36.2%.
Trade-off between accuracy and GiE in ENVISE: Table shows the accuracy of the BSN, the TN and their combined performance (i.e. ENVISE) for all six II. super-class pairs the CIFAR-100 dataset. Comparing row 2 and row 3 in Table 6 we observe a very minimal loss in accuracy in ENVISE with respect to the TN on different super-classes of the CIFAR- -00 dataset. In ENVISE. during inference, the BSN classifies the IL classes from the image stream. When the BSN misclassifies an IL class as an OL class, the image is then given to the TN for classification. This ensures that the overall accuracy of the system is bounded by the performance of the TN. Furthermore, from Figure 5 we observe that ENVISE has the highest GiE which indicates that, along with high computational efficiency, ENVISE also has minimal loss in overall accuracy compared to standalone TN.
Table 6: Comparison of accuracy between BSN, TN and ENVISE on the IL classes of CIFAR-100 dataset. The IL images that the BSN misclassifies as OD, are correctly reclassified by the TN.
Model
C1
C2
C3
C₁
C3
BSN
0.72
0.80
0.86
0.83
0.79
0.83
TN
0.76
0.81
0.89
0.86
0.83
0.84
ENVISE
0.75
0.81
0.88
0.84
0.81
0.84
A.4 ADDITIONAL QUANTITATIVE AND QUALITATIVE ANALYSIS
Complete comparison with baseline methods: We present a detailed comparison of ENVISE with the SOTA OL methods on CIFAR-100 and TI datasets in Table g We observe that ENVISE achieves lower FPR and detection eгror as compared to the baseline methods, Specifically, ENVISE outperforms the best performing baseline method (ODIN) by 23% and 25% in terms of mean FPR and mean detection eтror respectively. Furthermore, ENVISE also achieves higher AuROC and AuPR (outlier class) by outperforming ODIN by 9.5% and 15. 5% in terms of mean AuROC and mean AuPR (outlier class) respectively on CIFAR- 100 dataset. Table 9 also shows that ENVISE outperforms the baseline methods achieving lower FPR and detection error and higher AUROC and AuPR (II. and OL class) on TI dataset. Specifically, ENVISE achieve 9.3% and 30.4% lower mean FPR and mean detection error respectively as compared to the best performing baseline method (MCD). ENVISE also outperforms MCD by 5.5%, 14.1% and 3.1 1% in terms of mean AuROC, mean AuPR (inlier class) and mean AuPR (outlier class) respectively.
Performance of ENVISE is insensitive to specific IL. OL class pairs: As mentioned in Table 1, we evaluate ENVISE on meaningful IL \ OL class pairs that mimic real-life scenarios. However, we show that ENVISE is insensitive to the specific II ) OL class pairs created in Table 1. From Table 10 Table D and Table [2] we show that ENVISE outperforms all baseline methods when each IL class is compared to every other OL class by achieving lower FPR and detection error and higher
AuROC and AuPR on the CIFAR- -00 dataset.
Feature space separation: Figure I0 illustrates the comparison of ENVISE with all baselines in terms of the separation between IL and OL images in the feature space for all IL \ OL class pairs on CIFAR-100 dataset. To quantify the feature separation between the IL and OL samples, we compute the p-value using Wilcoxon's ' rank sum test Wilcoxon (1992) for the null hypothesis that the II and OL feature distribution are the same. Hence, lower p- value indicates better separation. We observe that ENVISE achieves better separation than bascline methods with the lowest pavalue. Furthermore, we also observe clusters of the IL images (in red) in the feature space which illustrates that ENVISE is capable of learning representations of IL images rather than memorizing labels from the TN.
Sub-classes within each super-class: Table 7 and Table g represents the sub-classes within each super-class of the CIFAR- 100 and tiny-imagenet (TI) datasets. These sub-classes are the original classes of these datasets respectively. Each super-class is denoted by C.s and Czo where x = [1, 2, 4, 5,6 6) representing the inlier and outlier super-class on CIFAR- 100 dataset. Furthermore, the
17

---

Under review as a conference paper at ICLR 2021
ED
CE
CS
ODIN
OECC
MCD
CAL
ENVISE
C
p 0.68
p = 0.55
p = 0.42
p 0.59
p 0.20
p = 0.28
p 0.39
0.041
C
0.48
0.52
0.55
0.36
0.18
0.053
0.47
0.021
C
p= 0.43
p= 0.57
p = 0.55
p= 0.31
p = 0.38
p= 0.048
p 0.53
0.033
C4
p= 0.16
p= 0.58
p = 0.46
p= 0.55
p= 0.028
p = 0.01
p 0.42
0.028
5
0.51
p 0.46
0.40
0.28
p 0.21
0.37
0.58
0.016
C6
p 0.25
p = 0.62
p = 0.85
p = 0.58
p 0.044
p = 0.16
0.49
p 0.043
Figure 10: Comparison of ENVISE with baseline methods in terms of separation between II (in red) and OL (in purple) samples in the feature space on CIFAR- dataset. The pavulue obtained from Wilcoxon's 's rank sum test is denoted below each visualization where lower value denotes better separation.
super-classes of the tiny-imagenet dataset denoted as Ty. and Tyo for inlier and outlier super-class, where = [1, 3, 4, 5). We use these notations for comparing ENVISE with the SOTA OL detection methods Table Table 10 Table and Table 12 respectively.
18

---

Under review as a conference paper at ICLR 2021
Table 7: Different sub-classes within each inlier and outlier super-classes on the CIFAR-100 dataset. These sub-classes represent the original classes in the CIFAR-100 dataset.
Super-classes
Sub-classes
water animals (Ci:)
beaver, dolphin, otter, seal, whale, aquarium fish, flatfish, ray shark, trout, crab, lobster
food containers (C1o)
bottles, howls, caps, cups, plates
flora (C2.)
orchids, poppies, roses, sunflower tulips, maple, oak, palm, pine, willow clock, keyboard, lamp, telephone, television apple, mushrooms, orange, pears, sweet peppers
houschold electric devices (C2o)
fruit and vegetables (Ca.)
houschold furmiture (Cao)
bed, chair, couch, table, wardrobe bee, beetle, butterfly, caterpillar, cockroach, snail, spider, worm
insects (Ca.)
manmade things (C4.)
bridge, castle. house, road, skyscrapper
camel, cattle, chimpanzee, elephant, kangaroo, fox, porcupine, possum racoon.skunk, hamster, mouse, rabbit, shrew, squirel. crocodile dinosaur, lizard, turtle, snake, bear. leopard, tiger, lion, wolf
animals (Csz)
people (Cho)
baby, boy, girl, man, women
outdoor scene (Co.)
cloud, forest, mountain, plain, sea
vehicles (Coo)
bicycle, bus, motorcycle, pickup truck train, lawn-mover rocket, streetcar tank, tractor
Table 8:
Different sub-classes within each inlier and outlier super-classes on the tiny-imagenet dataset. These sub-classes represent the original classes in the tiny-imagenet dataset.
Super-classes
Sub-classes
Labrador retriever, hog, Chihuahua, orangutan, chimpanzee, koala, lesser panda golden retriever, lion. baboon, African elephant, Yorkshire terrier,
animals (Tix)
bison, standard poodle, cougar, gazelle, ox. Egyptian cat Persian cat, German shepherd, guinea pig, bighorn
birds (Tis)
black stork, goose, king penguin, albatross, crane
European fire salamander, bee, scorpion, fly, black widow, ladybug, slug, cockroach, grasshopper, centipede, mantis, sulphur butterfly dragonfly, snail, boa constrictor, spider web, trilobite, tarantula
reptiles/ / insects (T2..)
aquatic animals (1'2o)
brain coral, American lobster, American alligator. tailed frog, spiny lobster sea slug, dugong, coral reef, bullfrog, sea cucumber, jellyfish, goldfish
edible items (Txz)
orange, pizza, mushroom, banana, ice cream, pomegranate, pretzel. mashed potato potpie, cauliflower, meat loaf, bell pepper, guacamole, lemon, icc lolly, confectionery
garments (Tao)
swimming trunks, apron, poncho, academic gown, military uniform, neck brace vestment, kimono, sombrero, fur coat, cardigan, bikini, miniskirt, bow tic sunglasses, sandal, sock, Christmas stocking
picket fence, candle, chain, rocking chair, torch, iPod
frying pan, dumbbell, water jug, teddy, plate, walking stick, computer keyboard bucket, comic book, sewing machine, remote control, teapot, barn, volleyball lawn mower, hourglass, desk, lampshade, bathtub, wok, CD player, rugby ball stopwatch, magnetic compass, space heater, plunger, backpack, wooden spoon broom, dining table, basketball, punching bag, refrigerator, umbrella
Household items (Taa)
places/ / manmade things (Tto)
seashore, water tower, triumphal arch, cliff dwelling. suspension bridge dam, steel arch bridge, cliff, monarch, obelisk, lakeside, fountain, altar, alp
Vehicles (Ta.)
school bus, lifeboat, jinrikisha, beach wagon, tractor, moving van, trollcybus police van, go kart, freight car, bullet train, sports car, gondola. limousine, convertible
beaker, parking meter, drumstick, reel, potter's wheel, beacon, acorn, gasmask, projectile cannon, cash machine, maypole, pay phone, flagpole, bannister, thatch, pill, bottle, pop bottle harbershop, hirdhouse, binoculars, organ, abacus, nail turnstile, heer bottle, oboe viaduct, scorehoard, harrel, pole, syringe, chest, butcher shop, espresso, snorkel, brass
Miscellaneous (Tso)
19

---

Under review as a conference paper at ICLR 2021
Table 9: Performance comparison of ENVISE with the baseline methods on different IL and OL classes CIFAR-100 and TI datasets as presented in Table 2 The + indicates smaller value is better and & indicates greater value is better.
Metric
Intier contice
P9
\ 190
190
05 15
65 1 180
12
1 160
'20 159
050 1
01:00
\
09300
690
130
022
300
9
65
09.
000
"2
2
000
050
650
999
122
9P0
823
55U
990
180
140
$20
E80
990
8L0
650
650
120
120
950
620
1+0
TPU
250
650
0P
680
130
160
100
130
160
160
390
060
120
6.0
980
020
000
580
060
180
SR0
980
120
420
SW0
290
860
580
820
004
120
220
620
420
Z20
820
9P0
620
690
Z60
890
0L0
680
060
2.90
6810
120
0.'0
L80
020
6.'0
060
5W0
280
580
220
P8O
290
62'0
9W'
777
254
150
020
62'0
820
620
9F0
9200
890
9910
020
0172
060
09
1210
98 0
1.:
96 0
06 0
13'0
53
98 0
53'0
09
6.'0
53
6L'0
0
08 0
22'0
08
7171
82 0
SF
9C'0
99'0
99 0
TL.0
2
06 0
0.0
9110
0+0
9
8P3
590
580
990
E773
227
LSU
05
650
S+0
291
600
FP
E+1
9F0
SPD
000
6E0
EPD
060
680
0670
1.0
060
98'
060
D38
61.0
90-0
SFD
251
TED
9900
L8
S9'
SPD
62.0
U9'
951
650
EFD
890
050
0F0
FFU
2FD
SED
SFD
Z60
960
560
P60
68'0
5
960
9P'0
720
610
6
95'0
26'0
L90
58'
190
11:0
85'0
UFD
PS
65
09'0
65
035
ZPD
I+0
SF0
42'0
890
C90
68'
69'
69
560
TM
16'0
11410
S6.0
260
60'
4.70
48'0
S610
233
16'0
88'
18'0
16'
6
SC0
68'
3060
8L'0
9F'
00'0
09'
6''0
191
057
11:0
055
150
914
34
32
22
SSh
T3
OP0
6.0
600
2F0
053
S60
520
SW0
890
S90
620
3C0
+20
060
690
990
SCD
020
457
PED
620
540
0F0
6230
280
090
200
633
390
0
9P0
E20
650
620
250
8P0
657
8E0
520
L2D
950
190
050
550
550
150
S50
050
OF0
LE0
1ED
620
5+5
8P0
560
880
011
050
050
600
33
94'0
P20
800
0.
15D
*+0
657
1E:
SED
L1D
9SD
PS'
E5D
0.0
S5D
55
6
0
EED
2+0
FFD
560
LS7D
01
280
01
860
50
LS7
62:0
0+0
0
19'0
8F-0
08
P20
6910
09'
99'
0910
19'0
85'0
09
90'0
PE
0E'0
IE'0
0
98'0
28'0
0
28'0
0
06'0
PS
450
5P0
65
S90
890
69
LL0
S20
850
290
291
190
190
790
991
050
9C'D
9E0
150
450
620
820
580
060
880
280
PS
180
3330
$900
82.00
150
1990
9910
591
18.0
94.0
9
99'0
E2:0
69'0
8910
12:0
0C'
00'0
02
15'0
8E'D
2800
F80
180
080
1.4
19
RL0
0
4910
090
08'0
58'0
LL'D
900
0
98'0
06
69'
990
07:2
6''
45'0
90'0
25'0
5
0
6''0
88'0
5W0
06'0
080
L80
560
C
L970
0
291
F27
11'0
581
16'0
E6'0
98'0
16'0
1610
18'0
151
68'0
9C'0
SL'
20'0
3.'
02"
RL'0
35'0
P34
69'0
BSSAND
20

---

Under review as a conference paper at ICLR 2021
Table 10: Comparison of ENVISE with the baseline methods IL class pairs C1 and C2 with every OL class on CIFAR-100. The 4 indicates smaller value is better and f indicates greater value is better.
(outtee) Aapa
Metric
Inller ouilice
99 110
F9
11
11
05. 112
010 "10
20 F1
11
11
F1
11
05 1 112
09. 12
112
11
"E ' 11
21 $1
110
112
9 19
11
CE < :12
115 11
2 \ 1.
1 12
44 \
\
00310
00 \
\
1
30
69
60
97'0
62
820
05
89
120
19
790
8F
6P'U
LFD
0.5
1r0
9F0
95
85'0
6+1
9F0
6P'
0.0
20
+60
P60
160
100
750
350
360
150
450
120
9C0
120
820
000
990
85
890
090
890
120
490
000
055
150
LPD
+00
8P0
500
050
050
050
6F0
9F0
+60
080
660
660
0'1
760
060
600
250
[P0
600
CFU
620
0
490
85
890
590
690
P2U
250
L90
190
SP0
600
6F0
054
9P0
0P0
050
2FD
6F0
9F0
260
060
6870
+80
0'1
960
060
62
09
99'
12:10
19
190
9
92.0
AL.00
TE:0
18
12:0
16:0
19
P2:0
62
55
0F'
SE
SP0
18'0
12:0
88
18'0
18'0
0600
1000
FP'
ES'0
25'0
0F'0
PE'
88'0
1ED
72:
190
85
2.7
055
95'0
290
60'0
0P'0
1+0
22'0
51'
19'0
0+'0
160
060
ES'
F60
E60
06'0
060
55'
0..0
2F0
120
69'
28'
590
450
02'0
690
59'
89'0
190
P5'
0F'0
LE'0
LE"O
RP0
0F0
SPO
160
180
880
688
860
460
560
DOW
65
$2.0
85
KS
85U
E2.0
88 0
1200
99'0
00'0
01 0
19
12:00
55
RE'O
6C
SE'0
88 0
O
E3'0
EW0
08
9
98
56
CAL
19'
R2'0
59'
19'0
60
09'0
F64
12:0
PS'
08900
P2'0
12'1
68
07'0
79.0
9
30'0
ZE'0
1F'0
02'0
62'0
9.'0
M9'0
89'0
88'0
99 120
24 159
150
10
12
29 150
10
12
120
120
10
12
" 12
$5 150
90 10
020
12
12
4E0 15
21 1 160
9 \ 12
005 ( 222
\ 10
\ 100
1E0
00 \
960 \
952
30
09
000
905
65
RE
0+
55
PF4
99'
01
290
190
0
190
85
190
05
75
117
000
005
LF0
SE0
IF0
260
2600
S60
560
160
062
PPD
PSU
8HD
300
120
290
FS
LS0
15D
98'
60
150
191
505
0910
RLD
550
4PD
550
SFD
SPD
800
1+0
580
9W'
68
2610
68'
68D
280
PS'
6+
62'0
013
1+0
P90
65'
15'
0910
S80
L90
190
05'
19D
09'0
65'
LE
L37
9+0
S.0
SPD
620
PPD
989
98'0
06.0
83D
7772
680
S5
090
50.
09'
12:
00'
19'
1270
98'
ZV0
7171
65'
19'
620
2E'0
6:'0
02'
0W0
02'0
68'0
27'2
.80
NICO
85
00.0
GF
8F
8F'
6
69
08 0
19'
IED
0133
191
ES00
P9'0
29:0
2910
22.0
30 0
0''
0333
28'0
E8'D
0
98.0
98'D
6.'0
0030
69'0
14:0
65'
15'
C5'D
55'
99'0
291
62
290
09'
190
1.'0
0550
09'0
8910
09'
SF'D
0+'0
12'0
EPD
050
RE'D
LR'D
2800
98'0
68'
5K0
98'0
DN
09'0
59'
08 0
55:0
1F'0
5
52.0
85'0
12'0
69'0
9
19'0
12'0
0
000
02'0
0
0+0
1+'
88'0
0
0
18'0
69
08'0
AAL
BSIANG
C270
1.70
08F
69'0
59'0
2630
29'0
68'0
19'D
EE7
8850
58'0
09'
68'
K270
S50
12'0
PED
09'0
69'0
89'0
0120
6''
21

---

Under review as a conference paper at ICLR 2021
Table 11: Comparison of ENVISE with the baseline methods IL class pairs C3 and C2 with every OL class on CIFAR-100. The 4 indicates smaller value better and f indicates greater value is better.
(outter) Aapa
Metri
Inlier ouilfic
99 100
44 \ 19
99 \ F62
05. 122
61 180
90 F0
10
R2
05. 1 102
60 ( RR
0 10
66 { 180
"E \ 19
100
21 (12
15
C \ 100
20. \, 190
10.
\ 102
1 F40
30 \
\
00310
00 \
\ \
\
1
30
69
09.
60
09
SLD
05D
023
157
0
00
157
SED
057
004
191
6FD
6.
EL73
65
0.7
R0'
0.:
SED
25
800
Z60
260
560
160
830
200
060
10
050
420
SPD
6F0
980
EP0
600
SP0
450
2F0
S80
IP0
CPU
+50
657
000
1P0
580
9F0
090
2P0
1+0
6P0
8PD
020
+#4
4P0
470
980
560
260
0L0
280
680
650
410
PP0
LN0
SP0
8P0
290
1+0
5PD
LND
IF0
UPU
650
RED
980
LFD
000
0P0
8F0
120
S+0
8F0
9W0
SWD
S60
0L0
250
580
52.0
11:0
22:10
98
69 0
9L'0
580
S2
12700
22:0
62
58
P2:0
05
22
95
0F
IF0
08'
1
27:0
T1:0
98
1,:0
SL.0
9
6910
7.0
290
15'
2FD
69
2400
8Y0
98'0
09'
S
Z.:0
66'0
25'D
000
1
003
9.'0
180
98'0
26'0
060
08'D
2C0
62.0
280
020
427
680
02'0
SPO
02'0
690
L90
L80
000
180
9E'0
8E'
00'0
SE'
PED
5W0
L80
02'0
06
68'0
2R0
82.
DOW
99'0
12.1
99 (1
69
95
891
69
12:0
88 0
12:0
F9
E2.
F2.0
98
56.0
09
69
62'0
15'0
61
62'0
6E
0
82'0
22:2
690
90-0
69'0
62.0
89'0
03'0
26:0
12610
MC'0
FS'0
16:1
16
55'0
06.
PS'0
5C'0
25'0
P1'0
25'0
06:0
02'2
9F0
99'0
62'
69 10
19
19
19
29 19
2
19
10
12
19
1
50
" 12
12 19
10
9 \ 19
22
4E0 20
1 12
\ 15
\ 119
2 170
25 1
sogo
2
95
30
69
00
190
000
0.
2FU
SP0
0
120
29
99
890
457
65
65
110
0S
55
PF4
000
2FD
S0
FFO
260
560
060
960
260
094
7A4
RED
52
SC
250
PWO
SPD
#00
+00
550
IFD
P80
GE'D
LED
600
1P0
LP'
LTD
9FD
50
50
560
LND
S90
060
96D
0'1
LED
FFD
13:0
0.0
F29
257
050
580
S+0
SP0
LSU
0F'D
S80
0FD
RED
920
SPD
1+D
L7D
8C'0
RPD
50
90
66D
LSD
06'0
99'0
060
9610
0'1
S5
59'
691
95'
090
0
190
29
#90
85'
29'
P9'
95'0
090
02'0
62'
0#'0
0
LZO
#+0
IFO
82'0
P00
99'
08'0
0
NICO
99'0
99'0
0F
LF0
05'0
0
88 0
691
99 0
19:0
591
22.0
PE'
SFD
9E 0
25'0
88'0
26
160
68D
98'0
0030
29'0
69'0
6910
9
6''0
99'
60
08'0
590
89'0
1910
69'
02'0
69'
P8'D
59'
59'0
0L'0
E3'D
M00
12
DPD
000
IE'D
88'0
06'0
28'0
H60
Z61
1T'D
580
DN
0
12:0
0+'0
0
19'0
09
0
19'0
9910
19'0
89'0
22'0
550
0
19'0
990
02:0
52'
SC'
SE'
L2'0
0
62'0
02'0
0
69'0
L97
CAL
BSIANG
C270
98'
127
89'0
1970
T27
06.0
13:0
6''0
89'0
E270
MCT
06'1)
03:0
69'0
8C.0
5L'
1270
O5'0
91
9
OF'
OCTO
LS'
1970
19'0
29'0
651
15'0
22

---

Under review as a conference paper at ICLR 2021
Table 12: Comparison of ENVISE with the baseline methods IL class pairs Cs and Cs with every OL class on CIFAR-100. The 4 indicates smaller value better and f indicates greater value is better.
(outter) Aapa
Metri
99 \ 190
14
180
02. 102
90 F0
\ 189
69 702
190
05, \ 102
09. 122
19
66 ( 100
"E \ 199
21 \
00 182
10
C 1 100
\, 120
(
\ 100
190
Inller ouiliie
\
90 \
00310
60 \
\
\
\
\
44
30
22
0
6L'
61
530
7870
RRD
98
65
6.
85.0
190
951
65
043
0000
1.'
EFD
00'
S+0
201
+60
P60
160
900
300
15
.60
FC4
62
120
020
610
020
220
65'
06
690
9
0L0
120
500
4PU
290
900
950
0P0
800
4P0
050
020
050
000
050
SPO
960
0'1
260
01
TGU
0-1
880
020
620
120
020
020
120
520
060
690
277
120
120
+P0
LPD
291
S0D
3000
LPD
SP0
LPD
050
6000
050
0.0
PF0
960
011
560
01
M6U
011
680
SP0
170
12
7C7
05
F9
06 0
24:0
RE
69
4C70
29'0
1.50
89
6910
19
OF
0+
69
DF0
62'0
P2:0
26.0
822
14:0
1000
OP'O
900
0+
01'0
58'
$5'0
99
690
890
EY0
1550
0.5
290
891
8910
6.'0
0C'0
9P'0
EPD
RE'0
SE1
PE'
060
260
060
060
F80
68'
610
6
SF0
0970
95'
92'0
L80
$.0
620
0.'0
02'0
P90
95
95'0
090
690
89'
6.'0
8E'0
P04
PFO
4ED
FED
PED
Z600
860
P60
160
L80
062
MD
FF'
61
6''0
6
891
09'0
1170
08
58
11:0
GR
99 0
8
29:0
Z2:0
85
6''0
FF
6''0
1F0
0+0
2E'0
28'0
88'0
0
27.0
18
02.
0.'0
2
69'00
95'1
29'0
29'0
58'0
96.10
18'0
58'0
LS'
M8'0
9C'0
62'0
69'
14'0
9
09'0
9E00
0
zz'2
65'0
26.0
69'0
7772
29'0
69 190
19
19
190
19
100
09 190
190
190
19
100
190
180
90
100
21 190
100
189
190
3E0 190
100
2 \ 190
005 \ 102
\ :90
102
\ 199
190
2
C2
952
30
09
000
905
0
99'
PSU
55
85
6F
95
85
8L.
6.
30
550
95
000
6F0
#F
260
EW'
0
960
560
065
891
061
691
L57
290
290
0.0
950
09'0
0500
PS
6+0
990
SN0
991
505
09/0
ESD
990
222
LED
SFD
IPD
2F0
9C0
9C'0
691
0L'3
780
180
0153
69'
63'0
6910
L570
P9D
2910
12:
657
09'0
OS'D
S5D
00'
LS7
190
9910
650
09'
PS10
990
C20
LED
PFD
0FD
2+0
9E'
7.4
8910
C5D
67
003
S5
09'
06'0
590
090
190
99'0
19'0
777
990
990
5W0
99
29'
79'
S90
59'
22'
02'0
6.0
0F0
IF'
22'
990
0'
18'
27'2
02.3
NICO
1810
99'0
12'
01.2
69'
P9
0..2
98'0
55:0
89
P910
15'0
C01
61
DE
D0 0
F8'D
68'0
68'0
28.0
58D
28'0
0030
0.'0
180
69'0
59'0
140
F2'0
2570
28'0
09'0
29'D
02'0
227
097
98'0
26.0
29'0
39'
92'0
S0'
0+'0
1200
25'
05'0
ER'0
S60
98'0
98'0
08'D
62
DD
52.0
52'0
L1:0
69'0
09'0
62'0
99'0
8910
02.0
12'0
52'0
09'0
69'0
02'0
09
62'0
25'0
82'0
0
06'0
2''0
69'
8900
0
19'0
9910
12'
AAL
BSIANG
23'0
06610
AC1
P270
9W'0
PED0
600
1670
88'1)
527
1C7
52'0
9T3
8170
9
6E'0
E5'
05'0
8570
551
19'0
23

---

Under review as a conference paper at ICLR 2021
Aftention map BBN from
Attention map ESN from second Highned prodiction
Attention map BEN from hightst prodictionn
Artention map BBN from wwonnd Hightet prodict
Incut Image
Ateentinn map TN
Input Image
Attention map
BSN
Nightel prodiction
Lor
Lat
0.93
0.68
0.21
0.95
c 0.68
0.48
0.29
0.91
== 0.77
o=
== 0.33
== 0.92
6= 0.71
= 0.51
C- 0.83
-- 0.56
C- 0.28
-- 0.89
C- 0.81
- 0.49
C. 0.28
C 0.99
0.89
0.52
0.39
0.97
0.74
0.88
0.21
0.96
C- 0.81
C- 0.61
C- 0.23
C- 0.85
C- 0.88
-
0.44
C- 0.39
C 0.89
Figure 11: The attention map visualization representing the regions of the image that the network focuses for classification. The confidence of correct classification is reported below each image
Attondion map BSN from
Attondion BSN from
Aitendion map BSN from niihhed
Actontion map EiE from secoond highted
Impul imagee
Atterzion map
Inpult Image
Atlention mep
awoznd highheat
Lgt
Agt
0.98
0.59
0.25
0.99
0.81
0.51
0.18
0.93
C- 0.91
C- 0.08
C- 0.25
c- 0.97
O 0.96
0.53
O- 0.19
C- 0.99
C- 0.96
C- 0.66
C- 0.22
C- 0.99
C- 0.92
C- 0.51
G- 0.38
C- 0.97
C- 0.78
6- 0.46
C-
0.7
C- 0.83
C- 0.83
C-
C- 0.23
C- 0.86
Figure 12: The attention map visualization representing the regions of the image that the network focuses for classification. The confidence of correct classification is reported below each image
24


Reviews:
[{"review_id": "r91sJ0u1KD2", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "A knowledge distillation framework is proposed for efficient object recognition. In this framework, the teacher network (TN) performs high accuracy prediction while two student networks (SN) mimic the prediction from TN. The first SN learns from TN while the second SN is a binarized form (BSN) of the first SN.  The design is made for online inference is that the BSN first recognizes the image, leaving the rare category objects to be recognized by the TN. Furthermore, an attention supervision scheme is proposed to enhance the CNN prediction by focusing on meaningful image content.  The proposed method has been validated on CIFAR-100 and Tiny-Imagenet.\n\nWhile the distillation for fast recognition and attention supervisions sound interesting, there are a few issues to make the current manuscript not convincing on the contribution side. The details are listed in the following:\n\n1. The major issue is on the motivation side of this framework design. In the paper, a BSN and a TN are both adopted to recognize objects. The reason claimed is for the efficient recognition for rare category objects. However, there is no motivation for why distillation can indeed solve this problem.  This reviewer agrees that objects from category distribute unevenly and class imbalance occurs during training. How class imbalance correlates to knowledge distillation is not clear. A common strategy is to adopt a focal-loss-based loss function to reduce contributions from easy samples. Suppose we want to use two CNNs for cascaded recognition, which is the core idea in this paper, the typical choice is to collect ordinary category objects for the first stage training and rare objects for the second stage training. A pure distillation from the TN does not ensure the differentiation of ordinary and rare categories.  \n\n2. The claim of online distillation is weird. Normally the CNN is trained offline and there is no sign of online model training in the manuscript. As online distillation appears everywhere in the manuscript, this reviewer doubts whether this claim is suitable for illustrating the CNN training process.\n\n3. The attention scheme is from Grad-CAM, which back-propagates CNN prediction to formulate attention supervision terms. The number of categories indicates the number of back-propagation during one training iteration. This computational complexity is tremendous compared to the fast convergence claim of BSN. There shall be some analysis on how to compute the attention maps efficiently in practice.\n\n4. In the experiments, the comparison shall be made to the sota object recognition methods (ResNet, EfficientNet based methods) besides outlier detection methods, As the proposed method focuses on efficient object recogntion. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "lRRMgf_SM1A", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The proposed work trains a teacher-student network using an online distillation paradigm. The student is a binarized network (BSN) trained to be accurate on frequent classes. An attention triplet loss is employed to improve the accuracy of the BSN and its ability to detect outlier classes. Faster convergency of BSN vs Real Valued Student network is claimed. A new metric to evaluate the actual gain in network efficiency is proposed. \n\nStrengths\n- The idea of using the attention triplet loss to improve the quality of attention maps of the BSN and thus increasing BSN accuracy is interesting\n- Experimental evaluation is very thorough\n\nWeaknesses\nThe main weakness of this work lies in the presentation and organization of the paper.\n- It is not clear how outlier detection is performed and why it is needed for the approach to correctly function. \n- It is not clear why at inference time false outliers are then processed by the teacher network. Is this because of the online setting? The setting should be better specified.\n- Sentence \"In conventional knowledge distillation, the attention map of the BSN can focus on the background\neven when the attention map of the TN emphasizes the semantically meaningful regions of the image.\", should be followed by one (possibly more) citations supporting this claim.\n- Lemma 3.1 should be followed by a sketch proof, being one of the main contribution, with the full proof in the appendix (as it is already).\n\nRelated work missing:\nExisting literature binarizing CNN in distillation exists:\n[a]Distillation Guided Residual Learning for Binary Convolutional Neural Networks, 2018\nTriplet losses are used in distillation:\nTriplet Loss for Knowledge Distillation, 2017 (arxiv), IJCNN(2020)\n\n\nThe main reason for my score regards the overall presentation of the paper. The main two contributions are the faster convergence and the use of triplet attention loss. The lemma should have been better highlighted with a sketch proof or at least some intuition so that readers not willing to sift through the appendix could get a grasp of it. The triplet attention loss should be better framed in the introduction (see above). Finally some of the mechanisms of the approach are unclear (how to get the OL score, why the TN must evaluate fp of the BSN) and so on. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "iv-nWrsv3AA", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors tackle the problem of efficient object recognition and outlier detection using online distillation. They propose to complement the standard distillation loss with a triplet loss using attention maps from the teacher to help the student focusing on the relevant part of the input images. They also provide a large set of experiments to validate the proposed approach. The paper is overall well written.\n\nI have two main concerns about the paper:\n- If my understanding is correct, the *hard-positive* samples used to compute the attention triplet loss are selected from the most probable class assigned by the TN while the *hard-negative* are using the second most probable class. Why is it desired for these first and second most probable categories to have different attention maps? Intuitively, if they are semantically similar it would make sense to focus on the same regions of the image to be able to distinguish them from each other.\n- Even though discussed in section A.3, I think that the claim \"BSN is adaptively trained only on IL class images without any knowledge of OL class images\" is incorrect (first paragraph of page 3). Since the TN is used as supervision and as a guide for the SN representation, a lot of knowledge about the OL classes can still be transmitted to the SN.\n\nAnd a few questions:\n- How does the SN perform when directly trained on all classes? It would give a tight lower bound to the performance of ENVISE, since some architectures used for the SN are already very competitive (especially Resnet 18 and 50).\n- It is surprising to see the BSN outperform the RvSN. Is there any reason for which the real-valued network can't converge to the same level of performance as the one achieved by the BSN?\n- Since table 5 shows that similar performances can be reached using a Resnet-18, why using a Densnet-201 for most of the experiments? It seems that it can improve the performance in terms of GiE but would just increase the overall inference latency of ENVISE. \n\n\nMinor remark:\n- Legends of fig 2b, 3 and 4 are too small.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "J0Hb5HG7NGk", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "- The idea is interesting to focus on the frequent labels by distilling a special binary network. But the technical details are not clearly presented and important experiments are missing. Overall, this paper does not reach the acceptance threshold.\n\nDetailed Comments:\n\n- Why is this method an 'online' distillation considering that the teacher network is pretrained and fixed? How do you define offeline distillation and online distillation?\n- \"The OL detector uses the softmax output of the BSN...\" How is the softmax output is used? Do you have a separate class named outlier apart from all the object categories? It should be made clear.\n- Why does BSN converge faster than RvSN considering that BSN is a binarized version of RvSN? What is the retionale behind this?\n- In the experimental part, only comparison to baseline methods are provided, but not to state-of-the-art efficient object recognition methods in the literature. Various methods have been proposed recently to improve the efficiency of an object recognition network, such as distillation, pruning, tensor decomposition, quatization. There should be comparison to these methods as well as baselines. Besides, the performance on Imagenet (not only tiny imagenet) is a common practice in the literature, but is missing in this paper.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes an online distillation method for efficient object recognition. The main idea is to employ a binary student network to learn frequently occurring classes using the pseudo-labels generated by a teacher network. In order to identify rare vs frequent classes, an attention triplet loss is used. The proposed scheme is empirically evaluated on CIFAR-100 and tiny-imagenet datasets.

The major and common concern from reviewers about this draft is the quality of presentation, which has made it difficult to read and understand the ideas and their underlying motivations. While specific instances of this were mentioned by the reviewers, and responded by the authors, the readability issues go beyond these instances. In fact, I could independently observe presentation issues different from those mentioned by the reviewers. For example, in Eq (1), what is eta (never defined before), what is the loss function here for which the gradient update leads to Eq (1)? The proof also is not clear and seems to have issues. For example, in Eq (13) what is the meaning of multiplying two vectors? If it means dot product, it should be denoted more clearly, e.g. as w^T w^* or <w,w^*>. In Eq (12), can alpha be negative? If not, should be clarified why, and if it can be negative, then (<w^*,w>)^2 >= (alpha n)^2 does not need to hold, but this inequality is used in Eq (14) anyway.

Overall, I think the submission can benefit from an overhaul of the writing. I encourage authors to resubmit after improving on that.
Decision: Reject