Title: ODIN: OUTLIER DETECTION IN NEURAL NETWORKS

Abstract: Adoption of deep learning in safety-critical systems raise the need for understanding what deep neural networks do not understand. Several methodologies to estimate model uncertainty have been proposed, but these methodologies constrain either how the neural network is trained or constructed. We present Outlier Detection In Neural networks (ODIN), an assumption-free method for detecting outlier observations during prediction, based on principles widely used in manufacturing process monitoring. By using a linear approximation of the hidden layer manifold, we add prediction-time outlier detection to models after training without altering architecture or training. We demonstrate that ODIN efficiently detect outliers during prediction on Fashion-MNIST, ImageNet-synsets and speech command recognition.

1. INTRODUCTION:

Thanks to the powerful transformations learned by deep neural networks, deep learning is applied in an increasing number of applications. But deep neural networks, as all data-driven models, tend to fail when input data differs from training data. Adopting deep learning in increasingly complex and possibly safety-critical systems makes it crucial to know not only whether the model's predictions are accurate, but also whether the model should predict at all. If the model is able to detect outlier observations post-training directly, the system can fall back to safe behaviour minimizing negative consequences of faulty predictions. By understanding the limits of models' learned representations and detecting when observations are not recognized, autonomous decision making based on deep learning can be improved.

In manufacturing process control, predictive models have long been used to predict process outcomes as well as detecting outlier input for decades (Eriksson et al., 2013; Kourti et al., 1996; Ferrer, 2014) . A widely used model for this purpose is Partial Least Squares regression (PLS) (Wold et al., 2001) , which project input data onto a set of linear latent variables prior to prediction. In the latent variable space, the distance from new observations to the training data distribution is used to detect outlier observations. The latent variables can also be used to approximate input observations, meaning that outliers can also be detected by measuring the distance to the latent variable subspace itself. However, the layers of a neural network learn a non-linear mapping from input to output data spaces rather than a single linear subspace. This makes it difficult to directly determine the limits of a neural network model's knowledge in the same manner as for a PLS model.

In this paper we present Outlier Detection In Neural networks (ODIN), a method for detecting outliers during prediction in deep neural networks. Based on principles long used in manufacturing process control, we propose using a linear approximation of intermediate activations to provide a fixed representation of the training data. By comparing new data to this fixed representation we are able to detect outliers during prediction without imposing constraints on architecture or training. This allows us to use ODIN as a plug-in method allowing reliable and safe autonomous decision making based on deep learning.

2. RELATED WORK:

A wide collection of methods allowing neural networks to describe uncertainty in predictions, which can be used to determine if new observations are outliers, have been proposed. For decades, many methods have been formulated within a Bayesian framework (Denker & LeCun, 1991; MacKay, 1992) allowing neural networks to predict probability distributions rather than point inferences. The predictive uncertainty can then be estimated by the entropy or variance of the predicted distribution. Gal & Ghahramani (2016) proposed MC-dropout, using prediction time dropout (Hinton et al., 2012) and Monte-Carlo sampling. In summary, MC-dropout make multiple predictions per inference while the network is randomly perturbed by drop-out which results in a predicted distribution.

A number of alternatives to using dropout to perturb Monte-Carlo samples have been proposed in recent years including: sampling based on batch-normalization parameters (Teye et al., 2018 ), model ensembles (Lakshminarayanan et al., 2017) , multiple prediction heads in a shared base network (Osband et al., 2016; Ilg et al., 2018; Mandelbaum & Weinshall, 2017) , variational inference of weight distribution instead of regular point weights (Blundell et al., 2015) and Laplace approximation of distributions from existing weights (Ritter et al., 2018) . However, the mentioned methods constrain either how the network is constructed (Lakshminarayanan et al., 2017; Osband et al., 2016; Ilg et al., 2018; Mandelbaum & Weinshall, 2017) or how the network is trained (Gal & Ghahramani, 2016; Teye et al., 2018) limiting their use in systems already in production. Several methods also rely on multiple inferences per prediction (Gal & Ghahramani, 2016; Teye et al., 2018; Blundell et al., 2015; Ritter et al., 2018) . This limits their use in real-time systems or systems with limited computational resources.

An alternative approach for estimating uncertainty in classification problems is presented by Chen et al. (2018) where linear classifiers are trained to classify the target output given intermediate layers of a given base model. The linear classifier outputs are then fed to a meta-model that is trained to estimate whether or not the base model is correct. Another alternative approach is proposed by Lee et al. (2018) that leverage Generative Adversarial Networks (GANs) (Goodfellow et al., 2014) to augment the original dataset with border-line outliers. A deep neural classifier is then trained to output high uncertainty for outlier observations and low uncertainty for the original observations. This method does however involve training a GAN that can be difficult to train to convergence (Mescheder et al., 2018) .

Anomaly detection is closely related to prediction time outlier detection, and there are many methods for flagging deviating observations. Non neural-methods include one-class support vector machines (Schölkopf et al., 2001) , local observation density (Breunig et al., 2000) , distances (Knorr et al., 2000) , isolation forests (Liu et al., 2008 ) and many others. A multitide of methods based on deep neural networks, typically autoencoders have been developed as well (Zhou & Paffenroth, 2017; Chalapathy et al., 2017; Zong et al., 2018; Oh & Yun, 2018) . Of particular relevance to this work is Oh & Yun (2018) , that use reconstruction residual as metric to flag outliers. Important to note is that outlier detection systems are based on training a separate model to detect deviant observations. Prediction time outlier detection, on the other hand, describes the limits of a predictive model's knowledge.

3. PREDICTION-TIME OUTLIER DETECTION:

In this section we briefly describe the Partial Least Squares regression model, and how its latent variable approximation of the input data space is used to detect outliers after training. We then describe how we can apply similar principles in neural networks by using a linear approximation of the hidden layer manifold, in a method we call Outlier Detection In Neural networks (ODIN).

3.1. OUTLIER DETECTION IN PARTIAL LEAST SQUARES REGRESSION:

Partial least squares regression (PLS) (Wold, 1975; Geladi & Kowalski, 1986 ) is a widely used regression model within manufacturing process control. Similar to Principal Component Analysis (PCA), PLS assumes that high-dimensional data resides in a sub-space of the original data space spanned by so called latent variables and formulated in terms of matrix decomposition. The PLS model is summarized as:

X = T P T + E Y = T C T + F (1)

where the n × m input matrix X is decomposed into n × k latent variable matrix T = [t 1 ... t k ] and m × k loading matrix P = [p 1 ... p k ] with residual matrix E. The n × p response matrix Y is predicted using T multiplied with response weight matrix C = [c 1 ... c k ] with residuals F . The latent variable-matrix T spans a orthogonal subspace of the data-matrix X and maximize the covariance between X and Y . Note that the PLS model of X is similar to how PCA approximates the data through matrix decomposition but PCA finds latent variables t i that maximize the variance in X rather than the covariance between two matrices.

The columns in T are typically calculated sequentially, where the first column t 1 is found through basis-vectors w 1 ∈ R m and c 1 ∈ R p solving the optimization problem:

maximize Cov(Xw 1 , Y c 1 ) = w T 1 E(XY T )c 1 s.t. ||w 1 || = 1 and ||c 1 || = 1 (2)

The corresponding loading vector p 1 is then chosen so that X1 = Xt 1 p T 1 is uncorrelated with t 1 . This is achieved by selecting p 1 as:

p 1 = X T t 1 ||t 1 || 2 2 (3)

The subsequent vectors t i , w i , p i where i ∈ [2, ..., k] are then calculated by repeating equations 2 and 3 using Xi = Xi-1 -t i-1 p T i-1 instead of X. The latent variable formulation means that PLS carries its own model of the training data and provides two ways of detecting outliers during prediction. Since new observations are projected to the low-dimensional sub-space spanned by the latent variables, both distance to the sub-space itself and distance to training observations within the sub-space can be used to detect outliers. Distance to sub-space is typically measured using the residual sum of squares (RSS). RSS for a new observation row vector x new ∈ R p is given by:

RSS = p j=1 (x new,j -xnew,j ) 2 (4)

where x new is approximated as

x new ≈ xnew = t new P T = x new P P T .

There are several ways to estimate the distance to training observations within the sub-space and a common choice is the Mahalanobis distance. The Mahalanobis distance is a well-used statistical distance measuring how many standard deviations away an observations is from the origin in a multivariate probability normal distribution. Given a fitted PLS model, the training data projections can be approximated as a multivariate normal distribution with covariance matrix C T = E(T T T ).

Then the Mahalanobis distance for x new is given by:

EQUATION

Alternatively, to compensate for using a linear model of a possibly non-linear manifold, a density based metric within the latent variable space may be used. For instance, by using the Local Outlier Factor (LOF) (Breunig et al., 2000) , observations within low-density regions may be flagged as outliers instead of only using the Mahalanobis distance.

3.2. OUTLIER DETECTION IN NEURAL NETWORKS (ODIN) USING LINEAR MANIFOLD:



APPROXIMATION:

In contrast to PLS, data are not typically linearly mapped to a single sub-space in deep neural networks. Instead, a neural network performs as a nested series of non-linear transformations. That is, the activation vector a i of an observation vector x from a layer i is given by:

a i = f i W i f i-1 W i-1 f i-2 ... f 1 (W 1 x) (6)

with weight-matrices W k and activation functions f k .

According to the manifold hypothesis, data is assumed to reside near a region of low dimensionality that may be highly entangled. One possible explanation for why deep neural networks work well is that they are able to disentangle complicated manifolds (Brahma et al., 2016) . If deep neural networks disentangle manifolds, we may find a transformation from a complicated data manifold to a manifold that is approximately linear. If this hypothesis holds true, we can apply a simple trick to add prediction time outlier detection to neural network models by building a model of the data representation within the model. Given n × m activation matrix

A i = [a i,1 ... a i,n ] T of training data X

, where the row vectors a i,k are given by equation 6, we approximate the activation manifold using PCA as:

A i ≈ T Ai P T Ai (7)

where the n × k latent variable matrix T Ai contain the projections of the A i onto the orthonormal sub-space spanned by columns of the m × k loading matrix P Ai .

Now we have a fixed orthogonal approximation of the activation manifold that we can use to detect outliers during prediction analogously to PLS (see 3.1). Meaning that we can measure the distance from new observations to the activation manifold as the residual sum of squares similar to equation 4 using the observation activation a i,new with projection t ai,new = a i,new P Ai :

RSS ai,new = m j=1 (a i,new -t ai,new P T Ai ) j 2 (8)

Similarily, the distance from training observations within the manifold can be measured using Mahalanobis distance or density based approaches as the Local Outlier Factor within the linear approximation. For the Mahalanobis distance, the covariance matrix of the activation projections

C T A i = E(T T Ai T Ai )

is used in in equation 5 as:

EQUATION

We choose to call our method Outlier Detection In Neural networks, ODIN, since we use the intermediate data representations within the neural network itself. In contrast to common Bayesian approaches, we do not perturb predictions to produce prediction distributions. We simply measure the distance from new observations to the training data to determine whether they are outliers or not.

4. EXPERIMENTS:

In the following sections we demonstrate how to detect outliers during prediction time using ODIN on different classification tasks. We choose classification tasks for demonstration since it is straightforward to simulate outliers by excluding a subset of the classes. We also explore how to choose what layer's activations to use and rank of PCA approximation. For comparison, we also perform outlier detection using MC-Dropout (Gal & Ghahramani, 2016) since it is well-established and straightforward to implement even though it has received criticism (Osband et al., 2016) .

4.1. FASHION-MNIST:

To provide a simple classification problem with outliers encountered during prediction, we use the Fashion-MNIST (Xiao et al., 2017) We evaluate ODIN using different outlier metrics (RSS, Mahalanobis distance and LOF), five levels of explained variance (R2) of the PCA model (50-99 %) and different layers of extracted features using the area under the receiver operating characteristic curve (ROC-AUC) as performance metric, (see Figure 1 , complete results in Table A .1). We calculate the ROC-AUC comparing how well test set observations are separated from outlier observations. For comparison, we also used MCdropout to calculate the image-wise entropy from 50 Monte Carlo samples per image and evaluated the results using ROC-AUC in the same way as ODIN.

All metrics clearly separate strong outliers (shoes) from the test set images (Figure 1 left) with RSS being most sucessful (ROC-AUC 0.97 compared to Mahalanobis 0.94 and LOF 0.91). There is a trend that they benefit from increased PCA R2. Surprisingly MC-dropout failed to detect shoe outliers (ROC-AUC 0.495). The subtle outliers (non-shoes) are significantly more difficult to detect (Figure 1 right), and LOF is most successful doing so (best ROC-AUC 0.71 compared to RSS 0.60, Mahalanobis 0.61 and MC-Dropout 0.63).

To conclude, the Fashion-MNIST experiment show that ODIN successfully detect outliers in a simple image classification problem. Strong outliers seem to be best detected by measuring distance to manifold while subtle outliers are better detected in low-density regions of the linear approximation using LOF.

4.2. CATS AND DOGS:

In order to provide a more complex example we demonstrate prediction time outlier detection using a pre-trained CNN on image synsets from ImageNet (Deng et al., 2009) . We train a cat vs. dog classifier on the cat-and dog-synsets from ImageNet and used the car-and horse-synsets as outliers. We used an Inception v3-network (Szegedy et al., 2016) pre-trained on ImageNet, freezing all Inception module weights during training. We replaced the penultimate layer with a hidden layer of 128 ReLu units and a single sigmoid output with 50 % dropout before and after the ReLu-layer and trained for 50 epochs using the Adam optimizer (Kingma & Ba, 2014) and achieved a test set accuracy of 93 %.

We extracted features from each inception module in the Inception-v3 network and pooled them feature-map wise using global average pooling. For each layer of features, we performed outlier detection with five levels of explained variance (R2) for the PCA model ( 50 We are able to convincingly detect cars using our cats vs. dogs classifier (best ROC-AUC for RSS, 0.96, Mahalanobis distance 0.94 and LOF 0.93). Horses are also detected as outliers even though they share visual features with cats and dogs (best ROC-AUC for RSS 0.76, Mahalanobis distance 0.75 and LOF 0.69). Since we used dropout for the last fully connected layers, we also performed MC-dropout achieving similar results (ROC-AUC: 0.86 for cars, 0.61 for horses). The degree of explained variance was not as influental in this experiment as in the Fashion-MNIST experiment, but both Mahalanobis distance and LOF fail to detect both cars and horses using 50 % R2. Interestingly, the performance of all metrics peak at inception module 8 where an auxilliary output was used during training on ImageNet (Szegedy et al., 2016) .

To conclude, the experiment on cats and dogs show that ODIN reliably detect outliers using a pretrained CNN on real-world images. ODIN performs slightly better than MC-dropout but does not rely on using dropout, or any type of constraint on the training procedure. In line with the results from the Fashion-MNIST experiment, higher PCA R2 produce more reliable results.

4.3. SPEECH COMMANDS:

To show that ODIN for prediction time outlier detection works for not only CNN-based image classification, we perform a speech command recognition experiment using a LSTM-based model. We use the Speech Commands dataset (Warden, 2018) that consists of 105 000 short utterances of 35 words recorded at 16 kHz sampling-rate. The words includes digits zero to nine, command words yes, no, up, down, left, right, on, off, stop, go, backward, forward, follow, learn and visual.

The dataset also include a set of arbitrary words bed, bird, cat, dog, happy, house, marvin, sheila, tree and wow. In our experiment, we train a classification model of both digits and command words and use the arbitrary words as outliers.

We transform the utterances into 64 Mel-Frecuency Cepstral Coefficients (Davis & Mermelstein, 1990) , using a frame-length of 2048 samples and frame-stride of 512 samples. We train a three layer bi-directional LSTM-model with 30 % dropout after each LSTM-layer and softmax output (see architecture in For outlier detection, we extracted training set features from the third LSTM-layer and fitted a PCAmodel explaining 99 % of the variance and chose RSS and Mahalanobis distance limits to be the 9th deciles of the training set distances. We then extracted features from the test set and outlier classes and projected them onto the PCA-model and calculated RSS and Mahalanobis distances. Using precision, recall and F1-score we evaluated outlier detection at the 9th deciles (see figure 3 , complete results in Table C .1). We also combined RSS and Mahalanobis distance classifications using OR and AND combined classification. For comparison, we also used MC-dropout with 10 Monte Carlo samples per utterance and calculated the sample-wise Shannon entropy. We performed outlier detection using the 9th decile of training set entropies as threshold, and evaluated MC-dropout in the same manner as ODIN.

Detecting outliers in the speech-commands dataset is difficult for both ODIN and MC-dropout with best word-wise F1-scores ranging from 0.25 for tree, which is phonetically similar to three, to 0.47 for house. ODIN consistently outperform MC-dropout. Additionally, since two metrics are used, we also have the opportunity to raise precision or recall by using AND-or OR-combination of classification according to the two metrics. Depending on the application, either precision or recall may be more important than the other.

The Speech Command experiment shows that ODIN performs well for recurrent neural networks on a speech recognition task in addition to image classification. We also demonstrate how it can be used in practice, by selecting classification threshold and evaluating our choice using precision and recall. We also show how combinations of the different metrics available may be used to tune the precision/recall ratio.

5. CONCLUSIONS:

Deep neural networks are powerful transformers that have shown great success in many applications. But, in order to adopt deep learning in safety-critical applications it is crucial to understand when new observations do not match the data used during training. To imitate linear latent variable models used in manufacturing process monitoring, we use a linear approximation of the hidden layer manifolds to measure distance to and within the manifold. We compare our results to MC-dropout, a well 

:



Figures and Tables:

Figure 3: F1-score (top), precision (middle) and recall (bottom) of outlier detection for the words used as outliers in the Speech Commands experiment comparing MC-Dropout and ODIN. Scores for AND-or OR-combining RSS and Mahal classifications are included to show how they influence the precision and recall ratio.

Figure 1: ROC-AUC of prediction time outlier detection on Fashion-MNIST detecting strong outliers represented by shoes (left) and subtle outliers represented as excluded non-shoe garments (right). Results from MC-dropout are shown as horizontal dashed lines.since all shoe-related information is absent from training data, and excluded clothes are more subtle outliers since the training data contain other upper body garments. We trained a small convolutional neural network (CNN, for architecture see FigureA.1) on five out of the ten classes. We used rmsprop(Tieleman & Hinton, 2012) optimization, categorical cross entropy loss function, batch size 128 for 10 epochs and kept 10 % of the images as validation set and achieved a test set accuracy of 97 %. To use for outlier detection, we extracted features from both max-pooling layers (without global average pooling) for all images.

Table content: <table><tr><td>ROC-AUC</td><td>0.00 0.25 0.50 0.75 1.00</td><td>LOF MaxPool-1</td><td>LOF MaxPool-2</td><td>Mahalanobis MaxPool-1 Shoes Mahalanobis MaxPool-2</td><td>RSS MaxPool-1</td><td>RSS MaxPool-2</td><td>LOF MaxPool-1</td><td>LOF MaxPool-2</td><td>Mahalanobis MaxPool-1 Non-Shoes Mahalanobis MaxPool-2</td><td>RSS MaxPool-1</td><td>RSS MaxPool-2</td><td>MC-dropout R2 50 % R2 80 % R2 90 % R2 95 % R2 99 %</td></tr></table>

ROC-AUC of prediction time outlier detection on Cats and Dogs experiment detecting strong outliers represented by cars (top row) and subtle outliers represented as horses (bottom row). Inception module, i.e. network depth, of extracted features used for outlier detection is shown on the X-axes. Results from MC-dropout are shown as horizontal dashed lines.

Table content: <table><tr><td/><td>1.00</td><td>RSS</td><td>Mahalanobis</td><td>LOF</td><td/></tr><tr><td>Car -ROC-AUC Horse -ROC-AUC</td><td>0.00 0.25 0.50 0.75 0.25 0.50 0.75 1.00 0.00</td><td>1 2 3 4 5 6 7 8 9 1011 Inception module</td><td>1 2 3 4 5 6 7 8 9 1011 Inception module</td><td>1 2 3 4 5 6 7 8 9 1011 Inception module</td><td>MC-dropout R2 50 % R2 80 % R2 90 % R2 95 % R2 99 %</td></tr><tr><td colspan="3">Figure 2:</td><td/><td/><td/></tr><tr><td/><td/><td/><td/><td colspan="2">-99 %) and different outlier</td></tr></table>

established Bayesian approach, and consistently detect outliers post-training without imposing any constraints on either architecture or training procedure. We demonstrate our method in two image classification experiments, with and without a pre-trained network, and a speech recognition example using a recurrent neural network. By defining the limits of our neural networks' knowledge, ODIN contribute to safer use of deep learning. Chong Zhou and Randy C. Paffenroth. Anomaly Detection with Robust Deep Autoencoders. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '17, pp. 665-674, 2017. doi: 10.1145/3097983.3098052. Bo Zong, Qi Song, Martin Renqiang Min, Wei Cheng, Cristian Lumezanu, Daeki Cho, and Haifeng Chen. Deep Autoencoding Gaussian Mixture Model for Unsupervised Anomaly Detection. In Sixth International Conference on Learning Representations, 2018. 1: ROC-AUC for all RSS configurations used in Cats and dogs experiment.

Table content: <table><tr><td colspan="4">Table A.1: ROC-AUC for all configurations used in Fashion-MNIST experiment. APPENDIX B CATS AND DOGS</td></tr><tr><td/><td/><td/><td>Non-Shoes</td><td>Shoes</td></tr><tr><td>Measure</td><td/><td>Features</td><td>PCA R2</td></tr><tr><td>LOF PCA R2 0.50</td><td colspan="3">MaxPool-1 0.50 0.80 0.90 Inception module 0.95 1 0.709278 0.497676 0.632584 0.806461 Car Horse 0.705666 0.904777 0.685191 0.894012 0.672836 0.905983 0.99 2 0.746097 0.504473 0.666276 0.900635 3 0.576762 0.470429</td></tr><tr><td/><td>4</td><td colspan="2">MaxPool-2 0.50 0.699465 0.505362 0.591984 0.783379</td></tr><tr><td/><td>5</td><td/><td>0.80 0.789033 0.517313 0.685097 0.878966</td></tr><tr><td/><td>6</td><td/><td>0.90 0.793035 0.540525 0.679432 0.890894</td></tr><tr><td/><td>7</td><td/><td>0.95 0.887417 0.538365 0.683418 0.898365</td></tr><tr><td colspan="4">0.99 0.934970 0.719343 0.679545 0.912277 Mahalanobis MaxPool-1 0.50 8 9 0.931100 0.693248 0.429212 0.725880 0.80 10 0.781649 0.703906 0.555856 0.804085 0.90 11 0.794187 0.725780 0.584215 0.893431 0.95 0.80 1 0.705079 0.568177 0.576945 0.940009 0.99 2 0.710353 0.516260 0.578156 0.919574 3 0.637867 0.526424</td></tr><tr><td/><td>4</td><td colspan="2">MaxPool-2 0.50 0.729584 0.535119 0.382974 0.427928</td></tr><tr><td/><td>5</td><td/><td>0.80 0.778540 0.518596 0.543957 0.590699</td></tr><tr><td/><td>6</td><td/><td>0.90 0.848603 0.540108 0.577311 0.780444</td></tr><tr><td/><td>7</td><td/><td>0.95 0.906682 0.567508 0.589486 0.850849</td></tr><tr><td>RSS 0.90</td><td colspan="3">0.99 0.943456 0.758081 0.605115 0.916437 MaxPool-1 0.50 0.931845 0.697327 0.602458 0.847860 0.80 10 8 9 0.821543 0.715034 0.591201 0.927887 0.90 11 0.867583 0.715177 0.571679 0.936623 0.95 1 0.681670 0.536555 0.574044 0.918533 0.99 2 0.702568 0.511546 0.587556 0.923430 3 0.682821 0.527521</td></tr><tr><td/><td>4</td><td colspan="2">MaxPool-2 0.50 0.781556 0.516929 0.564947 0.794195</td></tr><tr><td/><td>5</td><td/><td>0.80 0.773486 0.520306 0.589945 0.898555</td></tr><tr><td/><td>6</td><td/><td>0.90 0.820797 0.561280 0.603539 0.928036</td></tr><tr><td/><td>7</td><td/><td>0.95 0.889807 0.584810 0.603329 0.947959</td></tr><tr><td colspan="3">8 MC-dropout -9 10</td><td>0.99 0.958050 0.750493 0.569872 0.966362 -0.933917 0.679214 0.626128 0.493604 0.839283 0.712403</td></tr><tr><td/><td colspan="2">11</td><td>0.854041 0.701439</td></tr><tr><td>0.95</td><td>1</td><td/><td>0.698412 0.545075</td></tr><tr><td/><td>2</td><td/><td>0.741141 0.526304</td></tr><tr><td/><td>3</td><td/><td>0.691768 0.486097</td></tr><tr><td/><td>4</td><td/><td>0.775744 0.509923</td></tr><tr><td/><td>5</td><td/><td>0.802838 0.540941</td></tr><tr><td/><td>6</td><td/><td>0.837737 0.579207</td></tr><tr><td/><td>7</td><td/><td>0.905673 0.563418</td></tr><tr><td/><td>8</td><td/><td>0.960155 0.753794</td></tr><tr><td/><td>9</td><td/><td>0.933413 0.685189</td></tr><tr><td/><td colspan="2">10</td><td>0.848899 0.729979</td></tr><tr><td/><td colspan="2">11</td><td>0.836027 0.701428</td></tr><tr><td>0.99</td><td>1</td><td/><td>0.755548 0.549899</td></tr><tr><td/><td>2</td><td/><td>0.783529 0.524484</td></tr><tr><td/><td>3</td><td/><td>0.726799 0.488213</td></tr><tr><td/><td>4</td><td/><td>0.783420 0.499200</td></tr><tr><td/><td>5</td><td/><td>0.829163 0.502960</td></tr><tr><td/><td>6</td><td/><td>0.812442 0.559076</td></tr><tr><td/><td>7</td><td/><td>0.908622 0.565414</td></tr><tr><td/><td>8</td><td/><td>0.964234 0.739167</td></tr><tr><td/><td>9</td><td/><td>0.928095 0.691285</td></tr><tr><td/><td colspan="2">10</td><td>0.853077 0.715210</td></tr><tr><td/><td colspan="2">11</td><td>0.827277 0.701735</td></tr><tr><td colspan="2">MC-dropout -</td><td/><td>0.874272 0.618843</td></tr></table>

2: ROC-AUC for all LOF configurations used in Cats and dogs experiment.

Table content: <table><tr><td/><td/><td>Car</td><td>Horse</td></tr><tr><td>PCA R2</td><td>Inception module</td><td/></tr><tr><td>0.50</td><td>1</td><td colspan="2">0.747062 0.506228</td></tr><tr><td/><td>2</td><td colspan="2">0.712447 0.487221</td></tr><tr><td/><td>3</td><td colspan="2">0.669598 0.552026</td></tr><tr><td/><td>4</td><td colspan="2">0.728203 0.551993</td></tr><tr><td/><td>5</td><td colspan="2">0.699202 0.533573</td></tr><tr><td/><td>6</td><td colspan="2">0.603274 0.536457</td></tr><tr><td/><td>7</td><td colspan="2">0.567212 0.563133</td></tr><tr><td/><td>8</td><td colspan="2">0.383306 0.535602</td></tr><tr><td/><td>9</td><td colspan="2">0.591717 0.508432</td></tr><tr><td/><td>10</td><td colspan="2">0.571455 0.400476</td></tr><tr><td/><td>11</td><td colspan="2">0.077551 0.317552</td></tr><tr><td>0.80</td><td>1</td><td colspan="2">0.763223 0.551675</td></tr><tr><td/><td>2</td><td colspan="2">0.744682 0.576301</td></tr><tr><td/><td>3</td><td colspan="2">0.688884 0.548649</td></tr><tr><td/><td>4</td><td colspan="2">0.754682 0.512598</td></tr><tr><td/><td>5</td><td colspan="2">0.747763 0.531227</td></tr><tr><td/><td>6</td><td colspan="2">0.730615 0.581553</td></tr><tr><td/><td>7</td><td colspan="2">0.801094 0.624731</td></tr><tr><td/><td>8</td><td colspan="2">0.869052 0.611969</td></tr><tr><td/><td>9</td><td colspan="2">0.883470 0.625948</td></tr><tr><td/><td>10</td><td colspan="2">0.804274 0.518387</td></tr><tr><td/><td>11</td><td colspan="2">0.689432 0.634227</td></tr><tr><td>0.90</td><td>1</td><td colspan="2">0.780525 0.564537</td></tr><tr><td/><td>2</td><td colspan="2">0.764583 0.547487</td></tr><tr><td/><td>3</td><td colspan="2">0.711975 0.522324</td></tr><tr><td/><td>4</td><td colspan="2">0.733740 0.536643</td></tr><tr><td/><td>5</td><td colspan="2">0.767390 0.552443</td></tr><tr><td/><td>6</td><td colspan="2">0.764265 0.557794</td></tr><tr><td/><td>7</td><td colspan="2">0.820841 0.618635</td></tr><tr><td/><td>8</td><td colspan="2">0.902954 0.635213</td></tr><tr><td/><td>9</td><td colspan="2">0.916429 0.634161</td></tr><tr><td/><td>10</td><td colspan="2">0.805897 0.554186</td></tr><tr><td/><td>11</td><td colspan="2">0.799998 0.676703</td></tr><tr><td>0.95</td><td>1</td><td colspan="2">0.747972 0.579920</td></tr><tr><td/><td>2</td><td colspan="2">0.757642 0.549691</td></tr><tr><td/><td>3</td><td colspan="2">0.675628 0.514725</td></tr><tr><td/><td>4</td><td colspan="2">0.741327 0.556291</td></tr><tr><td/><td>5</td><td colspan="2">0.774473 0.529374</td></tr><tr><td/><td>6</td><td colspan="2">0.748180 0.580589</td></tr><tr><td/><td>7</td><td colspan="2">0.815929 0.623394</td></tr><tr><td/><td>8</td><td colspan="2">0.915607 0.654467</td></tr><tr><td/><td>9</td><td colspan="2">0.908326 0.652526</td></tr><tr><td/><td>10</td><td colspan="2">0.815633 0.567475</td></tr><tr><td/><td>11</td><td colspan="2">0.836487 0.684016</td></tr><tr><td>0.99</td><td>1</td><td colspan="2">0.746557 0.587880</td></tr><tr><td/><td>2</td><td colspan="2">0.764177 0.591805</td></tr><tr><td/><td>3</td><td colspan="2">0.699465 0.553276</td></tr><tr><td/><td>4</td><td colspan="2">0.755077 0.535602</td></tr><tr><td/><td>5</td><td colspan="2">0.758136 0.537180</td></tr><tr><td/><td>6</td><td colspan="2">0.773310 0.582332</td></tr><tr><td/><td>7</td><td colspan="2">0.826850 0.636759</td></tr><tr><td/><td>8</td><td colspan="2">0.923139 0.684981</td></tr><tr><td/><td>9</td><td colspan="2">0.926922 0.674828</td></tr><tr><td/><td>10</td><td colspan="2">0.846399 0.610927</td></tr><tr><td/><td>11</td><td colspan="2">0.864984 0.703895</td></tr><tr><td colspan="2">MC-dropout -</td><td colspan="2">0.874272 0.618843</td></tr></table>

3: ROC-AUC for all Mahalanobis distance configurations used in Cats and dogs experiment.

Table content: <table><tr><td/><td/><td>Car</td><td>Horse</td></tr><tr><td>PCA R2</td><td>Inception module</td><td/></tr><tr><td>0.50</td><td>1</td><td colspan="2">0.708335 0.541325</td></tr><tr><td/><td>2</td><td colspan="2">0.616837 0.470078</td></tr><tr><td/><td>3</td><td colspan="2">0.718214 0.559318</td></tr><tr><td/><td>4</td><td colspan="2">0.705265 0.498235</td></tr><tr><td/><td>5</td><td colspan="2">0.645487 0.496645</td></tr><tr><td/><td>6</td><td colspan="2">0.571521 0.534713</td></tr><tr><td/><td>7</td><td colspan="2">0.553561 0.486930</td></tr><tr><td/><td>8</td><td colspan="2">0.261918 0.527005</td></tr><tr><td/><td>9</td><td colspan="2">0.390027 0.471438</td></tr><tr><td/><td>10</td><td colspan="2">0.341882 0.479771</td></tr><tr><td/><td>11</td><td colspan="2">0.082321 0.443434</td></tr><tr><td>0.80</td><td>1</td><td colspan="2">0.702853 0.506118</td></tr><tr><td/><td>2</td><td colspan="2">0.692360 0.500746</td></tr><tr><td/><td>3</td><td colspan="2">0.636694 0.448763</td></tr><tr><td/><td>4</td><td colspan="2">0.738049 0.471821</td></tr><tr><td/><td>5</td><td colspan="2">0.763015 0.545338</td></tr><tr><td/><td>6</td><td colspan="2">0.751699 0.487402</td></tr><tr><td/><td>7</td><td colspan="2">0.850522 0.507993</td></tr><tr><td/><td>8</td><td colspan="2">0.914905 0.640597</td></tr><tr><td/><td>9</td><td colspan="2">0.900169 0.632538</td></tr><tr><td/><td>10</td><td colspan="2">0.662997 0.630082</td></tr><tr><td/><td>11</td><td colspan="2">0.626464 0.695660</td></tr><tr><td>0.90</td><td>1</td><td colspan="2">0.702995 0.518343</td></tr><tr><td/><td>2</td><td colspan="2">0.710210 0.517258</td></tr><tr><td/><td>3</td><td colspan="2">0.610357 0.459399</td></tr><tr><td/><td>4</td><td colspan="2">0.711855 0.533957</td></tr><tr><td/><td>5</td><td colspan="2">0.759704 0.506984</td></tr><tr><td/><td>6</td><td colspan="2">0.807125 0.517784</td></tr><tr><td/><td>7</td><td colspan="2">0.882330 0.551248</td></tr><tr><td/><td>8</td><td colspan="2">0.912471 0.718576</td></tr><tr><td/><td>9</td><td colspan="2">0.920168 0.670431</td></tr><tr><td/><td>10</td><td colspan="2">0.710473 0.635235</td></tr><tr><td/><td>11</td><td colspan="2">0.777926 0.689542</td></tr><tr><td>0.95</td><td>1</td><td colspan="2">0.686494 0.533299</td></tr><tr><td/><td>2</td><td colspan="2">0.721098 0.532510</td></tr><tr><td/><td>3</td><td colspan="2">0.617374 0.508837</td></tr><tr><td/><td>4</td><td colspan="2">0.743926 0.505614</td></tr><tr><td/><td>5</td><td colspan="2">0.795448 0.555107</td></tr><tr><td/><td>6</td><td colspan="2">0.815403 0.526479</td></tr><tr><td/><td>7</td><td colspan="2">0.879336 0.553616</td></tr><tr><td/><td>8</td><td colspan="2">0.929740 0.733433</td></tr><tr><td/><td>9</td><td colspan="2">0.928424 0.684038</td></tr><tr><td/><td>10</td><td colspan="2">0.781117 0.693994</td></tr><tr><td/><td>11</td><td colspan="2">0.821159 0.693193</td></tr><tr><td>0.99</td><td>1</td><td colspan="2">0.702316 0.536051</td></tr><tr><td/><td>2</td><td colspan="2">0.710956 0.535755</td></tr><tr><td/><td>3</td><td colspan="2">0.653425 0.507094</td></tr><tr><td/><td>4</td><td colspan="2">0.793847 0.522784</td></tr><tr><td/><td>5</td><td colspan="2">0.796423 0.527992</td></tr><tr><td/><td>6</td><td colspan="2">0.841915 0.568659</td></tr><tr><td/><td>7</td><td colspan="2">0.889687 0.595533</td></tr><tr><td/><td>8</td><td colspan="2">0.939224 0.724782</td></tr><tr><td/><td>9</td><td colspan="2">0.931264 0.669642</td></tr><tr><td/><td>10</td><td colspan="2">0.833341 0.700463</td></tr><tr><td/><td>11</td><td colspan="2">0.844568 0.704048</td></tr><tr><td colspan="2">MC-dropout -</td><td colspan="2">0.874272 0.618843</td></tr></table>

1: F1-score, precision and recall for all outlier words in the Speech Commands experiment.

Table content: <table><tr><td/><td/><td>F1-score Precision</td><td>Recall</td></tr><tr><td>Method</td><td>Word</td><td/></tr><tr><td>MC-dropout</td><td>bed</td><td colspan="2">0.401790 0.427691 0.378848</td></tr><tr><td/><td>bird</td><td colspan="2">0.439555 0.459788 0.421027</td></tr><tr><td/><td>cat</td><td colspan="2">0.394107 0.423164 0.368784</td></tr><tr><td/><td>dog</td><td colspan="2">0.353033 0.397995 0.317199</td></tr><tr><td/><td>happy</td><td colspan="2">0.337388 0.379331 0.303797</td></tr><tr><td/><td>house</td><td colspan="2">0.229379 0.284513 0.192144</td></tr><tr><td/><td colspan="3">marvin 0.253497 0.307327 0.215714</td></tr><tr><td/><td>sheila</td><td colspan="2">0.167971 0.214615 0.137982</td></tr><tr><td/><td>tree</td><td colspan="2">0.114615 0.142017 0.096077</td></tr><tr><td/><td>wow</td><td colspan="2">0.330767 0.378954 0.293453</td></tr><tr><td>Combined AND</td><td>bed</td><td colspan="2">0.300694 0.500000 0.214995</td></tr><tr><td/><td>bird</td><td colspan="2">0.270776 0.474515 0.189438</td></tr><tr><td/><td>cat</td><td colspan="2">0.258042 0.457393 0.179714</td></tr><tr><td/><td>dog</td><td colspan="2">0.227068 0.431012 0.154135</td></tr><tr><td/><td>happy</td><td colspan="2">0.225473 0.421896 0.153846</td></tr><tr><td/><td>house</td><td colspan="2">0.402760 0.597209 0.303833</td></tr><tr><td/><td colspan="3">marvin 0.197153 0.390141 0.131905</td></tr><tr><td/><td>sheila</td><td colspan="2">0.216491 0.407661 0.147379</td></tr><tr><td/><td>tree</td><td colspan="2">0.109530 0.226786 0.072200</td></tr><tr><td/><td>wow</td><td colspan="2">0.334311 0.542283 0.241639</td></tr><tr><td>Combined OR</td><td>bed</td><td colspan="2">0.420017 0.375098 0.477160</td></tr><tr><td/><td>bird</td><td colspan="2">0.407906 0.369685 0.454942</td></tr><tr><td/><td>cat</td><td colspan="2">0.387927 0.353131 0.430330</td></tr><tr><td/><td>dog</td><td colspan="2">0.359437 0.337883 0.383929</td></tr><tr><td/><td>happy</td><td colspan="2">0.395170 0.359856 0.438169</td></tr><tr><td/><td>house</td><td colspan="2">0.474430 0.419086 0.546616</td></tr><tr><td/><td colspan="3">marvin 0.335507 0.317853 0.355238</td></tr><tr><td/><td>sheila</td><td colspan="2">0.410965 0.369188 0.463403</td></tr><tr><td/><td>tree</td><td colspan="2">0.250456 0.231028 0.273451</td></tr><tr><td/><td>wow</td><td colspan="2">0.445418 0.399925 0.502591</td></tr><tr><td colspan="2">Mahalanobis distance bed</td><td colspan="2">0.376307 0.408853 0.348560</td></tr><tr><td/><td>bird</td><td colspan="2">0.355674 0.396193 0.322674</td></tr><tr><td/><td>cat</td><td colspan="2">0.398107 0.427201 0.372723</td></tr><tr><td/><td>dog</td><td colspan="2">0.329524 0.379205 0.291353</td></tr><tr><td/><td>happy</td><td colspan="2">0.341977 0.384102 0.308179</td></tr><tr><td/><td>house</td><td colspan="2">0.464785 0.482671 0.448178</td></tr><tr><td/><td colspan="3">marvin 0.264140 0.318334 0.225714</td></tr><tr><td/><td>sheila</td><td colspan="2">0.436551 0.455180 0.419387</td></tr><tr><td/><td>tree</td><td colspan="2">0.150050 0.181452 0.127914</td></tr><tr><td/><td>wow</td><td colspan="2">0.444995 0.469420 0.422986</td></tr><tr><td>RSS</td><td>bed</td><td colspan="2">0.371544 0.404442 0.343595</td></tr><tr><td/><td>bird</td><td colspan="2">0.354417 0.394534 0.321705</td></tr><tr><td/><td>cat</td><td colspan="2">0.272933 0.321119 0.237322</td></tr><tr><td/><td>dog</td><td colspan="2">0.285948 0.340026 0.246711</td></tr><tr><td/><td>happy</td><td colspan="2">0.318928 0.363920 0.283836</td></tr><tr><td/><td>house</td><td colspan="2">0.426921 0.454789 0.402272</td></tr><tr><td/><td colspan="3">marvin 0.299346 0.350128 0.261429</td></tr><tr><td/><td>sheila</td><td colspan="2">0.225788 0.275249 0.191395</td></tr><tr><td/><td>tree</td><td colspan="2">0.242328 0.273181 0.217737</td></tr><tr><td/><td>wow</td><td colspan="2">0.356695 0.400941 0.321244</td></tr></table>

Peer review (iclr2019):
 This paper studies how to better recognise classes that are found in the training set, and identify those that are not in the training set, using neural network methods. They call this as "outlier detection in neural networks". I have several questions about this paper. First, the definition of outlier is ambiguous. The classes that are not in the training set do not necessarily belong to outliers. In this respect, I felt the evaluations are problematic, as the unused data have been considered as outliers. The performance metrics used are therefore not entirely appropriate, as they are used to evaluate how good is the classifier. Even you define the unused data as outliers, such outliers cannot be directly considered as outliers in the network, a drift of the training model from the actual physical model should be quantified with different inputs to the system. Second, the actual contribution is the incorporation of equation (7) and (8) into the network layers, based on a linear approximation - something like applying PCA type of operation on the activations in different layers. There is no validation on the assumptions made, or any empirical study on the assumptions made about the "disentangle manifolds" and "approximately linear". Third, the paper lacks proofread and there are some typos and grammtical issues visual at several places, such as "systems raises", "that uses reconstruction residual", "multitide" "see 3.1", "CNN:s", etc. Fourth, in the experiments, the linear operation is applied to different layers and different networks, what*s the reasoning behind this? Some specific comments: How t_new is obtained in equation (4)?