SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Guy Bar-Shalom
Department of Computer Science Technion - Israel Institute of Technology guy.b@cs.technion.ac.il
Yonatan Geifman
Deci.AI
yonatan@deci.ai
Ran El-Yaniv
Department of Computer Science Technion - Israel Institute of Technology, Deci.AI rani@cs.technion.ac.il
To deploy and operate deep neural models in production, the quality of their predictions, which might be contaminated benignly or manipulated maliciously by input distributional deviations, must be monitored and assessed. Specifically, we study the case of monitoring the healthy operation of a deep neural network (DNN) receiving a stream of data, with the aim of detecting input distributional deviations over which the quality of the network's predictions is potentially damaged. Using selective prediction principles, we propose a distribution deviation detection method for DNNs. The proposed method is derived from a tight coverage generalization bound computed over a sample of instances drawn from the true underlying distribution. Based on this bound, our detector continuously monitors the operation of the network over a test window and fires off an alarm whenever a deviation is detected. Our novel detection method performs on-par or better than the state-of-the-art, while consuming substantially lower computation time (five orders of magnitude reduction) and space complexity. Unlike previous methods, which require at least linear dependence on the size of the source distribution for each detection, rendering them inapplicable to "Google-Scale" datasets, our approach eliminates this dependence, making it suitable for real-world applications. Code is available at https://github.com/BarSGuy/Window-Based-Distribution-Shift-Detection.
A wide range of artificial intelligence applications and services rely on deep neural models because of their remarkable accuracy. When a trained model is deployed in production, its operation should be monitored for abnormal behavior, and a flag should be raised if such is detected. Corrective measures can be taken if the underlying cause of the abnormal behavior is identified. For example, simple distributional changes may only require retraining with fresh data, while more severe cases may require redesigning the model (e.g., when new classes emerge).
In this paper we focus on distribution shift detection in the context of deep neural models and consider the following setting. Pretrained model  $f$  is given, and we presume it was trained with data sampled from some distribution  $P$ . In addition to the dataset used in training  $f$ , we are also given an additional unlabeled sample of data from the same distribution, which is used to train a detector  $D$  (we refer to this as the detection-training set or source set). While  $f$  is used in production to process a stream of emerging input data, we continually feed  $D$  with the most recent window  $W_{k}$  of  $k$  input elements.
The detector also has access to the final layers of the model  $f$  and should be able to determine whether the data contained in  $W_{k}$  came from a distribution different from  $P$ . We emphasize that in this paper we are not considering the problem of identifying single-instance out-of-distribution or outlier instances [28, 21, 22, 16, 38, 35, 34, 11], but rather the information residing in a population of  $k$  instances. While it may seem straightforward to apply single-instance detectors to a window (by applying the detector to each instance in the window), this approach can be computationally expensive since such methods are not designed for window-based tasks; see discussion in Section 3. Moreover, we demonstrate here that computationally feasible single-instance methods can fail to detect population-based deviations. We emphasize that we are not concerned in characterizing types of distribution shifts, nor do we tackle at all the complementary topic of out-of-distribution robustness.
Distribution shift detection has been scarcely considered in the context of deep neural networks (DNNs), however, it is a fundamental topic in machine learning and statistics. The standard method for tackling it is by performing a dimensionality reduction over both the detection-training (source) and test (target) samples, and then applying a two-sample statistical test over these reduced representations to detect a deviation. This is further discussed in Section 3. In particular, deep models can benefit from the semantic representation created by the model itself, which provides meaningful dimensionality reduction, that is readily available at the last layers of the model. Using the embedding layer (or softmax) along with statistical two-sample tests was recently proposed by Lipton et al. [29] and Rabanser et al. [37] who termed solutions of this structure black-box shift detection (BBSD). Using both the univariate Kolmogorov-Smirnov (KS) test and the maximum mean discrepancy (MMD) method, see details below, Rabanser et al. [37] achieve impressive detection results when using MNIST and CIFAR-10 as proxies for the distribution  $\mathcal{P}$ . As shown here, the KS test is also very effective over ImageNet when a stronger model is used (ResNet50 vs ResNet18). However, BBSD methods have the disadvantage of being computationally intensive (and probably inapplicable to read-world datasets) due to the use of two-sample tests between the detection-training set (which can, and is preferred to be the largest possible) and the window  $W_{k}$ ; a complexity analysis is provided in Section 5.
We propose a different approach termed Coverage-Based Detection (CBD), which builds upon selective prediction principles [9, 14]. In this approach, a model quantifies its prediction uncertainty and abstains from predicting uncertain instances. First, we develop a method for selective prediction with guaranteed coverage. This method identifies the best abstaining threshold and coverage bound for a given pretrained classifier  $f$ , such that the resulting empirical coverage will not violate the bound with high probability (when abstention is determined using the threshold). The guaranteed coverage method is of independent interest, and it is analogous to selective prediction with guaranteed risk [14]. Because the empirical coverage of such a classifier is highly unlikely to violate the bound if the underlying distribution remains the same, a systematic violation indicates a distribution shift. To be more specific, given a detection-training sample  $S_{m}$ , our coverage-based detection algorithm computes a fixed number of tight generalization coverage bounds, which are then used to detect a distribution shift in a window  $W_{k}$  of test data. The proposed detection algorithm exhibits remarkable computational efficiency due to its ability to operate independently of the size of  $S_{m}$  during detection, which is crucial when considering "Google-Scale" datasets, such as the JFT-3B dataset. In contrast, the currently available distribution shift detectors rely on a framework that requires significantly higher computational requirements (this framework is illustrated in Figure 3 in Appendix A). A run-time comparison of those baselines w.r.t. our CBD method is provided in Figure 1.
In a comprehensive empirical study, we compared our CBD algorithm with the best-performing baselines, including the KS approach of [37]. Additionally, we investigated the suitability of single-instance detection methods for identifying distribution shifts. For a fair comparison, all methods used the same (publicly available) underlying models: ResNet50, MobileNetV3-S, and ViT-T. To evaluate the effectiveness of our approach, we employed the ImageNet dataset to simulate the source distribution. We then introduced distribution shifts using a range of methods, starting with simple noise and progressing to more sophisticated techniques such as adversarial examples. Based on these experiments, it is evident that CBD is overall significantly more powerful than the baselines across a wide range of test window sizes.
To summarize, the contributions of this paper are: (1) A theoretically justified algorithm (Algorithm 1), that produces a coverage bound, which is of independent interest, and allows for the creation of selective classifiers with guaranteed coverage. (2) A theoretically motivated "windowed" detection
algorithm, CBD (Algorithm 2), which detects a distribution shift over a window; this proposed algorithm exhibits remarkable efficiency compared to state-of-the-art methods (five orders of magnitude better than the best method). (3) A comprehensive empirical study demonstrating significant improvements relative to existing baselines, and introducing the use of single-instance methods for detecting distribution shifts.
We consider the problem of detecting distribution shifts in input streams provided to pretrained deep neural models. Let  $\mathcal{P} \triangleq P_X$  denote a probability distribution over an input space  $\mathcal{X}$ , and assume that a model  $f$  has been trained on a set of instances drawn from  $\mathcal{P}$ . Consider a setting where the model  $f$  is deployed and while being used in production its input distribution might change or even be attacked by an adversary. Our goal is to detect such events to allow for appropriate action, e.g., retraining the model with respect to the revised distribution.
Inspired by Rabanser et al. [37], we formulate this problem as follows. We are given a pretrained model  $f$ , and a detection-training set,  $S_{m} \sim \mathcal{P}^{m}$ . Then we would like to train a detection model to be able to detect a distribution shift; namely, discriminate between windows containing in-distribution (ID) data, and alternative-distribution (AD) data. Thus, given an unlabeled test sample window  $W_{k} \sim Q^{k}$ , where  $Q$  is a possibly different distribution, the objective is to determine whether  $\mathcal{P} \neq Q$ . We also ask what is the smallest test sample size  $k$  required to determine that  $\mathcal{P} \neq Q$ . Since typically the detection-training set  $S_{m}$  can be quite large, we further ask whether it is possible to devise an effective detection procedure whose time complexity is  $o(m)$ .
Distribution shift detection methods often comprise the following two steps: dimensionality reduction, and a two-sample test between the detection-training sample and test samples. In most cases, these methods are "lazy" in the sense that for each test sample, they make a detection decision based on a computation over the entire detection-training sample. Their performance will be sub-optimal if only a subset of the train sample is used. Figure 3 in Appendix A illustrates this general framework.
The use of dimensionality reduction is optional. It can often improve performance by focusing on a less noisy representation of the data. Dimensionality reduction techniques include no reduction, principal components analysis [1], sparse random projection [2], autoencoders [39, 36], domain classifiers, [37] and more. In this work we focus on black box shift detection (BBSD) methods [29], that rely on deep neural representations of the data generated by a pretrained model. The representation extracted from the model will typically utilize either the softmax outputs, acronymed BBSD-Softmax, or the embeddings, acronymed BBSD-Embeddings; for simplicity, we may omit the BBSD acronym. Due to the dimensionality of the final representation, multivariate or multiple univariate two-sample tests can be conducted.
By combining BBSD-Softmax with a Kolmogorov-Smirnov (KS) statistical test [33] and using the Bonferroni correction [3], Rabanser et al. [37] achieved state-of-the-art results in distribution shift detection in the context of image classification (MNIST and CIFAR-10). We acronym their method as BBSD-KS-Softmax (or KS-Softmax). The univariate KS test processes individual dimensions separately; its statistic is calculated by computing the largest difference  $Z$  of the cumulative density functions (CDFs) across all dimensions as follows:  $Z = \sup_{z}|F_{\mathcal{P}}(z) - F_{Q}(z)|$ , where  $F_{Q}$  and  $F_{\mathcal{P}}$  are the empirical CDFs of the detection-training and test data (which are sampled from  $\mathcal{P}$  and  $Q$ , respectively; see Section 2). The Bonferroni correction rejects the null hypothesis when the minimal p-value among all tests is less than  $\frac{\alpha}{d}$ , where  $\alpha$  is the significance level and  $d$  is the number of dimensions. Although less conservative approaches to aggregation exist [20, 30], they usually assume some dependencies among the tests. The maximum mean discrepancy (MMD) method [18] is a kernel-based multivariate test that can be used to distinguish between probability distributions  $\mathcal{P}$  and  $Q$ . Formally,  $MMD^{2}(\mathcal{F},\mathcal{P},Q) = ||\pmb{\mu}_{\mathcal{P}} - \pmb{\mu}_{Q}||_{\mathcal{F}^{2}}^{2}$ , where  $\pmb{\mu}_{\mathcal{P}}$  and  $\pmb{\mu}_{Q}$  are the mean embeddings of  $\mathcal{P}$  and  $Q$  in a reproducing kernel Hilbert space  $\mathcal{F}$ . Given a kernel  $\mathcal{K}$ , and samples,  $\{x_1,x_2,\ldots ,x_m\} \sim \mathcal{P}^m$  and  $\{x_1',x_2',\dots,x_k'\} \sim Q^k$ , an unbiased estimator for  $MMD^{2}$  can be found in [18, 42]. Sutherland et al. [43] and Gretton et al. [18] used the RBF kernel  $\mathcal{K}(x,x') = e^{-\frac{1}{2\sigma^2} ||x - x'||_2^2}$ , where  $2\sigma^2$  is set to the median of the pairwise Euclidean distances
between all samples. By performing a permutation test on the kernel matrix, the p-value is obtained. In our experiments (see Section 6.4), we thus use four population based baselines: KS-Softmax, KS-Embeddings, MMD-Softmax, and MMD-Embeddings.
As mentioned in the introduction, our work is complementary to the topic of single-instance out-of-distribution (OOD) detection [28, 21, 22, 16, 38, 35, 34, 11]. Although these methods can be applied to each instance in a window, they often fail to capture population statistics within the window, making them inadequate for detecting population-based changes. Additionally, many of these methods are computationally expensive and cannot be applied efficiently to large windows. For example, methods such as those described in [41, 27] extract values from each layer in the network, while others such as [28] require gradient calculations. We note that the application of the best single-instance methods such as [41, 27, 28] in our (large scale) empirical setting is computationally challenging and preclude empirical comparison to our method. Therefore, we consider (in Section 6.4) the detection performance of two computationally efficient single-instance baselines: Softmax-Response (abbreviated as Single-instance SR or Single-SR) and Entropy-based (abbreviated as Single-instance Ent or Single-Ent), as described in [4, 6, 21]. Specifically, we apply each single-instance OOD detector to every instance in the window and in the detection-training set. We then use a two-sample t-test to determine the p-value between the uncertainty estimators of each sample.
Finally, we mention [14] who developed a risk generalization bound for selective classifiers [9]. The bound presented in that paper is analogous to the coverage generalization bound we present in Theorem 4.2. The risk bound in [14] can also be used for shift-detection. To apply their risk bound to this task, however, labels, which are not available, are required. CBD detects distribution shifts without using any labels.
4 Proposed Method - Coverage-Based Detection
In this section we present Coverage-Based Detection (CBD), a novel technique for detecting a distribution shift based on selective prediction principles (definitions follow). We develop a tight generalization coverage bound that holds with high probability for ID data, sampled from the source distribution. The main idea is that violations of this coverage bound indicate a distribution shift with high probability. In Section 6.2, we offer an intuitive demonstration that aids in understanding our approach.
4.1 Selection with Guaranteed Coverage
We begin by introducing basic selective prediction terminology and definitions that are required to describe our method. Consider a standard multiclass classification problem, where  $\mathcal{X}$  is some feature space (e.g., raw image data) and  $\mathcal{Y}$  is a finite label set,  $\mathcal{Y} = \{1,2,3,\dots,C\}$ , representing  $C$  classes. Let  $P(X,Y)$  be a probability distribution over  $\mathcal{X} \times \mathcal{Y}$ , and define a classifier as a function  $f: \mathcal{X} \to \mathcal{Y}$ . We refer to  $P$  as the source distribution. A selective classifier [9] is a pair  $(f,g)$ , where  $f$  is a classifier and  $g: \mathcal{X} \to \{0,1\}$  is a selection function [9], which serves as a binary qualifier for  $f$  as follows,
$$
(f, g) (x) \triangleq \left\{ \begin{array}{l l} f (x), & \text {i f} g (x) = 1; \\ \text {d o n ' t k n o w}, & \text {i f} g (x) = 0. \end{array} \right.
$$
A general approach for constructing a selection function based on a given classifier  $f$  is to work in terms of a confidence-rate function [15],  $\kappa_f: \mathcal{X} \to \mathbb{R}^+$ , referred to as CF. The CF  $\kappa_f$  should quantify confidence in predicting the label of  $x$  based on signals extracted from  $f$ . The most common and well-known CF for a classification model  $f$  (with softmax at its last layer) is its softmax response (SR) value [4, 6, 21]. A given CF  $\kappa_f$  can be straightforwardly used to define a selection function:  $g_{\theta}(x) \triangleq g_{\theta}(x|\kappa_f) = \mathbb{1}[\kappa_f(x) \geq \theta]$ , where  $\theta$  is a user-defined constant. For any selection function, we define its coverage w.r.t. a distribution  $\mathcal{P}$  (recall,  $\mathcal{P} \triangleq P_X$ , see Section 2), and its empirical coverage w.r.t. a sample  $S_k \triangleq \{x_1, x_2, \ldots, x_k\}$ , as  $c(\theta, \mathcal{P}) \triangleq \mathbb{E}_{\mathcal{P}}[g_{\theta}(x)]$ , and  $\hat{c}(\theta, S_k) \triangleq \frac{1}{k} \sum_{i=1}^{k} g_{\theta}(x_i)$ , respectively.
Given a bound on the expected coverage for a given selection function, we can use it to detect a distribution shift via violations of the bound. We will now formally state the problem, develop the bound, and demonstrate its use in detecting distribution shifts.
Problem statement. For a classifier  $f$ , a detection-training sample  $S_{m} \sim \mathcal{P}^{m}$ , a confidence parameter  $\delta > 0$ , and a desired coverage  $c^{*} > 0$ , our goal is to use  $S_{m}$  to find a  $\theta$  value (which implies a selection function  $g_{\theta}$ ) that guarantees the desired coverage, with probability  $1 - \delta$ , namely,
$$
\Pr_ {S _ {m}} \left\{c (\theta , \mathcal {P}) <   c ^ {*} \right\} <   \delta . \tag {1}
$$
This means that under coverage should occur with probability of at most  $\delta$ .
Algorithm 1: Selection with guaranteed coverage (SGC)
1 Input: detection-training set:  $S_{m}$ , confidence-rate function:  $\kappa_{f}$ , confidence parameter  $\delta$ , target coverage:  $c^{*}$ .
2 Sort  $S_{m}$  according to  $\kappa_{f}(x_{i})$ ,  $x_{i} \in S_{m}$  (and now assume w.l.o.g. that indices reflect this ordering).
3  $z_{\mathrm{min}} = 1, z_{\mathrm{max}} = m$
4 for  $i = 1$  to  $k = \lceil \log_2m\rceil$  do
5  $z = \lceil (z_{\mathrm{min}} + z_{\mathrm{max}}) / 2\rceil$    
6  $\theta_{i} = \kappa_{f}(x_{z})$    
7 Calculate  $\hat{c}_i(\theta_i,S_m)$    
8 Solve for  $b_{i}^{*}(m,m\cdot \hat{c}_{i}(\theta_{i},S_{m}),\frac{\delta}{k})$  {see Lemma 4.1}  
9 if  $b_{i}^{*}(m,m\cdot \hat{c}_{i}(\theta_{i},S_{m}),\frac{\delta}{k})\leq c^{*}$  then  
10  $z_{\mathrm{max}} = z$    
11 else  
12  $z_{\mathrm{min}} = z$    
13 end if
14 end for
15 Output: bound:  $b_k^* (m,m\cdot \hat{c}_k(\theta_k,S_m),\frac{\delta}{k})$ , threshold:  $\theta_{k}$
A threshold  $\theta$  that guarantees Equation (1) provides a probabilistic lower bound, guaranteeing that coverage  $c$  of ID unseen population (sampled from  $\mathcal{P}$ ) satisfies  $c > c^*$  with probability of at least  $1 - \delta$ . For the remaining of this section, we introduce the selection with guaranteed coverage (SGC) algorithm (Algorithm 1), which outputs a bound  $(b^{*})$  and its corresponding threshold  $(\theta)$ .
The SGC algorithm receives as input a classifier  $f$ , a CF  $\kappa_{f}$ , a confidence parameter  $\delta$ , a target coverage  $c^{*}$ , and a detection-training set  $S_{m}$ . The algorithm performs a
binary search to find the optimal coverage lower bound with confidence  $\delta$ , and outputs a coverage bound  $b^{*}$  and the corresponding threshold  $\theta$ , defining the selection function. A pseudo code of the SGC algorithm appears in Algorithm 1.
Our analysis of the SGC algorithm makes use of Lemma 4.1, which gives a tight numerical (generalization) bound on the expected coverage, based on a test over a sample. The proof of Lemma 4.1 is nearly identical to Langford's proof of Theorem 3.3 in [26], p. 278, where instead of the empirical error used in [26], we use the empirical coverage, which is also a Bernoulli random variable.
Lemma 4.1. Let  $\mathcal{P}$  be any distribution and consider a selection function  $g_{\theta}$  with a threshold  $\theta$  whose coverage is  $c(\theta, \mathcal{P})$ . Let  $0 < \delta < 1$  be given and let  $\hat{c}(\theta, S_m)$  be the empirical coverage w.r.t. the set  $S_m$ , sampled i.i.d. from  $\mathcal{P}$ . Let  $b^*(m, m \cdot \hat{c}(\theta, S_m), \delta)$  be the solution of the following equation:
$$
\underset {b} {\arg \min } \left(\sum_ {j = 0} ^ {m \cdot \hat {c} (\theta , S _ {m})} \binom {m} {j} b ^ {j} (1 - b) ^ {m - j} \leq 1 - \delta\right). \tag {2}
$$
$$
\boldsymbol {P r} _ {S _ {m}} \left\{c (\theta , \mathcal {P}) <   b ^ {*} (m, m \cdot \hat {c} (\theta , S _ {m}), \delta) \right\} <   \delta . \tag {3}
$$
The following is a uniform convergence theorem for the SGC procedure stating that all the calculated bounds are valid simultaneously with a probability of at least  $1 - \delta$ .
Theorem 4.2. (SGC - Uniform convergence) Assume  $S_{m}$  is sampled i.i.d. from  $\mathcal{P}$ , and consider an application of Algorithm 1. For  $k = \lceil \log_2 m \rceil$ , let  $b_{i}^{*}(m, m \cdot \hat{c}_{i}(\theta_{i}, S_{m}), \frac{\delta}{k})$  and  $\theta_{i}$  be the values obtained in the  $i^{th}$  iteration of Algorithm 1. Then,
$$
\boldsymbol {P r} _ {S _ {m}} \left\{\exists i: c (\theta_ {i}, \mathcal {P}) <   b _ {i} ^ {*} (m, m \cdot \hat {c} _ {i} (\theta_ {i}, S _ {m}), \frac {\delta}{k}) \right\} <   \delta .
$$
Proof (sketch - see full proof in the Appendix B.1). Define,  $\mathcal{B}_{\theta_i} \triangleq b_i^*(m, m \cdot \hat{c}_i(\theta_i, S_m), \frac{\delta}{k})$ ,  $\mathcal{C}_{\theta_i} \triangleq c(\theta_i, \mathcal{P})$ , then,
$$
\begin{array}{l} \mathbf {P r} _ {S _ {m}} \left\{\exists i: \mathcal {C} _ {\theta_ {i}} <   \mathcal {B} _ {\theta_ {i}} \right\} = \sum_ {i = 1} ^ {k} \int_ {0} ^ {1} d \theta^ {\prime} \mathbf {P r} _ {S _ {m}} \left\{\mathcal {C} _ {\theta^ {\prime}} <   \mathcal {B} _ {\theta^ {\prime}} \right\} \cdot \mathbf {P r} _ {S _ {m}} \left\{\theta_ {i} = \theta^ {\prime} \right\} \\ <   \sum_ {i = 1} ^ {k} \int_ {0} ^ {1} d \theta^ {\prime} \frac {\delta}{k} \cdot \mathbf {P r} _ {S _ {m}} \left\{\theta_ {i} = \theta^ {\prime} \right\} = \sum_ {i = 1} ^ {k} \frac {\delta}{k} = \delta . \\ \end{array}
$$
We would like to note that in addition to Algorithm 1, we also provide a complementary algorithm that returns a tight coverage upper (rather than lower) bound, along with the corresponding threshold. Details about this algorithm and its application for detecting distribution shifts are discussed in Appendix F.
4.2 Coverage-Based Detection Algorithm