SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Our coverage-based detection (CBD) algorithm applies SGC to  $C_{\mathrm{target}}$  target coverages uniformly spread between the interval [0.1, 1], excluding the coverage of 1. We set  $C_{\mathrm{target}} = 10, \delta = 0.01$ , and  $\kappa_f(x) = 1 - \mathrm{Entropy}(x)$  for all our experiments; our method appears to be robust to those hyper-parameters, as we demonstrate in Appendix E. Each application  $j$  of SGC on the same sample  $S_m \sim \mathcal{P}^m$  with a target coverage of  $c_j^*$  produces a pair:  $(\hat{b}_j^*, \theta_j)$ , which represent a bound and a threshold, respectively. We define  $\delta(\hat{c}|b^*)$  to be a binary function that indicates a bound violation, where  $\hat{c}$  is the empirical coverage (of a sample) and  $b^*$  is a bound,  $\delta(\hat{c}|b^*) = \mathbb{1}[\hat{c} \leq b^*]$ . Thus, given a window of  $k$  samples from an alternative distribution,  $W_k \sim Q^k$ , we define the sum of bound violations,  $V$ , as follows,
$$
V = \frac {1}{C _ {\text {t a r g e t}}} \sum_ {j = 1} ^ {C _ {\text {t a r g e t}}} \left(b _ {j} ^ {*} - \hat {c} \left(\theta_ {j}, W _ {k}\right)\right) \cdot \delta \left(\hat {c} \left(\theta_ {j}, W _ {k}\right) \mid b _ {j} ^ {*}\right) = \frac {1}{k \cdot C _ {\text {t a r g e t}}} \sum_ {j = 1} ^ {C _ {\text {t a r g e t}}} \sum_ {i = 1} ^ {k} \left(b _ {j} ^ {*} - g _ {\theta_ {j}} \left(x _ {i}\right)\right) \cdot \delta \left(\hat {c} _ {j} \mid b _ {j} ^ {*}\right), \tag {4}
$$
where we obtain the last equality by using  $\hat{c}_j\triangleq \hat{c} (\theta_j,W_k) = \frac{1}{k}\sum_{i = 1}^k g_{\theta_j}(x_i)$ . Considering Figure 2b in Section 6.2, the quantity  $V$  is the sum of distances from the violations (red dots) to the linear diagonal representing the coverage bounds (black line).
Algorithm 2: Coverage-Based Detection
1 // Fit  
2 Input Training:  $S_{m}, \delta, \kappa_{f}$ ,  $C_{\text{target}}$   
3 Generate  $C_{\text{target}}$  uniformly spread coverages  $c^{*}$   
4 for  $j = 1$  to  $C_{\text{target}}$  do  
5 |  $b_{j}^{*}, \theta_{j} = \mathsf{SGC}(S_{m}, \delta, c_{j}^{*}, \kappa_{f})$   
6 end for  
7 Output Training:  $\{(b_{j}^{*}, \theta_{j})\}_{j=1}^{C_{\text{target}}}$   
8 // Detect  
9 Input Detection:  $\{(b_{j}^{*}, \theta_{j})\}_{j=1}^{C_{\text{target}}}, \kappa_{f}, \alpha, k$   
10 while True do  
11 Receive window  $W_{k} = \{x_{1}, x_{2}, \ldots, x_{k}\}$   
12 Calculate  $V$  [see Equation (4)]  
13 Obtain p-value from t-test,  $H_{0}: V = 0, H_{1}: V > 0$   
14 if  $p_{value} < \alpha$  then  
15 Shift Detected ← True  
16 Output Detection: Shift Detected,  $p_{value}$   
17 end if  
18 end while
When  $Q$  equals  $\mathcal{P}$ , which implies that there is no distribution shift, we expect that all the bounds (computed by SGC over  $S_{m}$ ) will hold over  $W_{k}$ , namely  $\delta (\hat{c}_j|b_j^*) = 0$  for every iteration  $j$ ; in this case,  $V = 0$ . Otherwise,  $V$  will indicate the violation magnitude.
Since  $V$  represents the average of  $k\cdot C_{\mathrm{target}}$  values (if at least one bound violation occurs), for cases where  $k\cdot C_{\mathrm{target}}\gg 30$  (as in all our experiments), we can assume that  $V$  follows a nearly normal distribution [25] and perform a t-test to test the null hypothesis,  $H_0:V = 0$  against the alternative hypothesis,  $H_{1}:V > 0$  .The null hypothesis is rejected if the p-value is less than the specified significance level  $\alpha$  A pseudocode of our coverage-based detection algorithm appears in Algorithm 2.
To fit our detector, we apply SGC (Algo
rithm 1) on the detection-training set,  $S_{m}$ , for  $C_{\mathrm{target}}$  times, in order to construct the pairs  $\{(b_j^*,\theta_j)\}_{j = 1}^{C_{\mathrm{target}}}$ . Our detection model utilizes these pairs to monitor a given model, receiving at each time instant a test sample window of size  $k$  (user defined),  $W_{k} = \{x_{1},x_{2},\ldots ,x_{k}\}$ , which is inspected to see if its content is distributionally shifted from the underlying distribution reflected by the detection-training set  $S_{m}$ .
In our experiments, we use SciPy's stats.ttest_1samp implementation [44] for the t-test.
Our approach encodes all necessary information for detecting distribution shifts using only  $C_{\mathrm{target}}$  scalars (in our experiments, we set  $C_{\mathrm{target}} = 10$ ), which is independent of the size of  $S_{m}$ . In contrast, the baselines process the detection-training set,  $S_{m}$ , which is typically very large, for every detection they make. This makes our method significantly more efficient than the baselines, see Figure 1 and Table 1 in Section 5.
5 Complexity Analysis
This section provides a complexity analysis of our method as well as the baselines mentioned in Section 3. Table 1 summarizes the complexities of each approach. Figure 1 shows the population-based approaches run-time (in seconds) as a function of the detection-training set size, denoted as  $m$ .
All population-based baselines are lazy learners (analogous to nearest neighbors) in the sense that they require the entire source (detection-training) set for each detection decision they make. Using only a subset will result in sub-optimal performance, since it might not capture all the necessary information within the source distribution,  $\mathcal{P}$ .
Detection Method	Space	Time
MMD	O(m2+k2+mk)	O(d(m2+k2+mk))
KS	O(d(m+k))	O(d(m log m+k log k))
Single-instance	O(k)	O(k)
CBD (Ours)	O(k)	O(k)
Table 1: Complexity comparison. **bold** entries indicate the best detection complexity.  $m$ ,  $k$  refers to the detection-training size (or source size), and window size, respectively.  $d$  refers to the number of dimensions after dimensionality reduction.
In particular, MMD is a permutation test [18] that also employs a kernel. The complexity of kernel methods is dominated by the number of instances and, therefore, the time and space complexities of MMD are  $\mathcal{O}(d(m^2 + k^2 + mk))$  and  $\mathcal{O}(m^2 + k^2 + mk)$ , respectively², where in the case of DNNs,  $d$  is the dimension of the embedding or softmax layer used for computing the kernel, and  $k$  is the window size. The KS test [33] is a univariate test, which is applied on each dimension separately and then aggregates the results via a Bonferroni correction. Its time and space complexities are  $\mathcal{O}(d(m\log m + k\log k))$  and  $\mathcal{O}(d(m + k))$ , respectively.
The single-instance baselines simply con
duct a t-test on the heuristic uncertainty estimators (SR or Entropy-based) between the detection-training set and the window data. By initially determining the mean and standard deviation of the detection-training set, we can efficiently perform the t-test with a time and space complexity of  $\mathcal{O}(k)$ .
The fitting procedure of our coverage-based detection algorithm incurs time and space complexities of  $\mathcal{O}(m\log m)$  and  $\mathcal{O}(m)$ , respectively. Each subsequent detection round incurs time and space complexities of  $\mathcal{O}(k)$ , which is independent of the size of the detection-training (or source) set. Our method's superior efficiency is demonstrated both theoretically and empirically, as shown in Table 1 and Figure 1, respectively. Figure 1 shows the run-time (in seconds) for each population-based detection method as a function of the detection-training set size, using a ResNet50 and a fixed window size of  $k = 10$  samples. Our detection time remains constant regardless of the size of the detection-training set, as opposed to the baseline methods, which exhibit a run-time that grows as a function of the detection-training set size. In particular, our method achieves a five orders of magnitude improvement in run-time over the best performing baseline, KS-Softmax, when the detection-training set size is  $1,000,000^4$ . Specifically, our detection time is  $5.19 \cdot 10^{-4}$  seconds, while KS-Softmax's detection time is 97.1 seconds, rendering it inapplicable for real-world applications.
6 Experiments - Detecting Distribution Shifts
In this section we showcase the effectiveness of our method, along with the considered baselines, in detecting distribution shifts. All experiments were conducted using PyTorch, and the corresponding code is publicly available for replicating our results<sup>5</sup>.
2With preprocessing in the fitting stage, this reduces to  $\mathcal{O}(d(k^2 +mk))$  per detection decision.
${}^{3}$  MMD detection-training set is capped at 1,000 due to kernel method's unfavorable size dependence.
4This specific experiment is simulated synthetically (since the ImageNet validation set size is 50,000).
<sup>5</sup> https://github.com/BarSGuy/Window-Based-Distribution-Shift-Detection.
Figure 1: Our method outperforms the baselines in terms of scalability, with a significant five orders of magnitude improvement in run-time compared to the best baseline, KS-Softmax, when using a detection-training set of  $m = 1,000,000$  samples. One-  $\sigma$  error-bars are shadowed.
Our experiments are conducted on the ImageNet dataset [7], using its validation dataset as proxies for the source distribution,  $\mathcal{P}$ . We utilized three well-known architectures, ResNet50 [19], MobileNetV3-S [24], and ViT-T [8], all of which are publicly available in timm's repository [45], as our pre-trained models. To train our detectors, we randomly split the ImageNet validation data (50,000) into two sets, a detection-training or source set, which is used to fit the detectors $^3$  (49,000) and a validation set (1,000) for applying the shift. To ensure the reliability of our results, we repeated the shift detection performance evaluation on 15 random splits, applying the same type of shift to different subsets of the data. Inspired by [37], we evaluated the models using various window sizes,  $|W_{k}| \in \{10, 20, 50, 100, 200, 500, 1000\}$ .
6.1.1 Distribution Shift Datasets
At test time, the 1,000 validation images obtained via our split can be viewed as the in-distribution (positive) examples. For out-of-distribution (negative) examples, we follow the common setting for detecting OOD samples or distribution shifts [37, 21, 28], and test on several different natural image datasets and synthetic noise datasets. More specifically, we investigate the following types of shifts:
(1) Adversarial via FGSM: We transform samples into adversarial ones using the Fast Gradient Sign Method (FGSM) [17], with  $\epsilon \in \{7 \cdot 10^{-5}, 1 \cdot 10^{-4}, 3 \cdot 10^{-4}, 5 \cdot 10^{-4}\}$ . (2) Adversarial via PGD: We convert samples into adversarial examples using Projected Gradient Descent (PGD) [31], with  $\epsilon = 1 \cdot 10^{-4}$ ; we use 10 steps,  $\alpha = 1 \cdot 10^{-4}$ , and random initialization. (3) Gaussian noise: We corrupt test samples with Gaussian noise using standard deviations of  $\sigma \in \{0.1, 0.3, 0.5, 1\}$ . (4) Rotation: We apply image rotations,  $\theta \in \{5^{\circ}, 10^{\circ}, 20^{\circ}, 25^{\circ}\}$ . (5) Zoom: We corrupt test samples by applying zoom-out percentages of  $\{90\%, 70\%, 50\%\}$ . (6) ImageNet-O: We use the ImageNet-O dataset [23], consisting of natural out-of-distribution samples. (7) ImageNet-A: We use the ImageNet-A dataset [23], consisting of natural adversarial samples. (8) No-shift: We include the no-shift case to check for false positives. To determine the severity of the distribution shift, we refer the reader to Table 3 in Appendix C.
6.2 Maximizing Detection Power Through Lower Coverages
In this section, we provide an intuitive understanding of our proposed method, as well as a demonstration of the importance of considering lower coverages, when detecting population-based distribution shifts. For all experiments in this section, we employed a ResNet50 [19] and used a detection-training set which was randomly sampled from the ImageNet validation set.
We validate the effectiveness of our CBD method (Algorithm 2) by demonstrating its performance on two distinct scenarios, the no-shift case and the shift case. In the no-shift scenario, as depicted in Figure 2a, we randomly
(a) No-shift case.
Figure 2: Visualization of our detection algorithm, Algorithm 2.
(b) Shift case.
selected a 1,000 sample window from the ImageNet dataset that did not overlap with the detection-training set. We then tested whether this window was distributionally shifted using our method. The shift scenario, as depicted in Figure 2b, involved simulating a distributional shift by randomly selecting 1,000 images from the ImageNet-O dataset. Each figure showcases the target coverages of each application, the coverage bounds provided by SGC, the empirical coverage of each sample (for each threshold), and the bound violations (indicated by the red circles), when they occur.
Both figures show that the target coverages and the bounds returned by SGC are essentially identical. Consider the empirical coverage of each threshold returned by SGC for each sample. In the no-shift case, illustrated in Figure 2a, it is apparent that all bounds are tightly held; demonstrated by the fact that each empirical coverage is slightly above its corresponding bound. In contrast, the shift case, depicted in Figure 2b, exhibits bound violations, indicating a noticeable distribution shift. Specifically, the bound holds at a coverage of 0.9, but violations occur at lower coverages, with the maximum violation magnitude occurring around a coverage of 0.4. This underscores the significance of using coverages lower than 1, and suggests that the detection power might lay within lower coverages.
6.3 Evaluation Metrics
In order to benchmark our results against the baselines, we adopt the following metrics to measure the effectiveness in distinguishing between in- and out-of-distribution windows: (1) Area Under the Receiver Operating Characteristic  $\uparrow$  (AUROC) is a threshold-independent metric [5]. The ROC curve illustrates the relationship between the true positive rate (TPR) and the false positive rate (FPR), and the AUROC score represents the probability that a positive example will receive a higher detection score than a negative example [10]. A perfect detector would achieve an AUROC of  $100\%$ . (2) Area Under the Precision-Recall  $\uparrow$  (AUPR) is another threshold-independent metric [32, 40]. The PR curve is a graph that depicts the relationship between precision  $(\mathrm{TP} / (\mathrm{TP} + \mathrm{FP}))$  and recall  $(\mathrm{TP} / (\mathrm{TP} + \mathrm{FN}))$ . AUPR-In and AUPR-Out in Table 2 represent the area under the precision-recall curve where in-distribution and out-of-distribution images are considered as positives, respectively. (3) False positive rate (FPR) at  $95\%$  true positive rate (TPR)  $\downarrow$ , denoted as FPR@95TPR, is a performance metric that measures the FPR at a specific TPR threshold of  $95\%$ . This metric is calculated by finding the FPR when the TPR is  $95\%$ . (4) Detection Error  $\downarrow$  is the probability of misclassification when TPR is fixed at  $95\%$ . It's a weighted average of FPR and complement of TPR, given by  $\mathrm{P_e} = 0.5(1 - TPR) + 0.5FPR$ . Assuming equal probability of positive and negative examples in the test set. We note that  $\uparrow$  indicates larger value is better, and  $\downarrow$  indicates lower values is better.
6.4 Experimental Results
In this section we demonstrate the effectiveness of our CBD method, as well as the considered baselines. To ensure a fair comparison, as mentioned in Section 6.1, we utilize the ResNet50, MobileNetV3-S, and ViT-T architectures, which are pretrained on ImageNet and their open-sourced code appear at timm [45]. We provide code for reproducing our experiments<sup>5</sup>.
Table 2 presents a comprehensive summary of our main results, which clearly demonstrate that CBD outperforms all baseline methods in the majority of cases (combinations of model architecture, window size, and metrics considered), thus providing a clear indication of the effectiveness of our approach. Our method demonstrates the highest effectiveness in the low to mid window size regime. Specifically, for window sizes of 50 samples,
our method outperforms all baselines across all architectures, with statistical significance for all considered metrics, by a large margin. This observation is true for window sizes of 100 samples as well, except for the MobileNet architecture, where KS-Softmax shows slightly better performance in the metrics of Detection Error and FPR@95TPR. The second-best performing method overall appears to be KS-Softmax (consistent with the findings of [37]), which seems to be particularly effective in large window sizes, such as 500 or more. CBD stands out for its consistent performance across different architectures. In particular, our method achieves an AUROC score of over  $90\%$  across all architectures tested when using a window size of  $50+$ , while the second best performing method (KS-Softmax), fails to achieve such a score even with a window size of 100 samples, when using the ResNet50 architecture.
It is evident from Table 2, that as far as population-based detection tasks are concerned, neither of the single-instance baselines (SR and Entropy) offers an effective solution. These baselines show inconsistent performance across the various architectures considered. For instance, when ResNet50 is used over a window size of 1,000 samples, the baselines scarcely achieve a score of  $90\%$  for the threshold-independent metrics (AUROC, AUPR-In, and AUPR-Out), while other methods achieve near-perfect scores. Moreover, even in the low sample regime (e.g., window size of 10), these baselines fail to demonstrate any superiority over population-based methods.
Finally, based on our experiments, ResNet50 appears to be the best model for achieving low Detection Error and FPR@95TPR; and a large window size is essential for achieving these results. In particular, our method and KS-Softmax both produced impressive outcomes, with Detection Error and FPR@95TPR scores under 0.5 and 1, respectively, when a window size of 1,000 samples was employed. Moreover, we observe that when small window sizes are desired, ViT-T is the best architecture choice. In particular, our method applied with ViT-T, achieves a phenomenal  $86\%+$  score in all threshold independent metrics (i.e., AUROC, AUPR-In, AUPR-Out), over a window size of 10 samples, strongly out-preforming the contenders (around  $20\%$  margin). Appendix D presents the detection performance on a representative selection of distribution shifts, analyzed individually.
Architecture	Method	Window size
AUROC ↑ / AUPR-In ↑ / AUPR-Out ↑ / DetectionError ↓ / FPR@95TPR ↓
10	20	50	100	200	500	1000
ResNet50	KS	Softmax Embeddings	61/67/62/34/67	73/74/74/31/64	87/90/85/13/27	89/89/89/15/29	94/95/92/7/14	99/99/99/2/4*	100/100/100/0.4/0.9
72/74/73/28*56*	68/73/74/24/48	81/84/79/18/37	75/76/79/22/44	76/79/79/20/40	84/87/84/13/26	86/88/84/13/26
MMD	Softmax Embeddings	54/61/56/36/72	62/65/62/37/72	73/76/72/29/56	73/73/78/33/59	79/79/79/35/54	83/85/83/15/30	85/85/85/22/37
75/72/77/38/70	79/78/79/29/57	87/87/86/18/37	83/86/81/15/30	83/85/83/14/29	83/85/83/17/32	83/86/82/13/26
Single-instance	SR Entropy	56/65/55/34/68	72/73/72/32/63	71/75/72/28/56	77/79/79/25/50	84/85/83/19/40	87/88/87/14/28	88/88/89/15/30
64/69/63/32/64	73/73/73/32/63	74/78/73/26/52	80/80/81/23/47	84/85/84/17/35	87/87/87/15/31	90/90/91/13/26
CBD (Ours)	78/70/82*42/84	88*91*87*15*30*	95*95*93*97*17*	93*93*92*10*20*	97*97*97*5/10	98/98/98/4/7	100/100/100/0.4/0.9
MobileNetV3-S	KS	Softmax Embeddings	71/72/75/32*63*	84*84/83/21/43	89/91/88/13/27	92/93/91/10/20	95/97/94/5/11	96/96/97/6/11	100/100/100/1/2
63/67/63/37/75	65/66/67/37/75	77/78/76/27/54	72/73/76/27/53	84/83/86/22/43	86/87/86/15/30	79/81/81/18/36
MMD	Softmax Embeddings	75/73/75/38/72	78/78/78/30/59	86/89/82/17/32	86/89/84/14/26	87/88/86/14/28	89/90/88/12/24	90/91/88/11/22
67/67/68/39/75	66/67/68/37/74	72/77/71/23/47	75/75/79/28/53	89/87/87/20/39	81/82/81/21/40	82/86/80/15/30
Single-instance	SR Entropy	58/62/60/37/74	65/70/66/30/60	86/87/86/19/39	86/88/84/15/30	93/93/95/5/10	96/97/95/5/10	98/98/97/3/7
52/61/57/36/72	64/70/65/29/57	85/86/86/17/33	87/88/85/15/29	93/93/94/11/22	96/96/96/6/12	98/99/98/3/7
CBD (Ours)	80*73/82*40/80	80/85/76/20/40	94*95*93*8*15*	94/94*94*11/21	95/96/95/6/13	97/98/96/4/8	99/99/99/1/2
ViT-T	KS	Softmax Embeddings	62/68/66/30/59	85/86/83/19/41	82/82/83/21/42	90/91/91/11/22	88/89/90/11/22	95/96/95/5/11	98/98/98/3/6
68/66/71/38/76	76/83/73/19/38	82/86/80/17/34	81/82/81/20/39	81/84/80/17/34	76/75/82/22/44	84/83/86/19/38
MMD	Softmax Embeddings	58/59/64/44/82	69/70/73/38/69	77/80/77/22/44	75/80/76/20/40	80/86/78/15/29	89/91/90/12/23	93/94/92/7/15
61/59/68/46/85	74/77/74/26/53	82/84/80/20/40	80/82/83/21/39	82/81/82/20/40	78/76/81/22/44	77/78/79/24/45
Single-instance	SR Entropy	67/70/70/31/61	78/76/77/29/58	76/79/76/23/45	89/90/86/14/29	91/93/91/9/20	97/97/97/5/11	99/99/98/2/5
69/74/69/27/53	79/78/78/27/55	75/78/74/22/44	89/89/90/15/31	86/87/87/14/28	93/94/93/7/14	97/97/97/4/9
CBD (Ours)	89*86*90*28/56	91*87/92*24/47	94*93*95*13*25*	95*96*96*8*15*	97*96*97*8/16	98*98/98/4/9	99/99/99/3/6
Table 2: Comparison of different evaluation metrics over ResNet50, MobileNetV3-S, ViT-T with the discussed baselines methods. The best performing method is highlighted in bold; we add the superscript * to the bolded result when it is statistically significant.
We presented a novel and powerful method for the detection of distribution shifts within a given window of samples. This coverage-based detection algorithm is theoretically motivated and can be applied to any pretrained model. Due to its low computational complexity, our method, unlike typical baselines, which are order-of-magnitude slower, is practicable. Our comprehensive empirical studies demonstrate that the proposed method works very well, and overall significantly outperforms the baselines on the ImageNet dataset, across a number of neural architectures and a variety of distribution shifts, including adversarial examples. In addition, our coverage bound is of independent interest and allows for the creation of selective classifiers with guaranteed coverage.
Several directions for future research are left open. Although we only considered classification, our method can be extended to regression using an appropriate confidence-rate function such as the MC-dropout [12]. Extensions to other tasks, such as object detection and segmentation, would be very interesting. In our method, the information from the multiple coverage bounds was aggregated by averaging, but it is plausible that other statistics or weighted averages could provide more effective detections. Finally, an interesting open question is whether one can benefit from using single-instance or outlier/adversarial detection techniques combined with population-based detection techniques (as discussed here).
This research was partially supported by the Israel Science Foundation, grant No. 710/18.
[39] David E Rumelhart, Geoffrey E Hinton, Ronald J Williams, et al. Learning internal representations by error propagation, 1985.
[40] Takaya Saito and Marc Rehmsmeier. The precision-recall plot is more informative than the roc plot when evaluating binary classifiers on imbalanced datasets. PloS one, 10(3):e0118432, 2015.
[41] Chandramouli Shama Sastry and Sageev Oore. Detecting out-of-distribution examples with gram matrices. In International Conference on Machine Learning, pages 8491-8501. PMLR, 2020.
[42] Robert J Serfling. Approximation theorems of mathematical statistics, volume 162. John Wiley & Sons, 2009.
[43] Danica J. Sutherland, Hsiao-Yu Tung, Heiko Strathmann, Soumyajit De, Aaditya Ramdas, Alex Smola, and Arthur Gretton. Generative models and model criticism via optimized maximum mean discrepancy. In International Conference on Learning Representations, 2017. URL https://openreview.net/forum?id=HJWHIKqgl.
[44] Pauli Virtanen, Ralf Gommers, Travis E Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, et al. Scipy 1.0: fundamental algorithms for scientific computing in python. Nature methods, 17(3):261-272, 2020.
[45] Ross Wightman. Pytorch image models. https://github.com/rwrightman/pytorch-image-models, 2019.
A Shift-Detection General Framework