SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[1] Charu C Aggarwal. An introduction to outlier analysis. In Outlier analysis, pages 1-34. Springer, 2017.
[2] Maxime Alvarez, Jean-Charles Verdier, D'Jeff K Nkashama, Marc Frappier, Pierre-Martin Tardif, and Froduald Kabanza. A revealing large-scale evaluation of unsupervised anomaly detection algorithms. arXiv preprint arXiv:2204.09825, 2022.
[3] Jonathan Baxter. A model of inductive bias learning. Journal of artificial intelligence research, 12:149-198, 2000.
[4] Liron Bergman and Yedid Hoshen. Classification-based anomaly detection for general data. In International Conference on Learning Representations, 2020.
[5] Paul Bergmann, Michael Fauser, David Sattlegger, and Carsten Steger. Mvtec ad-a comprehensive real-world dataset for unsupervised anomaly detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9592-9600, 2019.
[6] Markus M Breunig, Hans-Peter Kriegel, Raymond T Ng, and Jorg Sander. Lof: identifying density-based local outliers. In Proceedings of the 2000 ACM SIGMOD international conference on Management of data, pages 93-104, 2000.
[7] Hila Chefer, Shir Gur, and Lior Wolf. Generic attention-model explainability for interpreting bi-modal and encoder-decoder transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 397-406, 2021.
[8] Bingqing Chen, Luca Bondi, and Samarjit Das. Learning to adapt to domain shifts with few-shot samples in anomalous sound detection. In 2022 26th International Conference on Pattern Recognition (ICPR), pages 133-139. IEEE, 2022.
[9] Xiaoran Chen and Ender Konukoglu. Unsupervised detection of lesions in brain mri using constrained adversarial auto-encoders. In MIDL Conference book. MIDL, 2018.
[10] Sungha Choi, Seunghan Yang, Seokeon Choi, and Sungrack Yun. Improving test-time adaptation via shift-agnostic weight regularization and nearest source prototypes. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXXIII, pages 440–458. Springer, 2022.
[11] Gregory Cohen, Saeed Afshar, Jonathan Tapson, and Andre Van Schaik. Emmist: Extending mnist to handwritten letters. In 2017 international joint conference on neural networks (IJCNN), pages 2921-2926. IEEE, 2017.
[12] Niv Cohen and Yedid Hoshen. Sub-image anomaly detection with deep pyramid correspondences. arXiv preprint arXiv:2005.02357, 2020.
[13] Lucas Deecke, Robert Vandermeulen, Lukas Ruff, Stephan Mandt, and Marius Kloft. Image anomaly detection with generative adversarial networks. In Joint European conference on machine learning and knowledge discovery in databases, pages 3-17. Springer, 2018.
[14] Lucas Deecke, Lukas Ruff, Robert A Vandermeulen, and Hakan Bilen. Transfer-based semantic anomaly detection. In International Conference on Machine Learning, pages 2546-2558. PMLR, 2021.
[82] Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. Transactions on Machine Learning Research, 2022.
[83] Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, et al. Florence: A new foundation model for computer vision. arXiv preprint arXiv:2111.11432, 2021.
[84] Shen Zhang, Fei Ye, Bingnan Wang, and Thomas G. Habetler. Few-shot bearing anomaly detection via model-agnostic meta-learning. 2020 23rd International Conference on Electrical Machines and Systems (ICEMS), pages 1341–1346, 2020.
[85] Chong Zhou and Randy C Paffenroth. Anomaly detection with robust deep autoencoders. In Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining, pages 665-674, 2017.
[86] Chong Zhou, Chen Change Loy, and Bo Dai. Denseclip: Extract free dense labels from clip. arXiv preprint arXiv:2112.01071, 2021.
A Justifications of Assumptions A1-A3
As follows, we provide justifications for assumptions A1-A3. Following the justification, we also discuss possibilities to remove or mitigate the assumptions.
A1 Assuming an available meta-training set is widely adopted in few-shot learning or meta-learning [21, 22, 32, 54] and domain generalization [46, 80]. In practice, the meta-training set can be generated using available covariates. For example, for our tabular data experiment, we used the timestamps; in medical data, one could use data collected from different hospitals or different patients to obtain separate sets for meta-training; and in MVtec-AD, we used the other training classes except for the target class to form the training set. We also provided an ablation study on the number of classes in the meta-training set (Tab. 7). We found even in the extreme case where we only have one data class in the training set, the trained model still provides meaningful results.
There are multiple ways to mitigate this assumption. If one does not have a meta-training set at hand, one can train their model on a different but related dataset, e.g., train on Omniglot but test on MNIST (see results below under this setting. We still get decent AUC results on MNIST).
Anomaly ratio	1%	5%	10%	20%
AUROC	84.4±2.4	85.2±2.5	84.3±2.5	82.2±2.4
A2 Batch-level prediction is a common assumption used in robustness literature [10, 49, 53, 67, 76]. In addition, batch-level predictions are widely used in real life. For example, people examine Covid19 test samples at a batch level out of economic and time-efficiency considerations<sup>6</sup>. To relax this assumption, our method can easily be extended to score individual data by presetting the sample mean and variance in BatchNorm layers with a collection of data. These moments are then fixed when predicting new individual data. Empirically, to understand the impacts of the batch size on the prediction performance, we conducted an ablation study with as small a batch size as three in the experiments.
A3 Besides being supported by the intuition that anomalies are rare, this is consistent with most of the data used in the literature. ADBench<sup>7</sup> has 57 anomaly detection datasets (with an average anomaly ratio of  $5\%$ ), all matching our assumption that the normal data take the majority in each dataset.
We provide a simple mathematical argument for the validity of A3, showing that a mini-batch with a majority of anomalies is very unlikely to be drawn for a sufficiently large mini-batch size  $B$ . Let  $p < 1/2$  denote the fraction of anomalies among the data and define  $\Delta = 1/2 - p > 0$ . For every data point  $\mathbf{x}_i$  in the batch, let  $y_i \sim \text{Bernoulli}(p)$  encode whether  $\mathbf{x}_i$  is normal ( $y_i = 0$ ) or abnormal ( $y_i = 1$ ). The variable  $S_B := y_1 + \dots + y_B$  thus counts the number of anomalies in the batch, so that  $S_B < B/2$  means that the majority of the data is normal. We want to show that the violation of A3 is unlikely, that is,  $P(S_B \geq B/2)$  is small. By Hoeffding's inequality, since  $0 \leq y_i \leq 1$  for all  $i$ , it follows that  $P(S_B \geq B/2) = P(S_B - \mathbb{E}[S_B] \geq B/2 - Bp) \leq \exp\left(-2B(0.5 - p)^2\right) \leq \exp\left(-2B\Delta^2\right)$ , which converges to zero exponentially fast when  $B \to \infty$ .
B Generalization to an Unseen Distribution  $P_{*}$
This section aims to provide a proof for Thm. 2.1. Inspired by Fallah et al. [19], we derive an upper bound of the generalization error of our meta-training approach on unseen distributions. The error is described in terms of the data distributions transformed by batch-norm-involved feature extractors.
Definition B.1. Given a sample space  $\Omega$  and its  $\sigma$ -field  $\mathcal{F}$ , the total variation distance between two probability measure  $P_{i}$  and  $P_{j}$  defined on  $\mathcal{F}$  is
$$
\left\| P _ {i} - P _ {j} \right\| _ {T V} = \sup  _ {A \in \mathcal {F}} \left| P _ {i} (A) - P _ {j} (A) \right| = \sup  _ {f: 0 \leq f \leq 1} \left| \mathbb {E} _ {x \sim P _ {i}} [ f (x) ] - \mathbb {E} _ {x \sim P _ {j}} [ f (x) ] \right| \tag {8}
$$
$^{6}$ https://www.fda.gov/medical-devices/coronavirus-covid-19-and-medical-devices/pooled-sample-testing-and-screening-testing-covid-19
<sup>7</sup>https://github.com/Minqi824/ADBench
Now we split the neural network into two parts: the first part is the layers before (including) the last batch normalization layer, referred to as feature extractor  $\mathbf{z} = f_{\theta}(\mathbf{x})$ , and the second part is the layers after the last batch normalization layer, namely the anomaly score map  $S(\mathbf{z}) = S(f_{\theta}(\mathbf{x})) = S_{\theta}(\mathbf{x})$ .  $S(\mathbf{z})$  may involve learnable parameters, but we omit the notations for conciseness. The split allows us to separate the effects of batch normalization layers on the generalization error of unseen distributions.
Under the transformation of  $f_{\theta}$  consisting of batch normalization layers, we have the data distribution transformed into the distribution of adaptatively centered representations
$$
P _ {j} (\mathbf {x}) \stackrel {\mathbf {z} = f _ {\theta} (\mathbf {x})} {\Longrightarrow} P _ {j} ^ {z} (\mathbf {z}), \quad j = 1, \dots , K, * \tag {9}
$$
resulting in  $P_{j}^{z}$  with  $\mathbb{E}_{P_j^z}[\mathbf{z}] = 0$  and  $\mathrm{Var}_{P_j^z}[\mathbf{z}] = 1$ .
Assume the mini-batch is large enough so that the mini-batch means and variances are approximately constant across batches, i.e., the batch statistics in batch normalization layers are equal to the population-truth values. Consequently, when  $\mathbf{x}_1,\ldots ,\mathbf{x}_B\stackrel {\mathrm{i.i.d.}}{\sim}P_j$  which constitutes  $\mathbf{x}_{\mathcal{B}}$ , their latent representations  $\mathbf{z}_i\coloneqq \mathbf{f}_\theta^i (\mathbf{x}_B)\stackrel {\mathrm{i.i.d.}}{\sim}P_j^z$ . For  $i = 1,\dots ,B$ . Then the expectation of the batch-level losses are
$$
\begin{array}{l} \mathbb {E} _ {\mathbf {x} _ {\mathcal {B}} \sim P _ {j}} \left[ \frac {1}{B} \sum_ {i = 1} ^ {B} L [ S _ {\theta} ^ {i} (\mathbf {x} _ {\mathcal {B}}) ] \right] = \mathbb {E} _ {\{\mathbf {z} _ {i} \sim P _ {j} ^ {z} \} _ {i = 1} ^ {B}} \left[ \frac {1}{B} \sum_ {i = 1} ^ {B} L [ S (\mathbf {z} _ {i}) ] \right] \\ = \frac {1}{B} \sum_ {i = 1} ^ {B} \mathbb {E} _ {\left\{\mathbf {z} _ {i} \sim P _ {j} ^ {z} \right\} _ {i = 1} ^ {B}} \left[ L \left[ S (\mathbf {z} _ {i}) \right] \right] \\ = \mathbb {E} _ {\mathbf {Z} \sim P _ {j} ^ {z}} [ L [ S (\mathbf {z}) ] ] \tag {10} \\ \end{array}
$$
Assumption B.2. For any parameters (if any), the loss function  $L[S(\cdot)]$  is bounded by  $C$ .
We now quantify the generalization error to an unseen distribution  $P_{*}$  by the difference between the expected loss of data batches of  $P_{*}$  and the one of meta-training distribution.
$$
\begin{array}{l} \left| \mathbb {E} _ {\mathbf {x} _ {\mathcal {B}} \sim P _ {*}} \left[ \frac {1}{B} \sum_ {i = 1} ^ {B} L \left[ S _ {\theta} ^ {i} (\mathbf {x} _ {\mathcal {B}}) \right] \right] - \frac {1}{K} \sum_ {j = 1} ^ {K} \mathbb {E} _ {\mathbf {x} _ {\mathcal {B}} \sim P _ {j}} \left[ \frac {1}{B} \sum_ {i = 1} ^ {B} L \left[ S _ {\theta} ^ {i} (\mathbf {x} _ {\mathcal {B}}) \right] \right] \right| (11) \\ = \left| \mathbb {E} _ {\mathbf {z} \sim P _ {*} ^ {z}} [ L [ S (\mathbf {z}) ] ] - \frac {1}{K} \sum_ {j = 1} ^ {K} \mathbb {E} _ {\mathbf {z} \sim P _ {j} ^ {z}} [ L [ S (\mathbf {z}) ] ] \right| \quad (\text {b y E q . (1 0)}) (12) \\ = C \left| \mathbb {E} _ {\mathbf {z} \sim P _ {*} ^ {z}} \left[ \frac {L [ S (\mathbf {z}) ]}{C} \right] - \frac {1}{K} \sum_ {j = 1} ^ {K} \mathbb {E} _ {\mathbf {z} \sim P _ {j} ^ {z}} \left[ \frac {L [ S (\mathbf {z}) ]}{C} \right] \right| \quad (\text {b y A s s u m p t i o n B . 2}) (13) \\ \leq C \sup  _ {0 \leq L / C \leq 1} \left| \mathbb {E} _ {\mathbf {z} \sim P _ {*} ^ {z}} \left[ \frac {L [ S (\mathbf {z}) ]}{C} \right] - \frac {1}{K} \sum_ {j = 1} ^ {K} \mathbb {E} _ {\mathbf {z} \sim P _ {j} ^ {z}} \left[ \frac {L [ S (\mathbf {z}) ]}{C} \right] \right| (14) \\ = C \left\| P _ {*} ^ {z} - \frac {1}{K} \sum_ {j = 1} ^ {K} P _ {j} ^ {z} \right\| _ {T V} \quad (\text {b y}) (15) \\ \end{array}
$$
This result suggests the generalization error of the loss function is bounded by the total variation distance between  $P_{*}^{z}$  and  $\frac{1}{K}\sum_{j = 1}^{K}P_{j}^{z}$ . The batch normalization re-calibrates all  $P_{j}$  such that  $P_{j}^{z}$  centers at the origin and has unit variance, making the distributions similar. Thus the total variation gets smaller after batch normalization, lowering the generalization error upper bound.
The limitation of this analysis is we assume the batch statistics are population-truth moments (mean and variance) in  $\mathbf{z}_i\stackrel {\mathrm{i.i.d.}}{\sim}P_j^z$ . So we cannot analyze the effects of the batch size  $B$  during training and testing. That said, we provide empirical evaluations on different batch sizes  $B$  at test time in Supp. I.1.
The training procedure of our approach is simple and similar to any stochastic gradient-based optimization. The only modification is to take into account the existence of a meta-training set. See Alg. 1.
Algorithm 1: Training procedure of ACR
Input:  $K$  interrelated training distributions  $P_{1},\dots ,P_{K}\stackrel {\mathrm{i.i.d.}}{\sim}\mathcal{Q}$  Mixing rate  $\pi$  Deep anomaly detector model parameters  $\theta$  Sub-sample size  $M$  Mini-batch size  $|\mathcal{B}|$  learning rate  $\alpha$  Number of training iterations  $T$    
Output: Optimized anomaly detector with parameter  $\theta_T$    
1 Randomly initialize  $\theta$    
2 Construct  $P_1^\pi ,\dots ,P_K^\pi$  (Eq.(6))   
3 for iteration t in [1,..,T] do   
4 Sample  $M$  tasks  $\{\mathbf{x}_{\mathcal{B}_m}\}_{m = 1}^M$  from all  $K$  task distributions  $\{P_1^\pi ,\dots ,P_K^\pi \}$    
5  $\theta_t\gets \theta_{t - 1} - \alpha \nabla_\theta \frac{1}{M}\sum_{m = 1}^M L(\mathbf{S}_{\theta_{t - 1}}(\mathbf{x}_{\mathcal{B}_m}))$  (Eq. (7))   
6 end
Figure 2: Illustration of batch normalization for AD with two tasks  $P_1^\pi$  and  $P_2^\pi$ . The method (batch-)normalizes the data in  $P_j^\pi$  separately. If each  $P_j^\pi$  consists mainly of normal samples, most samples will be shifted close to the origin (by subtracting the respective task's mean). As a result, the samples from all tasks concentrate around the origin in a joint feature space (gray area) and thus can be tightly enclosed using, e.g., one-class classification. Samples from the test task are batch normalized in the same way.
D Toy Example with Batch Normalization
An important component of our method is batch normalization, which shifts and re-scales any data batch  $\mathbf{x}_{\mathcal{B}}$  to have sample mean zero and variance one. Batch normalization also provides a basic parameter-free zero-shot batch-level anomaly detector (Eq. (4)). In Fig. 2, we show a 1D case of detecting anomalies in a mixture distribution. The mixture distribution composes of a normal data distribution (the major component) and an abnormal data distribution (the minor component). Eq. (4) adaptively detects anomalies at a batch level by shifting the normal data distribution toward the origin and pushing anomalies away. Setting a user-specified threshold allows making predictions.
CLIP-AD [51]. CLIP (Contrastive Language-Image Pre-training [59]) is a pre-trained visual representation learning model that builds on open-source images and their natural language supervision signal. The resulting network projects visual images and language descriptions into the same feature space. The pre-trained model can provide meaningful representations for downstream tasks such as image classification and anomaly detection. When applying CLIP on zero-shot anomaly detection, CLIP prepares a pair of natural language descriptions for normal and abnormal data:  $\{l_n = "A\text{photo of {NORMAL_CLASS}}\}$ ,  $l_a = "A\text{photo of something}"\}$ . The anomaly score of a test image  $\pmb{x}$  is the relative distance between  $\pmb{x}$  to  $l_n$  and  $\pmb{x}$  to  $l_a$  in the feature space,
$$
s (\boldsymbol {x}) = \frac {\exp \left(\left\langle f _ {x} (\boldsymbol {x}) , f _ {l} \left(l _ {a}\right) \right\rangle\right)}{\sum_ {c \in \left\{l _ {n} , l _ {a} \right\}} \exp \left(\left\langle f _ {x} (\boldsymbol {x}) , f _ {l} (c) \right\rangle\right)},
$$
where  $f_{x}$  and  $f_{l}$  are the CLIP image and description feature extractors and  $\langle \cdot, \cdot \rangle$  is the inner product. We name this baseline CLIP-AD.
Compared to our proposed method, CLIP-AD requires a meaningful language description for the image. However, this is not always feasible for all image datasets like Omniglot [41], where people can't name the written characters easily. In addition, CLIP-AD doesn't apply to other data types like tabular data or time-series data. Finally, CLIP-AD has limited ability to adapt to a different data distribution other than its training one. These limitations are demonstrated in our experiments.
OC-MAML [22]. One-Class Model Agnostic Meta Learning (OC-MAML) is a meta-learning algorithm that taylors MAML [21] toward few-shot anomaly detection setup. OC-MAML learns a global model parameterization  $\theta$  that can quickly adapt to unseen tasks with a few new task data points  $S$ , called a support set. The new-task adaptation takes the global model parameters to a task-specific parameterization  $\phi(\theta, S_t)$  that has a low loss  $L(Q_t; \phi(\theta, S_t))$  on the new task  $t$ , represented by another dataset  $Q_t$ , called a query set. OC-MAML uses a one-class support set to update the model parameters  $\theta$  with a few gradient steps to get  $\phi$ . To learn an easy-to-adapt global parameterization  $\theta$ , OC-MAML directly minimizes the target loss on lots of training tasks. Suppose there are  $T$  tasks for training. The following loss function is minimized
$$
l (\theta) = \frac {1}{T} \sum_ {t = 1} ^ {T} \mathbb {E} _ {S _ {t} \sim p _ {S} ^ {t}, Q _ {t} \sim p _ {Q} ^ {t}} [ L (Q _ {t}; \phi (\theta , S _ {t})) ], \tag {16}
$$
where  $p_S^t$  is task  $t$ 's support set distribution and  $p_Q^t$  is the query set distribution. During training, the support set contains  $K$  normal data points where  $K$  is usually small, termed  $K$ -shot OC-MAML. The query set contains an equal number of normal and abnormal data and provides optimization gradients for  $\theta$ . During test time, OC-MAML adapts the global parameter  $\theta$  on the unseen task's support set  $S^*$ , resulting in a task-specific parameter  $\phi(\theta, S^*)$ . The newly adapted parameters are then used for downstream tasks.
OC-MAML is not a zero-shot anomaly detector and requires  $K$  support data points to adapt compared to our method. Our method is simpler in training as it doesn't need to adapt to the support set with additional gradient updates, characterized in the function  $\phi(\theta, S)$ . OC-MAML is also different in batch normalization. Rather than the original batch normalization, OC-MAML first computes the batch moments using the support set and then normalizes both the support and query set with the same moments. However, the computed moments can be noisy when the support set size is small. In our experiments, we adopt a 1-shot OC-MAML for all image data.
ResNet152 [27]. Because batch normalization is an effective tool for zero-shot anomaly detection (see Fig. 1b), we directly apply batch normalization on extracted features from a pre-trained model. We then compute the anomaly score as the Euclidean distance between a feature vector and the origin in the feature space. Our experiments use a ResNet152 model pre-trained on ImageNet as a feature extractor and extract its 2048-dimension penultimate layer output as the final feature vector. Upon computing the features of an input batch through batch normalization layers in ResNet, two variants are available: using the batch statistics from training or re-computing the statistics of the test input batch itself. We name the former variant ResNet152-I and the latter ResNet152-II. Baseline ResNet152 doesn't optimize the feature extractor jointly with the zero-shot detection property of batch normalization. Hence the extracted pre-trained features are not optimal for the zero-shot AD.
ADIB [14]. In addition to zero-shot and few-shot anomaly detectors, we also compare with the state-of-the-art deep anomaly detector ADIB [14] which use pre-trained image features and additional data for outlier exposure in training. We use a "debiased" subset of TinyImageNet as the outlier exposure data for CIFAR100 as suggested in Hendrycks et al. [30], use EMNIST [11] as the outlier exposure data for MNIST as suggested in Liznerski et al. [51], use OrganC and OrganS datasets [81] as outlier exposure data for OrganA, and use half of the training data as normal data and half of the training data as auxiliary outliers for Omniglot.
Practical Training and Testing. On visual anomaly classification and tabular AD, we construct training and test distributions using labeled datasets<sup>8</sup>, where all  $\mathbf{x}$  from the same class  $j$  (e.g., all 0's in MNIST) are considered samples from the same  $P_{j}$ . The dataset  $\mathcal{Q}$  (e.g., MNIST as a whole) is the meta-set of all these distributions.
For training and testing, we split the meta-dataset into disjoint subsets. In the MNIST example, we define  $P_0, \dots, P_4$  as the distributions of images with digits 0 - 4 and use them for training. For testing, we select a single distribution of digits not seen during training (e.g., digit 5) as the "new normal" distribution  $P_*$  to which we adapt the model. The remaining digits (6 - 9 in this example) are used as test-time anomalies. To reduce variance, we rotate the roles among digits 5 - 9, using each digit as a test distribution once. $^9$
F.1 Implementation Details on Image Data for Anomaly Detection
Hyperparameter Search. We search the hyperparameters on a validation set split from the training set, after which we integrate the validation set into the training set and train the model on that. Then we test the model on the test set.
On CIFAR100-C, we construct the validation set on the training set of the primitive CIFAR100. We randomly select 20 classes as the validation set and set the remaining classes to be the training dataset at validation time. We search the neural network architecture (layers (3,4,5,6), number of convolutional kernels (32, 64, 128), and the output dimensions (8, 16, 32, 64, 128) while fixing the kernel size by  $3 \times 3$ . For the learning rate, we search values 0.1, 0.01, 0.001, 0.0001, and 0.00001, after which we search finer values in a binary search fashion. We also search the mini-batch size  $B$  (30, 60) and the number of sub-sampled tasks  $M$  (16, 32, 64) at each iteration. We select the combination that trade-off the convergence speed and optimization stability. When selecting the anomaly ratio  $\pi$  (Eq. (6)), we test 0.99, 0.95, 0.9, 0.8, 0.6 and find the results are quite robust to these values. So we fix  $\pi = 0.8$  across the experiments.
On non-natural image datasets (OrganA, Omniglot, and MNIST), we search hyperparameters on the validation set of Omniglot and use the searched hyperparameters on all datasets. Specifically, at validation time, we randomly split the Omniglot into 1200 classes for training and 423 classes for validation. After optimizing the hyperparameters, we constantly use the first 1200 classes for training and the remaining 423 classes for testing. The searched hyperparameters are the same as the ones in CIFAR100-C described above.
Training Protocols. We train the model 6,000 iterations on CIFAR100 data, 10,000 iterations on Omiglot, and 2,000 iterations on MNIST and OrganA. Each iteration contains 32 training tasks; each task mini-batch has 30 (for datasets other than CIFAR100) or 60 (for CIFAR100) points sampled from  $P_{j}^{0.8}$ . All 32 training tasks' gradients are averaged and incur one gradient update per iteration.