FLUID: A Unified Evaluation Framework for Flexible Sequential Data
Authors: Matthew Wallingford
Aditya Kusupati
Keivan Alizadeh-Vahid
Aaron Walsman
Aniruddha Kembhavi
Ali Farhadi
Venue: Accepted by TMLR ()

Abstract: Modern machine learning methods excel when training data is IID, large-scale, and well labeled. Learning in less ideal conditions remains an open challenge. The sub-fields of few-shot, continual, transfer, and representation learning have made substantial strides in learning under adverse conditions, each affording distinct advantages through methods and insights. These methods address different challenges such as data arriving sequentially or scarce training examples, however often the difficult conditions an ML system will face over its lifetime cannot be anticipated prior to deployment. Therefore, general ML systems which can handle the many challenges of learning in practical settings are needed. To foster research towards the goal of general ML methods, we introduce a new unified evaluation framework – FLUID (Flexible Sequential Data). FLUID integrates the objectives of few-shot, continual, transfer, and representation learning while enabling comparison and integration of techniques across these subfields. In FLUID, a learner faces a stream of data and must make sequential predictions while choosing how to update itself, adapt quickly to novel classes, and deal with changing data distributions; while accounting for the total amount of compute. We conduct experiments on a broad set of methods which shed new insight on the advantages and limitations of current techniques and indicate new research problems to solve. As a starting point towards more general methods, we present two new baselines which outperform other evaluated methods on FLUID.

Published in Transactions on Machine Learning Research (3/2023)
FLUID: A Unified Evaluation Framework for Flexible Sequential Data
Matthew Wallingford
mcw244 @cs. www.::swwwwssssss..e.... edu
University of Washington
Aditya Kusupati University of Washington
Keivan A Alizadeh University of Washington
keitomalkzadeeh@@maallcoom
Aaron Walsman University of Washington
ceronuullsmann@@@aaa.coom
Aniruddha Kembhavi
onibaallenai.or
Allen Institute for Artificial Intelligence
Ali Farhadi
ali@cs. uw. edu
University of Washington
Reviewed on OpenReview: https: // openreview. net/ forum? U.JBKWASSH
Abstract
Modern machine learning methods excel when training data is large-scale, well labeled, and matches the test distribution. Learning in less ideal conditions remains an open challenge. The sub-fields of few-shot, continual, transfer, and representation learning have made substantial strides learning under adverse conditions, each affording distinct advantages through methods and insights. These methods address different challenges such as data arriving sequentially or scarce training examples, however often the difficult conditions MT. system will face over its lifetime cannot be anticipated prior to deployment. Therefore, general ML systems which can handle the many challenges of learning in practical settings are needed. To foster research towards the goal of general ML methods, we introduce a new unified ovaluation framework- FLUID (Flexible Sequential Data). FLUID intogrates the objectives of few-shot, continual, transfer, and representation learning while enabling comparison and integration of techniques across these subfields. In FLUID, learner faces a stream of data and must make sequential predictions while choosing how to update itself, adapt quickly to novel classes, and deal with changing data distributions; while accounting for the total amount of compute. We conduct experiments on broad set of met which shed new insight on the advantages and limitations of current techniques and indicate new research problems to solve. As a starting point towards more general methods, we present two new baselines which outperform other evaluated methods on FLUID.
1 Introduction
Modern ML methods have demonstrated remarkable capabilities, particularly in settings with large-scale labeled training data drawn IID. However, in practice the learning conditions are often not so ideal. Consider general recognition system, key component many computer vision applications. One would expoct such system learn from new data distributions, recognize classes with few and many examples, revise the set of known classes novel ones seen, and update itself over time using new data.

---

Published in Transactions on Machine Learning Research (3/2023)
Standard Supervised Learning
Task-Style Continual Learning
Standard Few-Shot Learning
train many example distributions
single
training phase
a single evaluation phase same distribution
train then evaluate sequential distributions
train and evaluate unseen distributions
FLUID
learner must decide when truin and on which examples in order to maximize performance while minimizing MACS
Data Stream
model evaluated on each incoming data point
Training
Evaluation
evaluation methodology supports arbitrary changes in data distribution
Matching colors indicate matching distributions
Figure 1: Comparison of supervised (top-left), continual (top-middle), and few-shot learning (top-right) with FLUID (bottom). The learner (grey box) accumulates data (dotted path), trains on given data (filled nodes), then is evaluated (empty nodes). The size of the node indicates the scale of the training or evaluation. Each color represents a different set of classos.
Various subfields such as few-shot, contimual, transfer, and representation learning have made substantial progress on the challenges associated with learning in non-ideal settings. The methods from these fields excel when the deployment conditions be anticipated and align with specific scenarios. For example, few-shot methods perform well when the number of new classes and examples per class are few and known in advance. Similarly, continual learning techniques improve performance when data from new distributions arrive in fixed-size batches at predictable intervals. However, in many applications the exact conditions cannot be known a priori and are likely to change over time. For example, computer vision systems for autonomous self-checkout systems must account for changing inventories, distribution shifts in background and lighting between stores, and a long-tailed distribution of products, among other real-world challenges (Polacco & Backes, 2018; Wankhede et al., 2018). This calls for general methods that can handle plethora of scenarios during deployment.
To make progress towards such general methods, new evaluations which reflect the key aspects of learning in practical settings are essential. But, what are these aspects? We posit the following as some of the necessary elements: (1) Sequential Data In many application domains the data streams in. ML methods must be capable of learning from sequential data and new distributions.
(2) X-shot Data often has different number of examples for each class (few for some and many for others). Current evaluations assume prior knowledge of which data regime (few-shot, many-shot, etc.) new classes will be from, but often this cannot be known in advance. (3) Flezible Training Phases Practical scenarios rarely delineate when and how system should train. ML systems should be capable of making decisions such as when to train based on incoming data, which data to train with, and whether to update all parameters or just the classifier. (4) Compule Aware Real-world systems often have computational constraints not only for inference but also for training. ML systems should account for the total compute used throughout their lifetime. (5) Open-world As new data is encountered the set of known classes may change over time. Learning practical settings often entails recognizing known classes while detecting when data comes from new classes.
2

---

Published in Transactions on Machine Learning Research (3/2023)
With these elements in consideration we introduce FLUID (flexible sequential data), unified evaluation framework. FLUID integrates the objectives of few-shot, continual, transfer, and representation learning into a simple and realizable formulation along with a benchmarkable implementation. In FLUID, a learner is deployed on a stream of data from an unknown distribution and must classify incoming samples one at a time while deciding when and how to update based on newly received data.
We conduct extensive experiments with FLUID on a broad set of methods across subfields. The experimental results quantitatively demonstrate the current limitations and capabilities of various ML approaches. For example, find that current few-shot methods do not scale well to more classes and varying number of examples. Similarly, we observe that catastrophic forgetting is a significant challenge in the FLUID setting which is not solved by existing continual learning approaches. Finally, we briefly investigate the unexplored problem of update strategies for deciding when and how to train efficient on incoming data. The framework, data and models will be open-sourced.
We make the following contributions:
1. We propose a new evaluation framework, FLUID, which unifics the objectives of fow-shot, contimual, transfer, and self-supervised learning into a simple benchmark that enables comparison and integration of methods across related subfields and presents new research challenges.
2. We present empirical findings from experiments with FLUID which demonstrate the utility of more general evaluations. Specifically, we find that cxisting few-shot methods do not scale well to the FLUID setting which has more classes and varying number of examples per class. Larger networks perform better for few-shot classes, contrary to prevailing thought few-shot works which train light-weight models to avoid overfitting (Snell et al., 2017; Finn et al., 2017; Sun et al., 2019; Xu et al., 2020). We find this discrepancy to be caused by meta-training decreasing performance for larger networks whereas supervised pretraining does not. We observe in the FLUID setting that freezing network parameters prevents catastrophic forgetting and learns novel classes better than existing continual learning methods and suggests significant room for improvement.
3. We introduce two baselines, Exemplar Tuning & Minimum Distance Thresholding, which outperform evaluated methods in FLUID while matching performance in supervised and few-shot settings.
2 Related Work
We discuss the key aspects of FLUID in the context of other works and evaluations. We compare existing frameworks in Table 1 and discuss FLUID in the context of real-world applications in appendix R.
Sequential Data and Continual Learning New data is an inevitable consequence of our dynamic world and learning over time is a long-standing challenge (Thrun, 1996). In recent years, continual learning (CL) has made notable progress on the problem of learning in a sequential fashion (Li & Hoiem, 2017; Kirkpatrick et al., 2017; Rebuffi et al., 2017; Aljundi et al., 2018; 2019; Riemer al., 2019). Several setups have been proposed in order to evaluate systems' abilities to learn continuously and primarily focus on catastrophic forgelling, phenomenon where models drastically lose accuracy on old tasks when trained on new tasks. The typical CL setup sequentially presents data from each task then evaluates on current and previous tasks (Li & Hoiem, 2017; Kirkpatrick et al., 2017; Rebulli et al., 2017). Recent variants have proposed a task-free setting where the data distribution changes without the model's knowledge (Harrison et al., 2019; Riemer et al., 2019; He et al., 2020a; Wortsman et al., 2020; Sun et al., 2020).
Recently several works have empirically investigated the disconnect between existing continual learning evaluations and real-world applications (Prabhu et al., 2020; Hussain et al., 2021; Onl, 2022). Such works show that implicit details in the experimental setups can lead to significantly different methods and empirical conclusions. For example, Prabhu et al. (2020) show that not accounting for compute allows for an unrealistic method which retrains from scratch to drastically outperform sophisticated CL methods. In FLUID, we aim to address unrealistic assumptions and carefully consider each detail of the experimental design.
There are two assumptions in existing CL evaluations which we remove in FLUID. The first assumption is that data will be received in large batches with ample data for every class in the task. This circumvents
3

---

Published in Transactions on Machine Learning Research (3/2023)
Table 1: We categorize existing evaluations frameworks aimed at learning in practical settings. X: absence & - not applicable.
presence;
Property
Open-wald
Sequeatial
Variable Batch-size
Fen-shol
Many-shut
Compate Aware
Mernory Coustrnieed
Flexible Training
Distribution Staik
Non- Statiouarry
Framework
Regresettation Learning
X
Transee Learting
K
/
Thsk-hnsed Continall Learning
X
x
/
Task free Continaal Learning
K
V
Few-shot Learning
X
X
Generlized Few shot Learning
K
Streaming Pereeption
X
X
Open Long: Triled Recognition.
X
Data Stream Classifiation
J
Test Time Trining
OSAKA
FLUID (Ours)
fundamental challenge of sequential data which is learning new classes from only a few examples. Consider the common scenario which a learner encounters an instance from a novel class. The system must determine that it belongs to new class with no previous examples (zero-shot learning and out-of-distribution detection). The next time instance from the category appears, the system must be capable of one-shot learning, and forth. In other words, few-shot learning is an emergent requirement of learning from sequential data. The second assumption is that the training and testing phases will be delincated to the system. Deciding when to train and which data to train on is an intrinsic challenge of learning continuously.
Some CL evaluations include a memory cache to store images, typically between 0- 1000 examples from previous tasks. We argue that setting a specific memory constraint, particularly this small, too constraining. Meth should account for memory, but the FLUID framework does not explicitly restrict memory during streaming. Note that all methods use memory caching during the sequential phase except for the Nearest Class Mean (NCM) baseline.
Data stream classification (Gomes et al., 2019; Wankhade el al.; Stefanowski & Brzezinski, 2017; Bifel el al., 2010) has worked on the problem of learning from sequential data. This line of work primarily focuses on traditional classification and not image recognition. At a high level FLUID has similar goals as data stream classification. FLUID differs in its implementation while integrating the preexisting fields of few-shot, transfer, representation, and continual learning.
Most closely related to FLUID is the OSAKA benchmark (Caccia et al., 2020). Caccia et al. (2020) propose a more general scenario which unifies meta- metannna,ning meta-continual learning, and continual meta learning and addresses limitations of the previous evaluations. FLUID builds upon this direction of research with a few key differences. First, FLUID accounts for compute consumed throughout training, which is important metric to consider when flexible training phases are allowed. Second, FLUID samples from a long-tail distribution and evaluates accuracy with respect to class frequency (head and tail class accuracy). This allows us study the trade-off between meta-learning methods which excel in the few-shot regime and methods which are better with large-scale data. Last, conduct the experiments at the ImageNet scale. We find that the larger-scale setting leads to new empirical findings such as meta-training not scaling to larger networks and more data.
One key distinction that OSAKA and other CL frameworks incorporate that is not included in FLUID is a non-stationary distribution. In general, FLUID differs from traditional continual learning formulations (van de Ven et al., 2022; De Lange et al., 2021) in that the goal to perform well on the deployment distribution, rather than current and previously seen distributions. In FLUID the system adapts to one unknown distribution shift, whereas traditional CL frameworks change distributions over multiple episodes.
Few-shot and X-shot Learning Learning from few examples for some classes is an inherent aspect of the real-world. Learning from large, uniform datasets (Russakovsky et al., 2015; Lin et al., 2014) has been the primary focus of supervised learning while few-shot learning has gained traction as a subfield (Ravi & Larochelle, 2017; Hariharan & Girshick, 2017; Oreshkin et al., 2018; Sun et al., 2019).
While few- learning is a step towards more generally applicable ML methods, the framework has assumptions that are unlikely to hold in practical settings. The experimental setup for few-shot is typically
4

---

Published in Transactions on Machine Learning Research (3/2023)
the n-shot k-way evaluation. Models are trained on base classes during meta-training and then tested on novel classes during mela-testing. The n-shot k-way experimental setup is limited in two respects. n-shot k-way assumes that a model will always be given exactly n examples for k classes at test time which is unrcalistic. Second, most works only evaluate 5-way scenarios with 1, 5, and 10 shots. Realistic settings often have a mix of classes from both the high and low data regime. Recently, more general variants of the few-shot benchmark have been proposed which introduce variable shot numbers and greater domain difference between datasets (Chao et al., 2016; Triantafilouu et al., 2020; Dumoulin et al., 2021).
FLUID naturally integrates the few-shot problem into its framework by sequentially presenting data from a long tail distribution and evaluates systems across spectrum of shots and ways. Our experimental results on canonical few-shot methods indicate that methods are overly tuned to the specific conditions of the few-shot evaluation which indicates the need for more general experimental frameworks such as FLUID.
Flexible Training Phases Current experimental sctups dictate when models will be trained and tested. Ideally, an ML system should decide when to train itself, what data to Irain on, and what to optimize for (Cho et al., 2013). By removing the assumption that training and testing phases are fixed and known in advance, FLUID provides benchmark for tackling the unexplored challenge of learning when to train.
Compute Aware ML systems capable of adapting to their environment over time must account for the computational costs of their learning strategies as well as of inference. Prabhu et al. (Prabhu et al., 2020) showed that current CL frameworks do not measure total compute and therefore a naive, but compute-hungry strategy can drastically outperform state of the art methods. Previous works have focused on efficient inference (Rastegari et al., 2016; Howard et al., 2017; Kusupati et al., 2020; 2021; 2022; Lin et al., 2021; Wallingford ct al., 2022) and some on training costs (Evci ct al., 2020). In FLUID mcasure the total compute for both learning and inference over the sequence.
Open-world Practical scenarios entail inferring in an open world - where the classes and number of classes are unknown to the learner. Few-shot, continual, and traditional supervised learning sctups assume that test samples can only come from known classes. Previous works explored static open-world recognition (Liu et al., 2019; Bendale & Boult, 2015; Kong & Ramanan, 2021; Vaze et al., 2021; Radford et al., 2021) and the related problem of out-of-distribution detection (Hendrycks & Gimpel, 2016; Masana et al., 2018; Lee). FLUID is a natural integration of sequential and open- open-world learning where the learner must identify new classes and update its known class set throughout the stream.
3 FLUID Evaluation Details
FLUID evaluation is designed to be simple and general while integrating the key aspects outlined in section 2.
Table 2: The evaluation metrics used in the FLUID framework to capture the performance and capabilities of various algorithms.
Metric
Description
Overall Accuracy
Aceuracy over the sequence.
Mean Per-Class
Accuracy averaged over all classes in the sequence.
Accuracy
Total Compute
MAC operations for all compute over the sequence.
Unseen Class
AUROC for the detection of OOD samples.
Detection
Classes in the sequence belong fall into 4 subcategories:
1) Pretraining -eetraining-Hsnd:: > 50 samples & in pretraining.
Cross-Sectional
2) Pretruining- Tail: < 50 samples & in pretraining.
Accuracies
3) Novel-Head: > 50 samples & not in pretraining.
4) Novel- Tail: S 50 samples & not in pretraining,
5

---

Published in Transactions on Machine Learning Research (3/2023)
Formulation Let learning system S be composed of model, fo x y, and update strategy, U
T
fo x U (21, ue) + for where U (Tz, M1) is the training data collected up to time T. Model, fo, may be r=1
t=1
initialized using pretraining data D = (+)--
At each new time step, t+I 1, the model is given a sample, It, and provides a class label, fe(2 E {l...K + 1], for K known classes. In other words, the sample may belong to one of K previously soen classes, or a new class. The model output is evaluated with respect to the true label, 1(ff((z(41) = }0+1], and (æ Mt+1) is added to the training set. Tf yt+1 is from a new class, the set of known classes is updated accordingly. Next. the model, fa, may be updated according to U using all previously observed data. This process is repeated for some total number of time steps.
Systems are evaluated on a suite of metrics including the overall and mean class accuracy throughout the stream along with the total compute required for updates and inference.
Data In this paper, we evaluate methods with FLUID using a subset of ImageNet-222% (Deng et al., 2009). Traditionally, few-shot learning used datasets like Omniglot (Lake et al., 2011) & MiniTmagenet (Vinyals et al., 2016) and continual learning focused on MNIST (LeCun, 1998) & CIFAR (Krizhevsky et al., 2009). Some recent continual learning works have used Split-ImageNet (Wen et al., 2020). The aforementioned datasets are mostly small-scale and have very few classes. We evaluate on the ImageNet-22K dataset to present new challenges to cxisting modcls. Recently, the INaturalist (Van Horn et al., 2018; Wortheimer & Hariharan, 2019) and LVIS (Gupla et al., 2019) dalasets have advocated for heavy-tailed distributions. We follow suit and draw our sequences from heavy-tailed distribution.
The dataset consists of pretraining dataset and 5 different soquences of images for streaming (3 test and 2 validation sequences). For pretraining we use the standard ImageNet-1K (Russakovsky el al., 2015). This allows us to leverage existing models built by the community as pre-trained checkpoints. Sequence images come from ImageNet-22K3 after removing ImageNet-IK3 IK's images. Each test sequence contains images from 1000 different classes, 750 of which do not appear in ImageNet-1K. We refer to the overlapping 250 classes as Pretrain classes and the remaining 750 as Novel classes. Each sequence is constructed by randomly sampling images from heavy-tailed distribution of these 1000 classes. Each sequence contains 90000 samples, where head classes contain > 50 and tail classes contain S 50 samples. The sequence allows to study how methods perform combinations of pretrain vs novel, and head vs tail classes. In Table 3, we show results obtained for sequence 5, and the Appendix I shows results across all test sequences. More comprehensive statistics on the data and sequences are in Appendix B.
Pretraining Supervised pretraining (He et al., 2016) on large annotated datasets like ImageNet facilitates the transfer of Iearnt representations help data-scarce downstream tasks. Unsupervised learning methods like autoencoders (Ischannen et al., 2018) and more recent self-supervised methods (Jing & Tian, 2020; Purushwalkam & Gupta, 2020; Gordon et al., 2020) like Momentum Contrast (MoCo) (He et al., 2020b) and SimCLR (Chen et al., 2020a) have begun to produce representations as rich that of supervised learning and achieve similar accuracy on various downstream tasks.
Before the sequential phase, we pretrain our model on ImageNet-IK. In our experiments, we compare how different pretraining strategies (contrastive learning, meta-training, & supervised training) perform under more adverse conditions. We find new insights such as contrastive representations perform significantly worse on few-shot classes compared their supervised counterparts in the FLUID evaluation.
Evaluation metrics Table 2 defines the evaluation metries in FLUID to gauge the performance of the algorithms.
4 Baselines and Methods
In this soction, we summarize the basclines, other methods, and our proposed basclines, Exemplar Tuning and MDT. Additional details about the methods and implementation can be found in Appendix D and Appendix E respectively.
6

---

Published in Transactions on Machine Learning Research (3/2023)
Standard Training and Fine-Tuning We evaluate standard model training (update all parameters in the network) and fine-tuning (update only the final linear classifier) with offline batch training. We ablate over the number of layers trained during fine-tuning in Appendix F.
Nearest Class Mean (NCM) Recently, multiple works (Tian et al., 2020; Wang et al., 2019) have found that Nearest Class Mean (NCM) is comparable to state-of-the-art few-shot methods (Sun et al., 2019; Oreshkin et al., 2018). NCM in the context of deep learning performs a 1-nearest neighbor search in feature space with the centroid of each class as neighbor. We pretrain neural network with linear classifier using softmax cross-entropy loss, then freeze the parameters to obtain features.
Few-shot Methods We evaluate the following methods: MAML (Finn et al., 2017), Prototypical Networks (PTN) (Snell et al., 2017), Weight, Imprinting (Qi et al., 2018), ProtoMAML (Triantafilou el al., 2020), SimpleCNAPS (Bateni et al., 2020), New Meta-Baseling (Chen et al., 2020b) and ConstellationNet (Xu et al., 2020).
PTN trains deep feature embedding using 1-nearest neighbor with class centroids and soft nearest. neighbor loss. Parameters are trained with meta-training and backprop.
MAML is a gradient- gradient-based approach which uses second-order optimization to learn parameters that can be quickly adapted to a given task. We tailor MAML. to FLUID by pretraining the model according to the objective in Appendix D and then tune during the sequential phase.
Weight Imprinting initializes the weights of a cosine classifier as the class centroids, then fine-tunes with a learnable temperature. For further analysis of Weight Imprinting and comparison to Exemplar Tuning see Appendix M.
is same in implementation as NCM except a phase of meta-training is done after standard batch training.
For further details on the remaining the methods and how they are implemented in FLUID see Appendix D.
Continual Learning (CL) Methods We evaluate Learning without Forgetting (LwF) (Li & Hoiem, 2017), Elastic Weight Consolidation (EWC) (Kirkpatrick et al., 2017), Dark Experience Replay (DER) (Buzzega et al., 2020), & Experience Replay with Asymmetric Cross- Entropy (ER-ACE) (Caccia et al., 2022) to observe whether continual learning techniques improve performance in FLUID.
LwF leverages knowledge distillation (Buciluă et al., 2006) to retain accuracy on previous training data without storing it. EWC enables CL in a supervised learning context by penalizing the total distance moved by the parameters from the optimal model of previous tasks weighted by the corresponding Fisher information. Unlike LwF, EWC requires stored data, typically the validation set, from the previous tasks. In FLUID, we use LwF and EWC to retain performance on pretrain classes. For the continual learning methods we train all network parameters according to standard training procedures (Appendix E). For further details on ER-ACE and DER see Appendix D.
Out-of-Distribution (OOD) Methods We evaluate two methods proposed by Hendryeks & Gim- pel (Hendrycks & Gimpel, 2016) (HG) and OLTR (Liu et al., 2019) along with our proposed OOD baseline. The HG bascline thresholds the maximum probability output of the softmax classifier determine whether sample is OOD.
We propose the baseline, Minimum Distance Thresholding (MDT), which utilizes the minimum distance from the sample to all class representations, C,. In the case of NCM the class representation is the class mcan and for a linear layer it is the ith column vector. For distance function d and a threshold of distribution if: I (min; d (c;, < t). MDT with a Nearest Class Mean classifier can be derived from a Dirichlet process mixture model (Hjort et al., 2010), where a sample is considered to be out of distribution if it is assigned to a new cluster. The concentration parameter for the Chinese Restaurant process can be related to the out of distribution threshold 5 as:
sample is out
a
20 log
(1 + g) d/2
7

---

Published in Transactions on Machine Learning Research (3/2023)
P is the covariance scaling of the gaussian prior over the cluster means, -~~ NN pl) and a is the scaling for the isotropic covariance of each gaussian cluster, . (1zz. aT). Similar to the DP-Means derivation, a goes to O, the probability of a sample x being assigned to a new cluster goes to 1 when the distance of x to the closest cluster exceeds T.
Other metric learning techniques have proposed using distance to detect out of distribution examples (Tee; Masana al., 2018). MDT primarily differs from these works in that it can be used with a standard classification network and can be performed in single forward pass with negligible extra compute.
Exemplar Tuning (ET) We present a new baseline that leverages the inductive biases of instance-based methods and parametric deep learning. The traditional classification layer is effective when given a large number of examples but performs poorly when only a few examples are present. On the other hand, NCM and other few-shot methods are accurate in the low data regime but do not significantly improve when more data is added. Exemplar Tuning (ET) synthesizes these methods in order initialize class representations accurately when learning new classes and to have the capacity to improve when presented with more data. We formulate each class representation (classifier), C,, and class probability as the following:
C 1 Σ (x; n RED, [f(=;))] + +r ply= Eies eCs
(1)
where f(z; 8) is parametrized neural network, is a Icarnable residual, is the number of class examples, and Dj are all examples in class i. C2 is analogous to the i-th column vector in a linear classification layer.
The class centroid (the first term of C₁ in Eq 1) provides an accurate initialization from which the residual term r, can contimue to learn. Thus ET is accurate for classes with few examples (where deep parametric models are inacourate) and continues to improve for classes with more examples (where few-shot methods are lacking). In our experiments, the centroid is updated after each sample for minimal compute and batch train the residual vector with cross-cntropy loss according to the same schedule as fine-tuning (sce Appendix E for details).
We compare ET to initializing a cosine classifier with class centroids and fine-tuning (Weight Imprinting). Exemplar Tuning outperforms Weight Imprinting and affords two significant. advantages besides better accuracy. 1) ET has two frequencies of updates (fast instance-based and slow gradient-based) which allows the method to quickly adapt to distribution shifts while providing the capacity to improve over a long time horizon. 2) ET automatically balances between few-shot & many-shot performance, unlike Weight Imprinting which requires apriori knowledge of when to switch from centroid-based classification to fine-tuning.
5 Experiments and Analysis
We evaluate an array of methods from few-shot, continual, self-supervised learming, and dustr--diittibution detection in the proposed FLUID framework. We present a broad set of empirical findings which validate the need for more general evaluations such as FLUID and suggest future research directions. Table 3 displays a comprehensive set of metrics for the set of methods (outlined in Sec 4). Throughout this section, we will refer to rows of the table for specific analysis. For a summary of the main insights see Sec 5.7.
5.1 Few-shot Analysis
We evaluate three groups of methods to understand the effects of meta-training on generalization to novel classes and to gauge the overall utility of current few-shot methods in FLUID. The first group consists of methods which are purely meta-trained (Prototypical Networks and MAML). The second group consists of methods that do not utilize meta-training (Weight Imprinting, NCM, and fine-tuning). Finally, we evaluate methods which use both meta-training and standard batch training (ProtoMAML, SimpleCNAPS, Nonstellationnet,, Meta-Baseline).
We obscrve the methods that purely meta-train (PTN and MAML) do not perform well in the largo-scale FLUID setting with over 30% lower overall accuracy than the NCM baseline (Table and Figure 2-a). One might argue that PTN and MAML could simply scale to the larger setting by increasing model capacity. However,
8

---

Published in Transactions on Machine Learning Research (3/2023)
Table 3: Performance the suite methods (outlined in Section 4). We present several accuracy metrics Overall, Mcan-pen tean-per--llas well as accuracy bucketed into categories: Novel Novel-Head, Naill-Tail, Pretrain-Hest and Pretrain-Tal (Pretrain refers classes present in the ImageNet-1K dataset).
Pretrain Strategy
Novel Head (=50)
Pretrain
Novel Tail (<50)
Pretrain Tail (<50)
Mean Per-Class
GMACs! (x10%)
Method
Overall
Head (=50)
Backbone Conv-1
(a) Prototypical Netwurks
Meta
11.63
22.03
6,90
13.26
11.13
15.98
0.06
(b) MAML
Meta
2.86
2.02
0.15
0.10
1.10
3.64
2.20
Backbone- ResNet18
(c) Prototypical Networks
Meta
8.61
16. 98
6.79
12.71
9.50
11.14
0.15
(d) Coastellatio
Sup+Meta.
39.26
65.35
52.91
40.21
46.13
0.16
Mese-Bee
Sup-+-Met
40.47
67.03
27.53
53.87
40.23
47.62
0.16 5.73
ProtoMAML
Sup. Meta
11.68
69.48
28.91
51.16
12.94
19.25
5.73
(x) SimpleCNAPS
Sup. Meta
41.59
68.79
25.79
52.16
41.82
48.23
0.16
(h) Weight Imprinting
Sup.
40.32
67.46
15.35
34.18
32.69
48.51
0.16 5.73
(i) OLTR
Sup.
40.83
40.00
17.27
13.85
27.77
45.06
0.16
6.39
()) LwF
Sup. 30.07
67.50
7.23
56,96
31.02
48.76
22.58 / 45.16
C1
(k) EWC
Sup.
39.03
70.84
16.59
47.18
34.89
50.39
12 12:29
(l) DER
Sup.
35.34
74.11
10.64
52.05
32.30
19.07
11.29
(m) ER-ACI
Sup.
33.13
69.61
49.00
12.29
Boselines
(u) Fine- tune
Sup.
43.41
77.29
58.77
41.54
53.80
0.16 5.73
Standard Training
Sup.
38.51
68.
16.90
33.99
49.46
11.29
(p) NCM
Sup.
42.35
72.69
31.72
56,17
43.44
50.62
0,15
(q) Exemplar Tuning (Ours)
Sup.
48.85
75.70
27.93
45.73
43.61
18.16
0.16 5.73
Weight Imprinting
MoCa
16.77
26.98
6.19
8.69
12.60
22.90
0.16
5.73
OLTR
MoCo
34.60
33.77
39.92
MoCo
Fine-tune
MoCo
14.49
27.59
8.91
26.86
0.16 5.73
(1) Standard Training
MoCa
26.63
45.02
9.63
20.54
21.12
35.60
11.29
(v) NCM
MoCo
19.24
31.12
14.40
21.96
18.99
22.90
0.15
(w) Exemplar Tuning (Ours)
MoCo
31.50
46.21
12.90
21.10
24.36
39.61
0.16 5.733
0.7
0.7
0.6
0.6
Rolling Acouracy
0.5
ET (ours)
Rolling Accuracy
0.5
Fine-tune
LwF
0.4
NCM
0.3
PTN
0.3
0.2
MAML
ProtoMAML
0.2
NCM DenseNet181
0.1
0.1
NCM ResNet50
0.0
NCM ResNet18
0
15000
30000
45000
60000
75000
0.0
15000
30000
45000
60000
of Samples Seen
# Samples Seen
(b)
Figure 2: (a) Compares the accuracy of varions methods over the stream of data. (b) Compares the accuracy of NCM on novel classes across network architectures. Contrary to prevailing thought, we find that deeper networks generalize better to novel fow-shot classes.
few-shot works indicate that training with deeper and overparameterizet networks decrease performance (Sun et al., 2019; Oreshkin et al., 2018; Snell et al., 2017; Finn et al., 2017). We verify this observation, noting that the 4-layer convnet PTN (Table 3-a) outperforms the ResNet18 PTN in overall and novel class accuracy.
The prevailing thought in few-shot literature has been that smaller networks overfit less to base classes, and therefore methods use shallow networks develop techniques constrain deeper ones. We find evidence the contrary, that deeper networks generalize better to novel classes when using standard batch sampling (see

---

Published in Transactions on Machine Learning Research (3/2023)
Figure 2-b). Given that NCM and PTN differ only in the use of meta-training, the experiments indicate that meta-training is responsible for the lower performance of deeper networks. This evidence is further reinforced by the fact that Meta-Baseline performs worse than NCM with the inclusion of meta-training (Table 3-e.p).
For the more recent few-shot methods (ProtoMAMI, Meta-Basse Baseline, ConstellationNet) that utilize a combi- nation of meta-training and mini-batch training we observe significantly better results compared to purely meta-trained methods. However, we find that these recent few-shot methods are outperformed by the NCM baseline for novel, pretrain, tail and head classes (Table 3) indicating that meta-training in its current form reduces performance in settings with varying number of examples.
Typically meta-training is performed with fixed number of classes (way) and number of examples (shot). In FLUID the number of examples and classes are changing, therefore standard meta-training may not be suitable to match the test conditions. We conduct experiments to observe whether changing the meta-training procodure to better reflect. the test conditions improves performance of the model. We ablate over the way and shot number as well as randomly sample the shot and way Uhroughout training. Results can be found in Appendix P. We find that changing the shot dyper-paramen changes which part of the distribution the model performs well on (tail and head), but the overall accuracy does not significantly improve.
5.2 Continual Learning Analysis
We evaluate Learning without Forgetting (LwF), Elastic Weight Consolidation (EWC), Dark Experience Replay (DER), and Experience Replay with Asymmetric Cross Entropy (ER-ACE). For analysis, we compare to the baselines NCM, standard training, and fine-tuning, LwF and EWC are prominent CL methods while DER and ER-ACE are recent methods with state-of-the-ar performance.
We find that catastrophic forgetting of the pretrain parameters is a significant challenge in the FLUID evaluation. Standard training of the parameters on the sequential data degrades not only the accuracy for pretrain classes, but also for novel classes compared to CL methods and freezing network parameters (table 3). We hypothesize that large-scale pretraining provides better features even for novel classes compared to training on the smaller sequential data set. A similar observation was made by Hayes & Kanan (2020) in a CL setting. Further evidence for this can be found in the update stralegies sention where we observe that standard training on sequential data for too many epochs reduces overall accuracy (Section 5.6 Figure 3).
Our experiments show that freezing the feature extractor (NCM and fine-tuning) more effoctive in preventing calastrophic forgetting than existing CL methods. Specifically, NCM obtains 8% (Table 3 j-m) higher mean-per-class accuracy compared to existing CL methods and accuracy. This result indicates that there is significant room for progress in reducing forgetting in the FLUID setting and motivates the need for new evaluations which more closely model the challenges faced by real-world ML systems. For scenarios in which the pretraining data is radically different from the target. distribution the above conclusion may not hold, such as for permuted MNIST.
obtains 4% higher overall
The notable differences between FLUID and standard CL formulations are the inclusion of pretraining, flexible training, one distribution shift rather than multiple, and measuring performance only on the deployment distribution. We contend pretraining is a reasonable inclusion real-world vision systems have access to large datasets such as ImageNet, though some scenarios, especially those outside the domain of computer vision, may not afford pretraining.
5.3 Exemplar Tuning
We find that ET (Table 3-w) has significantly higher overall and mean-class accuracy than other evaluated methods and uses similar compute as fine-tuning. Figure 2-a shows how ET quickly adapts to new classes and continues learn the standard data regime (high accuracy at the start and end of the stream). Finally, we show that ET outperforms simple NCM + fine-tuning (Weight Imprinting) by ~00% in addition to the practical ad advantages outlined in section 4.
10

---

Published in Transactions on Machine Learning Research (3/2023)
0.5
Positive Rate
0.8
Rolling Rollig Accureoy
0.4
Accuracy
0.45
1K
0.40
0.4
Epochs
BK
8
NDT + ET(Ours)
0.35
10K
0.01
0.2
20K
0.1
Modo ST -i 0.01
OLTR
0:30
MoCe 0.1
0.0
16000
30000
46000
60000
75000
0.0
0.2
0.4
0.6
0.8
1.0
10"
# of Samples
False Positive Rate
MACs
(a)
(b)
(c)
Figure 3: (a) Accuracy of standard training with MoCo & supervised pretraining. Unexpectedly, MoCo accuracy falls during the initial streaming phase. (b) ROC curves for unseen class detection. MDT oulperforms all OOD baselines evaluated in FLUID. (c) Standard training accuracy curve for range of training frequencies & epochs showing that over training can lead to lower accuracy. MACs x total gradient updates.
5.4 Novel Class Detection and MDT
We mcasure AUROC for detecting new classes throughout the sequence and present in Figure 3-b. HG baseline + ET, OLTR, and MDT + ET achieve 0.84, 0.78 and 0.92 AUROC scores respectively. The performance of Minimum Distance Thresholding (MDT) indicates that standard recognition networks are well suited for detecting out-of-distribution classes and can be done simultancously with classification. We compare MDT and HG baseline with other classifiers such as NCM and fine-tuning in Appendix L.
5.5 Representation Learning Analysis
We observe unexpected behavior from contrastive MoCo (He et al., 2020b) and VINCE (Gordon et al., 2020) representations in the FLUID setting. For fine-tuning the classifier of a MoCo representation, we find that accuracy is less than 1% and 4.96% on novel and pretrain tail classes respectively (Table 3-t). In comparison the supervised counterpart (Table 3-n) obtains 23.56% and 58.77% accuracy respoctively. We conclude that this difficulty is due to learning the linear classifier because NCM with MoCo (Table 3-v) does not exhibit the same drop performance. Figure 3-a shows other unexpected behavior which MoCo accuracy drastically decreases initially when standard training, then begins improving after 10K samples. This behavior is not observed for supervised pretraining and occurs for range of learning rates. We argue that this is related to learning mixture of pretrain and novel classes which is the primary difference bet ween FLUID and previous downstream tasks. The significantly lower accuracy of MoCo representations on novel-tail classes while fino-tuning (Table 3-t) further reinforces this hypothesis. These observations and insights are also observed for VINCE (Gordon et al., 2020), a similar cantrastive method (see Appendix G). These results validate the utility of an evaluation such as FLUID which assess the capabilities of methods more generally.
5.6 Update Strategies
We investigate trade-offs between compute cost and accuracy of simple update strategies and leave the challenges of learning adaptive update strategies FLUID as future work.
For update strategies, ablate over the frequencies and number of training epochs per update (Figure 3-c & 6) and measure the trade-off in accuracy and total compute cost. We conduct these experiments for fine-tuning (Figure 6 in Appendix O) and for standard training (Figure 3-c) on ResNet18 model with supervised pretraining.
We observe that training for too many total epochs (training frequency x epochs) with standard training (Figure 3-c) decrcases overall accuracy, though for fine-tuning accuracy asymptotically improves (Figure in Appendix O). We hypothesize that the optimal amount of training balances the features learned from ImageNet-1K with those from the smaller, imbalanced streaming data. This aligns with continual learning
11

---

Published in Transactions on Machine Learning Research (3/2023)
experiments that indicate large-scale pretrained features trained more data outperform specialized features. These initial experiments are intended to illustrate the new problems that FLUI presents for future research. The results indicate that there is significant room for improvement in both efficiency and accuracy with new strategics for training networks under streaming conditions which we leave for future work.
5.7 Summary of Insights
1. Representative few-shot methods do not scale well to FLUID with a varying number of examples per class and more classes. Supporting Evidence: Prototypical Networks (PTN) and MAML perform 30% worse in overall accuracy compared to basolines of fine-tuning and nearest class mean (NCM) (Table a-b, n-p). Recent state-of-the-art methods ProtoMAML, SimpleCNAPS, and ConstellationNet are outperformed by the baseline NCM for both novel and pretrain classes.
The current formulation of meta-training inhibits few-shot methods from scaling to more data and larger architectures in the FLUID setting. Supporting Evidence: PTN decreases in all accuracy metrics when increasing architecture size from Conv-4 to ResNet18 (Table 3 a, c) while NCM increases in all accuracy metrics with larger models (Fig 2b). PTN & NCM differ only in that PTN uses meta-training while NCM uses standard batch training.
3. Catastrophic forgetting is significant challenge in the FLUID setting which is not solved by existing continual learning approaches and large-scale pretraining changes the types of methods which are effective for preventing forgetting. Supporting Evidence: Freezing the network parameters (Fine- tuning and NCM) obtains higher accuracy on novel and pretrain classes compared to all evaluated CL methods (Table 3).
4. Contrastive self-supervised reprosentations perform significantly worse on novel classes compared to those of supervised when learning from a mix of novel and pretrain classes FLUID. Supporting Evidence:
Fine-tuning from MoCo (Table 3) and VINCE (Appendix Table 6) obtain 0.1% and 1.69% accuracy on novel tail classes. Supervised fine-tuning obtains 23.56% on the same cross-section of data. This drastic gap between supervised & MoCo representation is absent in the original work by He et al. (2020b) when fine-tuning to COCO and other downstream tasks.
6 Limitations and Future Work
Throughout this paper, we studied various methods and settings in the context of supervised image classifica- tion, a highly explored problem ML. While we do not make design decisions specific to image classification, incorporating other mainstream tasks into FLUID potential next step. Also while the FLUID framework is agnostic to any particular data set, our experiments and conclusions anchored in the computer vision domain. Across the experiments in this paper, we impose some assumptions about the learning conditions, albeit only a few, on FLUID. For example, we currently assume that FLUID has access to labels as the data streams in. One exciting future direction is to add the semi- or un-supervised aspects to FLUID. Relaxing these remaining assumptions to bring FLUID closer to real world conditions is an interesting future direction.
7 Conclusions
We introduce FLUID, a unified evaluation framework designed to facilitate research towards more general methods capable of handling the challenges of learning in deployment settings. FLUID enables comparison and integration of solutions across few-shot, transfer, continual and representation learning, & distribiittii detection while introducing new research challenges like how and when to update model parameters based on incoming data. Through our experiments with FLUID on wide-range of methods we show the limitations and merits of existing solutions. For example, few-shot methods do not scale well to settings with more classes and varying number examples and freezing network parameters prevents catastrophic forgetting better than representative continual learning methods when in the FLUID sctting. As a starting point for solving the new challenges, we present two baselines, Exemplar Tuning & Minimum Distance Thresholding, which outperform existing methods on FLUID..
12

---

Published in Transactions on Machine Learning Research (3/2023)
References
Online continual learning for progressive distribution shift (ocl-pds): A practitioner's perspective. 2022.
Rahaf Aljundi, Marcus Rohrbach, and Tinne Tuytelaars. Selftess sequential learning, arXiv preprint arXiv:186..00 806.0541,, 2018.
Rahaf Aljundi, Klaas Kelchtermans, and Timne Tuytelaars. Task-free continual learning. Proceedings of the IEEE conference on computer vision and pattern recognition, 2019.
Peyman Bateni, Raghav Goyal, Vaden Masrani, Frank Wood, and Leonid Sigal. Improved few-shot visual classification. Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 14493-14500 2020.
Sara Beery, Grant Van Horn, and Pietro Perona. Recognition in terra incognita. In Proceedings of the European conference on computer vision (ECCV), pp. 356-473, 2018
Abhijit Bendale and Terrance Boult. Towards open world recognition. In Proceedings of the IEEE conference on computer vision and poltern recognilion, pp. 1893 1992-1902, 2015.
Albert Bifet, Geoff Holmes, Richard Kirkby, and Bernhard Pfahringer. Moa: Massive online analysis. Journal of Machine Learning Research, 11:1601-1604, 2010.
Erik Brynjolfsson, Yu Jeffrey Hu, and Michael D Smith. From niches to riches: Anatomy of the long tail. Sloon management review, 47(4):67-71, 2006.
Cristian Buciluă, Rich Caruana, and Alexandru Niculescu-Mizil. Model compression. In Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 535 541, 2006.
Pietro Buzzega, Matteo Boschini, Angelo Porrello, Davide Abati, and Simone Calderara. Dark experience for general continual learning: a strong, simple baseline. Advances in neural information processing systems, 33:15920-15930, 2020.
Lucas Caccia, Rahaf Aljundi, Nader Asadi, Tinne Tuytclaars, Joelle Pincau, and Eugene Belilovsky. New insights on reducing abrupt representation change in online continual learning. In International Conference on Learning Representations, 2022.
Massimo Caccia, Pau Rodriguez, Oleksiy Ostapenko, Fabrice Normandin, Min Lin, Lucas Caccia, Issam Laradji, Irina Rish, Alexandre Lacoste, David Vazquez, et al. Online fast adaptation and knowledge accumulation: a new approach to continual learning, arXiv preprint Xiv:220003. 05856, 2020.
Wei-Lun Chao, Soravit Changpinyo, Boqing Gong, and Fei Sha. An empirical study and analysis of generalized zero-shot learning for object recognition in the wild. In European conference on computer vision, pp. 52-68. Springer, 2016.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. arXiv preprint Nii:20002005700 2020a.
Yinbo Chen, Xiaolong Wang, Zhuang Liu, Huijuan Xu, and Trevor Darrell. A new meta-baseline for few-shot learning. arXiv preprint arXin:2003.042: 04390, 2020b.
Jaeik Cho, Taeshik Shon, Ken Choi, and Jongsub Moon. Dynamic learning model update of hybrid-classifiers for intrusion detection. The Journal of Supercomputing, 64(2):522 526, 2013.
Tamás Czimmermann, Gastone Ciuti, Mario Milazzo, Marcello Chiurazzi, Stefano Roccella, Calogero Maria Oddo, and Paolo Dario. Visual-bases defect detection and classification approaches for industrial applica- tions-a survey. Sensors, 20(5):1459, 2020.
Matthias De Lange, Rahaf Aljundi, Mare Masana, Sarah Parisot, Xu Jia, Aleš Leonardis, Gregory Slabaugh, and Tinne Tuytelaars. A continual learning survey: Defying forgetting in classification tasks. IEEE transactions on poltern analysis and machine intelligence, 44(7):3366 3385, 2021.
13

---

Published in Transactions on Machine Learning Research (3/2023)
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 TEFE conference computer vision and pollern recognation, pp. 248 #48-255. leee, 2009.
Vincent Dumoulin, Neil Houlsby, Utku Evci, Xiaohua Zhai, Ross Goroshin, Sylvain Gelly, and Hugo Larochelle. Comparing transfer and meta learning approaches on a unified lew-shot classification benchmark. arXiv preprint arXte:2104.06633 2021.
Utku Evei, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen. Rigging the lottery: Making all tickets winners. In Proceedings of the International Conference on Machine Learning, 2020.
Chelsca Finn, Picter Abboel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp- 1126-1135. JMLR. org, 2017.
Heitor Murilo Gomes, Jesse Read, Albert Bifet, Jean Paul Barddal, and João Gama. Machine learning for streaming data: state of the art, challenges, and opportunities. ACM SIGKDD Explorations Newsletter, 2019.
Daniel Gordon, Kiana Ehsani, Dieter Fox, and Ali Farhadi. Watching the world go by: Representation learning from unlabeled videos. arXiv preprint arXiv:2003.0790 2020.
Agrim Gupta, Piotr Dollar, and Ross Girshick. Lvis: A dataset for large vocabulary instance segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. $356-5364, 2019.
Bharath Hariharan and Ross Girshick. Low-shot visual recognition by shrinking and hallucinating features. In Proceedings of the IEEE International Conference on Computer Vision, pp. 3018 3027, 2017.
James Harrison, Apoorva Sharma, Chelsea Finn, and Marco Pavone. Continuous meta-learning without.
tasks. Advances in neural information processing systems, 2019.
Tyler L Hayes and Christopher Kanan. Lifelong machine learning with deep streaming linear discriminant analysis. In Proceedings of the IEEE/CVF conference on compuler vision and pallern recognilion workshops, pp. 220-221, 2020.
Jiangpeng He, Runyu Mao, Zeman Shao, and Fengqing Zhu. Incremental learning in online scenario. arXie preprint Xrxi:22003.100 13191, 2020a.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 770-778 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 172999738, 2020b.
Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distributiio examples neural networks. arxi preprint orXiv:1610.02) 02136, 2016.
Nils Lid Hjort, Chris Holmes, Peter Müller, and Stephen G Walker. Bayesian nonparametrics, volume 28. Cambridge University Press, 2010.
Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang. Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural works for mobile vision applications. arXiv preprint arXiv: 04861, 2017.
Aman Hussain, Nithin Holla, Pushkar Mishra, Helen Yannakoudakis, and Ekaterina Shutova. Towards a robust experimental framework and benchmark for lifelong language learning. In Thirty-fifth Conference Neural Information Processing Systems Datasets and Benchmarks Track (Round 1), 2021.
14

---

Published in Transactions on Machine Learning Research (3/2023)
Longlong Jing and Yingli Tian. Self-supervised visual feature learning with deep neural networks: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Barbinsk,, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 2017.
Shu Kong and Deva Ramanan. Opengan: Open-set recognition via open data generation. In Proceedings of the IEEE International Conference on Computer Vision, 2021.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Citeseer, 2009.
Aditya Kusupati, Vivek Ramanujan, Raghav Somani, Mitchell Wortsman, Prateek Jain, Sham Kakade, and Ali Farhadi. Soft threshold weight reparameterization for learnable sparsity. In Proceedings of the International Conference on Machine Learning, 2020.
Aditya Kusupati, Matthew Wallingford, Vivek Ramanujan, Raghav Somani, Jae Sung Park, Krishna Pillutla, Prateek Jain, Sham Kakade, and Ali Farhadi. Llc: Accurate, multi-purpose learnt low-dimensional binary codes. In Advances in neural informalion processing systems, 2021.
Adilya Kusupati, Gantavya Bhatt, Aniket Rege, Matthew Wallingford, Adilya Sinha, Vivek Ramanujan, William Howard-Snyder, Kaifeng Chen, Sham Kakade, Prateck Jain, et al. Matryoshka representation learning. arxiv preprint arXiv:2205.131, 2022.
Brenden Lake, Ruslan Salakhutdinov, Jason Gross, and Joshua Tenenbaum. One shot learning of simple visual concepts. In Proceedings of the annual meeting of the cognitive science society, volume 33, 2011.
Yann LeCun. The mnist database of handwritten digits. http://yan.. lecun. com/exib//misse 1998.
Kimin Lee. A simple unified framework for detecting distriiiittintution samples and adversarial attacks. In Advances in neural information processing systems workshop.
Zhizhong Li and Derek Hoiem. Learning without forgetting. IEFE transactions on pottern analysis and machine intelligence, 40(12):2935 2947, 2017.
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Europeon conference on computer vision, pp. 740-755. Springer, 2014.
Zhiqiu Lin, Deva Ramanan, and Aayush Bansal. Streaming self-training via domain-agnostic unlabeled images. arxio preprinl Xix::10:4...0 03309, 2021.
Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang, Boqing Gong, and Stella X Yu. Large-scale long-tailed recognition in an open world. In Proceedings of the IEFF Conference on Computer Vision and Pattern Recognition, pp. 2537 2546, 2019.
Mare Masana, Idoia Ruiz, Joan Serrat, Joost van de Weijer, and Antonio M Lopez. Metric learning for novelty and anomaly detection. In Proceedings of the British Machine Vision Conference, 2018.
Boris Oreshkin, Pau Rodríguez López, and Alexandre Lacoste. Tadam: Task dependent adaptive metric for improved few-shot lcarning. In Advances in Neural Information Processing Systems, pp. 721-731, 2018.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems, pp. 8024-8035, 2019.
Alex Polacco and Kayla Backes. The amazon go concept: Implications, applications, and sustainability.
Journal of Business and Management, 24(1):79 92, 2018.
Ameya Prabhu, Philip H.S. Torr, and Puncet K. Dokania. Gdumb: simple approach that questions our progress in continual learning. In Proceedings of the European Conference on Computer Vision, 2020.
15

---

Published in Transactions on Machine Learning Research (3/2023)
Senthil Purushwalkam and Abhinav Gupta. Demystifying contrastive self-supervised learning: Invariances, augmentations and dataset biases. arXiv preprint or arXiv22077.130 13916, 2020.
Hang Qi, Matthew Brown, and David G Lowe. Low-shot learning with imprinted weights. In Proceedings of 2018.
the IEEE conference on computer vision and pattern recognition, pp.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. arXiv preprint rrXx::2103.00020, 2021.
Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classification using binary convolutional neural networks. In European conference on computer vision, pp. 525-542. Springer, 2016.
Sachin Ravi and Hugo Larochelle. Optimization as model for few-shot learning. International Conference on Learning Representations; 2017.
Sylvestre Alvise Rebulli, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 2001-2010, 2017.
Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, and Gerald Tesauro. Learning to learn without lorgetting by maximizing transfer and minimizing inference. Proceedings of the IEEE conference on computer vision and pattern recognition, 2019.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, al. Imagenct large scale visual recognition challenge. International journal of computer vision, 115(3):211 252, 2015.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In Advances in neural information processing systems, pp. 4077-4087, 2017.
Jerzy Stefanowski and Dariusz Brzezinski. Stream classification, 2017.
Erik Sudderth and Michael Jordan. Shared segmentation of natural scenes using dependent pitman-yor processes. Advances in neural information processing systems, 21, 2008.
Qianru Sun, Yaoyao Liu, Tat-Seng Chua, and Bernt Schiele. Meta-transfer learning for few-shot learning. In Proceedings of the IEEE Conference on Computer Vision and Pottern Recognition, pp. 403 412, 2019.
Yu Sun, Xiaolong Wang, Liu Zhuang, John Miller, Moritz Hardt, and Alexci A. Efros. Test-time training with self-supervision for generalization under distribution shifts. In ICML, 2020.
Sebastian Thrun. Is learning the n-th thing any easier than learning the first? Advances in neural information processing systems, 1996.
Yonglong Tian, Yue Wang, Dilip Krishnan, Joshua B Tenenbaum, and Phillip Isola. Rethinking few-shot image classification: a good embedding is all you need? arXiv preprint ar arXiv:200s. 11539, 2020.
Eleni Triantafillou, Tyler Zhu, Vincent Dumoulin, Pascal Lamblin, Utku Evci, Kelvin Xu, Ross Goroshin, Carles Gelada, Kevin Swersky, Pierro-Antoine Manzagol, et al. Meta-dataset: A dataset of datasets for learning to learn from few examples. International Conference on Learning Representations, 2020.
Michael Tschannen, Olivier Bachem, and Mario Lucic. Recent advances in autoencoder-hasel representation learning. ari preprint arXiv:1812.05063 2018.
Devis Tuia, Benjamin Kellenberger, Sara Beery, Blair R Costelloe, Silvia Zuffi, Benjamin Risse, Alexander Mathis, Mackenzie W Mathis, Frank van Langevelde, Tilo Burghardt, et al. Perspectives in machine learning for wildlife conservation. Nature communications, 13(1):792, 2022.
16

---

Published in Transactions on Machine Learning Research (3/2023)
Gido M van de Ven, Tinne Tuytelaars, and Andreas S Tolias. Three types of incremental learning. Nature Machine Intelligence, pp- 1-13, 2022.
Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The inaturalist species classification and detection dataset. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 8769-8778, 2018.
Sagar Vaze, Kai Han, Andrea Vedaldi, and Andrew Zisserman. Open-set recognition: A good closed-set classifier is all you need. arXiv preprint arXiv:2110. 00.06007, 2021.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot 2016.
learning. In Advances in newral information processing systems, pp.
Matthew Wallingford, Hao Li, Alessandro Achille, Avinash Ravichandran, Charless Fowlkes, Rahul Bhotika, and Stefano Soatto. Task adaptive parameter sharing for multi-task learning. In Proceedings of the LEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7561-7570 7570, 2022.
Yan Wang, Wei-Lun Chao, Kilian Q Weinberger, and Laurens van der Maaten. Simpleshot: Revisiting ncarest-neighbor classification for fow-shot learning. arXiv preprint arXiv:1911.04683 2019.
Kapil K Wankhade, Snehlata S Dongre, and Kalpana Jondhale. Data stream classification: a review.
Kirti Wankhede, Bharati Wukkadada, and Vidhya Nadar. Just walk-out technology and its challenges: A case of amazon go. In 2018 International Conference on Inventive Reseorch in Computing Applications (ICIRCA), pp. 254 257. IEEE, 2018.
Yeming Wen, Dustin Tran, and Jimmy Ba. Batchensemble: an alternative approach to efficient ensemble and lifelong learning. In International Conference on Learning Representations, 2020.
Davis Wertheimer and Bharath Hariharan. Few-shot learning with localization realistic settings. In Proceedings of the IEEE Conference on Computer Vision and Pottern Recognition, pp. 6558 6567, 2019.
Mitchell Wortsman, Vivek Ramanujan, Rosanne Liu, Aniruddha Kembhavi, Mohammad Rastegari, Jason Yosinski, and Ali Farhadi. Supermasks in superposition. Advonces in Neurol Information Processing Systems, 2020.
Woijian Xu, Huaijin Wang, Zhuowen Tu, et al. Attentional constellation nets for few-shot learning. In International Conference on Learning Representations, 2020.
17

---

Published in Transactions on Machine Learning Research (3/2023)
A
FLUID Procedure
Algorithm 1 describes the high level implementation directives of FLUID framework.
Algorithm FLUID Procedure
Input: Task T
Input: ML sys.: (pretrained) model update strategy
Output: Evaluations: E, Operation Counter: C
function FLUID(T. (f.
2: Evaluations E ==(]
3: Datapoints D D=
4:
Operation Counter C 0.
while streaming do
6: Sample (x, from T
prediction p [(x) (A operations)
8:
Flag indicates if is new unseen class
E.insert( {y, p,
10:
D D.insert( (xx, v})
Update f using with (B operations)
12:
C' C+= A+ B
13:
end while
14:
return E,
15:
end function
B Dataset Information
The five sequences we pair with FLUID are constructed from ImageNet-22K (Deng et al., 2009). Two sequences (1-2) are for validation, and three (3-5) are for testing. Each sequence contains 1,000 classes; 250 which are in ImageNet-1IK (Russakovsky et al., 2015) (pretrain classes) and 750 of which are only in ImageNet-22K (novel classos). For the test soquences, we randomly select the classes without replacement ensure that the sequences do not overlap. The validation sequences share pretrain classes because there are not enough pretrain classes (1000) to partition among five sequences. We randomly distribute the number of images per class according to Zipf's law with = (Figure 4). For classes without enough images, we fit the Zipfian distribution as closely as possible which causes slight variation in sequence statistics seen in Table 4.
Table 4: Statisties for the sequenoes images used FLUID, Sequences 1-2 are for validation and Sequence 3-5 are for lesting. The images from ImageNet-22k are approximately fit to a Zipfian distribution with 250 classes overlapping with ImageNet-11 and 750 new classes.
Sequence
Number of Images
Min # of Class Images
Max of Class Images
1
89030
1
961
2
87549
21
961
3
90133
14
961
86988
6
892
5
89921
10
961
C
Dataset License
ImageNet does not explicitly provide license.
18

---

Published in Transactions on Machine Learning Research (3/2023
1000
800
Number of Images
600
Head
Classes
400
Tail Classes
200
50
0
200
400 463
600
800
1000
Sorted Classes
Figure 4: The distribution of samples over the classes for Sequences Classes with less than samples are considered in the tail and samples with greater than equal to 50 samples are considered in the head for the purpose of reporting.
More Method Details
Nearest Class Mean Each class mean, my is the average feature embedding of all examples in class
m, = Evec, fo(z); where C2 is the set of examples belong to class and fo is the deep feature embedding of x. Class probabilities are the softmax of negative distances between and class means:
(d(m,.fu(xx)
P(y= = i|x) = Eg e f6(())
(2)
MAML The gradient update for MAML is: 0 O+ 0 B.V Etoppry Ly, (Fo:) where 0f are the parameters after making gradient update given by: 02 + 0 a a VoLT. (fe).
OLTR The network consist two parts 1) A feature extractor consist a ResNet backbone followed by a modulated attention and 2) classifier and memory bank that are used to classify the output of the feature extractor. Training done in 2 stages; In the first stage the feature extractor is trained. In the second stage the feature extractor and classifier are fine-tuned while samples accumulated in the memory bank.
Weight Imprinting Weight Imprinting initializes the weights of the cosine classification layer, then performs fine-tuning using all of the data with a learnable temperature to scale the logits. Weight Imprinting can be thought of as NCM with cosine similarity as the metric for determining the closest neighbor, then performing fine-tuning. To use Weight Imprinting in sequential setting, rather than a few-shot setting, we must decide when to bogin fine-tuning. In the original formulation, finc-tuning was performed after the centroids were calculated using the entire data set, but in the sequential setting do not have access to the entire data set. until streaming ends. Therefore NCM on validation data. In a real-world scenario it would be difficult to obtain such information, but we employ such a strategy to provide an upper-bound for the performance of Weight Imprinting in the sequential setting.
choose to begin fine-tuning when the accuracy of fine- fine-tuning exceeds
ProtoMAML ProtoMAML, initializes the classification layer as nearest class mean in accordance with Prototypical Networks, then performs second order updates according the MAML objective. Weight Imprinting and ProtoMAMT. differ in that ProtoMAML, is trained with meta-training after it is initialized
19

---

Published in Transactions on Machine Learning Research (3/2023)
with the batch trained backbone. We use the first order variant in accordance with (Triantafillou et al., 2020)
for computational efficiency.
E Implementation Details
In this section, we discuss how methods are adapted with respect to FLUID. Some methods are intuitively applied with little modification, and some require interpretation for how they should be adapted.
Offline Training For all experiments (Table 3) that require offline training (fine-tuning, Weight Imprinting, standard training, ET and LwF), except OLTR, we train each model for 4 epochs every 5,000 samples observed. An epoch includes training over all previously seen data in the sequence. Experiments in Figure show that training for epochs every 5,000 samples balanced sufficient accuracy and reasonable computational cost. Fine-tuning experiments use a learning rate of 0.1 and standard training uses 0.01 for supervised pretraining. For MoCo pretraining finc-tuning uses learning rate of 30 and standard training uses 0.01. All the experiments use the +GD+Momnnuun optimizer with 0.9 momentum.
Instance-Based Updates All instance- based methods (NCM, ET, Weight Imprinting, Prototypical Networks) are updated after every sample as it takes no additional compute compared to batch updates.
Meta-Training For Prototypical-Networt and MAMT. we meta-train from scratch with the n-shot k-way paradigm. We use 5-shot 30-way in accordance with the original works (Snell et al., 2017) (Finn al., 2017). We trained according We meta-train for 100 epochs with a learning rate of 0.01 and reduce it by 0.5 every 40 epochs. For ProtoMAML and SimpleCNAPS we train according to the Meta-Dataset routine (Triantafillou et al., 2020). Meta-Datasel splits the 1000 classes into train, validation, and test. We meta-train with all 1000 classes for fair comparison and sample between 5 and 50 classes in accordance with the original work. ConstellationNet training consists of both standard batch and veta-training. We train with 5-shot 30-way with the hyper- parameters provided in their codebase (Xu et al., 2020).
Exemplar Tuning We initialize the residual vectors as zero. ET is trained according to the specifications of instance-based updates and oflline training simultaneously.
Weight Imprinting For Weight Imprinting, we transition from NCM to fine-tuning after 10,000 samples as we observed that the accuracy of NCM saturated at this point in the validation sequence. We use a learning rate of 0.1 while fine-tuning.
Learning Without Forgetting We adapt Learning Without Forgetting to the FLUID task by freezing a copy of the model after pretraining which is used for knowledge distillation. Not all pretraining classes are seen during streaming so only softmax probabilities for classes seen during the stream are used in the cross-entropy between the soft labels and predictions. We use temperature of 2 to smooth the probabilities in accordance with (Li & Hoiem, 2017). We swept over X values between 2,... 1}. mean per-class and overall accuracy. Training is done according to the specifications given in the offline training portion of this section.
We found .2 maximized
Elastic Weight Consolidation We adapt Elastic Weight Consolidation (Kirkpatrick et al., 2017) to the FLUID task by freezing a copy of the model after pretraining as the optimal model of the pretrain task. We then use the validation set of ImageNet-1K corresponding to the 250 classes being used for the computation of Fisher information per-parameter. For every training step in FLUID, a penalty is added based on the distance moved by the parameters from the base model weighted by the Fisher information. The Fisher information is calculated at the start of every flexible train step to mitigate catastrophic forgetting. Training is done according to the specifications given in the offline training portion this section. Depending on how frequently the Fisher information is computed, the compute associated increases over the standard training costs. We use the default hyper-parameten of .1 for À and .9 for a where the loss is: [()) = aL (0) + Σ, 3 P. (0: 0,,))
DER and ER-ACE We train DER and ER-ACE with the standard specifications reported in the offline training portion of section E. For DER, at each offline training phase we store the logits and use them compute the KL divergence term in the next offine training phase. For a we swept over values between (...A. and found .5 to be most effective.
20

---

Published in Transactions on Machine Learning Research (3/2023)
For ER-ACE we apply the loss given in the paper:
C ace UX - c old curr ) ce(X C curr)
Where Cotd contains the 1000 pretraining classes and Courr contains the novel classes which have been seen during streaming.
Both methods are performed at the end of each offine training routine. The replay buffer size for both methods is 90,000 for fair comparison to other methods. In Table 14 we compare the baseline NCM with DER. and ER-ACE with smaller replay buffer sizes. Note the NCM bascline uses no replay buffer while other CL methods do use a replay buffer to store data during the sequential phase.
OLTR For OLTR (Liu et al., 2019), update the memory and train the model for 4 epochs every 200 samples for the first 10,000 samples, then train 4 epochs every 5,000 samples with 0.1 learning rate for classifier parameters and 0.01 for feature extraction parameters which is in accordance with the specifications of the original work.
Pretraining We use the PyTorch (Paszke al., 2019) ResNet18 and ResNet50 models pretrained on supervised ImageNet-1K. We the models from Gordon et al. (Gordon et al., 2020) for the MoCo (He et al., 2020b) self-supervised ImageNct-1K pretrained models. MoCo-ResNet188 and MoCo-ResNet50 get top-1 validation accuracy of 44.7% and 65.2% respectively and were trained for 200 epochs. For fine-tuning and ET with MoCo, we report the results with a learning rate of 30 which is suggested by the original work when learning on frozen features. All other learning rates with MoCo are the same as with supervised.
F Training Depth for Fine Tuning
We explored how training depth affects the accuracy of a model on new, old, common, and rare classes. For this set of experiments, we vary the number of trained layers when fine-tuning for 4 epochs every 5,000 samples on ResNet18 with a learning rate 0.01 on Sequence (validation). The results are reported Table 5. We found that training more layers leads to greater accuracy on new classes and lower accuracy pretrain classes. However, we observed that the number of fine-tuning layers did not significantly affect overall accuracy for results the test sequences (3-5) only report fine-tuning of one layer (Table 3).
Table 5:
The results for fine-tuning various numbers layers with a learning rate of .01 on Sequence 2. Training more layers generally results in higher accuracy on novel classes, but lower accuracy on pretrain classes. The trade-off between novel and pretrain accuracy balances out so the overall accuracy is largely unaffected by the depth of training.
# of I Layers
Novel- Head (>-50)
Pretrain- Head (>50)
Novel- Tail (<50)
Pretrain- Tail (<50)
Mean Per-Class
Overall
41.32
80.96
17.13
66.52
39.19
56.87
2
41.55
80.79
17.40
67.03
39.43
56.79
45.82
78.59
19.08
59.52
40.73
57.23
46.96
75.44
19.87
53.97
40.39
57.04
46.76
75.72
19.97
54.04
40.41
57.04
G Results for VINCE ResNet18 backbone on Sequence 5
We report all performance metrics for sequence 5 in Table 6 for ResNet18 backbone trained via VINCE (Gordon et al., 2020). VINCE a self-supervised representation learning method that focuses on leveraging video as a natural form of augmentation for contrastive Icarning. These results corroborate the findings of Table 3 which uses ResNet18 backbone trained via MoCo (He et al., 2020b) further solidifying the insights drawn on self-supervised representation learning methods.
21

---

Published in Transactions on Machine Learning Research (3/2023)
Table Results on sequence 5 with ResNet18 backbone trained using VINCE (Gordon et al., 2020)
Pretrain
Novel
Pretrain
Novel
Pretrain Tail (<50)
Mean Per- Class
GMACk
Method
Overall
Strategy
Head (>50)
Head (>50)
Tail (<50)
(x109)
Backbone
ResNet18
(a) Fine-tune
VINCE
1.89
7.25
26.27
0.16 / 5.73
(b) Standard Training
VINCE
24.60
32.06
6.38
9.63
16.17
32.95
11.29
(c) NCM
VINCE
15.96
22.32
12.32
16.08
15.20
18.28
0.15
(d) Exemplar Tuning
VINCE
26.84
37.03
7.32
11.30
18.11
35.44
0.16 5.73
H Results for ResNet50 backbone on Sequence 5
We report all performance metrics for sequence 5 in Table 7 for ResNet50 backbone. These results corroborate
the findings of Table which uses ResNet18 backbone.
Table 7: Continuation of Table results on sequence with ResNet50 backbone.
Pretrain
Novel Head (>50)
Pretrain Head (>50)
Novel
Pretrain Tail (<50)
Mean Per-Class
GMAC&L (x10%)
Method
Overall
Strategy
Tail (<50)
Backbone
ResNet50
Fine-tune
MoCo
14.42
43.61
0.22
13.40
11.85
31.35
0.36
13.03
(b) Fine-tune
Sup.
47.78
82.06
27.53
66.42
46.24
57.95
0.36
13.03
(c) Standard Training
MoCo
26.82
42.12
10.50
21.08
21.32
35.44
38.36
(d) Standard Training
Sup.
43.89
74.50
21.54
50.69
39.48
54.10
38.36
(e) NCM
MoCo
30.58
55.01
24.10
45.37
32.75
36.14
0.35
(f) NCM
Sup.
45.58
78.01
35.94
62.90
47.75
52.19
0.35
(g) LwF
Sup.
21.52
5.49
38.74
20.69
30.57
38.36/76.72
(h) EWC
Sup.
43.84
76.03
21.22
53.64
39.89
54.59
40.36
(i) Exemplar Tuning
MoCo
28.86
54.03
7.02
20.82
21.89
40.13
0.36
13.03
( Exemplar Tuning
Sup.
52.95
82.27
28.13
57.15
48.02
62.41
0.36
13.03
Results For Other Sequences
We report the mean and standard deviation for all performance metrics across test sequences 3-5 in Table 8. Note that the standard deviation relatively low so the methods are consistent across the randomized sequences.
J Results for FLUID Places365
We reproduced the experiments long-tailed version of Places365 and find results consistent with those the FLUID variant of ImageNet. Results can be found in table 9
K Prototypical Network Experiments
We benchmarked our implementation of Prototypical Networks on few-shot basclines to verify that it is correct. We ran experiments for training on both MinilmageNet (Vinyals et al., 2016) and regular ImageNet-1k and tested our implementation on the MiniImagoNet test set and FLUID (Sequence 2). We found comparable results those reported by the original Prototypical Networks paper (Snell et al., 2017) (Table 10).
L Out-of-Distrriution Ablation
In this section we report AUROC and F1 for MDT and softmax for all baselines. In section we only included OLTR, MDT with Exemplar Tuning, and ET with maximum softmax (Hendrycks Baseline). Additionally,
22

---

Published in Transactions on Machine Learning Research (3/2023)
Table 8: Averaged results for all methods evaluated on Sequences 3-5, See Table 3 for the computational cost (GMACs) for each method and more information about each column.
Novel
Pretrain Head (>50)
Novel
Pretrain Tail (>50)
Mean Per-Class
Method
Pretrain
Overall
Head (>50)
Tail (<50)
Backbone Conv-4
Prototype Networks
Meta
5.02-+0.05
9.71+-0.11
0.64+0.01 0.13
1.27+0001 0.1:0.01
3.25++0.33
7.82++0.09
MAML
Meta
2.93=10.01
2.02+00002
1.1140.02
3.6110006
Backbone ResNet18
Prototype Networks
Meta
8.72-10.09
16.84=0.14
7.06++0.03
12.98:10.04
9.46++0.08
11.19::0.2
MletaBBaeeline
Sup./Meta
41.73:10.57
66.51-2.37
27.51+11.3
53.69-1..77
39.32.10.71
17.74110.63
Fine-tune
Moco
5.31
15.95-1.27
0.03.10
26.23
10.6110.23
18.52.10.98
Fine-tune
Sup.
43.2.10.5
74.55
22.7911.21
59.63.1.002
40.9
53.06100.65
Standard Training
Moco
26.9=0.27
42.39-3.04
9.140.74
21.11#00511
20.76=0.32
34.85+0.75
Standard Training
Sup.
38.82++0.49
65.88+2.32
16.15+0.83
44.3+0.91
33.63+0.38
48.81+0057
NCM
Moco
19.31+0.06
30.02+1.69
14.21+0.46
22.06+00.52
18.86+0.13
22.14+1.24
NCM
Sup.
41.68+00.5
70.05+2.29
31.24+00.8
57.23+0.97
42.87+0.62
47.89+-0.76
OLTR
MoCo
41.47++0.03
31.48-0.01
17.48+0.01
9.81+0.01
22.03+0
38.33+-0.01
OLTR
Sup.
51.1910.37
37.02-0.51
24.14=0.14
13.77:0.24
27.620.28
44.46:10.44
Exemplar Tuning
Moco
32.57=1.54
43.48=0.4
0.39
12.81=0.12
18.46=0.35
39.25=1.20
Exemplar Tuning
Sup.
46.36+2231
69.34-0.53
23.48=1.23
45.82+0.32
42.93=0.17
57.56+0.56
Backbone ResNe
Fine-tune tune
Moco
45.95:10.26
3.31+0.32
26.23=0.07
0.08
10.64=0.21
18.52:1.02
Fine-tune
Sup.
47.59:10665
80.14=1.71
06.99
66.92+1.
45.6220.6
57.48+0.47
Standard Training
Moco
43.93++0773
71.72=3.18
20.84+0.92
51.4310.68
38.94+0.9
53.45+1.73
Standard Training
Sup.
47.5910..45
80.14-2.59
26.6910.79
66.92.1.91
45.6210.47
57.4810.56
NCM
Moco
30.15:10.48
53.84-1.05
23.99110.53
44.1111.11
32.2710.92
35.45110.6
NCM
Sup.
45.46.10.95
76.55=1777
35.4710.82
65.62:1.57
47.77-0.65
52.2210.55
Exemplar Tuning
Moco
28.46:1304
40.42-1.33
7.5712.11
14.3614.14
19.5412.63
32.0712.37
Exemplar Tuming
Sup.
19.24+1.55
75.78+1.88
26.67+-2.17
55.63+2.31
44.15+1.44
62.35+1.02
Table 9: The FLUID evaluation applied to Places365 with ResNet18 architecture. ET outperforms basclines in overall and mean-per-class (MPC) acc.
Novel Head
Pretrain Head
Novel Tail
Pretrain Tail
MPC
Method
Acc.
Acc.
NCM
24.2
71.1
16.9
71.1
24.1
41.4
PM [8]
25.1
81.3
8.9
82.3
25.2
49.8
Fine-Tune
25.1
83.8
7.8
83.8
24.7
50.0
ET (Ours)
29.9
84.6
10.5
84.6
27.8
54.6
visualize the accuracy curves for in-distribution and out-of-distrrbatiio samples as the rejection threshold vary (Figure 5). All the OOD experiments presented in Figure 5 and Table 11 were run using ResNet18. Minimum Distance Thresholding (MDT) threshold distances but also similarity metrics can be used. MDT generally works better than maximum softmax when applied to most methods.
The results NCM and Exemplar Tuning using softmax and dot product similarity in comparison to OLTR are shown table 11. The F1-scores are low due the large imbalance between positive and nogative classes. There are 750 unseen class datapoints vs 90000 negative datapoints. Table 11 shows that cosine similarity (MDT) is better than softmax the OLTR model for most. methods.
23

---

Published in Transactions on Machine Learning Research (3/2023)
Table 10: Our implementation Prototypical Networks on MinilmageNet FLUID. o Results from Snell et al (2017).
MinilmageNet Way 5 Shot
Method
Backbone
Train Set
FLUID
Prototypical Networks
Cony
MinilmageNet
69.2
14.36
Prototypical Networks
Conv 4
ImageNet (Train)
42.7
15.98
Prototypical Networks
Conv
MinilmageNet
68.2
Correcty labeled percentage
8-
.8-
6-
.6
-6
IND
0OD
000
00D
00D
-2-
.2
.2
2-
0.0
0.2
0.4
0.6
0.8
1.0
0
1
2
6
1
2
0.0
0.2
0.4
0.6
0.8
1.0
minimum casine distance
maximum similarity lel
maximum similarity le1
probability
(a) NCM+MDT
(b) ET + MDT
(c) Fine- Tune + MDT
(d) OLTR
Correctly labeled percentage
.8
.4
.4
.2
IND
IND 000
00D
0OD
1.0 1.2 1.8 2.0 2.2 max probability
0.0
0.6
0.8
0.2
0.6
1.0
0.0
0.4 0.6
0.8
1.0
max probability
max probability
max probability
(e) NCM+
(I) ET + Softmax
(g) Fine-Tune + Softmax
(h) Full train + Softmax
Figure 5: The accuracy for the in-distribution (IND) and out-of-distributio (OOD) samples the threshold for considering a sample ont-of-distribution varies. The horizontal axis the threshold value, and vertical axis is the accuracy. Intersection of the IND and OOD curves at higher accuracy generally indicates better out-of-distribution detection for a given method.
Table 11: The out-of-disttrbbution performance each method on sequence score achieved by choosing the best possible threshold value.
report the AUROC and the
NCM
NCM
Exemplar Tuning
Exemplar Tuning +MDT
Standard Training
Standard Training +MDT
Fine-Tune +Softmax
Fine-Tune +MDT
Metric
OLTR
Softmux
+DT
SSfftmax
+ Softmax
AUROC
0.07
0.85
0.84
0.92
0.59
0.53 0.02
0.68
0.72
0.78
F1
0.01
0.20
0.10
0.20
0.03
0.06
D.10
0.27
M Weight Imprinting and Exemplar Tuning Ablations
In Table 13, we ablate over various softmax temperature initializations with Weight Imprinting. We learn the temperature as described in (Qi et al., 2018), but find that initial value affects performance. We report the best results in the main paper. We also ablate over the similarity metrics use ET. We find that the dot product (linear) the best measure of similarity for ET..
24

---

Published in Transactions on Machine Learning Research (3/2023)
Table 12:
Comparison ER-ACE and DER with NCM for varying buffer sizes. ER-ACE and DER are designed for
smaller buffer sizes therefore compare these methods against the NCM baseline which buffer size of
Novel
Pretrain
Novel Tail (<50)
Pretrain Tail (>50)
Mean Per-Class
Method
Pretrain
Backhone
Overall
Head (>50)
Head (>50)
DER (buller 5000)
Sup
R18
33.93
71.57
8.75
49.55
26.85
46.35
AR-ACE (bulfer 5000)
Sup
R18
31.49
69.83
13.52
48.11
29 29.20
41.71
DER (Buffer 10000)
Sup
R.18
34.51
72.79
9.61
50.94
30.39
47.79
ER-ACE (Buffer 10000)
Sup
R18
32.15
70.52
14,26
48.94
32.23
44.38
NCM (Buffer 0)
Sup
R18
42.35
75.70
31.72
56.17
43.44
50.62
Table 13: Comparison Weight Imprinting and Exemplar Tuning with different classifiers and initiial temperatures. Exemplar Tuning with a linear layer performs significantly better than all other variants.
Novel Head (>50)
Pretrain
Novel (<50)
Pretrain Tail (>500
Mean Per-Class
Method
Pretrain
Backbone
Overall
Head (>500
Weight Imprinting (s 1) Sup
R18
36,58
63.39
9.32
21,80
26.85
46.35
Weight Imprinting (s
Sup
R18
36.58
63.39
9.32
21.80
26.85
46.35
Weight Imprinting
Sup
R18
40.32
67.46
15.35
34.18
32.69 22.69
48.51
Weight Imprinting
Sup
31.18
32.66
34.77
28.94
46.67
Exemplar Tuning (Cosine)
Sup
R18
33.90
18.22
4.84
1.88
11.72
31.81
Exemplar Tuming (Euclidean)
Sup
R18
43,40
66.32
21.66
42.06
37.19
61.62
Exemplar Tuning (Linear)
Sup
R18
48.85
75.70
23.93
45.73
43.61
58.16
N Exemplar Tuning on Standard Recognition Tasks
On Mini-ImageNNe (Vinyals et al., 2016), for 5-shot 5-way Exemplar Tuming with a ResNet10 backbone obtains an accuracy 72.1% compared to 68.2% for Prototypical Networks. Exemplar Tuning accuracy on Net-eN (Liu al., 2019) with a ResNet18 backbone is 42.1% while a standard linear layer gets 41.9%.
O Update Strategies
Figure 6 has the accuracy vs MACs trade-off for fine-tuning across various update strategies.
P Meta- Training Ablations
Table 14: Ablation over the shot and way number for meta-training during the pretraining stage of FLUID. Randomly sampling indicates uniform sampling over the shot and way between the indicated interval. Models are trained from scratch for 100 epochs on ImageNet-1k -kk with an initial learning rate of and cosine annealing. Random sampling is performed with Prototypical Networks.
Novel Head (>50)
Novel Tail (<50)
Method
Pretrain
Backbone
Overall
Head (>50)
Tail (>-50)
Per-Class
Protoypical Networks (Sishot 20lway
Sup
R18
8.64
16.98
6.79
12.74
9.50
11.14
Protoypical Networks (S-shot 100-way)
Sup
R18
9:23
16.71
7.67
11.48
9.46
11.05
Protoypical Networks (5-shot 600-wwy)
Sup
R18
8.16
15.63
7.24
11.29
9.41
10.37
Protoypical Networks (5-shot 1000-wwy)
Sup
R18
7.94
14.85
5.11
9.62
7.70
8.95
Protoypical Networks (20-shot 100-wwy)
Sup
R18
9.57
17.13
5.91
11.31
9.37
10.95
Protoypical Networks (30-sbos 100-way)
R18
17.26
5.17
9.18
10.71
Protoypical Networks (100-shot 100-way)
Sup
R18
9.77
16.97
5.08
10.55
8.89
10.24
Protoypicul Networss (200-shot 100-way)
Sup
R18
8.38
15.29
4.83
10.48
8.78
10.05
Random Sampling (Shot 100). Way (20) 100)]
Sup
R18
16.85
7.38
Random Sampling (Shot 1000), Way 1000))
Sup
R18
1.12
10.31
2.02
6.75
4.74
5.91
25

---

Published in Transactions on Machine Learning Research (3/2023)
0.60
0.55
Accuracy
0.50
16
Interval
8
K
0.45
4
Epochs
2K
0.40
5K
2
10K
0.35
20K
0.30
1.5 x 10
1.7x10
1.9 10 MACs
2.1 x 10
Figure 6: The plot compares the accuracy and MACs for various update strategies when fine-tuning.
Non-Stationary Distribution
this section we run preliminary experiments to observe how a non-stationary distribution affects the empirical conclusions about continual learning methods. We alternate every 10,000 samples between the 3 lesting sequences which have disjoint novel classes as discussed in section for 100,000 total samples. For these experiments evaluate the standard continual learning methods and baselines. We observe that the gap between continual learning methods and baselines is smaller compared to the results found in table 3. In particular, the continual Icarning methods better prevent forgetting for the pretrain head and tail classes compared to standard training. Overall, we find similar empirical conclusions that pretraining enables methods which freeze the network parameters to outperform traditional continual learning methods.
Table 15: The performance of continual learning methods and baselines evaluated on a non-slationary distribution. All experimental hyper-parameters such update interval, learning rate, and training epochs are the same as in the original experiments.
Pretrain Strategy
Novel Head (>50)
Pretrain
Novel Tail (<50)
Pretrain Tail (<50)
Mean Per-Class
GMACs (x10%)
Method
Overall
Head (>50)
Backbone
18sNet18
(i) LwF
Sup.
29.39
69.51
6.62
56.20
27.31
43.29
22.58 45.16
(j) EWC
Sup.
30.03
70.17
14.80
46.34
29.84
44.19
12.29
(k) DER
Sup.
31.89
72.74
13.59
52.44
31.51
45.47
11.29
(m) Fine-tune
Sup.
33.51
74.75
17.99
57.17
33.75
48.73
0.16 3.73
(n) Standard Training
Sup.
32.49
62.64
15.33
41.26
28.37
43.76
11.29
NCM
Sup.
35.25
69.27
18.97
56.67
34.02
46.13
0.15
(p) Exemplar Tuning
Sup.
36.85
73.70
19.93
45.73
35.61
49.16
0.16 3.73
R FLUID Properties in Real-World Applications
In this section detail current, real-world applications of machine learning and how they relate to FLUID.
26

---

Published in Transactions on Machine Learning Research (3/2023)
Computer vision for monitoring biodiversity (Beery et al., 2018; Tuia et al., 2022). Rescarchers camera traps to identify and track animal populations in order to monitor ecological systems. Beery et al. (2018) discuss the challenges of deployment which are similar to those in the FLUID evaluation. Specifically, they cite difficuly in generalizing to new environments, lack examples for rare classes, and lengthy, intermittent data collection. We've listed the relevant details of monitoring biodiversity as they pertain to FLUID.
The model is pretrained on wildlife data which differs from the deployment distribution.
The camera trap systems are deployed and sequentially collect data from the new environment.
As with many real-world datasets, species observation data exhibits long- tail distribution (Beery et al., 2018; Sudderth & Jordan, 2008). Rare species must be classified using only few examples.
The system may be queried and updated any time using previously collected data.
The system has finite compute for retraining.
New data comes at irregular intervals and not in fixed-size batches.
New species or animals must be detected and added to the set of known classes.
Autonomous self-checkout systems (Polacco Backes, 2018; Wankhede al., 2018) such as Amazon Go use computer vision along with a variety of sensors detect which items customers have selected, enabling the purchasing of products without the need for cashier. Below we've listed the relevant aspects of autonomous self-checkout as they pertain to FLUID.
New products and additional training examples are added to the system over time.
Images of products used for pretraining differ from real-world data. Each store presents a distribution shift in the visual background, product placement, and lighting (Polacco & Backes, 2018).
Products in the tail of the distribution have less available real-world examples making classification of rare items more difficult. (Brynjolfsson et al., 2006).
The vision system must recognize which objects held by customers are store products and which are not (out-of-distributtin detection).
Compute for retraining the models finite.
Visual defect detection is the process of identifying defects imperfections in images videos. It is commonly used in manufacturing to ensure that products meet specific standards. Computer vision models identify issues including scratches, cracks, dents, or other types of damage on surfaces, as well as defects in the alignment or positioning of components (Czimmermann et al., 2020). We've listed the key details defect detection that relate to FLUID.
Models are pretrained on open-source data sets then fine-tuned to fit the client data which often differs from the pretrain distribution.
Defect examples collected over time and used to update the model.
The distribution of defect types is
and often examples of defects are rare.
The system ideally should detect new types of defects even if they are outside the training set.
Defect examples come at irregular intervals and not fixed-size batches.
Compute for retraining the models is finite.
27


Reviews:
[{"review_id": "OsEtHBrUg8", "reviewer": "Reviewer_B1ZX", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. The paper reads well and is easy to follow.\n\n2. Empirical exploration includes many methods from the studied learning settings and offers comparisons that previously have not been made.\n\nWeaknesses:\n\n1. The motivation is weak. It is unclear why such an evaluation may be beneficial in practice.\n\n2. The algorithmic novelty in the proposed framework is limited and incremental. I am not sure if this level of novelty is sufficient for TMLR. This work may be suitable for a dataset and benchmark venue.\n\n3. The overall empirical contribution is not very novel as existing datasets are used and only the evaluation framework is novel.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper develops a new evaluation framework for learning settings that involve processing sequentially arriving data that enables comparison of methods, including few-shot learning and continual learning, across several learning settings in this area. Extensive experimental exploration is offered to study the shortcomings of existing methods and then research questions are defined for future exploration.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I believe I have not reviewed a manuscript similar to this work that lacks any algorithmic novelty and is not introducing a new dataset. I find it challenging how this work can be improved. I have the following comments which may help:\n\n1. Problem motivation needs to be clarified further. I am not convinced that there is a need to have a unified evaluation framework for the learning settings described in the paper. It is claimed, \"in many applications the exact conditions cannot be known a priori and are likely to change over time\". I tend to disagree with this statement because the user likely can predict which setting is more appropriate. Could you give some examples of those \"many applications\"? I like to see a clear motivation with connections to real-world applications for this work.\n\n2. Catastrophic forgetting is a major concern in CL. I would like to see more comparisons and justification why the existing benchmarks are not sufficient for this purpose.\n\n3. I would like to see algorithmic novelty, even at a simple level. I have difficulty convincing myself that this work is suitable for a venue such as TLMR. I understand empirical explorations are extensive but they lack to convey a clear justification for this work.\n\nI am open for further discussion about the contributions of this work.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "WCuVHR417c", "reviewer": "Reviewer_jwLE", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\nCanonical continual-learning evaluation is, in my opinion, far from practical. FLUID is a sensible step towards a more realistic continual-learning evaluation. The empirical study is extensive and offers some insights. \n\n**Weakness**\n\nMy main criticism is that FLUID is quite close to a continual-learning scenario proposed over two years ago, namely OSAKA [1]. I understand that staying informed of all related literature can be challenging. This one, however, should definitely be referenced and properly contrasted. Listed below are OSAKA's properties:\n- Open-World = yes. (the set of known classes may change over time)\n- Sequential = yes.\n- variable batch-size = N/A (i.e. not a requirement) \n- few-shot and many-shot = yes \n    - OSAKA is essentially a continual few-shot learning problem, such that the systems will be tested few-shot the first time a data distribution appears and many-shots later.\n- compute-aware = No.\n- memory constrained = N/A.\n- Flexible training = Yes.\n   - in OSAKA, systems sporadically encounter new data distribution and have to autonomously discern if the data is OoD and requires retraining. \n- non-IID: Yes.\n\nBoth settings offer a pertaining phase and a continual learning phase that includes new data distributions.\nBoth settings feed the data online to the learners and only reveal the labels after the prediction. \nIn short, FLUID adds the aspect of compute awareness to the OSAKA's evaluation. \nNoteworthily, OSAKA is more flexible as $p(y|x)$ is allowed to shift during the \"life/deployment\" of the systems.\n\n[1] Online fast adaptation and knowledge accumulation (OSAKA): a new approach to continual learning\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work proposes a new holistic evaluation (FLUID) for continual learning systems. \nSome aspects of the fields of few-shot and open-world are introduced in the framework.\nThe evaluation is compute-aware, an important consideration for continual-learning systems.\n\nA thorough empirical study demonstrates the effectiveness of new baselines for FLUID over approaches proposed for continual, few-shot and representation learning.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Properly contrast FLUID with OSAKA, as mentioned above.\n\nSome minor suggestions that could improve the manuscript.\n- link Minimum DIstance Thresholding (MDT) with non-parametric expansion methods like, e.g., Chinese-restaurant processes.\n- \"From the experiments, we find that initializing networks with standard pretraining mitigates the effect of catastrophic forgetting in its current formulation, indicating the need for new evaluation frameworks.\" Although it is reasonable to pretrain representations with ImageNet when faced with a continual vision problem, more practical learning scenarios might not enjoy such a broad pretaining dataset. It is thus unclear if not incurring forgetting after ImageNet pretraining is an argument for needing new evaluation frameworks. ", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "aFLucXR6rM", "reviewer": "Reviewer_8uhi", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The paper is written and illustrated in a clear style, and it reads very pleasantly.\n- The need for a benchmark combining the objectives of continual, few-shot, transfer and representation learning is well-motivated.\n- The proposed benchmark is well-described, and I expect there is a good chance that it will be used and adopted by the research community.\n- The reported empirical comparisons seem to be performed to good standards.\n- Code is provided, and the code seems to be well-documented and neatly organized.\n\nIn many ways I think this paper is suitable for publication in TMLR; I think the proposed benchmark / evaluation framework and accompanying code would be useful resources for the research community.  \n\nHowever, I’m afraid there is a critical issue with this paper that makes it in its current form unsuitable for publication. I think this issue is addressable, but it requires a major revision.\n\nThis issue is that the sequence of data in the proposed benchmark does not have any non-stationarity. The benchmark is set up such that the data are always sampled i.i.d. from the same underlying distribution. This is problematic firstly because the paper claims it considers the Non-IID aspect (e.g., Table 1), but even more so because a large focus of the paper is on continual learning and continual learning methods, while a defining characteristic of continual learning is non-stationarity in the training data.\n\nThe only change in underlying distribution considered in the benchmark is from the pre-training phase to the sequential learning phase, but evaluation is only on the data distribution from the sequential learning phase, making that this falls under the realm of transfer learning, not continual learning (e.g., see Fig 5 of De Lange et al. 2022 TPAMI, https://doi.org/10.1109/TPAMI.2021.3057446).\n\nProbably related to the lack of non-stationarity, the compared continual learning methods also seem to be implemented in rather artificial ways. EWC and LwF are implemented to prevent forgetting of the pre-training data, but the model is not evaluated on the pre-training data, making it unsurprising that these approaches do not perform well in these experiments. With regards to the replay-based methods (DER and ER-ACE), it is not entirely clear how these methods are implemented (e.g., my understanding is that all data is stored, but this is also the case with offline training; how are these then different?). I was also not able to find how hyperparameter-values (e.g., the regularization strength of EWC) were selected and/or which values were used in the end.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes the FLUID benchmark / evaluation framework that aims to evaluate a combination of several key aspects of learning in practical settings, such as sequentially arriving data, X-shot learning and out-of-distribution detection. While fields such as continual learning, few-shot learning and transfer learning tend to focus on evaluating only one of such aspects, this paper aims to jointly evaluate them at the same time. It is argued that this is important because otherwise there is the risk that methods will be developed that are very good at dealing with one aspect, but fail when simultaneously confronted with other aspects. To support this claim, a comprehensive empirical comparison is performed, and it is reported that methods specifically designed for either continual learning or few-shot learning indeed do not perform well on the FLUID benchmark, as they are outperformed by relatively simple baselines.\n", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "As indicated above, a critical issue to be addressed is the lack of non-stationarity in the benchmark. Either the specification of the benchmark must be changed such that it includes relevant forms of non-stationarity, or the descriptions / interpretations of the benchmark must be changed. If the authors choose the latter option, I would probably also recommend leaving out the continual learning methods from the comparison, to avoid the risk of misinterpretation.\n\nOther issues that might be good to keep in mind when rewriting the paper:\n- One finding of the paper is that meta-training based few-shot methods such as prototypical networks and MAML do not perform well on the proposed FLUID benchmark. But I’m wondering whether this finding is correct / fair. It seems to me that the meta-training for these methods is done on test sequences (5-shot 30-way) very different from the actual sequence that the method is tested on. Is it therefore not expected that these methods won’t perform well? Unless I’m missing something, at the moment the paper is written in a way to suggest this is a lot more surprising than it actually is.\n- Another finding claimed by the paper is that when a suitable pre-trained network is available, simple approaches (e.g., an NCM classifier on top of the frozen features) often perform very well. I think the paper demonstrates this in a comprehensive and convincing way, and as such I think this is an important contribution, but the insight that when a suitable pre-trained network is available simple approaches work well is not new, at least not in the continual learning field (e.g., Hayes & Kanan 2020 CVPR-W, https://arxiv.org/abs/1909.01520).\n\nAnother note I had is that on p18 the authors mention that they provide “a more than fair comparison”. What exactly is meant by “more than fair”? I guess the authors mean that the comparison is in some way stacked against the method that is found to perform best, but this would benefit from more explanation.\n", "broader_impact_concerns": "No concerns in this regard.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers agree that the latest version of the paper meets the criteria for acceptance at TMLR, congratulations!

The discussion period was thorough and I thank the authors for carefully considering the comments of the reviewers and updating their paper several times accordingly. In particular, the added connection to the literature and, even more importantly, the more precise discussion of the differences between the FLUID evaluation and the "standard" CL training/testing better emphasize the contribution of this paper. In my view, the improved clarity is likely to increase the usefulness of this contribution to the community.

Minor revision. You responded to reviewer B1ZX's comments regarding the motivation of the setting with a few interesting potential applications. I encourage you to add (part of) this discussion to the paper (e.g., in passing in the introduction or in a paragraph or two in the related works or the appendix). This will better justify this claim "However, in many applications, the exact conditions cannot be known a priori and are likely to change over time."
Decision: Accept (poster)