dataesr/scientific-paragraphs-categorization huggingface dataset:
Transferring knowledge between datasets The optimization of α and β parameters is impossible in class IL without memory, since exemplars of past classes are unavailable. To circumvent this problem, we hypothesize that optimal values of these parameters can be transferred between reference and target datasets, noted D r and D t respectively. The intuition is that these values are sufficiently stable despite dataset content variability. We create a set of reference datasets and perform a modified class IL training for them using the procedure described in Algorithm 1. The modification consists in exploiting a validation set which includes exemplars of classes from all incremental states. Validation set storage is necessary in order to optimize the parameters from Equation 2 and is possible since reference dataset training is done offline.
Note that backbone incremental models for D r are trained without memory in order to simulate the IL setting of target datasets D t. We then store bias correction parameters optimized for reference datasets in order to perform transfer toward target datasets without using a memory. For each incremental state, we compute the average of α and β values over all reference datasets. The obtained averages are used for score rectification on target datasets. This transfer uses the procedure described in Algorithm 2. The memory needed to store transferred parameters is negligible since we need 2×(2+3+. +S) = (S+2)×(S-1) floats for each dataset and S value. For S = {5, 10, 20} states, we thus only store 28, 108 and 418 floating-point values respectively. Algorithm 1: Optimization of calibration parameters inputs : A, D r s for s ∈ 1, S ▷ reference dataset randomly initialize M 1 ; M * 1 ← train(A; M 1 , D r 1 ) ; for s = 2.
S do M * s ← update(A; M * s-1 , D r s ) ; α k s ← 1, β k s ← 0 for each k ∈ 1, s ; foreach (x, y) ∈ D r s ▷ validation set do o s ← M * s (x) ; for k = 1. s do o k s ← adBiC(o k s ) = α k s o k s + β k s • 1 ; end q s ← σ(o s ) ; loss ← L(q s , y) ; (α 1 s , β ; M * 1 ← train(A; M 1 , D t 1 ); for s = 2. S do M * s ← update(A; M * s-1 , D s ) ; foreach (x, y) ∈ D t s ▷ test set do o s ← M * s (x) ; for k = 1. s do o k s ← adBiC(o k s ) = α k s o k s + β k s • 1 ; end q s ← σ(o s ) ; ŷ ← argmax y∈ 1,Ns (q s ) ; ▷ inference end end In Figure  3 , we illustrate optimal parameters obtained across R = 10 reference datasets which are further described in Section 4. We plot α k and β k values learned after S = 10 IL states, using LwF  [21]  and LUCIR  [14]  methods. Mean and standard deviations are presented for (a) LwF  [21]  (b) LUCIR  [14]  Figure  3 : Averaged α k (left) and β k (right) values computed for R = 10 reference datasets using LwF and LUCIR, at the end of an incremental process with S = 10 states.
past and current incremental states in the final state of the IL process. The parameter ranges from Figure  3  confirm that, while optimal values do vary across datasets, this variation is rather low and calibration profiles remain similar. Consequently, parameters are transferable. When R > 1, a transfer function is needed to apply the parameters learned on reference datasets to a target dataset. We transfer parameters using the averaged α k s and β k s values, obtained for the set of D r. In Section 4, we evaluate this transfer against an upper-bound oracle which selects the best D r in each state. The proposed approach adds a simple but effective linear layer to calibrate the predictions of backbone class IL methods. Consequently, it is applicable to any IL method which works without memory.
Evaluation In this section, we discuss: (1) the reference and target datasets, (2) the backbone methods to which bias correction is applied and (3) the analysis of the obtained results. The evaluation metric is the average top-1 accuracy of the IL process introduced in  [28] , which combines the accuracy obtained for individual incremental states. Following  [5] , we discard the accuracy of the first state since it is not incremental. We use a ResNet-18 backbone whose implementation details are provided in the supp. material.
Datasets Reference datasets. The preliminary analysis from Figure  3  indicates that bias correction parameters are rather stable for different reference datasets. It is interesting to use several such datasets in order to stabilize averaged bias correction parameters. In our experiments, we use 10 reference datasets, each including 100 randomly chosen leaf classes from ImageNet  [7] , with a 500/200 train/val split per class. There is no intersection between these datasets, as each class appears only in one of them. Target datasets. We test TransIL with four target datasets. They were selected to include different types of visual content and thus test the robustness of the parameter transfer. The class samples from the target datasets are split into 500/100 train/test subsets respectively.
There is no intersection between the classes from the reference datasets and the two target datasets which are sampled from Ima-geNet. We describe target datasets briefly hereafter and provide details in the supplementary material: • CIFAR-100  [18]  -object recognition dataset. It focuses on commonsense classes and is relevant for basic level classification in the sense of  [29]. • IMN-100 -subset of ImageNet  [7]  which includes 100 randomly selected leaf classes. It is built with the same procedure used for reference datasets and is thus most similar to them. IMN-100 is relevant for fine-grained classification with a diversity of classes. • BIRDS-100 -uses 100 bird classes from ImageNet  [7]. It is built for domain fine-grained classification. • FOOD-100 -uses 100 food classes from Food-101  [4].
Backbone incremental learning methods We apply adBiC on top of four backbone methods which are usable for class IL without memory: • LwF [28] -version of the original method from  [21]  which exploits distillation to reduce catastrophic forgetting for past classes. • LUCIR [14] -distillation-based approach which uses a more elaborate way of ensuring a good balance between model stability and plasticity. We use the CNN version because it is adaptable to our setting. • FT+ [23] -fine-tuning in which past classes weights are not updated to reduce catastrophic forgetting. • SIW [2] -similar to FT+, but with class weights standardization added to improve the comparability of prediction between past and new classes. We compare adBiC to BiC, the original linear layer from  [33]. We also provide results with an optimal version of adBiC, which is obtained via an oracle-based selection of the best performing reference dataset for each IL state.
Overall results Results from Figure  2  (right) indicate that the degree of forgetting depends on the initial state in which classes were first learned. Applying calibration parameters learned on reference datasets clearly reduces the imbalance of mean prediction scores and the bias toward recent classes. Results from Table  1  show that our method improves the performance of baseline methods for all but two of the configurations evaluated. The best overall performance before bias correction is obtained with LwF. This result confirms the conclusions of  [2, 23]  regarding the strong performance of LwF in class IL without memory for mediumscale datasets. With adBiC, LUCIR performs generally better than LwF for S = 5 and S = 10, while LwF remains stronger with S = 20 states. Results are particularly interesting for LUCIR, a method for which adBiC brings consistent gains (up to 16 accuracy points) in most configurations.
This indicates that averaging parameters is an effective way to aggregate parameters learned from reference datasets. However, it would be interesting to investigate more refined ways to transfer parameters from reference to target datasets to further improve performance. The comparison of target datasets shows that the gain brought by adBiC is the largest for IMN-100, followed by BIRDS-100, CIFAR-100 and FOOD-100. This is intuitive  Except for LwF, adBiC gains are larger for S = {5, 10} compared to S = 20. This result is consistent with past findings reported for bias correction methods  [23, 33]. It is mainly explained by the fact that the size of validation sets needed to optimize adBiC parameters is smaller and thus less representative for larger values of S. A larger number of states leads to a higher degree of forgetting.
Robustness of dataset knowledge transfer We complement the results presented in Table  1  with two experiments which further evaluate the robustness of ad-BiC. First, we test the effect of a different number of training images per class for reference and target datasets. We remove 50% of training images for target datasets to test the transferability in this setting. The obtained results, presented in Table  2 , indicate that performance gains are systematic for LwF and LUCIR, albeit lower compared to results in Table  1. Results are more mixed for SIW and FT+, but adBiC still has a positive effect in the majority of cases. This experiment shows that the proposed dataset knowledge transfer approach is usable for reference and target datasets which have a different number of training samples per class.
However, maintaining a low difference in dataset sizes is preferable in order to keep the transfer effective. Second, we assess the robustness of the method with respect to R, the number of available reference datasets. We select the FOOD-100 dataset because it has the largest domain shift with respect to reference datasets and is thus the most suitable for this experiment. We vary R from 1 to 9 and perform transfer with ten random samplings for each R value. Results obtained for LUCIR are reported in Table  3. Accuracy levels are remarkably stable for different values of R and significant gains are obtained even when using a single reference dataset. These results confirm that parameter transfer is effective even with few reference datasets, which is interesting considering that the computational cost S = 5 Raw R = 1 R = 2 R = 3 R = 4 R = 5 R = 6 R = 7 R = 8 R = 9 R = 10 44.
Conclusion We introduced a method which enables the use of bias correction methods for class IL without memory. This IL scenario is challenging, because catastrophic forgetting is very strong in the absence of memory. The proposed method TransIL transfers bias correction parameters learned offline from reference datasets toward target datasets. Since reference dataset training is done offline, a validation memory which includes exemplars from all incremental states can be exploited to optimize the bias correction layer. The evaluation provides comprehensive empirical support for the transferability of bias correction parameters. Performance is improved for all but two of the configurations tested, with gains up to 16 top-1 accuracy points. Robustness evaluation shows that parameter transfer is efficient when only a small number of reference datasets is used for transfer.
It is also usable when the number of training images per class in target datasets is different from that of available reference datasets. These last two findings are important in practice since the same reference datasets can be exploited in different incremental configurations. A second contribution relates to the modeling of the degree of forgetting associated to past states. While recency bias was already acknowledged  [23] , no difference was made between past classes learned in different IL states  [33]. This is in part due to validation memory constraints which appear when the bias correction layer is optimized during the incremental process. Such constraints are reduced here since reference datasets training is done offline and a refined definition of the BiC layer with specific parameters for each past state becomes possible.
The comparison of the standard and of the proposed definition of the bias correction layer is favorable to the latter. The reported results encourage us to pursue the work presented here. First, the parameter transfer is done using average values of parameters learned on reference datasets. A finer-grained transfer method will be tested to get closer to the oracle results reported in Table  1. The objective is to automatically select the best reference dataset in each IL state of a target dataset. Second, we exploit an adapted version of a bias correction method which was initially designed for class IL with memory. We will explore the design of methods which are specifically created for class IL without memory. Finally, while distillationbased methods outperformed methods which do not use distillation for the datasets tested here, existing results  [2, 23]  indicate that the role of distillation diminishes with scale.
Introduction In this supplementary material, we provide: • implementation details of adBiC and the tested backbone IL methods (Section A). • classes lists of target datasets used for evaluation (Section B). • additional figures highlighting the effects of adBiC on the tested backbone methods (Section C). • additional tables for the robustness experiment presented in Section 4. 4 of the paper (Section D). • results on PLACES-100 dataset (Table  5 ). • additional accuracy plots for all methods and datasets (Figures  5  and 6 ).
A. 2. Adaptive bias correction The correction of output scores is done in the same way for all methods. After the extraction of scores and labels for each model, batches are fed into a PyTorch  [26]  module which performs the optimization of adBiC parameters, or the transfer of previously learned parameters. Following  [23] , BiC and adBiC layers are implemented as pairs of parameters and optimized simply through backpropagation. Parameters α k s , β k s corresponding to each incremental state s are optimized for 300 epochs, with the Adam  [17]  optimizer and a starting learning rate of 10 -3. An L2penalty is added to the loss given in Equation 3 of the main paper, with a lambda of 5•10 -3 for α parameters and 5•10 -2 for β parameters.
B. Datasets description We provide in Table  4  the lists of classes contained in each of the target datasets we used for evaluation. Overall, IMN-100, the randomly sampled set of 100 leaf classes from Imagenet  [7] , is more diversified than CIFAR-100, which mostly contains animal classes. IMN-100 is visually varied between different types of objects, foods, animals, vehicles, clothes and events. CIFAR-100 contains, in addition to animals, some types of objects and vehicles. The FOOD-100 dataset and BIRDS-100 (extracted from ImageNet  [7] ) are more specialized than IMN-100 and CI-FAR-100 and are thus useful to test the robustness of our method on finer-grained datasets. PLACES-100 and FOOD-100 are target datasets which have a larger domain shift with ImageNet classes, and are thus useful to test the robustness of our method against domain variation.
D. Effect of the number of reference datasets In Table  6 , we provide a full set of results for the accuracies of adBiC with LwF, LUCIR, SIW and FT+ when varying the number R of reference datasets, following Table  5 : Average top-1 incremental accuracy using S = {5, 10, 20} states, for the PLACES-100 dataset with all and half of the training data. The PLACES-100 dataset is extracted from Places365  [35]. Similarly to the other datasets presented in the main paper, we randomly select a hundred classes from the original dataset to construct a dataset with a hundred classes. Results obtained are comparable to those obtained on the other datasets, despite the domain shift from ImageNet. Gains obtained over the backbone method are given in green, and the best results for each setting in bold.
+ 0. 6 33. 3 + 1. 5 24. 7 + 2. 5 w/ adBiC + O 54. 9 + 1. 9 47. 3 + 3. 3 32. 6 + 3. 5 55. 9 + 2. 1 44. 2 + 3. 1 33. 1 + 3. 9 55. 8 + 2. 1 44. 8 + 3. 0 33. 3 + 3. 2 44. 0 + 1. 1 34. 2 + 2. 4 25. 3 + 3. 1 Average top-1 incremental accuracy using S = {5, 10, 20} states. Results are presented for each method without parameter transfer and with BiC and adBiC transfer. The gain (green) and loss (red) in accuracy obtained with parameter transfer are provided for each configuration. Joint is an upper bound obtained using a standard training with all data available. O denotes a choice of the reference dataset by oracle, in which the best reference dataset for each state is selected for transfer. Best results for each setting (excluding the oracle) are in bold. A graphical view of this table is in the supplementary material.
Average top-1 incremental accuracy of adBiC-corrected models trained incrementally on FOOD-100 with LUCIR, for S = {5, 10, 20} states, while varying the number R of reference datasets. For R ≤ 9, results are averaged across 10 random samplings of the reference datasets (hence the std values). Raw is the accuracy of LUCIR without bias correction. of offline training is also reduced. Results with other methods for CIFAR-100 are provided in the supp. material.
Acknowledgments. This work was supported by the  European Commission  under European  Horizon 2020 Programme , grant number  951911 -AI4Media , and partially supported by the  LabEx PERSYVAL-Lab  ( ANR-11-LABX-0025-01 ) funded by the  French program  " Investissements d'Avenir ". This publication was made possible by the use of the FactoryIA supercomputer, financially supported by the  Ile-de-France Regional Council.
5 -0. 1 14. 8 + 0. 3 33. 0 + 1. 3 21. 9 -1. 3 13. 8 -0. 8 32. 3 + 2. 6 22. 5 -0. 8 12. 4 -1. 1 28. 6 -0. 1 20. 6 -0. 5 11. 8 -1. 5 w/ adBiC 31. 9 + 3. 0 23. 6 + 1. 0 15. 0 + 0. 5 34. 9 + 3. 2 23. 7 + 0. 5 15. 7 + 1. 1 34. 0 + 4. 3 25. 0 + 1. 7 14. 2 + 0. 7 30. 8 + 2. 1 22. 2 + 1. 1 14. 2 + 0. 9 w/ adBiC + O 32. 5 + 3. 6 24. 6 + 2. 0 15. 9 + 1. 4 35. 7 + 4. 0 24. 9 + 1. 7 16. 2 + 1. 6 34. 5 + 4. 8 25. 7 + 2. 4 15. 4 + 1. 9 31. 3 + 2. 6 22. 7 + 1. 6 14. 5 + 1. 2 Joint 72. 7 75. 5 80. 9 71. 03 as IMN-100 has the closest distribution to that of reference datasets. BIRDS-100 is extracted from ImageNet and, while topically different from reference datasets, was created using similar guidelines. The consistent improvements obtained with CIFAR-100 and FOOD-100, two datasets independent from ImageNet, shows that the proposed transfer method is robust to data distribution changes.
Accurate knowledge of the seasonal snow distribution is vital in several domains including ecology, water resources management, and tourism. Current spaceborne sensors provide a useful but incomplete description of the snowpack. Many studies suggest that the assimilation of remotely sensed products in physically based snowpack models is a promising path forward to estimate the spatial distribution of snow water equivalent (SWE). However, to date there is no standalone, open-source, community-driven project dedicated to snow data assimilation, which makes it difficult to compare existing algorithms and fragments development efforts. Here we introduce a new data assimilation toolbox, the Multiple Snow Data Assimilation System (MuSA), to help fill this gap. MuSA was developed to fuse remotely sensed information that is available at different timescales with the energy and mass balance Flexible Snow Model (FSM2).
We demonstrate the wide capabilities of MuSA through two snow data assimilation experiments. First, 5 m resolution snow depth maps derived from drone surveys are assimilated in a distributed fashion in the Izas catchment (central Pyrenees). Furthermore, we conducted a joint-assimilation experiment, fusing MODIS land surface temperature and fractional snow-covered area with FSM2 in a single-cell experiment. In light of these experiments, we discuss the pros and cons of the assimilation algorithms, including their computational cost.
In addition, some data assimilation frameworks are based on highly specific implementations tied to operational constraints  (Cluzet et al. , 2021). This situation prevents the development of reproducible snow data assimilation studies and challenges the comparison of the performance of different snow data assimilation algorithms. This is why we have developed a new open-source data assimilation toolbox, the Multiple Snow Data Assimilation System (MuSA). MuSA is an ensemble-based snow data assimilation tool. It enables the fusion of multiple observations with a physically based snowpack model while taking into account various sources of forcing and measurement uncertainty. It is an open-source collaborative project entirely written in the Python programming language. It should facilitate the development of snow data assimilation experiments, as well as the generation of snowpack reanalyses and nearreal-time snowpack monitoring.
Overview of the data assimilation system The core of MuSA is an energy and mass balance of the snowpack model, Flexible Snow Model (FSM2;  Essery, 2015). FSM2 has several options for the parameterizations of key processes related to the energy and mass balance of the snowpack. The most complex configuration is chosen by default in MuSA, leading to a more detailed simulation of the internal snowpack processes. Albedo is computed from the age of the snow, decreasing its value as snow ages and increasing it with fresh snowfalls. Thermal conductivity of the snowpack is computed as a function of the snow density. Snow density is computed considering overburden and thermal metamorphism. Turbulent energy fluxes are computed as a function of atmospheric stability. Meltwater percolation in the snowpack is computed using the gravitational drainage.
Although this is the default configuration, it is possible to choose any other FSM2 setup, which may result in slight performance differences in terms of both the computational cost and the accuracy of the model  (Günther et al. , 2019). As envisaged in  Westermann et al. (2022) , the potential to run multiple model configurations leaves the door open for pur-suing rigorous model comparison using the model evidence framework  (MacKay, 2003)  in the cryospheric sciences. MuSA was designed as a Python program encapsulating the FSM2 Fortran code (Fig. 1 ). It handles the forcing and initial files, as well as the FSM2 runs and outputs, by internally generating the required ensemble of simulations from simple configuration (config. py) and constants (constants. py) files that should be filled by the user to both configure the MuSA environment and define the priors, respectively.
Then, it solves most of the challenges of ensemble-based snow data assimilation frameworks for the user, as it internally handles the input/output (I/O) and parallelization while keeping track of the ensembles that should flow in different ways depending on the chosen data assimilation (DA) algorithm. The data assimilation algorithms are independently implemented on a grid cell basis, allowing both single-cell and parallel spatially distributed simulations. The outputs of MuSA consist of the posterior mean snow simulation from FSM2 (updated_idx_idy. csv), the posterior standard deviations of the FSM2 ensemble (sd_idx_idy. csv), information related to the posterior perturbation parameters and the observations (DA_idx_idy. csv), and the original simulation without perturbation (OL_idx_idy. csv).
Additionally, it is possible to store the posterior ensemble of every cell. The following data assimilation algorithms are currently implemented in MuSA: particle filter (PF;  Gordon et al. , 1993; van Leeuwen et al. , 2019) ; ensemble Kalman filter (EnKF;  Evensen, 1994; van Leeuwen, 2020) ; ensemble Kalman filter with multiple data assimilation (EnKF-MDA;  Emerick and Reynolds, 2012) ; particle batch smoother (PBS;  Margulis et al. , 2015) , a batch smoother variant of the PF; ensemble smoother (ES; van Leeuwen and  Evensen, 1996) , a batch smoother variant of the EnKF; ensemble smoother with multiple data assimilation (ES-MDA;  Emerick and Reynolds, 2013). In its current version, MuSA is able to assimilate the following variables: - SWE [mm] -snow depth [m] land surface temperature  [K]  fractional snow-covered area  [-]  albedo  [-]  sensible heat flux to the atmosphere [W m -2 ] latent heat flux to the atmosphere [W m -2 ].