Where is the Truth? The Risk of Getting Confounded in a Continual World
Authors: Florian Peter Busch
Roshni Ramanna Kamath
Rupert Mitchell
Wolfgang Stammer
Kristian Kersting
Martin Mundt
Venue: ICML 2025 spotlightposter (2025)
Keywords: Continual Learning
Confounding
Computer Vision
Shortcut Learning
Dataset
Primary area: deep_learning
TL;DR: We propose a dataset to study confounding in continual learning and show that overcoming forgetting is insufficient to avoid confounding

Abstract: A dataset is confounded if it is most easily solved via a spurious correlation which fails to generalize to new data. In this work, we show that, in a continual learning setting where confounders may vary in time across tasks, the challenge of mitigating the effect of confounders far exceeds the standard forgetting problem normally considered. In particular, we provide a formal description of such continual confounders and identify that, in general, spurious correlations are easily ignored when training for all tasks jointly, but it is harder to avoid confounding when they are considered sequentially. These descriptions serve as a basis for constructing a novel CLEVR-based continually confounded dataset, which we term the ConCon dataset. Our evaluations demonstrate that standard continual learning methods fail to ignore the dataset's confounders. Overall, our work highlights the challenges of confounding factors, particularly in continual learning settings, and demonstrates the need for developing continual learning methods to robustly tackle these.

Where is the Truth?
The Risk of Getting Confounded in a Continual World
Florian Peter Busch 12 Roshni Ramanna Kamath 12 Rupert Mitchell 1
Wolfgang Stammer 12 Kristian Kersting" 1234
Martin Mundt
Abstract
Task
Task
Task
Cumulative Training
(T:) (T, (T. Ta
A dataset confounded if most easily solved via spurious correlation which fails general- ize to new data. In this work, show that, in continual learning setting where confounders may vary in time across tasks, the challenge of mitigating the effect of confounders far exceeds the standard forgetting problem normally consid- ered. In particular, we provide formal descrip- tion of such continual confounders and identify that, in general, spurious correlations are easily ignored when training for all tasks jointly, but it is harder to avoid confounding when they are considered sequentially. These descriptions serve as a basis for constructing a novel CLEVR- continually confounded dataset, which we term the ConCon dataset. Our evaluations demonstrate that standard continual learning methods fail to ig- nore the dataset's confounders. Overall, work highlights the challenges of confounding factors, particularly in continual learning settings, and demonstrates the need for developing continual learning methods to robustly tackle these.
Arctic Fax (wkine)
Broccali (green)
Tiger (bluc)
89.3%
?
Joint Training
91.7%
Jullyriah
Snswmoblle
(T. T..
Figure I. Challenges of continual confounding. Each task is cu- rated with spurious color correlations potentially affecting a model in lcarning underlying discriminative features. The different color- based confounding features across sequential tasks demonstrate the impact the classification problem with joint and cumulative training, with a 2.4% gap in performance.
well during training but fail to generalize to test cases (Ye et al., 2024). This can lead to undesired behavior at deploy- ment both in terms of trustworthiness (e.g., does the model make decisions for the "right" reasons (Ross et al., 2017)) and overall generalization failure (e.g.. does the model still make correct predictions (Geirhos et al., 2020)). However, these settings so far have mainly been investigated in con- ventional, static learning settings (Wu et al., 2023; Sagawa et al., 2020), rather than more realistic, continual learning (CL) scenarios (Chen al., 2018) where models process data sequentially, creating fundamental constraint that they cannot access all data simultaneously (McCloskey & Cohen, 1989; Parisi et al., 2019). While prior research indicates that task ordering has minimal impact in such sequential train- ing (De Lange et al., 2022), the distribution confounders across tasks can impact model performance negatively.
1. Introduction
Confounding factors represent a dangerous issue for AI systems, often steering models toward shortcuts instead of true understanding (Lapuschkin al., 2019; Schramowski et al., 2020; Steinmann et al., 2024). Data is considered con- founded when models learn spurious correlations that work
"Equal contribution 'Computer Science Department, TU Darmstadt, Germany. "Hessian Center for Al (hessian.AD), Germany. 'German Research Center for Al (DFKI), Ger- many *TU Darmstadt, Centre for Cognitive Science, Ger- many. 'University of Bremen, Department of Mathematies and Computer Science, Germany. Correspondence Roshni Ramanna Kamath <rrshni kamath @wtdaaasttaaddddee... Martin Mundt <mundtm @ uni bremen.de>.
For example, consider an image classification problem on natural images, i.c., ImageNet (Deng et al., 2009). We conducted controlled experiment using the ResNet-18 model (He et al., 2016) on selected classes (cf. Fig. 1). We start by training the model on the first two classes, specif- ically arctic fox and jellyfish. We then introduce two addi- tional classes, namely broccoli and snowmobile, followed by tiger shark and tree frog. Note that each class highly
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

The Risk of Getting Confounded in a Continual World
correlated with specific colors: while, blue, or green. We observe that such a cumulative training setup achieves average accuracy of 89.3% (across 5 random seeds). Inter- estingly, when trained on all the data simultaneously i.e., joint training, the model yields an accuracy of 91.7%. This performance gap of 2.4% is substantial, despite utilizing the same model and hyperparameters, and the last training iter- ation of the cumulative setup using the exact same data as joint training. As images in the first task contain predomi- nantly white or blue backgrounds, we posit that the temporal accessibility constraints give rise to spurious correlations that a model could use for its predictions. These are hard to remedy, as CL limits any access to past observations. Such correlations are unlikely to be similarly challenging when presented in joint large dataset from the start - order becomes imperative. We posit that the challenges posed by confounding, especially in continual learning settings where data is not accumulated completely, exceed the stan- dard problems usually considered in continual learning and confounding studies separately.
In summary, our contributions are: i) We design a logical framework that allows us to systematically study (insidi- ous) continual confounding, ii) We use the logical frame- work to construct the ConCon dataset that manifests con- tinual confounding and iii) We empirically evaluate vari- ous ML and CL methods and show that they get (insidi- ously) confounded when trained on ConCon. We publish the ConCon dataset, https: / zenodo. org/ /o / /10. 5281 / zenodo. 10630481, and source code for generat- ing the dataset as well as experimental results of our work, https: //gg thub. www.mm/m.................. htae//mmlreeeeee concon.
2. Related Work
Confounding. The need for trustworthy Al models has led to the curation of several confounded datasets (Steinmann et al., 2024) that allow for tailored analysis of model be- havior. Many of these datasets lie in the visual domain, and due to both the ease of data curation itself, but potentially also to more controlled data settings, existing datasets are typically of a synthetic nature. Important mentions are Toy- Color (Ross et al., 2017), ColorMNIST (Rieger et al., 2020; Kim et al., 2019), Decoy-MNIST (Ross et al., 2017) and CLEVR-Hans? & 7 (Stammer et al., 2021). However, con- founded datasets based on natural images and confounders have also been introduced, e.g., the confounded hyperspec- tral plant dataset of Schramowski et al. (2020), the ISIC Skin Cancer dataset (Codella et al., 2019; Tschandl et al., 2018), Covid-19 (DeGrave et al., 2021) and CUB5Box (Bontempelli et al., 2023). An issue that can occur when analyzing Al models based on natural data versus controlled synthetic environment is that multiple spurious correlations can oc- cur, such that revising a model to ignore one correlation does not guarantee that it is free from the influence of fur- ther, possibly unknown ones (Friedrich et al., 2023). This makes synthetic datasets attractive alternatives, particularly for initial investigations of models and potential mitigation strategies. Despite their great utility in analyzing offline learning settings, none of the confounded datasets men- tioned above allow for investigating Al models in terms of confounding factors in the presence of change over time.
To this end, we systematically investigate model behavior in different confounding settings and show that machine learning (ML) models particularly struggle with continual confounding. We observe that this is so even if training on the joint dataset successfully prevents confounding; a phenomenon we will later refer to as insidious continual confounding. Thus, motivated by the observed disparities in the previous ImageNet-baste evaluation and the need for a controlled, systematic analysis, we first introduce a formal description that characterizes continual confounding and highlights its unique challenges. We next introduce Con- Con, controlled confounded visual dataset for continual learning based on CLEVR (Johnson et al., 2017). In Con- Con, there exists one ground truth rule which can be used to successfully classify all images. Additionally, each task has one task- speciiic confounder that can be used to solve the problem for that specific task. For any other task, the previous confounders are not informative. Ideally, an ML model should be able to extract the true rule underlying all tasks instead of focusing on the task-specific confounders. We evaluate the performance of ML and CL methods on this problem and highlight that indeed these focus on the con- founders. Moreover, they are unable to extract the ground truth rule, thereby failing to make correct predictions on unconfounded data. Expanding on this general concept of continual confounding, we finally empirically corroborate the greater challenge of insidious continual confounding, where a model trained jointly on all tasks can succeed at learning the ground truth rule, while at the same time failing to do so when all data accumulated sequentially.
Continual Learning. The challenge for enabling ML mod- els to adapt to new concepts while retaining previously acquired knowledge has led to the emergence of continual learning (Chen et al., 2018; Mundt et al., 2023b). This has resulted in the creation of plethora of many sequential dataset benchmarks for studying continual learning (Koh et al., 2021; Peng et al., 2019). Despite several nuanced vari- ations in set-up and evaluation focus (Mundt et al., 2022), the shared objective of all these datasets is to empower the conception of continual techniques that alleviate the catas- trophic interference (McCloskey & Cohen, 1989; Ratcliff, 1990). Here, it is assumed that it is strictly beneficial to
We include another experiment on a larger number of Ima- geNet classes in Appendix E, where the gap between joint and cumulative training is even larger.
2

---

The Risk of Getting Confounded in a Continual World
retain previous experience as much as possible to avoid forgetting- without the need to store all prior observed data (Hayes et al., 2021; Kudithipudi et al., 2022). Respective benchmarks typically convert existing datasets into disjoint incremental variants (e.g., permuted MNIST (Matan et al., 1990; Goodfellow et al., 2015), or class incremental CIFAR- 10 (Krizhevsky & Hinton, 2009) and ImageNet (Deng et al., 2009)), successively learn on individual (language) datasets (Biesialska et al., 2020), or simulate controlled distribution changes (Hess et al., 2021). Recently, neuro-symbolic con- tinual CLEVR (Marconato et al., 2023) followed in this spirit of devising a disjoint sequence of tasks, introducing the first dataset with clear underlying logical rules. However, follows prior art in focusing on incrementing, e.g., object type or color to highlight the role neuro-symbolic methods may play in knowledge accumulation. Existing continual learning datasets, while useful for investigating sequential adaptation, do not support the systematic analysis of varying confounders across tasks. With ConCon, we introduce the first CL dataset where confounders differ across tasks.
3.1. Formal Description of Continual Confounding
Let us now provide a formal description of the aforemen- tioned settings support systematic data generation. Let g refer to ground truth predicate, which maps images to boolean classification. Let ea refer to the t-th confounding predicate. For a task I, we have set of positíve examples Pj and a set of negative examples N₁. Pa and N₁ are drawn randomly from examples satisfying the defining predicates Pt and ny respectively. When we say the task is confounded with confounder ct, we mean that the task can be solved both with the ground rule g and with ct. From this may derive the following constraints on Pt. and ny: Pt S 9 A Ct+ n4 S -(g V ct), where A and V are logical conjunction and disjunction respectively, and we define the partial ordering x S y and only if x = y.¹ Intuitively, these conditions state that all examples in Pe satisfy both g and ct, and all examples in Mr satisfy neither g nor cz. The natural choice
Pt == g A Ct: =
:= V
(1)
is the most permissive regarding which examples are valid dataset members, and is therefore least permissive regarding the simultaneous solution of all tasks. We accordingly call this dataset choice strict. The reader will note that, in this case, the confounder of task / may in general occur in both the positive and negative set of other tasks t'. We de- fine our second dataset, di disjoint sinnt, such that confounders Ct are observed only in their own task, i.e., in neither the positive Per or negative N₂ sets of any other task 1'. We do this both because such datasets are encountered in practice and because it results in interesting consequences for the possible joint solutions of all tasks, as will be discussed shortly. The constraints defining the negative set of the di s joint dataset are
3. Continual Confounding
Recall the introductory example with changing confounders over time. While each point in time (task)² is clearly con- founded when considered separately, it will turn out that subtleties in the specification used when generating these confounded datasets can have drastic consequences for the extent to which a subset of tasks remains confounded when considered jointly. Two possible choices for such specifi- cations are shown in Figure 2. In line with confounders often encountered in reality, e.g., specific watermarks, the first specification, labeled dis joint in the upper half of Figure 2, requires that confounder is only observed in its respective task. This makes finding a simultaneous solution to all tasks easier, since learned behavior specific to the pres- ence of the confounder from one task need not be unlearned in order solve some other task. We will later demonstrate with this setting that the avoidance of catastrophic forgetting across differently confounded tasks is insufficient to ensure acceptable generalization to unconfounded data.
T A 1=1,4
T
Pr :==- gAaA A
CG
U4 :- g V
C
(2)
where we have used the summation-style notation V to indicate disjunction over all for ny, and N to indicate conjunction over all negations of confounders from tasks not equal to / for Pt-
The st strict ct specification (lower half of Figure 2), on the other hand, has no such requirements regarding the presence or labeling of confounders from one task in the examples of another. This results in much greater disparity between behavior when training separately versus jointly on the tasks, since images satisfying the confounding rule of one task may appear in the negative set of some other task. In fact, our later evaluations in this setting may challenge traditional CL wisdom with respect to memory rehearsal as a solution.
Let us now consider the implications for the predicate learned by our network for the tasks both jointly and sep- arately. Assuming that the learned predicate r achieves perfect accuracy on the sets P. and N₁ from some task t, we obtain the following constraints on r:
Pt S Sr -44-
"e consider to be a time step, but to be consistent with the norm of CL literature, we refer to / as lask, even if the objective remains the same, but the underlying data distribution changes.
A boolean algebra interpreted this way forms distributive lattice, ie.,, partially ordered set with least upper bound V b, greatest lower bound a A b, and in which V and A distribute over each other.
3

---

The Risk of Getting Confounded in a Continual World
Disjoint Task 2
disjunction exclusive disjunction the left, and from conjunction to disjunction the right. These are both weakenings of the constraints on since V c S V Ct now require that if
Task
Task 3
Unconf.
and N C S Vi=1 ct. Intuitively, 9 is true and exactly one is true, then the predicate must evaluate to true. And if is true then either must be true, or at least one confounder er must be true.
Strict
Logical Implications. By studying these conditions, we may make several interesting observations about the pos- sible behavior of our learner. Firstly, a result of the weakened right hand side, V C is a valid solution to the di jo case, (as can be seen by substitution into Equation 6). This corresponds to the fact that the rule "ex- ample is positive if it satisfies the confounder rule from any task we have seen so far" is a valid solution at all points in training. In combination with the weakened left hand side, it is additionally true that the learned rule may have arbitrary behavior in the simultaneous presence of multi- ple confounders. This corresponds to the fact that, in the dis joint setting, multiple confounders may never occur together in the same example during training. Importantly, none of these solutions are likely to be valid for the uncon- founded test (which used only g for data generation) since none of them even mention, g-
Task
Task 2
Task 3
Unconf.
Ground
@
Conf. Task
X Not present
Figure 2. Abstract Representation of the ConCon Dataset. The unconfounded dataset (rightmost column) generated using only the ground truth rule (tt). For all three tasks, this rule is still present positive and absent in negative images. All positive images contain the confounded object (Ø). For the strict dataset, only the respective confounders are avoided in the negative samples. In the dis joint dataset, the confounders of any task may not reappear in either positive or negative images.
To solve all tasks jointly, r must satisfy T such constraints simultancously. Unless all confounders Ct are the same, this combined constraint will differ from the individual con- straints, a situation we will refer to as continual confounding. This can be expressed as least upper bound (resp. great- est lower bound) for the left (resp. right) hand side of the constraints. In the strict case this gives
As we will later see empirically, the strict case is less vulnerable to this variety of pathology. Intuitively, this is because any predicate r jointly satisfying all task constraints must either equal or contain within itself a statement of g. and thereby have less prior probability than g. In partic- ular, none of the above trivial solutions to the dis joint case generalize to the strict one. It is also possible in the strict ct case to force tighter bound ing the choices of Ct. If the confounders are exhaustive, i.e. Vee = T, then there can be no example which sat- isfies g but not r. since g = g A T S r, where denotes logical truth. Dually, it is impossible to simultaneously satisfy all confounders, Nee//=1 c: = 1, then there can be no example which satisfies r but not g, since r S gV 1 g. where 1 denotes logical falsity. When both of these condi- tions are satisfied we have g S r S g. and the ground truth predicate g is the unique joint solution to all tasks.
T (igna)ers (gva).
by restrict-
(4)
simplifying
T g Vasrssv ct
(5)
by distributivity. Intuitively, this means that if g is true and at least one e is true, then the learned predicate r must evaluate true. And, dually, that if r is true then either g must be true, all confounders e must be true.
Insidious Continual Confounding, An interesting property of the strict dataset that while each task is by defi- nition confounded when considered in isolation, the joint dataset is likely to have the ground truth g as its a priori most probable solution. As we later see empirically, this means that, if presented with all tasks simultaneously. our learner is likely to correctly infer the ground truth rule.
Repeating the above process for dis joint, we obtain
T 9 V aa 1=1
Ve t=1
(6)
In principle, a continual learner could remember sufficient information from earlier tasks to reconstruct the ground truth rule g once has seen all tasks. However, continual learners are generally designed to retain information only
where VT_ C is the exclusive disjunction over tasks t, i.e., exactly one of the c must be true. It can be seen that both sides of the constraints have been weakened, from
4

---

The Risk of Getting Confounded in a Continual World
relevant for decision making from previously observed tasks.
Rules
Positive
Negative
If the continual learner has solved early tasks by exploiting confounders, it will likely not have retained information sufficient to derive the ground truth rule. To such a con- tinual learner, the joint task will simply seem inconsistent and impossible: perfectly remembers its decision rules for the perfect solution of earlier tasks, and yet they are incompatible. This occurs despite the fact that, if it had initially been presented with the full joint dataset containing all tasks, the learner would have been able to derive the ground truth rule. Furthermore, even if there is in principle sufficient information remembered from earlier tasks for the continual learner to reject the confounded decision rules, we will later see that continual training while accumulating all data seen so far (i.e., an infinite memory buffer) may still fail where joint training succeeds. Since this variety of confounding only occurs once the continual aspect of the problem is introduced, we refer to it as insidious continual confounding, a specific kind of continual confounding.
T1 (Conf.) T2 (Conf.)
Blue
Sphere and small enhe
Metal
Sphere and small cube
Ta (Conf.)
Large
Sphere and small cube
Unconf
Sphere and small
3.2. The ConCon Dataset
Based on the previous descriptions, we now introduce the ConCon dataset. In general, every data point is an image containing multiple objects with several attributes, building on the framework of CLEVR (Johnson et al., 2017). New data simulated using the Blender software, and the ground truth rules and confounders are specified in JSON files. For dis joint and strict ConCon dataset variants, we have defined 3 tasks based on a common ground truth rule, and each task has been assigned unique confounder. The presence of the ground truth rule determines the binary class assignment of each image. An image can be either positive or negative (i.e., ground truth rule or is not satisfied). In this continual setting, different tasks vary by the presence absence of confounders, whereas the class that should be predicted remains the same (see Figure 2). Specifically, we have chosen each image to contain 4 objects, with choice among 2 sizes, materials, 3 shapes, and colors. In addition, a universal unconfounded dataset contains images that are based only on the ground truth rule, Further details, including a dataset sheet, are provided in the appendix.
Figure 3. ConCon samples. Shown are positive and negative ex- amples from our strict dataset. Throughout tasks (Tı, T2, Ts), sphere and small cube (ground truth rule (+)) must be present a positive image but can not appear together ative image. Alternatively, each of the three tasks may be solved by exploiting a confounder (presence object, respectively (#)). The contents of the unconfounded images depend only on the ground truth rule.
neg-
blue, metal, or large
only satisfy the ground truth and contain the task-specific confounder, but they further do not contain confounders specific to other tasks. Hence, for the first task, all positive images contain sphere and small cube and blue object and no metal or large object, all positive images in the second task contain a sphere and small cube and metal object and no blue or large object, and all positive images in the third task contain a sphere and small cube and large object and no blue or metal object. The negative images contain neither sphere and small cube nor blue nor metal nor large, across all the tasks. All other object properties are randomly sampled.
For di joint and strict ConCon variants, the exact ground truth rule was chosen as: sphere and small cube, see Figure 3. The task-specific confounding rules are blue, metal, and large, respectively. In the st riice ct. setting, all positive images of a task satisfy the ground truth rule and contain the task-specific confounder. For example, for the first task, all positive images contain both a sphere and small cube and a blue object. Conversely, the negative images of task neither satisfy the ground truth rule nor contain the specific confounder.
An important consideration in ConCon is the distribution shifts caused by the different confounding features and how they change across tasks. Specifically, features that act as confounders one task can also appear as random features in other tasks. The distribution shift between the confounded tasks and the unconfounded dataset is a result of such fea- tures no longer acting as confounders, i.e., they might appear on both positive and negative images but are not informative with respect to the class label. Our synthetic data generation used for ConCon ensures minimal distribution shift between confounded and unconfounded datasets while still exhibit- ing confounding one but not the other. Thus, the resulting
In the dis joint case, the positive images of a task not
5

---

The Risk of Getting Confounded in a Continual World
minimal train-test mismatch is intentional and allows for investigating whether high model accuracy is caused by
on the unconfounded dataset where the only valid solution is the ground truth rule itself. For the NN, we report an accuracy of 96.96 = 0.44 and NeSy achieves 80.33 = 0.91 accuracy. This confirms that the models can successfully learn the ground truth rule in principle.
the model making predictions based on the confounding features or the ground truth features.
Following the specifications outlined above, we have sys- tematically generated a comprehensive dataset comprising of 3000 train. 750 validation, and 750 test samples for both positive and negative sets of each task resulting in 9000 images per task. The additional unconfounded dataset is generated with the same size as these splits. Overall, a total of 63000 images have thus been generated for the ConCon dataset to enable thorough investigation. Precise implemen- tation details are provided in the appendix.
Comparison of Methods. We investigate eleven methods on ConCon. First, we train both models on the entire con- founded dataset, where we would like to see the interplay of the confounding rules across all tasks in joint training. As complement, we evaluate the model performance when trained on the 3 tasks one after the other in a naive sequen- tial manner. To combat catastrophic forgetting expected in the latter, we use memory- -based CL methods such as ER (Chaudhry et al., 2019), DER (Buzzega et al., 2020), and BGS (Lee et al., 2024), the regularization--aase method EWC (Kirkpatrick et al., 2017), and FROMP (Pan et al., 2020) which is a mix of both. We also evaluate the per- formance of GDUMB (Prabhu et al., 2020) on ConCon that is competitive against numerous CL methods and of PNNs (Rusu et al., 2022), where the model is expanded with each new task. We also evaluated generative replay method (van de Ven al., 2020), results of which can be found in Appendix D. For all the methods requiring a mem- ory buffer, we fix the size to 100 and populate it according to what each method stipulates. Finally, we investigate cu- mulative and a shuffled baseline. In the cumulative case, we append the entire dataset from previous task(s) to the current one and continue training the model on the dataset accumu- lated so far. This is typically treated as an upper-bound bound to CL performance in the spirit of ER with infinite data mem- ory (Hayes et al., 2021), which will be questioned in our later findings.* To isolate the consequences of confound- ing, in the shuffled setup, all the data is shuffled randomly across tasks, i.e., the confounders are no longer aligned with the tasks. This breaks the temporal alignment of the confounders while maintaining the CL constraint of the cu- mulative setup. We provide more detailed information on the CL methods and the full training details in Appendix B.
In summary, ConCon is the first dataset to support the sys- tematic investigation of the role of confounders in CL. The synthetic domains prove especially valuable in isolating and analyzing how different training mechanisms interact with data distribution characteristies. More importantly, ConCon allows us to conceive even more challenging settings, where the reader may perhaps be surprised to find that the introduc- tion of the continual element alone is sufficient to produce confounding. And furthermore, this confounding, which was not otherwise present, even persists when sequentially accumulating the dataset with infinite memory.
4. Experimental ConCon Evaluation
We now investigate whether continual learning methods overcome the confounding pitfalls of ConCon. Specifically, we aim to answer the following two questions:
Q1. Does preventing catastrophic forgetting help with con- tinual confounding?
Q2. Do continual learning methods suffer from insidious continual confounding?
4.1. Setup and Evaluation
Models. For each experiment, we use ResNet-18 model (He et al., 2016) (henceforth referred to as neural network or NN) and the neuro-symbolic concept learner (Stammer et al., 2021) (henceforth simply called NeSy) that is based on the Slot Attention module (Locatello et al., 2020) and Set Transformer (Lee et al., 2019). Where the ResNet-18 18 model bases its decisions on learned visual features, the NeSy model first extracts object property representations before making a final prediction. We describe training details and hyperparameters for both models in Appendix B.
Evaluation Metrics. We measure the accuracy on the con- founded test datasets across all tasks after training the model on each task. To investigate whether the model gets con- founded, we further measure the accuracy on the uncon- founded test set. We omit the standard deviations, as they are negligible (we include further results on the standard deviations in Appendix C).
4.2. CL Methods Get Continually Confounded
We start by investigating the ConCon di joi nt case, with evaluation results featured in the Table 1. Results on both
Sanity Check. To investigate whether a model is able to generalize to the unconfounded dataset after training on the confounded dataset, it must be established that the models can succeed in learning the ground truth rule when no con- founders are present. To this end, we train both models only
"Since this is the largest possible buffer size, applying buffer- based methods like ER with larger buffer sizes is expected to yield results between what run and the cumulative setup.
6

---

The Risk of Getting Confounded in a Continual World
Table ConCon Evaluation on disjoint. Test accuracies on dis joint for and NeSy. Current task (T) performance high almost throughout. Naive sequential training and EWC suffer from forgetting, showcasing huge drop in performance on previous tasks. While other methods manage keep high accuracies on previous tasks, our results on di sjoint show that avoiding forgetting not help with performance on unconfounded data. This holds true for both NN and NeSy. Current Task Acc.
Old Task Acc.
Unconf.
Method
Ti
T2
T3
Ti @ @T3
T2 T2@T3
Naive Sequential
100.0
92.2
100.0
50.0
63.6
48.1
EWC (Kirkpatrick et al., 2017)
100.0
99.9
100.0
50.1
60.6
47.7
FROMP (Pan et al., 2020)
100.0
99.8
99.4
99.2
97.9
50.2
ER (Chaudhry et al., 2019)
100.0
99.6
100.0
100.0
99.6
50.1
NN
DER (Buzzega al., 2020)
100.0
99.9
100.0
99.2
98.1
50.2
BGS (Lee et al., 2024)
100.0
99.9
100.0
95.1
96.2
50.4
GDUMB (Prabhu et al., 2020)
64.1
62.0
69.5
63.6
61.2
48.8
disjoint
PNNs (Rusu et al., 2022)
100.0
99.8
100.0
100.0
99.8
47.9
Shuffled
100.0
99.9
100.0
100.0
99.8
50.0
Cumulative
100.0
100.0
100.0
100.0
99.9
49.9
Joint
100.0
99.9
100.0
N/A
N/A
50.0
Naive Sequential
99.5
99.9
100.0
50.0
58.1
50.0
EWC (Kirkpatrick et al., 2017)
99.6
100.0
99.9
50.2
67.2
50.2
FROMP (Pan et al., 2020)
99.6
98.9
98.4
97.9
97.3
50.5
NeSy
ER (Chaudhry et al., 2019)
99.5
99.8
99.9
99.2
99.3
50.7
DER (Buzzega et al., 2020)
99.6
99.7
99.9
98.6
96.9
50.9
BGS (Lee et al., 2024)
99.6
99.9
99.9
94.4
95.1
50.9
GDUMB (Prabhu et al., 2020)
99.4
97.8
98.5
98.0
98.2
50.3
PNNs (Rusu et al., 2022)
99.5
100.0
100.0
99.6
100.0
49.6
Shuffled
99.8
99.7
99.9
99.7
99.9
50.8
Cumulative
99.8
99.9
99.9
99.8
99.9
50.8
Joint
99.7
99.9
99.9
N/A
N/A
50.7
NN and NeSy are reported and grouped into three categories per method: current task accuracy, old task accuracy, and accuracy on the unconfounded dataset.
trained continually (shuffled) or on the full dataset at the same time (joint). Since even the joint training setup does not learn the ground truth rule, it is unsurprising that the continual learning methods fail to do so as well.
For both NN and NeSy, we can see that most models success- fully classify the current tasks. However, the performance of the "Naive Sequential" approach on old tasks illustrates the necessity of continual learning methods to avoid forget- ting. While EWC does not appear significantly alleviate the memory-based methods ER, DER, BGS, and FROMP (benefiting mainly from the stored past examples), as well as PNNs (benefiting from additional clas- sification heads) doing so successfully, thereby achieving close to 100% accuracy on all confounded tasks after hav- ing learned them once. GDUMB trains the model using only the buffer, resulting in bad performance for NN but is otherwise in line with the other memory-basee approaches. Nevertheless, all methods both NN and NeSy fail the unconfounded dataset, even for the "Cumulative" setup, where all previous instances are kept when training the model sequentially. The models get continually confounded. This also remains true in both cases where model trained on data from all tasks the same time, no matter if it is
Our results show that both models prefer learning a combi- nation of the confounded tasks over the ground truth rule, as they provide reliable shorteuts for solving individual tasks during sequential training. As highlighted by our logical analysis. this can be explained by models learning the disjunction of all confounding features instead of the ground truth rule, despite the models' general ability to learn the ground truth rule, as previously shown in our sanity check.
forgetting,
Ultimately, does preventing catastrophic forgetting help with continual confounding? Our evaluation on the di joint int dataset showed that, in this setting where learn- ing the disjunction of confounders is preferred over learning the ground truth rule, CL methods can successfully avoid forgetting. However, preventing forgetting does not help with making correct predictions on unconfounded data. In particular, the ground truth rule also not learned when using a maximally large memory buffer (cumulative training that stores all data) when using the NeSy approach.

---

The Risk of Getting Confounded in a Continual World
Table ConCon Evaluation on strict. Test accuracies on strict for NN and NeSy. All continual learning methods fail to achieve high accuracies previous tasks, especially for NN. Strikingly, even cumulative continual learning fails to find a solution on unconfounded data for strict with NN, despite joint training on confounded data succeeding. While shuffled training also sees a small drop in accuracy, a big gap joint training remains. This shows that the majority of the difference between joint and cumulative training is not caused by the CL setting, but by the confounding setup, highlighting manifestation insidious continual confounding. Current Task Acc.
Old Task Ace.
Unconf.
Method
T₁
T2
T3
Ti @ @T3
T2@T;
Naive Sequential
100.0
100.0
100.0
49.1
48.9
50.0
EWC (Kirkpatrick et al., 2017)
100.0
99.9
100.0
49.1
48.9
50.0
FROMP (Pan al., 2020)
100.0
59.9
69.2
67.8
59.1
51.1
ER
(Chaudhry al., 2019)
100.0
97.7
96.4
54.6
59.6
51.0
NN
DER (Buzzega et al., 2020)
100.0
98.7
96.5
52.8
55.5
49.1
BGS (Lee et al., 2024)
100.0
99.8
99.6
50.1
49.1
49.5
GDUMB (Prabhu et al., 2020)
92.5
62.7
53.6
57.5
54.2
50,4
strict
PNNs (Rusu et al., 2022)
100.0
99.8
99.8
100.0
99.8
49.2
Shuffled
79.9
93.4
95.3
95.2
96.7
89.5
Cumulative
100.0
87.0
88.8
85.9
91.0
72.6
Joint
98.2
99.2
98.9
N/A
N/A
95.7
Naive Sequential
99.7
100.0
100.0
49.1
49.2
49.8
EWC (Kirkpatrick et al., 2017)
99.9
99.9
100.0
49.1
49.4
50.1
FROMP (Pan al., 2020)
99.8
83.4
74.7
85.1
76.3
56.2
NeSy
ER (Chaudhry al., 2019)
99.8
99.8
98.8
68.0
83.6
61.2
DER (Buzzega et al., 2020)
99.7
99.4
99.5
58.8
73.2
56.5
BGS (Lee et al., 2024)
99.8
99.9
99.9
51.8
56.3
51.3
GDUMB (Prabhu et al., 2020)
99.4
85.6
82.7
80.1
84.2
56.7
PNNs (Rusu et al., 2022)
99.8
100.0
100.0
99.8
100.0
49.9
Shuffled
87.8
92.5
89.2
88.2
92.6
70.1
Cumulative
99.8
92.5
88.8
88.6
92.3
69.9
Joint
88.4
92.6
89.8
N/A
N/A
70.8
4.3. CL Methods Can Get Insidiously Confounded
incompatibility of confounders (i.e., applying rule for a task lask-specific confounding feature on another task results in no better than random prediction). EWC also shows no significant difference the naive training setup. However, differences between our dataset variants start to show when considering the memory- based approaches.
We have seen that CL methods do aid in alleviating forget- ting, but do not help prevent confounding when the disjunc- tion of confounders learned over the ground truth rule on the joint dataset. On our strict ct dataset, we will now empirically corroborate that there in fact exists an even more concerning case: the case where models suffer from being confounded primarily through learning continually.
Whereas most of these were previously able to mitigate forgetting fully across all tasks in the disjoing case, we now observe that ER, DER, and BGS perform not much better than naive sequential training in the strict see- nario. Importantly, both NN and NeSy models addition- ally fail to make accurate predictions on the unconfounded dataset. FROMP and GDUMB appear to sacrifice current task performance for old task performance, but this does not improve performance on unconfounded data in our ex- periments compared to, for example, ER and gives strictly worse results than the cumulative (ER with infinite memory) approach. All standard CL methods thus not only fail to perform well on the unconfounded data but also do not get close successfully avoiding forgetting, with accuracies old tasks not even exceeding 60% except for FROMP, where
The experimental results on the strict dataset are shown in Table 2. First, consider the only approach that outputs different predictions depending on which data the task is coming from: PNNs. Here, the model has different clas- sification heads for each task, enabling to fully avoid forgetting and perform well on current and past tasks. How- ever, this leads to learning the respective tasks individually, failing to identify the underlying ground truth rule that ap- plies to all tasks. Considering the more standard continual learning setups, is again unsurprising that previous tasks appear to be forgotten in the naive sequential approach. The model likely learns and focuses on the new, task-specific confounder every stage. This may be explained by the
8

---

The Risk of Getting Confounded in a Continual World
current task performance is much lower. This shows how confounded datasets can be especially challenging for CL methods, even when those successfully avoid forgetting in easier problems. While the cumulative approach, benefiting from its infinite buffer size, performs much better in com- parison, contrasting it with joint and shuffled training on all tasks reveals results that may strike the reader as surprising.
prevention of catastrophic forgetting is insufficient avoid confounding in dis joint settings, where even joint train- ing fails to generalize. Most importantly, we successfully confirmed the latter to not be true for our strict dataset, where joint training succeeds but continual cumulative train- ing is veritably insufficient due to the manifestation of in- sidious continual confounding. This stands stark contrast to the typical assumption that knowledge accumulation is the desirable solution in continual learning.
For NeSy, cumulative training and joint training perform similarly well, but they both do not reach their earlier base- line performance on the unconfounded dataset (70.76% vs 80.33%) More interestingly, we find the cumulative train- ing with NN to seemingly focus more on the confounded tasks than on the ground truth, resulting in predictions more than 20% worse than in the joint training setup, despite train- ing on the exact same data. The latter in fact empirically highlights how both NN and NeSy perform significantly better than random guessing when trained jointly on the strict dataset. Despite each task being confounded, the NN achieves an astonishing 95.68% accuracy on the un- confounded data, compared to the achievable 96.96% when trained on the unconfounded data directly (recall the initial sanity check). While some of the differences between joint and cumulative training may not be solely explained by the presence of confounders, but also by the continual learning setup itself, we observe in our shuffled experiment that train- ing continually on the same data yields substantially better results when the tasks are not aligned with the confounding features. In other words, we now experimentally observe our prior formalized insidious continual confounding. That is, training in a continual setup performs significantly worse than the joint setup, challenging the conventional wisdom that continual learning revolves around the accu- mulation of data and knowledge and questioning the role of cumulative training as a performance upper-bound bound (Hayes et al., 2021; Mundt et al., 2023b).
Our work thus provides the means to benchmark and further explore confounding in CL, for example, by investigating causal relationships (Mundt et al., 2023a) and related innova- tive new perspectives on, e.g., causal rehearsal (Churamani et al., 2023) or bias mitigation of spurious correlations (Lee et al., 2023). We envision that a deeper understanding of continual confounding can lead to overall better model per- formance in continual settings. Other modalities, such as text or time series, are not currently in the focus of ConCon and are left for future research.
Impact Statement
Our paper challenges the conventional view of cumulative learning an upper bound when evaluating continual learn- ing methods, thereby showing that such confounders have detrimental effect on model performance even in what is conventionally considered a best-case scenario. The issue of confounders in machine learning represents a significant challenge. Making predictions based on such can not only lead to incorrect behavior but may also lead unfair or discriminatory decisions (Schröder et al., 2023). In our spe- cific case of ConCon, the dataset consists of synthetically generated geometric shapes and thus does not contain any personal or otherwise private information. At present, we do not envision any scenario in which it could be used with malicious intent. By means of ConCon, this paper aims to improve the evaluation of continual learning methods when confounding might be present, inspire new approaches to deal with these issues, and facilitate systematic investigation of such approaches in the future.
In conclusion, we can thus answer our question "do con- tinual learning methods suffer from insidious continual confounding?" affirmatively. Our evaluation shows that CL in the presence of confounding is significantly more chal- lenging than preventing forgetting. Both our models prevent forgetring in the dis joint case, where the solution of all confounded tasks could be combined in a relatively simple manner (disjunction). However, this proved insufficient to learn the ground truth rule required perform as well on the unconfounded data as joint training in the strict case.
A Acknowledgements
This work is supported by the Hessian Ministry of Higher Education, Research, Science and the Arts (HMWK: projects "The Third Wave of Al" and "The Adaptive Mind"). It further benefited from the Hessian research priority pro- gramme LOEWE within the project "WhiteBox" and the EU-funded "TANGO" project (BU Horizon 2023, GA No 57 57100431). The authors thank Laura Boyette for the pre- liminary results on this research. Furthermore, the authors would like to thank Felix Divo for proofreading our paper and thus improving the clarity of the final manuscript.
5. Conclusion
We have introduced the first continually confounded dataset, ConCon, and have formally described its implications for systematic analysis. ConCon consists of two variants that implement different types of confounding. In our evalua- tions, we have empirically demonstrated that even perfect
9

---

The Risk of Getting Confounded in a Continual World
References
continual learning survey: Defying forgetting in classifi- cation tasks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 4A(7):336633885 2022.
Akhtar, M., Benjelloun, O., Conforti, C.,, Gijsbers, P.,
Giner-Miguelez, J., Jain, N., Kuchnik, M., Lhoest, Q., Marcenac, P. Maskey, M., Mattson, P., Oala, L., Ruyssen, P. Shinde, R., Simperl, E., Thomas, G., Tykhonov, S., Vanschoren, J., van der Velde, J., Vogler, S., and Wu, C.-J. Croissant: A metadata format for ml-ready datasets. In Proceedings of the Eighth Workshop on Data Manage- ment for End-to-Bo End Machine Learning, DEEM '24, pp- 1-6. Association for Computing Machinery, 2024. ISBN 9798400706100.
DeGrave, A. J., Janizek, J. D., and Lec, S. Al for radio- graphic COVID-19 detection selects shortcuts over signal. Nature Machine Intelligence, 3(7):610- 616611169166606 2021.
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pat- tern Recognition (CVPR), pp. 248-255, 2009.
Friedrich, F., Stammer, W., Schramowski, P., and Kersting, K. A typology for exploring the mitigation of shorteut behaviour. Nature Machine Intelligence, 5(3):319 330, 2023.
Bicsialska, M., Bicsialska, K., and Costa-jussà, M. R. Con- tinual lifelong learning in natural language processing: A survey. In Proceedings of the 28th International Confer- ence on Computational Linguistics (COLING). Interna- tional Committee Computational Linguistics, 2020.
Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, W., Wallach, H., III, H. D., and Crawford, K. Datasheets for datasets. Communications of the ACM (CACM), 64(12): 86-92, 2021.
Bontempelli, A., Teso, S., Tentori, K., Ciunchiglia, F., and Passerini, A. Concept-level debugging of part-prototype networks. In International Conference on Learning Rep- resentations (ICLR), 2023.
Geirhos, R., Jacobsen, J.-H., Michaelis, C.,, Zemel, R., Bren- del, W., Bethge, M., and Wichmann, F. A. Shortcut learn- ing in deep neural networks. Nature Machine Intelligence, 2(1 1):665 1)6365667,, 2020.
Buzzega, P., Boschini, M., Porrello, A., Abati, D., and Calderara, S. Dark experience for general continual learning: a strong, simple baseline. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M. F., and Lin, H. (eds.), Advances in Neural Information Processing Sys- lems (NeurIPS), volume 33, pp. 15920-15930. Curran Associates, Inc., 2020.
Goodfellow, I. J., Mirza, M., Xiao, D., Courville, A., and Bengio, Y. An empirical investigation of catastrophic for- getting in gradient- based neural networks. arXiv preprint arXiv: 1ete:33122222 12.6211, 2015.
Hayes, T. L., Krishnan, G. P., Bazhenov, M., Siegelmann, H. T. Sejnowski, T. J., and Kanan, C. Replay in Deep Learning: Current Approaches and Missing Biological El- ements. Neural Computation, 33(1 (1:1)))2229999 2950, 2021.
Chaudhry, A., Rohrbach, M., Elhoseiny, M., Ajanthan, T., Dokania, P. K. Torr, P. H. S., and Ranzato, M. On tiny episodic memories in continual learning. arXiv preprint arXiv: /902. 10486 2019.
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016.
Chen, Z., Liu, B., Brachman, R., Stone, P., and Rossi, F. Lifelong Machine Learning. Morgan & Claypool Publish- ers, 2nd edition, 2018
Churamani, N., Cheong, J., Kalkan, S., and Gunes, H. To- wards causal replay for knowledge rehearsal in continual learning. In Proceedings of The First AAAI Bridge Pro- gram on Continual Causality, volume 208 of Proceed- ings of Machine Learning Research (PMLR), pp. 63-70. PMLR, 2023.
Hess, T., Mundt, M., Pliushch, I., and Ramesh, V. A Proce- dural World Generation Framework for Systematic Eval- uation of Continual Learning. Neural Information Pro- cessing Systems (NeurIPS) Datasets and Benchmarks Track, 2021.
Johnson, J., Hariharan, B., van der Maaten, L., Fei-Fei, L., Ziticck C. L., and Girshick, R. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1988-1997, 2017.
Codella, N., Rotemberg, V., Tschandl, P., Celebi, M. E., Dusza, S., Gutman, D., Helba, B., Kalloo, A., Liopyris, K., Marchetti, M., et al. Skin lesion analysis toward melanoma detection 2018: A challenge hosted by the international skin imaging collaboration (isic). arXiv 2019.
preprint
Kim, B., Kim, H., Kim, K., Kim, S., and Kim, J. Learning not to learn: Training deep neural networks with biased data. In 2019 JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9012-90200 2019.
De Lange, M., Aljundi, R., Masana, M., Parisot, S., Jia, X., Leonardis, A., Slabaugh, G., and Tuytelaars, A
10

---

The Risk of Getting Confounded in a Continual World
Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. arXiv preprint arXiv: / 4 441.69980, 2014.
In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Pro- cessing Systems (NeurIPS), volume 33, pp. 11525-11538. Curran Associates, Inc., 2020.
Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Desjardins, G., Rusu, A. A., Milan, K., Quan, J., Ra- A., and et al. Overcoming
malho, T., Ctrabka-BBrwwing catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences (PNAS), 114(13): 3521-3526, 2017.
Marconato, E., Bontempo, G., Ficarra, E., Calderara, S., Passerini, A., and Teso, S. Neuro-symbolic continual learning: Knowledge, reasoning shortcuts and concept rehearsal. In Proceedings of the 40th International Con- ference on Machine Learning (ICML), volume 202 of Proceedings of Machine Learning Research, pp. 23915- 23936. PMLR, 2023.
Koh, P. W., Sagawa, S., Marklund, H., Xie, S. M., Zhang, M., Balsubramani, A., Hu, W., Yasunaga, M., Phillips, R. L.,, Gao, I., Lee, T., David, E.,, Stavness, I., Guo, W., Earnshaw, B. A., Haque, I. S., Beery, S., Leskovec, J., Kundaje, A., Pierson, E.,, Levine, S., Finn, C., and Liang, P. WILDS: A benchmark of in-the-wild distribution shifts. In Proceedings of the 38th International Conference on Machine Learning (ICML), volume 139 of Proceedings of Machine Learning Research, pp. 5637-5664. PMLR,
Matan, O., Kiang. R., Stenard, C. E., Boser, B. E., Denker, J., Henderson, D., Hubbard, W., Jackel, L., and LeCun, Y. Handwritten Character Recognition Using Neural Net- work Architectures. USPS Advanced Technology Confer- ence, 105):10031101,, 1990.
McCloskey, M. and Cohen, N. J. Catastrophic interfer- ence in connectionist networks: The sequential learning problem. In Psychology of Learning and Motivation, volume 24, pp. 109-165. Academic Press, 1989.
2021.
Krizhevsky, A. and Hinton, G. Learning multiple layers of features from tiny images. Master's 's thesis, Department of Computer Science, University of Toronto, 2009.
Mundt, M., Lang, S., Delfosse, Q., and Kersting, K. CLEVA- compass: A continual learning evaluation assessment compass to promote research transparency and compara- bility. International Conference on Learning Representa- tions (ICLR), 2022.
Kudithipudi, D., Aguilar-Simon M., Babb, J.. Bazhenov, M., Blackiston, D., Bongard, J., Brna, A. P., Chakravarthi Raja, S., Cheney, N., Clune, J., et al. Biolog- ical underpinnings for lifelong learning machines. Nature Machine Intelligence, 4(3):196-210, 2022.
Mundt, M., Cooper, K. W., Dhami, D. S., Ribeiro, A., Smith, J. S., Bellot, A., and Hayes, T. Continual causality: A retrospective of the inaugural aaai-23 bridge program. In Proceedings of The First AAAI Bridge Program on Con- tinual Causality, volume 208, pp. 1-10. PMLR, 2023a.
Lapuschkin, S., Wäldchen, S., Binder, A., Montavon, G., Samek, W., and Müller, K.-.. Unmasking clever hans pre- dictors and assessing what machines really learn. Nature Communications, 10(1):1 1-8, 2019.
Mundt, M., Hong, Y., Pliushch, I., and Ramesh, V. A wholistic view of continual learning with deep neural networks: Forgotten lessons and the bridge to active and open world learning. Neural Networks, 160:306-336, 2023b.
Lee, D., Jung. S., and Moon, T. Issues for continual learning in the presence of dataset bias. In Proceedings The First AAAI Bridge Program on Continual Causality, volume 208 of Proceedings of Machine Learning Research, pp. 92-99. PMLR, 2023.
Pan, P., Swaroop, S., Immer, A., Eschenhagen, R., Turner, R., and Khan, M. E. E. Continual deep learning by func- tional regularisation of memorable past. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Sys- tems (NeurIPS), volume 33, pp. 1453-4464, Curran As- sociates, Inc., 2020.
Lec, D., Jung, S., and Moon, T. Continual lcarning in the presence of spurious correlations: Analyses and a simple baseline. In International Conference on Learning Representations (ICIR), 2024.
Lee, J., Lee. Y., Kim, J., Kosiorek, A. R., Choi, S., and Tch, Y. W. Set transformer: A framework for conmutatiooninvariinn neural networks. In Proceedings of the 36th International Conference on Machine Learning (ICML), volume 97 of Proceedings of Machine Learning Research, pp. 3744-3753. PMLR, 2019.
Parisi, G. I., Kemker, R., Part, J. L.., Kanan, C., and Wermter, S. Continual lifelong learning with neural networks: A review. Neural Networks, 1 11::54-71, 2019.
Peng, X., Bai, Q., Xia, X., Huang, Z., Saenko, K., and Wang, B. Moment matching for multi-source domain adaptation. In 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 1406-1415, 2019.
Locatello, F., Weissenborn, D., Unterthiner, T., Mahen- dran, A., Heigold, G., Uszkoreit, J., Dosovitskiy, A., and Kipf, T. Object-centric learning with slot attention.

---

The Risk of Getting Confounded in a Continual World
Prabhu, A., Torr, P., and Dokania, Gdumb: A simple approach that questions our progress in continual learn- ing. In The European Conference Computer Vision (ECCV), 2020.
van de Ven, G. M., Siegelmann, H. T., and Tolias, A. S. Brain-inspired replay for continual learning with artifi- cial neural networks. Nature Communications, 11:4069, 2020.
Rateliff, R.
Connectionist Models of Recognition Mem- ory: Constraints Imposed by Learning and Forgetting Functions. Psychological Review, 97(2):285-31) 1990.
Wu, S., Yuksekgonul, M., Zhang, L., and Zou, J. Discover and cure: Concept-aware mitigation of spurious correla- tion. In Proceedings of the 40th International Conference on Machine Learning (ICML), volume 202 of Proceed- ings of Machine Learning Research, pp. 37765-37786. PMLR, 2023.
Rieger, I.., Singh, C., Murdoch, W., and Yu, B. Interpre- tations are useful: penalizing explanations to align neu- ral networks with prior knowledge. In Proceedings of the 37th International Conference on Machine Learning (ICML), volume 119 of Proceedings of Machine Learning Research, pp. 8116-8126, PMLR, 2020.
Ye, W., Zheng, G., Cao, X., Ma, Y., and Zhang, A. Spurious correlations in machine learning: A survey, arXiv preprint 1rX1::240.12715, 2024.
Ross, A. S., Hughes, M. C., and Doshi-Velez, F. Right for the right reasons: Training differentiable models by con- straining their explanations. In Proceedings of the 26th International Joint Conference on Artificial Intelligence (IJCAI), pp. 2662-2670, 2017.
Rusu, A. A., Rabinowitz, N. C., Desjardins, G., Soyer, H., Kirkpatrick, J., Kavukcuoglu, K., Pascanu, R., and Had- sell, R. Progressive neural networks. arXiv preprint arXiv: nea.l6606047777 2022.
Sagawa, S., Koh, P. W., Hashimoto, B., and Liang, P. Distributionally robust neural networks. In International Conference on Learning Representations (ICLR), 2020.
Schramowski, P., Stammer, W., Teso, S., Brugger, A., Her- bert, F., Shao, X., Luigs, H., Mahlein, A., and Kersting, K. Making deep neural networks right for the right scientific reasons by interacting with their explanations. Nature Machine Intelligence, 276):476-486, 2020.
Schröder, M., Frauen, D., and Feuerriegel, Causal fair- under unobserved confounding: neural sensitivity framework. In International Conference Learning Representations (ICLR), 2023.
Stammer, W., Schramowski, and Kersting, K. Right for the right concept: Revising neuro-symbolic concepts interacting with their explanations. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3618-3628 2021.
Steinmann, D., Divo, F., Kraus, M., Wüst, A., Struppek, L., Friedrich, F., and Kersting, K. Navigating shortcuts, spurious correlations, and confounders: From origins via detection to mitigation. arXiv preprint 15 2024.
Tschandl, P., Rosendahl, C., and Kittler, H. The ham I 10000 dataset, a large collection of multi-source dermatoscopic images common pigmented skin lesions. Scientific data, 5:160161, 2018.
12

---

The Risk of Getting Confounded in a Continual World
Appendix
This appendix complements the main body with:
A: Additional dataset details and its rules for generating positive and negative samples.
B: Training information for the NN (ResNet-18) and NeSy (Neuro-Symbolic concept learner) models.
C: Training curves further corroborate that mitigation of forgetting is insufficient to tackle insidious continual confound- ing, together with individual test accuracies the positive and negative images for deeper empirical investigation of the fallacies.
Experimental results on the ConCon dataset using a generative model.
E: An experiment comparing joint and cumulative training a dataset comprised of several ImageNet classes.
F: A statement on limitations of our paper and the introduced ConCon dataset.
G: A datasheet for datasets (Gebru al., 2021) document e-g., composition, collection process, and other aspects.
A. ConCon and the Code Framework
Along with our paper, we also publish the specific ConCon dataset", as well as the data generation platform and the training code" to facilitate further research on continual confounding. We include a croissant (Akhtar et al., 2024) metadata record in our HitHub repository." ConCon uses the BSD 2-Clause "Simplified" License, and our code for generating the data and running the experiments released under the MIT license.
A.1. Dataset Details
Recall that ConCon is built on top of CLEVR (Johnson et al., 2017) and thus consists of images representing geometric objects with different properties. Each image is an RGB image of size 224x224 depicting four objects. our dataset, an object can be described by four properties, namely shape (cube, sphere, cylinder), material (metal rubber), size (small large), and color (gray, red, blue, green, brown, purple, cyan, yellow). Data is generated with respect the ground truth rule and confounders described in the main paper. Relationships between objects are not used in ConCon. We show several samples in Figure 4.
As we describe in detail in our main paper, the positive instances in the dataset fulfill the ground truth rule (g) and the negative instances do not (-g). In addition, it is possible that confounding feature must apply (c.) or must not apply (-c)), depending on the task (denoted by subscript t) and the dataset variant (dis joint or strict). The ground truth g is defined as sphere and small cube. The confounders, in the order of tasks, are oj blue, egi metal, and cy. large. We summarize the rules that should hold for respective tasks in Tables (dis joint) and 4 (strict).
Table 3. Rules disjoint: confounding feature may only appear in the positive images of its respective task. A common rule is thus applied for all negative images across all tasks.
Positive
Negative
Task 1
g / A 02 A -3
C3 - C1
A 02
Task 2
g A A 02 A
ng A
C3
Task 3
gA - A 0 Acg
g A e1 A C2 A C3
A.2. Code Framework
possible to generate further instances or make changes to the data generation process by using our code repository. The logic in the form of ground truth and confounders, which determines the data generation, can be in separate files.
'tttps: :ttpps:///eenee /t/////ddd..rg////////.// http:////...////eed...oooo /wwp/dzedddoo/o/////////oo///o 10630481
5281
"https:// ://i thub. com/m1 httm://mereeeechccceeeeao concon Thttps://gt thub. cwm/comm///..m./ wowmm//rreeerrchcooo concc/ccccoooooo/oooooooooocoooo /bt wwtwoo///////eooc http:///ooo/ooaeaessattttttooooo jaon
13

---

The Risk of Getting Confounded in Continual World
Task (Conf.)
Task 2 (Conf.)
Task 3 (Conf.)
Task (Conf.)
Task 2 (Conf.)
Task (Conf.)
Uncomf
Rules
@
Positive
Negative
Rules
X
Disjoint
Strict
Unconf
Sphere and small cube
Blue
Metal
Large
Figure Samples the ConCon Dataset. The first four samples dataset variant task.
14

---

The Risk of Getting Confounded in a Continual World
Table Rules strict: confounding feature must appear in the positive images but may appear in the negative images of its respective task. The confounding features may, but are not required to, appear in images of other tasks. Positive
Negative
Task 1
9 AA1
9 ci
Task 2
9 g
9 02
Task 3
9 A C3
-g Cg
Note that the intended behavior for our data generation consists of using simple confounders that are only required to be satisfied on single object. Here, our code could easily be adapted, for example, support the generation of images based on multi-object confounders by changing the selection of object properties in the respective function. In a similar spirit, both CLEVR and ConCon are generated with the Blender graphics simulation software, allowing for the inclusion of entirely new attributes and confounding features themselves. As such, extensions to, e.g., novel color choices are realizable with either trivial or modest amounts of specificaaiion/cange in code, or other already available properties (such as relationships) could easily be leveraged in future dataset creation and empirical investigation. Whereas these even more challenging setups may certainly be of interest in the long term, we note that our presently chosen configuration already highlights the catastrophic behavior of investigated methods and exposes insidious continual confounding, making solving our generated ConCon variant first desirable.
B. Experimental Details
In this section, we give detailed information on the continual learning methods and the training hyperparameters used for ConCon training of NN and NeSy models, as well as a brief description of technical details.
Methods. Continual learning methods aim alleviate forgetting when training and updating models continually. We here give additional information on the CL methods that we consider in our evaluation in the main paper: i) EWC (Kirkpatrick et al., 2017) is method that regularizes weights relevant to the previous task, ii) FROMP (Pan et al., 2020) stores examples relevant to the previous task and performs functional regularization while training on the current tasks, iii) ER (Chaudhry et al., 2019) maintains a fixed-size memory buffer of random samples from prior tasks and interleaves them while training on the current task, iv) DER (Buzzega et al., 2020) also maintains a memory buffer that stores past samples and predictions, and in addition, the loss function includes a cross-entropy loss for the current task and a regularization term using the stored logits for past tasks, v) BGS (Lee al., 2024) tackles the problem of spurious correlations continual learning by storing samples that are balanced across the class labels and spurious features, vi) GDUMB (Prabhu al., 2020) stores uniformly balanced past samples but is not conventional continual learner as the model is trained from scratch every new task, and vii) PNNs (Rusu et al., 2022) progressively grow the network by adding new subnetworks for each task, expanding the structure and parameters of previously learned networks.
NN Training. We use a ResNet-18 (He et al., 2016) model, a convolutional neural network- architecture with skip connections. Following standard deep learning practice, the model is trained with an Adam optimizer (Kingma & Ba, 2014) with mini-batch size of 64 and a learning rate of 0.001. The model was trained for 50 epochs for all methods but FROMP, where we trained the NN model for 30 epochs, as the performance became worse when training for 50.
NeSy Training. We use neuro-symbolic concept learner (NeSy) (Stammer et al., 2021) that is equipped with Slot attention model and a Set Transformer. The Slot-Attention module of NeSy is pre-trained on the original CLEVR dataset (Locatello et al., 2020) for the purpose of extracting accurate neuro-symbolic intermediate representations (e.g., disentangled slots for color and shape). We use the (perfectly accurate as a consequence of simulation) structured information from the embeddings of the attention module to then learn only the Set Transformer for the ConCon dataset. The latter trained using an Adam optimizer (Kingma& & Ba, 2014) with mini-batch size of 64 and learning rate of 0.001. The model was trained for 150 epochs. We apply transformations resize the images to 128 x 128 to fit the constraints set by prior works (Stammer et al., 2021).
Hy dlpperparameter Tuning. We did not excessively tune hyperparameters as all models achieved good solutions when expected in their baseline scenario. As explained in the main body and discussed in much more detail in the next section, observed failure cases are systematic and structural in nature a consequence of the continual learning set-up rather than hyperparameter values.
15

---

The Risk of Getting Confounded in a Continual World
Technical Details. We created the dataset using an AMD EPYC 7742 64-Core Processor CPU. All models were built using PyTorch and trained and evaluated NVIDIA Tesla V100-SXM3-32GG with 32 GB of RAM. Training on single task took no more than one hour for any experiment, with the exception of PNNs, which sometimes exceeded one hour.
C. Further Analysis of Results
We now go into some further detail on our evaluation described the main paper. First, we comment on standard deviations across experimental seeds, which have previously been omitted since they were negligible. We then showcase set of training curves to corroborate our main body's results. Finally, additional empirical details for our main evaluation are shown. Here, we further explain the models' behaviors by considering accuracies the decomposed sets of positive and negative images in addition to their average.
C.1. Standard Deviations Across Experimental Repetitions with Random Seeds
In the main paper, we observed methods in the continual setting performing well on the tasks they were trained on but completely failing on the unconfounded dataset in the disjoim nt case and also not reaching the joint accuracy in the strict case. To facilitate visual interpretation and focus on the main insights, we omitted reporting of standard deviations arising from experimental repetition with five random seeds in the main body. We now provide more details on why we consider these deviations to be negligible overall, but in particular with respect to the drawn conclusions.
In Table 5, we see that GDUMB and FROMP are the two methods that generally have a larger standard deviation. GDUMB selects small random buffer of samples train on, ignoring the rest of the dataset. Hence, the data which the model is trained differs among seeds. FROMP stores samples that are close to the decision boundary of previous tasks. In the stri ct case, if the ground truth is not learned and instead decision boundary based on task-specific confounders is learned, then the selected samples are not helpful for learning the ground truth rule but, at the same time, a rule based on all confounders will not be able to classify images from all tasks correctly. As such, selecting samples close to the decision boundary may lead to unstable (i.e., high standard deviation) results. Overall, the mean accuracies of GDUMB and FROMP are sufficiently low that the higher deviations do not affect the conclusions drawn.
Other than these two methods, standard deviations smaller than 1 percentage point of accuracy on all current tasks except for the naive training on the NN, where one outlier causes a higher deviation, the third task for DER with still small standard deviation of 2.69, and for the shuffled and cumulative settings, where it goes up to 3.85. For the latter two, the higher deviation is probably due to the model being trained on equal amounts of data from different tasks at the same time, therefore not being able to prioritize single tasks as much as other tasks, which almost reach perfect accuracy and thus have smaller deviations. We observe stronger deviations for the naive setup as well as EWC for old tasks. This fits with our observations of both approaches not successfully combating forgetting, resulting in partly random behavior on previous tasks. While there are some other slightly higher standard deviations of ER, DER, and BGS old tasks as well, the largest deviation of these here is 5.79, which is still not relevant to any claims in our paper as the mean performances here are all below 60 and deviations as implied by the standard deviation still results in bad accuracies.
Most importantly, standard deviations on the unconfounded datasets are fairly small throughout. With the exception of the shuffled and cumulative settings, we encounter the largest standard deviation for the NN with FROMP at around 2.77, with all others being smaller. This highlights the consistency of the failure to learn the ground truth rule. On the shuffled training setup, standard deviation reaches 4.63 (less than 1 for NeSy) and on the cumulative training setup. standard deviation on the third task moves up to 5.78 for NN (less than 1 for NeSy). Whereas this is, in fact, large and relevant deviation from the mean, it is still big difference from the more than 20 points difference that is observed by comparing the joint and the cumulative settings or the more than 15 points difference between shuffled and cumulative. This further emphasizes the NN's preference for learning confounders over the ground truth in continual settings.
C.2. Training Converges on Confounded Data with Worse Results on Unconfounded Data
Recall that, in the strict setting, we saw the models and the NN in particular achieve high accuracy on the unconfounded dataset in the joint setting, while training on the same data in a cumulative manner resulted in an accuracy more than 20 points worse. This result is surprising since both setups are trained on the exact same datasets. Hence, one might wonder whether the model actually converged to a good accuracy during training in the cumulative setup. As we can see in Figure 5, both models indeed converge at close to perfect accuracy. For example, the NN ends 100% train accuracy after task and
16

---

The Risk of Getting Confounded in a Continual World
Table 5. ConCon Evaluation with Standard Deviations. Test accuracies dis joint and strict for NN and NeSy, including standard deviations. Coloring is white until standard deviation and then takes darker shades of purple until 10. With exceptions, most standard deviations relatively small. Most importantly, performance the unconfounded data is consistently bad, Current Task Acc.
Old Task Ace.
Unconf.
Method
Ti
T2
T3
Ti @T3
T2
Naive Sequential
100.0:10.00
92.2: 15.58
100.010.00
50.0:10.03
13.6:188.02
1..1
EWC (Kirkpatrick al., 2017)
100.0+0.00
99.9
100.0+0.00
50.1+0.21
60.6+13.97
47.7+2.18
FROMP (Pan et al., 2020)
100.0=0.000
99.8+0.14
99.4+1.00
99.2+1153
97.9+2.41
50.2+0.17
ER (Chaudhry et al., 2019)
100.010.00
99.6210.36
100.010.03
100.0=0.00
99.6:10.22
50.110.10
NN
DER (Buzzega et al., 2020)
100.0:10.00
99.9=0.07
100.0=1..03
99.2:11.11
98.1=1147
50.241008
BGS (Lee ct al., 2024)
100.0:10.00
99.9=0005
100.0=10.03
95.1
96.2+2003
50.4:0.20
GDUMB (Prabhu et al., 2020)
04.1=18..30
62.0==9.70
19.5:
63.6 10.71
61.2-+8.93
48.8+t0.66
disjoint
PNNs (Rusu et al., 2022)
100.0+0.00
99.8+0005
100.0+0.000
100.0=0.00
99.8+00.05
47.9++2.36
Shuffled
100.0=00.03
99.9=0.03
100.0=0.033
100.0 0.03
9..8
5..0
Cumulative
100.0:10.00
100.0:10.03
100.0:1.03
100.0=
99.910.08
49.9110.00
Joint
100.0::0.00
99.9=10.04
100.0:10.03
N/A
N/A
50,0+1.03
Naive Sequential
99.5=0.18
99.9=0.12
100.0:10.03
50.0+10.00
58. +8.81
50.0+0.07
EWC (Kirkpatrick al., 2017)
99.6=0.28
100.0+0.003
99.9++0.05
50.2++0.5
67.2+99.12
50.2++0.18
FROMP (Pan et al., 2020)
99.6=0.31
98.9=1.29
98.4110.69
97.911.45
97.311.72
50.510.23
NeSy
ER (Chaudhry et al., 2019)
99.5=0.17
99.8:10.05
99.9110.04
99.2:10.19
99.3:10.23
50.7:0.18
DER (Buzzega et al., 2020)
99.6=0.22
99.7=1013
99.9-0008
98.6:t0.64
96.911112
50,9+0.05
BGS (Lee et al., 2024)
99.6=0.20
99.9++0.09
99. 99.9+
94.4++2..2
95.1
50.9+0.19
GDUMB (Prabhu et al., 2020)
99.4-0.12
97.8+0.89
98.5+0072
98.0++0.86
98.2++0.90
50.3++0.15
PNNs (Rusu et al., 2022)
99.5=0.25
100.0:10.00
100.010.00
99. 99.6:10.27
100.0=1..00
49.6:1.233
Shuffled
99.8 0.09
99.7=10099
99.911.055
99.7110.13
99.9110.33
50.810.07
Cumulative
99.8 0.07
99.9=
99.9:10.10
99.8:10.07
99.9:10.33
50.8
Joint
99.700.10
99.9
99.9+-0008
N/A
N/A
50.7++0.44
Naive Scquential
100.011.00
100.0:10.00
100.0:10.00
49.1=10.05
48.9+1007
50.0+0.07
EWC (Kirkpatrick al., 2017)
100.011.000
99.9=1.10
100.0+10.05
49.1+:0.05
48.9+10.08
50.0+1000
FROMP (Pan al., 2020)
100.011.000
89.9-1849
69. 69.2214.92
67.81110.71
59.1116.35
51.112.77
ER (Chaudhry et al., 2019)
100.011.00
97.7+10.5
96.4:10.59
14.6
59.612.71
51.0::1.24
NN
DER (Buzzega et al., 2020)
100.011.000
98.711092
96.5122.69
52.8:14.13
55.5
49.11:.26
BGS (Lee et al., 2024)
100.010.00
99.811011
99. 99.6:10.16
50. 10.1
19.1
49.3
GDUMB (Prabhu et al., 2020)
92.5-2.77
62.7-8834
53.6122.2
57.5+1..0
54.211..9
50.4+10.0
strict
PNNs (Rusu et al., 2022)
100.011.000
99.8:10.09
99.8:10.42
100.010.00
99.8:10.09
49.2111.3
Shuffled
3919
93.4.13.85
95.311:214
95.2:13.21
96.7:11.76
89.5144.3
Cumulative
100.0:10.00
87.0=11.57
88.8:13.65
85.9-12.65
91.0:12.89
72.61:5.78
Joint
98.2-1.11
99.2=10.17
98.9-10.30
N/A
N/A
95.7++2.04
Naive Sequential
99.7=0.20
100.0+10.03
100.0+10.00
49.1-10008
49.2-10.09
49.8
EWC (Kirkpatrick et al., 2017)
99.9=0.08
99.911007
100.0110.3
49.1110.08
49.4:10.49
50.1110.19
FROMP (Pan al., 2020)
99.8=0.12
83.419.01
74.7=7.18
85.117..5
16.36
56.211.17
NeSy
ER (Chaudhry al., 2019)
99.8=0.17
99.8=10.15
98.8110.11
1..0
83.6:10.71
61.210.82
DER (Buzzega et al., 2020)
99.7=0.23
99,4=0.15
99.5++0.24
58.8+11.0
73.2+33.1
56,5#1.05
BGS (Lee et al., 2024)
99.8=0.12
99.9-1.03
99.9-10.08
51.811101
56.3+1.55
51.3+10.99
GDUMB (Prabhu al., 2020)
99.4-0.24
8..6
1..741166
80. 80.1412.42
84.2112.3
$6.7+1109
PNNs (Rusu et al., 2022)
99.8=0.07
100.0.1..33
100.0+1.000
99.8:10.09
100.0-0..03
49.9110055
Shuffled
8768
92.5
89.2
88.2-10.35
92.6:10.79
70.110.51
Cumulative
9.09
92.5=10.11
88.8+10.51
88.6:10.65
92.3+10.65
69.9
Joint
8844
92.6-10.21
89.8-10.23
N/A
N/A
70,8:10,33
17

---

The Risk of Getting Confounded in a Continual World
NN
NeSy
100
100
Disjoint
90
90
Accuracy
Train
Accuracy
Train
80
80
Validation Cont. Avg.
Validation Conf. Avg.
70
Conf.
70
Conf. Unconf.
Unconf.
60
60
50
50
t1
t.2
t.
tl
t2
t3
100
100
90
90
Strict
Accuracy
Accuracy
80
80
70
70
Train
Train
Validation
Validation
60
Conf. Avg.
60
Conf. Avg.
Conf.
Conf.
50
Unconf.
50
Unconf.
t1
t2
t3
t1
t2
t3
Figure 5. Learning Curves for the Cumulative Training Setup. Recall that in the cumulative setting, the dataset being trained on grows over time, i.e., datasets from previous tasks are concatenated with the currently observed one. This figure compares these curves to the test accuracy on the unconfounded dataset (denoted by Unconf.), task-specific confounded datasets (denoted by Conf.), and average accuracy calculated over accumulated confounded tasks (denoted by Conf. Avg.) at end of training, Average accuracy after T tasks is given A7 = + Σx as Observe that in the strict case, performance on the unconfounded data does increase during training, but the final performance is still poor. We expect that this can be explained hy the further observation of the divergence between train and validation curves in the st ct case.
99.5% accuracy after task Training on the full accumulated data in task peaks 99.96% and ends 99.68% accuracy, Despite this, performance the unconfounded dataset, while increasing with the number of tasks, remains significantly worse than the train accuracy. In the main paper, we showed that the easiest rule that can be learned to solve all confounded tasks jointly is the ground truth rule. If that is true, why does the NN achieve a near-perfect train accuracy after training on all tasks but fail on the unconfounded data where the ground truth also applies? By inspecting the validation and test values of the confounded datasets, we find that, despite coming from the same distribution of images, the accuracies on these are considerably worse than for the train images. It turns out that the cumulative setting has resulted in both models overfitting far more severely to the train distribution. thus underperforms on both confounded data from the same distribution and unconfounded data due general failure to fully generalize to images. We expect that more investigation of this overfitting will further illuminate the problem of insidious continual confounding.
C.3. Models Mostly Fail on Positive Images in di joint Settings
Our main body's evaluation the dis joint dataset showed that performance the unconfounded dataset is not much better than random guessing, even when forgetting is successfully avoided. In this part of the appendix, we go into further detail on our evaluation to not only examine the average accuracies, but also the models' performance on the positive and negative images individually. Through this analysis, we can then pinpoint the failure modes even more precisely.
We employ a new visualization, extending the purely tabular results presented before. Here, we measure the accuracy on the confounded test dataset across all the tasks after training the model on each task. To investigate whether and how the model
18

---

The Risk of Getting Confounded in a Continual World
N
NoSy
Positive
Negative
Positive
Negative
Naive Sequential
100
100.00
0:19
0.00
43.39
100.00 .0 100.00 100.00
39.07
99.00
30.16
22.08
36.08
99.25
98,99
98.88
50,05
80
Troe (Cent)
89.84
99.89
17.68
97.12
64.37
84.43
84.56
4.96
The
0.99
99.95
0.72
93.87
99.84
99.97
18.48
60
40
9.03
27.23
100.00
92.03
100.00
00 100.00
100.00
0.03
16.29
99.95
94.91
99.97
100 00 100.00
5:15
20
1a5k
UnCond
Task
lapk
Uniont
1a5k
Back
UrCont
Tack
Taok
Task
UnCont
Pert
Test
ICar (Cant) (tee))
Teek (C3E) (Cont) ((are)
100
99.97
99.20
99.95
99.55
100.00 .0 100.00 100.00
0.48
99.
99.55
99.04
99.57
99.07
99.95
99 89
2,03
80
Shufiled
Tra (Cont))
200.00
99.81
100.00
99.60
100.00
100.00 99.95
0.35
99.60
99.63
99.87
99.52
99.79
99.84
99.92
1.87
40
99.68
99.97
Taet
100.00
90.95
0.35
90.95
99.60
99.92
1.92
UnGort
Uncont
UnCont
UnConf
Konb
(Condy
8oonts
6onnt;
(Cart)
(Cont)
(Centi
(Cont)
(aan
Kort)
(Cont]
(cont)
Comulative
Tran (oer IGinA
0:19
0.00
Test
100 00 100.00
$9.07
8.99
47.84
THe (COe) (5ot) (CSt))
81
57:47
.100
880
100,00
99.95
1.44
97.44
Tla (00M))
100.00
100.00 100.00
5:12
Than (00r))
99.73
99.89
0.77
96.75
Trah
99.92
99.84
99.92
13.57
60
(Corl)
99.95
09.89
100.00
99.60
(Cond)
100.00
100.00
29.95
0.27
99.60
99.87
99.97
99.60
THE
99.97
99.95
99.89
1.92
20
Tack [Cont]
Thok (Cont)
UnCont
TOOK (OWNT:
Thick (Cone)
UnCont
Tusk (CO0C)
Task
Dook (Cant) Test
UnCont
Thok (Core)
The (Cont)
UnConf
(Cont)
(Cont)
(Comt)
Figure 6. Accuracies for Positive and Negative Images on disjoint. Test accuracies on both the NN and NeSy for Naive Sequential, Shuffled, and Cumulative. We can that the prediction error is result of positive images being incorrectly classified negative. This is consistent with our hypothesis that the model mostly bases its predictions on the presence of the confounding features, none of which appear in any negative images of the confounded tasks. On the unconfounded data, the high performances on positive images after tasks 2 and 3 indicate that a disjunction of confounding fcatures is learned since this is satisfied in most positive and only few negative images. Further results on the other continual lcarning methods can be found in Figure 7. (Best viewed in color)
gets confounded, we also measure the accuracy on the unconfounded dataset. These results are illustrated via an inter-task confusion matrix in Figures and 7. Each row shows the results for a continual learning setup on both NN (left) and NeSy (right). Instead of the average accuracies already discussed in the main paper, we here plot the results on the positive and negative sets of images. We additionally color the fields corresponding to their accuracy, going from incorrect predictions (0%; red) over random predictions (50%; blue) to correct predictions (100%, white).
First, consider the shuffled approach. Here, tasks are not aligned with confounders, which is why performance on all tasks is very similar. Our hypothesis of the model learning a disjunction of confounders fits the results, as all tasks are predicted correctly but most negative images of the unconfounded datasets are incorrectly classified as positive. In the following, we go into further detail on the continual learning setups that are affected by confounders to further explain the models' behavior there.
The results of GDUMB NN set themselves apart from the other methods. We will quickly cover the other methods and then return to GDUMB. With the exception of GDUMB, predictions the negative set across all tasks and methods are mostly or even fully correct, and the classification errors are almost exclusively concentrated on the positive set. In other words, the models seem to predict the negative class too often, hence failing on the positive set. This can be explained by the model predicting the positive class whenever the confounding feature is present (it is never present on negative images of confounded tasks). The model also performs well on all previous tasks in FROMP, ER, DER, BGS, PNNs, and the cumulative setting, indicating that some combination of confounders is learned.
We can get some more insights into the model behavior by relating it to the performance on the unconfounded dataset. For all methods, performance on both positive and negative images is around 50% after the first task. Here, the model appears to make predictions based on the first confounding feature, blue, as in the unconfounded dataset, this roughly matches the proportion of images with and without a blue object. However, almost all images are predicted to be positive for the two following tasks. Our results on the memory- based methods FROMP, ER, DER, BGS, GDUMB, as well as PNNS and the cumulative setup, which show good performance on all current and previously seen tasks, lead us to hypothesize that the
19

---

The Risk of Getting Confounded in a Continual World
NN
NeSy
Positive
Negative
Positive
Negative
(Sant) (CCM
THi (CW]] (COM)) (6
1100
100,00
1.79
0.00
44.45
100.00 00 100.00 100.00
58.29
99:71
25:47
22.03
55 55.04
99.47
99.20
99.20
52.
80
EWC
Tran Boots
74.61
99.79
2.40
96.83
Tea
99.95
99.95
99.92
5.55
Thah
3.79
99.97
1.97
94.24
Tohh
99.95
99.95
99.95
16.32
40
(Corl
21.20
100.00
(CSmc)
100 00 100.00
Tark (Cwi
0.51
99.95
100.
99.95
4.99
(Cont)
(Crent
(0ont3
(Cent)
(Cont) (Cont)
(Cent:
(Cont)
(Cont)
10ont3
(Cont]
(Cont)
FROMP 7Oa (Grr ICor
0.03
0:03
43.09
(0001 ICed
100.00
.0 100.00 100.00
59.23
Tack (owd
17.12
49.81
Tra (OHR)) (EWI
THE
98.99
56.00
100
200.00
99.63
4.75
97.25
TEAM
99.89
99.95
99.92
5:25
99. 04
99.84 .5.
17.95
97.47
98.
38.91
98. 98.00
98.88 61
9.84
60
(Cont)
99.95
96.77
99.95
99.63
Thak
(Can))
98.43
99.04
98.83
0.72
98.19
98.08
99.89
99.60
Thok
(Coi)
97.68
96.62
96.91
1.44
20
Teck
Tesk
Texk:3 (Cant)
UnCont
Teck
Tank
Tedc (Cont) 7est
UnCont
Took (Cand)
Taxk
UnCont
Taak
Tack
Teac (Can))
UnCond
(Gont)
[Cond]
(Cant)
(Cone)
(Cont)
(Cant) Fest
(Gont)
(Cont)
Fext
Toa (Gane) (Cort)
(ESH (COR))
Tosk (Cont)
THA (CHH)) (Cor))
-100
200.00
14.24
14.00
47.25
100.00
100.00 100.00
$5.47
99.81
31.12
23.97
56.77
99.23
98.96
98.83
50.96
800
BR
100.00
B.56
Tohk
60
200.00
99.15
99.97
99.55
100.00
100.00
100.00
0. 0.69
98:40
98.72
99.95
99.42
99.97
99
99.9 92
2.08
20
Taak
Task 2
Taxe
UnCont
Taut
Task 2
Taek
Un/ant
Task
Taxt 2
Taak
Thak
Task
Trae
UnCont
Tex
Test
Test
DER Trai (Co00) (Contt
ICOEI (OmL)
TON TMME (ESI6 (Con))
200.00
2.16
0.00
04.72
100.00
100.00 100.00
58.67
99.81
12.61
23.01
48.48
99.39
99.17
99.15
57.17
800
99.92
99.84
3.25
97.49
100.00
00 100.00 100.00
5.20
Than
98.37
99.81
7.25
96.67
99.76
99.68
99.71
13.97
400
(Dme))
99.47
100.00
99.97
100.00
94.08
99.33
99.73
89
2.37
Taak-2
UnCont
Task (Cant)
UnCont
Task
UnCont
UnCond
[Comt
(Camt)
(Coml.)
(Cevi.1
[Com]]
[Cort)
(CoL))
(CO0)) (CRL) (Can))
IOald (Co1))
TraM (ORE) (5SeL) (Con))
-100
100.00
0.16
0.00
43:17
200.00
100.00 100.00
59.29
99.79
12.97
20.77
48.35
99.17
99.23
57.17
BGS
TrA
Korr (Cil
07.52
09.89
4.27
07.55
09.97
100.00 100.00
4.85
Then
06.21
99.87
2.05
96.53
99.84
99.84
99.87
13.57
60
40
90.19
92.45
100.00 00
99.52
100.00
00 100.00
99.97
1.33
88.
90.35
99.95
99.09
99.95
99 99.95
99.99 95
2.72
UhCond
[Condi
(0ont) Test
(Cunt)
(Cord [Con)) Ferr
(ConE)
(Cuml3
[Con()
(Cor()
[C
(oon))
Test
00.59
67.28
40.85
63.47
39.77
39.55
39.3 33
43.25
99.8
2:19
44.53
THE Toc
(Con)) (00E)
99.60
99.47
93.33
59.31
-100
GDUMB
1790
(Centi
28.96
24.40
0.05
25.15
(2ee1)
99.35
99.63
99.63
83.92
97.63
4.00
94.53
98.24
97.97
98.37
10.77
60
(Cand)
27.28
22.45
39.04
81.89
99.89
99.87
99.97
15.73
98.64
99.20
99.84
99.52
Tare
97.36
97.15
97.15
1.09
20
Task
2
1a5k33
UnCort
Tock (cont)
Tosk 2
Took (CorT)
Uncont
Tock
Task
Task
UnCont
Task3
Baok
Tisk
UnCont
[Cont]
(Con
(CanC) Tax!
Tax?
(CSH (CHRE) (Car))
Korl
1oee (CHH) (Cent) (0HE)
100.00 00
1.07
0:13
45,57
100.00 00.00 100.00 100.00
58.13
98. 96
4.45
4.13
48.6
99.95
99 95
99 97
54.05
800
Ns
(Cont)
100.00
99.73
0.72
95.47
200.00
00.00 99.95
100.00
100.00
0.00
32.37
Paah
99.95
100.00
99.97
79.25
40
100.00
99.73
100.00
91.68
100.00
99.95
100.00
99.20
100
100.00
93.23
THE
99.95
100.
100.00
5.89
20
Thok
Tysk 2
Thok
UnCont
Dosk
Thok
Tosc
UnCont
Tysk
Tark
Tack
UnCont
Thck
Tasi
Tisk
UNCont
Figure 7. Accuracies for Positive and Negative Images on disjoint. Test accuracies on both the NN and NeSy for different continual learning methods. As for the baseline methods, we see that the model failures on confounded tasks are due to incorrectly predicted positive images, while too many unconfounded images are incorrectly predicted as belonging to the positive class. (Best viewed in color)
20

---

The Risk of Getting Confounded in a Continual World
disjunction of all confounding features is learned. In any image of the unconfounded dataset, it is very likely that least one confounding feature is present, hence resulting in predictions for the positive class. For the naive setup and EWC, the higher accuracies on the positive images can also partly be explained by the data generation process: for both metal and large, it is more likely for an unconfounded image to contain the respective features than not to contain them.
Now we return to GDUMB, which, training on only the buffer, learns something more random than the other methods (also see Table 5), resulting in more unpredictable results after training on task 1. After task 2, GDUMB follows similar patterns to the other methods but with worse results. GDUMB fails to learn the ground-trun truth rule and also does not learn a solution that makes correct predictions on all confounded tasks (such as the disjunction).
With respect to NeSy, the same patterns can be observed. Performance on task I after training on task 2 is worse than for the NN, possibly indicating a stronger focus on the confounder of the current task. We believe that the marginally better accuracies on some not-yet-seen tasks (forward transfer) can partially be explained by the model learning some aspect of the ground truth rule due to its neurosymbolic representation at the core.
Overall, results on the positive and negative images show that the negative set of confounded tasks can be consistently predicted correctly. This entirely plausible and perhaps even expected, as the negative set used in all tasks follows shared rules for generation. In turn, errors introduced by incorrect predictions on positive images. The accuracies the unconfounded dataset conform with our hypothesis of the models learning a disjunction of confounders, rule that is mostly satisfied on positive unconfounded images.
C.4. Models Base Predictions on Task-Specific Confounders in strict Settings
NN
NoSy
Positive
Negative
Positive
Negative
Naive Sequentiol
-100
100.00
41.73
42.00 00
43.07
1000000
60.67
59.33
99.84
42.72
43.89
99.49
57.52
60.49
50.08
Traes
(Cont]
33.44
100.00
93.33
95.15
9.12
100.00
6.00
7.65
93.57
99.97
03.49
94.69
13.07
100.00
20.48
12.19
60
40
94.40
93.17
100.00
94.05
4.67
100.00
5.17
94.24
93.39
100.00
94.64
Tyst
4.05
4.93
000.00
5.04
Task3
Uniont
Unioon
UhCord
(err
[Condy
(00nE)
(cont)
(Core)
(Cont)
(Cont]
(Conf)
(Can()
(Cor()
(Cont)
(oon))
(Cant) (C0r8)
THE Toot (Con)) ((oe)
-100
93.04
68.24
80.59
01.20
66.60
66.67
87.68
93.36
55.31
95. $5.89
90.85
04.59
86.72
79.63
94.37
93.92 92
55,65
Shaffldd
(Cont)
96.40
93.31
87.79
89:12
7lah
83.95
93.49
95.87
75:15
96.08
90.51
83.81
85.81
80.32
93.36
55.92
60
(Cine)
07.23
96.21
92.11
94.03
[Cael
93.15
97.23
98.53
85.00 65.01
96.00
96.05
90.61
84.51
85.23
80.40
94.61
93.5 9.899
55.01
20
Tock
Tosk
UnCont
Tock
Thok 2
Topk
Uncont
Took
Tosk 2
Tock3
UnCont
Tack
Tosk
Task
UnCont
[Cont]
CCont'
Taxt
Taut
Text
Taxe
Trae (Conts (Can))
(Cant: (cae)
Tare (CHE)) (CRr)) (Cad)
Cumulative
100.00
41.73
42.00
43.07
1.0000 .0.
56.85
60.67
59.33
99.63
41.49
41.73
42.88
99.97
58.00
61.31
60.61
80
79.31
$9.95
66.56
Prah
82.00
94.69
46.23 29
58.48
97.47
87.60
69.33
77.36
97.31
41
57.44
40
ICeett
94.59
89.79
83.09
83.81
ICoed
77.20
92.27
94.45
61.30
96.16
90.77
83.65
85.89
TheE
81.01
93.89
93.87
54.00
20
Tysk2
UnCont
TIKE
Thok >
Uncont
Tosk1
Task
UnCont
Thok
Taskk
UNCann
(Canf)
(Cent)
(Conf)
Figure& 8. Accuracies for Positive and Negative Images on strict. Test accuracies both the NN and NeSy for Naive Sequential, Shuffled, and Cumulative. For naive sequential, performance on positive and negative images is relatively balanced after task 1. For tasks 2 and the respective confounding features more likely to be present, so the model tends to classify images as "positive" in the naive sequential approach. This is due to the first confounding feature (blue) having more alternative values (there are eight colors) than the confounding features metal and large, which both have only one alternative property value. In the shuffled setup for NN, the accuracies are similar across tasks, with only the unconfounded accuracy improving with additional data. Cumulative also improves with additional data, but stays behind shuffled in terms of accuracy. (Best viewed in color)
For the strict dataset, our evaluation in the main paper demonstrated the problematic manifestation of insidious continual confounding, i.e. a model's performance when trained on all confounded tasks jointly is considerably higher than when trained on the same data in cumulative, continual setup. In the latter, largely fails.
21

---

The Risk of Getting Confounded in a Continual World
NN
NoSy
Positive
Negative
Positive
Negative
100.00
41.73
42.00
43.07
100.00 .00
$6,83
60.67
59.33
99.79
42.37
42.45
43.63
Stte Took
(C30t (Com)
99.92
57.60
60.00 09.69
60.29
80
EWC Tra (Cend
93.33
92.68
10.37
6.56
Tre
NCWL
93.71
95.17
99.89
16.83
11.07
10rr
94.51
93.17
100.00 000.00
94.91
3.65
6.67
99.9 92
5.09
IorrO
94.19
93.41
99.97
94.72
4.11
5:31
100.00
5.39
20
Tack
lock 2
1ask.
UnCard
1aok
Unionm
Task
lock
Coni
lock
laok
Task
UnCornf
Pest
FROMP Tras (Cont)
100.00 00.00
41.73
42 00
13.02
(Cint) (0ot))
100.00 .0
56.75
60 67
59.9
(Cunt) 10r10
99.77 09.79
42.45
42.16
43.57
Teee
(Cont) (tae)
99.07
57 60
45
59
-100
-80
89.07
60.93
23.04
58.56
71.07
58.91
46.27
43:52
Twe
95.55
73:09
65.33
67.49
79.84
93.76
60.85
51.60
40
ICind
Text
53.52
UnCont
Uncont
UrCont
UnCont
(00mt)
60ont:
(Cont)
(ConT)
(Cent)
(Cont)
(Cant)
(Cont)
(0ont)
(GHO
100
ER
TONN (COME)
89.84
95.33
81.12
83.57
23.13
99.97
16.72
27.49
93:49
99.68
90.43
93.55
41.89
99.92
32:21
31.63
60
(Coedl
86.53
84.27
92.80
81.84
[Coel
22.75
34.96
99.97
20:13
94.05
93.41
98.56
93.39
THE (C3051
42.03
73.87
99.04
29.07
20
Task
UnCort
Tack
Uncont
Dokk3 (Gan))
UnCoet
Thckk)
Thek (Cont)
LNCond
(Con))
[Cond]
(Cont) Text
(Cont)
(Care)
(Cont) Pest
(Cont)
(Conf)
(Gonr)
(Conn]
Test
[COO)]
100.00
41.71
42.00
43.07
(Den) [Core]
100.00 .0
56.88
60. 69
59.33
99. 99.68
44.21
43.39
44.80
THE (CoM)) [r361
99.65
57.92
59.76
60.27
1000
DER
TOV Eannt)
97.84
87.57
90.05
700
99.09
11:57
15.85
93.76
99.31
90.24
93.36
33.81
32.67
60
(Chel)
87.65
86.25
93.39
83.65
The
(CHi)
17.97
24.80
99.68
34.61
(CWi)
03.79
93.09
99.25
93.71
23.81
53.36
99.79
10.33
20
Thek
Tysk 2
The
UnCont
Thak
Tusk
UnCont
Took
Thet 3
Tuak3
UnCont
Thek3
Thet (Cont)
UnCont
(Cant)
Text
Post
Teo!
(ort)) (Caed)
(Cmml) (oor)) (CHe)
(ont) IOWK
TONN TME (onee) (Orel
100.00
41.73
42.00
43.07
100.00 .0.0
56.88
60.67
59.33
09.68
43.71
42.75
44.36
99.87
57.92
59.95
60.19
600
BGS
Trai
92.53
99.52
91.47
93.49
Trah
13:25
100.00
8.16
10.05
That
93.71
99.84
92.05
94.59
16.
99.97
69
15.
40
90.64
88.93
99.17
89.71
9.49
9:23
99.97
9.39
94.19
93.33
39.89
94.37
9:44
19.36
0.21
20
2
UnCont
Task
UnCont
Tuac
Task
UrCont
Tamk3
UnCone
(Cant)
(Car1)
(Cond)
[Cone)
(Cant)
Yest
(Cert)
(COe) (Citl))
IC-ld (Cont)
Trak (C-e0) (Cor)
-100
85,23
29.63
27.12
29.04
99.84
79.17
71. 11.60
75.23
99.28
41.95
41.17
42.29
99.55
60.72
62.99
GDUMB
TOa
45.71
27.33
20.27
20.75
Taak
Q4.
04.08
98.16
87.57
85.65
Tran
95.
05.33
76.88
56.77
61.92
78.45
94.32
55.71
59.89
40
(Cont)
8.61
(Cant)
97.68
99.60
56
94.27
tukk
(b01))
70,08
63.63
Jut (Cot))
95.4
49.79
(Conk)
[Cont]
Yest
(Can1)
(Cant)
[ConE3
(Cont)
(Cont.)
(Cant)
Test
[Can(]
(Corl)
(Com))
(Cand)
(Cori) (Cott)
(csed (Cant)
(Cont)
Trak (C (Cot))
-100
100.00
41.73
42.00
43.07
200.00 00.00
56.05
60.67
59.33
41.73
42.88
57.71
59.63
PNNs
100.00
99.57
92.32
Trae
100.00
99.97
6.77
9.47
Teaes (Cor)
94.32
95.39
100.00
99.97
5:73
6.13
60
(GorE)
100.00
99.37
99.35
93.04
100.00 .0.00
99.97
99.97
5.36
99.0
100 00
100.00
94.88
(CON)
99.97
99.97
100.00
4,90
20
Task-1 (COTE)
Talk
Task (C0nE)
UnCon/
Taak (COnT)
Task
Task (ConC)
UnCont
Taak (CANE)
Task
Takk (Can()
Unioor
Task-
Taak
Task (Con))
UnCon/
[COnC]
[cor(]
(ConT)
(Con()
(ConC)
Teg
Test
Test
Figure 9. Accuracies for Positive and Negative Images on strict. Test accuracies both the NN and NeSy for different continual learning methods. Similarly to naive sequential, performance positive and negative images is mostly balanced after task 1. In the second and third tasks, the models tend to classify images as "positive" more often, as confounding features are more likely to be present than not. (Best viewed in color)
22

---

The Risk of Getting Confounded in a Continual World
Figures 8 and 9 show the individually decomposed results for the strict dataset variant, analogous to the analysis of nt: the previous subsection. As for the di s joi nt. case, accuracies in the shuffled case are mostly unchanged throughout tasks because the tasks are not aligned with the confounding features. We can, however, see that additional data helps the model learn the ground truth rule, as accuracies on both positive and negative images go up in tasks 2 and 3. Considering the continual learning setups, we again see that the models appear to have learned the first confounder after the first task. However, due to this confounding feature being allowed to be present in other tasks, this now leads to more balanced predictions on future tasks. After training on tasks and 3, most predictions fail on the negative instead of the positive sets of images. This is again consistent with our hypothesis of the model learning the task-specific confounder. To be precise, both the confounders of tasks 2 and describe a property, which, in our dataset, only has two different possible values (metal or rubber; small or large). Since these properties are determined randomly if not specified, there is only small probability of that feature not being present. 8 Therefore, applying the rule based on the task- confounding feature mostly classifies any such image as positive, no matter the class, thus adding up to around 50% on average.
While the results for the CL methods look mostly similar, there are three methods with results different from the others. FROMP has worse results on positive but better results on negative images. Here, old task accuracy is improved at the cost of current task performance. While this indicates that something different from simply the current task confounder is learned, the unconfounded accuracy remains to be not much higher than 50%. The second outlier method, GDUMB, performs better on negative images but much worse on positíve ones. Since, just like with the cumulative training, the ratio of samples from each class is the same, it does not prioritize the current task confounder. However, it still fails to make correct predictions on many images of the confounded tasks and also gives bad predictions on the unconfounded dataset. Lastly, PNNs mostly fall in line with the other methods, but the task information allows them to correctly predict positive and negative images of all previous tasks. On the other hand, this still does not help with making correct predictions on the unconfounded data, where PNNs do not perform better than the other methods.
NeSy can be observed to perform less badly negative images. It is possible that it picks up on some parts of the ground truth rule more successfully than the NN. Nevertheless, as we have seen in the main paper, the overall performance on the unconfounded dataset still has a large margin of error.
In summary, the models mostly focus on the task-specific confounders in the strict case. Whether such a model performs better on one class or the other is then determined by the distribution of test images. In this strict variant of ConCon, confounding features are likely to be present on any image of other tasks and the unconfounded dataset, thus resulting in better predictions on positive and worse predictions on negative images.
D. Experiment Using Generative Models
Table 6. Test Accuracies for Generative Replay on the ConCon Dataset. While accuracies the current tasks are high, we observe catastrophic forgetting, resulting in low accuracies on old tasks. Accuracy on the unconfounded dataset is around 50% showing that the model does not successfully learn the ground truth rule.
Ti
T2
Ta
TTQT; 53.2+t6.30
T2QT3
UnConf 47.73=1.45
disjoint 100 strict 99.96=0008
88.78 14.29
99.28+1.022 99.58+00.4
59.33++1..02
98.54+0.51
48.84440.48
48.61
49.26-0088
In addition to the NN and NeSy models, we here want to investigate whether another type of model based on generative models is more successful in overcoming continual confounding. To this end, we use a Variational Autoencoder (VAB) architecture based on ResNet-18 following van de Ven et al. (2020). The VAE framework allows the model to generate synthetic samples of the ConCon dataset from previous tasks by sampling from the Icarned latent space, providing a mechanism for replay without storing actual historical data. This experiment uses the same hyperparameters as the experiment in the main body (see Appendix B). Table shows the experimental results on both the dis joint and strict variants of our ConCon dataset. The model achieves high accuracies the current tasks across both dis joint and stri ct settings, but fails at correctly predicting images from previous tasks, showcasing catastrophic forgetting. More critically (but, given the bad performance on previous tasks, also unsurprisingly), the model fails completely on the
'aac of the four objects in unconfounded image has chance of containing the material metal. The probability of such an image containing any metal object is, therefore, quite low. The same idea applies analogously large (on positive images, the small sphere part of the ground truth rule also affects the distribution of sizes).
23

---

The Risk of Getting Confounded in a Continual World
unconfounded dataset, indicating, like other continual learning methods, that it learns the task-specific confounders and not the ground truth rule.
E. Expanded ImageNet Experiments
Following up on our motivational experiment using ImageNet classes at the start of the main body, we now conduct a larger experiment using ImageNet classes to further investigate whether we again observe a gap between joint and cumulative training. To this end, we consider two categories: four-legged animals and non-four-legged animals. We define three tasks where each task contains data from eight ImageNet classes, half of which are four-legged animals. Additionally, in each task, this group of four ImageNet classes shares a similarity with respect to color. For example, the two resulting classes in task I contain either four-legged animals where the predominant color is white (e.g., arctic fox, polar bear), while the second class contains four-legged animals where the predominant color is blue ve.g..jelllliie
blue jay). Therefore, when training the model in a continual manner (cumulative training), it is likely that the model will start by learning a simpler rule in task one, by, for example, making predictions mostly based on color, the "four-legged" feature, or even other features that we did not consider. Following our insights from the experiments of the main body, we hypothesize that such behavior will lead the model having more trouble with correctly making predictions on all data compared to the joint setting, where it processes all the data at the same time and, hence, would not favor rules as much that are only consistent with a subset of all classes.
The exact description of the dataset and experimental results is shown in Table 7. We use a ResNet- 18 model and average results across 5 random seeds (other hyperparameters are the same as for most other experiments, see Appendix B). The table shows the accuracies after the model has been trained on all tasks. From joint training to cumulative training, we observe an average decrease in accuracy of 5.58 percentage points, a substantial degradation in performance. While the setup is slightly different from our continual confounding experiments in the main paper, does showcase that the continual learning setting tends to result in a less accurate model than the joint setup.
Table 7. Cumulative and Joint Training on ImageNet Data. At the top, we show what ImageNet classes the dataset consists of. Each task includes a pair of classes defined by "4-leggedness" and dominant color, resulting in six classes overall. Class examples are from ImageNet. At the bottom, we report cumulative and joint training accuracies after training on all tasks. On average, joint training has an accuracy of 5.58 points higher than cumulative training.
Task
Class
Description
ImageNet Classes
Co
Four-legged, white color Not four-legged, blue color
arctic fox, polar bear, white wolf, samoyed dog jellyfish, blue jay, blue shark, tench
Ti
C1
C2
Four-legged, green color Not four-legged, white color
cheetah, deer, lion, leopard snowmobile, sailboat, snowplough, rattlesnake
T3
C3
C4
Four-legged, blue color Not four-legged, other colors
hippo, crocodile, water buffalo, beaver eagle, spider, lifeboat, mushroom
T3
Cs
Accuracy After Training on All Tasks
Training Setting
Ti
T2
T3 84.40.00.86 86.25+1..15
Cumulative
76.30:1.:5 83.45-1.466
78.50 86.25+1.22
Joint
F. Limitations
ConCon is the first dataset designed to investigate how ML and CL methods behave in the case of continual confounding. Nevertheless, there are certain possible research directions on continual confounding variations where an extension of the current ConCon could be beneficial. While our code framework for generating datasets can be changed to allow for different kinds of continual confounding, the current implementation only aims to generate problems where confounders only appear on a single object. Multi-object confounders are not implemented as of now. Other types of confounders, such as relations between objects or positioning (e.g., there is a sphere left to a red object, there is a cube on the far left), are also not considered yet. Generally, any -LEVR-baser dataset will have limitations due to its synthetic nature. Investigating continually confounded real-world datasets thoroughly is thus left for future work.
24

---

The Risk of Getting Confounded in a Continual World
G. ConCon Datasheet
Motivation
For what purpose was the dataset created? (e.g.was there specific task in mind? Was there specific gap that needed to be filled? Please provide a description.)
The ConCon dataset was created to evaluate confounding in a continual learning setting, It consists of a learning task which can be solved by applying ground truth rule. There exists one unconfounded dataset where no confounder is present and the ground truth determines the class assignment. Additionally, there are two dataset variants, dis joint and strictt with three confounded tasks each. Each task consists of its own dataset. trained data from single task at the same time, going over all tasks in sequential order. For disjoint, the confounding features are only present in their respective tasks. For strict, the confounding features are only informative in their respective tasks, i.e. they may only appear in one class of the confounded task but in any class of the other tasks.
continual learning setting, a model is only
What (other) tasks could the dataset be used for? Are there obvious tasks for which it should not be used?
The primary purpose of ConCon is the investigation of continual confounding. It could also be used for other tasks, such as neuro-symbolic continual learning.
Who created this dataset (e.g., which team, research group) and on behalf of which entity ? (e.g., company, institution, organization)
The creators of the dataset are: Florian Peter Busch Roshni Kamath 1.2 Rupert Mitchell WWolfang Stammer 1.2 Kristian Kersting 1,2,344 and Martin Mundt The associated affiliations are: *Department of Computer Science, TU Darmstadt, Darmstadt, Germany, Hessian Center for Al (hessian.Al) Darmstadt, Germany, *German Research Center for Artificial Intelligence (DFKI), Darmstadt, Germany and 'Centre for Cognitive Science, TU Darmstadt, Darmstadt, Germany, University of Bremen, Germany.
Who funded the creation of the dataset? If there is an associated grant, please provide the name of the grantor and the grant name and number.
No funding was received for the creation of the dataset.
Any other comments?
No further comments.
Composition
What do the instances that comprise the dataset represent (e.g., documents, photos, people, countries)? Are there multiple types of instances (e.g., movies, users, and ratings; people and interactions between them; nodes and edges)? Please provide a description.
Each instance is an image representing four geometric objects with certain properties. An object can have any of three shapes (cube, sphere, cylinder), be either small or large, have either metal or rubber material and be one of eight colors (gray, red, blue, green, brown, purple, cyan, yellow).
How many instances are there in total (of each type, if appropriate)?
Each task consists of 9,000 instances. There is one unconfounded dataset and both the dis jistt and strict variants consist of three tasks each resulting in a total of 63,000 instances.
Does the dataset contain all possible instances or is it sample (not necessarily random) of instances from a larger set? If the dataset is a sample, then what is the larger set? Is the sample representative of the larger set (e.g., geographic coverage)? If so, please describe how this representativeness was validated/veriffed. not representative of the larger set, please describe why not (e.g., to cover a more diverse range of instances, because instances were withheld or unavailable).
25

---

The Risk of Getting Confounded in a Continual World
The dataset was generated synthetically. The entire space of possible samples is given by any combination of properties of the four objects included in an image and their placement. Additionally, images which do not conform to the dataset setup as determined by the ground truth rule and confounding features are not sampled. The dataset was sampled from the entire space of such permissible instances. The positioning of objects in an image is also determined randomly, avoiding images where an object is not visible (e.g. an object is hidden behind another).
What data does each instance consist of? "Raw" data (e.g., unprocessed text or images) or features? In either case, please provide description.
Each instance consists of raw image in png format showing the objects as described previously.
Is there a label or target associated with each instance? If so, please provide a description.
Each instance fulfills or does not fulfill a (ground truth) rule. Thus, instances could be seen as belonging to a positive or negative class, and are provided already separated into these categories.
Is any information missing from individual instances? If so, please provide description, explaining why this information is missing (e.g., because it was unavailable). This does not include intentionally removed information, but might include, e.g., redacted text.
No information is missing that relevant to the intents and purposes of the dataset's creation. We follow the typical omission of nuanced graphics engine simulation parameters.
Are relationships between individual instances made explicit (e.g., users' movie ratings, social network links)? If so, please describe how these relationships are made explicit.
Apart from sharing the rule determining the associated label, the dataset is constructed to be confounded. In di s jo nt, there is a confounding feature that is only present in the positive images of one specific tasks and does not appear anywhere else in dis joint. In strict, these confounding features are also present in the respective positive images and not the same task's 's negative images but they may appear randomly in other tasks. The unconfounded dataset is not confounded. No further relationships are made explicit.
Are there recommended data splits (e.g., training, develoopeentvalidation,, testing)? If so, please provide a description of these splits, explaining the rationale behind them.
Yes, each task consists of a training, validation and testing data split, with 3000, 750, and 750 instances, respectively. Additionally, we also have a universal unconfounded dataset with the same split as the tasks. There no difference in the data generation of these splits.
Are there any errors, sources of noise, or redundancies in the dataset? If so, please provide description.
There are no errors and no noise other that which is is implicitly caused by the simulation process. There a limited set of configurations resulting an instance. Nevertheless, this set large but redundancy may occur. Note, that even if two data items have the same object properties, they are vanishingly unlikely to also have the same positions.
Is the dataset self-contained, or does it link to or otherwise rely on external resources (e.g., websites, tweets, other datasets)? If it links to or relies on external resources, a) are there guarantees that they will exist, and remain constant, over time; b) are there official archival versions of the complete dataset (i.e., including the external resources as they existed at the time the dataset was created); c) are there any restrictions (e.g., licenses, fees) associated with any of the external resources that might apply to a future user? Please provide descriptions of all external resources and any restrictions associated with them, as well as links or other access points, appropriate.
The dataset is self-contained, however it does build on prior work. It is built on the CLEVR framework (Johnson et al., 2017), which has an open source license that allows us to make these modifications. Data generation also relies on the publicly accessible and free Blender software.
Does the dataset contain data that might be considered confidential (e.g., data that is protected by legal
26

---

The Risk of Getting Confounded in a Continual World
privilege by doctor-patient confidentiality, data that includes the content of individuals non-public communications)? If so, please provide description.
No.
Does the dataset contain data that, if viewed directly, might be offensive, insulting, threatening, or might otherwise cause anxiety? so, please describe why.
No.
Does the dataset relate to people? If not, you may skip the remaining questions in this section.
No, the dataset only contains synthetic objects.
Does the dataset identify any subpopulations (e.g., by age, gender)? If so, please describe how these subpopulations are identified and provide a description of their respective distributions within the dataset.
No.
Is it possible to identify individuals (i.e., one or more natural persons), either directly or indirectly (i.e., in combination with other data) from the dataset? If so, please describe how.
No.
Does the dataset contain data that might be considered sensitive in any way (e.g., data that reveals racial ethnic origins, sexual orientations, religious beliefs, political opinions or union memberships, or locations; financial or health data; biometric or genetic data; forms of government identification, such as social security numbers; criminal history)? If so, please provide a description.
No.
Any other comments?
No further comments.
Collection Process
How was the data associated with each instance acquired? Was the data directly observable (e.g., raw text, movie ratings), reported by subjects (e.g., survey responses), indirectly inferred/derived from other data (e.g., part-of-speech tags, model-based guesses for age or language) ? data was reported by subjects or indirectly inferred/derived from other data, was the data validated/verrfieed? If so, please describe how.
This dataset was generated synthetically and by simulation according to certain rules as explained previously, Data was generated using Blender and not sampled from a real dataset.
What mechanisms or procedures were used collect the data (e.g., hardware apparatus or sensor, manual human curation, software program, software API)? How were these mechanisms or procedures validated?
The source code used to generate the datasets is available at: https:: http:////gittub.ccoommeeeeeeeeee
If the dataset is a sample from a larger set, what was the sampling strategy (e-g., deterministic, proba- bilistic with specific sampling probabilities)?
Sampling was conducted in probabilistic manner where unassigned object properties are selected with uniform probability. To ensure all objects are recognizable, images where only a low amount of an object's pixels are visible (for example due to occlusion) are rejected and replaced with a new sample. This generation procedure may thus result non uniform distributions size and shape attributes.
27

---

The Risk of Getting Confounded in a Continual World
Who was involved in the data collection process (e.g., students, crowdworkers, contractors) and how were they compensated (e.g., how much were crowdworkers paid)?
Code was written and data generation conducted by the authors of this dataset and its respective paper. No subcontractors or crowdworkers were involved as all data is of simulated nature.
Over what timeframe was the data collected? Does this timeframe match the creation timeframe of the data associated with the instances (e.g., recent crawl of old news articles)? not, please describe the timeframe in which the data associated with the instances was created.
The conception of the dataset spanned the course of roughly two months. The data generation itself took on the scale of several hours on consumer grade hardware.
Were any ethical review processes conducted (e.g., by an institutional review board)? so, please provide description of these review processes, including the outcomes, as well as a link or other access point to any supporting documentation.
No, the entire dataset is purely synthetic and only consists of geometric objects, ethical review required.
Does the dataset relate to people? If not, you may skip the remaining questions in this section.
No.
Did you collect the data from the individuals in question directly, or obtain it via third parties or other sources (e.g., websites)?
No.
Were the individuals in question notified about the data collection? If so, please describe (or show with screenshots other information) how notice was provided, and provide a link or other access point to, or otherwise reproduce, the exact language of the notification itself.
No.
Did the individuals in question consent to the collection and use of their data? If so, please describe (or show with screenshots or other information) how consent was requested and provided, and provide a link or other access point to, or otherwise reproduce, the exact language to which the individuals consented.
No.
If consent was obtained, were the consenting individuals provided with a mechanism to revoke their consent in the future or for certain uses? so, please provide a description, as well as a link or other access point to the mechanism (if appropriate). No.
Has an analysis of the potential impact of the dataset and its use on data subjects (e.g., a data protection impact analysis) been conducted? so, please provide a description of this analysis, including the outcomes, as well as link or other access point to any supporting documentation.
No.
Any other comments?
No further comments.
Preprocesiiggddaaninggaaliiiigg
28

---

The Risk of Getting Confounded in Continual World
Was any preproessinggcdeennng/aalllinn of the data done (e.g., discretization bucketing, tokenization, part-of-speech tagging, SIFT feature extraction, removal of instances, processing of missing values)? If so, please provide a description. If not, you may skip the remainder of the questions in this section.
No preprocessing was involved in the creation of this dataset.
Was the "raw" data saved in addition to the preproceseedddeennedlaeeeed data (e.g., to support unantici- pated future uses)? If so, please provide a link or other access point to the "raw" data.
Since no preprocessing was conducted, the "raw" data and its label are the primary content of this dataset.
Is the software used to preprocess/clenn/laeel the instances available? If so, please provide a link or other access point.
No preprocessing involved in the creation of this dataset.
Any other comments?
No further comments.
Uses
Has the dataset been used for any tasks already? If so, please provide a description.
Yes, we conducted an evaluation on continual confounding. Here, we showed that machine learning models get confounded on the dis joint data. On the strict. data, found out that performance the unconfounded data is better when training on data from all confounded tasks jointly when compared to the continual learning setup.
Is there repository that links to any or all papers or systems that use the dataset? so, please provide link other access point.
There is no such repository at the time of the creation of this dataset sheet.
What (other) tasks could the dataset be used for?
The primary purpose of ConCon the investigation continual confounding, but it could also be used for other tasks, such as neuro-symbolic continual learning.
Is there anything about the composition of the dataset or the way it was collected and prepro- bessed/cleaned/laaeeeed that might impact future uses? For example, there anything that a future user might need to know to avoid uses that could result in unfair treatment individuals or groups (e.g., stereotyping, quality of service issues) other undesirable harms (e.g., financial harms, legal risks) so, please provide description. is there anything future user could do to mitigate these undesirable harms?
None that the authors aware of, due to the synthetic nature of the dataset consisting of simulated simple objects (like colored cubes and squares) coupled with logical rules.
Are there tasks for which the dataset should not be used? If so, please provide description.
Please refer to the previously asked similar question.
Any other comments?
No further comments.
Distribution
Will the dataset be distributed third parties outside of the entity (e.g., company, institution, organization) on behalf of which the dataset was created? If so, please provide description.
29

---

The Risk of Getting Confounded in a Continual World
Yes, the dataset is publicly available Zenodo and the source code is available at Github.
How will the dataset will be distributed (e.g., tarball on website, API, GitHub) Does the dataset have digital object identifier (DOI)?
We the thiry- open-access repository Zenodo Zenodo features unique digital object identifiers. The Zenodo DOI for our dataset is: 10. 5281/zenodo 10630482.
When will the dataset be distributed?
The dataset made available 9th February, 2024.
Will the dataset be distributed under copyright other intellectual property (IP) license, and/or under applicable terms of use (ToU)? If please describe this license and/or ToU, and provide link other access point to, or otherwise reproduce, any relevant licensing terms or ToU, as well as any fees associated with these restrictions.
The dataset is distributed under the BSD 2-Clause "Simplified" License.
Have any third parties imposed IP-based or other restrictions on the data associated with the instances? so, please describe these restrictions, and provide a link or other access point to, or otherwise reproduce, any relevant licensing terms, as well as any fees associated with these restrictions. No.
Do any export controls or other regulatory restrictions apply to the dataset or to individual instances? so. please describe these restrictions, and provide link or other access point to, or otherwise reproduce, any supporting documentation.
No.
Any other comments?
No further comments.
Maintenance
Who will be supportiigghosttnn/mmannniinnn the dataset?
We use Zenodo to host the dataset, a reputable open-access repository where the dataset has a high likelihood to persist long into the future. Maintaining is carried out by the authors and affiliated groups.
How can the owner/curator//maaggr of the dataset be contacted (e.g., email address)?
The creators of the datasets: Florian Peter Busch and Roshni Kamath can contacted at florian peter. @tu-@tttdaaat tu-darmstadt..e and roshni kamath tu-darmstadt.de respectively.
Is there an erratum? so, please provide a link or other access point.
There presently is no erratum. However, Zenodo offers an erratum functionality should it become relevant in the future.
Will the dataset be updated (e.g., to correct labeling errors, add new instances, delete instances)? so, please describe how often, by whom, and how updates will be communicated users (e.g., mailing list, GitHub)?
Currently, no immediate updates are envisioned.
hhttps://zendoo.ogg
30

---

The Risk of Getting Confounded in a Continual World
If the dataset relates to people, are there applicable limits on the retention of the data associated with the instances (e.g., were individuals in question told that their data would be retained for fixed period of time and then deleted)? If so, please describe these limits and explain how they will be enforced.
The dataset does not relate to people.
Will older versions of the dataset continue be supportee/hosteemmaiittinnn?? If so, please describe how. not, please describe how its obsolescence will be communicated to users.
Yes, Zenodo supports access of old versions using unique Digital Object Identifiers (DOI).
others want to extend/augment/biii on/contribute to the dataset, is there a mechanism for them to do so? If so, please provide a description. Will these contributions be validated/verffied? If so, please describe how. If not, why not? Is there process for communicatingddisrrbbbin these contributions to other users? If so, please provide a description.
The code for data generation is publicly available on Github https: otot Validation/Verffication of future contributions will not be in the scope of the authors.
Any other comments?
No further comments.
31


Reviews:
[{"review_id": "8CGsT6rHys", "reviewer": "Reviewer_SVA5", "summary": "This paper explores confounding in continual learning. The authors formally describe confounding factors that lead to poor generalization and introduce a CLEVR-based synthetic dataset (ConCon) to study these challenges. They evaluate several continual learning approaches on ConCon and show that these methods struggle to mitigate the influence of confounders, leading to degraded performance on unconfounded test data.\n\n### Update after rebuttal \nThe author's rebuttal addresses most of my concerns. Thus I have updated my score from 2 to 3.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the detailed and constructive feedback. We first try to clarify their more general concerns.\n\nFeatures that act as confounders in one dataset can also appear as random features in other datasets. The distribution shift between the confounded tasks and the unconfounded dataset is a result of such features no longer **acting** as confounders, i.e., they might appear on both positive and negative images but are not informative wrt the class label. Our synthetic data generation used for ConCon ensures minimal distribution shift between confounded and unconfounded datasets while still exhibiting confounding in one but not the other. Thus, the resulting minimal train-test mismatch is intentional and allows for investigating whether a high model accuracy is caused by the model making predictions based on the confounding features or the ground-truth features. We have included this explanation in section 3.2.\n\nIn addition to experience replay (ER), where we use a buffer of 100 samples, we also evaluate ER with an infinite buffer size. This is what we refer to as *cumulative training*. Any other buffer size larger than 100 should, therefore, result in accuracies between those of ER and cumulative training. We made changes to clarify this in Section 4.1.\n\nWe will now respond to the weaknesses (W) and questions (Q).\n\n**W1/Q1** \n\nIn this paper, there is no a priori distinction between confounding features and ground truth features. This is intentional, as we aim to investigate the models' capabilities in identifying the ground truth features without prior information. As a result, confounding is defined relative to the full set of tasks. How features that act as confounders in one task appear in other tasks depends on the dataset variant: \nIn the disjoint dataset, confounding features are only present in their respective task. They are identifiable in the sense that their absence in the other tasks makes them irrelevant and unhelpful for accurate prediction-making. In the strict dataset, confounding features appear in other tasks in both positive and negative images. They are, therefore, incompatible with the correct decision rule in new tasks.\n\nAny confounding features of the confounded tasks may appear in images of either class in the unconfounded dataset. However, they are uninformative and provide no value for prediction-making, as they do not act as confounders there.\n\n**W2**\n\nWe ran experiments on ConCon where we shuffled the training data before distributing it across tasks, thereby removing the alignment of confounders:\n\n|          | Task 1 | Task 2 | Task 3 | Task 1 @ Task 3 | Task 2 @ Task 3 | UnConf |\n| -------- | ------ | ------ | ------ | --------------- | --------------- | ------ |\n| Strict   | 79.85  | 93.4   | 95.32  | 95.18           | 96.72           | 89.52  |\n| Disjoint | 99.98  | 99.90  | 99.96  | 99.98           | 99.84           | 49.97  |\n\nFor strict, the accuracy on UnConf is much closer to joint (95.7) than to cumulative (72.6). This confirms that insidious continual confounding is responsible for a majority of the drop in accuracy.\n\nFor the ImageNet experiment, we align the confounders task-wise: Task 1: Arctic fox, snowmobile, Task 2: broccoli, tree frog, Task 3: tiger shark, jellyfish.\n\nWe obtained 89.93% joint and 89.0% cumulative accuracy. Unfortunately, any differences here are obscured by variance due to data loading order, so we cannot draw substantial conclusions.\n\n**Q2**\n\nThe gap arises because of the combination of confounders and the challenges of streaming data. As we also show in our new experiment on ConCon above, the gap between cumulative and joint training is more substantial when tasks are confounded.\n\n**W3**\n\nWe ran additional experiments on real-world data. Please refer to our response to reviewer iRoy.\n\n**Q3**\n\nIn real-world scenarios, we expect confounding to be imperfect. This could express itself in two ways:\n\n1. Only a subset of the data is confounded.\n2. The entire dataset is imperfectly confounded, i.e., in the training set, the confounders provide some limited bits of information about the class, but this relationship does not generalize on the test set.\n\n**W4**\n\nWe here show the results for generative replay:\n\n|          | Task 1 | Task 2 | Task 3 | Task 1 @ Task 3 | Task 2 @ Task 3 | UnConf |\n| -------- | ------ | ------ | ------ | --------------- | --------------- | ------ |\n| Strict   | 99.96  | 98.54  | 99.58  | 48.84           | 48.61           | 49.26  |\n| Disjoint | 100.0  | 88.78  | 99.28  | 53.2            | 59.33           | 47.73  |\n\nResults match those reported for other approaches in our paper. We have included it in the appendix. \nPlease let us know if the reviewer recommends any other methods to include.", "claims_and_evidence": "1. Sequential vs. Joint Training:  \nThe paper claims that sequential training (including cumulative training) is more challenging in the presence of confounders compared to joint training. Figure 1 and Table 2 indicate that joint training outperforms cumulative training. However, this may be due to the inherent difficulty of training on streaming data—non-convex optimization can lead to overfitting on biased, limited data, causing poor local minima. Thus, the performance gap might primarily reflect the challenge of learning from changing distributions rather than the effect of confounders alone.  \n\n2. Generalization in Continual Learning:  \nThe authors argue that even methods that prevent forgetting in continual learning fail to generalize well due to confounders, as shown in Table 1. While this observation may be valid, the experimental design appears to mix the issue with a train-test distribution mismatch. In task t, the training data includes both ground-truth features (g) and task-specific confounders (c_t), but the test set only contains the ground-truth feature g. This mismatch might be driving the performance gap, which is more an issue of domain adaptation than a failure of continual learning.", "methods_and_evaluation_criteria": "The paper primarily introduces a new concept and dataset rather than novel methods. It proposes a formal description of confounding and uses the ConCon dataset—with its controllable confounding factors—to evaluate various continual learning approaches. However, the evaluation might be skewed by the train-test mismatch, making it unclear if the poor generalization is due to continual learning shortcomings or simply the data distribution shift.", "theoretical_claims": "The paper presents a formal description of continual confounding using Boolean algebra to capture the interplay between ground-truth predicates and task-specific confounders. Although the theoretical framework is plausible, the assumptions require further clarification. In particular, the distinction between ground-truth features and confounders is not well defined; defining confounders solely based on the test set is problematic.", "experimental_designs_or_analyses": "The experimental evaluation is based on the ConCon dataset. Concerns include:\n1. Task Design: The current task design may not adequately isolate the effect of confounders from the inherent difficulty of learning on non-stationary data.\n2. Train-Test Mismatch: The performance gap might be largely due to the mismatch between training (with both g and c_t) and testing (with only g), which is more a domain generalization issue than a continual learning problem.\n3. Limited Real-World Evaluation: Relying exclusively on a synthetic dataset raises questions about the generalizability of the findings to real-world scenarios.", "relation_to_broader_scientific_literature": "The paper situates its contributions within the continual learning literature. It would benefit from a deeper discussion on the relationship between domain generalization and invariant risk minimization literature, both of which address issues of spurious correlations and distribution shifts.", "essential_references_not_discussed": "Including related work from the domain generalization literature—where similar challenges of spurious correlations are addressed—would help strengthen the context for the paper’s contributions.", "other_strengths_and_weaknesses": "Strengths:\n1. Studies an interesting and underexplored problem in continual learning, which is related to dataset confounding. \n2. Introduces a novel, controlled synthetic dataset (ConCon) to study confounding effects.\n\nWeaknesses:\n1. The paper needs a clearer distinction between ground-truth features and confounders. If a factor is deemed confounding, it should ideally be identifiable in both the training and test sets; otherwise, the evaluation might be misleading.\n2. The design suffers from a train-test mismatch that may be driving the observed performance gaps. The performance differences in Figure 1 could stem from challenges associated with streaming data rather than solely from the presence of confounders. Consider redesigning the experimental tasks (e.g., grouping tasks by different background colors) to better isolate the impact of confounders. Specifically, Task 1 could involve classification between images with a blue background (e.g., jellyfish and shark), while Tasks 2 and 3 could focus on white and green backgrounds, respectively. Comparing the final performance between this revised design and the original design would help assess the influence of confounding factors.\n3. The reliance on the proposed synthetic dataset limits the applicability of the findings. Evaluating on real-world or multi-class datasets would provide stronger evidence for the claims.\n4.  It is helpful to incorporate more baselines. Such as generative reply [1], which is also an important baseline in continual learning. \n\n[1] Van de Ven, Gido M., Hava T. Siegelmann, and Andreas S. Tolias. \"Brain-inspired replay for continual learning with artificial neural networks.\" Nature communications 11.1 (2020): 4069.", "other_comments_or_suggestions": "1. The memory size used in experiments (e.g., 100 samples) may be too small to reflect practical scenarios; larger memory sizes should be considered, especially in table 2.\n\n2. If the goal is to evaluate model performance under different distributions, exploring domain adaptation techniques might be more appropriate than a pure continual learning approach.", "questions_for_authors": "1. How do you ensure a clear and consistent distinction between ground-truth features and confounders in the ConCon dataset?\n2. Could the performance gap between joint and sequential training be primarily attributed to the challenges of streaming data (non-convex optimization and local minima) rather than confounders?\n3. How do you envision the proposed framework and findings translating to real-world continual learning scenarios, where data distributions might differ less drastically between training and testing?\n\nFor additional details on potential weaknesses and suggested modifications, please refer to the previous discussion.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UcjfrG2bNv", "reviewer": "Reviewer_yrkD", "summary": "The paper presents the confounder problem in the continual learning regime, which is novel. It also establishes a benchmark with clear logical definitions and potentially highlights a new direction for studies in the continual learning field to improve the overall performance.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their positive feedback and we will include the suggested references in our related work section. We are happy to read that they agree that our experiments are sufficient to support our claims. Nevertheless, we invite the reviewer to also take a look at our responses to the other reviewers, where we add additional experiments that support our claims even further.", "claims_and_evidence": "The claims made in the work are clear and logical.", "methods_and_evaluation_criteria": "The evaluation methods presented in this work are novel and valid for use in continual learning scenarios to investigate whether the model is truly learning in a sequential setting rather than relying on shortcuts.", "theoretical_claims": "Not applicable.", "experimental_designs_or_analyses": "The experimental design is scientific, and the related continual learning methods are evaluated on the new benchmarks.", "relation_to_broader_scientific_literature": "N/A", "essential_references_not_discussed": "Here are some related continual learning works the authors may consider including:\n\n[1] Liu, Y., Zhu, W., & Ren, S. (2022). Navigating memory construction by global pseudo-task simulation for continual learning. Advances in Neural Information Processing Systems, 35, 7342-7355.\n\n[2] Farajtabar, M., Azizan, N., Mott, A., & Li, A. (2020, June). Orthogonal gradient descent for continual learning. In International conference on artificial intelligence and statistics (pp. 3762-3773). PMLR.", "other_strengths_and_weaknesses": "The work is well-written and complete. The continual confounding is clearly defined through logical rules and effectively explained with text and images. The dataset benchmark, ConCon, constructed also makes sense.", "other_comments_or_suggestions": "More models could be tested on the dataset for evaluation, but it is not necessary. The ResNet and NeSy models based on transformers used in the experiments are sufficient to support the statements.", "questions_for_authors": "None so far.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "StowrUyuiV", "reviewer": "Reviewer_KoqF", "summary": "This paper introduces the concept of continual confounders. A data contains confounders when a model trained on the data can fit the training data using spurious correlations but fails to generalize at test time. Continual confounders are ones that control distributions across a continual set of tasks. The paper also introduces a dataset, ConCon, a simulated dataset built on the framework of CLEVR using Blender software consisting of images of various objects with varying textures and colors in a 3D space.\n\nSection 3 introduces the concept of continual confounding. Continual confounders are divided into two categories, disjoin and strict. Disjoint confounders are only observed in their respective tasks. A model which fits continually disjoint confounders need not to unlearn the confounders of previous tasks to learn a new task. Strict confounders on the other hand, may appear in other tasks. Section 3.1 provides a rigorous definition of confounders. Section 3.2 introduces the ConCon dataset that contains disjoint and strict variants. The dataset contains various objects such as spheres, cubes, and cylinders in different sizes, colors, and textures. The task is binary classification to determine whether a sphere and small cube exists in the image. The confounders are the blue, metal, and large.\n\nIt is argued that static confounders are insidious continual confounders that if a model is trained jointly on the dataset, it can generalize well by not utilizing confounders but learned continually, it will fail to generalize. As such avoiding catastrophic forgetting is not enough for continual learners in such a setup.\n\nThe experiments evaluate the performance of continual learning methods for training two architectures, ResNet-18, and NeSy. The continual learning methods include replay methods and regularization methods. The following observations are made:\n- All CL methods on both models fail on unconfounded held-out test sets on the disjoint variant of ConCon. Similarly on the static variant all methods except for joint and cumulative training do not generalize.\n- Preventing catastrophic forgetting on the disjoint dataset does not help with making correct predictions on unconfounded data.\n- Training in a continual setup performs significantly worse than the joint setup.\n- Continual learning methods suffer from insidious continual confounding.\n\n## Update after rebuttal\nI thank authors for their response including clarifications and new results with PNNs. I recommend incorporating the response into a revision. I maintain a weak accept rating for this work as I believe this dataset may be valuable for the community in studying continual learning methods but understand other reviewers have remaining concerns about the dataset being synthetic.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback and positive evaluation of our work. We would like to comment on a few points in their review.\n\n**Related Work on Distribution Shifts**\n\nThank you for suggesting further related work. We have included the references and discussed their relation to ConCon and the goal of our paper.\n\n**Goal of This Paper and ConCon**\n\nThe goal of our paper is to \"improve the evaluation of continual learning methods when confounding might be present and also inspire new approaches to deal with these issues. The ConCon benchmark is designed to facilitate systematic investigation of such methods in the future.\" We included these sentences in our paper. The disjoint dataset represents a simple confounded setting where confounding features only appear on certain images. A good example of such confounding is watermarks. We show that a challenge for CL methods on disjoint datasets is not only to avoid forgetting but also to learn the ground-truth features and not just the disjunction of confounders. The confounding in our strict dataset is less obvious, as confounding features may appear as regular, random features in other tasks. For example, snowy backgrounds might at first only be associated with polar bears, but in later classes, they also appear on other images in winter. Perhaps surprisingly so, we show that this type of insidious continual confounding can deteriorate model performance compared to non-continual learning settings. Our goal is, thus, to highlight these challenges and dangers related to confounding in continual settings. Our results on insidious continual confounding also question the use of cumulative training as the gold-standard upper-bound in continual learning. \n\nOur response to reviewer iRoy about our contributions might also be of interest.\n\n**Effects of Pre-Training and Implications on Adaptive Architectures**\n\nOur NeSy model is pre-trained on the CLEVR dataset. However, we see that it does not perform well on the unconfounded datasets, where the accuracies are worse than for the Resnet-18 model (70.8 vs 95.7). Pretraining, in general, can both be helpful and harmful when it comes to continual confounding, depending on what the model is trained on. If the pre-trained model has learned the ground-truth prediction, it is unlikely to unlearn that even on confounded data. However, our comparison of joint and cumulative training shows how a model pre-trained (tasks 1 and 2) to focus on confounders performs worse than a model trained from scratch (joint training).\n\nFollowing the reviewer's suggestion, we ran the Progressive Neural Networks (PNNs) method on the two variants of our ConCon dataset for the NN model and added them to tables 1 and 2 :\n\n|          | Task 1 | Task 2 | Task 3 | Task 1 @ Task 3 | Task 2 @ Task 3 | UnConf |\n| -------- | ------ | ------ | ------ | --------------- | --------------- | ------ |\n| Strict   | 100.0  | 99.77  | 99.76  | 100.0           | 99.7            | 49.2   |\n| Disjoint | 100.0  | 99.84  | 100    | 100.0           | 99.6            | 47.89  |\n\nWe expand the ResNet-18 model when it encounters new tasks.\nData from confounded tasks always use their respective output heads, and the unconfounded data uses the most recent head (task 3).\nGenerally speaking, we do not expect adaptive architectures to help against continual confounding by themselves. Since the model first learns to focus on confounding features, thereby learning simple solutions and setting model weights accordingly, expanding the architecture does not automatically help learn the correct, more complex ground-truth rule. We hypothesize that such a model must be able to unlearn the previously learned behavior in order to ignore the confounders and learn the ground-truth feature instead. Therefore, we argue that adaptive architectures are unlikely to help against continual confounding, although specialized approaches might be useful.\n\n**Question on Insidious Continual Confounding**\n\n> Is insidious continual confounding limited to strict confounders? Or is insidious continual confounding a property that various confounder types can have including strict confounders?\n\nConfounders / confounding features themselves do not exhibit insidious continual confounding. Insidious continual confounding arises when a model is continually trained on confounded data, where confounding causes it to make predictions with lower accuracy than in a joint training scenario. (Note that confounding that does not change across tasks would not cause insidious continual confounding as joint training would suffer from it in the same way; see our evaluation on the disjoint dataset.) Whether insidious continual confounding occurs thus depends on both the type of confounding and the model and training procedure.\n\n**Typos**\n\nWe thank the reviewer for their feedback on typos. We will correct them for the camera-ready version.", "claims_and_evidence": "The paper makes the following claims and provide evidence in a synthetic setup:\n- Continual learning methods suffer from insidious continual confounding where the model can generalize if trained jointly on the data but does not generalize when trained continually.\n- Preventing catastrophic forgetting does not help with making correct predictions on unconfounded data.\n- The ConCon dataset is a benchmark for evaluating the impact of confounders in continual learning.", "methods_and_evaluation_criteria": "The paper does not propose a new method. It provides a new evaluation benchmark that controls the confounders over tasks and provides meaningful observations.", "theoretical_claims": "The paper provides a rigorous definition of confounders but no theoretical claims.", "experimental_designs_or_analyses": "The construction of the ConCon dataset with disjoint and strict variants is sound. The results on these datasets also match the intuition.", "relation_to_broader_scientific_literature": "This work is related to two literatures on continual learning and spurious correlations. The connection between these two literatures has not been explored before and is interesting to study.", "essential_references_not_discussed": "The paper is missing references to works on spurious correlations and related benchmarks. For example:\n- Koh, P. W., et al., Wilds: A benchmark of in-the-wild distribution shifts. ICML 2021.\n- Sagawa, S., et al. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. ICLR 2020.\n\nA discussion and comparison is needed, especially in terms of the types of spurious correlations identified in prior works.", "other_strengths_and_weaknesses": "Strengths:\n- The paper introduces an interesting concept of continually confounded datasets, rigorously defines them and introduces an evaluation that provides novel insights.\n\nWeaknesses:\n- It is not clear whether the goal of the benchmark is to inspire model architecture design in the future or better continual training methods or to suggest some continual tasks are not learnable by continual learners. The paper needs to clarify its goal.\n- The evaluation focuses on only two models, ResNet-18 and NeSy trained from scratch on the data. For example, what happens if a pretrained model is fine-tuned on the data? What are the implications for continual learning methods with adaptive architectures such as Progressive Neural Networks (Rusu, Andrei A., et al. \"Progressive neural networks.\" (2016))?", "other_comments_or_suggestions": "Typos:\n- Line 41: Resnet -> ResNet\n- Line 86: a -> an\n- Line 120: introduction example -> introductory example\n- Section 3: it may be helpful to add a summary table for comparison between disjoint and strict continual confounders and some of their characteristics. For example, which learner (with or without replay) in which setup (continual vs jointly) on which confounders (strict, disjoint) can fit/generalize.\n- Line 254: comprising of -> comprises", "questions_for_authors": "- Is insidious continual confounding limited to strict confounders? Or is insidious continual confounding a property that various confounder types can have including strict confounders?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aml1mS8zhK", "reviewer": "Reviewer_iRoy", "summary": "The paper explores a nuanced aspect of continual learning related to confounding data. It highlights how confounding data can create shortcuts by fostering spurious correlations, ultimately hindering the generalization ability of continual learning methods. The authors demonstrate the effects of confounding data on sequential continual learning using a confounding dataset generated from CLEVR. Their findings show that conventional continual learning methods struggle in this setting, underscoring the need for more robust approaches.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback. In the following, we would like to address their two concerns.\n\n**Synthetic vs Real-World Datasets**\n\nIn our paper, we introduce a real-world experiment on the popular ImageNet dataset and observe a surprising discrepancy between joint and cumulative training. By introducing the ConCon dataset, we are able to systematically investigate variants of continual confounding and eliminate other (potentially unknown) factors present in real-world images. This enables us to eliminate alternative causes of unexpected model behavior and allows for analyzing the effects of continual confounding in isolation.\n\nOur real-world experiment introduced in section 1 shows the same behavior as we observed in our experiments on ConCon: cumulative training results in a lower accuracy than joint training. In order to further highlight that this behavior can also occur on natural images, we also include another real-world experiment involving a higher number of ImageNet classes. Here, we even observe a 5% difference between cumulative and joint training. Please consider the experimental details and results below (\"New Real-World Experiment on ImageNet\"). We have included this experiment in the appendix.\n\n**New Real-World Experiment on Imagenet**\n\nIn this experiment, we select several ImageNet classes such that in each of the three tasks, we have a group of 4 ImageNet classes representing 4-legged animals and 4 ImageNet classes not representing 4-legged animals. Each group of 4 classes is assigned one label, resulting in 6 labels for the model to learn. See the table below for the class selection. The resulting groups are correlated with colors (Task 1: white vs blue, Task 2: green vs white, Task 3: blue vs non-blue). We hypothesize that for each task in isolation, learning to make predictions based on the confounding feature \"color\" is easier than learning the ground-truth features that correspond to \"animal with 4 legs\".\n\n|                     Task 1                      |                    Task 2                     | Task 3                                  |\n| --------------------------------------------- | ------------------------------------------- | --------------------------------------- |\n| arctic fox, polar bear, white wolf, samoyed dog |         cheetah, deer, lion, leopard          | hippo, crocodile, water buffalo, beaver |\n|     jellyfish, blue jay, blue shark, tench      | snowmobile, sailboat, snowplough, rattlesnake | eagle, spider, lifeboat, mushroom       |\n\nAs for our other experiments, we ran cumulative and joint training on a Resnet-18 model and averaged the results of 5 different seeds. The results for joint training and for cumulative training after covering all three tasks are shown in the table below:\n\n| Cumulative | 76.3%  | 78.5%  | 84.4%  |\n| ---------- | ------ | ------ | ------ |\n| Joint      | 83.45% | 86.25% | 86.25% |\n\nAcross all tasks, we observe an average decrease in accuracy of 5.58 percentage points, a substantial degradation in performance. This provides further evidence that continual confounding can manifest in real-world images and that it does deteriorate model performance. This experiment has been added to the appendix.\n\n**Contribution Despite a Lack of Solutions for Continual Confounding**\n\nThis paper is the first to investigate continual confounding. We not only showcase that this is an issue on real-world data but also introduce a thorough framework that provides a foundation for systematically studying different variants of confounding based on CLEVR, where one has complete control over the distribution of the generated images. ConCon, the resulting benchmark on continual confounding, is, therefore, one of many dataset and benchmark papers that are common in the machine learning community and fall under the ICML call for papers as \"datasets that are of interest to the machine learning community\". Our findings include a discussion on insidious continual confounding, a phenomenon where cumulative training yields worse accuracies than joint training. This by itself is an important contribution, as it questions the use of cumulative training as the gold-standard upper-bound for evaluating continual learning methods, which is a common approach in continual learning research.\n\nWe fully agree that developing methodological solutions for overcoming our identified issue of continual confounding is very important. However, we consider this to be out of the scope of this dataset paper, which aims to identify and analyze the issue. We believe that ConCon will be of great use in evaluating such novel approaches. However, doing so requires further research and more in-depth analysis than could be provided by this paper.", "claims_and_evidence": "The authors claim to study continual confounding using ConCon dataset and they also provide evidence supporting the claim but evidence does seem strong since it lacks evidence on other real world datasets.", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "Yes", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "Majority of the Continual Learning literature focuses on issue of Catastrophic Forgetting while this paper focuses on continual confounding which adds value to this field.", "essential_references_not_discussed": "References are adequate.", "other_strengths_and_weaknesses": "## Strengths\n\n- The paper is well-written and easy to follow.\n- It offers a novel perspective on continual learning by focusing on the impact of confounding data, rather than the commonly studied issue of catastrophic forgetting. This adds value to the field, where most literature centers around catastrophic forgetting.\n\n## Weaknesses\n- The ConCon dataset, used to study the impact of confounding data, appears somewhat artificial and monotonous.\n- While the paper effectively identifies a critical challenge in continual learning, it does not propose any solutions to address it.", "other_comments_or_suggestions": "None Found", "questions_for_authors": "While the paper presents a novel perspective on continual learning, it lacks strong evidence of continual confounding due to the absence of evaluations on real-world datasets. Additionally, it does not propose any solutions, leaving the community without a clear direction for addressing and mitigating the identified issue.\n\n- Is it possible to evaluate on the dataset similar to the figure 1 shown in the paper?\n- Why the authors have not investigated the solution of the problem described in the paper?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper received three weak accepts and one accept, reflecting generally positive reception with some reservations. The work addresses an important and underexplored issue in continual learning, dataset confounding, and introduces a novel synthetic benchmark to systematically study these effects. The paper is well-written and the continual confounding scenarios are clearly defined and illustrated. However, reviewers noted areas for improvement, including the need for a clearer separation between ground-truth features and confounders, potential train-test mismatches affecting results, and the limitations of relying solely on synthetic data. Suggestions were made to include additional baselines and to evaluate real-world datasets to improve generalizability. Overall, the dataset and framework have the potential to be a valuable tool for the community, and the authors are encouraged to incorporate reviewer feedback in the final version.
Decision: Accept (spotlight poster)