Permutation Invariant Learning with High-Dimensional Particle Filters

## Abstract

Abstract Sequential learning in deep models often suffers from challenges such as catastrophic forgetting and loss of plasticity, largely due to the permutation dependence of gradient-based algorithms, where the order of training data impacts the learning outcome. In this work, we introduce a novel permutation-invariant learning framework based on high-dimensional particle filters. We theoretically demonstrate that particle filters are invariant to the sequential ordering of training minibatches or tasks, offering a principled solution to mitigate catastrophic forgetting and loss-of-plasticity. We develop an efficient particle filter for optimizing high-dimensional models, combining the strengths of Bayesian methods with gradient-based optimization. Through extensive experiments on continual supervised and reinforcement learning benchmarks, including SplitMNIST, SplitCIFAR100, and ProcGen, we empirically show that our method consistently improves performance, while reducing variance compared to standard baselines. Project website and code is available here .

## 1 Introduction

What is the optimal order for training data? This question is fundamental to understanding how the sequencing of training data impacts machine learning model performance. In sequential learning settings, such as continual learning and lifelong learning, the sequencing of training data plays a crucial role in determining model performance. When models are trained on ordered minibatches of data, poor ordering—referred to as ”poor permutations”—can result in catastrophic forgetting and loss of plasticity (Wang et al., ; Abel et al., ).

In continual learning, models process tasks in a specific sequence. Unlike conventional training, where minibatch data is randomized, continual learning often relies on a strict sequence, making models prone to overfitting on newer tasks while losing performance on older tasks. This is known as catastrophic forgetting, where new information erases prior knowledge, severely degrading performance on earlier tasks (Kim & Han, ; van de Ven et al., ).

Similarly, in lifelong reinforcement learning (LRL), agents must adapt to new tasks sequentially. The order in which these tasks are presented can lead to loss of plasticity, limiting the agent’s ability to adapt to new environments (Muppidi et al., ; Lyle et al., ; Abbas et al., ; Sokar et al., ). This poor ordering can further manifest as negative transfer or primacy bias, where learning earlier tasks biases the agent towards those tasks, impeding adaptation to new tasks (Nikishin et al., ; Ahn et al., ).

To address these challenges, we propose a shift in perspective, viewing the problem through the lens of permutation invariance. By developing learning algorithms that are invariant to the order of data presentation, we can mitigate catastrophic forgetting and loss of plasticity. Our key insight is the use of particle filters, a probabilistic tool widely used in state estimation, to achieve this goal.

Particle filters excel at dynamically estimating system states from noisy data and are grounded in Bayesian inference (Thrun et al., ; Doucet et al., ; Jonschkowski et al., ; Karkus et al., ; Corenflos et al., ; Pulido & van Leeuwen, ; Maken et al., ; Boopathy et al., ). However, their application to modern machine learning has been limited due to scalability issues in high-dimensional settings. In contrast, gradient-based optimization techniques such as gradient descent efficiently handle high-dimensional spaces but lack the probabilistic framework offered by particle filters.

In this work, we bridge this gap by proposing a novel particle filter designed specifically for high-dimensional learning. We show that by adapting particle filters to high-dimensional learning problems, we can achieve more robust permutation-invariant learning. Our approach provides a new perspective on training in sequential settings and also addresses the core challenges of catastrophic forgetting and loss of plasticity in a principled manner.

Our contributions are threefold:

- •
Theoretically, we demonstrate that particle filters enable permutation-invariant learning, where the algorithm’s output remains consistent regardless of the training data order. We further show that this property naturally mitigates catastrophic forgetting and loss of plasticity.
- •
We introduce a simple, gradient-based particle filter specifically tailored for high-dimensional parameter spaces. This filter retains the essential features of traditional particle filters while being computationally efficient and well-suited for typical machine-learning optimization tasks.
- •
Through empirical evaluations on continual learning and lifelong reinforcement learning benchmarks, including SplitMNIST, SplitCIFAR100, and ProcGen, we show that our proposed particle filter achieves better performance and reduced variance compared to standard baselines. Additionally, we demonstrate that integrating this particle filter with continual learning and LRL methods increases overall performance and reduces performance variance.

## 2 Related Work

### 2.1 Particle Filters

Particle filters, or sequential Monte Carlo methods, are widely used for state estimation in non-linear and non-Gaussian settings. They represent probability distributions through a set of samples (particles), providing flexibility in capturing complex dynamics (Doucet et al., ). In fields such as robotics, particle filters have been applied successfully to localization and mapping problems, where they handle uncertainty and non-linearities effectively (Thrun, ). However, a key limitation is their scalability: as the dimensionality of the problem increases, the number of particles needed grows exponentially, making them less practical in high-dimensional spaces like those in machine learning (Bengtsson et al., ). Recent efforts have focused on improving particle filter scalability through adaptive resampling and dimensionality reduction techniques (Li et al., ), but these approaches have not fully bridged the gap for large-scale machine learning applications. Our work addresses this gap by proposing a high-dimensional particle filter that is computationally efficient and well-suited for machine learning tasks.

### 2.2 Bayesian Model Averaging

Bayesian model averaging (BMA) is a powerful technique for integrating uncertainty into model predictions by averaging across multiple models (Hoeting et al., ; Wasserman, ). By weighting model predictions based on their posterior probabilities, BMA can provide more robust predictions and better capture model uncertainty compared to single-model approaches (Raftery et al., ). In modern machine learning, BMA has been employed to enhance performance and uncertainty estimation, notably in ensemble techniques (Lakshminarayanan et al., ; Wortsman et al., ). However, BMA has not been extensively explored in the context of continual or permutation-invariant learning, where uncertainty over tasks and sequential data plays a crucial role.

In this work, we demonstrate the benefits of particle filters in high-dimensional machine learning settings. We then describe a particular particle filter that functions as a BMA technique, and demonstrate its advantages empirically.

## 3 Particle Filters for Learning Problems

In this section, we first theoretically demonstrate two beneficial properties of particle filters generally on learning problems, namely 1) permutation-invariance and 2) avoidance of catastrophic forgetting and loss of plasticity. We then describe a particular particle filter suitable for high-dimensional learning problems.

### 3.1 Setup

Consider a learning problem that provides a sequence of loss functions of model parameters, and the goal of learning is to minimize the sum of the loss functions. For instance, the loss function at each time step might correspond to the cross-entropy loss on a minibatch of points for a classification problem. We denote the model parameters at time $t$ as $x_{t}\in\mathbb{R}^{d}$ and the loss function at time $t$ as $L_{t}\in\mathbb{R}^{d}\to\mathbb{R}$. The goal is to find an $x$ minimizing $\sum_{t=1}^{T}L_{t}(x)$, where $T$ is the total number of updates.

How can we apply particle filters to this learning problem? To do this, we suppose that instead of learning a single model, we learn a distribution of models following a Bayesian approach. Specifically, suppose that at time $t=0$, we initially start with a prior distribution of candidate models; each $L_{t}$ corresponds to an observation that updates the likelihood of each model. Specifically, we suppose that the likelihood of the model $x$ is set as:

$$ $P(L_{t}|x)=e^{-L_{t}(x)}$ (1) $$

This likelihood function increases the likelihood of models that achieve lower loss values. We denote the prior distribution of models as $p_{0}$ and the posterior distribution after having observed $L_{1}$ through $L_{t}$ as $p_{t}$. Then, $p_{T}$ is given by:

$$ $p_{T}(x)=Z_{T}p_{0}(x)\Pi_{t=1}^{T}P(L_{t}|x)=Z_{T}p_{0}(x)e^{-\sum_{t=1}^{T}L _{t}(x)}$ (2) $$

where $Z_{T}$ is a normalization factor that ensures $p_{T}$ integrates to $1$. Observe that this posterior places high density in regions where the summed loss is low.

Particle filters enable the computation of $p_{T}(x)$ by incrementally computing estimates of $p_{t}(x)$. Specifically, given $p_{t}(x)$, we may compute $p_{t+1}(x)$ as:

$$ $p_{t+1}(x)=\frac{p_{t}(x)P(L_{t+1}|x)}{\int p_{t}(x^{\prime})P(L_{t+1}|x^{ \prime})dx^{\prime}}=\frac{p_{t}(x)e^{-L_{t+1}(x)}}{\int p_{t}(x^{\prime})e^{- L_{t+1}(x^{\prime})}dx^{\prime}}$ (3) $$

This Bayesian update equation may often be intractable to compute exactly, particularly when $p_{t}$ does not have a known parametric form. Instead of tracking $p_{t}$ exactly, particle filters track an estimate $\hat{p}_{t}$ instead:

$$ $\hat{p}_{t}(x)=\sum_{i}w_{t}^{(i)}\delta(x-x^{(i)}_{t})$ (4) $$

where $\delta$ is a delta function, $x^{(i)}_{t}$ represents the $i$th particle at time $t$ and $w_{t}^{(i)}$ represents the weight of the particle at time $t$. Each particle filter then has a different method of estimating the Bayesian update of Equation 3. After all updates are complete, an ensemble of particles is available, each of which is an estimate of the global minimizer of $\sum_{t=1}^{T}L_{t}(x)$. We denote the output distribution of a particle filter initialized at $\hat{p}_{0}$ trained on a sequence of loss functions $L_{1},...L_{T}$ as $\hat{p}_{0}[L_{1},...L_{T}]$.

Since particle filters aim to approximate Bayesian updates, we suppose that each update outputs a set of particles close to the true posterior. To formalize this, suppose that there exists a symmetric, non-negative discrepancy measure $D(p,q)$ that satisfies the triangle inequality:

$$ $D(p,q)\leq D(p,r)+D(r,q)$ (5) $$

for all $p,q,r$. Furthermore, suppose $D(p,p)=0$ for all $p$.

Now, suppose that the particle filter satisfies the following two conditions:

$$ $D(\hat{p}[L],\hat{q}[L])\leq CD(\hat{p},\hat{q})$ (6) $$

and

$$ $D(\hat{p}[L],\frac{p(\cdot)e^{-L(\cdot)}}{\int p(x)e^{-L(x)}dx})\leq CD(\hat{p },p)+\epsilon$ (7) $$

for some constants $C$ and $\epsilon$. $C$ can be interpreted as the estimation error propagating from the previous iteration, while $\epsilon$ can be viewed as error between each particle filter update and the true Bayesian update. This allows the discrepancy at time $T$ to be bounded as:

$$ $D(\hat{p}_{0}[L_{1},...L_{T}],\frac{p(\cdot)e^{-\sum_{t}L_{t}(\cdot)}}{\int p( x)e^{-\sum_{t}L_{t}(x)}dx})\leq C^{T}D(\hat{p}_{0},p_{0})+\epsilon\frac{C^{T}- 1}{C-1}$ (8) $$

This decomposes the discrepancy at time $T$ as a term depending on the initial discrepancy $D(\hat{p}_{0},p_{0})$ and a term depending on the incremental discrepancy $\epsilon$.

### 3.2 Permutation-invariance

Next, we demonstrate that particle filters are approximately permutation-invariant: they produce an output that is nearly invariant to the ordering of loss functions $L_{t}$. We show that $\hat{p}_{0}[L_{1},...L_{T}]$ is similar to $\hat{p}_{0}[L_{\sigma_{1}},...L_{\sigma_{T}}]$ where $\sigma_{1},\sigma_{2},...\sigma_{T}$ is a permutation of $1,2,...T$.

###### Theorem 1 .

Suppose $\sigma_{1},\sigma_{2},...\sigma_{T}$ is a permutation of $1,2,...T$ such that $N$ swaps of adjacent elements are required to convert $\sigma_{1},\sigma_{2},...\sigma_{T}$ to $1,2,...T$. Denote the initialized particle filter as $\hat{p}_{0}$. Then,

$$ $D(\hat{p}_{0}[L_{1},...L_{T}],\hat{p}_{0}[L_{\sigma_{1}},...L_{\sigma_{T}}]) \leq 2N\epsilon C^{T-2}\frac{C^{2}-1}{C-1}$ (9) $$

See Appendix A for a proof. This result demonstrates that particle filters are approximately permutation invariant, especially over small sequences. Standard learning algorithms such as gradient descent are notably not permutation-invariant: they tend to be highly dependent on the ordering of data points. Permutation-invariance enables learning algorithms with less stochastic outputs: in a perfectly permutation-invariant particle filter, the only potential sources of randomness are the initial selection of particles and the randomness in the particle filter updates themselves.

###### Theorem 1 .

### 3.3 Avoiding catastrophic forgetting and loss of plasticity

Now, we demonstrate that particle filters naturally avoid catastrophic forgetting and loss of plasticity. Catastrophic forgetting can be formalized in our framework as the phenomenon where a learning algorithm trained on a sequence of losses $L_{1},...L_{T}$ performs poorly on the earlier losses it is trained on. Similarly, loss of plasticity corresponds to performing poorly on later losses. We provide an upper bound on the loss at any point in training:

###### Theorem 2 .

Suppose that all loss functions are bounded in range $[0,M]$. Suppose that there exists a constant $k$ such that for all loss functions $L$ and distributions $p,q$:

$$ $\mathbb{E}_{x\sim p}[L(x)]-\mathbb{E}_{x\sim q}[L(x)]\leq kD(p,q)$ (10) $$

Also, suppose the particle filter guarantees:

$$ $\mathbb{E}_{x\sim\hat{p}[L]}[L(x)]\leq\beta\mathbb{E}_{x\sim\hat{p}}[L(x)]$ (11) $$

for all $L$ and $\hat{p}$. Then,

$$ $\mathbb{E}_{x\sim\hat{p}_{0}[L_{1},L_{2},...L_{T}]}[L_{i}(x)]\leq\beta M+2kT \epsilon C^{T-2}\frac{C^{2}-1}{C-1}$ (12) $$

See Appendix B for a proof. We make two key assumptions in this theorem: the difference in average loss under two different distributions can be bounded by $D$ and that each step in the particle filter reduces the loss on which it is trained by at least a fixed factor. We believe that the first assumption may in many cases be reasonable if the loss function is sufficiently slow-changing: small changes in the distribution over $x$ should not change the average loss value. The second assumption may also be reasonable under many settings for effective particle filters as well as other standard learning algorithms; with a fixed loss function, it corresponds to a linear convergence rate. Gradient descent, for example, satisfies this assumption on loss functions satisfying the Polyak-Łojasiewicz inequality. The resulting bound on the loss $L_{i}$ guarantees that the performance can be no worse than if there had only been a single update $L_{i}$ (yielding loss at most $\beta M$) plus an additional error term.

###### Theorem 2 .

### 3.4 Gradient-based particle filter

Given the desirable properties of particle filters in learning problems, here, we describe a particular particle filter well suited to the high-dimensional spaces found in most machine learning settings.

Suppose that our particle filter’s particle distribution $\hat{p}_{t}(x)=\sum_{i}w_{t}^{(i)}\delta(x-x^{(i)}_{t})$ represents another distribution $\tilde{p}_{t}(x)$ constructed as:

$$ $\tilde{p}_{t}(x)=Z\sum_{i}w_{t}^{(i)}e^{-\frac{||x-x^{(i)}_{t}||^{2}}{2\sigma^ {2}}}$ (13) $$

where $\sigma$ is a variance parameter and $Z$ is a normalizing constant. Essentially, $\tilde{p}_{t}(x)$ replaces each delta function in $\hat{p}_{t}(x)$ with a isotropic Gaussian. We derive the particle filter’s update on $\hat{p}_{t}(x)$ as an approximation of the optimal Bayesian update applied to $\tilde{p}_{t}(x)$. Observe that given a prior of $\tilde{p}_{t}(x)$ and the loss function $L_{t+1}$, the posterior is proportional to:

$$ $e^{-L_{t+1}(x)}\sum_{i}w_{t}^{(i)}e^{-\frac{||x-x^{(i)}_{t}||^{2}}{2\sigma^{2}}}$ (14) $$

We manipulate this expression until it can be expressed in the form of Equation 13. First, we make a linear approximation of $L_{t+1}$ centered at each particle $x^{(i)}_{t}$:

$$ $L_{t+1}(x)\approx L_{t+1}(x^{(i)}_{t})+\nabla L_{t+1}(x^{(i)}_{t})^{T}(x-x^{(i )}_{t})$ (15) $$

Pulling $e^{-L_{t+1}(x)}$ into the summation and applying this approximation:

$$ $\sum_{i}w_{t}^{(i)}e^{-\frac{||x-x^{(i)}_{t}||^{2}}{2\sigma^{2}}-L_{t+1}(x^{(i )}_{t})-\nabla L_{t+1}(x^{(i)}_{t})^{T}(x-x^{(i)}_{t})}$ (16) $$

Grouping terms:

$$ $\sum_{i}w_{t}^{(i)}e^{-\frac{||x||^{2}}{2\sigma^{2}}+[\frac{x^{(i)}_{t}}{ \sigma^{2}}-\nabla L_{t+1}(x^{(i)}_{t})]^{T}x}e^{-\frac{||x^{(i)}_{t}||^{2}}{2 \sigma^{2}}-L_{t+1}(x^{(i)}_{t})+\nabla L_{t+1}(x^{(i)}_{t})^{T}x^{(i)}_{t}}$ (17) $$

Completing the square in the exponent:

$$ $\sum_{i}w_{t}^{(i)}e^{-\frac{||x-x^{(i)}_{t+1}||^{2}}{2\sigma^{2}}}e^{\frac{|| x^{(i)}_{t+1}||^{2}}{2\sigma^{2}}-\frac{||x^{(i)}_{t}||^{2}}{2\sigma^{2}}-L_{t +1}(x^{(i)}_{t})+\nabla L_{t+1}(x^{(i)}_{t})^{T}x^{(i)}_{t}}$ (18) $$

where $x^{(i)}_{t+1}=x^{(i)}_{t}-\sigma^{2}\nabla L_{t+1}(x^{(i)}_{t})$. Simplifying the constant terms:

$$ $\sum_{i}w_{t}^{(i)}e^{-\frac{||x-x^{(i)}_{t+1}||^{2}}{2\sigma^{2}}}e^{\frac{ \sigma^{2}||\nabla L_{t+1}(x^{(i)}_{t})||^{2}}{2}-L_{t+1}(x^{(i)}_{t})}$ (19) $$

Observe that under our linear approximation, $L_{t+1}(x_{t+1}^{(i)})=L_{t+1}(x^{(i)}_{t})-\sigma^{2}||\nabla L_{t+1}(x^{(i)}
_{t})||^{2}$. Thus, we may write the expression as:

$$ $\sum_{i}w_{t}^{(i)}e^{-\frac{||x-x^{(i)}_{t+1}||^{2}}{2\sigma^{2}}}e^{-\frac{L _{t+1}(x_{t+1}^{(i)})+L_{t+1}(x^{(i)}_{t})}{2}}$ (20) $$

Finally, we define $w_{t+1}^{(i)}=w_{t}^{(i)}e^{-\frac{L_{t+1}(x_{t+1}^{(i)})+L_{t+1}(x^{(i)}_{t})
}{2}}$ to arrive at our final approximation of the posterior:

$$ $\sum_{i}w_{t+1}^{(i)}e^{-\frac{||x-x^{(i)}_{t+1}||^{2}}{2\sigma^{2}}}$ (21) $$

We represent this posterior with particles $x^{(i)}_{t+1}$ and respective weights $w^{(i)}_{t+1}$.

We summarize the update equations of this particle filter below:

$$ $x^{(i)}_{t+1}=x^{(i)}_{t}-\sigma^{2}\nabla L_{t+1}(x^{(i)}_{t})$ (22) $$

$$ $w_{t+1}^{(i)}=w_{t}^{(i)}e^{-\frac{L_{t+1}(x_{t+1}^{(i)})+L_{t+1}(x^{(i)}_{t}) }{2}}$ (23) $$

Algorithm 1 shows the full pseudocode of the filter. For simplicity, we do not normalize the weights of the particles at each iteration; this can be done once at the end of training. Intuitively, this filter updates the positions of the particles with gradient descent, but reweights the particles based on their performance at the old and new points, with lower-loss particles weighted higher. Like gradient descent and other gradient-based optimization procedures, this particle filter is well suited for optimization on high-dimensional spaces, while retaining the properties of a particle filters outlined in the prior sections such as permutation-invariance and avoidance of catastrophic forgetting. Figure 1 illustrates how our method converges to well-performing regions of the parameter space over time.

Figure: Algorithm 1 Gradient-based particle filter

Figure: Figure 1: Illustration of how our particle filter converges to well-performing regions of the parameter space over the course of training on SplitMNIST. The plot is constructed by using tSNE to map the particles into two dimensions, then representing each particle with a unimodal Gaussian of fixed variance.
Refer to caption: extracted/5964499/figures/twoDensityDistribute.jpg

#### Theoretical guarantees

Observe that this particle filter is built on gradient descent; thus it inherits the theoretical convergence guarantees of gradient descent. In particular, unlike gradient-free particle filters, this approach is suited for high-dimensional spaces just as gradient-based optimization methods require many fewer iterations relative to gradient-free methods in high-dimensions. What separates this particle filter from simply a model average of $N$ models independently trained with gradient descent? Unlike a simple model average, this approach retains the Bayesian estimation properties of a traditional particle filter; namely, its output is an approximation of the Bayesian posterior. This allows it to maintain the desirable properties of particle filters described earlier.

Next, we demonstrate that the particle filter indeed maintains fidelity to the Bayesian updates it is based on. Specifically, we demonstrate in a simplified setting that the given two particles with the same prior probability at initialization, the particle filer produces an output exactly matching the probability ratios of the true posterior at the final particle locations:

###### Theorem 3 .

Suppose particles $x^{(i)}$ and $x^{(j)}$ are initialized with the same prior probability:

$$ $p_{0}(x^{(i)}_{0})=p_{0}(x^{(j)}_{0})$ (24) $$

Furthermore, suppose that all loss functions are linear:

$$ $L_{t}(x)=g_{t}^{T}x+b_{t}$ (25) $$

Then,

$$ $\frac{p_{T}(x^{(i)}_{T})}{p_{T}(x^{(j)}_{T})}=\frac{w^{(i)}_{T}}{w^{(j)}_{T}}$ (26) $$

See Appendix C for a proof. This theorem guarantees that the particle filter indeed maintains weights in accordance with the true posterior distribution. Thus, it achieves the best of both worlds: it can perform optimization in high-dimensions while also approximating Bayes optimal solutions.

###### Theorem 3 .

## 4 Experiments and Results

In this section, we empirically validate the permutation-invariance of our gradient-based weighted particle filter (hereafter referred to as the weighted particle filter or WPF) and demonstrate its effectiveness in mitigating catastrophic forgetting and loss of plasticity across continual and lifelong learning benchmarks.

#### Continual Learning Experiments:

We evaluate our weighted particle filter on continual learning benchmarks SplitMNIST (LeCun & Cortes, ), SplitCIFAR100 (Krizhevsky, ), and ProcGen (Cobbe et al., ). SplitMNIST is divided into 5 ”super class” splits, and SplitCIFAR100 into 20, both for class-incremental learning. For ProcGen, we use image-action trajectory datasets from the games Starpilot, Fruitbot, and Dodgeball, partitioned into 15 levels sampled using the hard distribution shift mode, similar to Mediratta et al. ().

We compare our weighted particle filter against established continual learning methods: Synaptic Intelligence (SI), Elastic Weight Consolidation (EWC), and Learning Without Forgetting (LWF) (Zenke et al., ; Kirkpatrick et al., ; Li & Hoiem, ). Since our particle filter is architecture-agnostic, we also combine it with SI, EWC, and LWF to evaluate their joint effectiveness.

In the ProcGen continual learning experiments, we use a supervised behavior cloning policy as our base model and compare it against other baseline particle filters also using supervised behavior cloning. All methods are implemented with identical architectures and learning parameters to ensure a fair comparison.

After training, we measure the average accuracy/return of both the weighted particle filter and the baseline models across all splits/levels using 10 different shuffled permutations of epoch splits. To assess permutation invariance, we calculate the task-specific variance in accuracy/return across these 10 permutations. These experiments are designed to evaluate the particle filter’s resistance to catastrophic forgetting.

Our weighted particle filter uses 100 particles, with test accuracy evaluated as a weighted average across particles. We compare this approach with three additional particle filter baselines: (1) a standard particle filter, (2) a gradient-based particle filter without weighting, and (3) traditional gradient descent (single particle). This allows us to assess the impact of particle weighting and the benefits of the Bayesian framework. The standard particle filter serves as a benchmark to evaluate performance on high-dimensional problems, and it operates by resampling particles based on their training loss performance. The gradient-based particle filter without weighting (referred to as averaging particles) is included as a baseline to determine the effectiveness of particle weighting. Full implementation details can be found in the appendix.

Table: Table 1: Average accuracy and normalized variance across classes over 10 permutations for the weighted particle filter, standard particle filter methods, continual learning baselines, and continual learning baselines combined with the weighted particle filter on SplitMINST and SplitCIFAR100.

Figure: Figure 2: Average accuracy versus normalized task variance plots for both SplitCIFAR100 and SplitMNIST. The bottom right region of each plot represents the ideal scenario of high accuracy and low task-specific variance.
Refer to caption: x1.png

#### LRL Experiments:

In the lifelong reinforcement learning setting, we adopt the setup from Muppidi et al. () and conduct experiments on the ProcGen games Starpilot, Fruitbot, and Dodgeball. Distribution shifts are introduced by sampling new procedurally generated levels every 2 million time steps. The agent’s performance is evaluated based on the average normalized return over the course of the lifelong experiment. Additionally, we measure the normalized variance across each level for 10 different permutations of the lifelong level sequences.

We evaluate proximal policy optimization (PPO) (Schulman et al., ), along with other LRL methods designed to prevent loss of plasticity—specifically, PPO combined with TRAC (Muppidi et al., ) and PPO combined with EWC (Kessler et al., )—each tested both with and without our weighted particle filter.

### 4.1 Avoiding Catastrophic Forgetting and Loss of Plasticity

#### Performance Against Other Filters:

Tables 1 and 2 provide a summary of the performance comparison between our weighted particle filter and the baseline particle filters in both continual learning and lifelong RL experiments. Our weighted particle filter consistently achieves higher mean accuracy (averaged over classes and permutations) on SplitMNIST, SplitCIFAR100, and ProcGen Behavior Cloning datasets compared to the baseline particle filter, averaging particles, and traditional gradient descent (single particle).

Furthermore, Figure 3 demonstrates that our weighted particle filter is more resistant to loss of plasticity in LRL experiments compared to PPO using gradient descent. This underscores the advantage of incorporating particle weighting into the training process. This effect may align with the conclusions of Lyle et al. (); Sokar et al. (); Muppidi et al. (); Kumar et al. (), suggesting that when a model is not overly specialized to a specific task is better able to adapt to new tasks. In our approach, maintaining multiple particles—some tuned to domain-specific tasks and others oriented towards different sequential tasks—enables the agent to switch to well-performing particles when adapting to new environments, thereby preserving plasticity.

While theoretically, the baseline particle filter has the advantages of a Bayesian approach, because of the lack of gradient-based optimization, it fails. This lack of gradient-based optimization in high dimensions means it is essentially making random guesses, leading to performances that are close to what would be expected by chance. While gradient descent might show improved results in the latest epoch, it typically does so at the expense of previous epochs’ performance. Therefore, when the performance is averaged across all epochs, the result is diminished, approaching close to random chance levels.

#### Performance Compared to and Combined with Continual Learning and LRL Methods:

The results presented in Table 1 indicate that our weighted particle filter not only successfully avoids catastrophic forgetting but also outperforms all other methods in the SplitMNIST setting. In the SplitCIFAR100 dataset, our model closely competes with the top-performing continual learning model, LWF.

Interestingly, the greatest benefit is observed when we combine continual learning methods with our weighted particle filter. In all cases, the addition of the weighted particle filter increases accuracy.

In the lifelong RL experiments, our weighted particle filter consistently outperforms PPO + EWC across all games. Similar to the continual learning experiments, we observe that combining the weighted particle filter with PPO + EWC or PPO + TRAC results in an increase in average normalized return, demonstrating the effectiveness of these combined approaches.

Table: Table 2: Comparison of Average Normalized Return and Variance of Normalized Return for Particle Methods and LRL baselines with Supervised BC and PPO on both continual and lifelong setups of Dodgeball, Starpilot, Fruitbot. Higher return and lower variance is better.

Figure: Figure 3: Mean episode reward curves for the lifelong setups of Starpilot, Fruitbot, and Dodgeball, comparing PPO and PPO with the weighted particle filter. The results indicate that PPO with the weighted particle filter has greater resistance to the loss of plasticity observed in standard PPO.
Refer to caption: x2.png

#### Permutation Invariance

A distinctive feature of our gradient-based weighted particle filter is its permutation invariance. To validate this property, we evaluated the average normalized variances over classes or levels across 10 permutation runs for each experiment and each method in both the continual learning and lifelong reinforcement learning setups. Each run involved training on a different order of class datasets for SplitMNIST and SplitCIFAR100, or on a different order of levels for the ProcGen games.

Tables 1 and 2 show that, in all experiments, our Weighted Particle Filter exhibited lower variance compared to gradient descent. Additionally, when comparing continual learning or lifelong RL methods with and without the particle filter, we observe that the Weighted Particle Filter consistently increased performance/return and reduced variance.

Figure 2 effectively illustrates this relationship in the SplitMNIST and SplitCIFAR100 experiments. The bottom right region of each plot represents the ideal scenario of high accuracy and low task variance. It is evident from both plots that this optimal region is dominated by either the Weighted Particle Filter alone or continual learning methods combined with the Weighted Particle Filter, demonstrating the advantages of our approach.

## 5 Conclusion

Poor permutations of training data, such as strictly ordered minibatches, can lead to catastrophic forgetting and loss of plasticity. To overcome this challenge, we theoretically demonstrated that particle filters can be permutation-invariant, allowing them to mitigate the issues associated with poor ordering of training data. This permutation invariance offers a principled solution to avoiding catastrophic forgetting and preserving plasticity throughout learning.

Our results further highlight the effectiveness of a simple, gradient-based weighted particle filter in continual, lifelong, and permutation-invariant learning. Notably, our particle filter is domain-agnostic, significantly improving performance and reducing performance variance in both lifelong reinforcement learning and supervised continual learning settings. Moreover, our approach shows greater resistance to catastrophic forgetting and loss of plasticity. The success of our method lies in the combination of gradient-based updates, which make it suitable for high-dimensional problems, and Bayesian weight updates. Our approach paves the way for broader applications of particle filter methods in high-dimensional state spaces, particularly in modern machine learning.

## Appendix A Proof of Theorem 1

###### Proof.

We first show that $\hat{p}[L_{1},L_{2}]$ is similar to $\hat{p}[L_{2},L_{1}]$. Observe that applying true Bayesian updates $L_{1}$ and $L_{2}$ to $\hat{p}$ following Equation 3 yields:

$$ $Z\hat{p}(x)e^{-L_{1}(x)-L_{2}(x)}$ (27) $$

for some normalizing constant $Z$, which is invariant to the ordering of the loss functions. By Equation 8, we have:

$$ $D(\hat{p}[L_{1},L_{2}],Z\hat{p}(\cdot)e^{-L_{1}(\cdot)-L_{2}(\cdot)})\leq C^{2 }D(\hat{p},\hat{p})+\epsilon\frac{C^{2}-1}{C-1}=\epsilon\frac{C^{2}-1}{C-1}$ (28) $$

∎

since $D(\hat{p},\hat{p})=0$. Similarly, we have:

$$ $D(\hat{p}[L_{2},L_{1}],Z\hat{p}(\cdot)e^{-L_{1}(\cdot)-L_{2}(\cdot)})\leq \epsilon\frac{C^{2}-1}{C-1}$ (29) $$

By the triangle inequality, we have:

$$ $D(\hat{p}[L_{1},L_{2}],\hat{p}[L_{2},L_{1}])\leq 2\epsilon\frac{C^{2}-1}{C-1}$ (30) $$

Now, we bound the discrepancy between $\hat{p}[L_{1},L_{2}]$ and $\hat{p}[L_{2},L_{1}]$ when we apply $k$ additional updates $L_{3}$ through $L_{2+k}$. By Equation 6, we have:

$$ $D(\hat{p}[L_{1},L_{2},L_{3},...L_{2+k}],\hat{p}[L_{2},L_{1},L_{3},...L_{2+k}]) \leq 2\epsilon C^{k}\frac{C^{2}-1}{C-1}$ (31) $$

We may apply this inequality to bound the discrepancy between particle filter outputs when any two adjacent losses are swapped:

$$ $D(\hat{p}[L_{1},L_{2},...L_{i-1},L_{i},L_{i+1},L_{i+2},...L_{T}],\hat{p}[L_{1} ,L_{2},...L_{i-1},L_{i+1},L_{i},L_{i+2},...L_{T}])\leq 2\epsilon C^{T-2}\frac{ C^{2}-1}{C-1}$ (32) $$

Thus, with $N$ swaps, using the triangle inequality, the discrepancy may be bounded as:

$$ $D(\hat{p}_{0}[L_{1},...L_{T}],\hat{p}_{0}[L_{\sigma_{1}},...L_{\sigma_{T}}]) \leq 2N\epsilon C^{T-2}\frac{C^{2}-1}{C-1}$ (33) $$

###### Proof.

## Appendix B Proof of Theorem 2

###### Proof.

We first bound the difference in loss between $\hat{p}_{0}[L_{1},L_{2},...L_{i-1},L_{i},L_{i+1},L_{i+2},...L_{T}]$ and $\hat{p}_{0}[L_{1},L_{2},...L_{i-1},L_{i+1},L_{i+2},...L_{T},L_{i}]$. By Theorem 1, we have:

$$ $D(\hat{p}_{0}[L_{1},L_{2},...L_{i-1},L_{i},L_{i+1},L_{i+2},...L_{T}],\hat{p}_{ 0}[L_{1},L_{2},...L_{i-1},L_{i+1},L_{i+2},...L_{T},L_{i}])\leq 2T\epsilon C^{T -2}\frac{C^{2}-1}{C-1}$ (34) $$

Applying the bound on the difference of $L$ under different distributions:

$$ $\mathbb{E}_{x\sim\hat{p}_{0}[L_{1},L_{2},...L_{i-1},L_{i},L_{i+1},L_{i+2},...L _{T}]}[L_{i}(x)]-\mathbb{E}_{x\sim\hat{p}_{0}[L_{1},L_{2},...L_{i-1},L_{i+1},L _{i+2},...L_{T},L_{i}]}[L_{i}(x)]\leq 2kT\epsilon C^{T-2}\frac{C^{2}-1}{C-1}$ (35) $$

Now, applying the reduction in loss by training on $L_{i}$:

$$ $\mathbb{E}_{x\sim\hat{p}_{0}[L_{1},L_{2},...L_{i-1},L_{i},L_{i+1},L_{i+2},...L _{T}]}[L_{i}(x)]\leq\beta\mathbb{E}_{x\sim\hat{p}_{0}[L_{1},L_{2},...L_{i-1},L _{i+1},L_{i+2},...L_{T}]}[L_{i}(x)]+2kT\epsilon C^{T-2}\frac{C^{2}-1}{C-1}$ (36) $$

Finally, applying the absolute bound on the loss:

$$ $\mathbb{E}_{x\sim\hat{p}_{0}[L_{1},L_{2},...L_{T}]}[L_{i}(x)]\leq\beta M+2kT \epsilon C^{T-2}\frac{C^{2}-1}{C-1}$ (37) $$

∎

###### Proof.

## Appendix C Proof of Theorem 3

###### Proof.

First, observe that since the losses are linear, particle $i$ at time $t$ has position:

$$ $x_{t}^{(i)}=x_{0}^{(i)}-\sigma^{2}\sum_{\tau=1}^{t}g_{\tau}$ (38) $$

Next, observe that the weight of particle $i$ at the end of training may simply be expressed as the product of all weight updates:

$$ $w^{(i)}_{T}=e^{-\frac{1}{2}\sum_{t=1}^{T}L_{t}(x_{t}^{(i)})+L_{t}(x_{t-1}^{(i) })}$ (39) $$

We omit the normalizing constant for notational convenience. Using the linearity of $L_{t}$ and the update equation $x_{t}^{(i)}=x_{t-1}^{(i)}-\sigma^{2}g_{t}$:

$$ $w^{(i)}_{T}=e^{-\sum_{t=1}^{T}g_{t}^{T}(x_{t-1}^{(i)}-\frac{1}{2}\sigma^{2}g_{ t})+b_{t}}$ (40) $$

Now, expanding $x_{t-1}^{(i)}$ in terms of $g_{t}$:

$$ $w^{(i)}_{T}=e^{-\sum_{t=1}^{T}g_{t}^{T}(x_{0}^{(i)}-\sigma^{2}\sum_{\tau=1}^{t -1}g_{\tau}-\frac{1}{2}\sigma^{2}g_{t})+b_{t}}$ (41) $$

Rearranging terms:

$$ $w^{(i)}_{T}=e^{-\sum_{t=1}^{T}g_{t}^{T}x_{0}^{(i)}+\sigma^{2}\sum_{t=1}^{T}( \sum_{\tau=1}^{t-1}g_{t}^{T}g_{\tau}+\frac{1}{2}g_{t}^{T}g_{t})-\sum_{t=1}^{T} b_{t}}$ (42) $$

Rewriting the double summation:

$$ $w^{(i)}_{T}=e^{-\sum_{t=1}^{T}g_{t}^{T}x_{0}^{(i)}+\frac{1}{2}\sigma^{2}\sum_{ t=1}^{T}\sum_{\tau=1}^{T}g_{t}^{T}g_{\tau}-\sum_{t=1}^{T}b_{t}}$ (43) $$

Rearranging terms again:

$$ $w^{(i)}_{T}=e^{-\sum_{t=1}^{T}g_{t}^{T}[x_{0}^{(i)}-\frac{1}{2}\sigma^{2}\sum_ {\tau=1}^{T}g_{\tau}]-b_{t}}$ (44) $$

Observe that $x_{T}^{(i)}=x_{0}^{(i)}-\frac{1}{2}\sigma^{2}\sum_{\tau=1}^{T}g_{\tau}$; thus, $x_{0}^{(i)}-\frac{1}{2}\sigma^{2}\sum_{\tau=1}^{T}g_{\tau}=\frac{1}{2}(x_{T}^{
(i)}+x_{0}^{(i)})$. Using this and the linearity of $L_{t}$:

$$ $w^{(i)}_{T}=e^{-\frac{1}{2}\sum_{t=1}^{T}L_{t}(x_{0}^{(i)})+L_{t}(x_{T}^{(i)}) }=e^{-\sum_{t=1}^{T}L_{t}(x_{T}^{(i)})-\sigma^{2}\frac{1}{2}\sum_{t=1}^{T}L_{t }(\sum_{\tau=1}^{T}g_{\tau})}$ (45) $$

Next, note that $p_{T}(x_{T}^{(i)})$ is given by:

$$ $p_{T}(x_{T}^{(i)})=p_{0}(x_{T}^{(i)})e^{-\sum_{t=1}^{T}L_{t}(x_{T}^{(i)})}$ (46) $$

where we again omit normalizing constants for convenience. Finally, applying the same equations for particle $j$

$$ $\frac{w^{(i)}_{T}}{w^{(j)}_{T}}=\frac{e^{-\sum_{t=1}^{T}L_{t}(x_{T}^{(i)})- \sigma^{2}\frac{1}{2}\sum_{t=1}^{T}L_{t}(\sum_{\tau=1}^{T}g_{\tau})}}{e^{-\sum _{t=1}^{T}L_{t}(x_{T}^{(i)})-\sigma^{2}\frac{1}{2}\sum_{t=1}^{T}L_{t}(\sum_{ \tau=1}^{T}g_{\tau})}}=\frac{e^{-\sum_{t=1}^{T}L_{t}(x_{T}^{(i)})}}{e^{-\sum_{ t=1}^{T}L_{t}(x_{T}^{(j)})}}=\frac{p_{T}(x_{T}^{(i)})}{p_{T}(x_{T}^{(j)})}$ (47) $$

∎

###### Proof.

## Appendix D Experimental Setup and Details

#### SplitMNIST task:

In this task, our objective is to sequentially address a series of five binary classification tasks derived from the MNIST dataset. These tasks are designed to distinguish between pairs of digits, presenting a unique challenge in each case. The specific pairings are as follows:

- •
Digits 0 and 1 ({0v1})
- •
Digits 2 and 3 ({2v3})
- •
Digits 4 and 5 ({4v5})
- •
Digits 6 and 7 ({6v7})
- •
Digits 8 and 9 ({8v9})

#### Split CIFAR100 Task:

This task involves the sequential solution of 20 different 5-class classification tasks. Each task is associated with a distinct category comprising a specific group of objects or entities. The categories, along with their corresponding class labels, are listed below:

- •
Aquatic mammals: {beaver, dolphin, otter, seal, whale}
- •
Fish: {aquarium fish, flatfish, ray, shark, trout}
- •
Flowers: {orchid, poppy, rose, sunflower, tulip}
- •
Food containers: {bottle, bowl, can, cup, plate}
- •
Fruit and vegetables: {apple, mushroom, orange, pear, sweet pepper}
- •
Household electrical devices: {clock, computer keyboard, lamp, telephone, television}
- •
Household furniture: {bed, chair, couch, table, wardrobe}
- •
Insects: {bee, beetle, butterfly, caterpillar, cockroach}
- •
Large carnivores: {bear, leopard, lion, tiger, wolf}
- •
Large man-made outdoor things: {bridge, castle, house, road, skyscraper}
- •
Large natural outdoor scenes: {cloud, forest, mountain, plain, sea}
- •
Large omnivores and herbivores: {camel, cattle, chimpanzee, elephant, kangaroo}
- •
Medium-sized mammals: {fox, porcupine, possum, raccoon, skunk}
- •
Non-insect invertebrates: {crab, lobster, snail, spider, worm}
- •
People: {baby, boy, girl, man, woman}
- •
Reptiles: {crocodile, dinosaur, lizard, snake, turtle}
- •
Small mammals: {hamster, mouse, rabbit, shrew, squirrel}
- •
Trees: {maple tree, oak tree, palm tree, pine tree, willow tree}
- •
Vehicles 1: {bicycle, bus, motorcycle, pickup truck, train}
- •
Vehicles 2: {lawn mower, rocket, streetcar, tank, tractor}

#### ProcGen Environment:

We use the ProcGen games Starpilot, Dodgeball, and Fruitbot, which employ procedural content generation to create new levels (corresponding to specific seeds) upon episode reset. We specifically use the hard mode to introduce distribution shifts and ensure the tasks are sufficiently challenging for both lifelong reinforcement learning and continual behavioral cloning.

#### Level Characteristics:

- •
Observation: The observation space consists of an RGB image of shape 64x64x3, representing the state of the environment at each time step.
- •
Action Space: The action space is discrete, with up to 15 possible actions depending on the game.
- •
Reward: Rewards are provided in either dense or sparse formats, depending on the specific game.
- •
Termination Condition: A boolean value indicates whether the episode has ended.

#### Offline Data Collection:

- •
Levels Used:
–
Levels [0, 15): These levels are used for collecting trajectories, specifically for the lifelong RL setup. The agent in our BC experiments is trained sequentially, observing level 0 for 2 million steps, followed by level 1, and so on.
- •
Expert Policies Training:
–
We follow a similar setup to Mediratta et al. (). A strong and well-trained PPO policy is used, which was trained for 20 million steps on 200 levels of each game. This approach ensures that the policy generalizes well and acts as a proficient expert agent for collecting trajectories.
- •
Dataset Generation:
–
Expert Dataset: To generate the expert dataset, we rolled out the final checkpoint of the pretrained PPO model (i.e., the expert policy) across the 15 training levels (splits), collecting 100,000 transitions per level.

#### Lifelong RL Setup:

For the lifelong reinforcement learning setup, we followed the same experimental protocol as Muppidi et al. (). In the ProcGen experiments, individual game levels were generated using a seed value as the start_level parameter, which was incremented sequentially to create new levels. Every 2 million steps, a new level was introduced to the agent using the hard distribution mode. To assess permutation invariance, the sequence of start-level seeds was permuted 10 times, providing a diverse set of training orders for evaluation.

#### Model details:

Our Gradient-based particle filter uses 100 particles. Particles are initialized randomly from PyTorch’s nn.module network parameters. A small amount of noise is injected into these parameters in the beginning of training to increase exploration of the solution space. Our Gradient-descent implementation uses the same code, except we initialize the particle filter with only one particle. The averaging particle filter simply takes the average of the accuracies of all of the particles. The baseline particle filter does the following:

- 1.
Resamples particles from the existing pool with probabilities proportional to the exponential of the negative loss associated with each particle. .
- 2.
Applies perturbations to particles, enabling the exploration of the solution space.
- 3.
Updates the weights of the particles based on the new loss.

We have also incorporated three continual learning methods: SI, EWC, and LWF (van de Ven et al., ). Each of these methods has been implemented following the default, method-specific settings as prescribed in the (van de Ven et al., ) code implementation. These three models used a ”pure-domain” setting.

For a detailed implementation of our particle filter, please refer to our code submission.

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 3
presentation: 3
questions: 1. Can you explain the notation $\hat{p}[L]$?
2. Can you explain what do you mean in L147? Does the particle filter verify two competing conditions: estimating the full distribution $p_t$ and estimating the global minimizer of $p_t$? If all particles are estimates of the global minimizer, how come they represent the full distribution? 
3. Why do you think it is possible to verify (6) and (7) for particle filters in high-dimensional problems?
4. Can you comment on exponential nature of your bounds in $T$. Aren't they strictly worse that linear bounds in $T$? Any sublinear regret bounds from online optimization seems to be much tighter for the problem setting at hand.
soundness: 1
strengths: The paper identifies an interesting perspective based on permutation invariance to tackle two challenges observed in sequential learning. Experimentally, the results seem to correlate with this idea.
summary: In sequential learning, the permutation dependency of prevalent optimization algorithms such as gradient descent might suffer from catastrophic overfitting -- severe degradation in performance on earlier tasks -- or loss of plasticity -- limited adaptive capabilities to the new tasks. Nearly permutation-invariant training strategies can address these challenges. This paper proposes particle filters as an instantiation of nearly permutation-invariant training strategies and presents guarantees under some ideal assumption on the quality of the particle filters. To deal with the high-dimensional nature of machine learning papers, a gradient-based particle filter is proposed and evaluated on continual learning and lifelong reinforcement learning.
weaknesses: 1. Idealization of the particle filter updates: it is unclear why (6) and (7) hold for suitable constants. I recommend authors to give some examples so that this is more understandable.

2. The bounds in (8), (9), (12) are all exponential in $T$ and are potentially vacuous. These exponential terms in $T$ are considered as some constants and not discussed at all. I recommend authors to explain why they think these constants are small. As an example, consider Theorem 2 and (12). The loss is upper bounded by $M$ and I don't see why the exponential term is going to be much smaller than $M$ and it seems like (12) is vacuous.

3. It is unclear why the approximate solution in (15) satisfies the Bayesian properties of particle filters. This is stated in L329-331 but no justification isn't given apart from Theorem 3 which is for a very restricted setting. Therefore, I am skeptical that this is any different from N models independently trained with gradient descent.

---

confidence: 3
contribution: 4
initial_rating: 8
presentation: 4
questions: See weaknesses.pure 

Further questions/suggestions:
- It seems that you're providing an initial connection between particle filters and gradient descent which seems to provides a lot of potential for future work in extending these methods, e.g. with adaptive gradients. You note the potential for such future work in the last sentence, but I'm curious if you have any directions for future work that you are particularly excited about?
- L. 256: Under linear approximation, yes, but in practice, this equality is approximate, right? So when defining $w_{t+1}^{(i)}$, do you use the approximation or the exact quantity for $L_{t+1}(x_{t+1}^{(i)})$?
- A minor point: I think formatting table 1 to clearly delineate continual learning baselines versus continual learning baselines + particle filters (e.g. by using two different columns for the baseline alone versus the baseline + PF) would make it easier for the reader to compare them both to each other and to the Particle Methods above.
soundness: 4
strengths: This is an excellent paper. The authors nicely set up the motivation by explaining prevailing challenges in continual learning, clearly explain why particle filters can address this challenge, and then explain their own method. The benchmark experiments nicely demonstrate the strength of their method. Indeed, I believe that this method could inspire a host of follow-up research, further digging into how to combine particle filters and gradient descent-based optimization. I therefore strongly recommend acceptance.

Below are a few parts of papers I appreciated in particular:
- The introduction really clearly sets up the lens of permutation invariance, which provides a great motivation for particle filters.
- Section 3 provides a set of mathematical insights that make this intuition rigorous.
- The introduced algorithm is theoretically principled and I really appreciated that the authors were able to explain it so succinctly in the main text.
- I thought the authors' finding that this method can be combined with a range of other continual learning methods was really insightful and moved beyond a mere benchmark comparison.
summary: The authors introduce a novel particle-filter based method for learning in deep neural networks. This method uses a set of particles and updates each particle using a local first-order approximation to the loss, which theoretically grounds both the updates to the particle itself and its weight with respect to the overall set of particles. This method therefore combines the appealing properties of gradient descent with respect to high-dimensional weight spaces with appealing properties of particle filters, in particular their (approximate) permutation independence. They then demonstrate that this method improves performance on a number of continual and lifelong learning benchmarks, in particular when combined with other continual learning methods.
weaknesses: I have two primary concerns I'd like to see the authors address:

**1 Further related work**

Your method seems related to the use of ensembles for continual learning, e.g. [1-3]. Could you discuss the relation of your paper to this prior work?

**2 Connection between sections 3.2-3.3 and 3.4**

In sections 3.2 and 3.3 you provided a set of guarantees about particle filters under assumptions (6) and (7). You then introduce your own method, but don't explain whether this method meets these assumptions. Could you discuss whether your method meets these assumptions and if so, what these constants are for your method? If it is unknown whether your method meets these assumption, could you make that clear?

1. Rypeść, Grzegorz, et al. (2024). "Divide and not forget: Ensemble of selectively trained experts in Continual Learning." The Twelfth International Conference on Learning Representations. https://openreview.net/forum?id=sSyytcewxe
2. Soutif–Cormerais, A., Carta, A. & van de Weijer, J.. (2023). Improving Online Continual Learning Performance and Stability with Temporal Ensembles. Proceedings of The 2nd Conference on Lifelong Learning Agents. https://proceedings.mlr.press/v232/soutif-cormerais23a.html.
3. Wen, Yeming, Dustin Tran, and Jimmy Ba (2020). "BatchEnsemble: an Alternative Approach to Efficient Ensemble and Lifelong Learning." International Conference on Learning Representations. https://openreview.net/forum?id=Sklf1yrYDr

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 2
questions: - as the particle filter 'replicates' the model N times - how would a system of N times the size perform (potentially with other regularizations)? There are many architectures to make such a system efficient as well (e.g. Mixture of Expert ...). An experiment analyzing the performance as a function of model size (could be for N=1, but should be compared to the method proposed) should be added to clarify this.
soundness: 3
strengths: + the method outperforms, or performs very well against the baselines. In addition, it is shown that method is complementary, and when combined with other methods, can improve their performance.
+ the idea of the paper can be followed well (though due to the complexity of the problem some central derivations are in the appendix)
+ moving towards more real-world problems, the topic of continual learning is an important one to tackle for the community
summary: The paper "Permutation Invariant Learning with High-Dimensional Particle Filters" proposes a particle-filtering inspired continual learning system. A bound is dericed to show that the method is approximately permutation invariant, and a connection to the avoidance of catastrophic forgetting is established. Under the assumpation of Gaussian probability masses around the particles, a gradient-based update is derived. Empirically the method is shown to perform very competitive on two continual learning supervised datasets (SplitMNIST and SplitCIFAR100), and in one reinforcement learnign setting (ProcGen).
weaknesses: - The document should contain a section on Limitations. One thing that should be mentioned is that the algorithm needs N times more memory to keep the model parameters (or compute) for N being the particles, than a simple standard method with N=1. In particular, given that the community moved to very large models, this is a potentially very big limitation. If you made any design choices to overcome this limitation this should be discussed (or metnioned as future work).
- the writing could be made more accessible. Overall the authors do a good job in explicating the ideas. However, the paper would profit from some high-level summaries and previews before more complex topics. For example, I'd recommend adding a summary description of the derivation in section 3.4 (Gaussian approximation of the particles and Taylor approximation + simplification of terms) before actually doing it. It should also be discussed that the algorithm is independently operating on all i (?) - the only coupling is in the ensembling with the weights. This is important for parallelization, which should be discussed. 
- In the abstract, it should be mentioend that the permutation invariance is approximate and not exact. E.g. change the text 'we introduce a novel permutation-invariant ...' to 'we introduce a novel approximately permutation-invariant ...'.
- In Table 1 please make the best performing nubmers bold

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: Apart from aforementioned concerns, there are some specific questions that I wish to hear from authors.

Major doubts:
*    For all conducted experiments, are there any numbers regarding dimension of particles can be reported?
*    Though WPF can be combined seamlessly with other schemes, what is the additional computation cost introduced by flowing multiple particles? From my understanding, the cost increases linearly with number of particles. Any specific runtime comparisons between WPF and baseline methods, as well as how the runtime scales with the number of particles?
*    In Theorem 3, linear loss function is assumed. However, in classification task setup (e.g. SplitMNIST,SplitCIFAR100), I presume the loss is the CrossEntropy loss, am I right? If sequential losses are CrossEntropy losses, then follow-up questions pops up: how to justify probability density matches in this case?  How Theorem 3 relates to the experimental setup with non-linear loss functions?
*    If in Algorithm $1$, we set number of particles $N=1$, is it equivalent to a fixed step size gradient-descent? If so, in experiments, is the benchmark GD implemented with fixed step size? 
*    Following up the previous question, what is the  hyperparameter $\sigma$'s role in controlling the WPF's performance. I didn't find the hyperparameter setup $\sigma$ in Section 4. Please specifiy the number used in experiments and if possible, justify the choice of such numbers.

Minor questions/suggestions:
*    In line 117, what does $L_t\in \mathbb{R}^d\rightarrow \mathbb{R}$ mean? If $L_t$ is a loss function, then please cosider revise the notation.
*    Is BC listed in Table 2 abbreviation of "Behavior Cloning"?
*    Considering adding statement in Figure 3 to indicate the uncertainty band is under one-times standard deviation, two-times standard deviations or three-times standard deviations. Two blue lines is not a good idea either.
*    Line 461, please consider avoid using colloquial statement like "interestingly".
soundness: 2
strengths: The idea to apply particle filter technique is not particularly novel. However, the efficient realization in a high-dimensional parameter space and show its improvement in continual learning setup is a solid contribution. The author conduct extensive experiments to quantitatively justify the claim.
summary: This paper discusses a particle filter scheme (named as WPF) which tentatively provide a solution of continual tasks learning given high-dimensional parametrized models. WPF method reweighs each particle samples by manipulating the posterior distribution of parameters with gradient of task losses which are computed in a sequential order. Authors justify that WPF methods is permutation-invariant, can prevent notorious forgetting behavior and can prevent "loss of plasticity". There are some experiments conducted over SplitMNIST, SplitCIFAT1100 and ProcGen to show that: i) WPF can be a replacement of vanilla gradient-based optimizer and it can combine with loss regularization techniques (SI, LWF, EWC in this paper) to show performance improvement under both CL and LRL environments ii)  WPF effectively reduces the variance under both CL and LRL environments;
weaknesses: The major disappointment is the inconsistency between what authors attempt to justify in theory and what practical experiments reflect. Frist of all, the notation setup in Section 3.1 is leading confusion. In Equation (2)(3), a better statement of the posterior distribution is to define $p_t$ under conventional setup: $p_t(x):= p(x_t|L_{1:t})$, and it will improve the readability. While defining the distribution with sequential observation of loss using $\hat{p}_0[L_1,L_2,\ldots,L_T]$ is valid, the sudden show up of $\hat{p}[L]$ in line 158 is not properly defined. Adding some notation explanations would be appreciated.

Secondly, I personally did not view permutation invariance as a contribution bulletin. Theorem 1 (line 177-182) justify that a distance metric $D$ is bounded by some constant (including $N,\epsilon,T,C$), and these constants do not have any constraints to indicate if the distance metric converges. Throughout the paper authors fail to give one example of a practical formula of metric $D$. Moreover, the author then justify in line 182-183 that Equation (9) implies particle filters are "**approximately** permutation invariant", without any further qualitative analysis. I am expected to see if there are some experiments to justify/support Theorem 1 , but the only statement in line 510-525 is the performance improvement of reweighted particles which convey readers that "variance reduction implies permutation invariant". Please justify why permutation invariant can be verified by computing variance of repetitive experiments. I understand that WPF is capable of achieving better performance and lower variance under 10 different permutations runs, but this has nothing to do with the statement in Theorem 1.

To strengthen the claim, there are two potential improvements. One way is to show that variance-reduced gradient descent scheme (rather than using a fixed-step gradient descent scheme as per baseline), fail to achieve better overall performance over many runs of permutations. One classical variance-reduced gradient descent scheme is  [SVRG](https://papers.nips.cc/paper_files/paper/2013/hash/ac1dd209cbcc5e5d1c6e28598e8cbbe8-Abstract.html). Another way is to assign a computable metric (e.g. Wasserstein distance) and quantitatively validate theorem 1.

Besides, even though I assume all proof is correct, the author fails to bring attention in experiments to see which part of the result reflect that theorem 2 is correct, which is an upper bound of loss. All reported numbers are mean performance by weighted average of model parameters, and its variance accordingly. Moreover, two assumptions stated in theorem 2 is not validated in experiments as well.


Lastly, an equivalent statement of author's implementation, after reading algorithm 1, is: "WPF repeat gradient descent $N$ times, where $N$ is the number of particles, and reweigh each particles by an exponential term along the path." Then, I am expected to see authors justify why this is not considered as an Mixture-of-Expert(MoE) scheme, and authors ignored the existence of MoEs at all. It would be better if, at least, discuss or compare MoE schemes under this continual learning setups.

This paper has some presentation flaws of experimental details as well, please refer to questions for details.

Meta-review: The paper tackles the problem of dependence of training on batch ordering in machine learning. In particular, the authors argue that issues such as forgetting and loss of plasticity can be addressed if the training is invariant to batch ordering. The authors note that the true Bayesian posterior can be decomposed as a product of conditional updates over the data. They then develop a particle filter scheme for approximating the true Bayesian posterior. They show that this method achieves good performance in continual learning and lifelong RL settings.

Strengths:
- The paper is well-motivated and targets an important problem
- The proposed method is simple and makes intuitive sense
- Authors derive new theoretical bounds related to permutation invariance and forgetting with particle filtering methods
- The empirical results reported are promising

Weaknesses:
- The authors state that Bayesian model averaging has not been explored in continual learning. I believe this is not true, the idea that true Bayesian inference would resolve forgetting is very well known and motivated multiple papers, for example see [1, 2, 3, 4]. Similarly, particle evolution methods are not a new idea for approximate Bayesian inference, see e.g. [5, 6]. So the main novelty on the paper is specifically in applying (a possibly novel) particle filtering scheme to Bayesian continual learning.
>  Overall, BMA has not been extensively explored in the context of continual or permutation-invariant learning (Line 103)
- As the reviewers pointed out, the bounds have exponential terms $C^T$, which are potentially vacuous, depending on unknown constants $C$.
- The presentation should be improved. In particular, I (and several reviewers) was confused by the notation $\hat p(L)$. I think the authors should explain carefully what a particle filter is and how its update rule works.
- In the end, the method reduces to training an ensemble of independent SGD solutions, with a weighting scheme.
  + A priori, this seems very unlikely to work. In particular if the batch ordering is the same for all ensemble members, then they all should experience similar levels of forgetting. No weighting on the ensemble members can fix forgetting.
- Possibly most importantly, the experiments do not provide sufficiently strong evidence that the method is working well. In order to trust the results, the experiments should be conducted in a setting that was considered by prior work, with the same architecture.  Causes for concerns:
  + The authors do not specify the architecture they are using.
  + On the image datasets, the absolute performance is very low: 48-80% on SplitMNIST and 19-29% on SplitCIFAR100. 
  + It is not clear to me if the same batch ordering is used across all ensemble components (particles). If that is the case, it is not clear why there is such extreme deviation in performance between ensemble members.
  + It is not clear how exactly the ensemble is evaluated. I am assuming it is the accuracy with averaged predictions, but the paper says
> with test accuracy evaluated as a weighted average across particles (L404)

Decision recommendation: Based on the arguments above, I recommend rejecting the paper in its current form. I believe that the paper could make a strong contribution if the following issues are addressed: (1) clarity of presentation, (2) adding all details on the experiments, (3) careful literature review of related work in Bayesian deep learning, and most importantly (3) significantly improved experimental evaluation.

[1] Online Structured Laplace Approximations For Overcoming Catastrophic Forgetting
Hippolyt Ritter, Aleksandar Botev, David Barber

[2] Continual Learning Using Bayesian Neural Networks
HongLin Li, Payam Barnaghi, Shirin Enshaeifar, Frieder Ganz

[3] Bayesian Incremental Learning for Deep Neural Networks
Max Kochurov, Timur Garipov, Dmitry Podoprikhin, Dmitry Molchanov, Arsenii Ashukha, Dmitry Vetrov

[4] A Unifying Bayesian View of Continual Learning
Sebastian Farquhar, Yarin Gal

[5] Stein Variational Gradient Descent: A General Purpose Bayesian Inference Algorithm
Qiang Liu, Dilin Wang

[6] Bayesian Inference with Anchored Ensembles of Neural Networks, and Application to Exploration in Reinforcement Learning
Tim Pearce, Nicolas Anastassacos, Mohamed Zaki, Andy Neely

Decision: Reject