Rewiring Neurons in Non-Stationary Environments
Authors: Zhicheng Sun
Yadong MU
Venue: NeurIPS 2023 spotlight (2023)
Keywords: continual learning
reinforcement learning
brain-inspired learning
TL;DR: We propose a new rewiring approach for continual reinforcement learning.

Abstract: The human brain rewires itself for neuroplasticity in the presence of new tasks. We are inspired to harness this key process in continual reinforcement learning, prioritizing adaptation to non-stationary environments. In distinction to existing rewiring approaches that rely on pruning or dynamic routing, which may limit network capacity and plasticity, this work presents a novel rewiring scheme by permuting hidden neurons. Specifically, the neuron permutation is parameterized to be end-to-end learnable and can rearrange all available synapses to explore a large span of weight space, thereby promoting adaptivity. In addition, we introduce two main designs to steer the rewiring process in continual reinforcement learning: first, a multi-mode rewiring strategy is proposed which diversifies the policy and encourages exploration when encountering new environments. Secondly, to ensure stability on history tasks, the network is devised to cache each learned wiring while subtly updating its weights, allowing for retrospective recovery of any previous state appropriate for the task. Meanwhile, an alignment mechanism is curated to achieve better plasticity-stability tradeoff by jointly optimizing cached wirings and weights. Our proposed method is comprehensively evaluated on 18 continual reinforcement learning scenarios ranging from locomotion to manipulation, demonstrating its advantages over state-of-the-art competitors in performance-efficiency tradeoffs. Code is available at https://github.com/feifeiobama/RewireNeuron.

Rewiring Neurons in Non-Stationary Environments
Zhicheng Sun, Yadong Mu' Peking University, Beijing, China {sunzc, (rrnnzccmydd)pppp .yddm@pkuuu.....mm.m.mm het.e/ppkuueeeeee..eeeeee edu. cn
Abstract
The human brain rewires itself for neuroplasticity in the presence of new tasks. We are inspired to harmess this key process in continual reinforcement learning, prioritizing adaptation to non-stationary environments. In distinction to existing rewiring approaches that rely pruning dynamic routing, which may limit network capacity and plasticity, this work presents a novel rewiring scheme by permuting hidden neurons. Specifically, the neuron permutation is parameterized to be end-to-end learnable and can rearrange all available synapses to explore a large span of weight space, thereby promoting adaptivity. In addition, introduce two main designs to steer the rewiring process in continual reinforcement learning: first, multi-mode rewiring strategy is proposed which diversifies the policy and encourages exploration when encountering new environments. Secondly, to ensure stability on history tasks, the network is devised cache each learned wiring while subtly updating its weights, allowing for retrospective recovery of any previous state appropriate for the task. Meanwhile, an alignment mechanism is curated achieve better -lstaiittysttability tradcoff' by jointly optimizing cached wirings and weights. Our proposed method comprehensively evaluated on 18 continual reinforcement learning scenarios ranging from locomotion to manipulation, demonstrating its tradeoffs. htt.r///eeeiiioooomaaaaarrrrrrrr htt///eeeiieeeeoonno
advantages over sfate-of-the- competitors in performance-efficiene Code is available at https::///// http///gttttbbtt wttp://w.w.ooo.feffffefe.o....ee www.a//eooaaaaaaaaeeeaaoaaaaeeaa
1 Introduction
Despite remarkable advances in artificial neural networks, they have yet emulate the adaptivity of the human brain in realistic, non-stationary environments. A plausible explanation for this disparity from neuroscience is that the brain gains additional structural plasticity by constantly rewiring itself In fact, with extensive rewiring. the brain could even confront more adverse scenarios such stroke [TT 68 By contrast, the wiring of artificial neural networks is usually optimized for single domain 77, 72 70 and remains fixed thereafter, thus exhibiting limited adaptivity in the real world. This comparison highlights the potential of rewiring neurons in enabling neural networks to continually engage and learn in dynamically evolving environments, which is referred to as the problem of continual reinforcement learning
throughout one's 's lifetime [9] 35 67
53
A few preliminary studies have exploited this rewiring process in deep learning with other research scopes. They rewired neural networks through pruning (37 28 or dynamic routing 43 55 74 to
allow more efficient or input-dependen inference at test time. Following this line of work, mainstay of continual learning methods [41] 32] 233 learned task-specific masks on neuron wiring alleviate catastrophic interference [42] between different tasks. Unfortunately, existing rewiring approaches that rely on selecting a subnetwork are typically prone to capacity shrinkage over time and therefore may not be competent for continual adaptation to new reinforcement environments.
Corresponding author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

reuse weights
permute neurons by (12)(345)
(a) Synaptic level
(b) Network level
Figure 1: Illustration of rewiring neurons via permutation. (a) In micro, permuting neurons allows reusing existing synaptic connections to create functionally new computation units. (b) macro, layerwise permutation of hidden neurons provides extra structural plasticity in continual learning. Compared to alternative approaches, this method enjoys the higher capacity of the full network.
To harness the adaptivity the rewiring process as in the human brain, we propose novel rewiring scheme by neuron permutation, which fully reuses network weights to achieve structural plasticity. motivating example is presented in Figure Ial where instead of modifying synaptic strengths, one can directly permute neurons to populate certain desired functionalities. In more general cases, neuron permutation may serve as a complementary mechanism to weight updates to facilitate the learning process, it encodes numerous variations by rewiring existing synapses. For instance, tiny neural network shown in Fig. encapsulless over 10,000 wirings to be induced by permutation, with the number growing exponentially w.r.t. network depth and even more dramatically w.r.t. width. And thanks to the recent differentiable sorting algorithms 22 50 it can be efficiently parameterized for end-to-end learning and thus integrated for adaptation
scenarios.
In the context of continual reinforcement learning, we introduce two rewiring designs to emphasize exploration and stability respectively, which are the two essential aspects of the problem. To enhance exploration novel environments, multi-mode rewiring strategy is devised. In detail, it promotes policy diversity by randomly selecting rewiring mode at each step, while effectively sharing knowledge across different modes via a distillation loss. To enforce stability on past tasks, we let the network cache each learned wiring and regularize its weight changes, allowing previous states to recovered at low memory cost. Meanwhile, an alignment mechanism is proposed that jointly refines cached wirings and the latest weights, complementing plasticity.
In summary, our contributions are as follows. (1) We propose a novel rewiring scheme by permuting hidden neurons, allowing for additional structural plasticity under non- non-stationarity. (2) To encourage exploration in new environments, a multi-mode rewiring strategy is introduced that increases policy diversity. (3) To ensure stability history tasks, the network caches each learned wiring and aligns with the newest weights. (4) Experiments on continual reinforcement learning scenarios from Brax and Continual World clearly demonstrate the strengths of our method.
2 Related work
Permuting neurons. The effectiveness of neuron permutation has been demonstrated in aligning
differently trained neural networks 38 (59. Subsequent literature [6]
exploited permutation
symmetries of hidden neurons [29] for finding low-loss connectors 13 19 between loss minima.
Further applications of permutation symmetry include designing equivariant neural architectures [46] 77 and facilitating continual learning [47]. Our work is orthogonal to the aforementioned studies in that we use permutation to produce functionally diverse networks, rather than simply aligning them.
The key technique behind permutation learning is differentiable sorting. A number of works 1 10 15 approximated permutation with doubly stochastic matrices and ran the Sinkhorn algorithm Petersen et al. 48 49 cast differentiable sorting as a series of continuous swap operations. However, both methods involve multiple forward iterations, rendering them inefficient for longer permutations.
Therefore, more lightweight alternative via unimodal row stochastic relaxation 22 50 is adopted
our experiments. Among various efforts to embed this technique into the network structure are the first to advocate it for promoting plasticity in continual learning.

---

Continual learning. To address the major issues in continual learning [53] (or lifelong learning (64). such as catastrophic forgetting [4] and the stability dilemma three main approaches and parameter iso-
have been developed: memory replay [SI 58 weight regularization X 39
32 23 Within the last category, Mallya and Lazebnik 41 pioneered the use of pruning [37 28 to freeze important parameters for stability, but at the cost of capacity shrinkage.
lation [54 41
Ke et al. [32 resorted to dynamic routing [43 [55) for sharing knowledge between different tasks. Gurbuz and Dovrolis 23 iteratively pruned and grew neuron connections in sparse neural network to maintain fixed capacity. In comparison, our work leverages all model parameters with no additional pruning step, and can therefore be easily adapted to different scenarios.
Continual reinforcement learning. Since the earliest proposals [53] continual learning has been intertwined with reinforcement learning [6]] for control and robotics. In this intersecting area, a main stream of studies [36 g 71 made specific assumptions on the Markov decision process to
theoretically maximize reward amidst non-stationarity. Several notable works 3 45 52 employed meta-learning [57] for faster adaptation to new environments. There are also counterparts 54 20
that focused on continual learning of policy networks at the parameter level. Along this direction, Rusu et al. (54) proposed to dynamically grow the policy network for incorporating new knowledge. Kaplanis et al. 31 distilled knowledge [30] from previously learned policies to ensure stability, Recently, Gaya et al. [20] suggested that learning a subspace of policies strikes a good balance between model size and performance. Following these efforts, continue to push the performance- efficiency frontier with the proposed neuron rewiring scheme.
3 Method
In this section, we begin with a brief formulation of continual reinforcement learning (Section 33 Then, a novel rewiring approach is introduced in Section 3.2, along with its emphasis on two aspects: exploration (Section (.3) and stability-plastiiity (Section 3.4), both deemed crucial to the problem. Finally, we remark its connection neuroscience observations in Section 3.5
3.1 Problem formulation
Continual reinforcement learning concerns learning over non-stationary environments, which are characterized by a sequence of T tasks. The goal is to obtain a well-performed global policy x (a t)
that takes a state and task identifier t [1.. T input, and outputs distribution over actions To achieve this, a policy network is employed. Following the current literature [69] (20), we adopt a
multi-layer perceptron architecture comprising a series of alternating affine layers W (Wi)
and element wise activation functions a. The policy inference is formulated as:
Y = Wioooo oo c... .o WiX,
where X and Y denote the input and the predicted policy, respectively. This problem poses significant challenges on the policy network, as it must quickly adapt environmental changes through efficient exploration, while not catastrophically forgetting the previously learned policy.
We address the problem from a structural perspective, focusing on the design of the policy network. Unlike many existing approaches of this type that grow separate policy networks for each task at linearly increasing memory cost, our solution involves only two networks: one for the current task and one for the previous task. We will refer to the current network by default, and when necessary, use the superscripts / and 1 to distinguish between these two networks.
3.2 Rewiring via permutation
Inspired by the brain's 's remarkable adaptivity by rewiring itself 35 67 we seek to incorporate similar process into our policy network. Ideally, this would provide additional plasticity to standard weight learning in non-stationary environments. However, the implementation is not trivial due to major architectural differences between the brain and our network. Most notably, the brain exhibits sparse connectivity, whereas our network consists of fully connected layers.
For simplicity, we will omit biases from our presentation, while it is fairly easy to extend the following discussion to include biases.
3

---

1
2
t
sort
align
sort
(a) Learning to rewire
(b) Multi-mode rewiring
(c) Memory of wirings
Figure 2: Illustration of our proposed rewiring framework. (a) The rewiring process, formulated as neuron permutation, is parameterized by vectors indicating the neuron order within each layer, which can be learned via a differentiable sorting algorithm. (b) In the presence of a new task, our network simultaneously learns K different wirings diversify the policy and promote exploration. (c) Each learned wiring is cached and then continually refined to align with the newest weights.
To this end, we propose a novel rewiring scheme that exploits the layered structure the network. While each layer has intricate connections and rewiring within it may seem complex, we take alternative approach by rewiring between layers through neuron permutation, as shown in Fig. Tb This allows reusing existing synapses redirect the information flow within the network, thereby inducing new wirings with varying functionalities. Formally, is described as inserting a series of permutation matrices P = (Pi L-1 into the network architecture:
= WL WOO000 Py-Woo- o.......... oa P.W.X.
(2)
Note that we permute only the hidden neurons layer E [1.. 1], since the input and output formats are assumed to be fixed. This formulation also differs from permutation symmetry works (62)
16 2], as our focus is on populating different wirings rather than equivalent ones.
Then, a differentiable sorting algorithm is employed to learn the permutations, as illustrated in Fig. We follow Prillo and Eisenschlos (50] parameterize each permutation P1 with score vector et. detail, this score vector ty uniquely corresponds to an order 21, which can then be mapped bijectively permutation matrix. Thus, the permutation can be computed forward pass:
2a
Pt I :. 21 argsort(u)) (v))
where I is an identity matrix. Since both the indexing and argsort operators are non-differentiaabe. the following smoothed proxy is used instead during backpropagation:
(- (d((sor(()))1 1o
, 1o
P softmax
4
where d(::)) uses the L1 distance and 1 is an all-ones vector. This way we can backpropagate loss errors through permutation and learn the score vector es in an end-to-end manner. For clarity, the following text will represent the induced wiring as Pt, omitting parameterization details involving et-
In addition, we would like to highlight two aspects of the proposed rewiring scheme:
Parameter efficiency. Neuron permutation is highly parameter-efficien In network of width each layer of permutation learning requires only ((n) parameters. And after training, they can be compressed into discrete order with even smaller memory footprint. This distinguishes our method from incorporating additional linear layers, which introduces a quadratic parameter overhead that is undesirable in a dosource-constraiing continual learning setting.
Adaptivity. Neuron permutation exploits variety of structural variations to facilitate adaptation. There are over 10 1520 potential wirings in our adopted policy network, and the number continues to grow factorially w.r.t. network width and exponentially w.r.t. depth. Moreover, each wiring maps the current network weight to distinct weight point, thus covering large span of weight space. Together with the parameter efficiency, it significantly enhances the adaptivity of our network in non-stationary environments, which will be validated through experiments.

---

3.3 Rewiring for exploration
In applying the proposed rewiring scheme continual reinforcement learning, there are two key challenges that need to be addressed. The first challenge is to efficiently explore the policy space when encountering a novel environment, order to avoid being trapped suboptimal solutions. This notion of exploration has been emphasized in many related areas, such as maximum entropy reinforcement
learning [78 24 25]. unsupervised skill discovery [17]. and few-shot adaptation [34 21
To encourage exploration in new environments, we propose a multi-mode rewiring strategy that maintains set of different wirings to induce diverse policies, shown in Fig. 2b At each step, the agent randomly samples mode to select from the following candidate wirings:
Pt E
}.
(5)
where K is the total number of modes, and the mode is shared across all layers. Also, to ensure an equivalent learning rate, we multiply the learning rate of each wiring by K.
Figure 3 iillustrates the policies induced by multi mode rewiring. Interestingly, these policies naturally exhibit a diverge-and-converge pattern without any constraints on the wirings. That is, they diverge rapidly at first, but then gradually converge to a high similarity. This implies that multi-mode rewiring enables the agent explore various policies before finally settling on some effective ones. Thus, we can safely select one of the wirings for subsequent evaluation and training.
w/o LgL
175
IN
XL droggaee
We further distill knowledge [30] across different wirings for knowledge sharing, which resembles the idea of mu- tual learning (76 27 but in parameter-efficient manner. Kullback- (KL) divergence is used as the loss:
2002
4000
6002
800k
Figure 3: Evolution of Kullback-Leible divergence between different policies in- duced by multi-mode rewiring at K = 3.
(W, P) = Ex [DKL (Te (+Is)]
(6)
where TT is the policy induced by the k-th wiring, and we only backpropagate through one wiring to save computational cost. As shown in Fig. 3l it significantly speeds up the convergence within modes.
3.4 Rewiring for stability- -laatityyll
The second major challenge in continual reinforcement learning is the mubility-plasticin dilemma 77. where an agent could either suffer severe performance degradation on history tasks due to catastrophic interference [42], or fail to adapt to new environments if an overly rigid constraint is imposed to mitigate forgetting. This requires careful maintenance of old policies when acquiring new ones.
Caching each wiring. Our base solution is to cache each learned wiring and regularize the weights, as shown in Fig. 2c Intuitively, this would allow recovery of both previous weights and wirings. thereby preserving the learned policies. On the wiring side, thanks to the parameter efficiency of permutation, it yields a very small storage overhead (less than 0.5% per wiring in our network) compared to caching the entire network. On the weight side, a typical choice would be L2 regularization, which has proven mitigate forgetting well in many continual reinforcement learning scenarios (69 20]
Leeg (W')
Σ W WW--w--'1 112
(7)
However, such a scheme with weight regularization is usually prone to the stability-plasticity dilemma mentioned above, since the plasticity of the weights is largely reduced by the weight regularizer.
Aligning wirings with weights To address the reformulate the problem into the misalignment between the latest weights and cached wirings. For example, the forgetting since the last training stage can be reproduced by substituting the newest weights Wi' and the last cached wiring p into the inference procedure in Eq. (
dilemma in weight regularization,
Y ... - o....
(8)
5

---

Instead of framing it as a shift in the weight space (as in EWC [3], SI [7],, etc.), we view it as the misalignment between updated weights and fixed wiring. This provides new perspective on the loss of plasticity: existing schemes always align the weights to the wiring, while the wiring remains fixed, resulting in an overly rigid constraint. So we ask, why not align the wiring to the weights?
Inspired by this, we propose a new alignment mechanism that jointly refines the wiring and the weights align with each other. Specifically, the optimization of each cached wiring is achieved by inserting two wiring adapters P/ and P." before and after it. In this way, the wiring in Eq. (8) is adapted to:
Y 00.0000000 P/P! P
Wi o...X. X.
adupters P!-
Note that the element-wise activation functions can be interchanged with permutation layers, yielding an equivalent form in which the adapters could alternatively be interpreted as adapters on the weights:
Y = ...... .0. P
P P/_ 1...X X.
(10)
adapters on W.F
From here, we can switch back from the misalignment interpretation to the weight shift interpretation. In order to mitigate forgetting, the adapted weights should be close to the previous weights. which yields the following regularizer on both the wiring and the weights:
Lsp(W', P', PP) - 'P.
(11)
And after the wiring adapters are learned, they are used refine all cached wirings. For instance, the last cached wiring is updated as:
Pf - P/P' P!T.
(12)
Replacing the previous weight regularizer with this new alignment mechanism improves stability- plasticity in two aspects: (I) The weight constraint is relaxed, since the weight is not necessarily constrained near the previous one. (2) With aligned wirings to counteract weight changes, smaller
coefficient on the regularizer will suffice to mitigate forgetting.
Finally, the proposed rewiring designs are incorporated into the existing reinforcement learning framework (Soft Actor-Critic 25 in our case) using weighted sum of the standard training objective (maximum reward plus maximum entropy) and the two newly introduced losses LKL and Lsp with coefficients a and B. The rest of the training specifications remain unchanged.
3.5
Connection neuroscience
Since our inspiration comes from the rewiring process in the brain, it is natural to review and compare our designs with relevant neuroscience observations. Here, we would like to remark some noteworthy similarities and distinctions between our rewiring approach and the actual process.
Interdependence of weights and wiring. It is elucidated that the weight and wiring changes in the brain are not mutually exclusive, but could take place simultaneously to facilitate plasticity [9]. In our approach, similar line of thinking is followed by updating both weights and wiring during adaptation to new environments. Moreover, we introduce an alignment mechanism (Eqs. (11) and (12) that effectively coordinates these two updates promote stability-plastici
Layerwise locality. In vivo imaging shows that long-range axonal connections are remarkably stable, while local dendritic synapses are more plastic (66). This suggests that rewiring neighboring neurons may be the preferred approach. Given the layered structure of artificial neural networks, we adhere to rewiring neurons within the same layer. It should also be noted that there are other levels of locality to consider 40 but we find that incorporating layerwise locality already suffices.
Absence of sparsity. The main difference between our method and brain rewiring is that we opt for dense and intensive rewiring scheme, whereas the brain rewires neurons sparsely (9. This due to the fact that the human brain has vastly greater number of neurons (estimated at 86 billion 4 compared to the neural network we used (with hundreds of neurons). Hence, to adequately address the problem, need to leverage all available connections to maximize capacity.

---

Table 1: Average performance (f) and model size (.) on Brax scenarios across HalfCheetah, Ant and Humanoid. The results on HalfCheetah and Ant are averaged over 4 different scenarios, and you may find the full results in Appendix B.I. The baseline performances are summarized by Gaya et al. [20] Our method achieves near state-of-the-art performance with less than 60% of the parameters.
HalfCheetah
Ant
Humanoid
Method
Performance
Model size
Performance
Model size
Performance
Model size
FT-1
0.62 = 0.29
1.0
0.52 = 0.26
1.0
0.71 + 0.07
1.0
FT-L2
0.38 = 0.15
2.0
0.78 = 0.20
2.0
0.68 0.28
2.0
PackNet
41
0.85 = 0.14
2.0
1.08 + 0.21
2.0
0.96 + 0.21
2.0
EWC
133
0,43 + 0.24
3.0
0.55 + 0.24
3.0
0.94 + 0.01
3.0
PNN
[54
1.03 0.14
8.0
0.98 + 0.31
8.0
0.98 + 0.26
4.0
SAC-N
1.00
8.0
1.00 0.38
1.00 0.29
4.0
FT-N
1.16 0.20
8.0
0,97 + 0.20
8.0
0.65 + 0.46
4.0
CSP
20
1.27 + 0.27
5.4
1.11 + 0.17
3.9
1.76 + 0.19
3.4
Ours
1.17 + 0.15
2.1
1,22 + 0.11
2.1
1.78 + 0.22
2.0
Experiments
4.1 Experimental settings
Environments. We use 18 continual reinforcement learning scenarios from Brax and Continual World: (1) Brax [8] 20) contains 9 locomotion scenarios over domains: HalfCheetah, Ant and Humanoid. HalfCheetah and Ant each include 4 scenarios focused on forgetting, transfer, robustness, or composi- tionality. Each scenario has 8 tasks with different dynamics and up to million interactions per task. For Humanoid, there is one scenario with sequential tasks and budget of 2M interactions per task. (2) Continual World [69] a manipulation benchmark built on Mera-World World 73] and MuJoCo [65], featuring scenarios with tasks (CW3) and one scenario with 10 tasks (CW 10), both with varying reward function and a budget of 1M interactions per task. More details are provided in Appendix A.
Baselines. Our method is compared to commonly used baselines including PNN [54]. EWC [33]. PackNet [41] and CSP (20), as well as simpler strategies such as finetuning a single model (FT-1), finetuning with L2 regularization (FT-L2), training models from scratch for each task (SAC-N) and finetuning with checkpoints for each task (FT-N). Note that replay-based approaches are not included, since our formulation assumes access carlier replay buffers. The configurations for each method are described Appendix A.2. We report the baseline results from [69 20).
Metrics. Average performance and model size are selected as the evaluation criteria, following 20) The first metric is defined on Brax as the cumulative reward normalized by the base result of SAC-N and on Continual World as the success rate, both averaged over all tasks seen at the end of training. For model size, we use the maximum number of parameters divided by that of single policy (FT-1). Further indicators including forgetting and forward transfer are introduced in Appendix A.3. Unless otherwise noted, the new performances on Brax are averaged over 10 different seeds.
Implementation details. We build on the SaLinA library 12 and adopt Soft Actor-Critic (SAC) 25
with autotuned temperature (26) as the underlying algorithm. Both the actor and the critic are 4-layer perceptions with 256 hidden neurons per layer, while the actor also includes task-specific heads [69]. Their training configurations follow (20). For our method, we choose the new hyperparameters K, a. and B via grid search for each scenario, and provide sensitivity analysis in Section 4.3 The score vectors Eq. 44) are initialized with an arithmetic sequence rescaled to [o. 1], and the temperature by default. Furthermore, the computational cost is discussed in Appendix A.4.
4.2
Main results
The results on Brax are summarized in Table Overall, our approach achieves strong results with a relatively small model size, demonstrating the effectiveness rewiring neurons continually adapting new environments. Specifically, we observe from the results that: (1) Our method yields state-of-the-arr performance on the and Humanoid domains, with less than 60% of the parameters used by the previous best-performing method CSP. (2) On the HalfCheetah domain, where our
7

---

FT-1
FT-1
FT-L2
PackNet
PackNet
1.2
PNN
1.0
Pefommnee
EWC
Peffemszze
SAC-N
1.0
0.3
SAC-N
CSP
0.6
FT-N
0.8
CSP-S
CSP
Ours
0.4
Ours
0.6
Ours-L
Medel stoe
Moveet s00e
(a) HalfChectah scenarios
(b) HalfCheetah/forgeetin scenario
Figure 4: Performance-size tradeoffs for the relatively underperforming HalfCheetah scenarios. To compare similar sizes, CSP-S reduces the network width to 175, while Ours-L expands it to 384. Our approach is on par with the leading method CSP and outperforms many the larger baselines.
Table 2: Success rate(f) Continual World.
Table 3: Ablation studies of each rewiring design the WalfChhetaah/orrgettee scenario. We start with the baseline method FT-L.2 and add designs incrementally. indicates rewiring without caching the wirings.
Method
CW3
CW10
FT-1
0.29
0.10
0.10 1 0.01
FT-L2
0.22 1 0.11
0.48 1 0.05
PackNet
41
0.49 1. 0.12
0.83 1 0.02
Rewire
Multi-mode
Alignment
Performance
EWC
0.32 0.11
0.66
0.67 + 0.32
PNN
S4
0.56 +: 0.14
0.63 + 0.07
0.76 + 0.29
SAC-N
0.60 + 0.14
0.73 + 0.12
0.95 = 0.24
FT-N
0.65 + 0.11
0.77 + 0.09
1.14 - 0.22
CSP
0.65 0.08
0.81 0.06
1.15 + 0.25
Ours
0.67 + 0.09
0.84 + 0.02
1.31 = 0.21
performance is comparatively lower, we manage to surpass multiple baseline methods (PNN, SAC-N and FT-N) that have model size four times larger, and remain par with the state-of-the-art method CSP using less than half the parameters. (3) Across all three domains, our method consistently outperforms existing baselines with similar model size (PackNet and FT-L2) by large margins. (3) Across all three domains, our method consistently outperforms existing baselines.
In addition, Fig. illustrates the performance-size tradeoff of different methods on HalfCheetah scenarios. While these are underperforming scenarios for our method, it still maintains a competitive performance-sin tradeoff against alternative methods. In detail, our work outperforms models of comparable size (PackNet and FT-L2) by a large performance margin of more than 0.3, while also reducing the number of parameters by over 50% compared to the leading method CSP.
We further compare the performance on Continual World Table Moving to another continual reinforcement learning domain with manipulation tasks, our method consistently achieves state-of- the-art performance. The high success rates on both short 3-task scenarios and the longer 10-task scenario confirm the competence of our method in continually adapting new tasks.
4.3 Ablation studies
The following analyses are performed on the HallChectab//forgeting scenario alone, due to the high time cost of conducting ablation experiments all scenarios. We choose this scenario because is relatively representative in terms of reflecting the overall performance, as indicated in Appendix B.
Effectiveness of rewiring. We first verify the effectiveness of rewiring neurons in improving network adaptivity when encountering new environments. This is achieved by comparing the performance evolution to that of the vanilla finetuning approach. As shown by the performance curve in Fig. [5a] our rewiring approach exhibits a substantial speedup in adaptation to an unseen scenario, reaching an average reward of 2000 four times faster than the baseline finetuning method. And afterward, this method achieves a significantly higher final reward over 3000, even without any additional design. These results clearly demonstrate the efficacy of the proposed rewiring approach.

---

Recal
18
Ours
Rewire
Ours
FT
Ours w/o TxEL
2006
410k
6032
KO0K
2004
6071
600R
5002
Skp ofitiiraaion
Step of ancrection
(a) Effectiveness of rewiring and multi-mode
(b) Effectiveness of the distillation loss LxLL
Figure 5: Evolution of performance in the first stage of HalfCheetah/forgetting scenario. Ours denotes the full method, Rewire indicates rewiring without the multi-m mode strategy. FT denotes finetuning. The curves depict the median, with shaded areas showing 95% bootstrap confidence interval around the mean. Our method yields the highest final performance, proving the effectiveness each design.
1.3
Polomam
Ours
PackNet
Figure 6: Trend of the mean performance (over runs) w.r.t. the main hyperparameters: the number of modes K, the weight a for the distillation loss LKL, and the weight / for the alignment loss Lsp-
Effectiveness of multi-mode. The primary goal of multi-mode rewiring encourage exploration, which we validate with the performance curves in Fig. 5a Although the full model exhibits slower initial learning around step 200k due to the divergence multi-mode policies (as shown Fig. B ultimately outperforms upon convergence and achieves final reward of nearly 4000. As a result, the inclusion of the multi-mode design increases the performance by over 0. Table 3 We also compare the exploration efficacy of our multi-mode strategy to an existing exploration method called pink noise [14, with results presented in Appendix B.3 confirming the superiority of our approach. In addition, the ablation study in Fig- 5b shows that incorporating explicit convergence guidance improves both mean and median rewards, justifying the use of the distillation loss LII.-
Effectiveness of memory. This part of the model includes two designs: caching learned wirings, and aligning them with the weight. While both designs effectively address the problem of catastrophic forgetting, we regret that we cannot provide a detailed demonstration here, as the baseline FT-L2 method already achieves a high level of stability. On the other hand, these designs offer the advantage of plasticity, which is evident in the overall performance. According to the results presented in Table the inclusion of the caching of each wiring crucial for our method, as its removal would lead to substantial performance degradation of about O. 19. Similarly, the alignment mechanism contributes significantly to the performance, improving by more than 0. Further studies of the alignment mechanism by comparing the performance evolution are provided in Appendix B.3.
Hyperparameter analysis. Figure deppicts the impact of the newly introduced hyperparameters K, and / on the performance. Overall, regardless of the hyperparameter setting, our method consistently outperforms the baseline PackNet with a similar model size, demonstrating its non- sensitivity to hyperparameters. Specifically, for the number of modes K, the performance peaks at 3 and then declines. We suspect that the performance drop after that is related the reduced number of optimization iterations assigned to each wiring. For the distillation weight the performance increases stably within a reasonable range. For the alignment weight 3,, we mix it proportionally with the weight on the L2 regularizer when the model is under-regularized, in order to have fair comparison between the two regularizers. can be seen that performance increases significantly by incorporating our alignment mechanism, until the model becomes less plastic under a large weight.

---

5 Conclusion
In this work, we exploit the rewiring process of neural networks for continual reinforcement learning. Concretely, we propose the permutation of hidden neurons as a complement to the standard weight learning, and automate with differentiable sorting algorithms. The learned permutation efficiently reuses all existing synapses to explore a multitude of weight variations, thus promoting plasticity. Furthermore, we introduce multi-mode rewiring strategy and an alignment mechanism address the challenges of exploration and -lassiity within the problem. Finally, the effectiveness of our method is validated on a variety of continual reinforcement learning benchmarks.
Acknowledgement:
This work is supported by National Key R&D Program of China 20022ID0160005) and Beijing Natural Science Foundation (Z.19001).
Broader impact. Our research contributes to the progress of biologically inspired neural networks and has potential uses in developing more efficient and adaptive Al systems for real-world environments. However, we recognize that safety-critical scenarios, such as autonomous driving, prioritize stability over other concerns, and system as dynamic ours may not be appropriate for these applications. is essential to consider the adverse effects of implementing our method in such contexts.
Limitations. (1) The experimental setup does not cover replay-based continual learning strategies, nor more complex network architectures. As a result, the compatibility with them remains unclear. (2) Our method relies on task identifiers and cannot handle online task-free settings, which are closer to real scenarios but more difficult. (3) future work, we also need to devise a more biologically plausible approach that modulates rewiring intensity throughout the learning process.
References
[1] Ryan Prescott Adams and Richard S Zemel. Ranking via Sinkhorn propagation. arXiv preprint arXiv:1106.111 1925, 2011.
[2] Samuel K Ainsworth, Jonathan Hayase, and Siddhartha Srinivasa. Git re-basin: Merging models modulo permutation symmetries. In International Conference on Learning Representations, 2023.
[3] Maruan Al-Shedivat, Trapit Bansal, Yura Burda, Ilya Sutskever, Igor Mordatch, and Pieter Abheel. Continuous adaptation via meta-lcarming in nonstationary and competitive environments. In International Conference on Learning Representations, 2018
[4] Frederico AC Azevedo, Ludmila RB Carvalho, Lea T Grinberg, José Marcelo Farfel, Renata EL Ferretti, Renata EP Leite, Wilson Jacob Filho, Roberto Lent, and Suzana Herculano-Houzeel Equal numbers of neuronal and nonneuronal cells make the human brain isometrically scaled-up primate brain. Journal of Comparative Neurology, 513(5):532-5411 2009.
[5] Guillaume Bellec, David Kappel, Wolfgang Maass, and Robert Legenstein. Deep rewiring: Training very sparse deep networks. In International Conference on Learning Representations, 2018.
[6] Johanni Brea, Berfin Simsck, Bernd Illing, and Wulfram Gerstner. Weight-space symmetry deep networks gives rise to permutation saddles, connected by equal-loss valleys across the loss landscape. arXiv preprint (ixi::99070029///1 2019.
[7] Gail A Carpenter and Stephen Grossberg. A massively parallel architecture for a self-organizing neural pattern recognition machine. Computer Vision, Graphics, and Image Processing, 37(1):54-115 1987.
[8] Yash Chandak. Georgios Theocharous, Chris Nota, and Philip S Thomas. Lifelong learning with a changing action set. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 3373-3380, 2020.
[9] Dmitri B Chklovskii, BW Mel, and K Svoboda. Cortical rewiring and information storage. Nature, 78700077)78,-7788 2004.
[10] Marco Cuturi, Olivier Teboul, and Jean-Philippe Vert. Differentiable ranks and sorting using optimal transport. In Advances in Neural Information Processing Systems, pages 6861-6871, 2019.
[11] Numa Dancause, Scott Barbay, Shawn B Frost, Erik Plautz, Daofen Chen, Elena V Zoubina, Ann M Stowe, and Randolph J Nudo. Extensive cortical rewiring after brain injury. Journal of Neuroscience, 15144)910177-0017 2005.
[12] Ludovic Denoyer, Alfredo De la Fuente, Song Duong, Jean- Baptiste Gaya, Pierre- Alexa-dre Kamienny, and Daniel H Thompson. SaLinA: Sequential learning of agents. arXiv preprint arXiv:2110.07910, 2021.
10

---

[13] Felix Draxler, Kambis Veschgini, Manfred Salmhofer, and Fred Hamprecht. Essentially no barriers in neural network energy landscape. In International Conference on Machine Learning, pages 1309-1318. 2018.
[14] Onno Eherhard, Jakoh Hollenstein, Cristina Pinneri, and Georg Martius, Pink noise is all you need: Colored noise exploration in deep reinforcement learning. In International Conference on Learning Representations. 2023.
[15] Marvin Fisenherger, Aysim Toker, Laura I Teixé Florian Bemard, and Daniel Cremers. A unified framework for implicit Sinkhom diffcrentiation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 509-518, 2022.
[16] Rahim Entezari, Hanic Sedghi, Olga Saukh, and Behnam Neyshabur, The role of permutation invariance in linear mode connectivity of neural networks. In International Conference on Learning Representations, 2022.
[17] Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function. In International Conference on Learning Representations, 2019.
[18] C Daniel Freeman, Erik Frey, Anton Raichuk, Sertan Girgin, Igor Mordatch, and Olivier Bachem. Brax-a differentiable physics engine for large scale rigid body simulation. arXiv preprint arXiv:2 13f.:21,...1281,, 2021.
[19] T Garipov, P Izmailov, AG Wilson, D Padoprikhin, and D Vetrov. Loss surfaces, mode connectivity, and fast ensembling of DNNs. In Advances in Neural Information Processing Systems, pages 778998898, 2018.
[20] bean-Baptiste Gaya, Thang Doan, Lucas Caccia, Laure Soulier, Ludovic Denoyer, and Roberta Raileanu. Building subspace of policies for scalable continual learning. In International Conference Learning Representations, 2023.
[21] Jean- Ban-ttttiste Gaya, Laure Soulier, and Ludovic Denoyer. Learning a subspace of policies for online adaptation in reinforcement learning. In International Conference Learning Representations, 2022.
[22] Aditya Grover, Eric Wang, Aaron Zweig, and Stefano Ermon. Stochastic optimization sorting networks via continuous relaxations. In International Conference on Learning Representations, 2019.
[23] Mustafa B Gurbuz and Constantine Dovrolis. NISPA: Neuro-inspired continual learning in sparse networks. In International Conference on Machine Learning, pages 8157-8174, 2022.
adaptation for
[24] Tuomas Haarnoja, Haoran Tang, Pieter Abheel, and Sergey Levine. Reinforcement learning with deep energy-based policies. In International Conference on Machine Learning, pages
2017.
[25] Tuomas Haamoja, Aurick Zhou, Pieter Abbecl, and Sergey Levine. Soft actor-critic: Off pollcy maximum entropy deep reinforcement learning with a stochastic actor. In International Conference on Machine Learning, pages 1861-1870, 2018.
[26] Tuomas Haamoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Schoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Picter Abbeel, and Sergey Levine. Soft actor-critic algorithms and applications. arXiv preprint 1012:v:112..500000 2018.
[27] Bo Han, Quamming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-tcaching: Robust training of deep neural networks with extremely noisy labels. In Advances in Neural Information Processing Systems, pages 8536-8546, 2018.
[28] Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural networks. In Advances in Neural Information Processing Systems, pages 1135- 1143,114,,22 2015.
[29] Robert Hecht-Nielsen. On the algebraic structure of feedforward network weight spaces. In Advanced Neural Computers, pages 129- 135. Elsevier, 1990.
[30] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprini 1rXxiv:503.02533 2015.
[31] Christos Kaplanis, Murray Shanahan, and Claudia Clopath. Policy consolidation for continual reinforce- ment learning. In International Conference on Machine Learning, pages 3242-3251, 2019.
[32] Zixuan Ke, Bing Liu, Nianzu Ma, Hu Xu, and Lei Shu. Achieving forgetting prevention and knowledge transfer in continual learning. In Advances in Newral Information Processing Systems, pages 2021.

---

[33] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinn Barwinska, et al. Overcoming catastrophic forgetting neural networks. Proceedings of the National Academy of Sciences, 114(13):5521-3222 2017.
[34] Saurabh Kumar, Aviral Kumar, Sergey Levine, and Chelsea Finn. One solution is not all you need: Few-shot extrapolation via structured MaxEnt RL. Advances in Neural Information Processing Systems. pages 8198-8210, 2020. 2020.
[35] Jean-Vincems Le Bé and Henry Markram. Spontaneous and evoked synaptic rewiring in the neonatal neocortex, Proceedings of the National Academy of Sciences, 103(35):13214 2006.
[36] Erwan Lecarpentier and Emmanuel Rachelson. Non stationary Markov decision processes, worst-case approach using model-based reinforcement learning. In Advances in Neural Information Processing Systems, pages 7216-7225, 2019.
[37] Yann LeCun, John Denker, and Sara Solla. Optimal brain damage. In Advances in Neural Information Processing Systems, pages 598 088-605,110 1989.
[38] Yixuan Li, Jason Yosinski, Jeff Clune, Hod Lipson, and John Hopcroft, Convergent learning: Do different neural networks learn the same representations? In International Conference Learning Representations, 2016.
[39] Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(12):2935-29) 2944,, 2017.
[40] Ziming Liu, Eric Gan, and Max Tegmark. Seeing is believing: Brain-inspired modular training for mechanistic interpretability. arXiv preprint arXiv:2305.08746 2023.
[41] Arun Mallya and Svetlana Lazebnik. PackNet: Adding multiple tasks to a single network by iterative pruning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7765-7773, 2018.
[42] Michael McCloskey and Neal Cohen. Catastrophic interference in connectionist networks: The sequential lcarning problem. Psychology of Learning and Motivation, 24: 165, 1989.
[43] Mason McGill and Pietro Perona. Deciding how to decide: Dynamic routing artificial neural networks. In International Conference on Machine Learning. pages 2363-2372, 2017.
[44] Gonzalo E Mena, David Belanger, Scott Linderman, and Jasper Snoek. Learning latent permutations with Gumbel-Sinkhorn networks. In International Conference on Learning Representations, 2018.
[45] Anusha Nagabandi, Chelsea Finn, and Sergey Levine. Deep online learning via meta-lcaming: Continual adaptation for model-based RL. In International Conference on Learning Representations, 2019.
[46] Aviv Navon, Aviy Shamsian, Idan Achituve, Ethan Fetaya, Gal Chechik. and Haggai Maron. Equivariant architectures for learning in deep weight spaces. In International Conference on Machine Learning, pages 25790-25816, 2023.
[47] Fidel A Guerrero Peña. Heitor Rapela Medeiros. Thomas Dubail, Masih Aminbeidokhti, Eric Granger. and Marco Pedersoli. Re-basin via implicit Sinkhom differentiation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20237-20246, 2023.
[48] Felix Petersen, Christian Borgelt, Hilde Kuehne, and Oliver Deussen. for scalable sorting and ranking supervision. In International Conference on Machine Learming, pages 1346,8555, 2021.
Differentiable sorting networks
[49] Felix Petersen, Christian Borgelt, Hilde Kühhne, and Oliver Deussen. Monotonic differentiable sorting networks. In International Conference on Learning Representations, 2022.
[50] Sebastian Prillo and Julian Eisenschlos. SoftSort: A continuous relaxation for the argsort operator. In International Conference on Machine Learning, pages 7793-7802, 2020.
[51] Sylvestre- Alliis Rebufti, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. iCaRL: Incre- mental classifier and representation learning. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 2001-2010, 2017.
[52] Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, and Gerald Tesauro. Learning to learn without forgetting by maximizing transfer and minimizing interference. In International Conference on Learning Representations, 2019.
12

---

[53] Mark B Ring. Continual learning in reinforcement environments. PhD thesis, University of Texas at Austin, 1994.
[54] Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Ko- ray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint 1606. 1606.0667,, 2016.
[55] Sara Sabour, Nicholas Frosst, and Geoffrey E Hinton. Dynamic routing between capsules. In Advances in Neural Information Processing Systems, pages 3859-3869, 2017.
[56] Michael E Sander, Pierre Ablin, Mathieu Blondel, and Gabriel Peyré. Sinkformers: Transformers with doubly stochastic attention. In International Conference on Artificial Intelligence and Statistics, pages 3515-3530, 2022.
[57] Jürgen Schmidhuber. Evolutionary principles in self-referential learning, lcarning how to learn: The meta meta-meta...hook. Diploma thesis, Technische Universität Milnchen, 1987.
[58] Hanul Shin, Jung Kwon Lee, Jaehong Kim, and Jiwon Kim, Continual learning with deep generative replay. In Advances in Neural Information Processing Systems, page 2994-3003, 2017.
[59] Sidak Pal Singh and Martin Jaggi. Model fusion via optimal transport. Advances Neural Information Processing Systems, pages 22045-22055, 2020.
[60] Richard Sinkhorn. A relationship between arbitrary positive matrices and doubly stochastic matrices. The Annals of Mathematical Statistics, +5(2):87--879 1964.
[61] Richard S Sutton and Andrew G Barto. Reinforcement Learning: An Introduction. MIT press, 2018.
[62] Norman Tatro, Pin-Y Yu Chen, Payel Das, Igor Melnyk, Prasanna Sattigeri, and Rongjie Lai. Optimizing mode connectivity via neuron alignment. In Advances in Neural Information Processing Systems, pages 15300-15311, 2020.
[63] Yi Tay, Dara Bahri, Liu Yang, Donald Metzler, and Da-Cheng Juan. Sparse sinkhorn attention. In International Conference on Machine Learning, pages 9438-9447, 2020.
[64] Sebastian Thrun Tom Mitchell. Lifelong robot learning. Robotics and Autonomous Systems. 151--2):25-4, 1995.
[65] Emanuel Todorov, Tom Erez, and Yuval Tassa. MuloCo: A physics engine for model-based control. In JEEE/RSJ International Conference on Intelligent Robots and Systems, pages 5026-5033, 2012.
[66] Joshua T Traachtenher,, Brian E Chen, Graham W Knott, Guoping Feng, Joshua R Sanes, Egbert Welker, and Karel Svoboda. Long-term in vivo imaging of experience-dependee synaptic plasticity in adult cortex. Nature, (2006917):7887777 2002.
[67] Arjen Van Ooyen and Markus Butz- Outz-Ostendorf. The Rewiring Brain: A Computational Approach to Structural Plasticity in the Adult Brain. Academic Press, 2017.
[68] lan Winship and Timothy H Murphy. In vivo calcium imaging reveals functional rewiring of single somatosensory neurons after stroke. Journal of Neuroscience, 28(26)6552-666) 2008
[69] Maciej Wolczyk, Michal Zajac, Razvan Pascanu, Lukasz Kuciński, and Piotr Mitoš. Continual world: robotic benchmark for continual reinforcement learning. In Advances in Neural Information Processing Systems, pages 28496 28510 10, 2021.
[70] Mitchell Wortsman, Ali Farhadi, and Mohammad Rastegari. Discovering neural wirings. In Advances in Neural Information Processing Systems, pages 2684-2694, 2019.
[71] Annie Xie, James Harrison, and Chelsea Finn. Deep reinforcement learning amidst continual structured non-stationarity. In International Conference Machine Learning, pages 11 11339-11 1403, 2021.
[72] Saining Xie, Alexander Kirillov, Ross Girshick, and Kaiming He. Exploring randomly wired neural networks for image recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1284-1293, 2019.
[73] Tianhe Yu, Deirdre Quillen. Ryan C Julian, Avnish Narayan, Hayden Shively, Adithya Bellathur, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on Robot Learning, pages 1094 1100, 2019.
13

---

[74] Kun Yuan, Quanquan Li. Shaopeng Guo. Dapeng Chen, Aojun Zhou, Fengwei Yu, and Ziwei Liu. Differentiable dynamic wirings for neural networks. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pages 327-336, 2021.
[75] Friedemann Zenke, Ben Poole. and Surya Ganguli. Continual learning through synaptic intelligence. In International Conference on Machine Learning, pages 3987-3995, 2017.
[76] Ying Zhang. Tao Xiang, Timothy M Hospedales, and Huchuan Lu., Deep mutual learning. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4320 4328, 2018.
[77] Allan Zhou, Kaien Yang, Kaylee Burns, Yiding Jiang, Samuel Sokota. Zico Kolter, and Chelsea Finn. Permutation equivariant neural functionals. In Advances Neural Information Processing Systems, 2023.
[78] Brian D Ziehart, Andrew Maas, J Andrew Bagnell, and Anind K Dey. Maximum entropy inverse reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 1433- 1438, 2008.
[79] Barret Zoph Quoe Le. Neural architecture search with reinforcement learning. International Conference on Learning Representations, 2017.
14


Reviews:
[{"review_id": "fmqYWKxnXl", "reviewer": "Reviewer_CtSV", "summary": "This work studies the continual reinforcement learning setting. This paper proposes a method to permute the neurons in the network, and these permutations allow the exploration of a large part of the weight space. The proposed method caches weights from the prior tasks, which helps to mitigate forgetting. An additional rewiring strategy is proposed to encourage exploration when the task changes.", "questions": "The paper only considers a setting where the agent cannot access replay buffers from the previous tasks. But this seems like an arbitrary choice. If the agent has a replay, why can it not store experience from previous tasks? Can you please elaborate on the rationale for not allowing methods to store experience from previous tasks?", "limitations": "The empirical evaluation is this work is weak. I recommend a new paper by Patterson et al. (2023) to understand how to perform proper empirical analysis in RL.\n\nI will consider raising my score by 3 points or more if the authors improve the empirical evaluation and show that the current results hold with more runs.\nSpecifically, I want the authors to perform at least ten runs (30 would be ideal) for all experiments and show the 95% bootstrapped confidence interval.\n\nPatterson, A., Neumann, S., White, M., & White, A. (2023). Empirical Design in Reinforcement Learning. arXiv preprint arXiv:2304.01315.\n\nEDIT\nI have updated by score based on the new results shared the authors", "rating": 6, "confidence": 3, "soundness": 1, "presentation": 2, "contribution": 3, "strengths": "The proposed method is memory efficient, which is essential for lifelong learning systems as they might encounter an arbitrarily long sequence of tasks. The memory complexity is only O(k*d), where d is the number of neurons in the network and k is the number of tasks. \n\nI like the focus on ensuring continual exploration in new environments. This issue is generally overlooked in the literature, and the emphasis on continual exploration strengthens this work.\n\nThe empirical evaluation is performed on a wide range of environments. This evaluation can provide a detailed assessment of the proposed method.", "weaknesses": "Although the ideas presented in this paper are interesting, the empirical evaluation could be more rigorous. \nAll the experiments are performed with just three random seeds (line 259), meaning the results presented in the paper are not statistically significant. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the very helpful comments. Here are our responses to the concerns raised.\n\n[W1] Evaluation could be more rigorous\n\n* Having reading the paper by Patterson et al. [1], we wholeheartedly concur that our work stands to gain from a more rigorous evaluation. To this end, during the rebuttal period we have exhausted all GPU resources that we can assess to run the full method on all HalfCheetah and Ant scenarios with 10 seeds. This process has consumed over 3 GPU-months. We assure that the remaining experiments will be performed soon and included in the revised paper. In the interim, we present the current results, accompanied by a 95% bootstrap confidence interval (around the mean), as following.\n\n  |                  | HalfCheetah performance | 95% confidence interval       | Ant performance | 95% confidence interval       |\n  | ---------------- | ----------------------- | ------------ | --------------- | ------------ |\n  | Forgetting       | 1.31 $\\pm$ 0.21         | [1.11, 1.40] | 1.46 $\\pm$ 0.15 | [1.36, 1.55] |\n  | Transfer         | 1.42 $\\pm$ 0.19         | [1.29, 1.52] | 0.76 $\\pm$ 0.07 | [0.71, 0.79] |\n  | Robustness       | 1.07 $\\pm$ 0.12         | [0.98, 1.13] | 0.73 $\\pm$ 0.11 | [0.68, 0.81] |\n  | Compositionality | 0.88 $\\pm$ 0.09         | [0.81, 0.92] | 1.95 $\\pm$ 0.11 | [1.87, 2.00] |\n  | Aggregate        | 1.17 $\\pm$ 0.15         | [1.04, 1.24] | 1.22 $\\pm$ 0.11 | [1.15, 1.29] |\n\n\n  And below is the updated comparison table with some baselines in [2]. It can be seen that our method remains competitive even with more training runs.\n\n  |         | HalfCheetah performance | Model size | Ant performance | Model size |\n  | ------- | ----------------------- | ---------- | --------------- | ---------- |\n  | PackNet | 0.85 $\\pm$ 0.14         | 2.0        | 1.08 $\\pm$ 0.21 | 2.0        |\n  | PNN     | 1.03 $\\pm$ 0.14         | 8.0        | 0.98 $\\pm$ 0.31 | 8.0        |\n  | FT-N    | 1.16 $\\pm$ 0.20         | 8.0        | 0.97 $\\pm$ 0.20 | 8.0        |\n  | CSP     | 1.27 $\\pm$ 0.27         | 5.4        | 1.11 $\\pm$ 0.17 | 3.9        |\n  | Ours    | 1.17 $\\pm$ 0.15         | 2.1        | 1.22 $\\pm$ 0.11 | 2.1        |\n\n* In order to enhance the evaluation process, we have made the following efforts: (1) In our rebuttal, the majority of comparative results are accompanied by a 95% bootstrap confidence interval (around the mean) derived from 10 individual runs. (2) The curve plots presented in our work feature both a 95% confidence interval and the median for a comprehensive analysis.\n\n[Q1] Why assume no access to previous replay buffers\n\n*  The primary concern is memory usage, as highlighted in recent literature [2,3]. Gaya et al. [2] reported that for HalfCheetah, the replay buffer used by all methods requires around 1GB per task, while for Humanoid it takes around 15GB per task. Storing experience from previous tasks would result in a large memory overhead, which is undesirable in memory-sensitive situations. Therefore, we adopt the setting without access to previous replay buffers, which also allows direct comparisons with the baselines in [2].\n\nReferences\n\n[1] Patterson, A., Neumann, S., White, M., & White, A. (2023). Empirical design in reinforcement learning. *arXiv preprint arXiv:2304.01315*.\n\n[2] Gaya, J. B., Doan, T., Caccia, L., Soulier, L., Denoyer, L., & Raileanu, R. (2023). Building a subspace of policies for scalable continual learning. In *International Conference on Learning Representations*.\n\n[3] Khetarpal, K., Riemer, M., Rish, I., & Precup, D. (2022). Towards continual reinforcement learning: A review and perspectives. *Journal of Artificial Intelligence Research*, *75*, 1401-1476."}, {"review_id": "zgKlmnrfym", "reviewer": "Reviewer_cvgK", "summary": "The paper proposes a new method for continual reinforcement learning. The idea is to leverage a neuron rewiring mechanism implemented as additional permutations of neurons between the NN weights. In the proposed algorithm, several such permutation sets are maintained, which correspond to different policies that can be used. Having those multiple policies helps to maintain exploration. As a regularization mechanism, a variant of L2 is used for combined NN weight matrices and permutation matrices. Experimental evaluation shows the strong performance of the proposed method, which often matches or outperforms baseline approaches while limiting parameter overhead.", "questions": "[Q1] Are rewiring mechanisms used for actor only, or both actor and critic? Have you tried different options here?\n\n[Q2] Suggestion for an additional experiment (optional): one can see your approach as a way of maintaining a cheap set of ensemble models, each starting from one base model and modifying it with some small adapting module. It would be interesting to compare performance to similar solutions (not previously used in continual RL AFAIK), e.g. https://arxiv.org/abs/2002.06715 \n\nSome additional (minor) suggestions:\n\n[Q3] Figure 5 - it is not super clear what is “Rewire” here.\n\n[Q4] Figure 4 - not very easy to read (e.g. because of 4 different triangle shapes) - maybe colors would be easier to read?\n\n[Q5] - “Validation step” label in some of the figures - a nicer alternative would be something more interpretable like training steps.\n", "limitations": "IMO authors sufficiently covered the limitations in a dedicated paragraph in the last section.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "[S1] Presented solution is quite simple and elegant. Neuron permutations provide both an effective parametrization of the NN and the mechanism for having cheap ensembles.\n\n[S2] Experimental results are compelling, the presented approach is demonstrated to match or beat previous approaches and limit the parameter overhead.\n\n[S3] Ablation experiments and analyses are provided, providing further insights into the method.\n", "weaknesses": "[W1] Although the paper is mostly easy to read, some sections would benefit from editing. Especially Section 3.4 seems wordy and sometimes contains unnecessary or misleading statements, e.g. line 195: “In this situation, one may interpret…” - I have to say I don’t understand this sentence. See also the Questions section of the review for more (mostly minor) suggestions.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We deeply appreciate your valuable suggestions, and we would like to address your main concerns as follows:\n\n[W1] Misleading statements in Section 3.4\n\n* We apologize for any confusion around Section 3.4. Specifically, in line 195, our aim was to present a new interpretation of catastrophic forgetting: Instead of framing it as a shift in weight space (as in EWC, SI [1], etc.), we view it as the misalignment between updated weights and fixed wiring. This perspective led to our proposal to align the wiring with the weights. We will further revise this section to improve its clarity and conciseness.\n\n[Q1] Rewiring details\n\n* The rewiring mechanism is only used for the actor. We have not rewired the critics because many baselines (such as PackNet and PNN) in Brax [2] and Continual World [3] default to standard critics. Rewiring the twin critics would impose significantly higher computational costs than those baselines.\n\n[Q2] Comparison with ensemble methods\n\n* Following your suggestion, we implement BatchEnsemble [4] in continual RL, which learns rank-one \"fast weights\" for adapting to each new task. The results on the HalfCheetah/forgetting scenario are as follows:\n\n  |               | H/F performance | 95% confidence interval       | Model size |\n  | ------------- | --------------- | ------------ | ---------- |\n  | BatchEnsemble | 0.94 $\\pm$ 0.23 | [0.81, 1.08] | 1.1        |\n  | Ours          | 1.31 $\\pm$ 0.21 | [1.11, 1.40] | 2.1        |\n\n  While BatchEnsemble is very efficient, its performance is limited by the expressiveness of rank-one matrices (the main weights are frozen during learning). Our method, on the other hand, jointly learns the weights and the wirings under an alignment scheme (Eq. 11), thus achieving better results.\n\n[Q3] \"Rewire\" in Figure 5\n\n* \"Rewire\" represents rewiring (Section 3.2) but without the multi-mode strategy (Section 3.3). We have revised the figure caption to provide a brief explanation. Please refer to Figure 2 in the global response.\n\n[Q4] Readability of markers in Figure 4\n\n* Thank you for the valuable suggestion. We've now added colors to differentiate the various triangle markers, and the updated version is included as Figure 1a in the global response.\n\n[Q5] More interpretable x label in Figures 3 and 5\n\n* We have replaced \"Validation step\" with \"Step of interaction\" in the global response. The \"training steps\" mentioned in the reviewing comment can be easily inferred from this. For example, on the HalfCheetah/forgetting scenario, the actor is updated every two steps of interaction, while the critics are updated every step.\n\nReferences\n\n[1] Zenke, F., Poole, B., & Ganguli, S. (2017). Continual learning through synaptic intelligence. In *International conference on machine learning* (pp. 3987-3995).\n\n[2] Gaya, J. B., Doan, T., Caccia, L., Soulier, L., Denoyer, L., & Raileanu, R. (2023). Building a subspace of policies for scalable continual learning. In *International Conference on Learning Representations*.\n\n[3] Wołczyk, M., Zając, M., Pascanu, R., Kuciński, Ł., & Miłoś, P. (2021). Continual world: A robotic benchmark for continual reinforcement learning. *Advances in Neural Information Processing Systems*, *34*, 28496-28510.\n\n[4] Wen, Y., Tran, D., & Ba, J. (2020). BatchEnsemble: an alternative approach to efficient ensemble and lifelong learning. In *International Conference on Learning Representations*."}, {"review_id": "qujO0UU2tk", "reviewer": "Reviewer_AnPE", "summary": "The authors propose a new architectural method for continual reinforcement learning. The method relies on training not just the weights of the network, but also the arrangement of the neurons in each layer (implemented as permutation vectors based on a learned score vector for each layer of neurons). \n\nThe method continually adjusts previously saved permutations/rewirings, by ensuring that they would produce similar outputs when applied to the new weights, despite constant changes in the actual weights (IIUC). \n\nIn its current form, the method requires task IDs, but no replay buffers.\n\nVarious experiments suggest that the method is competitive with other established methods with similar requirements, despite its conceptual simplicity.\n\n", "questions": "- Eq. 3: While I understand NumPy notation, some readers might not. Please explain it a bit more.\n\n- Eq. 4: This seems to be the most important equation, but it is difficult to understand. What is the d() function? Apparently it is not the dimensionality of the layers, which is also (confusingly) called d in the next paragraph? Please clarify this.\n\n- Figure 3 shows the divergence and convergence between the various policies, but not their performance. It would be useful to have an additional curve indicating (say) median performance.\n\n- Section 3.4: IIUC, the method here is to continuously adjust the previously found wirings P(t-k), so that they will produce similar outputs as they did with their original weights w(t-k), when applied to the new weights w(t), despite the fact that the weights w(t) keep changing (if so, a one-sentence explanation of this type would be helpful).\n\n- It seems that you are learning W(t) and P’/P’’ together, in parallel, through the loss in equation 11. An alternative would be to simply learn the new W(t) in isolation, then afterwards compute the P’/P’’ “offline”, by applying Eq 11 with frozen W(t) (since Eq. 11 doesn’t seem to require any interaction with the environment). Why not do the latter? \n\n- Can you please confirm whether you simply train one new rewiring (or more precisely one new P’/P’’ pair) per task? That is, does the ’t’ index on Pt/Wt correspond to the tasks? Or is there something else involved?\n\n\n\n\n", "limitations": "The authors note some limitations of their approach in the Conclusion. These do not seem deal-breaking to me, since many of these limitations are shared by other approaches requiring task IDs. ", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The method is novel (to my knowledge) and elegant.\n\n- The results seem interesting.\n\n- The exposition is (mostly) clear", "weaknesses": "I don't see any major weakness, except for the limitations noted by the authors in the Conclusion. I would appreciate some clarifications, as detailed below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for providing valuable feedback. Our responses to your questions are below.\n\n[Q1] NumPy notation in Eq. 3\n\n* Thanks for pointing out the NumPy notation in $\\boldsymbol{I}[\\boldsymbol{z}_l,:]$. It rearranges the rows of the identity matrix $\\boldsymbol{I}$ according to the indices $\\boldsymbol{z}_l$, resulting in a permutation matrix. For example, consider $\\boldsymbol{v}_l=\\begin{pmatrix}0&4&2\\end{pmatrix}^\\top$ and $\\boldsymbol{z}_l=\\mathrm{argsort}(\\boldsymbol{v}_l)=\\begin{pmatrix}0&2&1\\end{pmatrix}^\\top$, then we have:\n  $$\n  \\begin{aligned}\n  \\boldsymbol{P}_l&=\\boldsymbol{I}[\\boldsymbol{z}_l,:]\\\\\\\\\n  &=\\begin{pmatrix}1&0&0\\\\\\\\0&0&1\\\\\\\\0&1&0\\end{pmatrix}.\n  \\end{aligned}\n  $$\n\n[Q2] The $d()$ function in Eq. 4\n\n* $d()$ is a differentiable almost everywhere, semi-metric function (such as $L_1$ distance, $L_2$ distance) applied pointwise [1]. In this paper we use the $L_1$ distance. Below is a demonstration following the previous example, with $\\tau=1$:\n  $$\n  \\begin{aligned}\n  \\hat{\\boldsymbol{P}}_l&=\\mathrm{softmax}\\left(\\frac{-d(\\mathrm{sort}(\\boldsymbol{v}_l)\\boldsymbol{1}^\\top,\\boldsymbol{1}\\boldsymbol{v}_l^\\top)}{\\tau}\\right)\\\\\\\\\n  &=\\mathrm{softmax}\\left(-d(\\begin{pmatrix}0&0&0\\\\\\\\2&2&2\\\\\\\\4&4&4\\end{pmatrix},\\begin{pmatrix}0&4&2\\\\\\\\0&4&2\\\\\\\\0&4&2\\end{pmatrix})\\right)\\\\\\\\\n  &=\\mathrm{softmax}\\begin{pmatrix}0&-4&-2\\\\\\\\-2&-2&0\\\\\\\\-4&0&-2\\end{pmatrix}\\\\\\\\\n  &\\approx\\begin{pmatrix}0.867&0.016&0.117\\\\\\\\0.107&0.107&0.787\\\\\\\\0.016&0.867&0.117\\end{pmatrix}.\n  \\end{aligned}\n  $$\n  The result is a continuous relaxation of the permutation matrix $\\boldsymbol{P}_l$, which allows end-to-end learning.\n\n* As for the naming confusion, we will rename the dimensionality of the layers to $n$ to avoid any ambiguity.\n\n[Q3] Performance of multi-mode policies\n\n* Following your suggestion, we've included performance curves in the global response (Figure 2). Figure 2a demonstrates that, despite the slower initial learning of multi-mode policies from policy divergence, they ultimately outperform upon convergence. Figure 2b shows that incorporating explicit convergence guidance improves both mean and median rewards, verifying the impact of convergence on performance.\n\n[Q4] Interpretation of Section 3.4\n\n* Your understanding is correct. Our method continuously adjusts the cached wirings to counteract weight changes, thereby improving stability. We will add a brief explanation like this in Section 3.4 for clarification.\n\n[Q5] Computing $\\boldsymbol{P}'$/$\\boldsymbol{P}''$ afterwards\n\n* In this case, the learning objective for $\\boldsymbol{W}^t$ (Eq. 11) is computed using outdated $\\boldsymbol{P}'$/$\\boldsymbol{P}''$ (from the last round). Therefore, its performance is likely to be inferior to joint learning, for which we stayed with the original implementation.\n\n[Q6] Training details\n\n* We train one new wiring $\\boldsymbol{P}^t$ and one new $\\boldsymbol{P}'$/$\\boldsymbol{P}''$ pair per task, where the index $t$ on $\\boldsymbol{P}^t$ represents the current task. The rest of the wirings ($\\boldsymbol{P}^{t-k},k>0$) remain frozen and are only updated at the end of each task, so no additional training cost is involved.\n\nReference:\n\n[1] Prillo, S., & Eisenschlos, J. (2020). SoftSort: A continuous relaxation for the argsort operator. In *International Conference on Machine Learning* (pp. 7793-7802)."}, {"review_id": "mbTMmUmTz1", "reviewer": "Reviewer_nWMB", "summary": "The authors propose a means to efficiently expand the capacity of a neural network, namely connection permutations. The approach interleaves permutations matrices between layers of a neural network, such that input-output relationships can be adapted during learning in addition to learning the weight matrices. The authors focus in particular on RL in non-stationary environments, where adapting to new tasks can result in catastrophic forgetting of older tasks. \n\nThe permutations are learned with an existing differentiable approximation to the argsort operator. In addition, the authors propose to (a)  allow the agent to cache permutations from previous tasks; (b) have the agent sample from multiple permutations to encourage exploration; and (c) align cached permutations with the latest weight matrices to ensure that previously learned policies are still consistent with the latest network, preventing catastrophic forgetting. Evaluating on a collection of continual RL tasks, the authors show SOTA or near-SOTA performance while using significantly fewer parameters than other techniques.", "questions": "- See the point above about whether the model indirectly receives information about when to cache the weights. If so, are there means (e.g. information-theoretic surprise) by which the model could detect a task change and initiate weight caching itself?", "limitations": "The authors include a short assessment of Limitations in the Conclusions. I don't foresee any significant negative societal impact.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "Originality:\nThe argsort relaxation is an existing method, and differentiable permutations have been used for sorting inputs and in the attention maps of transformers. The authors also point out that neuron permutations have previously been used to explore network alignment. However, a permutation-based approach to network topology does not appear to have been exploited to manage the stability-plasticity tradeoff. This is a novel insight and a meaningful step beyond the existing literature, to the best of my knowledge.\n\nClarity:\nOverall I found the presentation of the main idea to be clear, with helpful diagrams, although there were some issues on specific points (see below).\n\nSignificance:\nThe application of rewiring alongside weight learning has potential impact beyond non-stationary RL, including in other continual learning tasks, RL with exploration and either RL or non-RL tasks comprised of modular sub-tasks. For this reason, the proposed method is a valuable contribution to the literature.", "weaknesses": "Eq. 11 appears to require caching the previous state of the weights as well as the permutations, which could be a significant memory overhead for large networks.\n\nIt's not clear to me, but it appears that the weights are cached on presenting a new task in a rule-based way; i.e. an external signal to the network indicating that the agent has started a new task. Does this mean that the model technically receives more information than the baselines, in the form of a signal to cache?\n\nOne clarity issue was in the computational neuroscience vs. ML direction of the paper. Based on the title in particular, I expected the focus to be on biologically plausible non-stationary RL / navigation, and it took me well into the Related Work to realize that it was primarily an ML paper. I would probably include a reference to permutation in the the title if you want it to be picked up by the ML and RL communities (e.g. \"Permuting Neurons in Non-Stationary Reinforcement Learning\"). On the other hand, I think the title is fine if the authors primarily expect to go down the comp neuro / bio plausible route with this work as suggested in the Conclusions.\n\nI found the definitions of the ablations to be somewhat confusing in part because they were presented differently in the main text than the paper. The paper starts by describing rewiring and multi-modality together in 3.3, then caching and alignment together in 3.4, while the ablation table has rewiring, then caching, then multi-modality, then alignment. Can you include paragraph sub-headers in the description for each of these interventions in 3.3 and 3.4 to make it easier for the reader to turn back and match up with the ablation table?\n\nCan you re-state what K and (especially) alpha and beta correspond to in the subtitle of Figure 6?\n\nOverall I think the paper is strong. A higher rating (9/10) would likely require a more theoretically rigorous analysis of the impact of permutations on network capacity, a demonstration of lift on tasks outside of RL / non-stationary RL, or both.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the very constructive comments. We would clarify as follows.\n\n[W1] Memory overhead\n\n* Our method requires caching one previous weight $\\boldsymbol{W}^{t-1}$ and all previous permutations. Among them, the weight accounts for most of the overhead, while the permutations prove to be highly parameter-efficient (see lines 142 and 186). Overall, the memory cost is only slightly more than twice the base size, remaining competitive with other existing methods.\n\n[W2] Issue with task identifiers\n\n* The model indeed receives an \"external signal\" indicating the start of a new task, known as the task identifier. However, it is commonly used by the compared methods. For instance, PNN and CSP expand the model upon each new task, while EWC computes the Fisher information matrix at the end of each task. Therefore, our method does not acquire more information than these baselines.\n\n[W3] Direction in computational neuroscience vs. ML\n\n* This paper aims to approach an ML problem from a bio-inspired route, with \"Rewiring Neurons\" in the title underscoring our neuroscience motivation. We've touched on some related discussion in Section 3.5 and the Conclusion, and will continue to revise the paper to make our intent clearer.\n\n[W4] Presentation of the ablation table.\n\n* We apologize for the different order of the ablation table, as it goes from rewiring (Section 3.2), then caching (Section 3.4), then multi-mode (Section 3.3), then alignment (Section 3.4). To help readers locate each design in the main text, we will add two paragraph sub-headers \"Caching each wiring\" and \"Aligning wirings with weights\" in Section 3.4.\n\n[W5] Hyperparameters in Figure 6\n\n* $K$ is the number of modes in multi-mode rewiring (Eq. 5), $\\alpha$ is the coefficient for the distillation loss $L_{\\textrm{KL}}$ (Eq. 6), and $\\beta$ is the coefficient for the regularization loss $L_{\\textrm{SP}}$ (Eq. 11). For clarification, we will re-state the definition of $\\alpha$ and $\\beta$ (originally in line 218) in Figure 6.\n\n[W6-1] Impact of permutation on network capacity\n\n* The permutation layers do not affect the network capacity. Instead, they facilitate exploration of the existing weight space (by enabling traversal from one weight point to another via permutation transforms). This largely enhances the network's adaptivity, as shown in the continual RL experiments.\n\n[W6-2] Demonstration on non-RL tasks\n\n* We have not yet conducted such experiments, but looking beyond RL tasks presents intriguing challenges. Take visual tasks as an example: neuroscience research [1] indicates that synapses in the visual cortex exhibit much higher stability. This could imply a need for new rewiring approaches with more moderate rewiring intensity.\n\n[Q1] Detecting task changes\n\n* This has been investigated in the field of task-free continual learning [2] where task identifiers are absent. For example, Rao et al. [3] suggested maintaining a buffer for poorly modeled samples and then expanding the model when the buffer reaches a critical size. Ardywibowo et al. [4] proposed to detect task changes using an energy-based novelty score.\n\nReferences\n\n[1] Grutzendler, J., Kasthuri, N., & Gan, W. B. (2002). Long-term dendritic spine stability in the adult cortex. *Nature*, *420*(6917), 812-816.\n\n[2] Aljundi, R., Kelchtermans, K., & Tuytelaars, T. (2019). Task-free continual learning. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition* (pp. 11254-11263).\n\n[3] Rao, D., Visin, F., Rusu, A., Pascanu, R., Teh, Y. W., & Hadsell, R. (2019). Continual unsupervised representation learning. *Advances in neural information processing systems*, *32*, 7647-7657.\n\n[4] Ardywibowo, R., Huo, Z., Wang, Z., Mortazavi, B. J., Huang, S., & Qian, X. (2022). VariGrow: Variational architecture growing for task-agnostic continual learning based on Bayesian novelty. In *International Conference on Machine Learning* (pp. 865-877)."}, {"review_id": "IIx1C9VJpv", "reviewer": "Reviewer_NgVj", "summary": "The paper presents a bio-inspired rewiring technique to improve deep reinforcement learning (DRL), especially in continual learning in non-stationary environment. The rewiring is implemented using permutation matrix P for all the hidden layers in MLP. There are several benefits. First, by using a set of different wirings, the agent executes more diverse policy which is claimed to improve exploration. Second, rewiring with a novel regularization trick can alleviate the stability-plasticity dilemma in continual learning. The effectiveness of the second claim is proven by robotic control experiments. The proposed rewiring method achieves SOTA with less network parameters. Ablation studies are also presented for better understanding the technique.\n\n---- post rebuttal -----\n\nI appreciate the authors' reply. I lead toward acceptance.", "questions": "- What's the computational cost and time consumption of your method comparing to others despite having less model parameters? \n- Is your method compatible with evolutionary algorithms if using non-differentiable sorting for P?\n- What's the scalability of the rewiring technique? More specifically, will the performance improve if you make your model larger (comparable with CSP in Table 1)?", "limitations": "The authors have discussed limitation reasonably in the manuscript.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The writing is clear and easy to follow\n- The idea is interesting and well motivated\n- The proposed method is well evaluated with diverse tasks and ablation studies\n- Connections with neuroscience is well discussed", "weaknesses": "- The claim in 3.3 lack supporting evidences. Although ablation studies show a performance degragation without multi-mode, it is unclear how it relates to \"exploration\". There are various ways of improving exploration, e.g., simply using pink noise for action (https://openreview.net/forum?id=hQ9V5QN27eS). If the author would like to claim their methods' advance on exploration, comparison with other methods should be performed. And there should be empirical or theoretical evidence to show such an advantage in terms of \"enable the agents to explore unseen environment (line 168)\" rather than Fig.3 or overall performance.\n\n- The proposed method achives similar performance (I would say no statistically significant difference from CSP) while using less parameters, which is a good thing. However, since the network is small (a few FC layers of width 256),  such an advantage is not obviously important. I believe it would be much more exciting if the authors can show that (1) rewiring largely outperforms CSP with simular model size (could be very small network that can work onsingle chip microcomputer) or (2) performs similar or better than CSP with less energy consumption in tasks that require heavy computation resource.\n\n- Although MLP is a foundamental neural network structure, it is unclear wether the rewiring technique works well for Transformers, CNNs, RNNs, etc.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the very constructive comments. Our responses are provided below.\n\n[W1] Justification for multi-mode strategy\n\n* In Figure 2a of the global response, we compare the exploration efficacy of our multi-mode strategy against pink noise [1]. While the single-mode baseline with pink noise exhibits rapid initial learning, its performance plateaus over time. In contrast, our full method with multi-mode strategy effectively avoids this suboptimal situation and achieves the highest final performance.\n* In addition, we recommend that the reviewer refer to Figure 5 and line 299 in the main paper, where a similar empirical justification for multi-mode rewiring is provided.\n\n[W2] Additional comparison with CSP\n\n* To compare at \"similar model sizes\", we experiment with CSP using a reduced network width of 175 (denoted as CSP-S). This aligns the model size of the two methods on the HalfCheetah/forgetting scenario. As shown in the results below, our method achieves slightly higher mean performance than CSP-S at similar small sizes.\n\n  |       | H/F performance | 95% confidence interval       | Model size |\n  | ----- | --------------- | ------------ | ---------- |\n  | CSP-S | 1.27 $\\pm$ 0.15 | [1.14, 1.32] | 2.3        |\n  | Ours  | 1.31 $\\pm$ 0.21 | [1.11, 1.40] | 2.1        |\n\n[W3] Exploration of network architectures\n\n* Currently, our work is focused on MLP due to its standard use in the continual RL benchmarks, and the observations we've made already provide important insights into the field. Meanwhile, we recognize the value of exploring rewiring techniques in other architectures such as Transformers and CNNs, possibly moving on to datasets like Split ImageNet. We will continue to investigate bio-inspired approaches like rewiring with more sophisticated architectures and tasks in the future.\n\n[Q1] Computational efficiency\n\n* In Appendix A.4, we have shown that our method outperforms CSP in terms of time efficiency. Here is another comparison using multiply-add operations (MACs) on the HalfCheetah/forgetting scenario:\n\n  |      | MACs (M) | Model size |\n  | ---- | -------- | ---------- |\n  | FT-1 | 0.14     | 1.0        |\n  | PNN  | 1.08     | 8.0        |\n  | CSP  | 0.63     | 4.5        |\n  | Ours | 0.48     | 2.1        |\n\n  As can be seen, our method has a lower computational cost than both PNN and CSP, despite the need for two forward passes (to compute the distillation loss $L_{\\textrm{KL}}$ between wirings).\n\n[Q2] Compatibility with evolutionary algorithms\n\n* Thank you for bringing attention to this line of work [2,3]. Integrating evolutionary algorithms into our model appears to be difficult, due to the considerable engineering effort required for their fitness functions. And the efficiency may not be comparable to differentiable sorting. Nevertheless, evolutionary algorithms represent a more biologically plausible solution, since there is no clear evidence that the brain implements differentiable algorithms, so they are still worth exploring.\n\n[Q3] Scalability\n\n* We validate the scalability of our method by expanding the network width to 384 (denoted as Ours-L), which allows a direct comparison with the original CSP results as follows:\n\n  |        | H/F performance | 95% confidence interval       | Model size |\n  | ------ | --------------- | ------------ | ---------- |\n  | Ours   | 1.31 $\\pm$ 0.21 | [1.11, 1.40] | 2.1        |\n  | Ours-L | 1.38 $\\pm$ 0.10 | [1.31, 1.42] | 4.6        |\n  | CSP    | 1.41 $\\pm$ 0.07 | -            | 4.5        |\n\n  The comparison shows that Ours-L has a noticeable performance improvement, closing the gap with CSP. For a more intuitive illustration, please see Figure 1b in the global response.\n\nReferences\n\n[1] Eberhard, O., Hollenstein, J., Pinneri, C., & Martius, G. (2023). Pink noise is all you need: Colored noise exploration in deep reinforcement learning. In *International Conference on Learning Representations*.\n\n[2] Scharnow, J., Tinnefeld, K., & Wegener, I. (2005). The analysis of evolutionary algorithms on sorting and shortest paths problems. *Journal of Mathematical Modelling and Algorithms*, *3*, 349-366.\n\n[3] Bassin, A., & Buzdalov, M. (2020). The (1+($\\lambda$, $\\lambda$)) genetic algorithm for permutations. In *Proceedings of the 2020 Genetic and Evolutionary Computation Conference Companion* (pp. 1669-1677)."}]

Author rebuttal:
We thank all reviewers for the insightful comments, which are important for improving our work. Alongside our individual responses, we have meticulously prepared a PDF file containing figures that effectively address numerous frequently raised concerns. Below is a concise summary of these figures.

* Figure 1: Performance-size tradeoff. It includes additional comparisons with CSP (Reviewer NgVj) from 10 runs (Reviewer CtSV) and improved readability (Reviewer cvgK).

* Figure 2: Evolution of performance for multi-mode policies (Reviewers AnPE and NgVj), featuring 95% confidence interval (Reviewer CtSV) and median (Reviewer AnPE). It also uses more interpretable x labels and a revised caption (Reviewer cvgK).

Meta-review: Meta Review for Rewiring Neurons in Non-Stationary Environments

As summarized by reviewer nWMB, this work proposes a means to efficiently expand the capacity of a neural network, namely connection permutations. The approach interleaves permutations matrices between layers of a neural network, such that input-output relationships can be adapted during learning in addition to learning the weight matrices. The authors focus in particular on RL in non-stationary environments, where adapting to new tasks can result in catastrophic forgetting of older tasks.

The permutations are learned with an existing differentiable approximation to the argsort operator. In addition, the authors propose to (a) allow the agent to cache permutations from previous tasks; (b) have the agent sample from multiple permutations to encourage exploration; and (c) align cached permutations with the latest weight matrices to ensure that previously learned policies are still consistent with the latest network, preventing catastrophic forgetting. Evaluating on a collection of continual RL tasks, the authors show SOTA or near-SOTA performance while using significantly fewer parameters than other techniques.

Most reviewers agree that this work is original, clear, and potentially significant. A clear acceptance.
Decision: Accept (spotlight)