Title: Balancing Plasticity and Stability with Fast and Slow Successor Features
Primary area: reinforcement learning
Keywords: successor features, deep reinforcement learning, synaptic consolidation, plasticity, stability

Abstract: A hallmark of intelligence is the ability to adapt in non-stationary environments, yet deep Reinforcement Learning (RL) agents often struggle in such settings. Most prior studies introduce non-stationarity through abrupt shifts in features or dynamics, whereas real-world changes might be more gradual and continuous. Moreover, existing methods have largely attributed these struggles to either plasticity or stability loss at the level of Q-values, yet it remains unclear how these factors affect representations such as Successor Features (SFs). To address these concerns, we modify existing 3D Miniworld and MuJoCo environments to incorporate continuous non-stationary changes and use them to identify whether poor performance arises from a loss of plasticity or stability. We find that methods preserving stability, such as synaptic consolidation, achieve better performance than those focused on plasticity. Motivated by this finding, and prior evidence that SFs reduce interference in non-stationary settings, we investigate whether SFs provide a better target than Q-values for consolidation. Across both environments, we find that applying a neuro-inspired synaptic consolidation mechanism to SFs yields superior performance. To better understand the benefits of consolidating SFs, we conduct a cross-attention analysis to probe the relative contributions of the consolidation variables. We find that consolidation is most effective when SFs are stabilized across multiple timescales, as different timescales capture complementary aspects of learning. Together, these results show that multi-timescale consolidation of predictive representations is critical for robust RL in continuous non-stationary environments.