Title: Zero-Shot Adaptation of Behavioral Foundation Models to Unseen Dynamics
Venue: NeurIPS 2025
Accepted: no
Keywords: zero-shot reinforcement learning

unsupervised reinforcement learning

successor measure
Abstract: Behavioral Foundation Models (BFMs) proved successful in producing policies for arbitrary tasks in a zero-shot manner, requiring no test-time training or task-specific fine-tuning. Among the most promising BFMs are the ones that estimate the successor measure learned in an unsupervised way from task-agnostic offline data. However, these methods fail to react to changes in the dynamics, making them inefficient under partial observability or when the transition function changes. This hinders the applicability of BFMs in a real-world setting, e.g., in robotics, where the dynamics can unexpectedly change at test time. In this work, we demonstrate that Forward–Backward (FB) representation, one of the methods from the BFM family, cannot distinguish between distinct dynamics, leading to an interference among the latent directions, which parametrize different policies. To address this, we propose a FB model with a transformer-based belief estimator, which greatly facilitates zero-shot adaptation. We also show that partitioning the policy encoding space into dynamics-specific clusters, aligned with the context-embedding directions, yields additional gain in performance. These traits allow our method to respond to the dynamics observed during training and to generalize to unseen ones. Empirically, in the changing dynamics setting, our approach achieves up to a 2x higher zero-shot returns compared to the baselines for both discrete and continuous tasks.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes methods to enable zero-shot adaptation of Behavioral Foundation Models (BFMs) to environments with unseen transition dynamics. The authors identify that a BFM with vanilla Forward–Backward representation (FB) fails to generalize across different dynamics because it conflates experiences from multiple environments, causing “interference” among learned policies. To overcome this, the paper introduces (1) Belief-FB (BFB), which conditions the FB model on a latent belief state of the environment’s dynamics using a transformer encoder, and (2) Rotation-FB (RFB), which further partitions the latent policy space by dynamics by sampling policy vectors from a von Mises-Fisher distribution aligned with the inferred context. Theoretical results show that vanilla FB’s performance degrades as more environments with different dynamics are included, whereas the proposed RFB can remove this dependence. Empirical evaluations on gridworld-like navigation, a pointmass maze, and a MuJoCo Ant with varying wind demonstrate that BFB and RFB achieve substantially higher zero-shot returns than baselines for both seen and unseen dynamics.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n1. The paper is well-structured and clearly written. It includes a comprehensive literature review and clear background info, and the proposed methods are novel and well-motivated. \n2. Convincing experiments have been performed on both discrete and continuous environments with error bars reported. BFB and RFB show large performance improvements over state-of-the-art baselines on multiple tasks for both seen and unseen dynamics. Visualization of latent space and ablation studies on data diversity and context length further confirm the paper’s claims about disentanglement and context capture.\n3. The paper provides regret bounds analysis that shows the worst case regret grows with the number of environments (Theorem 1) and suggests partitioning the latent space into non-overlapping clusters can eliminate that growth depending on the number of training environments k (Theorem 2).\n\n**Weaknesses**\n1. The context encoder $f_{dyn}$ is trained to infer dynamics-related hidden variables from visitation data (although it can be reward-free and generated by random policy) of the test environment. In the zero-shot evaluation, the agent needs to be given a “small batch of transitions” from the new context. In real-world application, obtaining such visitation data from the new environment could still be a challenge. \n2. The scalability of current context encoder $f_{dyn}$ design is limited. Accommodating a larger number of distinct dynamics would likely require (1) increasing the dimensionality of the latent space h to separate all modes, (2) collecting a larger and more diverse offline dataset to cover every new context, and (3) growing the transformer’s capacity to process longer or higher-dimensional input. Together, these raise both sample complexity and compute/memory costs, potentially making real-time inference on resource-constrained platforms infeasible .\n3. Empirical runtime is not reported. Such discussion would further clarify the approach’s computing resource requirement and efficiency.\n\nQUESTIONS:\n1. What future developments do you think could improve the scalability of your methods, so it can handle more complex environments or much longer trajectories?\n2. With insights from this paper, do you think it’s possible to train a single BFM that generalizes not only across different dynamics of the same type of environment, but also across different environment types (e.g. both Randomized-Doors and Randomized-Pointmass)?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

comments: SUMMARY:
The paper investigates zero shot adaption of behavioral foundation models (BFMs) based on successor features. It has two parts: First they identify that the forward backward representation (FB) cannot distinguish between distinct dynamics leading to interference among latent directions that define policies. This can lead to problems for the model to adapt to changes in the dynamics of the task/environment. As a fix for this issues, they propose a transformer based belief estimator (BFB) and an enhancement thereof (RFB). Using different RL tasks, they demonstrate that their two approaches outperforms existing approaches but a substantial margin.

STRENGTHS AND WEAKNESSES:
Strengths:
- The paper is generally well written, figures are high quality, proofs and other details are provided in the Appendix. 
- The paper addresses a relevant problem. 
- The paper demonstrates substantial improvements on the chosen tasks. 
- The paper provides theoretical backup for their claims and observations. 

Weaknesses:
- There could be a bit more discussion whether and how the method can scale to bigger problems and what the challenges would be. However, the authors do mention scaling in future work, albeit only in one sentence.

QUESTIONS:
I don't have any immediate questions apart from a brief discussion on the scalability (see weaknesses).

LIMITATIONS:
Are discussed in section 5.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-01

comments: SUMMARY:
The paper proposes to improve a Forward-Backward (FB)-based behavioral foundation model for zero-shot adaptation to unseen dynamics. With both theoretical analysis and empirical observations, the paper poses the policy averaging problem of the previous FB approach on unseen dynamics, and proposes two solutions that infer a hidden context variable.

STRENGTHS AND WEAKNESSES:
The method is theoretically grounded and well-motivated. The conceptual experiments provide insightful analyses on the behavior of the proposed methods and baselines in the dynamics-changing scenarios.

The major weakness is the reliance on random exploration. This can be limited in complex environments where a random policy cannot effectively cover the possible transitions, or the required context length should be largely increased. The inefficiency of the transformer architecture in encoding long sequences further amplifies the limitation.

QUESTIONS:
* In the experiment with randomized four-rooms, why does the test return decrease as the context length increases after ~100? Does this also happen in the pointmass task when the context length is further increased?

* Would the performance become optimal if we give an oracle trajectory that contains all possible transitions instead of the random-exploration-context?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-03