Optimistic Task Inference for Behavior Foundation Models
Authors: Thomas Rupf
Marco Bagatella
Marin Vlastelica
Andreas Krause
Venue: ICLR 2026 Oral (2026)
Keywords: Behavior Foundation Models
Zero-Shot Reinforcement Learning
Deep Reinforcement Learning
Fast Adaptation
Primary area: reinforcement learning
TL;DR: We propose an algorithm for fast online task inference in behavior foundation models.

Abstract: Behavior Foundation Models (BFMs) are capable of retrieving high-performing policy for any reward function specified directly at test-time, commonly referred to as zero-shot reinforcement learning (RL). While this is a very efficient process in terms of compute, it can be less so in terms of data: as a standard assumption, BFMs require computing rewards over a non-negligible inference dataset, assuming either access to a functional form of rewards, or significant labeling efforts. To alleviate these limitations, we tackle the problem of task inference purely through interaction with the environment at test-time. We propose OpTI-BFM, an optimistic decision criterion that directly models uncertainty over reward functions and guides BFMs in data collection for task inference. Formally, we provide a regret bound for well- trained BFMs through a direct connection to upper-confidence algorithms for linear bandits. Empirically, we evaluate OpTI-BFM on established zero-shot benchmarks, and observe that it enables successor-features-based BFMs to identify and optimize an unseen reward function in a handful of episodes with minimal compute overhead.

Published as a conference paper at ICLR 2026
OPTIMISTIC TASK INFERENCE FOR BEHAVIOR FOUNDATION MODELS
Marin Vlastelica' Andreas Krause' ETT Zürich, Switzerland Max Planck Institute for Intelligent Systems, Germany
Thomas Rupf' Marco Bagatella 12
{ (thrupÉ, mwagatella, www.saalllllaaaaaaaaoooomoo krausea @ethz. ch
ABSTRACT
Behavior Foundation Models (BFMs) are capable of retrieving high-performing policy for any reward function specified directly at test-time, commonly referred zero-shot reinforcement learning (RL). While this is very efficient process in terms compute, it can be less so in terms of data: as a standard assumption, BFMs require computing rewards over non-negligible inference dataset, assuming either access to a functional form rewards, or significant labeling efforts. To alleviate these limitations, inference purely through interaction with the environment at test-time. We propose OpTI-BFM, an optimistic decision criterion that directly models uncertainty over reward functions and guides BFMs in data collection for task inference. Formally, we provide a regret bound for well-trained BFMs through direct connection to upper-confidence algorithms for linear bandits. Empirically, we evaluate OpTI--BFM on established zero-shot benchmarks, and observe that it enables successor- paacessor-features-boe features-based BFMs to identify and optimize an unseen reward function in handful of episodes with minimal compute overhead."
tackle the problem of task
INTRODUCTION
Zero-shot reinforcement learning (Touati et al., 2023) has gradually gained relevance powerful generalization of standard, single-reward RL (Sutton & Barto, 2018; Silver et al., 202 I). Zero-shot agents are designed to distill optimal policies for set of reward functions from unlabeled, offline data (Touati & Ollivier, 2021; Agarwal et al., 2024; Park et al., 2024b; Jajoo et al., 2025). As these methods scale to more complex and broader environments, they are often referred to Behavior Foundation Models (BFMs) (Park et al., 2024b; Tirinzoni et al., 2025). At their core, the majority of BFMs rely on Universal Successor Features (Ma al. (2020), USFs). These methods build upon explicit representations of states (features): given a set of policies, the expected discounted sum of features along each policy's 's trajectory may be estimated completely offline (c.g.. through TD learning) (Dayan, 1993; Barreto et al., 2017). As long as reward function lies within the span of features, zero-shot policy evaluation can be performed through simple scalar product, which in turn enables zero-shot policy improvement, i.e., each policy is implicitly paired with reward function and updated towards optimality (Touati & Ollivier, 2021).
Once this unsupervised pre-training phase is complete, USFs yield set of learned policies. Given a reward function that is linear the features, its optimal policy is by construction indexed by the linear weights describing the reward function in the basis of features, which can in turn be interpreted as task embeddings. The process of finding the optimal policy, which we refer to as task inference, thus corresponds solving linear regression problem. While this process is remarkably efficient in terms of compute, it retains strict requirements in terms of data: a dataset of labeled (state, reward) pairs needs to be provided. In simple settings, this necessity is not particularly problematic: BFMs have been largely trained in simulations (Touati et al., 2023; Park et al., 2024b; Tirinzoni et al., 2025), which makes generating reward labels for the pre-training dataset, or additional data, particularly convenient, Realistically, however, (i) the pre-training dataset might be unavailable or proprietary and, most importantly, (ii) labeling states with rewards might incur significant costs. For instance,
Cooe available https://g/tt http:///ggtttutttt thub. com htt.:////o////oaooootto//////// -bfm.

---

Published as a conference paper at ICLR 2026
Unsupervised Pre-training via USF
Online Task Inference
ral) update
OpTI-BFM
(ooo (dh. (2). ). (dl
80
750
72gg
72
Offline Task Inference
infer
Tog
((s,,,5)))
[(s.r))
Figure 1: In contrast to the standard offline task inference pipeline for BFMs, we explore an alternative online framework: instead of producing a point-estimate of the task embedding from an existing dataset, we actively collect data to build a belief over task embeddings, which results milder labeling requirements and fast retrieval of near-optimal policy.
when BFMs pre-trained directly from pixels, evaluating success from single image is not straightforward or cheap operation.
In order to alleviate these issues, explore alternative framework for task inference, which instead aims to collect a small amount of data directly during deployment (see Figure 1). This removes the need to access the pre-training dataset (i) and may require fewer labels (ii), as the data can be collected actively. To navigate this setting, we propose OpTI-BFM, decision criterion that curates a sequence of task embeddings. Starting from an uninformed prior, OpTI-BFM leverages the linear relationship between features and rewards to update its belief over the space of rewards. A BFM conditioned on meaningfully chosen task embeddings can then interact with the environment for a few steps: labeling the states that are visited allows accurate task estimation with minimal labeling efforts. Crucially, if the underlying BFM is well-trained, we show that control reduces to a bandit problem over tasks. Under this assumption, we can provide regret guarantees for OpTI-BFM. When considering established zero-shot benchmarks the Deepmind Control Suite (DMC) (Tassa et al., 2018), we find that OpTI-BFM requires only handful of episodes to correctly identify the task, eventually matching or surpassing the performance that the standard offline reward inference pipeline achieves with significantly more data.
2 PRELIMINARIES
Notation We model the environment as reward-free Markov Decision Process (MDP) M (S, A, 7) where S and A are state and action spaces, respectively, P(ds'|s, a) is proba- bility measure describing the likelihood of transitions, u(ds) is a measure describing the initial state distribution, and is discount factor. Given policy π S - A(A) and state-action pair (sp, an) E A we use E. Su, ag π| to denote expectations w.r.t. state-action sequences (s)))))))20 defined by sampling a (T(s) and $1+1 P(st, az). For given reward function r r: S + R. we define the policy'ss 's state-action value function as the discounted sum of future rewards OF (No, (n) Eizo "B (r(+)) 80, 00; r).
Successor Features and Behavior Foundation Models For specific feature map Φ S + Rd Successor Features (Barreto et al. (2017), SFs) generalize state-action value functions by modeling the expected discounted sum of features under policy
(50, 0o)
Σ ('E(((s)) 8o, ao.
(1)
SFs allow zero-shot policy evaluation for any reward that lies in the span of the features: if
z ((s) for some = E Rdd the Q-function is linear function of SFs:
Q7 an) = Σ nd Er(())))) so, ag. .|
Σ 7 E ((s.) so. Op, ग| = =T ( an)
(2)
120
120
2

---

Published as a conference paper at ICLR 2026
A similar structure also holds for value functions: (s) = 2) when defining (s)
[-- // 1/0//(( (s, Behavior Foundation Models (BFMs) generally" capitalize on the opportunity of evaluating policies for multiple reward functions, by additionally learning family of parameterized policies (ez) with respect to all rewards in the span of features & (Borsa et al., 2018; Touati & Ollivier, 2021; Park al., 2024b; Agarwal et al., 2025b). Concretely, BFMs train family of parameterized policies so that each policy Ty is optimal for the reward function r(s) = <(s)
Tz (a(s)) E arg max, 1= (s. for each Z.
(3)
In practice, the set of policies, their SFs, and features e may be represented through function approximation: Ty (als) ~ <<(a|s, 2), 1 a) s 1b(8, a, and 6(s) & (bs)). 2 is a low- dimensional space, whose elements can be seen as task embeddings, as they represent reward functions in the feature basis. The low dimensionality of task embeddings is key component enabling efficient task inference in this work.
Given reward function inference, the task embedding 2, parameterizing the optimal learned policy T. is found by minimizing the residual between r(s) and o(s) projectin r onto the span of Φ). Given task inference dataset D (st) this may be solved in closed form:
arg min, E EewD((((s) - ({s))) Covp(o) Ee~D((())r(s)]
(4)
where Covp(e)-1 B=.='=(o(ssd(()))
This process of mapping from a reward function r to an (approximately) optimal policy Ts, what makes BFM based on USFs capable of zero-shot RL (Touati et al., 2023), i.e., they can produce an optimal policy for a previously unseen reward function. When the expectation over D is computed exactly, and lies in the span of Φ, then Ts, is guaranteed to be the optimal policy (Touati & Ollivier, 2021). However, in practice, the expectation is approximated through sampling, which requires (i) the availability of the task inference dataset D (potentially a subset of the pre-training data) and (ii) providing reward labels for each state & E D. As this can be an expensive operation, potentially requiring human supervision, we will propose an alternative online framework for retrieving 2r-
3 OPTIMISTIC TASK INFERENCE FOR BEHAVIOR FOUNDATION MODELS
3.1 SETTING: TASK INFERENCE AT TEST-TIME
We consider alternative framework for task inference in BFMs, designed to remove the necessity for storing pre-training data and, principally, to decrease the required number of reward labels. We focus on an online setting, in which the agent and directly control the collection of the data used to estimate z. While the choice of z will be uninformed the beginning, it will ideally be possible to rapidly identify the correct task, and thus select the z that retrieves the optimal policy, i.e., that coincides with the true task embedding.
update the task embedding during deployment,
More formally, we start from pre-trained based BFM, providing a set of parameterized policies (πz)æεz as well as SF estimates ri=== of features é. While we will now consider finite-horizon SF estimates a) (so) > 7t E [Ø(s;) 80. n2] ] to streamline the presentation and analysis, we remark that the algorithm can be easily instantiated in infinite-horizon settings, as is done in our empirical evaluation. The agent interacts with the environment in episodic setting with horizon H and initial state distribution Mo; instead of directly selecting actions, it will select a task embedding 2 at each step t, and execute action sampled from the respective policy a, Tz, (-1st). This action will result in observing a new state 81, as well as the reward r, of this transition, which constitutes the only source of information about the task?.
We define the discounted return of the k-th episode as
G = - (
(5)
Thhee are BFMs that are not based USFs. We refer the reader to Agarwal al. (2025a) for comprehen- sive overview.
'e assume that each environment interaction provides a reward label; if the agent can additionally control when request a reward label, more efficient schemes are possible, see Appendix B.2
3

---

Published as a conference paper at ICLR 2026
where SkH Moo, ag Tz. (+1st) and 81+1 P(-lse,a)) The goal of the agent is now simply to minimize the expected regret over n episodes
RA = E ER= G G1.
(6)
where G3 denotes the discounted return choosing 2, in each step, and the expectation is w.r.t. (4o- the MDP dynamics, the action distributions, and the choices of the task embeddings. Intuitively, the agent needs to follow decision rule that maps the observed history of states and rewards to task embedding: (sp, ru.... St, Tt) - 2++1 E Z. The selected task embeddings should induce informative trajectories with respect to the reward function, while largely avoiding suboptimal behavior. This setting is reminiscent of well-developed research directions, namely exploration in the space of behavioral priors (Singh et al., 2020) and fast adaptation (Sikchi al., 2025); however, existing methods ignore the underlying structure connecting rewards and task embeddings, which we will show may be leveraged efficiently achieve near-optimal performance.
3.2 METHOD: OPTI-BFM
Our method leverages a core feature of BFMs: for well-trained USFs, the expectation of the return of the k-th episode (Eq. (5)) from initial state so is approximately linear w.r.t. the successor features of the policy conditioned the k-th task embedding zy: (x))),2) 2, B[Gi Tzcl where 2, is the optimal task embedding, and initially unknown. This simple property has significant implication: Policy search reduces online optimization of linear function, which has been extensively studied in the bandit literature (Dani et al., 2008; Abbasi- Yadkori et al., 2011). 1). Building upon these fundamental results, we propose Optimistic Task Inference for BFMs (OpTI-BFM) in order to efficiently explore the space of behaviors while controlling suboptimality.
Interestingly, the same approximately linear relationship that connects SF' and returns, also exists between features and rewards: o(s) 2, = r(s). Following the latter property, OpTI-BFM keeps track of a leaat-s estimate of 2, given the previously observed transitions
2 arg min Σ (n:=)(s))TT)) T2) + +1112122 Z
a Σ ((s))(())))
Σ ((s))r (7)
using Iz-reuularization to ensure the inverse exists. Rewriting this as
= Ei- ri. where V Ala + ((s))()))(s)))
(8)
allows OpTI-BFM to not only track the mean estimate 2, but also confidence ellipsoid C, around &, that contains the true task embedding z, with high probability:
C = E Rd ||= 2 1l S }
where 3r controls the Mahalanobis distance. The estimation of confidence sets allows optimistic behavior in each step by choosing the task embedding 2, which is believed to attain the largest return among those in the confidence set:
21 arg max w 2). WEC,
(10)
SEZ
Intuitively, this procedure conditions the BFM on the most "promising" task embedding among those that are compatible with rewards observed so far. Note that this algorithm has one crucial difference to Upper Confidence Bound (UCB)- algorithms for linear contextual bandits (Abbasi- Yadkori et al., 2011; I; Dani et al., 2008), two different contexts involved: the features ø, which are used for online regression and for estimating the confidence interval, and the successor features vh, which instead used in the acquisition function in Eq. (10). We will discuss that using for regression results in tighter estimates in Appendix A.2.
Algorithm 1 instantiates OpTI-BFM for online task inference. We will establish guarantees for OpTI-BFM in the next section, and then describe how Eq. 10 may be optimized in practice, or avoided altogether with Thompson Sampling (TS) variant, among others.
4

---

Published as a conference paper at ICLR 2026
Algorithm 1 One episode of online task inference with OpTI-BFM
Require: BFM with dh a). ((s), and (a)s),, starting state so Mo, online Least Squares estimator (2n-1, Vn- (potentially initialized with past experience) for I. 0, 0,..., H do
Find 2z E arg ma
2 max wo (11,))))
> Optimism w.r.t. cumulative reward.
Execute action dg nzz (-|s;)
Observe reward rt, next state St+1
Update (2n+t-1, Vn++-1) with (0(ss), re) through Eq. 8 b Update based on
3.3 GUARANTEES
Leveraging direct connection to principled algorithms for linear bandits (Dani et al., 2008; Abbasi- Yadkori et al., 2011). I). we provide regret guarantees for OpTI- -FMM We note that this is a crucial property for online task inference, which could otherwise fail to gather informative data, and converge to suboptimal solutions. For simplicity, we study a variant of OpTI-BFM that only updates its decision rule at the beginning of each episode, i.e., have constraint allows us to leverage results from linear contextual bandit literature.
2 for & (kH) This additional
We show that OpTI-BFM approaches the performance of the optimal policy ne, (Eq. (4)) under the following assumptions:
(AI) Perfect USF for our setting: for every (s,0, E S x A E we have (/ u0,
Ei= A'E (o(s;) 50, ao] and n. (a|s) > 0 = a E arg max vsss, a, 2)
(A2) Linear Reward: r is in the span of features á up to i.i.d. mean-zero o-subgaussian noise Tec- i.e. r(st) = ((x)) T 2, + 7)
(A3) Optimization Oracle: the OpTI-BFM objective, Eq. (10), can be computed exactly.
(A4) Bounded norms: 1|2 I S and [|o(-)]l I L for some S >> and >> 0.
Assumptions (A1) and (A2) are instrumental to recovering theoretical guarantees, but we found OpTI-BFM to perform well even when they are violated (see Section 5). Note that for sufficiently large horizon / the mismatch between the finite discounted sum of features we assume here and the infinite one we have in practice is negligible: the lz-error by L /(1 - 7). Assumption (A3) may be empirically motivated by the efficiency of finding a good approximate solution (see Section 4). In this setting, we can show that OpTI-BFM has sublinear regret.
Proposition 1. (informal) Under assumptions Al-A4, in episodic discounted MDP, if OpTI- BFM (Algorithm 1) only updates 2t at the start of each episode, it incurs an expected regret of R2 S Ô (d)n).
Proof. We prove formal version, Proposition 5, in Appendix A. The proof is similar to the standard regret bounds for LinUCB/OFUL. (Dani et al., 2008; Abbasi-Yadkori et al., 2011) except that the confidence interval is updated H-times per step with features that differ from the context features of the bandit.
PRACTICAL ALGORITHM
Having introduced and analyzed OpTI-BFM, we now discuss practical implementation, and present some additional variants
The main challenge for a practical implementation of OpTI-BFM lies in optimizing the decision criterion in Eq. (10), involving two continuous spaces Z and C, and highly non-lineat map + v((,2), Fortunately, BFMs are pre-trained such that that w s arg max z w (' 2). i.c. the optimal policy for a task described by w is the one conditioned on w itself. In practice, as training is not perfect, we do not strictly rely on this property, and still search 2 over C, instead of Z:
2t E arg max maxwec, w (5),
(11)
*We provide an empirical comparison to this variant in Section 5.3.
'One additional variant is presented in Appendix B.2 We additionally consider radius of 2,9, instead of for this confidence set.
5

---

Published as a conference paper at ICLR 2026
We note that, as C2 shrinks, so does the decision space, and with it the complexity of the optimization problem. Finally, as commonly done for linear UCB (Remark 1), we can reformulate the objective as
arg max max w T (b( 2) = arg max (s 2) T Ê + Be (1,, =)llv= ZEC, wEC;
(12)
=EC.
which allows to optimize over just variable. In practice, we found simple sampling approach to be sufficient to find an approximate solution. We optimize the final objective through random shoot- ing with budget of n 128 candidates, each of which is evaluated through forward pass of the suc- cessor feature network vp in the BFM. Sampling from the ellipsoid C may be done efficiently by apply-
1/2 B&C uniform samples from the unit ball (Barthe al., 2005). ing the push-forward ç - 2 V
A practical implementation may also benefit from efficient updates to the key parameters: 21-1. V. and V==| We do so by keeping track of the information vector Σ - ØeT' and the Cholesky factor of V,, i.e. V22 R2 R. This enables updates and/or recomputation of all components in (cd))
(Gill et al., 1974; Seeger, 2004) making OpTI-BFM very cheap (see Appendix B.I).
Thompson Sampling Variant The connection between least squares confidence sets and Bayesian linear regression is well established in contextual linear bandit literature (Kaufmann et al., 2012; Agrawal & Goyal, 2013). Concretely, we may interpret Op'I's 's internal components at time-step 1 as Gaussian posterior (2 V over task embeddings that stem from a Bayesian linear regression with prior (0, La) on the data rob//, (where a is hyper-parameter in practice). Intuitively, this can be thought of as starting from a prior on behaviors and then refining it over time until it converges to a single behavior. Given this Bayesian interpretation, it is then natural to consider a Thompson Sampling (TS) approach, where we simply sample behavior from the posterior, i.e. task embedding ~ N(2,, V.-1), which foregoes the optimization of the UCB version. We evaluate this variant extensively in Section 5.
Non-stationary Rewards Variant Because of its online nature, OpTI-BFM can potentially adapt to reward functions that change over time. To this end, we consider a variant leveraging a simple idea from non-stationary bandit literature: weighing old data points less than new ones in the least squares estimator (Russac et al., 2020). Concretely, we introduce a new hyper-parameter 0 < p S 1 and weight data from past time-step at time-step with weight p²=3. We evaluate OpTI-BFM in setting with non-stationary rewards in Section 5.4.
5 EXPERIMENTS
The empirical evaluation is divided in several subsections, each of which will address specific question, as their titles suggest. In the following, we first detail some evaluation choices.
To evaluate performance of various adaptation/inference algorithms we consider the environments Walker, Cheetah, and Quadruped from the established EXORL (Yarats et al., 2022) benchmark, with four different tasks (i.e. reward functions) each. We describe full experimental protocols in Appendix E. In all figures, error-bars and shaded regions represent nin-maxx-nnterva over training seeds.
Environments
Methods We choose Forward-Backwant (FB) framework (Touati & Ollivier, 2021) state-of- the-art BFM. We adhere to the standard training and evaluation protocol for FB (Touati et al., 2023), and described it in detail Appendix D. We apply OpTI-BFM on top of this BFM, as well a variant, OPTI-BFM-TS. We further consider LoLA (Sikchi et al., 2025), an approach based on policy search that was originally introduced for fast adaptation. In compliance with our online task inference setting, we initialize LoLA with an uninformed choice of 2, and estimate on-policy returns in the standard episodic fashion, without privileged resets. Besides these three learning algorithms, we evaluate "Random" and "Oracle" baselines that serve as a lower and an upper bound on possible performance respectively. The former executes a random embedding in each step, i.e. 24 Unif(2) Z), whereas the latter executes the optimal policy ne,, where 2, is attained by solving the linear regression problem in Eq. (4), assuming privileged access to labeled data or the reward function. We follow standard practice and approximate 2, with a large budget of 50k labeled

---

Published as a conference paper at ICLR 2026
cheetah
quadruped
walker
Performance
1.0
1.0
1.0
0.8
0.8
0.6
0.5
0.4
0.4
-
2
8
10
8
10
10
Episode
Episode
Episode
-x-
Random
LoLA
OPTI-BFM-TS (ours)
-
OpTI-BFM (ours)
Figure 2: Mean relative performance over 10 episodes of interaction DMC. OpTI-BEM recovers
Oracle performance in 5 episodes. We report per-task absolute performance in Fig. 14 in Appendix C.
cheetah
quadruped
walker
Relative Performance
1.0
0.8
1.0
0.8
0.6
0.8
0.6
0.4
0.6
0.4
0.2
10a
10a
10
102
10
10
10a
10a
10
RND
Random
OpTI-BFM-TS (ours)
OpTI-BFM (ours)
Figure 3: Relative performance after different # of environment interactions. OpTI-BFM is consis- tently among the top performers for all environments and time-steps. We show per task performance in Fig. 15 in Appendix C.
samples from the pre-training dataset in practice (Touati al., 2023; Agarwal al., 2025b). not indicated otherwise, "relative performance" is relative to the Oracle performance.
5.1 How DOES OPTI-BFM COMPARE TO OTHER ONLINE TASK INFERENCE METHODS?
We first evaluate all methods in the online task inference setting described in Section 3.1, and report episodic returns (Eq. (5)) in Fig. 2. We find that OpT1-BFM recovers Oracle performance on all tasks within episodes (5k environment steps) interaction. We observe significant gap between the optimistic strategy of OpTI-BFM and its TS variant in Cheetah. Nevertheless, TS remains promising approach, as it avoids any optimization problem, and simply samples task embeddings from its current belief. Finally, and performs blackbox policy search, makes slower progress which is better visible over 50 episodes in Fig. 16 in Appendix C. This result is consistent with existing ablations initializing LoLA random task embedding (see Sikchi et al. (2025), Figure 5), as the case in our setting.
find that LoLA, which ignores the linear structure of the problem
5.2 IS THE DATA COLLECTED ACTIVELY BY OPTI-BFM INFORMATIVE?
While the previous evaluations focus episodic returns, equivalently regret minimization, now evaluate the quality of the inferred task embeddings, e.g., in the case of OpTI-BFM, how well does тe perform? In practice, to ablate away any bias la-regularized estimators, we compute the task embedding nn by minimizing the squared eтror in Eq. (4) over the dataset of the first n observed transitions of each method, and evaluate the corresponding policy T.. the same environment and task. We compare with two baseline data sources: (i) random trajectories from RND (Burda et al., 2018), which represents a aask-agnottic exploration approach, and (ii) the first n samples from our Random baseline, which rolls out a random policy learned by the BFM. Figure shows the average relative performance of T., in each environment. We find that the data from an actively exploring source, i.c. OpTI-BFM and RND, outperforms the passive Random approach. Furthermore, we can see that OpTI-BFM and its TS variant tend to be more data-efficient than RND, which can be traced back task-awareness
5.3 ARE FREQUENT UPDATES IMPORTANT?
The regret bound provided in Section 3.3 assumes a version of OpTI-BFM that only updates the task embedding it executes the start of each episode. In this section aim experimentally
7

---

Published as a conference paper at ICLR 2026
cheetah
ruped
wall Liee
1.00
1.00
0.75
0.75
Relative
0.50
0.25
0.25
0.25
10
10
10
Episade
Episode
Episode
OPTI-BFM-TS (ours)
OpT1-BFM:TS-E (ours)
OPT (ours)
OpTI-BFM.EE (ours)
Figure 4: Relative performance our methods, and their variants that keep task embeddings fixed for each episode (-EP). Updating the task embedding during the episode leads to faster convergence. For longer episode evaluations see Fig. 16 in Appendix C.
speedup
sloodown
speedup
slowdown
Horizontal Velodly
7.5
Horizontal Veloaty
Horzontal Horbontall Velodty
6
5.0
Hartoonaal
4-
2-
2.5
0.0
10000 20000
30000
10000
20000
30000
10000 20000 30000 Step
10000
20000 30000 Step
Step
Step
p-
Reference
p=
p=10
(a) OPTI
(b) OPTI-BFM-TS
Figure 5: Horizontal velocity of OpTI-BFM in custom velocity tracking tasks in DMC Walker for different decay rates p. OpTI-BFM can adapt to non-stationary reward functions when decaying the weight of old observations.
quantify this gap in theory and practice by evaluating this episodic version of OpTI-BFM. Fig. shows that changing the latent every step. The episodic versions do reach equal performance eventually as shown in Fig. 16 in Appendix C. Intuitively, this faster improvement may be explained by the fact that the default version of OpTI-BFM can adjust faster to new information.
episode-level leads to slower improvement than changing
5.4 CAN OPTI-BFM ADAPT TO NON-STATIONARI REWARDS?
is conceivable that OpTI-BFM can adapt to reward functions that are changing over time because of its online nature. To investigate whether this is possible, we introduce two new tasks for the Walker environment that closely resemble the wa k and run tasks but change the velocity target over time: speedup and s owdown increase and decrease the velocity target respectively after an initial "burn-in" phase. We evaluate OpTI-BFM in a single, 30k-step, episode in Figure 5, which shows that a direct application of OpTI-BFM (p 1.0) struggles to adapt the changing reward function after converging to a fixed one in the first 10k steps. As soon as p is reduced, OpTI-BFM tracks the velocity target accurately. Moreover, observe that, if p is too small, the uncertainty is not reduced quickly enough, and the agent can converge to suboptimal behavior.
5.5 CAN OPTI-BFM SARM-STARE FROM LABELED DATA?
In settings that provide a small amount of labeled data D {(s r, )= from the beginning (Sikchi et al., 2025), OpTI-BFM may be warm-started by updating the least squares estimator n-times with the given data. As shown in Fig. 6, when doing so on high-quality i.i.d. data from the training dataset, the performance of OpTI-BFM quickly improves, showing that OpTI-BFM can additionally be seen an extension the traditional task inference BFMs.
6 RELATED WORK
Behavior Foundation Models This work builds upon USFs (Dayan, 1993; Barreto et al., 2017; Ma et al., 2020), and specifically on BFMs that utilize SFs to train a parameterized policy and the corresponding parameterized SFs. common approach for learning SF-based BFMs is to
8

---

Published as a conference paper at ICLR 2026
wa LK
run
wa bac Kwa
run backward
300
800
200
250
Retum
Retur
Return
600
Retum
150
200
400
100
150
100
200
3
2
1
Episodi
Episode
Episode
Episode
Cracle
==
5=
100
Figure 6: Return of OpTI-BFM DMC Cheetah when warm-starting with n i.i.d. labeled states from the training dataset. Initial performance increases quickly grows. We report full results in Fig. 17 in Appendix C.
approximate of successor measures (Blier et al., 2021; Touati & Ollivier, 2021; Agarwal al., 2025b), but other options spanning from spectral decomposition of random visitations (Wu et al., 2018) to implicit value learning (Park al., 2024b) also exist. In this regard, we refer the reader to Agarwal et al. (2025a) for a broader overview of BFMs and unsupervised RL. Among these approaches, the Forward Backward (FB) framework (Touati & Ollivier, 2021) has risen to prominence, as it achieves state-of-the-ar performance in standard continuous control tasks (Touati et al., 2023), while being scalable to humanoid control (Tirinzoni et al., 2025). Recent work has explored various applications of FB's zero-shot capabilities, such as imitation learning (Pirotta et al., 2023), epistemic exploration (Urpí et al., 2025), and constrained RL (Hugessen et al., 2025). Among related works, our method most closely related to Sikchi al. (2025), which however focuses different problem: fine-tuning a zero-shot policy, which is already initialized through standard offline task inference. On the other hand, our goal is to find strong policy purely through environment interaction, without relying on preexisting labeled data. The two approaches might nevertheless be combined.
Reward Learning The task inference goal of OpTI-BFM fundamentally connected to the reward learning problem in RL. Among the many instantiations, rewards may be inferred from scalar evaluations (Knox & Stone, 2009; MacGlashan et al., 2017), preferences (Fürnkranz et al., 2012; Christiano et al., 2017), or from other types of feedback (Jeon et al., 2020). A powerful extension to reward learning explores active strategies for query selection, largely adopting a Bayesian perspective (Bryık et al., 2020; Wilde et al., 2020). While none of these works are aimed at BFMs, several are close in spirit: for instance, Lindner et al. (2021) propose an information-diress similar goal to ours: inferring the task in a way which does not necessarily reduce the model error, but quickly produces an optimal policy.
method to achieve
Linear Bandits Linear Contextual Bandits are well-studied problem in bandit literature (Abe & Long, 1999; Dani et al., 2008; Abbasi-Yadkori et al., 201 Kaufmann et al., 2012; Agrawal & Goyal, 2013). This is influential to this work in two ways. First, OpTI-BFM is inspired by UCB methods in this setting (Dani et al., 2008; Abbasi-Yadkon et al., 201 I), Second, we utilize guarantees and ideas from this literature (Lattimore Szepesvári, 2020) to produce a regret bound for OpTI--FF B-M in an episodic setting.
7 CONCLUSION
By drawing a connection between practical work in scalable Behavior Foundation Models and rigorous algorithms for linear optimization, we proposed an algorithm that may efficiently infer the task hand and retrieve well-performing policy in high-dimensional, complex environments. By minimizing the number of reward evaluations necessary for task inference, this algorithm can enable BFM to be applied beyond domains in which rewards are readily available, for instance when learning directly from pixels.
OpTI-BFM is designed to perform task inference in a minimal number of episodes: this is mainly pos- sible the search space is minimal. While updating the task embedding alone enables great sample efficiency, fine-tuning additional components of the BFM may provide even better performance the long run (Sikchi et al., 2025). Moreover, theoretical guarantees only cover slower, episode-level

---

Published as a conference paper at ICLR 2026
updates: extending these results to (empirically stronger) per-step represents important direction for future formal works.
While OpTI-BFM builds upon linearity between features and rewards, it does not make any significant assumption on the structure of the feature space Z: understanding the propertics of its elements both formally and practically constitutes important avenue for future work. BFMs have demonstrated scalability up to complex domains, while maintaining structured properties (linearity!) an em- bedding space: we believe that this is ripe field for application of more theoretically principled approaches, which may in turn be impactful beyond regular domains.
ACKNOWLEDGEMENT
We thank Núria Armengol Urpí and Georg Martius for their help throughout the project. Marco Bagatella supported by the Max Planck ETH Center for Learning Systems. We acknowledge the support from the Swiss National Science Foundation under NCCR Automation, grant agreement 51NF40 180545.
REFERENCES
Abbasi-Yadkori, Y., Pál, D., and Szepesvári, C. Improved Algorithms for Linear Stochastic Bandits. In Advances in Neural Information Processing Systems, volume 24. Curran Associates, Inc., 2011.
Abe, N. and Long, P. Associative reinforcement learning using linear probabilistic concepts. 06 1999.
Agarwal, S., Sikchi, H., Stone, P., and Zhang, A. Proto Successor Measure: Representing the Space of All Possible Solutions of Reinforcement Learning, November 2024.
Agarwal, S., Chuck, C., Sikchi, H., Hu, J., Rudolph, M., Niekum, S., Stone, P., and Zhang, A. A Unified Framework for Unsupervised Reinforcement Learning Algorithms. In Workshop on Reinforcement Learning Beyond Rewards @ Reinforcement Learning Conference 2025, July 2025a.
Agarwal, S., Sikchi, H., Stone, P., and Zhang, A. Proto Successor Measure: Representing the Behavior Space of an RL Agent, March 2025b.
Agrawal, S. and Goyal, N. Thompson Sampling for Contextual Bandits with Linear Payoffs. In Proceedings of the 30th International Conference on Machine Learning, pp. 127-135, PMLR, May 2013.
Barreto, A., Dabney, W., Munos, R., Hunt, J.J., Schaul, T., Hasselt, H. P., and Silver, D. Successor Features for Transfer in Reinforcement Learning. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
Barthe, F., Guedon, O., Mendelson, S., and Naor, A. probabilistic approach to the geometry of the 'ell_ ball The Annals of Probability, 33(2), March 2005. ISSN 0091-1798. doi: 10.1214/009 17904000008874.
Bryık, E., Huynh, N., Kochenderfer, M. and Sadigh, D. Active weeference-based gaussian process regression for reward learning. arXiv preprint arXiv:2005.02555, 2020.
Blier, L., Tallec, C., and Ollivier, Y. Learning Successor States and Goal--ependent Values: Mathematical Viewpoint, January 2021.
Bogunovic, I. and Krause, A. Misspecified gaussian process bandit optimization, 2021.
Borsa, D., Barreto, A., Quan, J., Mankowitz, D.,, Munos, R., van Hasselt, H., Silver, D., and Schaul,
T. Universal Successor Features Approximators, December 2018.
Burda, Y., Edwards, H., Storkey, A., and Klimov, O. Exploration by Random Network Distillation, October 2018.
Christiano, P. F., Leike, J., Brown, T., Martic, M., Legg, S., and Amodei, D. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017.
10

---

Published as a conference paper at ICLR 2026
Dani, V., Hayes, T. P., and Kakade, S. M. Stochastic Linear Optimization under Bandit Feedback. 21st Annual Conference on Learning Theory, 2008.
Dayan, P. Improving Generalization for Temporal Difference Learning: The Successor Representation.
Neural Computation, 5(4):613 624, July 1993. ISSN 0899-7667. doi: 10.1 19.11662ccc..199555...1111
Fürmkranz, J., Hüllermeier, E., Cheng, W., and Park. S.-H. Preference-based reinforcement learning: formal framework and policy iteration algorithm. Machine learning, 89(1): 2012.
Ghosh, A., Chowdhury, S. R., and Gopalan, A. Misspecified linear bandits, 2017.
Gill, E., Golub, G. H., Murray, W., and Saunders, M. A. Methods for modifying ma- trix factorizations. In Mathematics of Compulation, volume 28, pp. 505-535, 1974. doi: 10.1090/S0025-57188 19778-9974-03455586
Hugessen, A., Wiltzer, H., Neary, C., Zhang, A., and Berseth, G. Zero-Shot Constraint Satisfaction with Forward- Backward Representations. In Workshop Reinforcement Learning Beyond Rewards @ Reinforcement Learning Conference 2025, July 2025.
Jajoo, P. Sikchi, H., Agarwal, S., Zhang, A., Niekum, S., and White, M. Regularized Latent Dynamies Prediction a Strong Baseline For Behavioral Foundation Models. In Workshop on Reinforcement Learning Beyond Rewards @ Reinforcement Learning Conference 2025, July 2025.
Jeon, H. J., Milli, S., and Dragan, A. Reward-rational (implicit) choice: A unifying formalism for reward learning. Advances in Neural Information Processing Systems, 33:4415 2020.
Kaufmann, E., Cappe, O., and Garivier, A. On Bayesian Upper Confidence Bounds for Bandit
Problems. In Proceedings of the Fifteenth International Conference on Artificial Intelligence and Statistics, pp. 592-600 PMLR, March 2012.
Kiefer, and Wolfowitz, J. The equivalence of two extremum problems. Canadian Journal of Mathematics, 12:363 - 366, 1960.
Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization, 2017.
Knox, W. B. and Stone, P. Interactively shaping agents via human reinforcement: The tamer framework, In Proceedings of the fifth international conference on Knowledge capture, pp. 9-16, 2009.
Lattimore, and Szepesvári, C. Bandit Algorithmss Cambridge University Press, edition, July 2020. ISBN
978-1-108-48682--5 doi: 10.1017/9781 108571401.
Lindner, D., Turchetta, M., Tschiatschek, S., Ciosek, K., and Krause, A. Information directed reward learning for reinforcement learning. Advances in Neural Information Processing Systems, 34: 3850-3862, 2021.
Ma, C., Ashley, D. R., Wen, J., and Bengio, Universal Successor Features for Transfer Reinforce- ment Learning. January 2020.
MacGlashan, J., Ho, M. K., Loftin, R., Peng, B., Wang, G., Roberts, D. L., Taylor, M. E.,, and Littman, M. L. Interactive learning from policy-dependent human feedback. In International conference on machine learning, pp. 2285-2294, PMLR, 2017.
Park, S., Frans, K., Eysenbach, B., and Levine, OGBench: Benchmarking Offline Condiiondd
RL, October 2024a.
Park, S., Kreiman, T., and Levine, S. Foundation Policies with Hilbert Representations, May 2024b.
Pirotta, M., Tirinzoni, A., Touati, A., Lazaric, A., and Ollivier, Y. Fast Imitation via Behavior Foundation Models. In The Twelfth International Conference on Learning Representations, October 2023.
Russac, Y. Vernade, C., and Cappé, O. Weighted Linear Bandits for Son-Stationary Environments, March 2020.
11

---

Published as a conference paper at ICLR 2026
Seeger, M. Low Rank Updates for the Cholesky Decomposition. 2004.
Sikchi, H., Tirinzoni, A., Touati, A., Xu, Y., Kanervisto, A., Niekum, S., Zhang, A., Lazaric, A., and Pirotta, M. Fast Adaptation with Behavioral Foundation Models, April 2025.
Silver, D., Singh, S., Precup. D., and Sutton, R. S. Reward is enough. Artificial Intelligence, 299: 103535, October 2021. ISSN 0004-3702. doi: 10.10166j.artim..22222 2021113333..11
Singh, A., Liu, H., Zhou, G., Yu, A., Rhinehart, N., and Levine, S. Parrot: Data-driven behavioral priors for reinforcement learning. arXiv preprint arXiv:2011.10022 2020.
Sutton, R. S. and Barto, A. G. Reinforcement Learning: An Introduction. The MIT Press, second edition, 2018.
Tassa, Y. Doron, Y., Muldal, A., Erez, T., Li, Y., Casas, D. d. L., Budden, D., Abdolmalcki, A., Merel, J., Lefrancq, A., Lillicrap, T., and Riedmiller, M. DeepMind Control Suite, January 2018.
Tirinzoni, A., Touati, A., Farebrother, J., Guzek, M., Kanervisto, A., Xu, Y., Lazaric, A., and Pirotta,
M. Zero-Shot Whole-Body Humanoid Control via Behavioral Foundation Models, April 2025.
Touati, A. and Ollivier, Y. Learning One Representation to Optimize All Rewards, October 2021.
Touati, A., Rapin, and Ollivier, Y. Does Zero-Shot Reinforcement Learning Exist?, March 2023.
Tucker, A. D., Biddulph, C., Wang, C., and Joachims, Bandits with costly reward observations, In Proceedings of the Thirty-Ninth Conference on Uncertainty in Artificial Intelligence, pp. 2147- 2156. PMLR, July 2023.
Urpi, N. A., Vlastelica, M., Martius, G., and Coros, S. Epistemically-guideer exploration, July 2025.
Wilde, N., Kulić, D., and Smith, L. Active preference learning using maximum regret. In 2020 JEEF/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 10952-10959. IEEE, 2020.
Wu, Y., Tucker, G., and Nachum, O. The laplacian in rl: Learning representations with efficient approximations. arXiv preprint arXiv: 181000080000 2018.
Yarats, D., Brandfonbrener, D., Liu, H., Laskin, M., Abbeel, P., Lazaric, A., and Pinto, L. Don't Change the Algorithm, Change the Data: Exploratory Data for Offline Reinforcement Learning, April 2022.
12

---

Published as a conference paper at ICLR 2026
A PROOFS
This section describes the proof for Proposition 1. It closely follows Chapters 19 and 20 from Lattimore & Szepesvári (2020). We proceed as follows:
We reintroduce the setting with more convenient notation in Appendix A.1.
We show that running linear regression feature-reward pairs is at least efficient as running linear regression on SF-return pairs (Appendix A.2)
We cite a result that grants the optimal choice of 3s in Appendix A.3.
We proceed with the standard linear contextual bandit regret bound proof with confidence based feature-reward pairs (Appendix A.4).
A.1 SETTING
Throughout this section will the double subscript / denote the (-th time-step in the episode. Since we will analyze a version of OpTI-BFM that only switches task embedding at the start of each episode, and since know that the best task embedding is always Ey (Eq. 4), we define 21...., 2n be the chosen task embeddings for n episodes. Let Φx.t = 6(sk.)) be the feature observed in step of episode k. Note that this quantity is random variable; throughout this section we use simple expectation E[[] to indicate the expectation over initial states, action distributions of policies, MDP dynamics, and decisions of the algorithm. We will still use [ ı,, to denote the expectation over rolling out policy π from state
A.1.1 ASSUMPTIONS
For convenience, repeat assumptions here with the notation
(A1) Perfect USF for our setting: for every (s, 0, 2) S A x Z we have
1
(SO. ao. 2) ''B (o(s)) u.z Sp. Sp.aul
Σ
(13)
and
n. (a)s)
E arg max x), aeAA
(14)
(A2) Linear Reward: is in the span of features ò up to i..dd mean-zero o-subgaussian noise 7/, i.e.
Th = ((5k,)) T 2r + Tkk
(15)
(A3) Optimization Oracle: the OpTI-BFM objective in Eq. (10) can be computed exactly.
(A4) Bounded norms: ||=||| S and ||o(-)]l2 5 L for some S > and 0.
A. REGRET
We also repeat the definitions of discounted return of the k-tth episode (Eq. (5)) and expected regret (Eq. (6)) for consistent notation.
G₂ = -11-1 T' R2 = E Ei G Gr.
(16)
(17)
where Cy the discounted return achieved by n., which is optimal for the the reward r(.) = (.:) 2, under AI. We now also define the expected discounted return of episode for task embedding
Gk.z
E G
T.. 8k.0
(18)
13

---

Published as a conference paper at ICLR 2026
Notice that this implies that E|G Gk... and that R. EE k= Gk.z- Gk.... the latter of
which is the term we want to bound at the end of this section. We may now show that the expected discounted return is linear in the SFs:
Gk.= E
Gk
21; 8k.0
(19)
E
-i T T2 9k.0
(20)
Ei=o n (zr, 26 (8k,)))
T2z 8k,0
(21)
2r, E
Σ === 72 (()
Tss 8k.0
(22)
Al (Zr, (8k.0. =))
(23)
Thus, our setting resembles a contextual bandit with context 8g ~ μo. action space and an expected payoff that is linear in 15(80, =), with the one notable difference being that more information is available on the pay-off feedback: it is not available in terms of return GR alone, but additionally sequence of features-rewart pairs (Øk,e, "
A.1.3 OPTI-BFM
Let us finally reintroduce the least squares estimator with the double subscript notation:
H
H-1
= =
where
V, = AI
(24)
A=1
t=0
Jen
t=0
From this we then redefine the confidence sets at the start of episode k as
C fz E Rd ||2 2n Snlilly. S Bk}.
(25)
Defining the UCB operator
(CB max 2), w),
(26)
WEC&
then OpTI-BFM picks 2 = arg max UCB for the k-th episode, which we assume can be attained exactly in A3.
A.2 RETURN-LEVEL VS. REWARD-LEVEL FEEDBACK
In this section we show a few properties conveying the intuitive fact that, if 2, is estimated from richer, reward-level feedback (ΦR,2, Tk.s), confidence sets may be tigher than those produced by return-level feedback 1SH-1y 7 &x,e, Ee (rr,,)) which can be though of as an aggregation of H datapoints of the former (up expectation over dynamics).
For this analysis we define the empirical SF of episode l: as
H-
Σ
7' Due
and Wn = Al + Σ J
(27)
t=
Note that we have E.
Vhe
8k.0
(/S,,,0 21). We further define
A =
Σ T
(28)
so that we have V,, = XI + Σ Ag. Finally, the following constant" will become useful later:
H-1
- 2H
CH Σ
7
(29)
0
We now show that the precision matrices obtained through reward-level feedback (V.) grow least as fast as those obtained through return-level feedback (W,) over the course of one episode.
Note that cz for H
14

---

Published as a conference paper at ICLR 2026
Proposition 2. We have, in Loewner order,
duvi & cH As.
(30)
Proof.
This a direct application of a Cauchy- Schwarz incquality: for any 2,
H
2
x. Th x = (x: x²2)²=
Σ
7
@kit
22
Σ (x )x.))
-CHx
A
From this, immediately follows Vn grows least fast W,, over all episodes. Proposition 3. We have, in Loewner order,
1
Vn 7
Wn
(31)
CH
Proof. Using Proposition 2, we know that
1
A >
(32)
CH
Summing over episodes on both sides, we get
A
(33)
CH
k=1
By adding A/ on both sides, we can write
Al +
Σ A v Al +
(34)
CH
1=1
k=1
On the left, we recognize
Al + Σ
A Vn.
(35)
k=1
while on the right we have
Al -
1
Al
+
JR TT
+ 1
1
Al
1
Wn
+
1
1
Al,
CH
CH
CH
CH
CH
1=1
which in turn yields
(1 (1-1)
1
Vn >
Wn
Al
>
Wn.
(36)
CH
CH
CH
where the last step follows from the fact that CH 2 1.
We now derive a useful property from this result, that will later help us to apply the well-known elliptical potential Lemma. Intuitively it shows that reward-level feedback leads to confidence ellipsoids at least as tight as those of return-level feedback.
Proposition 4. We have for any
Rd
| S / /CH ||/||
(37)
Proof. By Proposition
V W.. where both matrices are positive definite by construction. As the inverse thus reverses order,
(38)
V,-1 & cnWW-
(39)
As consequence, one can show that, for any x,
||=||,, vx Vn S V CH W.: veH |
(40)
n
15

---

Published as a conference paper at ICLR 2026
A.3 CONFIDENCE ELLIPSOIDS
We will now take take the chance to cite a standard result for optimal choices of radii Sa. which guarantee that the true parameter is contained in confidence sets (Eq. (9))i each time-step with high probability.
Theorem 1. (Adapted from Abbasi- Yadkori et al. (2011) Theorem 2)
Given confidence sets C₁ as defined in Eq. (25), and if rewards are in the span of e to o-subgaussian
zero-mean noise (A2), then for any 8 (0, 1) the following holds: Pr[ak E N+ |2, $ Csl S with
det Vi-1 X
BA VĀS + o log
+ +2 2og
(41)
This is the classic uniform-in- -i--tme- self-normalized bound based on supermartingales. Notice that the theorem makes no assumptions about the action selection process nor the process that controls the state-changes, except that they are measurable in the natural filtration the corresponding time-step, which is the case for our decision rule (OpTI-BFM) BFM) and our episodic MDP setting. We remark that, since by definition V2 > V, for 2 s, we have that 3, 2 B, as well.
A.4 REGRET BOUND
We now prove formal version of Proposition 1.
Proposition 5. Under assumptions AI-A4, with choices for 3x as in Theorem 1, if OpTI-BFM is applied the beginning of each episode and selects the task embedding z arg max UCBx(z) for the k-th episode, incurs an expected regret of (dd/T))
Proof. We start off by investigating the event E that 2, E C2 Vk E {1...., m), i.e., is in all confidence sets, which occurs with probability at least 8 according to Theorem
Since we will refer to both empirical and "expected" SFs (1) and v respectively), we introduce the natural filtration FR-1 generated by the history up to the start of episode k. in particular, 8k,0, Zk, 1 WR-1 are Fm-r-measurablee Intuitively, conditioning on Fx-1 allows us to limit randomness to within the current episode. Crucial for will be,
(SK.D =x) E J
[r-1]
(42)
We will proceed as in the standard regret bound for linear contextual bandits (Lattimore & Szepesvári, 2020) Let žạ = arg maxwec, (1K(8).0 =x), w) be the task embedding that maximizes the UCB. We then have that
(v 2,), 2+) I UCB, (2,) < (EB) = (wb(s,, zw), Zx).
(43)
We can therefore bound the instantancous regret eμ by
e Gk.z, Gk.z
(44)
(W(Sn,0; 2r), - (1<(8n,0 =x);2z)
(45)
S (w(su.0= 2x), - -Wb(sn,0 2x), 2r)
(46)
(v/,00 2x), Ex - -z))
(47)
Conditioning on previous events (F and applying the Cauchy- inequality pointwise, we have
EE [ek
F 1] < E [(bb( 24), Ex - zr) Fx-1] ( Es - 2zr) Fx
(48)
= E
(49)
1 | 1| - 2r zrllvr-1 Fam]]
< E
(50)
Since we in event E we have 2, E Ck. furthermore, En Cs by definition. So, using the definition of the confidence set Eq. (25), we have
- - 2, fllv,_, 2B, & 28,.
(51)
16

---

Published as a conference paper at ICLR 2026
This is where this proof diverges from the standard regret bound: note that V6-1 is sum of features and not empirical SFs vb; making immediate application of the elliptical potential lemma non- trivial. We can however recall the comparison of least square estimation with ceward-feedback and return-feedback from Appendix A.2. In particular, one apply Proposition 4 to upper- bound the V-norm with the W -norm that depends on the empirical SFs:
Pk
S
VCH
4k
(52)
W
Combining these two bounds results in
E [ek
Fk-1)
S
EE
2/3n, v
CH
2/h
Fk
(53)
W
-(1->") 1-7
Recalling that, by assumption A4,
and
H-1 S & AI
V XI +
(54)
so ||a|| S - 1/2 112 |alllv we can write that, pointwise for all kt
KTzz Ex - 2x
min
2,3
Joh
28n VCH
J W
(55)
L(1 H
S 23,, min
th VCH
(56)
Vx(1 -
Weel
=
Combining the two bounds inside the expectation (as they hold pointwise), obtain
E
jex
Fk < E 23, min B, CH
F
Fn-1
(57)
Using the fact that min a, bx} I max a, b} min (1, x for a, b out of the expectation:
O, we can then take some constants
28,, min I1 |
E lex Fx max / CH E
Fx-1
(58)
We now sum over episodes. To do this, we use the tower rule on ex, i.e. Blen] = EEE [ek
FR II..
Rn E
EN
E
[ex]
Σ E [E 1=1
[x-1]]
(59)
k=1
Applying our (pointwise) bound on E
[ex
Fx
and the tower rule again, we get.
Rn = Σ E E FR-1l] <
2a
E Bn
min
1,
[1/
Fa
(60)
k=
k=
}
2nx
3n min
1,
xx
(61)
Using linearity of expectation and then applying the Cauchy- SchwarZ inequality over episodes (pointwise), prepare the sum the application of the elliptical potential Lemma on (1x))_
17

---

Published as a conference paper at ICLR 2026
and (W&) (pathwise):
1, 1/1105a) War
R - 2a
min
(62)
k=1
min { 1,
< 2xx E
8mg
(63)
with
1=1
det Wn
< 2o
Bugi
2n log
Lemma
(64)
A
Inserting our choice for 3n we obtain
( (VAS+ + o ( deet Vn-1 + 2 log(1/6)
)
det Wn
Rs S 2x
V 2n log
(65)
A
We recall that, so far, we investigated the expected regret conditional on the event & that E C2 Ak E {1. ................................ n]. which occurs with probability least according to Theorem We now choose
5 = 1/n. Using Proposition 6 to upper-boume the determinants (pointwise), and recalling the fact
that J <
L(1-") (by A4), we obtain
Rn 2a
Vās+o (Vcloo
dd + nHL2
+ log
2nd log
dd
11
(1-3)²
dd
$ (dvn)
(66)
We finally investigate the complementary event E², We remark that the discounted return in each episode can be bounded by constants using the Cauchy- Cauchy-Schwarz inequality:
H
LS
1G = E
Σ Y 1'
Tea 84,0
LS
Σ
7
(67)
t=0
It thus holds that Rs <
2LS T-
under 80
To combine the analysis for both events, let R. Ex= e/ be the empirical regret. By the law of total probability, have that
R₂ = E[Rm]] = [PrE] E Rm
Σ + Pr
E
Rm [c]
(68)
1 2LS
$1 (1.0
+
(69)
1-7
{ddm,
(70)
which concludes the proof.
A.5 REGRET BOUND UNDER WEAKER ASSUMPTIONS
Our empirical evaluations are performed in setting which violates two assumptions described in A.1.1: perfect USF estimation (A1) and linearity of rewards (A2). This section provides study of the algorithm's behavior under these violations taking some ideas from existing theory of misspecified
bandits (Ghosh et al., 2017; Bogunovic & Krause, 2021; Lattimore & Szepesvári, 2020).
We start by replacing Al and A2 by weaker assumptions that quantify the mismatch between ó and and between and its projection z,T respectively:
18

---

Published as a conference paper at ICLR 2026
(AI') For every (s, (s, E x Z we have
2))
- EE | 0 SE
(71)
and
(a(s)) > 0 + E arg max (8, 2)_
(72)
aEA
(A2') For reward we have for each state sk,
(73)
where the expectation is over i..d. mean-zero o-subgaussian noise 1e,t
We may thus write za as biased version the true "**
=s)
(74)
and, similarly, rewrite rewards as
Fk ((SK,)) 2, (sk,)) + Me.t-
(75)
We then establish uniform-in-time bound as in Theorem by inflating 3 account for the newly introduced bias in the reward. Considering the distance between the current mean estimate - and 2,, we can decompose this into
t-1
||r - 2-11ll,_,
Zp
dil 2y
V,
Dell
V
Σ
(s))
(76)
1=1
i=1
1/-1
i-1
1-1
24 Vi-12 + A2₁ - -
6il - V Σ (
(77)
V.-1
6-1
1-1
24 2, V=l
Ažp +
Σ
de +
Σ o.)r(ss)
(78)
1=1
i=1
Ve-1
1-1
A2
2i71 + Σ
S (3))
(79)
A2 +
6:77
Φ (s:))
(80)
[=
19

---

Published as a conference paper at ICLR 2026
The first term is covered by Theorem 1, while the new bias term can be bounded through similar techniques to those used in the main proof:
1-1 Σ (s))
A (s:) (s:)lv
1lly properties
(81)
| v=l vz
A2'
(82)
| v=l
Vi-1 Ve-1
(83)
| v
Cauchy-Schware
(84)
min L2
x: 11
(85)
I.
Σ min (1. 1 16
(86)
VX
V2LC
det V/
V
tlogg
Lemma
(87)
VX
det Vo
As a result, we can establish a uniform-in in-time guarantee if is inflated as
V2LC VX
det Vi det Vo
=/= B +
ttlog
(88)
Continuing to build the regret bound, we first notice that the mismatch also impacts the discounted return:
Gk.=
E2
G =z 8k,0
(89)
E
11 -1
T2. Sn.0
(90)
A2'
Ei= 7hth (Zr, 66 ((sk.))) ))> + A, (sk.e) :iz sk,0
(91)
cr, (2r,
E ΣH /d ((1,)) 81.0 1) + IHt-i 2 &, (SK,e)
Tz, Tz, 8k.0
(92)
(Zr, e (sk,0, 2)) + E Etilo ( T2,
(93)
(z,, 36* (32.0, 2)) -
(94)
We can then proceed along the main proof in Appendix A.4 as normal, substituting 3' for 3 throughout. Given the suboptimal vi Eq. (43) now becomes
( (s) & (CB)(2+) & UCB
(Wb (Sk.0, 2x))
(95)
20

---

Published as a conference paper at ICLR 2026
Then, in the bound on the instantaneous regret (Eq. (44)), we get an added term.
Ek Gk.s, - GK.zK
(96)
(Sk,0, 2r), 2x) - (10 (8k,0, 2k), +
26
(97)
I-7
(81.0, =x); _b) - -(00 (81,0, 2k), 2r) + 7-7
26
(98)
(0 (5k,0) 2x), 5x - 2x) + I-7
26
(99)
12' (W(SE,0; 2k); Ex 2 A 2x); 5k 2r)
25
(100)
1-7
(1b(0n,0 =x), -x) + A (5k.0= =k =x)]lv=a
1 5vllve-1
26
Cauchy-Schwarz
1-7
S (1b(k 2x), - 2,) + A 2/3 2
(101)
1-7 Amaax(Vi 1 VA
Where we used
|,lly, 1 v V V
S
|
(102)
in the last step.
We can treat the added term 28%
& VX I-
2 separately throughout, bounding it by
23 n
(103)
after summing over the episodes. As a result, inserting our new 8y in the regret bound, we obtain
(d d1*+HHT3
V2L
dd nHT2
Rn 2a
VAS VAS+ + tottoaloo ilog
2 log
ndll log
dd
dd
now
dd 12(1--411
x
2nd
[1-
28
yA
Vx
(104)
new
Because of the new term appearing 8 and the final constant term, the regret bound is longer sublinear, but instead
R2 <
+ (n(vdd))
(105)
which is also the final regret when incorporating the event E This bound indicates that, while the confidence set may shrink initially, the two mismatches may introduce an irreducible uncertainty in the size of the respective missmatches that cannot be resolved. Crucially, the suboptimality incurred directly connected to the degree mismatch considered in assumptions (Al') and (A2').
A.6 USEFUL PROPERTIES
A.6.1 DETERMINANTS
We list few useful properties that are used in the proofs above Proposition 6. Let Φ1. .1...., on be sequence of vectors with || < L for all ( 1,..., n, and
V. = X Ei= T.O.T
det Vn Ad
dA + nL?
log
< d dlog
(106)
dA
Thiis approximation of Amax (V=)) eigenvalues of Vi evolve time.
S 1/A is rather crude and could be improved by investigating how the
21

---

Published as a conference paper at ICLR 2026
Proof. First, note that
det(AI) = X and Tr(AI) - da
(107)
For Vn we have, using the AM-GM inequality,
- 2 T.V,
Tr AI nL?
det V = II -
-
(108)
d
because ||+||| S L.. Thus we have
det Vn
yd nL? d)
log
< dlog
(109)
Ad
A.6.2 ELLIPTICAL POTENTIAL LEMMA
We here state a standard result, commonly referred to as the elliptical potential lemma, for complete- ness.
Lemma 1. Assume Vo > 0 be positive definite and ler || & for all = 1,...,7, n, and Vi = Vo + E.=100 2.0..
be a sequence of vectors with
det Vn
min || 12: S 2
(110)
det Vo
Proof. First, note that we have for 2 0
min(1,u] S logg(1 + u)
(111)
and use it to get
Σ min 1obll 1 s Σ log (1 |10el1 :)
(112)
t=1
Then, notice that for t 2 we have
V/2 = V + oioT 1/22 112
0100 (1+V
Tv:1/ v2-1
(113)
so, taking the determinant on both sides we get
det V. = det V det (I [ V/ 1/2 2i / V/ = det Vi-1 (1 + 11 11 (((=
(114)
where used det B) det A-det B and det (1 + +un ) 1 || Telescoping the previous result, we get
det Vn det Vo
II t=1
(1 || (|
(115)
which implies
det Vn2 det Vn
log [ + | 1 = =log
(116)
which proves the claim.
22

---

Published as a conference paper at ICLR 2026
A.6.3 UCB OPTIMISATION
We here state the derivation of a well-known property of LinUCB for completeness.
Remark 1. We want to show that
arg max max w T (b(8t,,) = arg max (b(8,, T
2 + Be + 2)ll
(117)
SEC, wEC;
SEC,
Focusing on the inner max term, we have
max w (/(( wECs
max
w (s),2)
(118)
|| 3e 8.ll
Sa.
Sa.
& 1Tb((82 ()
max u (8),))
(119)
| Sa.
S 2 - 1T (T6)))) + max Ilullys,, (1068), 1
(120)
Mally 18a
= 2 - 1 T (s), + Be (1 ,)llv,
(121)
where the equality is allained by
(82,
u² Br [8 2)]|
(122)
B ADDITIONAL EXPERIMENTS
B. HoW MUCH COMPUTE DOES OPTI-BFM NEED?
To estimate the compute cost of OpTI-BFM, measure the time for the action selection and update steps on an Nvidia RTX 4090, skipping the first few calls to avoid measuring JIT compilation time. Table shows that OpTI-BFM and OpTI-BFM-TS -T are about 5x and 4x slower than running just the policy (Oracle) respectively.
Table 1: Computational cost of OpTI-BFM compared to running just the policy on an Nvidia RTX 4090 GPU. OpTI-BFM is about 5x and OpTI (E11-BFM-TS TS about 4x slower.
Oracle (Tz,)
OpTI-BFM
OPTI-BFM-TS
Time per Step
0.772 ms
3.567 ms
2.756 ms
Frequency
1386 Hz
280 Hz
363 Hz
B.2 REQUESTING REWARD LABELS EXPLICITLY
We have so far assumed that each environment interaction provides a reward label, and is thus synonymous with the labeling cost. One could also consider setting where the agent decide whether it wants to observe the reward label in each transition. very simple approach this setting threshold the D-gap (Kiefer & Wolfowitz, 1960; Lattimore & Szepesvári, 2020), which describes how much the confidence ellipsoid shrinks for new dr.
( = log det + ).0,) - log det Vz-1 log (1 lo
(123)
and only requesting ri if ((o:) 2 K. We evaluate labeling efficiency for this variant: for each labeling budget (# Samples), we measure average rewards at the time step in which the budget is exhausted, i.e. "E Ei=1 Tt- This is presented in Fig. 7 (see also Fig. 19 and Fig. 20 for per-task results). We observe that x trades-off environment interaction and labeling cost: for higher k, good performance is achieved with fewer labels, but more episodes are required to request the same number of labels. Given a specific cost for labels, a slightly more involved thresholding scheme (Tucker et al., 2023) could potentially be used to decide the threshold apriori. Surprisingly, this variant of OpTI-BFM can maintain the performance of the main algorithm, especially in easier tasks, while reducing the amount of labeled data by one order of magnitude.
This approach can be directly applied to OpTI-BFM by introducing
23

---

Published as a conference paper at ICLR 2026
walk
run
walk bac Kward
run backward
0.20
Average Reward
Aeerage Reward
Ruerage Reward
0.8
Average Reward
0.75
0.2
0.6
0.15
0.50
0.10
0.4
0.1
0.25
0.05
0.00
10
10
10
10
10'
10
10
10
10
10
10a
10
10
102
10
10
@Samples
e Samples
@ammlles
Samples
OPT (x= 0.0)
0.03
k=0.1
0.3
OpT (K- 0.0)
0.03
K-
K 0.3
Figure 7: Average reward in Cheetah at different numbers of requested reward labels (# Samples) of our main methods for different information thresholds k. We stop interaction after 30k environment steps. trades-off interaction cost with labeling cost. More than one order of magnitude less reward labels can still result the same performance in easier tasks! We report per method results of all environments in Fig. 20 and Fig. 19
cheetah
quadruped
walker
Reladie Performance
1.2
1.2
1.0
1.0
1.0
0.8
0.8
0.6
0.6
0.6
0.4
Episode
Episode
Episode
OpTI-BFM (ours)
OpTI-BFM (ours)
Figure 8: Relative performance after different # of environment interactions of OpTI-BFM and OpTI-BFM +grad; the latter is a variant that optimizes the USB objective through gradient steps. Performances are similar; we report per-task results in Fig. 18 in Appendix C.
B.3 FIRST-ORDER OPTIMIZATION OF THE UCB OBJECTIVE
This section introduces a variant of OpTI-BFM which uses gradient ascent to optimize the UCB objective:
arg max (i ()) 2 + Br (1),,)))) 2)l ZEZ
(124)
as opposed the random shooting approach we describe in Section 4. We validate this variant empirically on wa walker tasks, relying on the Adam (Kingma & Ba, 2017) optimizer with learning rate of 0.1 to perform 4, 8, or 16 gradient steps in each step, and maintaining the optimizer state throughout. At each step, we utilize 4 to warm-stan starrttart the optimization procedure. We report overall return curves in Fig. 8, and an ablation of compute costs Fig. 9. We observe that, given sufficient number of gradient steps, this variant approaches the main instantiation of our method performance. However, compute requirements for gradient ascent scale linearly with the number of gradient steps, and surpass those of random shooting. Considering that state embeddings are, for most BFMs, relatively oow-dimensional (c.g. d = 50), we conclude that random shooting approach should be preferred, it is also less prone to fall into local optima. For very high-dimensional task spaces, first-order optimization remains however a promising option.
B.4 OPTI-BFM FOR LOCO-NAVIGATION
To evaluate effectiveness beyond the locomotion experiments presented in the main text, we eval- uate OpTI--BM in antmaze medii um-navi gate-vO, um--navgate--o, from the OGBench (Park et al., 2024a) benchmark. By default, this environment defines an indicator function reward that is if the piloted quadruped reaches certain goal position, at which point the episode terminates immediately. To make this environment consistent with our theory, we change the task to "holding" the goal position: the episode does not terminate until 1000 steps are accumulated. Furthermore, the radius of the sparse reward to cover the maze cell containing the goal, as seen in Fig. 10b in green, reaching the goal exactly is otherwise challenging during online learning. As is common in this
slightly increase
24

---

Published as a conference paper at ICLR 2026
stand
wi Lk
run
TLip
400
800
800
Return
Retur
Retur
300
600
600
400
400
400
100
200
2
8
10
6
10
2
8
10
6 Episode
10
Episode
Episode
Episode
16 samples
64 samples
2 steps
8 grad. steps
32 samples
128 samples
4 grad. steps
16 grad. steps
(a) Absolute performance for different # of samples or gradient steps.
325
300
275
250
225
200
16
16 32
64
128
grad. steps
samples
OpTI-BFM
(b) Inference speed Hz for different # samples or gradient steps.
Figure 9: Performance and inference speed of OpTI-BFM, and a gradient-based variant (OpTI-BFM
+grad). The performance gap moderate, but the computational cost gradient-based variant generally higher.
taskl
task2
task3
taske
task5
0.5
Soctans Make
0.75
Success Rote
0.75
D.6
0.75
0.50
0.50
0.50
0.2
0.25
0.2
0.00
0.00
0.00
10
Eptsdd
Episode
Episode
(a) Success Rate over 30 episodes.
Oracie
OPTI-8FM
(b) The first 9 trajectories (left-to-right) of Oracle and OpTI-BFM in Task 2. The initial state is in the top left (red) and the goal reach and stay the cell in the bottom right (green). which assigns reward of each step. OpTI-BFM explores the whole maze before finally exploiting the reward it has seen in the green cell. We attribute any perceived exploration bohavior of the Oracle to the high behavior cloning coefficient that is necessary train policies that perform well in this setting.
Figure 10: OpTI-BFM can infer goal-reaching task in the OGBench (Park et al., 2024a) loconaviga- tion environent antmaze- -eaaeeeeededd di um homm-navigate-v00
benchmark (Park et al., 2024a), we introduce a behavior cloning coefficient 0.01 to the actor loss in pre- pre-training, the pre-training data contains near-expert navigation behavior as opposed to the high-coverage exploration data in EXORL. Results shown Fig. 10 are consistent with our main evaluation: OpTI-BFM reach near-Oracle performance within a handful of episodes.
25

---

Published as a conference paper at ICLR 2026
walk
wi Lk bac Backward rd
run ba
200
750
Return
Return
200
Return
500
Return
150
100
600
250
50
400
0
Episede
Episede
Episede
Episede
Oracle
Rendom
d-
g-100.0
2- 300.0
d-1000.0
Figure 11: Performance of OpTI-BFM over 10 episodes in the Cheetah environment with mismatch between db and & (see Eq. (125)) of magnitude a. Performance deteriorates as a increases.
B.5 OPTI-BFM WITH INACCURATE SUCCESSOR FEATURES
To investigate how robust OpTI-BFM is to violations of assuption A1, we evaluate its performance when introducing a systematic mismatch between the discounted sum of Φ and tb. Concretely, we let OpTI-BFM use
(s 2) = ( 2) MLP(2;
(125)
where MLP is two layer network with ReLU activation and L2-normalized outputs, and 0 are randomly sampled weights, which we resample at the start of task inference. This additional MLP introduces systematic, z-dependent bias, whose magnitude may be controlled through the hyperparameter a. To avoid confounding factors as much as possible, we adopt an optimization procedure that samples from the whole 2 to optimize the UCB objective, increasing the number of samples to n 512. As expected, Fig. 11 shows that performance deteriorates as we increase or very large values of a (+x 1000) are necessary render OpTI-BFM completely uninformative, and approach performance of the random baseline. As the norm of & is generally bounded by vd ~ 350, these constitute significant perturbations.
B.6 OPTI-BFM WITH NON-LINEAR REWARDS
This section investigates how a violation of assumption A2, i.e. linearity of rewards in the features ø impacts performance of OpTI-BFM. In order to isolate this effect as much as possible from other factors (e.g.. how hard the task specified by reward function is), we consider a family of reward functions with increasingly larger orthogonal components to the feature space. Starting from an existing reward function r(s). project it to the feature space, obtaining 2,, and then extract its orthogonal component e(s) = r(s) - (ss 2, by training a small network. We can then control linearity of rewards and investigate performance under the reward function
To (s) = z, d(s) + a e(s).
(126)
Note that (s) = 2,T which has 0 projection error, and (s) recovers r(s). While this technique can approximately disentangle reward components, for a >> 1 the reward function might increase in scale: episodic performance might thus actually grow with projection error. Furthermore, the function approximation of e might be inaccurate, meaning that a does not allow directly tuning the projection error. We can nonetheless compare the return of OpTI--BMM (upon convergence, i.e. at its 10th episode) with that of the Oracle, over range of mean absolute projection errors, as Fig. 12. While the results are increasingly noisy, the error increases far beyond ranges encountered for existing rewards, OpTI-BFM seems to suffer slightly more in specific tasks looks-generally achieving worse performance compared to the Oracle for similarly misaligned reward functions. This may be explained by the fact that OpTI-BFM relies on linearity for both data collection and task inference, while the Oracle relies on independently collected data.
B.7 CAN OPTI-BFM LEARN FROM NOISY REWARDS?
In this section we evaluate the performance of OpTI-BFM under noisy reward feedback. To this end, we add zero-mean Gaussian noise to the observed rewards with different standard deviations a. As predicted by Eq. (66), we see in Fig. 13 that convergence is slower for higher standard deviations. Adjusting the hyper-parameter 3 = 10 for o = 10 did improve performance slightly, but with a noise level that is 10x higher than the reward of the environment r, [-1, 1] convergence remains slow.
26

---

Published as a conference paper at ICLR 2026
wa Lk
run
walk bac kward
run. ba kwwa rd
Retur at 10th epicade
3000
1000
1000
2000
500
1000
1000
-500
0
0.0
0.5
1.0
1.5 2.0
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
15 2.0
Vz
$ Tz1
V,
I6,
Ira
$ Zz
Oracle
OpTT-9FM
Original nnn
Figure 12: Return of the 10th episode of Oracle and OpTI-BFM in the Cheetah environment when artificially increasing or decreasing the projection error of the reward function onto ø through learned network.
walk
run
walk backward
run bac kward
1000
1000
800
750
Return
Retumn
Retur
Retur
W
400
250
200
0
20
Episode
Episode
Episade
Episode
1=10
Oracle
Figure 13: Performance of OpTI-BFM over 10 episodes in the Cheetah environment when adding Gaussian noise with different standard deviations to the rewards observes. Convergence slows down as the noise is increased.
27

---

Published as a conference paper at ICLR 2026
C
FULL EXPERIMENTAL RESULTS
stand
wa Lk
run
flip
800
400
600
Return
Return
600
Return
300
Return
600
400
400
200
400
200
100
2
8
10
6
10
2
10
6
10
Episode
Episode
Episode
Episode
Oracle
Random
LOLA
OPT--FM-TS (ours)
OPTI-3FM tours)
DMC Walker
walk
walk backward
backward
800
200
Return
Retur
Retur
Retur
1.50
600
200
500
100
100
250
50
200
10
10
Episode
Episode
Episode
Episode
Orade
Random
LoLA
-
OpT M-TS (ours)
OpTI-FFM (ours)
(b) DMC Cheetah
stand
jump
walk
run
500
800
600
400
Retuum
Return
Retur
400
300
400
200
200
200
100
Episode
Episede
Episode
Episode
Orade
Random
LOLA
OpT M--TS (ours)
OpTI-3FM (ours)
(c) DMC Quadruped
Figure 14: Episode Return over 10 episodes (10k steps) of interaction in DMC.
D TRAINING PROTOCOL
D.1 FORWARD BACKWARD FRAMEWORK
We use the forward backward (FB) framework (Touati & Ollivier, 2021) as the base BFM throught our experiments. To fully explain FB, we quickly introduce the successor measure (Dayan, 1993; Blier et al., 2021) = it is defined as
M (su, ao ao. X)=
Σ
Pr(ss E X X|s0, on, m).
(127)
and can be thought of as the discounted state occupancy of policy æ when starting in 80, ao- FB then learns the low-rank decomposition of the successor measure density w.n.t. the empirical dataset measure Derain (ds):
M a, ds = F(s,a,)) B(s ) (drannds") (ds+).
(128)
where the policy should satisfy
(als) - E arr as)
(129)
The successor measure and successor features are closely related, since we have that
1 (s, = / MT ds (s ds(sst
(130)
28

---

Published as a conference paper at ICLR 2026
stand
walk
run
flip
800
400
Retum
Return
300
Retur
600
400
200
400
100
102
102
101
10a
102
10
10a
10a
10
104
107
10
RND
Random
OpTI-BFM:TS (ours)
OpTI-BFM (ours)
50k i.i.d Samples
DMC Walker
walk
run
walk backward
run, backward
800
200
Return
Retum
200
Reture
150
400
400
100
100
200
50
10a
10a
10
10a
10a
10a
10a
10a
10
10a
10a
10
AND
Random
OPTI-9FMITS (ours)
Tp FM (ours)
50k L.d. Samples
(b) DMC Cheetah
ju jump
500
Retum
600
Return
600
Return
400
700
500
400
300
600
400
500
200
10c
10
10
10
10
10
10a
10a
10
10a
10a
RND
Random
OpT--FMM-S (ours)
OpTI-3FM (ours)
50k Samples
(c) DMC Quadruped
Figure 15: Per task results of Fig. 3. We show absolute performance here and include the Oracle performance (gray line).
Following Touati Ollivier (2021) (Theorem 13), we thus have for FB
/ (Ms+)MM(ss (s,,,))(
/ B(st)B(st) )
(131)
=EgDDan [B(st (oottB((st)) ((s,))]F(ssa,
(132)
=Cov Fs,
(133)
so F(s, a, 2) are SFs of features ((s) (CoVDain B) B(s). Task inference for FB with labeled dataset 'D then effectively becomes
= (CovDΦ))-E(() E(d(s)r(s)]
(134)
= (CovD ((Cov B) B)) (Cov B) B))
(135)
= (CoVD B)
(136)
(Cov
(137)
which is consistent with Touati & Ollivier (2021) (Propasition 15), In practice we pre-compute
(CovDrm B) with 50k samples from Durain after training.
D.2 PRE-TRAINING
As is the standard in zero-shot RL benchmarks (Touati et al., 2023; Agarwal al., 2025b), train our FB model using an offline dataset collected with RND (Burda et al., 2018; Yarats et al., 2022) consisting of 10M transitions.
29

---

Published as a conference paper at ICLR 2026
stand
wa Lk
run
flip
800
600
800
Return
600
Return
600
Return
300
Return
400
400
200
200
10
20
30
40
50
10
20
30
40
50
10
20
30
40
50
10
20
30
40
50
Episode
Episode
Episode
Episode
Oracle
Random
LoLA
OPTI-BFM-TS (ours)
OPTI-FFM (ours)
DMC Walker
walk
walk backward
run backward
300
1000
200
750
Return
600
Raturn
200
500
100
100
250
200
Episode
Episade
Episade
Episode
Oracle
Random
LoLA
OpTI-BFM-TS (ours)
OPT (ours)
(b) DMC Cheetah
stand
jump
walk
run
500
800
600
Retum
600
Return
400
Retur
400
Retum
300
200
200
100
20 30
40
50
20
40
50
Episode
Episede
Episode
Episede
Random
OpTi-BFM-TS (ours)
OPT1-BFM (ours)
(c) DMC Quadruped
Figure 16: Episode Return over 50 episodes (50k steps) of interaction in DMC when only allowing switching of the task embedding at the start of episodes. Note that LoLA resamples its task embedding every 50-250 steps (rrper-parameter)
FB is trained using losses, Firstly, the successor measure loss (Blier et al., 2021)
Cra(2) Ba,as''D, D/rn EgtnD
2F(s,a,2) B(s')
(138)
+ - B(s
(139)
where F, B are target networks, secondly, an orthogonality regularizing loss on B
Lorth EF gnnDan s'a
| [B( B(s) (())((s)))
(140)
and thirdly a DDPG-style loss for n::
Cx(=) EgwDin Ea-as (+le) [ (=,,
(141)
using the reparameterization trick for Gaussian policies.
D.3 ARCHITECTURE AND HYPER-PARAMETER
We choose to follow the implementation details of the most recent work on FB (Tirinzoni et al., 2025), specifically their implementation for DMC in the released code-base. The B-network is 3-layer MLP. The F is an ensemble of size with each two 2-layer MLPs to encode the arguments (s, a) and (s, z) that then concatenated and fed into another 2-layer MI.P. uses two 2- MI.Ps to
30

---

Published as a conference paper at ICLR 2026
stand
wa
run
flip
900
600
800
Return
Return
700
Rcturn
300
Return
500
600
600
400
200
400
500
300
2
10
10
10
6
10
Episode
Episode
Episode
Episode
Oracle
10
25
50
100
(a) DMC Walker
walk
backward
backward
300
800
200
800
Return
Return
Return
200
400
100
600
100
200
Episode
Episode
Episode
Episode
Oracle
カ 0
n=
10
n=
n=
0 100
(b) DMC Cheetah
stand
jmm
walk
run
1000
500
700
Retur
800
600
Retur
600
Retum
400
600
500
400
400
400
200
8
10
Episade
Episade
Episade
Episade
(c) DMC Quadruped
Figure 17: Return of OpTI-BFM when warm-starting with n i.i.d. labeled states from the pre- pre-training dataset.
encode the arguments s and (s, 2) and then also another 2-layer MLP to map from the concatenated layers to a mean μ.. The policy then a truncated Gaussian with fixed standard deviation of a 0.2 that truncates at 1.5o. The rest of the pre-training hyper-parameters are listed in Table 2 below. The task embedding space of FB the d-dimensional hyper-sphere since the optimal policy n. invariant to the scale of the reward function. Note that OpTI-BFM cannot make this simplification because it tries to estimate the hidden parameter of the reward function and not the latent that is plugged into the policy. We can recover the latter easily with an L2 normalization.
E EXPERIMENT PROTOCOL
We evaluate methods common zero-shot RL benchmark (Touati al., 2023; Agarwal et al., 2025b). The benchmark consists of the Walker, Cheetah, and Quadruped environments from DMC (Tassa et al., 2018) with four tasks (reward functions) each. Note that each task has randomized initial states, so when evaluating single episode performance, e.g. Fig. 15, we report the mean over 20 episodes. And when evaluating task inference performance multiple episode, e.g., Figs. 2 and 16, we report the mean over 10 trials. All error-bars and shaded regions denote min-max-ine max-intervals over three training seeds around mean performance.
CUSTOM WALKER VELOCITY TASKS
The custom tasks implement for DMC Walker Fig. 5 consist of only the velocity tracking components the st and, walk, and run tasks. Further note that, by default, higher velocity is always allowed: running policy will also perform rather well in standing. For this reason, modify
31

---

Published as a conference paper at ICLR 2026
stand
wa
run
flip
600
Return
Return
Return
Return
500
600
300
600
400
200
400
300
2
10
6
10
10
10
Episode
Episode
Episode
Episode
Oracia
OpTI-BFM (ours)
OpTI-BFM grad. (ours)
(a) DMC Walker
walk
backward
backward
900
800
200
Return
250
Return
600
Return
150
200
700
100
150
600
100
200
50
Episode
Episode
Episode
Episodo
Oracle
OpTI- (aurs)
OPT grad, (ours)
(b) DMC Cheetah
stand
jmp
wa ak
run
1000
800
600
400
Retur
Return
Retum
Retum
600
400
300
400
200
400
200
8
10
Episade
Episade
Episade
Episade
OpTI-BFM (ours)
OpTi-BFM-grad, (ours)
(c) DMC Quadruped
Figure 18: Return of OpTI-BFM and OpTI- +grad. that use random-shooting shooting and gradient ascent respectively to optimize the UCB objective. The two variants perform very similarly.
the velocity reward bonus so that it is if the velocity matches the target exactly, and then linearly tapers 0.5 when off by From there, the reward then directly drops to
F INFERENCE
For each experiment and method perform grid-search over typer-parameters, We then choose the hyper-paranen parameters with highest overall cumulative return per environment for each method to report performance.
F.I OPTI-BFM
We found both OpTI-BFM and OpTI-BFM-TS to be very robust to the hyper- hyper-parameters tested. For each method we consider a single parem--aramettr with three values each: For UCB, we test a fixed 8, {1.0,0 0.1, 0.0013. For TS, test parameters where held constant. Specifically, p = 1 and X = not reported otherwise (Fig. 5). The number of samples for the UCB optimization was 128 throughout if not specified otherwise.
{0.1, 0.001, 0.001,0.0001) This means all other
F.2 LoLA
For LoLA we consider range of hhype--paaaeeted to trade-off exploration, update frequency, update step-size, and variance the gradient. We search combinations of:
horizon length and task embedding update rate (50, 250) as in (Sikchi et al., 2025);
32

---

Published as a conference paper at ICLR 2026
stand
wa
run
flip
Average Reward
Avarage Reward
0.8
Average Roward
Avarage Reward
0.6
0.8
0.6
0.3
0.4
0.4
0.2
10
10
10
10
10
10
10
10
10
10
10
* Samples
#ammples
Samples
#Samples
FM (x= 0.0)
0.03
0.3
(a) DMC Walker
walk
run
walk backward
run backward
0.20
Average Reward
0.8
Average Reward
Average Reward
0.8
Reward
0.15
0.6
0.6
0.4
0.1
0.4
0.10
0.2
0.2
0.05
0.0
0.0
0.0
0.00
10
10a
10
10a
10
10
10
10a
10a
10
101
10a
10
@Samples
#Samples
#Samples
#Samples
K=0.3
(b) DMC Cheetah
stand
jump
walk
run
Average Reward
0.8
Average Reward
0.6
Average Reward
0.6
Average Reward
0.4
0.6
0.4
0.2
0.2
0.1
10
10a
10a
10
10
10a
10
Samples
Samples
Samples
#Samples
OpTI-BFM
(c) DMC Quadruped
Figure 19: Average reward of OpTI-BFM at different numbers of requested reward labels (# Samples) for different information thresholds k. We stop interaction after 30k environment steps. k trades-off interaction cost with labeling cost. More than one order of magnitude less reward labels can still result the same performance in easier tasks.
learning rate {0.1, 0.05} in (Sikchi et al., 2025);
standard deviation of the Gaussian on the task embedding (0.05, 0.1, 0.2).
The batch size reported in Sikchi et al. (2025) is 5-10: we thus consider replay buffer of 1000, which results in effective batch size of 20, 10, and 4 for the respective update rates.
33

---

Published as a conference paper at ICLR 2026
stand
walk
flip
0.6
Arerage Reward
Arorage Roward
0.8
Arcrage Reward
0.4
Avorage Reward
0.8
0.6
0.4
0.2
0.2
10
10
10
10
10
10"
10
10a
10
102
10
10"
10
10
10
10
#Samples
#Samples
#Samples
#Samples
OFT (x= 0.0)
0.03
xm 0.1
km
0.3
(a) DMC Walker
walk
walk backward
backward
Average RoRard
Average Reward
0.2
Average Reward
Average Reward
0.10
0.4
0.1
0.2
0.05
0.2
0.0
0.00
10
10a
10"
10
101
10
10
10
10
10a
10
10*
10
10
10
10
#Samples
#Samples
#Samples
#Samples
OFTI-BFM-TS (c= 0.0)
0
x= 0.1
K= 0.3
DMC Cheetah
stand
jump
walk
Average Reward
0.8
Average Roward
Average Reward
Average Roward
0.6
0.4
0.4
0.2
0.1
10
10a
10P
10
10
10a
10a #Samples
10*
10
10a
10
10
10
10a
10a
10*
# 4ampples
Samples
#5amples
OPM (x- D.D)
0.03
K-
0.1
0.3
(c) DMC Quadruped
Figure 20: Average reward of OpTI-BFM-TS at different numbers of requested reward labels (# Samples) for different information thresholds K. Table
Name
Value
discount ?
0.98
batch size
1024
# training steps
2M
optimizer
adam
learning rate
le-4
target network update factor
0.01
weight of Cottho
1.0
Q-value penalty
0.5
fixed actor standard deviation actor sample noise clipping sampling
0.2
0.3
50% Durai and 50% Random
dimension d
50 L2 normalization
B network final activation
B network hidden dimension
256
F network hidden dimension
1024
マ network hidden dimension all networks first activation all other activations
1024
Layernorm Tanh
relu
34


Reviews:
[{"review_id": "nRxU354mKg", "reviewer": "Reviewer_afmy", "summary": "This paper proposes OpTI-BFM, an optimistic decision criterion that directly models uncertainty over reward functions and guides BFMs in data collection for task inference. Authors frame this online task inference problem as a linear bandit problem and maintain a probabilistic belief (a confidence ellipsoid) over the true task embedding z_r by performing real-time least-squares regression on reward-level feedback. Experiments on zero-shot benchmarks show that OpTI-BFM matches or surpasses offline reward inference methods with much less data.", "questions": "- How is the method's robustness to misspecification? For example, how does OpTI-BFM perform if the true reward function r(s) has a significant non-linear component, or if the pre-trained BFM is of lower quality (violating A1)? How does its degradation compare to the offline \"Oracle\" regression, which would also suffer from this misspecification?\n  \n- How does the random shooting for UCB optimization scale? Have you explored the sensitivity to this number of samples? Would a gradient-based approach be more robust or scalable?\n  \n- Could the theoretical guarantees be extended to the step-level update case?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- Tackles a significant and practical bottleneck. Online task inference with only a few active-interaction episodes is important for real-world applications.\n  \n- Solid theoretical guarantees via connections to linear bandit algorithms.\n  \n- Good empirical performance with additional experiments and analysis, e.g. episode-level updates and non-stationary rewards.", "weaknesses": "- The paper operates under assumptions of a perfect BFM / successor feature model. It is unclear what would happen for the theoretical guarantees or how the empirical results would change with approximation errors.\n  \n- The formal regret bound is proven for an episodic-update variant of OpTI-BFM. However, the experiments (Sec.5.3, Fig. 4) show that the step-update variant is empirically superior and converges much faster. While it is a positive result that the practical algorithm is even better, it means the theory doesn't formally cover the best-performing algorithm presented.\n  \n- The optimistic search for z is currently done by random shooting and may fail for larger, more complicated spaces.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uHE7d7iM1z", "reviewer": "Reviewer_eb95", "summary": "The authors tackle a key limitation of behavioural foundation models (BFMs) based on Universal Successor Functions (USFs): the need for a dataset of labeled (state, reward) pairs. For cases where such data may be impractical or prohibitively costly to acquire, an alternative framework based on actively collecting a smaller amount of data online during deployment is explored.\n\nTo navigate this new framework the authors propose OpTI-BFM(Optimistic Task Inference for Behavioural Foundation Models) which leverages the linear relationship between features and rewards to update its belief over the space of rewards (i.e., incrementally improves its estimate of the task embedding using observed rewards). OpTI-BFM maintains a confidence ellipsoid over possible task embeddings and selects actions optimistically to efficiently explore the reward space.\n\nLeveraging the fact that policy search for well-trained USF-based BFMs reduced to online optimisation of a linear function, a regret bound for OpTI-BFM in an episodic setting is established - the expected regret over n episodes $R_n\\leq \\mathcal{O}(d\\sqrt{n})$. OpTI-BFM is evaluated empirically using a common zero-shot RL benchmark (ExORL) consisting of Walker, Cheetah and Quadruped environments with four reward functions each. OpTI-BFM achieves oracle performance (upper bound) within five episodes (5k steps) on all tasks, outperforming LoLA and the “Random” (lower bound) baselines.", "questions": "Could the authors comment on whether OpTI-BFM generalises beyond continuous control environments?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "The authors introduce a new framework for task inference in BFMs without labeled offline (state, rewards) data. In this framework, the relationship between BFM policy search and linear bandits is exploited to develop, and prove a regret bound for, the OpTI-BFM algorithm for online task inference. OpTI-BFM is timely and tackles the problematic requirement for labeled data with implications for many real world applications. The empirical results support the efficacy of OpTI-BFM in three standard zero-shot tasks (Walker, Cheetah, Quadruped), outperforming LoLA and reaching Oracle-level performance within 5 episodes.", "weaknesses": "- Whilst the experiment section is quite strong already it could be improved further if the authors are able to show the performance of OpTI-BFM on an alternative environment to those of the DeepMind Control suite. e.g. an alternate task with pixel observations.\n\n- There is not much discussion of how OpTI-BFM could be deployed for real-world use. The authors say that their method would enable BFMs to work “beyond domains in which rewards are readily available”, but it is not obvious to me how it would interact with a real environment to get immediate reward labels. It would be appreciated if this could be explained further by the authors.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9Hf9a1RnIC", "reviewer": "Reviewer_REKP", "summary": "The paper adresses task inference for BFMs without relying on a labeled “task‑inference” dataset at test time. Instead of estimating a point task embedding offline, the agent actively interacts with the environment and updates a belief over reward parameters. The core idea is to view policy search over task embeddings as linear bandit optimization that comes from the insight that for USF-based BFMs, the relationship between successor features and returns is approximately linear. So the paper proposes OpTI‑BFM, which maintains a least‑squares estimate over the unknown reward weights, then selects the task embedding optimistically. It provides theoretical sublinear regret bounds $O(d\\sqrt(n))$ demonstrates empirically that the method can identify tasks within 5 episodes on DMC benchmarks.", "questions": "- Add controlled experiments that systematically break assumptions. For example inject bounded bias into $\\psi$ and report regret vs.\\ SF error or create rewards with a tunable component orthogonal to $\\phi$ or ablate sub-Gaussian noise level. Reporting regret/performance vs.\\ the projection error $\\|r - \\phi^\\top z\\|$ would make the empirical section align with A2/A1. \n\n- Provide either 1) a theoretical extension to per-step updates, or 2) a head-to-head comparison that keeps the episodic-only variant as the default and shows the practical gap in other settings too, with a candid discussion of why the bound should still be true.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- One of the core original ideas is the new task‑space bandit formulation for BFMs. The paper formulates online task inference as linear bandit optimization in the task-embedding space: with well-trained USFs, the expected episode return is approximately linear in the successor features of the policy conditioned on a task embedding, i.e., $\\mathbb{E}[\\hat{G}_k \\mid s_0, \\pi_z] \\approx \\langle \\psi(s_0, z), z_r \\rangle$. This lets the agent choose $z$ using a rule over confidence sets on the unknown reward weights $z_r$ (Eqs.~(7)--(10), \\S3.2). The ``two-context'' twist (estimating with features $\\phi$ but acting with successor features $\\psi$) is new in the BFM literature and differentiates the method from standard LinUCB.\n- Another contribution shows that running least-squares on reward-level pairs $(\\phi, r)$ yields tighter confidence sets than regressing on episode-level returns and empirical SFs $(\\tilde{\\psi}, \\hat{G})$.\n- The paper has efficient implementation details that are non-trivial. For example: the optimizer is computationally light and table 1 shows that the machinery is deployable in real-time control. \n- The paper is mathematically sound and strong.", "weaknesses": "- The theory assumes perfect USFs and that the policy conditioned on $z$ is (near) optimal for reward $z$ (A1), strictly linear rewards with sub-Gaussian noise (A2), and an optimization oracle for Eq.~(10) (A3) and are introduced before Algorithm1 on p.5. In practice, USFs are learned with function approximation and the acquisition is solved by random shooting. The paper notes ``we found OpTI-BFM to perform well even when [A1--A2] are violated'' but does not quantify robustness to misspecification. \n- Also the regret bounds are proven for a variant that only updates $z$ at episode starts (\\S3.3), while the recommended/practical algorithm updates $z$ every step, which is empirically much better (Fig.4). So the bound doeen't exactly cover the method actually used. (p.5; Fig.4 p.8).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pQycvz9lpO", "reviewer": "Reviewer_pxZn", "summary": "This paper proposes an ICRL (Interactive or In-Context Reinforcement Learning) algorithm based on Behavior Foundation Models (BFMs). The method performs online task inference by estimating the task embedding from observed rewards during inference, enabling dynamic policy adjustment. The authors provide partial theoretical analysis and show that the proposed method can achieve near-oracle performance within only a few episodes. Moreover, the approach can be extended to handle non-stationary reward settings.", "questions": "1. How does the proposed estimation procedure perform in sparse-reward settings (e.g., when the reward is only given at the end of an episode)?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The method is grounded on a solid theoretical foundation.\n\n2. It is innovative and computationally efficient.\n\n3. The approach is extensible and may inspire further research in this direction.", "weaknesses": "1. The experimental section only compares against Oracle and LoLA; it should include comparisons with other ICRL methods and evaluate optimization speed under out-of-distribution (OOD) settings.\n\n2. In Appendix A5.3, line 1036, equation (84), the variable _x_ should likely be _ψ_.\n\n3. In Algorithm 1, the formula for updating the estimator should reference the corresponding equation number for clarity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)