APC-RL: Exceeding data-driven behavior priors with adaptive policy composition
Authors: Finn Rietz
Pedro Zuidberg Dos Martires
Johannes A. Stork
Venue: ICLR 2026 Poster (2026)
Keywords: Reinforcement Learning
Normalizing Flows
Demonstrations Data
Behavior Prior
Learned action space
Primary area: reinforcement learning
TL;DR: We show how to use RL behavior priors of arbitrary quality to accelerate online learning while avoiding performance degradation when priors are misaligned with the target task.

Abstract: Incorporating demonstration data into reinforcement learning (RL) can greatly accelerate learning, but existing approaches often assume demonstrations are optimal and fully aligned with the target task.
In practice, demonstrations are frequently sparse, suboptimal, or misaligned, which can degrade performance when these demonstrations are integrated into RL.
We propose Adaptive Policy Composition (APC), a hierarchical model that adaptively composes multiple data-driven Normalizing Flow (NF) priors.
Instead of enforcing strict adherence to the priors, APC estimates each prior's applicability to the target task while leveraging them for exploration. 
Moreover, APC either refines useful priors, or sidesteps misaligned ones when necessary to optimize downstream reward.
Across diverse benchmarks, APC accelerates learning when demonstrations are aligned, remains robust under severe misalignment, and leverages suboptimal demonstrations to bootstrap exploration while avoiding performance ceilings caused by overly strict adherence to suboptimal demonstrations.

Published as a conference paper at ICLR 2026
APC-RL: EXCEEDING DATA-DRIVER BEHAVIOR PRI- ORS WITH ADAPTIVE POLICY COMPOSITION
Finn Rietz*
Pedro Zuidberg dos Martires Department of Computer Science Örebro University
Department of Computer Science Örebro University
Fakultetsgatan 1, Örebro, Sweden finn. cen..iiete rietz@oru.se
Fakultetsgatan Örebro, Sweden pedro. sabeergddos-aan de..deerggd-mmmrrreeeeee::
Johannes Andreas Stork Department of Computer Science Örebro University Fakultetsgatan 1, Órebro, Sweden Nohannesandress storkeoru.se
ABSTRACT
Incorporating demonstration data into reinforcement learning (RL) can greatly accelerate learning. but existing approaches often assume demonstrations are optimal and fully aligned with the target task. In practice, demonstrations are frequently sparse, suboptimal, or misaligned, which can degrade performance when these demonstrations are integrated into RL. We propose Adaptive Policy Composition (APC), a hierarchical model that adaptively composes multiple data- driven Normalizing Flow (NF) priors. Instead of enforcing strict adherence to the priors, APC estimates each prior's 's applicability to the target task while leveraging them for exploration. Moreover, APC either refines useful priors, sidesteps misaligned ones when necessary to optimize downstream reward. Across diverse benchmarks, APC accelerates learning when demonstrations are aligned, remains robust under severe misalignment, and leverages suboptimal demonstrations to bootstrap exploration while avoiding performance degradation caused by overly strict adherence to suboptimal demonstrations.
INTRODUCTION
Demonstration data are often used to make reinforcement learning (RL) (Sutton Barto 2018)
feasible in challenging tasks. Examples are regularizing the policy to imitate demonstrated actions (Lu et al. 2023 Zhu et al. 2018 Hester et al. (018)) and generative modeling of demonstrated actions or action sequences (Pertsch al.) 2021] Yang et al., 2022] Singh al. 2021). However, these approaches often make implicit but crucial assumptions about demonstrations, e.g. complete coverage of the state space and optimality, which unrealistic many practical settings. In fact, strictly adhering to the data even when the demonstrations are suboptimal or sparse is the main reason for failure when the dataset and the online RL task are misaligned (i.e., some way sub-optimal) [Dong al. (2025); Kong et al. (2024); Zhang et al. (2023). To avoid this, we should therefore only utilize demonstration data in reinforcement learning where and as long as it is pertinent for the online RL task.
In this paper, we present Adaptive Policy Composition (APC), a novel and flexible method of using demonstration data in RL, that does not rely on optimal and complete demonstrations. In APC, decide based on online feedback where and how long to rely on the data, instead always adhering the demonstrations. We propose hierarchical RL approach consisting of higher-level selector that decides between several actors on the lower level. Importantly, there are always exactly one prior-free and at least one data-driven, prior-based actor the lower level. Using several prior-based
"Corresoonding Author

---

Published as a conference paper at ICLR 2026
actors is possible and allows to include several distinct demonstration datasets that contain various behaviors. Our prior- prior-based based actors pre-train their own behavior prior with their demonstrations while the prior-free actor learns from scratch. Unlike previous approaches (Pertsch et al. 202I] Yang et al.) 2022] Singh et al. 202T), the inclusion of the prior-free actor provides APC with complete flexibility to diverge from demonstrations if necessary to optimize the online RL task, for example, when no demonstrations apply. Our approach allows the use of standard (off-policy) learning algorithms to optimize these actors.
Our contributions are threefold: (i) Algorithmic: We introduce APC, a novel compositional policy architecture that combines prior-based and prior-free actors under an adaptive selector. We further propose two key mechanisms reward- reward-sharing scheme that enables data-efficient training across actors, and a learning-fres arbitrator selector that mitigates primacy bias-both of which are crucial for robust and efficient performance. (ii) Empirical: We demonstrate that APC achieves strong robustness under demonstration misalignment, consistently outprrfoming prior methods such as PARROT and imitation learning baselines. (iii) Analytical: Through ablations, we identify the selector design and reward sharing as critical components for enabling stable and efficient exploration.
2 BACKGROUND
2.1 REINFORCEMENT LEARNING
RL problems are formalized as Markov Decision Processes (MDPs). A MDP is a tuple M = (S, A., p, 7), where S is the state space, A is the action space. r A + R is reward function, P:S S x A + denotes the discrete-time state- transition-kernn and y E (0, is a discount factor. that maximizes the discounted return objective
The goal in RL is to find policy S x A [0, 1]
Σ rr(ss an)
(TT ((TT)
where s, E S, a, E A. and (T ~ π) is a shorthand for denoting trajectories with actions sampled form the policy and the state evolving according to p. RL algorithms optimize the objective in equation| |
to identify the optimal policy x = max, J(T) which inherently requires a good exploration strategy, well as balancing the exploration versus exploitation trade-off. Uninformed exploration such as e-greedy (Sutton & Barto 2018) or temporally extended Brownian motion (Uhlenbeck & Ornstein 1930) appeal due their simplicity but often fall short in complex, long-horizon, sparse-reward MDPs. Exploration based on prior knowledge, e.g., demonstrations, can be more useful in such cases, which motivates our choice of composing multiple data-driven behavior priors.
2.2 DATA-DRIVEN POLICY PRIORS WITH GENERATIVE LATENT SPACE MODELING
Generative models of demonstrated actions implement effective data-driven priors for RL policies that learn in their latent space (Pertsch et al. 2021 Yang et al. 2022 Singh et al. (022). In this paper, we follow the PARROT (Singh al. 2021) approach and learn
conditiood Normalizing Flow (NF) (Rezende & Mohamed 20155 Papamakarios et al., 2021) model T(z;
with parameters é. The NF maps latent actions z from the base distribution ((z 0, I) the complex, multi-modal, per-state action distribution of the demonstration dataset D = (s,, a-) The NF is learned with likelihood maximization and matches the dataset distribution with the generative distribution
(())= = - (E log (T a; Φ); 0, + log det J (a: + const.,
(2)
where Ma;s Φ) T (a; Φ) = z is the inverse NF and J is the Jacobian (from the change of data- prior for RL by learning policy in the NF's
variables thcorem). The NF I serves latent space Z. Specifically, a latent policy Tz x Z - [0, 1] outputs a latent action z, which is then deterninistically mapped to MDP action T(z; s) refer to the resulting policy distribution in the MDP action space as the prior-based actor Ta(a (a s).
A through the NF transformation. We
RL in the latent action space Z of pre-trained NF is beneficial because it focuses learning and exploration on useful, demonstrated actions in A. Moreover, NFs are multimodal and allow a simple, unimodal latent policy n. induce a multimodal distribution in the action space. Finally, unlike
2

---

Published as a conference paper at ICLR 2026
PARROT
State
Poliyy
Reploy buffer
algo
74
B
APJ
Priors
Inverse transform
Replay buffers
RL algos.
qulii
Shate
Selector
Tull tralaed
Env
Bo)
SAC
29)
pall
AlO pracosioal
n
misznn sharing tric
Figure I: Architecture overview. (a): PARROT Singh et al. 2021 features a single latent policy and NF prior. (b): Our method uses high-level selector compose multiple latent policies and NF priors. A prior-free actor (index (0), dashed border) learns directly in the action space. The selected latent policy and NF prior (cyan-colored arrows) are executed at time t. A reward-sharing trick (magenta-colored arrows) allows us to compute the latent coordinate z corresponding to the executed action ag, and use the transition at time to also update the other actors that were not selected.
other latent space generative models used as data-driven priors in RL such VAEs (Pertsch et al. 2021 Yang et al. 2022 NFs are invertible. This means that the prior-based actor, Ta s). in theory, undo the behavior prior learn any desired policy. However, in practice this has been found to infeasible, as reported by (Singh et al.] (2021) and confirmed in our evaluation, leading to permanent influence of the behavior prior and resulting in failure under misalignment. This serves as the primary motivation for our method, which we present in the following section.
METHOD: ADAPTIVE POLICY COMPOSITION
Assume that we are given n 2 demonstration datasets D(1), D(,),...,,DD D(n),, each D()) consisting of state-action pairs, (s, a) E D(U), without reward signal or temporal order. APC leverages these demonstrations to learn overall policy aaa s) by pre-training set of prior-based actors, each based on its own dataset () (see See. 2.2 and then learning latent policies from reward feedback by interacting with the online RL task in all actors, including the prior-free actor. APC is a hierarchical reinforcement learning approach: The overall policy (a s) is composition of the lower-level actors, controlled by a high-level selector that decides which actor to execute in each state. The prior- based actors solve tasks efficiently when demonstrations performance under misalignment. In comparison, the prior-free actor lacks demonstration guidance retains full flexibility learn from reward feedback alone. This compositional architecture allows APC to both exploit distinct behavoir priors for efficient exploration and various aspects of the target task, while the prior-free actor can overcome limitations of the prior-based actors.
aligned but fail to achieve optimal
The remainder of this section is organized as follows: model, consisting of multiple prior-based actors, the prior-fres free actor, and the high-level selector. overview is provided in Fig. Sec. 3.2 describes the online learning procedure for the lower-level actors, and Sec. 3.3 introduces a crucial technique for robust and efficient learning with multiple NE priors. Finally, Sec. 3.4 presents the key design of our high-level selector.
Sec. 3.1 formalizes the compositional policy
3.1 COMPOSITIONAL POLICY MODEL
Our policy model composes multiple lower-level actors to efficiently solve the online RI. task. The higher-level selector T is conditional categorical distribution πæ Cat(po(s), (s(),.... Pn (s)) with support 3 E {0, 1,..., n that decides which lower-level actor to use in state s. The values (s(s) the probabilities of selecting the i-th lower-level actor in state s, which means that c = i
s) = (s). The set of lower-level actors consists of the prior-free actor na and n prior-based actors (1)
ma 1 < S The prior-free actor πa
(0) identity flow ( (z; s) and the latent policy
3

---

Published as a conference paper at ICLR 2026
(0) Sx 2 [0, 1]. while each of the prior-based actors na (1) T2
consists of NF that is pre-trained
on the demonstration dataset D( as described in Sec. 2.2
2.2 and a latent policy nT : x 2 - [o, 1].
To guarantee invertible flows, we also set |2| = |Al. The overall policy's s action distribution,
x
a
(8)) ( (a: ())ssss))))))))))))) + log det Jace s)]: TB( Tzz
likelihood of a under x[3")
change of variables
a mixture over lower-level actors, weighted according to the selector TH and computed based on change of variables with the latent policies and pre-trained NFs. In equation 3 we use 3 to index latent policies and NFs for notational clarity.
n) and latent actions z E Z. Instead APC uses MDP actions E A, selector actions 3 E {0, 1,.,,...,,1 of sampling directly from the complex density π (a s) in equation at each time step t, we first obtain Be from the selector Th and then sample a latent action Z₁ from the chosen lower-level actor's
(8.) which we deterministically transform with the corresponding NF T(B) to obtain latent policy π2
the action a, for the online RL task. Our approach to learning <a s) is to learn the selector and
each of the latent policies separately, as detailed in the following sections.
3.2 LOWER-LEVEL ACTOR LEARNING
As described above, our model contains + 1 latent policies T2 that independently interact with the online RL task when they are chosen by the higher-level selector g, and thus observe transitions (s. z. s'). We opt to run + 1 parallel instances of Soft Actor-Critic (SAC) (Haarnoja et al. 2018), one for each of the lower-level actors, though any other off-policy RL algorithm that supports continuous action spaces could be used instead. We denote g(() as the parameters for the latent policies and g)(0) for the latent Q-functions. Each SAC optimizes entropy-regularize RL objective and learns a unimodal Gaussian policy with so-called actor and critic updates. For more details on SAC, we refer to (Haarnoja et al., 2018). Note that the parameters ((()) of the pre-trained NFs ( are not updated during online learning, only the SAC parameters change, as per /Singh al. 2021 For more implementation and model details, we refer to App.
We maintain separate replay buffers B(0), ((() _B)),.,,,, B(n) for the n + latent policies, which is motivated by the fact that the same latent coordinate can correspond to different MDP actions under the different NF behavior priors. This implies that the reward r; resulting from the latent action zy only provides valid feedback signal for the low-level actor that generated a, Thus, at every time step we fill the replay buffer B( with the transition (s,, Zz, Tt, St+1) and update the parameters g(&) and vy(&e) of the selected lower-level actor on a batch of transitions sampled from buffer ((.)) Next, we introduce a crucial mechanism that ensures both robust and efficient learning with multiple (prior-based) lower-level actors.
3.3 REWARD SIARINGG TRICK FOR BALANCED ACTOR UPDATES
(9:) of the lower-level The basic learning scheme outlined above updates only the latent policy m2 actor selected at time step f. As the number of actors increases, this allocation of experience reduces sample efficiency: transitions are spread across multiple replay buffers, resulting in fewer updates per actor relative to the total number of environment interactions. More importantly, this setup can bias the higher-level selector: if a suboptimal prior-based actor initially outperforms, for example, the prior-free actor, the selector may overcommit to it, preventing crucial exploration of alternative actors that could potentially later outperform the actor that initially performs best.
To address this issue, we exploit the invertibility of NFs. Any executed action a, can be mapped into the latent space of every prior-based actor via z: T st). where the same environment action a gets mapped to different latent coordinates z, * z! This happens because different demonstration datasets ( and D(1) induce different behavior priors and, as such, different transfor- mations ( and TÚ), Thus, using the inverse 7 of these learned transformations, transitions can also be constructed for all actors * 3e that were not selected at time t, and their replay buffers B() can be populated with the constructed transitions. In this variant, each replay buffer receives transition
4

---

Published as a conference paper at ICLR 2026
at every step, enabling all actors update continuously, independent of which one produced the executed action at time Our ablation results confirm that this feedback shabbin mechanism is essential: it not only improves sample efficiency but also prevents primacy bias a tendency of RL algorithms to "overfit early experiences that damages the rest of the learning process" (Nikishin et al. 2022
Xu et al.
2024 by ensuring fair learning progress for all actors.
3.4
HIGH-LEVEL SELECTOR
The high-level selector n determines which lower-level actor to execute in each state. We adopt learning-fres arbitnator (Russell Zimdars, 2003) design for implementing the selector, where the selection probabilities (s) are derived directly from the value estimates V (s) of the lower-level actors. Concretely, the probability of selecting lower-level actor / is
1 pr = exp L (V))))))
Z = exp (1)
which defines a categorical distribution Th = Cat(Po (s) .......... (s)), where 1j is a temperature parameter that controls the sharpness of the distribution. However, because lower-level agents implemented with SAC, which does not provide a direct value function estimate, we approximate (*((ss) via Monte Carlo estimates of each lower-level agent's Q-function. In practice, this amounts to sampling a single latent action ( from each actor and evaluating (s( N exp( 1 (s, )!))), which yields an unbiased, though high- variance, estimate.
Implementing the selector learning- free arbiter provides two advantages: First, it eliminates the computational overhead of training a separate high-level neural network policy with additional parameters and gradient updates. Second, it circumvents the instabilities of hierarchical RL that arise when jointly optimizing higher- and lower-level agents, including sensitivity to learning-rate tuning, non-stationarity of the lower-level policies, primacy bias, and exploration difficulties. Our ablation results confirm that this design substantially improves stability and performance compared to learned higher-level policies.
EVALUATION AND ANALYSIS
We evaluate APC across continuous-come control benchmarks to assess its robustness under demonstration misalignment and efficiency with access to demonstrations aligned with the online RL task. Our experiments address four central questions: (i) Can APC remain robust under severe demonstration misalignment, avoiding the performance degradation observed in prior methods? (fi) Can APC effectively exploit well-aligned demonstrations to accelerate learning? (iii) Can APC exceed the performance of suboptimal demonstrations? (iv) Which architectural components are critical for cnabling such robust exploration?
4.1 ENVIRONMENTS
We test APC on the environments shown in Fig. 2
All environments, including data collection procedures and experimental setups, are described in greater detail in Appendix
Maze Navigation: Based the well-known D4RL benchmark (Fu et al., 2020). Starting from the center, point mass agent must reach different goal locations in simple maze. We refer the four goal locations as separate tasks. The state contains the agent's current position and velocity, and actions correspond to acceler- ations the 2D plane.
EC
Figure 2: Our environmental testbed, from left to right: Maze Navigation, the different goals marked in red. Franka Kitchen, with the manipu- lation targets marked in yellow. Car Racing.
Based on Guppt et al. 2019), kitchen environment where a Franka Emika Panda robot needs to solve various manipulation tasks. The state is in 5R and contains symbolic
Franka Kitchen:
5

---

Published as a conference paper at ICLR 2026
Ours SAC
PARROT
100%s
100%
Qfitte
0%
100%
IL
0%
100%
100%
0.6
14
1.6
1.0
166
Steps until. success
le6
Steps until % success
166
(a) Experiment (i): Average performance on tasks with misaligned demonstrations, showing that PARROT and IL fail to Icarn, while APC solves the tasks oven faster than from-scratch SAC, despite the misaligned prior.
(b) Experiment (ii): Average performance on tasks with aligned demonstrations, showing that PAR- ROT solves the tasks most quickly, while APC considerably outperforms IL, QFilter, and from- scratch SAC.
Figure Time success in the PointMaze Navigation environment. Each method was executed for most 1.5M environment steps; each experiment was repeated with three random seeds. Bars indicate the step at which the cross- seed average running success rate reached 100%, or the final success rate after 1 1.5M steps if convergence was not achieved earlier (shorter bars are better). Percentage annotations denote the cross-seed average running success rate at that time (3 seeds).
information about the agent and the manipulatable objects in the kitchen. Actions correspond 9D joint velocities.
Car Racing: A top-down car racing environment from the Gymnasium suite (Towers al. 2024). The task is to drive fast laps on the racing track, and actions correspond to steering and braking or accelerating.
4.2 BASELINES
We include the following baselines. (SAC) (Haarnoja al. 2018) serves standard, from-scratch RL. baseline for continuous action spaces. Comparing with this baseline reveals the acceleration in learning due including (task-aligned) demonstration data and potential performance degradation due to misaligned demonstrations. Our next baseline is a simple yet powerful imitation learning (IL) approach Lu et al. (2023) that regularizes the policy by enforcing high likelihood for state-action pairs in the demonstration data. (QFilter) (Nair et al. 2018) is an extension of the IL baseline that only includes the imitation loss for (s, a) demonstration tuples that have a higher Q-value than the action sampled from the online learning policy in state s. This baselines therefore also has the ability to exclude misaligned demonstrations from negatively affecting online performance. Lastly, (PARROT) (Singh al. 2021 serves to reveal the increased adaptability of our method when using NF prirrs
4.3 EXPERIMENTS & FINDINGS
(i) APC shows robustness under demonstration misalignment: We first evaluate robustness against demonstration misalignment in the PointMaze Navigation environment. Each method (except SAC) is provided with expert demonstrations ( for one task from the four possible goals, and then evaluated on the remaining three tasks for which the demonstrations are misaligned. As shown in Fig. 3a both PARROT and imitation learning (IL) fail to reliably solve the three tasks with the misaligned prior: after 1.5M steps their cross-seed running success rates remain at 0% and 7%, respectively. In contrast, APC reliably converges to 100% success in roughly 0.5M steps. Surprisingly, APC even outperforms from-scratch SAC despite the misaligned prior, indicating that APC can exploit misaligned priors for efficient exploration. Imitation learning combined with the QFilter is also able to avoid complete performance degradation due to the misaligned demonstrations on the PointMaze environment, but learns considerably slower than APC, due relying more heavily accurate Q-function estimates and lacking the ability to exploit pre-trained behavior priors.
We observe consistent trends in the higher- dimees-ai FrankaKitchen environment. End-effector trajectories sampled from pre-trained NF priors (Fig. 4a) exhibit diverse behaviors, but similarity in trajectory geometry or task semanties is not predictive of transfer success (Fig. 4b). When

---

Published as a conference paper at ICLR 2026
Hunge Cobinet
Bottom Burser
Macrevouv
$11de CAbiney
Light Seifcl
Top
(a) 3D FrankaKitchen end-effector trajectories sampled from the per-task pre-trained NF priors. The red dot indicates the same starting state for cach task; the trajectories have been shifted to save space.
Microwave
Kettle
Light Switch
Slide Cabinet
100s
100%s
120
07
100%
0%
100%
11%
53%
100%
DOWS
0.1M
0.5M
L.O
0.1M
0.5M
1.0M
0.1M
0.5M
1.0M
0.1M
0.5M
1.0M
Hinge Cabinet
Top Burner
Bottom Burner
34%
Ours
Parrot
0%
0%
TT.
SAC
60%
0%
QPilter
100%
1000
0.1M
0.5M
1.0
0.IM
0.5M
1.0M
0.1M
0.5M
1.0M
(b) Time to success when using prior data from different tasks (panel titles) while optimizing the mi crovave task. Bars indicate the step at which the cross-seed average running success rate reached 100%, or the final success after 1M steps if convergence was not achieved earlier (shorter hars are hetter). Percentage annotations denote the cross-seed average running success rate at that time (3 seeds).
Figure 4: Results on FrankaKitchen's 's mi microwvve task, which requires opening the microwave door. APC efficiently solves the task when exposed to aligned demonstrations (experiment (ii), top-left panel) while remaining robust under demonstration misalignment (experiment (i), remaining panels).
demonstrations misaligned, PARROT and IL suffer severe losses in sample efficiency fail entirely, underscoring their brittleness, APC, however, consistently solves the target task across all configurations, demonstrating strong robustness even in complex MDPs. This contrast highlights the importance of APC's ability to bypass misaligned priors, which is essential for adaptability under demonstration misalignment. Results for additional target tasks are reported in the appendix.
(ii) APC efficiently exploits aligned priors: We next consider the case where the demonstration prior fully aligned with the online RL task. As expected, PARROT achieves the fastest convergence in this setting (Fig. 3 right; Fig. 4b) top left), since its behavior cloning pre-training step allows it to solve the task optr-oppimally just by random sampling in the latent space. APC, despite having access to the same near-optimal prior, must additionally learn accurate Q-function estimates for the arbiter to identify the beneficial prior, resulting in slightly slower convergence. Compared to the IL baseline, APC converges substantially faster on PointMaze and slightly slower the FrankaKitchen environment, with neither method dominating overall. Importantly, all methods strictly outperform from-scratch SAC, confirming that all approaches are able to exploit the aligned demonstrations to accelerate learning. These results show that APC's increased adaptability under misalignment does not come at the cost of significant sample inefficiency in the aligned setting.
(iii) APC exceeds suboptimal demonstrations: We further evaluate APC in the CarRacing envi- ronment under a different but equally challenging form of demonstration misalignment. Instead of optimal demonstrations from related tasks, we collect x 30k transitions from a human driver on the target track. This dataset is used to pre-train the NF behavior prior and is also provided to the IL baseline. The left panel in Fig. shows the resulting return curves. SAC, trained from scratch, reaches optimal performance ( 900 return) after roughly 250k steps. PARROT, however, only marginally improves over the mean human score, indicating that the suboptimal prior imposes strict performance ceiling. The IL baseline eventually surpasses human performance, but learning is slowed considerably by the imperfect demonstrations. QFilter also exceed the human performance and is less strongly affected than by the sub-optimal demonstrations than IL, achieving optimal returns after roughly 100k steps. APC achieves optimal returns in circa 30k steps, outperforming SAC and all demonstration guideed baselines. These results show that APC can exploit suboptimal priors to warm- start learning and guide exploration, while avoiding the performance ceilings that constrain existing
7

---

Published as a conference paper at ICLR 2026
800
600
Roturn
Raturn
400
Oars
Achitrutor (reword sharing)
200
Arbitrutor (no rewerd sharing)
QFille
Net (reward shartng)
Human
Net reward sharing)
100k
150k
200k
250k
50k
100k
Stops
Figure 5: Return curves the car racing environment, the shaded area corresponds to one standard deviation around the mean, averaged over three seeds. Left: Experiment (iit) showing APC's performance relative to our baselines. Right: Experiment (iv) shows the performance of multiple APC ablations.
Episode
Prior-based
-700
Episode 20
Prior based
Prior-free
500
Episade 10
Prior-Free Prior-Free Prtor-besed
Prior-besed
Eptaodic
Episode
Episode 0-
Prior Prior-Fres
1030
Episode
Prior free
750
Epiontic return
Episode 20
Prior-Free
Prior- besed
500
Episode
Prsor-hased
250
Episode 5
Prtor-Tree
Episode 0
Prior based Prior-free
800
1000
Figure 6: Qualitative visualization for experiment (iv) showing selector decisions the CarRacing task. The rows in both panels correspond selected evaluation episodes. Along each row, time steps progress from left right, and each marker indicates whether the prior-based or prior-free actor was chosen at that step. Top panel: Decisions made by the parameter-free arbitrator selector with lower-level reward sharing. Bottom panel: Decisions made by learned selector without reward sharing. This shows that the arbitrator selector exploits the prior-free actor to achieve high returns, while the learned selector overcommits to the prior-based actor and achieves subpar returns. An extended figure showing all ablations and more episodes can be found App.
methods. This makes APC especially valuable in practical settings where suboptimal demonstrations readily available, while expert demonstrations might be scarce.
(iv) Arbitrator and reward sharing are crucial for exploration: To disentangle the contributions of the arbitrator selector architecture (Sec. 3.4) and the reward-shaaee sharing trick for efficient learning (Sec. 3.3 we performed an ablation study in the CarRacing environment.
We compare four variants: (i) full method with arbitrator selector and reward sharing, (ii) an arbitrator selector without reward sharing, (iii) hierarchical learned selector with reward sharing between the latent actors, and (iv) hierarchical learned selector without reward sharing between the latent actors. The learned hierarchical selector is also optimized with SAC and attempts to maximize the same environment reward as the latent actors. action space is discrete, parameterizing Categorical distribution representing the choice over which lower-level actor to execute at each step t.
The corresponding return curves the right panel of Fig, 5 (right) show a clear separation: only our full method consistently achieves optimal return. Two observations explain these results. First, replacing the arbitrator-style selector with a learned high-level policy introduces strong primacy bias. As shown in Fig. (bottom), the learned selector overcommits to using the prierd actor, as it initially achieves higher return than the randomly initialized prior-fit free actor. Once this bias is reinforced, the prior-free free actor rarely used and cannot quickly improve, though it could
8

---

Published as a conference paper at ICLR 2026
ultimately surpass the NF prior. In contrast, the arbitrator avoids this failure mode by directly comparing value estimates across actors, without learning an additional policy.
However, second, the arbitrator alone does not suffice. Without reward sharing, transitions collected by the prior-based actor benefit only that actor, further amplifying its dominance. Using the arbitrator alongside reward sharing (Fig- 6 top) ensures that all actors are updated on every transition, enabling the prior-free actor to learn from higher-quality trajectories produced by the prior-based actor. This allows rapidly improve its Q-value estimates and eventually outperform the prior-based actor.
These results show that both mechanisms are critical: the arbitrator mitigates primacy bias, while reward sharing ensures fair competition through data-efficient learning. Our design combines both to achieve robust exploration and accelerated learning without premature convergence to suboptimal actors.
4.4 SUMMARY OF RESULTS
Across environments and settings, our experiments demonstrate three consistent findings, First, APC remains robust under misaligned demonstrations, reliably solving target tasks where PARROT and IL fail, and in some cases even outperforming from-scratch SAC by exploiting misaligned priors for exploration. Second, APC effectively leverages aligned demonstrations: while slightly slower than PARROT under perfectly aligned priors, APC consistently outperforms from-scratch SAC, showing that its added flexibility does not compromise sample efficiency. Third, APC exceeds suboptimal demonstrations by bootstrapping from imperfect data without imposing performance ceilings from the data. Finally, our ablation studies confirm that the arbitrator-style selector and seward-sharigg mechanism are both necessary to prevent primacy bias and ensure fair, data-efficient competition among actors. Together, these results highlight APC's robustness, efficiency, and adaptability across diverse demonstration settings.
RELATED WORK
Normalizing Flows for RL In addition to using pre-trained NFs as data-driven action priors, NFs have also been used to replace simple unimodal Gaussian policies with richer, multimodal distributions, with the reported benefit of improved exploration and sample efficiency (Ward et al. Mazoure al. 2020). In these approaches, the NF parameters are learned jointly with the policy during online RL, effectively treating the flow as part of the policy network. In contrast, our method keeps the NF prior fixed after pre-training it demonstrations. NFs have also been leveraged to enforce safety constraints by mapping the action space into a constraint-respecting action subspace (Brahmanage et 2023 Chen et al. (023). similar to "invalid action masking" techniques (Kalweit et al. 2020 Huang & Ontañón 2022, Rietz et al. 2024). Our use of flows differs from these works: rather than masking out forbidden actions, we bias the policy by searching in the NF prior's 's latent space to guide exploration towards behaviors observed in the demonstration data. Using pre-trained NFs as demonstration-de driven action priors was introduced by [Singh al.) (2021), who argue that the invertible nature of NFs allows for flexible adaptation to the online task. Our results, however, show that a misaligned NF prior is hard to escape in practice, and that our hierarchical design, which explicitly allows the agent to bypass misaligned priors, greatly increases adaptability and robustness under distribution shift.
Skill-based Hierarchical Learning A large body of work exploits hierarchical architectures to accelerate learning by introducing temporal abstraction. In these approaches, high-level policy selects between discrete options or "primitives" (Sutton et al. 1999. Fox et al.) 2017: Ajay al.) 2020; Kulkarni et al.) 2016). which may be obtained from demonstrations, learned through un- supervised exploration (Eysenbach et al.) 2018 Park et al.. 2024 2023). or provided as scripted controllers (Nasiriany et 20|] Chitnis et al. construct a continuous latent embedding of skills" (Pertsch et al. 2021 Yang et al. 2022 Rana
et al. 2020). Other methods instead
et al. 2022), and solve downstream tasks by searching over the latent space. The works lack means for adapting when the primitives or skills do not suffice for solving the task. An exception to this is MAPLE (Nasiriany al. 2022), which also learns an online policy to improve upon the given scripted controllers. Our work differs from these since we do not leverage temporal abstraction for exploration but instead focus on robustness and adaptability under demonstration misalignment.

---

Published as a conference paper at ICLR 2026
Offline to Online RL A complementary line of work accelerates online RL by leveraging pre- collected offline datasets MDP transitions D (s, a, r, s (Levine et al. 2020 Xie al. 2021). Ball et al. (023)) balance online and offline data through joint sampling in off off-po RL. Nair al. 2020) pre-train policy offline and constrain the subsequent online policy remain close to it. Zhang et al. 2023), Hu et al. (2024) pre-train both the policy and value function, and then refine the value function online while using both the offline and online policies as proposal distributions. Kong et al. (2024) adopt similar proposal-policy scheme but periodically reset the online policy to counteract primacy bias. While effective, these methods primarily target the distributional shift between offline and online RL. Crucially, they also require reward-labeled data for pre-traie tr-rianniin, whereas our approach relies only on unlabeled demonstrations to train NF priors.
Learning from Demonstrations Learning from demonstration has long history in RL. Most approaches incorporate demonstrations through explicit imitation losses that encourage the policy to stay close to the demonstrated behavior (Ross et al.
201
Hester et al. 2018 Goecks et al. 2019
Fujimoto & Gu 202 Lu et al. 2023 Tiapkin et al. 2024.. generally assuming that demonstrations
are aligned with the target task and offer no mechanism to cope with substantially misaligned demonstrations. Inverse RL methods infer the reward function from demonstrations and subsequently optimize it with RI. (Abbeel & Ng. 2004; Ziebart et al. 20088 Ho & Ermon 2016). but likewise depend demonstrations that are near-optimal. Some recent works account for suboptimal demonstrations (Nair et al. [201 8 Zhao et al.) 2022] Hu et al. [2024] Dong et al. 20225] Cramer et al.
2025), but lack APC's ability to adaptive compose multiple distinct behavior priors with prior-free actor.
6 LIMITATIONS AND DISCUSSIONS
An apparent shortcoming of APC lies in its high computational overhead that scales linearly with the number of latent actors, since each actor is updated separately with SAC. While reward sharing improves sample efficiency, maintaining multiple parallel learners increases wall-clock time and limits scalability to larger sets of behavior priors.
Although APC is designed to remain robust under demonstration misalignment and distribution shift, may still fail in adversarial or contrived scenarios. If many severely misaligned priors all bias exploration toward ieee-irrleeant regions of the state-space, and if the reward signal is uninformative about this sub-optimality, then each actor's Q- Q-values might not allow the arbitrator to distinguish and avoid misaligned behaviors.
FUTURE WORK
We see addressing the computational demands of APC as important future work. This could be approached by maintaining and updating shared, central critic, while heuristically updating only the selected actor at time t, instead of updating all full actor-critics at each step. This might substantially reduce computational overhead and wall-clock time, while preserving APC's adaptive behavior and robustness under demonstration misalignment and distribution shift.
CONCLUSION
This paper proposes Adaptive Policy Composition (APC), hierarchical RL architecture that com- poses multiple NF priors with a prior-free fallhack actor under an adaptive selector. By combining a parameter-free arbitrator with reward sharing, APC ensures data-efficient learning across all actors and avoids primacy bias, enabling robust demonstration -guuudd exploration even under misalignment. Our experiments across diverse benchmarks show that APC leverages aligned demonstrations, re- mains robust under misalignment, and exceeds suboptimal demonstrations by using priors to bootstrap exploration. These findings demonstrate that APC is general approach for integrating imperfect demonstrations into online RL without impairing performance, thereby bridging the gap between data-driven priors and reward-driven adaptation.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This work was partially supported by the Wallenberg AI, Autonomous Systems and Software Program (WASP) funded by the Knut and Alice Wallenberg Foundation.
REFERENCES
Pieter Abbeel and Andrew Ng. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the twenty-firsi international conference on Machine learning. pp. 1, 2004.
Anurag Ajay, Aviral Kumar, Pulkit Agrawal, Sergey Levine, and Ofir Nachum. Opal: Offline primitive discovery for accelerating offline reinforcement learning. arXiv preprint arXiv:2010.13611, 2020.
PhilipJ J Ball, Laura Smith, Ilya Kostrikov, and Sergey Levine. Efficient online reinforcement learning with offline data. In International Conference on Machine Learning, pp. 1577-1594, PMLR, 2023.
Janaka Brahmanage, Jiajing Ling, and Akshat Kumar. Flowpg: action-constrained policy gradient with normalizing flows. Advances in Neural Information Processing Systems, 2023.
Changyu Chen, Ramesha Karunasena, Thanh Nguyen, Arunesh Sinha, and Pradeep Varakantham. Generative modelling of stochastic actions with arbitrary constraints in reinforcement learning. Advances in Neural Information Processing Systems, 36:39842-39854, 2023.
Rohan Chitnis, Shubham Tulsiani, Saurabh Gupta, and Abhinav Gupta. Efficient bimanual manip- ulation using learned task schemas. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pp. 1159-1155. IEEE, 2020.
Emma Cramer, Lukas Jäschke, and Sebastian Trimpe. Cheq-ing the box: Safe variable impedance learning for robotic polishing. arXiv preprint arXiv:2501. 007985, 2025.
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real NVP. In 5th International Conference on Learning Representutions, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Truck Proceedings. OpenReview.net, 2017. URL httttt::///ppereeeeeee met/fore
Perry Dong, Alec M. Lessing, Annie S. Chen, and Chelsea Finn. Reinforcement learning via implicit imitation guidance, 2025. URL
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function. arXiv preprint arXiv:1802.06074, 2018.
Roy Fox, Sanjay Krishnan, Ion Stoica, and Ken Goldberg. Multi-level discovery of deep options. arXiv preprint arXiv: 17X3:v/7003.824 2017.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning, 2020.
Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. Advances in neural information processing systems, 34:20132-20145, 2021.
Vinicius G Goecks, Gregory M Gremillion, Vernon J Lawhern, John Valasek, and Nicholas R Waytowich. Integrating behavior cloning and reinforcement learning for improved performance in dense and sparse reward environments. arXiv preprint arXiv: 2019.
Abhishek Gupta, Vikash Kumar, Corey Lynch, Sergey Levine, and Karol Hausman. Relay policy learning: Solving long-horizon tasks via imitation and reinforcement learning. In Leslie Pack Kaelbling, Danica Kragic, and Komei Sugiura (eds.). 3rd Annual Conference on Robot Learning. CoRL 2019, Osaka, Japan, October 30 November 2019, Proceedings, vol- ume 100 of Proceedings of Machine Learning Research, pp. 1035-1037. PMLR, 2019. URL http::/ppocceee ngs ml www.pra........a....m..mmmmmm.re wttps///w////us////eeee wttp://wtt/.........oo html

---

Published as a conference paper at ICLR 2026
Tuomas Haamoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference machine learning. pp. 1861-1870. Pmlr, 2018.
Todd Hester, Matej Vecerík, Olivier Pietquin, Mare Lanctot, Tom Schaul, Bilal Piot, Dan Horgan, John Quan, Andrew Sendonaris, Ian Osband, Gabriel Dulac-Arnold, John P. Agapiou, Joel Z. Leibo, and Audrunas Gruslys. Deep q-learning from demonstrations. Sheila A. Mcllraith and Kilian Q- Weinberger (eds.), Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018, pp. URL httph:tt//// /wwi:/ooiiioo org 10. 1609 www.:16699/a...vo. ww2.1.111.... 11757
AAAI Press, 2018. doi: 10. 10.1609/AAALLV321.11177
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. Advances in neural information processing systems, 29, 2016.
Hengyuan Hu, Suvir Mirchandani, and Dorsa Sadigh. Imitation bootstrapped reinforcement learning. In Dana Kulic, Gentiane Venture, Kostas Bekris, and Enrique Coronado (eds.), Robotics: Science and Systems XX, Delft. The Netherlands, July 15-19, 2024, 2024. doi: 10. 10.116677RRSS.222.XX.00 URL https: / /oi org/ 10. 11w.1116677555 RSS 2024 XX. 056
Shengyi Huang and Santiago Ontañón. A closer look at invalid action masking in policy gradient algorithms. In Roman Barták, Fazel Keshtkar, and Michael Franklin (eds.), Proceedings of the Thirty-Fifth International Florida Artificial Intelligence Research Society Conference, FLAIRS 2022, Hutchinson Island, Jensen Beach, Florida, USA, May 15-18, 2022. Florida Online Journals, 2022. doi: ...2473/FLAIRS.V3SL... 17351.13555
135..30584. URL http::ttp://// / org 10. 30t../2433/332
Gabriel Kalweit, Maria Hügle, Moritz Werling, and Joschka Boedecker. Deep inverse q-learning with constraints. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hisua--Tien Lin (eds.), Advances in Neural Information Processing Systems
33: Annual Conference Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020. URL
neurips cc pctppppaee///////aaaeee//eee/e/. .tml
Rui Kong, Chenyang Wu, Chen -iao Gao, Zongzhang Zhang, and Ming Li. Efficient and stable offline-to-onling reinforcement learning via continual policy revitalization. In Proceedings the Thirty-Third International Joint Conference on Artificial Intelligence, IJCAI 2024, Jeju, South Korea, August 3-9, 2024, pp. 4317-4325. ijeai.org, 2024. URL WtRl:ttps:// www. Jecai wwr.ca..orgoo org/ 2eneeeding///2224/55
Tejas D Kulkarni, Karthik Narasimhan, Ardavan Saeedi, and Josh Tenenbaum. Hierarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation. Advances in neural information processing systems, 29, 2016.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. CoRR, abs/2005.01644 2020. URL https://
httgg//bs/2005.. 01643
Yiren Lu, Justin Fu, George Tucker, Xinlei Pan, Eli Bronstein, Rebecca Roelofs, Benjamin Sapp. Brandyn White, Aleksandra Faust, Shimon Whiteson, et al. Imitation is not enough: Robustifying imitation with reinforcement learning for challenging driving scenarios. In 2023 IEEE/RSJ International Conference Inielligent Robots and Systems (IROS), pp. 7553-7560. IEEE, 2023.
Bogdan Mazoure, Thang Doan, Audrey Durand, Joelle Pineau, and Devon Hjelm. Leveraging exploration in off-policy algorithms via normalizing flows. In Conference Robot Learning, pp. 430-444. PMLR, 2020.
Ashvin Nair, Bob McGrew, Marcin Andrychowicz. Wojciech Zaremba, and Pieter Abbeel. Over- coming exploration in reinforcement learning with demonstrations. In 2018 IEEE international conference on robotics and automation (ICRA), pp. 6292-6299. IEEE, 2018
12

---

Published as a conference paper at ICLR 2026
Ashvin Nair, Murtaza Dalal, Abhishek Gupta, and Sergey Levine. Accelerating online reinforcement learning with offline datasets. CoRR, abs/2006.09359, 2020. URL htllatts//aaxiii....
abs/2005 09359
Soroush Nasiriany, Huihan Liu, and Yuke Zhu, Augmenting reinforcement learning with behavior primitives for diverse manipulation tasks. In 2022 International Conference on Robolics and Automation, ICRA 2022, Philadelphia. PA, USA, May 23-27, 2022, pp. 7477-7484. IEEE, 2022. doi: 10. (0..109//RA444499.00000 (2w..666399222222222222 URL https: / //w.i./diii/// org. 10. 1109/ ICRA46639 2022. 9812140
Exgenii Nikishin, Max Schwarzer, Pierluca D'Oro, Pierre-Luc Bacon, and Aaron Courville. The primacy bias in deep reinforcement learning. In International conference on machine learning, pp- 16828-16847, PMLR, 2022.
George Papamakarios, Eric T. Nalisnick, Danilo Jimenez Rezende, Shakir Mohamed, and Balaji Lakshmmnnrayyanan Normalizing flows for probabilistic modeling and inference. J. Mach. Learn. Res., 22:57:1-57:64, 2021. htt.h: 1ttp://r.raaapppe//////11111111
html
Seohong Park, Oleh Rybkin, and Sergey Levine. Metra: Scalable unsupervised rl with metric-aware abstraction. arXiv preprint arXiv:2310.08897. 2023.
Seohong Park, Tobias Kreiman, and Sergey Levine. Foundation policies with hilbert representations. arXiv preprint arXiv:2402. 15567, 2024.
Karl Pertsch, Youngwoon Lee, and Joseph Lim. Accelerating reinforcement learning with learned skill priors. In Conference on robot learning, pp. 188-204 PMLR, 2021.
Krishan Rana, Ming Xu, Brendan Tidd, Michael Milford, and Niko Sünderhauf. Residual skill policies: Learning an adaptable skill-based action space for reinforcement learning for robotics. In Karen Liu, Dana Kulic, and Jeffrey Ichnowski (eds.). Conference on Robot Learning, CoRI. 2022, 14-18 December 2022, Auckland, New Zealand, volume 205 of Proceedings of Machine Learn- ing Research, pp. 2095-2104. PMLR, 2022. URL v205/rana23a.0 http:/aannaaa.hmlll
mlr. press/
Danilo Jimenez Rezende and Shakir Mohamed. Variational inference with normalizing flows. In Francis R. Bach and David M. Blei (eds.), Proceedings of the 32nd International Conference on Machine Learning. ICML 2015, Lille, France, 6-11 July 2015, volume 37 of JMLR Workshop and Conference Proceedings, pp. 1530-1538. JMLR. JMLR.org, 2015. URL htttp::/proceeeeee mlr. wtt.ppess/v///eeeeeeecooco
html
Finn Rietz, Erik Schaffernicht, Stefan Heinrich, and Johannes A. Stork. Prioritized soft q- decomposition for lexicographic reinforcement learning. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024.
URL htttps:://ppereeiiiiee net.iiewweeeeeeeee.eme..ee.eeeee fortt/foumm20 hetrrmmmddd000yyyyG...
Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pp. 627-635, JMLR Workshop and Conference Proceedings, 2011.
Stuart J Russell and Andrew Zimdars. Q-decomposition for reinforcement learning agents. In Proceedings of the 20th international conference on machine learning (ICML-03), pp. 656-663, 2003.
Mohit Sharma, Jacky Liang, Jialiang Zhao, Alex LaGrassa, and Oliver Kroemer. Learning compose hierarchical object-centric controllers for robotic manipulation. arXiv preprint arXiv:201 1.04627, 2020.
Avi Singh, Huihan Liu, Gaoyue Zhou, Albert Yu, Nicholas Rhinehart, and Sergey Levine. Parrot: Data-driven behavioral priors for reinforcement learning. In 9th International Conference Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. hiRp:hhttts::/penreeete
net. forum?i d=Yuuv WOFeRR
13

---

Published as a conference paper at ICLR 2026
Richard S. Sutton and Andrew G. Barto. Reinforcement Learnin An Introduction. The MIT Press. 2nd
second edition, 2018. URL http:pp//iiccmmaaeeeeeeeeeeeooo. html
Richard S Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: framework for temporal abstraction in reinforcement learning. Artificial intelligence, 11((1-2):81121) 1999.
Danil Tiapkin, Denis Belomestny, Daniele Calandriello, Eric Moulines, Alexey Naumov, Pierre Perrault, Michal Valko, and Pierre Menard. aamoontraaion-regulaazzie RL. In The Twelfth International Conference on Learning Representations, 2024. URL net/ htt///ouummid-aaappdoooo
Mark Towers, Ariel Kwiatkowski, Jordan Terry, John Balis, Gianluca De Cola, Tristan Deleu, Manuel Goulão, Andreas Kallinteris, Markus Krimmel, Arjun KG, et al. Gymnasium: A standard interface for reinforcement learning environments. arXiv preprint arXiv:2407.17032 2024.
George E Uhlenbeck and Leonard S Ornstein. On the theory of the brownian motion. Physical review, 36(5):823, 1930.
Patrick Nadeem Ward, Ariella Smofsky, and Avishek Joey Bose. Improving exploration in soft-actor- critic with normalizing flows policies. arXiv preprint arXiv:1906002771, 2019.
Tengyang Xie, Nan Jiang, Huan Wang. Caiming Xiong, and Yu Bai. Policy finetuning: Bridg- ing sample-efficient offline and online reinforcement learning. Advances in neural information processing systems, 34:27395-27400 2021.
Guowei Xu, Ruijie Zheng, Yongyuan Liang, Xiyao Wang, Zhecheng Yuan, Tianying Ji, Yu Luo, Xiaoyu Liu, Jiaxin Yuan, Pu Hua, Shuzhen Li, Yanjie Ze, Hal Daumé III, Furong Huang, and Huazhe Xu. Drm: Mastering visual reinforcement learning through dormant ratio minimization. In The Twelfth International Conference on Learning Representations, 2024. URL https
/oppnnevviww net wtt.://fidddiooooeeoeioooomoi MSe
Quantao Yang, Johannes Stork, and Todor Stoyanov. Mpr-rl: Multi-prion regularized reinfoormmmme learning for knowledge transfer. IEEE Robotics and Automation Letters, 7(3):7652-7659, 2022.
Haichao Zhang, Wei Xu, and Haonan Yu. Policy expansion for bridging offinnett-online re- inforcement learning. In The Eleventh International Conference on Learning Representa- tions, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL https //openreview.o net www.eeoooummioooom -
Yi Zhao, Rinu Boney, Alexander Ilin, Juho Kannala, and Joni Pajarinen. Adaptive behavior cloning regularization for stable offline toniie--t--oline reinforcement learning. In 30th European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning, ESANN 2022, Bruges, Belgium, October 5-7, 2022, 2022. doi: 10. 442////AAAA22222225022222222 http::://doi..ooo 000. 14428442eee8 /et////annn//2// ESS20221100
URL
Yuke Zhu, Ziyu Wang. Josh Merel, Andrei A. Rusu, Tom Erez, Serkan Cabi, Saran Tunyasuvunakool, János Kramár, Raia Hadsell, Nando de Freitas, and Nicolas Heess. Reinforcement and imitation learning for diverse visuomotor skills. Hadas Kress-Gazit, Siddhartha S. Srinivasa. Tom Howard, and Nikolay Atanasov (eds.), Robotics: Science and Systems XIV. Carnegie Mellon University, Pittshurgh, Pennsylvania, USA, June 26-30, 2018, 2018. doi: 10. 10.15607/RSS.20 X0V..X...009 URL
http::/www..oool..ooooeeeeeg.gg. wttp:////.gorrr////////////////
html
Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, Anind K Dey, et al. Maximum entropy inverse reinforcement learning. In Aaai, volume 8, pp. 1433-1438. Chicago, II., USA, 2008
14

---

Published as a conference paper at ICLR 2026
A
REPRODUCIBILITY STATEMENT
Source code is available hupps//ggthub..omm/ferrskkk---
B ENVIRONMENT DETAILS
B.1 MAZE NAVIGATION
We adopt the maze navigation environment from D4RL. (Fu et al. 2020); however, we customize the maze layout as shown in Fig. 2 The agent corresponds to simple point mass, with actions A R corresponding to linear force exerted on the point. The observation space S E R contains the agent's current (I, y) position and velocity. The task encoding, defined by one of four distinct goal locations, is not part of the observation and must be inferred from the reward signal. This still yields a standard, fully observable MDP for each separate task.
The reward function is dense and defined as the exponential of the negative Euclidean distance between the agent and the goal. To encourage short episodes, we subtract a constant penalty of -1 each step. Episodes start from random position near the maze center, terminate successfully when the agent reaches within 0.5 units of the goal, and are truncated after 400 steps.
For each goal location i, we generate demonstration datasets D(L), ..... D(4) using extensively pre- trained, optimal policies ((*) Specifically, from (1)= and recording the resulting (s, a) pairs. These datasets are used either to pre-train the NF priors (one per task) or directly as input to the IL baseline, depending on the evaluation setting.
collect 100 episodes per task by sampling actions
B.2 FRANKAKITCHEN
We use the FrankaKitchen environment introduced by Gupta al. (0119) which features seven distinct manipulation tasks: opening the microwave door, pushing the kettle onto the correct stove burner, turning the bottom burner by rotating the corresponding knob, turning on the top burner by rotating the corresponding knob, flipping the 1 light swit to the on position, opening the s sliding cabine net door, and opening the hinge cabinet door. The state space S 5259 contains symbolic features describing all manipulable objects, along with the robot's 's joint angles and velocities. The action space task identity is not included in the state and must instead be inferred from the reward signal, yielding standard, fully observable MDP for each individual task.
R corresponds to joint velocity commands. The one-hot
To facilitate exploration and accelerate training, we replace the original sparse rewards with dense reward function. This modification was necessary given the high computational burden of evaluating a combinatorial number of tasks and prior settings for multiple seeds. Let Pee IR3 denote the end-effector position, computed as the midpoint of the left and right gripper fingers, and let ponj E R denote the position of the target object for the current task. We define the end-effector distance term
as
Fee -a, |eee- Pobjlla
(5)
with scaling factor 0.5. For each task k the environment additionally provides achieved goal (k) state gach and a desired goal state gies.
(R) We can thus compute a task success distance term as
(k)
Ptask gach
gáes 12,
(6)
which encourages the agent to bring the target object into its goal configuration (c.g., microwave door fully open). Our final dense reward function is then given by
Rguccesss
Teaskl S c,
r(s, a)
Tee + Ttasks otherwise,
where Reueress 100 a large completion bonus.
For each of the seven tasks i, we construct demonstration datasets D(I), ...,, D(7) using extensively pre-trained, optimal policies (()+, Each dataset consists of 100 episodes collected by executing (()+ and recording the resulting (s, a) pairs. These datasets are either used to pre-train task-specific NE
priors or passed directly the II. baseline, depending on the evaluation setting.
15

---

Published as a conference paper at ICLR 2026
B.3 CARRACING
We use the CarRacing environment from the Gymnasium suite (Towers et al. 2024), which requires driving fast laps top-down race track. The simulated planar car follows simplified vehicle dynamics that include skidding and varying friction across terrain types (asphalt vs. grass). The continuous action space is A R2, corresponding steering, acceleration, and braking. While the original environment provides pixel observations, we extract a symbolic representation directly from the simulator.
The symbolic observation space captures the vehicle's 's relative position, orientation, and motion with respect the track. At each timestep, the agent exposed to the following symbols:
Track-edge distances: signed distances to the left and right road boundaries, (thert, dright).
Heading error: orientation difference A between the car's heading and the tangent of the nearest track centerline, wrapped [- m..
Velocities: forward and lateral velocity components in the car's 's local frame, (Nowd, (side), and the angular velocity w.
Lookahead waypoints: relative positions of the next L 5 centerline waypoints in the car's local coordinate frame, {(T,. 1y)}
Formally, the observation vector is
[hee,, dright; A0, Uwwd, Uside: W, 21, VL,
IL.
R6222
(8)
which yields S R166 for = 5. Episodes begin with the car at rest at a fixed position centered on the track, and enforce deterministic resets such that the track layout remains identical across episodes.
We the environment's unmodified reward function: each step incurs a penalty of -0.1, and the agent receives a reward of + +1000/M, where M is the number of track-tiles visited during the current episode.
This environment contains only single task - driving efficiently on the fixed track. We collect demonstration dataset D(I) by recording 10 trajectories from a human driver, with an average return of approximately 250. This dataset is used either to pre-train the NF prior directly input the IL baseline.
C TRAINING AND MODEL DETAILS
C.I SAC
Our main learning algorithm is Soft Actor-Critic (SAC) (Haarnoja al. 2018). We follow the standard learning procedure described by Haarnoja et al. (018)) without modification, and use largely the same hyperparameters across environments (Tab. 1). with minor adjustments the discount factor and Polyak target coefficient to stabilize training, particularly in the CarRacing environment with its high-magnitude rewards. SAC employed in all of our experiments: (i) as from-scratch baseline, (ii) to implement the IL baseline, and (iii) to train the latent policies of the lower-level actors within APC and PARROT. To make for a fair and consistent comparison, the same SAC hyperparameters are used for all methods in each experiment.
C.2 NORMALIZING FLOW BEHAVIOR PRIOR
We implement the NF prior using a conditional version of the real NVP architecture (Dinh et al., 2017), which is composed of multiple affine coupling layers. Each affine coupling layer splits the input x E RD into two parts and computes the output y by applying a scale-and-shift transformation to one part, conditioned the other:
YP X(id],
Y(d+ = X[(+1:D] @ exp (u(X s)) (X s).
where v and q are neural networks that additionally take the state s as input, to learn different transformations in different states. Concretely, we implement v and g fully connected ML.ss with
16

---

Published as a conference paper at ICLR 2026
Table SAC hyperparameters used across environments.
Hyperparameter
PointMaze
Franka
CarRacing
Number of parallel environments
10
Replay buffer size
10t
x 106
1 x 10c 0.995
Discount factor
0.995
0.995
Polyak target coefficient
0.005
0.01
0.005
Batch size
256
256
256
Learning starts
10
x 10
x 10
Policy learning rate
x 10 4
3 X 10
3 x 10
Q-function learning rate
x 10
1 10-3
x 10-3
Entropy coefficient
0.1
0.1
0.005
Entropy autotune
False Fully-connected [512, 256] Adam
False Fully-comnected [512, 256]
False Fully-connected [512. 256] Adam
Actor network type
Actor hidden layer widths
Actor optimizer
Adam
Actor activation function
tanh
tanh
tanh
Critic network type
Fully-connected
Fully-connected [512, 256]
Fully-connected [512, 256]
Critic hidden layer widths
[512, 256]
Critic optimizer
Adam
Adam
Adam
Critic activation function
tanh
tanh
tanh
hyperparameters summarized Tab. 2. To increase expressivity, we interleave each affine coupling layer with a parameter-free flip transformation layer that reverses the order of the input dimensions.
Pre-training. Each NF prior is pre-trained on demonstration dataset D using maximum menximum-likelaee likelihood estimation 2). To improve stability, particularly settings with low-variance near-unimodal action distributions, we add two regularization terms: (i) an inverse-consistency penalty La encourag- ing nearby actions in real space to map to similar latent codes, and (ii) a Css encouraging local smoothness in the mapping from latent to real actions. The overall loss for training the NF prior is
penalty
(()) = E (Ta 0) + log det Jr, (a; dic Lio + dB Lis
(10)
where
| (a Caj
T
Lie En Eas.e.
(11)
E 1 +
[[To(z + 5z; T [z; 3)
Ls Ea,.ö
(12)
8
1
+
where e, and 8= are noise vectors sampled from zero-centered Gaussians with standard deviation 0.01, and E small term for numerical stability, Aic and A control the strength of the respective penalties.
Online usage. During online RL phase, the NF priors are used only for inference: latent action sampled from a latent policy is transformed into environment action T(2;; Optionally, via the feedback- shobbick-shhring mechanism (Sec. 3.3 the inverse mapping T is applied to compute latent codes for other actors' policies. Importantly, the latent policies never backpropagate through the NF prior. From their perspective, the NF is simply part of the environment and affects the transition and reward dynamics.
For additional background real NVPs and their behavior priors in RL, refer to Dinh et al. (201)); Singh et al. (202T)
17

---

Published as a conference paper at ICLR 2026
Table Normalizing Flow (NF) prior hyperparameters used across environments.
Hyperparameter
PointMaze
FrankaKitcher
CarRacing
Number of coupling layers
10
10
Hidden layer widths of y,
[256]
[256]
[256]
Activation function
ReIU
ReLU
ReLU
Base distribution covariance
0.2
0.2
0.2
Learning rate
1 x 10-4
1 x 10
1 x 10
Batch size
64
64
1024
Number of training epochs
100
100
100
Gradient clipping norm
1.0
1.0
1.0
Inverse-consistency penalty Ale
1 X 10
1 10-3
1 10
Forward-smoothness penalty Ats Optimizer
1 x 10-3
x 10-3
10-3
Adam
Adam
Adam
18

---

Published as a conference paper at ICLR 2026
D ADDITIONAL FRANKAKITCHEN RESULTS
Microwave
Kettle
Light Switch
Slide Cabinet
100%
100%
100%
100%
100%
100%
1100%
0%
0%
100%
10036
1156
4796
33%
100%
100%
100S
0.1M
0.5M
1.0M
0.1M
0.5M
1.0M
0.1 1
0.5M
1.0M
0.1M
0.5M
1.0M
Hinge Cabinet
Top Burner
Bottom Burner
0%
34%
17%
Ours
Parrot
100%
100%
IL
0%
0%
0%
SAC
66%
0%
93%
QFilter
100%
1.00%
100%
0.1M
0.5M
1.0M
0.1M
0.5M
1.0M
0.1M
0.5M
1.0M
Time to success when using prior data D, from different tasks (panel titles). Bars indicate the step at which cross-seed average running success rate reached 100%, the final success rate after steps if convergence was not achieved earlier (shorter hetter). Percentage annotations denote cross-sced average running success rate at that time (3 seeds). Dashed vertical lines indicate the convergence time of APC.
(a) Replica of Fig. 4b
Microwave
Kettle
Light Switch
Slide Cabinet
100%
Success rale
50%
0.1M
0.5M
1.0M 0.1M
0.5M Top Burner
1.0M 0.1M
0.5M
1.0M
0.1M
0.5M
1.0M
Hinge Cabinet
Bottom Burner
Success rate
100%
Ours
Parrot
50%
SAC
Qfilter
0%
0.1M
0.5M
1.0M 0.1M
0.5M
1.0M 0.1M
0.5M
1.0M
(b) Success rate over time corresponding to the above bar plot, using prior data from different tasks (panel titles). The shaded area corresponds standard deviation across three random seeds.
Figure 7: Results on FitnkaKitchen's microwave task, which requires opening the microwave door.
19

---

Published as a conference paper at ICLR 2026
Microwave
Kettle
Light Switch
Slide Cabinet
100%
100%
100%
100%
50%
0%
100%
90%
50%
100%
50%
73%
73%
0.iM
0.5M
1.0M
0.1M
0.5M
1.0M
0.1M
0.5M
1.0M
0.1M
0.5M
1.0M
Hinge Cabinct
Top Burner
Bottom Burner
0%
9%
100%
Ours
100%
100%
100%
Parrot
IL
0%
72%
100%
SAC
0%
50%
100%
QFilter
73%
73%
73%
0.1M
0.5M
1.0M
0.1M
0.5M
1.0M
0.1M
0.5M
1.0M
(a) Time to success when using prior data D, from different tasks (panel titles). Bars indicate the step which the cross-sced average running success reached 100%, or the final success rate after 1M steps if convergence was not achieved earlier (shorter is better). Percentage annotations denote the cross-seas seed average running success rate at that time (2 sceds). Dashed vertical lines indicate the convergence time of APC.
Microwave
Kettle
Light Switch
Slide Cabinet
Success
50%
0.1M
0.5M
1.0M O.1M
0.5M Top Burner
1.0M 0.1M
0.5M
1.0M
0.1M
0.5M
1.0M
Hinge Cabinet
Bottom Burner
Success rate
100%
Ours
Parrot
50%
SAC
Qfilter
0%
0.1M
0.5M
1.0M 0.1M
0.5M
1.0M 0.1M
0.5M
1.0M
(b) Success rate over time corresponding to the above bar plot, using prior data from different tasks (panel titles).
The shaded area corresponds standard deviation
random seeds.
Figure 8: Results on FrankaKitchen'ss bott om burner task, which requires turning the knob to turn on one of the bottom-row stove burners.
20

---

Published as a conference paper at ICLR 2026
Microwave
Kettle
Light Switch
Slide Cabinet
100%
100%
50%
100%
100%
100%
100%
0%
100%
100%
82%
1.7%
1000%
50%
60%
60%
60%
0.iM
0.5M
1.0M
0.1M
0.5M
1.0M
0.1M
0.5M
1.0M
0.1M
0.5M
1.0M
Hinge Cabinct
Top Burner
Bottom Burner
13%
71%
100%
Ours
Parrot
100%
100%
100%
IL
100%
100%
100%
SAC
51%
100%
100%
QFilter
60%
50%
60%
0.1M
0.5M
1.0M
0.1M
0.5M
1.0M
0.1M
0.5M
1.0M
(a) Time to success when using prior data D, from different tasks (panel titles). Bars indicate the step which the cross-sced average running success reached 100%, or the final success rate after 1M steps if convergence was not achieved earlier (shorter is better). Percentage annotations denote the cross-scas seed average running success rate at that time (2 sceds). Dashed vertical lines indicate the convergence time of APC.
Microwave
Kettle
Light Switch
Slide Cabinet
Success
50%
0.1M
0.5M
1.0M O.1M
0.5M Top Burner
1.0M 0.1M
0.5M
1.0M
0.1M
0.5M
1.0M
Hinge Cabinet
Bottom Burner
Success rate
100%
Ours
Parrot
50%
SAC
Qfilter
0%
0.1M
0.5M
1.0M 0.1M
0.5M
1.0M 0.1M
0.5M
1.0M
(b) Success rate over time corresponding to the above bar plot, using prior data D, from different tasks (panel titles). The shaded area corresponds to one standard deviation two random seeds.
Figure Results FrankaKitchen's light switch task, which requires flipping the light switch
21

---

Published as a conference paper at ICLR 2026
Microwave
Kettle
Light Switch
Slide Cabinet
100%
28%
2996
100%
100%
100%
100%
100%
100%
0%
0%
0%
1%
75%
75%
o.iM
0.5M
1.0M
0.iM
0.5M
1.0M
0.1M
0.5M
1.0M
0.1M
0.5M
1.0M
Hinge Cabinet
Top Burner
Bottom Burner
0%
100%
50%
Ours
100%
100%
93%
Parrot
IL
10p%
100%
100%
SAC
100%
0%
QFilter
75%
75%
75%
0.1M
0.5M
1.0M
0.1M
0.5M
1.0M
0.1M
0.5M
1.0M
(a) Time to success when using prior data D, from different tasks (panel titles). Bars indicate the step which the cross-sced average running success reached 100%, or the final success rate after 1M steps if convergence was not achieved earlier (shorter is better). Percentage annotations denote the cross-scos seed average running success rate at that time (2 sceds). Dashed vertical lines indicate the convergence time of APC.
Microwave
Kettle
Light Switch
Slide Cabinet
Success
50%
0.1M
0.5M
1.0M 0.1M
0.5M
1.0M 0.1M
0.5M
1.0M
0.1M
0.5M
1.0M
Hinge Cabinet
Top Burner
Bottom Burner
Success rate
100%
Ours
Parrot
50%
SAC
Qfilter
0%
0.1M
0.5M
1.0M 0.1M
1.0M 0.1M
0.5M
1.0M
(b) Success rate over time corresponding to the above bar plot, using prior data D, from different tasks (panel titles). The shaded area corresponds to one standard deviation two random seeds.
Figure 10: Results FrankaKitchen's top burner task, which requires turning the know turn on one of the top-row stove burners.
22

---

Published as a conference paper at ICLR 2026
Microwave
Kettle
Light Switch
Slide Cabinet
100%
100%
100%
100%
T00%
100%
100%
100%
0%
100%
5%
50%
50%
100%
100%
100%
100%
0.iM
0.5M
1.0M
0.iM
0.5M
1.0M
0.1M
0.5M
1.0M
0.1M
0.5M
1.0M
Hinge Cabinct
Top Burner
Bottom Burner
79%
100%
100%
Ours
Parrot
100%
100%
100%
IL
100%
100%
100%
SAC
100%
T00%
59%
QFilter
100%
100%
100%
0.1M
0.5M
1.0M
0.1M
0.5M
1.0M
0.1M
0.5M
1.0M
(a) Time to success when using prior data D, from different tasks (panel titles). Bars indicate the step which the cross-sced average running success reached 100%, or the final success rate after 1M steps if convergence was not achieved earlier (shorter is better). Percentage annotations denote the cross-scas seed average running success rate at that time (2 sceds). Dashed vertical lines indicate the convergence time of APC.
Microwave
Kettle
Light Switch
Slide Cabinet
Success
50%
0.1M
0.5M
1.0M 0.1M
0.5M
1.0M 0.1M
0.5M
1.0M
0.1M
0.5M
1.0M
Hinge Cabinet
Top Burner
Bottom Burner
Success rate
100%
Ours
Parrot
50%
SAC
Qfilter
0%
0.1M
0.5M
1.0M 0.1M
0.5M
1.0M 0.1M
0.5M
1.0M
(b) Success rate over time corresponding to the above bar plot, using prior data D, from different tasks (panel titles). The shaded area corresponds to one standard deviation two random seeds.
Figure 11: Results on FrankaKitchen's slide cabinet task, which requires sliding open the top-right cabinet door.
23

---

Published as a conference paper at ICLR 2026
Microwave
Kettle
Light Switch
Slide Cabinet
0%
0%
0%
0%
43%
0%
0%
0%
0%
0.1M
0.5M
1.0M
0.1M
0.5M
1.0M
0.1M
0.5M
1.0M
0.1M
0.5M
1.0M
Hinge Cabinet
Top Burner
Bottom Burner
100%
0%
Ours
Parrot
100%
0%
0%
IL.
0%
SAC
100%
0%
0%
QFilter
0.1M
0.5M
1.0M
0.1M
0.5M
1.0M
0.1M
0.5M
1.0M
(a) Time to success when using prior data D, from different tasks (panel titles). Bars indicate the step which the cross-sced average running success rate reached 100%, or the final success rate after 1M steps if convergence was not achieved carlier (shorter is better). Percentage annotations denote the cross-seed average running success rate that time (2 seeds). Dashed vertical lines indicate the convergence time of APC.
Microwave
Kettle
Light Switch
Slide Cabinet
100%
Success
50%
AN
0.1M
0.5M
1.0M O.1.M
0.5M
1.0M 0.1M
0.5M
1.0M 0.1M
0.5M
1.0M
Hinge Cabinet
Top Burner
Bottom Burner
Ours
Parrot
Success
50%
IL
SAC
Qfilter
0.1M
0.5M
1.0M 0.1M
0.5M
1.0M 0.1M
0.5M
1.0M
(b) Success rate time corresponding to the above bar plot, using prior data D, from different tasks (panel titles). The shaded area corresponds to one standard deviation across two random seeds.
Figure 12: Results on FrankaKitchen's h nge cabinet task, which requires opening the top-left "hinge" type cabinet door. Due to the difficulty of the task, all methods fails at solving the task when not using an aligned prior.
24

---

Published as a conference paper at ICLR 2026
Microwave
Kettle
Light Switch
Slide Cabinet
100%
100%
100%
100%
100%
100%
100%
100%
100%
100%
100%
89%
37%
100%
100%
100%
0.iM
0.5M
1.0M
0.1M
0.5M
1.0M
0.1M
0.5M
1.0M
0.1M
0.5M
1.0M
Hinge Cabinct
Top Burner
Bottom Burner
100%
100%
100%
Ours
Parrot
100%
100%
100%
IL
0%
0%
0%
SAC
53%
100%
16%
QFilter
100%
100%
100%
0.1M
0.5M
1.0M
0.1M
0.5M
1.0M
0.1M
0.5M
1.0M
(a) Time to success when using prior data D, from different tasks (panel titles). Bars indicate the step which the cross-sced average running success reached 100%, or the final success rate after 1M steps if convergence was not achieved earlier (shorter is better). Percentage annotations denote the cross-scos seed average running success rate at that time (2 sceds). Dashed vertical lines indicate the convergence time of APC.
Microwave
Kettle
Light Switch
Slide Cabinet
Success
50%
0.1M
0.5M
1.0M 0.1M
0.5M
1.0M 0.1M
0.5M
1.0M
0.1M
0.5M
1.0M
Hinge Cabinet
Top Burner
Bottom Burner
Success rate
100%
Ours
Parrot
50%
SAC
Qfilter
0%
0.1M
0.5M
1.0M 0.1M
0.5M
1.0M 0.1M
0.5M
1.0M
(b) Success rate over time corresponding to the above bar plot, using prior data D, from different tasks (panel titles). The shaded area corresponds to one standard deviation two random seeds.
Figure 13: Results on FrankaKitchen's kettle task, which requires sliding the kettle the stove burner.
25

---

Published as a conference paper at ICLR 2026
E ADDITIONAL ABLATION RESULTS
Fpieode 100
Proce tree
based
800
Epinode 30
Prior-hesed Prioe-frew
Epesode 20
Prior Prior-hand
Prtor-tnna
Egisodic
Fpisode
Proo-Iree
200
Prioe-hused
Egisode
Prior based
200
400
600
000
1000
(a) Arbitrator selector, with reward sharing.
Episodo 100
Prior haadd
800
Fpioode 50
Proe based Proce free
Episode 20
Prior-heseed Price-free
400
Epesode 10
Prior based Prtor-rnas
200
Epiende
Proee free
Priot-hosed
200
400
600
800
1000
(b) Arbitrator selector, no reward sharing.
Prion-ussdd
Episode
Epasode 50
Priox based
Episode 20
Pnoe-based Vnoc free
400
Epiondin Wooooii..coo
Prio-hhssd Price-rree
trpssde 10
Episode
booe-aased Price-reee
Epúsode
Prioe-besee Prioe-Truw
(c) Learning hierarchical selector, with reward sharing jpg
Episode 100
Priur-rree
Prinr-froe
600
Epliodi netura
Ppisode
Prior based
300
Eposode 10
Priur-lused Priur-free
200
Egisode 5
Prior based Prior free
Episade 0
Prior-based Prior-free
200
400
800
1000
(d) Learning hierarchical selector, reward sharing.
Figure 14: Complete selector-action plot from the ablation study on the car racing environment, extension of Fig.
26

---

Published as a conference paper at ICLR 2026
Number of priors
SAC
13 12
APC, (partially) aligned
APC, fully misaligned
50k
100k
150k
200k
250k
300k
350k
Environment Interactions
Figure 15: SAC APC's time until 100% cross-seed success, when using different numbers of behavior priors, on the PointMaze environment. The z-error bars indicates the variance over five seeds and the different tasks. APC's "(partially) aligned" variant here means that APC's set of priors includes the behavior prior for the current target task (but also n misaligned ones), while the "fully misaligned" variant means that the other tasks. The beneficial exploration bias due to the aligned prior is weakened by the increasing number of misaligned priors, which explains the increase time until success when using more priors. Nevertheless, as long as APC has access to the aligned prior, it performs better or par with SAC, highlighting its ability to avoid negative exploration bias from misaligned priors.
priors consists solely misaligned priors for
F ADDITIONAL POINTMAZE RESULTS
F.1 NUMBER OF PRIORS
We also study the effect of the number of misaligned priors on APC. For this, we run APC on the PointMaze environment with access to two, three, all four behavior priors for the four goal locations. As before, we separately consider the performance when the set priors (partially) aligned, meaning it contains a behavior prior that is optimal for the current task, or fully misaligned, when the set priors only contains behavior priors for other tasks.
For this analysis, standard exponent the negative Fuclidean distance reward with the negative Fuclidean distance directly. The exponent of the negative Fuclidean uninformative, evaluating to almost everywhere, except very close to the goal, effectively creating sparse reward setting. In such settings, the effect of misaligned priors is amplified, since there (almost) feedback for learning about the negative influence of misaligned priors, allowing them to continuously bias the exploration in a negative fashion. Using the negative Euclidean distance directly yields more informative gradients throughout the maze, which allows us to better analyze how APC behaves as the number of priors increases.
slightly denser reward for the PointMaze reward by replacing the
The results are shown in Figure T5 As can be seen, with an increasing number of priors, the beneficial exploration bias from the aligned behavior prior is weakened. This happens because initially, before the Q-functions contain meaningful estimates, the misaligned behavior priors can (negatively) influence exploration by biasing the agents towards wrong goal locations. However, once the per-actor Q-functions correctly reflect lower values for the misaligned priors, they receive less weight under the arbitrator and their impact diminishes, allowing APC to exploit the aligned prior or prior-free free actor.
Furthermore, we find that APC, when given of fully misaligned priors, performs slightly worse than from-scrated SAC, due to the misaligned priors biasing exploration negatively, until their Q- values reflect low utility. However, this represents contrived adversarial setting that include in this analysis for pedagogical reasons and completeness, but note that is very unlikely in practice.
27

---

Published as a conference paper at ICLR 2026
Misaligeed
0.3
100k
250c
400k
100K
250k
100k
300k
230k
000k
1034
250k
400k
100%
250L
400k
enviiommre stops
Endirnmen Steps
Drvisriment Stops
Eniiformen stops
Enstonment stops
Figure 16: Arbitrator sensitivity analysis with respect to the Boltzmann temperature The arbitrator largely insensitive towards the temperature, however, robustness in the misaligned prior case deteriorates with very high temperature 7 100.
F.2 ARBITRATOR TEMPERATURE SENSITIVITY
We evaluate the sensitivity of the arbitrator its lemperature coefficient 1). Recall that the arbitrator selects among actors using Boltzmann categorical distribution C... with selection probabilities
pi(s) =
2 exp (s))
Σ exp (H)(
(13)
The temperature 77 regulates how strongly value differences influence the selection: small v yields more peaked distribution, while large 7) biases the distribution towards uniform. Figure 16 reports running success rates for temperatures on logarithmic grid, T E 0.11,0.1,.... 100, evaluated on the PointMaze environment and using single prior.
Overall, the arbitrator is mostly insensitive the temperature choice. The only notable degradation appears for large temperatures and when given misaligned prior, which is expected: In this setting, value differences between the misaligned prior- actor and the prior-free actor remain too small relative the temperature, leading the arbitrator to sample both nearly uniformly and preventing effective filtering of the misaligned prior.
A practical heuristic for choosing 7 could be based on the reward scale: low reward magnitudes (and therefore small value differences) could be amplified with a smaller 7. whereas high reward magnitudes might benefit form using larger n, to counteract large value differences between similar behaviors.
28


Reviews:
[{"review_id": "GflnVdehLU", "reviewer": "Reviewer_add8", "summary": "The authors propose a hierarchical RL approach to leveraging suboptimal/misaligned demonstration data; specifically, they propose Adaptive Policy Composition (APC), which trains multiple Normalizing Flow (NF) prior policies and chooses one based on alignment with the target task. Crucially, APC uses two key mechanisms to maintain performance in the face of non-optimal / misaligned demonstrations: (1) reward-sharing, which exploits NF invertibility to map each action token to all policies' latent spaces, enabling continuous updates across all policies and not just the one which took the action, (2) a parameter-free \"arbitrator\" high-level selector, which simply selects the policy based on the value estimate of the lower-level actors.\n\nThis method surpasses PARROT [1] as well as standard from-scratch RL and imitation learning baselines in the misaligned and suboptimal demonstration settings and is only slightly slower to converge compared to PARROT given a fully-aligned prior. Ablations demonstrate that both mechanisms are critical for performance: the arbitrator avoids overcommitting to the prior-based actor and maintains balanced usage of actors, and reward-sharing ensures all actors are updated continuously.\n\n[1] Singh, Avi, et al. \"Parrot: Data-driven behavioral priors for reinforcement learning.\" arXiv preprint arXiv:2011.10024 (2020).", "questions": "Could you elaborate on how to get the Normalizing Flow reward-sharing method to work for discrete action spaces?\n\nPerforming some sensitivity analysis on the temperature parameter for the \"parameter-free\" arbitrator (which is therefore a bit of a misnomer) could be useful as well.\n\nAre there any pathological situations in which, due to the distribution of demonstration data, APC would learn significantly slower than the learning-from-scratch baseline, or is there an argument for why this cannot occur?\n\nIt would be clearer to explicitly state the number of actors per evaluation environment (e.g. one of them only had a single prior-based actor), and it would be very interesting to find the effect on performance as the number of actors (and therefore demonstration datasets) scaled. Could you have number of actors != number of datasets (e.g. suppose the number of tasks is extremely high)?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "Leveraging suboptimal / misaligned demonstration data for reinforcement learning is a significant research topic, given that real-world scenarios are unlikely to have perfectly aligned demonstration data.\n\nThe proposed method is a novel contribution; in particular, the reward-sharing trick which exploits invertibility of normalizing flows which enables continuous updates for all actors.\n\nThe experimental section is executed thoroughly and with high quality; the ablations are convincing that both mechanisms are critical for final performance. Overall the paper is well-motivated and design choices at each step is clear.", "weaknesses": "APC’s framework trains multiple parallel SAC agents and evaluates each, so (as typical for methods that target the low-sample regime) while the sample-efficiency is increased, the total compute and (likely) wall-clock time may actually increase. Clarifying the scale of this extra compute (and e.g. memory for the additional replay buffers) per task suite could be helpful.\n\nBesides Franka Kitchen, evaluation settings are relatively simple (Maze Navigation and Car Racing); demonstrating this works on additional task suites would be much more convincing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eyhBRhUkv9", "reviewer": "Reviewer_TwB7", "summary": "This paper introduces Adaptive Policy Composition (APC), a framework that learns multiple data-driven Normalizing Flow (NF) priors from different demonstration datasets. Building on these NF priors, the method trains multiple latent policies and corresponding low-level actors in online environments. A reward-sharing trick leverages the invertibility of NF priors, allowing different actors to learn from the same transition. A high-level selector is then learned to determine which low-level actor to invoke at each state. The proposed approach effectively addresses the challenge of suboptimal demonstration data in real-world scenarios. Experimental results show that APC outperforms the PARROT baseline.", "questions": "1. If I understand correctly, the authors use the selector to choose a single actor at each step rather than forming a mixture of multiple actors. How would the performance be affected if the method instead used a mixture of different actors? If such a mixture could hinder actor training, could it still be beneficial after the actors have been trained?\n\n2. What is the main difference between APC and existing methods that rely on a single data source? If other Learning-from-Demonstration (LfD) methods were extended to integrate the multiple demonstration datasets used in this work, would they achieve similar performance?\n\n3. From Figures 3(b) and 4(b), the proposed APC method does not show substantial improvement over PARROT or SAC on aligned data (with slower convergence). Can I assume that the method mainly provides advantages when the demonstration data are misaligned?\n\n4. I am interested in the performance of APC on the D4RL benchmarks. How does it compare with state-of-the-art LfD methods trained on datasets with different levels of expertise? Would APC still outperform those methods if only a single expert demonstration dataset were used?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The reward-sharing trick is particularly interesting, as it allows different actors to learn from a single data source, thereby improving sample efficiency.\n\n2. The paper presents comprehensive experimental results across multiple environments, demonstrating the effectiveness of the proposed method.", "weaknesses": "1. Since the proposed APC method learns multiple actors from the environment, it may incur additional computational costs compared to traditional methods (e.g., PARROT). Moreover, the approach requires multiple demonstration datasets, which may not be applicable in real-world scenarios.\n\n2. The ablation study in the paper is also not sufficiently strong. For instance, it lacks an analysis of performance under limited demonstration sources (e.g., using only a single dataset) and does not include comparisons with state-of-the-art Learning-from-Demonstration (LfD) methods using optimal or suboptimal data.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "S9VpHzN0WG", "reviewer": "Reviewer_fyEH", "summary": "The paper proposes APC-RL, which combines multiple prior-based actors (using normalizing flows) with a prior-free actor under a value-based selector. It introduces a “reward-sharing” trick that reuses experience across actors via NF invertibility. Experiments on PointMaze, FrankaKitchen, and CarRacing claim faster learning and robustness to misaligned demonstrations.", "questions": "1. How is demonstrator misalignment quantified?\n\n2. Why not directly share rewards without invertibility if the training is off-policy?\n\n3. How efficient is the selector? Does it scale with many actors?\n\n4. Would this method still outperform if retraining from scratch without demos?\n\n5. How would APC-RL compare to HER with Q-filter on similar tasks?", "limitations": "", "rating": 2, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Tackles a relevant problem: overcoming poor demonstrations in RL.\n\n2. Simple and modular architecture that integrates easily with SAC.\n\n3. Reward-sharing via NF inversion is conceptually neat and potentially sample-efficient.", "weaknesses": "1. Efficiency and justification gaps: The approach may be inefficient in multiple ways: maintaining several actors, evaluating each at every step, and managing multiple replay buffers. The paper neither analyzes this computational overhead nor justifies why keeping multiple actors is preferable to simply retraining a single policy once demonstrations start degrading performance.\n\n2. Limited generality: The method is evaluated only with SAC, with no discussion of applicability to other off-policy algorithms.\n\n3. Invertibility not clearly needed: Since SAC is off-policy, rewards could be shared directly without NF inversion. The necessity of invertibility is not well justified.\n\n4. No clear intuition: The paper details how training occurs but not why the composition improves learning or stability.\n\n5. Weak experimental design: Few baselines, undefined notion of “misalignment,” and only 2–3 seeds. The paper should also compare against established demonstration-filtering methods such as HER with demonstrations (Q-filter) to contextualize the claimed benefits.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cIJxwcLSUT", "reviewer": "Reviewer_LZVv", "summary": "This work builds upon substantial advancements in behavioral priors for online reinforcement learning, and proposes an end-to-end technique to balance and control the guidance received from multiple, potentially misaligned priors. Given N data sources, the method trains N+1 policies: one acting over real actions, and N acting over the latent space of frozen normalizing flows estimating the state-conditional action distribution in each data source (as pioneered by PARROT). These policies, as well as their values, are simply trained through SAC; the softmax of the value can be used as a policy-selection criterion for online data collection. Finally, the connected data by each policy can be relabeled according to the transformations induced by each prior, and used for training every other policy. The authors evaluate the proposed methods on a simple 2D maze, as well as on simulated kitchen and driving environments. Across the board, results confirm that the proposed method is less sample efficient than PARROT when the prior is well aligned (likely due to the overhead of value learning for high-level action estimation), but prevent catastrophic failure when the prior is misaligned, while consistently outperforming a solution learning from scratch (as long as the prior is not adversarial). The submission finally provides ablations and a visualization of the policy switching pattern in the racing environment.", "questions": "## Minor concerns and questions\n- line 202: a definition and discussion of \"primacy bias\" might be informative.\n- equation 4: I would recommend simply expressing the policy through $\\propto$, and avoiding an explicit normalization constant.\n- equation 4: did the authors also evaluate a greedy high-level policy? i.e. simply an argmax of the values\n- line 218: did the authors actually measure high variance? Would more actions samples result in better empirical performance?\n- line 244: why is a non-standard shape chosen for the D4RL maze.\n- line 310: Fig.3 right instead of left\n- Figure 5: how is the network-based high-level actor trained?\n- line 716: this would be appreciated.\n\n## Conclusion\nThis work is clearly motivated, the algorithm is principled and sufficiently supported by empirical evidence, particularly using both simulated and human-collected data. Despite computational and modeling limitations (as highlighted above), and assuming that questions are sufficiently addressed, I would currently recommend acceptance.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The method is clear, principled and well motivated.\n- The paper is well-written and easy to follow; claims are well supported and not overstated.\n- The experimental section is well-designed and sufficient to analyze the most important facets of the algorithm. Notably, it includes both simulated and human-collected data.", "weaknesses": "- The computational overhead induced by the method is perhaps the biggest limitation: a linear number of agents need to be trained as the number of data sources increases. Have the authors considered conditional actor and critic architecture (i.e. a single critic network, conditioned on the policy's index)?\n- A further constraint is the reliance of this method on invertible generative models in order to effectively relabel actions. Normalizing flows are in general capable models, but other methods (such as flow matching) seem to be prevalent empirically. Can the method be generalized to a broader class of architectures for modeling priors?\n- The work fails to acknowledge that the proposed method may only alleviate partially misaligned priors. Consider a setting in which the agent is initialized in the top left corner of the maze, and the prior is trained on data reaching the top right corner, while the downstream task involves reaching the bottom left corner. Exploration, in this case, will be biased towards an useless direction, and it is hard to imagine that the method would outperform learning from scratch. Moreover, the normalizing flow trained on the prior would often be queried OOD. This is of course understandable, but I believe it should be acknowledged explicitly in a limitation section.\n- There are several options for dynamically incorporating behavioral priors into downstream RL, including [1] and [2] (although I believe [2] does not directly apply as uncertainty estimates are not available nor informative). Although they have not been applied in the exact same setting, a comparison would be informative.\n\nReferences:\n[1] Bagatella et al., SFP: State-free Priors for Exploration in Off-Policy Reinforcement Learning, TMLR 2022\n[2] Cramer et al., CHEQ-ing the Box: Safe Variable Impedance Learning for Robotic Polishing, arXiv 2025", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)