Leveraging Fully-Observable Solutions for Improved Partially-Observable Offline Reinforcement Learning
Authors: Chulabhaya Wijesundara
Andrea Baisero
Gregory David Castanon
Alan S Carlin
Robert Platt
Christopher Amato
Venue: Accepted by TMLR ()

Abstract: Offline reinforcement learning (RL) is a popular learning framework for control problems where online interactions with the environment are expensive, risky, or otherwise impractical.
Existing offline RL methods commonly assume full observability of the state, and therefore there is a lack of offline RL methods that are specialized for the more general case of partially-observable control.
To address this gap, we propose Cross-Observability Conservative Q-Learning (CO-CQL), an offline RL algorithm for partially-observable control that leverages fully-observable expert policies in an asymmetric learning setting.
To motivate the use of fully-observable experts for partially-observable control, we formalize Cross-Observability Optimality Ratio (COOR), a theoretical measure of cross-observability that quantifies the benefit of learning asymmetrically from a fully-observable expert, and Cross-Observability Approximation Ratio (COAR), an estimation of COOR computable from trained policies.
Our empirical evaluation on a wide variety of partially-observable challenges demonstrates that CO-CQL is able to exploit the guidance of fully-observable experts to outperform other state-of-the-art offline algorithms.

Published in Transactions on Machine Learning Research (09/2025)
Leveraging Fully-Observable Solutions for Improved Partialy--Observable Offline Reinforcement Learning
Chulabhaya Wijesundara
wijerundonc.chhorhhaastee..d
Khoury College of Computer Sciences
Northeastern University
STR
Andrea Baiscro
Khoury College of Computer Sciences Northeastern University
Gregory Castañón
gregory.castononnstt.. us
STR
Alan Carlin
alan.com carlin@strr us
STR
Robert Platt
p.platt@nortthester..co edu
Khoury College of Computer Sciences Northeastern University
Christopher Amato
c.omato mmmtto@@nrthhestten...e
Khoury College of Computer Sciences Northeastern University
Reviewed on OpenReview: https: // openreview. net/ forum? id= e9p4
Abstract
Offline reinforcement learning (RL) is popular learning framework for control problems where online interactions with the enviromment are expensive, risky, otherwise imprac- tical. Existing offline RL methods commonly assume full observability of the state, and therefore there is a lack of offline RL methods that are specialized for the more general case of partially-observabbe control. To address this gap, we propose Cross-Observallee Conservative Q-Learning (CO-CQL), an offline RL algorithm for trol that leverages fully-observable expert policies in an asymmetric learning setting. To motivate the use of fully-observable experts for partially-observabee control, formalize Optimality Ratio (COOR), a theoretical measure of cross-observability that quantifies the benefit of learning asymmetrically from fully-observable expert, and Cross-Obses Obscrvabbbyyyy Approximation Ratio (COAR), an estimation of COOR computable from trained policies. Our empirical evaluation on a wide variety of partially-observable challenges demonstrates that CO-CQL able exploit the guidance of fully-observable experts to outperform other state-of-the-art offline algorithms.
con-
1 Introduction
Standard online reinforcement learning (RL) involves learning from interactions with the environment (Stt- ton & Barto 2018). However, many real-world control problems are inconducive online interactions due high costs and risks (Levine et al. (2020), e.g., autonomous driving and healthcare. To address this issuc, offinee RT (a.k.a batch RL (Fujimoto et. al. 2019h) has emerged as data-driven solution where agents learn

---

Published in Transactions on Machine Learning Research (09/2025)
from pre-collected interactions (eevine et al.) 2020). Many such approaches (Kumar et al. 2020 Kostrikov
et al. 2021: Fujimoto & Gu 2021 ) are developed assuming fully-observable (FO) control, whereas real-world
control problems are often partially-observable (PO), featuring noisy observations and hidden information.
In this work, we address offline partially observable RL through the lens of asymmetric RL, where the
learning agent can take advantage of privileged state information (Pinto et al., 2018
Baisero & Amato
2022 Baisero al., 2022) and/or a fully-observable expert policy (Weihs et al. 2021 Warrington et al,
2021 Xguyen et al. 2022 Shenfeld et al. 2023) during training. Though the use of state information during
deployment is in clear conflict with the premise of partial observability, the availability of such information during training is possible and beneficial in a number of realistic scenarios, c.g., when training in simulation (via access to the simulation state), and when training in a highly controlled environment for deployment in a less controlled one (via the placement of additional sensors in the training environment).
Our asymmetric partially-observabll offline RL sctting assumes access to a datasct containing both states and observations, and to fully-observable expert policy (which can itself be trained using the same dataset). Given these prerequisites, method learns a partiaaly--obsevablee policy entirely offline without requiring online interactions. This reflects settings where access to a simulated or highly controlled training environ- ment is available but expensive or limited, and where deployment must rely solely on partial observations, such as autonomous driving (Chen et al. 2019 Codevilla et al. 2018; Ge et al., 2020), robotic locomo- tion (Kumar et al. 2021 Margolis et al. 2022), 1 and robotic grasping (Pinto et al.] 2018; Chen et al., 2023
This setting captures a broad range of practical cases: (a) when both a simulator and fully-observable expert are available, (b) when a dataset is pre-collected and only expert supervision is needed, and (c) when the fully-observable expert is handcrafted or scripted and the simulator is only used to log data. Our method applies in all such scenarios, and furthermore, although rare, our method can also clearly be applied in the setting when both a mixed observabibbyyy dataset and fully-observable expert are available without requiring a simulator.
Our contributions in this work are summarized as follows: (a) We motivate asymmetric learning of partial- ly-observable tasks from fully-observable experts, and propose Cross- Observability Oplimality Ratio (COOR), as a theoretical measure to quantify the utility of fully-observable experts for partially- tasks; (b) We propose Cross- Observability Approrimate Ratio (COAR) practical estimation of COOR; (c) We develop coos-Obbereubbllty Conservalive Q-Leurning (CO-CQL), new oflline RL algorithm that exploits asymmetric learning for partially-obseraable control by combining conservative Q-function regularization and behavior cloning objectives; (d) We perform an empirical evaluation of CO-CQL and other competitive baselines on a wide variety of identified partially-observable control task archetypes; and (e) We perform an analysis of robustness to dataset quality for CO-CQL. Our results remonstrate that in environments with high COOR, even simple behavior cloning from fully-observable experts is able to achieve good performances. Our hybrid CO-CQL method can leverage this guidance in high-COOR environments while improving upon it by optimizing the standard RL objective. In environments with low COOR, CO-CQL is still able to exploit the fully-observable guidance when useful, and ignore it when not.
2
Related Work
We briefly summarize other related work in offline RL and partially-observable control.
Offiine Fully-Osservable RL.
Behavior Cloning (BC) (Bain & Sammut 1999] employs supervised learn-
ing to learn directly from dataset of expert demonstrations. Twin-Delaye DDPG with Behavior Cloning (TD3+BC) Fujimoto & Gu 2021) uses BC as an auxiliary objective on a policy primarily trained via TD3 (Fjimoto et al. 2018). Conservative Q-Learning (CQL) (Kunar et al. 2020 applies a conservative regularizer on value models to reduce overestimation biases. While CQL and TD3+ BC aim to minimize the actions, Implicit Q-Learning (IQL.) (Kostrikov et al., [021] avoids
sampling of the issue by learning a value function from the dataset's stato-action pairs. All of these methods are primarily designed and evaluated on fully-observable control, whereas we address ceriially-obserablle control.
2

---

Published in Transactions on Machine Learning Research (09/2025)
Online Partially-Obserrabble RL. Although many works have focused online oully-observable RL, far fewer address partially-observable control (Weihs et al. 2021: Warrington et al., 2021 Cross- Observbbbvvyyylittyy Soft Imitation Learning (COSIL) (Nguyen et al.) 2022) is the most similar to our work in using a fully- observable expert as part of an auxiliary behavior cloning objective. The key difference is that COSIL is an online algorithm, whereas ours is fully offine, with no environment leedback during learning. Furthermore, our algorithm requires key modifications to better fit the offline setting, c.g., additional regularization of the Q-function reduce distribution shift and Q-value overestimation.
Offline Partially-Observabie RL. Offline partially-observable RL is challenging and under-cxplored
area, as it inherits difficulties from both offline RL and partially-observable RL. Guo et al. 2022) recently proposed a theoretical approach to solving linear POMDPs via causal inference in provably efficient manner. Concurrently, Lu et al. (2022) introduced Proxy variable Pessimistic Policy Optimization (P30) which also uses causal inference for ostimating pessimistic value functions. Howover, both works lack empirical evaluation, and primarily remonstrate theoretical advances to the field. In contrast, our proposed CO-CQL method is a practical algorithm that we are able to evaluate empirically. Gu al. (2023) and Hong et al.) (2023) explore representation learning in the offline setting. The former learns discrete proxy representations of states, which are then mapped to observation histories through secondary encoder; a policy is learned to predict actions based on these proxy representations. The latter shows that combining bisimulation metric with an offline algorithm can allow for learning of more compact observation history representations, thereby improving overall sample efficiency. Instead of high-level observation history abstraction, we focus on whether low-level MDP expert supervision can improve partially-observable offline learning. We hypothesize that the reprosentation learning techniques of Gu et al. (2023) and Hong et al. (2023) could be integrated into our approach in future work to improve sample efficiency and performance.
3 Background
In this section, we briefly summarize the background in partially-obserrabee control relevant to our work.
Partially- Partially-Observall Observable Markov Decision Processes. A partially- Markov decision process (POMDP) (Astrōm 1965) is discrete-time control problem defined by tuple (S, Po, po, T, state space S, action space A, observation space O, starting distribution po(s), stochastic transition function T(s, a, s'), stochastic observation function O(a, s, o), reward function R(s, a), and discount factor y.
7),), with
Contrary to fully-observable agents, a partially-obbserabbe agent observes the state indirectly via noisy and stochastic observations. As a result, partially-obbeevablee policies act based on their observable action- observation history h, (Oo,
H. The objective of partially-observable RL is train a history-based policy æ: H. + AA that maximizes the total expected episodic return JRL E E 7 R(se, at)].
Offline Reinforcement Learning. Offline RL is a form of osf-policy RL. Assuming fully-observable control, the agent is trained on static dataset of fully-observable transitions DMDP {(s. a, s') ) generated by behavior policy тg. The goal of offline RL is to train a policy æ to maximize the RL objective JRL using the offline dataset rather than online interactions. In partially-observable control, the dataset contains observation data (typically in history form) rather than state data DPOMDP = a, {()),,,;)))))>.
Learning a partially-observable policy solely on observation data extremely difficult. Our work employs an asymmetric framework to aid training by assuming access to an offline dataset that includes both observation and state data D = ((h, 8, a, r, 8', ) and a well-performing fully-observable expert μr: S AA.
Conservative Q-Learning (CQL). Kumar et al. 2020 develop state-of-the-art offiine RL algorithm for fully-observable control that employs the principle of pessimism (Buckman et al. 2020) to prevent value overestimation. In their work, Kumar et al. (2020) propose two forms for CQL. One form of CQL combines a standard DQN objective (Mnih et al. 2015)
1 Jeot = BRandnd (r+ max ((s',d)) - Q(s,a)) + +QR())
(
(1)
2
a'
3

---

Published in Transactions on Machine Learning Research (09/2025)
with conservative value regularizer, R(Q) EgwD [max, -( Bs,aDD (Q(saa) that minimizes the gap
between maximal and in-distribution values. When the action space is continuous, an auxiliary policy model p(s) s argmax,, Q(s, a) is trained to estimate maximal values.
Another form CQL combines soft actor-critic (SAC) objectives with the value regularizer,
Jčq = By-D,aw a(a Q(s, a)]
(2)
eQ Eaandon (r ( (or (Q(s', a(d - Q(s,,)) AR(Q).
(3)
Cross- Dooas-bbeevvability Soft Imitation Learning (COSIL) Nguyen et al. 2022 propose an online RL method that exploits pre-trained fully-observable agent to guide the training of partially-observable agent via an objective that incorporates a policy divergence D as a psendo-reward,
JCOSIL E
Σ rs (R(s),))) aD (H(s)))) x(h)))
(4)
4 Leveraging Fully-Observable Solutions for Partially-Observable Offline RL
In this section we first introduce theoretical framework to justify learning via cross-observability, then practical approximation to it that is computable, and finally CO-CQL, our proposed algorithm based on conservative cross-observabiliiyy imitation learning for oflline RL.
4.1 Cross Optimality Ratios
Let (1) C denote the subsets of actions that optimal for partially-observable agent and fully-obserrable agent respectively acting
history h and state It is fundamental property of partially-obserrabbe control that (h) and Aso(s) are not generally equivalent, and agents acting upon different information may have to take different actions to act optimally. However, this is not necessarily always the case, and many control problems feature a significant. overlap between optimal partially-observable and fully-observable actions. We quantify this notion as cross-observabilliy optimality ratio.
Definition 4.1 (Cross-Olee Observability Optimality Ratio (COOR)). For any given history and state, p² (h,, 8) E [0, 1] is the ratio of optimal fully-observable actions that are also optimal for partially-observabl control.
(P) (:o [A)
p" (h, 8)
(5)
Simplified HeavenHell Example. Consider the simplified variant of Heavenllell (Blai & Geffner 1998) shown in Figure [ The agent must identify and reach the good exit while avoiding the bad exit. fully- observable agent directly observes the good exit loca- tion, and moves accordingly. A partially-observable agent must first visit an oracle to identify the exits and reduce its state uncertainty, and then backtrack to reach the good exit. Although the optimal partially- observable policy and the optimal fully- oblly-bbservable pol- icy differ, there are several history-state pairs (about two thirds) with overlapping optimal behaviors.
Agent Starting
Optimal POMDP Action
Optimal Action
Figure 1: Simplified HeavenHell A partially- observable agent must visit an oracle to gather in- formation, while a fully-observable agent does not.
Approximate Ratio. Computing COOR requires expert domain knowledge of all optimal actions for both partially-observable and fully-observable agents-a clearly impractical assumption. To approximate COOR in practice, we propose measure the overlap between a well-trained partially-observable policy and expert fully-observable policy.
4

---

Published in Transactions on Machine Learning Research (09/2025)
Definition 4.2 (Cross- Ovoss-Observabiiityy Approximation Ratio (COAR)). Let and j be well-trained partially- observable and fully-observable policies respectively, and [0, be freely chosen probability threshold. Then, ((4,,) s) E [] is the ratio of q-likely fully-observable actions that are also q-likely according to the partially-observable policy,
p(h, 8) E. [T h) 2 g) (a(a 8)
(6)
Σ. I [m(a
q
In practice, each indicator is an estimate of the optimal subsets Apo (h) and Aro (s). Examining the performance of a trained partially-observable policy in conjunction with the corresponding COAR, we can estimate the task cross-observabilityy We will perform such an analysis in our evaluation.
On the role of COOR. COOR is not used directly in our algorithm, and we do not derive bounds based on it. Instead, it serves a motivational and analytical role: it formalizes the intuition that some partially-obseraable control problems share significant overlap with their fully-observable counterparts, and that leveraging fully-obserrable guidance can be beneficial. We show empirically in Sect 5.2.5) that COAR correlates strongly with the effectiveness of behavior cloning from fully--bservaale experts. This validates both COOR and COAR useful tools for interpreting the success of asymmetric offline learning.
4.2 Oioervbbssyvaabiiityy Conservative Q-Learning for Partially-Observabll Offline RL
CO-CQL makes several extensions to CQL and COSIL: (a) To adcquately handle the partially-observable data, state-based models are replaced with history-based models (e-g.: T(s) becomes v(h), Q(s, a) becomes Q(h, a)), using recurrent networks to process sequential data; (b) To exploit the fully-observable expert, we add a BC auxiliary loss to the policy objective; and (c) To handle discrete control problems, we employ the specialized discrete implementation of SAC (Christodoulou, 2019).
We adapt conservative regularization R(Q)= EA-D [max, Q(h, a)] - (Q a)] to partially- control. As in SAC, we use this in a value objective adapted offline partially-observabll control,
r + (Taan) [Q(hao, a') a llogT(d T(a') hao)],
(7)
J8o-CQL = E. EnaroND [w Q(h, a))] XR(Q).
(8)
where hao denotes the extended observation history: the current history h, followed by action a, and the resulting new observation o.
The entropy temperature objective of SAC is likewise adapted to offline RL via
Joo-CQ = a (a) |- log (a hh] H,
(9)
where H is the target entropy value. Lastly, the policy objective of SAC is adapted to the offline partially- observable control and combined with an auxiliary BC term based on imitating the fully-observable expert,
Jeo-cQL En [Ba=<()) log ((a h) - Q(h, +) BD (p(s) x(b))]
(10)
where 3 is BC scaling coefficient and D is divergence measure between fully-observable and partially- obscrvable policies whose form depends on the action space.
We interpret this behavior cloning term as imitation learning that tries to project fully-observable behavior into partially-observable behavior space. In an online setting such as that of COSIL (Nguyen et al. 2022). the bchavior cloning term is useful reduce exploration time, as the fully-observable expert has already done some amount of exploration during its own training. Because this notion of exploration does not exist in the offline setting, CO-CQL focuses on exploiting the recommendations of the expert policy to resolve blind spots that would result from partially-observable policy being trained exclusively the dataset.
Continuous Control. When the action space is continuous, we approximate the expectations over actions in Equations 7 and (10) via Monte Carlo (MC) estimation. Further, we employ an expected mean squared difference error D(u(s) || T(h)) Eu=n(s)a><(()) [|]u -a]] as the divergence measure between fully-observable and partially-observable policies (again estimated via MC).
5

---

Published in Transactions on Machine Learning Research (09/2025)
Discrete Control. When the action space is discrete, the expectations over actions in Equations and 10 are computed exactly via full enumeration, i.e,
i
9
(hao)( [Q(aoo, d') a log (aa' hao)] Σ x(a' hao) (Q(hoo, a') r(a' hao)),
(11)
(1) log 1:]] Σ v(a h)) log (t).
(12)
Further, we employ cross-entropy D(((s) || [(h)) = -a Eda 8) log h) as a divergence measure.
Caveats Context-Independeen Behavior Cloning. It is clear that different history-state contexts imply varying degrees of cross-obserrabilitty as reflected by COOR p" ah, Ideally, we would want to regulate the amount / of behavior cloning in context-dependent fashion, e.g., based on COOR values. Unfortunatcly, COOR is currently a thcorctical construct that not available in practice, and oven the corresponding COAR a) only computable post-training. Though more sophisticated methods are likely desirable, we are currently limited to applying single scalar factor chosen on the properties of the control problem whole. Finding better ways to regulate B represents important focus for future work.
5 Evaluation
In this section we provide an overview of our experimental setup. We perform two types of evaluations: an analysis of learning performance comparing CO-CQL to other baselines, and an analysis on how robust CO-CQL is to the quality of the training dataset. The Appendix contains more detailed discussions on control problems, datasets, architectures, and hyperparameters.
Baselines. We implement non-asymmetric baselines of direct behavior cloning of the fully-observable ex- pert (BC), discrete and continuous variants of recurrent CQL and IQL, and continuous variant of recurrent TD3+BC. During training CQL, IQL, and TD3+ BC use observation historics, while asymmotric CO-CQL uses both observation histories and states.
Evaluation Metrics. Each method is evaluated over 5 independent learning runs. Each is periodically evaluated by computing average performance over 10 sample episodes. The trained policies of all methods only use observation histories during evaluation. For each method, we report the mean and standard error of these average performances, over the 5 runs.
5.1 Control Problems
We identify 4 prototypical challenges concerning partial observability. Each challenge is evaluated on a corresponding dataset as described in Appendix A.6
Noisy Observations The agent observes a noisy version of the state corrupted by Ganssian noise. We employ modified versions of HalfCheetah and LunarLander [Brockman et al. 2016
Latent State Dimensions The agent consistently obscrves some but all state dimensions. We employ modified versions of HalfCheelah and LunarLander (Brockman et. al.) 2016 with observations that maintain position information but lack velocity information.
Eioornation-Gaherinn The agent must first obtain key information, and then rely on long-term mem- orization to remember it and complete the task. We employ a hard variant of HeavenHell BBli &
Geffner 1998 and MemoryFourRooo Rooms [Baisero et al. 2021
Field of View The agent observes environment via a limited field of view. This is similar to latent state dimensions, with the difference that the dimensions that remain latent change based on the agent's 's behavior. We employ DyntmicObstacles and KeyDoor (Baiscro et al. 2021
In addition the above, we perform an analysis on dataset robustness for CO-CQL based on modified versions of PaltPole (Brockman al. 2016) and simplified HeavenHell (Blai & Geffner 1998

---

Published in Transactions on Machine Learning Research (09/2025)
BC
CQL
CO-CQL
IQL
TD3 + BC
4000
Return
0.75
2900
Eeitedi
Epitodic
500
0.25
0,00
18 Training Timestaps
500
1.
Taining Timestaps
Training Timestops
Vainiin Timesbeps
(a) Noisy HalfCheetah.
(b)
Latent HalfCheetah.
(c) HHrd HeavenHell.
(d)
Syyamiiobbstackee
-100
2.5
-100
-150
2.0
-2.5
200
-250
-300
300
200 400 K. 600 B00K
1
200 4.0 K 609 200
1M
200
4.00
600
800
200 409
600 800
Iraining limesteps
Traring Timesteps
Training Timesteps
Training Timesteps
(e) Noisy LunarLander.
(f) Latent LamarLander.
(g) MemoryFourRooms.
(h) KeyDoor.
Figure 2: Performance mean and standard error (5 runs).
5.2 Results
Scctions 5.2.1 to 5.2.4 contains performance analysis of evaluated methods the partially-observaale
challenges. Section 5.2.5 contains an analysis on the behavior cloning coefficient. B of CO-CQL and its relationship to COOR. Section 5.2.6 contains an analysis of robustness to dataset quality for CO-CQL.
5.2.1 Noisy Observation Analysis
Noisy observations require the agent to learn to filter the noise over multiple observations in order to recover better estimation of the underlying state. The usefulness of behavior cloning form fully-observable expert likely depends the amount of noise injected relative to the complexity of the underlying state dynamics, with less noise and simpler dynamics making behavior cloning more directly useful, whereas more noise and more complex dynamics make behavior cloning less useful.
Noisy HalfCheetah. This lask requires precise control of lightly coupled joints, and we expect the addition of noise to have a major effect on optimal control. Consequently, we expect BC to struggle. CO-CQL performs the best, followed closely by TD3+BC and CQL. We hypothesize that (unlike IQL and naive BC), CO-CQL, TD3+BC, and CQL are helped in the partially-observal Additionally for CO-CQL, the behavior cloning regularizer provides stabilizing signal while learning on noisy observations. This indicates the usefulness of our behavior cloning regularizer in its ability to exploit even lower amounts of overlap. In Figure 2a we see that indeed BC performs the worst.
setting by the use of conservative regularization.
Noisy LunarLander. In contrast to the above, the dynamics of this task are likely simple enough where we expect. BC to perform well despite the introduction of observation noise. In Figure 2e] we see that indeed BC performs almost the best (behind CO-CQL.). CO-CQL is able to exploit its BC component this environment, leading to the best performance. Other baselines struggle to learn from the noisy experiences alone, demonstrating the benefits of exploiting existing cross-observaailityy overlaps.
5.2.2 Latent State Dimensions Analysis
In principle, the latent state ficlds of these tasks are inferrable by integrating historic observations, c.g., a joint angular velocity can be estimated well by extaaplatingg observed angular positions. Instead of observing this information directly, it the history model that now burdened with the additional task to learn perform this inference. Consoquently, we expect high COOR and good performance by BC. However, Figures 2b and 2 show that this is not the case, indicating that the history models fail appropriately estimate the missing state dimensions when guided by BC alone.
7

---

Published in Transactions on Machine Learning Research (09/2025)
Fblly-Observable Expert
COL
CO-COL
13.3%
14.3%
15.4%
13.8%
0.0%
0.0%
0.0%
0.9%
0.9%
0.9%
0.9%
0.0%
0.0%
0.0%
4.0%
4.4%
3.8%
0.0%
0.0%
0.0%
14.3%
X
0.9%
X
Reaches goal
4.9%
Doese't visit priest
14.3%
Rarely reaches goal
9%
4.0%
Priest Location
14.6%
25%
50%
130%
2.9%
Visite priset
26.4%
Goal Location
0.0%
0.0%
8.6%
7.9%
16.0%
Figure Hard HeavenHell visitation densities for the fully-observable expert (left), CQL (center), and CO-CQL (right), calculated across 100 episodes where the good exit is left. The fully-observable expert ignores the oracle altogether, introducing issues in pure BC-based learning. CQL succeeds in visiting the priest but then fails to exploit the information gained. CO-CQL succeeds at balancing both sub-goals.
Latent HalfCheetah. Although BC performs poorly, we find that CO-CQL is still able to exploit the fully- observable guidance and perform better than other basclines. This plausibly indicates that the RL objective is useful to learn useful history representations, given which fully-observable guidance can be exploited.
Latent LumarLander Again, we note that CO-CQL performs much better than BC, indicating that the fully-observable guidance is not sufficient. to learn an appropriate history model. In this task with simpler dynamics, note that CO-CQL and CQL perform similarly, with CO-CQL taking small lead.
5.2.3 Information-Gatherie Analysis
In problems that require information gathering and memorization, purcly cloning the fully-observable expert results in suboptimal partially-observabbe behaviors that are unable to gather information. In partially- observable navigation tasks, this means that the partially-observable policy trained BC alone can best guess the correct navigation goal with 50 % chance of success. Figure 3 shows HeavenHelll visitation densities for the fully-observable expert, CQL, and CO-CQL, demonstrating that a fully-observable expert cannot demonstrate information gathering experience, but can still be used by CO-CQL to learn how to exploit the information after it has been gathered.
Hard HellenHell. informmation-gatheiing plays fundamental role in this task, and we expect see a stark difference in performance, depending on whether fully-observable guidance is used and whether it is used adequately. In Figure 2c] we see that most. methods fail to perform well. BC fails because it has no guidance how to visit the oracle, whereas CQL and IQL fail to solve the task as whole. Figure provides visual and numerical brcakdown of learned policy state visitation behavior across the fully-obscrvable expert, CQL, and CO-CQL. The fully-observable expert, which also represents the policy learned through simple behavior cloning, goes directly to an exit while not performing any informationggatheiing behavior (as shown in the figure, the oracle location is visited 0% of the time). Standard CQL on the other hand, learns the opposite extreme of behavior, where it excessively visits the oracle and rarcly follows through to the good exit. In contrast to both of these approaches, CO-CQL is able to exploit. the fully-observable guidance of how to reach the good exit, while also learning to gain the key piece of information by visiting the oracle. As shown in the figure, the state distribution for the CO-CQL agent is spread more evenly, particularly with higher visitation for the starting state which is expected for an agent that needs to visit the oracle and then double back through the starting state to reach an exit.
MemoryFour Rooms. In this task, BC goes back to outperforming IQL and CQL. This likely because it is possible that the agent observes the beacon (the key piece of information) on its way to the good exit, which was not possible HeavenHeell Hence, the guidance of BC is at least sometimes more suitable even for partially-observabb control. CO-CQL is able to exploit this guidance to perform significantly better than either IQL and CQL, and slightly better than BC.
5.2.4 Field of View Analysis
Both tasks are geal-ng tasks performed from the agent's first-person point of view. In such tasks, the partial observability limits the agent's view, meaning that the agent may have to explore the layout to
8

---

Published in Transactions on Machine Learning Research (09/2025)
Episodic Return
0.0
Episodic Return
0.0
B 1le55
B 0.1
0.5
B Bee4
B 0.5
1e-3
-200
B 0.9
0.0
le-2
B 1.0
1e-1
B 10.0
250 k 500 750 k M
0 250 k 500 k 750 k
1 M Training Timesteps
Training Timesteps
(a) Hard HeavenHell.
(b) Noisy Lunarlander.
Figure 4: Performance mean and standard error (5 runs).
figure out the location of goals and subgoals; however, the information-gatl is comparatively low, the agent is typically able to gain broader view of the environment by moving turning slightly. Consequently, expert guidance to play an important role.
component of these lasks
expect these environments to have high COOR, and fully-observable
Dynamicobstaclles This lask contains stochastically moving obstacles that need to be avoided while reaching goal. Figure 2d confirms that indeed BC performs very well, better than methods like CQL that do not exploit fully-observable expert guidance. Nonetheless, CO-CQL is able to exploit this guidance and combine it with the main RL objective to achieve the best learning performance.
KeyDoor. This task contains navigation subgoals (the key and the door) that must be reached before the final exit. Once again, Figure 2h confirms that BC performs very well, demonstrating the benefit. of exploiting fully-observable expert guidance in environments with high COOR. Once again, CO-CQL is able to exploit this guidance to achieve the best learning performance.
5.2.5 Cross Ooss--Ottimality Ratio Analysis
We analyze how the behavior cloning coefficient B influences the COAR estimate, and how both relate to learning perfor- mance. Figure 4 shows the performance of CO-CQL on Hard HeavenHell and Noisy LunarLander for different values of 3, while Table T contains the corresponding COAR estimates. Tn Hard Heavenleell the distance the oracle and the good exit are similar, resulting in known COOR of about 66 %. In con- trast, the COOR of Noisy Lundrr is not known exactly, but we can make an informed guess that it is higher, as the task does not require much information-gatherie
Table 1: Average COAR (5 runs).
(a) HeavenHell
(b) Lunor
8 COAR
B
COAR
42.2 %
0.0
62.2 %
1e-5
69.2 %
0.1
69.9%
le-4
62.6%
0.5
83.4%
1e-3
59.2%
0.9
87.6%
le 2
76.7 %
1.0
90.6 %
r
le-1 1
85.8 %
10.0
75.9%
Overall, the results in Figures 4a and confirm that low COOR requires lower
Hard HeavenHell. In this task, we see that both high behavior cloning (3 1c-1) and low behavior cloning (3 = 0) result in poor performance. This is expected for tasks where guidance is useful up to some amount, but too much guidance detrimental to ieoommationggtterre
refereeifern Figure 4a with Table Ia] we note that high performing CO-CQL policies (8 E (1c-5, 1c-4, (e-3)) are associated with a COAR of about 66 %, broadly matching the true COOR. This confirms that the overlap between the learned policy and the fully-observable expert matches the overlap between the optimal partially-observable policy and the fubly-bbservable expert.
Noisy LunarLander. In Figure we notice that the higher COOR of this environment translates into higher values of 3 performing better. Even in such high overlap scenarios, there often benefit in avoiding too high values of 8, which allows CO-CQL to learn from the fully-observable expert, but also adapt to the
9

---

Published in Transactions on Machine Learning Research (09/2025)
CQL
CO-CQL
IQL
Episodic Return
Episodic Return
Episodic Return
150
150
150
100
100
100
50
50
50
200
400 600 k 800 k
1
200
400
600
800
1M
200 k 400 k 600 800
1 M
Training Timesteps
Training Timesteps
Training Timesteps
(a)
Latent CartPole, 100 % expert.
(b)
Latent CartPole, 50 % expert.
(c)
Latent CartPole, 0% expert.
Episodic Return
Episodic Return
Episodic Return
0.5
0.5
0.5
0.0
200 k
400 600 k 800 k
M
200 k 400 k 600 k 800 Training Timesteps
1
200
400 k 600 k 800 k Training Timesteps
1 M
Training Timesteps
(d) Easy HeavenHell, 100 % expert.
(e) Easy HeavenHell, 50% % expert.
Easy HeavenHell, % expert.
Figure 5: Performance mean and standard error (5 runs).
remaining partial observability. Nonetheless, CO-CQL demonstrates robustness towards 3, as even extremely high values (33 10.0) performs quite similar to the best value (3 1.0).
5.2.6 Robustness to Dataset Quality Analysis
We evaluate the robustness of CO-CQL and baselines to datasel quality on Latent CartPole and Easy HeavenHell. For each control problem, we create datasets made up from partially-observabl expert and random demonstrations in relative ratios of 100 %, 50 %, and 0 % respectively. Results are shown in Figure
Latent CartPole. In this task, we expect partial observability to have minimal impact on the resulting optimal policy, and a high cross-oplimalityy ratio. As expected, BC performs very well regardless of dataset quality. CO-CQL is able to exploit the high cross-optimality and achieve similar robustness across dataset qualities. In contrast, CQL and IQL are more susceptible to the dataset quality,
Easy HeavenHell. In this task, partial observability has much bigger impact on the resulting optimality, and the cross-optimality is much more limited. As expected, BC fails regardless of the dataset quality, whereas the other more data-driven methods perform better. TQL performs well with high quality data, but is strongly affected by the quality as well. In contrast, CQL and CO-CQL perform well across all types of datasets. Of these two, CO-CQL exhibits the better performance due to its ability to exploit the fully-observable expert guidance when useful and ignore it when not.
6 Conclusion
In this work, we demonstrated that fully-observable experts can be used aid the training of partially- observable agents in offline RL. We provide two novel measures of cross-observabiiity COOR, which is grounded by relevant theoretical insights, and COAR, which is practical approximation useful to analyze the cross-observability properties associated with trained policies. Our algorithmic contribution, CO-CQL, combines a standard RL objective with an auxiliary behavior cloning objective, so as to effectively exploit both sources. Our empirical evaluation demonstrates that CO-CQL succeeds in exploiting fully-observabbe guidance diverse variety of partially-observable tasks and, critically, CO-CQL able learn well even when the task cbssrvbbbeyyyyiiity is relatively low. Most importantly, CO-CQL able learn solve sub-tasks that cannot be learned by the fully-observable expert alone.
10

---

Published in Transactions on Machine Learning Research (09/2025)
References
Michael Bain and Claude Sammut. A Framework for Behavioural Cloning. In Machine Intelligence 15, Intelligent Agents [St. Catherine's College, Oxford, July 1995), pp. 103 129, GBR, January 1999. Oxford University. ISBN
Andrea Baisero and Christopher Amato. Unbiased asymmetric reinforcement learning under partial ob- servability, In Proceedings of the 21st International Conference on Autonomous Agents and Multiagent Systems, AAMAS '22, pp. 44-52, Richland, SC, 2022. International Foundation for Autonomous Agents and Multiagent Systems. ISBN 9781450392136.
Andrea Baisero, Sammie Katt, and Christopher Amato. gym-gridverse: Gridworld domains for fully and partially observable reinforcement
2021.
Andrea Baisero, Brett Daley, and Christopher Amato. Asymmetric DQN for partially observable reinforce- ment learning. In James Cussens and Kun Zhang (eds.), Proceedings of the Thirly-Eighll Conference on Uncertainty in Artificial Intelligence, volume 180 of Proceedings of Machine Learning Research, pp. 107-117. PMLR, 01-05 Aug 2022. URL batapss//proceeddngs.. mlr. http:ig..........sssssseseeeeeee (tress//1800/aassro2222 html
Andrew G. Barto, Richard S. Sutton, and Charles W. Anderson. Neuronlike adaptive elements that can solve difficult learning control problems. IEEE Transactions on Systems, Man, and Cybernetics, SMC-13 15):834-846 September 1983. ISSN 2168-2909. doi: 10010//TMMC.188..633333 IEEE Transactions on Systems, Man, and Cybernetics.
Conference Name:
Bonet Blai and Hector Geffner. Solving large POMDPs using real time dynamic programming. In Working Notes Fall AAAI Symposium on POMDPs, volume 218, 1998. URL https://bonetblaii, batinnttlaiigggtuu... io/
reports/fall98-ae pdf]
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. OpenAI Gym, June 2016. URL http:///aaaa...orr//////6666.../ arXiv:1606.0155: 01540 [cs].
Jacob Buckman, Carles Gelada, and Mare G. Bellemare. The Importance of Pessimism in Tixed-Datasee Policy Optimization, November 2020. URL
06799 arXiv:2009.06799 [es].
Dian Chen, Brady Zhou, Vladlen Koltun, and Philipp Krähenbühl. Learning by cheating. In Conference on Robot Learning (CoRL), 2019.
Tao Chen, Megha Tippur, Siyang Wu, Vikash Kumar, Edward Adelson, and Pulkit Agrawal. Visual dexterity: In-hand reorientation of novel and complex object shapes. Science Robolics, 8(84), November 2023. ISSN 2470-9476. doi: 10.1126 10.1112/sciroboticss.o sitrobotics..dee244.. URL http://dx doi. org/ htw.../1/////ccooiicc.oooo 2dt. adc9244
Petros Christodoulou. Soft Actor-Critic for Discrete Action Settings, October 2019. URL http://arxiv.
org/abs/1910.0720 arXiv:1910.07209 [cs, stat].
Felipe Codevilla, Matthias Millerr Antonio López, Vladlen Koltun, and Alexcy Dosovitskiy. End-to-
end driving via conditional imitation learning. In 2018 IEEE International Conference on Robolics and Automation (ICRA), pp. 1-9 9. IEEE Press, 2018. doi: M0.1199/CCRA2201..44600 URL https wtt.oo///1...11//1.... 84664877
Scott Fujimoto and Shixiang Gu. A Minimalist Approach to Offline Reinforcement Learning. November 2021. URL
Scott Fujimoto, Herke Hoof, and David Meger. Addressing Function Approximation Error in Actor-Critic Methods. In Proceedings of the 35th International Conference on Machine Learning, pp. 1587-1596. PMLR, July 2018. URL ptttps://proceedingas mlr fresss//88/fffuinooooolll. http::///errreeerraaaa.ee//sssep html ISSN: 2640-3498.
Scott Fujimoto, David Meger, and Doina Precup. Off-Policy Deep Reinforcement Learning without Explo- ration, August 2019. URL htttp://arxdv.orr/bbb///ae
arXiv:1812.002000 [cs, stat].
11

---

Published in Transactions on Machine Learning Research (09/2025)
Bolun Ge, Bin Yang, and Quanzeng Wang. Urban driving based condition imitation learning and multi- period information fusion. In Proceedings of the 2020 6th International Conference on Compuling and Artificial Intelligence, ICCAL '20, pp. 491 491, New York, NY, USA, 2020. Association for Computing Machincry. ISBN 9781450877089. doi: 14.144//9444555344446333 URL https: http::://oo.org/11... 1145/ 3404555. 34046399
Pengjie Gu, Xinyu Cai, Dong Xing, Xinrun Wang, Mengchen Zhao, and Bo An. Offline RL with discrete proxy representations for generalizability in POMDPs. In Thirty-seventh Conference Neural Information Processing Systems, 2023. URL https://openrevieen wwwpeeewwwweeeeoruunnnie..m net.foormnieecJo66222
Hongyi Guo, Qi Cai, Yufeng Zhang, Zhuoran Yang, and Zhaoran Wang. Provably Efficient Offline Reinforcement Learning for Partially Observable Markov Decision Processes. In Proceedings of the 39th International Conference on Machine Learning, pp. 8016 8038. PMLR, June 2022. URL https: 7/proceedings www.mmrl.pp..... www.m....r/.i12.... wttpa///vvvegggooaaaeee html
ISSN: 2640-3498.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor. In Proceedings of the 35th International Conference on Machine Learning, pp. 1861-1870. PMTR, July 2018. URL httpps:/procceedings mlr.
preess/800/haarooja11883 html ISSN: 2640-3498.
Joey Hong, Anca Dragan, and Sergey Levine. Offline rl with observation histories: Analyzing and improving sample complexity, 2023.
Leslie Pack Kaelbling, Michael Littman, and Authony R Cassandra. Planning and acting in partially observable stochastic domains. Artificial intelligence, 101(1-2):99 134, 1998.
Πlya Kostrikov, Ashvin Nair, and Sergey Levine. Offine Reinforcement Learning with Implicit Q-Learning. October 2021. URL https://openreviev. http://www....rrrrr/////////////
Ashish Kumar, Zipeng Fu, Deepak Pathak, and Jitendra Malik. Rma: Rapid motor adaptation for legged robots, 2021.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative Q-Learning for Offline Reinforcement Learning. In Advances in Newral Information Processing Systems, volume 33, pp. Curran Associates, Inc., 2020. URL. Abstract. html
(B222061122a1113221166666668
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems, November 2020. htttt:://aaxiv.orr/bbbbb.. 016433 MrXiv:2005.01644 [cs, stat].
Miao Lu, Yifei Min, Zhaoran Wang, and Zhuoran Yang. Pessimism in the Face of Confounders: Provably Efficient Offline Reinforcement Learning in Partially Observable Markov Decision Processes. September 2022. URL https::/ wttp:/arrrvieeeneettrrrrrraaaeee wtt.::/ooorrmm/dd-PP0000 UbV
Gabriel Margolis, Ge Yang, Kartik Paigwar, Tao Chen, and Pulkit Agrawal. Rapid locomotion via reinforce- ment learning. In Robotics: Science and Systems, 2022.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Mare G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattic, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 518(7540):529-55) February 2015. ISSN 1476-4687. doi: nature14236 Number: 7540 Publisher: Nature Publishing Group.
(/s88/nature14444 URL htttt::////e httpp::://wwwwauuree....
nature www.:natw........aaaaa
Hai Huu Nguyen, Andrea Baiscro, Dian Wang, Christopher Amato, and Robert Platt. Leveraging Fully Observable Policies for Learning under Partial Observability. August 2022. URL https::/openrevvier. het/forumniidppp-OPPB..rr
12

---

Published in Transactions on Machine Learning Research (09/2025)
Lerrel Pinto, Marcin Andrychowicz, Peter Welinder, Wojeiech Zaremba, and Pieter Abbecl. Asymmet- ric actor critic for image-based robot learning. In Hadas Kress-Gazit, Siddhartha S. Srinivasa, Tom Howard, and Nikolay Atanasov (eds.), Robotics: Science and Systems XIV, Carnegie Mellon Univer- sity, Pittsburrh,, Pennsylvania, USA, June 26-30, 2018, 2018. doi: N015607/RSS 2018.000...000 0.08. URL
http://g...rrrr////////1// html
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy oplimiza- tion algorithms. CoRR, 0os//177.06330 2017. URL http://arxiv..rrrtt wttp//////////////////////////o/ .63477
Idan Shenfeld, Zhang-Wet Wei Hong, Aviv Tamar, and Pulkit. Agrawal. Tgrl: An algorithm for teacher guided re- inforcement learning, In Proceedings of the 40th International Conference on Machine Learning, ICML'23. JMLR.org, 2023.
Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. The MIT Press, second edition, 2018. URL http://incomlleeeideas.. hetp//boo////h/bebboooe 2nd. .ttll
Andrew Warrington, Jonathan W. Lavington, Adam Scibior, Mark Schmidt, and Frank Wood. Robust Asymmetric Learning in POMDPs. In Proceedings of the 38th International Conference on Machine Learn- ing, pp. 11013-1 11023. PMLR, July 2021. URL http/:/t:://rr mlr mwr.mmrrrreeeee html ISSN: 2640-3498.
Luca Weihs, Unnat Jain, Tou-Jen Liu, Jordi Salvador, Svetlana Lazebnik, Aniruddha Kembhavi, and Alex Schwing. Bridging the Imitation Gap by Adaptive Insubordination. November 2021. URL https://
openreview wwt.:iiwwwwwwffoeooo www.neefooooddwwleeoeeeomm Pi-
K. J Aström. Optimal control of Markov processes with incomplete state information. Jour- nal of Mathematical Analysis and Applications, 10(1):174-22) 205, February 1965. ISSN 0022-247X. doi: 10.1016 02.1016/00222220 (65)90154-X. URL http::/www...ciaaadddee.......tt cot/////eeeiiiaeetrriii////pppp
0022247X6590154X
A Appendix
A.1
Low COOR Case Study
We consider simplified version of the famous partially-observable Tiger environment (Kaelbling et al. 1998), where there is a 2-room grid with actions Listen, GoLeft, GoRight. The treasure is randomly placed left right each episode. The fully-observable optimal policy always goes straight to the goal; the partially- observable optimal policy must first take Listen (reveals the treasure side) and then go to the revealed side. Before information is acquired, the optimal fully-observable action (go directly) is not optimal for the partially-observable policy; after listening, they align. Thus, across most (history, state) pairs before listening, the overlap of optimal actions near zero, iee,, low COOR by construction, while it becomes high only after information acquisition. This illustrates why a strong BC term may be harmful in low-overlap phases and motivates smaller B until overlap increases.
This simplified environment can be further extended to zero COOR version, where the Listen action immediately ends the episode. In this variant, the optimal fully-observable policy always goes straight to the goal, and the optimal partially-observable policy always takes the Listen action to avoid recciving a penalty from blindly going to the wrong side. Thus, the overlap of optimal actions zero. In this case, the 3 BC coefficient should be set to zero to gracefully degrade CO-CQL to standard CQL because there is no MDP overlap exploit.
A.2 Compensating For The Loss Function
Because BC divergence is helpful primarily when fully-observable and partially-observalle optima overlap, we hypothesize two approaches for compensating for low-overlap contexts. One advantage gating, where the BC term is only included when the difference in Q-values for dataset action MDP expert action
13

---

Published in Transactions on Machine Learning Research (09/2025)
is larger than specified threshold. Second, the BC term could be down-weiigtted based the uncertainty of the actor policy, measured through entropy. In either case, CO-CQL could be set to fall back to use the dataset's s actions rather than the MDP expert's actions. In this work we intentionally keep & global for simplicity and validating the asymmetric learning approach and leave these extensions for future work.
A.3 Environments
(a) CarlPole.
(b) Lander
Legend
aus
Hinge joint
Slide joint (giobal)
Hinge (Bood)
Hinge joint (giobal
(c) HaljCheetah.
(d) HeovenHell
Figure 6: Environments.
This section contains detailed descriptions of the environments used in our evaluation.
A.3.1 CartPole
CartPole [Barto et al. 1983) is classic fully-observable control problem where the agent must learn to balance a pole on moving cart. The agent receives a positive reward of 1 for each timestep that the pole does not fall. Its state space is continuous with dim(S) = 4 dimensions representing the position and velocity of the cart, and the angular position and velocity of the pole. Its action space discrete with 2 actions, indicating whether the agent moves left or right. Figure 6a shows a rendering of CartPole.
Latent CartPole. This variant generates observations by removing velocity information from the state, reducing the observation dimensions dim(O) from 4 to 2.
A.3.2 LunarLander
LunarLander [Brockman et al.) 2016) is a classic fully-observable control problem where the agent must learn to pilot and land a spacecraft without crashing. The agent receives a mixture of sparse and dense rewards depending whether the agent landed safely or crashed, and how close gets to the landing spot. Its state space is continuous with dim(S) = dimensions denoting the 2D position, 2D velocity, 1D angular position, 1D angular velocity, and 2 binary dimensions representing whether either leg of the spacecraft is touching the ground. Tts action space discrete with 4 actions denoting any combination of left and right engine being on off. Figure 6bl shows rendering of LunarLander.
Noisy LunarLander. This variant generates observations by adding Gaussian noise N(u 0, o² = 0.04) to the state.
Latent This variant generates observations by removing velocity information from the state, reducing the observation dimensions dim(O) from to
14

---

Published in Transactions on Machine Learning Research (09/2025)
A.3.3 HalfCheetah
HalfCheetah [Brockman et al. 2016 is a classic fully-observable control problem where the agent must learn to apply joint torques to make a cross-sectional robot run. The agent receives positive dense rewards for moving forward, and negative dense rewards for using large actions torques (to promote efficient movement). Its state space is continuous with dim(S) 17 dimensions denoting the 6D angular positions of the controlled joints, the 6D angular velocities of the controlled joints, the 1D angular position of the uncontrolled head joint, the 1D angular velocity of the uncontrolled head joint, the 1D position of the head tip (only horizontal); and the 2D velocitics of the head tip (horizontal and vertical). Its action space is contimuous with dim(A) = representing the 6D lorques applied to the controlled joints. Figure 6c shows a rendering of HaljCheelah.
Noisy HalfCheetah: This variant generates observations by adding Gaussian noise N = 0, = 0.04) to the state.
Latent HalfCheetah:
This variant generales observations by removing velocity information from the state, reducing the observation dimensions dim(O) from 17 8.
A.3.4 HeavenHell
HeavenHell (Blai & Geffner 1998 is a classic partially-observable control problem where the agent must identify which of two exits is good and reach it. The agent receives positive sparse reward of +1 only upon reaching the good exit, and a negative sparse reward of -1 only upon reaching the bad exit, regardless of whether has acquired the appropriate information. Its state space is discrete, with a number states that depends on the environment size, denoting the agent's position in the grid, and the location of the good exit. Its observation space is discrete, with number of observations that depends on the environment, size, denoting the agent's position, except when the agent is at the oracle, in which case the observation denotes the location of the good exit. Its action space is discrete with (A) 4 representing the four directions where the agent can move (north, south, east, west).
Easy HeavenHell:
This variant has 10 locations for the agent, and 2 possible locations for the good exit, adding up to dSm(S) 10 2 = 20 states and dO) = 10 - 1 + 2 = 11 observations.
Hard HeavenHell:
This variant has 14 locations for the agent, and 2 possible locations for the good exit,
adding up to dim(S) = 14 2 = 28 states and (O) = 14 - 1+ 2 = 15 observations.
A.3.5 GridVerse
Grid [Baisero et al. 2021 is framework for customizable 2D grid world control problems that allows for both full state observability and partial observability. In our evaluation, use three predefined tasks: DynamicObstacles, KeyDoor, and MemoryFFurRoomss For each of these tasks, we employ 7 x 7 states and 2 x 3 first-person POV observations.
The states of Grid environments are represented by the following components:
Grid.
The grid field is x 7 x 3 tensor of categorical data representing three channels of information for each of the 7 x 7 grid tiles. The first channel contains categorical index representing the tile type, e.g., wall, empty, exit, door, key, etc. The second channel contains (when relevant) information on the stale of the tile, e.g., whether a door tile is open, closed, or locked. The third channel contains (when relevant) color information on the object, which may be relevant for the dynamics of the environment, e.g., yellow door may only be opened by yellow key but not red key.
Agent ID Grid. The agent ID grid field is a 7 binary matrix representing the agent's location.
Agent
The agent field is a 6D array containing the pose of the agent, with dimensions representing the agent coordinates, and 4 dimensions representing one-hot encoding of agent orientation.
15

---

Published in Transactions on Machine Learning Research (09/2025)
(a) DynamicObstacles state.
(b) KeyDoor state.
(c) Memory Rooms state.
(d) DynamicObstacles obs.
(e) KeyDoor obs.
(f) MemoryFourHoomm obs.
Figure 7: Grid Verse environments.
Item The item field a 3D array representing the item (if any) that being carried by the agent. The item's encoding the same as the grid field, i.e., object type, object status, and object color. In the three environments we employ, this component not strictly necessary (not even for KeyDoor). ignore this field.
The observations of Grid Verse environments are represented by similar components:
Grid.
The grid field is a 2 x 3 x 3 tensor of categorical data representing three channels of information for each of the x 3 tiles observable by the agent. The channels encode the same information as in the corresponding state field.
Agent ID Grid. The agent ID grid field is 2 x bianry matrix representing the agent's location within its obscrvation area.
Item. This the same field the state. As in the of the state, we ignore this field.
The actions of Grid environments vary by environment. All share common movement actions that allow the agent to step forward, backword, left, and right, as well as to turn left and right. In KeyDoor, additional actions allow the agent to pick up/ drop down the key and activate the door.
A.3.6 DynamicObstackes
In Obstacles, the agent must reach goal while avoiding moving obstacles. Figures 7a and respectively show a sample state and observation. Agent and goal positions are fixed on two opposite corners of the grid, whereas the locations and movements of the obstacles are stochastic. In the fully-observable version of this task, the agent has full view of the grid and the obstacles, inchuding those behind, and avoiding the obstacles is easier than in the partially-obseevaable version of this task with limited field of view.
7
The agent receives composite reward obtained by adding the following components:
A dense living reward of 0.005;
A dense reward of +0.2 for moving closer to the goal;
A dense reward of -0.2 for moving further away from the goal;
16

---

Published in Transactions on Machine Learning Research (09/2025)
A sparse reward of -1 for bumping into wall or an obstacle; and
A sparse reward of +5 for reaching the goal.
An episode terminates upon bumping into wall, an observable, or reaching the goal.
A.3.7 KeyDoor
In KeyDoor; the agent must reach a goal located behind a locked door; to do so, the agent must first complete the subtasks of locating and picking up the key and opening the door. Figures 7b and 70 rospectively show a sample stale and observation. Positions of key, door, and goal are randomized, as well as the size of both rooms. In the fully-observablee version of this task, the agent has full view of the grid, key, door, and goal. Since we ignore the item field of both states and observations, the fully-obsorvable agent must infer whether it is holding the key based whether it is observable in the grid or not, whereas the partially-observable agent must remember whether it has picked up the key in the past.
The agent receives a composite reward obtained by adding the following components:
dense living reward of -0.05;
dense reward of +0.2 for moving closer to the goal;
dense reward of -02 for moving further away from the goal;
A sparse reward of +1 for picking up the key;
A sparse reward of -1 for dropping the key;
A sparse reward of +1 for opening the door;
A sparse reward of -1 for closing the door; and
A sparse reward of +5 for reaching the goal.
An episode terminates upon reaching the goal.
A.3.8 MemoryFourRooms
In MemoryFourlRomms, the agent must reach a good exit while avoiding bad exit; to identify which exit is which, the agent must first explore the grid until it finds a beacon whose color matches the good exit. Figures 7c and 7 respectively show a sample state and observation. The layout of the four rooms are randomized, as well as the positions of agent, beacon, and exits. The colors of the the exits and beacon are randomly sampled from the set (red, green, blue, yellow). In the fully-observable version of this task, the agent has full view of the grid and is able to immediately identify the good exit's 's identity and location, whereas in the partially-observable version, the agent must first find explore the grid to find both beacon and exit.
The agent receives a composite reward obtained by adding the following components:
A dense living reward of -0.05;
A sparse reward of +5 for reaching the good exit; and
sparse reward of -5 for reaching the bad exit.
An episode terminates upon reaching either exit.
17

---

Published in Transactions on Machine Learning Research (09/2025)
A.4 Architectures
Each environment provides state and information data in variety of formats. This requires to implement several different model architectures to handle these formats, the specifics of which are detailed below. Most architectures rely an overall model that is sequentially composed of a model to learn observation representations ions (the nalure of which depends on the environment), followed by a single recurrent layer, followed by two fully-connected (FC) layers. In each case, only observation histories are considered, and the action-component of the history is ignorod.
A.4.1 HeavenHell
HeawenHell provides states and observations as categorical scalars. We therefore use an embodding layer generale encodings of those values before processing with the rest of the network.
Actor Embedding(256)) LST6) FC(256), ReLU, FC(dim(A)), Softmax
Critic (2beddingg256),, (256, FC(256), ReLU, FC(dim(A))
A.4.2 CartPole/LunarLandee
CartPole and LunarLander provide states and observations as 1D feature vectors. We use a fully-connected layer to process these vectors.
Actor FC(64), ReLU, LSTM(64), FC(64), ReLU, FC(dim(A)), Softmax Critic FC(64), ReLU, (64),644), FC(64), RcLU, FC(dim(AA)
A.4.3 HalfCheetah
HalsCheeiah provides states and observations 1D feature vectors. We use fully-connected layer to process these vectors. The policy network outputs both action means and log standard deviations. The critic network computes action-values by combining observations and actions in its inputs and mapping them to a scalar output.
Actor FC(64), ReLU, L64M(64), FC(64), ReLU, FC(2 dim(AA)
Critic FC(64), ReLU, (64), FC(64), ReLU, FC(1)
A.4.4 GridVerse
Grid provides states and observations dictionaries containing data in various formats, categor- ical, some integral, and some continuous. Due to this additional complexity, we show the architecture used for Grid Verse environments as graphical diagrams in Figure S
A.5 Expert Policies
this section, we briefly explain how we got expert policies used in our methods and evaluations, and to fill our datasels. A more in depth--liccssson into the dataset composition is available Appendix A.6
Fully-Observable Expert Policies. A key factor in our work is that fully-observable policies are usually siguificantly easier to train than partially-observablle policies, and that we may be able to exploit fully- observable guidance help the training of partially-obseraabe policies. In principle, is possible to train such oully-observable policies directly using standard offline methods on the same datasets that would be used for the partially-observable policy. However, in practice, to simplify this component and have strogge
18

---

Published in Transactions on Machine Learning Research (09/2025)
Grid
Embedding
Concat
ReLU Linear
LSTM
ReLU Linear
Linear
Softmax
Action Probs
Agent ID Grid
Embedding
(a) Verse Actor Architecture.
Grid
Embedding
Concat
ReLU Linear
LSTM
ReLU Linear
Linear
Q-Values
Agent ID Grid
Embedding
(b) Verse Critic Architecture.
Figure 8: Verse Architectures.
Table 2: Datasets composition and sizes, and CO-CQL 8 Coefficients
Environment
Dataset Composition
Dataset Size
B
Hard HeavenHell
(Policy) 100 % Random
1 k episodes
0.0001
Easy HeavenHell
(Policy) 100 % Expert
100 timesteps
0.001
Easy HeavenHell
(Policy) 50 Expert, 50 % Random
100 timesteps
0.01
Fasy HeavenHell
(Policy) 100 % Random
100 timesteps
0.0001
Latent CartPole
(Policy) 100 % Expert
1 M timesteps
0.02
Latent CartPole
(Policy) 50 % Expert, 50 % Random
1 M timesteps
0.02
Latent CartPole
(Policy) 100 % Random
1 M timesteps
1.0
Noisy LunarLander
(Policy) 100 % Trained
1 k episodes
0.9
Noisy HalfCheetah
(Policy) 100 % Trained
1 k episodes
0.5
Latent LunarLander
(Policy) 100 % Trained (Policy) 100 % Trained
1 k episodes 0.0001
Latent HalfCheetah
1k episodes
0.4
MemoryFourRooms
(Trajectories) 50 % Success, 50 % Failure (Traecctories) 50 % Success, 50 % Failure
1 k episodes
0.1
KeyDoor
1 episodes
0.1
ObmmaiiOOstacles (Trajectories) 50 Success, 50 % Failure
k episodes
optimality guarantees on the fully-observable expert, we employ online RL methods. To obtain the fully- observable experts for HeavenHell, CartPole, LunarLander, and HalfCheetah, we employ standard non- recurrent SAC (Haarnoja et al., 2018). To obtain the fully-observable experts for GridVerse, we employ standard non-recurrent PPO (Schulman al. 2017).
It should be additionally noted that fully-observable experts also be obtained, depending the setting, without requiring any training at all, c.g. a coded heuristic could act as an expert policy, or bchavior cloning onto a human expert.
Partially-OObeervable Expert Policies. To create some of the datasets used our evaluation, we will need way to generate expert partially-observabbe trajectorics. To generate those trajectories, we train pabteally policies using recurrent SAC (Haarnoja et al. 2018
A.6 Datasets
The composition and size of each dataset varies per environment, shown in Table In the table, we use the following labels:
(Policy) Random. This label denotes trajectories obtained by running random policy.
19

---

Published in Transactions on Machine Learning Research (09/2025)
(Policy) Expert. This label denotes trajectories obtained by running partially-observable policy trained as described in Appendix A..5 Further, were able to verify that the task is being solved well, e.g,, by meeting known return threshold that is considered to be significant for the task.
(Policy) Trained.
This label also denotes trajectories obtained by running partially-observable policy trained as described in Appendix A.5 However, in this case were not able to verify that the task is being solved well, so the policy is still sub-optimal in some regard.
(Trajectory) Success. This label denotes trajectories that are sampled from random policy, but also filtered so that the correct goal is reached. The filtering process does not account for whether the task was solved efficiently even in manner that is congruent to optimal partially-observable control, e.g., trajectory is deemed successful even if it reaches good exit without having made a corresponding observation necessary to identify it from a bad exit.
(Trajectory) Failure. This label denotes trajectories that are sampled from random policy, but also filtered so that the correct goal is not reached.
All datasets will be made publicly available upon publication.
A.7 Hyperparameters
In this section, we show in details the hyperparameters used to train CO-CQL and the other baselines.
A.7.1 CO-CQL Hyperparameters
The general hyperparameters used for CO-CQL for training across all of our datasets are shown in Table 3 The specific 3 coefficient used with each of our datasets is shown in Table 2
We tune primarily the behavior cloning coefficient 8 and the CQL alpha threshold value. All other hyperpa- rameters are carried over un-modified from standard online, non-recurrent SAC as well as non-recurrent CQL (besides batch size and history length which we choose based on computational capacity). We performed simple grid-searching to determine our /3 coefficients.
We find from the COAR analysis in section 5.2.5 that if one has intuition regarding the MDP/POMI /PODP overlap of an environment, then the search space over the optimal B parameter can be shortened; if the expected overlap is higher, then one should start with higher values of 3 and vice versa.
A.7.2 CQL Hyperparameters
The general hyperparameters used for recurrent CQL for training across all of datasets shown in Table 3
A.7.3
IQL Hyperparameters
The general hyperparametees used for recurrent TQT. for training across all of our datasets are shown in Ta-
ble 5
A.7.4
BC Hyperparameters
The general hyperparameters used for recurrent BC for training across all our datasets are shown in Table
A.7.5 TD3+BC Hyperparameters
The general hyperparameters used for recurrent TD3+BC for training across all of our datasets are shown
in Table 7
20

---

Published in Transactions on Machine Learning Research (09/202))
Table 3: CO-CQL Hyperparameterss
Hyperparameter
Environment
Value
Discount
all
0.99
Batch Size
all
32
HeavenHell
Full episodes
Grid Verse
History Length
CartPole
LunarLander
4 timesteps
HalfCheetah
HeavenHell
3e-5
Grid Verse
Actor Learning Rate
CartPole
Lunar HalfCheetah
3e-4
Critic Learning Rate
all 3e-4
Actor Update Frequency
all
2
Target Network Update Frequency
all
Target Entropy Scaling
all Noisy HalfCheetah
0.3
2
Latent HalfCheetah
Latent LanorLander
1
CQL
Noisy Lander
HeavenHiell
Verdeerse
10
CartPole
21

---

Published in Transactions on Machine Learning Research (09/2025)
Table 4: CQL Hyperparameters
Hyperparameter
Environment
Value
Discount
all
0.99
Batch Size
all
32
HeavenHell
Full episodes
Grid Verse
History Length
CartPole
LunarLander
4 timesteps
HalfCheetah
HeavenHell
3e-5
Grid Verse
Actor Learning Rate
CartPole
Lunar HalfCheetah
3e-4
Critic Learning Rate
all 3e-4
Actor Update Frequency
all
2
Target Network Update Frequency
all
Target Entropy Scaling
all Noisy HalfCheetah
0.3
2
Latent HalfCheetah
Latent JanorLander
1
CQL
Noisy Lander
HeavenHiell
Verse
10
CartPole
22

---

Published in Transactions on Machine Learning Research (09/2025)
Table 5: IQL. Hyperparameters
Hyperparameter
Environment
Value
Discount 7
all
0.99
Batch Size
all
32
HeawenHell GridVerse Full episodes
History Length
CartPole LunarLander HalfCheetah
4 timesteps
Actor Learning Rate
all
3e-4
Critic Learning Rate
all
3e 4
State- Value Function Learning Rate all
3e-4 4
Actor Update Frequency
all 2
Target Network Update Frequency
all
1
IQL
all
0.7
IQL 3
all
3.0
Table 6: BC Hyperparameters
Hyperparameter
Environment
Value
Discount
all
0.99
Batch Size
all
32
HeavenHell
Full episodes
GeidVerse
History Length
CartPole
LunorLander HalfCheetah
4 timesteps
Actor Learning Rate
all
3e-4
Actor Update Frequency
all
2
23

---

Published in Transactions on Machine Learning Research (09/2025)
Table 7: TD3+BC Hyperparametens
Hyperparameter
Environment
Value
Discount ?
HalfCheetah, LunarLander
0.99
Batch Size
HalfCheetah, LunarLander
32
History Length
HalfCheetah, LunarLander
timesteps
Actor Learning Rate
HalfCheetah, LunarLander
4
Critic Learning Rate
HolfCheetah, LunarLander 3e-4 4
Actor Update Frequency
HalfCheelah, LunarLander
Target Network Update Frequency
HalfCheetah, LamarLander
TD3+BC a
HalfCheetah, LunarLander
2.5
Exploration noise
HalfCheetoh, LunarLander
0.1
Policy noise
HalfCheetah, LunarLander
0.2
Noise clip
HalfCheetah, LunarTander
0.5
24


Reviews:
[{"review_id": "p6Q7BKLqL5", "reviewer": "Reviewer_AWnz", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths -\n\n1. New Theoretical Contribution\n- Well-defined metrics (COOR and COAR) for quantifying when fully-observable experts can help\n- Clear mathematical framework that bridges theory and practice\n\n2. Practical Algorithm Development\n- CO-CQL effectively combines behavior cloning with offline RL\n- Successfully handles both discrete and continuous action spaces\n- Works across diverse partially-observable scenarios\n\n3. Strong Empirical Validation\n- Comprehensive testing across different types of partial observability\n- Clear performance improvements over baselines\n- Good analysis of robustness to dataset quality\n\nWeaknesses -\n\n1. Limited Theoretical Analysis\n- Lacks formal convergence guarantees\n- Could benefit from theoretical bounds on performance\n\n2. Hyperparameter Sensitivity\n- Context-independent behavior cloning coefficient (β) despite context-dependent optimal values\n- Limited analysis of sensitivity to other hyperparameters\n\n3. Practical Considerations\n- Could provide more guidance on expert policy selection\n- Limited discussion of computational requirements and scalability\n- Missing analysis of sample efficiency\n\nOverall, the strengths outweigh the limitations, and the weaknesses are resolvable through additional analysis", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces a new way to teach AI agents to make decisions when they can only partially see their environment, using pre-recorded data rather than live practice. The key innovation is a method called CO-CQL that learns from two sources: the pre-recorded data and an \"expert\" AI that has full visibility of the environment. The researchers first developed mathematical tools (COOR and COAR) to measure when it's helpful to learn from the fully-visible expert. They then created CO-CQL, which combines learning from the recorded data with copying the expert's behavior when appropriate. They tested their method on various challenges, like when the AI has noisy sensors or limited vision, and found it performed better than existing approaches. The method is particularly clever because it can figure out when to trust the expert's advice and when to ignore it. This research is important because most current methods assume AI agents can see everything in their environment, which isn't realistic for many real-world applications like robotics or self-driving cars.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Hyperparameter Analysis\n\n- Add basic sensitivity analysis for behavior cloning coefficient β\n- Provide clearer guidelines for selecting β across different environments\n- This is critical as β is a key parameter affecting performance and current guidance is limited\n\n2. Theoretical Analysis\n\n- Add basic convergence analysis for CO-CQL\n- Explore theoretical relationship between COOR and performance\n- Would strengthen theoretical foundations but not essential for practical utility", "broader_impact_concerns": "No significant ethical concerns", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "kOZJ7TFBeI", "reviewer": "Reviewer_5Y9P", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- Tackles under explored problem - RL for PO control\n- Problem formulation via asymmetric learning framework where agents can lean on richer state info during training  before deploying in the real world\n- Introduction of COOR and COAR concepts, a fresh way to think about the problem with how much FO and PO optimal actions overlap\n- Broad testing across diverse tasks and consideration of dataset quality variations\n\nWeaknesses:\n\n- Not a big leap in Novelty - the work builds on existing work like COSIL &, CQL\n- COOR and COAR feel disconnected, these two concepts seem more of motivational than practical as they are not directly used in proposed CO-CQL\n- Emperical results are thin as the provided results does not have performance numbers\n- The work assumes having a dataset with both states and observations during training, which is a big ask for truly PO setting\n- No theoritcal guarantees about CO-CQL behavior", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "- Work introduces fresh take on offline RL for PO control using FO expert policies without needing online environment interactions. This makes it practical for cases like self driving or robotics where richer data will be there during training but not on deployment\n- new theoretical measure called  Cross-Observability Optimality Ratio (COOR), a nice way to formalize the intuition that FO guidance can help PO tasks. \n- Propose emperical way to compute COOR by defining COAR (Cross Observability Approximation Ratio) as a workable alternative to COOR\n- Proposes new algorithm (Cross-Observable Conservative Q-Learning) that mixes conservative Q-learning with a behavior cloning loss to learn from FO experts\n- Tested proposed CO-CQL on a bunch of different PO control tasks\n- Robustness analysis via experiements for dataset variations\n- Have some useful guidance for practitioners like their observation that tasks where FO and PO actions align a lot, even simple behavior cloning can do well and that CO-CQL can push that performance further", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- The paper does not clearly articulate how CO-CQL advances (fundamentally) beyond existing methods. e.g., the addition of behavior cloning is extension of TD3+BC (as given in paper). Why is this proposed hybrid approach needed when BC alone performs well in high COOR env. ? A stronger justification is needed. \n\n- Also as a continuation, if the authors can clearly articulate how C0-CQL advances beyond TD3+BC, CQL ... may be by demostrating unique scenarios where C0-CQL outperforms simpler BC or standard CQL in low-COOR environments, that would strengthen this work\n\n- COOR seems like a motivational tool rather than a component in C0-CQL and COAR as an approximation that relies on well-trained policies and probability threshold. How is this threshold chosen? how is COAR computed in practise?  The empirical validation of COAR's correlation with BC performance will strengthen the robustness of this approach\n\n- The HeavenHell example oversimplifies the problem, as it assumes a specific scenario where FO and PO actions overlap significantly. How does the proposed COOR and COAR generalize to more complex PO tasks with less overlap?\n\n- Also analysis of the same example is not convincing: the FO expert ignores the oracle, yet CO-CQL learns to visit it? The work says, \"CO-CQL learns to exploit the fully-observable guidance of how to reach good exit, while learning key piece of info.\" How does the policy network arbitrate between these conflicting objectives? A deeper analysis here is required, without it it reads more accidental than by design \n\n- The comparison against baselines (e.g., CQL, IQL, BC) is not rigorous. e.g.,, the paper claims CO-CQL outperforms other methods but Table 2 only lists dataset compositions and \\beta without performance metrics. The lack of quantitative results (e.g., average returns, success rates) makes it impossible to verify the claims proposed in the paper. \n\n- For baselines, the discussion should more directly address why cloning a potentially OOD FO expert is superior to cloning the in-distribution behavior policy? A more insightful baseline (something like: where BC loss clones the dataset actions instead of the FO expert's actions?) would strengthen this work\n\n- The dataset compositions are varied, but the paper does not discuss how these variations impact CO-CQL’s performance relative to baselines. Why not compare CO-CQL against a purely BC approach in high-COOR environments, given the paper’s admission that BC performs well in such cases?\n\n- The paper assumes access to a dataset containing both states and observations, which is a strong assumption for offline PO RL. In many real-world scenarios (at least from my experience in process manufacturing, ads and healthcare), only observation data may be available during deployment. The work briefly mentions scenarios where state information is accessible during training like simulation, but it does not address how CO-CQL would perform with observation-only datasets. Could you add more details around it? because without it, it would limit the method’s applicability to truly PO settings. \n\n- Include a theoretical analysis of CO-CQL’s convergence or robustness, even if simplified as this would provide confidence in its performance to readers.\n\n\nMinor comments:\n\n- The dataset compositions in Table 2 include mixtures of expert, random, and trained policies, but the process for generating expert policies is only briefly described. The paper states that experts are trained using the same dataset, but no details are provided on the training algorithm. More info here will be helpful\n\n- The paper briefly mentions that representation learning techniques could be integrated with CO-CQL in future work. However, it does not explain why these techniques were not considered in the current evaluation, given their relevance to PO RL. A comparison to such methods would have strengthened the paper’s positioning in the literature.\n\n- The \\beta coefficients in Table 2 vary very widely (from 0.001 to 1.0), but no explanation/analysis is given on how these diverse ranges or  these hyperparameters would have on performance.", "broader_impact_concerns": "I don't see any significant ethical implications in this work as it focuses on technical advancements in offline RL and from my reading I don't see social concerns either", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "BOckTxTpbw", "reviewer": "Reviewer_M5qG", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n* The problem studied in this paper is well-motivated. \n\n* The presentation is clear and easy to follow.\n\n* Empirical results are analyzed in detail. I am particularly interested in the latent state dimension example, as we usually do not have access to the higher-order information such as velocity in many control problems.\n\n**Weaknesses**\n\n* In the abstract, it claims that the proposed CO-CQL algorithm leverages fully-observable expert policies in an asymmetric learning setting, without mentioning the need of a well-trained partially-observable policy $\\pi$ which appears in Definition 4.2 and serves as a keystone in CO-CQL. I wonder if the proposed approach still works when there is only a well-trained fully-observable policy $\\mu$ available as in the usual setting of offline RL.\n\n* One of the core contributions of this paper is the proposed COAR metric which serves as a practical estimate of COOR. However, it does not contribute to the proposed CO-CQL algorithm at all. While I understand that it still has its strength in evaluating the post-training performance, its importance is not as clear as suggested.\n\n* The proposed algorithm is compared only with other baseline methods that are not supposed to handle the partially-observable problems. A more convincing way to demonstrate its strength is to compete with approaches that are particularly designed for this setting. An example can be found in the paper by Morad et al. (https://arxiv.org/pdf/2303.01859).\n\n* There is a lack of concrete approach to transition from partially-observable data to sequential data suggested in Section 4.2 (namely, from $\\pi(s)$ to $\\pi(h)$).\n\n* The selection of hyperparameter can be critical to the performance of the proposed method as shown in Figure 4 and in Table 1. Can we get a better understanding towards it than those elaborated in section 5.2.5? \n\n* Experiments are rather simple. Since the main contribution of this paper is on the empirical side, it would be more convincing if more difficult experiments (e.g., hopper, walker and humanoid) can be conducted and presented.\n\nOverall, this paper needs a major revision to clarify its contributions and provide more convincing evidence.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a method that leverages fully-observable expert policies to partially-observable control settings. Particularly, it incorporates historical data into the network input to mitigate the influence of partial observability.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "* What does *hao* mean in $Q(hao, a')$ in Equation 11? It also appears in other equations.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)