Leveraging Fully Observable Policies for Learning under Partial Observability
Authors: Hai Huu Nguyen
Andrea Baisero
Dian Wang
Christopher Amato
Robert Platt
Venue: CoRL 2022 Poster (2022)
Keywords: Partial Observability
Imitation Learning
Fully Observable Expert

Abstract: Reinforcement learning in partially observable domains is challenging due to the lack of observable state information. Thankfully, learning offline in a simulator with such state information is often possible. In particular, we propose a method for partially observable reinforcement learning that uses a fully observable policy (which we call a \emph{state expert}) during training to improve performance. Based on Soft Actor-Critic (SAC), our agent balances performing actions similar to the state expert and getting high returns under partial observability. Our approach can leverage the fully-observable policy for exploration and parts of the domain that are fully observable while still being able to learn under partial observability. On six robotics domains, our method outperforms pure imitation, pure reinforcement learning, the sequential or parallel combination of both types, and a recent state-of-the-art method in the same setting. A successful policy transfer to a physical robot in a manipulation task from pixels shows our approach's practicality in learning interesting policies under partial observability.

Leveraging Fully Observable Policies for Learning under Partial Observability
Hai Nguyen, Andrea Baisero, Dian Wang, Christopher Amato, Robert Platt Khoury College of Computer Sciences Northeastern University, Boston, MA, United States nguyen. hail@nottheasterrn edu
Abstract: Reinforcement learning in partially observable domains is challenging due to the lack of observable state information. Thankfully, learning offline in a simulator with such state information often possible. In particular, we propose method for partially observable reinforcement learning that uses fully observ- able policy (which we call a state expert) during offline training to improve online performance. Based on Soft Actor-Crite (SAC), our agent balances performing actions similar to the state expert and getting high returns under partial observ- ability. Our approach parts of the domain that are fully observable while still being able to learn un- der partial observability. On six robotics domains, our method outperforms pure imitation, pure reinforcement learning, the sequential or parallel combination of both types, and recent state-of-the-art method in the same setting. successful policy transfer to a physical robot in manipulation task from pixels shows our approach's 's practicality in learning interesting policies under partial observability.
leverage the fully-observable policy for exploration and
Keywords: Partial Observability, Imitation Learning, Fully Observable Experts
Introduction
Many real-world robotics control problems feature some degree of par-
tial observability, but policy learning this setting remains a significant challenge in robotics [1]. While there are many reinforcement learn- ing methods that address partial observability in theory [2, 3, 4, 5], their performance remains questionable in practice in interesting partially ob- servable (PO) domains which require long-term memorization and active information gathering [6. 7J. In contrast, the setting of fully observable (FO) control has featured the success of many powerful reinforcement learning (RL) algorithms (e.g., [8, 9, 10, 1 11)). Unfortunately, full ob- servability only holds for a small portion of realistic robotics problems.
Cod Bouch muiching
thyer the right
Cuttral
Experi
Figure 1: correct goal object, state expert takes the red path directly, while a partially observable agent must first take the green path to identify the correct goal object, then take the red path.
To reach the
In this work, we attempt to leverage good fully observable policies (state experts) available only during offline training to help train PO policies that can execute online. We rely the setting of offline training and online execution, a successful RL framework where an agent can use "privileged" information such the state [12, 13, 14, 15] or the belief about the state [6] during offline training, e.g., from simulators, to effi- ciently learn PO policies that are later can be deployed without the access to the privileged information anymore. In this work, the privileged in- formation is not just the state itself but also the state expert. Our setting can be illustrated in a navigation task (Figure 1), which requires an agent to navigate to an unknown goal object on the right, identifiable by an object the left side. While the optimal behavior under partial observability is to first navigate leftwards to identify the goal object, the state expert is able to move the goal object directly. Despite being sup-optimal from the PO perspective, the state expert can provide experience during training leading to the goal object, which is potentially useful for both exploration and part of the policy needed in the PO case after the goal object identified.
6th Conference on Robot Learning (CoRI. 2022), Auckland, New Zealand.

---

In this work, we propose to perform a form of cross-observabiiiiyy soft imitation learning (COSIL), i.e, softly projecting the behavior of the FO agent into a PO agent which still tries maximize its own performance. FO experts are generally easier to obtain than PO ones, which often require more impractical computations, egg,, some sufficient statistics of the entire history like the belief states, which requires the true environment dynamics [16]. The resulting COSIL agent balances behaviors that imitate the FO agent with behaviors that are necessary for PO optimality. Experimental results on six robotics domains show that our proposed COSIL method performs significantly better than pure imitation learning, pure reinforcement learning, the sequential or parallel combination of the two types of learning, and other state-of-the-an methods in the same setting. Moreover, policy with pixel observations learned in simulation is transferred successfully to physical robot, showing COSIL's ' ability to learn policies that can handle the complexities of the real world. Additional details can be seen at our project website https: //sttes...ss/eeee///////e/eeeees //sitesgoooglleeee. e. http//eeewwcolllleeeeeeeo cor1
2 Related Work
Imitation learning methods such as behavioral cloning (BC) [17] and DAgger [18] perform pol- icy learning by minimizing supervised loss between the expert's actions and the agent's. These agents are trained on a dataset of encountered states and expert actions and are utilized successfully in tasks like autonomous flight [19, 20] or self-driving [21]. Several methods combine reinforce- ment learning and imitation learning (often from limited expert demonstrations), e.g. DQID [22], DDPGfD [23], and GAIL [24]. However, these methods are designed for Markov decision processes (MDP), i.e., both the expert and the student can observe the environment state. Closest to our work are ADVISOR [25] and Asymmetric DAgger (A2D) [15]. which leverage FO experts during train- ing for PO tasks. A2D operates in setting requiring differentiable and modifiable state expert, jointly trained from scratch (using the environment states) with the student agent, while we assume given, fixed state expert. In the same setting as ours, ADVISOR adaptively changes the weighting per state between the imitation loss and the reinforcement learning loss (see the appendix for a brief description of ADVISOR). However, it also requires training an additional PO actor to mimic the state expert to compute the weighting. Therefore, both ADVISOR and A2D must wait until the additional actor (ADVISOR) and the state expert (A2D) are properly trained before utilizing them to better train the main agent. In contrast, our method can utilize the state expert immediately.
3 Background
In this section, we introduce the background topics required to understand our contributions, i.e., partially observable Markov decision processes, and the Soft Actor-Critic algorithm.
3.1 Partially Observable Markov Decision Processes
A partially observable Markov decision process (POMDP) [26] is defined by a tuple (8, 22 Po: T, R, 7, where S, A, and Ω are the state space, the action space, and the ob- servation space, respectively. The initial state is sampled according to the starting state dis- tribution (s()). and the next states are sampled according to the stochastic transition function Pr(s' s, a) = T(ss, a. s'). The agent does not observe the states but rather receives observations sampled from the observation function Pr(o a, s') O(s', a, o). In order to act optimally, PO agent must, in general, condition its actions on the entire cettionobseevatton history h1 = (act. ogt), i.e. all the observable information has seen so far [27]. Denoting the space of histories as H. the goal is to find a history policy :: H A(A) which maximizes the expected 7ddiscounted return J+ E ''R(s;, a.)].
Although our work focuses on PO control, it also involves FO agents modeled as a state policy :: S + ((A). To differentiate them elearly, we denote history policies as x and state policies p.
3.2 Soft Actor-Critic
Soft Actor-Critic (SAC) [10, 28] addresses the FO maximum-entropp control problem, i.e., the prob- lem of finding a policy f which solves a given FO control problem while maintaining high action- entropy when possible. SAC does this by extending the standard RL objective with a weighted

---

entropy term,
Y (R(s), +1) +H((((s)))
J = E
=0
where (u(s)) Ba) log p(aa s)] the entropy of policy μ at the given state s, and is a trade-off coefficient which determines the relative importance between the RL objectíve and the entropy-min maximization objective. The entropy term can be interpreted as supplementary pseudo- reward given to the agent, which is high for high-entropy policies and low for low-entropy policies. Therefore, the agent will seek not only to maximize the entropy of the policy in the visited states but also to visit states associated with high-entropy policy.
In practice, SAC is off- off-poliey learning algorithm that employs replay buffer containing past transitions D = {(s, a, r. 8') } parametric value model Qo: i==- SAC trains a parametric policy model pa: S - AA, and a
R. The policy model trained by maximizing
(0) Da-De [Q0(s, a) - a po(a {]]
(2)
while the value model is trained to minimize
(Q(Ø)= BeanD (R(s, 7 [V - (o(s,a))2
(3)
2
V(s) (e) (Qs(s, a) aog (o(a) s)]
(4)
where Q2 is a frozen target model that is updated at a slower pace than Qa to improve stability.
Hyperparameter a plays a central role in SAC, determining how much high-entropy states are preferred pure rewards. Choosing a reasonable can be difficult since it is not directly inter- pretable, and a good value depends dynamically on the current policy's 's expected returns and entropy. Haarnoja al. [28] proposed to automatically adjust or by minimizing its own objective,
Ja (a) = L...s [- log [o(s)] - aaf,
(5)
where H is given target entropy. In practice, Equation (5) modulates such that it increased if the current policy entropy is lower than the target entropy and vice versa. In contrast to choosing a value of a, choosing a value of H is much simpler since it is broadly interpretable as the logarithm of the number of actions that we want the max- max-entropy policy to consider in an average state.
3.3 Soft Actor-Critic for Partially Observable Control
Although primarily designed for FO control problems and state policies μl, SAC can be easily adapted to handle PO control problems and history policies т. like most (if not all) model-free learning algorithms. Two main changes need to take place for SAC: First, all appearances of a state s in the equations and models of SAC must be replaced with a respective history h. This also im- plies the use of recurrent neural network component the overall architecture of policy and value models, e.g., an LSTM [29] or a GRU [30]. Second, the replay buffer must be structured in order to contain and extract (truncated) episodes rather than individual transitions.
4 Cross-Observabillt, Soft Imitation Learning
SAC is based on the premise of max-entropy control and is designed to find policies that solve the control task while pertaining high entropy. This is achieved by extending the standard RL objective with an entropy component which not only pushes the policy model to be more stochastic for any given state but also acts as a pseudo-reward that pushes the policy to visit future states where the policy can be more stochastic. Inspired by this dual effect, we aim to employ a similar technique to exploit FO expert knowledge to train a PO agent.
Consider an offline training scenario where an FO expert j is available, egg., obtained via a planning procedure, a pre-trained model, or a model trained jointly with the PO agent. We propose formu- lating pseudo-reward for the PO agent æ based on the expected similarity with the FO agent 14, expressed as the following cross-bbbervabiiity saft imitation learning (COSIL) objective.
J E
Σ 7 (R(s,, ar) - DD(((()), -aD(((s),,a(())))
(6)
3

---

where ( is some divergence measure between the action- distributions of and 9, e.g., the KL divergence D(u(s), (h)) = Ee-p(s) [log pla s) - log x(a h)], or the total variation divergence ((p(s), (h)) maxalp(als) ()] among other options.
Minimizing the divergence D( x (hh) can be interpreted as a form of cross-observabiiiyy imi- tation learning, which tries to project FO behaviors as PO behaviors. On its own, such form of imitation learning is known to be sub-optimal, e.g., it is not generally possible to replicate the be- havior of μ with я. Further, FO behaviors are known to be, in general, sub-optimal for PO control, e.g., PO agents might need to engage in information-gatherii not exhibit. An agent which strictly minimizes the divergence D would incur an optimality gap between its resulting performance and the true optimal performance (see the appendix).
behaviors which an FO agent would
Despite these issues, we argue that applying trade-off between the pure RL objective and the imitation learning objective in a soft fashion, i.e. using the divergence D as a pseudo-reward can be quite beneficial. We are directly inspired by SAC, which applies similar trade-off between the pure RL and the max-entropy objectives to establish itself as a state-of-the-eee model-free RL algorithm. In SAC too, the max-entropy objective alone is insufficient and inadequate when considered alone and only becomes beneficial when combined with task rewards as a pseudo-reward. Additionally, the FO expert μ is likely be a good source of exploratory behavior even for the PO task. The actions chosen by ja are undoubtedly related to the overall task, and is likely beneficial for the PO agent to explore them thoroughly. Second, there are many PO tasks where the agent can and/or must reach low state uncertainty to solve the tasks. Under such low state uncertainty situations, the optimal PO and FO behaviors overlap strongly, if not perfectly (as long as the state uncertainty is maintained low enough). In such situations, the optimality gap becomes low, and the cross-obeervability imitation learning task becomes beneficial. Finally, because the soft-imitation task is encoded pseudo- reward, the PO agent is generally pushed to achieve situations where there is low state uncertainty, which tends to be very beneficial for PO agents.
Similar to Equation (5), we formulate a minimization objective for a,
Ja(a) = aD En,SoD [(p(()
which modulates a dynamically in order to maintain an expected divergence D. Like the target entropy H in SAC, D is an important hyperparameter that indicates how different the policies x and I are allowed to be on average. The semanties of D depend on the choice of divergence function D(r(h), ps))). and good values of D are likely to be domain-dependeer grid search to find the best value. Please see the appendix for the detailed algorithm.
In practice, we perform
5 Experiments
We perform experiments a diverse set of robot and navigation domains with discrete (D) and continuous action spaces (C), proprioceptive and pixel observations. We focus on the setting where the state experts are given during training, i.e., the setting of ADVISOR [25].
5.1 Domains
Below, we only briefly describe the domains and the behavior of the state experts; see the appendix for more details.
(D) Bumps-1D. [6] Two movable homogeneous bumps rest on a table, with robotics finger mov- ing horizontally above them (Figure 2a). When an episode starts, the positions of all entities are randomized, but the left-right order between the bumps is maintained. The agent must push the rightmost bump (blue) to the right without disturbing the left bump (red). There are four action combinations: move left or right, each with compliant or stiff finger. The agent does not know which bump is rightmost - it must infer using the history of the finger's 's deflected angles and posi- tions while touching both bumps with compliant finger before pushing right the blue bump with stiff finger.
(D) Bumps-2D. [6] The same finger is now constrained to move in a plane above two bumps of different sizes (Figure 2b). must reach the bigger bump (red) and stay there to get the only non- zero reward. However, it is unaware of which bump is bigger and, therefore, must make contact with

---

(a) Bumps-1D 1D
(b) Bumps-2D
(c) Mini-Memory
(d) Car-Flag
(c) LunarLander (f) Block- Block-Picking
Figure 2: Our experimental domains. The first three domains have discrete actions and the last three
feature continuous actions. Block-Pte Block-Picking is the only domain with pixel observations.
both bumps at least once, inferring the bumps' relative sizes from the angular displacement of the finger. The state expert, however, knows the position of the bigger bump and can go there directly.
(D) Minigrid-Memory. [31] An agent (red triangle) must go to matching object located on the right side with an object (a ball or key) located in a small room on the left side (Figure 2c). The grid layout, the matching object type, and the agent's starting position are randomized per episode. The agent might start in a cell where it cannot see the object; therefore, an optimal agent must first gather information by going/turning left into the small room to see the object. We chose the biggest version of this environment [31] with a size of 17.
(C) Car-Flag. [32] A car must reach the green flag, which can be on either the left or right side. The agent normally can only observe the car's position and velocity, but if it is near the blue flag, it can also observe the green flag" side (left/right). The optimal policy will be navigating the green flag only after visiting the blue flag to find the right side. In contrast, the optimal state expert will always go left right towards the green flag, whose position is known the expert.
LunarLander-V. We consider two versions of the classic LunarLander en- vironment [33] where the agent only observes subsets of the full state (see the appendix for more details). During training, the expert, on the other hand, can observe the full state. Masking parts of the state to MDPs into POMDPs common in previous work [34, 35, 5]. However, these do- mains do not explicitly require active information gathering because the missing information (e.g.. velocities) can be inferred only by memorizing short observation histories (e.g.. positions).
(C) JunarLander-a
(C) Block-Picking. Two homogeneous blocks rest on a table. The red block is fixed to the table and cannot be moved. In contrast, the blue one is movable, and the agent will accomplish the task it picks the blue block. The agent takes in a top-down colorless depth image; therefore, it must gather information by trying move both blocks to check for movability. The state expert knows the poses the movable block during training and can always perform a successful pick. This task particularly challenging if learning from scratch because the exploration is hard, the reward is sparse, the action space is continuous, and can potentially be long-horizon.
5.2 Baselines
We compare COSIL against diverse of pure imitation learning agents, pure reinforcement learn- ing agents (general and specialized for POMDPs, on-policy, and off off-policy), and the combination of both types in several ways. If not described explicitly, all of these agents are memory-based
ADVISOR. ADV-On is the original on-policy ADVISOR [25] that is built off PPO [9]. For a fair comparison with COSIL (off-policy), implemented off-policy version of ADVISOR (ADV- Off) using SAC for continuous action spaces [10, 28] and discrete action spaces [36].
DAgger [18] is chosen to represent imitation learning methods instead of naive behavioral cloning agent, because DAgger often performs better.
TD3 refers to recurrent version of TD3 [37] for domains with continuous action spaces, imple- mented in [34].
SAC refers to two recurrent versions SAC for continuous actions [10, 28) and for discrete ac- tions [36), followed the implementation by [34].
BC2SAC pre-trai with behavioral cloning for Pac% of the total training timesteps before being trained with SAC.
5

---

1.0
0.8
OM
Bumps 2D
0.6
0.4
0.4
Minigrid Memory
Bumps- ID
0.2
0.2
0.0
0.0
10k
20c
300
50K
50k
80k
200k
200k
3008
400k
Environment emment Step
Environment Stop
Environment Stop
100
Block Picking
0.8
Rotun
0.6
LunarLander-P
0.4
Car- Flag
0.2
-300
0.0
75A Envirorment Stop
4000
60k
20x
Predronment
Provinomment Step
(Ours)
DAgger
DC+SAC
Ramdom
Expert
Figure 3: Learning curves of all methods (4 seeds with the shaded areas denoting 1 standard devia- tion). Note that all agents (except Random and State Expert) are memory-based agents.
BC+SAC is trained using a weighted combination of behavioral cloning loss LBC and RL losses CRI, ie., the agent is trained by optimizing L WBC + CRI.
VRM a state-of-the-ar off-policy model-based agent [3] for POMDPs with continuous action spaces by combining a recurrent variational dynamic model and SAC agent.
Random, State Expert. While a random agent takes actions randomly, state expert is either trained with full state access or is hand-coded using the domain knowledge.
Additional Baselines include recurrent PPO [9] implemented in [38], on-policy POMDP special-
ized method DPFRL [2], and a non-recurren SAC (Markovian SAC [34]) that uses observations instead of histories. We report the performance of these baselines in the appendix.
5.3 Experiments
Depending on each domain, we record either the success rates or the returns of evaluation agents averaged over ten episodes. For SAC-based agents, we turn off exploration during evaluation. The reported results are averages over four seeds with shaded areas denoting one standard deviation with optional smoothing if needed. For Random and State Expert, their performances are averaged over 100 episodes and visualized as horizontal lines.
Metrics.
Results. The performance of agents is shown in Figure (the performance in LunarLander-V is similar to LumarLander-P and reported the appendix). Overall, COSIL (green) is compa- rable to ADV-Off in Bumps-1D, LunarLander-P, Block-Picking but considerably better in the remaining three domains, which we argue have larger optimality gaps. DAgger performs quite well in LunarLander-P, suggesting that this domain's optimality gap small. This is expected for this domain because the partially observable angular velocities of the vehicle can be inferred from a few recent fully observable angles. Our agent can also perform well in this domain, signifying the ability to work in domains with a small optimality gap. Other baselines that do not use the state expert struggle to learn efficiently, especially in Block-Picking, where random exploration will have a minuscule chance of picking the correct object. This explains the complete failures of naive baselines such as SAC TD3, and highlights the advantage of exploiting the state expert for better exploration. BC2SAC performs poorly across all domains. This way, reinforcement learning often unlearns imitation learning policies when optimizing RL losses. The poor performance illustrated by sudden drops whenever reinforcement learning starts. BC+SAC does not seem to perform well either, often not significantly better than DAgger. A fixed static combination like BC+SAC can be sub-optimal because the optimal PO policy can sometimes act vastly differently from the FO expert.

---

0.5
0.8
Mormel
0.5
Unean Doaay
Success ttaes
0.6
Aft-Off
9.6
0.5
BC
0.4
Rancort
0.4
am 0.2
0-0.
Expect
0=07
D=gs
b)
0.0
0.0
0.0
Envronment
Environment
Figure 4: Additional results: COSIL in MinsGrid-Crossinng Performance of COSIL in Car-Flag when: b) limeartylepponentaalyy decaying a, varying a E {0.0, 0.3, 0.6.0.9); c) vary- ing D E {0.5, 0.6, 0.8, 0.9, 1.0, 1.1).
For instance, the PO agent often must act to gather more information about the state. contrast, the state expert hardly has to do that, given its full knowledge of the state. Therefore, it is difficult for BC+SAC to behave optimally by fixing the weights between BC and RL loss. In fact, this observa- tion has indeed motivated ADVISOR, which adaptively changes the weights between the BC loss and the PPO [9] loss for each state (see the appendix for more details).
Training & Hyperparameter Tuning. All agents are trained using complete episodes (truncated too long). On-policy agents (i.e., DPFRL, PPO, and ADV-On) use 5-10x more samples than other off-poliie off-policy agents to train. We perform grid search over relevant hyperparameters of each method (see the appendix for more details), then select the best combinations report.
5.4 Additional Experiments
Performance in Domains with Small Optimality Gap. When the optimality gap is small, e.g. when the state expert is nearly optimal under both full and partial observability, the best learning strategy is to purely mimic the state expert. We demonstrate this point a variant of [31] with grid size of 25 and 10 crossings (see the appendix for details about the domain). This domain has a very small optimality gap: the paths that lead to goal cell of an optimal PO agent (which only observes local area around it) and state agent (which observes the whole grid world) are quite similar. Figure 4a shows that both COSIL (with D = 0) and ADV. Off are outperformed by behavioral cloning expected. In the case of COSIL, by setting D 0, a will be increased to penalize the agent heavily it behaves differently from the state expert (see Equation (6)) gradually morphing COSIL into behaving like BC.
Not adapting a over time. In Figure 4b, we illustrate the performance of COSIL in Car-Flag without adapting to reach the target divergence D. We investigated reducing 1 - 0 lin- carlylexponentially over time or keeping it fixed at different values {0.0, 0.3, 0.6, 0.9}. The figure shows that adapting is superior to other ways of modulating in this domain. However, similar to SAC, there is no hard constraint that a must be always adapted.
Performance with varying D. We investigate the sensitivity of COSIL in Car-Flag when varying D from 1.1 to 0.5 with step of 0. Figure 4c shows COSII. is stable with half of the tested range.
5.5 Real Robot Experiments
We transfer the policies learned in simulation in Block-Picking to a Universal Robot URS arm mounted with Robotiq 2F-85 gripper (Figure 5). We use two cameras: one Occipital Structure Sensor (Cam 1) and one Microsoft Azure Kinect camera (Cam 2), and combine their point clouds to create depth image by using orthographic projection at the gripper's position. To better transfer, during training, we add uniform noise to the positions and the yaw angle of the two blocks and add Perlin noise [39, 40] to the simulated depth images. We pick the best policies in the simulation of ADV-Off and COSIL, roll out ten episodes for each, and count the number of successful picks.
During the deployment, when the agent manipulates the supposedly immovable block, we temporar- ily disabled the movements along 2, U.9 axes (see coordinates in Figure 2f) so that the agent will manipulate the other block after a while trying to move the earlier block without observing any
7

---

movements, The disabling is necessary because when the agent observes that the immovable block movable, will persistently manipulate that block. The temporal blocking allows to avoid heav- ily gluing the immovable block to the table. However, it also leads to cases that the agents never see during training (the agents desire to move the gripper, but nothing moves due to the blocking), but both COSIL and ADV-Off seem to generalize well.
Combine Point Clouds
Comeea
Ortiogaphicc Projection
Method
Simulation
Real-world
Atrkkspace
ADV-Off 9/10
6/10
COSIL
9/10
8/10
Table I: Result of transferring policies learned in simulation the real world in Block-Picking.
Figure 5: Experimental robot setup.
Table shows the performance comparison of pick successes between COSIL and ADV-Off. The results show that COSIL policy learned in simulation can actually perform well in the real world, in a PO task that would be very challenging to learn without the aid from a state expert during training. Empirically, our policies are more robust to rotational and translational misalignment of the two blocks (see videos on our project website), leading to more successful picks. However, similar success rates are expected with a bigger sample size, given their similar performance in simulation.
Limitations
COSIL can bias exploration negatively if the state expert is heavily sub-optimal under partial observability, being equal to giving very poor demonstrations. In this case, is doubtful that any method can utilize FO experts to aid in learning optimal PO policies. To illustrate, we modify Bumps-20 such that staying on top of the bigger bump without visiting the smaller bump in the past will be penalized heavily (-100). Previously, such behavior of the state expert did not incur any penalty. Moreover, maintain Markovian rewards (e.g., the reward only depends on the last state and action), we add a flag indicating whether the smaller bump was visited to the observation. Figure shows that both COSIL and ADV-Off are outperformed by SAC, with ADV-Off failing learn completely. For COSIL, setting high value of D might alleviate the situation as COSIL will become closer to normal SAC agent.
coe 10und
0.20
0.25
00:
u.e
0.58
200
42%
432
1000
Figure 6: An example when COSIL can be less perfor- mant than SAC.
Another limitation is the requirement of constant access to expert actions at every state during offline training, which might be impractical. One potential fix is using our method with variant of SAC that learns from limited demonstrations, such as one proposed by Liu et al. [41]. Another possible direction for addressing this issue is to use method proposed by Gangwani et al. [35] with limited demonstration trajectories given by state expert instead of POMDP expert.
7 Conclusion
This paper tackles POMDPs in robotics from an unconventional angle: leveraging fully observable policies during offline training to better train partially observable policies that later can be deployed online. We introduce cross-observabiiity soft imitation learning, which balances achieving high
PO performance with acting similarly to the FO expert. With COSIL, the PO agent can exploit the FO expert for targeted exploration and information gathering during offline training. COSIL outperforms the state-of-the-arr method in the same setting in several robotics domains. COSII. also performs better than other conventional approaches and can learn policies that are successfully transferred to physical robot.

---

Acknowledgments
This material is supported by the Army Research Office under award number W91 the U.S. Office of Naval Research under award number N00014-19-1-2131; NSF grants 1816382, 1830425, 1724257, and 1724191.
References
[1] H. Kurniawati. Partially observable markov decision processes and robotics. Annual Review of Control, Robotics, and Autonomous Systems, 511::553-277, 2022.
[2] X. Ma, P. Karkus, D. Hsu, W. S. Lee, and N. Ye. Discriminative particle filter reinforcement learning for complex partial observations. International Conference on Learning Represen- tations, 2019.
[3] D. Han, K. Doya, and J. Tani. Variational recurrent models for solving partially observable control tasks. In International Conference Learning Representations, 2019.
[4] M. Igl, L. Zintgraf, T. A. Le, Wood, and S. Whiteson. Deep variational reinforcement learning for pomdps. In International Conference on Machine Learning, pages 2117-2126. PMLR, 2018.
[5] L. Meng, R. Gorbet, and D. Kulić. Memory-based deep reinforcement learning for pomdps. 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 5619-5626. IEEE, 2021.
[6] H. Nguyen, B. Daley, X. Song. C. Amato, and R. Platt. Belief-grounded networks for ac- celerated robot learning under partial observability. In Conference on Robot Learning, pages 1640-1653. PMLR, 2021.
[7] H. Nguyen, Z. Yang, A. Baisero, X. Ma, R. Platt, and C. Amato. Hierarchical reinforcement learning under mixed observability, arXiv preprint arXiv:2204.0m 00898, 2022.
[8] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, al. Human-level control through deep rein- forcement learning. nature, (18(7544):55995333 2015.
[9] Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, Proximal policy optimization algorithms. arXiv preprint arXiv: 1wtxXiv:17000000 1707.06447 2017.
[10] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International conference on machine learning, pages 1861-1870. PMLR, 2018.
[11] V. Mnih, A. P. Badia, M. Mirza, A. Graves, Lillicrap, T. Harley, D. Silver, and K. Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International conference on machine learning, pages 1928-1937. PMLR, 2016.
[12] A. Baisero, B. Daley, and C. Amato. Asymmetric dqn for partially observable reinforcement learning. In The 38th Conference on Uncertainty Artificial Intelligence, 2022.
[13] A. Baisero and C. Amato. Unbiased asymmetric reinforcement learning under partial ob- servability. In Proceedings of the 21st International Conference on Autonomous Agents and Multiagent Systems, pages 44-52, 2022.
[14] L. Pinto, M. Andrychowicz, Welinder, Zaremba, and Abbeel. Asymmetric actor critic for image-based robot learning. In 14th Robotics: Science and Systems, RSS 2018. MIT Press Journals, 2018
[15] A. Warrington, J. W. Lavington, A. Scibior, M. Schmidt, and E. Wood. Robust asymmetric learning in pomdps. In International Conference on Machine Learning, pages 11013-11023. PMLR, 2021.

---

[16] L. P. Kaelbling, M. L. Littman, and A. R. Cassandra. Planning and acting in partially observ- able stochastic domains. Artificial intelligence. 101(1-2):99-33, 1998.
[17] M. Bain and C. Sammut. A framework for behavioural cloning. In Machine Intelligence 15, pages 103-129, 1995.
[18] S. Ross, G. Gordon, and D. Bagnell. A reduction of imitation learning and structured predic- tion to no-regret online learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pages 627-635. JMLR Workshop and Conference Pro- ceedings, 2011.
[19] S. Ross, N. Kelik-Barkhudarov, K. S. Shankar, A. Wendel, D. Dey, J. A. Bagnell, and M. Hebert. Learning monocular reactive uav control in cluttered natural environments. In 2013 IEEE international conference on roboties and automation, pages 1765-1772. IEEE, 2013.
[20] A. Giusti, Guzzi, D. C. Cireşan, F.-L. He, J. P. Rodríguez, F. Fontana, M. Faessler, C. Forster, J. Schmidhuber, G. Di Caro, et al. A machine learning approach visual perception of forest trails for mobile robots. IEEE Robotics and Automation Letters, 1(2):661-66, 2015.
[21] M. Bojarski, D. Del Testa, D. Dworakowski, B. Firner, B. Flepp. P. Goyal, L. D. Jackel, M. Monfort, U. Muller, J. Zhang, al. End to end learning for self-driving cars. arXiv preprint arXiv: 1604.003337 2016.
[22] T. Hester, M. Vecerik, O. Pietquin, M. Lanctot, T. Schaul, B. Piot, D. Horgan, J. Quan, A. Sendonaris, I. Osband, et al. Deep q-learning from demonstrations. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
[23] M. Vecerik, T. Hester, J. Scholz, F. Wang, O. Pietquin, B. Piot, N. Heess, T. Rothörl, T. Lampe, and M. Riedmiller. Leveraging demonstrations for deep reinforcement learning on robotics problems with sparse rewards. arXiv preprint arXiv:1707.088// 2017.
[24] J. Ho and S. Ermon. Generative adversarial imitation learning. Advances in neural information processing systems, 29, 2016.
[25] L. Weihs, U. Jain, I.J. Liu, J. Salvador, S. Lazebnik, A. Kembhavi, and A. Schwing. Bridging the imitation gap by adaptive insubordination. Advances in Neural Information Processing Systems, 34, 2021.
[26] K. J. Astrom. Optimal control of markov decision processes with incomplete state estimation. Math. Anal. Applic., 10: 174-205, 1965.
[27] S. P. Singh, T. Jaakkola, and M. I. Jordan. Learning without statc-estimation in partially observ- able markovian decision processes. In Machine Learning Proceedings 1994, pages 284-292. Elsevier, 1994.
[28] T. Haarnoja, A. Zhou, K. Hartikainen, G. Tucker, S. Ha, J. Tan, V. Kumar, H. Zhu, A. Gupta, Abbeel, al. Soft actor-critic algorithms and applications. arXiv preprint arXiv:1812.05900 2018.
[29] S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural computation, 9(8): 1780, 1997.
[30] K. Cho, B. Van Merriënboer, D. Bahdanau, and Y. Bengio. On the properties of neural machine translation: Encoder-decoder approaches. arXiv preprint 4rXiv:1409.1259, 2014.
[31] M. Chevalier-Boisvert, L. Willems, and S. Pal. Minimalistic gridworld environment for openai gym. https://gitthb: www.//nwxiieeccgggmmmmmm...
2018.
[32] H. Nguyen. Pomdp robot domains.
https://github.co http://tttub..commmabbbgguu
pomdp- domains, 2021.
[33] G. Brockman, V. Cheung, I.. Pettersson, J. Schneider, Schulman, J. Tang, and Zaremba. Openai gym. arXiv preprint 1606v:1606.013 2016.
10

---

[34] T. Ni, B. Eysenbach, and R. Salakhutdinov. Recurrent model-free rl can be strong baseline for many pomdps. In International Conference on Machine Learning, pages 16691-16723. PMLR, 2022.
[35] T. Gangwani, J. Lehman, Q. Liu, and J. Peng. Learning belief representations for imitation learning in pomdps. Uncertainty in Artificial Intelligence, pages 1061-1071. PMLR, 2020.
[36] Christodoulou. Soft actor-critic for discrete action settings.
arXiv preprint
arXtv:1910.07207, 2019.
[37] S. Fujimoto, H. Hoof, and D. Meger. Addressing function approximation error in actor-critic methods. In International conference on machine learning, pages 1587-1596. PMLR, 2018.
[38] I. Kostrikov. Pytorch implementations of reinforcement learning algorithms. attps:// github. http://tttrttkko//oooaa-a------- gail gail,, 2018.
[39] James, A. Davison, and E. Johns. Transferring end-to-end visuomotor control from sim- ulation to real world for multi-stage task. In Conference on Robot Learning, pages 334-343. PMLR, 2017.
[40] K. Perlin. An image synthesizer. ACM Siggraph Computer Graphics, 19(3):887-296, 1985.
[41] G. Liu, L. Zhao, P. Zhang, J. Bian, T. Qin, N. Yu, and T.-Y. Liu. Demonstration actor critic. Neurocomputing, 434:194-202, 2021.
[42] Achiam. Spinning Up in Deep Reinforcement Learning. 2018.
[43] E. Coumans and Y. Bai. Pybullet, python module for physics simulation games, robotics and machine learning. 2016.
[44] D. Wang, C. Kohler, X. Zhu, M. Jia, and R. Platt. Bulletarm: An open-source robotic ma- nipulation benchmark and learning framework. In The International Symposium of Robotics Research, 2022.
[45] E. Todorov, T. Erez, and Tassa. Mujoco: physics engine for model-based control. In 2012 IEEE/RSJ international conference on intelligent robots and systems, pages 5026-5033. IEEE, 2012.
[46) A. Rafinn, A. Hill, A. Gleave, A. Kanervisto, M. Ernestus, and N. Dormann. Stable-baselines3: Reliable reinforcement learning implementations. Journal of Machine Learning Research, 22 (268):1-8 2021.
[47] D. P. Kingma and J. Ba. Adam: method for stochastic optimization. arXiv preprint arXiv:1412.6980. 2014.


Reviews:
[{"review_id": "vsWy0scscM6", "reviewer": "Reviewer_oqag", "summary": "This paper proposes Cross-Observability Soft Imitation Learning (COSIL), a method for augmenting partially-observable RL training with a fully-observable RL policy to improve performance during training. The method introduces the COSIL objective, which is similar to the SAC critic but swaps out the entropy term for a distance metric measuring the similarity between the partially-observable policy and fully-observable policy. The paper studies experimental settings with varying amounts of optimality gap, which consists of the behavior policy performance difference between expert partially-observable policies and fully-observable policies. Specifically, they study 6 different environments that include some form of partial observability, such as withholding geometric or physical properties of the scene.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The paper is easy to follow and well written\n- The motivation and comparison with SAC is helpful\n\nWeaknesses:\n- The significance is limited because the empirical settings seem quite contrived (ie. adding features to the environment that are not standard in robot learning). It feels that each of the experimental settings introduce partial observability in fairly forced ways by adding specific MDP adjustments to normal robot learning tasks. More relevant experiments can be motivated by similar works in multi-modal learning that consider state masking or modality masking.\n- Similar to the above point, it’s unclear how widely applicable COSIL is or whether it will only work in specific partially observable MDPs; with a very suboptimal state expert, performance can actually be worse, as shown in Figure 6. This may require very careful selection of when and how to apply COSIL.\n- The performance of COSIL in low optimality gap environments is not great\n- The real world performance improvement is hard to gauge, perhaps due to the small evaluation sample size. The evaluation also introduced a large sim-to-real gap as mentioned in 5.6.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The motivation with SAC is elegant and helps introduce an original and interesting method. My concern is that the impact is limited by the empirical results, which are mediocre even on fairly contrived settings. This seems to suggest challenges for the applicability and significance of the method, even if it is sound and novel.", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "- Were the fully observable policies trained in parallel or pretrained? In section 2, it’s claimed that a drawback of ADVISOR and A2D are that the state expert must be pretrained, but it’s not clear which setting the experimental settings use. Are there experiments comparing using a fixed pretrained state expert vs. a state expert that is trained in parallel with the partially observable expert?\n- Were the BC baselines in 5.2 fully observable or partially observable experts?\n- The baseline for ADV-OFF seems poor in the most complex domain, block picking. The variance is high, and there is a strange dip early in training. Can you run more seeds?\n"}}, {"review_id": "lIprd4TEBeW", "reviewer": "Reviewer_a2e5", "summary": "This work studies an imitation learning setting in which the learner is subjected to partial observability while the expert has full observability. The proposed approach introduces a regularization term for the actor to encourage the learner to match the action distribution of the expert, which observes the full state. The strength of this term is controlled by \\alpha, which is updated by dual gradient descent to match some target divergence. The resulting method COSIL is compared to an array of algorithms on a series of environments with designed partial observability.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: The paper is overall well-written, and the proposed method COSIL is compared to a strong array of algorithms. COSIL performs either comparably to or better than alternative methods, including ADVISOR which is designed with the same objective in mind. A policy learned by COSIL in simulation is also shown to transfer well onto a physical robot.\n\nWeaknesses: As pointed out in the limitations section,\n- to compute the divergence term during training, we require the expert actions at every state visited by the learner, and\n- the state expert, while optimal under full observability, can be suboptimal under partial observability.\n\nThe second weakness is somewhat concerning to me. The setup studied in this work is a hybrid of RL and IL in that both environment rewards and expert action labels are provided at each time-step, which is strictly more informative than the RL setting and the IL setting. With additional information, I would expect an algorithm designed for this setup should converge to optimal performance or perform better than algorithms that use only rewards and only labels. However, as depicted in Fig. 4a, BC performs better than COSIL when the expert is suboptimal under partial observability.\n\nTo account for the potential suboptimality of the state expert, the value of \\bar{D} needs to be carefully chosen so that we allow the learner to find a better solution than a suboptimal expert. It seems that too-small values can lead to worse success rates (Fig. 4c) and I suspect that the best choice of \\bar{D} would depend on the environment and expert.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Overall, this paper studies a practical setting and shows improvements on existing algorithms for this setup. The proposed method is intuitive and rigorously evaluated across multiple environments.\n\n=============\n\nPost-rebuttal: I appreciate the additional analysis on sensitivity to \\bar{D} and adapting \\alpha. It's helped me understand how alternative design choices affect COSIL.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {"issues": "Questions:\n- Why does the transferred ADV-Off policy perform worse in the real world when the performances are similar in simulation?\n- In Fig. 4b, how does COSIL perform with fixed values for \\alpha? I’m curious whether we need to adapt the strength of the divergence at all.\n- Would training for longer in Minigrid-Memory and Block-Picking lead to better converged performance for COSIL?"}}, {"review_id": "MAFBcvCczN1", "reviewer": "Reviewer_6hqs", "summary": "- goal: RL for partial observability\n- idea: leverage good fully observable policies (state experts) to help train partially observable policies\n- introduced objective: minimizing divergence (e.g. KL) between expert and partially observable agent\n- training in such a way is suboptimal, but applying objective in a soft fashion is beneficial\n- car-flag: agent learns to accelerate only once it gains more information\n- lander: policy less aggressive than expert due to partial observability\n- block-picking: agent doesn’t see color. quickly moves to other block once it sees it cannot pick it. real setup: use sim policies with real depth maps.\n- Results:\n  - does well across the board compared to dagger and RL methods\n  - method performs well in both optimality gap and no gap.\n- limitation: requires constant access to expert actions at every state during training. proposes fix by learning from limited demonstrations.\n", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "strengths:\n- paper is pretty well explained\n- policy can handle partial observability.\n- it seems that the policies are doing as well as the expert but in a more careful manner until they observe the right information because they aren’t as confident as the state expert, this seems like a good thing.\n- good results across all tasks, whether there is optimality gap or not\n\nweaknesses:\n- it’s not very clear to me in what real world scenario you have access to a state expert?\n- not clear how practical or useful this is in the real world\n- real setup is pretty limited\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Well written paper, good results on narrow setups, but rather limited real setup, and not clear how practical or useful this is in the real world.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {"issues": "- maybe the authors can motivate better / clarify real life scenarios where the approach is directly applicable? (where a state expert is available)\n- The video and voice over is too fast, it’s hard to follow\n"}}, {"review_id": "2gs3a6X-2-lo", "reviewer": "Reviewer_Spnc", "summary": "The paper is interested in reinforcement learning in partially observed tasks. Specifically, the approach aims to make use of an online expert with full observability to improve the training of an SAC agent. The key insight is that the fully observable experts actions may not always be optimal for the partially observable agent.\n\nThe methods two main components are:\n- Adding a BC term to the SAC reward.\n- Dynamically adjusting the weight of the BC term to map to a target BC term. \n\nExperiments suggest that the method outperforms SAC+BC, DAgger, and other relevant baselines on some simulated and one real robot task.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "*Strengths*\n\n- Studying training RL policies under partial observability is an interesting and relevant problem for robotic learning.\n- The motivation makes sense in partially observable tasks, a fully observable expert actions may not always be optimal for the partially observed agent.\n- The paper has experiments on a physical robot. \n\n*Weaknesses*\n\nFirst, I have some questions about the method:\n- The proposed method as I understand it simply amounts to adding a BC loss to the reward with an online queryable expert at each state. Section 4 frames this as a KL divergence, but looking in the appendix the online expert is deterministic and this is simply amounts to an MSE loss. This added BC objective is commonplace in many IL+RL algorithms algorithms, the paper could do a much better job at explaining how their instantiation is different from prior work.\n- In general, I don't understand what about the method is \"Soft\" Imitation Learning. The introduction and Section 4 emphasizes \"soft\" but what does this mean and how does it describe the method? I found the paragraph in lines 145-158 unclear in explaining this.\n- I also don't understand fully why the weight on the BC term is adapted to hit some target divergence? I understand that the method is trying to balance maximizing reward vs. matching the potentially suboptimal expert, but I would expect the choice of alpha to do this. It would be good to see an ablation of the COSIL method with the adaptive re-weighting part removed. \n\nSecond, I have some questions about the experiments:\n- What is different between the proposed method and the SAC+BC baseline? Is it just that in COSIL the BC loss is part of the reward? I find it surprising that performs so much worse than COSIL considering how similar they seem. \n- I would imagine that if the expert is suboptimal as described, the pure SAC agent would eventually outperform COSIL. It would be interesting to see at what point SAC starts to outperform. \n\nThird, overall the paper clarity could be improved significantly. The paper could better explain what about the current method is novel and why it is important for partially observed tasks. It should also have more details about the method implementation in the main text, including the algorithm block which is currently in the appendix. \n\nI also have some larger questions about the practicality of the problem setup. For example, the assumption that there is an online expert who gives action labels not just on current online states, but any state the agent wants to query (e.g. for target states)  is strong. For example a human teleoperator cannot do this in the real world, which seems to limit the approach largely to simulation. And if the policies are learning in simulation, then sample efficiency is less of a concern, and pure SAC which will likely eventually reach higher asymptotic performance would be a better choice. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Overall the paper has some solid insights around learning in partial observability and using a state based expert. But the paper currently does not clearly explain and motivate the method, and I have some open questions about the method, experiments, and general setup that hopefully can be addressed in the rebuttal. ", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Poor: Difficult to follow and understand.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "issues": "See Strengths and Weaknesses.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)