Learning to Look: Seeking Information for Decision Making via Policy Factorization

Authors: shivin  dass
jiaheng  hu
ben  abbatematteo
peter  stone
roberto  martín-martín
Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-06-07 10:29:04

Abstract: Many robot manipulation tasks require active or interactive exploration behavior in order to be performed successfully. Such tasks are ubiquitous in embodied domains, where agents must actively search for the information necessary for each stage of a task, e.g., moving the head of the robot to find information relevant to manipulation, or in multi-robot domains, where one scout robot may search for the information that another robot needs to make informed decisions. We identify these tasks with a new type of problem, factorized Contextual Markov Decision Processes, and propose DISaM, a dual-policy solution composed of an information-seeking policy that explores the environment to find the relevant contextual information and an information-receiving policy that exploits the context to achieve the manipulation goal. This factorization allows us to train both policies separately, using the information-receiving one to provide reward to train the information-seeking policy. At test time, the dual agent balances exploration and exploitation based on the uncertainty the manipulation policy has on what the next best action is. We demonstrate the capabilities of our dual policy solution in five manipulation tasks that require information-seeking behaviors, both in simulation and in the real-world, where DISaM significantly outperforms existing methods. More information at https://robin-lab.cs.utexas.edu/learning2look/.

Main paper: 
 introduction: 
 Intelligent decisions can only be made based on the right information. When operating in the environment, an intelligent agent actively seeks the information that enables it to select the right actions and proceeds with the task only when it is confident enough. For example, when following a video recipe, a chef would look at the TV to obtain information about the next ingredient to grasp, and later look at a timer to decide when to turn off the stove. In contrast, current learning robots assume that the information needed for manipulation is readily available in their sensor signals (e.g., from a stationary camera looking at a tabletop manipulation setting) or rely on a given low-dimensional state representation predefined by a human (e.g., object pose) that also has to provide the means for the robot to perceive it. In this work, our goal is to endow robots with the capabilities to learn to perform information-seeking actions to find the information that enables manipulation, using as supervision the quality of the informed actions and switching between active perception and manipulation only based on the uncertainty about what manipulation action should come next.Performing actions to reveal information has been previously explored in the subfields of active and interactive perception. In active perception [1, 2, 3] , an agent changes the parameters of its sensors (e.g., camera pose [4, 5, 6] or parameters [7, 8, 9] ) to infer information such as object pose, shape, or material. Interactive perception [10] solutions go one step further and enable agents to change the state of the environment to create information-rich signals to perceive kinematics [11, 12] , ma-8th Conference on Robot Learning (CoRL 2024), Munich, Germany.Figure 1 : DISaM for tasks with information-seeking behavior. To make the right decision in a task (e.g., what beverage to pick or in what dining set to place it), a robot may need to seek taskrelevant information (the time of day to decide the beverage or the person at the table to choose where to place it). We formalize such information-seeking tasks as factorized contextual MDPs and solve them with a dual policy collaborative approach where an information-seeking policy (π IS ) takes active perception actions to search for the right contextual information, and an informationreceiving policy (π IR ) consumes this retrieved context to select the right manipulation actions. terial [13] , or other properties [14, 15, 16, 17] . However, these solutions fully factorize the problem into perception and subsequent task-oriented manipulation, creating the need for a human to specify what information to gather. Belief space planning approaches [18, 19, 20] jointly reason about how actions of an agent reduce uncertainty, and how this, in turn, leads to better actuation. However, correctly predicting the information that future actions may reveal requires accurate forward and observation models that are generally not available, leading to plans that grow quickly in complexity with the number of actions and the intricacy of the state and sensor spaces. Instead of planning, other solutions to Partially Observable Markov Decision Processes (POMDPs) [21] rely on reinforcement learning to train a unified agent with memory that first finds information and then acts based on it. However, finding such long sequences of actions is prohibitively challenging, especially with sparse task reward, thus limiting their applicability to simple settings [22, 23, 24] .Our main insight is to formulate these manipulation tasks with information-seeking behavior as Contextual Markov Decision Processes (CMDP) [25] and solve them with two separate collaborative policies, one controlling an information-seeking (IS) agent and the other an information-receiving agent (IR). We present DISaM, a Dual Information-Seeking and Manipulation policy solution for CMDPs, where an IS agent acts to determine the context, while the IR agent acts based on it to accomplish an overall manipulation task. The context represents the information that is unknown to the agent a priori and has to be discovered by the IS to enable IR to take the right subsequent actions. For instance, the context could be a task label represented either as a one-hot vector or a language instruction. In DISaM, both agents can share the same embodiment, as in our cooking example above, or have separate embodiments, for instance when one agent is a scout and the other is an operator awaiting information, but in both cases, our policy factorization matches a natural factorization of the agents based on different action and/or observation spaces. Two main benefits result from factorizing into collaborative IS and IR policies to solve the CMDP. First, if we assume that the correct context is given, we may render the IR-learning problem fully observable and solve it with a simpler solution like imitation learning (IL). Second, the factorization transforms the original long-horizon problem into two shorter-horizon ones, providing an opportunity to exploit additional supervision at the interface; in DISaM, we use the IR policy to provide intrinsic reward for the IS policy, making the IS-learning problem a simple POMDP. These design decisions, on top of making the training process more efficient and tractable, make DISaM compatible with a variety of existing recent IL models in robotics [26, 27, 28, 29] . Finally, at test time, DISaM can simply balance between the information-seeking and the manipulation behaviors (IS and IR policies) based on the uncertainty the IR policy has about the next action to take, using an ensemble architecture.Our contributions include 1) a general formalism for tasks with information-seeking behavior as factorized CMDPs (fCMDPs), and 2) our proposed dual-policy solution for fCMDPs, DISaM. We carry out multiple experiments in simulation and on real robots, where we demonstrate the capabilities of DISaM to enable robots (single or multi-robot systems) to seek and use the right information for manipulation. Additionally we balance between exploring for context and exploiting it, even with multiple stages that require several seeking/acting behavior switching phases. Our proposed approach substantially outperforms the baselines in 3 simulation and 2 real world tasks. 
related work: 
 POMDPs and CMDPs. POMDPs provide a principled framework for decision making under partial observability. They have been extensively studied in the robotics literature, with applications including localization, navigation, manipulation, and human-robot interaction [18, 30, 31] . The intractability of exactly solving POMDPs is well-established [32, 33] , necessitating approximation algorithms [34, 35, 36] . Scaling POMDP planning to real-world settings with image observations remains a challenge [30, 31, 37, 38] , and the assumption of known dynamics and observation models remains prohibitively restrictive in open world settings. Belief-space task-and-motion planning methods similarly require substantial engineering of predicates, skills, and perception routines [39, 40] . Applying model-free RL with some notion of memory (e.g. recurrent neural networks) has achieved some success in partially-observable problems [22, 23, 24] , though these methods generally struggle in long-horizon, hard exploration, sparse reward settings like finding and exploiting task-oriented information in manipulation.Contextual MDPs (CMDPs) [25] or Latent MDPs [41] have been proposed as a special class of POMDPs in which the reward and transition dynamics are determined by a (typically unobserved) context, effectively parameterizing a family of MDPs. This model captures the case in which latent variables change slowly over time or are constant within an episode, e.g., serving a user a web application [25] . Theory about these problems has been developed in multi-task [42, 43, 44] , modelbased [45] , continual [46] , and inverse RL settings [47] , with applications ranging from medical decision making [48] to human-robot interaction [49] . We build on this framework to develop a CMDP model of active perception in which one agent seeks information to determine the latent context and a second acts to maximize the context dependant reward. MPC dual-control methods [50, 51] follow a similar motivation, simultaneously optimizing for system identification and a control objective, but require knowledge of the dynamics of the system and designation of the uncertain parameters.Active and Interactive Perception. Active perception [1, 2, 3] refers to the ability of an agent to intelligently seek out informative observations. Example applications include next-best-view planning for 3D reconstruction [4, 5, 6] and active perception for grasping [52, 53, 54] . When the agent is also equipped with information-gathering manipulation capabilities, the problem is referred to as interactive perception [10] , which has been studied in the context of object segmentation [14, 55, 16] , articulation model estimation [11, 56, 12] , material property estimation [13] , grasping [15, 57, 58] , scene understanding [17] , and beyond. While these approaches demonstrate the necessity and complexity of information-seeking behaviors, they typically define a priori the information that must be recovered from the scene. In contrast, our goal is to enable agents to discover what needs to be inferred through reinforcement.Active perception has also been considered in the reinforcement learning setting [59, 60, 61, 62, 63] . Similar to our factorization, Liu et al. [62] decouple exploration and exploitation in a meta-RL setting but assume that a single exploration phase is sufficient for gathering all task relevant information. Some works learn to crop observations to select salient visual inputs [64, 65, 66] . Recent work has developed RL methods capable of jointly learning manipulation behaviors with active vision to achieve clear view of the manipulator [67, 68, 69, 70, 71, 72, 73] . For example, Shang and Ryoo [71] propose jointly training manipulation and sensory policies by developing an intrinsic reward based on action prediction, encouraging the camera to view the manipulator. Similarly, Göransson [72] train active perception policies to predict sensor observations of the manipulator. These works generally fail to enable sophisticated information gathering behaviors and instead focus on viewpoint selection to facilitate short-horizon manipulation tasks. In contrast, we seek to enable agents to search for task-relevant information, decoupling information seeking from manipulation. 
problem formulation: factorized contextual markov decision processes: 
 We formulate the problem of solving a manipulation task with information-seeking behavior as a Contextual Markov Decision Process (CMDP), a special form of MDP where the dynamics and the reward depend on a hidden contextual variable [25, 45, 47] . A CMDP is represented by the tuple M C = (C, S, A, O, M(c)), where c ∈ C is the set of contexts, s ∈ S is the set of states, a ∈ A is the set of actions , o ∈ O is the set of observations , and M(c) is a function that maps the context to an MDP M(c) = (S, A, O, T c , R c , ρ c ), where T c : S × A × S → [0, 1] is the transition function, R c : S × A → R is an overall task-reward function, and ρ c is the initial state distribution, all dependent on the context. The goal is to find the policy, π c : S → A, that maximizes the expected total discounted sum of rewards,E c [ ∞ t=0 γ t R c (s t , a t ) | s 0 ].Naively training a single policy to solve the CMDP is challenging and usually requires a complex iterative procedure [25, 74] . Instead, we make the important observation that in many practical CMDP problems, including robotic manipulation, the action space of the original agent can be naturally factored into two parts, A = A IS ∪ A IR , where one part of the action space A IS is only useful for context seeking (e.g. a moving camera) while the other part A IR is sufficient to complete the task given the right context (e.g. a robot manipulator). Given this assumption, we propose to factorize the CMDP into a factorized CMDP (fCMDP) with two sub-problems that can be addressed with two collaborative policies trained separately: an information-seeking (IS) policy, π IS : O IS → A IS , that searches and provides context information c IS ∈ C to a second policy, an information-receiving (IR) policy, π IR : O IR × C → A IR , that consumes the context and takes reward-seeking actions based on it. Specifically, for each IS observation O IS , the information is generated by a given or learned function, f c : O IS → C. As different information may be required across the span of a task, these policies would operate iteratively, where π IS continuously searches for information that allows π IR to make the right decision. We provide further details about the formulation in Appendix A.1.Our proposed factorization of the CMDP problem provides new opportunities that our proposed method, DISaM, exploits. In the following we explain how DISaM trains a solution for the proposed fCMDP, focusing especially on the less explored problem of how to train collaborative informationseeking policies. 
disam, dual information-seeking and manipulation policies for fcmdp: 
 In DISaM, we model a partially-observed manipulation problem as an fCMDP, and devise a solution to train a policy for each of the factorized subproblems: an information-seeking policy, π IS θ and an information-receiving policy, π IR . Additionally, we do not assume that f c , the function mapping observations from the IS policy into context for the IR policy, is given; instead, we learn this mapping from IS observations into context, E ϕ : O IS → C. DISaM utilizes a novel optimization procedure that first trains the IR agent on the original task with the reward of the fCMDP assuming access to the true context, and then trains the information-seeking agent to infer the context for the IR agent.To that end, it minimizes a loss based on the difference between the IR policy conditioned on the IS-provided context vs. ground truth context * .In the following, we first explain how the IR agent is instantiated (Sec. 4.1); then, we discuss our novel iterative optimization procedure that utilizes intrinsic rewards generated by the IR agent to train the information-seeking agent (Sec. 4.2); and finally, we provide details on how the uncertainty In Phase 1, we learn the information-receiving policy π IR that takes in ground-truth context information and controls the movement of the robot. In Phase 2, we learn an information-seeking policy π IS as well as an image encoder E ϕ such that the context can be correctly reconstructed from the camera observation. Once all parts are trained, together they create a system that takes in image observations and controls both the robot and the camera.of the IR policy is used in DISaM to determine, in deployment, when the collaborative dual agent needs more information to complete the current task, querying the information-seeking agent, or when it is ready to pursue the task objective via the IR agent (Sec. 4.3).The IR policy takes the current observation and controls the IR agent to interact with the environment, conditioned on the context. We assume that only the actions of the IR agent can achieve the original manipulation goal. In DISaM, we exploit the factorization in the fCMPD and assume we can train the IR policy with access to the ground truth context: π IR : O IR × C → A IR . In this case, the IR agent is essentially operating in a standard MDP and can be optimized relatively easily using either imitation learning or reinforcement learning. In this work we opt for imitation learning and collect a dataset of expert human demonstrations, D = {τ 1 , τ 2 ...τ n }, that lead to the manipulation goal, labeled with ground truth context, c. While in general CMDPs, the context determines the reward, transitions, and initial state distribution, in our experiments we focus on context-dependent rewards. Thus, the context provides information about the goal of the task, including relevant subgoals. In principle, this context can be represented in any format as long as it can express multiple goals; we primarily consider task IDs, but also explore using natural language instructions as the context. Given the dataset of expert demonstrations with ground truth context, we train π IR with behavior cloning (Fig. 2 , Phase 1). The observation space for the IR agent in our experiments, o IR ∈ O IR , consists of images of the robot's workspace acquired by a stationary camera, as well as robot proprioception. Other forms of training the IR policy (e.g., with reinforcement learning) may be possible; we leave the exploration of these alternatives to future work. Our simple IL solution to train the information-receiving policy aligns with multiple recent works [26, 27, 28, 29] that provide trained goal-conditioned policies conditioned on task labels (IDs or language).Once an IR policy, π IR , is trained, DISaM uses it to train an information-seeking agent that includes the IS policy, π IS θ , and an encoder mapping observations of the IS agent into context for the IR policy, E ϕ . The factorization in the fCMDP assumes that the actions from the IS agent alone are not able to directly achieve the original manipulation goal, rather they are sufficient only for determining the context required by the IR agent to solve the task.The observations for the information-seeking agent in our experiments, o IS ∈ O IS , consist of images from the camera controlled by IS policy, the camera's pose, and, in the case of an interactive IS agent, proprioceptive signals from a robot arm. The observation encoder, E ϕ : O IS → C, then converts the observations from the IS agent into contexts to be passed to the IR agent. In cases where the IS agent may need to collect different context information depending on the IR agent's state, we augment the observations of the IS agent with observations from the IR agent, O ′ IS = O IS ∪ O IR . A possible objective for the information-seeking agent is to maximize the original reward of the fCMDP, i.e., the original reward of the manipulation task that the IR policy optimizes. However, the task reward may be sparse, posing significant challenges to the training process of the IS agent. To address this, DISaM leverages a key insight: we can approximately optimize for the task reward by assuming that the IR policy makes the right decisions if provided with the right context. By comparing the actions taken by the IR policy based on the IS-provided context, a IR ∼ π IR (o, c IS ), with the IR policy actions based on the ground truth context (assumed to be provided during training), a GT ∼ π IR (o, c GT ), we can train the IS policy to seek and encode the context information that minimizes the action differences at each time step. Hence, the objective for IS agent is:EQUATIONDISaM uses the distance L between the two action distributions (e.g., cross entropy) to define reward for the IS policy π IS θ , given asR IS = α -β • L(π IR (o, c IS ), π IR (o, c GT ))), with hyperparameters α and β. This objective is denser than sparse task rewards and can be optimized efficiently with reinforcement learning (Fig. 2 , Step 2.2). On the other hand, the ground truth context c GT also enables training the encoder, E ϕ , via supervised learning (Fig. 2 , Step 2.1).We optimize these two objectives iteratively, alternating between training E ϕ while keeping π IS θ fixed, and vice versa, as visualized in Fig. 2 and summarized in Algorithm 1 in Appendix A.2. While the sections above describe the training objectives of the IS agent and the IR agent, we still need to specify a way to decide when to query the information-seeking agent and when to hand control back to the IR agent during deployment. As shown in Fig. 3 , DISaM uses the uncertainty of the IR agent to determine when to act and when to seek new information. Specifically, DISaM maintains an ensemble of encoders E ϕ (o IS t ). At each decision-making step, DISaM samples n contexts {c i t } n i=1 , from these encoders and conditions the IR agent on each of them to generate n action distributions, {π IR (o, c i IS )} n i=1 . DISaM then computes the average KLdivergence between each pair of action distributions as a measure of uncertainty [75] . If the value of uncertainty is higher than some threshold, δ, DISaM assumes that the IR agent does not have sufficient context information to generate reliable actions, and it queries the information-seeking agent to seek further context information. When the uncertainty drops below δ, it indicates that the IR policy is confident about the next manipulation action to take, and DISaM executes the IR policy until the uncertainty rises again. A pseudocode of DISaM full test-time system operation is provided in Appendix A.3 with a discussion of the hyperparameter δ. 
experimental evaluation: 
 We evaluate our method on five distinct tasks in both simulation and the real world. The Cooking (Fig. 4a ) task involves preparing and serving a side dish, with decisions based on the diner's preference, the clock, and the serving area. In Walls (Fig. 4b ), a pick-and-place task, the block to be picked is identified by a hidden matching block, while the placement region is determined by another block on the right. The Assembly (Fig. 4c ) task requires the IR agent to assemble nuts on pegs according to instructions found in a drawer. In Button (Fig. 4d ), the IS agent turns on We first train the IR agent on data collected in these environments using a behavior cloning objective.In the sim environments, the IR agent's action space consists of skills, and the IS agent's action space consists of camera pan/tilt actions as well as navigation actions in Walls and manipulation skills in Assembly. In real world, we use transformer-based visuomotor policies [76] as the IR policies and collect demonstrations using TeleMoMa [77] . Using the trained IR policies, we train the IS policy as described in Sec. 4.2. To improve the real world training time, we opt for a mixed sim and real training setup described in Appendix A.6. The trained IS policy is then deployed in the environment to seek the correct information and provide it to the IR policy. The control between IR and IS is exchanged based on the test time protocol described in Sec. 4.3. We compare our method with the following baselines (more details in Appendix A.7),• DISaM (reward): An ablation of our method which uses task rewards to train the IS agent instead of our proposed intrinsic reward. From initial experimentation we found that sparse rewards are too difficult to learn from, so we provide hand designed rewards at the end of certain key stages.• Full RL: A reinforcement learning baseline that jointly optimizes the IR and IS agents using the task reward with PPO [78] . We use stage rewards as described above to train this policy as well.• Random Cam: A test time algorithm where the IS agent performs random walks (see Appendix A.7) and uses a pretrained encoder E ϕ along with IR policy's uncertainty (similar to our test time protocol) to determine when to hand control back to the IR policy.• Sampled Context: Providing IR with randomly sampled context vector from the set of contexts.In our experimental evaluation, we aim to answer the following questions.1) Can our proposed framework effectively learn to seek, infer, and exploit contextual information?In Fig. 5 we summarize the results of our experiments and observe that DISaM significantly outperforms the baselines in all simulation tasks. Specifically, in longer horizon tasks that involve multiple stages -Cooking, Walls and Teatime -the baselines struggle whereas DISaM is able achieve high success rates. We provide the stage-wise evaluation results in Appendix A.5. Qualitatively, we observe that Full RL uses the IS policy infrequently, leading to inadequate context recovery and suboptimal performance. Random Cam demonstrates reasonable performance but struggles with large action spaces and requires more time to obtain relevant observations. In contrast, DISaM, by separating information-seeking and manipulation, enables efficient context recovery and improved task completion. We further discuss DISaM's failure modes and robustness to noisy observations in Appendix A.9 and A.10 respectively.2) How important is it to optimize the behavior cloning loss in DISaM's objective instead of sparse task reward? We ablate DISaM by reinforcing the IS policy with the true stage rewards instead of our proposed intrinsic reward, shown by DISaM (task reward) in Fig. 5 . While learning directly from the task reward is challenging due to reward's sparsity, resulting in low success rates, we can see that DISaM's performance is significantly boosted by the use of intrinsic reward. The evaluations are performed across three seeds with 50 rollouts each in sim environment and 1 seed with 10 rollouts in the real world tasks. Across all 5 tasks, DISaM significantly outperforms the baselines.3) Can DISaM handle different representations of context to define the task? We compare DISaM with a variant, DISaM (language), which uses CLIP embeddings of sentences as context vectors, rather than a onehot encoding. This is challenging since learning a mapping from observations to language embeddings is difficult for the observation encoder. We compare between the two in the Cooking and Walls task as shown in Fig. 5 . In Cooking, surprisingly we observe only a small performance drop. Walls, however, is more challenging due to the IS agent needing to move the camera in order to see around the walls, and a modest drop in performance is observed. This agent still outperforms the baselines, suggesting that using language as context is possible in DISaM. We provide further details about the architecture and language instructions in Appendix A.4. 
limitations: 
 While our proposed framework provides a foundation for addressing complex information-seeking tasks, there are several parts of DISaM that we could improve in the future. First, DISaM does not employ observation history or memory in the IS agent, which could enable more complex information-gathering behaviors, especially the ones incorporating long-horizon temporal dependencies or perceiving not only static images but movements and behaviors of other agents. Second, while our fCMDP formalism does not require it, for optimal performance DISaM requires that there is minimal overlap between the action spaces of information-seeking and informationreceiving agents. Otherwise one agent could counteract the other. This assumption prevents more subtle interactions between information seeking and acting, for example in mobile manipulators where motion of the base moves the onboard cameras and the end-effectors at the same time, and is a critical avenue for future work. Third, even though DISaM uses intrinsic reward based on the action discrepancy loss to enhance the density of rewards, learning long horizon information-seeking behavior is still challenging, for instance DISaM requires long training times for long horizon tasks such as Walls (sim). We expect that using some form of exploration bonus can significantly improve the sample efficiency during training. We plan to address the above challenges of DISaM in future work to handle more sophisticated problems in robotics that involve complex manipulation skills and information-seeking behaviors. 
conclusion: 
 We presented DISaM, a novel dual-policy solution for manipulation tasks with information-seeking behaviors. DISaM provides a solution with two policies for the two subproblems in factorized contextual MDP: 1) finding the context and 2) manipulating based on a given context. We exploit the factorization by training both policies consecutively, using simple IL for the manipulation policy that then use it to provide rewards to train the information-seeking policy. We demonstrated empirically the potential of DISaM's dual solution to solve complex multi-stage problems alternating between information-seeking and manipulating phases based on the uncertainty of the next action, both in simulation and the real-world. We believe our proposed fCMDP formalism can be applied to other problems with partial observability requiring active perception (with other sensor modalities, with more agents); we plan to explore that next. 
 
 Appendix: As introduced in Sec. 3, we model our problem as a contextual Markov Decision Process (CMDP) and propose a novel factorization of it into two subproblems that can be addressed with two collaborative policies trained separately: an information-seeking (IS) policy, π IS , that searches and provides context to a second policy, an information-receiving (IR) policy, π IR , that consumes the context and takes reward-seeking (manipulation) actions based on it. In the factorized CMDP (fCMDP), we assume that the trajectories generated by the optimal information-receiving policy, [a * (t = 0), . . . , a * (T )], will achieve maximum return in the CMDP, R c , due to the values in some of the dimensions of the action vector, independently of the values in others, and that these actions can be inferred from a subspace of the observation space. Similarly, we assume that the trajectories generated by an optimal information-seeking policy will reveal the true context because of the values in some of the dimensions of the action vector, independently of the values in others, and that this true context can be inferred from a subset of the observation space. We factorize the action and observation spaces of each agent in the fCMDP based on what action dimensions are necessary to control to achieve the task reward vs. to reveal information, such that:π IS : O IS → A IS and π IR : O IR × C → A IR , with O = O IS ∪ O IR .Therefore, only IS actions can lead to IS observations with information to infer the right context, and IR actions can change the state toward the overall task goal. The context can be inferred by the IS agent by mapping its observation(s) into a context with a given or learned function, f c : O IS → C. However, for this function to map to the right context (i.e. the one that leads the IR policy to accumulate the highest return), the IS policy needs to take the right actions that reveal an information-rich observation. There is no constraint in the overlap between action and observation spaces of both policies but our method performs best in cases where there is little or no overlap.This factorized CMDP (fCMDP) matches a natural factorization into agents with different action and observation spaces, e.g., when the IS agent controls the head of a humanoid and the IR agent controls the arm and manipulates the environment, or when the IS agent is a navigating agent scouting the environment for an IR agent that waits for the contextual information to act in front of a table.Alg. 1 includes the pseudo-code for training DISaM's Information Seeking (IS) agent. Specifically, DISaM trains the IS agent by iterating between the IS policy optimization loop and the encoder optimization loop. The policy optimization uses on-policy data whereas the encoder optimization utilizes data sampled from a replay buffer aggregated during IS policy training. The IR agent takes action if and only if it can reconstruct the true IR actions based on the information c IS provided by the IS agent.Alg. 2 includes the pseudo-code of the DISaM's deployment solution. During deployment, DISaM relies only on environmental observations to make decisions (no oracle or ground truth); the uncertainty of the IR policy is compared against a hyperparameter, δ, to determine when to query the IS agent or when to execute IR's actions (see Fig. 3 ). To compute the uncertainty, DISaM samples n contexts {c i t } n i=1 from the ensemble of trained encoders (see Sec. 4.3), E ϕ , and conditions the IR agent on each of them to generate n action distributions, {π IR (o, c i IS )} n i=1 . DISaM then computes the average KL-divergence between each pair of action distributions as a measure of uncertainty. Due to the difference in the scale of the KL-divergence, the threshold δ needs to be adapted to each action space. However, we found our method relatively robust to this parameter: we simply use δ = 0.5 for all skill-based tasks (discrete action space), and δ = 1e5 for visuomotor tasks (continuous action space), working well across different tasks. ψ =argmin ψ EncoderLoss(c IS , c GT )▷ Optimize E ψ using gradient descent 30: end procedure In several of our tasks in simulation, the context is specified using a language instruction that specifies the goal for the task. Each task stage is specified with a different instruction; the set of possible language instructions for an entire task results from the Cartesian product of possible instruction for each stage. Thus, the language instructions used in the Cooking (sim) task include {"Lift up the bread", "Grasp the meat"}×{"Cook for a short amount of time", "Cook until it is welldone"}×{"Place the pot on the red region", "Put the pot on the green area"}. The instructions used in the Walls (sim) task include {"Pick up the blue cube", "Lift the wooden cube"}×{"Place the cube on the red region", "Put the cube on the green area"}. When the environment is initialized we select the sentences that correspond to the correct instructions, concatenate them to form a single sentence, and process them with CLIP [79] to generate a language feature that acts as contextual information about the goal of the task.Since language embeddings are continuous vectors, we model them as Gaussian mixture models using Mixture Density Networks (MDNs) [80] as the prediction head for E ϕ and use the negative log-likelihood loss to train E ϕ . This is different from how we model and train the encoder when contexts are one-hot vectors. In that case, while using the same backbone network of E ϕ as language Algorithm 2 DISaM Deployment {c i t } n i=1 ← E ψ (o IS ) 5: A dist ← {π IR (o IR , c i IS )} n i=1 6:Uncertainty u ← PairwiseKL(A dist ) until episode done 15: end procedure contexts, we replace the MDN head with a categorical distribution and train E ϕ using the crossentropy loss.In our experiments, three tasks consist of multiple stages leading to several phases of rising uncertainty, exploration with IS, and decreasing uncertainty, which are dynamically handled by our deployment procedure (Appendix A.3). Specifically, a stage may arise when the entire context cannot be reconstructed using a single information-seeking observation, o IS , triggering a phase in the dual policy execution where the IS policy searches and provides the IR agent with the partial context necessary to complete the current stage before moving to the next one. For example, in the Cooking (sim) task, the side dish to prepare, the cooking time, and the serving location are determined by the IS policy looking over to different locations: the dinner, the clock, and the serving region respectively; each of these parts is denoted a "stage" of the task. In Walls (sim), the stages are picking and placing, with the block to pick determined by an identical block placed behind the wall on the left and the placing region determined by the color of the block on the right. Finally, in the Teatime (real) task, the agent must look at the time to determine the appropriate drink, then the person to determine the serving location. Note that knowledge of these stages is not required by DISaM, which automatically infers the necessary information to collect at each stage through the intrinsic reward of minimizing the IR policy loss.We evaluated DISaM, ablations, and baselines on each stage in isolation in addition to the overall task performance reported in the main text. Fig. 6 shows the success rates for each method for each stage of the multi-stage task as well as the overall success rate. In the simulation tasks, we performed 50 policy rollouts per seed and calculated the success rate of each stage as (# of stage successes)/(# times stage reached) to avoid accumulating in the results the effect of the previous stages. In the real environment, 10 rollouts were performed by resetting the environment to the initial state for each stage and evaluating the agent. The methods that receive task reward information (DISaM (task reward) and Full RL) use a reward provided at the end of each stage rather than the sparse task reward.DISaM (language) and DISaM (task reward) are generally competitive in the individual stages for Cooking (sim), but the stage-wise failure rates compound resulting in lower overall success. On the more challenging Walls (sim) and Teatime (real), the variants are less competitive in each stage. Across all tasks, the baselines Full RL, Random Cam, and Sampled Context generally perform poorly in all stages, with the exception of Random Cam succeeding in some of the simpler stages (e.g., Cooking (sim) stage A, Walls (sim) stage B). Full RL achieves consistent success Performing reinforcement learning training procedures in the real world is challenging. Among the known challenges [81] , two are significantly problematic: safety of the agent and the environment, and the need for constant resets (usually by a human). In our real-world training procedure we propose solutions for these two challenges, leading to an autonomous and safe training process: we developed a mixed sim-real training setup for training the IS policy and observation encoder using DISaM in real-world tasks.Firstly, we create a sim version of the real-world task and collect data in both environments to train a corresponding IR policy. For IS policy training (see Fig. 2 and Alg. 1), we train the IS policy and encoder in the real world and generate the real-world context vectors but we exploit the simulator to interpret the context vector with the IR policy and provide rewards to the real IS policy. This eliminates the need to roll out the real IR policy, minimizing the resets and human supervision required during training. Once the IS policy is trained, we can then deploy it with the real IR policy following the system depicted in Fig. 3 and described in Alg. 2.When the domains of the training and testing IR policy are different, we cannot use the IR observations to inform the IS policy decisions. One limitation arising from this is that we cannot do tasks requiring multiple stages: the IS policy cannot utilize the IR observations to learn which information is being queried. Hence, in the Teatime (real) task, we separately train and evaluate the IS and IR policy in the two separate stages of the task, but consider them as a single task with two stages in our evaluations as they execute consecutively. Another issue that results from training the IS policy in the real world is the constant need to update the information in the environment that represents context, e.g., the time on the clock or the person sitting next to the table. Since in this work, we focus on the visual cues to represent the context, we choose to portray this information on screens, which allows us to programmatically update the context altering easily the environment whenever the visual cue needs to be updated (as described in Appendix A.11).While in the simulation we use separate cameras for the IR and IS agents, in the real-world tasks both agents share the same camera, mounted on the head of a PAL Tiago++ [82] . We enable this by choosing a resting head position for the IR agent and resetting back to it whenever we switch to the IR policy deployment phase from the IS policy deployment phase. This is analogous to installing a second camera stationary pointing at the manipulation area. Additionally, during the IS deployment phase, we store the latest observation from the IR deployment phase and use it to calculate the uncertainty of the IR policy.Below we provide more information and implementation details of the baselines, mode is rare since the observation encoder requires significantly less training samples than the IS policy to achieve adequate performance.• Failure of the IR policy: Even when the IS policy and the observation encoder have correctly identified the visual cue, the IR policy may fail during execution. This failure mode is typically observed in the real world setting where data collection is expensive and we could not collect sufficient data for the IR policy to always be successful when given the correct context.We visualize the distribution of failures of DISaM on the three multi-stage tasks Cooking, Walls and Teatime task in Fig. 7 . We note that IR policy failure is a common failure in the real world due to high data collection costs leading to limited availability of training data.In simulation, once the policies have been fully trained (e.g., the IS policy), the full DISaM dual system performs with very high performance, close to 100% success. As a proxy for the real world, we performed an additional experiment by adding varying levels of Gaussian noise to the observations of both the IS and the IR policy, representing sensor noise. We present results for the the 3 simulation tasks with varying levels of noise in Fig. 8 where noise scale represents the standard deviation of the zero centered Gaussian noise.We observe that DISaM remains robust with low levels of noise and that after sufficient increase, the most significant errors are caused by failure to encode the images into the correct context since the noisy images are out-of-distribution for the encoder and make the image encoders overly confident about specific contexts, thus misleading the IR policy to take incorrect actions.In Fig. 9 we visualize rollouts in all tasks. In the following we describe the observation and action spaces of the IR and IS agents in each task (summarized in Table 1 ).Walls (sim): Shown in Fig. 9a , the IR agent is required to pick a block and place it on one of the serving regions. The color of the block to pick is determined by an identical block placed behind a wall on the table on the left and the color of the block on the table on the right corresponds to the correct serving region. Here the IR agent is the robot hand that uses the images from the eye-in-hand camera along with it's proprioceptives as observations and outputs a skill ID to choose between pick, 


Decision: Strengths:
- A challenging and important problem is addressed. 
- The paper is clearly written, well-describing the main ideas.
- The proposed idea seems to be technically sound, relatively well-supported by evaluations. 

Weaknesses:
- More extenstive analysis of the results would be recommended. 
- Comparison with more diverse class of techniques could be possible. 
- Limitations are not well-addressed. 
- Novelty needs to be better clarified.

Reviews: review: title: title: Review of the Paper on Dual Policy Manipulation with Information-Seeking Behavior
paper_summary: paper_summary: This paper introduces a novel framework, DISaM, which is designed for robotic manipulation tasks requiring information-seeking behavior. It addresses the challenges posed by the complexity inherent in Partially Observable Markov Decision Processes (POMDPs) and Reinforcement Learning (RL) by breaking down the tasks into two sub-problems within a Factorized Contextual Markov Decision Processes (fCMDP) framework. The framework trains an information-seeking policy using Reinforcement Learning and an information-receiving policy using imitation learning. The innovative use of intrinsic rewards based on the discrepancy between predictive and actual actions significantly enhances the training effectiveness, particularly in complex robotics scenarios involving information gathering.
main_review: main_review: **Strengths**

1. The paper successfully identifies a practical subset of POMDP challenges and introduces an effective dual-policy approach within the fCMDP framework, specifically designed for tasks that involve active information gathering.
2. The introduction of intrinsic rewards that measure discrepancies between actions within different contexts is a clever strategy that enhances the density of rewards, facilitating more efficient RL training for the information-seeking policy.
3. The empirical evaluation includes a variety of robotic scenarios, convincingly demonstrating the robustness and effectiveness of the proposed method across both simulated and real-world environments.

**Weaknesses**

1. The paper lacks a thorough discussion on the generalizability of the approach to different environments, object variations, and task complexities, which raises questions about its practical deployment in diverse settings.
2. There is an absence of discussion regarding failure cases, which are crucial for understanding the limitations and potential areas for improvement of the proposed methods.
3. The assumption that the context is fully observable based solely on immediate observations, as discussed in the conclusion, is somewhat restrictive and may not hold in more complex tasks.
4. The requirement for specifying a base policy through expert demonstrations for the information-receiving policy could limit the method's applicability where expert data is scarce or hard to obtain.
questions: questions: 1. It would be beneficial to provide a clearer definition and examples of 'context' as used in the framework, as its current description in the introduction can be confusing without concrete examples.
2. Further clarification on the method's generalizability to different object locations, shapes, numbers, and environmental conditions is necessary to assess its broader applicability.
3. The paper should address the lack of discussion on failure cases to provide a more comprehensive understanding of the method’s limitations and reliability.
4. Insights into how this dual-policy approach could be scaled or adapted to handle multiple tasks simultaneously would be valuable for its potential implementation in complex robotic systems.
limitations: limitations: 1
review_summary: review_summary: I recommend a weak accept of this paper. However, this recommendation could shift to a strong accept contingent upon the authors' responses to the concerns raised in the rebuttal phase.
title: title: A factored approach to exploration and task execution
paper_summary: paper_summary: This paper presents a technique for solving manipulation tasks which involve exploration. The technique involves a factorized policy for information seeking and information receiving, where the information seeking policy is trained using the information receiving policy. The approach is validated on both simulated and real-world manipulation scenarios which require exploration.
main_review: main_review: Overall, the paper is well-motivated and the methods are thoroughly presented. The experiments include real-robot tasks and demonstrate the method works. More thorough analysis of the results and limitations to better understand the methods would improve the quality of the paper.

**Strengths:**

* The methodology is well-explained overall and describes an interesting approach to a highly relevant problem in robotics of active exploration.
* The results show significant improvement against baselines and ablations, including on the real robot.

**Points of feedback:**

The paper would benefit from a more thorough analysis of the results. While the success rate improves with the proposed method, it would be helpful to better visualize what factors impact this, e.g. through showing what the causes of the failures in the proposed method and baselines are. Given the factored nature of the problem, it would be interesting to show success rate for each policy stage. 

The limitations should be expanded, ideally independent to the conclusion, to better explain the failure modes of the method. Notably, the experiments display fairly simple skills with a small number of objects. It would be useful to discuss how this method might generalize to more complex tasks or scenes.

The specific tasks defined are not particularly convincing from a robotics context. For example, checking the time, setting timers, or downloading recipes are trivial to do with an internet connection, without the robot needing to look at a clock or to watch TV in real time. This does not detract from the technical contributions, as it is understandable that experiments are contrived to demonstrate the methods, but the simplicity of the skills, primarily in the information seeking portion, suggest limited generalization capability. The points would be more clearly supported by more realistic scenarios. For future work, the authors may consider collaborative human-robot tasks, e.g. checking which stage of the recipe a cook is on to determine which tool or ingredient to bring, or object search.

Some details of the formulation (Section 3) could use additional clarification. Specifically:
* Is the CMDP defined over discrete or continuous space? 
* The context variable could benefit from further explanation since it is a key component of the work. Is it a continuous parameter? A discrete class?
* Does the method require that there is no intersection between $A_{IS}$ and $A_{IR}$? Some tasks could involve overlap between these action spaces, e.g. moving an object out of the way (IS) to pick up an object of relevance (IR) both involve a “pick” action.
* It’s a bit unclear how the “factorized” component of the formulation affects the CMDP formulation described. Are these considered two separate CMDP problems?

*Other points of feedback:*
* Does this technique generalize to more than two factorized policies? For example, for chaining more complex tasks?
* The language task representation is an interesting approach but is underexplored and under-defined in the paper. The authors may consider removing it as it seems out of scope of the contributions of the paper and could be more thoroughly explored as future work.
questions: questions: The following points are the major suggestions to be addressed in the revised manuscript:

1. What are the failure modes of the method? What are the success rates of each policy stage individually?
2. Further discuss the limitations of the method proposed.

Details and other points of feedback are provided above.
limitations: limitations: 1
review_summary: review_summary: The proposed method is well-described, original, and shows improved results on real and simulated experiments. Some lack of analysis and discussions of the limitations, as well as the somewhat simplistic nature of the experiments, cast doubt on the potential impact as currently written.
title: title: Promising work around active information gathering, but some concerns regarding novelty and assumptions
paper_summary: paper_summary: This paper leverages factored, contextual markov decision processes for accomplishing manipulation tasks that consist of information gathering and execution steps. The authors key insight is that, for some tasks, it is possible to explicitly partition the action space into completely separate "information seeking" and "information receiving" stages, thus reducing the overall problem complexity and horizon length. They use imitation learning to train the receiving part, and a cross-entropy-based distance metric to train encoders for the information seeking part.   Experiments are done across multiple simulated and real-world robot tasks including basic assembly and cooking "pick-and-place"-esque tasks. Their results show their method achieves higher success compared to baselines of a full unfactored PPO method, and a random walk information seeking method.
main_review: main_review: This paper addresses significant emerging problems within robotic manipulation - namely tasks in which the robot is responsible for both information gathering about its context, and executing on the task at the same time. The authors do a good job laying out the computational drawbacks of existing CMDP methods which attempt to solve for a single un-factored policy.

In terms of clarity, the paper was overall easy to follow. The figures did a good job illustrating both experimental task setup and overall learning pipelines. One minor issue I had was with the problem formulation in section 3. Key details about the formulation are sequestered to the supplementary material. I found this a little odd, as my understanding is that the theoretical setup and assumptions of the fCMDP is one of the key contributions of the paper. Perhaps the few extra details could be moved to the main text? (Looking at the appendix, I see that there are only two paragraphs anyway, and much of these paragraphs repeat information from the main text).

On comparison to related work - I think the paper could be improved by comparison to a slightly wider range of techniques. Despite the paper being named "Dual Policy Manipulation..." I was surprised to find no references in the related literature and baselines to any "Dual Control" techniques --- A popular paradigm within model-predictive-control for balancing information seeking behaviour with execution. If this is not a fair comparison, the authors should note why not somewhere. Additionally, there have been rich developments in TAMP and symbolic-planning communities which solve exactly these kinds of problems (long-horizon "kitchen"-type problems under partial information) e.g.,

C. R. Garrett, C. Paxton, T. Lozano-Pérez, L. P. Kaelbling and D. Fox, "Online Replanning in Belief Space for Partially Observable Task and Motion Problems," 2020 IEEE International Conference on Robotics and Automation (ICRA)

These would seem like the natural baselines for comparison, rather than just raw PPO.

My main concern however is on novelty, and the reasonableness of the assumptions made in the task setup. For instance they assume they can train the IR policies "with access to the ground truth context". However, after the IS policy is trained, my understanding is that the learned policy is then evaluated on exactly the same task. If we are assuming access to the ground truth of the eval task during training, why bother?

Further, the authors state in their limitations that DISaM "does not employ observation history or memory in the IS agent" and assumes "a hard partitioning of action space into information seeking and information receiving actions". They state they intend to build on these limitations in future work...but to me, this seems to be a bit of a Catch-22 --- the assumptions of a hard partition and ignorance of observation history are the *core assumptions* necessary for them to formulate their key contribution (factoring the problem into IS and IR). I am left wondering whether the tasks the solve are realistic analogies of real-world manipulation tasks, and a bit unclear on significance of the contribution.
questions: questions: - What do you think of the comparison of this method to TAMP-like methods, and MPC-"Dual Control" methods?
- Have I misunderstood something about the training assumptions? Why is it reasonable to assume access to ground truth context during training, then test the policy on an identical task?
- Do you see the fundamental assumptions of your fCMDP formulation (e.g., hard-partitioning of IS and IR) as reasonably applying to realistic real-world scenarios? Further, if you intend to lift these assumptions in future work, do you think it can be done in a way that incrementally builds on this framework? Or would the entire formulation have to be rethought?
limitations: limitations: 2
review_summary: review_summary: A potentially strong formulation of fCMDPs with solid experiments, but I am uncertain about the significance of the technical contribution, and about some of the assumptions made to get it to work.