Learning to Look: Seeking Information for Decision Making via Policy Factorization
Authors: Shivin Dass
Jiaheng Hu
Ben Abbatematteo
Peter Stone
Roberto Martín-Martín
Venue: CoRL 2024 (2024)
Keywords: Active Vision
Manipulation
Interactive Perception
TL;DR: We study the problem of active perception in the manipulation domain and identify a subset of tasks within it as factorized CMDPs. We then propose DISaM, a dual-policy solution to enable information-seeking behavior in learnt agents.

Abstract: Many robot manipulation tasks require active or interactive exploration behavior in order to be performed successfully. Such tasks are ubiquitous in embodied domains, where agents must actively search for the information necessary for each stage of a task, e.g., moving the head of the robot to find information relevant to manipulation, or in multi-robot domains, where one scout robot may search for the information that another robot needs to make informed decisions. We identify these tasks with a new type of problem, factorized Contextual Markov Decision Processes, and propose DISaM, a dual-policy solution composed of an information-seeking policy that explores the environment to find the relevant contextual information and an information-receiving policy that exploits the context to achieve the manipulation goal. This factorization allows us to train both policies separately, using the information-receiving one to provide reward to train the information-seeking policy. At test time, the dual agent balances exploration and exploitation based on the uncertainty the manipulation policy has on what the next best action is. We demonstrate the capabilities of our dual policy solution in five manipulation tasks that require information-seeking behaviors, both in simulation and in the real-world, where DISaM significantly outperforms existing methods. More information at https://robin-lab.cs.utexas.edu/learning2look/.

Learning to Look W Seeking Information for Decision Making via Policy Factorization
Shivin Dass', Jiaheng Hu', Ben Abbatematteo Peter Stone' : Roberto Martin-Martin'
'The University of Texas at Austin, 'Sony AI (sdass, jiahengh, abba, pstone, robertomm) (ateroommeuttteeeeee @utexas edu
Abstract: Many robot tasks require active or interactive exploration behavior order to be performed successfully. Such tasks are ubiquitous in embodied do- mains, where agents must actively search for the information necessary for each stage of task, e.g., moving the head of the robot to find information relevant to manipulation, or in multi-robot domains, where one scout robot may search for the information that another robot needs to make informed decisions. We identify these tasks with a new type of problem, factorized Contextual Markov Decision Processes, and propose DISaM, dual-policy solution composed of an information-seeking policy that explores the environment find the relevant con- textual information and an informatiion-rcceviin policy that exploits the context to achieve the manipulation goal. This factorization allows to train both poli- cies separately, using the information-receiviei one to provide reward to train the information-seeking policy. At test time, the dual agent balances exploration and exploitation based on the uncertainty the manipulation policy has on what the next best action is. We demonstrate the capabilities of our dual policy solution in five manipulation tasks that require information seeking behaviors, both in simulation and in the real-world, where DISaM significantly outperforms existing methods. More information at
Keywords: Active Vision, Manipulation, Interactive Perception
Introduction
Intelligent decisions can only be made based on the right information. When operating in the envi- ronment, an intelligent agent actively seeks the information that enables select the right actions and proceeds with the task only when it is confident enough. For example, when following video recipe, chef would look at the TV to obtain information about the next ingredient to grasp, and later look at a timer to decide when to turn off the stove. In contrast, current learning robots assume that the information needed for manipulation readily available their sensor signals (e.g., from a sta- tionary camera looking at a tabletop manipulation setting) or rely on given www-dimensional state representation predefined by a human (e.g., object pose) that also to provide the means for the robot to perceive it. this work, our goal is endow robots with the capabilities learn to perform iiformation-seekinn actions find the information that enables manipulation, using supervision the quality of the informed actions and switching between active perception and manipulation only based on the uncertainty about what manipulation action should come next.
Performing actions to reveal information has been previously explored in the subfields of active and interactive perception. active perception [1. 2, 3], an agent changes the parameters of its sensors (e.g., camera pose [4, 6] parameters [7. 8, 9]) infer information such object pose, shape, material. Interactive perception [10] solutions go one step further and enable agents to change the state of the environment to create information-rich signals to perceive kinematics [11, 12), ma-
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

IS
12 25
π
20
Figure 1: DISaM for tasks with information-seeking behavior. To make the right decision in task (e.g., what beverage to pick or in what dining set to place it), a robot may need to seek task- relevant information (the time of day to decide the beverage or the person at the table to choose where place it). We formalize such infornn tasks as factorized contextual MDPs and solve them with dual policy collaborative approach where an information-secking policy (T" takes active perception actions to search for the right contextual information, and information- receiving policy (xr (i) consumes this retrieved context to select the right manipulation actions.
terial [13], or other properties [14, 15, 16, 17]. However, these solutions fully factorize the problem into perception and subsequent task-oriented manipulation, creating the need for a human to specify what information to gather. Belief space planning approaches [18, 19, 20] jointly reason about how actions of an agent reduce uncertainty, and how this, in turn, leads to better actuation. However, correctly predicting the information that future actions may reveal requires accurate forward and ob- servation models that are generally not available, leading to plans that grow quickly in complexity with the number of actions and the intricacy of the state and sensor spaces. Instead of planning, other solutions to Partially Observable Markov Decision Processes (POMDPs) [21] rely on reinforcement learning to train a unified agent with memory that first finds information and then acts based on it. However, finding such long sequences of actions prohibitively challenging, especially with sparse task reward, thus limiting their applicability to simple settings [22, 23, 24].
Our main insight is to formulate these manipulation tasks with information-seekee Contextual Markov Decision Processes (CMDP) [25] and solve them with two separate collaborative policies, one controlling an information-seeking (IS) agent and the other an information--rceevingg agent (IR). We present DISaM, a Dual informationnSeeking and Manipulation policy solution for CMDPs, where an IS agent acts determine the context, while the IR agent acts based on it to accomplish overall manipulation task. The context represents the information that is unknown to the agent priori and has to be discovered by the IS to enable IR to take the right subsequent actions. For instance, the context could be a task label represented either language instruction. In DISaM, both agents can share the same embodiment, as in our cooking example above, or have separate embodiments, for instance when one agent is scout and the other an operator awaiting information, but both cases, our policy factorization matches a natural factorization of the agents based on different action and/or observation spaces. Two main benefits result from factorizing into collaborative IS and policies to solve the CMDP. First, we assume that the correct context is given, may render the IR-learning problem fully observable and solve it with simpler solution like imitation learning (II.). Second, the factorization transforms the original long-horizon problem into two shorter-horizon ones, providing an opportunity to exploit additional supervision at the interface; in DISaM, we use the IR policy to provide intrinsic reward for the IS policy, making the IS -Saleennn problem a simple POMDP. These design decisions, on top of making the training process more efficient and tractable, make DISaM compatible with variety of existing recent IL models in roboties [26. 27, 28, 29]. Finally, at test time, DISaM can simply balance between the cearmattionseekinng and the manipulation behaviors (IS and IR policies) based on the uncertainty the IR policy has about the next action to take, using ensemble architecture.
behavior as
one-hot vector

---

Our contributions include 1) a general formalism for tasks with information-seekingg behavior as factorized CMDPs (fCMDPs), and 2) our proposed dual-policy solution for fCMDPs, DISaM. We carry out multiple experiments in simulation and on real robots, where we demonstrate the capabili- ties of DISaM to enable robots (single or multi-robot systems) to seek and use the right information for manipulation. Additionally we balance between exploring for context and exploiting it, even with multiple stages that require several seeking/acting behavior switching phases. Our proposed approach substantially outperforms the baselines in 3 simulation and 2 real world tasks.
Related Work
POMDPs and CMDPs. POMDPs provide a principled framework for decision making under par- tial observability. They have been extensively studied in the robotics literature, with applications including localization, navigation, manipulation, and human-robot interaction [18, 30. 31]. The intractability of exactly solving POMDPs is well-established [32, 33), necessitating approxima- tion algorithms [34, 35, 36]. Scaling POMDP planning to real-world settings with image obser- vations remains a challenge [30, 31, 37, 38], and the assumption of known dynamics and observa- tion models remains prohibitively restrictive in open world settings. Belicf-space task-and and-motion planning methods similarly require substantial engineering of predicates, skills, and perception rou- tines [39, 40]. Applying model-free RL with some notion of memory (e.g. recurrent neural net- works) has achieved some success in partially-observable problems [22, 23, 24], though these meth- ods generally struggle in long-horizon, hard exploration, sparse reward settings like finding and exploiting task-oriented information in manipulation.
Contextual MDPs (CMDPs) [25] or Latent MDPs [41] have been proposed as a special class of POMDPs in which the reward and transition dynamics are determined by (typically unobserved) context, effectively parameterizing family of MDPs. This model captures the case in which la- tent variables change slowly over time
constant within episode, e.g., serving a user web application [25]. Theory about these problems has been developed in multi-task [42, 43, 44], model- based [45], continual [46], and inverse RL settings [47], with applications ranging from medical de- cision making [48] to human-robo interaction [49]. We build on this framework to develop a CMDP model of active perception which one agent seeks information to determine the latent context and a second acts to maximize the context dependant reward. MPC dual-control methods [50, 51] follow similar motivation, simultaneously optimizing for system identification and a control objective, but require knowledge of the dynamics of the system and designation of the uncertain parameters.
Active and Interactive Perception. Active perception [1, 2, 3] refers to the ability of an agent to intelligently seek out informative observations. Example applications include next- best planning for 3D reconstruction [4, 5, 6] and active perception for grasping [52, 53, 54]. When the agent is also equipped with information-gathering manipulation capabilities, the problem is referred to as interactive perception [10], which has been studied in the context of object segmentation [14, 55, 16], articulation model estimation [11, 56, 12], material property estimation [13], grasping [15, 57, 58), scene understanding [17], and beyond. While these approaches demonstrate the necessity and complexity of informationn-sekknng behaviors, they typically define a priori the information that must be recovered from the scene. In contrast, our goal is to enable agents to discover what needs to be inferred through reinforcement.
Active perception has also been considered in the reinforcement learning setting [59, 60, 61, 62, 63). Similar to our factorization, Liu et al. [62] decouple exploration and exploitation in a meta-Ri RI. set- ting but assume that a single exploration phase is sufficient for gathering all task relevant informa- tion. Some works learn to crop observations to select salient visual inputs [64, 65, 66]. Recent work has developed RL methods capable of jointly learning manipulation behaviors with active vision to achieve clear view of the manipulator [67, 68, 69, 70, 71, 72, 73]. For example, Shang and Ryoo [71] propose jointly training manipulation and sensory policies by developing an intrinsic reward based action prediction, encouraging the camera view the manipulator. Similarly, Göransson [72] train active perception policies to predict sensor observations of the manipulator. These works
3

---

generally fail to enable sophisticated information gathering behaviors and instead focus on view- point selection to facilitate short-horizon manipulation tasks. In contrast, we seek to enable agents to search for task-relevant information, decoupling information seeking from manipulation.
3 Problem Formulation: factorized Contextual Markov Decision Processes
We formulate the problem of solving a manipulation task with information- seeking behavior as a Contextual Markov Decision Process (CMDP), a special form of MDP where the dynamics and the reward depend on hidden contextual variable [25, 45, 47]. CMDP is represented by the tuple C' is the set of contexts, the set of actions, E O is the set of observations, and M(()) is function that maps the context to an MDP (c) (S, A, O,O, Re: Pc), where T S x A x S - [0, 1] is the transition function, R. S x A + R is an overall task-reward function, and Pe is the initial state distribution, all dependent on the context. The goal is to find the policy, Te S + A, that maximizes the expected total discounted sum of rewards, E. I Re(, ac) so].
Mc S, A, O, Mc)), where
S is the set of states, a E A is
Naively training single policy to solve the CMDP is challenging and usually requires complex iterative procedure [25, 74]. Instead, we make the important observation that in many practical CMDP problems, including robotic manipulation, the action space of the original agent can be naturally factored into two parts, A = Ais U AIR, where one part of the action space Ays is only useful for context seeking (e.g. a moving camera) while the other part AiR is sufficient to complete the task given the right context (e.g- a robot manipulator). Given this assumption, we propose to factorize the CMDP into factorized CMDP (ICMDP) with two sub-problems that can be addressed with two collaborative policies trained separately: an information-seeking (IS) policy, n1S O1s Ars, that searches and provides context information cıs C to second policy, an information-recceving (IR) policy, π IR OnR C - Ain that consumes the context and takes reward-seeking actions based on it. Specifically, for each IS observation Oıs, the information is generated by given or learned function, fe Ois - C. As different information may be required across the span of a task, these policies would operate iteratively, where π 18 continuously searches for information that allows x IR make the right decision. We provide further details about the formulation in Appendix A.I.
Our proposed factorization of the CMDP problem provides new opportunities that our proposed method, DISaM, exploits. In the following we explain how DISaM trains a solution for the proposed fCMDP, focusing especially on the less explored problem of how to train collaborative information- seeking policies.
DISaM, Dual Information-Seeking and Manipulation Policies for fCMDP
In DISaM, we model a partially-observe manipulation problem as an ICMDP, and devise a solution to train a policy for each of the factorized subproblems: an coommation-seeiing policy, x and an information--reeivingg policy, IR Additionally, we do not assume that Se, the function mapping observations from the IS policy into context for the IR policy, is given; instead, we learn this mapping from IS observations into context, Eg O1s C. DISaM utilizes novel optimization procedure that first trains the IR agent on the original task with the reward of the fCMDP assuming access to the true context, and then trains the information-ssekkng agent to infer the context for the IR agent. To that end, it minimizes a loss based on the difference between the IR policy conditioned on the IS IS-provided context vs. ground truth context
In the following, we first explain how the IR agent is instantiated (Sec. 4.1); then, we discuss novel iterative optimization procedure that utilizes intrinsic rewards generated by the IR agent to train the information-sseeking agent (Sec. 4.2); and finally, we provide details on how the uncertainty
"In general, actions by the IR policy based on the ground truth context may not lead to the highest task return (e.g.. if the IR policy is far from optimal): in those cases, it would be better to directly optimize IS based on task reward, However, assume that IR policy is good approximation of the optimal policy once trained.

---

Phase 1
Phase 2
(o Car agd
Octimize superved lomning
Oys
O
dg
IS Pollcy
[H Hollyy
lnooder
L
Envronment
Ground Irum Contod
Hollyy
t OyR
Egt
BC
Encocerr
Step 2.1: Iterative Encoder Training
IR Policy training via mitation learning
Preckcted
Contool
Step 2.2: Itorative Training of IS Policy
Figure 2: Two learning stages of DISaM. In Phase I, we learn the informatiio-receevin policy Ir that takes in ground-truth context information and controls the movement of the robot. In Phase 2, learn an information-seeking policy TTi well as image encoder Es such that the context can be correctly reconstructed from the camera observation. Once all parts are trained, together they create system that takes in image observations and controls both the robot and the camera.
of the IR policy is used in DISaM to determine, in deployment, when the collaborative dual agent needs more information to complete the current task, querying the information seeking agent, or when it is ready to pursue the task objective via the agent (Sec. 4.3).
4.1 Cntermatiio-Receivvinn Agent
The IR policy takes the current observation and controls the IR agent to interact with the environ- ment, conditioned on the context. We assume that only the actions of the IR agent can achieve the original manipulation goal. In DISaM, we exploit the factorization in the ICMPD and assume we can train the IR policy with access to the ground truth context: π IR Oi x C - A₁R- this case, IR agent is essentially operating a standard MDP and can be optimized relatively easily using either imitation learning or reinforcement learning. In this work we opt for imitation learning and collect dataset of expert human demonstrations, D (TI, T2...Tn). that lead the manipulation goal, labeled with ground truth context, While in general CMDPs, the context determines the reward, transitions, and initial state distribution, in our experiments we focus on context-dependent rewards. Thus, the context provides information about the goal of the task, including relevant sub- goals. In principle, this context can be represented in any format as long as it can express multiple goals; we primarily consider task IDs, but also explore using natural language instructions as the context. Given the dataset of expert demonstrations with ground truth context, we train IR with behavior eloning (Fig. 2, Phase 1). The observation space for the IR agent in our experiments, OIR E OIR, consists of images of the robot's workspace acquired by a stationary camera, well as robot proprioception. Other forms of training the IR policy (e.g., with reinforcement learning) may be possible; we leave the exploration of these alternatives to future work. Our simple IL solution to train the information-reccivingg policy aligns with multiple recent works [26, 27, 28, 29] that provide trained goal-conditioned policies conditioned on task labels (IDs or language).
4.2 Information-Seeking Agent
Once IR policy, -. IR is trained, DISaM uses train information-ssekinng agent that includes the IS policy, T/S. and encoder mapping observations of the IS agent into context for the IR policy, Eg. The factorization in the fCMDP assumes that the actions from the IS agent alone are not able to directly achieve the original manipulation goal, rather they are sufficient only for determining the context required by the IR agent to solve the task.
The observations for the information-sseeking agent in our experiments, 0is E Ors, consist of im- ages from the camera controlled by IS policy, the camera's 'ss pose, and, in the of interactive IS agent, proprioceptive signals from robot arm. The observation encoder, Fgg O - C, then
5

---

converts the observations from the IS agent into contexts to be passed to the IR agent. In cases where the IS agent may need to collect different context information depending on the IR agent's s state, we augment the observations of the IS agent with observations from the IR agent, O's = Ois U Onn-
A possible objective for the information-seeking agent is maximize the original reward of the ICMDP, i.e., the original reward of the manipulation task that the IR policy optimizes. However, the task reward may be sparse, posing significant challenges to the training process of the IS agent. To address this, DISaM leverages a key insight: we can approximately optimize for the task reward by assuming that the IR policy makes the right decisions provided with the right context. By com- paring the actions taken by the IR policy based on the IS IS-provided context, aiR ~ T cis), with the IR policy actions based on the ground truth context (assumed to be provided during training), AGT π _2 (o, CGT), we can train the IS policy seek and encode the context information that minimizes the action differences at each time step. Hence, the objective for IS agent is:
arg min Ei C(T 18 (OIR, Eg (0(1s)) π IR (01R, CGT)]. 8,6
(I)
DISaM uses the distance L between the two action distributions (c.g., cross entropy) to define reward for the IS policy Th. given as R1s = a - B L( (o, (1,)),), π 2 (o, cGT))), with hyperparameters and B. This objective is denser than sparse task rewards and can be optimized efficiently with reinforcement learning (Fig. 2, Step 2.2). On the other hand, the ground truth context CGr also enables training the encoder, Egg via supervised learning (Fig. 2. Step 2.1).
We optimize these two objectives iteratively, alternating between training Eg while keeping = fixed, and vice versa, as visualized in Fig. 2 and summarized in Algorithm 1 in Appendix A.2.
4.3 Deciding between Information-Secking and Information- Actions at Test Time
While the sections above describe the training objectives of the IS agent and the agent, we still need to specify a way to decide when query the information-seeking agent and when to hand control back to the IR agent during deploy- ment. As shown in Fig. DISaM uses the uncertainty of the IR agent to determine when to act and when to seek new in- formation. Specifically, DISaM maintains an ensemble of en- coders ). At each decision- decision-making step, DISaM sam- ples n contexts {et )=1) from these encoders and conditions the IR agent on each of them to generate n action distributions, (T (o, e DISaM then computes the average KL- mea-
Follyy
a/s
Pheckoted Contedds
encodle
Ii Polkyy
dj
1 1"
Endroomment
Unicetsiity rowe Itae
4,
dys
divergence between each pair of action distributions sure of uncertainty [75]. If the value of uncertainty is higher than some threshold, 6, DISaM assumes that the IR agent does not have sufficient context information to generate reliable ac- tions, and it queries the intormation-seeking agent to seek fur- ther context information. When the uncertainty drops below 6, it indicates that the IR policy is confident about the next manipulation action to take, and DISaM ecutes the IR policy until the uncertainty rises again. A pscudocode of DISaM full test-time system operation is provided in Appendix A.3 with a discussion of the hyperparameter 8.
Figure 3: Deployment of DISaM. When uncertainty over IR's next action is low, DISaM follows the IR actions; when the uncertainty is high, DISaM follows IS policy.
5 Experimental Evaluation
We evaluate our method on five distinct tasks in both simulation and the real world. The Cooking (Fig. 4a) task involves preparing and serving a side dish, with decisions based on the diner's preference, the clock, and the serving area. Walls (Fig. 4b), pick-and-place task, the block to be picked is identified by hidden matching block, while the placement region is deter- mined by another block on the right. The Assembly (Fig. 4c) task requires the IR agent assemble nuts on pegs according to instructions found in drawer. In Button (Fig. 4d), the IS agent turns on

---

(a) Cooking
(b) Walls
(c) Assembly
(d) Button
(c) Teatime
Figure 4: Tasks in our evaluation of DISaM. We evaluated DISaM on simulation tasks Cooking, Walls, Assembly and two real-world tasks with the Tiago robot Button, and Teatime. These tasks each require different information-ann -aattrrinnnntt strategies, and demonstrate the sophisticated active and interactive information gathering capabilities of DISaM.
a TV by pressing a red button, and the IR agent serves the next ingredient based on what appears on the screen. Finally, in Teatime (Fig. 4c), the IS agent checks the time, selects the appropriate drink, and serves it to the person seated at the table. Further task details are provided in Appendix A.
We first train the IR agent on data collected in these environments using behavior cloning objective. In the sim environments, the IR agent's action space consists of skills, and the IS agent's 's action space consists of camera pan/tilt actions well as navigation actions in Walls and manipulation skills in Assembly. In real world, we use transformer- collect demonstrations using TeleMoMa [77]. Using the trained IR policies, we train the IS policy as described Sec. 4.2. To improve the real world training time, we opt for a mixed sim and real training setup described in Appendix A.6. The trained IS policy is then deployed the environment to seek the correct information and provide exchanged based on the test time protocol described in Sec. 4.3. We compare our method with the following baselines (more details in Appendix A.7),
visuomotor policies [76] as the IR policies and
the policy. The control between IR and IS
DISaM (reward): : An ablation of method which uses task rewards to train the IS agent instead of our proposed intrinsic reward. From initial experimentation found that sparse rewards too difficult to learn from, so we provide hand designed rewards at the end of certain key stages.
Full RL: A reinforcement learning baseline that jointly optimizes the IR and IS agents using the task reward with PPO [78]. We stage rewards as described above to train this policy well.
Random Cam: A test time algorithm where the IS agent performs random walks (see Ap- pendix A.7) and uses pretrained encoder Egg along with policy's uncertainty (similar to our test time protocol) to determine when hand control back to the IR policy.
Sampled Context: Providing IR with randomly sampled context vector from the set of contexts.
In our experimental evaluation, we aim to answer the following questions.
1) Can our proposed framework effectively learn seek, infer, and exploit contextual information? In Fig. 5 we summarize the results of our experiments and observe that DISaM significantly outper- forms the baselines in all simulation tasks. Specifically, in longer horizon tasks that involve multiple stages Cooking, Walls and Teatime - the baselines struggle whereas DISaM is able achieve high success rates. We provide the stage-wise evaluation results in Appendix A.5. Qualitatively, we observe that Full RL uses the IS policy infrequently, leading to inadequate context recovery and suboptimal performance. Random Cam demonstrates reasonable performance but struggles with large action spaces and requires more time to obtain relevant observations. In contrast, DISaM, by separating information-seeking and manipulation, enables efficient context recovery and improved task completion. We further discuss DISaM's failure modes and robustness to noisy observations in Appendix A.9 and A.10 respectively.
2) How important to optimize the behavior cloning loss in DISaM's objective instead of sparse task reward? We ablate DISaM by reinforcing the policy with the true stage rewards instead of our proposed intrinsic reward, shown by DISaM (task reward) in Fig. While learning directly from the task reward is challenging due to reward's sparsity, resulting in low success rates, can that DISaM's performance is significantly boosted by the use of intrinsic reward.
7

---

1.0
Figure 5: Evaluation results. The evaluations are performed across three seeds with 50 roll- outs each in sim environment and seed with 10 rollouts in the real world tasks. Across all 5 tasks, DISaM significantly outperforms the baselines.
Success Rate
0.8
0.6-
L Teatime
0.4
0
Cooking (sim)
Walls (sim)
Assembly (sim)
Button
DISaM
DISaM (task reward)
Random Cam Sempled Context
DISaM (language)
Full RL
3) Can DISaM handle different representations of context define the task? We compare DISaM with a variant, DISaM (language), which uses CLIP embeddings of sentences as context vectors, rather than a onehot encoding. This is challenging since learning mapping from observations to language embeddings difficult for the observation encoder. We compare between the two in the Cooking and Walls task as shown in Fig. 5. Cooking, surprisingly we observe only small performance drop. Walls, however, is more challenging due to the IS agent needing to move the camera in order to see around the walls, and modest drop in performance is observed. This agent still outperforms the baselines, suggesting that using language context is possible in DISaM. We provide further details about the architecture and language instructions in Appendix A.4.
6 Limitations
While our proposed framework provides foundation for addressing complex ieformationnseekingg tasks, there are several parts of DISaM that we could improve in the future. First, DISaM does not employ observation history or memory in the IS agent, which could enable more complex cnormattiongatheriinn behaviors, especially the ones incorporating long-horizon temporal depen- dencies or perceiving not only static images but movements and behaviors of other agents. Sec- ond, while our fCMDP formalism does require it, for optimal performance DISaM requires that there is minimal overlap between the action spaces of information-ssekknng and information- receiving agents. Otherwise one agent could counteract the other. This assumption prevents more subtle interactions between information seeking and acting, for example in mobile manipulators where motion of the base moves the onboard cameras and the end-effectors at the same time, and is a critical avenue for future work. Third, even though DISaM uses intrinsic reward based on the ac- tion discrepancy loss enhance the density of rewards, learning long horizon behavior still challenging, for instance DISaM requires long training times for long horizon tasks such Walls (sim). We expect that using some form of exploration bonus can significantly improve the sample efficiency during training. We plan to address the above challenges of DISaM in future work to handle more sophisticated problems in robotics that involve complex manipulation skills and information-sseking behaviors.
ceermattion-sekinng
7 Conclusion
We presented DISaM, a novel dual-policy solution for manipulation tasks with information-seekinn behaviors. DISaM provides a solution with two policies for the two subproblems in factorized contextual MDP: 1) finding the context and 2) manipulating based on a given context. We exploit the factorization by training both policies consecutively, using simple IL for the manipulation policy that then use it to provide rewards train the information-seekingg policy. We demonstrated empirically the potential of DISaM's dual solution to solve complex multi- problems alternating between information--seking and manipulating phases based on the uncertainty of the next action, both in simulation and the real-world. We believe our proposed fCMDP formalism can be applied to other problems with partial observability requiring active perception (with other sensor modalities, with more agents); we plan to explore that next.
8

---

Acknowledgments
We thank Arpit Bahety and Ruchira Ray for their feedback on the manuseript. This work took place at the Robot Interactive Intelligence Lab (RobIn) at UT Austin. Robin is supported in part by DARPA TIAMAT program (HR0011-24-4 1-24-9-0428), A part of this work also took place at the Learning Agents Research Group (LARG) at UT Austin. LARG research is supported in part by NSF (FAIN- 2019844, NRT-2125858), ONR (000014-1--2430 ARO (W91 (W911NF-33--0004)) Lockheed Martin, and UT Austin's Good Systems grand challenge. Peter Stone serves as the Executive Director of Sony AI America and receives financial compensation for this work. The terms of this arrangement have been reviewed and approved by the University of Texas Austin in accordance with its policy objectivity in research.
References
[1] R. Bajesy, Active perception. Proceedings of the IEEE, 76(8):966-1055, 1988.
[2] Aloimonos, I. Weiss, and A. Bandyopadhyay, Active vision. International journal of com- puter vision, 1:333-356, 1988.
[3] R. Bajesy, Y. Aloimonos, and J. K. Tsotsos. Revisiting active perception. Autonomous Robots, 42:177-196, 2018.
[4] R. Pito. A solution to the next best view problem for automated surface acquisition. IEEE Transactions on pattern analysis and machine intelligence, 11110)1016-10300 1999.
[5] W. R. Scott, G. Roth, and J.-F. Rivest. View planning for automated three-dimensional object reconstruction and inspection. ACM Computing Surveys (CSUR), 35(1):64-995 96, 2003.
[6] Delmerico, S. Isler, R. Sabzevari, and D. Scaramuzza. A comparison of volumetric infor-
mation gain metrics for active 3d object reconstruction. Autonomous Robots, 42(2):197-208, 2018.
[7] J. M. Tenenbaum. Accommodation in computer vision. Stanford University, 1971.
[8] T. D. Garvey. Perceptual strategies for purposive vision. 1976.
[9] K. Pahlavan and J.-O, Eklundh. A head-eye system-analysii and design. CVGIP: Image Understanding, 56(1):41-56, 1992.
[10] J. Bohg, K. Hausman, B. Sankaran, O. Brock, D. Kragic, S. Schaal, and G. S. Sukhatme. Inter- active perception: Leveraging action in perception and perception in action. IEEE Transactions on Robotics,
2017.
[11] D. Katz and O. Brock. Manipulating articulated objects with interactive perception. 2008 IEEE International Conference on Robotics and Automation, pages 272-277. IEEE, 2008
[12] R. Martin-Martín and O. Brock. Coupled recursive estimation for online interactive perception of articulated objects. The International Journal of Robotics Research, 41(8):741-777, 2022.
[13] J. Sinapov, T. Bergquist, C. Schenck, U. Ohiri, S. Griffith, and A. Stoytchev. Interactive object recognition using proprioceptive and auditory feedback. The International Journal of Robotics Research, 30(10):1250-1262 2011.
[14] G. Metta and Fitzpatrick. Early integration of vision and manipulation. Adaptive behavior, 11(2):109-12 2003.
[15] O. B. Kroemer, R. Detry, J. Piater, and J. Peters. Combining active learning and reactive control for robot grasping. Robotics and Autonomous systems, 58(9):1105-11 111011166,,,11111111111111111111 2010.

---

[16] S. Y. Gadre, K. Ehsani, and S. Song. Act the part: Learning interaction strategies for articulated object part discovery. In Proceedings of the JEEE/CVF International Conference on Computer Vision, pages 15752-15761, 2021.
[17] H. Jiang, B. Huang, R. Wu, Z. Li, S. Garg, H. Nayyeri, S. Wang, and Y. Li. Roboexp: Action- conditioned scene graph via interactive exploration for robotic manipulation. arXiv preprint arXiv:2402. 15487, 2024.
[18] L. P. Kaelbling, M. L. Littman, and A. R. Cassandra. Planning and acting in partially observ- able stochastic domains. Artificial intelligence, 101(1-2):9--34, 1998.
[19] R. Platt Jr, R. Tedrake, L. P. Kaelbling, and T. Lozano-Perez. Belief space planning assuming maximum likelihood observations. In Robotics: Science and Systems, volume 2, 2010.
[20] L.. P. Kaelbling and T. Lozano-Pérez. Integrated task and motion planning in belief space. The International Journal of Robotics Research, (2(9-10): 1000::199-111100000 1224-122,,20 2013.
[21] G. E. Monahan. State of the art-a survey of partially observable markov decision processes:
theory, models, and algorithms. Management science, 28(1):1-16, 1982.
[22] P. Mirowski, R. Pascanu, F. Viola, H. Soyer, A. J. Ballard, A. Banino, M. Denil, R. Goroshin, L. Sifre, K. Kavukcuoglu, et al. Learning to navigate in complex environments. arXiv preprint arXiv: 1.00611.063,.. 2016.
[23] V. Mnih, A. P. Badia, M. Mirza, A. Graves, T. Lillicrap, T. Harley, D. Silver, and K. Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International conference on machine learning, pages 1928-1937. PMLR, 2016.
[24] P. Anderson, Q. Wu, D. Teney, J. Bruce, M. Johnson, N. Sünderhauf, I. Reid, S. Gould, and A. Van Den Hengel. Vision-and Vision-and-language navigation: Interpreting "suuall-ggrunnde naviga- tion instructions in real environments. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 3674-3683, 2018.
[25] A. Hallak, D. Di Castro, and S. Mannor. Contextual markov decision processes. arXiv preprint arXiv: 1502.0229, 2015.
[26] A. Brohan, N. Brown, Carbajal, Y. Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Haus- man, A. Herzog, J. Hsu, et al. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint 1rX
2022.
[27] A. Brohan, N. Brown, Carbajal, Y. Chebotar, X. Chen, K. Choromanski, T. Ding, D. Driess, A. Dubey, C. Finn, al. Rt-2: Vision-languagg-action models transfer web knowledge to robotic control. arXiv preprint arXhv:2307.15818, 2023.
[28] A. Padalkar, A. Pooley, A. Jain, A. Bewley A. Herzog, A. Irpan, A. Khazatsky, A. Rai, A. Singh, A. Brohan, et al. Open x-embodiment: Robotic learning datasets and rt-x mod- els. arXiv preprint 1rX0v:2310008864, 2023.
[29] O. M. Team, D. Ghosh, H. Walke, K. Pertsch, K. Black, O. Mees, S. Dasari, J. Hejna, Kreiman, C. Xu, et al. Octo: An open-source generalist robot policy. arXiv preprint arXiv:2405.12213 2024.
[30] H. Kurniawati. Partially observable markov decision processes and robotics. Annual Review of Control, Robotics, and Autonomous Systems, 5:253-277, 2022.
[31] M. Lauri, D. Hsu, and J. Pajarinen. Partially observable markov decision processes in robotics:
A survey. IEEE Transactions on Robotics, 39(1) 2022.
[32] R. D. Smallwood and E. J. Sondik. The optimal control of partially observable markov pro- cesses over finite horizon. Operations research, 21(5): 1-0880881188 1973.
10

---

[33] M. L. Litiman. Memoryless policies: Theoretical limitations and practical results. 1994.
[34] T. Jaakkola, S. Singh, and M. Jordan. Reinforcement learning algorithm for partially observ- able markov decision problems. Advances in neural information processing systems, 7, 1994.
[35] Pineau, G. Gordon, and Thrun. Anytime point-based upproximations for large pomdps. Journal of Artificial Intelligence Research, 27:335-380, 2006.
[36] A. Ng and M. I. Jordan. Pegasus: A policy search method for large mdps and pomdps. arXiv preprint arXiv: 1tt1:1300..87 2013.
[37] K. Zheng, Y. Sung. G. Konidaris, and S. Tellex. Multi-resolution pomdp planning for multi- object search in 3d. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 2029. IEEE, 2021.
[38] T. Nguyen, V. Hrosinkov, E. Rosen, and S. Tellex. Language-conditioned observation models for visual object search. In 2023 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 10894-10901. IEEE, 2023.
[39] L. P. Kaelbling and T. Lozano-Pérez. Hierarchical task and motion planning in the now. In 2011 IEEE International Conference on Robotics and Automation, pages 1470-1477, IEEE, 2011.
[40] C. R. Garrett, C. Paxton, T. Lozano-Pérez, L. P. Kaelbling, and D. Fox. Online replanning in belief space for partially observable task and motion problems. 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 5678-5684. IEEE, 2020.
[41] J. Kwon, Y. Efroni, C. Caramanis, and S. Mannor. RI for latent mdps: Regret guarantees and a lower bound. Advances in Neural Information Processing Systems, 34:24523 22534,, 2021.
[42] M. E. Taylor and P. Stone. Transfer learning for reinforcement learning domains: A survey. Journal of Machine Learning Research, 10(7), 2009.
[43] E. Brunskill and L. Li. Sample complexity of multi reinforcement learning. arXiv preprint arXiv: 1309.681.. 2013.
[44] S. Sodhani, A. Zhang, and J. Pineau. Multi-task reinforcement learning with context-based representations. In International Conference on Machine Learning, pages 9767-9779. PMLR, 2021.
[45] W. Sun, N. Jiang, A. Krishnamurthy, A. Agarwal, and J. Langford. Model-based rl in con- textual decision processes: Pac bounds and exponential improvements over model-free ap- proaches. In Conference learning theory, pages 2898-2933. PMLR, 2019.
[46] S. Sodhani, F. Meier, Pineau, and A. Zhang, Block contextual mdps for continual learning. In Learning for Dynamics and Control Conference, pages 608-623 PMLR, 2022.
[47] S. Belogolovsky, Korsunsky, S. Mannor, C. Tessler, and T. Zahavy. Inverse reinforcement learning in contextual mdps. Machine Learning. 110(9):2295-2334 2021.
[48] L. N. Steimle, D. L. Kaufman, and B. Denton. Multi-model markov decision processes. IISE Transactions, 53(10): 124-1139, 2021.
[49] X. Chen, S. Zhang, P. Zhang, L. Zhao, and J. Chen. Asking before action: Gather information in embodied decision making with language models. arXiv preprint arXiv:2305. 15695, 2023.
[50] T. A. N. Heirung, B. E. Ydstie, and B. Foss. Towards dual mpc. IFAC Proceedings Volumes. 45(17):502-507, 2012.
[51] T. A. N. Heirung, B. E. Ydstie, and B. Foss. An mpe approach to dual control. IFAC Proceed- ings Volumes, 46(32):69-74, 2013.

---

[52] M. Breyer, L. Ott, R. Siegwart, and J. J. Chung. Closed-loop next-best-view planning for target-driven grasping. In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 1411-1416. IEEE, 2022.
[53] D. Morrison, Corke, and J. Leitner. Multi-view picking: Next-best-view reaching for im- proved grasping in clutter. In 2019 International Conference on Robotics and Automation (ICRA), pages 3762-8768. IEEE, 2019.
[54] S. Jauhri, S. Lueth, and G. Chalvatzaki. setiee-pereeetive motion generation for mobile ma- nipulation. arXiv preprint 1.Xiv:231000043333 2023.
[55] R. Martin-Martin, S. Höfer, and O. Brock. An integrated approach to visual perception of articulated objects. In 20/6 IEEE international conference on robotics and automation (ICRA), pages 5091-5097. IEEE, 2016.
[56] J. Sturm, K. Konolige, C. Stachniss, and W. Burgard. Vision-based detection for learning articulation models of cabinet doors and drawers in household environments. In 2010 IEEE International Conference on Robotics and Automation, pages 362-368. IEEE, 2010.
[57] K. Hsiao, P. Nangeroni, M. Huber, A. Saxena, and A. Y. Ng. Reactive grasping using optical proximity sensors. In 2009 IEEE International Conference Roboties and Automation, pages 2098-2105. IEEE, 2009.
[58] M. C. Koval, N. S. Pollard, and S. Srinivasa. Pre-and post-contact policy decomposition for planar contact manipulation under uncertainty. The International Journal of Robotics Re- search, 35(1-3):244-264, 2016.
[59] S. D. Whitehead and D. H. Ballard. Active perception and reinforcement learning. In Machine Learning Proceedings 1990, pages 179-188. Elsevier, 1990.
[60] S. D. Whitehead and D. H. Ballard. Learning to perceive and act by trial and error Machine Learning, 7:45 83, 1991.
[61] C. Tang, B. Abbatematteo, J. Hu, R. Chandra, R. Martín-Martin, and P. Stone. Deep reinforcement learning for robotics: A survey of wollwwrld successes. arXiv preprint arXiv:2408.0352 035539 2024.
[62] E. Z. Liu, A. Raghunathan, P. Liang, and C. Finn. Decoupling exploration and exploitation for meta-reinforcement learning without sacrifices. In International conference on machine learning, pages 1925-6935. PMLR, 2021.
[63] A. Xie, L. M. Bhamidipaty, E. Z. Liu, J. Hong, S. Levine, and C. Finn. Learning explore in pomdps with informational rewards. In Forty-first International Conference Machine Learning.
[64] James and A. J. Davison. Q-attention: Enabling efficient learning for vision-based robotic manipulation. IEEE Robotics and Automation Letters, 7(2):1612-1619, 2022.
[65] S. James, K. Wada, T. Laidlow, and A. Davison. Coarse-to-fine q-attention: Efficient learning for visual robotic manipulation via discretisation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13739-13748, 2022.
[66] Y.-L. Tai, Y. C. Chiu, Y.W. Chao, and Y.-T. Chen. Scone: A food scooping robot learning framework with active perception. In Conference on Robot Learning, pages 865- PMLR, 2023.
[67] R. Cheng, Agarwal, and K. Fragkiadaki. Reinforcement learning of active vision for manip- ulating objects under occlusions. In Conference Robot Learning, pages 422 431. PMLR, 2018.
12

---

[68] Y. Zaky, G. Paruthi. B. Tripp, and J. Bergstra. Active perception and representation for robotic manipulation. arXiv preprint arXiv:2003.06334, 2020.
[69] R. Jangir, N. Hansen, S. Ghosal, M. Jain, and X. Wang. Look closer: Bridging egocentric and third-person views with transformers for robotic manipulation. IEEE Robotics and Automation Letters, 7(2):3046-308 2022.
[70] Lv, Y. Feng, C. Zhang, S. Zhao, L. Shao, and Lu. Sam-rl: Sensing-aware model-based reinforcement learning via differentiable thysics-based simulation and rendering. In Robotics science and systems, 2023.
[71] Shang and M. S. Ryoo. Active vision reinforcement learning under limited visual observ- ability. Advances in Neural Information Processing Systems, 36, 2024.
[72] R. Göransson. Deep Reinforcement Learning with Active Vision on Atari Environments. PhD thesis, Master's Thesis. Lund University, 2022.
[73] M. K. Grimes, J. V. Modayil, P. W. Mirowski, D. Rao, and R. Hadsell. Learning to look by self-prediction Transactions on Machine Learning Research, 2022.
[74] S. Rezaei-Shoshtari, Morissette, F. Hogan, G. Dudek, and D. Meger. Hypernetworks for zero-shot transfer in reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 9579-9999 9579-9587, 2023.
[75] K. Menda, K. Driggs-Campbell and M. J. Kochenderfer. Ensembledagger. A bayesian ap- proach safe imitation learning. In 2019 IEEE/SS International Conference on Intelligent Robots and Systems (IROS), pages 5041-5048. IEEE, 2019.
[76] B. Liu, Y. Zhu, C. Gao, Y. Feng, Q. Liu, Y. Zhu, and Stone. Libero: Benchmarking knowl- edge transfer for lifelong robot learning. Advances in Neural Information Processing Systems, 36, 2024.
[77] S. Dass, W. Ai, Y. Jiang, S. Singh, J. Hu, R. Zhang, P. Stone, B. Abbatematteo, and R. Martin- Martin. Telemoma: A modular and versatile teleoperation system for mobile manipulation. arXiv preprint arXiv:2403.07866 2024.
[78] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1707.06377 2017.
[79] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, Agarwal, G. Sastry, Askell, Mishkin, J. Clark, G. Krueger, and I. Sutskever. Learning transferable visual models from natural language supervision. CoRR, abs/2103.00022 2021. URL https://arxiv.or
abs/2103. 00020.
[80] C. M. Bishop. Mixture density networks. 1994.
[81] H. Zhu, J. Yu, A. Gupta, D. Shah, K. Hartikainen, A. Singh, V. Kumar, and S. Levine, The ingredients of real-world robotic reinforcement learning. arXiv preprint arXiv:2004. 125700 2020.
[82] J. Pages, L. Marchionni, and F. Ferro. Tiago: the modular robot that adapts to different research needs. In International workshop on robot modularity, IROS, volume 290, 2016.
13

---

A Appendix
A.1 Factorized Contextual Markov Decision Process
As introduced in Sec. 3, we model our problem as a contextual Markov Decision Process (CMDP) and propose a novel factorization of it into two subproblems that can be addressed with two col- laborative policies trained separately: an information--xeeking (IS) policy, wis, that searches and provides context to second policy, an information-receivvn (IR) policy, ıT, that consumes the context and takes reward-seeking (manipulation) actions based on it. In the factorized CMDP (fCMDP), assume that the trajectories generated by the optimal informattion-receivin policy, (a"(t = ())= o). (T)), will achieve maximum return in the CMDP, R., due to the values in some of the dimensions of the action vector, independently of the values in others, and that these actions can be inferred from a subspace of the observation space. Similarly, we assume that the trajecto- ries generated by an optimal information-se values in some of the dimensions of the action vector, independently of the values in others, and that this true context can be inferred from subset of the observation space. We factorize the action and observation spaces of each agent in the fCMDP based what action dimensions are necessary to control to achieve the task reward vs. to reveal information, such that: n O Ais and n : OIR x C + AIR, with O O1s U U Therefore, only IS actions can lead to IS obser- vations with information to infer the right context, and IR actions can change the state toward the overall task goal. The context can be inferred by the IS agent by mapping its observation(s) into a context with a given or learned function, fo Ois + C. However, for this function to map to the right context (i.e. the one that leads the IR policy to accumulate the highest return), the IS policy needs to take the right actions that reveal information-rice observation. There is no constraint in the overlap between action and observation spaces of both policies but our method performs best in cases where there is little or no overlap.
policy will reveal the true context because of the
This factorized CMDP (ICMDP) matches a natural factorization into agents with different action and observation spaces, e.g., when the IS agent controls the head of a humanoid and the IR agent controls the arm and manipulates the environment, when the IS agent is navigating agent scouting the environment for IR agent that waits for the contextual information act in front of table.
A.2 IS Policy Training Procedure
Alg. I includes the pseudo-code for training DISaM's Information Seeking (IS) agent. Specifically, DISaM trains the IS agent by iterating between the IS policy optimization loop and the encoder optimization loop. The policy optimization uses on-policy data whereas the encoder optimization utilizes data sampled from a replay buffer aggregated during IS policy training. The IR agent takes action if and only if it can reconstruct the true IR actions based on the information eıs provided by the IS agent.
A.3 DISaM Deployment Procedure
Alg. includes the pseudo-code of the DISaM's deployment solution. During deployment, DISaM relies only on environmental observations make decisions (no oracle or ground truth); the uncer- tainty of the IR policy is compared against hyperparameter, 5, determine when query the IS agent or when to execute IR's actions (see Fig. 3). To compute the uncertainty, DISaM samples contexts {c)) from the ensemble of trained encoders (see Sec. 4.3), Eg. and conditions the IR agent on each of them to generate n action distributions, the average KL-divergence between each pair of action distributions as a measure of uncertainty. Due to the difference in the scale of the Kl.-divergence, the threshold 5 needs to be adapted to each action space. However, we found our method relatively robust to this parameter: we simply use 8 0.5 for all skill-based tasks (discrete action space), and 8 1e5 for visuomotor tasks (continuous action space), working well across different tasks.
co. ) DISaM then computes
14

---

Algorithm Iterative Optimization for the Information Seeking Agent
1: Initialize Information-seeking policy =b observation encoder Exg, Encoder Replay Buffer Be, Rollout Buffer B, switch threshold 'T
4: Recieve initial observations 018, 01R ,, 3: Empty B
2: for i in 2... K do
5: for i in 1.2..., K do
6: 019, OIR CollectRollout(oos- 0IR)
7: end for
8: Optimize m on with PPO objective
9: for in 1,2..., Kene do
018, CGT ~ Bg UpdateEncodee((oss.coo 12: end for
Sample batch from replay buffer
11:
13:
end for
14:
15: procedure COLLECTROLLOUTION) 018)
16: Take actions 01s ~ 18 (oıs) and obtain oss; CGT from the environment
17:
CI9 (d's)
18:
===am1x LossFoonnnnnnn Crs; car), -1)
19:
add (o1s, a1s, (ss) to B
20: add (ois. CGT) to Be
while LossFunc(orR CIs CCT) do
22:
Take actions aiR ~ك: (OSR, cGr) and obtain 0iR from the environment end while
23:
24:
return 018. OIR
end procedure
26:
procedure UPDATEENCODERROON CoT)
28:
Crs (ois)
* aargming Encoderl.oss(ers,.crT
D Optimize Erp using gradient descent
30: end procedure
31:
32: procedure LOSSFUNC(O1R cIS, CGT
33:
return Distance( (" Cis), n (OIR, CCT)
34: end procedure
A.4 Using Language as Context
In several of our tasks in simulation, the context is specified using language instruction that speci- fies the goal for the task. Each task stage is specified with different instruction; the set of possible language instructions for entire task results from the Cartesian product of possible instruction for each stage. Thus, the language instructions used in the Cooking (sim) task include "Liift up the bread". "Grasp the meat") x x (Cook for a short amount of time". "Cook until it is well- done") xx"Place the pot on the red region", "Put the pot on the green area"). The instructions used in the Walls (sim) task include ("Pick up the blue cube". "Lift the wooden cube x Place the cube the red region", "Put the cube on the green area" ") When the environment is initialized select the sentences that correspond the correct instructions, concatenate them to form single sentence, and process them with CLIP [79] to generate a language feature that acts as contextual information about the goal of the task.
Since language embeddings are continuous vectors, we model them as Gaussian mixture models using Mixture Density Networks (MDNs) [80] the prediction head for Es and the negative lig-liieldood loss to train Es- This is different from how model and train the encoder when contexts one-hot vectors. In that case, while using the same backbone network of Eg as language
15

---

Algorithm DISaM Deployment
1:
procedure DEPLOYAGENT(T)) IR Exp)
2: Recieve initial observations 018, 01R
3: repeat
4: {ci)i=1 E
5: Adia 4- 11T Gis)
6: Uncertainty PAirwiseKL(Adia)
7:
if < 5 then
8:
Take action aIR Adist Receive observations 01s,
9:
else
11:
Take action dis 2 (01s) Receive observations 018, OnR
13:
end if
14:
until episode done
15:
end procedure
contexts, we replace the MDN head with a categorical distribution and train Es using the cross- entropy loss.
A.5
Stage Wise Success Rates and Analysis
In our experiments, three tasks consist of multiple stages leading several phases of rising un- certainty, exploration with IS, and decreasing uncertainty, which are dynamically handled by our deployment procedure (Appendix A.3). Specifically, stage may arise when the entire context not be reconstructed using single information-seeking observation, 019, triggering a phase in the dual policy execution where the IS policy searches and provides the IR agent with the partial con- text necessary to complete the current stage before moving to the next one. For example, in the Cooking (sim) task, the side dish to prepare, the cooking time, and the serving location are de- termined by the IS policy looking over to different locations: the dinner, the clock, and the serving region respectively; each of these parts is denoted "stage" the task. Walls (sim), the stages picking and placing, with the block pick determined by an identical block placed behind the wall the left and the placing region determined by the color of the block on the right. Finally, in the Teatime (real) task, the agent must look at the time to determine the appropriate drink, then the person to determine the serving location. Note that knowledge of these stages is not required by DISaM, which automatically infers the necessary information to collect at each stage through the intrinsic reward of minimizing the IR policy loss.
We evaluated DISaM, ablations, and baselines each stage in isolation in addition the overall task performance reported in the main text. Fig. 6 shows the success rates for each method for each stage of the multi-stage task as well the overall success rate. In the simulation tasks, we performed 50 policy rollouts per seed and calculated the success rate of each stage as (# of stage successes) ((# times stage reached) to avoid accumulating in the results the effect the previous stages. In the real environment, 10 rollouts were performed by resetting the environment to the initial state for each stage and evaluating the agent. The methods that receive task reward information (DISaM (task reward) and Full RL) use reward provided the end of each stage rather than the sparse task reward.
DISaM (language) and DISaM (task reward) are generally competitive in the individual stages for Cooking (sim), but the stage-wise failure rates compound resulting in lower overall success. On the more challenging Walls (sim) and Teatime (real), the variants are less competitive in each stage. Across all tasks, the baselines Full RL, Random Cam, and Sampled Context generally perform poorly in all stages, with the exception of Random Cam succeeding in some of the simpler stages (e.g., Cooking (sim) stage A, Walls (sim) stage B). Full RL achieves consistent success
16

---

Cooking (slm
Teat (real)
1.0-
Success Raae
0.8-
0.6
0.4
0.2
Overall
Overall
Overall
Stage
Stage
Stage
D15aM
DISaM (lanquage)
DISaM (task reward)
RL
Random Cam
Sampled
Figure 6: Stage-wise success rates. Success rates for each method are plotted for each stage of the multi-stage tasks: Cooking (sim), Walls (sim), and Teatime (real).
only in the first stage of Cooking (sim) and Walls (sim), perhaps due to experiencing these early stages more frequently during training.
A.6 Real World Training leveraging Synced Simulation
Performing reinforcement learning training procedures in the real world is challenging. Among the known challenges [81], two are significantly problematie: safety of the agent and the environment, and the need for constant resets (usually by a human). In our real-world training procedure we propose solutions for these two challenges, leading to an autonomous and safe training process: we developed mixed sim-real training setup for training the IS policy and observation encoder using DISaM in real-world tasks.
Firstly, we create a sim version of the real-world task and collect data in both environments to train a corresponding IR policy. For IS policy training (see Fig. 2 and Alg. 1), we train the IS policy and encoder in the real world and generate the real-world context vectors but we exploit the simulator to interpret the context vector with the IR policy and provide rewards to the real IS policy. This eliminates the need roll out the real IR policy, minimizing the resets and human supervision required during training. Once the IS policy is trained, we can then deploy it with the real IR policy following the system depicted in Fig. 3 and described in Alg. 2.
When the domains of the training and testing IR policy are different, we cannot use the IR observa- tions to inform the IS policy decisions. One limitation arising from this is that we cannot tasks requiring multiple stages: the IS policy cannot utilize the IR observations learn which information is being queried. Hence, in the Teatime (real) task, separately train and evaluate the IS and IR policy in the two separate stages of the task, but consider them single task with two stages in our evaluations as they execute consecutively. Another issue that results from training the IS policy in the real world is the constant need to update the information in the environment that represents context, e.g., the time on the clock or the person sitting next to the table. Since in this work, we focus on the visual cues to represent the context, we choose to portray this information on screens, which allows us to programmatically update the context altering easily the environment whenever the visual cue needs be updated (as described in Appendix A.1
While in the simulation we use separate cameras for the IR and IS agents, in the real-world tasks both agents share the same camera, mounted on the head of PAL Tiago++ [82]. We enable this by choosing a resting head position for the IR agent and resetting back whenever we switch to the IR policy deployment phase from the IS policy deployment phase. This is analogous to installing second camera stationary pointing the manipulation area. Additionally, during the IS deployment phase, store the latest observation from the IR deployment phase and use it to calculate the uncertainty of the IR policy.
A.7 Baselines
Below provide more information and implementation details of the baselines,
17

---

DISaM (reward): An ablation of our method which uses task rewards to train the IS agent instead of our proposed intrinsic reward. This baseline is used to test if high-performance IS policy can be learned just by using task rewards. Instead of using sparse task rewards, which are difficult to learn from, we use hand-designed rewards at the end of key stages. While this stage-wise reward requires some domain knowledge to design, we found Specifically, after completing stage the reward obtained is + 10 in simulation environments and +5 in real-world settings. For all other timesteps, the reward is -0.1. Across all our experiments we use the same hyperparameters and network architectures as we used for DISaM.
be important for the learning of the IS policy.
Full RL: A reinforcement learning baseline that jointly optimizes the IR and IS agents. The policy architecture a shallow convolutional neural network to process image inputs followed by an MLP, that takes observations of both IR and IS agents as inputs and outputs an action over the combined action space of the two agents. The policy is trained using PPO [78] with the same stage-wise rewards as described above in the DISaM (reward) section.
Random Cam: A baseline that replaces the trained IS agent in DISaM with an agent that performs random movements but utilizes the trained encoder Esg to encode the context and hands control back to the IR policy when it's uncertainty lower than the prefixed threshold (similar to our test time protocol). Instead of sampling actions uniformly and randomly at every step, we perform weighted sampling by putting more weight the last action performed, enabling the policy to explore larger distances in the state space. Specifically, with probability 0.5 + "Lactions' the action of the policy stays the same as in the previous timestep and otherwise uniformly randomly sampled from among the remaining actions.
Sampled Context: We sample the context vector from a prior probability distribution over contexts and use it to produce IR actions using an IR policy trained with behavior cloning. In all our settings, the prior distribution consists of a uniform probability distribution over all possible contexts.
A.8 Comparing Visuomotor and Skill-based IR policies
The training procedure between visuomotor and skill-based IR policies is similar except for some minor differences that we note here. Primarily the distance metric L in Equation varies between the the two IR policies. When the IR policy is skill-based, L is the cross-entropy loss between the action distribution induced by the true and the predicted contexts. When the IR policy is visuomotor, we measure the negative log-likelihood of sampled predicted actions under the distribution of true actions. To demonstrate that the performance of DISaM is not affected by the type of IR policy used, we compare DISaM's performance trained with a skill-based DISaM (skill) and visuo- motor DISaM (motor) IR policy the Walls (sim) task. DISaM (skill) achieves an overall success rate of 47.67 + 0.58 and DISaM (motor) achieves an overall success of 47.33 + 0.58, mea- sured over 50 rollouts across 3 seeds. As we can see, the overall performance of the system stays consistent across both forms of the IR policy.
A.9 Failure Modes of DISaM
In the following we discuss the failure modes of DISaM.
Failure to observe the correct visual eue (IS policy failure): A failure mode of the IS policy in which the policy fails to take correct sequence of actions to observe the required visual cue and hence may not able to predict the context correctly. During evaluation, since we terminate the execution of the policy after fixed maximum number of environment steps, this failure mode typically arises in tasks where the sequence of correct actions be taken to observe the visual cue is large such in the simulated Walls task.
Failure to interpret the observation (Image encoder failure): Once the IS policy is looking at the correct visual cue, the observation encoder must encode that it could be properly handled by the IR policy. If the encoder has not been trained sufficiently or has made some spurious correlations with some other parts of the environment then this failure could arise. This failure
18

---

50
47
(a) Cooking Failures
(b) Walls Failures
(c) Teatime Failures
Figure 7: Distribution of failures for the multi-stage tasks (a) Cooking (b) Walls and (c) Teatime.
Pells
Assembly
Cooking
82
Parcontag
24
0.21
0.15
0.02
0.00
3.00
Noise
Success
S Policy Fallure
Image Enoodar Fallure
19 Policy Fallure
Figure 8: The distribution of success and failure of DISaM with varying levels of observation noise. For low sensor noise, DISaM achieves high performance but as the sensor noise increases the failure to map images to correct contexts starts to dominate.
mode is rare since the observation encoder requires significantly less training samples than the IS policy to achieve adequate performance.
Failure of the IR policy: Even when the IS policy and the observation encoder have correctly identified the visual cuc, the IR policy may fail during execution. This failure mode is typically observed in real world setting where data collection expensive and we could not collect sufficient data for the IR policy always be successful when given the correct context.
We visualize the distribution of failures of DISaM the three multi-stage tasks Cooking, Walls and Teatime task in Fig.7. We note that IR policy failure is common failure in the real world due to high data collection costs leading to limited availability of training data.
A.10 Robustness to Noisy Observations
In simulation, once the policies have been fully trained (e.g., the IS policy), the full DISaM dual system performs with very high performance, close to 100% success. As proxy for the real world, we performed an additional experiment by adding varying levels of Gaussian noise to the obser- vations of both the and the IR policy, representing sensor noise. We present results for the the 3 simulation tasks with varying levels of noise in Fig.8 where noise scale represents the standard deviation of the zero centered Gaussian noise.
We observe that DISaM remains robust with low levels of noise and that after sufficient increase, the most significant errors are caused by failure to encode the images into the correct context since the noisy images are out-of-distribution for the encoder and make the image encoders overly confident about specific contexts, thus misleading the IR policy take incorrect actions.
A.11 Tasks
In Fig.9 visualize rollouts in all tasks. In the following describe the observation and action spaces of the IR and IS agents in each task (summarized in Table 1).
Walls (sim): Shown in Fig.9a, the IR agent is required to pick a block and place on one of the serving regions. The color of the block to pick is determined by an identical block placed behind wall the table the left and the color of the block on the table on the right corresponds the correct serving region. Here the IR agent is the robot hand that uses the images from the eye-in-hand camera along with it's proprioceptives as observations and outputs skill ID to choose between pick,
19

---

Cooking
Walls Manipulation Skills
Assembly
Button
Teatime
Manipulation Skills
Manipulation Skills
6D Cartesian
6D Cartesian
IR Actions
Deltas
3D Nav. Deltas
Cam Pan/Tilt and XY Nav.
Cam Pan/Tilt
Cam Pan/Tilt +
IS Actions
Cam Pan/Tilt
Cam Pan/Tilt
Manip. Skills
Manip. Skills
Stages
2
2
Table Task Summary. The action space of the IR and IS agents as well the number of stages for each task in our experiments.
place and go-near actions. The IS policy observes the scene from floating camera with discrete actions that allow the camera to pan left/right, tilt up/down, move wwward/buckwarr/leff//ig. by some fixed amount.
Assembly (sim): Shown in Fig.9b, the IR agent is required to assemble the screw in the correct colored peg. The color of the correct peg is determined by similarly colored block placed inside one of the drawers the table on the right. Here the IR agent is the robot hand that uses the images from the eye-in-han camera along with it's proprioceptives as observations and outputs a skill ID to chooses between pick-place and go-near actions. The IS policy observes the scene from a floating camera and also can control the second robot arm near the drawers to interact with them. The IS policy has a discrete action space that allows to pan and tilt the camera, along with controlling the second robot amm to open the drawers and pick/place the blocks.
Cooking (sim): Shown in Fig.9e, the IR agent is required to cook meal. The proper meal to prepare is determined by the dish being prepared, the cook time depends on the timer and the serving region is determined by the check mark. Here the IR agent is the robot hand that uses the images from the eye-in-hand camera along with it's proprioceptives as observations and outputs a skill to choose between pick-place and go-near actions. The IS policy observes the scene from floating camera with discrete actions that allow the camera to pan left/right, and tilt up/down.
Teatime (real):
Shown in Fig.9c, the IR agent needs to determine the time of day to decide on the beverage to serve and look at the person at the table to choose where to place the beverage. Here the IR agent is Tiago++ robot that uses the images from it's head camera along with it's proprioceptives as observations and outputs a continuous delta action in the Cartesian space to control it's hand. The IS policy observes the scene from Tiago's head camera as well (see Appendix A.6) with discrete actions that allow the camera to pan left/right, and tilt up/down.
Button (real): Shown in Fig.9d, the IR agent is required to place the correct fruit in the bowl depending on the recipe. The IS policy is required to first turn the monitor on by pressing a button and then look at the screen to determine what fruit to use. Here the IR agent a Tiago++ robot's right hand that uses the images from it's head camera along with it's proprioceptives as observations and outputs continuous delta action in the Cartesian space to control the hand. The IS policy observes the scene from Tiago's head camera well (see Appendix A.6) and in addition to controlling the head movement, also controls Tiago's left arm. The IS policy has discrete action space that allows it to pan and tilt the camera, along with controlling the second robot arm to press at various positions on the table.
20

---

Task Messiization
A
Took Visuslzatioo
uuncrtain Ord wicc
fron lo pick
which
téear
View
Ungerbiity palicy
Unaetaniy i Piplcy
(a) DISaM rollouts Walls.
(b) DISaM rollouts Assembly.
uncatinn
oo san
intedd
Frll to iaee
IOODO
Uncottiity
Uncomsiity Polic
of Fi coloy
(c) DISaM rollouts on Teatime.
(d) DISaM rollouts on Button.
Task
IR uncencin what
IR Runcottn how cook
uncer tain where to
IR View
Unportaiitty poliyy
View
(c) DISaM rollouts Cooking.
Figure 9: Here we demonstrate the rollouts of DISaM on the tasks that we study. We show the uncertainty during the rollouts and 1) Highlight IR observations that have high uncertainty, and 2) Sequence IS observations before finding information that reduce uncertainty.
21


Reviews:
[{"review_id": "7XCSDpHx5p", "reviewer": "Reviewer_fHBz", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 5, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper introduces a novel framework, DISaM, which is designed for robotic manipulation tasks requiring information-seeking behavior. It addresses the challenges posed by the complexity inherent in Partially Observable Markov Decision Processes (POMDPs) and Reinforcement Learning (RL) by breaking down the tasks into two sub-problems within a Factorized Contextual Markov Decision Processes (fCMDP) framework. The framework trains an information-seeking policy using Reinforcement Learning and an information-receiving policy using imitation learning. The innovative use of intrinsic rewards based on the discrepancy between predictive and actual actions significantly enhances the training effectiveness, particularly in complex robotics scenarios involving information gathering.", "summary_of_recommendation": "I recommend a weak accept of this paper. However, this recommendation could shift to a strong accept contingent upon the authors' responses to the concerns raised in the rebuttal phase.", "technical_quality": 5, "clarity_of_presentation": 5, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 1, "questions_for_rebuttal": "1. It would be beneficial to provide a clearer definition and examples of 'context' as used in the framework, as its current description in the introduction can be confusing without concrete examples.\n2. Further clarification on the method's generalizability to different object locations, shapes, numbers, and environmental conditions is necessary to assess its broader applicability.\n3. The paper should address the lack of discussion on failure cases to provide a more comprehensive understanding of the method’s limitations and reliability.\n4. Insights into how this dual-policy approach could be scaled or adapted to handle multiple tasks simultaneously would be valuable for its potential implementation in complex robotic systems.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1PeAHNvyco", "reviewer": "Reviewer_ibsv", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper presents a technique for solving manipulation tasks which involve exploration. The technique involves a factorized policy for information seeking and information receiving, where the information seeking policy is trained using the information receiving policy. The approach is validated on both simulated and real-world manipulation scenarios which require exploration.", "summary_of_recommendation": "The proposed method is well-described, original, and shows improved results on real and simulated experiments. Some lack of analysis and discussions of the limitations, as well as the somewhat simplistic nature of the experiments, cast doubt on the potential impact as currently written.", "technical_quality": 4, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 1, "questions_for_rebuttal": "The following points are the major suggestions to be addressed in the revised manuscript:\n\n1. What are the failure modes of the method? What are the success rates of each policy stage individually?\n2. Further discuss the limitations of the method proposed.\n\nDetails and other points of feedback are provided above.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LGzZMaNRZ6", "reviewer": "Reviewer_iUtd", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 2, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper leverages factored, contextual markov decision processes for accomplishing manipulation tasks that consist of information gathering and execution steps. The authors key insight is that, for some tasks, it is possible to explicitly partition the action space into completely separate \"information seeking\" and \"information receiving\" stages, thus reducing the overall problem complexity and horizon length. They use imitation learning to train the receiving part, and a cross-entropy-based distance metric to train encoders for the information seeking part.   Experiments are done across multiple simulated and real-world robot tasks including basic assembly and cooking \"pick-and-place\"-esque tasks. Their results show their method achieves higher success compared to baselines of a full unfactored PPO method, and a random walk information seeking method.", "summary_of_recommendation": "A potentially strong formulation of fCMDPs with solid experiments, but I am uncertain about the significance of the technical contribution, and about some of the assumptions made to get it to work.", "technical_quality": 2, "clarity_of_presentation": 4, "potential_impact": 2, "robotics_focus": 4, "quality_of_limitations": 2, "questions_for_rebuttal": "- What do you think of the comparison of this method to TAMP-like methods, and MPC-\"Dual Control\" methods?\n- Have I misunderstood something about the training assumptions? Why is it reasonable to assume access to ground truth context during training, then test the policy on an identical task?\n- Do you see the fundamental assumptions of your fCMDP formulation (e.g., hard-partitioning of IS and IR) as reasonably applying to realistic real-world scenarios? Further, if you intend to lift these assumptions in future work, do you think it can be done in a way that incrementally builds on this framework? Or would the entire formulation have to be rethought?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Strengths:
- A challenging and important problem is addressed. 
- The paper is clearly written, well-describing the main ideas.
- The proposed idea seems to be technically sound, relatively well-supported by evaluations. 

Weaknesses:
- More extenstive analysis of the results would be recommended. 
- Comparison with more diverse class of techniques could be possible. 
- Limitations are not well-addressed. 
- Novelty needs to be better clarified.
Decision: Accept