The Value of Sensory Information to a Robot
Authors: Arjun Krishna
Edward S. Hu
Dinesh Jayaraman
Venue: ICLR 2025 Poster (2025)
Keywords: robotics
limited sensing
perception
imitation learning
reinforcement learning
planning
Primary area: reinforcement learning
TL;DR: A novel approach to study when and how frequently state-of-the-art robotic policies need to sense the world reveals many interesting insights and untapped efficiencies

Abstract: A decision-making agent, such as a robot, must observe and react to any new task-relevant information that becomes available from its environment. We seek to study a fundamental scientific question: what value does sensory information hold to an agent at various moments in time during the execution of a task? Towards this, we empirically study agents of varying architectures, generated with varying policy synthesis approaches (imitation, RL, model-based control), on diverse robotics tasks. For each robotic agent, we characterize its regret in terms of performance degradation when state observations are withheld from it at various task states for varying lengths of time. We find that sensory information is surprisingly rarely task-critical in many commonly studied task setups. Task characteristics such as stochastic dynamics largely dictate the value of sensory information for a well-trained robot; policy architectures such as planning vs. reactive control generate more nuanced second-order effects. Further, sensing efficiency is curiously correlated with task proficiency: in particular, fully trained high-performing agents are more robust to sensor loss than novice agents early in their training. Overall, our findings characterize the tradeoffs between sensory information and task performance in practical sequential decision making tasks, and pave the way towards the design of more resource-efficient decision-making agents.

Published as a conference paper at ICLR 2025
THE VALUE OF SENSORY INFORMATION TO A ROBOT
Arjun Krishna
Edward S Hu University of Pennsylvania
Dinesh Jayaraman
University of Pennsylvania ar jrasaas.. .ueeessssuueenn.nee.. edu
University of Pennsylvania
ABSTRACT
A decision- maaiiin agent, such as robot, must observe and react to any new task-relevant information that becomes available from its environment. We seek to study fundamental scientific question: what value does sensory information hold to an agent at various moments in time during the execution of task? Towards this, we empirically study agents of varying architectures, generated with varying policy synthesis approaches (imitation, RL, model-based control), on diverse robotics tasks. For each robotic agent, we characterize its regret terms of performance degradation when state observations are withheld from it at various task states for varying lengths of time. We find that sensory information is surprisingly rarely task-critical in many commonly studied task setups. Task characteristics such stochastic dynamics largely dictate the value of sensory information for a well- trained robot; policy architectures such planning vs. reactive control generate more nuanced second-order effects. Further, sensing efficiency is curiously cor- related with task proficiency: in particular, fully trained high-performing agents are more robust to sensor loss than novice agents early in their training. Overall, our findings characterize the tradeoffs between sensory information and task per- formance in practical sequential decision making tasks, and pave the way towards the design of more resource-efficien decision agents. Appendices, videos, and more at wttthtt://// //stte///iieesem goog wwwpo//l....oooooooooooo/oeeoo/o vosi rooooooooicc.
INTRODUCTION
Over time, as robot moves, information emerges in its environment (c.g. an occlusion is removed and a new object becomes visible, or a pair of dice settles on an outcome), some of it is captured by the robot's s sensors, and the robot then acts on this information. Any information not acquired sensing is lost to all downstream it is often taken for granted during the design of robot that more sensing always improves task performance. However, sensing not always an asset (Mason 1993 Erdmann Mason 1988). Sensing and the processing of sensed information entail the use of several key resources in robotics: computation latency, power, controller complexity, and by extension for learning robots, training data. Even discounting these resource costs, over-sensing can still hurt task performance by causing fragile robot behaviors that respond to task-irrelevant sensed details in the environment (Mason 1993).
prcision-making computations, and perhaps for this reason,
Understanding the relationship between the sensing setup (how much information is sensed, what is sensed, quality, timing, latency etc.) and the performance of a downstream policy is an important scientific problem in robotics and more broadly, all of sequential decision-making (Donald 1995
Valle 2019: Tishby & Polani 2010; Koditschek] [022 Xu et al., [202T] Majumdar et al. 2023).
In this paper, we first propose novel approach to empirically study one slice of this question: at what instances along the robot's 's trajectory does enviromment feedback reveal valuable information? In particular, we assume a fixed policy and a simulated perfect sensor that instantaneously reveals the full Markov state of the environment. In this setting, our approach (Section 55) characterizes: at what moments did access to the sensed information critically improve the robot's actions? We study several benchmark tasks spanning varying robot morphologies, and diverse policy synthesis approaches, particularly focusing on state-of-the-an art imitation and reinforcement learning techniques. Our analysis yields several interesting findings, of which highlight select few here. In most tasks, sensing only improves agent actions significantly at some rare moments during task execution. Further, the reliance environment feedback is similar for high-performing policies synthesized

---

Published as a conference paper at ICLR 2025
through very different techniques and represented with very different architectures. Finally, and rather iuunter-ntuiitively, policy performance is inversely correlated with sensor dependency. Well-trained policies respond to sensed information much less frequently. We summarize our contributions below:
We propose a novel approach to quantify the value of sensory information (VoSI) in complex task setups focusing environments with perfect sensing and deterministic dynamics.
With VoSI as a probing tool, we systematicalyy analyze diverse robotics- inspiieed tasks and state-of- the-art lookahead policies, revealing novel insights into the role of sensing.
While these robotic settings largely feature deterministic dynamics and perfect sensing, we also
validate VoSI in more toy tasks with stochastic dynamics and model or sensing noise.
We highlight the potential of VoSI for enabling efficient policy execution by demonstrating how to construct policy that only senses at key sensing moments.
RELATED WORK
Information Parsimony through Constrained Optimization. Agents in the real world often incur significant costs in acquiring and processing sensory information. The trade-off between maximizing perk-eerfrmmance and minimizing associated sensing costs has motivated extensive work on synthesizing efficient controllers under resource constraints. Hansen et al. (1996) ascribe fixed cost any sensing action and learn to optimize when sense along with maximizing a task objective in discrete MDPs. Recently. Treven et al. [024]; Holt et al. (2024) associate constraints with the number of interactions in continuous time control settings. Tishby & Polani (2010); Eysenbach et al. T) Lu et al. (2023) propose associating cost with the bits of information inferred from observations without any direct constraints on sensing itself. In control theory, the event-triggered formalism (Aström 2008 enables the design of controllers that process full-state information only when some coarsely sensed deviations are observed, thereby reducing the computational overhead. These works set up specific models for the resource costs entailed by sensing, and aim to generate good task policies under constraints on those costs. Complementing these approaches, we instead build general tool to understand the sensory requirements for fixed task performance behavior, as represented in a frozen controller. We do this by setting up and measuring the "value of sensory information". Our notions of task value are pertinent for efficient execution of that behavior under any resource cost models and constraints, when the robot looks to act efficiently. We show that our approach reveals interesting insights into the properties of ookkhhea-ppolictee on challenging tasks.
Value of Information (Vol) For Sequential Decision Making Agents. Howard (9966) first described the "value of information" (Vol) towards decision making, placing a value on the reduction of uncertainty about various random variables relevant to task. Inspired by this highly influential framework, we propose to measure the "value of sensory information" for a sequential decision making agent, to assign values to sensory measurements made at various moments in time. We are not, however, the first to apply Vol-related ideas in control. Notably, Flaspohler et al. (2020) build on Vol to synthesize near-optimal model-based planners for PoMDPs: they construct open loop "macro actions" that significantly reduce the planning complexity. Their macro-actions represent a restriction of the mixed loop policy executions that we study. Further, their experimental validation is restricted to stylized simple environments. MMjuuddar et al. (2023) characterize the task-relevant information potential of sensed information and provide fundamental bounds the performance achievable by any sensor-based policy. This characterization requires environments that can be expressed in simple analytic forms or that are tractable for sampling based assessments (requiring finite actions and short task horizon), and this is reflected in their experiments. Further, they do not offer insights on when to sense. In this work, we propose novel empirical analysis approach that can be applied to much more complex environments and policy representations than these prior works. Through it, we study state-of-the-an art robotic policy architectures, on standard benchmark tasks the robot learning literature, revealing insights about the value of sensory information to agent.
3 SETUP AND NOTATION
We model the environment as a controlled Markov Process (S, A, μ. P). where S is the set of states A is the set of control actions available to robot, p is the distribution of start states, and
2

---

Published as a conference paper at ICLR 2025
$0
dp
P(si|so: ao)
01
P(sr|so: ao) ((sslss,,))))
a2 $3
P(sa|83, aa)
$5
[00: 01 02]
(50)
[0g. anl (o:)
Illustration of a mixed loop (MixI.) execution of a look-ahead policy æ. At state so: the agent exccutes the open loop action plan 00:3 ~ (50) for the next h = 3 steps. Over this open loop execution, uncertainty accrues due to stochastic dynamics and/or an imperfect environment model, that is then resolved when the agent senses the state t 3.
Figure I:
P( |8t; (t) is the transition probability distribution specifying how the environment state 8 time evolves in response to robot action decision dg. We consider two common modes of "task specification". The first is imitation learning, where the agent must mimic expert actions closely. The second is reinforcement learning, where a reward function Tr = r(st, ar) specifies feedback, and the agent must maximize (undiscounted) cumulative rewards Ep > t= r())
Look-Ahear Policies. For our analysis, we consider the popular class of "look-ahead" policies A that, after observing the current state St. prescribe not just the immediate action a,, but an n-step "chunk" of future actions IT (ot:ttn((s)) This is a common feature in recently popular imitation learning architectures such as diffusion policy (Chi al. 2023). as well as popular planning-bas approaches such as the model-base RL architecture TD-MPC2 (Hansen et al. 2023).
Mixed Loop and Fixed-Rate Execution. Crucially for our analysis, such look-ahead policies permit execution in a "mixed loop mode". At any given state 8t, the agent can commit to executing any h S n steps of the action chunk a in an open loop manner. At time + h, the loop is closed corresponds to closed loop execution.
by sensing and reacting to S(+A (see Figure T.. Fixing Popular look-ahead policies often reduce resource usage for on-robot execution by sensing and acting periodically with fixed "execution period" words, the agent forgoes sensory information between times and t h We call this "fixed-rate" mixed loop execution. More generally, h < n need not be fixed. We denote the distribution actions for h timesteps conditioned on observing as 0cs+h
to reduce resource usage (Chi et al. (023). In other
To:h (-l.).
How FREQUENTLY Do AGENTS NEED To SENSE?
Our efforts study the value of sensory information start by envisioning a robotic agent that is provided with a look-ahead policy S - A", synthesized by any means of our choice e.g.. model-based control, imitation learning. or reinforcement learning. It must then decide "how to execute" this policy starting from some state Sp. in particular, whether to "sense" or "not" at each time instant 2,2.... As motivated in the introduction, each "sense" decision carries resource costs to real agents. However, for most of our investigation, rather than set specific resource costs, we instead study how much positive value in terms task performance improvements each instant of sensory information brings to the agent to offset any potential resource costs.
Mixed loop execution with lookahead policies enables us to empirically study the impact forgoing sensing any time(s) on the performance of the policy π. In the general mixed loop setting, for a time duration T starting at a sensed state sp, there are 2 possible executions, including fully closed loop (sense each moment) and fully open loop (never sense), amongst which agent must choose. For this first experiment, we restrict our attention to the simpler class of fixed-rate mixed loop executions with varying execution period h, to arrive relationship between the sensing budget (as reflected in the sensing rate 1/h), and task performance.
approximate understanding of the
We study 7 diverse robotic tasks, depicted in Figure 2 sourced from DM-control (Tassa et al. 2018). Robosuite (Zhu et al.) 2020; Mandlekar et al.) 2023). and the Push-T task (adopted from Chi et al. (2023)) The DM-Control tasks involve dynamic behaviors with varied robots and action spaces, like swinging up cartpole, catching a ball in an actuated cup, and spinning a wheel at high angular
'Te policy "action" is often issued low-level controller such PID that uses proprioceptive feedback.
3

---

Published as a conference paper at ICLR 2025
DM-Control
Robosuite
Push-T
T
swingup
cup-catch
finger-spin
square-nut
coffee-pod
threading
Figure 2: The seven robotic tasks considered in our analyses.
velocity with an actuated 2-jointed finger. Robosuite is popular robot learning benchmark for static tabletop object manipulation tasks: we study square-nus nut: picking up and placing a square nut around square peg, coffee -pod: placing coffee pod into coffee machine slot, and threading: high-precision task involving threading a thin needle through small hole on a light movable platform. Finally, Push-T is stylized 2D non-peeheneile manipulation task that involves complex contact dynamics between cylindrical end-effector (projected to a circle) and T-shaped object that must be pushed into target configuration.
For each task, we train state-based lookahead-pon -policies with TD-MPC2 (Hansen et al. 2023) and DiffusionPPlicy (DP) (Chi et al. (203)) which represent the state of the art in model-based RL and imitation respectively. TD-MPC2 failed to produce effective policies on the Robosuite manipulation tasks, and DP failed finger-spin and cup-catch tasks. We study the other policies, which all performed well. More details of training setup and data-collection are provided in Appendix A
Figure plots the best mean reward with 95% confidence intervals i.e. lowest regret achieved by fixed-rate mixed loop executions as the budget varies. "Normalized regret" is measured as a fraction of the mean closed loop execution reward, which is always positive our tasks. There several outstanding trends: most task policies in our study perform just as well as closed loop ( O normalized regret), even when executed once every 5 steps. The three Robosuite tasks are remarkable: for all these manipulation tasks, policy performance does not deteriorate substantially even when sensing happens only once, the beginning of each trajectory. We refer such "one-time-sensing time-sensing" execution as "open loop" in this paper. This surprising effectiveness of open loop executions is in line with observations made recent works Dasari et al.) (2022); Raffin et al. (2023); Wang et al.) (2024) for other standard robot learning benchmark tasks. Two tasks DM-control swingup and Push-T afford comparing TD-MPC2 and DP policies. Trends are largely similar for the two: on Push-I, TD-MPC2 appears be slightly more robust (lower regret) to reduced sensing. Finally, cup-catch, finger-spin and Push-T all show smoothly accelerating regret as the sensing rate decreases.
A Note On Regret Profiles and Task Complexities. Each task's regret profile may also be interpreted measure of its "sensory complexity": low regrets indicate low sensory complexity. threading
= square-nut < swingup < cup-catch < ffinger-sspp spin R Push-T. There is thus also quite a clear complexity ordering among task suites: Robosuite < DM-control < Push-T. Speculating a little, robot learning benchmarks for manipulation as represented by these three Robosuite tasks may be outliers in terms of how little sensing perceptual capabilities they require.
By this measure, the tasks in order of difficulty are: coffee-
4.1 WHICH CONDITIONS NECESSITATE SENSING?
To appropriately interpret these task regret profiles, we now analyze how task characteristics and policy architectures drive policy performance under reduced sensing.
The Sensing Needs of Optimal Infinitely Expressive Policies. Sensing has value to agent to the extent that it delivers information that the agent does not already know, or more explicitly, what the agent could not have predicted without sensing. Let us consider how a particular observation 8, can be informative to an optimal policy т", without limitations on its expressivity or capacity. Observe that construct one such optimal x² that operates by optimizing future actions a.r through the true environment transition function P(s'ls, [ss|ss a), either learned or manually specified. Now, if an agent operating with knows the current state 8t, then its uncertainty about 8 is the entropy of the
4

---

Published as a conference paper at ICLR 2025
DM-Control
Robosuite
Push-T
1.0
1.0
1.0
Normalized Regret
0.8
Nomalizzd Regret.
0.8
Normalized Regret
0.8
0.6-
0.6
0.6-
04
0,4
0.4
0.2
0.0
0.0
0.0
5
10
100
100
10
100
(CL)
1/(Sensing Rate) llog scale) OU
(CL)
1/(Sensing Rate) [log-scale] (OU
(CL
1/(Sensing Ratel [log-scale] (OL)
cup-catch (TD-MPC2) swingup (TD-MPC2)
threadlng (DP) coffee-pod (DP)
TD-MPC2 DP
singer-spin (TD-MPC2)
swingup
square-nut (DP)
Figure 3: Task regret profiles our main tasks, Plots demonstrating what fraction of closed loop policy returns are lost ("normalized regret") when a policy is instead run in fixed-rate mixed loop mode. We group the tasks into DM-control (left), Robosuite (middle), and Push-T (right).
four-rooms 11x11
Impact Dynamics Stochasticity (P)
Impact of Dynamics Model Error (E)
Normalized Regrei
50
130
700
50
100
200
ICU
1/:Sensing Rate) [ogastale]
(OL)
(CL
1/(Snngnng Rate) [Deescoee]]
(OU)
0.05 0.2
0.4
0.01
0.1
1.0
Figure 4: Task regret profiles our toy four- rour-rooms task (left) illustrate how stochastic dynamics (middle) and dynamics model error (right) necessitate more sensing.
transition distribution: (s-=1) P(s'[s a,) log P(s'|s oz). In other words, sensing st+1 delivers H (s++1) bits of information. It follows that the task-relevant information value of sensing 8141 is bounded as (st4) < (s{+1). If the dynamics P(s'|s, a) are always deterministic, then ($++1) = O: there is value to sensing 8,+1, by induction, any future states. This is intuitive: nt can precisely simulate 81+1: and can therefore act optimally without ever needing to actually sense the environment.
Thus, there is at least one task-optimal policy x² that can operate "open loop" (i.e., sensing only at t= 0) when sensing perfect and dynamics are deterministic. corollary of this result is that if the lask reward includes any non-zero cost e >> 0 associated with sensing, then x must be open loop.
To illustrate, consider simple 11x11 four-rooms discrete grid-world task where a robot is tasked to reach and stay at a fixed goal location starting from anywhere (see Figure 4 The robot has one stay-in-place action and 4 directional actions, and gets a sparse reward of when reaches the goal. We assume perfect sensing in all environments in this work, but we now introduce stochastic dynamics in four-rooms: an action other than the desired command a executed at random with probability p. We obtain a task-optimal model-based agent (T* f P) for each environment, and compute similar sensory-regret profiles for fixed-re -att-aate mixed loop executors of agents across different levels of stochasticity (p). Observe that an optimal agent gains no additional value from sensing after the first timestep under deterministic environment dynamics (p 0) and that the value gain from sensing (performance regret) is correlated with the level of stochasticity.
How Finite Expressivity Influences Sensing Needs: Complexity + Uncertainty. Our 7 simulated robotic tasks above have perfect sensing and deterministic dynamics, in most robotic benchmark tasks. Why then do agents still perform worse when they can't sense and react to environment states? We argue that for "finite policies" (e.g. finite in expressivity or training data), complexity plays similar role to stochasticity in the above analysis. Intuitively, finite policies struggle to simulate complex dynamics, such as of a ball bouncing off a rocky forest floor, or of a swimmer swimming through a perturbed river stream. This also applies to complexity in sensing or actuation: provided with a perfect high- high-resolution 3-D map of the forest floor or a flow field of the river stream, finite policies may not successfully process such observations into the task-relevant state. To see this in
5

---

Published as a conference paper at ICLR 2025
Influence of Stage of Training
Impact of Model Capacity
Normalized Regret
finger-spin
Push-T
Normalized Regret
finger-spin
1.0 Push-T
1.0
1.0
1.0
0.5
0.5
0.0
0.0
0.0
0.0
10
50 100
10
50
100
5
50 100
10
100
1/(Snninng Rate)
1/(Snnsing Rate)
1/(nniggg Rate)
(Sensing Rate) [log-scale]
[log-scale]
[log-scale]
[log tog-scale]
beginner
novice
expert
small
medium
large
Figure 5: Task regret profiles on two tasks, showing how increasing policy competency affects sensing needs, through (left) various training stages and (right) various model capacities.
our grid-world family, consider agents deterministic grid, but with different levels of model error e, such that (P, P) A (s, a), simulating the implications of finite expressivity (see Appendix for more details on how P generated). As Figure shows. these model errors generate similar regret profiles to true environment stochasticity above. On the robotic benchmark tasks finger-spin and Push-T we explore the implications of finite training data and codel-capacity by evaluating the task regret profiles of fixed-rate mixed loop executors of TD-MPC2 agent checkpoints different stages of training: early (beginner), middle (novice), and latest (expert) and different representation capacities (modulated by the size of the latent dimensions of the MLP) in Figure 5 (details of the experiment design Appendix [T.. line with the arguments above, agents with higher model capacity or those exposed to more training data appear degrade less at lower sensing rates. Note that the relative drop in performance requires some interpretation in the context of the reward function and absolute returns attained discuss this in Appendix
WHEN SHOULD AN AGENT SENSE?
We have interpreted the regret plots of Figure coarsely profiling the importance of sensory information in a task. However, those experiments only considered fixed-rate mixed loop policies and studied the impact of fixed sensing frequency task performance. They do not support finer-grained questions about the moments in time at which sensing is more valuable or less valuable. For example, in the push-T task, where contact dynamics only come into play when the end-effector is near the T. states at the beginning of an episode as the end-effector moves to the T might not require closed loop execution (as we will show later in Figure g). We now build on from the methodology of Section 4 to permit such detailed analysis.
The Value of Sensory Information (VoSI). Having sensed the state of the world at time t, we seek to understand how much rask-relevant value an agent loses by choosing to not sense the state of the world for h timesteps compared to operat- ing closed loop. To empirically characterize this state-wise re- gret, let us first consider ( ft). mixed loop "execution strategy" that observes state 8; at time t, then forgoes sensing for the next steps, and operates closed loop forever after. This strategy MMsL (st, h) induces distribution over trajectories TMixL h) of episode length T. For convenience, we denote the distribution induced by closed loop control (h. 0) as (8,, 0). Some TCL. and TMixt. trajectories are visualized in Figure for four-rooms with stochastic dynamics. The open loop phase of 7MixL trajectories are depicted in gray, and the closed loop phase that follows, in green.
Ten.
Figure 6: (Mk h)) and TCL (8o) for four-rooms.
(s.) := TMixt. (st,
The idea is simple: Mixt represents the best behaviors when sensing is withheld over the window (t, + At], and TCL in turn represents the best behaviors without any sensing restrictions. Thus, any deterioration in task performance from TCL to TMxL may be attributed to task-relevant sensory information lost during (t, t + h). and measure that deterioration also serves to quantify the "value of sensory information" (VoSI) in that window. When reward function is available, in

---

Published as a conference paper at ICLR 2025
tasks, an obvious choice of tesk-relevant deterioration measure is the loss of rewards of the mixed loop execution relative to a closed loop execution, until the end of the episode. Thus the VoSI is:
T
T
VOSI( h) = E
rt
E
Σ
rt
(c)
(s.1)
This is closely related to the task-level regret profiles analysis of Section fixed state 8, with increasing h, VoSI(s, h = [1, ...) represents a kind of state-wise regret profile: measures how task performance degrades when acting open loop starting from for increasing durations of time. We call these VoSI profiles. Later, we will define and compare other alternative measures of deterioration from TCL TMixL. to plug into VoSI computation in specific settings, such as when no reward is available.
4 When measured for
Before moving on to further experiments using this measure of the value of sensory information, a note: recall from the beginning of Section 4 that our study is limited to an agent considering various "executions" of a frozen look-ahead policy æ, distinguished only by the moments in time when is afforded access to sensory inputs 8t- One practical issue in this evaluation is that the standard pre-trained look-ahead policies π (such as TD-MPC2, DiffusionPolicy) are designed for closed loop performance, and would be operating out-of-distribution when executed in mixed loop mode. Thus TMixL as defined above might produce erratic behaviors, and might not produce the best execution of h subject to the constraint of not sensing within a time window. However, in our experiments, mixed loop TMxtt. executions of these policies appear to hold up well enough to produce coherent and interpretable findings, and those findings themselves align with the expectations we arrive through reasoning as in Section 1.1
Implementing VoSI Profiles. We measure VoSI on all 7 tasks from Figure 2 for TD-MPC2 (DM-control, Push-T) and DP policies (Robosuite). In all cases, we first execute the policies closed loop starting from the task's initial state distribution, and then measure VoSI on states 8 from these closed loop trajectories as follows. For each state s, we first reset the simulator to that state, and then run mixed loop strategies TMKx (s, h) with the open loop length varying from I to 100 steps. For each h, we generate 5 full trajectories starting at s and ending at episode termination (after ' steps). This permits a Monte Carlo estimate of the expectation over ( h) in the second term in Equation (T. For the expectation over TCL in the first term, we similarly use 5 trajectory rollouts to estimate this quantity. These are expensive evaluations: for each state for which we compute VoSI, we must generate 100x5 = 500 trajectories. For DP which is computationally expensive to run, we restrict the analysis to interesting states, ignoring states where the arm is primarily moving in free space (details in Appendix A.3)
presents a few representative VoSI profiles VoSI(s, h = (I.) from across all tasks, sorted into three prototypical profile "shapes".
The Informative Shapes Of VoSI Profiles. Figure 7
"Flatt VoSI profiles (Figure 7 1) occur when the agent derives very little value from sensing in (t, + fi] even at high h. In other words, having sensed the state t, the agent can operate without sensing, losing nearly no task reward. In Robosuite, as suggested carlier by the task regret profiles of Figure VoSI profiles are surprisingly flat even during object interactions (see e.g. Figure 1g). On DM-control and Push-T, VoSI profiles usually only become "flat" after the task goal is effectively achieved (e.g. the T is already in place in Push-T, see Figure 7 Ie), and the agent does not need to do much other than wait for episode termination. For swingup, VoSI profiles are flat both near the starting states where the dynamic swing action must be initiated (Figure 7 lb), and more surprisingly, even in the upright phase (Figure 7 Ic) up to about 70 steps (+= 0.7 seconds). Figure 8-11 offers some insight on this latter case: closed and open loop trajectories (FCL and TMILL) look remarkably alike, because good policies barely actuate the cartpole base in the upright phase.
"Gradual" VOSI profiles (Figure 2) involve steadily accruing VoSI as the window length increases, suggesting that the task-relevant value of sensing grows steadily with the delay in acquiring new information. Sustained and repetitive contacts as in Push-T and finger-spin (Figure (Figure 2a-e) often induce this VoSI prototype. This is understandable: contact is hard to model precisely, so the true realizations of contact dynamics as represented in sensed states always resolve some ambiguity when available. Figure 8 zooms into the Push-T state in (Figure 2e): over the course of push, small missteps from open loop execution cause steady drift from optimal trajectories. This requires extra recovery time when closed loop control resumes after h steps, affecting task reward.
7

---

Published as a conference paper at ICLR 2025
1b
1d
1c
Flat Degradation
VoSI
100
100
100
0
236
256
Gradual Degradation
2b
2c
2d
2
VoSi
h
100
100
100
100
100
Stepped Degradation
3b
3c
3o
VoSI
100
100
Figure 7: VoSI profiles of states encountered our experiments on robotic tasks can be broadly sorted into three prototypical profile shapes, shown here 1) flat, 2) gradual, and 3) stepped profiles.
2.0
h- 20
h 40
h 70
h- 80
h 100
TCL
0.5
Mill
TeL
0.29
20 40 60 8o 100
1.0
h-4
h=
h
0.75
Tcb
y
y
Y
20
40
60
30
:00
Figure 8: Time-aligned visualization of one closed loop trajectory TCL (s:) and the open loop phase of one mixed loop trajectory (St, h), starting from initial state s,. The VoSI profile of s, is plotted on the left, and indicates measurement of the task-relevant deviation between Tai. and Tiit--
"Stepped" VoSI profiles (Figure 3) involve sharp increases at some values of h followed by steady increases or flat stretches. These are indicative of phase changes in the task. For example, in cup-catch(Figguee 3c-e), 3c-e), failing
closely track the ball's complex dynamics as it is dynamically
8

---

Published as a conference paper at ICLR 2025
swingup
finger-spin
cup-catch
Push-T
VoSI
03
0.0
40
100
100
0.0
20
60
100
Figure 9: Visualization all the VoSI profiles overlayed for four tasks.
thrown up can make the difference between the ball landing in the cup, instead colliding with its lip. Similarly, in swingup, not sensing during a short and dynamic phase can prevent the agent from correcting for overshoot undershoot, causing sharp reward losses.
Finally, in Figure we take a step back and overlay the VoSI profiles for all states encountered in TCL executions each task. These plots reveal information about characteristic VoSI profiles in each task. Further, the dark "gradual" and "stepped" curves in Finger-spl and swi ngup correspond to frequently repeated states in these tasks involving periodic movements.
Observing VoSI Evolution Over Time. Our anal- ysis above has focused on individual states sampled from across all tasks, but we now plot the evolution of VoSI profiles of states encountered in a closed loop rollout of the cup-catch task in Figure [0] Each column here corresponds to the VoSI for a specific time-step in the following manner: the x-axis cor- responds to the time & at which the VoSI profile
VoSI
1.0
80
0.8
lifeOOffset (t, t
60
0.6
VoSI(s,, ft) starts, and the y-axis represents absolute times t2 = t + at which the VoSI is evaluated. In other words, (f1, t2) in the plot corresponds VoSI(st,, l2 - 41). Each row of this visualization thus lines up the VoSI measurements for future time instant /", from all past time instants / < l'.
40
20
20
40
60
80
Timestep (t.)
This makes it possible to easily evaluate temporal consistency trends. For example, all states t prior Figure 10: VoSI profiles over trajectory.
to 20 largely "agreed" on the same VoSI values assign all states. This consistency indicates that no task-relevant information was sensed during 0 < < 20. Further, the large VoSI values assigned futures beyond 20 all times before that moment indicate kind of event horizon. Sure enough, VoSI estimates at timesteps after / = 20 diverge from those before: the moment the agent had sensed at = 20, its value for sensing at future moments dropped 0. Rare critical information was observed at t = 20. We overlay visualizations of the states at 0, 20, 40 over the plot: t 20 represents a tipping point after which the ball is destined to fall into the cup, so that no further sensing is necessary. We provide more details and examples of such plots in Appendix B
Reward-Free and Interaction-Fres Measures of VOSI. As introduced in Equation T VoSI requires task rewards, and must be computed through additional interaction with the environment starting from each state. To make our analysis accessible outside of simulated RL benchmark tasks where these conditions can be easily met, study alternative implementations of VOSI, explained in detail in Appendix C Briefly, the state disagreement VoSI-S measures the mean state- wise distance of mixed loop trajectories from closed loop trajectories. This requires no reward function, but still requires the execution of the mixed-loop trajectory. When no more interaction with the environment possible, we propose the plan disagreement VoSI-P, which measures how much the look-ahead policy IT' action plans formed at differ from actions executed in the closed loop.
Table shows that these alternatives generally have high rank correlation with the reward loss-based VoSI from Equation (I) (VoSI-R). As show in Appendix C.I disagreements can often be traced down to some degen-
correlation
swingup
cup-catch finger-spin
push-T
A VoSI-R, VoSI-S) A VOSI-R, VoSI-P)
0.78
0.88*
0.98
0.53*
0.54
0.18*
0.73
0.37*
Table 1: Correlations between VoSI metrics.
9

---

Published as a conference paper at ICLR 2025
Deterministic Dynamics and Inaccurate Model
Stochastic Dynamics and Inaccurate Model
Stochastic Dynamics and Perfect Model
Normalized Regret
1.0
p= 0=
1.0
p= 8.2, 0.7
1.0
p 0.2, &
0.5-
0.5
0.0
0.0
50
100
200
50 100 200
50 100 200
1 (Sensing Rate) [log-scale]
1 1/(Snnsnng Rate) [log-scale]
1/ (Sensing Rate) [log-scale]
fixed
greedy
Figure 11: Visualization of task regrets profiles achieved for different scenarios on four-rooms by the fixed-rate mixed loop execution strategy and greedy strategy utilizing the state-wise VoSI profiles.
erate phases of task, such as after the is already the target position in Push-T so we discount them from computing the correlation". At this time, agent actions do not matter to task rewards (as measured in VoSI-R), but can generate potentially large action disagreements (VoSI-P).
Towards Synthesizing Efficient Execution Strategies. Our primary goals in this manuscript have been to develop a new toolkit to study foundational scientific questions in robotics and decision making, and get the first answers in some settings of practical interest. Answers these questions have many potential practical uses, and we now showcase a simple implementation that is within reach given the extensive analyses above: synthesizing execution strategies for look-ahead policy I that are efficient efficient in terms of sensing and other entailed resource costs.
We present a simple greedy strategy that sets the open loop execution length h= 7ggeedy (s: (B,T) T)). where s is the currently observed state, B is the task regret budget (how much reward we are willing lose relative to closed loop), T is the remaining episode length. Starting after sensing the state so at time & = 0, Tgreedy sets the largest horizon h such that (oSI h) < hB It then decrements the budget and time T' by VoSI(s, h)) and respectively (see Algorithm I for more details). Varying the initial budget available Rgreedy generates family of execution profiles with varying degrees of parsimony (measured as the average sensing rate achieved). For evaluation, recall the task regret where
analysis in Figures executions with varying fixed h. In Figure we compare the Mggeedy family against fixed rate executions variations of the four-rooms task from Section 4.1 Ngreedy effectively exploits VoSI to achieve better performance at most sensing rates than the naive fixed rate strategy. We compare the proposed strategy with baseline that is designed reduce sensing requirements in Appendix
plotted the normalized regret sensing rate of fixed rate
DISCUSSION AND CONCLUSIONS
We have presented a framework to empirically investigate sensory requirements of sequential decision- making agents through a value of information-inspired lens. Our framework permits a first investiga- tion of what sensory information matters and how much, to state of the art policy architectures, in representative robotic tasks. The insights here offer a first glimpse of the value of sensory information, and the efficiencies that understanding can enable.
However, our empirical study is still only a small step towards understanding the sensory needs of robots. The VoSI framework in this paper is limited to particular kind of robot: one that provided with pre-trained closed loop ahek- policy π, and is then charged with making decisions about when to sense such that the policy performance is not overly affected by the omitted sensory inputs. As such, while we might like to have insights on the value of sensory information based on the best possible behavior under each sensing pattern, our findings can only approximate this through what и can achieve. Even beyond this, we have studied when robots need to sense their environments under the most generous assumptions on the task and sensing apparatus: largely deterministic environments except in toy gridworld setting, and noise-free instantaneous actuation and sensing. In Appendices, we have included proof-of- coff-of-concept examples of VoSI analyses: in a cartpole swingup task with noisy vision-based state estimation (Appendix [). and in a multi-sensory realistic quadruped climbing task (Appendix[] J). Future work could further widen the scope of real-world tasks for studying VoSI, and study how can inform efficient policies.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENT
This research was supported in part by NSF CAREER Award 2239301, NSF Award 2331783, DARPA TIAMAT 12490421), and generous gift from AWS AI the ASSET Center for Trustworthy Al at Penn Engineering. The authors would like to thank the members of the Perception, Action, & Learning group at Penn for the insightful discussions, and the ICLR reviewers for their valuable and constructive feedback.
REFERENCES
Karl Aström. Event based control. In Analysis and design of nonlinear control systems: honor of Alberto Isidori, pp- 127-147. Springer, 2008.
Karl Johan Áström and Bo Bernhardsson. Comparison of periodic and event based sampling for first-order stochastic systems. IFAC Proceedings Volumes, 32(2):5006-5011 -500,, 1999. [2
Xuxin Cheng. Kexin Shi, Ananye Agarwal, and Deepak Pathak. Extreme parkour with legged robots. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pp. 11443- 11450. IEEE, 2024. 28
Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, and Shu- ran Song. Diffusion policy: Visuomotor policy learning via action diffusion. arXiv preprint arXiv:2303.04137 04000.04137... 2023.
15
Sudeep Dasari, Jianren Wang, Joyce Hong. Shikhar Bahl, Yixin Lin, Austin Wang. Abitha Thankaraj. Karanbir Chahal, Berk Calli, Saurabh Gupta, al. Rb2: Robotic manipulation benchmarking with a twist. arXiv preprint urXiv:2203.08098 2022.
Bruce Randall Donald. On information invariants in robotics. Artificial Intelligence, 72(1-2):217-304, 1995. 22
Michael Erdmann. Understanding action and sensing by designing action sensors. The
International journal of robotics research, 14(5):483-509 1995. 22
Michael A Erdmann and Matthew T Mason. An exploration of sensorless manipulation. IEEE Journal on Robotics and Automation, 4(4):369-379, 1988.
Ben Eysenbach, Russ R Salakhutdinov, and Sergey Levine. Robust predictable control. Advances in Neural Information Processing Systems, 34:27813-27825, 2021.
Genevieve Flaspohler, Nicholas A Roy, and John W Fisher III.. Belief-dependent macro-action discovery in pomdps using the value of information. Advances in Neural Information Processing Systems, 33:11 11s:1108-1111,, 2020. 2
Roy Fox and Naftali Tishby. Optimal selective attention in reactive agents. arXiv: 15..11152....... 2015. 23
arXiv preprint
Siddhant Haldar, Jyothish Pari, Anant Rai, and Lerrel Pinto. Teach robot to fish: Versatile imitation from one minute of demonstrations. arXiv preprint arXiv:2303.01497 2023. [8
Eric Hansen, Andrew Barto, and Shlomo Zilberstein. Reinforcement learning for mixed open-loop and closed-loop control. Advances in Neural Information Processing Systems. 9, 1996.
Nicklas Hansen, Hao Su, and Xiaolong Wang. Td-mpe2: Scalable, robust world models for continuous control. In The Twelfth International Conference on Learning Representations, 2023.
Wilhelmus PMH Heemels, Karl Henrik Johansson, and Paulo Tabuada. An introduction event- triggered and self-triggered control. In 2012 ieee 51st ieee conference on decision and control
(cde). pp. 3270-3285. JEEE, 2012. 22
Samuel Holt, Alihan Hüyük, and Mihacla van der Schaar. Active observing in continuous- time control. Advances in Neural Information Processing Systems, 36, 2024. 2
11

---

Published as a conference paper at ICLR 2025
Ronald A Howard. Information value theory. IEEE Transactions on systems science and cybernetics, 2(1):22-26, 1966. 2
Daniel E Koditschek. What is robotics? why do we need it and how can we get it? Annual Review of Control, Robotics, and Autonomous Systems, 4:1-33, 2021.
22
Ashish Kumar, Zipeng Fu, Deepak Pathak, and Jitendra Malik, Rma: Rapid motor adaptation for legged robots. arXiv preprint wwxivv2107004035 2021. 28
Steven M Valle. Sensor lattices: structures for comparing information feedback. In 2019 12th International Workshop on Robot Motion and Control (RoMoCo), pp. 239-246, IEEE, 2019. 22
Xiuyuan Lu, Benjamin Van Roy, Vikranth Dwaracherla, Morteza Ibrahimi, lan Oshand, Zheng Wen, ct al. Reinforcement learning, bit by bit. Foundations and Trends@ in Machine Learning, 16(6): 733-865 2023.
22 23
Yicheng Luo, Zhengyao Jiang, Samuel Cohen, Edward Grefenstette, and Marc Peter Deisenroth. Optimal transport for offline imitation learning. arXiv preprint arXiv:2303. 2023. [8
Bartosz Mačkowiak, Filip Matëjka, and Mirko Wiederholt. Rational inattention: A review. Journal
of Economic Literature, 61(1):226-273, 2023. 23
Anirudha Majumdar, Zhiting Mei, and Vincent Pacelli. Fundamental limits for sensor-based robot control. The International Journal of Robotics Research, 42(12):1051-1069 2023.
Ajay Mandlekar, Soroush Nasiriany, Bowen Wen, Iretiayo Akinola, Yashraj Narang, Linxi Fan, Yuke Zhu, and Dieter Fox. Mimicgen: A data generation system for scalable robot learning using human demonstrations. arXiv preprint arXiv:2310. 17596, 2023. 15
Matthew Mason. Kicking the sensing habit. Al magazine, 14(1):58-59) 1993.
Grace McFassel and Dylan A Shell. Reactivity and statefulness: Action-based sensors, plans, and necessary state. The International Journal of Robotics Research, 42(6):385-411, 2023. [2
Antonin Raffin, Olivier Sigaud, Jens Kober, Alin Albu-Schäffer, João Silvério, and Freek Stulp. A simple open-loop baseline for reinforcement learning locomotion tasks. arXiv preprini
1rXiv:2310.05880, 2023. 4
Christopher A Sims. Implications of rational inattention. Journal of monetary Economics, 50(3):
665-690, 2003. 23
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint 1801000800.00690, 2018. 3
Naftali Tishby and Daniel Polani. Information theory of decisions and actions. In Perception- cycle: Models, architectures, and hardware, pp. 601-636. Springer, 2010.
2 22 23
Lenart Treven, Bhavya Sukhija, Yarden As, Florian Dörfler, and Andreas Krause. When to sense and control? time-adamee adaatidee approach for continuous-time rl. arXiv preprint 16X1 2024.
Sebastian Trimpe and Raffaello D'Andrea. An experimental demonstration of a distributed and event-based state estimation algorithm. IFAC Proceedings Volumes, (4(1):8811-88) 2011. 22
Sander G Van Dijk, Daniel Polani, and Chrystopher I. Nehaniy. Hierarchical behaviours: getting the most bang for your bit. In European Conference Artificial Life, pp. 342-349. Springer, 2009. 23
Volodymyr Vasyutynskyy and Klaus Kabitzsch. Event-based control: Overview and generic model. In 2010 IEEF International Workshop on Factory Communication Systems Proceedings, pp. 271-279.
IEEE, 2010. 22
Jun Wang, Ying Yuan, Haichuan Che, Haozhi Qi, Yi Ma, Jitendra Malik, and Xiaolong Wang. Lessons from learning spin" pens". arXiv preprint arXiv:2407. 18902, 2024. 4
12

---

Published as a conference paper at ICLR 2025
Jingxi Xu, Bruce Lee, Nikolai Matni, and Dinesh Jayaraman. How are learned perception -ased controllers impacted by the limits of robust control? In Learning for Dynamics and Control, pp. 954-966. PMLR, 2021. [
Yulin Zhang and Dylan A Shell. Abstractions for computing all robotic sensors that suffice solve planning problem. In 2020 JEEE International Conference on Robotics and Automation (ICRA), pp. 1469-8475. IEEE, 2020. 22
Yuke Zhu, Josiah Wong. Ajay Mandlekar, Roberto Martin-Martin, Abhishek Joshi, Soroush Nasiriany, and Yifeng Zhu. robosuite: A modular simulation framework and benchmark for robot learning. arXiv preprint arXiv:2009, 22229 2020. 3
13

---

Published as a conference paper at ICLR 2025
TABLE OF CONTENTS
Appendices A.1 and A.2 outline the hyperparameters of the TD-MPC2 and DiffusionPoon Policy agents analyzed. Appendix A.33 provides details of the evaluation protocol for obtaining task-regret profiles and VoSI proflee.
Appendix A
Describes the experimental setup for the four-rooms experiments providing a description of the procedure used to obtain inaccurate models for the analysis. The pseudo- code for the proposed greedy strategy Mggrody is provided in Algorithm []
Appendix B:
Appendix C
Describes the alternate reward-free and interaction-free measures of VoSI i.e. the state-disagreemmnt (VoSI-S) and plan disagreement (VoSI-P).
Appendix Provides comparison of the two policy architecutres (DiffusionPolicy and TD-MPC2) on swingup and Push-T tasks.
Appendix E Provides more details on the the evolution of VoSI profiles over trajectory
Figures and 10 and includes more examples.
Appendix Discusses when comparing agents with different closed loop performance.
nuances about the relative performance degradation metric
Appendix Extends the related works presented in the main paper (i.e. Section 2
Appendix Offers an "event- cevent-triggered control" based approach point of comparison for the VoSI-based greedy efficient execution strategy presented at the end of Section 5
Appendix : Results of the VoSI analysis of an agent operating with noisy sensing on the swingup task.
Appendix J: Results of the VoSI analysis realistic quadruped climb-crate task exam- ining how staleness in depth information affects the policy's 's success rate.
A SETUP AND HYPERPARAMETERS
We additionally encourage the reader visit the project website at htttps://sites..geee google com/v com/view / /voii- robotics ics view more task-specific details and explore visualizations that highlight different slices of the data beyond the ones presented in the main paper. The code used to conduct the experiments will be made available on the website.
A.1 TD-MPC2
We use a JAX implementation of the TD-MPC2 algorithm and retain the default hyperparame- ters (listed in Table 2 associated with training the TD-MPC2 except for the experiments with varying model-sizes for which we alter the hidden dimensions of the neural architecture. This reproduction closes matches and exceeds the performance of the original PyTorch implementation. We obtain performant TD-MPC2 checkpoints for the dense-reward DM-control suite tasks considered in 200K steps and the Push-T task clipping the output actions to not move than 50 pixels in the PyMunk interface of the Push-T task to generate smooth motion profiles as the performant RL trajectories were used seed data for training the Diffusson Policy representations on these tasks. TD-MPC2 failed to produce meaningful behavior on the robosuite tasks potentially due the sparse reward nature of the task. To execute the TD-MPC2 agent in mixed-open loop manner disable warm-starting the model-based search and just rely on the latest state observation and use the policy prior to seed the action plan search.
trained up 2M steps with additional modification of
A.1.1 CHECKPOINT SELECTION FOR MODEL-STAGE AND MODEL-SIZA EVALUATION
To test the hypothesis of how model finiteness and expressivity influence the sensing needs of a TD-MPC2 agent. We design two studies the Push-T and finger-spin tasks: (1) select checkpoints at different stages of training (Table 3 (2) synthesize expert controllers starting from different representational capacities (keeping all other hyperparameters the same) (Table
14

---

Published as a conference paper at ICLR 2025
Table 2: Default hyperparameters of TD-MPC2 agent
Parameter
Value
encoder.layers
[256, 256]
encoder.Ir
le-4
world model.mlp-de
512
mordd_ moodel.latent-dde
512
world_moder value- dropout
0.01
world_mode
mww.l..mmmmeeeeee. nuu-vvalue-nets
5
world_model.nn somdel.num-bins
101
world_model.ss www.mmoooollll...oo. symlog- min
-10
worlddmodel.syme syyllogmmaaa max
10
world model.Ir
3c-4
world model. www.mmaa--aaaaaaa ---a--d-ooonn norm
20
mpc.horizon
3
mppi-iterations
6
population-size
512
num-elites
64
min-plan-std
0.05
max-plan-std
2
temperature
0,5
optim. warmstart
false
optim.batch-s..
256
aptim.discoum
0.99
optim.rho
0.5
optim.
cetim.coniittncyycom
20
optim.reward coef
0.1
optim.continue continue-cooff
1.0
optim.value-com value-coef
0.1
pepimmentropy-coe
lc-4
optim.tau
0.01
Table 3: TD-MPC2 checkpoints selected for characterizing task-regret profiles different stages of learning
Task
Level
Steps
Return
finger-spin
beginner
14K
44.37
novice
100K
117.39
expert
200K
133.25
Push-T
beginner
200K
25.62
novice
1M
58.53
expert
2M
66.59
A.2 DIFFUION-POLICY (DP)
We use the code adopted from the official repository (Chi et al. 2023) for training U-Net based DP agents and retain most of the hyperparameters and configurations from the original repository to train policies on the srate-based tasks (Table the only key difference is the extended prediction horizon lengths (close to the episode length) used for training, this was done in order to synthesize mixed loop executions with increasingly lower sensing rates. The Robosuite tasks are obtained from MimicGen repository (Mandlekar et al. 2023) as came with over 200 high quality demos for each task square-nut, coffee-podi and threading all requiring some degree of precise insertion. The success rates over 100 rollouts on these tasks are reported Table On the DM- Control suite and Push-T task train Diffusion Policy agents the dataset of 200 trajectories from
6
15

---

Published as a conference paper at ICLR 2025
Table 4: TD-MPC2 checkpoints selected for characterizing task-regret profiles at different model capacities
Task (Steps Trained)
Level
enc.mlp-dim
wm.mlp-ddi
wm.latent-dim
Return
finger-spin (200K)
small
64
64
32
124.33
medium
128
256
128
136.74
large
512
1024
1024
137.92
Push-T (2M)
small
64
64
32
33.833
medium
128
256
128
70.063
large 512
1024
1024
73.704
closed loop execution of the performant TD-MPC2 policy and were able synthesize meaningfully performant policies on swi ngup and Push-"! but failed on f Finger-spin and cup-catch tasks. We suspect that the non-smooth nature of the control signals of RL-controllers used supervise the imitation learning agent as potential reason for this, however note we did not perform an extensive hyperparameter search to try to synthesize the best policies. The issue of non-smooth target control signals for imitation is ameliorated in swingup with an explicit control cost penalty when training expert TD-MPC2 agents and on Push-T task with the actions constrained to generate smooth motions beneficial for imitation.
Table 5: Default hyperparameters of Diffusion Policy for task of episode length T
Parameter
Value
input ceiiteemed.com dim
256
step_embed _mmed_ddm
256
encoder layers
(512, 512)
unet.down dims
(256, 512, 1024)
kernel_size
5
n-groups num num wit.iddfffiioosseee steps
8
100
ema. power
0.75
dim
256
num_demos
200
train_epochs
3000
batch_size Ir weight decay
256
le-4
le-5
pred. horizon
min3300, T)
obs_horizon
1
act_horizon 4
Table 6: Success Rates for DP agents on Robosuite tasks
Task
Success Rate (%)
square-nut
coffee-pod
86
threading
80
A.3 TASK-REGRET PROFILE AND VOSI PROFILE EVALUATION PROTOCOL
We evaluate fixed-rate mixed loop execution strategy for both the lookahead policies all values of the fixed-rate execution horizons h {1.... ,T} where T represents the episode length the task and report the average return obtained over 100 rollouts of the mixed loop execution.
16

---

Published as a conference paper at ICLR 2025
Implementing VoSI Profiles. For both TD-MPC2 agent on DM-control and Push-T tasks, start by collecting a buffer states from 50 rollouts of closed loop executions of the policy. For each state in this buffer seek characterize VoSI(s, h), do this by reseting the simulator to that state and then run mixed loop strategies 1Mix1((s, h) with the open loop length varying from to 100 steps. For each h, we generate full trajectories (i.e., length of trajectory episode length 7) from s. Now this permits Monte Carlo estimate of the expectations over MMxx h)) in the second term in Equation ( the variance of the values underlying policy. For the expectation over TCL we similarly sample 5 trajectories from closed loop execution of π and estimate this quantity. Thereby yielding an estimate of VoSI(s, h). This is computationally expensive procedure especially for evaluating long-horizon policies for each state we generate on the order 500 trajectories, which starts becoming prohibitively expensive to run DP agent for given the higher inference costs. We therefore, examine the closed loop trajectories from the lens of the blan-disagreement metric (Appendix C) to identify states where the policy reacts to sensory information (which happens only at regions of object interaction observed in Figure [2) and perform the counterfactual examination only on states preceding such states alone effectively filtering states where there is extended free-space motion of the manipulator.
high we starting from a performant
square-nut
coffee-pod
threading
Time-Offsel Offset (B, - hl
280
160
240
200
120
30
120
120
2
40
40
0
120
100
40
60
120
160
200
240
250
50 120
160 200 240 250
Timestop (t,)
Timestep (t:)
Timestep (t,)
Figure 12: Visualization of the planning disagreement metric (VoSI-P) aligned along timesteps of representative rollouts of the Diffusion Policy agent on robosuite tasks. These identify characteristic object-interaction points near which expensive counterfactual return ettun-based metries evaluated.
17

---

Published as a conference paper at ICLR 2025
B GRID WORLD
Setup As identified in Section 4.1 there are two scenarios that justify processing environment feedback: (1) stochastic dynamics (2) inaccurate model of the dynamics. We perform a set of controlled experiments on the four-rooms grid world and study the sensing needs of a model- based controller that has a deterministic optimal closed loop policy π² : S A identified for each environment with stochasticity (p) by value-iteration and a dynamics world model that is used to sample lookahead action sequences. To study the axis of model suboptimality representation eгrоr in the dynamics model of the agent by finding transition probability matrix s.t. DK (s, a), P(s' s, a)) = V We identify such P by simply setting P(s [s, a) c(s, a) (s
simulate
[eEE(()) denotes a uniform prior [N(s))
a(s P(s'| 8, a), where (s (s)
the reachable neighbors (s)) (as effect of some action) of state The weighting term c(8, a) E [0, 1] is identified to satisfy the constraint on KL-divergence.
Task- Reegett Profile and VoSI Profile Evaluation Protocol The task-regret profiles (Figures 4 and [1) obtained by evaluating the performance of the execution strategies over 1000 rollouts upto horizon of 200.
The VoSI profiles used by the greedy execution strategy (Algorithm I are monte-carlo estimates of the expected reward loss (Equation (T)) for each state in the grid world by simulating 1000 rollouts of (s, (s, h) Yh {0,- ... 100}. Observe that ( 0) denotes the distribution of closed loop execution trajectories TcL. Therefore once rollouts of TMixL h) are independently sampled the value-of-sensing information VoSI(s, h) in a time-window after observing s can be estimated.
Algorithm 1 Greedy mixed loop execution Strategy (Mgredyy
Require:
a policy that can produce lookahead actions budget
B
VoSI(s, h): the value of sensory information profiles
T : the episode length
Strategy use the budget the equally distribute the cost of acquiring sensory information over the episode and commit to greedy strategy
1= 0
while < do B
8 T-t
f- max h s.t VOSI(s;, h) < h
B - B VoS(ss,,))
+ 11th++1
Execute (s))
end while
C DETAILS OF THE REWARD-FREE AND INTERACTION- FREE MEASURES OF VOSI
To expand the scope of VoSI measures to leverage trajectory discrepancy characteristics beyond just task rewards. We study a few simple alternate measures:
State Disagreement VoSI-S(s, h)) : (s.h) mi T E ||
112 mea- sures simple time-aligned state discrepancy of measure of distribution MiXL from TCL. One more broadly any optimal-transe transportt based formulation to characterize these state distribution disagreements as used in Luo et al. (202); Haldar et al. (2023) to obtain a reward signal from distribution matching perspective.
s Mixi
18

---

Published as a conference paper at ICLR 2025
aællz a measure akin to the imitation-loss which suggests the deviation of action plans from closed loop actions sequences executed the environment. Such metric does not need access to the distribution of TMix1. trajectories and can therefore be applied dataset of TCL trajectories.
Plan Disagreement VoSI-P(s,, h) (sa Σー= ||a.t
C.1 ON THE ERRONEOUS CORRELATION OF METRICS DUE TO TASK-COMPLETION
By visualizing state-aligned orderings of different alternative VoSI metrics in Figures and 14
where each row across the subplots correspond to the same state s with a few representative states visualized in the buckets. We observe that on states that denote task-completion or states near task completion, have close to flat VoSI-R profile, but exhibit varying degrees of VoSI-P or VoSI-$ due to stochasticity of the policy post task-completetion which bears no consequence on the reward achieved but can make the estimates of other trajectory disagreement metrics relying on states (VoSI-S) and actions (VoSI-P) quite noisy and thereby does not provide much signal. We therefore discount such states when presenting the rank-order correlation statistics Table to suggest how effective alternative metrics can be in computing notions similar to VoSI-R.
VoSI-R
Vo5I-P
VO5I-5
120
30 horizon In)
horizon
hortzon 31l
Figure 13: Push-T
VoSI-R
VoSI-P
VoSi-S
2000
1.0
e -
135
0.6
102
00
025
50
Figure 14: cup-catch
D COMPARING DIFFERENT POLICY REPRESENTATIONS
The Impact Policy Architectures. From Figure in Section 4 we observed for tasks swingup and Push-T T for which we have both TD-MPC2 and DP policies, that the planning based TD-MPC2 agent exhibited lesser degradation in task performance when constrained to operate at much lower sensing rates in comparison to reactive DP policy. To gain finer grained understanding of the sensory dependence of these policy representations we employ our method analysis and compute blate-wise regret profiles of these different policy representations on common task states and illustrate characteristic differences Figure 15 In swingup, we observe that DP exhibits much higher performance degradation in the swingup phase of the task when compared TD-MPC2, but degrading lesser during the balance phase of the task. For P we observe that DP policy critically relies sensing retain some performance in comparison TD-MPC2.
19

---

Published as a conference paper at ICLR 2025
0.75
VoSI
0.5
0.25
40
60
80
100
20
80
100
40
60
80
100
20
40
60
20
40
60
h
h
DP
TD-MPC2
Figure 15: Visualization VoSI profiles of TD-MPC2 (planning-hamne -hased) and DP (reactive-controllee))
along a few common task states.
E DISCUSSION ON VOSI PROFILES OVER A TRAJECTORY
In Figure we visualize overlayed VoSI profiles (with transparency alpha 0.02) for states sampled uniformly from closed loop rollouts of a fixed episode length. In tasks like cup-catch and Push-T the task objective is achieved much before the episode ends i.e. the ball is in the cup or the T object is aligned with the goal. At this point, VoSI profiles are flat because actions are trivial and the policy does not perform further meaningful actions. This why there is a high density of seemingly flat profiles for these tasks in Figure
For most figures in the paper we have visualized the evolution of VoSI profiles over the open loop horizon h. In Figure I0. we additionally convey how the VoSI profiles VoSI(s, ft) evolve not only over open loop horizon but also over the states s (starting from so to ST of a closed loop rollout). This means
mapping the scalar (oSI(st,, h) of two arguments (time index t of last observed state 8t-1, and open loop horizon h). We use colors to indicate the VoSI values. In Figure [O] the x-axis represents the time-indee index t of the last observed state St, and y-axis represents the offset open-loop horizon l2 = l + h. Each vertical column of colors represent VoSI profiles for a particular state varied horizons. The choice of adding the last-observed time offset in the y-axis i.e. using t + h rather than h helps highlight the structure in the evolution of VoSI over time. Tracking along the horizontal row 12 we can interpret how valuable re-sensing before f2 prior moments in time.
We provide more illustrative visualizations of the evolution of VoSI profiles over the course of rollout akin Figure 10 in Figure 16 for the swingup and finger-spin tasks. We observe characteristic periodicity in VoSI profiles along the trajectory in the balance-phass of the swingup task and peak-spinning phase of finger-spin i.e. any state in that window has similar VoSI profile indicating that one could execute fixed-rate mixed loop executors in such windows to sense at the boundaries of where there is value to gain from sensing and repeat.
VoSI
VoSI
10
Time-Offser Offset (t, h)
350
150
08
300
250
0.6
Time- Of-Offseet (tr
0.6
100
200
04
50
Time
200
0
100
200
0.0
40
100 120
140
Timestep (t,)
Timestep (t,)
Figure 16: Evolution of VoSI profiles trajectory for swingup (left) and finger-spin (right)
20

---

Published as a conference paper at ICLR 2025
F IMPACT OF TRAINING STAGE AND MODEL CAPACITY ON SENSING NEEDS OF THE POLICY
Figure shows drop in performance of mixed-loop execution relative closed-loop performance for each policy, but closed-loop performance for beginner and novice are often poor, meaning that there is "less to lose" when executing mixed-loop. For example, in the Push-T task, our "beginner" policy executed closed-loop gets a return of 25.62 compared to 66.59 for the expert. Thus, even if "beginner" deteriorates lower return with mixed-loop execution, this registers in the "relative regret" performance metric relatively small drop in task regret.
In Figure 17 we provide version of the plot where the regret scales are the same by computing the task performance regret with respect to the performance of the "expert" policy checkpoint for training stage (and "large" checkpoint for the model capacity experiment). From Figure [7 it becomes more evident that the expert is capable of retaining higher levels of performance sensing rate decreases.
Influence of Stage of Training
finger-spin
Push-T
1.0
1.0
0.6
0.6
alized
04
0.0
0.0
10
50
100
5
10
50
100
1/(Sensing Rate)
1/1Sensing Rete)
[log-scale]
[log-stale]
beginner
novice
expert
Impact Model Capacity
finger-spin
Push-T
1.0
1.0
Normalized lized Regret
0.6
0.6
0.4
0.0
0.0
10
50
100
50
100
1/(Sensing Rate)
1/1Sensing Rate) [log-stale]
[log-scate]
small
medium
large
Figure 17: Task regret profiles on two tasks with respect to the 'expert' (above) and 'large' (below) policy checkpoints ensuring that scales of the regret profiles are commensurate.
21

---

Published as a conference paper at ICLR 2025
G EXPANDED RELATED WORK
The questions studied in this work are a subset of the questions that Donald (1995) first posed to the robotics community, asking what information is needed for a robot to perform task, how the robot might acquire such information, and more. Their initial steps towards answering these questions developed the notion of "information invariants", which describe loosely speaking, level sets of task performance in terms of sensed or stored information and other resources such as speed and communication bandwidth among multiple agents. These foundational questions still persist Koditschek| (2021). One prominent line of inquiry is on sensor reductions or equivalent classes and sensor dominanceLa Valle 2019). which describes a framework for computing partial orderings amongst sensing setups for robots in task-agnostic manner. Related, /Zang & Shell (2020) provide a framework for searching jointly over sensor designs and plans for some task, and is demonstrated on simple tasks with small number of discrete states and actions. Erdmann (1995) set up specialized minimalistic sensors for a policy that "sense not the states but the applicable actions", they do so by starting from abstract idealized sensor to identify all states where a given action is guaranteed to make some progress in a task (as tracked by a change in progress-measure measure or value function defined on the state space). The applicable action-based subsets form a cover of the state space. The author then proposes task-specific minimalistic "action-based" sensor that is designed to capture just enough information of the state to identify atleast one action that is guaranteed to make progress in task, essentially setting up minimum cover problem. Note that in the process of the sensor design we have consequently also obtained an applicable action for each state with the guarantee that the strategy accomplishes the task under the sensor designed. More recently, McFassel & Shell (2023) have expanded the applicability of these ideas by increasing flexibility in the way progress measures are defined. The foundational framework laid out by Erdmann] (1995) serves as a powerful tool that provides insight into minimalistic sensor design, it limited to systems that can be fully modeled and analyzed analytically under restricted policy classes. While the framework provides insights into what to sense at every timestep for a closed loop policy and does not characterize when acquiring this information is critical. Furthermore, it does not characterize the degradation in performance achievable under a certain sensor design.
In control theory, the questions of devising control systems that are parsimonious with processing the full state information are studied extensively by prior works that propose event- based sampling and
event-triggered control (Áström & Bernhardsson 1999 Aström 2008 Vasyutynskyy & Kabitzschi
2010 Heemels et al. 2012).. Such systems are designed to change the prescribed control plan only when events (discrete variables that are function of a partial state of the system) are triggered. The event-triggered control literature, much like the works discussed in Section 2 aim to generate controllers that operate with limited sensing.
triggeed control typically (I) assumes that a low-cost sensor is constantly monitoring the state at each time and (2) defines a "trigger" event when a manually defined function of this sensory observation crosses a threshold. For example, in (Trimpe) & D' D'AAdrea (201 an event trigger is defined based on the deviation of the coarsely sensed state from a model 's predictions. In contrast, in this work we have sought to characterize for an arbitrary lookahead policy a notion of its reliance external feedback in-terms of its task performance. The insights from which can complement the event- ceit-ttiggered control literature by aiding with the design of events and triggers for efficient execution of the policies.
Outside of robotics, in machine learning and information theory, Tishby & Polani 2010), like us, present an information theoretical view of sequential decision making: they define the "information- to-go" property of a policy π based on the KL divergence from a prior, of the distribution of futures generated by म. They show that this quantity follows a Bellman-like recursion, which is useful to optimize "informationally constrained" policies that maximize task rewards while generating constrained ftormatiion-togoo These constraints are not directly on any information processing capabilities on the agent, but rather on "the information processed in the joint envermennv system", i.e., the surprisingness of policy-induced trajectories relative to the prior. The priors considered are uninformative, and their information-to- known. Our VoSI is affected by stochasticity even in fully known dynamics, as well as the agent's limited information processing capabilities (see Section 4.1 for discussion). Finally, they don't 't deal with when / how the information should be acquired. which is central to our approach. Their work's influence appears in recent works Eysenbach et al. (202 Lu et al. (2023) that apply similar constrained optimization techniques to generate policies that must extract less of the sensed information, but without any constraints when sensing is available. Closer to in this family of
vanishes when the dynamics is fully
22

---

Published as a conference paper at ICLR 2025
work is Van Dijk al. (2009). who demonstrate through evolutionary optimization that temporally extended options-styy architectures are preferred when information processing is constrained. Fox their notion of information-to-go. they measure deviation from a prior that includes an ergodic action distribution that is loosely related to the open-loop policy executions motivate the study of what information an agent should sense and retain for maximizing data efficiency during online reinforcement learning, where the data is ever-changing. They motivate an analysis of the cost- cost-per- bit of information (measured in entropy) sensed to benefit obtained (task return), and conduct experiments on online learning bandit settings to demonstrate implications of agent designs that factor in the cost of acquiring and retaining information for learning polices that minimize the regret accrued in trying reach a stationary target policy.
& &ishby 2013) extends (Tishby & Polani (2010)'s 's framework to POMDPs:
consider in this work. [Lu et al. (2023)
Finally, decision making is also of interest in economics, and sure enough, our questions have precedents that are studied this discipline Maćkowiak et al. (2023), starting from the foundational work of [ims 2003). "Rational inattention" posits that (human) agents cannot process all available information but can choose which pieces information to attend to, explaining many observed macroeconomic phenomena.
23

---

Published as a conference paper at ICLR 2025
H A POINT OF COMPARISON FOR THE REDUCED SENSING APPLICATION
Figure I1 shows a proof-of-concept for a potential application VoSI to reduce sensing needs when executing pre-trained policy. We use simple greedy approach that can adapt to test-time-specifed sensing costs. To offer a point of comparison for the VoSI-based greedy efficient execution strategy we consider "event-triggered control" based approach described below:
(described in Section
Event-triggered approach We note that "event-triggered control" literature (see Appendix G is relevant here as an approach that reduces the amount of expensive state sensing. However, note that does not completely forgo sensing at any moment but instead chooses sense at coarser fidelity until "an event is triggered".. For the four-rooms task, we define coarse sensors that identify the agents position in different sub-grids depicted in Figure 18 (left). We "trigger" event to sense the true MDP state when (a) the coarse sensor recognizes a transition between sub-grids, or (b) no change detected by the coarse sensor within specified timeout hyperparameter) when it is not within the goal region (green sub-grid). As the timeout parameter is varied for this baseline, we obtain the regret profile by measuring the task performance average sensing rate (which only includes the number of times the high fidelity state observation is queried).
Deterministic Dynamics and Inaccurate Model
Stochastic Dynamics and Inaccurate Model p 0.2, E 0.7
coarsely sensed sub-grids
Izrmalized Regret
= 0, E=
1.0
1.0
0.5
0.5
0.0
0.0
50 100 200
50 100 200
1 (Sensing Rate) [log-scale]
1/ (Sensing Rate) [log-scale]
fixed-rate greedy
event-triggered
Figure 18: Task regret profiles achieved for four-rooms by different execution strategies. Note that the event-triggered approach here coarsely senses the state of the environment every step i.c. the agent's presence in some sub-grid depicted on the left. The proposed greedy strategy or the fixed rate execution baseline does not have access to such additional information.
we observe that the event- wrigt-tiggeeee approach achieves higher task performance (lower regret) compared to proposed greedy strategy when constrained operate at lower sensing rates. The extra knowledge, in the form coarse sensing, proves to be a significant advantage for the event- triggered approach in the deterministic dynamics setting (Figure [8 middle).. but when dynamics is stochastic, our naive VoSI-based greedy strategy matches this approach performance, even without the sub-grid sensor. The coarsely sensed "room" based event triggers might not be very beneficial in environments with stochastic dynamics as the action plans necessary within a sub-grid might change widely as result of noise in transition dynamics this results the event- sent-triggered approach relying more on the heuristic timeout to obtain high-fidelity state observations.
From Figure 18
24

---

Published as a conference paper at ICLR 2025
I CASE STUDY: NOISY SENSING
We would like to demonstrate the applicability of the proposed VoSI analysis in settings beyond the scenarios primarily studied in the paper i.e. deterministic environments with perfect state sensing. We provide proof-of-concept application of VoSI as a probing tool to study an agent that operates under noisy sensing on the swingup task.
We train small CNN-based visual state estimator that has up to 5 cm and 2 degrees enrors in the predicted cart position and pole angle, which is quite sigmificant in this task. We then obtain a TD- MPC2 policy that operates these state estimates (see Appendix [.1 for more details) and conduct similar VoSI analysis (Section 5). As such, the value of new sensory measurements increases: this reflected in steeper VoSI profiles overall. However, trends remain similar: for example, stepped VoSI profiles during the swingup phase, and close to flat VoSI profiles even up to 70 steps of open-loop execution during the balancing phase when the pole is already close vertical. See Appendix I.2 for more details.
This establishes both that our VoSI framework applicable to many more general task setups than reported in the main paper, and that the findings reported in the main paper already offer useful intuitions for what results more general/complex task setups might look like.
I.1 TRAINING THE STATE ESTIMATOR AND POLICY
To create a dataset for the state estimator, we sample 20000 configurations with cart position (in meters) (-2 2), pole angle (degree) U8-180, 180). At each of these configurations capture a (128, 128, 3) RGB image from a front-view camera positioned 6.5m from the center of the rail. Figure shows some sample front view images and also shows the coverage of the configuration space in the training data.
We train CNN to predict the cart-position and pole-angle (cosine sine of the angle) from images by minimizing an L.2 loss over the dataset. The architecture of the CNN used is as follows:
[Conv (3x3, channels=32)
GroupNomm - ReLU AvgPool (2x2) -
Conv (3x3, channels=64) + Cooupoommgrrruppstt + ReLU > AvgPool (2x2) -
Conv (3x3, channels=128) GroupNorm
- ReLU - AvgPool (2x2)
Platten > ninear (32768, 256) ReIUU + linnear (256, 3)
The state estimates (predicted cart position and pole angle) obtained by the CNN have a more realistic еrror profiee that are illustrated in Figure by an uniform sampling of the configuration space. The nature of the error varies differently in the configuration space and can be thought of as providing coarser sensing of the true state of the system.
Sesuree0ovvee 20 separate test dataset of 20000 samples obtained
The lookahead TD-MPC2 policy operates the current and previous state estimates i.c. the observation at time t is or = (ft. 1t. The agent predicts the force to apply on the cart to swing the pole upright and balance it note that the instantaneous velocity information is not provided here.
Training Data Coverage
150
pole angle (degrees)
100
cart position (m)
Figure 19: Sample images the training dataset (left) and scatter plot illustrating the coverage of the configuration space in the training data (right).
25

---

Published as a conference paper at ICLR 2025
Cart Position Estimation Error
150
pole angle (degreos)
100
density
-50
-100
-150
-0.15
0.10
-0.5
060
005
0.50
0.1
(m)
cart position (m)
Pole Angle Estimation Error
150
0.8
pole angle (degretes)
500
density
-50
2
02
100
00
-2 error 9 y (degrees)
cart position (m)
Figure 20: The eтror characteristics of the CNN based state estimator. The histogram on the left illustrates the distribution of errors (predicted ground truth value) over states that are sampled uniformly from the configuration space. The plot the right indicates the errors over the tested configuration space.
1.2
VOSI ANALYSIS
TASK REGRET PROFILE ANALYSIS
Like the analysis presented in Section we start the investigation by analyzing the task regret profiles attained by fixed-rate mixed loop exccutions with varying execution period and obtain an approximate understanding of the relationship between sensing budget (as reflected in the sensing rate }.) and task performance.
The agent with noisy sensing achieves closed loop task performance of 150.45, To contrast the profile with "perfect sensing", we obtain the task-regret profile for another TD-MPC2 agent that provided the true cart positions and pole angles for timesteps and t 1. The agent under this "perfect sensing" setup achieves a closed loop performance of 164.46. However, do note that perfect sensing in this case does not accurately reflect the true Markov state of the system the instantaneous velocities are not provided to the agent. The task regret profiles for both these sensing conditions are presented in Figure 21
we observe that with the fixed-rate mixed loop execution of the policy with noisy sensing any deviation from operating closed loop results in performance drop. Whereas an agent provided with "perfect sensing" can tolerate 10 fold reduction in the amount of sensing. However, bear in mind that fixed rate mixed loop execution not necessarily the optimal sensing strategy and is merely useful probe to arriving at an approximate understanding of the relationship between the sensing budget and task performance. We use VoSI profiles (introduced Section (5) to provide more fine-grained analysis of the agent at the state level by following the same protocol described in Appendix A.3 to gather data for the analysis.
In Figure 21
CHARACTERISTIC VOSI PROFILES
In Figure we visualize overlay of VoSI profiles for states uniformly sampled from closed loop executions of the policy with noisy sensing. And similar to the insights of swingup task in the main paper observe "stepped"" VoSI profiles that involve sharp increase in regret some values h
26

---

Published as a conference paper at ICLR 2025
Normalized Regret
0.4
0.-
[CU]
1/(enngg Rate) [log-scalo]
[a OL]
nolsy sensing
Figure 21: Task regret profiles for agent with noisy sensing and perfect sensing on swingup task.
1.00
0.75
VoSI
0.50
0.25
20
80
100
Figure 22: VoSI profiles overlayed for agent with noisy sensing (similar Figure 9.
followed by steady increases or flat stretches, but these sharp increases occur at earlier open loop horizons á compared to the perfect sensing scenario examined in the main paper (Figure 9-1).
The sharp increases in task regret primarily appear during the swingup phase of the task where not sensing during short dynamic period of this phase can prevent the agent from correcting an undershoot which causes sharp loss in rewards. For example, for states encountered in the 'swingup phase" the VoSI profiles (Figure 23 Figure b) show steep decrease task performance when the agent forgoes sensing in short window as extended execution of open loop actions based on noisy state estimate in this dynamic phase results an undershoot of the pole (these are similar to insights we drawn from Figure 3a for agent with perfect sensing), As the agent starts entering the 'balance phase' of the task (Figure 23 + Figure 23 - Figure 23 the agent starts to be able to tolerate extended horizons of open loop action execution without losing any task performance (similar to the insights obtained for agent with perfect sensing from Figure 7 3b.Figere 7 ld which show close to flat profiles during the 'balance phase" of the task).
10
0.75
VoSI
0.5
0.25
20
100
20
40
60
80
100
26
40
60
80
100
20
40
60
80
100
20
60
80
100
Figure 23: Characteristic VoSI profiles observed on the swingup task for agent with noisy sensing.
27

---

Published as a conference paper at ICLR 2025
J VOSI ANALYSIS OF A COMPLEX ROBOTIC TASK: QUADRUPED CRATE CLIMBING TASK WITH DEPTH CAMERA AND PROPRIOCEPTION
In this appendix, we look to demonstrate proof-of-concepp VoSI application to still more complex and realistic robotic task: Unitree Gol quadruped performing a complex locomotion task, namely. climbing onto a crate. Aside from being more complex and realistic than the tasks studied thus far, this setting also facilitates the study of the value of specific sensors in multi-sensory setup.
In particular, in many robotics applications an agent is typically equipped with wide array of sensors that have different data acquisition and processing costs. For instance, our quadruped system comes equipped with joint encoders, force sensors on the feets, IMU and cameras estimate the proprioceptive state and the exteroceptive terrain conditions. Of these sensory streams, constantly processing visual information from cameras poses significant challenges due to the limited on-board compute and power constraints of the system. In this section, we demonstrate how the proposed VoSI analysis can be used to characterize the value of visual sensing, assuming that other sensors are always available.
CLIMB-CRATE TASK
In task setup, Unitree Gol quadruped must climb onto crate of varying heights using head mounted depth camera (see Figure 24). For every episode, we spawn a crate with varying height (10 25 cm) while keeping its length and width fixed 90 x 90 em. The robot is spawned at uniformly random distance (Im to 2m) away from the crate with slightly different approach angles (=1))). The robot gets forward velocity command that directs it to climb the crate, after successfully climbs the crate the command is zeroed out.
1.5
10
mg)
climb-crate
head-mountes depth camera
Figure 24: Illustration of the climb-crate task and the depth estimates from hhad-mmuntedd camera.
TRAINING
leverage the widely used two-stage teacher-student training paradigm (Cheng et al. 2024 Kumar al. 2021) to obtain policy that tolerate various degrees of staleness in the depth information while always receiving the latest proprioceptive information from the IMU and joint encoders.
During the RL training of the state-conditioned teacher policy robust velocity- walking gaits to emerge. Additionally, the teacher policy is trained on a wide range of terrain types with sensor noise and domain randomization. The teacher policy's behavior is then distilled to student policy on the evaluated climb-crate task. During distillation, along with training main closed loop policy network that operates the latest sensory readings (depth and proprioceptive) to predict the teacher actions, we train an axillary network that operates on the history of proprioceptive readings and stale depth information to predict the same teacher actions. This allows the student to operate on stale depth information by choosing to query this auxiliary network. We adopt the architecture in Cheng et al. (2024) with the exception of dropping the recurrence in 58x87 depth image with uniform random noise
dense rewards to encourage
depth processing. The student policy operates of 44 cm in the depth estimates and random pixel dropout (dropout_p = 0.05), while reusing the proprioceptive noise levels and domain randomization parameters from Cheng et al. (2024). The trained student policy attains a closed loop success rate of 79.4% over 500 simulation trials on the climb-crat task. While perform analyses in simulation, these policies are known transfer zero-shot to the real robot, so there reason to believe that our analyses could continue to hold for the transferred policies.
28

---

Published as a conference paper at ICLR 2025
VOSI ANALYSIS RESULTS
Our VoSI analysis is applied the depth sensing alone, while assuming that other sensors are always freely available, We closely follow the procedure outlined in Appendix [A.3] where we first obtain about 50 closed loop rollouts and then conduct the VoSI analysis on subsampled states along these rollouts where the depth information is foregone to various extents.
Similar to the VoSI analysis conducted in previous sections, we analyze the effect of withholding the depth information to various extents as the agent performs the task and illustrate the characteristic profiles observed in Figure 25] Recall that VoSI evaluations are measured relative to reward function: for simplicity, use the sparse reward to compute VoSI, where the agent obtains return of if it successfully climbs onto the crate and gets 0 otherwise.
In episodes where the crate heights are high (>> 20 cm) we observe that policy critically relies on depth information moments before has climb the crate (Figure (5) but as soon as has positioned one of it's feet the crate the depth information does not add much value (Figure 25) Furthermore, in episodes where the crate height is lower < 15 em the agent is able to successtully climb the crate without having access the most recent depth information (Figure This further highlights that the VoSI framework can provide useful insights to inform the deployment of such learned policies.
0.8
0.8
VoSI
0.6
VoSI
0.2
0.2
0.0
20
40
60
80
100
120
140
0
20
40
60
80
100
120
140
h
0.8
VoSI
0.6
0.2
0.0
20
40
60
80
100
120
140
Figure 25: Characteristic VoSI profiles measuring extents to which the Gol robot can operate without the latest depth information on the climb-crate task.
29


Reviews:
[{"review_id": "bQwYSm58hN", "reviewer": "Reviewer_Tq4f", "summary": "This paper presents a method to answer the question in the title. More specifically they approach the problem as how does the frequency of observation updates change policy performance. They perform many studies on various environments to answer this question.", "questions": "Baseline Comparisons\nThere are no baseline comparisons in this work. It seems to be that there is no comparable baseline to have here from the related work. That is reasonable to me but I wonder if for the results presented in figure 11 you might be able to have a baseline comparison? The idea is for different amounts of maximum regret you have some amount of sensory interaction. Is it possible to do some comparison here for the other works that also reduce sensory interaction?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "Research Problem\nVery good. In my view there are two research questions that this work attempts to answer. One is “how important is recent information to learned policies?”, the other is “what is the best way to determine how important is recent information to learned policies?”. Both questions are interesting and relevant to the field. \n\nNovelty\nVery good. I’m not very familiar with the sensing in agent learning literature but if their related work is complete (which it seems to be), their work has significant novelty.\n\nSignificance/Contribution\nVery good. This work could have a lot of significance in the field. The idea of not only discovering the importance of information to policies (and robots) but also the general idea of how to study this question could provide many other works a base to work off. My one complaint is that the introduction doesn’t give the authors perspective on their exact contribution with this work. Adding a few lines in with their contribution would be great as someone who is more broadly in the RL field but not the specific questions of sensing for RL and learning.\n\nAlgorithm Details\nGood. There isn’t an algorithm in this work exactly, but their method is described well. It isn’t an extremely complex idea (using policies that can predict multiple actions in the future) but from their description I believe I could recreate their work.\n\nExperimental Selection\nExcellent. Most of this paper is made up of experiments and analysis to answer their questions. I think the questions presented are well thought out.\n\nPresentation of Results\nGood. In general, the results are clear and the figures well done.  \n\nClarity\nVery good. In general, this paper is very well written and clear. I had no issues reading or understanding the ideas.\n\nFuture Work/Conclusion\nVery good. I think the future work presented in this paper answers the ideas I had for future work. I think the authors can build upon it in interesting ways and can continue to grow our understanding of these questions presented in the paper.", "weaknesses": "The one figure I’m still confused about is Figure 9. I don’t completely understand the figure even with the description in the text.\n\nStatistical Rigor\nI would like confidence intervals on all of the figures. They are on some but figures 3, 4, 5 and 11 don’t have them. Having CIs is critical to understanding the data.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "T47RjGjLys", "reviewer": "Reviewer_6eFu", "summary": "The paper empirically studies the value of sensory information in a variety of robotic tasks and for policies stemming from different learning approaches. This is achieved by mixing open-loop and closed-loop control. Performance degradation is analyzed and its correlations with sensing-frequency, task proficiency, task difficulty, etc. are analyzed. VoSI is proposed as a metric, as well as derived variants that are reward-free and interaction-free. Finally the paper shows a first attempt at using the VoSI profiles for control with parsimonious sensing. The paper studies 7 robotic tasks and a benchmark, which reveal insights.", "questions": "- I'd be really curious to see how the results change when introducing stochasticity/noise. I'd assume the VoSI profiles will change quite a bit, e.g. for the upright phase in swing-up\n- Another interesting aspect would be to analyze the influence of the sensing-frequency on generalization. E.g. is more frequent sensing more robust to changes in the environment, perturbation, etc.?\n- To push it even further you could consider analyzing what happens if you train the base-policies with domain randomization to make them more robust.\n\n## After rebuttal\nThanks for the detailed and constructive replies. I have increased my score (as indicated in my message below).", "limitations": "", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 1, "strengths": "The paper is well written and tackles an interesting and understudied problem, that has practical relevance in robotics.\nThe extensive experiments are well designed and executed. Having a diverse set of tasks and environments is appreciated.\nThe methodology for analysis is sound, the proposed metrics potentially more widely applicable, and the final proposal for using them for control interesting.\nThe paper is well written and easy to follow. The supplementary material is well done and provides many additional relevant details.", "weaknesses": "- In my opinion the paper studies entirely the wrong setting\n  - It only becomes clear rather late in the paper that all experiments (except 4-room) are done in a setting assuming perfect sensing and that everything is deterministic. That's exactly the opposite of how things work on a real robot and where the challenges lie. Sensing is going to be imperfect, and a robot has to deal with unknown perturbations, changes and variability in the environment/objects/tasks, etc. The authors mention those things as future work, but I'd argue that without considering those the paper isn't much more than an academic exercise in a setting that is simplified to a point it doesn't have any practical use for robotics and/or control.\n  - In a similar vain, I would not call the considered tasks 'robotic'. Some of them are more on the side of toy benchmark tasks, others are indeed robotics-inspired but still quite far from being representative for real robot tasks (see reasons above, and you already hinted at problems of Robosuit yourself [being essentially purely kinematic]).\n  - On the flip-side, the tackled problem really is only relevant for real robots. For other policy learning settings - simulated robots, or maybe entirely different domains - the usefulness seems rather limited. Also, how much of a need for parsimonious sensing there really is, is also a bit debatable - it highly depends on the robot embodiment (light-weight UAV vs. stationary robot arm vs. networked swarm).\n- The paper ignores a huge body of work. Work in the control community, in particular on event-based/event-triggered control/state-estimation, has studied this kind of problems since decades$^*$ and as a consequence come to the same insights. Overall the paper feels a bit like re-inventing the wheel (also e.g. the state-disagreement metric). Work on event-based sensing / event-based cameras also goes in a similar direction.\n- With the above background, this paper serves as a nice confirmation about what common knowledge in control and robotics would predict.\n  - When having the 'perfect world' assumption in mind, none of the findings are surprising, let alone counter-intuitive. In a perfect world, with perfect models, sensing etc. all tasks can be solved open-loop. Feedback is required when the models no longer hold (external perturbations, stochasticity, imperfect models) which you nicely show.\n  - And then we can also think about making things more robust, i.e., finding a strategy that can deal with imperfect models/sensing, usually at the cost of performance, e.g., sim2real approaches.\n- At least for the RL setting, I find the approach of using a policy that was trained closed-loop problematic as you point out in l. 339. It sounds a bit like \"we got lucky\". When instead training the policy with the parsimonious sensing, the policy will potentially change drastically to account for that. So the analysis is at best a proxy of what could be achieved under that setting.\n- Minor\n  - It would have been nice to promise releasing the code\n  - typo l. 313 \"hase\"\n  - typo l. 508 \"efficient efficient\"\n\n\n$^*$ See e.g.\n\nTrimpe, S. and D'Andrea, R., 2011. An experimental demonstration of a distributed and event-based state estimation algorithm. IFAC Proceedings Volumes, 44(1), pp.8811-8818.\n\nFor a cool real-world demonstration", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CRMMyOOZUy", "reviewer": "Reviewer_Reqd", "summary": "The paper investigates how frequently action sequences should be recomputed based on new observations with Diffusion Policy and TD-MPC2 on simulated benchmark tasks. The authors find that for many tasks, it is sufficient to recompute the action sequence at a low frequency, and for tasks from the Robosuite environment, it is even sufficient to use only the initial observations and execute entire rollouts open-loop. Furthermore, the authors identify stochasticity in the dynamics and modeling error as reasons why the action sequence needs to be updated based on new observations. They then devise a strategy for adapting the sensing and replanning frequency dynamically during the policy rollout.", "questions": "1. Line 54: In my opinion it is not counter-intuitive that the agent's performance is inversely correlated with the sensor dependency, particularly in the case of TD-MPC2. TD-MPC2 learns a dynamics model of the environment that it uses for planning. The longer the agent was trained, the more accurate the model is, which means that the agent needs to correct the model error less frequently with real data. Perhaps the authors could expand on why they consider this inverse correlation as counter-intuitive.\n\n2. In the plot for Push-T in Figure 10, it seems that most of the curves are quite flat, even though the contact dynamics should be relatively hard to predict for long horizons. Do all the flat curves correspond to cases where the T is almost at the target location or are there other situations in which the dynamics behave particularly predictable?\n\n3. Figure 11: For \"fixed-rate\" the sensing rate specifies the frequency at which the policy gets new sensory input and recomputes the action sequence. For the \"greedy\" configuration there is no fixed rate as the execution strategy adapts the h dynamically. What exactly is plotted on the x-axis for the \"greedy configuration\"? Is it the average sensing frequency?\n\n4. Line 835: If I understand correctly, this maximization is supposed to choose the largest h, so that the VoSI stays above the threshold. Shouldn't that rather be formulated as\n$\\mathrm{max}~ h ~~ \\mathrm{s.t.}~ \\mathrm{VoSI}(s_t, h) \\leq h \\delta$?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The paper conducts an intriguing investigation of how frequent sensor input is needed during the execution of imitation / reinforcement learning policies. The paper is well-written, the experiments are clear, and the results are presented nicely. It is very interesting that the Robosuite tasks can be solved entirely with open-loop control, which suggests that this simulated benchmark might not capture the intricacies of real-world robotics applications well.", "weaknesses": "1. The paper aims to provide insights for robotics tasks, yet all results are obtained only in simulation. The characteristics of simulated benchmark are clearly very different from real-world robotics tasks. Simulations tend to behave significantly more deterministically and predictably. Real-world robotics tasks typically involve significant noise e.g., from the sensors and the actuation, and there are hard to model effects like friction and contacts, which are simplified significantly in dynamics simulators. The fact that the Robosuite benchmarks can be solved by only taking the initial observation into account further indicates that these benchmarks are too simplistic and do not capture the intricacies of real robot tasks. Furthermore, it seems challenging to conduct a similar analysis on real robots since the method requires collecting multiple rollouts for each state. I would appreciate it if the authors could discuss these limitations and the expected differences if such an analysis were applied for a real task.\n\n2. It seems to me that the reward-free and interaction-free VoSI alternatives are still not very practical since these metrics cannot deal with situations in which there exist multiple possible strategies for solving a task. TD-MPC2 learns a dynamics model of the environment. Could we use the deviation between the rollouts predicted by the model and the true rollouts as a proxy for VoSI to circumvent the problem in this case? \n\n3. The sensing frequency adaptation scheme was only tested on a gridworld task. Due to the discrete nature of the gridworld, the tested model is either perfectly accurate or predicting some completely different state. In robotics applications (the focus of the paper), models are typically always somewhat accurate and the main issue is accumulating errors. Clearly there are certain states/actions that cause larger model error (e.g., when the robot is in contact), but still this kind of model error seems to be quite different from the gridworld setting. In a robotics setting, I assume that the optimal sensing strategy is significantly closer to the fixed-rate strategy since preventing error accumulation requires frequent corrections. Perhaps the authors could comment on whether they expect similar benefits of the adaptation strategy in a robotics scenario or on how the experiment could be made more realistic.\n\n4. I am unsure whether sensing frequency adaptation according to the VoSI can be useful for real-world robotics applications. In the current form calculating the VoSI requires multiple rollouts, which would defeat the purpose of increasing the computational efficiency. Furthermore, I am unsure whether a strategy to adapt the replanning frequency makes sense in typical real robotics applications. Usually the actions are executed at a fixed rate on the system, so the controller might be able to produce the action faster if it does not replan according to new sensor information, but this does not mean that the execution will be faster since then the controller just needs to wait according to the control frequency anyway. I would appreciate if the authors could expand on how this strategy could potentially be extended to real systems and what the benefits in practical applications could be.\n\n5. Line 52: \"Sensing only improves the agent actions significantly at some rare moments during task execution\". This statement seems to hold only for certain tasks. Row 2 in Figure 7 shows that for at least two tasks, the performance degrades continuously the lower the sensing frequency is. If the statement above were accurate, then we would see a steep increase in the VoSI at the critical timestep (as e.g., in the swingup examples). I encourage the authors to revise this statement and acknowledge that the observation only holds for certain environments.\n\n6. Line 290/291: The statement \"agents [...] exposed to more training data usually degrade less\" is not really backed by the data. This is only the case in in finger-spin, while in Push-T it seems to be the other way around. The authors should either provide more data to support the claim or revise the statement to make sure that it accurately reflects the data.\n\n7. Line 810: The way that U(s') is defined, it is a uniform distribution over all states, which would mean that the model predicts that the agent teleports randomly across the gridworld. I feel like it should rather be a uniform distribution over something like the neighboring states as this seems like a more realistic error model (a realistic model would probably get the states at least approximately correct, but might make mistakes in the details).\n\n8. There are relatively few tasks for which there are results for both Diffusion Policy and TD-MPC2 (only swingup and Push-T). It would be good to have results for more tasks with both methods to see how consistent the results are across the learning methodologies and policy architectures.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)