Privileged Sensing Scaffolds Reinforcement Learning
Authors: Edward S. Hu
James Springer
Oleh Rybkin
Dinesh Jayaraman
Venue: ICLR 2024 spotlight (2024)
Keywords: reinforcement learning
model-based reinforcement learning
world models
robotics
privileged information
asymmetric learning
multimodality
perception
sensing
Primary area: reinforcement learning
TL;DR: We study how privileged, training-time only observation streams can aid skill learning, and instantiate a MBRL algorithm that incorporates privileged sensing into all auxiliary, training-time components of RL to better train the policy.

Abstract: We need to look at our shoelaces as we first learn to tie them but having mastered this skill, can do it from touch alone. We call this phenomenon “sensory scaffolding”: observation streams that are not needed by a master might yet aid a novice learner. We consider such sensory scaffolding setups for training artificial agents. For example, a robot arm may need to be deployed with just a low-cost, robust, general-purpose camera; yet its performance may improve by having privileged training-time-only access to informative albeit expensive and unwieldy motion capture rigs or fragile tactile sensors. For these settings, we propose “Scaffolder”, a reinforcement learning approach which effectively exploits privileged sensing in critics, world models, reward estimators, and other such auxiliary components that are only used at training time, to improve the target policy. For evaluating sensory scaffolding agents, we design a new “S3” suite of ten diverse simulated robotic tasks that explore a wide range of practical sensor setups. Agents must use privileged camera sensing to train blind hurdlers, privileged active visual perception to help robot arms overcome visual occlusions, privileged touch sensors to train robot hands, and more. Scaffolder easily outperforms relevant prior baselines and frequently performs comparably even to policies that have test-time access to the privileged sensors. Website: https://penn-pal-lab.github.io/scaffolder/

Published as a conference paper at ICLR 2024
PRIVILEGED SENSING SCAFFOLDS REINFORCEMENT LEARNING
Edward S. Hu' James Springer'
Oleh Rybkin² Dinesh Jayaraman'
Uniiversity of Pennsylvania hueddseas. upenn.edu
UC Berkeley
ABSTRACT
We need to look at our shoelaces as we first learn to tie them but having mas- tered this skill, we can do from touch alone. We call this phenomenon "sen- sory scaffolding": observation streams that are not needed by master might yet aid a novice learner. We consider such sensory seaffolding setups for training artificial agents. For example, robot arm may need to be deployed with just a low-cost, robust, general-purpose camera; yet its performance may improve by having privileged craning-time--ne access to informative albeit expensive and unwieldy motion capture rigs or fragile tactile sensors. For these settings, we propose Scaffolder, a reinforcement learning approach which effectively ex- ploits privileged sensing in critics, world models, reward estimators, and other such auxiliary components that are only used training time, improve the target policy. For evaluating sensory scaffolding agents, we design new "S3" suite of ten diverse simulated robotic tasks that explore wide range of prac- tical sensor setups. Agents must use privileged camera sensing to train blind hurdlers, privileged active visual perception help robot arms overcome visual occlusions, privileged touch sensors to train robot hands, and more. Scaffolder easily outperforms relevant prior baselines and frequently performs comparably even policies that have test-time access to the privileged sensors. Website: /attps://penn-- lab qi www.itth.....sceeeee htt.coffoccce/rr//////
INTRODUCTION
is well-known that Beethoven composed symphonies long after he had fully lost his hearing- Such feats are commonly held to be evidence of mastery: for example, novice typists need to look at the keyboard to locate keys but with practice, can graduate to typing without looking. Thus, sensing requirements may be different during learning versus after learning. We refer this as "sensory scaffolding", drawing inspiration from the concept of scaffolding teaching mechanisms in psychology that provide temporary support for student (Wood et al. 1976 Vygotsky al. 2011), like training wheels when learning to ride a bicycle.
For artificial learning agents such robots, sensory scaffolding permits decoupling the observation streams required at test time from those that are used to train the agent. The sensors available in a deployed robot often decided by practical considerations such as cost, robustness, size, compute requirements, and ease of instrumentation, e.g., autonomous cars with only cheap and robust RGB camera sensors. However, those considerations might carry less weight at training time, so a robot learning practitioner may choose to scaffold policy learning with privileged information (Vapnik) & Vashist 2009) from extra sensors available only training. In the case of the cars above, the manufacturer might equip small fleet of training cars with expensive privileged sensors like lidar to improve RGB-only driving policies for eustomers to install in their cars.
What learning mechanisms might permit artificial agents to improve by exploiting such privileged, training-time sensors? Considering reinforcement learning (RL) algorithms, we observe that while their primary output is usually policy, they often employ elaborate training apparatus with value functions, representation learning objectives, reward estimators, world models, and data collection policies. While the output policy must only access pre- determined target sensors, key insight that this training apparatus offers natural routes for privileged observation streams to inffuence policy learning.

---

Published as a conference paper at ICLR 2024
Train
Deploy
Target
Privileged
Act
Touch
Camera
Improve
World Model
Exploration
Distill
Target Policy
Observe
Joints
Object Pose
Representation
Value
Available:
Available:
train/ deploy
train
Scaffolder
Figure 1: Learning a policy to operate from partial observations can be aided by access privileged sensors exclusively during training. Scaffolder improves world models, critics, exploration, and representation learning objectives to synthesize improved target policies.
This insight directly motivates Scaffolder, a novel model-based reinforcement learning (MBRL) ap- proach that "scaffolds" each training component of RI. by providing it access privileged sensory information. Fiiuure I shows a schematic. MBRL algorithms learn an environment simulator, or world model, from experience, and then train policies on synthetic experiences collected within this simulator, potentially mediated by value functions and reward estimators. In Scaffolder, rather than training a low-fidelity world model on the impoverished target observations, we train a "scaffolded world model" with privileged observations that more accurately models environment dynamics. This enables more realistic training experience synthesis, and better credit assignment using scaffolded value functions and reward estimators. Further, exploratory data collection, both in the real environ- ment and within the world model, can now better performed by employing a scaffolded explo- ration policy. Finally, the scaffolded observations also enable learning improved representations of the test-time target sensor observations.
Our key contributions are: (1) We study policy learning with privileged information in novel and practically well-motivated "sensory scaffolding" setting, where extra sensors are available to agent such robot at training time. (2) We propose Scaffolder, MBRL method that exten-
sively utilizes privileged observations to scaffold each auxiliary component of RL. (3) We validate new Sensory Scaffolding Suite (S3). $3
Scaffolder extensively against prior state of the contains ten diverse environments and sensor setups, including privileged active visual perception for occluded manipulation policies, privileged touch and pose sensors for dexterous manipulation policies, and privileged audio and sheet music for training "blind and deaf" piano- robots. See Figure (4) Through detailed analyses and ablation studies, we study the relative impacts of scaffolded learning mechanisms through which privileged sensing impacts policy learning. We find all components are important and that each component's contribution depends on task-specific prop- erties. Finally, show empirically that Scaffolder improves the agent's estimates of the true RL objective function, thus providing improved learning signals to drive policy improvement.
2 PROBLEM SETUP, NOTATION, & PRIOR WORK
The Sensory Scaffolding Problem Setting: Consider the setup Figure a robot policy must use only target observations o, proprioception, touch to pick up block, whose pose is unknown. common to model such problems as partially observable Markov decision process (POMDP) Kaelbling al. 1998) and train the policy - (a: or select appropriate robot actions at A. However, note Figure I that the training process has access to additional privileged camera observations of: thus, the policy π (a, | effectively trains in different, "scaffolded" POMDP with observations of Jo- off] These scaffolded observations of may still only be partial in that they don't t reveal the full environment state. Indeed they often are partial our evaluation settings, but they nevertheless contain strictly information about the environment state 8t E S than the
2

---

Published as a conference paper at ICLR 2024
Blind Cheetah
Noisy Monkey
Block
Occluded Pick-
Privileged
Camera
True State
Object Pase
Touch
Ceiling Cam
Wirist Cam
Target
Proprio
Noisy State
Proprio
Cam
Proprio
Touch
Occluded Cam
Sensory Scaffolding Suite (S3). We visualize four out of our diverse tasks, each exploring different restricted sensing scenarios such as proprioceptive-nly inputs, noisy sensors, images, and occluded moving viewpoints. We evaluate the enhancement of policy training using privileged sensors like multiple cameras, controllable cameras, object pose, and touch sensors. Refer to Appendix E.4 for details on all environments.
Figure 2:
target observations oi In the picking task, even though the policy will operate blind at test time, the learning procedure can use noisy knowledge of the block pose, as revealed through the privileged camera observations.
We now review several prior lines of work on reinforcement learning (RL) with privileged information which may apply to this sensory scaffolding problem. See AAppendix A for extended discussion. Perhaps the most straightforward vehicle for privileged information in RL reward. Analogous to labels in supervised learning, RL rewards specify the task, only present at training time, and may therefore exploit privileged sensors. For example, [Schenck & Fox (2017) instrument privileged thermal cameras to gauge fluid levels to train image- based pouring policy. In another interesting example, Huang et al. (2022a) exploit privileged interactive sensing behaviors for reward estimation, such as pulling at door to evaluate whether door locking task is prop- erly completed. Further, nearly all sensorimotor RL in simulation uses privileged low-dimensional simulator state folding in a limited way. More pertinent to us are methods that exploit privileged observations in other training-specific components of RL beyond just the reward. We discuss these below.
Prior Work:
s, inform task rewards and thus may seen as implementing sensory scaf-
Privileged Critics: Actor-critic methods commonly condition the critic on privileged information (i.e. v(o*)), usually low- oow-dimessional simulator state, while training the actor policy -- (a or) on target observations like high dimensional images (Pinto al., [2018) [Andrychowiiz et al. 2018). These methods assume that target observations contain full information about the state, and struggle when this assumption violated (Baisero Amato 2021).
Privileged Policies: These methods train a privileged teacher x to guide the student target policy क A common failure mode here is the "imitation gap" "eppns et al. 2021 Swamy et al, 2022): the student cannot recover the teacher's actions given impoverished inputs
This is typically mitigated by incorporating an additional reinforcement loss into the student's 's learning objective (Rajeswaran al. 2017] Weihs al. 2021 Nguyen al.) 2022 Shenfeld al. 2023
Aside from direct imitation (Chen et al. 20200) privileged teachers can also benefit the student's
exploration by sharing data (Schwab al.) 2019; Shenfeld al., 2023 Kamienny et 2020
Weigand et al. 2021) or defining auxiliary rewards for the student (Walsman et al. (0022).
Privileged World Models: Most privileged sensing strategies employ model-free RL methods,
with few exploring enhancements to model-based RL using privileged sensors. [Seo et al. (2023)
improves DreamerV2 (Hatner al., (022) by training the single-vier -iew policy representation on multi-view data. Recently, Informed Dreamer (Lambrechts et al. (023) improve DreamerV3's
(Hffner et al. 2023) representation and world modelling via privileged information prediction. Privileged Representation Learning Objectives: Privileged observations are commonly used to train representations for high-dimensional, image- based tasks, for example, by leveraging priv- ileged sensors like additional views (Sermanet et al. 2018: Seo et al. 2023) segmentations (Salter et al.) 202T). Several recent works have leveraged privileged simulator state for sim2real
applications [Lee al. 2020 Kumar al. 2021
Qi et al. 2023), training policy conditioned on
3

---

Published as a conference paper at ICLR 2024
target observations and predicted simulator states. These methods have demonstrated impressive results in quadruped locomotion and dexterous manipulation.
Summarizing, nearly all prior works focus on one route for privileged observations to influence tar- get policy learning, and many suffer from assumptions about the privileged observation of or target observation o that do not hold all settings. Scaffolder makes no such assumptions and exploits the scaffolded observations o+ [oo, o through multiple routes integrated into one cohesive RL algorithm. We also compare our method empirically against representative methods from each cat- egory of prior work, demonstrating significant and consistent gains on large suite of tasks.
3 Scaffolder: IMPROVING POLICY TRAINING WITH PRIVILEGED SENSORS
World Model
train to predict
4
m
ot
Tz+1
Representation
Credit Assignment
Exploration
Figure 3: Scaffolder uses scaffolded observations to improve all components of training: world modelling, credit assignment, exploration, and policy representation.
We now describe Scaffolder, model-based reinforcement learning (MBRL) approach that uses scaffolded observations to comprehensively improve training mechanisms. MBRL algorithms learn an environment simulator, or world model (WM), from data and then train policies inside the simula- tor, facilitated by learned reward estimators and value functions. In the sensory scaffolding setting, suggest conceptually simple improvement: train the WM scaffolded observations of in- stead of impoverished target observations o The scaffolded WM is then used to improve target policy training, credit assignment, and exploration. We also improve the target policy representation learning objective with ot. See Figure for a visualization.
Scaffolded World Model. We build on DreamerV3 (Hafner et al. 2023). MBRL method well- known for its generality (see Appendix C.T for Dreamer details). DreamerV3's WM is imple- mented Recurrent State-Space Model (Hainer et al. 2020). The world model serves pur- poses: it acts as an environment simulator to train the policy and serves as the policy's recurrent state encoder for aggregating observation history. Scaffolder learns additional "scaffolded" WM trained on o to replace the target WM (trained on o for policy training. We still retain and train the original target world model so we can encode states for target policy execution. We define both world models below.
Target
Scaffolded
Target
Scaffolded
Emhedding:
e emb (0,
et emb (ot)
Reward:
P (r [7]
pt af)
Dynamics:
Pe (2i 1 411)
p (=/
2 1))
Decoder:
Pg (o [
p (of (
Posterior:
9 (2 01-1,
do (=t
2 ac-1,
Continue:
Ps (cz 2,)
P p5 (o/)
Emb. Predictor:
p (ej
The embedding a low-dimensional representation the observation. The recurrent posterior acts the state encoder and infers the current state given history and current observation, while the dynamics predicts the current state given only history. The decoder, reward, and continue heads reconstruct the observations, rewards, and termination signals from the trajectory data. Both WMs are trained to encode and predict collected trajectories using Dreamer V3's world model learning objectives (see Appendix for full equations).
Scaffolded Critic and Reward. DreamerV3, without privileged sensing, optimizes the target policy Tg (a > by maximizing the policy's estimated return, which is computed by evaluating
4

---

Published as a conference paper at ICLR 2024
the learned reward and critic networks synthetic trajectories generated by the world model. Scaffolder improves the return estimate by using more accurate synthetic experience from the scaf- folded WM, and better reward and value estimates from the scaffolded reward p (r 22)) and critic (0=,2**) Note that the critic requires both 2 , to remain unbiased, see Appendix C.2 for details. To evaluate the policy, the scaffolded critic and reward networks require trajectories with scaffolded latent states. We describe how to acquire such trajectories in imagination.
Nested Latent Imagination (NLI): NLI is a procedure to roll out Fg inside the scaffolded world model to generate synthetic training data. See Figure 4 for visualization. At time the target policy selects action a; = Ta (2+ to ex- ecute inside the scaffolded WM. generating next scaffolded latent z and reward r241. To con- tinue the rollout by querying the policy time t + 1, the scaffolded =f+1 must be translated into its corresponding target latent 21+1. This is achieved by first employing new "target embedding prediction" model ps (07+1 = trained so that e7+1 x emb (0441) Then, ē,+1 ~ ( *(+)), fed into the target posterior To (ei+1 2 at, ( yields the desired translated target latent 2i41> compatible with the tar- get policy Tg- This completes one step of the trajectory generation and is repeated H times to create trajectory
p I z ax-1)
R
T++
Pi
pt ztt)
qs (2i zi-1 ac-1.
ng (azl 27)
ae
P5 2t)
(2F,,
Figure 4: Nested Latent Imagination.
[ ,0), .i,....... =/r =
Computing the TD(A) return: Now, we show how to improve target policy training with scaf- folded observations and components. All RL methods aim to maximize the true discounted policy return. Policy updates in Dreamer follow the gradient of the TD(A) return, an estimate of the true return that is mediated by the world model's latent state, reward predictions and value function. We modify Dreamer policy objective to use scaffolded dynamics, scaffolded reward estimator, and scaffolded critic instead of the impoverished target components. We highlight the terms that depend on the scaffolded components in blue. The target policy objective is defined as
T
J(0)=
Σ E 1=1
R + nH _e (as 2 [=]]]]
(1)
where the first term maximizes the policy's 's TD(A) return, the second term is an entropy regularizer, and the expectation is over the imagined trajectories generated using the above NLI procedure. The TD(A) return is the weighted average of n-step returns, defined as:
R = r yea ((1 A) = + AR241)
(=ī11
R2 =
This modified objective now benefits from the scaffolded components in the following ways. First, the imagined trajectories are generated with the scaffolded dynamics, which can better approximate the true dynamics compared to the target dynamics. Next, the reward and value estimates can now use zt, which contains potentially relevant information not available in 2 to improve credit as- signment. This all results in a more accurate TD(A) return estimate of the true return, which in turn better optimizes the policy. Thus, scaffolding the world model directly improves the RL training signals provided to the target policy T each update.
While only the WM and critic are directly involved in the policy update above, they are trained on exploration data, and the policy itself operates on representation These can both be improved by using privileged observations, as we discuss below.
Scaffolded Exploration Policy. We train a scaffolded task policy 5 (a+ I 2,,)) inside the scaffolded world model. While TH cannot directly run at test time since privileged observations would be missing, it can generate exploratory data to train oe The intuition is that the scaffolded मæ learner becomes more performant more quickly, so it explores task-relevant states beyond the current reach of Tg. We use a 1:1 ratio of मे and Fe rollouts for exploration. We choose not to enforce any imitation objective between ' and π to avoid the "imitation" gap SSection 2).
Scaffolded Representation Learning Objective. Finally, following prior works (Pinto et al. 2018 Lambrechts al. 2023) that have found it useful regress privileged information for representation
5

---

Published as a conference paper at ICLR 2024
learning, train an auxiliary decoder from 2- to the scaffolded observation ot, to improve 2". We expect this to be helpful when is recoverable from o .ic. inferring object poses from images.
EXPERIMENTS
We aim to answer the following questions. (1) Does Scaffolder successfully exploit privileged sens- ing to improve target policy performance? (2) How does Scaffolder compare to prior work on RL with privileged information? (3) What are the most critical components through which privileged observations influence target policy learning, and how do task properties affect this?
4.1 THE SENSORY SCAFFOLDING SUITE (S3) OF TASKS
We propose Sensory Scaffolding Suite (S3), a suite of 10 nobotics-based tasks to evaluate Scaffolder, baselines, and future approaches in the sensory scaffolding problem setting. See feeeFe 5| As mo- tivated in [Section I] robotics is a well-suited domain for studying sensory scaffolding; practical considerations such as cost, robustness, size, compute requirements, and ease of instrumentation of- ten incentivize operating with limited sensing on deployed robots. In addition to standard definitions for RL environments, S3 pre-defines privileged and target sensors. S3 tasks are more general and difficult than prior sensory scaffolding tasks in a variety of ways. S3 tasks have continuous, observa- tion and action spaces with complex dynamics. Furthermore, the privileged sensors are potentially tigh-dimensional and noisy observations, rather than ground truth, cow-dimensional simulator state. S3 defines performance scores for each task: success rate for pen and cube rotation, number of branches swung for Noisy Monkey, and returns computed from dense rewards for the other tasks.
Environment/Tase
Target Sensors
Priv. Scnsors
Propertics
Blind Pick
Proprio. Touch
2 Fixed Cams, Wrist Cam
Random object position
Blind Locomotion
Proprio
Random hurdles
Blind Deaf Piano
Proprio
Future Notes, Piano State
Deterministic
Blind Numb Pen
Proprio, Iniv/Goal Pose
Object Pose, Touch
Random init goal pose
Blind Numb Cube
Proprio, Init/Goal Pose
Object Pose, Touch
Random init goal pose
Noisy Monkey
Noisy proprio, Noisy branch pos.
True Proprio, Branch Pos.
Random noise
Wrist Pick-place
Proprio. Touch, Wrist Cam Proprio, Touch, Occluded Cam Proprio, Init/Goal Pose, Cam
Fixed Cams
Random object/ bin position
Occluded Pick-place
Wrist Cam
Random object bin position
Visual Pen
Object Pose, Touch
Random init goal pose
Visual Cube
Proprio, Init/Goal Pose, Cam
Object Pose, Touch
Random init goal pose
Figure 5: The Sensory Scaffolding Suite (S3) of tasks. See Appendixx E.4 for details.
We provide a high-level overview here; for more details, see (pppedixx E.4 Figure 5 provides key information about each task. Tasks 1-5 in S3 focus on taking away privileged vision, audio, and touch to train blind, deaf, and numb robot policies that operate mainly from proprioception at test time. The robots span manipulators, legged robots, and dexterous hands, and the tasks include object picking, hurdling, pen rotation, and piano playing.
Tasks 6-10 provide more target sensing to policies, particularly vision (cither as raw RGB images, or object recognition system outputs). Privileged senses
more informative, such multi- camera images, and "active" cameras that move with the robot's wrist. Equipped with the target sensors, monkey robots must swing between tree branches, dextrous hands must rotate objects, and 2-fingered robot arms must put objects away amidst occlusion on cluttered tables.
Baselines. In Section 2 we classified privileged RL methodologies based on the component they enhance with privileged information, such as the critic, policy, world model, and representation learning objective; we select representative baselines from each category to compare to Scaffolder. Wherever possible, we implement baselines over DreamerV3 (Hafner et al. (202)), the base MBRL agent for method. We overview the baselines below, see Appendix E.3 for more details.
Unprivileged: D3eamerV3, We train Dreamer only on target observations. DreamerV3 is known for its consistency across benchmarks, thereby serving strong unprivileged baseline.

---

Published as a conference paper at ICLR 2024
Critic: AAC. For privileged critic baseline, use Asymmetric Actor Critic (AAC) (Pinto et al.
2018). AAC is model-free; we find that it is much more sample-inefficiee so we train it for 100M steps in each environment (20 x more than Scaffolder).
Teacher Policy: Dreamer V3+BC. Here, we follow a common strategy from prior works (Weihs)
et al. 202T Nguyen al. 2022 Shenfeld al. 2023) train privileged teacher policy and
incorporate a teacher-imitating BC loss alongside RL rewards when training the target policy. We
implement this in DreamerV3 and call it prremeer3+BBe The weights for BC and RL objectives
are selected through hyperparameter search for each environment.
World Model: Informed Dreamer. [ambrechts al. (2023) extends DreamerV3 to exploit
privileged observations by incorporating new objective term to decode privileged observations
of from target Dreamer state &
Representation: RMA+, Kumarr et al. (2021) train policies with PPO using a privileged state representation analogous to 2 and regress from target observations o to 2² This is liable to fail when 2t may contain information that is not in but RMA has achieved impressive results for many robotics applications. To facilitate fast RMA training, we found it necessary to provide ground truth simulator state as privileged information; model-free, so we run it for 100M steps to permit it to learn meaningful policies.
call this RMA+. Like AAC, RMA+ is
Exploration Policy: Guided Obs. Finally, Guided Observability (Weigand al. 2021) collects better exploration data by dropping out the privileged information from policy inputs over time.
4.2
RESULTS
Scaffolder (ours)
DBeamerV3+BC Guided Obs
RMA+ (100M)
DreamerV3
Informed Dreamer
AAC (100M)
0.79
Median Score on $3 (10 Tasks)
Blind Pick
Blind Locomolion
300
@ore (Nfemalloo))
0.0
0:52
200
2000
0.4
0.36
100
1000
8-13
0.0
0.
Stops
Stops
Blind Deaf Piano
Blind Numb Pen
Blind Numb Cube
Noisy Monkey
500
0.6
400
0.4
88
15
0.2
84
200
82
100
0.0
0.0
2
Stages
Ied
Stag
Iwe
Sleps
Sleps
Wrist Pick-Place
Cocluded Pick-Place
Visual
Visual Cube
400
0.8
.00
300
0.4
200
200
0.2
0.50
100
0.0
0.25
0.00
0.20
0.75
0.00
0.50
0.75
Steps
5e5
Steps
165
166
155
Figure 6: Scaffolder performs better than baselines across all tasks in learning speed and final per- formance, showing its generality. For each method, we report median score and standard error, and each method is run with 4-10 seeds aside from AAC (100M) and RMA+ (100M).
We evaluate the training performance and final task scores of each method. The normalized final median scores in the top left of Figure 6 are computed using the performance of the unprivileged DreamerV3 baseline as a lower bound (0.0) and the performance of a privileged DreamerV3 model that is trained and evaluated on ot the upper bound (1.0). See Appendix E for more info.
7

---

Published as a conference paper at ICLR 2024
Figure 6eemonsttttes that Scaffolder achieves the highest aggregate median performance across all ten tasks. Scaffolder bridges 79% of the gap between o and o², just by having temporary access to ot training time. In other words, much of the gap between the observations o and ot might lie not in whether they support the same behaviors, but in whether they support learning them.
A closer look at the learning curves in Figure 6 reveals that Scaffolder learns more quickly in out of 10 tasks. Even with its limited environment sample budget (between 250K to 5M), it outperforms or competitive with AAC and RMA trained with 100M samples out of 10 tasks.
Scaffolder successfully exploits privileged sensors to learn performant policies despite severely lim- ited target sensors it plays imperfect, but recognizable rendition of "Twinkle Twinkle Little Star", deftly rotates cubes to goal orientations without any object pose information, and actively moves robot wrist camera to look for objects outside the field of view. See website for video examples of learned behaviors. Base V3 uniformly performs much worse.
The privileged baselines all exhibit a high degree performance variance. They may excel in certain tasks yet are oftentimes worse than the non von-privileged DreamerV3 baseline. Recall from Section 2 that these approaches focus on one route for privileged sensors to influence policy learning, and many make assumptions about the nature of privileged and target observations. This may be ill- suited to performing well on the diverse Sensory Scaffolding Suite. DreamerV3 operating on just the target sensors without any privileged info proves to be a surprisingly strong performer, even outperforming some prior approaches that access privileged information like AAC.
For example, RMA excels in Noisy Monkey, where regressing the privileged true state o² from noisy state o- is relatively easy, yet completely fails in Blind Pick. Indeed, we find that in environments like Blind Pick or Blind Locomotion with large information gaps between target and privileged ob- servations, RMA, DreamerV3+BC. and Informed Dreamer tend to suffer. RMA fails because the target observations o are not predictive of the privileged observations oP (e.g. proprioception is not usually predictive of object states). Informed Dreamer similarly fails because it also enforces the target world model to predict privileged observations from target observations. Finally, Dream- crV3+BC fails in such cases due to the large differences in optimal behavior between a privileged teacher and blind student the privileged teacher can directly pick up the object with vision, while the student policy needs to learn information gathering behavior to pick up the block.
614
3.45 2.00 225 140 1.45
Figure 7: Scaffolder discovers spiral search and robust hurdling for blind policies.
The project website showcases several interesting behaviors learned by Scaffolder and baselines. Scaffolder behaviors broadly fall into two categories depending on the task. Sometimes, it per- forms information gathering strategies in Blind Picking, Scaffolder performs spiral search over the workspace to find the block from touch alone. At other times, it acquires robust behaviors invariant to unobservables in Blind Locomotion, Scaffolder discovers robust run-and-jumg maneuvers to minimize collisions with unseen randomized hurdles. See Figure 7 for visualizations.
4.3 ABLATIONS AND ANALYSIS OF Scaffolder
As evidenced above, Scaffolder works well, but why? We first replace each privileged component with mon-privileged counterpart to assess component
contributions. "No Scaff. WM" opts for training the policy in the target world model over training in the privileged world model. "No Scaff. Critic" uses an unprivileged critic o. (s-) in R. for policy learning. "No Scaff. Explore" collects data with only the target policy. For "No Scaff. Repr.", the representation is trained to reconstruct target observations instead of privileged observations.
Figure 8 compares these ablations on 4 tasks. Overall, all ablations perform poorly relative to Scaf- Jolder, indicating the importance of each component. Interestingly, different scaffolded components
8

---

Published as a conference paper at ICLR 2024
Scaffolder
No Scalf. WM
No Scaff. Critic
No Scaff. Explore
No Scaff. Repr.
Blind Pick
Blind Cube Rolation
Blind Locomotion
Visual Cube
1.0
200
0.8
1.
0.8
Score
200
0.4
100
02
0.2
0.0
2
2
0.00
0.25
0.50
0.75
1.00
Stapps
Steps
Twe
Steps
Shapa
166
Figure 8: We ablate Scaffolder by replacing components with their non-privileged counterparts. All components are important, and the combined method performs best.
prove crucial for different tasks: exploration for Blind Pick, critic for Blind Cube Rotation, world model for Blind Locomotion, and representation for RGB Cube Rotation.
These task-specific trends may be due to the task diversity in S3. In Blind Pick, a blind policy struggles to locate the block, but the vision-based exploration policy can easily locate the block and populate the buffer with useful data. In Blind Cube Rotation, the privileged critic, with access to ob- ject pose, more easily estimate the object-centric reward. Blind Locomotion involves navigating unexpected hurdles, therefore a scaffolded world model, with access to vision, can accurately pre- diet future hurdles. Lastly in Visual Cube, the target policy must encode high-dimensional images, and the scaffolded representation objective encourages encoding of essential object pose and touch information. In all these cases, the combined Scaffolder method benefits from cohesively integrating these many routes for privileged sensing to influence policy learning, and performs best.
Scaffolder Improves RL Training Signals. We notice in Figure 8 that dropping the scaffolded WM and its corresponding scaffolded value sig- nificantly impact learning. Recall that these components are critical to estimate the policy return (Equation (I)). whose gradients directly deter- mine policy updates. We now examine these policy return estimates.
Scaffaided TD(M) Improvament
Let the scaffolded TD(A) return be Equation (1)) and the impover- ished "target" TD(A) return be the original Dreamer V3 policy objective, which substitutes the blue terms in Equation (1) with target dynamics Figure 9:
Comparing
Po (#7+1 ,,, Ge), target rewards P (r; 2, J. and target critic 2/ (zr)) TD(A) estimates.
We compute the mean absolute error (MAE) for each of these two estimates against the ground truth
Monte-Cari return R₁. In Blind Pick, every 25K steps while training, we collect ~ 1000 transitions
with the policy and compute RG
Figure plots Improvement (target return MAE scaffolded return MAE). The improvements are mostly positive, indicating that scaffolded TD(A) is more accurate. This provides evidence for why Scaffolder trains better policies than non-privileged counterparts its policy objective better approximates the true expected return, giving the policy optimizer a better training signal. We find similar trends in other environments in Appendix
5 CONCLUSION
We have studied how additional observation streams at training time can aid skill learning, intro- ducing a diverse S3 task suite aid this study. Scaffolder successfully exploits privileged sensors across these tasks to improve reinforcement learning signals and train better policies. While these results are promising, there is much room for future work. See Appendix for extended limitations and future work. Empirically, our task suite focuses on simulated robotic tasks, but other domains like real-world robotics and video games, each with their own forms of privileged sensing, warrant study, For real robots, it may be practical to estimate rewards through their sensors (Schenck & Fox, 2017 Fu et al., 2018 Haldar et al.) 2023) rather than assume black-box rewards from supervising humans. Here, we expect that improved reward estimates through privileged sensors would offer further advantages (see Appendix G. Our work also opens a window to intriguing deeper questions about the relationship between sensing and learning that it only begins to address: what environment information must a learning agent sense in order to perform a task, and how does this vary with tasks and learning phases? We believe that our findings will prove useful for such future research.
9

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENTS
This work supported by the NSF CAREER Award 2239301 and ONR award N00014-22-1- 2677. The authors would like to thank the anonymous reviewers for their constructive feedback and members of the GRASP lab and PAL group for their support.
7 REPRODUCIBILITY STATEMENT
To ensure reproducibility, we will release all code about Scaffolder, baselines, and Sensory Scaffold- ing Suite on the project website: https: pal 1ab wwwguitthh..... io/ stt.ssaaffllloooo Our supplementary has comprehensive overview of our method in Apppendi C experimentation protocol resource usage hyperparameters in Appendix E baseline details Appendix E.3 and environmental descriptions in Appendix E.4
REFERENCES
Marcin Andrychowicz, Bowen Baker, Maciek Chociej, Rafal Jozefowicz, Bob McGrew, Jakub Pa- chocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, al. Learning dexterous in-hand manipulation. arXiv preprint arXiv:1808.00177 2018.
Andrea Baisero and Christopher Amato. Unbiased asymmetric reinforcement learning under partial observability, arXiv preprint arXiv:2105.11674 2021.
Yevgen Chebotar, Ankur Handa, Viktor Makoviychuk, Miles Macklin, Jan Issac, Nathan Ratliff, and Dieter Fox. Closing the sim-to-real loop: Adapting simulation randomization with real world experience. arXiv preprint arXiv:1810.05687 2018.
Dian Chen, Brady Zhou, Vladlen Koltun, and Philipp Krähenbühl. Learning by cheating. In Con- ference on Robot Learning, pp. 66-75, PMLR, 2020.
Yuanpei Chen, Chen Wang, Fei-Fei, and C Karen Liu. Sequential dexterity: Chaining dexterous policies for long-horizon manipulation. arXiv preprint arXiv:2309.0098/ 2023.
Justin Fu, Avi Singh, Dibya Ghosh, Larry Yang, and Sergey Levine. Variational inverse control with events: A general framework for data-driven reward definition. Advances in neural information processing systems, 31, 2018.
Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. In International Conference on Learning Representations, 2020.
URL https:: wtwp://rrwweweeeeeeouroummmmmme wwwpeeeeeteedeeeeeeeeeeeeeeeeeee
Danijar Hafner, Kuang-Huei Lee, lan Fischer, and Pieter Abbeel. Deep hierarchical planning from pixels, 2022.
wwtp:/ivvorg//bbsbeeo 2206.
Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models. arXiv preprint arXiv:2301.04104 2023.
Siddhant Haldar, Vaibhav Mathur, Denis Yarats. and Lerrel Pinto. Watch and match: Supercharg- ing imitation with regularized optimal transport. In Conference on Robot Learning, pp. 32-43. PMLR, 2023.
Edward S. Hu, Kun Huang. Oleh Rybkin, and Dinesh Jayaraman. Know thyself: Transferable visual control policies through robot-awareness. In International Conference on Learning Representa- tions, 2022. URL
Kun Huang. Edward S. Hu, and Dinesh Jayaraman. Training robots to evaluate robots: Example- based interactive reward functions for policy learning. In 6th Annual Conference on Robot Leam- ing, 2022a. URL /ttttpa:///penrevieeee net forum w d1s
10

---

Published as a conference paper at ICLR 2024
Shengyi Huang. Rousslan Fernand Julien Dossa, Chang Ye, Jeff Braga, Dipam Chakraborty, Kinal Mehta, and João G.M. Araújo. Cleanrl: High-quality single-fit file implementations of deep rein- forcement learning algorithms. Journal of Machine Learning Research, 23(274):1-18, 2022b.
1342. html
Leslie Pack Kaelbling, Michael L Littman, and Anthony R Cassandra. Planning and acting in partially observable stochastic domains. Artificial intelligence, 101(1-2):99-134 1998.
Pierre-Alexanrre Kamienny, Kai Arulkumaran, Feryal Behbahani, Wendelin Boehmer, and Shi- mon Whiteson. Privileged information dropout in reinforcement learning. arXiv preprint arXiv:2005.09220, 2020.
Ashish Kumar, Zipeng Fu, Deepak Pathak, and Jitendra Malik. RMA: Rapid Motor Adaptation for Legged Robots. In Roboties: Science and Systems (RSS), 2021. doi: 10. N0.156077RSSS2222.XXX 011.
Gaspard Lambrechts, Adrien Bolland, and Damien Ernst. Informed POMDP: Leveraging Additional Information in Model-Based RL. arXiv preprint arXiv:2306.11488, 2023.
Joonho Lee, Jemin Hwangbo, Lorenz Wellhausen, Vladlen Koltun, and Marco Hutter. Learning Quadrupedal Locomotion over Challenging Terrain. Science Robotics, 2020. doi: 10.1126/ scirohotics.abc59966.
Hai Nguyen, Andrea Baisero, Dian Wang, Christopher Amato, and Robert Platt. Leveraging fully observable policies for learning under partial observability. In CoRL, 2022.
Xue Bin Peng, Pieter Abbeel, Sergey Levine, and Michiel Van de Panne. Deepmimic: Example- guided deep reinforcement learning of physics-based character skills. ACM Transactions On Graphics (TOG), 3714):1-14, 2018.
Lerrel Pinto, Marcin Andrychowicz, Peter Welinder, Wojeiech Zaremba, and Pieter Abbeel. Asym- metric actor critic for image-based robot learning. In Robotics: Science and Systems (RSS), 2018.
Haozhi Qi, Ashish Kumar, Roberto Calandra, Yi Ma, and Jitendra Malik. In-hand object rotation via rapid motor adaptation. In Conference on Robot Learning, pp. 1722-1732. PMLR, 2023.
Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. arXiv preprint /709. 2017.
Daniele Reda, Hung Yu Ling, and Michiel van de Panne. Learning to brachiate via simplified model imitation. In ACM SIGGRAPH 2022 Conference Proceedings, SIGGRAPH '22. Association for Computing Machinery, 2022.
Sasha Salter, Dushyant Rao, Markus Wulfmeier, Raia Hadsell, and Ingmar Posner. Attention- privileged reinforcement learning. In Conference on Robot Leurning, pp. 394-408. PMLR, 2021.
Connor Schenck and Dieter Fox. Visual closed-loop control for pouring liquids. In 2017 IEEE International Conference Robotics and Automation (ICRA). pp. 2629-2636. IEEE, 2017.
Devin Schwab, Tobias Springenberg, Murilo F Martins, Thomas Lampe, Michael Neunert, Abbas Abdolmaleki, Tim Hertweck, Roland Hafner, Francesco Nori, and Martin Riedmiller. Simultane- ously learning vision and feature-based control policies for real-world ball-in-a-cup. In Robotics: Science and Systems (RSS), 2019.
Younggyo Seo, Junsu Kim, Stephen James, Kimin Lee, Jinwoo Shin, and Pieter Abbeel. Multi-view masked world models for visual robotic manipulation. arXiv preprint arXiv:2302.024000 022000000400, 2023.
Pierre Sermanet, Corey Lynch, Yevgen Chebotar, Jasmine Hsu, Eric Jang, Stefan Schaal, and Sergey Levine. Time-contrastive networks: Self-supervised learning from video. In IEEE International Conference on Robotics and Automation, pp. 133--144, 2018.
Idan Shenfeld, Zhang-War Wei Hong, Aviv Tamar, and Pulkit Agrawal. Tgrl: An algorithm for teacher guided reinforcement learning. In International Conference on Machine Learning, 2023.

---

Published as a conference paper at ICLR 2024
Yanchao Sun, Ruijie Zheng, Xiyao Wang, Andrew E Cohen, and Furong Huang. Transfer RL across observation feature spaces via model- regularization. In International Confer- ence Learning Representations, 2022. URL https:: net/ net/forum?dd
7KdAoOsT81C
Gokul Swamy, Sanjiban Choudhury, J Bagnell, and Steven Z Wu. Sequence model imitation learn- ing with unobserved contexts. Advances in Neural Information Processing Systems, 35:17665- 17676, 2022.
Gyan Tatiya, Jonathan Francis, and Jivko Sinapov. Transferring implicit knowledge of non-visual object properties across heterogeneous robot morphologies. In 2023 IEEE International Confer- on Roboties and Automation (ICRA), pp. 11315-11321. IEEE, 2023.
Matthew E Taylor, Peter Stone, and Yaxin Liu. Transfer learning via inter-task mappings for tem- poral difference learning. Journal of Machine Learning Research, 8(), 2007.
Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojeiech Zaremba, and Pieter Abbeel. Do- main randomization for transferring deep neural networks from simulation to the real world. In IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 23-30. IEEE, 2017.
Vladimir Vapnik and Akshay Vashist. A new learning paradigm: Learning using privileged infor- mation. Neural networks, 22(5-6):544-557, 2009.
Lev Vygotsky et al. Interaction between learning and development. Linköpings universitet, 2011.
Aaron Walsman, Muru Zhang, Sanjiban Choudhury, Dieter Fox, and Ali Farhadi. Impossibly good experts and how to follow them. In The Eleventh International Conference on Learning Repre- sentations, 2022.
Stephan Weigand, Pascal Klink, Jan Peters, and Joni Pajarinen. Reinforcement learning using guided observability, arXiv preprint arXiv:2104. 10986, 2021.
Luca Weihs, Unnat Jain, lou-Jen Liu, Jordi Salvador, Svetlana Lazebnik, Aniruddha Kembhavi, and Alexander Schwing. Bridging the imitation gap by adaptive insubordination. In NewrIPS, 2021.
David Wood, Jerome Bruner, and Gail Ross. The role of tutoring problem solving. Journal of child psychology and psychiatry, 17(2):99-000, 1976.
Kevin Zakka, Philipp Wu, Laura Smith, Nimrod Gileadi, Taylor Howell, Xue Bin Peng, Sumeet Singh, Yuval Tassa, Pete Florence, Andy Zeng, and Pieter Abbeel. Robopianist: Dexterous piano playing with deep reinforcement learning. In Conference Robot Learning (CoRL), 2023.
Grace Zhang, Linghan Zhong, Youngwoon Lee, and Joseph Lim. Policy transfer across visual and dynamics domain gaps via iterative grounding. arXiv preprint 1rXiv:2107.00332 2021.
12

---

Published as a conference paper at ICLR 2024
A EXTENDED RELATED WORK
For the sake of brevity, we gave a broad high level overview of related works on sensory scaffolding. We now give more extensive overview.
First, the sensory scaffolding problem is related to the areas of domain adaptation, domain transfer, and sim-to-real transfer. Such works attempt to transfer an agent trained in source MDP to target MDP, differing observation space, action space, dynamics, rewards, or initial state distribution
(i.e. resets) (Tyloor et al., 2007: Tobin et al. 2017 Chebotar et al. 2018 Hu et al. 2022 Zhang
ct al. 2021 Peng et al., 20188] Chen et al. 2023). p. Sensory scaffolding can be thought as a special
case of observation space transfer, where the source observation space is superset (fo o²) of the target observation space in a POMDP setting. In contrast, prior work (Sun et al., 2022] Tatiya
al., 2023) focusing observation transfer typically assume mapping between the observation spaces to facilitate transfer, i.e. the target image observation space can recover the source simulator state (Pinto et al. 2018), and are formulated with MDPs in mind.
Next, the field of privileged imitation learning is related, but not identical, to sensory scaffolding (i.e. privileged reinforcement learning). In the privileged IL setup, one assumes access pre-existing expert policy with privileged access to information. and the objective for the partially observed student policy is to imitate the expert policy (Swamy et al., 2022, Weihs et al.) 2021). In sensory scaffolding, we study how privileged sensory streams at training time improve RL policy search for a target policy with diminished senses. We do not assume access to any existing target policy, and the target policy's 's objective is to maximize reward, not imitate privileged policy.
B EXTENDED LIMITATIONS AND FUTURE WORK
Scaffolder trains an additional scaffolded world model and exploration policy on top of the original world model policy, which adds a sizable amount of compute and resource burden. Training single dynamics model and policy that is shared across both o and o would reduce this burden, and we believe using ideas from the multi-task learning literature to be a good first step.
Through our ablations and analysis, we have good empirical understanding of how Scaffolder ben- efits from privileged observations. On the theoretical side, there work characterizing privileged imitation learning (Swamy et al., (022). critics (Baisero & Amato, 2021). world models (Lam- brechts et al. (023) and representation learning (Vapnik & Vashist, but concrete theoretical understanding of how privileged information affects the entire RL process and components, is is lacking.
On the practical side, while Scaffolder uses privileged observations to improve multiple training- time only RL mechanisms, future work can investigate better ways of leveraging privileged obser- vations. For example, the use of ot as reconstruction target for representation learning can likely be replaced with a better privileged representation learning objective.
Next, using additional sensors at training time brings in practical problems. More sensors ally results in increased computation and bandwidth requirements. Furthermore, many multimodal datasets are incomplete, i.e. certain modalities may be missing in subsets of the data.
C METHOD DETAILS
Here, we provide more in-depth exposition of Scaffolder and DreamerV3. To start, we set up DreamerV3. In the main paper, we chose to depart from D3eamerV3 notation and world model naming convention for two reasons. First, they POMDP settings, s refers to the true environment state. To avoid confusion with true environment state, we chose use z instead of for world model state. Next, they split up world model state s into two components, h, 2 to more accurately write out the exact computation graph. In the main paper, we defined world model state as component, to be more conceptually ergonomic.
for learned world model state, while in our
Now we default back to the Dreamer notation, and will define Scaffolder in terms of DreamerV3 notation to make comparison easier.
13

---

Published as a conference paper at ICLR 2024
C.I DREAMERV3
In DreamerV3, the recurrent encoder maps observations 2t to stochastic representations 2;. Then, the sequence model, recurrent network with recurrent state ht, predicts the sequence of represen- tations given past 0c-1- The world model state is the concatenation of fty and ž₁- The world model state is then used reconstruct observations, predict rewards rt, continuation flags Ct.
Sequence model:
h4 fo 1s(4--1 -- 04-1)
RSSM
Encoder:
2t g0 24,
Dynamics predictor:
2, (2(l)))))
Reward predictor:
6. (fflillie
Continue predictor:
è (o(cil
Decoder:
ā ~ P ( ( 2.)
Given the sequence batch of inputs æ1.T, actions d1:7, rewards 71.T. and continuation flags c1:7, the world model is trained minimize prediction loss, dynamics loss, and the representation loss.
( = Egg
1= Erred + Payn (0) + Prep (+))
(4)
We refer interested readers to Hafner et al. (2023) for the particular world model loss definitions. We use default hyperparameters from DreamerV3 for our method and baselines that use Dreamerv3.
Next, DreamerV3 trains Actor-Critie neural networks in imagination. The actor and critic operate over world model states s, = h4: 2t, and are defined as:
Actor:
~ (8)))))))
(5)
Critic:
(x(()) R Epg. [Ra]
The critic is trained to estimate values on imaginary trajectories generated by exccuting the actor policy with the learned dynamics. The actor policy is trained maximize the TD(A) returns of the imagined trajectories, defined below.
R - Tz + ycs (1 A) + AR(TT1)
R = (s(
(6)
Now, with the DreamerV3 components set up. we ready define Scaffolder comprchensively. and using DreamerV3 notation.
C.2 Scaffolder
Scaffolder trains two world models, one on scaffolded observations + = {,=, and one on target observations - The scaffolded world model is defined as:
Sequence model:
h fo + 11 0t-1) 0t-1)
RSSM
Encoder:
2 g0 tettthtaa att)
Dynamics predictor:
2t polăī (htt
(7)
Reward predictor:
A. polfelhtt 2
Continue predictor:
& ~ poleilhi 22t)
Decoder:
a ~ (at Iht 2,t)
The target world model is defined as:
Sequence model:
h5 1a(1T11:2 27 4--1)
RSSM
Encoder:
2, ~ (o(= (T
Dynamics predictor:
2 pol (hū)
Reward predictor:
A. (ffle 25²
Continue predictor:
&. (oclih ;2
Decoder:
$7 po 1h,,
14

---

Published as a conference paper at ICLR 2024
The world models are trained follows: We sample trajectory containing observations Tirro actions 01,T, rewards 71:7, and continuation flags c1:r- We then train the scaffolded world model as normal and then convert "ir to Zir by simply leaving out privileged observations o from the data before feeding it to the target world model.
Target Actor and Privileged Critic Next, we define the target actor and critic. The target actor operates over target world model state 8, {h, =t Crucially, the critic operates over both the scaffolded world model state st h 2t and the target world model state 5; Note 8 is still required because & contains info about the historical beliefs of the agent, which is not predictable from only the Markovian learned world model state information.
See (Baisero & Amato 2021 for more
The additional environmental information in s potentially makes value estimation easier for the privileged critic. Furthermore, the critic is trained on trajectories generated by the scaffolded world model, which should better approximate true dynamics. We highlighted in blue the places where privileged components are used instead of their original counterparts.
Target Actor:
Q: Te (arls;) Bi (sT, ,t)
Privileged Critic:
= E P RR
Now, the actor is trained with to maximize TD(A) returns R2 of generated trajectories. We improve all components within this objective with scaffolded observations, and highlight them in blue.
First, the policy objective is written as:
(()) EL_ E,, P [R]/ (1x r) (at|s;)
(10)
where we start by using the scaffolded world model to generate imaginary trajectories, using the Nested Latent Imagination procedure described in SSeecition 3
We further incorporate scaffolded components into the TD(A) return. Because we have access to st from generating trajectories in the scaffolded world model, we can use the scaffolded reward, continue, and critic to compute TD(A) return.
R = + 7Ca (11 A)o st1) AR
R = oie (5 ST)
(11)
Additional Exploration Policy We train an additional exploration policy a"; sT) that oper- ates over scaffolded world model state during training time, to collect informative trajectories. The intuition is that with better sensing, xe can solve the task more quickly and gather relevant data. To do so, we simply define separate actor critic networks for the exploration policy, that depend on st It is trained to maximize reward using standard imagination within the scaffolded world model.
Exploration Actor:
at ( ( R 2
(12)
Exploration Critie:
R2
We alternate between collecting episodes with the exploration policy and target policy in a 1:1 ratio.
Target Policy Representation We modify the target decoder to (ă h 2, so that it en- forces the representation to be more informative of the privileged observations, which is potentially useful for control.
C.3 NESTED LATENT IMAGINATION
Here, motivate and explain the nested latent imagination procedure more in detail. First, let's start with a brief overview DreamerV3' latent imagination procedure.
Recall that DreamerV3 uses the world model components (dynamics, posterior) for two roles: 1) as a environment simulator to generate synthetic experience for policy improvement, and 2) as a latent state encoder that aggregates observation history for the policy.
15

---

Published as a conference paper at ICLR 2024
R
2212
R
Teme
Up
p3 2 9c-1)
Prlat a--), ne (c,l() patri x1)
#
9 22-2:00-1:0E)
(z,)
Pă at)) meletiz)))
Latent Imagination
Nested Latent Imagination
Figure 10: Left: DreamerV3's Latent Imagination note that it only uses impoverished. target dynamics, rewards, and values to generate trajectories for policy improvement. Right: Scaffolder uses scaffolded dynamics, rewards and values.
Concretely, the dynamics p (2 2t at) the environment simulator, and the posterior p (2r 4[-1 0c-1, encodes present e; and history z;-1; at-1 into a compact latent state 24
Referring to the left side of Figure 10. DreamerV3's latent imagination generates trajectories by repeating the following three steps.
1. Policy Inference: At timestep i, the latent state žे is fed into the target policy x a to generate an action dz-
2. Dynamics Inference:
The current latent state and action (2, a.) are fed into the dynamics
p (2++1 _.,))) to generate future state 2++1-
3. Credit Assignment: With the future state as input, rewards p (re+11 2741) and value estimates v(2 are computed.
After running these steps for H timesteps, trajectory [=T =2 .... 2.... 2n is created. The policy is trained to maximize the TD(A) return of all the trajectories, which is a function of the dynamics, rewards, and values estimates.
Consider what happens to this procedure if the target observations are extremely impoverished, i.e. have a high loss of information from the scaffolded observations. Then the target dynamics model (2 a. ), rewards p (r++1 2(41) and value estimates v(((+1) will be inaccurate.
Instead, we propose to train an additional scaffolded world model, that can take advantage of the scaffolded observations to learn more accurate environmental simulator for generating trajectories, replacing the role the target dynamics in DreamerV3. However, because we still need to run the target policy, we retain and train the target world model components and use the target posterior to encode latent states from target observations.
Following the right side of Figure 10
1. Policy Inference: timestep we have the target latent state 2, and the scaffolded latent state 2 The target latent is fed into the target policy ( ||2," generate an action ae.
2. Scaffolded Dynamics Inference: The current scaffolded latent state and action (2f aa) are fed into the scaffolded dynamies p+ (=t =t, ax) generate future state 2
3. Scaffolded Credit Assignment: With the scaffolded future state as input, rewards
p+ (re+1 = and value estimates 1(2 are computed.
4. Target State Inference: At this point, we have the future scaffolded state z but the policy requires - to start step for the next iteration.
We can't directly convert or learn a mapping 2 + " since they encode fundamentally different information (see Baisero & Amato (222)). Instead, we learn mapping
zt + o since z - ot via DreamerV3's reconstruction objective, and o is a superset of o=
16

---

Published as a conference paper at ICLR 2024
By training the target embedding predictor p* (e- 2=*) where e is the low-level embed- ding of the target observation, we can now infer the target latent state =, with the target posterior ((,, 2 (T) With 2441 from the target posterior, we now can start step for the next iteration.
After running these 4 steps for H timesteps, trajectory r [2T 2 01, 22,2221000 ...2 ²H] is created. The policy is trained maximize the TD(A) return of the trajectories, which is function of the scaffolded dynamics, rewards, and values estimates. Later, we show that these scaffolded TD(A) estimates using scaffolded components are more accurate than the non-privileged TD(A) estimates, explaining why Scaffolder trains better policies than DreamerV3.
D RUNTIME AND RESOURCE COMPARISON
Our method builds off DreamerV3, so it is compute efficient, requiring only GPU and 4 CPUs for each run. Similarly, baselines building off of DreamerV3 like Informed Dreamer, DreamerV++BC, Guided Observability also are compute efficient. We train on Nvidia 2080ti, 3090, A10, A40, A6000, and L40 GPUs. contrast, model-free methods like RMA and AAC require large batches for PPO / Actor-Critic training, requiring 128 CPU workers to have fast and stable training.
Next, we give the runtime in hours for each method, for all experiments these are approximate, since we run methods over different types of GPUs and CPUs. In general, Scaffolder takes ~ 30%-40% longer to run than DreamerV3 due to training an additional world model and exploration policy, but Scaffolder is much more sample efficient and reaches higher per- formance with less environment steps, making up for slower wall time. Similarly, Dreamer VCCBC also slower compared to DreamerV3 because trains the teacher policy and teacher world model alongside the student policy and world model. In general, RMA and AAC take at least day to train reach 100M steps.
Figure II Note that
We plot the learning curves over hours rather than environment steps in Figure 12 largely the same as the learning curves over environment steps in Figuur 6 This is because Scaffolder is much more sample-efficient than baselines, reaching higher performance earlier in walltime even it takes more time per-step train the additional models.
The trends are
Scafffolder
Informed Dreamer
DreamerV3+BB
Guided Ohs.
RMA+<100M)
AAC(100M)
Dreamer
Blind Pick
18
14
18
14
36
36
14
Incomotion
24
Blind Deaf Piano
18
14
18
14
36
36
14
Blind Pen
36
Blind Cube
12
12
36
36
Noisy Monkey
20
12
12
36
36
12
Wrist Pick Pick place
22
16
16
36
36
16
Occluded Pick-Place
9
36
36
RGB Pen
3
36
36
RGB Cube
3
36
36
Figure 11: Approximate runtime in hours for each method, the tasks.
E EXPERIMENT DETAILS
E. EVALUATION PROTOCOL
In the main results in Figure 6. we report the score for each task. The score is task tpec-fpec metric of performance. We use return (sum of rewards) by default. The Pen / Cube rotation tasks use success rate and Noisy Monkey uses number of handholds. While the policy trains, we periodically evaluate the policy every 15000 training steps, and log the mean score over 15 evaluation episodes. We launch 4-10 seeds for each method, and report median and standard error of the evaluation scores in the learning curves over the seeds. We give each task sample budget ranging from 250K to 5M.
To compute the final normalized median scores, we perform the following steps for each task. First, obtain lower and upper bound on performance by running Dreamerv3 using either o of as inputs for the task's ss training budget. Next, we take each method's median score over seeds at the end of training and normalize by the lower and upper bound scores. We do this for each method.
17

---

Published as a conference paper at ICLR 2024
Scaffolder (ours)
DBeamerV3+BC
RMA+ (100M)
DreamerV3
Informed Dreamer
Guided Obs
AAC (100M)
Median Score S3 (10 Tasks)
Blind Pick
Blind Locomotion
0.8
300
3000
Store (Normaizee]
0..8
Scor
200
2000
0.44
0.36
100
1000
0.2
0.18
0:15
0
0.01
10
20
Hours
Houns
Blind Deaf Piano
Blind Numb Pen
Blind Numb Cube
Noisy Monkey
800
400
0.3
0.8
300
Saoe
0.4
200
0.2
100
0.2
0.0
0
10
20
10
10
15
5
10
20
Hours
Hours
Hours
Wrist Pick-Place
Occluded Pick-Place
Visual Pen
Visual Cube
400
0.4
0.8
300
Scons
200
200
0.6
100
0.2
0.4
100
0.1
0
100
0
10
20
10
20
2
9.0
255
5.0
1.5
10.0
Hours
Houm
Hours
Houra
Figure 12: We plot performance over walltime for Scaffolder and baselines.
We then average the normalized scores for each method for all tasks, and report this as the final median score.
E.2 Scaffolder HYPERPARAMETERS
Next, we describe the hyperparameter settings for Scaffolder. In short, we found hyperparameter search to be short and easy, due to the robustness of the DreamerV3 algorithm. For new environ- ments, we found that DreamerV3 only needs tuning for two hyperparameters, the model size and update to data (UTD) ratio. We follow an easy guideline for tuning these more complicated dy- namics generally require larger models, and tasks with harder exploration require more data and fewer updates (low UTD). As a result, we found hyperparameter settings that work for tasks with similar properties, as seen in Table 1
We did not tune these two DreamerV3 hyperparameters towards Scaffolder- rather, when we used model sizes and UTD ratios from when we ran DreamerV3 with privileged inputs reference method used for computing the upper bound scores. These same settings are then used for all DreamerV3 methods (Scaffolder. Informed Dreamer, DreamerV3+BC, Guided Observability) be consistent.
Table 1: DreamerV3 hyperparameters
Model, UTD
Small, 512 (Simple Dynamics, Easy Exploration)
Blind Pick, Blind Locomotion, Wrist Pick-Place, Occluded Pick Place Blind Deaf Piano Noisy Monkey, Blind Pen, Blind Cube, RGB Pen, RGB Cube
Large, 512 (Complex dynamics. Easy Exploration)
Large, 16 (Complex Dynamics, Hard Exploration)
18

---

Published as a conference paper at ICLR 2024
E.3 BASELINES
DreamerV3 We run DreamerV3 "out of the box" with only access to target observations for all tasks. See the prior hyperparameters discussion for details.
Informed Dreamer Informed Dreamer adds an additional loss term to DreamerV3 that pushes the world model to decode privileged information from the target observations. Informed Dreamer works best when the privileged information is predictable from the impoverished, target observation. Due to the simplicity of the method, which just decodes additional observations, we can just the DreamerV3 codebase directly to run Informed Dreamer by changing the decoder outputs.
DreamerV3+BC DreamerV3+BC trains two policies simultaneously, a teacher policy on privi- leged observations and a student policy on target observations. The teacher policy trains with the same objective as DreamerV3, while the student policy has combined reinforcement learning and behavior cloning objective that pushes the student to both maximize return and mimic the teacher's s actions. The BC objective assumes the optimal student and teacher have the same action distribu- tions for it to be helpful.
This assumption does not always hold true and violated multiple of S3's environments, in- cluding Wrist Camera Pick-and-Place, Occluded Pick-and-Place and Blind Pick. The student's RI. objective can help with this imitation gap, but DreamerV3+BC is still outperformed by numerous other methods that exploit privileged information training time. We implement DreamerV3+BC on top of D3eamerV3.
To find the best balance between RL and BC objective for each task, try BC weighting values of 0.1, 0.01, 0.001 for all 10 tasks, and report PreamerV3+Be results with the best weight for each task.
Guided Observability Guided Observability implements a schedule for removing privileged in- formation during training. The policy is initially given both privileged and target observations, but as training progresses, the privileged information is dropped out with increasing probability. This dropout probability linearly increases with training iterations until only the target observations remain. We used 50% cutoff for access to privileged information for all tasks, but this hyperpa- rameter can realistically be tuned for each environment.
Guided Observability is algorithm agnostic but was implemented on top of DreamerV3 for this base- lines so as to remain as comparable as possible with Scaffolder and other Dreamer-based baselines. Our implementation closely nimicked PO-GRL (Weigand et al., [021), but had one small modi- fication. In PO-GRL, dropped-out observations are directly placed the replay buffer. For our implementation, however, only scaffolded observations (o are placed in the replay buffer, whether they were dropped out during the data collection process or not. Privileged information sampled from the replay buffer for world model training is then probabilistically dropped training schedule. This mitigates the risk of privileged information leaking into updates late in the training process. We choose simple linear annealing schedule that anneals towards 0 by 50% of training for all tasks.
the same
RMA RMA shares the same policy between teacher and student, and assumes that the privileged teacher training uses low-dimensional privileged system information as input. Therefore, it is not conceptually or practically easy to incorporate noisy, high dimensional observations into teacher training. So instead, we just give RMA advantage by giving it access to privileged low dimen- sional states instead of privileged observations. We use the RMA codebase from (Qi et al. 2023) to run our experiments.
Asymmetric Actor Critie (AAC) Asymmetric Actor Critic PPitoo et al. 2018) conditions the critic on scaffolded observations, while only conditioning the actor target observations. AAC assumes that providing privileged information to the critic will allow it to better estimate the value of states and thus, improve policy learning.
19

---

Published as a conference paper at ICLR 2024
AAC was implemented top of cleanrl's (Huang al. 2022b) implementation of PPO. To account for the relative sample inefficiency of PPO and model free algorithms general, a 100M sample budget given to AAC for each task.
E.4
SENSORY SCAFFOLDING SUITE
A two-fingered robot arm must pick up randomly initialized block using only propri- oception (gripper position) and touch sensing. During training, it can use multiple privileged RGB camera imputs (two static cameras and one wrist camera).
Blind Pick
Observation Space:
Privileged observations include three low-resolution RGB cameras: two static cameras face the workspace from the front and side, while one egocentric camera is located in the wrist.
of:
o-: The target observation space includes the gripper's position, velocity, and open/close state. Additionally, the rohot has two binary touch sensors on each gripper finger.
Action Space: Gripper velocity and gripping force.
Reward: Task reward for successfully picking up the block and auxiliary rewards computed using distances between the gripper, block, and picking goal positions.
Blind Locomotion In the Half Cheetah task, a proprioceptive policy must run and jump over randomly sized and positioned hurdles. During training, privileged RGB images of the Half Cheetah and its nearby hurdles are given, allowing the agent to see obstacles before bumping into them (see Figure 2
Observation Space:
oP: RGB camera that tracks the Half Cheetah as it moves.
o Joint angles and velocities.
Action Space: Torques for each of the moveable joints on Half Cheetah.
Reward Function: Reward proportional to the Half Cheetah's nowward-moving velocity with penalty for incurred control costs.
Blind and Deaf Piano A pair of 30-DoF Shadowhands the Robopianist simulator (Zakka et al., 2023) must learn play "Twinkle Twinkle Little Star" using only proprioception. At training time, the policy has access to future notes, piano key presses, and suggested fingerings, which emulates having vision to see sheet music and hearing to determine which keys were pressed.
Observation Space:
oP: Future notes for the next ten control steps, suggested fingerings for each set of notes, current piano key presses, previous action, and previous reward.
o-: Joint angles and forearm position for each Shadowhand.
Action Space: Desired joint angles and forearm position for each Shadowhand.
Reward Function: Task reward for playing correct notes with a penalty incurred for incorrect notes. Additionally, auxiliary rewards are provided encourage the fingers to stay close the keys and minimize energy.
Blind and Numb Pen A proprioceptive policy must control a 30-DoF Shadowhand to rotate a pen from randomized initial orientation randomized desired orientation. The target policy receives joint angles, the initial orientation the pen, and the desired goal orientation. During training, the policy has access the object pose and touch sensors.
Observation Space:
oº: Pen pose and dense touch sensing.
- Joint angles, initial pen pose, goal pen pose.
20

---

Published as a conference paper at ICLR 2024
Action Space:
Joint angles.
Reward Function: Dense positive reward proportional to the similarity between the current and target pen orientations. A negative penalty incurred for dropping the pen.
Blind and Numb Cube Similar to Blind and Numb Pen rotation, proprioceptive policy must control a 30-DoF Shadowhand to rotate a cube from a randomized initial orientation to a randomized desired orientation.
Observation Space:
aº: Cube pose and touch sensing.
o= Joint angles, initial cube pose, goal cube pose
Action Space:
joint angles.
Reward Function: Dense positive reward proportional to the similarity between the current and target cube orientations. A negative penalty is incurred for dropping the cube.
Noisy Monkey Bars A 13-link gibbon must swing between a fixed set of handholds in 2D environment using the brachiation simulator (Reda et al. 2022). Handholds are placed far enough apart such that the gibbon must use its momentum to swing and fly through the air to reach the next handhold. To simulate imperfect sensors the target observations, while privileged observations represent true simulator states. To improve policy learning in this difficult control task, the relative position of the model's center of mass from a provided reference trajectory is used as an auxiliary reward.
robotic platform, Gaussian noise is added to
Observation Space:
oP: Privileged observations include true simulator states, including the gibbon model's over- all velocity and pitch, sin and cos of each joint angle, joint velocities, height of the current arm from the model's center of mass, and whether each hand currently grabbing or re- leased. Additionally, the relative position to the next handhold and the relative distance to the reference trajectory are provided the policy.
o The target observation space is equivalent to the privileged observation space but with added Gaussian noise. Gaussian noise is applied to the true environment states rather than the observations to more accurately represent noisy sensors. For example, noise is added to joint angles rather than the sin and cos of those angles that the policy receives.
Action Space: Desired joint angle offsets for each joint the gibbon model and binary grab/release action for each hand.
Noise: Gaussian noise with mean and 0.05 standard deviation independently applied each simulator state component. Angles are measured in radians and distance is measured in meters. For reference, the gibbon 0.63m tall.
The reward function includes three terms: a style term encourage the model to stay relatively upright and minimize jerky motions, tracking term to encourage the model's center of mass to stay close the provided reference trajectory, and a sparse task reward term for each additional handhold reached.
Reward Function:
Wrist Camera Pick-and-Place This task examines the impact of multiple privileged 3rd- red-person cameras learning target oetive-perception policy with wrist camera input. Given the wrist cam- cra's 's restricted field view, the target policy must concurrently learn to move to locate the block and bin while executing the task. This task mirrors real-world scenarios such mobile manipulation, where a controllable wrist camera with a limited field view might be the sole visual sensor.
Observation Space:
-o: Two static, low-resolution RGB cameras facing the workspace from the front and side. Proprioceptive information including the gripper's position, velocity, and open/close state, touch sensing from two binary touch sensors located on each gripper finger, and vision from low-resolution RGB egocentric camera located the wrist of the robot.
o=:
Action Space: Gripper velocity and force.
21

---

Published as a conference paper at ICLR 2024
Reward Function: Sparse task reward for placing the block in the bin and auxiliary rewards com- puted using distances between the gripper, block, and bin positions.
Occluded Pick-and-Place Rather than employing active visual perception test time, this task examines the impact of active-perception as privileged sensing at training time. The targel policy must use an RGB camera from an occluded viewpoint alongside proprioception and touch sensing to pick up a block behind shelf and place it into bin. Both object and bin locations are ran- domly initialized. During training time, the robot gets access to privileged wrist mounted camera, enabling it to perform active visual perception to locate the block during training (see Figure 2)
Observation Space:
oP: Two low-resolution RGB cameras: one static, unoceluded camera facing the workspace from the side and one pereeepereeption camera located in the wrist of the robot.
--: One occluded low-resolution RGB camera. Although this camera does not reveal any information about the block, it does encode the position of the randomly placed bin.
Action Space: Gripper velocity and force.
Reward Function: Sparse task reward for placing the block in the bin and auxiliary rewards com- puted using distances between the gripper, block, and bin positions.
Visual Pen Rotation We carry over the setup from the previous two blind and numb dexterous manipulation tasks with privileged object pose and contact sensors. In this task, however, the target policy gains access to a top-down RGB image (see [Figure 5) to rotate a pen from a randomized initial orientation randomized desired orientation. At training time, the policy has access the pen pose and touch sensing.
Observation Space:
o": Pen pose and dense touch sensing.
o Joint angles, One top-down, low-rusolution RGB camera. desired pen pose
Action Space:
Desired joint angles.
Reward Function: Dense positive reward proportional to the similarity between the current and target pen orientations. A negative penalty is incurred for dropping the pen.
Visual Cube Rotation Similarly to Visual Pen Rotation, visual target policy conditioned on top-down RGB image and proprioception must rotate block to desired orientation.
Observation Space:
Cube pose and dense touch sensing.
o= Joint angles, topdown RGB camera, desired cube pose.
Action Space:
Desired joint angles.
Reward Function: Dense positive reward proportional to the similarity between the current and target cube orientations. A negative penalty is incurred for dropping the cube.
22

---

Published as a conference paper at ICLR 2024
F ADDITIONAL TD()) ERROR EXPERIMENTS.
Scaffolded TD(A) vs. Target TD(A)
Let the scaffolded TD(A) return be Equation (1) and the impoverished "target" TD(A) return be the original DreamerV3 policy objective, which substitutes the blue terms in Equation (1) with target dynamies P (2
Scaffolded Adventage
2 ae), target rewards P& (Té ), and target critic 2 (2 We compute the mean absolute error (MAE) for each of these two estimates against the ground truth Monte-Carlo return Rt.
We evaluate this in additional enviromments, Blind Navigation and Blind Locomotion. In Blind Navigation, randomly initialized agent must find a randomly ini- tialized goal point in 15x15 gridworld. The privileged
10
Epach
Scaffolded TD(A) Target TD(A)
125
observation the location of the goal in every epsiode, while the target observation is the x.y location of the
Scaltalded Advantage
100
agent. Every 25K steps while training, we collect 1000 transitions with the policy and compute Rt.
50-
plots Advantage (target return MAE scaf- folded return MAE). If the target error is higher than the scaffolded estimate's 's error, then Advantage is posi- tive. The advantages mostly positive, indicating that scaffolded TD(A) more accurate. This provides evi- dence for why Scaffolder trains better policies than non- privileged counterparts - its policy objective better ap- proximates the true expected return, giving the policy op- timizer a better training signal.
Figure 9
-25
10
20
30
Epoch
Figure 13: Top: Blind Nav., Bottom:
Blind Locomotion
G SCAFFOLDER WITH REWARDS ESTIMATED FROM PRIVILEGED SENSORS
To simulate realistic real world RL scenarios, we estimate reward signals from noisy sensors rather than using rewards computed from simulator state. We modify the Blind Pick environment's 's reward function to use estimated object pose rather than ground truth object pose, acquired via color-segmentation from the privileged cameras. The estimated object pose now has noise due to partial occlusions of the block by the robot gripper.
Blind Pick
200
Score
100
We train Scaffolder and other methods on the estimated reward, and evaluate the ground truth return. As seen in Figure 14, all meth- ods suffer some performance degradation. Scaffolder gets around 200 return whereas in the original Blind Pick it gets 300. However, the performance trends are consistent. Looking at policy rollouts, find that only Scaffolder can reliably pick up the block and all other methods fail.
0
Steps
1e5
Figure 14: Scaffolder still outperforms baselines in the estimated reward setting.
23

---

Published as a conference paper at ICLR 2024
H Scaffolder ON PRE-EXISTING ENVIRONMENTS
We ran Scaffolder and DreamerV3 (operating only on target input, consistent with our other results) on other environments from the COSIL Nguyen al. (2022) paper, Bumps-2D and Car-Flag.
In Bumps-2D, a robot arm must find the bigger bump out of a pair of randomly initialized bumps in 2D grid, using only proprioception and touch sensing. In Car-Flag, a using just proprioception must find randomly initialized green flag. If the car finds blue flag, the green flag position is revealed, In both environments, the state expert can directly go the bigger bump / green flag, while the student policy must perform information gathering actions to solve the task.
Scaffolder
DreamerV3
1.0
1.0
0.8
Bumpe-2D
0.6
Car- Flae
0.0
25k
SGK
75k
100k
325K
150k
COSIL
SAC
ADV
DAgger
BC+SAC
EC2SAC
VAM
Random
State Expert
Figure 15: Performance of Scaffolder and DreamerV3 on two environments from COSIL.
new results are consistent with those reported in our paper on the S3 tasks: Scaffolder clearly outperforms DreamerV3, further establishing the versatility of our approach for utilizing privileged sensory information.
As seen in Figure 15
In addition, these experiments also enable new comparisons with the prior work evaluated in COSIL. Note that COSIL and its baselines operate in a slightly different setting from Scaffolder: they as- sume access perfect, hand-coded scaffolded policy, whereas Scaffolder trains both target and scaffolded policies from scratch, and in tandem, starting at environment step 0. This means that, when comparing learning curves, must remember that COSIL effectively "starts ahead" of Seaf- folder. Even with this handicap, Scaffolder easily outperforms COSIL on Bumps-2D, and matches Car-Flag. These results COSIL's own evaluation environments show further evidence of Scaffolder 's improvements over prior state of the art.
24


Reviews:
[{"review_id": "a23ywWAMgM", "reviewer": "Reviewer_xL3j", "summary": "This paper proposes Scaffolder, a MBRL method that extends DreamerV3 with privileged information in its modules. Scaffolder uses privileged world models and exploration policies to roll-out trajectories to train a better target policy. To ensure consistency between target and privileged latent, Scaffolder proposes to predict target latent from privileged latent, bottlenecked by target observation. Scaffolder outperforms baselines on the newly proposed S3 benchmark.", "questions": "- Real-world applications. Please see details in the weaknesses section above.\n- I am curious about one particular design choice. Why do the authors choose to predict target latent from privileged latent, bottlenecked by target observation. Why not usethe same latent, shared by both the privileged and target modules?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ The paper is well written and motivated. The presentation is clear.\n+ Strong empirical performance.", "weaknesses": "- I agree that it makes sense to evaluate the proposed method on the newly proposed benchmark, for motivations mentioned in the paper. However, the paper would still benefit from evaluating extra existing benchmarks, just for reference. \n- One major benefit of privileged information reinforcement learning is to train the target policy with privileged information in simulation, and deploy it in the real world where there is no privileged information. However, all experiments in the paper are purely in sim. Can the authors comment more on how well the presented approach will work in real-world applications?\n- In addition to the number of frames being the x-axis for figure 6, please also include one where x-axis is the wall-clock time. This way the community will have a better understanding of how the proposed method and baselines work on this particular environment set.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VLWdgtgeLN", "reviewer": "Reviewer_V3Vw", "summary": "This work proposes to utilize privileged sensory information to improve every component of model-based reinforcement learning, including world model, exploration policy, critic, and representation as well. This work provides extensive evaluation over 10 environments including different kinds of sensory data, showing the proposed method outperform all representative baselines. This work also provide detailed ablation study over all environments showing the", "questions": "1. Refer to weakness. \n2. Regarding some technical details, is a bit confusing:\n* In section C1, it says “We launch 4-10 seeds for each method”, what’s the exact meaning of using different number seeds across methods or across environments?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This work provides systematic analysis over different components in the “sensory scaffolding” setting, and proposes corresponding scaffolding counterparts  of every component in MBRL, except the policy during deployment. \n2. This work provides a promising evaluation comparison with multiple representative baselines, demonstrating that with the proposed pipeline, privilege information improves the sample efficiency as well as the final performance over wide-range of tasks.\n3. Through ablation study, this work shows different components in the system boost the performance in a different way, providing additional insight on how privileged information can be used in the future work.\n4. Experiment details are well presented in the Appendix, including runtime and resource comparison over different methods on different environments.\n5. The overall presentation of the work is good, considering the complexity of the system and amount of information delivered.", "weaknesses": "1. For scaffolded TD error comparison, it’s not clear why the comparison is conducted on Blind pick environment, since the gap between the proposed method and the version without scaffolded critic is much larger (at least in terms of relative gap) on Blind Cube Rotation environment. Also it would be great to see whether the estimate is close for tasks like Blind Locomotion (since the gap is small on that task). It seems there is some obvious pattern in the Figure 9, that the scaffolded TD is worse at 5, 10, 15 epoch and performs best on 7, 12, 18 epoch, it would be great to have some explanation for that.\n2. For some claims made in the paper, it’s actually not quite convincing. For “In other words, much of the gap between the observations o− and o+ might lie not in whether they support the same behaviors, but in whether they support learning them.”, some additional visualization like trajectory visualization might be helpful to strengthen the claim, since the similar reward score does not necessarily result in similar behavior. \n3. For runtime comparison, since the speed of given GPUs varies a lot, it might be better to compare the wall-time with similar system configuration, assuming the wall-time is consistent across different seeds.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LocwyzTRe6", "reviewer": "Reviewer_bAbn", "summary": "The learning process known as \"sensory scaffolding\" involves novice learners using more sensory inputs than experts. This principle has been applied in this study to train artificial agents. The researchers propose \"Scaffolder,\" a reinforcement learning method that utilizes privileged information during training to optimize the agent's performance.\n\nTo evaluate this approach, the researchers developed a new \"S3\" suite of ten diverse simulated robotic tasks that require the use of privileged sensing. The results indicate that Scaffolder surpasses previous methods and frequently matches the performance of strategies with continuous access to the privileged sensors.", "questions": "1. More clarification on the posterior and embedding component in the method part. \n2. More clarification of other scaffolding methods in the related work, no need for any experiments.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "This paper delves into a critical question within the field of reinforcement learning: how can we effectively use privileged information as a 'scaffold' during training, while ensuring the target observation remains accessible during evaluation? This question takes on an added significance in robotic learning, where simulation is a major data source.\n\nWhile there has been considerable research in this area, as detailed in the related work, this paper adds value to the existing body of knowledge, even without introducing novel methods. The proposed method may not be groundbreaking, but it offers a comprehensive examination of this issue from four perspectives: model, value, representation, and exploration.\n\nThis research serves as a valuable resource for those looking to deepen their understanding of the field. The excellent writing and presentation of this paper further enhance its contribution. Overall, despite the lack of methodological novelty, the paper is worthy of acceptance due to its systematic exploration and clear articulation of the subject matter.", "weaknesses": "1. Increasing the clarity around the Posterior and detailing how it is used to transition from the privileged latent state to the non-privileged latent state would greatly enhance understanding of the method.\n   \n2. The related work section could be expanded to include research papers that leverage privileged simulation reset to improve policy. These works also seem to align with the scaffolding concept presented in this paper. Papers such as [1][2] could be added for reference.\n\n3. In the experimental design, the wrist camera and touch features don't appear to be excessively privileged or substantially different from the target observations. It would be beneficial to experiment with more oracle-like elements in the simulator as privileged sensory inputs. For instance, the oracle contact buffer or geodesic distance could be considered.\n\n\n[1] DeepMimic: Example-Guided Deep Reinforcement Learning of Physics-Based Character Skills\n\n[2] Sequential Dexterity: Chaining Dexterous Policies for Long-Horizon Manipulation", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "G3h8JS8rxP", "reviewer": "Reviewer_Sbzi", "summary": "The authors propose Scaffolder, a model-based RL method that can leverage privileged information at training time. Here, priviledged information means an MDP where:\n\n* There is some true state $s$ with observations $o^+$, where $o^+$ includes privileged info we do not want to assume is available at inference time. (i.e. ground truth object state). This could be used to train a privileged policy $\\pi^+$ that cannot be used as-is for inference.\n* There is an observation $o^-$ for unprivileged / impoverished target observations, which our final policy $\\pi^-$ will depend on.\n* We would like the best $\\pi^-$ possible while leveraging information in $o^+$.\n\nThis problem has been studied in a number of recent works, often in a model free manner. This paper aims to leverage privileged information in a model-based manner.\n\nTo do so, the authors train 2 worlds models. The world model subroutine used is DreamerV3. One models privileged information $o^+$ and the other models target information $o^-$. In this summary I'll call them WM+ and WM-.\n\nA \"latent translator\" is learned to translate WM+ into an observation that WM- can use in its rollouts. Specifically: WM+ has internal latent state $z^+$. We fit a prediction model $p(e^-|z^+)$, where $e^- \\approx emb(o^-)$. Part of DreamerV3 is learning a posterior $q(z_{t+1}|z_t,a_t,e_{t+1}=emb(o_{t+1}))$ that infers latent state from history and current observation. By replacing the impoverised $e^- = emb(o^-)$ with a prediction driven by privileged latent $z^+$, we can channel some privileged information into the rollout of $z^-$, assuming that privileged information is eventually observable in unprivileged information.\n\nThis latent translator lets us use $\\pi^-(a|z^-)$ to rollout both WM+ and WM-, giving a sequence of latents $(z^+,z^-)$ from both world models. The learned reward function is then defined as $R(z^+,z^-)$ to allow observing privileged information in the critic.\n\nWe additionally fit a $\\pi^+$ directly in the privileged world model, using this solely to generate additional exploratory data (it is possible that some exploration behaviors are easier to learn or discover from privileged information). Last, a decoder is trained to map $z^-$ to $o^+$. To me this seems the least motivated, in that not all parts of $o^+$ should be predictable from $z^-$ in the first place, but it seems to empirically be effective.\n\nThe evaluation of Scaffolder is done in a variety of \"sensory blindfold\" tasks, mostly robotics based, where some sensors are defined as privileged and some are not. The method is compared to DreamerV3 on just target information, a few variants of DreamerV3 based on only fitting one world model with decoding of privileged information, and some model free baselines like slowly decaying use of privileged information, asymmetric actor critic, or using BC to fit an unprivileged policy to a privileged one.", "questions": "In general, the paper focuses on the online, tabula rasa case where we are in an entirely unfamiliar environment. How adaptable is this method to either the offline case, or the finetuning case where we have an existing policy / world model?", "limitations": "", "rating": 10, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "The paper provides a good overview of previous methods for handling privileged information, proposes an evaluation suite for studying the problem of privileged information, and proposes a modification of DreamerV3 that handles the information better than Informed Dreamer. There is a significant amount of machinery around Scaffolder, but it's mostly clear why the added components ought to be helpful for better world modeling and policy exploration. The model-free baselines used are pretty reasonable, and it is shown that Scaffolder still outperforms these model free methods even when the model free methods are given significantly more steps.\n\nFinally, the evaluation suite covers a wide range of interesting robot behaviors and the qualitative exploratory methods discovered to handle the limited state (i.e. spiraling exploration behavior) are quite interesting. The S3 suite looks like a promising testbed for future privileged MDP work, separate from the algorithmic results of the paper.", "weaknesses": "In some sense, Scaffolder requires doing 2x the world model fitting, as both the z^- and z^+ models need to be fit for the approach to work. In general, this is \"fair\" for model-based RL, which is usually judged in number of environment interactions rather than number of gradient steps, but it very definitely is a more complex system and this can introduce instability.\n\nA common actor-critic criticism is that the rate of learning between the actor and critic needs to be carefully controlled such that neither overfits too much to the other. Scaffolders seems to take this and add another dimension for the rate of learning between the privileged world model and unprivileged one, as well as the learning speed of the privileged exploratory actor $\\pi^+$ and target actor $\\pi^-$.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)