reviews: 
Comments: The authors propose Scaffolder, a model-based RL method that can leverage privileged information at training time. Here, priviledged information means an MDP where:

* There is some true state $s$ with observations $o^+$, where $o^+$ includes privileged info we do not want to assume is available at inference time. (i.e. ground truth object state). This could be used to train a privileged policy $\pi^+$ that cannot be used as-is for inference.
* There is an observation $o^-$ for unprivileged / impoverished target observations, which our final policy $\pi^-$ will depend on.
* We would like the best $\pi^-$ possible while leveraging information in $o^+$.

This problem has been studied in a number of recent works, often in a model free manner. This paper aims to leverage privileged information in a model-based manner.

To do so, the authors train 2 worlds models. The world model subroutine used is DreamerV3. One models privileged information $o^+$ and the other models target information $o^-$. In this summary I'll call them WM+ and WM-.

A "latent translator" is learned to translate WM+ into an observation that WM- can use in its rollouts. Specifically: WM+ has internal latent state $z^+$. We fit a prediction model $p(e^-|z^+)$, where $e^- \approx emb(o^-)$. Part of DreamerV3 is learning a posterior $q(z_{t+1}|z_t,a_t,e_{t+1}=emb(o_{t+1}))$ that infers latent state from history and current observation. By replacing the impoverised $e^- = emb(o^-)$ with a prediction driven by privileged latent $z^+$, we can channel some privileged information into the rollout of $z^-$, assuming that privileged information is eventually observable in unprivileged information.

This latent translator lets us use $\pi^-(a|z^-)$ to rollout both WM+ and WM-, giving a sequence of latents $(z^+,z^-)$ from both world models. The learned reward function is then defined as $R(z^+,z^-)$ to allow observing privileged information in the critic.

We additionally fit a $\pi^+$ directly in the privileged world model, using this solely to generate additional exploratory data (it is possible that some exploration behaviors are easier to learn or discover from privileged information). Last, a decoder is trained to map $z^-$ to $o^+$. To me this seems the least motivated, in that not all parts of $o^+$ should be predictable from $z^-$ in the first place, but it seems to empirically be effective.

The evaluation of Scaffolder is done in a variety of "sensory blindfold" tasks, mostly robotics based, where some sensors are defined as privileged and some are not. The method is compared to DreamerV3 on just target information, a few variants of DreamerV3 based on only fitting one world model with decoding of privileged information, and some model free baselines like slowly decaying use of privileged information, asymmetric actor critic, or using BC to fit an unprivileged policy to a privileged one.
Strengths: The paper provides a good overview of previous methods for handling privileged information, proposes an evaluation suite for studying the problem of privileged information, and proposes a modification of DreamerV3 that handles the information better than Informed Dreamer. There is a significant amount of machinery around Scaffolder, but it's mostly clear why the added components ought to be helpful for better world modeling and policy exploration. The model-free baselines used are pretty reasonable, and it is shown that Scaffolder still outperforms these model free methods even when the model free methods are given significantly more steps.

Finally, the evaluation suite covers a wide range of interesting robot behaviors and the qualitative exploratory methods discovered to handle the limited state (i.e. spiraling exploration behavior) are quite interesting. The S3 suite looks like a promising testbed for future privileged MDP work, separate from the algorithmic results of the paper.
Weaknesses: In some sense, Scaffolder requires doing 2x the world model fitting, as both the z^- and z^+ models need to be fit for the approach to work. In general, this is "fair" for model-based RL, which is usually judged in number of environment interactions rather than number of gradient steps, but it very definitely is a more complex system and this can introduce instability.

A common actor-critic criticism is that the rate of learning between the actor and critic needs to be carefully controlled such that neither overfits too much to the other. Scaffolders seems to take this and add another dimension for the rate of learning between the privileged world model and unprivileged one, as well as the learning speed of the privileged exploratory actor $\pi^+$ and target actor $\pi^-$.
Recommendation: 10: strong accept, should be highlighted at the conference
Reviewer confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness/correctness: 3 good
Clarity: 4 excellent