Kati8/ICLR_2023_2025 huggingface dataset:
authors: Edward S. Hu
authors: James Springer
authors: Oleh Rybkin
authors: Dinesh Jayaraman
reviews: 
Comments: The learning process known as "sensory scaffolding" involves novice learners using more sensory inputs than experts. This principle has been applied in this study to train artificial agents. The researchers propose "Scaffolder," a reinforcement learning method that utilizes privileged information during training to optimize the agent's performance.

To evaluate this approach, the researchers developed a new "S3" suite of ten diverse simulated robotic tasks that require the use of privileged sensing. The results indicate that Scaffolder surpasses previous methods and frequently matches the performance of strategies with continuous access to the privileged sensors.
Strengths: This paper delves into a critical question within the field of reinforcement learning: how can we effectively use privileged information as a 'scaffold' during training, while ensuring the target observation remains accessible during evaluation? This question takes on an added significance in robotic learning, where simulation is a major data source.

While there has been considerable research in this area, as detailed in the related work, this paper adds value to the existing body of knowledge, even without introducing novel methods. The proposed method may not be groundbreaking, but it offers a comprehensive examination of this issue from four perspectives: model, value, representation, and exploration.

This research serves as a valuable resource for those looking to deepen their understanding of the field. The excellent writing and presentation of this paper further enhance its contribution. Overall, despite the lack of methodological novelty, the paper is worthy of acceptance due to its systematic exploration and clear articulation of the subject matter.
Weaknesses: 1. Increasing the clarity around the Posterior and detailing how it is used to transition from the privileged latent state to the non-privileged latent state would greatly enhance understanding of the method.
   
2. The related work section could be expanded to include research papers that leverage privileged simulation reset to improve policy. These works also seem to align with the scaffolding concept presented in this paper. Papers such as [1][2] could be added for reference.

3. In the experimental design, the wrist camera and touch features don't appear to be excessively privileged or substantially different from the target observations. It would be beneficial to experiment with more oracle-like elements in the simulator as privileged sensory inputs. For instance, the oracle contact buffer or geodesic distance could be considered.


[1] DeepMimic: Example-Guided Deep Reinforcement Learning of Physics-Based Character Skills

[2] Sequential Dexterity: Chaining Dexterous Policies for Long-Horizon Manipulation
Recommendation: 8: accept, good paper
Reviewer confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness/correctness: 3 good
Clarity: 4 excellent
reviews: 
Comments: This work proposes to utilize privileged sensory information to improve every component of model-based reinforcement learning, including world model, exploration policy, critic, and representation as well. This work provides extensive evaluation over 10 environments including different kinds of sensory data, showing the proposed method outperform all representative baselines. This work also provide detailed ablation study over all environments showing the
Strengths: 1. This work provides systematic analysis over different components in the “sensory scaffolding” setting, and proposes corresponding scaffolding counterparts  of every component in MBRL, except the policy during deployment. 
2. This work provides a promising evaluation comparison with multiple representative baselines, demonstrating that with the proposed pipeline, privilege information improves the sample efficiency as well as the final performance over wide-range of tasks.
3. Through ablation study, this work shows different components in the system boost the performance in a different way, providing additional insight on how privileged information can be used in the future work.
4. Experiment details are well presented in the Appendix, including runtime and resource comparison over different methods on different environments.
5. The overall presentation of the work is good, considering the complexity of the system and amount of information delivered.
Weaknesses: 1. For scaffolded TD error comparison, it’s not clear why the comparison is conducted on Blind pick environment, since the gap between the proposed method and the version without scaffolded critic is much larger (at least in terms of relative gap) on Blind Cube Rotation environment. Also it would be great to see whether the estimate is close for tasks like Blind Locomotion (since the gap is small on that task). It seems there is some obvious pattern in the Figure 9, that the scaffolded TD is worse at 5, 10, 15 epoch and performs best on 7, 12, 18 epoch, it would be great to have some explanation for that.
2. For some claims made in the paper, it’s actually not quite convincing. For “In other words, much of the gap between the observations o− and o+ might lie not in whether they support the same behaviors, but in whether they support learning them.”, some additional visualization like trajectory visualization might be helpful to strengthen the claim, since the similar reward score does not necessarily result in similar behavior. 
3. For runtime comparison, since the speed of given GPUs varies a lot, it might be better to compare the wall-time with similar system configuration, assuming the wall-time is consistent across different seeds.
Recommendation: 8: accept, good paper
Reviewer confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness/correctness: 3 good
Clarity: 3 good
reviews: 
Comments: This paper proposes Scaffolder, a MBRL method that extends DreamerV3 with privileged information in its modules. Scaffolder uses privileged world models and exploration policies to roll-out trajectories to train a better target policy. To ensure consistency between target and privileged latent, Scaffolder proposes to predict target latent from privileged latent, bottlenecked by target observation. Scaffolder outperforms baselines on the newly proposed S3 benchmark.
Strengths: + The paper is well written and motivated. The presentation is clear.
+ Strong empirical performance.
Weaknesses: - I agree that it makes sense to evaluate the proposed method on the newly proposed benchmark, for motivations mentioned in the paper. However, the paper would still benefit from evaluating extra existing benchmarks, just for reference. 
- One major benefit of privileged information reinforcement learning is to train the target policy with privileged information in simulation, and deploy it in the real world where there is no privileged information. However, all experiments in the paper are purely in sim. Can the authors comment more on how well the presented approach will work in real-world applications?
- In addition to the number of frames being the x-axis for figure 6, please also include one where x-axis is the wall-clock time. This way the community will have a better understanding of how the proposed method and baselines work on this particular environment set.
Recommendation: 8: accept, good paper
Reviewer confidence: 2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness/correctness: 3 good
Clarity: 3 good