Kati8/ICLR_2023_2025 huggingface dataset:
authors: Jiajian Li
authors: Qi Wang
authors: Yunbo Wang
authors: Xin Jin
authors: Yang Li
authors: Wenjun Zeng
authors: Xiaokang Yang
reviews: 
Comments: This paper proposes a long short-term world model to address the limited imagination horizon of traditional world models.
The authors leverage MineCLIP to detect task goals and generate intrinsic rewards, enabling the model to jump directly to these goal states when feasible.
For policy optimization over these jump-based transitions, they train modules to predict future value, interval timesteps, and cumulative rewards.

The paper is well-organized, and its key contributions are:
(1) It introduces a method for generating target states in MineDojo (or possibly in other 3D-RPG games).
(2) It demonstrates the feasibility of training a world model with jumping transitions and optimizing policies over such transitions.
Strengths: 1. It introduces a method for generating target states in MineDojo (or possibly in other 3D-RPG games).
2. It demonstrates the feasibility of training a world model with jumping transitions and optimizing policies over such transitions.
3. The illustration is clear and the writing organization is good.
Weaknesses: 1. Since MineCLIP is an important tool for this work, I suggest the author include a brief introduction of what MineCLIP can do in section 3.2/3.2.1 or an appropriate position. This would help readers who are not familiar with research on MineDojo to understand this paper.

2. In the abstract, "We argue that the primary obstacle in open-world decision-making is improving the efficiency of off-policy exploration across an extensive state space." This seems not closely connected to the main contribution of this paper. I suggest paraphrasing it to highlight "across a long horizon" or something that is more related to the topic.

3. Though the method sounds promising for solving MineDojo tasks, it may not be a general method for all kinds of open-world games. Such as in 2D games or fixed camera control tasks.

    (a)  Before seeing the target for the first time in the training process, there won't be a reasonable goal-directed reward or jumping option, the exploration still requires extensive enumerates.

    (b) The crop and resize operation (or assumption) is only useful for 3D visual navigation tasks.

    (c) When the target is occluded, the world model still needs to perform step-by-step imagination.

If these are true, I suggest the authors include a sentence or so in the limitation section to clarify it.
Recommendation: 8
Reviewer confidence: 4
Soundness/correctness: 3
Clarity: 3
reviews: 
Comments: This paper studies Model-based RL in Open-World environment, specifically Minecraft. The authors propose a long short-term imagination method to improve the imagination process of the dreamer algorithm. The main idea is to use an affordance map to identify the distance object of interest and train a long-term transition model to skip the approaching steps so that the imagination can be more task-relevant. Results from 5 tasks in Minecraft showcase the improvement of the method over Dreamerv3 and other two baselines.
Strengths: - The paper is mostly well written. 
- The method proposed is novel and the results are promising comparing to the baselines.
Weaknesses: - Although the high-level idea is straight-forward, the implementation is overcomplicated. 
- The method feels very ad-hoc to the Minecraft tasks studied in this paper. It doesn't come into my mind about any other relevant tasks other than Minecraft where the proposed method can be applied.
Recommendation: 8
Reviewer confidence: 4
Soundness/correctness: 3
Clarity: 3
reviews: 
Comments: The paper introduces a novel hierarchical model-based reinforcement learning (MBRL) framework named Long Short-Term Imagination (LS-Imagine), designed to address the challenge of short-sightedness in open-world decision-making with high-dimensional visual inputs, such as MineDojo. In LS-Imagine, imagination horizon can be extended with a specialized world model. Specifically, a affordance maps are predicted to guide the jumpy switch. Two world models are trained to capture transitions at different temporal resolutions, short-term state transition and long-term state transition. Agent learning is conducted during imagination, a common strategy utilized in most background planning method.  The authors provided experimental results on Harvest tasks from the MineDojo benchmark, showed superior results compared with baseline methods.
Strengths: 1. The jumpy prediction technique within the long-term imagination framework is innovative as it departs from the fixed interval approach prevalent in previous work, offering increased flexibility in jumpy prediction
2. The paper is well-organized and clearly written.
Weaknesses: 1. The proposed method employs a hierarchical structure, yet the baseline comparisons are made with flat learning methods. Including comparisons with hierarchical MBRL methods like Director[1] could greatly strengthen the paper.
2. Equation 9 appears to have an inconsistency in the time indexing; should the bootstrapping term $R^\lambda_{t+1}$ be $R^\lambda_{t+\hat{\Delta}_{t+1}+1}$ ? 
3. The use of  $\lambda$ -return in evaluating the policy might introduce bias since it should be evaluated with on-policy data,, but the predicted jumpy state, $\hat{z}_{t+1}$, might not aligned with the learning policy.
4. The paper focuses on Harvest tasks. Including results from other complex, long-horizon tasks, such as the Tech Tree task group from MineDojo, would better demonstrate the framework’s effectiveness.

[1]: Hafner, Danijar, et al. "Deep hierarchical planning from pixels." Advances in Neural Information Processing Systems 35 (2022): 26091-26104.
Recommendation: 8
Reviewer confidence: 3
Soundness/correctness: 2
Clarity: 3