World2Minecraft: Occupancy-Driven Simulated Scenes Construction
Authors: Lechao Zhang
Haoran Xu
Jingyu Gong
Xuhong Wang
Yuan Xie
Xin Tan
Venue: ICLR 2026 Poster (2026)
Keywords: Embodied AI; World2Minecraft; 3D Semantic Occupancy Prediction; MinecraftOcc Dataset; Vision-Language Navigation
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Embodied intelligence requires high-fidelity simulation environments to support perception and decision-making, yet existing platforms often suffer from data contamination and limited flexibility. To mitigate this, we propose World2Minecraft to convert real-world scenes into structured Minecraft environments based on 3D semantic occupancy prediction. In the reconstructed scenes, we can effortlessly perform downstream tasks such as Vision-Language Navigation(VLN). However, we observe that reconstruction quality heavily depends on accurate occupancy prediction, which remains limited by data scarcity and poor generalization in existing models. We introduce a low-cost, automated, and scalable data acquisition pipeline for creating customized occupancy datasets, and demonstrate its effectiveness through MinecraftOcc, a large-scale dataset featuring 100,165 images from 156 richly detailed indoor scenes. Extensive experiments show that our dataset provides a critical complement to existing datasets and poses a significant challenge to current SOTA methods. These findings contribute to improving occupancy prediction and highlight the value of World2Minecraft in providing a customizable and editable platform for personalized embodied AI research. We will publicly release the dataset and the complete generation framework to ensure reproducibility and encourage future work.

Published as a conference paper at ICLR 2026
WORLD2MINECRAFT: OcCUPANCY-DRIVEN SIMU- LATED SCENES CONSTRUCTION
Lechao Zhang', Haoran Xu', Jingyu Gong', Xuhong Wang", Yuan Xie Xin Tan 12,+ "Shooo of Computer Science and Technology, East China Normal University 2 "hanggaai Artificial Intelligence Laboratory Shanghai Innovation Institute
($1275901049, 10235102421 (alaa5102227]estttt ecnsuu.ecnu..
wangxxhhngg@pjlab org. cn, (jygong, yxie, xtan) ecnu, edu. cn
ABSTRACT
Embodied intelligence requires high-fidelity simulation environments to support perception and decision-making yet existing platforms often suffer from data con- tamination and limited flexibility. To mitigate this, we propose World2Minecraft to convert real-world scenes into structured Minecraft environments based on 3D semantic occupancy prediction. In the reconstructed scenes, we can effortlessly perform downstream tasks such as Vision-Language observe that reconstruction quality heavily depends accurate occupaney prediction, which remains limited by data searcity and poor generalization existing models. We introduce low-cost, automated, and scalable data acqui- sition pipeline for creating customized occupancy datasets, and demonstrate its effectiveness through MinecraftOce, large-scale dataset featuring 100,165 im- ages from 156 richly detailed indoor scenes. Extensive experiments show that our dataset provides critical complement existing datasets and poses a sigmificant challenge to current SOTA methods. These findings contribute to improving occu- pancy prediction and highlight the value of World2Mineeraff in providing a cus- tomizable and editable platform for personalized embodied Al research. Project page: https tssspsst : //or d22m www.mmiinnirff.cmmmmmmmm waazrafttttthaaa github..bbi io/.
Aavigation(VLN)) However,
INTRODUCTION
Embodied intelligence aims to develop intelligent agents that can perceive, understand, and interact within complex environments. Progress in this field depends critically on the availability of high- fidelity, diverse simulation environments supported by robust datasets. Platforms like Habitat (Savva et al., 2019) is limited by their reliance on real-world scans, which not only yield scenes with visual and geometric artifacts but uneditable, limiting their utility for agents that need modify their environment. Minecraft is widely used for reinforcement learning (Cai et al., 2023; Li et al., 2025; Cai et al., 2024b; Zheng et al., 2025a; Cai et al., 2025b) for its customizable environments. However, its blocky graphics create stark reality gap despite recent domain inviiiiannaiiatt prompting efforts (Zhao al., 2024). These limitations highlight the need for new simulation platforms that are both highly flexible and editable, and also capable of maintaining visual realism.
Real-to-sim transfer presents an effective approach for this goal. However, current techniques like NeRF (Mildenhall et al., 2021) and 3D Gaussian Splatting (Kerbl et al., 2023; Ji et al., 2025; Tian al., 2025) often yield photorealistic views but uneditable representations that lack physical proper- ties. Similarly, CAD-based methods (Avetisyan al., 2019; Gümeli et al., 2022; Tyszkiewicz et al., 2022; Murali et al., (017) yield clean scenes but require precise instance segmentation and can not be directly used for downstream tasks. To reconcile realism with interactability, we utilize 3D semantic occupancy (Cao & De Charette, 2022). Unlike implicit fields, its discrete voxel structure naturally aligns with Minecraft blocks. This compatibility enables the direct translation of real-world scenes into editable, physically grounded environments, bypassing complex mesh-to-block conversions.
Inspired by this, propose World2Minecraft, framework that reconstructs real-world scenes as high-quality Minecraft environments by leveraging 3D semantic occupancy prediction as shown in Figure In contrast to existing methods, our approach is cost-effective, yields readily ed- itable scenes, and is directly applicable to downstream tasks such as Vision-Language Navigation
Coorrepoondigg author.

---

Published as a conference paper at ICLR 2026
(VLN) (Anderson et al., 2018; Gu et al., 2025). The framework operates by first predicting single- frame 3D semantic occupancy, then integrating multi-frame observations via camera parameters (Wu et al., 2024) to build unified semantic occupancy field of the complete scene. The resulting rep- resentation can be refined via a developed visual tool(as shown in Appendix J) before generating construction instructions for Minecraft. Executing these instructions faithfully reproduces the high- fidelity scene in Minecraft.
After reconstructing real-world scenes in Minecraft, we conducted extensive experiments on down- stream VLN tasks. To this end, we constructed MinecraftVLNN, a dataset composed of 1,059 samples from our reconstructed scenes, augmented with 2,483 additional samples from community-cres crreate large-scale scenes to increase scale and diversity. We defined two subtasks -Nxt- View and Next- Action-and and fine-tuned VWL-3. (Bai et al., 2025) and Owen2.5-VLL-77 (Wang et al., 2024) on each, achieving notable performance gains. Real-time navigation was successfully demonstrated by employing Gemini-2.5-Prr (Comanici et al., 2025) as the controller for an agent in the recon- structed Minecraft environments. However, the reconstruction quality remained sub-optimal for practical use. We identified that accurate semantic occupancy prediction is critical to reconstruction fidelity and scalability, yet it faces two major limitations: (1) heavy reliance large-scale, expen- sively annotated data (?), and (2) dataset constraints such as limited diversity, poor coverage, and sensor noise (Liu al., 2023), which hinder model generalization in complex real-world scenarios.
To advance the generalization of scene occupancy prediction, we introduce a novel, low-cost, and automated pipeline for generating customized semantic oceupancy datasets, which significantly reduces the traditional reliance on expensive manual annotation or limited real-world scans. We demonstrate its effectiveness through MinecraftOce, large-scale dataset produced by this pipeline, featuring 100,165 high-resolution images captured from continuous roomtour across 156 richly de- tailed indoor scenes constructed in Minecraft. By leveraging mods for physically- -based rendering and precise layout control, we automatically generate visually realistic environments with complex structures, diverse objects, and dynamic lighting, effectively narrowing the sim-to-real gap. Ex- periments show that current occupancy models perform poorly on MinecraftOce, revealing clear generalization limits. Moreover, when used as auxiliary training data, it enhances performance on real-world benchmarks like NYUv2(Silbermaa ing benchmark and an effective data augmentation resource for improving model robustness. In summary, the main contributions of this paper
al., 2012), confirming its dual role as a challeng-
follows:
We introduce World2Minecrafi, pipeline for real-world scene reconstruction in Minecraft via semantic occupancy prediction.
We conduct VLN task within the reconstructed scenes, during which we construct the MinecraftVLNN dataset to validate the practical utility of our approach.
We propose an automated pipeline for semantic occupancy data generation, and present the large-scale MinecraftOce benchmark, which exposes the generalization limits of existing methods and serves as effective training data for enhancing robustness.
RELATED WORK
2.1 DATA-DRIVED 3D SCENE GENERATION
In embodied intelligence research, real-to-sim lated Dnvironments-remaie a critical yet challenging task. Generative approaches, such as 3D- GPT (Sun et al., 2025a), SceneCraft (Yang al., 2024) and Styleshot (Gao al., 2025), excel in creating novel content from abstract inputs but not designed to faithfully reconstruct specific, existing real-world locations. While WonderWorld (Yu al., 2025) generates 3D worlds from single image but lack the semantic decomposability and editability. Similarly, CAD-based meth- ods (Avetisyan et al., 2019; Gümeli et al., 2022; Tyszkiewiez et al., 2022; Murali et al., 2017) yield clean and lightweight scene representations. However, they rely heavily on precise instance segmen- tation and accurate scale alignment between the retrieved CAD models and the real-world scene, which hinders their direct application in downstream tasks. Recent methods like LiteReality (Huang et al., 2025) simplify real-to-virtual conversion, yet remain limited in object and scene diversity, and cannot be directly used for downstream tasks. We propose low-cost and easily editable method for real-to-sim conversion by leveraging occupancy prediction, enabling the direct application of reconstructed environments to downstream tasks like VIN in Minecraft.
real-world scenes into simu-
2

---

Published as a conference paper at ICLR 2026
1) Transfer Reality Minecraft
Post process
The Reconstructed Minecraft Scene
W
MINESSAFT
Mono Occupancy prediction
O
kethlock - block1
minecroft
Og
kethlock - block
sciblock block4
Embodied Occupanccy prediction
Setblock blocks
2) Vision Language Novigation
Instrruction>
Proceed from your current location throghh the main
door, find refrigerator living room..
Next Niew
<Histery>
Current-view>
Reality Minecraft
move forward V
<Colcee>
<Choloe>
B. turn left
right
stop.
View
View
Figure I: Framework of World2Minecraff, which illustrates the process of reconstructing real- world scenes into Minecraft environments and conducting navigation within these scenes. 1) For the transfer reality Minecraft, RGB images are input into the occupancy prediction model, which is then postprocessed to generate instructions for reconstruction Minecraft. 2) VLN tasks involving Next-Vien and Next-Action performed within the reconstructed scenes.
2.2 INDOOR 3D OCCUPANCY PREDICTION DATASET
3D semantic understanding is fundamental task in computer vision, with extensive research fo- cused on point cloud segmentation (Gong et al., 202 laşb; Sun et al., 2024) and 3D occupancy pre- diction. However, the development of accurate models is hampered by the scarcity of large-scale, high-quality annotated data. Existing datasets, such as NYUv2 (Silberman et al., 2012), OceScan- (Wu et al., 2024), typically derived from
Net (Yu et al., 2024), and CimbodiedOce-Saanh.ee real-world scans. They consequently suffer from limitations including sensor noise, sparse annota- tions, and constrained object diversity, while being costly and time-consuming produce. These challenges underscore the need for more efficient data creation paradigm. In response, we propose an automated and labor-efficient pipeline for synthesizes high-fidelity voxel occupancy at a fraction of the cost, enabling scalable and diverse data generation for robust model training.
2.3 EMBODIED INTELLIGENCE RESEARCH IN MINECRAFT
Minecraft has been widely used for embodied intelligence and reinforcement learning research (Cai al., 2025a; Zheng et al., 2025a; Cai al., 2024b; Wang al., 2023), with many works built upon MineStudio (Cai et al., 2024a), streamlined open-source framework that unifies simulation and data management. To this end, ROCKET-1 (Cai et al., 2025b) leverages visual-temporal context prompting to master open- open-world interactions, JARVIS- VLLA (Li et al., 2025) post-trains large-scale vision- language models perform diverse in in-game tasks with keyboards and mouse, and GROOT (Cai et al., 2023) learns to follow instructions by watching gameplay videos. Despite these advances, all these approaches operate Minecraft's ' native blocky visuals, which exhibit a substantial reality gap compared real-world scenes. To address this limitation, integrate high-fidelity community mods, significantly narrowing the visual and structural gap and providing more effective simulation environment for embodied intelligence.
METHOD
3.1 PRELIMINARIES
Minecraft serves valuable platform for embodied AI research due to its voxel-based, spatially discretized world and consistent physical mechanics. However, the vanilla game presents limitations for perception-related tasks, including significant visual domain gap from reality, limited seman- tie diversity of blocks, and simplistic indoor structures. To address these issues, we developed a customized environment using different mods, including WorldEdit, Screen with Coordinates and TMEO. A detailed introduction to the standard environment and our modifications is provided in Appendix B.
3

---

Published as a conference paper at ICLR 2026
MinecraftVLN Dataset Construction
Position
Imgl:
EulerAngles
Timestamp
Img2: .....
>>>>
A/B/C/D
Proceed from your current location through the main door and find
<(<
The refrigerator in the living rom.
Answer
Next View/Next Action
Instruction
Next-View
Next- Actionction
Based the historical images, determine the Input>> current step. then choose next-view matching the next instruction.
Based the current images. determine the step. choose next-action image matching the next instruction.
<Input>
History
Instruction Proceed from your current location through main door. find the refrigerator
-Cureet-view>>
add
<History>
move forward
<ondidates
.Candidates>
turn left
turn right
the living room
D. stop.
<Output>
<Output>
Figure 2: Dataset Construction Pipeline for MinecraftVLN. We segment roomtour sequences into valid trajectories, then generate eestruction-followiin Arestton-AAsweer pairs using the collected coordinates and orientations to construct Next-View and Next-Action dataset.
3.2 WORLD2MINECRAFT. TRANSFER REALITY TO MINECRAFT
We propose World2Minecrafi, a framework that converts real-world scenes into Minecraft via 3D semantic occupancy prediction (Fig. I()). As detailed in Algorithm 1 (Appendix C), our method establishes an end-to-end pipeline from multi-view perception to executable Minecraft commands. The core of our method addresses 3D semantic occupancy prediction from sequence of first-person images I = {T, .2..... IN) along with their corresponding camera intrinsic parameters K. First, we employ monocular predictor Fmono that generates per-view semantic occupancy grids O mono from individual RGB images, where each voxel is assigned semantic label from C total classes:
O' mono F KC) 10,1,.... C XxXxxx
When the emr-image occupancy predictions obtained, leverage camera extrinsic parameters E merge them into unified 3D semantic representation O
Oweeme
F Fembodied ((Omoom iN KC, (0,1,...
To identify potential object centers, we first convert the multi-class semantic grid Oweme into binary occupancy grid Óninary. where voxels corresponding to any object class are marked as and empty voxels as O. We then compute local occupancy density map D on this binary grid by applying 3D convolution with a uniform kernel K E g Potential centers are identified by applying density threshold 7:
C C=[v/D(N D(v) 2 T.
K O Ô scene
where denotes the 3D convolution, and (x. 2) represents voxel coordinates. These initial center points C {c, ) are often redundant. To obtain accuarte and representative set of object locations, we cluster these points using the DBSCAN (Ester al., 1996) algorithm. Clustering is performed independently for the center points within each semantic class. This ensures that points with different semantic labels are not grouped together, preserving the categorical integrity of objects. It groups points based on distance threshold n, using the L2 norm the metric. Each resulting cluster is then represented by its centroid, forming a refined set of centers C::
C¹
Σ v v, g(x) = (v E p12 - unaas),
(4)
G
4

---

Published as a conference paper at ICLR 2026
MinecraftOcc Dataset Construction
Choe J: axis-aligned
The key Mod
Map Data
>>>
WorldEdlt
Fiix Plncx
Cose 2: diegona
Screenshot with Coordinates
W
Position: Y.
EulerAngles:
Timestump: WXX--X--XX
TMEO
Image Porometer Acquisition
Single -IngleImagee Occupancy Extraction Strategy
Occupancy Semantic
Data Construction
Dataset Construction Pipeline for MinecraftOce. We record coordinate data during roomtour, divide the viewpoint into two yaw-based cases to define view regions(the yellow indicates invisible areas; green indicates visible areas), and extract semantic occupancy from map data.
Figure 3:
This process yields refined set of centers C' [ck (K I M). where each centroid iden- tifies distinct object instance. Before final rendering. to ensure geometric fidelity, we employ a align each instance's occupancy grid O₂ with { }- We address orientation ambiguity by iterating through
eetrieval-based matching mechanism. Specifically, a candidate furniture library L = discrete of rotation angles selecting the optimal template and rotation that maximize spatial overlap:
OO n Rot TT
(7" = argmax 1.8
(5)
[O U (Ro
Once the optimal geometric representations are retrieved, they are translated into Minecraft building commands render the complete virtual scene.
3.3 ENABLING VLN IN MINECRAFT
Preparations. To conduct VLN in the reconstructed scene within Minecraft, we reconstructed all real-world indoor scenes from the validation set of EmbodiedOcc-ScanNe dataset (Wu et al., 2024) in Minecraft. Due to the limited accuracy of current prediction models, we selected 15 scenes for manual refinement to ensure high fidelity. However, observed that the limited scale of the reconstructed scenes resulted navigation dataset with relatively short and simple instructions. To Minecraft scenes, thereby increasing
address this, incorporated 5 additional community-crean the complexity and diversity of the instruction set (as shown in Figure and Table 8). An agent was subsequently directed perform room tours within these 20 selected scenes, generating a series of image sequences annotated with positions and orientations.
MinecraftVI.N Dataset Construction. As shown in Figure the dataset constructed by processing trajectories from the roomtour image sequence to extract meaningful navigation seg- ments. Human annotators provided detailed textual descriptions for each segment. Using a question- answering template, we generated 3,801 items in total(as shown in Table 9): 1,059 samples (Base) from the real-world reconstructed scenes and 2,483 samples (Extend) from the community-create scenes. The combined dataset (Combined) merges the aforementioned Base and Extend sets.
The MinecraftVLN dataset includes two distinct tasks: 1))Nxt- View Prediction:
The agent receives natural language instruction and a sequence of three historical images. It must first localize its current navigation step based on the instruction and visual context, and then predict the next most probable view. 2)xx--Acttonn Prediction: Given the instruction and the current view, the agent identifies its current progress within the instruction and predicts the next action (c.g., move forward, turn left) to comply with the instruction.
Conduct VLN in Minecraft. We conducted experiments in two key directions. First, we fine-tuned 0wen2.5-VL (Wang et al., 2024; Bai al., 2025) the MinecraftVLLN Second, deployed Gemini-2.5-Pe (Comanici et al., 2025) for direct embodied navigation control in Minecraft. De- tailed experimental setups and results presented in Sec. 4.3.
5

---

Published as a conference paper at ICLR 2026
Table I: Comparison between MinecrafiOce, NYUv2, and OceScanNet across key statistics.
Dutaset
Num, of Images
Num. of Scenes
Num, Classes
Total Semantic Voxels
Avg. Vonels per Seene
Image Resolution
NYUv2
1,449
10,786,528
~23,2K
640 x
OecSeanNet 65,119
674
13
201,215,233
~299. -4,7M
6:00 x 480
Minecraltice
100,165
156 (~11,00] rooms)
1,452
733,280,226
1920 x 11229
3.4 MINECRAFTOCC DATASET CONSTRUCTION
In this section, we will detail our automated semantic oceupancy generation pipeline and the MinecrafiOcc dataset built upon it.
Problem Definition. The 3D occupancy prediction task (as detailed in Sec. 3.2) infers the geometric and semantic structure of scene from a set of first-person images I, along with their camera intrinsic parameters K. and extrinsic parameters E
Image and Camera Parameter Acquisition. To acquire our dataset, used an automated mod tool, Screen with Coordinates, which simultancously captures first-person screenshots and records the corresponding camera pose (3D position and orientation). With this paired data of images and poses, we compute the corresponding intrinsic and extrinsic camera matrices for each image. The detailed methodology for deriving these matrices from the virtual camera's Field of View (FOV), position, and orientation is elaborated in Appendix D.
To generate semantic occupancy labels for each image, we define a fixed-size 3D) spatial volume V, with a minimum corner Vmin and a maximum corner Mmax- To accurately define the 3D spatial region based on the player's viewpoint, we cat- egorize the player's yaw angle 0 into two fundamental cases (in fact, all possible viewpoints can be ultimately categorized into these two cases) based on its orientation relative the world grid: als shown Case / in Figure 3), where the viewpoint is parallel to a coordinate axis, and diagonal(as shown Case Figure 3), where the viewpoint is directed along a 45-degree angle to the axes. For the axis-aligned case, we set the player's position Pplnyer = (xp, Up: =p) as the center of the V's back face. For the diagonal case, we set the Ppinyer as the volume's minimum corner V Miin- This logic is formalized by the calculation function
Single-Image Occupancy Extraction Strategy.
(Vmin, V Vmaz) = [[PPtegen 0, w. h, d)
(6)
where (w, h, d) are the dimensions of the volume, and each point a E satisfies the boundary
constraints Umin I S Umax
Due to the discrete nature of Minecraft's space, diagonal views often suffer from significant voxel loss at the periphery. To mitigate this, we designed a viewpoint-aware fallback strategy. This strat- egy supplements structural information from slightly adjusted neighboring viewpoints based on the original view, significantly enhancing the completeness of the data labels and improving stability and robustness during training. Specifically, we apply a correctional offset, e, to the bounding box corners Vmin and Vmaxx
v
min = Vinn +
v max
V + E
This adjustment expands the player's visible range, resulting in a field of view that more accu- rately reflects their actual perspective, while sacrificing only minimal portion of the depth-offfield area negligible loss in terms of overall visual coverage.
Occupancy Semantic Data Construction. To obtain semantic labels, we utilized the WorldEdir mod to extract the block types at these coordinates from the map data. This can be viewed as querying world-map function, Mworldd which maps any coordinate v to a semantic label from a set of all possible block types S = (air, stone, wood, ...). which allowed us to construct the final voxel- semantic occupancy representation, a grid O:
8y M Mavora((V), AAEA
(8)
0 = (sv VE
(9)

---

Published as a conference paper at ICLR 2026
View
View 2
View
Prediction
Construction
Reality
Minceraft
Reality
Minceraft
Reality
Minecraft
Figure 4: The reconstruction results from reality Minecraft presented above. As can observe that from View to View 3, the Reality row and the Minecraft row demonstrate a high degree of consistency. The Prediction column displays the predicted occupancy views from different perspectives of the same scene, while the corresponding reconstructed scenes in the Construction column align well with them.
EXPERIMENTS
4.1 EXPERIMENTAL SETUP
Datasets. Our experimental setup employs multiple datasets to evaluate different aspects of the proposed method and dataset. We employ the EmbodiedOcc-ScanNee dataset (Wu al., 2024) to train occupancy prediction model, which is the core of the World2Minecraft. For evaluating downstream task performance, Next-Action. We also introduce MinecraftOcc to evaluate the existing methods and employ the NYUv2 dataset for comparative analysis and mixed-training experiments.
introduce the MinecraftVLN dataset including Next- View and
Evaluation Metries. We employ standard metrics for each aspect of our evaluation: Accuracy for the VLN task, mean Intersection over Union (mloU) and Intersection over Union (IoU) for oceu- pancy prediction, and no-reference image quality metrics including Natural Image Quality Evaluator (NIQE), Perception-basee Image Quality Evaluator (PIQE), and Laplacian Variance (LV) to assess dataset realism (see Appendix H. for details).
4.2 THE RESULTS OF WORLD2MINECRAFT
Implementation Details. We employed the pre-trained EmbodiedOce (Wu et al., 2024) model to reconstruct real-world scenes from the full validation of SenooddiedOc-ScaanNee (Wu al., 2024) in Minecraft. From these reconstructions, we selected 30 scenes for meticulous manual refinement enhance their structural completeness and visual quality. Among these, 15 high-quality scenes were chosen construct the MinecraftVI.N dataset, which also serves as the platform for develop- ing and evaluating embodied navigation agents controlled by large language models. the initial automated reconstructions were suboptimal due to inherent model limitations, manual refinement ef- fectively restored geometrically consistent layouts, particularly in the accurate placement modern indoor furniture, demonstrated in the resulting scenes.
Analysis. The reconstruction results from reality to Minecraft are presented in Figure 4. As we observe that from View to View the Reality row and the Minecraft row demonstrate high degree of consistency. The Prediction column displays the predicted occupancy views from different perspectives of the same scene, while the corresponding reconstructed scenes in the Con-

---

Published as a conference paper at ICLR 2026
Table 2: Across three distinct MinecrafiVLN settings, the performance (Accuracy) of Qwen2.5- VL models (3B and 7B) on Next-View and Next-Action tasks under No Training, SFT, and RFT conditions is evaluated.
Dataset Composition
Owen2.5-VL-31
Owen2.5-VL-77
Task
No Train
SFT
RFT
No Train
SFT
RFT
Next-View Next-Action
0.2195
0.5610
0.2927
0.3905
0.5854
0,4390
Base
0.1943
0.7200
0.6343
0.3829
0.8000
0.6343
Next-View
0.2261
0.7087
0.3043
0.2913
0.6826
0.6043
Extend
Next-Action
0.3657
0.5437
0.6667
0.3786
0.6019
0.6343
Next-View
0.2288
0.5609
0.3137
0.2878
0.6642
0.6753
Combined
Next-Action
0.3037
0.4835
0.6570
0.3760
0.6281
0.6219
Figure 5: The result of a Gemini-2.5-Pry controlled agent performing VLN in our reconstructed scene. Following the natural language instruction "Go to the piano", the agent successfully navigates to the target step by step.
struction column align well with them, which collectively demonstrates the effectiveness of our World2Minecraft pipeline.
4.3 THE RESULT VIN IN MINECRAFT
Implementation Details. We conducted VLN in the reconstructed Minecraft environment. We first collected a navigation dataset within the 15 scenes mentioned in Sec. 4.2. However, we observed that the limited scale of the reconstructed scenes resulted in navigation dataset with relatively short and simple instruction sequences (denoted as Base). To mitigate this, we extended data collection to community-buile Minecraft scenes (denoted Extend), thereby increasing the complexity and diversity of the instruction set (as shown in Figure 6 and Table 8). The combined dataset (denoted Combined) was then used for evaluation. We evaluated two subtasks, Next- View and Next- Action, across all three dataset settings. For each setting, experiments were conducted under three conditions: No training, Supervised Fine-Tuning (SFT) based on I.Lama-Factory (Zheng et al., 2024), and Reinforcement Fine- Tunnnn (RFT) based on EasyRI (Zheng et al., 2025b), adopting Qwen2.5-VL(3B and 7B) models the base model. The results are summarized in Table
Analysis. The experimental results confirm that, as a baseline, the 7B model outperforms the 3B model in zero-shot settings, and both SFT and RFT lead significant performance gains. However, the optimal fine-tuning strategy is not uniform. On the more challenging multi-image Next-Vin View task, SFT proves more effective for the smaller 3B model. while the performance gap between SFT and RFT narrows considerably for the larger 7B model. Conversely, best method depends on the dataset: SFT excels on the Base set, whereas RFT shows superior performance on the more diverse Extend and Combined sets. These results collectively validate the effectiveness of our dataset and demonstrate the feasibility of conducting complex navigation tasks within the reconstructed Minecraft environments.
the Next-Action task, the
Application in Minecraft. We deploy Gemini-2.5.1 to control an agent performing VLN a World2Minecraft reconstructed scene. The agent successfully locates target piano by following natural language instructions "Go to the piano" as shown in Figure 5, demonstrating the practical utility of our environment for embodied AI.
8

---

Published as a conference paper at ICLR 2026
4.4 EXPERIMENTAL RESULTS ABOUT MINECRAFTOCC
Implementation Details. We first compared MinecraftOce with the NYUv2 and OccScanNet datasets in terms of scene count, image count, resolution, and image quality as shown in Fig- 1. To ensure objective evaluation, we randomly selected 100 images from each dataset to assess image quality using the NIQE, PIQE, and LV metrics. For the occupancy predic- tion experiments, and to ensure fair comparison with NYUv2 and OceScanNet, we mapped the 1,452 classes in MinecraftOcc to their 13 corresponding categories (see Appendix G for the detailed mapping). We initially evaluated several methods on our dataset three scales (8k, 50k, and 100k), including MonoScene (Cao & De Charette, 2022), NDCScene(Yae et al., 2023), ISO(Yu et al., 2024), and Symphonies (Jiang et al., 2024). Additionally, we conducted joint training experiments combining Symphonies with NYUv2 and the 8k-scale MinecrafiOce dataset. Analysis. Experimental results demonstrate that our MinecraftOce dataset significantly out- performs NYUv2 and OceScanNet in terms of image quantity and metries including NIQE, PIQE, and LV as shown in Table 3. As the re- sults in Table existing mainstream methods generally achieve relatively low performance, indicating the unique challenges presented by our dataset. Notably, MonoScene, which delivers average performance on NYUv2, demonstrates re- markable stability on MinecrafiOcc. We hypothesize that this is because other methods are overfitted to the NYUv2 dataset, leading to performance degradation when evaluated on different dataset. Furthermore, after joint training with the NYUv2 dataset on Symphonies( (JiannnneesJjnngggg et al., 2024), im- provements in both IoU(40.43) and mloU(+0.21) were observed, suggesting that the MinecrafiOcc dataset can effectively complement existing datasets, which is in Table 5.
Table 3: Image quality comparison across datasets using NIQE(L), PIQE(J), and LV(1).
Dataset
NIQE +
PIQE +
LV 1
NYUv2 OceScanNet
14.96
47.40
57,369
17.63
58.78
10,352
MinecraftOcc 9.97
45,23
274,305
Table 4: Minecraftoce Dataset results under different training settings.
comply
chair
cable
Setting Method
Precision
Recall
Monoscene
40.66
20.93
48.5
71,46
89.10
54.28
78.
32.14
0.00
11.20
12.56
13.15
8.67
1.94
10.85
NDC-Seme
37.06
17.82
46.5
65.57
88:42
46.01
79.
28.03
0.00
9.37
3.26
5.81
11.36
0.81
8.18
3.61
ISO
33.82
14.83
37.16
79.00
83.14
45. /5.42
78.33
27.08
0.00
1.30
0.13
2.49
1.44
0.40
3.45
0.07
Symphonics
39.11
21.56
49.30
65.42
$9.05
49.39
77.16
$1.06
2,80
12.24
13.27
13.26
13.77
4.95
11,11
8.13
Monoscene
39.51
19.61
$4,00
59.55
91.37
$2.93
83.99
27.68
2.57
9.70
5.82
8.78
5.53
3.25
9.56
5.90
NDC:
28
63.10
90..2
83.49
6.20
4.96
3.37
I5O
Symphonies
51.17
90.95
25.28
5.12
8.08
7,00
6.45
Monoscene
29.23
14.56
50.39
41.03
91.89
24.39
52.55
24.42
2.40
9.73
10.45
7.48
7,86
NDC Scene
25.08
12.95
41.09
47.02
88.91
26.12
52.81
22.83
1.11
6.93
8:26
6.86
6.41
2.76
4.61
3.85
100k
ISO
23.20
8.15
42.3 35
33.90
89 66
20.79
49.1 18.18
18.80
0.00
0.00
0.00
0.00
0.00
0.00
0.09
0.00
Syughanies
27.00
13.08
34.78
$7.23
86.14
26.05
61.02
21.27
2.97
6.06
5.90
5.74
4.79
2.50
4.53
3,05
Table Performance comparison on NYU V2 Dataset. * Represents the model trained on a mixture of the MinecraftOce 8k and NYUv2 training sets, and evaluated on the NYUv2 test set.
furniture
ceiling
window
objects
wall
chair
sofaa
table
Method
IoU
mioU
LMSCNet
33.93
15.88
4.49
88.41
4.63
0.25
3.94
32.03
15.44
6.57
0.02
14.51
4.39
AICNet
30.03
18.15
7.58
82.97
9.15
0.05
6.93
35.87
22.92
11.11
0.71
15.90
4.56
3DSketch
38.64
22.91
8.53
90.45
9.94
5.67
10.64
42.29
29.21
13.88
9.38
23.83
8.19
Monoscene
42.51
26.94
8.89
93.50
12.06
12.57
13.72
48.19
36.11
15.13
15.22
27.96
12.94
NDC-Scema
44.17
29.03
12.02
93.51
13.11
13.77
15.83
49.57
39.87
17.17
24.57
31.00
14.96
Symphonies
49.91
29.70
14.54
86.59
25.95
15.96
16.78
46.60
38.06
15.37
15.32
32.16
19.58
Symphonies*
50.34
29.91
13.96
88.55
26.18
17.26
17.22
45.83
38.94
17.38
12.29
32.58
18.88
4.5 COMPARISON WITH LAYOUT-BASED RECONSTRUCTION METHODS
Implementation Details. To validate the advantages of our occupancy-based reconstruction over layout- based methods, we compare World2Minecraftt with the recent indoor scene generation meth- ods: LPP (Feng et al., 2023). I-Design (Çelen et al., 2024), and LayoutVLM (Sun et al., 2025b). These methods generate scenes from textual descriptions but lack the geometric precision required for embodied AI tasks. We adapt them real-to-sim setting by converting input im-

---

Published as a conference paper at ICLR 2026
Table 6: Comparison with layout-based scene generation methods.
Method
OOB |
Collision +
Semantic
Visual f
Complete 1
Aesthetic 1
LayoutGPT
0.279
4.5
0.689
5,000
3.856
4.582
I-Dessin
0.423
0
0.884
6.001
4.734
5.352
LL
0
0.9
0.348
3.625
2.270
2.708
Morld2Minecraft (Ours)
0.024
0.2
0.913
6.145
5.186
6.022
Table 7: Detailed efficiency breakdown comparing our pipeline with building from scratch.
Metric Details
Build from Scratch
World2Minnccraft (Ours)
Improvement
Total Time (seconds).
482.00
70.38
Automated Process
5.88
Manual Refinement
482.00
64.50
7,5x
Total Operations
340.00
24.50
13.9
Addition Actions
319.30
9.70
32.9x
Deletion Actions
7.60
Orientation Adjustments
20.70
7.20
2.9
ages into textual descriptions using GPT-4o, which are then used generate scene layouts. For fair comparison, we use the same set of scenes from the Minecraft dataset that were manually re- fined in our World2Minecraft evaluation. Our evaluation employs six metries spanning functionality and aesthetics: OOB Rate (percentage of objects placed outside room boundaries), Collision Count (number of intersecting objects), Semantic Integrity (the ratio of generated semantic categories to the total categories present in the ground truth scene), and Visual Realism, Scene Completeness, and Aesthetic Atmosphere (perceptual scores rated by GPT-4o on scale of 1-10 assessing realism, completeness, and aesthetic appeal, respectively).
Analysis. As shown Table 6, World2Minecraft outperforms baselines across most metrics, no- tably achieving 0.913 in Semantic Iniegrity and 6.145 in Visual Realism. The minimal OOB Rate (0.024) and Collision Count (0.2) reflect superior spatial awareness, whereas layout-based meth- ods (e.g., LayoutGPT, LayoutVLM) struggle with geometric confficts and plausibility. This success stems from the combination of semantic occupancy prediction and shape-awar template matching, which captures fine-grained geometry to ensure the precise alignment vital for avoiding obstacles in downstream VLN tasks.
4.6 EFFICIENCY ANALYSIS OF MANUAL REFINEMENT
Implementation Details. We conducted efficiency experiment comparing World2Minecraft with refinement against building scenes entirely from scratch. For 15 scenes from the MinecraftVLN. Base dataset, we measured the total time and operation counts. Experienced builders created equiv- alent scenes from scratch as a baseline. The manual refinement involves three simple, lightweight operations: Deletion of floating artifacts, Completion of minor surface holes, and Adjustment of object orientations.
Results. As shown in Table 7, World2Minecraft with refinement requires only 70.38s per scene 7x reduction compared to building from scratch (482.00s). The refinement process itself involves an average of 24.5 operations (e.g., 9.7 hole fillings, 7.6 noise deletions, 7.2 orientation adjust- ments), compared to 340 operations for complete scene construction. This efficiency stems from our pipeline providing a high-quality initial reconstruction, requiring only minimal corrections. These corrections address imperfections inherent to any reconstruction algorithm, ensuring navigability for VLN tasks.
CONCLUSION
In this work, we introduce World2Minecraff. framework that converts real-world scenes into struc- tured Minecraft environments via 3D semantic occupancy prediction. We also propose scalable data construction pipeline and we build MinecraftOcc. large-scale dataset of diverse indoor scenes with voxel-wise semantic annotations. Our experiments demonstrate the utility of these reconstructed environments for downstream VLN tasks and establish MinecrafiOce's 's dual value as both challenging new benchmark that exposes limitations in ofte-of-theean models, and as powerful resource for augmenting existing real-world datasets. To foster reproducibility and future research, we will publicly release our complete framework and dataset.
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
Our work aims to advance embodied Al in an ethical way, focusing on safety, transparency, and re- peatability. All data in this study artificially created inside the Minecraft virtual world, which of- fers a controlled and consistent experimental setup. The MinecraftOcc and MinecraftVLN datasets built entirely from simulated scenes and include no real human data, private details, or identifi- able personal spaces. This approach avoids privacy issues that come with collecting real-world data. By working mainly in simulation, we also lower the safety risks and resource use typically involved in real robot experiments,
REPRODUCIBILITY STATEMENT
To reproduce the work presented in this paper, follow these steps in sequence:
1. Download the World2Mineccaft code along with the MinecraftOcc and MinecraftVIN
datasets.
2. Train the EmbodiedOce model using the provided configuration and training scripts.
3. Feed the predictions from the trained EmbodiedOce model into the World2Minecraft pipeline to generate corresponding Minecraft construction commands.
4. Prepare a Minecraft environment with the TMEO Mod installed and execute the generated commands to reconstruct the scenes.
5. Conduct Vision-Language Navigation (VLN) tasks using the provided evaluation scripts within the reconstructed Minecraft scenes.
6. Collect image sequences using the Screenshot with Coordinates tool for data acquisition purposes.
7. Extract map data using the WorldEdit utility to obtain scene layout information.
8. Generate occupancy data using the provided processing scripts create the final dataset format.
ACKNOWLEDGMENTS
This work was supported by the National Natural Science Foundation China (Grant Nos. 62302167, 62222602, 62502159, U23A20343. W2521174), the Shanghai Committee of Science and Technology (Grant Nos. 25511103300, 25511104302, 25511102700), the Natu- ral Science Foundation of Shanghai (Grant No. 257R1402135), the Natural Science Foun- dation of Chongqing (Grant Nos. CSTB2022NSCQ-QQ0000, CSTB2023NSCQ-MSXX0 CSTB2025NSCQ-GPA GPX0444 the Young Elite Scientists Sponsorship Program by CAST YESS20240780, and the Open Project Program of the State Key Laboratory of CAD&CG, Zhe- jiang University (Grant No. A2501).
REFERENCES
Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian Reid, Stephen Gould, and Anton Van Den Hengel. langgae navigation: Interpreting visually-grounded navigation instructions in real environments. In Proceedings of the IEEE con- ference on computer vision and pattern recognition, pp. 3674-3683, 2018.
Armen Avetisyan, Manuel Dahnert, Angela Dai, Manolis Savva, Angel X Chang, and Matthias Nießner. Scan2ead: Learning cad model alignment in rgb-d scans. In Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, pp. 2614-2623, 2019.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, al. Qwen2. 5-vl technical report. arXiv preprint 2rx2 2025.
Shaofei Cai, Bowei Zhang, Zihao Wang, Xiaojian Ma, Anji Liu, and Yitao Liang. Groot: Learning to follow instructions by watching gameplay videos. arXiv preprint arXiv:23 10.08235, 2023.
11

---

Published as a conference paper at ICLR 2026
Shaofei Cai, Zhancun Mu, Kaichen He, Bowei Zhang, Xinyue Zheng, Anji Liu, and Yitao Liang. Minestudio: A streamlined package for minecraft ai agent development. arXiv preprint
arXiv:2412.18293, 2024a.
Shaofei Cai, Bowei Zhang, Zihao Wang, Haowei Lin, Xiaojian Ma. Anji Liu, and Yitao Liang. Groot-2: Weakly supervised multi-modal instruction following agents. arXiv preprint arXiv:2412.10410, 2024b.
Shaofei Cai, Zhancun Mu, Haiwen Xia, Bowei Zhang, Anji Liu, and Yitao Liang. Scalable multi- task reinforcement learning for generalizable spatial intelligence in visuomotor agents. arXiv preprint arXiv:2557. 2025a.
Shaofei Cai, Zihao Wang, Kewei Lian, Zhancun Mu, Xiaojian Ma, Anji Liu, and Yitao Liang Rocket-1: Mastering open-world interaction with visual-temporal context prompting. In Pro- ceedings of the Computer Vision and Pattern Recognition Conference, pp. 12122-12131, 2025b.
Anh-Quan Cao and Raoul De Charette. Monoscene: Monocular 3d semantic scene completion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp-
3991-4001, 2022.
Ata Çelen, Guo Han, Konrad Schindler, Luc Van Gool, Iro Armeni, Anton Obukhov, and Xi Wang. I-design: Personalized I1m interior designer. In European Conference on Computer Vision, pp- 217-234. Springer, 2024.
Gheorghe Comanici, Eric Bieber, Mike Schackermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capa- bilities. arXiv preprint arXiv:2507.0626/ 2025.
Martin Ester, Hans-Peter Kriegel, Jörg Sander, Xiaowei Xu, al. A density-based algorithm for discovering clusters in large spatial databases with noise. In kdd, volume 96, pp. 226-231, 1996.
Weixi Feng, Wanrong Zhu, Tsu-Jui Fu, Varun Jampani, Arjun Akula, Xuehai He, Basu, Xin Eric Wang, and William Yang Wang. Layoutgpt: Compositional vi- sual planning and generation with large language models. In A. Oh, Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural In- formation Processing Systems, volume 36, pp. 18225-18250. Curran Associates, Inc., 2023. URL https: /ttps////////e//esseeeeee ngg neurips cc wwtpcccpppppeeeeeeeeeeeee files/ileeeeeeee wttpse/filsssspppeeeeeeeeee pappr/2023//////////// file/ 1.779e48555 45411944555557723333333333333
.df..
Junyao Gao, Yanan Sun, Yanchen Liu, Yinhao Tang, Yanhong Zeng, Ding Qi, Kai Chen, and Cairong Zhao. Styleshot: snapshot on any style. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025.
Jingyu Gong, Jiachen Xu, Xin Tan, Haichuan Song, Yanyun Qu, Yuan Xie, and Lizhuang Ma. supervsupe point cloud segmentation via gradual receptive field component reasoning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11673- 11682, 202
Jingyu Gong, Jiachen Xu, Xin Tan, Jie Zhou, Yanyun Qu, Yuan Xie, and Lizhuang Ma. Boundary- aware geometric encoding for semantic segmentation of point clouds. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 1432, 2021b.
Tianjun Gu, Linfeng Li, Xuhong Wang, Chenghua Gong, Jingyu Gong. Zhizhong Zhang, Yuan Xie, Lizhuang Ma, and Xin Tan. Doraemon: Decentralized ontology-aware reliable agent with enhanced memory oriented navigation. arXiv preprint 2rXiv:2505.21966 2025.
Can Gümeli, Angela Dai, and Matthias Nießner. Roca: Robust cad model retrieval and alignment from single image. In Proceedings of the IEEE/CV conference on computer vision and pattern recognition, pp. 4022-4031, 2022.
Zhening Huang, Xiaoyang Wu, Fangcheng Zhong, Hengshuang Zhao, Matthias Nießner, and Joan Lasenby. Litereality: Graphics-ready 3d scene reconstruction from rgb-d scans. arXiv preprint 00Xi225507.0286.. 2025.
12

---

Published as a conference paper at ICLR 2026
Yuzhou Ji, He Zhu, Junshu Tang, Wuyi Liu, Zhizhong Zhang. Xin Tan, and Yuan Xie. Fastigs: Speeding up language embedded gaussians with feature grid mapping. In Proceedings of the AAAI Conference Artificial Intelligence, volume 39, pp. 3922-3930, 2025.
Haoyi Jiang, Tianheng Cheng, Naiyu Gao, Haoyang Zhang, Tianwei Lin, Wenyu Liu, and Xing- gang Wang. Symphonize 3d semantic scene completion with contextual instance queries. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20258- 20267, 2024.
Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splat- ting for real-time radiance field rendering. ACM Trans. Graph., 139-) 2023.
Muyao Li, Zihao Wang, Kaichen He, Xiaojian Ma, and Yitao Liang. Jarvis-vla: Post-training large- scale vision language models to play visual games with keyboards and mouse. arXiv preprint arXiv:2503. 16365, 2025.
Haisong Liu, Haiguang Wang. Yang Chen, Zetong Yang. Jia Zeng, Li Chen, and Limin Wang. Fully sparse 3d panoptic occupancy prediction. CoRR, 2023.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes neural radiance fields for view synthesis. Communications of the ACM, 65(1):99-106, 2021.
Srivathsan Murali, Pablo Speciale, Martin R Oswald, and Mare Pollefeys. Indoor scan2bim: Build- ing information models of house interiors. In 2017 IEEE/RSJ International Conference on Intel- ligent Robots and Systems (IROS), pp. 6126-6133. IEEE, 2017.
Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, et al. Habitat: A platform for embodied ai research. In Proceedings of the IEEE/CVF international conference on computer vision, pp.
9339 9347, 2019.
Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus. Indoor segmentation and sup- port inference from rghd images. In European conference on computer vision, pp. 746-760. Springer, 2012.
Chunyi Sun, Junlin Han, Weijian Deng, Xinlong Wang, Zishan Qin, and Stephen Gould. 3d-gpt: Procedural 3d modeling with large language models. In 2025 International Conference on 3D Vision (3DV), pp. 1253-1263. IEFE, 2025a.
Yun Sun, Weiyu Liu, Siyi Gu, Dylan Lim, Goutam Bhat, Federico Tombari, Manling Li, Nick Haber, and Jiajun Wu. Layoutylm: Differentiable optimization of 3d layout via vision-language models. In Proceedings of the Computer Vision and Puttern Recognition Conference, pp. 29469- 29478, 2025b.
Tianfang Sun, Zhizhong Zhang, Xin Tan, Yong Peng. Yanyun Qu, and Yuan Xie. Uni-to-multi modal knowledge distillation for bidirectional lidar-camera semantic segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2024.
Qijian Tian, Xin Tan, Yuan Xie, and Lizhuang Ma. Drivingforward: Feed-forward 3d gaussian splatting for driving scene reconstruction from flexible surround-view input. Proceedings of the AAAI Conference Artificial Intelligence, volume 39, pp. 7374-7382, 2025.
Michal J Tyszkiewicz, Kevis-Kokits Maninis, Stefan Popov, and Vittorio Ferrari. Raytran: 3d pose estimation and shape reconstruction of multiple objects from videos with ray-traced transformers. In European Conference Computer Vision, pp. 21 1228. Springer, 2022.
Peng Wang. Shuai Bai, Sinan Tan, Shijie Wang. Zhihao Fan, Jinze Bai. Keqin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge, et al. Qwen2-vl: Enhancing vision-languee model's 's perception of the world at any resolution. arXiv preprint arXiv:2409. /2/91, 2024.
Zihao Wang, Shaofei Cai, Guanzhou Chen, Anji Liu, Xiaojian Ma, and Yitao Liang. Describe, explain, plan and select: Interactive planning with large language models enables open-world multi-task agents. arXiv preprint a02.0:2002.01560. 2023.
13

---

Published as a conference paper at ICLR 2026
Yuqi Wu, Wenzhao Zheng, Sicheng Zuo, Yuanhui Huang, Jie Zhou, and Jiwen Lu. Embodiedocc: Embodied 3d occupancy prediction for vision-based online scene understanding. arXiv preprint arXiv:2412.04380,, 2024.
Xiuyu Yang, Yunze Man, Junkun Chen, and Yu-Xiong Wang. Scenecraft: Layout-guided 3d scene generation. Advances in Neural Information Processing Systems, 37:82060-82084, 2024.
Jiawei Yao, Chuming Li, Keqiang Sun, Yingjie Cai, Hao Li, Wanli Ouyang, and Hongsheng Li. Ndc-scene: Boost monocular 3d semantic scene completion in normalized device coordinates space. In 2023 IEEE/CVF International Conference Computer Vision (ICCV), pp. 9421-9431. IEEE Computer Society, 2023.
Hong-Xing Yu, Haoyi Duan, Charles Herrmann, William T Freeman, and Jiajun Wu. Wonderworld: Interactive 3d scene generation from single image. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 5916-5926, 2025.
Hongxiao Yu, Yuqi Wang. Yuntao Chen, and Zhaoxiang Zhang. Monocular occupancy prediction for scalable indoor scenes. In European Conference on Computer Vision, pp. 38 04-54. Springer, 2024.
Cairong Zhao, Yubin Wang, Xinyang Jiang, Yifci Shen, Kaitao Song, Dongsheng Li, and Duoqian Miao. Learning domain invariant prompt for vision-language models. IEEE Transactions on Image Processing, 33:1348 1344-160,, 1360, 2024.
Xinyue Zheng, Haowei Lin, Kaichen He, Zihao Wang, Qiang Fu, Haobo Fu, Zilong Zheng, and Yitao Liang. Mcu: An evaluation framework for open-ended game agents. In Forty-second International Conference on Machine Learning, 2025a.
Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhanghhi Feng, and Yongqiang Ma. Llamafactory: Unified efficient fine-tuning of 100+ language models. In Pro- ceedings of the 62nd Anmual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), Bangkok, Thailand, 2024. Association for Computational Linguis- abs/2403.133335
tics. URL
Yaowei Zheng, Junting Lu, Shenzhi Wang, Zhangchi Feng, Dongdong Kuang, and Yuwen Xiong. Easyrl: An efficient, scalable, multi-modality rl training framework. https::/gitttti com/ hiyouga/ EasyR1, 2025b.
14

---

Published as a conference paper at ICLR 2026
A THE USE OF LARGE LANGUAGE MODELS
In the preparation of this manuscript, large language models (LLMs) were utilized solely for writing and presentation assistance, in accordance with their respective licenses and terms of use. Specif- ically, LLMs were employed to assist in language polishing to improve the fluency and clarity of selected sections, adjusting the layout and presentation of figures and tables, and generating aux- iliary code snippets for data processing and visualization tasks. It is important emphasize that all scientific content, methodological development, experimental design, data analysis, and conclu- sions remain the intellectual contribution of the authors, with LLM usage strictly limited to auxiliary editorial and presentational tasks.
B PRELIMINARIES
Minecraft, as an open-world sandbox game, is renowned for its highly flexible building mechanics and consistent physics, making significant foundation for constructing simulation environments in embodied Al research. This section systematically introduces its standard environment features, modding mechanism, and the high-fidelity extended environment we developed based on it.
B. NAIVE MINECRAFT ENVIRONMENT
The standard Minecraft environment provides voxel-based and spatially discretized world the entire world is divided into 1xlx1 unit voxel blocks, each corresponding to approximately 1m in the real world. This environment not only exhibits geometric regularity but also supports realistic physical interactions, demonstrating high degree of authenticity and simulation fidelity, especially in terms of lighting, gravity, and terrain dynamics, which closely mirror real-world physical laws.
However, the vanilla Minecraft environment has notable limitations embodied AI research. Its low-resolution, blocky visual style introduces significant domain gap when compared to real- world images, restricting its applicability in perception tasks. Furthermore, the default Minecraft blocks lack diversity and granularity, consisting mainly of abstract classes (e.g.. "wood", "stone") rather than semantically meaningful entities such as furniture. The indoor structures in the default environment overly simplistic and lack layout complexity, failing provide the spatial variety necessary for perception-dependent decision-making in embodied tasks.
B.2 MOD ABOUT MINECRAFT
In order to address the limitations described above, we introduced series of functional mods that significantly enhance environment construction, data collection, and visual realism. The modding system core extensibility mechanism of Minecraft, allowing modification enhancement of game functionality through custom code and resource packs. In this work, we mainly employed three mods, illustrated Figure 3, namely:
WorldEdit It provides efficient large-scale procedural scene generation and editing capabilities. It supports rapid creation, duplication, and modification of composite structures via scripting, greatly improving both the efficiency and diversity of constructing complex indoor environments. Addition- ally, allows easy access to the map data of target regions, including block types and coordinates, subsequent downstream processing and analysis.
Screen with Coordinates simultaneously records the player's current viewpoint frame along with the player's position coordinates and viewing orientation, represented in Euler angles, during ren- dering.
TMEO Texture and Mod Pack It introduces over 1,400 fine-grained, semantically labeled object models (c.g., furniture such as "Blingds lighting" and household items like "Crib infant beds"), substantially enriching the semantic diversity and object hierarchy of scenes. Coupled with its high- resolution physically-bee based material pack, it lays the foundation for subsequent Physically Based Rendering(PBR).
15

---

Published as a conference paper at ICLR 2026
B.3 EXTENDED MINECRAFT ENVIRONMENT
Building upon the aforementioned mods, constructed a high-fidelity, diversified extended Minecraft environment tailored for embodied Al tasks. This environment significantly surpasses the native platform in terms of semantic complexity, visual realism, and scalability.
By integrating the fine-grained objects and diverse structures from the TMEO mod, we developed an indoor environment system comprising 156 detailed scenes and approximately 1,600 rooms. These encompass multi-story architectural structures, complex spatial layouts, and dense furnish- ings, greatly enriching the semantic hierarchy and spatial diversity of the scenes.
On the visual level, leveraging high-definition textures, PBR shaders, and dynamic lighting mods, we achieved realistic shadows, reflections, and global illumination effects. This markedly reduces the domain gap between synthetic images and real-world scenes.
In terms of environment construction and data collection, tools like WorldEdit and Screen with Co- ordinates enabled the establishment of standardized pipeline. This supports the efficient generation of new scenes and the automatic acquisition of multi-mon annotated data, ensuring high reusability and extensibility.
Algorithm World2Minecraft: Reality-to- Virtual Transfer
Require: Input: Image set I (J,,.... .N
1:
Camera intrinsic parameters I. Camera extrinsic parameters & Pretrained models Fmono- Femb
2:
3:
Ensure: Output: Reconstructed Minecraft scene
4: procedure RECONSTRUCTSSENEECC K, Fmono: Femb)
5:
Omono +- 0
Initialize monocular predictions set
6:
for each image I, do O'nono f- Fmono (4, mono f- O mono U {0) mono
p Process each view
7:
Generate vir-view occupancy
8:
9:
scene t- F Fembodied O O KC, E)
>> Fuse multi-view predictions P Compute density map via 3D convolution Extract centers above threshold b Merge centers within distance
10:
D - K Oxcene
11:
C + (v Óscene D(v) TT
12:
C < Cluster (C, 17)
13:
M Translate T MoMinecraf((C")
Generate Minecraft building commands
14:
BaocuteCommands(MM
> Render scene in Minecraft
return MinecraftScene
Return reconstructed virtual scene
C ALGORITHM OF WORLD2MINECRAFT
Our proposed method, World2Minecraff,, formulates the reality-to-virtual transfer as scene recon- struction problem. The core pipeline, outlined in Algorithm 1, takes of multi-view images and camera parameters as input and produces executable commands to reconstruct the scene in Minecraft. The process consists of three main stages: multi-view semantic occupancy prediction, volumetric fusion and density-based filtering, and finally, virtual world generation.
Stage 1: Multi-view Semantic Occupancy Prediction. The algorithm begins by processing each input image 7; independently using a monocular prediction model FFmmno (Line 4-7). This model infers an mitial 3D semantic occupancy volume O' O for each view, capturing the geometry and semantics visible from that particular viewpoint. These per-view predictions are aggregated into set O mono.
Stage 2: Volumetric Fusion and Filtering, The individual occupancy volumes are then fused into consistent global scene representation Owene by an embodied model Fembodied, which utilizes the camera parameters to resolve inconsistencies and merge information (Line 8). To obtain a clean and structured representation suitable for building, convolution kernel to the fused occupancy (Line 9). Voxel centers with a density value exceeding threshold selected as candidate building locations C (Line 10). A clustering step (Line 11)
compute density map D by applying 3D
16

---

Published as a conference paper at ICLR 2026
further refines these candidates by merging those within small distance 77 reducing redundancy and ensuring structural coherence.
Stage 3: Virtual World Generation. The final stage translates the refined 3D centers C' into a sequence of Minecraft building commands .M (Line 12). These commands, which specify the placement of specific block types at 3D coordinates, executed to render the final scene within the Minecraft environment (Line 13), completing the transfer from reality to a semantically decomposed and editable virtual world.
D IMAGE AND CAMERA PARAMETER ACQUISITION
As shown in Figure 3, we use an automated mod tool, Screen with Coordinates, acquire data. This tool simultaneously captures first-person screenshots while recording the virtual camera's 's pose for each frame, including its 3D position and orientation (Fuler angles). From this paired data, we compute the intrinsic and extrinsic matrices of the camera for each image.
D.1 INTRINSIC CAMERA MATRIX
The intrinsic matrix I. is determined by the virtual camera's Field of View (FOV) and the image dimensions (W,H).
fx
0 Cz
=
fy
cy
(10)
0
i
Here, the focal lengths Say fy and the principal point (Cas cy) derived from the horizontal FOV, denoted as at
W
W
fa = fy 2 2tan(a/2)
Cx
Cy
(11)
2
2
D.2 EXTRINSIC CAMERA MATRIX
The extrinsic matrix & defines the transformation from the camera coordinate system to the world frame. It is constructed from the camera's 's position (Fp =p) and its orientation, which is defined by rotation matrix R.
R O
P
(12)
The rotation matrix R is derived from the yaw (8) and pitch (δ) angles provided by the game mod. To align the mod's Euler angle convention with standard right-hander coordinate system (e.g., camera: +X right, +Y down, +Z forward), we apply an offset of π to the angles.
The final orientation is achieved by composing two sequential rotations: first, a yaw rotation around the world's Y-axis, followed by pitch rotation around the camera's local X-axis. This corresponds to extrinsic YX Euler angle convention.
First, the yaw rotation matrix Ryaw is defined as rotation around the Y-axis by an angle of (0 + m):
cos(0 + T)
sin(0 + +T) cos - sin
R Ryyaw Ry (8
0
0
0
0
(13)
si 0 + T)
cos(# + x)_
sin 0 - cos 0
Next, the pitch rotation matrix Rpitch is defined as rotation around the X-axis by an angle of (Ø+T)):
1
0
0 sin (o T) cos( +
[
Repcch Rxx (
cos(o + T) sin( + T)
cos
sin
(14)
0 sin Φ cos
The final rotation matrix R is the product of these two matrices, with the yaw rotation applied first:
cos 0 sin 0 sin
sin
R Rpitch Ryaw
Φ
cos f sin Φ
(15)
sin O cos á sin Φ
cos 0 cos Φ
17

---

Published as a conference paper at ICLR 2026
E VIEWPOINT PROJECTION AND REDUNDANCY REMOVAL
To further improve label quality, we introduce view frustum culling mechanism based on cam- era projection. In early versions of the dataset, the occupancy regions associated with an image occasionally included irrelevant space outside the camera's field of view, which introduces noise. Therefore, we use the camera's intrinsic and extrinsic parameters to project the 3D voxel grid onto the 2D image plane. Let т IR3 + IR² be the camera projection function, which utilizes the intrinsic K and extrinsic & matrices to map world coordinate point v to image coordinates u. We retain only those voxels that project within the image boundaries. The final set of visible voxels, Vvisille, is defined as:
(16) By removing voxels that project outside the image, we ensure that the final 3D occupancy labels are precisely aligned with the image's field of view, thereby eliminating redundancy and preventing label misalignment.
Visibbl {v V (V, K,) E [0, W] [0, H]}
F DATASET ANALYSIS
Table 8: Statistical Summary of Instruction Lengths Across Different Tasks and Datasets. The table shows the count, mean, standard deviation (Std. Dev.), and quartiles for the length of navigation instructions.
Task
Dataset Type
Count
Mean
Std. Dev.
Min
25%
50%
75%
Max
Base
769
79.01
32.87
57.0
78.0
96.0
187
Next-Action
Extend
1351
171.37
67.10
62
117.0
156.0
232.0
334
Combined
2120
137.87
72.34
15
81.0
121.0
181.0
334
Base
290
87,41
33.57
28
64.0
87.0
99.0
187
Next-View
Extend
1132
173.23
66.70
62
121.0
156.0
232.0
334
Combined
1422
155.73
70.48
103.0
145.0
208.0
Instruction Length Distribution by MinecraftVLN
Next-View
Next-Action
0.014
have extend
0.012
combined
0.010
Density
0.008
0.006
0.004
0.002
100
200
300
200
250
300
350
400
Instruction Length (Characters)
Figure 6: A comparison of instruction length distributions across three datasets for the Next-View and Next-Action tasks. The extend dataset clearly contains shorter and more uniform instructions.
G SEMANTIC CLASS MAPPING FOR CROSS-DATASED EXPERIMENTS
To facilitate a fair and meaningful comparison between models trained on our MinecraftOcc dataset and those evaluated standard benchmarks like NYUv2, established many-to-one
18

---

Published as a conference paper at ICLR 2026
Table 9: Dataset statistics for Minecraftoce and Minecraft Minecraftoce is provided at three
scales (8k, 50k, 100k), while Minecraft includes three scales (Base, Extend, Combined) with two task settings: Next- View and Next-Action.
Dataset
Scale
Task
Train
Test
All
8k
6,100
2,024
8, 8,124
Minecraftoce
50k
39,914
10,245
50,159
100k
79,799
20,366
100,165
Base
Next- View
902
230
1132
Next-Action
1042
309
1351
Minecraft
Extend
Next- View
249
41
290
Next-Action
594
175
769
Combined
Next-Vier Next-Action
1151
271
1422
1636
484
2120
mapping from our 1,000+ fine-grained, Minecraft-specific classes (including 200 distinct lighting fixtures) to a standardized set of 12 common indoor semantic categories. This process creates shared semantic ground for consistent evaluation.
The mapping was designed to group Minecraft blocks and items based on their functional and struc- tural roles within an indoor scene. Table 10 summarizes this hicrarchy, providing the rationale for each target superclass along with a few representative examples from the MinecraftOce dataset. This standardized taxonomy is used for all cross-dataset experiments.
The complete, exhaustive mapping of all classes is provided as a jsoo file in our supplementary material to ensure full reproducibility.
Table 10: Summary of the mapping from fineggriined MinecraftOce classes to 12 target superclasses. Representative examples are provided for clarity.
Target Superclass
Mapping Rationale / Included Concepts
Example MinecraftOcc Classes
empty
A broad catch-all category for non-structural, transparent, or empty elements. Overhead structural surfaces and decorative ceiling elements.
tmeosultraacchaarglaaa. unco.uultra::diiodeeng... tmeooullraasshhgooiian...
ceiling
minecraft:bbicchpplanks
floor
Horizontal walking surfaces, including stairs, slabs, and floor coverings.
tmeo ultrrayyttiihhiootti.... mineersafttsttoeeebrickk.....
wall
Core vertical structural surfaces All types of blinds and shutters.
tmeov:dllantimen
window
Lmeov::balycchuang
chair
All pengsen-prsonn seating, including henches, stools, and dining chairs.
tmeo ultraacchnggdeng tmeo ultra:bataiyizi
bed
All types beds and associated bedding parts. Couches and other seating furniture.
ameo.ulttrardach.anngaa..
sofa
ameo.ultraasshaf..l222
table
Surfaces placing objects.
mme.polttaacccarrhhyyyuuxxxxx
tvs
All television sets and monitor General non-scating furniture, e.g., cabinets, shelves, sinks, and wardrobes.
furn
Lmeo.ultraryylgui...
tmeov:shujia... tmeov:penzai ultraayinxiangg...
objs
Miscellaneous functional and decorative items not part of the main structure.
H
IMAGE QUALITY METRICS
This appendix briefly describes the three no-reference image quality assessment metrics used in our evaluation:
Natural Image Quality Evaluator(NIQE), Measures how closely an image's 's statistical properties match those of natural images. Lower values indicate better, more natural quality.
19

---

Published as a conference paper at ICLR 2026
Perception-basee Image Quality Evaluator(PIQE), Assesses local distortions and artifacts in im- ages based on human visual perception. Lower values indicate fewer distortions and better percep- tual quality.
Laplacian Variance(LV). Quantifies image sharpness by measuring the variance of Laplacian- filtered responses. Higher values indicate sharper images with more detail.
These complementary metrics provide a comprehensive assessment of visual quality from different perspectives: NIQE evaluates global naturalness, PIQE detects local artifacts, and LV measures sharpness.
I IMPLEMENTATION DETAILS
This section provides comprchensive overview of the implementation details for the two main ex- periments conducted this study: Reinforcement Fine- Tini-Tunngg (RFT) and Supervised Fine- Tuning (SFT). All experiments used Qwen2. 5-VL the base model.
1.1 EXPERIMENT 1: REINFORCEMENT FINE-TUNING
The RFT experiment was conducted using the EasyRI framework. For this experiment, the vision tower of the model was unfrozen and trained alongside the language components. We employed the Generalized Rejection Policy Optimization (GRPO) algorithm. To regularize the policy, we incorporated a Suulback--eibber (KL) divergence penalty with a coefficient of 1.0 x 10-2, calculated using a low-variance estimator.
Dataset and Data Processing. The training and validation data were sourced from Parquet files, using content for prompts and answer for responses. Prompts were formatted via a custom Jinja template (me. jinja). The maximum prompt length was set to 4096 tokens, and the maximum response length was 1024 tokens.
Hardware and Training Configuration. The experiment was run on a single node with 8 GPUs, utilizing Fully Sharded Data Parallelism (FSDP) with full parameter sharding and CPU offloading for both model parameters and optimizer states to conserve memory. The rollout phase was acceler- ated with tensor parallel size of 2. Key hyperparameters summarized in Table 11.
1.2 EXPERIMENT 2: SUPERVISED FINE-TUNING (SFT)
The SFT experiment was conducted using the LLaMA Factory framework.
Model and Fine-tuning Strategy. In contrast to the RFT experiment, the vision tower and the multi- modal projector were kept frozen during SFT. Fine-tuning performed only the parameters of the language model component using a fall-paraeeter approach (f (Einetuning..typ:: full).
Dataset and Preprocessing. We utilized custom dataset named base.train.taskk. limiting the training to a maximum of 1000 samples. The data was formatted with the qwen2 vvl conversa- tion template, and the maximum sequence length was capped 8192 tokens.
Training Configuration. The model was trained for 5 epochs using the DeepSpeed ZeRO Stage 3 strategy and bfloat16 mixed precision. We employed cosine learning rate scheduler with 10% warmup period. No validation was performed during training. The detailed hyperparameters presented in Table 11.
J INTERACTIVE VISUALIZATION TOOL: SCENEFORGE
To facilitate the analysis and refinement of occupancy clustering results, we developed an interactive web-based visualization tool names SceneForge that provides Open3D-like 3D exploration capabil- ities. This tool plays a crucial role our World2Minecraft pipeline by enabling intuitive inspection and manual correction of semantic occupancy predictions.
20

---

Published as a conference paper at ICLR 2026
Table 1I: Comparison of key hyperparameters for the SFT and RFT experiments.
Parameter
RFT Setting
SFT Setting
General Strategy
Fine-tuning Type
Full- Full-parameter b
Training Precision
Algorithm
GRPO
Optimization
Optimizer
A 1.0 10
Adam W (implied)
Learning Rate
1.0 10
Weight Decay
1.0 x 10
LR Scheduler
Cosine
Warmup Ratio
0.1
Total Epochs
30
Batching Global Batch Size
128
Per-device Batch Size
1
Gradient Accumulation Steps
Effective Batch Size
128 (Global)
2
RFT-Specific Details
KL Coefficient (AKL.
1.0 x 10-2
Rollout Samples (n)
Training Temperature
1.0
TOOL OVERVIEW
The visualization tool is implemented as a standalone web application using D3.js for 3D rendering and interaction. supports the following key functionalities:
Multi-format Data Loading: The tool accepts both voxel-wise occupancy data (occ. (occ.json) and pre-computed center points (centers. json), with optional demo data for quick testing.
Interactive 3D Exploration: Users can rotate the view (mouse drag). zoom (scroll wheel), and pan (Shift + drag) to examine the scene from any angle, mimicking the interaction paradigm of Open3D.
Category-aware Visualization: Twelve semantic categories are color-coded and can be individually toggled on/off via an interactive legend, enabling focused analysis of specific object types.
Real-time Parameter Adjustment: Dynamic controls allow users to adjust voxel size, transparency, and center point size during visualization to optimize clarity.
Center Point Editing: An advanced editing mode supports manual refinement of object centers through:
Drag-and-dis repositioning of center points
Point deletion (right-click) and selective removal
Point splitting for fine- -reingraidd object separation
Bulk operations by category selection
J.1 INTEGRATION WITH RESEARCH WORKFLOW
The tool served two primary purposes in research:
1. Qualitative Analysis: During method development, used the tool to visually inspect clustering results, identify failure cases, and understand the limitations of automatic center prediction algorithms.
21

---

Published as a conference paper at ICLR 2026
2. Data Refinement: For the MinecraftVLN dataset creation, the editing capabilities allowed us to manually correct inaccurately predicted object centers, ensuring higher quality navi- gation environments.
J.2 TECHNICAL IMPLEMENTATION
The tool architecture consists of four modular components:
config. ja Configuration constants and color schemes
data js Data loading and processing utilities
projection. js 3D projection and rendering engine js User interface event handlers and state management
This web-based implementation ensures cross poss--patform compatibility without requiring complex dependencies, making accessible for researchers to reproduce and extend our work. The complete source code is available in our supplementary materials.
22


Reviews:
[{"review_id": "8CwAzYV8Xl", "reviewer": "Reviewer_HVMZ", "summary": "The paper introduces a real-to-simulation framework, World2Minecraft, which converts real-world observations into a Minecraft environment through semantic occupancy prediction. The authors further utilize the constructed Minecraft scenes to generate samples for vision-language navigation and additional data for training semantic occupancy prediction models. Experimental results show that this augmentation strategy holds potential to enhance task performance.", "questions": "* Task-specific scene usage: Why are only 15 manually refined scenes used for the VLN task, while all constructed scenes are utilized for semantic occupancy prediction? What specific refinements are required for VLN, and why are they unnecessary for occupancy prediction?\n* Scope of reconstruction: Since furniture assets come from an existing asset set, does the proposed pipeline only reconstruct the structural layout of the environment? If so, what are the key advantages over prior real-to-sim frameworks that similarly rely on existing furniture databases?", "limitations": "", "rating": 4, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "* Leveraging simulation to augment real-world task performance through real-to-sim techniques is a promising direction. \n* Developing an automated real-to-sim pipeline is highly valuable.", "weaknesses": "* Unclear real-to-sim pipeline： The process of converting real scenes into Minecraft is insufficiently detailed. It is unclear how furnitures are selected, how their poses are estimated, how occlusions are handled, and how the approach compares to existing real-to-sim methods. This raises doubts about how automated the proposed pipeline truly is.\n* If the pipeline relies on an existing furniture set (e.g., MOD assets), the method becomes similar to prior real-to-sim approaches that depend on existing object set. It is not obvious why constructing the environment specifically in Minecraft provides clear advantages. Additionally, the physics in Minecraft appears irrelevant for the presented tasks.\n* Manual refinements reduce practicality: For the VLN task, the authors mention manually refining 15 environments due to data quality issues, but do not detail the required operations or effort. This raises concerns about scalability. Furthermore, if data quality is limited, it is unclear whether using all constructed scenes benefits the semantic occupancy task—does every scene require manual refinement as well?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HpCgq2FuHE", "reviewer": "Reviewer_7kuq", "summary": "This paper introduces World2Minecraft, a pipeline to convert real-world scenes into editable Minecraft environments for embodied AI research using 3D semantic occupancy prediction. The authors find this process is limited by the poor quality of current occupancy models. To address this, they present MinecraftOcc, a large-scale, automatically generated dataset from 156 photorealistic, modded Minecraft scenes (100,165 images) to serve as a challenging benchmark and training resource. Experiments show MinecraftOcc exposes the generalization limits of SOTA models and improves their performance on real-world data (NYUv2) when used for joint training. The utility of the end-to-end pipeline is also shown via a new MinecraftVLN dataset created from the reconstructed scenes.", "questions": "1. Can you quantify the \"meticulous manual refinement\" required for the 15 scenes? How does this manual bottleneck affect the scalability of the World2Minecraft pipeline?\n\n2. Have you considered closing the loop: using World2Minecraft to reconstruct a real-world scene, and then using that reconstructed scene as input for your MinecraftOcc automated data generation pipeline?\n\n3. How do you ensure the coarse 1,452-to-13 class mapping (e.g., \"curtains\" to \"empty\") doesn't introduce significant label noise that negatively impacts model training and evaluation?\n\n4. How do you disentangle the models' poor performance on MinecraftOcc (Table 4) between a true \"generalization failure\" and a simple \"domain gap\" (i.e., real-world models performing badly on Minecraft's specific visual style)?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The World2Minecraft framework is a clever solution for creating editable, high-fidelity simulation environments, bridging the gap between static real-world scans and blocky, low-fidelity simulators.\n\n2. The MinecraftOcc dataset is a significant contribution, providing a large-scale (100k+ images), high-resolution, and perfectly-annotated data source for training and benchmarking 3D semantic occupancy models.\n\n3. The paper effectively demonstrates MinecraftOcc's value. SOTA models perform poorly on it, highlighting generalization gaps , while joint training with it improves performance on real-world data (NYUv2).\n\n4. The authors validate the full pipeline by creating a MinecraftVLN dataset from the reconstructed scenes and successfully training SFT/RFT agents for navigation tasks .", "weaknesses": "1. The World2Minecraft pipeline is not fully automated; its initial outputs were \"suboptimal\" and required \"meticulous manual refinement\" to be usable for the VLN dataset, undermining its scalability claims.\n\n2. The paper has two loosely connected parts: the World2Minecraft (real-to-sim) pipeline and the MinecraftOcc (sim-for-training) dataset. The dataset is not generated from the pipeline, which makes the overall story less cohesive.\n\n3. The \"Base\" MinecraftVLN dataset derived from the pipeline is small (15 scenes) with \"simple instructions\". The authors had to add 5 external community-built maps to achieve sufficient complexity.\n\n4. The mapping of 1,452 mod-specific classes to 13 standard classes for cross-dataset experiments is coarse and potentially noisy (e.g., \"curtains\" and \"lights\" are mapped to \"empty\"), which could confound the joint-training results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "E6LHTG6P2c", "reviewer": "Reviewer_Nuc5", "summary": "This paper focuses on transforming real-world scenarios into Minecraft structured environments, attempting to address issues such as data pollution and insufficient flexibility in embodied intelligent simulation platforms. \n\nThis paper proposes a new VLN method **MinecraftVLN** that enables the agent to complete navigation tasks in the reconstructed scene.\n\nThis paper also proposes a large-scale dataset **MinecraftOcc** to benefit the entire community.", "questions": "Could the authors provide a detailed explanation of the core research question? What challenges did you encounter while addressing this issue that resulted in the final pipeline? I do not perceive the uniqueness of World2Minecraft, nor do I see any significant insights here.\n\nAdditionally, do the authors have better experimental results? In my opinion, the current experimental outcomes do not convincingly demonstrate the advantages of your method and dataset.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "**The research question is important**: Existing simulation platforms (Habitat and AI2-THOR) are not editable, necessitating a real-to-sim approach. However, existing real-to-sim approaches lack physical properties and cannot be applied to downstream tasks. Therefore, the authors propose a solution based on 3D semantic occupancy prediction.\n\n**Constructed two large datasets**: Data collection is a very tedious task. The authors proposes a low-cost collection pipeline to generate a large number of images, indoor scenes, and navigation samples. These have contributed to the community and navigation tasks, alleviating the previous problem of insufficient data.", "weaknesses": "**The method lacks innovation**: The \"World2Minecraft framework\" proposed in the article is essentially a patchwork of the \"single-view 3D semantic occupancy prediction → multi-view fusion → Minecraft command generation\" pipeline, with no groundbreaking design in any of the steps. The entire process lacks original technical support and is more like an \"assembly experiment\" of existing tools. The authors fail to explain the core issues of this pipeline, why previous researchers haven't developed this pipeline, whether the simple assembly of such a pipeline is directly applicable, whether additional problems were encountered during the process, and how the authors propose carefully designed modules to address these issues. I don't see a research-based process, but rather a collection of engineering pieces.\n\n**The dataset's value may be limited**: I greatly appreciate the authors' meticulous effort in designing and collecting datasets for the benefit of the community and research. However, there are some concerns. The MinecraftOcc dataset has 156 scenes. While it contains \"approximately 1,000 rooms,\" it doesn't specify the diversity between scenes (e.g., whether there is overlap in room layout, furniture layout, or lighting conditions), making its value in testing model generalization capabilities unproven.\n\n**The introduction is poorly written**: When discussing the \"flaws of existing methods,\" it first criticizes the \"uneditability\" of Habitat and AI2-THOR, then mentions the \"lack of physical properties\" of NeRF and 3D Gaussian splatting, and finally turns to semantic occupancy prediction methods. This leads to a logical jump and a failure to form a coherent chain of argumentation.\n\n**The results are not very impressive**: The outcomes shown in Figure 4, Table 2, and Table 4 are not very impressive, and I did not perceive the advantages of the method and dataset proposed by the authors.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "i6BI36hLVt", "reviewer": "Reviewer_X8ju", "summary": "This paper introduces **World2Minecraft**, a framework for converting real-world scenes into Minecraft environments using 3D semantic occupancy prediction. Based on the reconstructed scenes, this paper also presents two datasets: **MinecraftOcc**, a large-scale benchmark for indoor 3D occupancy prediction, and **MinecraftVLN**, designed for Vision-Language Navigation tasks. Experiments demonstrate the framework’s scalability and effectiveness in improving model performance and enabling embodied AI research in editable virtual environments.", "questions": "See weaknesses. If these concerns are clarified with solid results, I would be happy to increase my score.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-written, with clear and intuitive figures that make the dataset construction process easy to understand.", "weaknesses": "1. The use of occupancy as an intermediate representation for real-to-sim scene reconstruction raises some questions. According to the paper, occupancy is not directly used to reconstruct Minecraft scenes but instead to identify object instance centers for reconstruction. Does this suggest that object instance or room layout information alone is sufficient for scene reconstruction? Why not use these representations directly, considering they are potentially easier to obtain than occupancy labels?  \n\n2. Based on the results in Table 5, adding **MinecraftOcc** training data only provides a marginal improvement in model performance (mIoU +0.21). This raises concerns about the actual impact of **MinecraftOcc** in improving current occupancy prediction models or supplementing existing datasets.  \n\n3. The paper lacks experiments demonstrating the contribution of the **MinecraftVLN** dataset to existing visual navigation models or its ability to complement existing datasets. Additional experiments are recommended to show that **MinecraftVLN** can improve the performance of current visual navigation models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)