Title: Distilling LLM Prior to Flow Model for Generalizable Agent’s Imagination in Object Goal Navigation
Venue: NeurIPS 2025
Accepted: yes
Keywords: Visual Navigation; Object Goal Navigation
Abstract: The Object Goal Navigation (ObjectNav) task challenges agents to locate a specified object in an unseen environment by imagining unobserved regions of the scene. Prior approaches rely on deterministic and discriminative models to complete semantic maps, overlooking the inherent uncertainty in indoor layouts and limiting their ability to generalize to unseen environments. In this work, we propose GOAL, a generative flow-based framework that models the semantic distribution of indoor environments by bridging observed regions with LLM-enriched full-scene semantic maps. During training, spatial priors inferred from large language models (LLMs) are encoded as two-dimensional Gaussian fields and injected into target maps, distilling rich contextual knowledge into the flow model and enabling more generalizable completions. Extensive experiments demonstrate that GOAL achieves state-of-the-art performance on MP3D and Gibson, and shows strong generalization in transfer settings to HM3D.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes GOAL, a generative method for object-goal navigation. Previous methods build incremental BEV semantic map and apply discriminative method to infer the position to navigate. On the contrary, this paper model objectnav as a generative problem. It proposes a novel framework to distill LLM's common knowledge into a flow-based model, which guesses unseen region to facilitate navigation planning. Experimental results on several benchmarks demonstrates the effectiveness of GOAL.\n\nSTRENGTHS AND WEAKNESSES:\n# Strength:\n1. Model map-based ObjNav as a generative problem is interesting and makes sense.\n2. The technical contribution is sufficient.\n3. During inference, LLM is not required, which may be friendly for on-device deployment.\n\n# Weakness:\n1. For 3D perception, the paper claims \"To mitigate this, we aggregate past RGB-D observations into unified point clouds representations and perform joint segmentation using 3D perception models, inspired by how humans implicitly integrate multi-frame observations.\" However, it seems the authors only leverage per-frame 3D segmentation using sparseconv, without modeling the inter-frame temporal information. Applying online 3D perception may achieve better performance [1][2][3].\n2. One advantage of GOAL is that it does not require LLM during inference. I'd like to know its inference speed compared with other methods. This should be demonstrated in experimental results.\n\n\n[1] Fusion-aware point convolution for online semantic 3d scene segmentation. CVPR 2020.\n\n[2] Memory-based adapters for online 3d scene perception. CVPR 2024.\n\n[3] Embodiedsam: Online segment any 3d thing in real time. ICLR 2025.\n\nQUESTIONS:\n1. The choice of generation. Why generate semantic map rather than other representations like 3D scene graph? State-of-the-art methods like UniGoal leverages 3D scene graph rather than BEV semantic map.\n2. Can this method be extended to more tasks rather than object goal?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe submission proposes GOAL, a generative flow-based framework that incorporates external knowledge from large language models to enhance the agent's ability to imagine unobserved regions in object goal navigation tasks. The method achieves state-of-the-art performance on MP3D and Gibson datasets and demonstrates strong generalization capabilities in transfer settings to HM3D.\n\nSTRENGTHS AND WEAKNESSES:\n## Strength\n- The method outperforms existing approaches on multiple benchmarks, including MP3D and Gibson.\n\n- The submission provides experiments and ablation studies, demonstrating the effectiveness of each component of the framework and its overall robustness.\n\n- The writing and illustration of the submission are convenient for understanding.\n\n## Weakness\n- The primary weakness of the submission is the lack of novelty in methods/architecture. Similar to SGM, it predicts unobserved maps to guide navigation, differing mainly in specific technical approaches. But I also question the necessity of the proposed methods.\n\n- I question the necessity of the proposed methods:\n  - Regarding the \"Distilling LLM Prior\", the data shown in Table 1 also indicate that the improvement brought by LLM is very limited (SR:47.6→48.8). And when using LLMs, the prompts in this submission only provide central objects without specific information about other actual objects surrounding them, lacking environmental distribution information. This approach is insufficient for handling complex environments.\n  - Regarding the \"Flow Model\", Table 2 shows that the dependent couplings method(SR:41.5) outperforms independent couplings (SR:39.0), but the improvement is marginal. Given that dependent couplings rely on extensive hyper-parameter tuning (D.1 Hyper-parameters tuning), I find it difficult to be convinced of the effectiveness of the proposed method.\n\n- Demonstrated in \"D.1 Hyper-parameters tuning\" of the submission, the hype-parameters mainly relies on experimental testing and optimization. The performance fluctuates significantly with several hyper-parameters, and there is a lack of reasonable explanation, which deepens the concerns about the reliability of the method.\n\n- Although the experiment tested LLMs from different manufacturers, they are all LLMs with a large number of parameters (for example, DeepSeekR1 has 671B parameters), lacking the ability test of small-parameter models. However, works such as SG-Nav, often achieve good results on models with relatively small parameters (such as LLaMA 7B in SG-Nav).\n\nQUESTIONS:\n- When using models that can run on consumer-grade GPU, such as LLaMA 7B, for training, what is the performance of this method?\n\n- Since the generated content of LLM only includes distance and confidence, is there a significant gap in the content generated by different LLMs? Please provide quantitative analysis or examples for illustration.\n\n- For hyper-parameters tuning, is there a more reasonable interpretation of the current performance curve? Is it possible to propose a more systematic optimization approach?\n\n- Table 2 takes the model's real-time performance into account, and dependent couplings achieve a significant reduction in inference time (from 17.13ms to 12.38ms). However, for an episode, it seems that map prediction for waypoint selection does not need to be performed at every step. So, how much delay will be caused in an entire episode?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work presents a generative flow-based framework that models the semantic distribution of indoor environments by bridging observed regions with LLM-enriched full-scene semantic maps.\n\nSTRENGTHS AND WEAKNESSES:\nThe authors show state-of-the-art performance on Gibson and MP3D datasets. Moreover, they show good generalisation on HM3D dataset.\n\nThe authors have not compared the performance of their model with RIM [1].\n\nThe generated maps rely on an LLM prompt to predict numerical values of distance and confidence, alongwith reasoning. I think there would be some randomness in the output of values for distance and confidence. As LLMs are autoregressive models, I am not sure how reliable they would be at predicting numerical values, also given that they are prone to hallucination. Perhaps, you could train a transformer encoder to predict the values of distance and confidence, instead of prompting an LLM.\nMoreover, the qualitative results in the appendix in Figure 8 show gym equipment and fireplace in multiple, distant places in the same environment. Usually in a house, all the gym equipment is present in one area and not scattered in different places. So, I am not sure how accurate and reliable are generated maps.\n\n[1] Chen, Shizhe, Thomas Chabal, Ivan Laptev, and Cordelia Schmid. \"Object goal navigation with recursive implicit maps.\" In 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 7089-7096. IEEE, 2023.\n\nQUESTIONS:\n1. Does the LLM give the same numbers for the distance and confidence, even when the same prompt is used as the input of the LLM? \n2. In Figure 8, the gym equipment and fireplace seem to be in multiple, distant places in the same environment. How accurate are the generated maps? Is there a numerical method to measure this?\n\nLIMITATIONS:\nI am not sure how reliable are the numerical values of distance and confidence that are an output of LLM. As the map is generated using this information from the LLM output, I am unsure how reliable is the quality of generated maps.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a framework for ObjectNav based on semantic map completion with a flow-matching-based generative model. It uses an LLM to perform a form of data-augmentation, injecting objects into semantic maps during training based on a pre-trained LLM’s prediction about likely co-occurring objects. At test time, the flow-matching generative model completes a plausible semantic map of the scene given the history of observations, which is then used to select navigation targets for ObjectNav. The method is compared against several baselines for navigation via envrionment imagination, showing SOTA and transfer to environments that the generative model is not trained on.\n\nSTRENGTHS AND WEAKNESSES:\n*Strengths*\n- This paper proposes a strong system for object navigation with several good ideas and engineering choices. Their LLM based map-augmentation scheme incorporates priors from LLMs effectively while avoiding the need for running an LLM at inference time, and the use of flow matching for map inpainting is a natural upgrade in methodology for this line of work based on more current inpainting research. \n- Paper is well written and the language is clear. Many related works are compared against and many relevant ablations are included. \n- The results are strong and clearly give SOTA performance.\n\n\n*Weaknesses*\n- Missing details on the “Effectiveness of data-dependent couplings” ablation: Inpainting/outpainting with diffusion models is already a well studied problem. While the inpainting formulation used in this paper is compelling and could make sense, there are very few details about the baseline used to justify the contribution of this non-standard inpainting scheme. UNet style models typically concatenate the conditioning to noise [1] while several different conditioning schemes are used for DiT based models [2,3]. This paper would be stronger if these methods were explicitly compared against, or one of these inpainting schemes were adopted directly. As is, there is not enough to justify the proposed inpainting scheme as a contribution.\n\n- Incomplete ablations against SGM. Some of the major differences between this work and SGM that need more experimentation:\n1) Scene segmentation vs per-frame segmentation\n2) The use of flow matching vs MAE\n3) LLM conditioning techniques (conditioning on features vs directly augmenting maps)\nSGM claims to do segmentation using single frame models (Mask-RCNN, RedNet) while this work uses scene segmentation. In Table 3 this work reports a 5 point SR increase from using scene segmentation. If the same increase were found from simply using scene segmentation with SGM, then SGM would actually be out-performing this method on MP3D, implying that either MAE or the SGM language conditioning technique is superior to the techniques from this paper. This paper would benefit from a comparison against SGM using scene segmentation, and SGM using this method’s language augmented maps as part of training.\n\n[1] Rombach, Robin, et al. \"High-resolution image synthesis with latent diffusion models.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022.\n\n[2] Nitzan, Yotam, et al. \"Lazy diffusion transformer for interactive image editing.\" European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2024.\n[3] Wu, Xian, and Chang Liu. \"DiTPainter: Efficient Video Inpainting with Diffusion Transformers.\" arXiv preprint arXiv:2504.15661 (2025).\n\nQUESTIONS:\nBased on equations 6,7,8 it seems as if there is no protection for augmenting objects being placed across a wall from it’s co-occurring object, making it’s geodesic distance much farther than the predicted distance. \n\nIt’s not clear to me exactly how segmentation is performed. As is it seems as if this work trained a sparse convolutional network from scratch on RGB pointclouds. If this is is the case there should be more details included about training.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-05"}