Tag Map: A Text-Based Map for Spatial Reasoning and Navigation with Large Language Models

Authors: mike  zhang
kaixian  qu
vaishakh  patil
cesar  cadena
marco  hutter
Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-06-07 03:13:27

Abstract: Large Language Models (LLM) have emerged as a tool for robots to generate task plans using common sense reasoning. For the LLM to generate actionable plans, scene context must be provided, often through a map. Recent works have shifted from explicit maps with fixed semantic classes to implicit open vocabulary maps based on queryable embeddings capable of representing any semantic class. However, embeddings cannot directly report the scene context as they are implicit, requiring further processing for LLM integration. To address this, we propose an explicit text-based map that can represent thousands of semantic classes while easily integrating with LLMs due to their text-based nature by building upon large-scale image recognition models. We study how entities in our map can be localized and show through evaluations that our text-based map localizations perform comparably to those from open vocabulary maps while using two to four orders of magnitude less memory. Real-robot experiments demonstrate the grounding of an LLM with the text-based map to solve user tasks.

Main paper: 
 introduction: 
 Scene understanding is fundamental for robots to generate actionable plans within their environment. A popular task for evaluating scene understanding is Object Goal Navigation (ObjectNav) [1, 2] , where the robot must navigate to a target object within an unknown scene. Two components have emerged as recurring aspects in recent works on ObjectNav. First, a Large Language Model (LLM) which plans navigation goals, and second, a map that serves as a memory module storing observed objects and which can ground the reasoning of the LLM.Map representations in previous works can be classified as either explicit or implicit. Explicit maps label locations as a class from a fixed set of classes [3, 4, 5] . This limits the tasks the map can handle but has the advantage of directly reporting the classes it contains. In contrast, implicit, or open vocabulary maps build upon the line of work in computer vision starting from CLIP [6] , later extending to object detection [7, 8] and segmentation [9, 10, 11, 12] , based on vision embeddings that are queryable by arbitrary text. Recent works in ObjectNav have considered maps where a location stores an embedding instead of a label [13, 14] . In theory, such maps can represent any objects, rooms, and affordances. In practice, there is no guarantee that the embeddings will capture the desired information. Moreover, embeddings cannot directly report what entities they represent. Instead, they must be queried to retrieve potential matches. For simple tasks along the lines of "go to <thing>", such maps can be directly queried for <thing>. However, additional processing is needed to identify relevant queries for more complex tasks lacking an explicit goal object or region.We propose a text-based map that can represent an extensive amount of semantic classes explicitly by leveraging multi-label image classification models trained to recognize thousands of semantic classes. Our map called the tag map, is a memory-efficient unstructured database storing viewpoints and the recognized entities for each viewpoint as text tags. Despite the minimal amount of information contained within, we find that tag maps can produce 3D localizations for both objects and rooms/regions with sufficient precision and recall to serve as useful navigation goals. Moreover, the explicit text nature of the tag map lends itself well for use in grounding LLMs.Our main contributions are the following:• A text-based map, building on top of large image classification models, capable of representing thousands of semantic classes from common and rare objects to rooms and regions.• A simple, yet effective, method to localize in 3D, the semantic classes from the text-based map.• A method for grounded LLM spatial reasoning over the text-based map to generate plans from given user task specifications.Through quantitative experiments, we demonstrate that the localizations from our map perform comparably in precision and recall against state-of-the-art open vocabulary maps while storing orders of magnitudes less information. Real-world robot experiments demonstrate the effectiveness of our map at grounding a LLMs to reason from task descriptions and generate feasible navigation plans. 
related work: 
 Semantic Map Representations. A semantic map represents the scene geometry and assigns semantic classes to that geometry. Earlier works leveraged 2D or 3D segmentation and object detection models trained on a fixed set of classes to generate class annotations which are stored in the map [15, 16, 17, 18, 19] . Semantic maps can be extended into scene graphs by further annotating relationships between labeled instances [20, 21, 22] , such as objects next to each other or objects contained in a room. Recent interest has been directed towards open vocabulary representations where rather than explicit classes, the map is annotated with embedding vectors queryable by arbitrary text. Such maps have attached embeddings to dense scene geometry [23, 24, 13] , to segmented instances or objects [25, 26, 14] , as well as into scene graphs [27, 28, 29] .Object Goal Navigation. While some works in ObjectNav applied end-to-end learning [30, 31, 32, 33, 34, 35] or explored semantics-agnostic exploration strategies [36] , many works have opted to maintain a map during runtime. Storing what has already been observed in a map can help in picking semantically relevant locations to explore further [37, 38] , as a more privileged input to a learned policy [3, 35] , to explore unseen frontiers [39, 40, 41, 42, 43] , and to reduce exploration when given followup goals [5] . Multiple works have applied LLMs to reason over the map semantics. Given a target object, commonsense reasoning from an LLM is useful for suggesting related objects and rooms for more efficient selection of regions, frontiers, or objects to explore [40, 42, 43, 44] .LLMs have also been applied to translate natural language user task specifications into actionable navigation goals within the map [13, 14, 5] . To give you the context of the environment, you are provided with a list of tags. Each tag describes something likely to be found in the environment.-dining room -fireplace ... Multi-Label Classification. A well-studied computer vision task [45, 46] , also known as image tagging, where the objective is to recognize the semantic classes contained within an image. In contrast to segmentation or detection, tagging does not require localization of the classes within the image. Tagging models are a mature technology with several commercially available APIs [47, 48, 49] that can recognize thousands of classes. In this work, we use the Recognize Anything (RAM) family of open-source models [50, 51] on par with most commercially available APIs. 
text-based map representation: 
 The core of our approach is the text-based tag map which stores the unique entities (tags) recognized by an image tagging model and relates them to the viewpoints they were recognized from. Its data structure can be implemented as a hash table, where the keys are the unique recognized entities and the values are references to the corresponding viewpoints. Each viewpoint stores a unique ID, its camera pose, and a depth statistic used later for localization. We emphasize that the viewpoint RGB and depth images are not stored in the map. An overview of our framework is shown in Fig. 2 . Building a tag map from RGB-D images and poses of a set of viewpoints is shown in Fig. 2 . From the depth image, we compute the mean and median depth, along with the 80th quantile depth value representing the far plane distance of the viewpoint frustum. A filtering step checks the mean and median depth and discards the frame if it is considered a close-up view. The RGB image is then forwarded through a tagging model ensemble, where each ensemble member receives a different centered cropped version of the image that removes a small portion of the edges. The final tags are the tags agreed on by the ensemble, filtering out false positive tags due to poorly observed objects around the edges of an image. Effects of the depth and crop ensemble filters are visualized in Fig. 3 . The tags, poses, and viewpoint frustum distances are registered in the tag map.Given a tag, multi-view consistency over the tag's corresponding viewpoints is used to generate localized regions in the tag's near vicinity, as demonstrated in Fig. 4 . The procedure is similar to Space Carving [52] , where viewpoint intersections are used to reconstruct 3D geometry. Unlike Space Carving, viewpoints in the tag map may be generated from partial views of an entity. The large vocabulary of image tagging models allows our framework to localize uncommon categories such as "thermostat", "cowboy boots" and "basketball court".Localizing a tag begins with retrieving that tag's corresponding viewpoints and mapping each viewpoint to its 3D frustum using the stored far plane distance. The space covered by the viewpoints is voxelized and each voxel is assigned votes corresponding to the number of viewpoints that contains it. The voxel votes represent a likelihood of the tag over the space. Localization proposals at a confidence level v are extracted by thresholding out voxels below v votes and clustering the remaining voxels using DBSCAN [53] . A confidence level of n indicates that a proposal is consistent across at least n viewpoints. We set a series of clustering thresholds to capture not just the global but also local maxima of the voxel likelihood. Lastly, a non-maximum suppression step is applied to remove proposals that contain within it proposals of higher confidence levels. Fig. 5 shows example coarse localizations for various tags of objects and locations.We opt for the simple method of grounding the LLM by appending the list of unique tags from the tag map into the prompt of the LLM, as shown in Figure 2 . The LLM is additionally given access to an API for requesting information from the tag map. The API exposes the following functions:• localize tag(tag): Returns the proposals for the tag along with their confidence levels.• region region dist(r1, r2): Computes the distance between regions r1 and r2.• point region dist(p, r): Computes the distance to reach region r from point p.The API enables the LLM to reason spatially about the tags in its prompt. For example, given a query of "go to the kitchen fridge", the LLM can localize "kitchen" and "fridge" and set the navigation goal to be the fridge proposal with a small distance to a kitchen proposal. 
evaluating coarse-grained localizations: 
 The coarse localizations produced by our framework generally do not precisely contain the localized entity as seen in Fig. 5 . However, such localizations can still be useful navigation goals for reaching the entities. Evaluating coarse localizations using metrics such as Intersection Over Union (IoU) fails to assess localizations in terms of "imprecise but still useful" navigation goals. For example, localizing a small area within a bedroom would be a useful goal for navigating to the bedroom but have poor IoU against the room's bounding box label.Our goal is to produce a proposal P that is useful for locating the desired entity. In other words, if the robot reaches the region P , it should be close enough to an instance of the desired entity E ∈ E. This is aligned with the objective of the Habitat ObjectNav Challenge [2] benchmark, where an agent is successful in navigating to a goal object if the distance between the agent and the nearest goal object is below 1.0 m1 and the object can be viewed with basic yaw and pitch movements of the camera. We quantify the usefulness of P as the expected length of the shortest path an agent must travel from P to reach any desired entity instance in the set E. We denote this quantity as the Proposal to Entities Distance (P2E) defined asEQUATIONwhere d(•, •) is the shortest path length function. We can evaluate P2E over a set of proposals and classify proposals with P2E under a set threshold (e.g. 1.0 m) as relevant. We refer to the portion of relevant proposals for that threshold as the Precision at Threshold and use it to measure the precision of that set of proposals.Symmetrically, we define the Entity to Proposals Distance (E2P) which evaluates for an entity instance E the shortest expected path length from E to reach any proposal for that entity. For a set of entity instances, we compute the portion with E2P below a threshold for a given set of proposals as a measure of the recall for that set of proposals, which we denote as the Recall at Threshold.The expected shortest path length is difficult to compute exactly so we approximate it by constructing a grid graph over the scene that avoids collisions with the scene geometry, allowing the computation of approximate shortest paths. Nodes of the grid graph are assigned to the proposals and entity instances and the expectation is approximated by averaging over the assigned nodes. Computing the approximations of P2E and E2P is equivalent to computing the Directed Average Hausdorff Distance (DAHD) [54] , a metric commonly used in medical image segmentation.Evaluations are performed on the Matterport3D dataset [55] . The evaluation is separated into object and room/region evaluations. Object evaluation is done on the 21 common object classes defined by the Habitat ObjectNav Challenge [2] . For regions, we evaluate all classes in Matterport3D except for "other room", "junk", "no label", "outdoor", "entryway/foyer/lobby", and "dining booth". These classes were ignored as they did not refer to actual rooms or corresponded to ambiguously labeled regions. When evaluating a tag map, each class label is mapped to a list of corresponding tags. Proposals are generated for all the corresponding tags that exist in the tag map and are grouped to form the set of proposals for the class. Unless stated otherwise, we report the precision and recall over all classes by averaging the precision and recall across classes. Further details on the implementation of the evaluation are found in Appendix D. 
comparison with embedding based maps: 
 Tag map localizations are compared against localizations from open vocabulary embedding-based maps using the proposed precision and recall metrics. We chose to compare against OpenScene [23] for 3D segmentation using a point cloud representation with an embedding for each point, and OpenMask3D [25] for 3D instance segmentation which stores 3D instance masks with embeddings for each mask. We query all embeddings with all class labels at once, then assign each embedding a class based on the best matching class following Peng et al. [23] . Another option is to query classes individually and assign embeddings a class if they match above a threshold as done in Lu et al. [26] . However, picking a suitable threshold is difficult and thresholds may not transfer across scenes. For OpenScene, we cluster the segmentation output to obtain instance bounding boxes following Lu et al. [26] The comparison results are reported in Tables 1 and 2 . For objects, the tag map outperforms both embedding-based maps for most precision and recall thresholds. However, the embedding-based maps have better precision when the threshold is set to a smaller value (e.g. 0.1 m), due to these methods storing geometric information to produce tighter instance bounding boxes. For regions, OpenMask3D achieves the best precision across thresholds as it produces smaller instance proposals that are often precisely located within the region label. However, these smaller proposals have lower recall as the larger size of region labels results in higher E2P. Conversely, OpenScene tends to produce much larger proposals for region classes resulting in lower E2P and better recall. The tag map achieves good precision for region classes but the worst recall compared to the other methods. Note that larger precision-recall thresholds are used for the regions as region labels are much larger than object labels. We also compared the memory usage of the maps in Fig. 6 . The tag map uses much less memory than the embedding-based maps as storing the embedding vectors requires significantly more memory than text-based tags. The comparison is especially stark against OpenScene which stores per-point embeddings for a scene point cloud.The recent work of Chang et al. [5] proposed to use store viewpoint CLIP embeddings and to search over the embeddings when given an object query. We modify our coarse localization method to retrieve viewpoints using CLIP instead of using the relevant tags. To make the comparison fair, for both methods, we retrieve only the top K most confident viewpoints. For tags, we use the tag prediction confidences from the tagging model. We also include the unmodified tag localization method as a baseline, denoted as "all views". Precision and recall comparisons for the two methods at different thresholds are reported in Fig. 7 . Retrieval via tags consistently outperforms CLIP Figure 7 : Comparing the effect of retrieving viewpoints using tags against using CLIP embeddings on coarse localization precision and recall. We compare retrieving the top {1, 5, 10, 20} most confident viewpoints for each method. For using tags, we also include "all views", which retrieves all tagged viewpoints. Using tags consistently outperforms CLIP embeddings in precision for both object and region classes as well as consistently better recall for objects. retrieval in terms of precision for the same K, suggesting that the performance of image embeddings is behind that of models specialized in recognition. For object classes, tag-based retrieval also outperforms CLIP in terms of recall. We observe a general trend of trading off precision for recall for increasing K which is to be expected. For object classes, retrieving all relevant viewpoints ("all views") results in a significant increase in recall while this only produces a slight increase in recall for region classes. This is because a single scene only contains a few large region instances but many smaller object instances. Therefore, only a few views are necessary to recall all region instances.Recall For conciseness, crop ensemble filtering is referred to as "ensemble" and depth filtering as "depth".We ablate the tagging model, the crop ensemble filter, and the depth filter. We compared the tagging models RAM [50] and its successor RAM++ [51] , finding that RAM++ did not outperform RAM in our evaluations. This is likely because RAM++ is designed to improve the recognition of rare classes and our evaluation is limited to more common classes from Matterport3D. The depth filter was found to have an insignificant impact on the evaluation performance. We believe this is because the depth filtering removes uninformative views which often generate incorrect tags that are generally unrelated to the classes considered in the evaluation (see Fig. 3 ). Removing the crop ensemble filter results in a notable drop in precision with a smaller gain in recall, suggesting that the crop ensemble filter provides a favorable tradeoff between filtering false positive tags and over-filtering true positives. Ablation results over object classes are reported in Table 3 while region class results are reported in Appendix A.1.There are some faucets where I can get water Figure 8 : An example of grounded navigation using a tag map on a real robot. A user asks the system to assist with some task through a chat interface. First, the LLM uses the tag map context to propose relevant entities for the task. Next, the LLM queries the tag map API for additional information on the entities of interest which it then uses to reason spatially and produce a navigation plan.Integration of the tag map with an LLM for grounded navigation was demonstrated on the legged robot ANYmal [56] . A dataset from a lab/office scene was collected and viewpoint poses were computed using COLMAP [57, 58] to construct the tag map and a scene mesh. The mesh is used for visualization and to build a pose graph connecting neighboring viewpoints in the tag map that have a traversable path to each other. The pose graph produces global path plans that are tracked by a local planner based on Cao et al. [59] . We used GPT-4 [60] as the LLM and leveraged its function calling capabilities to integrate it with the tag map API. The LLM is allowed to make multiple function calls before giving its response. This allows the LLM to, for example, localize relevant tags using localize tag() and then reason about them spatially using region region distance() in a single response. This can be thought of as the LLM generating scene graph annotations between the localized proposals. We also prompt the LLM to consider the localization confidence levels such that it will prefer selecting more confident localizations to navigate to.A user is given a chat interface for conversing with the LLM which allows the LLM to ask the user for additional clarifying information and for the user to provide follow-up tasks building on top of previous queries. Fig. 8 demonstrates examples of grounded navigation addressing tasks given by a user. We focus on tasks where the relevant entities for solving the task are not explicitly mentioned and must be inferred by the LLM through the tag map context. In addition to the robot demonstrations, a more thorough evaluation of the Tag Map grounded navigation pipeline over a set of test user queries is presented in Appendix F. 
conclusions, limitations, and future work: 
 This work presented a text-based map that can represent an extensive set of entities, including rare objects and regions, and coarsely localize them in 3D. Being text-based allowed the map to seamlessly ground an LLM through its prompt. Interfacing the LLM's function calling with the map enabled further spatial reasoning by the LLM on the localized entities. The tagging model used in this work was prone to producing false positive tags which risked polluting the context given to the LLM. However, the false positive tags were often spurious concepts having little to do with the more practical tasks we tested and were mostly ignored by the LLM. Though sometimes the LLM generated infeasible plans due to such tags. For the evaluations conducted on Matterport3D, we were limited to the viewpoints available in the dataset.  
 
 Appendix: You are a helpful robot assistant.To give you some knowledge of your environment, you are provided with a set of tags. Each tag describes something that's likely to be found in the environment. You can use these tags to help you with assisting the user.Note that the tags are not perfect. Some tags may be incorrect and the tags do not cover everything in the environment. Therefore, try to pick the tag which you are most confident in.You can query for the regions in the environment where a tag is localized and also get the confidence level for each region.When making a statement about a tag, do not say that the tag is definitely in the environment, instead reference the confidence level.If there are no tags directly related to the user's request, suggest tags which may be related to the user's request and ask the user if they would like to know more about the suggested tags.List of tags in the format ' Matterport3D includes a raw class label for all labeled objects. For example the raw labels "arm chair" and "bean bag chair" are grouped into the common object class of "chair". Both "arm chair" and "bean bag chair" are within the set of classes able to be recognized by the tagging model. Because of this, for each common object class, we identify the corresponding raw classes and map them to matching tags within the vocabulary of the tagging model. We also manually identified tags that corresponded to a common object class and included them in the mapping. The mapping for the common object classes to tags are defined as follows:[id] -[tag]' 0 -<tag 0> 1 -<tag 1> . . . N -bathtub: bath, jacuzzi We relabeled regions labeled "familyroom" and "lounge" as "living room" since these labels described analogous regions. Similarly, we also relabeled regions labeled "toilet" as 'bathroom".To evaluate embedding-based methods, the class labels are mapped to strings which are then encoded into text embeddings for querying the map embeddings.We followed the authors of OpenScene and OpenMask3D and applied the prompting method of "a {} in a scene" for object class labels. We modified the prompt to be grammatically correct for the following object classes: For viewpoint retrieval using CLIP, we mapped each label to a string and used the ImageNet80 ensemble prompting method from Radford et al. [6] to create the text embedding.chestFor object classes, we used the label directly as the string, except for the following classes where we modified the string to be more grammatically correct or more accurately reflect the class semantics: chestThe performance of the LLM and Tag Map grounded navigation pipeline is evaluated on a set of user queries that could be asked of a home assistant robot. The evaluation is done on a scan of the lab environment used for the real robot experiments in Sec.5.4. We assume that the environment is entirely traversable such that the robot can reach any goal location from any starting location.A total of 25 user queries were evaluated. The grounded navigation for a query is considered successful if the LLM grounded on the Tag Map context suggests an instance of a semantic class that is relevant to addressing the query and if that instance is correctly localized from the Tag Map The image tagging model mistakenly recognizes an umbrella in the image.It's getting quite cold in here boiler ✗The boiler tag describes a kettle, but the LLM misinterprets it as a home heating boiler.It's getting quite cold in here, could you bring me something? coat ✓In contrast to the previous query, the LLM interprets this query as bringing an item to warm the user instead of trying to warm the space .Take my phone and charge it charger ✓Help! my shirt is on fire extinguisher ✓ Please prepare a location for the party this evening terrace ✓ OpenScene wrongly labels a table that looks like a bed. OpenMask3D wrongly labels a wall as a bed.8th Conference on Robot Learning (CoRL 2024), Munich, Germany.In the challenge, another 0.1m is allowed, thus the distance threshold is actually 1.1 m. 


Decision: Update:
After the rebuttal, the reviewers are in consensus about the paper's novelty, contribution, and execution. The proposed method works well and seems easy to adopt by others. I will recommend acceptance for this paper.

Original:
The reviewers find the proposed method novel and interesting, but they share some concerns regarding its generalizability under different scenarios as well as various experimental details. I ask the authors to carefully address all reviewer comments during the rebuttal period.

Reviews: review: title: title: Well Written and Well Motivated Paper with Some Missing Evaluations
paper_summary: paper_summary: This paper proposes a memory efficient text based map for storing recognized entities and their viewpoints as text tags. This information is sufficient to produce 3D localizations for objects, rooms, and regions at sufficient granularity for vision-based navigation. These localizations, when combined with an LLM provided with a simple API interface, enables flexible generation of navigation plans given a task specification from a user.
main_review: main_review: The paper is written quite clearly and is well motivated. Tag maps have significantly lower memory footprint compared to embedding based maps and I really appreciate the thorough nature of the ablations and the analysis of the precision/recall of localization from tagmap vs. a CLIP-based retrieval. The robot results on the ANYmal platform are also nice, but lack sufficient detail for me to understand whether the stated improvements from tagmap actually manifest when performing grounded navigation.

It is however a bit strange that tag maps exhibit such low performance in terms of recall on region classes but such good recall for objects. Further discussion regarding this discrepancy would be much appreciated. A big weakness of the paper is that there are no quantitative results provided regarding how often the tagmap based navigation system (when integrated with the LLM) is actually able to reach the desired regions. There is a nice qualitative example showing how tag map is applied to navigation on the ANYmal platform, but there are no quantitative results evaluating the full system. I feel that this is quite important for this work, because tag map does not uniformly outperform embedding based maps in terms of precision/recall for both objects and region classes. As a result, it is critical to evaluate whether tag map can lead to concretely better (or at least similar) navigation results when combined with an LLM planner compared to existing embedding based maps.
questions: questions: 1. Could you expand on why exhibit such low performance in terms of recall on region classes but such good recall for objects?
2. Can you provide concrete quantitative results evaluating how often tagmap based planning leads to the robot navigating to the correct location in either simulation on real experiments?
3. Can you repeat #2 with embedding-based mapping techniques to establish that tagmap can at least maintain their performance while achieving a lower memory footprint?

If the above points are addressed, I would be very happy to update my score.
limitations: limitations: 3
review_summary: review_summary: Generally, I am positive on the paper and think that tagmap is well motivated and explained quite clearly. The main issue with the paper is the lack of quantitative metrics indicating whether tagmap can achieve similar grounded navigation results when integrated with an LLM-based planner when compared to embedding based maps. If this comparison can be clearly established quantitatively, then I am happy to update my score, but as it stands I am on the fence between Accept/Reject, but leaning Accept.
title: title: Well written paper introducing a novel approach for text-based semantic mapping and localization in unseen environments for object goal navigation tasks. Answers to some questions regarding the inference time of this approach, utility in real-time applications, incorporating feedback and comparisons to 3D scene graph based methods are desired.
paper_summary: paper_summary: The paper proposes a method for building text-based maps of unknown scenes for object goal navigation tasks. The map consists of viewpoint IDs and associated semantic classes recognized in the image by a multi-label classification model. Course localization of a desired tag is done by reconstructing the 3D geometry using RGBD images at the tag’s corresponding viewpoints. Finally, an LLM, equipped with localization and relative distance calculation APIs, can be prompted with the tag map to find a path to the desired tag. Course localization from these text-based maps perform comparably to open vocabulary maps while using orders of magnitude less memory.
main_review: main_review: Strengths:

The paper is well written and provides an interesting new approach for building text-based maps with thousands of  semantic classes.
 
Extensive evaluations and comparison to competitive baselines show that these text-based maps perform comparably to open vocabulary maps while using orders of magnitude less memory

Real world experiments in  a lab/office scene using the legged robot ANYmal demonstrate how a user can interact with the LLM and answer clarifying questions 

The authors clearly discuss some limitations of this approach and interesting directions for future work

Ablations are very useful in understanding the different components of the method


Weaknesses:

It seems like, in the proposed approach, using LLMs for planning is not necessarily needed. If the task is clearly specified with both the region and the entity, the relevant tags can be directly extracted from the tag-map. The authors point out that they focus on tasks “where the relevant entities for solving the task are not explicitly mentioned and must be inferred by the LLM through the tag map context”. This is a fair assumption but not critical to the overall method.

Although it is advantageous to have low memory usage, a much more desirable feature for real-time grounded navigation tasks can be low inference time. A discussion on the how much slower is this approach compared to prior works (OpenScene, OpenMask3D) could be useful
questions: questions: Can the proposed text-based map be generated in real time? Additionally, given a partially constructed map can a good course localization be done if the object is detected and terminate exploration? Furthermore, can an LLM be used to direct exploration to relevant parts of the scene using the partially built tag-map?

How long does it take to generate the course localizations and plans using the proposed method as compared to prior work. Can it be used for real-time navigation tasks?

If the proposed localization and plan turns out to be incorrect what are effective/efficient ways of replanning?

Can the proposed approach be extended to control in continuous environments (VLN-CE tasks) i.e. without relying on matterport3D viewpoints? And, how?

How does this method compare to 3D scene graph representations such as ConceptGraphs, which are also open-vocabulary, object-centric and compact?
limitations: limitations: 2
review_summary: review_summary: The work presents an interesting new approach for building text-based maps with thousands of semantic classes. Localization is done only for the desired tags thus requiring much less memory. This is a novel approach that can be useful for object-goal navigation tasks.
title: title: Review of Tag Map: A Text-Based Map for Spatial Reasoning and Navigation with Large Language Models
paper_summary: paper_summary: The paper presents a method to generate a data structure of localized tags from scene data. Semantic classes are obtained from the map through localization. The generated map is then used as context for a LLM in order to assist in navigation of a robot.
main_review: main_review: # Quality
The paper is well-polished. The problem of using implicit tag maps rather than explicit tag maps is defined in the beginning, and the paper establishes the context of the problem in the introduction. The method is accompanied by benchmarks and real-world experiments with a quadrupedal robot navigating a building floor. Comparisons are made against existing embedding-based tag maps. Ablation tests are also performed on the components of the tag map generation pipeline.

# Clarity
The paper explains the motivation for the project, as well as the methods used in the approach. The process of creating the tag map is explained, including the step involving localizing the tags with respect to the map viewpoints. The paper defines the metrics used in evaluating coarse-grained localization. Limitations could use some more elaboration, such as the presence of false-positive tags.

# Originality
The paper makes use of existing models in order to generate the data structure, and it also uses an existing LLM to evaluate the effectiveness of the model. The novelty comes from the data pipeline that takes RGB-D data as input and produces the tag map. GPT-4 is used as the LLM that calls the exposed API of the tag map.

# Significance
An explicit text map allows the use of separately-trained models to use the data structure without additional training on implicit embeddings. The approach can expose an API for LLMs to query in order to obtain positional data for text queries.

# Strengths
- Project is readily applicable to other uses in robotics and AI.
- Straightforward method makes implementation trivial.

# Weaknesses
- The paper mainly reuses existing models to generate the text map and does not feature any new model architectures.
questions: questions: How often does the approach produce false positive tags? Are the false positives more common with certain tags than others?
limitations: limitations: 2
review_summary: review_summary: The paper applies existing models to produce a useful data pipeline to produce textual maps of scenes. This has obvious applications in robot planning and as queries for LLMs. No new models were presented in the paper.
title: title: A nice novel and compact representation of the scene
paper_summary: paper_summary: The authors proposed to associate text tags and their corresponding bounding boxes generated in 2D with overlapping 3D view frustums to create a semantic scene representation.
main_review: main_review: Strengths:
1. In short, the idea proposed in the paper is novel and the experiment showed its effectiveness and compactness compared to open-vocabulary mapping alternatives like OpenScenes and OpenMask3D. However, I have a few doubts regarding the evaluation which will be described in the next session. 
2. The flow of the paper is excellent and the idea is concisely described. However, some design details are missing from the paper or appendix.

Weakness and questions:
1. The idea of using overlapped frustum to represent objects is indeed compact but it has two major problems: (1) the frustum is prone to be under-segmented in 3D, meaning that the points inside the overlapped frustum tend to contain irrelevant artifacts, and (2) the quality of the overlapped frustum is highly dependent on the directions and positions of those multi-views associated with the same tag. Regarding (2), for example, you need to view objects from more than 2 views with large baselines to have constraints on the 3D space for those 2D tags. If an object is viewed only once which might happen during exploration, the frustum might represent a large space that is associated with a certain tag. As a result, the method might work well in a closed space where the objects are close to the wall which helps constrain the frustum but might fail easily in open and clutter space with objects in the center. This implicit viewpoint assumption is mentioned in the limitations but I think it is a major issue of the method and fatal to the generalization ability of the method.
2. The metrics P2E and E2P are not fair and biased. Undersegmentation or undersegmentation are preferred in this metric. Imagine that the predicted point cloud (or bbox) is larger than the desired target point cloud (or bbox) and is the superset of the target, the path connecting the two will be 0. So is the other way around. I would like to see the 3D bounding box precision or recall, and the navigation (or planning) success rate from a random position to the retrieved object (stop position <1 meter) in the simulator which are reproducible.
3. Tables 1 and 2 do not mention P2E and E2P which are introduced in Sec. 4.1. I don't know what distance is used in the table. I hope the author can address this confusion by adding captions under the table or mentioning it in the experiment part when introducing the table.
4. I expect some qualitative results comparing OpenScenes, OpenMask3D, and the proposed method like visualizing the bounding boxes of some instances. I think it might be also a good way to help authors figure out the potential drawbacks of the method.
5. I wonder why the authors only limited the semantic classes to 21 classes in Matterport3D which supports 40 categories. Excluding some meaningless labels, there are still at least 30 meaningful categories in the dataset. I hope the authors can justify their choice.
6. More details regarding the depth statistics are needed in the paper or appendix. Line 106-107 say "mapping each view106 point to its 3D frustum using the stored depth statistics." but I think it is a bit ambiguous. What is depth statistics and how is it exactly done?
questions: questions: The short questions listed here are corresponding to the bullet points in the review weakness section. Please refer to the review section for more details and answer them:
1. The method is prune to under-segmentation and the quality of the overlapped frustum is highly dependent on the viewing points. How did you solve this?
2. The metric of P2E and E2P are not fair and biased. Correct me if I am wrong. But I would also like to see the 3D bounding box precision or recall, and the navigation (or planning) success rate from a random position to the retrieved object (stop position <1 meter) in the simulator which are reproducible.
3. Tables 1 and 2 do not mention P2E and E2P. What is the threshold distance used in the tables? State it in the caption.
4. Is it possible to show some qualitative results comparing to OpenScenes and OpenMask3D?
5. Why do the authors only consider 21 categories among 40 in Materport3D?
6. Explain more regarding line 106-107.
limitations: limitations: 2
review_summary: review_summary: If the authors can address my concerns mentioned in the review, I will raise the recommendation to one level higher.