SLAMuZero: Plan and learn to Map for Joint SLAM and Navigation

Venue: icaps-conference_org_ICAPS_2024_Conference
Published: 2023-12-08 01:47:34

Abstract: MuZero has demonstrated remarkable performance in board and video games where Monte Carlo tree search (MCTS) method is utilized to learn and adapt to different game environments. This paper leverages the strength of MuZero to enhance agents’ planning capability for joint active simultaneous localization and mapping (SLAM) and navigation tasks, which require an agent to navigate an unknown environment while simultaneously constructing a map and localizing itself. We propose SLAMuZero, a novel approach for joint SLAM and navigation, which employs a search process that uses an explicit encoder-decoder architecture for mapping, followed by a prediction function to evaluate policy and value based on the generated map. SLAMuZero outperforms the state-of-the-art baseline and significantly reduces training time, underscoring the efficiency of our approach. Additionally, we develop a new open source library for implementing SLAMuZero, which is a flexible and modular toolkit for researchers and practitioners.

Main paper: 
 introduction: 
 Imagine you find yourself in a new city and need to reach a specific destination. How would you like to navigate your way out? Like most people, you would likely rely on a map. You also need to first identify "your position" on the map. This scenario draws an analogy to robotic navigation, where a robotic is tasked with exploring a new room (Figure 1a ) while building a map on the top of it (Figure 1b ).Simultaneous Localization and Mapping (SLAM) is a widely employed technique in the fields of robotics and computer vision. It allows robots and devices to simultaneously create a map of their environment while determining their own position within that environment. SLAM enables a robot or sensor to navigate and understand an unknown or partially unknown space by both building a map of the surroundings and continuously updating its own position relative to that map, which is crucial for autonomous vehicles, drones, and other robotic systems that need to operate in unfamiliar or changing environments (Placed et al. 2023) .The rapid advancement in deep reinforcement learning (DRL), the widespread availability of user-friendly simulators (Beattie et al. 2016; Savva et al. 2019) , and the accessibility of open-source datasets from cameras and sensors (Xia et al. 2018; Chang et al. 2017; Ramakrishnan et al. 2021) ). This paper introduces a novel learning framework referred as SLAMuZero, designed to address the joint challenges of SLAM and navigation. Within this framework, we leverage the power of Muzero, a highly regarded planning module in DRL known for its remarkable performance in 50 handling complex action spaces across a variety of game environments (Schrittwieser et al. 2020; Hubert et al. 2021; Antonoglou et al. 2021) . Our contributions include:1. We introduce SLAMuZero for joint SLAM and navigation. SLAMuZero extends the search process in MuZero 55 by introducing a SLAM module that decodes the predicted map from the hidden state for path planning. SLA-MuZero outperforms the state-of-the-art baseline and significantly reduces training time. 2. We provide a modular and extendable library for im-60 plementing SLAMuZero. This library provides a userfriendly toolkit for deploying MCTS-based algorithms and allows users to readily adapt MuZero and its variants to existing RL frameworks.The rest of the paper is organized as follows: Sec. 2 dis-65 cusses related work. Sec. 3 presents preliminaries regarding MuZero. Sec. 4 introduces our proposed SLAMuZero. Sec. 5 introduces the solution approach. Sec. 6 presents numerical results and Sec. 7 concludes. ). An end-to-end learning scheme is adopted by (Chen and Gupta 2019) for active SLAM conducted on the Habitat platform (Savva et al. 2019) . Chaplot et al. 2020b uses a hierarchical learning scheme to achieve better performance for navigation tasks across different indoor environments. Oh and Cavallaro 2019 proposed a learning method that predicts the future camera view from a current state-action pair for exploration. Gottipati et al. 2019 leverages convolutional neural network for perception and model-free deep reinforcement learning for acting to disambiguate the agent pose within a reference map. Placed and Castellanos 2020 uses laser measurements as inputs to D3QN and guided the training with the reward to solve active SLAM. However, the current DRL methods for active SLAM lack the necessary planning abilities required for agents to navigate unknown environments effectively. To address this issue, our SLA-MuZero integrates a SLAM module with MuZero, resulting in enhanced planning capabilities and learning efficiency. 
preliminary muzero: 
 MuZero (Schrittwieser et al. 2020 ) is designed for RL tasks and is capable of learning to master complex environments and games without any prior knowledge of their rules. At the core of MuZero is Monte-Carlo Tree Search (MCTS) (Schrittwieser et al. 2020; Browne et al. 2012) , a technique that has demonstrated superhuman performance across various game scenarios. MuZero is structured into three phases. In the planning phase, MuZero first encodes past observations into a hidden state. Subsequently, it executes MCTS, where the process initiates at the root node. At each encountered leaf node, MuZero employs a prediction function to evaluate the hidden state and generate priors for candidate actions (child nodes), along with computing the value associated with the node. At each non-leaf node, MuZero selects a child node recursively, continuing this process until it reaches a leaf node. Concurrently, it updates the hidden state in a recurrent manner with a dynamics function, which takes the previous hidden state and selected actions as input.In the acting phase, an action is sampled from a distribution proportional to the visit count of the root node's children. In the training phase, for the evaluation function, the policy is aligned with the visit count distribution, while the value is trained to approximate the n-step bootstrapping return. For 115 the dynamic function, the model is trained to minimize the difference between the predicted and the observed reward. 
slamuzero: 
 In this section, we introduce SLAMuZero, which consists of representation, SLAM, prediction and dynamics mod-120 ules. The workflow of SLAMuZero is illustrated in Figure 3. In our SLAMuZero, the representation module h and SLAM module q constitute the Encoder-Decoder architecture, which is a mapping between sensor output and the estimated environment. The representation module encodes the 125 raw observations into a hidden state. The model is unrolled for K steps, where the hidden state is recurrently updated through dynamics module g. At each hypothetical step k, the hidden state is decoded by the SLAM module to generate the map and location as m k t , l k t . The policy and value 130 are computed from the internal map and location m k t , l k t by the prediction module,p k t , v k t = f (m k t , l k t ), which guides the search tree expansion. The search is performed at each timestep t. The next action a t+1 is selected according to the visit count for each action from the root node of search tree. 135 We now introduce each module and how four modules are jointly trained in SLAMuZero (Figure 2 ).• Representation: The hidden state is the output of the representation module h, which takes raw observations as input, such as an RGB image of the room to be ex-140 plored. In this work, the module takes images as input, which have a resolution of 256x256 and consist of three channels. These three channels are rescaled to the range [0,1]. Given the large spatial resolution of the RGB observations, the representation module employs a se-145 quence of convolutions with a stride of 2 to reduce the spatial resolution. The output is a hidden state with a resolution of 16x16 and 64 channels. the policy p k and value v k are output by the prediction module f . In the prediction module, the policy head and value head maintain the spatial resolution while altering the number of channels, followed by linear layers to map the size of the output to the number of actions and the value support size, respectively.• Dynamics: Given a hidden state s k-1 and a candidate action a k-1 , the dynamics module g produces an immediate reward r k and a new hidden state s k = g(s k-1 , a k-1 ). The module initially encodes the action into a single panel with the same resolution as the hidden state, rescaled to the range [0,1]. This panel is then appended to the hidden state and passed through 8 Residual Blocks to generate the next hidden state with the same shape. To enhance the stability of the process, the hidden state is rescaled to fall within the range [0, 1] at each step. Additionally, the loss is multiplied by 1 K to ensure that the unroll step size has no impact on the magnitude of the total gradient.The algorithm for SLAMuZero is summarized in Alg. 1. For one sub trajectory sampled from replay buffer, for each timestep t, the representation module h generates the initial hidden state s 0 t from the past observations o 1:t from the sampled trajectory. The model is subsequently unrolled recurrently for K steps. At each step k, five pairs of quantities are collected to calculate the loss, namely the predicted policy p k t with the action probability from the root node π t+k , the preidcted value v k t with the n-step bootstrapping value z t+k , the predicted reward r k t with the actual reward re-190 ceived u t+k , the predicted map m k t with the ground truth map d t+k , the predicted location l k t with the actual location e t+k . Then, the dynamics module g receives the hidden state s k-1 t from the previous step and the real action a t+k as inputs to generate the next hidden state s k t . The model employs 195 a loss function to minimize the discrepancies between predicted and actual rewards, values, and policies, as well as the discrepancies between estimated and actual maps and locations, commonly referred to as the "SLAM error". This loss function is defined as follows:200 l t (θ) = K k=0 l r (u t+k , r k t ) + l v (z t+k , v k t ) + l p (π t+k , p k t ) + l map (d t+k , m k t ) + l pose (e t+k , l k t ) + c||θ|| 2 (1)where, l r , l v , l p , l map , l pose are errors for reward, value, policy, map, location, respectively. c||θ|| 2 is a regularized term capturing the learning process parameterized by θ.Algorithm 1: SLAMuZero Obtain loss according to Equ. 1 13:m k t , l k t = q(s k t , θ) 7: p k t , v k t = f (m k t , l k t ,θ ←optimise(l, θ) 14: end forTo implement and evaluate the proposed SLAMuZero, we 205 propose an open-source library tailored for customizing MuZero algorithms. This library is seamlessly integrated with the Habitat platform, which offers a realistic and controlled setting for testing, to perform visual based robot navigation tasks (Figure 4 ). The data source is detailed in Sec. 6. 210We now introduce the design of the open-source library to facilitate the implementation of our proposed SLAMuZero. Despite the existence of numerous RL frameworks, such as stable-baselines (Raffin et al. 2021) , rllib (Liang et al. 2018) , and ElegantRL (Liu et al. 2021) , the implemen-215 tation of MCTS-based algorithms and their adaptation to specific tasks remains a challenge. The widely used Effi-cientZero (Ye et al. 2021 ), LightZero (Niu et al. 2023) suffer from difficulties in extending the algorithm to accommodate customized loss designs, customized neural network 220 designs, and easy adaptation to different frameworks or environments. To address these issues, we develop a library A primary interface is provided to specify the search algorithm, network designs, as well as the learner, where loss and optimizer can be easily customized. The library employs mctx (Danihelka et al. 2021) as the backend for performing the search process, which can be fully compiled just-intime and run in parallel, ensuring an efficient process. Designed for flexibility and extensibility, our library simplifies the adaptation of MCTS-based RL to prevailing RL frameworks. It empowers users to innovate with advanced neural networks, experiment with novel loss functions, and test in customized environments. More details can be found in (https://anonymous.4open.science/r/Anomalous-F517/). 
6 experiment: 
 In this section, we conduct numerical experiments and make a comparison of our method and baselines.The exploration task involves the agent navigating in an un-240 familiar environment with the goal of exploring as much space as possible (see Figure . 1). The agent receives observations in the form of RGB images from the visual sensor and pose information from the motion sensor. To guide the agent in the environment, we utilize an analytical path planner (Sethian 1996) , which takes the agent's pose, the map, and the goal as inputs to calculate the robot's action sequence. The reward in this task is defined as the exploration ratio of the entire room, and the action is defined as the location on the predicted map.The evaluation metric is the percentage of area explored in the scene (Cov), which is the ratio of coverage to the maximum possible coverage in the corresponding scene. The parameters for the Gibson scene are the same as those used in Neural SLAM (Chaplot et al. 2020a ). Other parameters are listed in Table 1 . The proposed SLAMuZero is trained on the Gibson dataset with a maximum of 1 million frames in contrast to 10 million frames for other baselines. This indicates a significant improvement on training efficiency of our method. The result in (Savva et al. 2019) 0.737 RL + Res18 (Chaplot et al. 2020a) 0.747 RL + Res18 + AuxDepth (Mirowski et al. 2016) 0.779 RL + Res18 + ProjDepth (Chen and Gupta 2019)  
 
 Appendix:  


Decision: The paper studies a problem of navigating in an unknown environment and suggests a combination of Monte-Carlo Tress Search (MCTS) with several learnable simultaneous-localization-and-mapping (SLAM) modules to solve the problem.

The strength of this (short) paper is that the combination of MCTS and SLAM is novel and is showed to outperform several baselines. On the other side, the empirical evaluation does not seem to be comprehensive. Including more competitors and, probably, evaluating on different problem setups (as currently only the exploration scenario is addressed) is worthwhile.

Another concern is that it is questionable whether the paper fits the ICAPS scope. Indeed, ICAPS community is interested in robotic problems as well as in machine learning, but the 'planning' component should still be clearly articulated. Here the latter is present in the form of Monte-Carlo Tree Search, a well-known method which is not significantly modified. Indeed, the main novelty lies outside the 'planning' domain (in integrating MCTS with the other task-specific methods). Consequently whether this contribution is significant from the Planning point of view is questionable. I wonder whether the inclusion of this paper into the ICAPS'24 program is a win-win for both the authors and the community. It may be the case that this paper would get more attention and impact if presented at ML/Robotics conference and, at the same time, it is not clear what 'planning lessons' the ICAPS community can learn from the paper.

Finally, as the study involves training deep neural networks the authors are encouraged to pay more attention to privacy and surveillance concerns, biases and assumptions built into the model and training data, and the environmental impact of the computation needed to train deep learning methods.

Reviews: review: strength_weakness: strength_weakness: The paper has some weaknesses and the work seems premature for publication. (The following comments have been edited to reflect the rebuttal from the authors)
1. One issue is that the evaluation is not very strong. The comparisons in Table 2 are all just taken directly from Chaplot et al. 2020a (except for MuZero and SLAMuZero which the authors ran but with a different set of parameters and scenes). Additionally, the results only show metrics from the Gibson validation set, not on generalization to the MP3D environments.
2. The paper also has only one mention of the path planner they use, it would be useful to have some additional details. 
3. There is no indication of what kind of hardware was used to run the experiment and whether that hardware setup matches the baseline comparisons.
4. Also, the results do not mention how long it took to train their method. If the paper's method only improves the coverage ratio by 0.013 but takes 10 times longer to train, it may not be as much of an advancement. The paper does say that it improves on training efficiency (1 million frames vs 10 million for other methods), but that still does not address the training time.
5. Another issue is that the paper should be careful about conflating navigation and exploration. Exploration is usually considered one aspect of navigation, but the most common navigation task is PointGoal. A system trained for exploration may not be successful at PointGoal navigation.

Additionally, there are many issues with the writing and organization that make the paper difficult to follow.
For example, there are multiple typos: "a robotic", "Muzero" "Table. 2", "learned". There are also run-on sentences and inconsistent reference styles ("adopted by (Chen and Gupta 2019)" vs "Oh and Cavallaro 2019 proposed"). The term "sub trajectory" is used without explanation. The explanation of MuZero is not clear, especially in the second half starting at line 108. Figure 3 is referenced before Figure 2, even though Figure 2 comes first in the paper. It is unclear what Figure 3 represents. Figure 4's reference in the text does not relate to the content of Figure 4. Gibson scene is mentioned on line 254 before it is described in the next paragraph.
questions: questions: 1. How would a robot using your method have the resolution of the ground truth map available to it? If the environment is unknown then it would not have this information, but you provide it to your system (line 152-153).
2. Why is the goal of the method to explore as much space as possible? (line 241) Should it be to explore as much as possible within a given time limit or step count?
strength_weakness: strength_weakness: The only problem I have with the paper is related to the experiments and the experimental analysis, which are not sufficient in my view: 
1. I would like to see results for another dataset, to give the results more credibility.
2. Similar experiments are required for a different evaluation metric. Specifically testing computation time on the fly (while the robot is acting) is interesting, as it might be that this approach is much more computationally demanding relative to other existing approaches. 
3. Further statistical results (in addition to the empirical averages shown) are required in the presentation. 
4. Analysis of the learning process is also relevant, in particular to examine the stability of the learning convergence.

It is perfectly fine to add all the above in an appendix.
questions: questions: 1. Can you explain how the ground truth map and pose (d_t and e_t) are obtained in a physical environment (i.e., a non-simulated one)?
2. Please explain how you used the analytical path planner that is described in the "Experimental set-up". Where does it fit in the experiments?
strength_weakness: strength_weakness: The authors used the Gibson dataset to train and valid their approach with other learning approaches. It would be better if the authors could test their SLAMuZero with different parameters. But considering this is a short paper, this should be a minor concern.

Some small issues: 
   - please be careful with the "extra lines" in Figures 2 and 3;
   - the illustrations of decoded states in figures 2 and 3 are too small. For example, in figure 3, it's difficult to see the difference of robot's location $l^k$ along different trajectories;
   - In line 149, "the map $m^k$ and location $l^k$ in the map $m^k$" seems a bit strange, do you mean "the map $m^k$ and location $l^k$ in the decoded state"?
questions: questions: 1. Currently, SLAMuZero receives observations of RGB images. In more practical robotics applications, robots are often equipped with a wide range of sensors. Therefore, can SLAMuZero be easily adapted to multimodal inputs? 

2. In real applications, is the dynamic module $g$ easy to obtain? Given $g$ takes a hidden state and an actual robot action as inputs, and outputs a new hidden state and an instant reward.