Learning Simultaneous Navigation and Construction in Grid Worlds 
Authors: Wenyu Han
Haoran Wu
Eisuke Hirota
Alexander Gao
Lerrel Pinto
Ludovic Righetti
Chen Feng
Venue: ICLR 2023 poster (2023)
Keywords: Navigation
Localization
Construction
Deep reinforcement learning
Representation learning

Abstract: We propose to study a new learning task, mobile construction, to enable an agent to build designed structures in 1/2/3D grid worlds while navigating in the same evolving environments. Unlike existing robot learning tasks such as visual navigation and object manipulation, this task is challenging because of the interdependence between accurate localization and strategic construction planning. In pursuit of generic and adaptive solutions to this partially observable Markov decision process (POMDP) based on deep reinforcement learning (RL), we design
a Deep Recurrent Q-Network (DRQN) with explicit recurrent position estimation in this dynamic grid world. Our extensive experiments show that pre-training this position estimation module before Q-learning can significantly improve the construction performance measured by the intersection-over-union score, achieving the best results in our benchmark of various baselines including model-free and model-based RL, a handcrafted SLAM-based policy, and human players. Our code is available at: https://ai4ce.github.io/SNAC/.

Published as a conference paper at ICLR 2023
LEARNING SIMULTANEOUS NAVIGATION AND CON- STRUCTION IN GRID WORLDS
Wenyu Han, Haoran Wu, Eisuke Hirota, Alexander Gao, Lerrel Pinto, Ludovic Righetti, Chen Feng" New York University
ABSTRACT
We propose study a new learning task, mobile construction, enable agent to build designed structures in 1/2/3D grid worlds while navigating in the same evolving environments. Unlike existing robot learning tasks such as visual nav- igation and object manipulation, this task is challenging because of the interde- pendence between accurate localization and strategic construction planning. pursuit of generic and adaptive solutions to this partially observable Markov de- cision process (POMDP) based on deep reinforcement learning (RL), we design Deep Recurrent Q-Network (DRQN) with explicit recurrent position estimation in this dynamic grid world. Our extensive experiments show that pre-training this position estimation module before Q-learning struction performance measured by the intersection-owe -oveetttioooov--iiiiin score, achieving the best results in benchmark of various baselines including model-free and model-based RI,, a handcrafted basedd policy, and human players. Our code is available at: https://aiiee gi github......... io/ SNAC/.
significantly improve the con-
INTRODUCTION
Intelligent agents, from animal architects (e.g., mound-building termites and burrowing rodents) human beings, can simultaneously build structures while navigating inside such dynamically evolving environment, revealing robust and coordinated spatial skills like localization, mapping, and planning. Can we create artificial intelligence (AI) perform similar mobile construction tasks?
To handcraft such an Al using existing robotics techniques is difficult and non non-trivial. A fundamental challenge the tight interdependence of robot localization and long-term planning for environment modification. If GPS and techniques alike are not available (often due to occlusions), robots have to rely on simultaneous localization and mapping (SLAM) or structure from motion (SfM) for pose estimation. But mobile construction violates the basic static-enviromment assumption in classic vi- sual SLAM methods, and even challenges SIM methods designed for dynamic scenes (Saputra et al., 2018). Thus, need to tackle the interdependence challenge to strategically modify the environ- ment while efficiently updating a memory of the evolving structure in order to perform accurate localization and construction, shown in Figure 1.
Deep reinforcement learning (DRL) offers another possibility, especially given its recent success in game playing and robot control. Can deep networks leam generic and adaptive policy that controls the AI to build calculated structures as temporary localization landmarks which eventually evolve into the designed one? To answer this question, we design an efficient simulation environment with a series of mobile construction tasks in 1/2/3D grid worlds. This reasonably simplifies the environ- ment dynamics and sensing models while keeping the tasks nontrivial, and allows us focus on the aforementioned interdependence challenge before advancing to other real-world complexities.
To show the tasks in grid worlds are still non-trivial and challenging, we benchmark the performance of several baselines, including human players, a handcrafted policy with rudimentary SLAM and planning, some model-free DRL algorithms which have achieved state-of-the-art performance in other learning tasks (see Table for comparisons), and model-based DRL using Deep Q-Networks (DQN) augmented with Monte Carlo tree search (MCTS) as planning mechanism. Although our
"Corresoonding author, cfengenyu.edu

---

Published as a conference paper at ICLR 2023
Where
Slart trmm here
C
Maa
Macs
dlanding
want
sdbert from hene!
lolt corner.
build Stal
g
Figure I: Challenge in mobile construction. An Al needs to navigate in an environment (square area) and build a structure according to a design (stacked cubes in a/e). The Al (a-d) builds poorly because it learns localization and construction jointly from observations. Specifically, similar structures seen in (c) confuse the Al due to the wrong localization. In contrast, a better construction is achieved in (e-h) using pre-trained localization
in this non-static environment.
Loc
Plan
Env-Mod
Env-Eval
Manipulation (Fan et 2018; Yang al., 2019;
al., 2020; et 2020)
X
Robot Locomotion (Duan et 2016)
x
x
x
X
Visual Navigation (Zhu al,, 7, Gupta et al., 2017; ot al, 2018; Zeng et 2020)
x
Atari (Mnih al, 2013)
x
vxx
x
Minecral (Oh al., 2016; Guss et al., 2019; Platanios et al., 2020)
x
x
x
x
First Person-Shooting Lample & Chaplor, 2017)
8
Real-Time Strategy Games (Syunacve of al,, 2010; Jadorberg ot al, 2019)
X
Physical Reasoning Bapst et (2019); Bakhtin al. (2019)
7
Mobile Construction (Ours)
Table I: Mobile construction vs. existing learning tasks. Loc: robot localization. Plan: long-term planning. Env-Mod: environment structure modification. Env-Eval: evaluation of the accuracy of environment modifications. This shows the novelty of the task with fundamentally different features than typically benchmarked tasks, requiring joint efforts of localization, planning, and learning.
tasks may seem similar to other grid/pixelized tasks such as Atari games (Mnih et al., 2013), the results reveal the significantly worse performance of those bascline algorithms, especially model- free DRL methods, than the human baseline.
recent study (Stooke al., 2021) found that decoupling representation learning from RL policy learning can outperform the joint learning of the two in standard RL algorithms. Inspired by this, we propose to pre-train an explicit position estimation module using recurrent neural networks in the above DRL baselines. Our experiment results show that this proposed method outperforms other RL baselines.
In summary, our contributions include:
a suite of novel and easily extensible learning tasks focusing on the interdependent local- ization and planning problem, which are released open-source fast Gym environments; comprehensive benchmark of baseline methods, which demonstrates the learning chal- lenge in these tasks;
effective approach which combines DRQN with explicit position estimation deep network, outperforming other baselines;
detailed ablation study providing insights about the causes of the challenge, which could inspire future algorithms to solve the problem more effectively.
2 RELATED WORKS
RL baselines. With the great success of model-free RL methods in game-plia gaaee-playing (Mnih et al., 2013; Silver et al., 2016) and robot control (Cheng et al., 2019; Zhang al., 2015), consider family
2

---

Published as a conference paper at ICLR 2023
of common model-free RL methods baselines. Since our tasks are in grid worlds, similar to many Atari games such as the Pac-Man, our first choice is DQN (Mnih et al., 2013). However, DQN has limited capability to tackle POMDP due to its poor latent state representation from long-term history, we benchmark DRQN (Hausknecht & Stone, 2015) which uses recurrent Q-network for integrating historical information. Since our tasks may require explicit long-term planning, similar to (Silver et al., 2016; Bapst et al., 2019), we also include a model-based method which uses DQN as the prior and MCTS (Coulom, 2006) the planning mechanism.
In addition, we add an actor-critic-based baseline, Proximal Policy Optimization (PPO) (Schulman al., 2017). Whereas the standard policy gradient method performs one gradient update per data sample, PPO enables multiple epochs of minibatch updates by a novel objective with clipped proba- bility ratios. We also include Rainbow (Hessel al., 2017), SAC (Haarnoja al., 201 8), and DRQN combined with Hindsight Experience Replay (HER) (Andrychowicz et al., 2017) as additional base- lines in supplementary Section B.
Representation learning for RL. Although the aforementioned RL methods work well many robot learning tasks, it is still challenging to end-to-end learn meaningful representations and control policies jointly via DRL (Stooke et al., 2021), I). Recently, variety of studies improve the performance of RL methods through representation learning, such as internal feature learning (Lample & Chaplot, 2017), and auxiliary tasks (Jaderberg et al., 2016; Mirowski et al., 2016). Echoing these studies, we also find in our experiments that direct end-to-end DRL from raw observations without any representation learning could not solve our task efficiently. Therefore, we propose introduce position-related representation learning module in our DRL framework, and pre-train this module to decouple its training from DRL.
Position-related representation learning is not new in DRL. For example, DRL-based visual naviga- tion studies (Zhu et al., 2017: Chaplot et al., 2020; Datta et al., 2021) use neural networks to extract from visual inputs latent representations encoding position information for RL agents to navigate in static environment. Differently, our task involves a dynamically evolving environment. While work like (Hoeller et al., 2021) indeed navigates robots in dynamic environments using latent state representations in RI, distinctively, our task further involves active modification of the environment. Thus, we do not include this line of work our baseline.
Non-learning baseline. Besides the above methods, one may wish to see the performance of more classical approaches such as either dynamic (Saputra et al., 2018) or active (Mu et al., 2016) vi- sual SLAM which has been studied for decades in robotics, even though they are not designed to work in dynamic environments like ours, nor do they study the planning for better construction curacy. Therefore, we implement a naive handcrafted policy (pseudo code in Algorithm 1 in the supplementary Section B) with basic localization and planning modules The localization module borrows the idea from visual SLAM (Saputra et al., 2018) which relies on finding the common features through successive images to estimate the robot's pose. The planning module simply controls the robot to always build at the nearest possible location.
non-learning baseline.
MOBILE CONSTRUCTION IN GRID WORLD
3.1 TASK OVERVIEW
We formulate mobile construction task as 6-tuple POMDP (S, D), in which a robot required to accurately create geometric shapes according to design D in a grid world. The state space S is represented as S = G x P, where G is space of all possible grid states G storing the number bricks at each grid, and P is a space of all possible robot positions p in the grid world.
At each time step, the robot takes an action E A, either moving around or dropping brick at or near its location. Moving a robot will change its location according to the unknown probabilistic transition model T [p. G, a). Dropping brick will change the grid state G at (for ID & 2D tasks) or near (for 3D tasks) position p without any uncertainty for simplicity. This is reasonable because, in mobile construction settings, motion uncertainty is often the key challenge to ensure accuracy (Sandy et al., 2016).
The robot make local observation O of G centering around its current location, with sensing region defined by a half window size W,, We pad the constant value -1 outside the grid world boundary to ensure valid observations in P. which is distinguished from empty (= 0) or filled
3

---

Published as a conference paper at ICLR 2023
(> 0) grid states. This could help the robot localize itself near the boundary. Finally, the design D E G is simply a goal state of the grid world the robot needs to achieve, and R(s, a; D) is the reward function depending this design.
The aforementioned interdependence challenge localization and planning is reflected via two factors in this setting. First, partial observability makes robot localization necessary. Second, the environment uncertainty in 7 simulates real-world scenarios where the motion control of the mobile robot imperfect and the odometry is error-prone. We implement this by sampling the robot's moving distance d in each simulation time step from uniform distribution.
3.2
TASK VARIATIONS
Next, we specialize the above formulation into a suite of progressively more challenging tasks which varies in the setup of dimension, design D type, and obstacle mechanism, as summarized in Table and Figure 2.
Y-Axis
Design
Design D
Dosign D
Obsorvation
Obsonvatin
X-lots
8
X-Axis
Observation
X-Axis
(a) ID environment
(b) 2D environment
(c) 3D environment
Figure 2: (a) ID grid world environment: a robot moves along the x-axis and drops bricks (red) along -y direction. The two vertical blue dash lines indicate its sensing region for vector o. The (5, 1, 0, 1) in this case. The blue curve is the design D. (b) 2D environment: robot moves in 2D plane and lays bricks (red/yellow: in/correct dropping). The blue box is its sensing region for which is a 2D array with size (2W, + 1)?. The green ring is the ground-truth design D. (c) 3D environment: a robot moves 2D x-y plane and lays bricks (dark blue cubes). The bottom left/right is the 3D/top view. The blue box is similar to 2D. The red ring in the bottom left is the topmost surface of D.
Dimension
1D lefe--eft/-right, drop-brick current location
2D
3D
Move-lefi/-righh hwwp:elll-ngghtuuu--oowwo drop-brick location
Move-lefi-rrigh/ wwihhiiiiirighh--orwaaaaa horph//-owwar----o wowwantt-auckwaattt
Action
drop-brick left/righ/froon/ceur side
Grid state
R
Observation
+1
0. 1
R R2W,, +1x(2,
Constant & Variable
Dense Sparse
Obstacle
Table 2: Detail setups each character for 1/2/3D environment. Each character is described in Section 3 and illustrated in Figure 2. In 1D environments, the grid state G R W is vector, where W is the width the environment and o E 22,,,+ is vector with size 2W, + 1. The grid state G for 2D and 3D environment are 2D binary matrix and 2D matrix with width W and height H respectively. We only add obstacle mechanism in 3D environment.
Dimension. We vary our tasks by the grid world dimension in 1/2/3D respectively. The robot is restricted moving along the x-axis in ID and x-y plane for and 3D environments.
Constant vs. Variable Design. The design of our tasks could be either constant variable. The constant design task requests a robot to build the same shape D in all episodes in both training and testing. For the variable design task, the design D will vary for each episode. For variable design task, to account for real-world situations where robots should have access to the design, augment the local observation with additional information such the design D as 4-tuple Oeno (o, N., Nr. D), where N, is the number of actions the robot has taken, and N, is the number of bricks used.

---

Published as a conference paper at ICLR 2023
Dense Sparse Design. A dense design is a solid shape, and sparse design is an unfilled shape (Figure A.2e and Figure A.2f).
Obstacles. Different from ID and 2D tasks where the robot size is ignored, the robot in the 3D tasks occupies grid so its motion could be obstructed by the built bricks.
Stop criteria. Each episode ends when N₂ Smaz or No Nemaz. where the former is the maximum number of steps and the latter the maximum number of bricks (the integrated area of the design D within each episode). We set N smar reasonably large to ensure the design completion. In addition to the same stop criteria as in the ID/2D cases, the game will stop when the robot is obstructed by the built bricks and cannot move anymore in 3D environment.
4 RECURRENT POSITION ESTIMATION IN DYNAMIC ENVIRONMENTS
After formulating mobile construction tasks, benchmarked our naive model-free DRL baselines on these tasks. We found that these DRL policies perform worse than our human baselines, es- pecially 2D and 3D tasks. We believe the low performance is due to the difficulty in learning meaningful representations and an effective control policy jointly via RL training alone. Especially, the aforementioned interdependence challenge of mobile construction tasks requires the agent to lo- calize itself in a dynamic environment where the surrounding structure could change after the agent build a new brick. Inspired by some recent studies (Stooke al., 2021; Lample & Chaplot, 2017; Jaderberg et al., 2016; Mirowski et al., 2016) which decouple representation learning from RL, we propose our method which combines (1) a pre-trained localization network (L-Net) to estimate the current agent position and (2) DRQN to select the best action based on the predicted positions and observations. Section 4.1 introduces our proposed method and section 4.2 explains the training process of L-Net.
Overall Pipeline
DRON Baseline
L-Net
DRON
Linaer
(-
DRRN
L-Net
DIKON
Elty
LSUM
Linor
Figure 3: Illustration of the DRQN baseline and our proposed method. (a) We use the standard DRQN model which takes the raw observation og and goal design D as input and outputs an action a, in each time step as our baseline. (b) Our pipeline contains two main components: (1) pre-trained L-Net which an LSTM taking the 0t-1, at-1, hidden state h4-1, and the predicted robot position pt-1 from the previous frame and o, of the current frame as input and outputting predicted fi and hidden state ht in the current step. Then, we crop a local design D.rop centering at the predicted position from goal D. This Dr has the same size 0t- (2) we feed the Derop and o, to a DRQN to make an action a, of current step. The detailed training process of L-Net is explained in section 4.2, and shown in supplementary Figure B.I
4.1 DRQN WITH L-NET
Our method is shown in Figure 3, consisting of two main components: (1) an L-Net which is pre- trained via supervised learning, using an LSTM to predict position P. at each time step (2) standard DRQN which takes predicted positions P.. observations or and cropped goal design Drop input and outputs the actions a, current frame.
5

---

Published as a conference paper at ICLR 2023
Let use the 2D dynamic task as an example to explain the overall pipeline. Firstly, the raw observations 0t-1, Oy of the previous and current time steps respectively. are flattened to vectors. Then these two vectors are concatenated with the predicted position Pt-1 and the action a in the previous time step. This concatenated vector 2 and hidden state 14-1 are fed to an LSTM to output hidden feature Y/: and its hidden state h4- Finally, linear layer is used to map the input hidden feature to predicted position Pt. The L-Net which is composed of the LSTM and the linear layer pre-trained through supervised learning.
As shown in Figure 3a, the observation or and the design D are given to the DRQN to output an action at in the baseline. We tried this way of inputting information to DRQN similarly in our pipeline in Figure 3b, and we additionally input the predicted position Pe to the DRQN in our method at the beginning. But the resulting performance is not significantly better than our model-free RI. baselines. This suggests that the raw input (the predicted robot position and the complete design) is not an efficient representation for DRQN-based training.
Inspired by our human volunteers who choose to only refer to a local region of the D near the current position to make decisions when solving our tasks, we crop local design D centering at the predicted position Pt from the complete design D. Then, the D Ptt and or are given to the DRQN- policy network to output an action. Note that when training this pipeline, the pre-trained L-Net is frozen and only the initial position po is assumed known. All other positions Pele=1.2,... used in each DRQN step are predicted by I.-Net based on predicted positions f from its previous time step.
4.2 L-NET PRE-TRAINING
Since the L-Net is trained apart from the overall pipeline training by supervised learning, we collect a dataset which contains 30000 episodes of games. Each game leads to replay buffer (M.)li=12.....N where M; = (o--1, 04-1, Pi-1,, 0:). Each data in such buffer is collected by let- ting the agent perform random actions until an episode ends. When training the I.-Net, we randomly sample a chunk of data Mi L. where L is a constant sequence length, from the en- tire sequence. As shown in supplementary Figure I. we firstly feed the data M++1 and a randomly initialized hidden state h4 to the L-Net that predicts position Pt+1 and its hidden state he+1- Note that we always use the first ground-trutt (GT) position Pt as input and the input positions of following steps are all outputs from previous steps. This means do not perform teacher-forcing train the LSTM. Then, we just repeat steps and compute L predicted positions f1.l:=<t1,,+2,.... 1.t.2..... Finally, the L-Net is trained by minimizing the L2 loss between the predicted positions and GT positions.
EXPERIMENTS
We conduct comprehensive experiments to test baselines and our proposed method on mobile con- struction tasks. In this section, we only pick representative baselines from different classes show the results. All experiment results are listed in supplementary Table 7 and the best qualitative re- sults are shown supplementary Figure C.3. The evaluation metric, reward function, and training protocol are described in this section.
Baselines. We choose two Q-learning methods, an actor-critic method, a search-based method, a non-learning method, and human players as representative baselines. Detailed architecture designs and hyperparameter setups for each algorithm are explained in the supplementary Section B.
DQN. For constant design tasks, we an MLP-based Q network. For variable design tasks, the design is mapped to a feature vector by a convolutional layers.
DRQN. We add one recurrent LSTM layer to the Q network as the DRQN baseline.
PPO. We benchmark discrete PPO in the Stable Baselines (Hill et al., 2018).
DON+MCTS. For this mosel-based method, we simply combine DQN with standard MCTS algorithm, similar to Bapst et al. (2019).
Handcrafted policy. It consists of basic localization and planning modules.
Human. A simple GUI game that allows human player to attempt tasks in identical environments with the same limitations of partial observability and step size uncertainty.
They act as the agent, using the ARROW keys to maneuver, and SPACE to drop brick.

---

Published as a conference paper at ICLR 2023
Evaluation metric: we use the Intersection over Union (IoU) score evaluation criteria which is measured between the terminal grid state G" and the ground -ouund-truth design D. Here, we use IoU as our evaluation criteria because it is more straightforward for us to evaluate the quality of the built structure to the ground-truth design D. The loU is defined as:
GH 1DD ToU = GT JD
min(G" Σ PEP
(1)
max (GT (p), D(p))
Reward function. The reward function for ID and 3D environments is shown in Table 3. For the 2D enviromment, R is simply designed as: the agent gets 5/0 for dropping bricks at correct/wrong positions and gets for movement. Note that in the 2D environment we tried to add similar penalty in Table 3 for incorrect brick dropping, but we found this led to more frequent ac- tions of moving instead of dropping bricks because of the higher chance of negative rewards of the latter (especially for sparse plans), Consequently, this led to bad performance of all tested methods. Therefore, we removed the penalty from the reward function in all 2D environments.
Training protocol. To validate the proposed framework and its robustness, trained with the same set of 4 random seeds and averaged results are reported. For the constant de- sign tasks in ID/2D/3D, we test the trained agent for 500 times for each task and report the average and the standard deviation (stddev) of IoU scores among 500 tests. For the variable design tasks, we randomly generated 500 ground-truth designs and split them to 8/1/1 for training/vlliaationnesttg
baselines are
Drops brick
on D
Drups brick below D
Drops brick over D
Move
Obstructed
10
10
X
Table 3: Reward function for ID and 3D tasks.
Following the same protocol as constant de- sign tasks, we test all baselines on the testing set for 500 times for each task and report the average and the stddev of IoU scores among 500 tests.
5.1 BENCHMARK RESULTS
10
2D Dene
20 Sporse
30 Dense
3D Sparse
5:50
0.50
5.00
0.00
0.00
DMOV
1,00
1.00
5.00
PPO
DON-METS
38 Varibble
2
0,50
0:0
0.25
DMOX-LLntt
Constorz
Verisble
Chrstani
Varibble
Conderi
Varibble
Constant
Virielee
Constert
Variablee
841-19777
0631
TeU: 201
Boxt demo
BRON HLan
DRON F.ack
DUN-Lasi
DRON-Liet
DRRNN-Loet
DRUNNLLot
DRON-Laet
DRON-Lan
DROON
DRON+Loet
L0U: 0010
0002
Int: 0.750
lall: OniS
IA 2664
Ren demmo RL. Live
DRON
DRUN
bRgg
DRON
DROGN
PPO
DON-MMTT
DgN
DON-MCTS
Figure 4: Experiment results of baselines and our method. (a) & (b) present the average and the
stddev (errorbars) of IoU of examined methods on constant and variable design tasks in ID/2D/3D environments. (c) & (d) show the best qualitative results of RL w/ and w/o L-Net, respectively.
Figure 4 summarizes the quantitative testing results and the best qualitative testing results of the baselines and our method. In general, we see that the performance of each method drops as the dimension of the environment increases. This clearly shows that the larger the exploration space and that the more complex the construction mechanisms are (e.g., obstacles in 3D tasks, further studied in Section 5.2), the worse the tested method performs. Especially in 3D environments, most of the baselines learn ineffective policies or fail to learn any policies. Mcanwhile, most of the methods consistently perform worse the sparse design tasks than the dense ones. We believe that such poor performance is due to the much sparser reward signals in sparse design tasks.
Moreover, notice that all RL baselines have more difficulties dealing with variable design tasks compared to constant ones. We posit that the RL baselines lack the capacity to learn effective representations for variable designs. DRQN is expected to be the best among all RL baselines in
7

---

Published as a conference paper at ICLR 2023
most of the tasks because it can learn better from long period of history. Compared with all examined RL baselines, our proposed method achieves better performance consistently in all tasks. In some cases, it can even surpass the human baseline and the handcrafted policy which relies on prior knowledge of the environment setups. Next, we will compare them in detail.
RL w/ v.s. w/o L-Net. Besides the proposed method described in section 4, we also include another baseline which combines DQN with L-Net to further test the effectiveness of our method. We find that RL w/ L-Net methods outperform all other RL baselines consistently among all tasks. For ID constant tasks, the DRQN w/ L-Net achieves 0.896 ave IoU score, which even surpasses the human baseline (IoU 0.83). It also performs the best among other learning- based baselines in ID variable tasks. As shown in Figure 4(c), DRQN w/ L-Net can perfectly build the Gaussian-shaped design. In contrast, most of the RI. baselines are only able to build a flattened shape (See the first column of Figure C.3 in supplementary). All these suggest the importance of representation learning that enables a reasonable position estimation, even in our dynamic environments.
On the 2D constant dense task, all methods reach similar performance. However, the gap between RL w/ L-Net and RL w/o L-Net becomes significantly larger in more difficult 2D tasks, such as the 2D constant sparse task. The DRQN w/ L-Net can achieve a reasonable performance of median IoU 0.301 and best IoU 0.545 in the 2D variable sparse task where others only learn very limited policies. The qualitative results in Figure 4(c) show that DRQN w/ L-Net can complete half of the triangle shape in best cases while RI. baselines can only drop a few bricks around the goal design (see the 6th column of Figure 4(d)). This indicates that with the help of position prediction by the pre-trained L-Net, DRQN is able to learn& a better construction policy.
Similarly, the DRQN w/ L-Net method outperforms RL baselines in 3D tasks. From the bar chart of Figure 4(a), we can see that DRQN w/ L-Net can still learn reasonable policies which achieve average IoU 0.3074 and 0.315 on the 3D constant dense and sparse tasks. Our method is almost able to construct the complete hollow cylinder in Figure 4(c). Moreover. it is remarkable that DRQN w/ L-Net manages to learn a rough construction policy while other RL baselines fail to learn any useful policies. We conclude that explicit localization via representation learning is crucial for tackling 3D tasks, though the performance is still limited due to the potential challenges of obstacles.
Handcrafted policy. Although our handcrafted method performs seemingly well on ID tasks, one should not draw hasty conclusion that it must be the ultimate direction for solving mobile con- struction. Because the good performance of this method relies on prior knowledge of the distribution parameters of the probabilistic transition model T: our handcrafted localization has a very low error rate (especially in ID settings) by leveraging this privileged information, which is not accessible in other baseline methods and in reality. Moreover, from Figure 4, we can see that the obstacle mech- anism, the sparse designs, and the environment uncertainty in 2D and 3D are still challenging for this method. We believe learning- based methods have a good potential to adaptively address these variations, while solving them one by one via different handcrafted methods is less effective.
Human baseline. We collected 30 groups of human test data, totaling 490 episodes played, and report the average IoU shown in the Figure 4 with dot line. All humans were required to play the same maximum number of steps for each game as other baselines. From the feedback, we found that humans could learn effective policies (building landmarks to help localization) very efficiently in at most few hours, which is much sooner than training an RL model.
5.2
ABLATION STUDY
Shape
loU
DRUN
40PS1
20
-[lbenstoo)
DRUN
3D
We performed ablation studies to comprehen- sively analyze the reasons associated with the poor RL baseline perfor- mances on 2D and 3D tasks. We identify four potential challenges: (I) the obstacles in 3D environments, (2) the lack of localization information, (3) the step size d uncertainty, and (4) the lack of landmarks in the environment for localization. We use DRQN results on constant tasks as the basis of this ablation study and all the ablation experiments were conducted using the same setup and test criteria described in Section 5. We also try imitation learning to train DRQN with expert experience and conduct an ablation study for Rainbow (see supplementary Section C).
0.315
-0033
+0U12
0.334
+0.151
6071
+1.725
Danse
Stálev
0.132
-0005
-11032
40,037
-0,002
0.145
-11009
0.724
-0.027
40.121
-0.591
40.156
0
+1.714
0.338
0.09
Spans:
Stckee
-0037
-0076
-0.10
0015
0.2
+0.186
Table 4: Ablation study on constant design tasks using DRQN. 1/4: expecting better/worse performance.
8

---

Published as a conference paper at ICLR 2023
Obstacles. To explore the influ- ence of the obstacles, we conducted two experiments: adding the obsta- cle mechanism moving it from the 3D one (Figure 5). As shown in Table 4, the performance of the 3D task increases significantly from 0.071 to 0.799 for dense de- signs. Similarly, when we add obsta- cles into 2D, the IoU score drops by more than 40%. These results sug- gest that the presence of obstacles is an important reason for the poor per- formance on 3D tasks.
a 2D task and re-
Figure 5: Influences of obstacles. Top rows show the base- line results 2D and 3D constant tasks. The first two im- ages of bottom row show results of adding obstacle to 2D world. Compared with baseline, the performance drops tably. The last two images of bottom row show results of re- moving obstacle from 3D world. The performance increases significantly, especially for 3D dense design task.
Localization. From the previous experiments, we conclude that good position estimation via repre- sentation learning could help RL lear better construction policies. Moreover, it is interesting to test what are RL methods' performance upper bound with the help of perfect localization. To test this, provided GT position p for DRQN on 2D constant tasks. From Table help of position information, 2D constant sparse perfectly solved (see Figure 6a).
that with the
Environment uncertainty. Besides the limited sensing range, the random step size could be another reason for the poor performance of baselines. Therefore, we conduct an experiment where the step size uncertainty is removed. From Table 4, can see that the IoU increases by more than 40% compared to the sparse designs (see Figure 6b).
(a) Add GPS
(b) Fixed step size
(c) Initial environment
(d) Add landmark results
Figure 6: Influence of localization and environment uncertainty. Either adding GPS in (a) or remov- ing step size uncertainty in (b) largely improves DRQN on sparse design tasks compared with the first two columns on the top row of Figure I.Inluence of landmarks. (c) We randomly add land- marks in the initial environment (marked as gray triangle). (d) Compared with the baseline, adding landmarks improves DRQN obviously both dense and sparse design tasks.
Landmarks. An empty initial environment lacks landmarks widely used in SLAM methods for the robot to localize itself. This could be another potential challenge of mobile construction tasks. We conduct an experiment: randomly adding some landmarks in the initial environment. From Table 4, we can see that IoU increases dramatically both on dense and sparse tasks (see Figure 6c and 6d).
CONCLUSION AND FUTURE WORK
To stimulate the joint effort of robot localization, planning, and deep RL research, we proposed suite of mobile construction tasks, where we benchmarked the performance of common model-free and model-based deep RL algorithms, handcrafted policy with basic localization and planning, and human baselines. Meanwhile, we propose our method which incorporates pre-trained L-Net and DRQN to solve these tasks. The experiment results indicate that learning an explicit position estima- tion can effectively improve the performance of RL methods on mobile construction tasks. Although the performance of the proposed method is limited on the hardest tasks, we believe augmenting RL. frameworks with representation learning a promising direction to solve the interdependence chal- lenge in mobile construction tasks. We believe that the limitations of current methods could be overcome by designing a localization function which is able to localize more precisely in dynamic environments. In the future, plan further extend our mobile construction task suite with more features such as allowing new action of explicitly placing landmarks, physics-base based simulation in continuous worlds, and multi-agent mobile construction.

---

Published as a conference paper at ICLR 2023
7 REPRODUCIBILITY STATEMENT
To make all our experiment results reproducible, we submit codes with hyperparameters used for each method on each task in the supplementary material. We also provide an instruction on how to use our code to reproduce the experiment results.
ACKNOWLEDGEMENT
The research supported by NSF CPS program under CMMI-1932187. The authors gratefully thank our human test participants, Dongdong Liu and Armand Jordana for implementing the SAC and PPO baselines and the helpful comments from Bolei Zhou, Zhen Liu, and the anonymous re- viewers, and also Congcong Wen for paper revision.
REFERENCES
Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAl Pieter Abbeel, and Wojeiech Zaremba. Hindsight experience play. In Advances in neural information processing systems, pp. 5048-5058, 2017. 3, 15
Hadi Ardiny, Stefan Witwicki, and Francesco Mondada. Construction automation with autonomous mobile robots: A review. In 2015 3rd RSI International Conference on Robotics and Mechatronics (ICROM), pp. 418 424. IEEE, 2015. 14
Peter Auer, Nicolo Cesa-Biancha and Paul Fischer. Finite-time analysis of the multiarmed bandit problem. Machine learning, 47(2):235 256, 2002. 17
Anton Bakhtin, Laurens van der Maaten, Justin Johnson, Laura Gustafson, and Ross Girshick. Phyre: A new benchmark for physical reasoning. arXiv preprint urXiv:198.00566,, 2019.
Victor Bapst, Alvaro Sanchez-Gonzales Carl Doersch, Kimberly Stachenfeld, Pushmeet Kohli, Pe- ter Battaglia, and Jessica Hamrick. Structured agents for physical construction. In International Conference on Machine Learning, pp. 464-474. PMLR, 2019. 2, 3, 6,
M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253-279, Jun 2013. ISSN 1076-9757. doi: 10.1613/jair.992. 18
T Bonswetch, Daniel Kobel, Fabio Gramazio, and Matthias Kohler. The informed wall: applying additive digital fabrication techniques on architecture. Proceedings of the 25th annual conference of the association for computer-aides design in architecture, pp. 1899-495,2 2006. 14
Jonas Buchli, Markus Giftthaler, Nitish Kumar, Manuel Lussi, Timothy Sandy, Kathrin Dörfler, and Norman Hack. Digital in situ fabrication-challenges and opportunities for robotic in situ fabrication in architecture, construction, and beyond. Cement and Concrete Research, 1 11::66 75, 2018. 14
Devendra Singh Chaplot, Ruslan Salakhutdinov, Abhinav Gupta, and Saurabh Gupta. Neural topo- logical slam for visual navigation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12875-12884, 2020. 3
Richard Cheng, Gábor Orosz, Richard M Murray, and Joel W Burdick. End-to-end safe reinforce- ment learning through barrier functions for safety-critical continuous control tasks. In Proceed- ings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 3387-3395, 2019. 2
Petros Christodoulou. Soft actor-critic for discrete action settings. arXiv preprint urXiv:1910.0722 2019. 16
Rémi Coulom. Efficient selectivity and backup operators in monte-carlo tree search. In International conference on computers and games, pp. 72-83. Springer, 2006.
10

---

Published as a conference paper at ICLR 2023
Samyak Datta, Oleksandr Maksymets, Judy Hoffman, Stefan Lee, Dhruv Batra, and Devi Parikh. Integrating egocentric localization for more realistic voint-goal navigation agents. Conference on Robot Learning, pp. 313-328. PMLR, 2021. 3
Yan Duan, Xi Chen, Rein Houthooft, John Schulman, and Pieter Abbeel. Benchmarking deep reinforcement learning for continuous control. In International Conference on Machine Learning, pp. 1329-1338, 2016. 2
Linxi Fan, Yuke Zhu, Jiren Zhu, Zihua Liu, Orien Zeng, Anchit Gupta, Joan Creus-Costa, Silvio Savarese, and Li Fei-Fei. Surreal: Open-source reinforcement learning framework and robot manipulation benchmark. In Conference on Robot Learning, pp. 767-782, 2018.
Meire Fortunato, Mohammad Gheshlaghi Azar, Bilal Piot, Jacob Menick, Ian Osband, Alex Graves, Vlad Mnih, Rémi Munos, Demis Hassabis, Olivier Pietquin, Charles Blundell, and Shane Legg. Noisy networks for exploration. CoRR, abs//706. 102990 2017. 15
Markus Giftthaler, Timothy Sandy, Kathrin Dörfler, Ian Brooks, Mark Buckingham, Gonzalo Rey, Matthias Kohler, Fabio Gramazio, and Jonas Buchli. Mobile robotic fabrication at 1: scale: the in situ fabricator. Construction Robotics, 1(1-4):3-14, 2017. 14
Saurabh Gupta, James Davidson, Sergey Levine, Rahul Sukthankar, and Jitendra Malik. Cognitive mapping and planning for visual navigation. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 2616-2625, 2017.
William H Guss, Brandon Houghton, Nicholay Topin, Phillip Wang, Cayden Codel, Manuela Veloso, and Ruslan Salakhutdinov. Minerl: large-scale dataset of minecraft demonstrations. arXiv preprint arxiv: / 13440, 2019.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. ICML, pp. 1856-1865,
2018. 3
Matthew Hausknecht and Peter Stone. Deep recurrent q-learning for partially observable mdps. arXiv preprint arXiv: 1XX.vv557700627 2015.
Matteo Hessel, Joseph Modayil, Hado van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning, 2017. 3, 16, 18
Ashley Hill, Antonin Raffin, Maximilian Ernestus, Adam Gleave, Anssi Kanervisto, Rene Traore, Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert, Alec Rad- ford, John Schulman, Szymon Sidor, and Yuhuai Wu. Stable baselines, 2018. 6, 15
David Hoeller, Lorenz Wellhausen, Farbod Farshidian, and Marco Hutter. Learning a state repre- sentation and navigation in cluttered and dynamic environments. IEEE Robotics and Automation Letters, 6(3):6081-5008883 2021. 3
iRobot@. irobot@root@ coding robot. 14
Max Jaderberg, Volodymyr Mnih, Wojeiech Marian Czarnecki, Tom Schaul, Joel Z Leibo, David Silver, and Koray Kavukcuoglu. Reinforcement learning with unsupervised auxiliary tasks. arXiv preprint 1rXiv:/61..05330 2016. 3,
Max Jaderberg, Wojeiech M Czamecki, lain Dunning, Luke Marris, Guy Lever, Antonio Garcia Castaneda, Charles Beattie, Neil C Rabinowitz, Ari Morcos, Avraham Ruderman, et al. Human- level performance in 3d multiplayer games with population-based reinforcement learning. Sci- ence, 36.(6443)8599-855, 2019.
Sasa Jokic, Petr Novikov, Stuart Maggs, Dori Sadan, Shihui Jin, and Cristina Nan. Robotic posi- tioning device for three-dimensional printing. arXiv preprint arXiv:1406.3400 2014. 14
Yann Labbé, Sergey Zagoruyko. Igor Kalevatykh, Ivan Laptev, Justin Carpentier, Mathieu Aubry, and Josef Sivie. Monte-carlo tree search for efficient visually guided rearrangement planning. IEEE Robotics and Automation Letters, 5(2):3715-3722) 2020. 2
11

---

Published as a conference paper at ICLR 2023
Guillaume Lample and Devendra Singh Chaplot. Playing fps games with deep reinforcement learn- ing. In Proceedings of the Thirty- First AAAI Conference on Artificial Intelligence, pp. 2017. 2, 3,5
Chengshu Li, Fei Xia, Roberto Martín-Martinn and Silvio Savarese. Hrl4in: Hierarchical rein- forcement learning for interactive navigation with mobile manipulators. In Conference on Robot Learning, pp. 603-616, 2020. 2
Lucas Galvan Marques, Robert Austin Williams, and Wenchao Zhou. A mobile 3d printer for coop- erative 3d printing. In Proceedings of the 28th Annual International Solid Freeform Fabrication Symposium. University of Texas, 2017. 14
Nathan Melenbrink, Katja Rinderspacher, Achim Menges, and Justin Werfel. Autonomous anchor- ing for robotic construction. Automation in Construction,, 120:103391, 2020. 14
Piotr Mirowski, Razvan Pascanu, Fabio Viola, Hubert Soyer, Andrew Ballard, Andrea Banino, Misha Denil, Ross Goroshin, Laurent Sifre, Koray Kavukcuoglu, et Learning to navigate in complex environments. arXiv preprint arXiv:1611.0367. 2016. 3, 5
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wier- stra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5600 2013. 3,
Kaichun Mo, Haoxiang Li, Zhe Lin, and loon-Young Lee. The adobeindoornav dataset: Towards deep reinforcement learning based real-world indoor robot visual navigation. arXiv preprint arXiv:1802.08824, 2018. 2
Beipeng Mu, Matthew Giamou, Liam Paull, Ali-akbar Agha-mohammadi, John Leonard, and Jonathan How. Information- based active slam via topologiaal feature graphs. In 2016 IEEE 55th Conference on Decision and Control (CDC), pp.
IEEE, 2016. 3
Cristina Nan. A new machinecraft: A critical evaluation of architectural robots. In International Conference on Computer-Aides Architectural Design Futures, pp. 422-438, Springer, 2015. 14
Junhyuk Oh, Valliappa Chockalingam, Satinder Singh, and Honglak Lee. Control of memory, active perception, and action in minecraft. arXiv preprint arXiv:/605.000 /28, 2016. 2
Emmanouil Antonios Platanios, Abulhair Saparov, and Tom Mitchell. Jelly bean world: A testbed for never-ending learning. arXiv preprint arXiv:2002.06300, 2020. 2
Timothy Sandy, Markus Giftthaler, Kathrin Dörfler, Matthias Kohler, and Jonas Buchli. Au- tonomous repositioning and localization of in situ fabricator. In 2016 IEEE International Conference on Robotics and Automation (ICRA), pp. 1852-2858. IEEE, 2016. 3
Muhamad Risqi U Saputra, Andrew Markham, and Niki Trigoni. Visual slam and structure from motion in dynamic environments: A survey, ACM Computing Surveys (CSUR), 51(2):1-36, 2018. 3,3
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1707.06377 2017. 3
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, loannis Antonoglou, Veda Panneershelvam, Mare Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484 2016. 2,
Adam Stooke, Kimin Lee, Pieter Abbeel, and Michael Laskin. Decoupling representation learning from reinforcement learning. In International Conference on Machine Learning, pp. 9870-9879. PMLR, 2021. 2, 3,
Gabriel Synnaeve, Nantas Nardelli, Alex Auvolat, Soumith Chintala, Timothée Lacroix, Zeming Lin, Florian Richoux, and Nicolas Usunier. Torchcraft: a library for machine learning research real-time strategy games. arXiv preprint arXiv: 1611 1.06255 2016. 2
12

---

Published as a conference paper at ICLR 2023
Justin Werfel, Kirstin Petersen, and Radhika Nagpal. Designing collective behavior in termite- inspired robot construction team. Science, 343(6172):754-75, 2014. 14
Brian Yang, Jesse Zhang, Vitchyr Pong, Sergey Levine, and Dinesh Jayaraman. Replab: A repro- ducible low-cost arm benchmark platform for robotic learning. arXiv preprint 190Kiv::905004444 2019. 2
Fanyu Zeng, Chen Wang, and Shuzhi Sam Ge. A survey on visual navigation for artificial agents with deep reinforcement learning. IEEE Access, 24135426-354444 2020. 2
Fangyi Zhang, Jürgen Leitner, Michael Milford, Ben Upcroft, and Peter Corke. Towards vision- based deep reinforcement learning for robotic motion control. arXiv preprint arXiv:1511.0377
2015. 2
Xu Zhang, Mingyang Li, Jian Hui Lim, Yiwei Weng, Yi Wei Daniel Tay, Hung Pham, and Quang- Cuong Pham. Large-scale 3d printing by a team of mobile robots. Automation Construction, 95:98-106, 2018. 14
Yuke Zhu, Roozbeh Mottaghi, Eric Kolve, Joseph Lim, Abhinav Gupta, Li Fei-Fei, and Ali Farhadi. 'Target-rrvven visual navigation in indoor scenes using deep reinforcement learning. In 2017 IEEE international conference robotics and automation (ICRA), pp. 3357-3364. IEEE, 2017. 2,
13

---

Published as a conference paper at ICLR 2023
A MOBILE CONSTRUCTION TASKS
Mobile construction robots. Recently have seen a rising trend of 3D printing using mobile robots all around the globe for construction and manufacturing (Werfel et al., 2014; Jokic et al., 2014; Ardiny et al., 2015; Nan, 2015; Marques al., 2017; Buchli et al., 2018; Zhang et al., 2018; Melenbrink et al., 2020). All of those are carefully engineered systems that either assume some global localization ability or only work for specific scenarios, which restricts their feasibility in large scale. Moreover, none of them address the aforementioned challenge from theoretical perspective. To stimulate both robot localization and planning and deep RL research community to work on this problem, we design efficient mobile construction simulation environment with series of tasks in 1/2/3D grid worlds. Several types of goal design D used in each task of 1/2/3D environments shown in Figure A.2. We also show some real world examples of mobile construction tasks in Figure A.
LY-Axis
DesignD
X-Axis
Observation
a ID environment
Design
Design
CREW vellicn
X-fais
X-Axis
(b) 2D environment
(b) 3D environment
Figure A.1: a) A real world example ID environment (left). a programmed wall project (Bon- swetch al., 2006), and ID grid world environment (right). b) A real world example of 2D envi- ronment iRobot (left), painting robot and 2D grid world environment (right). c) A real world example of 3D environment, mobile robot construction project (Giftthaler et al., 2017) (top) and 3D) grid world environment (bottom).
For the ID constant tasks, consider three types of shape designs D RW: sin function curve, Gaussian curve and Step function curve (see Figure A.2a to Figure A.2c). For the dynamic tasks, we generate D (Figure A.2d) based on the following equation: D sin(bæ - e).rrere (3, 12), 6 E {1, 2,,2,3). 3}. and с x)) The coefficients b and are chosen randomly for each episode. For 2D variable tasks shown Figures A.2g and A. A.2h,three vertexes triangle are randomly picked within the grid world. The setup of designs D in 3D is similar to the ones in 2D as shown in Figures A.2i-A.21.
B BASELINE SETUP
Several common model-free and model-based RL baselines, and one handcrafted policy are consid- ered in our paper. Here, we fix the environment constants as follows: (1) for ID environments, half window size W. 2. and the environment width W 30; (2) for 2D and 3D environments W. and W = H = 20. Detailed architecture designs and hyperparameter setups for each baseline are explained as follows.
14

---

Published as a conference paper at ICLR 2023
M
(a)
(b)
(c)
(d)
(e)
(I)
(h)
(k)
Figure A.2: Example plans. 1) (a-d) shows the curve shapes for ID tasks. 2) (e/f) are design shapes for 2D constant dense and sparse tasks and (g/h) are 2D variable tasks; 3) (i/j) are 3D constant tasks and (k/l) are 3D variable tasks. The 3D red patches are the top-most surfaces of ground-truth plans.
DQN. For the constant design tasks, we use an MLP with three hidden layers containing 164, 128] nodes with ReLU activation function for each layer. Three convolutional layers with kernel size of 3, and ReLU activation functions are used to convert the ground-truth D to feature vector for variable design tasks. We train DQN on each task for 3,000 episodes. Batch size is 2,000, and replay buffer size 50,000. For tuning the learning rate, we try relatively small value 1e-8 to make sure convergence initially and gradually increase it until finding a proper value. Addition- ally, instead of only using the current frame, we try to stack ten frames of historical observations in the replay buffer. This is similar to how the original DQN handles history information for Atari games (Mnih et al., 2013), but no significant difference was found.
DRQN. We benchmark DRQN by simply adding one recurrent LSTM layer the Q network in the DQN. The hidden state dimension is 256 for LSTM layer for all tasks. We train it for 10,000 episodes with batch size of and replay memory size of 1,000.
We augment the DRQN baseline with hindsight experience re- play (Andrychowicz et al., 2017) as another baseline. At the end of each training episode, the transitions (of (o a", a(s",,';D) ottt) of each time step t will be relabeled as (o' a., R(s ( a'; G") oene), where we change the D to the grid state G at the terminate step T. Both transitions stored into the replay buffer. We train this DRON+Hindsight for 10,000 episodes with batch size of 64 and replay memory size of 1,000.
Hindsgin
PPO. We benchmark PPO in discrete settings using the Stable Baselines implementation (Hill al., 2018). We train PPO for 10 million time steps with a shared network of 3 layers of 512 neurons with tanh activation function. For the hyperparameters, we the 1D constant environment the learning rate, the batch size, the number of minibatches size, and the clipping threshold. We found that the most sensitive parameters were the batch size and the minibatch size and chose the following values: x 10" for the batch size, 1 x 102 for the number of minibatches, 2.5 x 10 for the learning rate and 0.1 for the clipping threshold.
Rainbow. For the Rainbow implementation, we use 3 noisy hidden layers (Fortunato et al., 2017) with 128 nodes in each layer, and ReL.U nonlinear activation functions. Rainbow has large set of
2D
3D
Variable
Consiani
Vanablle
Conslani
Variable
Dense
Sparse
Dense
Spurse
Deuse
Spurso
Dense
Sparse
0.0078
0.0123
0.0069
0.0033
0.0040
0.0011
0.0064
0.0072
0.006
0.0058
Simulation time (scconds)
Max
0.0136
0.023
0.0072
0.0046
0.0065
0.0029
0.0313
0.0316
0.0209
0.0268
Stdday
0.0003
0.0006
0.00004
0.0003
0.0011
0.0003
0.0052
0.0051
0.0046
0.0044
Table Simulation time of our environments. We test each simulation environment for 500 episodes of games on Intel(R) Core(TM) i9-9920X CPU @ 3.50GHz using a single thread and report the average, maximum, and standard deviation of the simulation time of each game. Our environments can be simulated faster by batch processing.
15

---

Published as a conference paper at ICLR 2023
L-Net Training
he
L-Net
Est. Preition
GT Position
LSTM Call
L Ch P
Ye+1
Linear
Pc+1
P++1
O
hp+1
L-Net
a++1
Oc+1
LSTM Cell
L Peod
Y++2
Linear
Po42
Po42
P:+1
oc+2
Loop steps
h
Jane spanitidita expless
L-Net
ar+1-1
O++1-1
LSTM Call
L Pess
Petz
Linear
pC+z
Pe+l
PLtl-1
Loss:
Le PeenPee
De+s.
Figure B.I: Illustration of training process of L-Net.
hyperparameters, as each of the six components adds additional hyperparameters. We used those suggested in (Hessel et al., 2017) as a starting point, but they led poor results on our specific task and environment designs. As grid search over such large hyperparameter space was impractical, we used a random search approach. Based on empirical results, the algorithm was most sensitive to learning rate, as well as Vinin Vmaas and Natomas which define the value distribution support predicted by the distributional Q-network. Generally, Vmii value of -5, Vmaz of 35, and Natoms = 101 provided stable performance, and were chosen heuristically based on approximate range of discounted rewards possible in our environments. We used a prioritized experience replay buffer of size 1 10 with priority exponent of 0.5, and starting importance sampling exponent 3 of 0.4. Additionally, we used multi-step returns with n = 3, and noisy network ag 0.1. Finally, we used learning rate of 5 10-5 for ID and 2D, and 10-4 for 3D environments.
SAC. We use the implementation of (Christodoulou, 2019) for SAC in discrete settings which has automatic tuning mechanism for entropy hyperparameters. We use learning rate 3 10 for target networks for most plans. We use ReLU activation functions for the hidden layers and Softmax for the final layer of the policy network. We use interpolation factor 7=5 10 3 for target networks and the start steps before running the real policy is 400 with mini batch size 64. We first search the main hyperparameters based on 1D constant case. Next, we use different network architectures for relatively complex 2D and 3D cases with the same hyperparameters as 1D constant case. For 1D environment, we use hidden layers with 64 nodes each for both actor and critic networks. For 2D environment, we use hidden layers with 512 nodes each for the variable design and 3 hidden layers containing [64, 128, 64] nodes for the constant design. A 5-layer network architecture containing 164,128,256, 128,64] nodes is applied in the 3D cases.
DQN+MCTS. For this model-based method, we simply combine DQN with standard MCTS algo- rithm, similar to Bapst al. (2019). In each time step, the tree search method uses the current state as the root node and the latest Q network as a guide to evaluate values for each node the search tree. When the search process ends, the action of is selected based on the highest visit counts among
16

---

Published as a conference paper at ICLR 2023
all possible actions from the root node. In all experiments, use the true environment simulator as our transition model. We use same architecture design for each task as we use in the DQN exper- iments. The rollout of each tree search is 20 and the Upper Confidence Bound (UCB) (Auer et al., 2002) is used to balance the exploration and exploitation in the search process. The UCB constant set to 0.5 for each experiment. We use learning rate 10-3, batch size 2000, and replay buffer size 50,000 for all tasks and train the DQN on each task for 3000 episodes.
Handcrafted policy. Besides these RL baselines, also consider handcrafted policy with basic localization and planning modules. The localization module borrows the idea of SLAM which uses the common features between successive frames localize the robot. For the planning modules, we simply let the robot move to the nearest empty grid which should be built. The detail handcrafted algorithm is shown in Algorithm The algorithm divided into two sub-modules, localization and planning. Localization receives current location 4, current observation or and next observation o++1 to determine next position 44+1 by finding the common features from two successive observations or and 0.+1- We borrow this idea from visual SLAM which uses similar mechanism to localize. The planning function will find all candidate target location Leundidate for next step by comparing the current observation with the ground truth design D. Based on these candidate positions, the agent will decide whether to move to nearest candidate position or build at current position. The priority action space Aprior is used to decide the action when the robot does not receive any specifie action commend from the nearest planning policy. This priority action will change when robot touches the boundary of the environment. This mechanism helps robot explore more spaces of the unknown environment.
Human. For assessing human performance, we made a simple GUI game (details and video samples of the game in supplementary material) that allows a human player to attempt our tasks in identical environments with the same limita- tions of partial-observabiiity and step size un- certainty. They act as the agent, using the AR- ROW keys to maneuver, and SPACE to drop brick. The human baseline GUI game is shown in Figure B.2. In constant task environments, human players are required to complete the task without the access to the ground-truth design, while in variable task environments, they can reference the current ground truth design. Ad- ditionally, players can toggle between training or evaluation mode. In training mode, they can view per-step reward as well as their cumulative reward over the episode, whereas in evaluation mode, they can only see the number of bricks used and steps taken. For each episode played. players reported their episode-loU.
(a) Training (ID).
(b) Evaluation (2D).
Figure B.2: Game GUT for measuring human per- formance. (a) 1D Variable design: user can see the ground truth design in dynamic environments. (b) 2D Constant design in evaluation mode: only step and brick count are shown, while rewards are hidden.
C BENCHMARK RESULT
In this section, we show the quantitative results of each baseline on all tasks in Table 7. We also explain imitation learning and the Rainbow method ablation study here.
Learning from expert experience. Because most of the model-free and model-based RL methods cannot perform well on 2D and 3D tasks due to the afore- mentioned challenges, we are curious to study whether learning from expert experience could be one potential solution to mobile construction tasks. Therefore, we collect expert experiences on 2D constant sparse and variable dense tasks via our handcrafted policy which performs better than the RL methods. Then we train the DRQN baseline with these (instead of random) ex- periences in the initial replay buffer. Table 6 shows that learning from expert can indeed boost the performance of DRQN, especially for the worst perform- ing cases.
Comstant spuss
Asg
0.016
DRQN
Sidder
0.078
0,014
Min
0.2
Ang
0.33
0.3
Expert experience!1)
Sidder
0.054
0.08
Min
0,30
0.12
Table 6: Quantitative results of training DRQN with expert experience.
17

---

Published as a conference paper at ICLR 2023
0.7
3.3
carllice jos double))
rarlbiow tio muthccepl
3.1
too pricitzzd)
Colocdess
Figure C.I: Training history of IoU: the full Rainbow algorithm does not always outperform its pruned configurations.
Rainbow method ablation study. We observed similar performance across most tasks when com- paring the base DQN algorithm with DQN plus Rainbow. To better understand the effect of each individual extension on the whole, we conducted six separate runs: in each, we removed one ex- tension from the complete Rainbow algorithm, similar to (Hessel et al., 2017). Figure C.I displays the loU running average over training in the 2D variable environment for each of these pruned configurations. Overall, we find that the complete Rainbow algorithm initially learns faster than all but one of the pruned configurations, suggesting that the combination of optimizations indeed leads to better sample-efficiency in the learning process. Removing noisy networks (reverting to an epsilon-greedy exploration approach) led to the largest decrease in learning efficiency, Over longer horizon, the top testing performance plateaus to nearly the same across all configurations. In the case of distributional learning, multi-step learning, and double DQN, removing each individual actually improved top test performance, from 0.676 to 0.685, 0.701, and 0.714 respectively. While this study not exhaustive, it suggests that the Rainbow algorithm does not easily generalize from the Arcade Learning Environment (Bellemare et al., 2013) (which it was designed for) to our specific tasks.
Construction accuracy with respect position estima- tion error. We also conduct an experiment to test how the position prediction precision of L.-Net could effect the construction performance of DRQN agent. We test DRQN w/ L-Net on 2D variable sparse task for 500 times and get average L, distance between predicted and GT positions and loU of each game. As shown in the Fig- ure C.2, we find that the construction performance of the policy expected to effected by the accuracy posi- tion estimation by L-Net. With higher localization error, the construction will also be poorer. This indicates that designing appropriate localization module is key path to solving mobile construction tasks.
Anrigg piobes econiissee
Figure C.2: Results of construction curacy with respect to position estima- tion error. The red line shows linear lation between two examined values
18

---

Published as a conference paper at ICLR 2023
ID
2D
3D
loU
Constant
Variahle
Constant
Variable
Constant
Variable
Dense
Spurse
Spurse
Dense
Sparse
Dense
Sparse
Avg
0.83
0.881
0.803
0.606
0.853
0,93
0.843
0.759
0.871
0.863
Human
Min
0.448
0.686
0.306
0.053
0.254
0.156
0.3
0,232
0.313
0.101
Handcraft
Avg
0.948
0,995
0.953
0.655
0.633
0.42
0.336
0.679
0.353
0.379
0.839
0.279
0.029
0
Stddev
0.03
0.013
0.034
0.177
0.218
0.249
0.09
0.221
0.178
0.267
DQN
Avg
0.799
0.781
0.744
0.234
0.061
0,044
0.024
0.022
Min
0.439
0.353
0.164
Stádev
0.061
0.092
0.075
0.052
0.060
0.042
0.014
0.013
Avg
0.861
0,850
0.818
0.538
0.427
0.068
0.071
0.009
DRQN
Min
0.67
0.502
0.724
0.2
Stddev
0.034
0.075
0.032
0.078
0.165
0.053
0.048
0.015
DRON-Hindsight
Avg
0.842
0.804
0.645
0.191
0.361
0.045
0.069
0.054
Min
0.67
0.467
0.545
0.083
0.018
Stddev
0.034
0.075
0.171
0.045
0.02
0.424
01,38
0.142
0.14
0.053
0.021
0.013
0.009
Min
0.189
0,136
0.06
0.023
0
0
Stddev
0.102
0.13
0.034
0.019
0.047
0.033
0.011
0.014
0.788
0.794
0.65
0.294
0.358
0.140
0.317
0.007
Min
0.57
0.446
0.457
0.179
0
0
0.04
0
Stddev
0.039
0.065
0.055
0.038
0.114
0.068
0.092
0.012
Avg
0.823
0.792
0.725
0.507
0.072
0.021
0.112
Rainbow
Min
0.533
0.484
0.175
0.172
0
0.022
Stddev
0.064
0.061
0.093
0.093
0.087
0.035
0.038
DQN-MCTS
Avg
0.745
0.738
0.752
0.154
0.135
0.020
0.032
0.015
0.013
0:007
Min
0.059
0.313
0.587
0.031
0
0
0.013
0
0
Stddev
0.109
0.084
0.075
0.046
0.106
0.030
0.015
0.01
0.02
0.012
Avg
0.829
0.799
0.869
0.662
0.687
0.065
0.098
DQN w/ LeNet
Min
0.388
01366
0.693
0.246
0.069
0
0.015
0.008
Stddev
0.043
0.072
0.038
0.094
0.132
0.066
0.053
0.024
Avg
0.896
0.863
0.854
0.684
0.7
0,301
0.374
0.315
0.143
0.053
DRQN w/ L-Net
Min 0.624
0.38
0.696
0.289
0.132
0
0.072
0.049
0
0
Stddev
0.059
0.079
0.036
0.091
0.137
0.151
0.107
0.095
0.076
0.041
Table 7: Benchmark quantitative results. Empty cells indicate the agents failed at the these task without learning any control policy. Blue purple represent best performance of learning-basee methods w/ & w/o L-Net.
19

---

Published as a conference paper at ICLR 2023
DON
1:th: DUAGE
JUU 0.73
660-0.855
0.340
8.136
140:6.145
2.00
(MU 0.064
I:l:: 0.093
300--3.00
DAGN
kelt
P
DON-MCTS TS
8,266
P
SAC
[el: 0,935
Tel: 0,352
W0100 388
14U:80055
100:8204
Iel:: 0827
Jel:: 0,18
BAU 0,993
BIKO
DAC: GO0S
IAU: 6015
Iet::
IAU 0,900
DRON
fal:
JK0--.544
Iall: BUACO
9-355
IADI 8:00
1e1: 1.000
lol: 1,000
(A): LO00
1ol::10000
1000
Iel: RAM
lel: 0.64
Tell: Lave
Bui 0.19
Figure C.3: The best testing cases of baselines and our method on all tasks.
20

---

Published as a conference paper at ICLR 2023
Algorithm Handcrafted Policy
1: function MAIN
2: Initialize environment and get initial observation Ot=0
3: Get initial location Pr=0
4: Initialize priority action space Aprior
5: for t t 0, NN do
6:
Ge: A
Ot, Aprior)
7:
Execute a and obtain next observation 04+1, reward, and terminal sign Done from en-
vironment.
8:
Pt+1 LOCAIIZZTIINN(, Ot, 0c41, 0c)
9:
Done then
10:
Break
11:
function LOCALIZATIINNP.O:00H...
12:
Common Feature = FEATUREMATCHINGIOO, 0r+1)
if Common Feature is empty or ot+1 then
14:
Determine Pt+1 using Odometry, here we assume step size is always 1. return Pi-1
else
16:
Determine step size using Common Feature. Calculate P++1 using step size and a;. return Pc+1
18:
function PLANNING((,, Ot, Aprior)
19:
Peandidan: COMPARE(O;, D(p())
Update Aprior based the current boundary condition if P PPaaddiaate is empty then
21:
22:
0t random sampled from Aprior return at Aprior
23:
else
24:
Find nearest location Phear from Peandidate
25:
Prear Pt then return a=Drop brick, Apriar
26:
else
27:
Determine action a, based on the corresponding direction of Poear to Pr return Cty Aprior
21


Reviews:
[{"review_id": "wBXcxAKBbz", "reviewer": "Reviewer_SRjN", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents the mobile construction POMDP to formulate the task of a mobile agent placing blocks according to some design, while using observations of its (dynamic) environment to navigate. The paper introduces various instantiations of, and complications to, this task, such as operating in 1/2/3D settings and with/without obstacles. Finally, the paper demonstrates that a deep recurrent Q-network (DRQN) in concert with a localization network (L-net) outperforms baseline approaches and achieves good results on some test cases but leaves room for future work to succeed in others.", "summary_of_the_review": "This paper serves as a solid foundation for future works to explore more sophisticated approaches to the mobile construction task, as well as more sophisticated formulations of the task itself that capture more real world complexities. The paper does an excellent job of exploring the performance and limitations of existing approaches, and presents a new and somewhat novel approach that outperforms the baselines. The clarity could use some work, and there is some concern about the reproducibility of the experiments. If these issues were addressed, I would feel quite comfortable upgrading my rating.\n\nUpdate: Based on author responses and revised manuscript, I am upgrading my rating to an 8. Please see comments for details.", "strength_and_weaknesses": "Strengths\n------------\nNovelty\n- The mobile construction POMDP is interesting and novel\nSignificance\n- RL for mobile construction is highly relevant to a growing subfield in robotics\n- The proposed DRQN+L-Net approach significantly outperforms the baselines in the most challenging test cases\n- The problem specification is quite general and extensible for future works to build upon and compare against\n- The ablation study is very well done and points towards promising future directions\nClarity\n- The paper is mostly clear and concise\n- Figures are detailed and effective at visualizing the task and results\n- Tables 1, 2, and 4 are effectively summarize a great deal of information clearly\nSoundness\n- The POMDP formulation, choice of evaluation metric (IoU), and other design decisions seem sound\n\nWeaknesses\n-----------------\n- Marginal novelty algorithmic contribution\n- Proposed POMDP is similar enough to existing grid-world type problems that there are not many totally new/unexpected insights\n- Feels like a small step towards a more realistic model/simulation of mobile construction\n- Font sizes in tables and figures are way too small (unreadable when printed, hard to read on screen without extreme zoom)\n- Some grammatical and spelling issues to address\n- \"So we removed the penalty for better performance\" -- better performance on what algorithm(s)? If this is to improve the performance only of the proposed approach, it seems inappropriate to make it part of the experimental setup.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "LPwCJiDvtg", "reviewer": "Reviewer_cAFg", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors introduce a new problem of mobile construction, where an autonomous agent must construct a grid world according to a design (target grid world state) that is fed as input to the agent. To tackle this problem, the authors de-couple the approach into (1) learning to localize the agent via an L-Net, and (2) zoom in on the design based on the localization info and feed that, along with the observation, as input to a DNN to predict the actions.", "summary_of_the_review": "My main concern is how the findings and contributions of this paper can be applied more generally. For example, the authors propose the L-Net as a useful decoupling, but this approach is specific to the grid world setup. In more realistic setups, such as iGibson or Habitat, many navigation approaches already predict localization. Also, the experimental findings which show that explicit localization prediction is helpful for the mobile construction task, and that adding obstacles in the grid worlds reduces performance drastically, are somewhat obvious.\n\nUpdate: Based on the other reviewers' comments and the authors' responses, I've updated my review to be 6.  I agree with the authors' responses that decoupled representations are helpful for this task, and that the findings are not as obvious as I initially thought.", "strength_and_weaknesses": "## Strengths\n* The authors introduce an interesting problem of mobile construction, and develop a grid world setup for experimenting on this task.\n* The authors propose a novel approach of localization via an L-Net and action prediction via Deep Recurrent Q Network.\n* The authors include extensive experiments: comparisons against human performance, ablations, and comparisons against standard RL baselines (e.g. PPO, DQN) and hand-crafted policies under different setups (e.g. obstacles, no location uncertainty, etc.)\n\n## Weaknesses\n* It's not clear to me that the findings in the grid world experiments will transfer easily to more complicated setups, such as mobile construction and mobile manipulation in the real world, in a 3D physics-based simulator, or in minecraft.\n* Moreover, most of today's learning-based approaches for learning-based navigation/mobile manipulation (e.g. Savinov et al. semiparametric topological mapping; learning to explore via active neural SLAM) already utilize localization information (and same with non-learning based approaches ie. SLAM), so the finding that decoupling localization from action is helpful for mobile construction is also not very surprising.\n\n## Notes\n* Regarding the reward and adding penalties: \"we found this led to more frequent actions of moving instead of dropping bricks\". It may be useful to add a \"time\" penalty for each action that doesn't result in a positive reward so that the agent can be more efficient in construction.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "6U77eQbmoWD", "reviewer": "Reviewer_fEQb", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces a new task, mobile construction, to evaluate the abilities of an agent to localise within a dynamic environment. The goal is to navigate an environment to build a structure, either in 1, 2 or 3D specified as a goal. The observation available to the agent is the state of the environment in its current neighbourhood. The fact that agents modify themselves the appearance of the environment lead to challenges in terms of localization and mapping. \n\nDifferent baselines from the literature are considered, and compared against humans and handcrafted baselines. The main takeaway is that the task is challenging, and thus an interesting direction to pursue. Authors also provide a study of the current limitations of agents, by varying the available given inputs.", "summary_of_the_review": "This work introduces an interesting task, along with an evaluation of diverse baselines from the literature. Compared with other robotics, Embodied AI tasks that also necessitate localisation and mapping, the observations to the agent are much further from real-world observations. However, this should not be considered as a weakness as it allows to properly isolate the studied capacity, i.e. localization in a dynamic environment. Moreover, the ability of the agent to modify its environment makes it an interesting and challenging task for current SLAM methods.\n\nI am however concerned with the much stronger performance of handcrafted baselines, compared with all other agents, and in particular the agent equipped with the L-Net module introduced in this work. Authors should elaborate on what additional prior knowledge these handcrafted baselines have access to, as it looks to me like the design of L-Net is also guided by some prior knowledge about the task at hand.", "strength_and_weaknesses": "Strengths:\n1. The introduced task targets an important problem, i.e. localization, and challenges current SLAM and sfM methods due to the dynamic status of the environment caused by construction.\n2. Authors evaluate a variety of baselines from the literature, and experiments are thorough. The ablation study on the impact of modifying the given input to the agent is interesting, and allows to better understand the source of current agent limitations.\n3. The paper is generally well written.\n\nWeaknesses:\n1. Compared with other tasks, e.g. in Embodied AI, the environment the agent evolves in is not photorealistic. However, the simple nature of the task can be a good thing to isolate core problems to be solved before to move to environments that are closer to what humans experience in the real world. Moreover, as shown in the paper, the task can not be considered as trivial as most policies do not get strong performance. I would thus tend not to consider lack of realism as a weakness here.\n2. Handcrafted methods beat all other baselines in most cases. However, authors mention they have access to more prior knowledge. I would like authors to elaborate on this, as it is an important concern in this work.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "1m6H3_mP5CE", "reviewer": "Reviewer_FZE1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new challenge for AI -- Mobile Construction. In Mobile Construction an agent is initialized in an environment and tasked with building a given structure. The challenge of the task is that it is long horizon and the agent must operate from local observations.\n\nThere are 3 versions of the task that are proposed, 1 in 1D, 1 in 2D, and 1 in 3D. This gives an increasing amount of difficulty.\n\nTowards solving this task, the authors propose to explicitly learn localization and incorporate it into the agent by cropping and transforming the target design to be relative to the agents current location.\n\nThe proposed method outperforms all the various baselines in the most challenge 3D case.", "summary_of_the_review": "Overall, my main concern with this paper is lack of evaluation of the environment's simulation performance. This seems like a key part of characterize given that the environment is one that, at least in my understand, should be able to be simulated very quickly.", "strength_and_weaknesses": "## Strengths\n\n- The proposed task seems challenging for RL but conceptually quite simple. A nice mix.\n- The comparisons to existing methods are well done and thorough.\n- The proposed method is effective and the way of incorporating positional information is interesting.\n- Ablations are well done.\n\n## Weaknesses\n\nNo simulation speed numbers are given for the environment itself. Given that this seems to be a visually simplistic environment, I imagine it can be quite fast, but specific numbers would be very useful.\n\nOn that note, it also seems like the environment doesn't support batched simulation. I encourage the authors to think about supporting this as such an environment seems idea for this and a faster environment almost always gains more traction.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster