MIRA: Mental Imagery for Robotic Affordances
Authors: Yen-Chen Lin
Pete Florence
Andy Zeng
Jonathan T. Barron
Yilun Du
Wei-Chiu Ma
Anthony Simeonov
Alberto Rodriguez Garcia
Phillip Isola
Venue: CoRL 2022 Poster (2022)
Keywords: Neural Radiance Fields
Rearrangement
Robotic Manipulation

Abstract: Humans form mental images of 3D scenes to support counterfactual imagination, planning, and motor control. Our abilities to predict the appearance and affordance of the scene from previously unobserved viewpoints aid us in performing manipulation tasks (e.g., 6-DoF kitting) with a level of ease that is currently out of reach for existing robot learning frameworks. In this work, we aim to build artificial systems that can analogously plan actions on top of imagined images. To this end, we introduce Mental Imagery for Robotic Affordances (MIRA), an action reasoning framework that optimizes actions with novel-view synthesis and affordance prediction in the loop. Given a set of 2D RGB images, MIRA builds a consistent 3D scene representation, through which we synthesize novel orthographic views amenable to pixel-wise affordances prediction for action optimization. We illustrate how this optimization process enables us to generalize to unseen out-of-plane rotations for 6-DoF robotic manipulation tasks given a limited number of demonstrations, paving the way toward machines that autonomously learn to understand the world around them for planning actions.

MIRA: Mental Imagery for Robotic Affordances
Lin Yen-Chen', , Pete Florence", Andy Zeng", Jonathan T. Barronn Yilun Du', Wei-Chiu Ma', Anthony Simeonov', Alberto Rodriguez Garcia', Phillip Isola' MIT Google
Abstract: Humans form mental images of 3D scenes support counterfactual imagination, planning, and motor control. Our abilities to predict the appearance and affordance of the scene from previously unobserved viewpoints aid us in performing manipulation tasks (e.g,, 6-DoF kitting) with level of ease that is currently out of reach for existing robot learning frameworks. In this work, we aim to build artificial systems that can analogously plan actions on top of imagined images. To this end, we introduce Mental Imagery for Robotic Affordances (MIRA), an action reasoning framework that optimizes actions with novel-view synthesis and affordance prediction in the loop. Given of 2D RGB images, MIRA builds consistent 3D scene representation, through which we synthesize novel orthographic views amenable to pixel-wise affor- dances prediction for action optimizationn. We illustrate how this optimization process enables us to generalize to unseen out-of-plane rotations for 6-DoF robotic manipulation tasks given a limited number of demonstrations, paving the way toward machines that autonomously learn to understand the world around them for planning actions.
Keywords: Neural Radiance Fields, Rearrangement, Robotic Manipulation
1
Introduction
Suppose you are shown a small, unfamiliar object and asked if could fit through an M-shaped slot. How might you solve this task? One approach would be to "rotate" the object in your mind's eye and see if, from some particular angle, the object's profile fits into an M. To put the object through the slot would then just require orienting it to that particular imagined angle. In their famous experiments on "mental rotation". Shepard & Metzler argued that this is the approach humans of novel shapes [1]. Decades of work in psychology have documented numerous other ways that "mental images", i.e. pictures in our heads, can aid human cogmition [2]. In this paper, we ask: can we give robots a similar ability, where they use mental imagery to aid their spatial reasoning?
when reasoning about the relative poses
Fortunately, the generic ability to perform imagined translations and rotations of scene, also known as novel view synthesis, has seen a recent explosion of research in the computer vision and graphics community [3, 5). Our work builds in particular upon Neural Radiance Fields (NeRFs) [6], which can render what scene would look like from any camera pose. We treat a NeRF as robot's "mind's ss eye", a virtual camera it may use to imagine how the scene would look were the robot to reposition itself. We couple this ability with an affordance model [7), which predicts, from any given view the scene, what actions are currently afforded. Then the robot must just search, in its imagination, for the mental image that best affords the action wishes to execute, then execute the action corresponding to that mental image. We test this framework on 6-DoF rearrangement tasks [8], where the affordance model simply predicts, for each pixel in a given camera view, what the action-value of picking (or placing) that pixel's coordinates. Using NeRF virtual camera for this task has several advantages over prior works which used physical cameras:
Out-of-plane rotation. Prior works have applied affordance maps to 2-dimensional top-down camera views, allowing only the selection of top-down picking and placing actions [7, 9, 10]. We instead formulate the pick-and-place problem as action optimization process that searches across different novel synthesized views and their affordances of the scene. We demonstrate that this optimization process can handle the multi-modality of picking and placing while naturally supporting actions that involve out-of ofl-nee rotations.
6th Conference Robot Leaming (CoRL 2022), Auckland, New Zealand.

---

Collect Perspeclive Images Optimize NeRF
Render Novel Views w/ Orthographic Ray Casting
Feed Novel Views into the Policy
Execute Action
X
Polley
Polley
M-
place
(a)
(b)
(c)
(d)
Figure 1: Overview of MIRA. (a) Given set of multi view RGB images as input, optimize a neural radiance field representation of the via volume rendering with perspective ray casting. (b) After the NeRF is optimized. we perform volume rendering with orthographic ray casting render the scene from V viewpoints. orthographic images are fed into policy predicting pixel-wise action-values that correlate with picking and placing success. (d) The pixel with the highest action-value is selected, and its estimated depth and associated view orientation used to parameterize the robot's motion primitive.
The rendered
(M)
(h)
Real-world Results. MIRA is sample efficient learning vision basdd 6-DoF manipulation tasks: packing flosses (a-b), packing metal cubes (c-d), and putting the metal sphere into cups (c-f). These tasks are challenging for methods that rely on 3D sensors as objects contain thin structures or are composed of specular or semiitransparent material. MIRA is able to solve these tasks because only requires RGB inputs.
Figure 2:
Orthographic ray casting. NeRF trained with images from consumer cameras can be used to synthesize novel views from novel kinds of cameras that are more suitable to action reasoning. Most physical cameras use perspective projection, in which the apparent size of an object in the image plane is inversely proportional to that object's distance from the camera - relationship that any vision algorithm must comprehend and disentangle. NeRF can instead create images under other rendering procedures; we show that orthographic ray casting is particularly useful, which corresponds to non-physical "camera" that is infinitely large and infinitely distant from the scene. This yields images in which an object's size the image plane invariant to its distance from the camera, and its appearance is equivariant with respect to translation parallel to the image plane. In essence, this novel usage of NeRF allows us to generate "blueprints" for the scene that complement the inductive biases of algorithms that encode translational equivariance (such as ConvNets).
RGB-only, Prior rearrangement methods [11, 12, 13] commonly require 3D sensors (e.g. via structured light, stereo, or time-of time-of-flghh), and these are error-prone when objects contain thin structures or are com- posed of specular or semi transsrrnpp materials a common occurrence (see Fig. for examples). These limitations drastically restrict the of tasks, objects, and surfaces these prior works reason over.
We term our method Mental Imagery for Robotic Affordances, or MIRA. To test MIRA, we perform experiments in both simulation and the real world. For simulation, extend the Ravens [9] benchmark to include tasks that require 6-DoF actions. Our model demonstrates superior performance existing state- of-the-art methods for object rearrangement [14, 9]. despite not requiring depth sensors. Importantly, the optimization process with novel view synthesis and affordance prediction in the loop enables our framework to generalize to contribute framework that uses NeRFs as the scene representation to perform novel view synthesis for precise object rearrangement, (ii) an orthographic ray casting procedure for NeRFs rendering that facilitates the policy's translation equivariance, (iii) extended benchmark of 6-DoF manipulation tasks in Ravens [9], and (iv) empirical results on a broad range of manipulation tasks, validated with real-robot experiments.
object configurations, where the baselines struggle. In summary, we

---

2 Related Works
2.1 Vision-based Manipulation.
Object-centric. Classical methods in visual perception for robotic manipulation mainly focus on rep- resenting instances with 6-DoF poses [15, 16, 17, 18, 19, 20, 21]. However, 6-DoF poses cannot represent the states of deformable objects granular media, and cannot capture large intra-category variations of unseen instances [11]. Altemative methods that represent objects with dense descriptors [22, 23, 24] or keypoints [11, 25, 26, 27] improve generalization, but they require dedicated data collection procedure (c.g., configuring scenes with single objects).
Action-centric. Recent methods based on end-to-end learning directly predict actions given visual obser- vations [28, 29, 30, 31, 32, 33j. These methods can potentially work with deformable objects or granular media, and do not require any object-specific data collection procedures. However, these methods are known to be sample inefficient and challenging to debug. Recently, several works [9. 13, 34, 35, 36, 37, 38] have proposed to incorporate spatial structure into action reasoning for improved performance and better sample efficiency, Among them, the closest work to ours is Song al. [13] which relies on view synthesis to plan 6-DoF picking. Our work differs in that it 1) uses NeRF whereas [13] uses TSDF [39], 2) does not require depth sensors, 3) uses orthographic image representation, 4) does not directly use the camera pose as actions, and 5) shows results on rearrangement tasks that require both picking and placing.
2.2 Neural Fields for Robotics
Neural fields have emerged as promising tool to represent 2D images [40], 3D geometry [41, 42), appearance [6, 43, 44], touch [45], and audio [46, 47]. They offer several advantages over classic representations (e.g., voxels, point clouds, and meshes) including reconstruction quality, and memory efficiency. Several works have explored the usage of neural fields for robotic applications including localization [48, 49], SLAM [50, 51, 52], navigation [53], dynamics modeling [54, 55, 56, 57J, and reinforcement learning [58]. For robotic manipulation, GIGA [59] and NDF [12] use occupancy networks' feature fields to help action prediction. However, both of them rely depth cameras to perceive the 3D geometry of the scene, while MIRA does not require depth sensors and thus can handle objects with reflective or thin-structured materials. Dex-NeRF [60] infers the geometry of transparent objects with NeRF and determines the grasp poses with Dex-Net [30]. However, it only predicts the 3-DoF grasping pose and does not provide a solution for pick-conditionee placing; NeRF-Supervision [61] uses NeRF as a dataset generator to learn dense object descriptors for picking but not placing. In contrast, MIRA capable of predicting both 6-DoF picking and cick-conditioned placing. Model-based methods [54, 55] use NeRFss as decoders to learn latent state representations for model predictive control; NeRF-RL [58] instead uses the learned latent state representations for downstream reinforcement learning. Compared to these works, MIRA uses imitation learning to acquire the policy and thus avoid the conundrum of constructing reward functions. Furthermore, MIRA enjoys better sample efficiency due to the approximate 3D rotational equivariance via novel-view synthesis. The last but not least, MIRA demonstrates real-world results on three 6-DoF kitting tasks while these works primarily focus on simulation benchmark.
3 Method
Our goal is to predict actions a,, given RGB-only visual observations or, and trained from only limited number of demonstrations. We parameterize our action space with two-pose primitives a; = (Tpic Tplace), which are able to flexibly parameterize rearrangement tasks [9]. This problem is challenging due to the high degrees freedom of a, (12 degrees of freedom for two full SE(3) poses), a lack of information about the underlying object state (such as object poses), and limited data. Our method (illustrated in Fig. 1) factorizes action reasoning into two modules: 1) continuous neural radiance field that can synthesize virtual views of the scene novel viewpoints, and 2) optimization procedure which optimizes actions by predicting per-pixel affordances across different synthesized virtual pixels. We discuss these two modules in Sec. 3.1 and Sec. 3.2 respectively, followed by training details in Sec. 3.3.
3

---

(a) 3D scene
(b) Perspective ray casting
(c) Orthographic ray casting
Figure 3: Perspective Orthographic Ray Casting, (a) 3D world showing two objects, with the camera located the top. (h) The procedure of perspective ray casting and perspective rendering of the scene. The nearby object large, distant object is small, and both objects appear "tilted" according to their position. (c) The procedure of
orthographic ray casting and an orthographic rendering of the scene, which does not correspond to any real consumer camera, wherein the size and appearance of both objects are invariant to their distances and equivariant to their locations. able to By using NeRF synthesize these orthographic images, which correspond non-physical cameras, construct RGB inputs that are equivariant with translation.
3.1 Scene Representation with Neural Radiance Field
To provide high-fidelity novel-view synthesis of virtual cameras, we represent the scene with a neural radiance field (NeRF) [6]. For our purposes, key feature of NeRF is that it renders individual rays (pixels) rather than whole images, which enables flexible parameterization of rendering at inference time, including camera models that are non-physical (e.g., orthographic cameras) and not provided in the training set.
To render a pixel, NeRF casts a ray r(() =0+)= td from some origin o along the direction passing through that pixel on an image plane. In particular, these rays are casted into field Fe whose input a 3D location x= (z) and unit-norm viewing direction d, and whose output an emitted color c= (r.g.b) and volume density Along each ray, K discrete points {x =r(tx)) ) K are sampled for use as input Fe, which outputs set of densities and colors (a J (Fe d) Volume rendering [62] with a numeri- cal quadrature approximation [63] is performed using these values produce the color C(r) of that pixel:
C(r)=
Te 1-exx((ooR(te+)) (otttttitt))))))))))))) C2.
T3 =e=e (- (-E ohe (te (te(tt++-ttt)) W<k
where Ti represents the probability that the ray successfully transmits to point r(te). At the beginning of each pick-and-place, our system takes multi-view posed RGB images as input and optimizes θ by minimizing photometric loss Cphoro = EEEE [|((]] C(r) -(r)) using some sampled set of rays re R. where C(r) is the observed RGB value of the pixel corresponding ray in input image. In practice, we use instant-NGP [64] to accelerate NeRF training and inference.
Orthographic Ray Casting, In Fig. we illustrate the difference between perspective and orthographic cameras. Though renderings from a NeRF Fe are highly realistic, the perspective ray casting procedure used by default in NeRF's volume rendering, which we visualize in Fig. 3(b), may cause scene content to appear distorted or scaled depending on the viewing angle and the camera's field of view - more distant objects will appear smaller in the image plane. Specifically, given a pixel coordinate (u,v) and camera pose (R.t), NeRF forms ray r r= (o.d) using the perspective camera model:
(u f d=R
o=t,
(U Co S/5
This model a reasonable proxy for the geometry of most consumer RGB cameras, hence its use by NeRF during training and evaluation. However, the distortion and scaling effects caused by perspective ray casting degrades the performance of the downstream optimization procedure that takes as input the synthesized images rendered by NeRF, as we will demonstrate in results (Sec. 3.4). To address this issue, modify the rendering procedure of NeRF after it is optimized by replacing perspective ray casting

---

with orthographic ray casting:
-u-cz)/ fz o=t+R (0 (w-Cy) Cy
U d= R 0
(3)
To our knowledge, our work is the first to demonstrate that when a NeRF model is trained on perspective cameras, may directly be utilized to render orthographic images. Such a result both non-obvious and surprising because rays are now rendered using different volume rendering procedure from the one used during training time, and the orthographic rendering process essentially corresponds to an rendering test on the NeRF model.
We visualize this procedure in Fig. 3(c). Orthographic ray casting marches parallel rays into the scene, so that each rendered pixel represents a parallel window of 3D space. This property removes the dependence between an object's appearance and its distance to the camera: an object looks the same if either far nearby. Further, all rays for a given camera rotation R are parallel, this provides equivariance the in-plane camera center Ca.Cy. These attributes facilitate downstream learning to be equivariant to objects' 3D locations and thereby encourages generalization. As open-source instant-NGP [64] did not support orthographic projection, we implement this ourselves as a CUDA kernel (see Supp.).
Our decision choosing an orthographic view of the scene draws inspiration from previous works that have used single-view orthographic scene representations [16, 9], but critically differs in the following two aspects: (a) create orthographic scene representations by casting rays into radiance field rather than by point-cloud reprojection, thereby significantly reducing image artifacts (see Supp.); (b) we rely on muewi-view RGB images to recover scene geometry, instead of depth sensors.
3.2 Policy Representation: Affordance Raycasts in a Radiance Field
To address paramm-arrmeteerize actions, we formulate action selection an optimization problem over synthesized novel-view pixels and their affordances. Using our NeRF-based scene representation, we densely sample V camera poses around the workspace and render images I,, = Fee for each pose, Ve, = 1..1,.V... One valid approach is to search for actions directly in the space of camera poses for the best Te., but orders of magnitude computation may be saved by instead considering actions that correspond each pixel within each image, and sharing computation between all pixels in the image (e.g.. by processing each image with just single pass through ConvNet). This (i) extends the paradigm of pixel-wise affordances [7] into full 6-DOF, wwel-view-enabllee action spaces, and (ii) alleviates the search over poses due to translational equivariance provided by orthographic rendering (Sec. 3.1).
Accordingly, we formulate each pixel in each synthesized view as parameterizing a robot action, and we learn dense action-value function E which outputs per-pixel action-valuee of shape Hx given novel-view image of shape I Hx33 Actions selected by simultaneously searching across all pixels in all synthesized views tig
U. e = arggmnn E(J,,
Vo.=0,1,...VV
(4)
where the pixel up and the associated estimated depth d(u;) from NeRF are used to determine the 3D translation, and the orientation of Tez used determine the 3D rotation of the predicted action. Our approach employs single ConvNet that is shared across all views and use multiple strategies for cquivariance: 3D translational equivariance is in part enabled by orthographic raycasting and synergizes well with nanslationalll-equiieee
dense model architectures for E such as ConvNets [65, 66), meanwhile 3D rotational equivariance is also encouraged, as synthesized rotated views can densely cover novel orientations of objects.
While the formulation above may be used predict the picking pose Tpick and the placing pose Tplace independently, intuitively the prediction of Tpick affects the prediction of Tplace due to the latter's 's geometric dependence on the former. We therefore decompose the action-valle value function into (i) picking and (ii)
5

---

(a)
Figure 4: Simulation qualitative results. MIRA only requires RGB inputs and can solve different 6-DoF tasks: (a)
hanging-disks, (b) place-red-in-green, (c) stacking-object, and (d) Block-insertion.
pick-conditioned placing, similar prior work [9]:
Pick: pick argmin Shick(eapus :lppcck;
Vpiick= = ,V
(5)
1pec
"placo="paxc =
argmin Fiplace (1 Slygue: picks Ppick) Makke
= 0,,,".,V
(6)
where Pspaace uses the Transport operation from Zeng al. [9] to convolve the feature map of T.j around pick with the feature maps of ( / for action-value prediction. We refer readers to [9] for details on this coupling.
3.3 Training
We train the action-value function with imitation learning. For each expert demonstration, we construct a tpple D (hr M ] where INje and INi... are the synthesized images whose viewing directions are aligned with the end-effeccor's rotations; upus and uficce are the best pixels in those views annotated by experts. We draw pixels {ūاlå o Ufick o ufacce Nos from randomly synthesized images Ing as negative samples. For brevity, omit the subscript for pick and place and present the loss function that is used to train both action-value functions:
Eo(l.u
C(D)= -logp [ [u"[i, [
(ir)) p(u" (("""i,,neg ( (1,)))))=== e-Eo(I.u)
Now + و e -
A key innovation of our objective function compared to previous works [9, 10, 36, 67] is the inclusion Nia
of negative samples (ú, ji=l from imagined views Lug. We study the effects of ablating negative samples in Sec. 3.4 and show that they are essential for successfully training action-value functions. In practice, we batch the image that contains the positive pixel and other synthesized views for the forward pass. Every pixel in these images are treated as samples to compute the loss.
We execute experiments both simulation and real-world settings evaluate the proposed method across various tasks.
3.4 Simulation Experiments
Environment. We propose four new 6-DoF tasks based on Ravens [9] and use them as the benchmark. We show qualitative examples of these tasks in Fig. 4 and summarize their associated challenges the supplementary materials. Notably, place-red-in-gree
a relatively cluttered environment where 5-10 distractor objects are randomly spawned and placed; hanging-diske and stacking-objectt require the policy to generalize novel objects that not seen during training. All simulated experiments conducted in PyBullet [68] using Universal Robot UR5e with suction gripper. The input observations for MIRA are 30 RGB images from different cameras pointing toward the center. For all the baselines, we additionally supply the corresponding noiseless depth images. Each image has resolution of 640 x 480. The camera has focal length 450 and camera center (c2,cy) = (320,240) Demonstrations are collected with motion planner that can access the ground-truth states of objects.
Evaluation. For each task, perform evaluations under two settings: in-distribution configures objects with random rotations (0,.0, [- 2 [-), This is also the distribution used

---

to construct the training set. out-of-distribution instead configures objects with random rotations (0.-0, - T E We note that these rotations are outside the training distribution and also correspond to larger out-of-plane rotations. Thus, this setting requires stronger generalization. We use binary score (0 for failure and for success) and report results on 100 evaluation runs for agents trained with 1,10,100 demonstrations.
Baseline Methods. Although our method only requires RGB images, we benchmark against published baselines that additionally require depth images as inputs. Form2Fit [14] predicts the placing action by estimating dense descriptors of the scene for geometric matching. Transporter-SA(() and Transporter-SSE() are both introduced in Zeng et al. [9]. Although Transporter-SE(2) not designed to solve manipulation tasks that require 6-DoF actions, its inclusion helps indicate what level task success can be achieved on the shown tasks by simply ignoring out-of-plane rotations. Transporter-SE(3) predicts 6-DoF actions estimate SE(2) actions, and then feeding them into regression model to predict the remaining rotational (Tx:;y)) and translational (z-height) degrees of freedom. Additionally, we benchmark against baseline, GT-State MLP, that assumes perfect object poses. It takes ground truth state (object poses) as inputs and trains an MLP to regress two SE(3) poses for Tpick and Tplace.
by first using Transporter-SE(()
innsertion in distribation paises
block loockiinsetton
place raace-red--in--aa greens in distribution poses
place greens out-of distributionn
out-of distribution poses
Method
10
100
10
100
10
100
10
100
GTaState MLP
0
0
0
0
Form2Fir [14]
0
10
0
35
79
96
21
30
61
Teusporter-88(2)
30
25
Thansporter-SB(3) [9]
26
77
0
20
22
29
77
86
23
20
Ours
27
hanging-disks a-ddisiibunico-ppsss
hanging-diin out-of-distrine mut-of-ditribution-ppe
stacking-ohjent in-distribution-poer
stacking-objeets
ouloot-istilxniiooppess
Method
100
I
10
100
1
10
10
GT-Slate MLP
0
0
0
0
Form2Fir [14]
0
4
3
1
0
4
Teusporter-88(2)
3
Tramsooter-SBB [9]
6 66
75
0
32
20
0
42
40
0
16
34
Ours
Table 1: Quantitative results. Task success rate (mean %) vs. # of demonstration episodes (1, 10, 100) used in training. Tasks labeled with in an-distribution configure objects with random rotations 00,,0, E 0.0 [[-m.r)).
This also the rotation distribution used for creating training set. Tasks labeled with outtof-diittriution configure objects with rotations (0..0, f, E -- . E (ii) larger out-of-plane rotations.
I. 11.0g | m,))) that are (i) outside training pose distribution and
Results. Fig. 5 shows the average scores of all methods trained on different numbers of demonstrations. GT-State MLP fails completely; Form2Fit cannot achieve 40% success rate under any setting; Both Transporter-SE(2) and Transporter-SE(3) ~70% success rate when the object poses are sampled from the training distribution, but the success rate drops to ~30% when the object poses are outside the training distribution. MIRA outperforms all baselines by a large margin when there are enough demonstrations of the task. Its success rate is ~00% under the setting of in-distribution and ~80% under the setting of out-of-disrribution. The performance improvement over baselines demonstrates generalization ability thanks to the action optimization process with novel view synthesis and affordance prediction in the loop.
able achieve
in-distribation
ont- ont-of-distriuutiinn
100
Transpoter-SE(e)
Transpoter-S(g))
Average
CT: State MLP
40- 20
40
20
Training
T'iiiinn
Figure 5: Average of all methods under both in-distribution and out-of-distributas settings.
Interestingly, found that MIRA sometimes performs worse than
7

---

baselines when only 1 demonstration is supported. We hypothesize that this is because our action-value function needs more data to understand the subtle differences between images rendered from different views in order to select the best view. We show the full quantitative results in Table 1.
Ablation studies. To understand the importance of different components within our framework, we benchmark against two variants: (i) ours w/ perspective ray casting, and (ïi) ours w/o multi-view negative samples in Eq. (7). We show the quantitative results in Table 2. We find that ours w/ perspective ray casting fail to learn the action-value functions because the perspective images (a) contain distorted appearances of the objects that are challenging for CNNs to comprehend and (b) the groudtruth picking or placing locations may be occluded by the robot arms. We visualize these challenges in the supplementary materials. Our method with orthographic ray casting circumvents both challenges by controlling the near/far planes to ignore occlusions without worrying about the distortion and scaling. Ours w/o multi-view negative samples also fails to learn reliable action-value functions, and this may be due distribution shift between training and test: during training, it has only been supervised to choose the best pixel given an image. However, it is tasked to both select the best pixel and best view during the test time.
block insertion
block insertion
indiistribution poses
out tuttof-diittrhuutimmm poses
Table 2:
Ablation studies. We study the effects of ablating or- thographic ray casting or multi- view negative samples from our system.
Method
1
10
100
1
10
100
Ours perspective
0
0
Ours w/o niet negatives
0
11
13
0
2
4
Ours
0
84
89
0
74
78
3.5 Real-world Experiments
We validate our framework with three kitting tasks in the real world and show qualitative results in Fig. 2. Additional qualitative results and video can be found in the supplementary material. Our system consists of a URS arm, a customized suction gripper, and a wrist-mounted camera. We show that our method can successfully (i) pick up floss cases and pack them into transparent containers, (ii) pick up metal cubes and insert them into the cases, and (iii) pick up stainless steel ice sphere and place it into 10+ different cups configured with random translations and out-of of--eane rotations. See Fig. 2 for qualitative results. These tasks are challenging because (i) they include objects with reflective transparent materials, which makes these tasks not amenable to existing works that require depth sensors [9. 12, 14), and (ii) they require out-of-plane action reasoning. The action-value functions are trained with 20 demonstrations using these cups. Demonstrations are supplied by humans who teleoperate the robot through a customized user interface. At the beginning of each sock-and-place, our system gathers 30 1280 x 720 RGB images of the scene with the pristed camera. These 30 locations are sampled from a circular path on the table and the robot moves the camera each of them sequentially through inverse kinematics. Each image's 's camera pose is derived from the robotic manipulator's 's end-effector pose and calibrated transformation between the end-effector and the camera. This data collection procedure is advantageous as industrial robotic manipulators feature sub-nillimeter repeatability, which provides accurate camera poses for building NeRF, In practice, we search through V 121 virtual views that uniformly cover the workspace and predict their affordances for optimizing actions. The optimization process currently takes around 2 seconds using a single NVIDIA RTX 2080 Ti GPU. This step can be straightforwardly accelerated by parallelizing the computations with multiple GPUs
Limitations And Conclusion
In terms of limitations, our system currently requires training NeRF of the scene for each step of the manipulation. An ins [64] requires approximately 10 seconds to converge using single NVIDIA RTX 2080 Ti GPU, and moving the robot arms around to collect 30 multi-view RGB images of the scene takes nearly 1 minute. This poses challenges to apply MIRA to tasks that require real-time visio-motor control. We believe observing the scene with multiple mounted cameras or learning prior over instant- NGP could drastically reduce the runtime. In the future, we plan to explore the usage of mental imagery for other robotics applications such navigation and mobile manipulation.
8

---

References
[1] R. N. Shepard and J. Metzler. Mental rotation of threc-dimensional objects. Science, 171(3972): 701-703, 1971.
[2] A. Richardson. Mental imagery. Springer, 2013.
[3] F. Dellaert and L. Yen-Chen. Neural volume rendering: Nerf and beyond. arXiv preprint arXiv:2101.05204, 2020.
[4] Xie, Takikawa, S. Saito, O. Litany, Yan, N. Khan, F. Tombari, Tompkin, V. Sitzmann, and S. Sridhar. Neural fields in visual computing and beyond. Computer Graphics Forum, 2022.
[5] A. Tewari, J. Thies, B. Mildenhall, P. Srinivasan, E. Tretschk, W. Yifan, C. Lassner, V. Sizzmann,
R. Martin-Brualla, S. Lombardi, et al. Advances in neural rendering. In Computer Graphics Forum, 2022.
[6] Mildenhall, P. Srinivasan, M. Tancik, J. T. Barron, Ramamoorthi, and R. Ng. Nerf: Repre- senting scenes neural radiance fields for view synthesis. In ECCV, 2020.
[7] A. Zeng. Learning visual affordances for robotic manipulation. PhD thesis, Princeton University, 2019.
[8] D. Batra, X. Chang, S. Chemova, A. J. Davison, Deng, V. Koltun, S. Levine, J. Malik, I. Mordatch, R. Mottaghi, al. Rearrangement: A challenge for embodied ai. arXiv preprint arX19:2011.01975. 2020.
[9] A. Zeng, P. Florence, J. Tompson, S. Welker, J. Chien, M. Attarian, T. Armstrong, I. Krasin, D. Duong, V. Sindhwani, and J. Lee. Transporter networks: Rearranging the visual world for robotic manipulation. CoRL, 2020.
[10] M. Shridhar, L. Manuelli, and D. Fox. Cliport: What and where pathways for robotic manipulation. In CoRL, 2021.
[11] L. Manuelli, W. Gao, Florence, and R. Tedrake. kpam: Keypoint affordances for category-level robotic manipulation. In ISRR, 2019.
[12] A. Simeonov, Y. Du, A. Tagliasacchi, J. B. Tenenbaum, A. Rodriguez, P. Agrawal, and V. Sitzmann. Neural descriptor fields: Se (3)-equivariant object representations for manipulation. In ICRA, 2022.
[13] S. Song, A. Zeng. J. Lee, and T. Funkhouser. Grasping in the wild: Learning 6dof closed-loop grasping from low-cost demonstrations. In IROS, 2020.
[14] K. Zakka, A. Zeng, J. Lee, and S. Song. Form2lit: Learning shape priors for generalizable assembly from disassembly. In ICRA, 2020.
[15] M. Zhu, K. G. Derpanis, Y. Yang, Brahmbhart, M. Zhang, C. Phillips, M. Lecce, and K. Daniilidis. Single image 3d object detection and pose estimation for grasping. In ICRA, 2014.
[16] A. Zeng, K.-T. Yu, S. Song, D. Suo, F. Walker, A. Rodriguez, and J. Xiao. Multi-view self-supervised deep learning for 6d pose estimation in the amazon picking challenge. In ICRA, 2017.
[17] Y. Xiang, T. Schmidt, V. Narayanan, and D. Fox. PoseCNN: A convolutional neural network for 6d object pose estimation in cluttered scenes. In RSS, 2018.
[18] X. Deng, Y. Xiang, A. Mousavian, C. Eppner, T. Bretl, and D. Fox. Self-supervised 6d object pose estimation for robot manipulation. In ICRA, 2020.
[19] H. Wang, S. Sridhar, J. Huang, J. Valentin, S. Song, and L. J. Guibas. Normalized object coordinate space for category-level 6d object pose and size estimation. In CVPR, 2019.

---

[20] X. Chen, Z. Dong, J. Song. A. Geiger, and O. Hilliges. Category level object pose estimation via ECCV, 2020.
neural analysis-yy-synthesi..
[21] X. Li, H. Wang. L. Yi, J. Guibas, A. L. Abbott, and S. Song. Category-level articulated object pose estimation. In CVPR, 2020.
[22] P. R. Florence, L. Manuelli, and R. Tedrake. Dense object nets: Learning dense visual object descriptors by and for robotic manipulation. In CoRL,, 2018.
[23] P. Florence, L. Manuelli, and R. Tedrake. Self-supervised correspondence in visuomotor policy learning. RA-L, 2019.
[24] P. Sundaresan, J. Gramnen, B. Thananjeyan, A. Balakrishna, M. Laskey, K. Stone, J. E. Gonzalez, and
K. Goldberg. Learning rope manipulation policies using dense object descriptors trained on synthetic depth data. In ICRA, 2020.
[25] T. D. Kulkarni, A. Gupta, C. Ionescu, S. Borgeaud, M. Reynolds, A. Zisserman, and V. Mnih. Unsupervised learning of object keypoints for perception and control. NeurlPS, 32, 2019.
[26] X. Liu, R. Jonschkowski, A. Angelova, and K. Konolige. Keypose: Multi-view 3d labeling and keypoint estimation for transparent objects. In CVPR, 2020.
[27] You, I.. Shao, Migimatsu, and J. Bohg. Omnihang: Learning to hang arbitrary objects using contact point correspondences and neural collision estimation. In 2021 IEEE International Conference Robotics and Automation (ICRA), pages 5921-5922 IEEE, 2021.
[28] S. Levine, C. Finn, T. Darrell, and P. Abbeel. End-to-end training of deep visuomotor policies. JMLR, 2016.
[29] D. Kalashnikov, A. Irpan, P. Pastor, J. Ibarz, A. Herzog, E. Jang, D. Quillen, E. Holly, M. Kalakrish- nan, V. Vanhoucke, and S. Levine. Scalable deep reinforcement learning for vision-basee robotic manipulation. In CoRI., 2018.
[30] J. Mahler, J. Liang. S. Niyaz, M. Laskey, R. Doan, X. Liu, J. A. Ojea, and K. Goldberg. Dex-net 2.0: Deep learning to plan robust grasps with synthetic point clouds and analytic grasp metrics. arXiv preprint / 2017.
[31] A. Zeng, S. Song, K.-T. Yu, E. Donlon, F. R. Hogan, M. Bauza, D. Ma, O. Taylor, M. Liu, E. Romo, et al. Robotic pick-and beck-and-place place of novel objects in clutter with domain image matching. In ICRA, 2018.
grasping and cross-
[32] A. ten Pas, M. Gualtieri, K. Saenko, and R. Platt. Grasp pose detection in point clouds. 1JRR, 2017.
[33] Mousavian, C. Eppner, and D. Fox. 6-dof graspnet: Variational grasp generation for object manipulation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019.
[34] J. Wu, X. Sun, A. Zeng, S. Song, J. Lee, S. Rusinkiewicz, and T. Funkhouser. Spatial action maps for mobile manipulation. arXiv preprint 2rXiv:2000.09111,,2000 2020.
[35] S. James, K. Wada, T. Laidlow, and A. J. Davison. Coarse-to-fine q-attention: Efficient learning for visual robotic manipulation via discretisation. In CVPR, 2022.
[36] H. Huang, D. Wang, R. Walter, and R. Platt. Equivariant transporter network. In RSS, 2022.
[37] D. Wang, R. Walters, and Platt. So (2) equivariant reinforcement learning. In ICLR, 2022.
[38] X. Zhu, D. Wang, O. Biza, G. Su, R. Walters, and R. Platt. Sample efficient grasp learning using equivariant models. In RSS, 2022.
[39] B. Curless and M. Levoy. A volumetric method for building complex models from range images. In Proceedings of the 23rd annual conference Computer graphics and interactive techniques, 1996.
10

---

[40] T. Karras, M. Aittala, S. Laine, E. Härkönen, J. Hellsten, J. Lehtinen, and T. Aila. Alias free generative adversarial networks. NeurlPS, 34, 2021.
[41] J. Park, P. Florence, J. Straub, R. Newcombe, and S. Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In CVPR, 2019.
[42] L. Mescheder, M. Oechsle, M. Niemeyer, S. Nowozin, and A. Geiger. Occupancy networks: Learning 3d reconstruction in function space. In CVPR, 2019.
[43] V. Sitzmann, M. Zollhöfer, and G. Wetzstein. Scene representation networks: Continuous 3d-structure- aware neural scene representations, In NeurlPS, 2019.
[44] V. Sitzmann, Rezchikov, B. Freeman, J. Tenenbaum, and F. Durand. Light field networks: Neural scene representations with single-evaluation rendering. NeurIPS, 34, 2021.
[45] R. Gao, Y.-Y. Chang, S. Mall, L. Fei-Fei, and J. Wu. Objectfolder: A dataset of objects with implicit visual, auditory, and tactile representations, In CoRL,, 2021.
[46] Sitzmann, J. Martel, A. Bergman, D. Lindell, and G. Wetzstein. Implicit neural representations with periodic activation functions. In NeurlPS, 2020.
[47] A. Luo, Y. Du, M. J. Tarr, J. B. Tenenbaum, A. Torralba, and C. Gan. Learning neural acoustic fields. arXiv preprint arXiv:2204006226 2022.
[48] L. Yen-Chen, P. Florence, Barron, A. Rodriguez, P. Isola, and T.-Y, Lin. iNeRF: Inverting neural radiance fields for pose estimation. In IROS, 2021.
[49] A. Moreau, N. Piasco, D. Tsishkou, B. Stanciulescu, and A. de La Fortelle. Lens: Localization enhanced by nerf synthesis. In Conference on Robot Learning, 2022.
[50] E. Sucar, S. Liu, Ortiz, and A. Davison. iMAP: Implicit mapping and positioning in real-time. In ICCV, 2021.
[51] Z. Zhu, S. Peng, V. Larsson, W. Xu, H. Bao, Z. Cui, M. R. Oswald, and M. Pollefeys. Nice-slam: Neural implicit scalable encoding for slam. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12786-122 12796, 2022.
[52] J. Ortiz, A. Clegg, J. Dong, E. Sucar, D. Novotny, M. Zollhoefer, and M. Mukadam, isdf: Real-time neural signed distance fields for robot perception. In RSS, 2022.
[53] M. Adamkiewiez, T. Chen, A. Caccavale, R. Gardner, P. Culbertson, Bohg, and M. Schwager. Vision-only robot navigation in neural radiance world. RA-L, 2022.
[54] Y. Li, S. Li, V. Sitzmann, P. Agrawal, and A. Torralba. 3d neural scene representations for visuomotor control. In CoRL, 2021.
[55] D. Driess. Z. Huang, Y. Li, R. Tedrake, and M. Toussaint. Learning multi-object dynamics with compositional neural radiance fields. arXiv preprint arXiv:2202. /1855, 2022.
[56] Y. Wi, P. Florence, A. Zeng, and N. Fazeli. Virdo: Visio-tactile implicit representations of deformable objects. In ICRA, 2022.
[57] Shen, Z. Jiang, Choy, L. J. Guibas, Savarese, A. Anandkumar, and Y. Zhu. Acid: Action- conditional implicit visual dynamics for deformable object manipulation. In RSS, 2022.
[58] D. Driess, Schubert, P. Florence, Y. Li, and M. Toussaint. Reinforcement learning with neural radiance fields. arXiv preprint arXiv:2206.01634 2022.
[59] Z. Jiang Y. Zhu, M. Svetlik, K. Fang, and Y. Zhu. Synergies between affordance and geometry: 6-dof grasp detection via implicit representations. In RSS, 2021.

---

[60] J. Ichnowski*, Y. Avigal*, J. Kerr, and K. Goldberg. Dex-NeRF: Using a neural radiance field to grasp transparent objects. CoRI,, 2020.
[61] L. Yen-Chen, P. Florence, J. T. Baron, T.-Y. Lin, A. Rodriguez, and P. Isola. NeRF-Supervision: Learning dense object descriptors from neural radiance fields. In ICRA, 2022.
[62] J. T. Kajiya and Herzen. Ray tracing volume densities. SIGGRAPH, 1984.
[63] N. Max. Optical models for direct volume rendering. IEEE TVCG, 1995.
[64] T. Müller, A. Evans, C. Schied, and A. Keller. Instant neural graphies primitives with multireoolution hash encoding. In SIGGRAPH, 2022.
[65] LeCun, Y. Bengio, et al. Convolutional networks for images, speech, and time series. The handbook of brain theory and neural networks, 3361(10):1995, 1995.
[66] J. Long, E. Shelhamer, and T. Darrel.. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3431-3440, 2015.
[67] D. Seita, P. Florence, J. Tompson, E. Coumans, V. Sindhwani, K. Goldberg, and A. Zeng. Learingg to rearrange deformable cables, fabrics, and bags with goal-conditioned transporter networks. In 202/ IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2021.
[68] E. Coumans and Y. Bai. Pybullet, python module for physics simulation for games, robotics and machine learning. 2016.
12


Reviews:
[{"review_id": "zzuUYwhzQfU", "reviewer": "Reviewer_jzYJ", "summary": "This paper presents MIRA, a supervised pick-and-place manipulation framework that uses a NeRF-based scene representation to efficiently learn 6-DoF policies. The framework first builds an NeRF of a tabletop scene with 30 images captured from various viewpoints. A number of sampled viewpoints are rendered to generate novel perspectives of the scene. Then, actions are predicted by selecting the best pixel location to pick and place (separately) over all the rendered perspectives. This approach is able to learn robust 6-DoF policies without any depth data, which makes it applicable to objects with reflective, glossy, and translucent surfaces that are typically hard to model with commodity depth sensors. Experimental results include comparisons against TransporterNets, Form2Fit, and ground-truth state MLPs. The results show that MIRA significantly outperforms these baselines, specifically in scenes with out-of-distribution object poses, all without using explicit depth information.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths** \n\n+ Overall, the idea of using NeRFs with action-centric perception 6-DoF manipulation is quite novel and an interesting direction for the field. Naive image-to-action agents often lack the 3D prior for efficiently learning 6-DoF actions. Even with sufficient data, image-based models struggle to be robust to scene changes. Further, NeRFs also circumvent common issues with depth-cameras like modeling specular, glossy, translucent surfaces, and small objects. While there are some limitations to MIRA's approach, the formulation here has a lot of potential to be extended to general problems in vision-based manipulation. It’s exciting to see how future works build on this approach. \n\n+ The experiments cover a good set of baselines for action-centric perception. These evaluations test settings with both in-distribution and out-of-distribution object poses. The results also include ablations that investigate specific contributions like orthographic projection and multi-view negative sampling for EBM training. The simulated experiments are set in an open-source simulator (Ravens), and should be easily reproducible, especially if the authors release their code and setup. The real-world results are quite compelling, particularly the results with shiny and translucent objects.  \n\n+ The writing and presentation of methods is mostly clear. Figure 1 provides a good overview of the framework. Section 3 does a good job of providing preliminaries on NeRFs and orthographic ray-casting. The supplementary video also provides a good overview of the approach and highlights its benefits. \n\n**Weaknesses** \n\n- One key limitation (acknowledged in the limitations section) is that MIRA requires 30 multi-view RGB images, which takes 1 minute to capture with a robot arm. Optimizing the NeRF requires another 10 seconds. While this might be feasible for simple pick-and-place tasks, it’s not easily applicable to dynamic tasks that require real-time visuomotor control. For instance, while opening a cabinet or pouring water into a mug, it’s hard to capture 30 images with the same manipulator that is interacting with the object. However, given the rapid-pace of advancements with neural radiance fields, it might be possible to build 3D scene representations with limited viewpoints like in pixelNeRF (Yu et al. 2020). Nonetheless, the contribution here to quasi-static 6-DoF pick-and-place tasks itself is sufficient and interesting to the robot-learning community.  \n\n- Section 3.2 and 3.3 could benefit from additional details on the approach. More specifically, are all the multi-view images batched and processed with the ConvNet in a single-pass? Isn’t this expensive? I also assume it’s one ConvNet with shared weights for all the images, or is each image processed with independent ConvNets and kept spatially consistent across scenes? What is the ratio of positive to negative samples for the EBM training? And how dense are these samples – can you use every pixel in the image as training samples, like in TransporterNets? \n\n- Similarly, Section 4.2 (or an appendix section) could provide additional details on the image-capture process. How are the 30 multi-view images captured for the real-robot experiments? Is this a predefined path executed with IK or a motion-planner? This information might be useful for practitioners to understand how to get a good coverage of the tabletop. Also, were all 121 virtual images crucial for precise 6-DoF actions? Would it be possible to use less virtual images? Perhaps this could be investigated with simulated experiments by varying the number of virtual images and studying how it affects performance in Ravens tasks. \n\nReferences: \n\nYu et al. 2020 – https://arxiv.org/abs/2012.02190\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "MIRA presents a novel and interesting approach for using NeRFs with action-centric perception for  pick-and-place manipulation in 6-DoF settings. The ideas and experiments from the paper are very relevant to the CoRL community. ", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "Mostly additional details regarding the approach. See the Weaknesses section above for details. ", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {}}, {"review_id": "dEMU-_cpwwo", "reviewer": "Reviewer_YteW", "summary": "This paper uses a state-of-the-art neural representation, i.e., neural radiance fields (nerfs), to plan robot manipulation actions. Using a set of 2D images, acquired from a diverse range of angles/poses, a 3D volume of the scene is optimized, from which an orthographic view is rendered and fed into a policy that predicts a pick-and-place action value. In essence, the diverse set of initially acquired views allows synthesizing new views that might result in selecting a better manipulation position. \n", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "strengths:\n- a good use-case of nerfs for robotic manipulation\n- a good range of evaluations, including real robot experiments\n\nweaknesses:\n- no methodological novelty\n- too many views initially required to build nerf\n- too long optimization/computation time prior to action prediction + very long time to gather the initial views -> infeasible for realistic manipulation tasks\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper investigates a straight-forward application of neural representations for robot manipulation. In particular, neural radiance fields (nerfs) are optimized from multiple views that allows constructing the 3D volume of the scene, which in turn enable better manipulation action planning. The evaluations and the experiments cover a good range and provides convincing evidence on the usability of the approach. The major drawbacks stand as the computation time and the initial time cost of acquiring 30 images from different views, both of which diminishes the value proposition of the work significantly.\n", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "- an in-depth analysis on the effect of number of initial images.\n- related work: Nerf for Robotics subsection needs to provide a context as to the difference of the proposed approach to the mentioned prior work.\n- related work: how does this work compare to [R1,2] \n- where do expert demonstrations come from?\n- evaluation of and discussion on how/whether the same approach is applicable to 2/3-finger based grasping/picking actions. \n\n[R1] D. Driess et al., \"Learning Multi-Object Dynamics with Compositional Neural Radiance Fields\", arXiv preprint, 2022.\n[R2] D. Driess et al., \"Reinforcement Learning with Neural Radiance Fields\", arXiv preprint, 2022."}}, {"review_id": "cdsvquTo27", "reviewer": "Reviewer_DpD8", "summary": "The method enables the robot to apply 6-DOF pick-up and place actions given RGB-only image of the environment with objects. Instead of directly learning 6 DOF pose, the system learns to predict the best-projected plane of the scene that enables an orthogonal approach of the end-effector to this projected plane for successful pick-up and place. NeRFs enable rendering of the scene with orthogonal projection from any arbitrary camera pose, given a number of scenes from different camera poses. NeRFs is used to generate a number of projections from sampled camera poses, and the problem is reduced to learning affordances and 3 DOF (x,y,yaw) parameters of pick-up and place actions.\n\nThe method is verified in the simulator in different environments, and in the real robot in an insertion task. The performance is shown to surpass the performance of a number of SOTA methods in the simulator. The method does not require depth information, yet achieves good performance in 6 DOF action execution.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The strenghts of the paper:\n- Instead of RGB-D data, the method works with RGB only data. This enables the system to work with reflective and this structured objects.\n- Object detection is not required. The affordances are learned in pixel level.\n- Orthogonal projection enables better affordance prediction compared to perspective projection.\n- The method works well in out-of-distribution validation cases compared to other methods.\n- The paper is very well written.\n- The method is verified with experiments that require generalization capabilities.\n\nThe weak points in the paper:\n- Many stated contributions can be attributed to the existing NeRF method. As such, the used scene representation created by NeRF is not novel. The orthogonal projection is not challenging with NeRFs. Assigning affordances to pixels of scenes is also not novel.\n- The paper combines the following: Generating a number of projected scenes using NeRF and assigning affordances to the pixels in these scenes. This approach is smart, but it corresponds to an engineering effort of combining existing SOTA methods, rather than proposing a highly original/novel method with significant impact to the field.\n- The method is compared with baselines such MLP, Form2Fit, Transporter-SE(2/3). This paper out-performed the other methods probably because of the underlying NeRFs method, however NeRF is not a contribution of this paper. In order to better identify/show the contribution, the paper could have been compared with other existing control methods that use NeRF such as [54] or [60].\n- There is no object detection step, yet, the method was not verified in cluttered settings with more complex actions (such as grasping with a gripper).\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper utilizes the NeRF in a smart way in order to create potential projections of an arbitrary environment and reduces 6DOF end-effector pick-up/place pose search into a search of projections + orthogonal approach with 3DOF for pick-up/place. Although this is clever and shown to be effective compared to the existing methods, this corresponds to a combination of existing methods rather than a proposal of an original method with high impact, theoretically. Practically, as the contribution is on the control part, the proposed system needed to be compared with other NeRF-based methods.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {"issues": "- Is is possible to compare with other NeRF based control methods? [54] makes 6 DOF control. Can [60] be extended to 6DOF control?\n- Is taking 30 snapshots from different camera poses feasible and scalable in the real world?\n- How well does the method perform in cluttered environments?"}}]

Author rebuttal:

Decision: Accept (poster)