GNFactor: Multi-Task Real Robot Learning with Generalizable Neural Feature Fields
Authors: Yanjie Ze
Ge Yan
Yueh-Hua Wu
Annabella Macaluso
Yuying Ge
Jianglong Ye
Nicklas Hansen
Li Erran Li
Xiaolong Wang
Venue: CoRL 2023 Oral (2023)
Keywords: Robotic Manipulation
Neural Radiance Field
Behavior Cloning

Abstract: It is a long-standing problem in robotics to develop agents capable of executing diverse manipulation tasks from visual observations in unstructured real-world environments.  To achieve this goal, the robot will need to have a comprehensive understanding of the 3D structure and semantics of the scene. In this work, we present $\textbf{GNFactor}$, a visual behavior cloning agent for multi-task robotic manipulation with $\textbf{G}$eneralizable $\textbf{N}$eural feature $\textbf{F}$ields. GNFactor jointly optimizes a neural radiance field (NeRF) as a reconstruction module and a Perceiver Transformer as a decision-making module, leveraging a shared deep 3D voxel representation. To incorporate semantics in 3D, the reconstruction module incorporates a vision-language foundation model (e.g., Stable Diffusion) to distill rich semantic information into the deep 3D voxel. We evaluate GNFactor on 3 real-robot tasks and perform detailed ablations on 10 RLBench tasks with a limited number of demonstrations. We observe a substantial improvement of GNFactor over current state-of-the-art methods in seen and unseen tasks, demonstrating the strong generalization ability of GNFactor. Project website: https://yanjieze.com/GNFactor/

GNFactor: Multi-Task Real Robot Learning with Generalizable Neural Feature Fields
Yanjie Ze!
Ge Yan
Yueh-Hua Wu
Annabella Macaluso
Yuying Ge
Jianglong Ye
Nicklas Hansen
Li Erran Li Xiaolong Wang²
Shanghai Jiao Tong University PCC San Diego
University of Hong Kong #AWS AI, Amazon
"Equal Contribution
yanjieze.com/GNFacoorr
Abstract: It is long-standing problem in robotics to develop agents capable of executing diverse manipulation tasks from visual observations in unstructured real-world environments. To achieve this goal, the robot needs to have compre- hensive understanding of the 3D structure and semantics of the scene. In this work, we present GNFactor, a visual behavior cloning agent for multi-task robotic ma- nipulation with Generalizable Neural feature Fields. GNFactor jointly optimizes generalizable neural field (GNF) reconstruction module and Perceiver Transformer decision- making module, leveraging shared deep 3D voxel representation. To incorporate semantics in 3D, the reconstruction module uti- lizes a vision- vision-language foundation model (e.g., Stable Diffusion) to distill rich semantic information into the deep 3D voxel. We evaluate GNFactor on 3 real robot tasks and perform detailed ablations on 10 RLBench tasks with a limited number of demonstrations. We observe substantial improvement of GNFactor over current satte-of-theearr methods seen and unseen tasks, demonstrating the strong generalization ability of GNFactor.
Keywords: Robotic Manipulation, Neural Radiance Field, Behavior Cloning
Introduction
One major goal of introducing learning into robotic manipulation is to enable the robot to effectively handle unseen objects and successfully tackle various tasks in new environments. In this paper, we focus on using imitation learning with few demonstrations for multi-task manipulation. Using imi- tation learning helps avoid complex reward design and training can be directly conducted the real robot without creating its digital twin in simulation [1, 2, 3, 4]. This enables policy Icarning di- verse tasks in complex environments, based on users' instructions (see Figure 1). However, working with a limited number of demonstrations presents great challenges in terms of generalization. Most of these challenges arise from the need to comprehend the 3D structure of the scene, understand the semantics and functionality of objects, and effectively follow task instructions based on visual cues. Therefore, comprehensive and informative visual representation of the robot's ss observations serves crucial foundation for generalization.
The development of visual representation for robot learning has mainly focused on learning within a 2D plane. Self-supervised objectives are leveraged to pre-train the representation from the 2D image observation [6. 7, 8] or jointly optimized with the policy gradients [9, 10, 11]. While these approaches improve sample efficiency and lead to more robust policies, they are mostly applied to relatively simple manipulation tasks. To tackle more complex tasks requiring geometric understand- ing (e.g., object shape and pose) and with occlusions, 3D visual representation learning has been recently adopted with robot learning [11, 12]. For example, Driess et al. [12] train the 3D scene representation by using NeRF and view synthesis to provide supervision. While it shows effective- ness over tasks requiring geometric reasoning such as hanging a cup, it only handles the simple
7th Conference Robot Learning (CoRL. 2023), Atlanta, USA.

---

"Tura the Foucee*
"Open the Top Microwave Door" "Place the Tea Pot on the Stove"
Left
Kindben'
Kttchen
Feature
Right
Figure 1: Left: Three camera views used in the real robot setup to reconstruct the feature field generated by Stable Diffusion [5]. We segment the foreground feature for better illustration. Right: Three language- conditioned real robot tasks across different kitchens.
scene structure with heavy masking in a single-task setting. More importantly, without semantic understanding of the scene, it would be very challenging for the robot to follow the user's language instructions.
In this paper, we introduce learning language-conditione policy using a novel representation leveraging both 3D and semantic information for multi-task manipulation. We train Generalizable Neural Feature Fields (GNF) which distills pre-trained semantic features from 2D foundation mod- els into the Neural Radiance Fields (NeRFs). We conduct policy learning upon this representation, leading to our model GNFactor. It is important to note that GNFactor learns an encoder to extract scene features in feed-forward manner, instead of performing per-scene optimization in NeRF. Given a single RGB-D image observation, our model encodes it into a 3D semantic volumetric fea- ture, which then processed by Perceiver Transformer [13] architecture for action prediction. To conduct multi-task learning, the Perceiver Transformer takes in language instructions get task embedding, and reason the relations between the language and visual semantics for manipulation.
There are two branches of training in our framework (see Figure 3): (i) GNF training. Given the collected demonstrations, we train the Generalizable Neural Feature Fields using view synthesis with volumetric rendering. Besides rendering the RGB pixels, we also render the features of the foundation models in 2D space. The GNF learns from both pixel and feature reconstruction at the same time. To provide supervision for feature reconstruction,
apply vision foundation model (e.g., pre-trained Stable Diffusion model [5]) extract the 2D feature from the input view as the ground truth. In this way, we can distill the semantic features into the 3D space in GNF. (ii) GNFactor joint training. Building on the 3D volumetric feature jointly optimized by the learning objectives of GNF, we conduct behavior cloning to train the whole model end-to-end.
For evaluation, we conduct real-robot experiments three distinct tasks across different kitchens (see Figure 1). We successfully train single policy that effectively addresses these tasks in different scenes, yielding significant improvements over the baseline method PerAct [3]. We also conduct comprehensive evaluations using 10 RLBench simulated tasks [14] and 6 designed general- ization tasks. We observe that GNFactor outperforms PerAct with an average improvement of 1.55x and 1.57x, consistent with the significant margin observed the real-robot experiments.
2 Related Work
Multi-Task Robotic Manipulation. Recent works in multi-task robotic manipulation have led to significant progress in the execution of complex tasks and the ability to generalize to new scenar- ios [15, 2, 1, 16, 17, 3, 18, 19J. Notable methods often involve the use of extensive interaction data to train multi-task models [2, 1, 16, 17]. For example, RT-1 [1] underscores the benefits of task- agnostic training, demonstrating superior performance in real-world robotic tasks across a variety of datasets. To reduce the need for extensive demonstrations, methods that utilize keyframes which encode the initiation of movement have proven to be effective [20, 21, 22, 23, 24]. PerAct [3] employs the Perceiver Transformer [13] to encode language goals and voxel observations and shows its effectiveness in real robot experiments. In this work, we utilize the same action prediction frame-

---

RealSense Camera (Right)
RealSense Camera (Teft)
Robotic
(a) RGB observations for 10 RLBench tasks.
RealSensel
Camera (ttont)
(b) Sampled vlews for GNF training simulation. Figure 2: Simulation environments and the real robot setup. We show the RGB observations for 10 RLBench tasks in Figure (a), sampled views GNF Figure (b), the real robot setup Figure (c).
Real rebut
work as PerAct while we focus on improving the generalization ability of this framework by learning a generalizable volumetric representation under limited data.
Learning (RL/IL). To improve manipulation policies by leveraging visual information, numerous studies have concentrated on enhancing 2D vi- sual representations [8, 7, 6, 25], while for addressing more complex tasks, the utilization of 3D representations becomes crucial. Ze et al. [1]] incorporates deep voxel-based 3D autoencoder in motor control, demonstrating improved sample efficiency compared to 2D representation learning methods. Driess et al. [12] proposes to first learn state representation by NeRF and then use the frozen state for downstream RL tasks. While this work shows the initial success of utilizing NeRF in RL, its applicability in real-world scenarios is constrained due various limitations: egg,, the requirement of object masks, the absence of robot arm, and the lack of scene structure. The work closest to ours is SNeRL [26], which also utilizes vision foundation model in NeRF. However, sim- ilar to NeRF-RL [12]. SNeRL masks the scene structure to ensure functionality and the requirement for object masks persists, posing challenges for its application in real robot scenarios. Our proposed GNFactor, instead, handles challenging muti-task real-world scenarios, demonstrating the potential for real robot applications.
3D Representations for Reinforcement/mmittae
Neural Radiance Fields (NeRFs). Neural fields have achieved great success in novel view synthesis and scene representation learning these years [27, 28, 29, 30, 31, 32), and recent works also start to incorporate neural fields into robotics [33, 34, 35, 12, 26]. NeRF [29] stands out for achieving pho- torealistic view synthesis by learning an implicit function of the scene, while it requires per-scene optimization and thus hard to generalize. Many following methods [36, 37, 38, 39, 40, 41, 42] propose more generalizable NeRFs. PixeINeRF [43] and CodeNeRF [37] encode 2D images as the input of NeRFs, while TransINR [36] leverages vision transformer to directly infer NeRF parame- ters. A line of recent works [44, 45, 46, 47, 48, 49] utilize pre-trained vision foundation models such as DINO [50] and CLIP [51] as supervision besides the RGB image, which thus enables the NeRF to learn generalizable features. In this work, we incorporate generalizable NeRF to reconstruct different views in RGB and embeddings from a pretrained Stable Diffusion model [5].
3 Method
In this section, we detail the proposed GNFactor, multi-task agent with 3D volumetric represen- tation for real-world robotic manipulation. GNFactor is composed of volumetric rendering module and a 3D policy module, sharing the same deep volumetric representation. The volumetric render- ing module learns a Generalizable Neural Feature Field (GNF), to reconstruct the RGB image from cameras and the embedding from a vision- language foundation model, e.g., Stable Diffusion [5]. The agnostii nature of the lasion- embedding enables the volumetric representation to learn generalizable features via neural rendering and thus helps the 3D policy module better handle multi-task robotic manipulation. The task description is encoded with CLIP [51] to obtain the task embedding T. An overview of GNFactor is shown in Figure 3.

---

RGB-D (Front View)
Pllace the Teapot Stove"
Tokenize
Language Encoder
Deep
Otrans
Patchily
Grot
Percelver
Goallide
Voxel Encoder
Copen
Renderer
Robot State
Diflusion Feature
Figure 3: Overview of GNFactor. GNFactor takes RGB-D image as input and encodes it using voxel encoder transform it into feature in deep 3D volume. This volume then shared by modules: vol- umetric rendering (Renderer)
robot action prediction (Perceiver). These two modules are jointly trained, which optimizes the shared features to not only reconstruct vision-language embeddings (Diffusion Feature) and other views (RGB), but also to estimate accurate Q-values (Quass. Que Qcalliee. Qopea).
3.1 Problem Definition
To effectively address complex real-world robotic problems, we structure the observation space as a 3D voxel space O E 11000x3 x3 as opposed to the commonly used 2D images [1, 2, 7, 8]. The 3D voxel observation originates from an RGB-D image captured by single front camera with known extrinsic and intrinsic parameters, ensuring our method's practical applicability in the real world. In addition the front camera view used for policy training, we also gather additional views for training the GNF. We collect only RGB images for these additional views instead of RGB-D images. In real-world scenarios, we use k 2, while in simulated environments, we set h: = 19.
The action the robot arm with gripper represented by translation Otrans E RR.3 rotation azot E
R (360/5)x3 gripper openness aopen E [0, 1], and collision avoidance acollision E [0, 1]. For the rotation Gro each rotation axis is discretized into instructs the motion planner regarding the necessity avoid collisions, which is crucial
bins. The collision avoidance parameter Geollision
tasks
encompasses both contact-based and non-contact-based motions.
Due to the inefficiency of continuous action prediction and the extensive data requirements that come with it, we reformulate the behavior cloning problem as a first extract keyframes from expert demonstrations using the following metric: frame in the tra- jectory is keyframe when joint velocities approach zero and the gripper's open state remains stant. The model then trained to predict the subsequent keyframe based current observations. This formulation effectively transforms the continuous control problem into discretized keyframe- prediction problem. delegating the internal procedures to the RRT-connect motion planner [53] in simulation and Linear motion planner in real-world xAmm7 robot.
problem [3, 52]. We
3.2 Learning Volumetric Representations with Generalizable Neural Feature Fields
In our initial step, we transform the RGB-D image into 1003 voxel. Then the 3D voxel encoder encodes this 3D voxel and outputs our volumetric representation e E F 10003 x128 To enhance the volumetric representation e with structural knowledge and language semanties, we learn General- izable Neural Feature Field (GNF) that takes the deep volume e as the scene representation and the model learned by reconstructing the additional views and the features predicted by 2D vision- language foundation model [5]. The entire neural rendering process is described as follows.
We denote Ux 128 as the sampled 3D feature for the 3D point x using the volumetric represen- tation v. Ux is formed with trilinear interpolation due to the discretized nature the volume v. Our GNF primarily consists of three functions: (i) one density function oxx (x)) 3-3+128 - R4, that maps the 3D point x and the 3D feature ex to the density о, (ii) one RGB function c(x, d, fx) R3+3+128 + 3 that maps the 3D point the view direction d, and the 3D feature Ux to color, and (iii) one vision-language embedding function f(x, d, Ux) 13+3++22 + 1g that maps the 3D

---

point x, the view direction d. and the 3D feature to the vision- language embedding. In Figure 3, the corresponding components of these three functions are illustrated. Given a pixel's camera ray r(t) = + td, which is defined by the camera origin o E R view direction and depth t with bounds [mn, 4r]. the estimated color and embedding of the ray can be calculated by:
C(r,u)= f"d T(t)a(((t) (x(o)c(((t)) d, )xtt))dt
Jr. F(r,u) = I"d
(1)
(x))) (x(
where T(t) = exp (- Sin (s))ss). The integral is approximated with numerical quadrature in the implementation. Our GNF is then optimized to reconstruct the RGB image and the vision-language embedding from multiple views and diverse scenes by minimizing the following loss:
Lreccm
Σ [I((r)) PER
C(r) + Area || [F()) F(
where C(r) is the ground truth color, F(r) is the ground truth vision-languad embedding generated by Stable Diffusion, the set of rays generated from camera poses, and Xteu is the weight for the embedding reconstruction loss. For efficiency,
sample bua rays given one target view, instead of reconstructing the entire image. To help the GNF training, we use coarse-to-fine hierarchical structure as the original NeRF [29] and apply depth-guided sampling [54] in the "fine" network.
3.3 Action Prediction with Volumetric Representations
The volumetric representation is optimized not only to achieve reconstruction of the GNF module, but also to predict the desired action for accomplishing manipulation tasks within the 3D policy. As such, we jointly train the representation v to satisfy the objectives of both the GNF and the 3D policy module. In this section, we elaborate the training objective and the architecture of the 3D policy.
We employ Perceiver Transformer [3] to handle the high-dimensional 3D volume, the robot's proprioception, and the language feature. We first condense the shared volumetric representation into a volume of size 20 x 128 using a 3D convolution layer with kernel size and stride of 5, followed by a ReLU function, and flatten the 3D volume into a sequence of small cubes size 8000 x 128. The robot's 's proprioception is projected into a -e-dimensional space and concatenated with the volume sequence for each cube, resulting in a sequence of size 8000 256. We then project the language token features from CLIP into the same dimensions (77 x 256) and concatenate these features with combination of the 3D volume, the robot's 's proprioception state, and the CLIP token embedding. The result is sequence with dimensions of 8077 256.
input, i.e., the
This sequence is combined with a learnable positional encoding and passed through the Perceiver Transformer, which outputs sequence of the same size. We remove the last 77 features for the ease of voxelization [3] and reshape the sequence back voxel of size 20 256. This voxel is then upscaled to 1003 x 128 with trilinear interpolation and referred to Upr- UpT is shared across three action prediction heads (Qupea- Quanss Qree. Qcollide in Figure 3) to determine the final robot actions at the same scale as the observation space. To retain the learned features from GNF training, we create a skip connection between our volumetric representation v and Upr. The combined volume feature (v, Upr) is used to predict a 3D Q-function Qurans for translation, as well as Q-functions for other robot operations like gripper openness (Qopen). rotation (Qoo), and collision avoidance (Qulliik). The Q-function here represents the action values of one timestep, differing from the traditional Q-function in RL that for multiple timesteps. For example, in each timestep, the 3D Quans value would be equal to 1 for the most possible next voxel and 0 for other voxels. The model then optimizes the cross-entropy loss like classifier,
Laction EEy. [log Virans Ey... [log Vrull ] Ey,, [log Vopen Ey,,as [log Veollide (3)
where V; = softmax @.) for Q: E [Qrans Qopen 2ma, Qcollide] and Y, E [Ymans Yoors Yopen Yooliae]]
is the ground truth one-hot encoding. The overall learning objective for GNFactor is as follows:
LONFactor Laction + Areoon
(4)
5

---

RLBench
Real Robot
Multi-Task (10)
Generalization
Multi-Task (6)
Generalization (6)
Success Rates
40
28.3
60
54.8
40
31.7
40
33.3
18.0
20.4
28.3
20
20
20 16.7
PerAct
GNFactor (ours)
Figure 4: Main experiment results. We present the average success rates in hoth the multi-task and gener- alization settings across RLBench tasks and real robot tasks. The enror bar represents one standard deviation. The number in the bracket denotes the number of tasks.
where Arecon is the weight for the reconstruction loss to balance the scale of different objectives. To train the GNFactor, we employ joint training approach which the GNF and 3D policy module are optimized jointly, without any pre-training. From our empirical observation, this approach allows for better fusion of information from the two modules when learning the shared features.
4 Experiments
In this section, conduct experiments answer the following questions: (i) Can GNFactor surpass the baseline model in simulated environments? (ii) Can GNFactor generalize to novel scenes in simulation? (iii) Does GNFactor learn a superior policy that handles real robot tasks in two different kitchens with noisy and limited real-world data? (iv) What are the crucial factors in GNFactor to ensure the functionality of the entire system? Our concluded results are given in Figure
4.1 Experiment Setup
For the sake of reproducibility and benchmarking, conduct primary experiments in RLBench simulated tasks. Furthermore, to show the potential of GNFactor in the real world, we design set of real robot experiments across two kitchens. We compare our GNFactor with the strong language- conditioned multi-task agent PerAct [3] in both simulation and the real world, emphasizing the universal functionality GNFactor. Both GNFactor and PerAct use the single RGB-D image from the front camera as input to construct the voxel grid. the multi-task simulation experiments, we also create stronger version of PerAct by adding more camera views input to fully cover the scene (visualized in Figure 10). Figure 2 shows simulation tasks and the real robot setup. We briefly describe the tasks and details are left in Appendix B and Appendix C.
Simulation. We select 10 challenging anguage-connitione manipulation tasks from the RLBench tasksuites [14]. Each task has least variations, totaling 166 variations. These variations encompass several types, such as variations in shape and color. Therefore, achieve high success rates with very limited demonstrations, the agent needs to learn generalizable knowledge about manipulation instead of merely overfitting to the given demonstrations. We use the RGB-D image of size 128 x 128 x 3 from the single front camera as the observation. To train the GNF, we also add additional 19 camera views to provide RGB images as supervision.
Real robot. We the xArm7 robot with a parallel gripper in real robot experiments. We set up two toy kitchen environments to make the agent generalize manipulation skills across the scenes and designed three manipulation tasks, including open the microwave door, turn the faucet, and relocate the teapot, as shown in Figure 1. We set up three RealSense cameras around the robot. Among the three cameras, the front one captures the RGB-D observations for the policy training and the left/right one provides the RGB supervision for the GNF training.
Expert Demonstrations. We collect 20 demonstrations for each RLBench task with the motion planner. The task variation is uniformly sampled. We collect 5 demonstrations for each real robot task using a VR controller. Details for collection remain in Appendix D.
Generalization tasks. To further show the generalization ability of GNFactor, design additional 6 simulated tasks and 3 real robot tasks based on the original training tasks and add task distractors. Training details. One agent trained with two NVIDIA RTX3090 GPU for 2 days (100k iterations) with batch size of 2. The shared voxel encoder of GNFactor is implemented as lightweight 3D UNet with only 0.3M parameters. The Perceiver Transformer keeps the same number of parameters PerAct [3] (25.0 2M parameters), making comparison with PerAct fair.

---

Table I: Multi-task test results on RLBench. We evaluate 25 episodes for each checkpoint on 10 tasks across seeds and report the success rates (%) of the final checkpoints. Our method outperforms the most competitive baseline PerAct [3] with an average improvement of 1.55x and still largely surpasses PerAct with 4 cameras as input. The additional camera views are visualized Figure
Method/ / Task
close jar
open drawer sweep dustpan
off grill
tap
Average
PerAct
18.7+82
51.7=186
0.0ka0
40.0e17.0
38.7
PerAct (4 Cameras)
21.3+74
41.0113
0.0+00
65
46.7+3.5
GNFactorr
25.3:6.8
76.0
28 0=15.0
37.3
50.7+8.2
Method/
block
put in drawer
drag stick
push buttons
PerAct
18.7 13.7
2.7
5.3150
18.71124
6..
20.4
PerAct Cameras)
16.0+14.2
6.7.0..8
12.0=a.3
9.3+1.9
5.3
22.7
GNFactor
20.0_15.0
0.0190
37 17.3
18.7
4.0133
31.7
Table 2: Generalization to unseen tasks RLBench. We evaluate 20 episodes for each task with the final checkpoint across seeds. We denote "L." a larger object, "S" as smaller object, "N" as new position, and "D" as adding a distractor. Our method outperforms PerAct with an average improvement 1.f.1.5. 1.57x.
Method / Task
drag (D)
slide (L)
slide (8)
open (n)
turn (1)) push (D)
Average
PerAct GNFactor
6.644.7
33.3:1..7
5.0=4.1
25.0=10.8
18.316.2
20.0
18.0
46.713306
25.014.11
6.71662
31.716.2
28.3:24
31.712.
28.3
4.2 Simulation Results
We report the success rates for multi-task tests on RLBench in Table and for generalization to new environments in Table 2. We conclude our observations as follows:
Dominance of GNFactor over PerAct for multi-task learning. As shown by Table and Fig- ure 4, GNFactor achieves higher success rates across various tasks compared to PerAct, particularly excelling in challenging long-horizon tasks. For example, in sweep to dustpan task, the robot needs to first pick up the broom and use the broom to sweep the dust into the dustpan. We find that GNFactor achieves success rate of 28.0%, while PerAct could not succeed at all. In simpler tasks like open drawer where the robot only pulls the drawer out, both GNFactor and PerAct perform reasonably well, with success rates of 76.0% and 54.7% respectively. Furthermore, observe that enhancing PerAct with extra camera views does not result in significant improvements. This underscores the importance of efficiently utilizing the available camera views.
Generalization ability GNFactor new tasks. In Table 2, we observe that the change made on the environments such as distractors impacts all the agents negatively, while GNFactor shows better capability of generalization on out of tasks compared to PerAct. We also find that for some challenging variations such as the smaller block in the task slide (S), both GNFactor and PerAct struggle to handle. This further emphasizes the importance of robust generalization skills.
Ablations. We summarize the key components in GNFactor that contribute to the success of the volumetric representation in Table 4. From the ablation study, we gained several insights:
() Our GNF reconstruction module plays a crucial role in multi- task robot learning. Moreover, the RGB loss is essential for learning consistent 3D feature in addition to the feature loss, especially since the features derived from foundation models are not inherently 3D consistent.
Table 4: Ablations. We report the averaged success rates 10 RI- Bench tasks. "DGS" is short for depth sampling. "-x" means replacing.
Ablation
Success Rate (%)
(ii) The volumetric representation benefits from Diffusion fea- tures and depth-guided sampling, where the depth prior is uti- lized to enhance the sampling quality in neural rendering. An intuitive explanation is that GNF, when combined with DGS, becomes more adept at learning depth and 3D structure infor- mation. This enhanced understanding allows the 3D representa- tion to better concentrate on the surfaces of objects rather than the entire volume. Moreover, replacing Stable Diffusion with DINO [50] or CLIP [51] would not result in similar improvements easily, indicating the importance of our vision-language feature.
GNFactor w/o. GNF objective w/o, RGB objective
24.2
27.
w/o. Diflusion:
30.0
Diffusion + DINO Diffusion CLIP w/o. DGS w/o, skip connection
30.4
29.2
27.6
33.2
Apa 0.01 - 1.0
35.2
Ausost

---

Table Multi-task test results on real robot. We evaluate 10 episodes for each task and report the resulting success rate (%). We denote "door" as "open door", "faucet" as "turn faucet", and "teapot" as "relocate teapot". The number in the parenthesis suggests the kitchen ID and "d" suggests testing with distractors.
Method Taask door (1)
faucet (1) teapot (1) door (1,4) faucet (1,4) teapot (1,4)
Average
PerAct
80
0
10
50
GNFactor
10
80
10
30
50
30
Method door
faucet (2) teapot (2) door (2,4) foucat (2,d) teapot (2,d)
PerAct
10
50
0
10
30
0
22.5
GNFactor
70
40
20
40
30
43.3
(iii) While the use of skip connection is not a new story and we merely followed the structure of PerAct, the result of removing the skip connection suggests that our voxel representation, which distills features from the foundation model, plays critical role in predicting the final action.
(iv) Striking careful balance between the neural rendering loss and the action prediction loss is critical for optimal performance and utilizing information from multiple views by our GNF module proves to be beneficial for the single-view decision module.
Furthermore, provide the view synthesis in the real world, generated by GNFactor in Figure 5 and Figure 6. We also give the quantitative evaluation measured by PSNR [29]. We observe that the rendered views are somewhat blurred since the volumetric presentation learned by GNFactor is optimized to minimize both the neural rendering loss as well as the action prediction loss, and the rendering quality is largely improved when the behavior cloning loss is removed and only the GNF is trained. Notably, for the view synthesis in the real world, we do not have access to ground-truth point clouds for either training or testing. Instead, the point clouds are sourced from RealSense cameras and are therefore imperfect. Despite the limitations in achieving accurate pixel-level reconstruction results, we focus on learning semantic understanding of the whole scene from distilling Diffusion features, which is more important for policy learning.
4.3 Real Robot Experiments We summarize the results of our real robot experiment in Table 3. From the experiments, GNFactor outperforms the PerAct baseline on almost all tasks. Notably, in the teapor task where the agent is required to accurately determine the grasp location and handle the teapot from a correct angle, PerAct fails to accomplish the task and obtains zero success rate across two kitchens. We observed that is indeed challenging to learn a delicate policy from only demonstrations. However, by incorporating the representation from the embedding of vision-language model, GNFactor gains an understanding of objects. As such, GNFactor does not simply overfit the given demonstrations.
The second kitchen (Figure 1) presents more challenges due to its smaller size compared to the first kitchen. This requires higher accuracy to manipulate the objects effectively. The performance gap between GNFactor and the baseline PerAct becomes more significant the second kitchen. Importantly, our method does not suffer the same performance drop transitioning from the first kitchen to the second, unlike the baseline.
We also visualize our 3D policy module by Grad-CAM [55], as shown in Figure 7. We use the gradients and the 3D feature map from the 3D convolution layer after the Perceiver Transformer to compute Grad-CAM We observe that the target objects are clearly attended by our policy, though the training signal only the Q-value for a single voxel.
Conclusion and Limitations
In this work, we propose GNFactor, a visual behavior cloning agent for real-world multi-task robotic manipulation. GNFactor utilizes a Generalizable Neural Feature Field (GNF) to learn a 3D volu- metric representation, which also used by the action prediction module. We employ the vision- language feature from the foundation model Stable Diffusion besides the RGB feature to supervise the GNF training and observe that the volumetric representation enhanced by the GNF is helpful for decision-making. GNFactor achieves strong results in both simulation and the real world, across 10 RLBench tasks and 3 real robot tasks, showcasing the potential of GNFactor in real-world scenarios.
One major limitation of GNFactor the requirement of multiple views for the GNF training, which can be challenging scale up the real world. Currently, use three fixed cameras for GNFactor, but it would be interesting to explore using cell phone to randomly collect camera views, where the estimation of the camera poses would be a challenge.
8

---

Acknowledgment. This work was supported, in part, by the Amazon Research Award, Cisco Fac- ulty Award and gifts from Qualcomm.
References
[1] A. Brohan, N. Brown, Carbajal, Y. Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Haus- man, A. Herzog, J. Hsu, et al. Rt-1: Roboties transformer for real-world control scale. arXiv, 2022.
[2] E. Jang, A. Irpan, M. Khansari, D. Kappler, F. Ebert, C. Lynch, S. Levine, and C. Finn. Be-z: Zero-shot task generalization with robotic imitation learning. In CoRL, 2022.
[3] M. Shridhar, L. Manuelli, and D. Fox. Perceiver-actor: A multi-task transformer for robotic manipulation. In CoRL, 2023.
[4] M. Dalal, A. Mandlekar, C. Garrett, A. Handa, R. Salakhutdinov, and Fox. Imitating task and motion planning with visuomotor transformers. arXiv, 2023.
[5] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-resolution image syn- thesis with latent diffusion models. In CVPR, 2022.
[6] S. Parisi, A. Rajeswaran, S. Purushwalkam, and A. Gupta. The unsurprising effectiveness of pre-trained vision models for control. In ICML, 2022.
[7] Nair, Rajeswaran, V. Kumar, C. Finn, and Gupta. R3m: universal visual represen- tation for robot manipulation. arXiv, 2022.
[8] I. Radosavovic, T. Xiao, S. James, P. Abbeel, J. Malik, and T. Darrell. Real-world robot learning with masked visual pre-training. In CoRL, 2023.
[9] M. Laskin, A. Srinivas, and P. Abbeel. Curl: Contrastive unsupervised representations for reinforcement learning. In ICML,, 2020.
[10] N. Hansen and X. Wang. Generalization in reinforcement learning by soft data augmentation. In ICRA, 2021.
[11] Y. Ze, N. Hansen, Chen, M. Jain, and X. Wang. Visual reinforcement learning with self- supervised 3d representations. RA-L, 2023.
[12] D. Driess, I. Schubert, P. Florence, Y. Li, and M. Toussaint. Reinforcement learning with neural radiance fields. NeurIPS, 2022.
[13] A. Jaegle, E. Gimeno, A. Brock, O. Vinyals, A. Zisserman, and J. Carreira. Perceiver: General perception with iterative attention. In ICML, 2021.
[14] S. James, Z. Ma. Arrojo, and A. Davison. Rlbench: The robot learning benchmark & learning environment. RA-L, 2020.
[15] R. Rahmatizadeh, P. Abolghasemi, L. Bölöni, and S. Levine. Vision-based multi-task manip- ulation for inexpensive robots using end-to-end learning from demonstration. In 2018 IEEE international conference on robotics and automation (ICRA), pages 3758-3765. IRRE, 2018.
[16] M. Shridhar, L. Manuelli, and D. Fox. Cliport: What and where pathways for robotic manipu- lation. In CoRL, 2022.
[17] D. Kalashnikov, A. Irpan, P. Pastor, J. Ibarz, A. Herzog, Jang, D. Quillen, Holly, M. Kalakrishnan, V. Vanhoucke, et al. Qt-opt: Scalable deep reinforcement learning for vision- based robotic manipulation. arXiv, 2018

---

[18] T. Yu, D. Quillen, Z. He, R. Julian, K. Hausman, C. Finn, and S. Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on robot learning, pages 1090-1100. PMLR, 2020.
[19] R. Yang, H. Xu, Y. Wu, and X. Wang. Multi-task reinforcement learning with soft modular- ization. Advances in Neural Information Processing Systems, 33:4767-4777, 2020.
[20] S. Song, A. Zeng, J. Lee, and T. Funkhouser. Grasping in the wild: Learning 6dof closed- loop grasping from low-cost demonstrations. IEEE Robotics and Automation Letters, 5(3): 4978 4985, 2020.
[21] A. Murali, A. Mousavian, C. Eppner, C. Paxton, and D. Fox. 6-dof grasping for target-driven object manipulation in clutter. In 2020 IEEE International Conference on Robotics and Au- tomation (ICRA), pages 6232-6238. IEEE, 2020.
[22] A. Mousavian, Eppner, and D. Fox. 6-dof graspnet: Variational grasp generation for object manipulation. In ICCV, 2019.
[23] Z. Xu, Z. He, and S. Song. Universal manipulation policy network for articulated objects. RA-L, 2022.
[24] Y. Li, S. Agrawal, J.-S. Liu, S. K. Feiner, and S. Song. Scene editing as teleoperation: A case study in 6dof kit assembly. In IROS, 2022.
[25] N. Hansen, Z. Yuan, Y. Ze, T. Mu, A. Rajeswaran, H. Su, H. Xu, and X. Wang. On pre-training for visuo-motor control: Revisiting leaming-from-ssratch baseline. In ICML, 2022.
[26] D. Shim, S. Lee, and J. Kim. Snerl: Semantic-aware neural radiance fields for reinforcement learning. ICMI., 2023.
[27] Y. Chen, S. Liu, and Wang. Learning continuous image representation with local implicit image function. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8628-8638, 2021.
[28] L. Mescheder, M. Occhsle, M. Niemeyer, S. Nowozin, and A. Geiger. Occupancy networks: Learning reconstruction function space. Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 4460 4470, 2019.
[29] B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1):99-106, 2021.
[30] M. Niemeyer, L. Mescheder, M. Oechsle, and A. Geiger. Occupancy flow: 4d reconstruction by learning particle dynamics. In Proceedings of the IEEE/CVF international conference on computer vision, pages 5379-5389, 2019.
[31] J. Park, P. Florence, J. Straub, R. Newcombe, and S. Lovegrove. Deepsdf: Learning con- tinuous signed distance functions for shape representation. In Proceedings of the IЕEE/CVF conference on computer vision and pattern recognition, pages 165-174, 2019.
[32] V. Sitzmann, M. Zollhöfer, and G. Wetzstein. Scene representation networks: Continuous 3d-structure-awer aware neural scene representations. Advances in Neural Information Processing Systems, 32, 2019.
[33] Z. Jiang, Y. Zhu, M. Svetlik, K. Fang, and Y. Zhu. Synergies between affordance and geometry: 6-dof grasp detection via implicit representations. arXiv preprint arXiv:2 2021. [34] Y.-C Lin, Florence, A. Zeng. T. Barron, Y. Du, W.-. Ma, A. Simeonov, A. Garcia, and P. Isola. Mira: Mental imagery for robotic affordances. In Conference on Robot Learning, pages 1916-1927. PMLR, 2023.
10

---

[35] Y. Li, S. Li, V. Sitzmann, P. Agrawal, and A. Torralba. 3d neural scene representations for visuomotor control. In Conference on Robot Learning, pages 112-123. PMLR, 2022.
[36] Y. Chen and X. Wang. Transformers as meta-learners for implicit neural representations. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XVII, pages 170-187. Springer, 2022.
[37] W. Jang and L. Agapito. Codenerf: Disentangled neural radiance fields for object categories. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pages 12949 12958, 2021.
[38] K.-E. Lin, Lin, W.-S. Lai, T.-Y. Lin, Y.-C. Shih, and R. Ramamoorthi. Vision transformer for nerf-based view synthesis from single input image. In Proceedings of the IEEE/CVE Winter Conference on Applications of Computer Vision, pages 806-815, 2023.
[39] J. Reizenstein, R. Shapovalov, P. Henzler, L. Sbordone, P. Labatut, and D. Novotny. Common objects in 3d: Large-scale learning and evaluation of real-life 3d category reconstruction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10901- 10911, 2021.
[40] K. Rematas, R. Martin-Brualla, and V. Ferrari. Sharf: Shape-conditioned radiance fields from a single view. arXiv preprint arXiv:2102.08866 2021.
[41] A. Trevithick and B. Yang. Grf: Learning a general radiance field for 3d representation and rendering. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pages 15182-15192, 2021.
[42] Q Wang,
Wang, K. Genova, P. Srinivasan, H. Zhou, J. Barron, Martin-Brualla, N. Snavely, and T. Funkhouser. Ibrnet: Learning multi-view image- based rendering. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1690-4699, 2021.
[43] A. Yu, V. Ye, M. Tancik, and A. Kanazawa. pixelnerf: Neural radiance fields from one or few images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4578-4587, 2021.
[44] V. Tschernezki, I. Laina, D. Larlus, and A. Vedaldi. Neural feature fusion fields: 3d distillation of self-supervised 2d image representations. arXiv, 2022.
[45] S. Kobayashi, E. Matsumoto, and V. Sitzmann. Decomposing nerf for editing via feature field distillation. NeurIPS, 2022.
[46] Ye, N. Wang, and X. Wang. Featurenerf: Learning generalizable nerfs by distilling founda- tion models. arXiv, 2023.
[47] Kerr, C. M. Kim, K. Goldberg, A. Kanazawa, and M. Tancik. Lerf: Language embedded radiance fields. arXiv preprint arXiv:2303.099 095555 2023.
[48] K. M. Jatavallabhula, A. Kuwajerwala, Q. Gu, M. Omama, Chen, S. Li, G. Iyer, S. Saryazdi, N. Keetha, Tewari, al. Conceptfusion: Open-se multimodal 3d mapping. arXiv preprint arXiv:2302.07221, 2023.
[49] K. Blomqvist, F. Milano, J. Chung, L. Ott, and R. Siegwart. Neural implicit vision-language feature fields. arXiv preprint arXiv:2303. 10962, 2023.
[50] M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin. Emerging properties in self-supervised vision transformers. In ICCV, 2021.

---

[51] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervi- sion. In ICML, 2021.
[52] James, K. Wada, T. Laidlow, and J. Davison. Coarse-to-fine q-attention: Efficient learning for visual robotic manipulation via discretisation. In CVPR, 2022.
[53] S. Klemm, Oberländer, A. Hermann, A. Roennau, T. Schamm, M. Zollner, and R. Dill- mann. Rrte-connect: Faster, asymptotically optimal motion planning. In 2015 IEEE interna- tional conference robotics and biomimetics (ROBIO), pages 1670-1677. IEEE, 2015.
[54] H. Lin, Peng, Z. Xu, Y. Yan, Q. Shuai, H. Bao, and X. Zhou. Efficient neural radiance fields for interactive free-viewpoint video. In SIGGRAPH Asia, 2022.
[55] R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE international conference on computer vision, pages 6 618-626, 2017.
[56] Y. You, J. Li, S. Reddi, J. Hseu, S. Kumar, S. Bhojanapalli, X. Song, J. Demmel, K. Keutzer, and C.-J. Hsieh. Large batch optimization for deep learning: Training bert in 76 minutes. arXiv, 2019.
12

---

A
Visualizations
With Action Loss
Without Action Loss
RGB
Feature
Laft
PSNR-13.88
PSNR-22.65
Front
PSNR-15.01
2NNR-21.2
Right
PSNR-14.67
PSNR 19. 96
Figure 5: View synthesis of GNFactor in the real world. PSNR computed for quantitative evaluation.
The visualization with the action loss is relatively blurred compared to that without the action loss. The noisy rendering is mainly because, in inference, do optimize per-step for rendering but just perform one feedforward obtain the feature.
RGB
Feature
Figure 6: More novel view synthesis results. Both RGB and features are synthesized. We remove the action loss here for hetter rendering quality. Videos available vanjieze.com//NNPacoor
RGB-D Image
Grad-CAM
RGB-D Image
Grad-CAM
Figure 7: Visualize the 3D policy module by Grad-CAM [55]. Though the supervision signal is only the
Q-value for single voxel during the training process, observe in visualizations that the target objects clearly attended by our policy. Videos are available on yanjiize.com//GFacce
13

---

B Task Descriptions
Simulated tasks. select 10
tasks from RLBench [14], all of which involve at least variations. See Table 5 for an overview. Our task variations include randomly sampled colors, sizes, counts, placements, and categories of objects, totaling 166 different variations. The set of colors have 20 instances: red, maroon, lime, green, blue, navy, yellow, cyan, magenta, silver, gray, orange, olive, purple, teal, azure, violet, rose, black, and white. The set of sizes includes 2 types: short and tall. The set of counts has 3 instances: 2, 3. The placements and object categories specific to each task. For example, open drawer has 3 placement locations: top, middle and bottom. In addition to these semantic variations, objects are placed on the tabletop at random poses within a limited range.
Table Language-conddttooed tasks in RLBench [14].
Task
Variation Type
of Variations
Avg. Keyframs
Languuge Template
close jar
color
20
"close jar
open drawer
placement
3
3,0
"open the druwer
sveep to dustpan
size
4.6
"sweep dirt to the dustpan"
off grill
category
2
5.0
"take the off the grill"
turn tap
placement
2
2.0
"turm - tap"
slide
"slide block to target"
put in drawer
placement
3
12.0
"put the item the drawer"
drag stick
color
20
6.0
"use the stick to drag the cuhe onto the target"
push
color
50
"push the button, [then button]"
stack blocks
color, count
60
14.6
"stack blocks"
Generalization tasks in simulation. We design additional tasks where the scene is changed based on the original training environment, to test the generalization ability of GNFactor. Table gives an overview of these tasks. Videos are also available on yanjieze.com/GNNacooo.
Table 6: Generalization tasks based on RLBench.
Task
Base
Change
drag (D)
drag stick
add two colorful buttons on the table change the block size to a larger one change the block size to a smaller one change the position of the drawer change the position of the tap add two colorful jar on the table
slide (L) slide block
slide (S) slide block
open (n) open drawer
turn (N) turn tap
push (D) push buttons
Real robot tasks. In the experiments, we perform three tasks along with three additional tasks where distracting objects are present. The door task requires the agent to open the door on an mircowave, a task which poses challenges due to the precise coordination required. The faucet task requires the agent to rotate the faucet back to center position, which involves intricate motor control. Lastly, the teapor task requires the agent to locate the randomly placed teapot in the kitchen and move on top of the stove with the correct pose. Among the three, the teapot task is considered the most challenging due to the random placement and the need for accurate location and rotation of the gripper. All 6 tasks are set up in two different kitchens, as visualized in Figure 8. The keyframes used in real robot tasks are given in Figure 9.
C Implementation Details
Voxel encoder. We use a lightweight 3D UNet (only 0.3M parameters) encode the input voxel 1003 x 10 (RGB features, coordinates, indices, and occupancy) into deep 3D volumetric rep- resentation of size 1003 128. Due to the cluttered output from directly printing the network,
14

---

(a) Kitchen
(b) Kitchen
Figure 8: Kitchens. We give closer view of our two kitchens for real robot experiments. The
figures are captured in almost the same position to display the size difference between the two.
time
Turn Faucet
Open Oven
Relocate Teapot
Figure 9: Keyframes for real robot tasks. We give the keyframes used in our 3 real robot tasks
across 2 kitchens.
15

---

provide the PyTorch-Style pseudo-code for the forward process as follows. For each block, we use a cascading of one Convolutional Layer, one BatchNorm Layer, and one LeakyRel.U layer, which is common practice in the vision community.
def forward x)
conv0 self convo # 100"3x8
conv2 self. conv2 www.eeeee....ceeeeeoeeaooaaaeaca
convi (convO)) 50"3x16
conv4 self. cow.e(sse..coon3(( conv3(conv2)) 25"3x32
self conv6(sell conv((eel..coovv((con)))
13"3x64
conv4 self self.conv7(x) 25"3x32
conv2 self celf.conv9(:: (x) 50'3x16
x self.conv_one conv_out (convO self. (x))1(x)) 100"3x128
return
Generalizable Neural Field (GNF), The overall network architecture of our GNF is close to the original NeRF [29] implementation. We use the same positional encoding as NeRF and the encoding function is formally
r(p) = (sin (2 mp) cos (2mmp) ... sin (24-1
S TD) cos (24-1mp) mp)
(5)
This function applied to each of the three coordinate values and we set L = 6 in our experiments. The overall position encoding is then 36-dimensional. The input of GNF thus a concatenation of the original coordinates (RR)), the position encoding (R.3), the view directions (TR3), and the voxel feature 12228), totaling 170 dimensions. Our GNF mainly consists of ReanetFCBlocks, in which skip connection is used. The input feature is first projected to 512 with linear layer and fed into these blocks, and then projected to the output dimension 516 (RGB, density, and Diffusion feature) with cascading of one ReLU function and one linear layer. We provide the PyTorch-Style pseudo-code for the networks as follows. GNF
Linear (in__iiinffe (it_i/a_eeeee1100..111 out fettureauues-,1,, bias-True),
(0-4) R
(fc_0): Linear(in (in_ (in_features=512, out out_featuree=612, bias=True)
(fc_1); Linear (in (in_ffeeee fea_tueuree=51,, out 0 bias=True) (activation): ReLU()
)
ReLU(), (ineaarin_feturre--55 out out_ fut_features=516, bias=True)
Percevier Transformer. Our usage of Percevier Transformer is close to PerAct [3]. We use attention blocks to process the sequence from multi-modalities (3D volume, language token, and robot proprioception) and output a sequence also. The usage of Perceiver Transformer enables us to process the long sequence with computational efficiency, by only utilizing small of latents to attend the input. The output sequence is then reshaped back to voxel to predict the robot action. The Q-function for translation is predicted by 3D convolutional layer, and for the prediction of openness, collision avoidance, and rotation, we use global max pooling and spatial softmax operation to aggregate 3D volume features and project the resulting feature to the output dimension with a multi-layer perception. We could clarify that the design for the policy module is our contribution; for more details please refer PerAct [3] and its official implementation on https:/ thub. httpp:///ggiabb....//raaaa//g//p wwt./ppaaa/pppaac.c.tt...ac
D Demonstration Collection for Real Robot Tasks
For the collection of real robot demonstrations, we utilize the HTC VIVE controller and bases- tation track the 6-DOF poses of human hand movements. We then use triad-openvr package ltttps://github com ctt.tuu...ooorrriiettttt.t TrriadSmiitrrad.c...
employ SteamVR and accurately map human operations onto the xAmn robot, enabling to interact with objects in the real kitchen.
16

---

We record the real-time pose of xAmn and 640 x 480 RGB-D observations with the pyrealsense2 https////ppp..ooppoojeeeeee httt//yyrraaseeeee1) Though the image size is different from our simulation setup. we use the same shape of the input voxel, thus ensuring the same algorithm is used across the simulation and real world. The downscaled images (80 60) are used for neural rendering.
E
Detailed Data
Besides reporting the final success rates in our main paper, we give the success rates for the best sin- gle checkpoint (i.e., evaluating all saved checkpoints and selecting the one with the highest success rates), shown in Table 7. Under this setting GNFactor outperforms PerAct with larger margin. However, do not use the best checkpoint in the main results for fairesss.
We also give the detailed number of success in Table 8 for reference in addition to the success rates computed in Table 2.
Table 7: Multi-task test results RLBench. We report the success rates for the best single checkpoint for reference. We could observe GNFactor surpasses PerAct by large margin.
Method/ / Task
jar
open
aveep dustpan
grill
tap
Average
PerAct GNFactor
22.725.
62.72132
0.0
46.7
36.029. 45.3:3
40.0
77.3
40.0
66.7282
Method / Task
slide block
put in drawer
drag stick
push buttons
stack blocks
PerAct
22
9.3s5.0
12 0s6.s
18.7
5.3+L0
23.6
GNFactor
18.7:10.5
10.71124
73.3113
20.0ga.
8.0g0.0
40.0
Table 8: Detailed data for generalization to novel tasks. We evaluate 20 episodes, each across 3 seeds, for the final checkpoint and report the number of successful trajectories here.
Generalization
PerAct
GNFactor w/o. Diffusion
GNFactor
drag (D)
2, 0, 2
15, 2,
18, 5, 5
slide (L)
6, 6,6,8
1,10,10 10
6, 5,5,4 4
slide (S)
0,2,1
6,1,5 6,3,3
0,3, 1
push (D)
4,4,5
7,6,6
open (N)
6,2,7
5,2,9
8,5,6
turn (N) 4,5,2
2,7,2
6, 6,6,5
F
Stronger Baseline
To make the comparison between our GNFactor and PerAct fairer, we enhance Peract's input by using 4 camera views, as visualized in Figure 10. These views ensure that the scene is fully covered. is observed our experiment results (Table 1) that GNFactor which takes the single view as input still outperforms PerAct with more views.
G Hyperparameters
We give the hyperparameters used in GNFactor as shown Table 9. For the GNF training, we use a ray batch size bay = 512, corresponding to 512 pixels to reconstruct, and use Atea = 0.01 and X 0.01 to maintain major focus on the action prediction. For real-world experiment, we set the weight of the reconstruction loss 1.0 and the weight of action loss 0.1. This choice was based observation that reducing the weight of the action loss and increasing the weight of the reconstruction loss did not significantly affect convergence but did help prevent overfitting to
17

---

Front
Wrist
Left
Right
Figure 10: Visualization of 4 cameras used for the stronger PerAct baseline. To enhance the PerAct baseline, add more views as the input of PerAct. These views pre-defined in RLBench, making sure the observation covers the entire scene.
a limited number of real-world demonstrations. We uniformly sample 64 points along the ray for the "coarse" network and sample 32 points with depth-guided sampling and 32 points with uniform sampling for the "fine" network.
Table Hyperparameters used in GNFactor.
Variable Name
Value
training iteration
100k
image size input voxel size
128 128 X 3
100 x 100 x 100
batch size
2
optimizer
LAMB [56]
learning rate ray batch size bray weight for reconstruction loss Arecon weight for embedding loss Xfeat number of transformer blocks number of sampled points for GNF number of latents in Perceiver Transformer dimension of Stable Diffusion features dimension of CLIP language features hidden dimension of NeRF blocks
0.0005
512
0.01
0.01
6
64
2048
512
512
512
18


Reviews:
[{"review_id": "yS45DFLmF8", "reviewer": "Reviewer_asyz", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposed an improved version of PerAct for better generalization ability. Specifically, The author proposed Generalizable Neural feature Fields (GNF) as a visual representation of the scene, and combine it with the perceiver transformer-based action module in PerAct. RGB and semantic feature reconstruction via volume rendering is used to aid in learning the GNF.", "summary_of_recommendation": "Based on the strengths and weaknesses section, the paper's empirical results and usage of semantic feature reconstruction are interesting, I would recommend weak accept for the paper.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "- Why the input feature of GNF is 170 in L460, it seems to be inconsistent with the definition in L141-L144. I'm also confused why the input of σ,c,f is $R^{3}$, $R^{3 \\times 3}$, $R^{3 \\times 3}$ respectively. Also, what is the size of the positional encoding?\n- More explanation on ablation results, why Diffusion works better than DINO?\n- Why might depth-guided sampling (DGS) have a more significant impact on the model's performance even than semantic features, considering its originally proposed for speeding up rendering, to my understanding?\n- I would suggest the author avoid using the term \"NeRF\" to describe GNF (e.g. in L7) to prevent confusion (see weakness)\n- Missing reference [1]\n\nReference:\n\n[1] Z. Jiang et al., GIGA: \"Synergies Between Affordance and Geometry: 6-DoF Grasp Detection via Implicit Representations\", RSS, 2021", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths:\n\n\n- The paper is well-organized and easy to follow\n- Experiments in both simulation and real-world are conducted\n- The proposed methods outperform the baseline in empirical results\n\nWeakness\n\n- The view synthesis results are poor, and it's unclear if GNF learns visual features. The author may consider ablative studies on the RGB reconstruction module (e.g. only remove RGB reconstruction loss and keep the semantic feature reconstruction loss). Also, the authors may need to provide more details on how the GNF is implemented to capture high-frequency geometry and texture (e.g. details on positional encoding), as the local features $v_x$ lack high-frequency information (since it's an interpolation of a discretized feature grid)\n\t\n- The proposed GNF seems to be different from NeRF, as NeRF is an implicit field that takes position (positional encoder) as input, while GNF uses both position and local features (interpolation of a feature grid), which is more similar to GIGA [1]. Only the volume rendering part shares similarity with NeRF. The author may explain more about why they are not using the original NeRF architecture. \n\t\n- More details on how PerAct is trained in the experiments section. Is the voxel grid constructed from multiple RGB-D images or just one RGB-D image (as in GNFactor)? Using only one RGB-D camera to train PerAct may lead to an unfair comparison, as GNFactor utilizes additional RGB images, gaining more information regarding the scene during training. While PerAct may learn poorly in certain scenarios due to limited data, such as when the target object is occluded in the RGB-D image.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "TQSdTAI7Hp4", "reviewer": "Reviewer_dBGA", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes a method for performing language-conditioned actions across multiple tasks using a neural feature field representation. The robot is given a natural language description of the task and expected to accomplish it. To do this, first an RGBD image from the robot’s perspective is captured and converted to a voxel grid. Next it is fed to a learned voxel encoder which outputs a voxel feature grid. This is then condensed and concatenated with both proprioceptive and CLIP features from the natural language description of the task. The result of this is fed through the perceiver network, which outputs Q-values for the robot’s actions. The results show that this method outperforms the baseline on both simulated and real robot tasks.", "summary_of_recommendation": "The methodology proposed in this paper is novel and interesting. The results clearly show that it improves over the chosen baseline. As such, I think this paper makes a meaningful contribution to the literature. So long as the authors can address my concerns about the clarity of the paper, I would give this paper a weak acceptance recommendation. If the authors are able to add additional baseline comparisons, I would bump that up to a strong acceptance recommendation.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "The biggest area for improvement during the rebuttal period is clarity. There are some things in the paper I am unclear about. For example, in section 3.3 it says on line 167 that the voxel grid is “condensed” and then on line 177 it is “upscaled.” How? Please clarify which method is used for these (a couple words is likely sufficient, e.g., “using trilinear downsampling”). On a bigger scale, it is unclear how the robot actually executes actions. Section 3.1 mentions keyframes and using a motion planner between them, but which planner? Furthermore, the perceiver part of the network outputs what the paper terms “Q-values.” Are these Q-values as in Q-learning used by many RL methods? Or are these the next desired kinematic state for the planner to plan to? If the latter, please pick a different variable than “Q” as this will be very confusing to anyone from the RL field, and if the former please describe in much more detail how actions are selected. I suspect the latter is the case but the choice of variable name confused me for quite some time.\n\nSmall nit: The sentence spanning lines 233 and 234 lists “GNFactor and PerAct” in that order but lists the results “54.7% and 76.0%” in the opposite order, with PerAct achieving 54.7% and GNFactor achieving 76.0% (at least according to Table 1). Please swap.\n", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The biggest strength and main contribution of this paper is the way in which the voxel encoder is trained. In addition to a loss on the output Q-values, a secondary loss on the reconstruction of the scene is used. Specifically, the encoded voxel grid is interpreted as a neural radiance field and is re-rendered from another viewpoint using the NeRF volumetric rendering equation. The loss between this image and the ground truth RGB image taken from this viewpoint is used to condition the voxel encoder. Additionally, a feature image is rendered from the same viewpoint using the same technique and compared to the features generated by Stable Diffusion for that image. These two extra loss terms enforce that the voxel encoder encodes both correct geometry and meaningful semantics, significantly improving its performance as shown in the results.\n\nThe main weakness of the paper is the results are not very persuasive. The goal is to show that the robot can generalize to new tasks not seen during training using this method, but the average generalization success rate is only around 30%. It’s hard to say if the robot is really succeeding or not. However, the inclusion of the PerAct baseline goes a long way to alleviating this concern. It is clear that the methodology in this paper is doing something because the baseline is even worse at generalization. Without this baseline that would be impossible to tell. One improvement the authors might consider to this paper would be to add additional baselines, particularly ones that vary significantly or use alternative kinds of inputs. This would help put in context the performance of the proposed method. Although I will note that even without additional baselines, I would recommend this paper for acceptance, so they aren’t strictly necessary, more of a nice finishing touch.\n", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}, {"review_id": "Id_NHW8T0JF", "reviewer": "Reviewer_Lzg7", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper presents an approach for end-to-end multi-task robotic manipulation based on a language prompt. At test time, the system: a) encodes a third-view RGB-D image of the robot environment into a 3D-voxel-based volume and further takes as input: b) the robot state, c) a text-based description of a task to perform. Combining a), b), c), the system predicts a set of actions that the robot arm should execute. The same inputs are provided during training, which follows a multi-task learning paradigm: 1. Coordinate-based MLPs are conditioned on the voxel-based feature grid a) and decoded using volume rendering to reconstruct both RGB and foundation-model features of a novel view. 2. a), b) and c) are combined to estimate Q-values for the actions to perform. The approach outperforms previous methods in real-world and synthetic tasks. The main claimed contribution consists in generalization improvement, attributed to the used 3D representation and to the features from a foundation model.", "summary_of_recommendation": "While partially incremental, the proposed approach effectively combines tools, resulting in a performance improvement over previous works. \nConsidering also that the method was tested on a real-world system, overall I recommend the paper for acceptance. However, I would like the authors to clarify the points I raised above, and in particular to further support the claim that the improvement in generalization is a result of the use of a 3-D representation and of the features distilled from a foundation model.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "- The main point that I would like the authors to further support is the claim that the generalization ability comes from the use of a 3D representation and of features distilled from a foundation model, or in other words, from the multi-task training paradigm. As noted in L252-255, the rendering reconstructions are quite poor in quality, and by looking at Table 4 it seems like increasing the importance of the rendering losses results in a decreased success rate? Was a baseline with $\\lambda_\\textrm{recon}=0$ or at least with $\\lambda_\\textrm{feat}=0$ tried? Such an ablation would really show the importance of the multi-task training.\n- For the same reason, it would be great if Table 3 included variances like Table 1 and 2, to show how large the improvement of generalization is w.r.t. PerAct.\n- L86: Why is the \"lack of scene structure\" a limitation? What exactly is meant by \"scene structure in this context\"? Similarly, the sentence \"SNeRL masks the scene structure to ensure functionality\" is unclear: What is meant by \"masking\", \"scene structure\", and \"ensuring functionality\" (of what?) in this context?\n- How is the coordinate frame of the voxel grid chosen? In particular, how are the encoded camera-centric features backprojected into the voxel volume?\n- How important is it that the novel-view synthesis during training is performed for keyframes defined according to the description L115-L117? Would the results change significantly if less constrained viewpoints (e.g.,  with a varied gripper's open state, or with non-zero joint velocity) were used?\n- Could you elaborate on the need to go back to a voxel representation (L176)? I see that this is practically used for the skip connection, but was the impact of this evaluated? For instance, how important is the skip connection?\n- L257: What is meant by \"components\"?\n- L269-270: \"GNFactor gains an understanding of objects\" sounds like an unsupported claim. I encourage the authors to rephrase or the further support this claim.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "# Summary\n**Strengths**:\n- The chosen combination of pre-existing tools proves to be effective in improving the performance in real-world and synthetic manipulation tasks, w.r.t. the baselines.\n- The approach was tested on a real-world system.\n- Videos and visualizations of the performed real-world and synthetic experiments are provided.\n- The quality and clarity of the manuscript are overall good.\n\n**Weaknesses**\n- Additional experiments could be performed to further support the main claimed contribution that the improvement in generalization is a result of the used 3-D representation and the distilled features from a foundation model (cf. \"Questions for Rebuttal\")\n- The system is fairly incremental.\n\n--------\n# Detailed review\n\n## Originality and significance\n\nThe system is largely inspired by previous work (mainly PerAct) and is in this respect fairly incremental. However, the chosen combination of existing tools (language-conditioned Perceiver for action prediction, generalizable neural fields, and distillation of features from a foundational model) is interesting, and is shown to provide measurable improvements over previous works.\n\n### Quality and clarity\n- The manuscript is generally well-written, with only a limited amount of typos, although occasionally clarity, language and sentence structure could be improved (cf. below and \"Questions for Rebuttal\").\n- The number of results presented, as well as the format of the presentation, is satisfactory, but additional ablations could be performed to strengthen the contribution (cf. \"Questions for Rebuttal\").\n- The implementation details essential for reproducibility are presented (either in the main paper or in the Supplementary).\n\n## Suggestions and minor questions\n### Structure / clarity\n- L13: \"[...] substantial improvement\": Improvement in what exactly?\n- L25-26: It would be great if an example was provided for why \"these challenges arise from the need to comprehend the 3D structure of the scene, understand the semantics and functionality of the objects [...]\", etc.\n- L37-38: \"simple scene structure with heavy masking in a single-task setting\". This sentence is unclear or not properly introduced. In particular: what does \"masking\" mean or refer to in this context?\n- L88: The name NeRF-RL has not been introduced or linked to a reference before in the text.\n- L93: Is [28] based on neural fields?\n- L100: \"the generalizable feature\" is vague. Perhaps \"generalizable features\" would read better.\n- L116: So far no mention of a \"trajectory\", \"gripper\" or a robot arm has been made and this terms might sound confusing to the reader. It might be beneficial to first state clearly that the problem involves a robot arm with a gripper. Similarly, some restructuring of Sec. 3.1 could be beneficial for understanding:\n  - The camera configurations are introduced in L120-123, but frames are already mentioned in the previous paragraph.\n  - L124-127 mention the collection of additional images to train the GNF, and in particular of RGB images rather than RGB-D ones, but it would be easier to follow this explanation if the training paradigm are requirements of each module were introduced first.\n- L127: In what unit of measure is the translation regressed?\n- L133: For additional clarity, I suggest pointing the reader here to the additional implementation details (e.g., the architecture of the encoder) introduced in App. B and Sec. 4.1.\n- L146-147: The point \"the transformation from the Deep 3D Volume to the 2D input for the Renderer corresponds to the density function\" is not clear from Fig. 3, which in fact does not mention \"density\" at all.\n- Eq. (2): Minor: The previously-introduced $v$ argument was dropped in $\\hat{\\mathbf{C}}(\\mathbf{r}, v)$ and $\\hat{\\mathbf{F}}(\\mathbf{r}, v)$.\n- L167: How exactly is \"the shared volumetric representation\" \"condensed\" into a lower dimensional volume? Through some form of pooling?\n- L169: What is meant exactly by \"robot's proprioception\"? That is, what does the robot state consist of exactly?\n- L178: Why is $Q_\\textrm{open}$ not mentioned here?\n- Tables 1, 2, and 3 should include either in the number or in the caption the percent sign, since (I'm assuming) the success rate is expressed as percentage values here.\n- While understandable, the symbols in Table 4 (particularly $\\rightarrow$) could be clarified.\n\n### Additional references\nAdditional references that distill pre-trained foundation models into 3-D representations could be included:\n- Peng et al. \"OpenScene 3D Scene Understanding with Open Vocabularies\", CVPR, 2023.\n- Murthy Jatavallabhula et al., \"ConceptFusion: Open-set Multimodal 3D Mapping\", RSS, 2023.\n- Blomqvist et al., \"Neural Implicit Vision-Language Feature Fields\", arXiv/2303.10962, 2023.\n\n### Language\n- L9-10: Repetition of the same concept: \"To incorporate semantics in 3D[....], to distill rich semantics into the deep 3D voxel\".\n- Tense coherency: In L39 and L47-50, the future tense is used, unlike the rest of the paragraph, where the present tense is used.\n- L50: \"reason the relations\" is not correct English.\n- L93-95: This sentence is not cohesive/contains a repetition: \"NeRF[30] stands out [...], while NeRF requires [...]\".\n- L99: \"which thus enabling\" is not correct English.\n- Caption of Table 1: \"finally saved checkpoints\" is a bit clunky. Perhaps \"final checkpoint\" would sound better.\n- L239: \"handle\". Missing: handle what?\n- L250: For coherency with the other occurencies, \"the GNFactor\" should be \"GNFactor\".\n\n### Typos and formatting\n- \"e.g.\" -> \"\\eg\" (L10, L56).\n- \"ground-truth\" (noun) -> \"ground truth\" (L57). \"ground truth\" (adjective) -> \"ground-truth\" (L258)\n- L72-73: em-dashes (--) should be used instead of simple dashes.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "-RQZNwAhrqs", "reviewer": "Reviewer_obdr", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper presents GNFactor, a behavior-cloning agent that uses Neural Radiance Fields (NeRFs) to improve generalization with persistent 3D representations. GNFactor builds-off PerAct, a prior method for 3D manipulation, by adding an auxiliary NeRF objective that reconstructs RGB and Diffusion features from multiple viewpoints. This objective improves robustness and generalization by enforcing 3D and semantic consistency. Experiments are conducted in both simulation and real-world environments. In simulation, GNFactor significantly outperforms PerAct in task success, especially in tasks with unseen object instances of different sizes. In real-robot experiments, GNFactor outperforms PerAct by two-fold.  \n", "summary_of_recommendation": "GNFactor proposes a novel objective to improve vision-based behavior-cloning policies. While prior works have investigated either 3D reconstruction or pre-trained vision features for better generalization, GNFactor combines them into a single objective. The experimental results and ablations indicate this objective makes a significant improvement. Overall, GNFactor is a valuable contribution to the manipulation community. \n\n**Post Rebuttal**:\nThe authors clarified concerns, and conducted additional experiments to better understand the framework. Happy to raise the score to Strong Accept.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "Summarizing questions from above:\n- Why exactly are the rendered views grainy and blurry?\n- Have the authors trained the NeRF without action loss? \n- Have the authors tried using CLIP vision features instead of diffusion features?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths:\n- Using NeRF rendering and diffusion consistency to improve the generalization performance of behavior-cloning agents is an interesting and novel approach. Prior works have either looked at 3D reconstruction (eg. MIRA) or pre-trained features (R3M) to improve policies, but GNFactor simply puts them together. This combination might be a great way of adapting internet pre-trained 2D models like CLIP/StableDiffusion to robotics tasks that involve 3D understanding. \n- GNFactor achieves compelling quantitative results over PerAct (a prior state-of-the-art). The experiments are conducted in a benchmark simulation environment, and a code-release has been promised by the authors, so the results should be easily reproducible. \n- Overall, the paper is well-written, and the supplementary materials are helpful. The figures are informative. The experiments cover a good range of tasks, ablations, random seed comparisons. The real-robot videos and simulation rollouts on the website and supplementary video are also nice. \n- GNFactor works with real-robots. One major concern with NeRFs for robotics is the issue of painfully slow inference at test-time. Even the fastest NeRFs like Instant-NGP take a few seconds to optimize scenes. But GNFactor only uses the NeRF objective at train-time, and test-time inference is just a forward-pass.\n\nWeaknesses:\n-  The related work is missing a discussion on MIRA (Chen et al, 2021) and NeRF for dynamics (Li et al, 2020). The first work predicts affordances in rendered NeRF images. The second work uses NeRF reconstruction to enforce 3D persistence in visual dynamics models. While GNFactor uses NeRFs in a different context, a short discussion on these prior works might still be helpful to readers.\n- Why are the rendered views in Figure 4 grainy and blurry? This quality is far worse than photorealistic renderings commonly found in NeRF papers. Section 4.2 briefly hints that this could be caused by “optimiz[ing] both neural rendering and action prediction loss”. Did the authors try training without the action prediction loss, and did that generate photorealistic renderings? It’s interesting that the NeRF loss still boosts task performance, despite the poor rendering quality. It might be worth investigating more deeply about this.\n- In Section 4.2 (Ablations): did the authors try using CLIP vision features? Since the language is encoded with CLIP’s language encoder, wouldn’t CLIP vision features be better than Diffusion vision features since they are already aligned? If it turns out empirically that Diffusion features are indeed the best, then that would be good to know for readers. \n- Minor: Table 1 - The average column could be merged between the two rows to avoid confusion. \n- Minor: In Line 232 - “GNFactor achieves an impressive success rate of 28.0%” -> “impressive” is a bit of an overclaim for 28.0%\n\nReferences:  \nLin et al, 2022 – https://arxiv.org/abs/2212.06088  \nLi et al, 2021 –  https://arxiv.org/pdf/2107.04004.pdf ", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces GNFactor, a robotic agent designed for multi-task manipulations based on visual observations in unstructured real-world scenarios. GNFactor combines a neural radiance field (NeRF) for scene reconstruction and a Perceiver Transformer for decision-making, using a unified deep 3D voxel representation. To infuse semantics in 3D, GNFactor integrates a vision-language foundation model like Stable Diffusion. Evaluations on a real robot and RLBench reveal that GNFactor significantly outperforms existing techniques, showcasing strong generalization capabilities.

A common consensus from the reviewers indicates that while the success rate isn't necessarily groundbreaking, the innovative approach, comparison with baselines, and evidence of general improvement over previous methods make the work a valuable contribution to the field.

More specifically, the reviewers unanimously acknowledged the structure and clarity of the paper, the evaluations in both simulated and real-world environments, and the superior performance over the baselines in empirical results. The reviewers also acknowledged the novelty of combining NeRF rendering and diffusion consistency to enhance the generalization performance of behavior-cloning agents.

Given the rebuttal from the authors and the discussions with the reviewers, I recommend accepting this paper. However, the authors shall revise the paper based on the reviewers' comments by integrating the suggested improvements and responses to the reviewers' concerns.
Decision: Accept (Oral)