Reinforcement Learning with Neural Radiance Fields
Authors: Danny Driess
Ingmar Schubert
Pete Florence
Yunzhu Li
Marc Toussaint
Venue: NeurIPS 2022 Accept (2022)
Keywords: RL
NeRF
Computer Vision
Representation Learning
Robotic Manipulation
Neural Implicit Representations

Abstract: It is a long-standing problem to find effective representations for training reinforcement learning (RL) agents. This paper demonstrates that learning state representations with supervision from Neural Radiance Fields (NeRFs) can improve the performance of RL compared to other learned representations or even low-dimensional, hand-engineered state information. Specifically, we propose to train an encoder that maps multiple image observations to a latent space describing the objects in the scene. The decoder built from a latent-conditioned NeRF serves as the supervision signal to learn the latent space. An RL algorithm then operates on the learned latent space as its state representation. We call this NeRF-RL. Our experiments indicate that NeRF as supervision leads to a latent space better suited for the downstream RL tasks involving robotic object manipulations like hanging mugs on hooks, pushing objects, or opening doors.
Video: https://dannydriess.github.io/nerf-rl

Reinforcement Learning with Neural Radiance Fields
Danny Driess"
Ingmar Schubert' TU Berlin
Pete Florence Google
Yunzhu Li MIT
Mare Toussaint Berlin
TU Berlin
Abstract
It is long-standing problem to find effective representations for training re- inforcement learning (RL) agents. This paper demonstrates that learning state representations with supervision from Neural Radiance Fields (NeRFs) can im- prove the performance of RL compared to other learned representations low-dimensional, hand-engineered state information. Specifically, we propose to latent space describing
train an encoder that maps multiple image observations the objects in the scene. The decoder built from a latent-conditioned NeRF serves the supervision signal to learn the latent space. An RI algorithm then operates on the learned latent space as its state representation. We call this NeRF-RL. Our experiments indicate that NeRF supervision leads to latent space better suited for the downstream RL tasks involving robotic object manipulations like hanging mugs on hooks, pushing objects, or opening doors. Video: https: /ttpss//daanpy /ttp:///adnniiieeemrmm github. io/nerf-rl
Introduction
The sample efficiency of reinforcement learning (RL.) algorithms crucially depends on the representa- tion the underlying system state they operate [1, 2, 5, 6, 7]. Sometimes, low-dimensional (direct) representation of the state, such as the positions of the objects in the environment, is consid- ered to make the resulting RL problem most efficient [2].
However, such www-dimensional, direct state representations can have several disadvantages. On the hand, a perception module, e.g., pose estimation, is necessary in the real world to obtain the representation from raw observations, which often is difficult to achieve in practice with sufficient robustness. On the other hand, if the goal is learn policies that generalize shapes [8], using a low-dimensional state representation is often impractical. Such scenarios, while challenging for RI, common, e.g., in robotic manipulation tasks.
different object
Therefore, there a large history of approaches that consider RL directly from raw, high-dimensione observations like images (e.g.. [9, 10]). Typically, an encoder takes the nigh-dimensional input and maps to a low-dimensional latent representation of the state. The RL algorithm (e.g., the Q-function the policy network) then operates perception module is necessary, the framework can extract information from the raw observations that relevant for the task, and the RL agent, in principle, may generalize in which, e.g., object shapes are varied. While these are advantages in principle, jointly training encoders capable of processing high-dimensional inputs from the RL signal alone challenging. To address this, one approach is to pretrain the encoder on a different task, e.g., image reconstruction [1, 4, 11], multi-view consistency [6], time-constrastive task [3]. Alternatively, auxiliary loss on the latent encoding can be added during the RL procedure [5].
the latent vector state input. This way, separate
challenging environments,
In both cases, the choice of the actual function has significant influence on the usefulness the resulting latent space for the downstream
architecture and associated (auxiliary) loss
"qual contribution. Correspondence: danny. dri dwwess@gmmill..ommmmm com
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

RL task. Especially for image data, convolutional neural networks (CNNs) are commonly used for the encoder [12]. However, 2D CNNs have a 2D (equivariance) bias, while for many RL tasks, the 3D structure of our world is essential. Architectures like Vision Transformers [13, 14] process images with no such direct 2D bias, but they often require large scale data, which might be challenging in RI. applications. Additionally, although multiple uncalibrated 2D) image inputs can be used with generic image encoders [15], they do not benefit from 3D inductive biases, which may help for example in resolving ambiguities in 2D images such as occlusions and object permanence.
Recently, Neural Radiance Fields (NeRFs) [16] have shown great success in learning to represent scenes with neural network that enables to render the scene from novel viewpoints, and have sparked broad interest in computer vision [17]. NeRFs exhibit a strong 3D inductive bias, leading to better scene reconstruction capabilities than methods composed of generic image encoders (c.g., [18]..
In the present work, we investigate whether incorporating these 3D inductive biases of NeRFs into learning a state representation can benefit RL. Specifically, we propose to train an encoder that maps multiple RGB image views of the scene latent representation through an auto-encoder structure, where a (compositional) NeRF decoder provides the self-supervision signal using an image reconstruction loss for each view.
In the experiments, we show for multiple environments that supervision from NeRF leads to a latent representation that makes the downstream RL procedure more sample efficient compared to supervision via a 2D CNN decoder, a contrastive loss on the latent space, or even band-engineered perfect low-level state information given as keypoints. Commonly, RI. is trained on environments where the objects have the same shape. Our environments include hanging mugs on hooks, pushing objects on a table, and door opening scenario. In all of these, the objects' shapes are not fixed, and we require the agent to generalize over all shapes from a distribution.
To summarize our main contributions: (i) we propose to train state representations for RL with NeRF supervision, and (ii) we empirically demonstrate that an encoder trained with a latent-conditioned NeRF decoder, especially with an cojject-ommositionalll NeRF decoder, leads to increased RL perfor- mance relative to standard 2D CNN auto-encoders, contrastive learning, or expert keypoints.
2 Related Work
Neural Scene/Object Representations in Computer Vision, and Applications. To our knowledge, the present work is the first to explore if neural scene representations like NeRFs can benefit RL. Outside of RL, however, there has been a very active research field in the area of neural scene representations, both in the representations themselves [19, 20, 21, 22] and their applications; see [23, 24, 17] for recent reviews. Within the family of NeRFs and related methods, major thrusts of research have included: improving modeling formulations [25, 26), modeling larger scenes [26, 27], addressing (re- rre)llighingg [28, 29, 30], and an especially active area of research has been in improving speed, both of training and of inference-time rendering [31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41]. In our case, we are not constrained by inference-time computation issues, since we do not need to render images, and only have to run our latent-space encoder (with a runtime of approx. 7 ms on an RTX3090). Additionally of particular relevance, various methods have developed condiiccndd [42, 43, 44] or conpositional//ojeetoooo 53], although they, nor other NeRF-style methods to our knowledge, have been applied to RL. Neural scene representations have found application across many fields (i.e.. augmented reality and medical imaging [54]) and both NeRFs [55, 56, 57, 58] and other neural scene approaches [59, 60, 61, 62] have started to be used for various problems in robotics, including pose estimation [55], trajectory planning [56]. visual foresight [11, 53], grasping [59, 57], and rearrangement tasks [60, 61, 58).
approaches for NeRFs [45, 46, 47, 48, 49, 50, 51, 52,
Learning State Representations for Reinforcement Learning, One of the key enabling factors for the success of deep RL is its ability to find effective representations of the environment from high-dimensional observation data [10, 63]. Extensive research has gone into investigating different ways to learn better state representations using various auxiliary objective functions. Contrastive learning is a common objective and has shown success in unsupervised representation learning in computer vision applications [64, 65). Researchers built upon this success and have shown such learning objectives can lead to better performance and sample efficiency in deep RL [66, 67]. where the contrasting signals could come from time alignment [68, 3), camera viewpoints [69], and different sensory modalities [70], with applications in real-world robotic tasks [6. 71]. Extensive efforts
2

---

have investigated the role of representation learning in RL [72], provided detailed analysis of the importance of different visual representation pretraining methods [73]. and shown how we can improve training stability in the face of multiple auxiliary losses [74]. There is also range of additional explorations on pretraining methods with novel objective functions (e-g., bisimulation metrics [75] and temporal cycle-consistency loss [76]) and less-explored data sources (e.g., in-the- wild images [77] and action-free videos (78]). Please check the survey for more related work in this direction [79]. Our method is different in that we explicitly utilize decoder that includes strong 3D inductive biases provided by NeRFs, which we empirically show improves RL for tasks that depend on the geometry of the objects.
3 Background
3.1 Reinforcement Learning
action is This work considers decision problems that can be described discrete-time Markov Deci- sion Processes (MDPs) M = A. T, y, R. Po). and A are the sets of all states and actions, respectively. The transition probability (density) from s to s' using
T(s' s. The agent receives a real-valued reward R(s, a, s') after each step. The dis- count factor y E [0, 1) trades off immediate and future rewards. Po : S + R is the dis- tribution of the start state, RL algorithms try to find the optimal policy n S where x² argmax, _t=o " 1 sr.ae), (s.))) [[[s, at 8t-1)] Importantly, in this work, we consider RL problems where the state a encodes both the position and the shape of the objects in the scene. We require the RL agent generalize over all of these shapes at test time. We can therefore think of the state as a tuple s = (sp. sz). where sp encodes positional information, and 8, encodes the shapes involved. We focus the experiments R(s, a, s') = Rg >> for s' E Sg and R(s, a, 8') = 0 for E Sy, where the volume of Sg C S is much smaller than the volume of S. The state space usually is low-dimensional or a minimal description of the degrees of freedom of the system. In this work, we consider that the RL algorithm has only access particular, this means that the policy has observations as input a ~ T(- | y). Since we assume that the underlying state (sp; 8,) is fully observable from y, we can treat y like state for MDP.
IR..
sparse reward settings, meaning
(nigh-dimensional) observation y Y of the scene (e.g., RGB images). In
Reinforcement Learning with Learned Latent Scene Representations. The general idea of RL with learned latent scene representations is to learn encoder Q that maps an observation y y to a k-dimensional latent vector 2 = P0yy) E Z C R$ of the scene. The actual RL components, c.g., the Q-function or policy, then operate
its state description. For a policy this means that the action a ~ T- | 2) (- ((y) is conditional on the latent vector instead of the observation directly. The dimension of the latent vector is typically (much) smaller than that of the observation
space ), but larger than that of the state space S.
3.2 Neural Radiance Fields (NeRFs)
The general idea of NeRF, originally proposed by [16], is to learn function f= (a, c) that predicts the emitted RGB color value c(z) E R and volume density (zx) E R>0 at any 3D world coordinate R3, Based on f. image from arbitrary view and camera parameters can be rendered by computing the color (rr) R3 of each pixel along its corresponding camera ray r(a) = r(0) + ad through the volumetric rendering relation
C(r) - 1 1 an
T do with Ty(r, = exp ((((w)) (-xpt_/"otrw))dd
=exp
(1)
Here, r(0) E F is the camera origin, d E R23 the pixel dependent direction of the ray and a, af E IR the near and far bounds within which objects are expected, respectively. The camera rays are determined from the camera matrix K (intrinsics and extrinsics) describing the desired view.
4 Learning State Representations for RL with NeRF Supervision
This section describes our proposed framework, in which we use latent state space for RL that is learned from NeRF supervision. For learning the latent space, use encoder-decoder where the
3

---

Reconstructions
NeRF Decoder
Observations
D
Train offline, then freeze
Latent
Encuder
Space
Ω
2
renderings
multi-view
masks"
image
RL. Policy
Actions
multi-view
π
a
Train online
Figure I: State representation learning for RL with NeRFs. First, the encoder and NeRF decoder are trained with supervision from a multi-view reconstruction loss on an offline dataset. Then, the encoder's 'ss weights are frozen, and the latent space is used state input train policy with RL. *Masks of individual objects are only required for the compositional variant of our encoder.
decoder a latent-conditioned NeRF, which may either be a global [42, 43, 44] or a compositional NeRF decoder [53]. To our knowledge, no prior work has used such NeRI-derived supervision for RL. In Sec. 4.1 we describe this proposition, Sec. 4.2 provides an overview of the encoder-decode training. Sec. 4.3 and Sec. 4.4 introduce options for the NeRF decoder and encoder, respectively.
4.1 Using Latent-Conditioned NeRF for RL
We propose the state representation 2 on which an RL algorithm operates a latent vector produced by encoder that maps images from multiple views latent z, which is trained with (compositional) latent-conditioned NeRF decoder. that this framework is beneficial for the downstream RL task, as it produces latent vectors that represent the actual 3D geometry the objects in the scene, can handle multiple objects well, as well as fuse multiple views in consistent way deal with occlusions by providing shape completion, all of which is relevant to solve tasks where the geometry is important. There are two steps to our framework, as shown in Fig. First, we train the encoder decoder from dataset collected by random interactions with the environment, i.e., we do not yet need a trained policy. Second, we take the encoder trained in the first step, which we leave frozen, and use the latent space train an RL policy. Note that we investigate two variants of the auto-encoder framework, global one, where the whole scene is represented by one single latent vector, and compositional one, where objects represented by their own latent vector. For the latter, objects are identified by masks in the views.
will be verified in experiments, we hypothesize
4.2 Overview: Auto-Encoder with Contet-Conditiooed NeRF Decoder
Assume that observation y (1 K " of the scene consists of RGB images I' 33xhx i 1, 1,..., V taken from V many camera views, their respective camera projection matrices Kt R334 (including both intrinsics and extrinsics), and per-view image masks M I:V. For global NeRF decoder, these are global background masks Mi, {0, 1] f and for compositional NeRF decoder as in [53], these are sets of binary masks Mj E {0, 1] hxw that identify the objects
j m in the scene in view i. The global case is equivalent to
1, M M The encoder maps these posed image observations from the multiple views into a set of latent vectors 21:m, where each 2j represents each object in the scene separately in the compositional case, or the single all objects in the scene. This is achieved by querying Ω on the masks M.V.V i.e.
= (I-VV KLW KLWYV
MM.V) E R
(2)
for object j. The supervision signal to train the encoder is the image reconstruction loss
L | M D (Ω (TL.V KLV MEm) 1²
on the input view where the decoder D renders an image D(2 K) for arbitrary views specified by the camera matrix K from the set of latent vectors 21:m- Both the encoder and decoder

---

are trained end-to-end at the same time. The target images for the decoder are the same in both the global and compositional case: the dlobal-masked image I' a Mis (o is the clement-wise product). In the compositional case this can be computed with M V j=1 Mj. By fusing the information from multiple views of the objects into the latent vector from which the decoder has to be able to render the scene from multiple views, this auto-em anto-encoder framework can learn latent vectors that represent the 3D configurations (shape and pose) of the objects in the scene.
4.3 Condit--ondittooed NeRF Decoder Details
Global. The original NeRF formulation [16] learns a fully connected network f that represents one single scene (Sec. 3.2). In order to create decoder from NeRFs within an auto-encoder to learn latent space, we condition the NeRF [(,;=) ) on the latent vector E IR* [42, 43, 44J. While approaches such as [42, 43, 44] use the latent code to represent factors such as lighting or category- level generalization, in our case the latent code is intended to represent the scene variation, i.e., shape
and configuration of objects, such that a downstream RL agent may use this as a state representation.
Compositional. In the compositional case, the encoder produces a set of latent vectors 21:m describing each object j ....., m individually, this leads m many NeRFs cz),,, f(x,2) 2,). J 1.... m with their associated volume density Jj and color value cj. Note that while one could use different networks fy with their own network weights for each object, we have single network for all objects. This means that both the object's pose as well as its shape and type are represented through the latent code 2j. order to force those conditioned NeRFs to learn the 3D configuration of each object separately, we compose them into global NeRF model with the composition formulas (proposed e.g., by [80, 81]): a(x) = Em (x)), c(z) ल(w) ンド=1 (z) As this composition happens in 3D space, the latent vectors will be learned such that they correctly represent the actual shape and pose of the objects in the scene with respect to the other objects, which we hypothesize may be useful for the downstream RL agent.
ff(x)
4.4 Encoder Details
The encoder f operates by fusing multiple views together to estimate the latent vector for the RL task. Since the scientific question of this work is investigate whether decoder built from NeRFs train the encoder end-to-end is beneficial for RL, we consider two different encoder architectures. The first one is 2D CNN that averages feature encodings from the different views, where each encoding is additionally conditioned on the camera matrix of that view. The second one is based on a learned 3D neural vector field that incorporates 3D biases by fusing the different camera views in 3D space through 3D convolutions and camera projection. This way, we are able to distinguish between the importance of 3D priors incorporated into the encoder versus the decoder.
Per-image CNN Encoder ("Image encoder"). For the global version, we utilize the network archi- encoder choice. In order work with multiple objects in the compositional case, we modify the architecture from [11] by taking the object masks into account as follows. For each object j, the 2D CNN encoder computes
tecture from [11]
SMLP ECNN (T' M3) , .
2j -CNN (T!:,, K.V. M.. 1MLP
V Σ
(FNNN
(4)
ECNN is a ResNet-18 [82] CNN feature extractor that determines a feature from the masked input image I Mj of object for each view i, which is then concatenated with the (flattened) camera matrix. The output of the network gMLP is hence the encoding of each view, including the camera information, which is averaged and then processed with fMLP, to produce the final latent vector. Note that in the global case, we set m = 1, Mi = Min such that ΩCNN produces a single latent vector.
Neural Field 3D CNN Encoder ("Field encoder""). Several authors [43] have considered to in- corporate 3D biases into learning an encoder by computing pixel-aligned features from queried 3D locations of the scene to fuse the information from the different camera views directly in 3D space, We utilize the encoder architecture from [53], where the idea is to learn a neural vector field & NM:V] 3 + E over 3D space, conditioned on the input views and masks. The features of de are computed from projecting the query point into the camera coordinate system from the respec- tive view. To turn ø into a latent vector, it is queried on workspace set X, E Rdx xha xex (a 3D
5

---

grid) and then processed by 3D convolutional network, i.e., 2, EAD CNN (o [rlv : M. (R))). This method differs from [43, 83, 60] by computing a latent vector from the pixel-aligned features.
5 Baselines/ / Alternative State Representations
In this section, we briefly describe alternative ways of training an encoder for RL, which we will investigate in the experiments as baselines and ablations. For details, refer to the appendix.
This baseline uses a standard CNN decoder based on deconvolutions instead of NeRF to reconstruct the image from the latent representation, similar to [1]. Therefore, with this baseline we investigate the influence of the NeRF decoder relative to CNN decoders. We follow the architecture of [11] for the deconvolution part for the global case. In the compositional case, we modify the architecture to be able to deal with a set of individual latent vectors instead of a single, global one. The image
Cony, Autoencoder:
Daecomv (9MLP( m Σド=1 2y), K) is rendered from 21:m by first averaging the latent vectors and then processing the averaged vector with fully connected network SMLP, leading to an aggregated feature. This aggregated feature is concatenated with the (flattened) camera matrix K describing the desired view and then rendered into the image with Daecovy In the experiments, we utilize this decoder as the supervision signal to train the latent space produced by the 2D CNN encoder from Sec. 4.4. In the compositional version, the 2D CNN encoder (4) use the same object masks as the compositional NeRF-RL variant.
Contrastive Learning. As an alternative to learning encoder via a reconstruction loss, the idea of contrastive learning [84] is to define a loss function directly on the latent space that tries to pull latent vectors describing the same configurations together (called positive samples) while ones representing different system states apart (called negative samples). A popular approach to achieve this is with the InfoNCE loss [85, 64]. Let y/ and Y: be two different observations of the same state. Here, denotes perturbed/auummened version of the observation. For a mini-batch of observations { ú.) }-) after encoding those into their respective latent vectors 2; (ya,, ăn (ūū;) with the encoder Q the loss for that batch would use (z,, 2,) as a positive pair, and (xi, as a negative pair, or some similar variation. crucial question in contrastive learning is how the observation y perturbed/augmenntd into ÿ to generate positive and negative training pairs, described in the following.
CURL. In CURL [5], the input image is randomly cropped to generate y and ÿ. We closely follow the hyperparameters and design of [5]. CURL operates on a single input view and we choose a view for this baseline from which the state of the environment can be inferred as best possible (Fig. 17).
Multi View CURL. This baseline investigates if the neural field 3D encoder (Sec. 4.4) can be trained with contrastive loss. As this encoder operates multiple input views we double the number of available camera views. Half the views are the same as the other experiments, the other half are captured from sightly perturbed camera angles. We use the same loss as CURL, but with different contrastive pairs- rather than from augmentation, the contrastive style is taken from TCN [68]: the positive pairs come from different views but at the same moment in time, while negative pairs come from different times. Therefore, this baseline can be seen as a multi-view adaptation of CURL [5].
Direct State / Keypoint Representations. Finally, we also consider a direct, low- diwemminnional representation of the state. Since we are interested in generalizing over different object shapes, we consider multiple 3D keypoints that are attached at relevant locations of the objects by expert knowledge and observed with a perfect keypoint detector [8]. See Fig. 2b for a visualization of those keypoints. The keypoints both provide information about object shape and its pose. Furthermore, as seen in Fig. 2b, they have been chosen to reflect those locations in the environment relevant to solve the task. Additionally, we report results where the state is represented by the poses of the objects as this cannot represent object shape, in this case we use a constant object shape for training and test.
6 Experiments
We evaluate our proposed method on different environments where the geometry of the ob- jects in the scene is important to solve the task successfully. Please also refer to the video pettps:/ddamydrieeee
github. to/nerf-rl Commonly, RL is trained and evaluated on sin- gle environment, where only the poses are changed, but the involved object shapes are kept constant. Since latent-conditioned NeRFs have been shown to be capable of generalizing over geometry [43],

---

we consider experiments where we require the RL agent to generalize over object shapes within some distribution. Answering the scientific question of this work requires environments with multi-view observations and for the compositional versions object masks as well. These are not provided in standard RI. benchmarks, which is the reason for choosing the environments investigated this work. We use PPO [86] as the RI. algorithm and four camera views in all experiments. Refer to the appendix for more details about our environments, parameter choices, network architectures, and training times.
6.1 Environments
Mug on Hook. In this environment, adopted from [87] and visualized in Fig. 2b, the task is to hang& mug on hook. Both the mug and the hook shape are randomized. The actions are small 3D translations applied to the mug. This environment is challenging as we require the RL agent to generalize over mug and hook shapes and the tolerance between the handle opening and the hook is relatively small. Further, the agent receives a sparse reward only if the mug has been hung stably. This reward is calculated by virtually simulating a mug drop after each action. If the mug does not fall onto the ground from the current state, a reward of one is assigned, otherwise zero.
Planar Pushing. The task in this environment, shown in Fig. 3b, is push yellow box-shaped objects into the left region of the table and blue objects into the right region with the red pusher that can move in the plane, i.e., the action is two dimensional. This is the same environment as in [53] with the same four different camera views. Each run contains a single object on the table (plus the pusher). If the box has been pushed inside its respective region, a sparse reward of one is received, otherwise zero. The boxes in the environment have different sizes, two colors and are randomly initialized. In this environment, we cannot use keypoints for the multi-shape setting, as the reward depends on the object color; we evaluate the keypoints baseline only in the single shape case (Appendix).
Door Opening. Fig. 4b shows the door environment, where the task is open sliding door with the red end-effector that can be translated in DoFs as the action. To solve this task, the agent has to push the door handle. As the handle position and size is randomized, the agent has learn to interact with the handle geometry accordingly. Interestingly, as can be seen in the video in the supplementary material, the agent often chooses to push on the handle only at the beginning, as, afterwards, it is sufficient to push the door itself at its side. The agent receives a sparse reward if the door has been opened sufficiently, otherwise, zero reward is assigned.
6.2 Results
Figs 2a, 3a, 4a show success rates (averaged over 6 independent experiment repetitions and over 30 test rollouts per repetition per timestep) as a function of training steps. Also shown are the 68% confidence intervals. These success rates have been evaluated using randomized object shapes and initial conditions, and therefore reflect the agent's 's ability to generalize over these.
In all these experiments, a latent space trained with compositional NeRF supervision as the decoder consistently outperformed all other learned representations, both in terms of sample efficiency and asymptotic performance. Furthermore, our proposed framework with compositional NeRF even outperforms the expert keypoint representation. For the door environment, the 3D neural field encoder plus NeRF decoder (NeRF-RL comp. + field) reaches nearly perfect success rates. For the other two environments, the compositional 2D CNN encoder plus NeRF decoder (NeRF-RL comp. + image) was slightly better than with the neural field encoder but not significantly. This shows that the decoder built from compositional NcRF is relevant for the performance. not so much the choice of the encoder.
Training the 3D neural field encoder with a contrastive loss as supervision signal for different camera views as positiveenegative training pairs is not able to achieve significant learning progress in these scenarios Multi-CURL. However, the other contrastive baseline, CURL, which has a different encoder and uses image cropping as data augmentation instead of additional camera views, is able to achieve decent performance and sample efficiency on the door environment, but not for the pushing environment. the mug environment, CURL initially able to make learning progress comparable to our framework, but never reaches a success rate above 59% and then becomes unstable. Similarly, the global CNN autoencoder baseline shows decent learning progress initially on the mug and pushing scenario (not for the door), but then becomes unstable (mug) never surpasses 50% success rate
7

---

encoder
decoder
comp.
NeRF
loss
NeRF- RI.
comp.+ficld
3D CNN
comp. 3D NeRF
image reconstr.: L.2
comp.timage
2D CNN
comp. 3D NeRF
image reconstr:: 1.2
(ours)
global+image
2D CNN
global 3D NeRF
X
image reconstr:: L2
Cony. Autoencoder,
2D CNN
comp. 2D CNN 2D CNN
image reconstr.: L2
Cony, Autoencoder, g
2D CNN
X
X
image reconstr.: L2
CURL Multi-CURL
CNN
contrast: InfoNCE
CNN
contrast: InfoNCE
Keypoints
chosen by expert knowledge and perfect extraction
Table Overview of the different state representation learning frameworks.
NoRF-RL. field
comp. image
0.8
Com, Autoemedkr,
Com.. Autneneader, CURL
1
Saages Rate
0.6
Multi-CUR
Keypolnts
5
(b) Left: Blue coordinate frames denote the four camera poses. Red points are expert keypoints. Right: NeRF renderings (different scenes).
Number of experienced transitions N 10
(a) Learning curve.
Figure 2: Mug on hook environment. (b) shows an example scene and NeRF renderings
NcRF-RL comp.
NeRU-RL comp. + image
NcRF-RI glohil a image
Conw Autoencoder,
Cons, Automeeder,
CURT
0.4
0.2
0.00
0.25
0.50
0.75
1.00
1.25
1,50 1,75
(b) Left: Blue coordinate frames denote the four camera poses. Yellow and blue areas are goal regions where the objects should be pushed to. Right: NeRF renderings.
Number experienced transitions N 10"
(a) Learning curve.
Figure 3: Pushing environment. (b) shows NeRF renderings for different scenes.
NeRF RL comp field NcRI-RL camp. image
0.8
Coms. Autvencoder,
Cumy, Autoenooder,,
0.6
Muli-CURL Keypaints
0.4
0.2
(b) Left: Blue coordinate frames denote the four camera poses. Right: NeRF renderings.
Number of experienced transitions N 10
(a) Learning curve.
Figure 4: Door environment. (b) shows NeRF renderings for different scenes.

---

(pushing). Such variations in performance or instable learning across the different environments have not been observed with our method, which is stable in all cases.
The compositional variant (NeRF-RL comp.) of our framework achieves the highest performance. Since the conv. comp. autoencoder baseline has worse performance than its global variant, composi- tionality alone is not the sole reason for the better performance of our state representation. Indeed, the global NeRF-RL image variant in the pushing env. is also better than all other baselines.
In the appendix Sec. A.I, we find a positive correlation between NeRF reconstruction quality and RL performance. Furthermore, it turns out that the performance of our framework is not significantly affected when we pretrain the encoder with less data (Sec. A.2). In Sec. A.3, we investigate the influence of the number of input views on the RL performance. In the pushing scenario, only two or even one input view are sufficient for good performance. However, for tasks that require more 3D) understanding such the mug scenario, we observe drop in performance when reducing the number of views from 4 to 2.
7 Discussion
Why NeRF provides better supervision. The NeRF training objective (I) strongly forces each f(.,zg 2ÿ) to represent each object in its actual 3D configuration and relative to other objects in the scene (compositional case), including their shape. This implies that the latent vectors 2, have to contain this information, i.e., they are trained to determine the object type, shape and pose in the scene. In the global case, 21 has to represent the geometry of the whole secne. As the tasks policies to take the geometry of the objects into account, we hypothesize that a latent vector that is capable of parameterizing a NeRF reconstruct the scene in the 3D space has to contain enough of the relevant 3D information of the objects also for the policy to be successful.
consider require
Masks. In order for the auto-encoder framework to be compositional, it requires object masks. We believe that instance segmentation has reached a level of maturity [88] that this is a fair assumption to make. As we also utilize the individual masks for the compositional conv. autoencoder and the multi-view CURL baseline, which do not show good performance, it indicates that the masks are not the main reason that our state representation achieves higher performance. This is further supported by the fact that the global NeRF-RL RL variant which does not rely on individual object masks on the pushing scenario achieved a performance higher than all baselines, i.e., masks will increase the performance of NeRF-RL as they enable the compositional version, but they do not seem essential.
Offline/Online In this work, we focused on pretraining the latent representation offline from a dataset collected by random actions. During RL, the encoder is fixed and only the policy networks are learned. This has the advantage that the same representation can be used for different RL tasks and the dataset to train the representation not necessarily has to come from the same distribution. However, if a policy is needed to explore reasonable regions of the state space, collecting a dataset offline to learn latent space that covers the state space sufficiently might be more challenging for offline approach. This was not an issue for our experiments where data collection with random actions was sufficient. Indeed, we show generalization over different starting states of the same environment and with respect to different shapes (within distribution). Future work could investigate NeRF supervision in online setup. Note that the reconstruction loss via NeRF is computationally more demanding than via a 2D CNN deconv. decoder or a contrastive term, making NeRF supervision as an auxiliary loss at each RL training step costly. One potential solution for this is to apply the auxiliary loss not at every RL training step, but with a lower frequency. Regarding computational efficiency, this is where contrastive learning has an advantage over our proposed NeRF-based decoder, as the encoding with CURL can be trained within half day, whereas the NeRF auto-encoder took up to 2 days to train for our environments. However, when using the encoder for RL, there is no difference in inference time.
Multi-View.
The auto- uuto-encoder framework we propose can fuse the information of multiple camera views into a latent vector describing an object in the scene. This way, occlusions can addressed and the agent can gain better 3D understanding of the scene from the different camera angles. Having access to multiple camera views and their camera matrices is an additional assumption we make, although we believe the capability utilize this information is an advantage of our method.

---

8 Conclusion
In this work, we have proposed the idea utilize Neural Radiance Fields (NeRFs) to train latent spaces for RL. Our environments focus on tasks where the geometry of the objects in the scene is relevant for successfully solving the tasks. Training RL agents with the pretrained encoder that maps multiple views of the scene to a latent space consistently outperformed other ways of learning a state representation and even keypoints chosen by expert knowledge. Our results show that the 3D prior present in compositional NeRF as the decoder is more important than priors in the encoder.
Broader Impacts. Our main contribution is a method learn representations that improve the efficiency of vision-based RL. which could impact automation. As such, our work inherits general ethical risks of AI, like the question of how address the potential of increased automation in society,
Acknowledgments
The authors thank Russ Tedrake for initial discussions; Jonathan Tompson and Jon Barron for feedback on drafts; Vincent Vanhoucke for encouraging latent NeRFs.
This research has been supported by the Deutsche Forschungsgemeinschatt (DFG, German Research Foundation) under Germany's s Excellence Strategy EXC 2002/1 "Science of Intelligence". project number 390523135. Danny Driess thanks the International Max-Planck Research School for Intelli- gent Systems (IMPRS-IS) for the support. Ingmar Schubert acknowledges support by the German Academic Scholarship Foundation. Yunzhu Li acknowledges support by Amazon.com Services LLC,, PO# #2D-063 10236 and the Wistron Corporation.
References
[1] C. Finn, X. Y. Tan, Y. Duan, T. Darrell, S. Levine, and P. Abbeel. Deep spatial autoencoders for visuomotor learning. In 2016 IEEE International Conference on Robotics and Automation (ICRA), pages 512-519. IEEE, 2016.
[2] R. Jonschkowski, R. Hafner, Scholz, and M. Riedmiller. Pves: Position-veloose encoders for unsupervised learning of structured state representations, arXiv preprint arXiv:1005.09800 2017.
[3] D. Dwibedi, J. Tompson, C. Lynch, and P. Sermanet. Learning actionable representations from visual observations. 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 1577-1584. IEEE, 2018.
[4] T. D. Kulkarni, A. Gupta, C. Ionescu, S. Borgeaud, M. Reynolds, A. Zisserman, and V. Mnih. Unsupervised learning of object keypoints for perception and control. Advances in neural information processing systems, 32, 2019.
[5] M. Laskin, A. Srinivas, and P. Abbeel. Curl: Contrastive unsupervised representations for reinforcement learning. In International Conference on Machine Learning, pages 5639-5650. PMLR, 2020.
[6] L. Manuelli, Y. Li, P. Florence, and R. Tedrake. Keypoints into the future: Self-supervised correspondence in model-based reinforcement learning. arXiv preprint arXiv:2009.05085, 2020.
[7] M. Vecerik, J.-B. Regli, O. Sushkov, D. Barker, R. Pevceviciute, T. Rothörl, C. Schuster, R. Hadsell, L. Agapito, and J. Scholz. S3k: Self-supervised semantic keypoints for robotic manipulation via multi-view consistency, arXiv preprint arXiv:2009. 20XV::2009..47111. 2020.
[8] L. Manuelli, W. Gao, P. Florence, and R. Tedrake. kpam: Keypoint affordances for category- level robotic manipulation. arXiv preprint 1903v: 06684, 2019.
[9] V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602, 2013.
[10] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, Ostrovski, et al. Human-level control through deep rein- forcement learning. nature, 5 518(7540):529-555
2015.
[1I] Y. Li, S. Li, V. Sitzmann, Agrawal, and A. Torralba. 3d neural scene representations for visuomotor control. In Conference Robot Learning, pages 112-123, PMLR, 2022.
10

---

[12] S. Lange and M. Riedmiller. Deep auto-encoder neural networks in reinforcement learning, In The 2010 International Joint Conference on Neural Networks (IJCNN), 2010.
[13] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[14] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenbom, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al. An image is worth 16x16 words: Transformers for image recogmition at scale. arXiv preprint arXiv:2010. 1929, 2020.
[15] I. Akinola, J. Varley, and D. Kalashnikov. Learning precise 3d manipulation from multiple uncalibrated cameras. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages
IEEE, 2020.
[16] B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In European conference on computer vision, pages 405-421. Springer, 2020.
[17] Dellaert and L. Yen-Chen. Neural volume rendering: Nerf and beyond, 2021.
[18] S. A. Eslami, D. Jimenez Rezende, F. Besse, F. Viola, A. S. Morcos, M. Garnelo, A. Ruderman,
A. A. Rusu, I. Danihelka, K. Gregor, et al. Neural scene representation and rendering. Science, 360(6394)) 100-)1210, 2018
[19] J. Park, P. Florence, J. Straub, R. Newcombe, and S. Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 165-174, 2019.
[20] L. Mescheder, M. Oechsle. M. Niemeyer, S. Nowozin, and A. Geiger. Occupancy networks: Learning 3d reconstruction in function space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4460 1660-4470, 2019.
[21] Z. Chen and H. Zhang. Learning implicit fields for generative shape modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9939, 2019.
[22] V. Sitzmann, M. Zollhöfer, and G. Wetzstein. Scene representation networks: Continuous 3d-structure-aware neural scene representations. Advances in Neural Information Processing Systems, 32, 2019.
[23] Y. Xie. T. Takikawa, S. Saito, O. Litany, S. Yan, N. Khan, F. Tombari, J. Tompkin, V. Sitzmann, and S. Sridhar. Neural fields in visual computing and beyond. arXiv preprint arXiv2211.111466 2021.
[24] A. Tewari, J. Thies, B. Mildenhall, Srinivasan, E. Tretschk, Y. Wang. C. Lassner, V. Sitz- mann, R. Bruuala, S. Lombardi, et al. Advances in neural rendering, arXiv preprint
arXiv:2111 111111..050 1.05849, 2021.
[25] J. T. Barron, B. Mildenhall, M. Tancik, P. Hedman, R. Martin-Brualla, and P. P. Srinivasan. Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5855-5864, 2021.
[26] J. T. Barron, B. Mildenhall, D. Verbin, P. P. Srinivasan, and P. Hedman. Mip-nerf 360: Un- bounded anti-aliased neural radiance fields. arXiv preprint
2021.
[27] M. Tancik, V. Casser, X. Yan, S. Pradhan, B. Mildenhall, P. P. Srinivasan, J. T. Barron, and H. Kretzschmar. Block-nerf: Scalable large scene neural view synthesis. arXiv preprint arXiv:2202.05223. 2022.
[28] M. Boss, R. Braun, V. Jampani, T. Barron, C. Liu, and H. Lensch. NeRD: Neural reflectance decomposition from image collections.
2020.
[29] Srinivasan, Deng, X. Zhang, M. Tancik, B. Mildenhall, and J. T. Barron. NeRV: Neural re- flectance and visibility fields for relighting and view synthesis. http:///ariv.or//d////11200 2020.
[30] X. Zhang, P. P. Srinivasan, B. Deng, P. Debevec, W. T. Freeman, and J. T. Barron. Nerfactor: Neural factorization of shape and reflectance under an unknown illumination. http::/arri..or/las22//0000000
2021.
[31] L. Liu, J. Gu, K. 7. Lin, T.-S. Chua, and C. Theobalt. Neural sparse voxel fields. In Advances in Neural Information Processing Systems (NeurIPS), volume 33, 2020.
11

---

[32] D. Lindell, J. Martel, and G. Wetzstein. Autolnt: Automatic integration for fast neural volume rendering. https://arri.oorrdbb2222221/2//2
2020.
[33] D. Rebain, W. Jiang, S. Yazdami, K. Li, K. M. Yi, and A. Tagliasacchi. DeRF: Decomposed 2020.
radiance fields. https://araih..rrrabb/0111..1..
[34] T. Neff, P. Stadlbauer, M. Parger, A. Kurz, H. Mueller, R. A. Chaitanya, A. S. Kaplanyan, and M. Steinberger. DONeRF: Towards Real-Time Rendering of Compact Neural Radiance Fields using Depth Oracle Networks. Computer Graphics Forum, 40(4), 2021. ISSN 1467-8659. doi: 10.111 14340. URL https:// org/10. 1111/cgf. 14340.
[35] S. J. Garbin, M. Kowalski, M. Johnson, J. Shotton, and J. Valentin. Fastnerf: High-fidelity neural rendering at 200fps. https://arsiix.or//u////033 10380, 2021.
[36] C. Reiser, S. Peng. Y. Liao, and A. Geiger. Kilonerf: Speeding neural radiance fields with thousands of tiny mips. 2021.
[37] A. Yu, R. Li, M. Tancik, H. Li, R. Ng, and A. Kanazawa. Plenoctrees for real-time rendering of neural radiance fields. In arXiv, 2021.
[38] S. Lombardi, T. Simon, G. Schwartz, M. Zollhoefer, Y. Sheikh, and J. Saragih. Mixture of volumetric primitives for efficient neural rendering, 2021.
[39] A. Yu, S. Krilovich-Keil, M. Tancik, Q. Chen, Recht, and A. Kanazawa. Plenoxels: Radiance fields without neural networks. arXiv preprint arXiv:2112.05131, 2021.
[40] V. Sitzmann, S. Rezchikov, W. Freeman, J. B. Tenenbaum, and F. Durand. Light field networks: Neural scene representations with single-evaluation rendering. In arXiv, 2021.
[41] T. Müller, A. Evans, C. Schied, and A. Keller. Instant neural graphics primitives with a multiresolution hash encoding. ACM Trans. Graph., 41(4): July 2022. doi: URL https: /tt./ii..oo/ org//oo//// 10. 1155/1152229. 35301177.
10.1 241114///222233353001
[42] R.
N. Radwan, M. S. Sajjadi. T. Barron, A. Dosovitskiy, and D. Duckworth. Nerf in the wild: Neural radiance fields for unconstrained photo collections. In Proceedings of the IEEE/CVE Conference on Computer Vision and Puttern Recognition, pages 7210-7219, 2021.
[43] A. Yu, V. Ye, M. Tancik, and A. Kanazawa. pixelnerf: Neural radiance fields from one or few images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4578-4581 2021.
[44] Q. Wang. Z. Wang, K. Genova, P. P. Srinivasan, H. Zhou, J. T. Barron, R. Martin-Brualla, N. Snavely, and T. Funkhouser. Ibrnet: Learning multi-view image-based rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4690 4699, 2021.
[45] K. Zhang, G. Riegler, N. Snavely, and V. Koltun. NERF++: Analyzing and improving neural radiance fields. http::/arrxt.orr//a//21000.00000
2020.
[46] M. Niemeyer and A. Geiger. GIRAFFE: Representing scenes as compositional generative neural feature fields. htpps//arxiiooor//l////11.1..000 2020.
[47] M. Guo, A. Fathi, J. Wu, and T. Funkhouser. Object-centric neural scene rendering.
https://arxivoorgaas20000 2..02200222202..... 2020.
[48] W. Yuan, 7.. Lv, T. Schmidt, and S. Lovegrove. Star: Self-supervised tracking and reconstruction of rigid objects in motion with neural rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13114-13152. 2021.
[49] Z. Wang, T. Bagautdinov. S. Lombardi, T. Simon, J. Saragih, J. Hodgins, and M. Zollhöfer. Learning compositional radiance fields of dynamic human heads. 2020.
[50] J. Ost, F. Mannan, N. Thuerey, J. Knodt, and F. Heide. Neural scene graphs for dynamic scenes. https///artivorr/aass/0111111 10339, 2020.
[51] H.-X Yu, I.. J. Guibas, and J. Wu. Unsupervised discovery of object radiance fields. arXiv preprint arXiv:2107.07905, 2021.
[52] B. Yang. Y. Zhang, Y. Xu, Y. Li, H. Zhou, H. Bao, G. Zhang, and Z. Cui. Learning object- compositional neural radiance field for editable scene rendering. International Conference on Computer Vision (ICCV), October 2021.
12

---

[53] D. Driess, Z. Huang, Y. Li, R. Tedrake, and M. Toussaint. Learning multi-object dynamics with compositional neural radiance fields. XrXiv preprint arXiv:2202.11855, 2022.
[54] H. Zhang, R. Wang. J. Zhang, C. Li, G. Yang, P. Spincemaille, T. Nguyen, and Y. Wang. Nerd: Neural representation of distribution for medical image segmentation. arXiv preprint WMXiv:2103.40022 2021.
[55] L. Yen-Chen, P. Florence, J. T. Barron, A. Rodriguez, P. Isola, and T.-Y. Lin. iNeRF: Inverting neural radiance fields for pose estimation. IROS, 2021.
[56] M. Adamkiewicz, T. Chen, A. Caccavale, R. Gardner, P. Culbertson, J. Bohg, and M. Schwager. Vision-only robot navigation a neural radiance world. IEEE Robotics and Automation Letters,
7(2):460644613, 2022.
[57] J. Ichnowski, Y. Avigal, J. Kerr, and K. Goldberg. Dex-nerf: Using a neural radiance field to grasp transparent objects. arXiv preprini arXiv:21 10.11177, 2021.
[58] L. Yen-Chen, P. Florence, J. T. Barron, T.-Y. Lin, A. Rodriguez, and P. Isola. NeRF- :upervinnn Learning dense object descriptors from neural radiance fields. In IEEE Conference on Robotics and Automation (ICRA), 2022.
[59] K. Karunratanakul, J. Yang. Y. Zhang, M. J. Black, K. Muandet, and S. Tang. Grasping field: Learning implicit representations for human grasps. In 2020 International Conference on 3D Vision (3DV). pages 333-344. IEEE, 2020.
[60] J.-S. Ha, D. Driess, and M. Toussaint. Learning neural implicit functions as object representa- tions for robotic manipulation. arXiv preprint arXiv:21 12. 02.0422, 2021.
[61] A. Simeonov, Y. Du, A. Tagliasacchi, J. B. Tenenbaum, A. Rodriguez, P. Agrawal, and V. Sitz- mann. Neural descriptor fields; Se (3)-equivariant object representations for manipulation. arXiv preprint arXiv:21
2021.
[62] Y. Wi, P. Florence, A. Zeng, and N. Fazeli. Virdo: Visio-tactile implicit representations of deformable objects. arXiv preprint arXiv:2202. 00868, 2022.
[63] D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. Van Den Driessche, Schrittwieser,
I. Antonoglou, V. Panneershelvam, M. Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):4844489, 2016.
[64] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597-1607. PMLR, 2020.
[65] K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages
2020.
[66]] A. Srinivas, M. Laskin, and P. Abbeel. Curl: Contrastive unsupervised representations for reinforcement learning. arXiv preprint arXiv:2004.044 043460044 2020.
[67] B. You, O. Arenz, Y. Chen, and J. Peters, Integrating contrastive learning with dynamic models for reinforcement learning from images. Neurocomputing, 2022.
[68] P. Sermanet, C. Lynch, Y. Chebotar, J. Hsu, E. Jang, S. Schaal, S. Levine, and G. Brain. Time-contrastive networks: Self-supervised learning from video. In 2018 IEEE International Conference Robotics and Automation (ICRA), pages 1134-114] 141. IEEE, 2018.
[69] A. Kinose, M. Okada, R. Okumura, and T. Taniguchi. Multi-view dreaming: Multi-view world model with contrastive learning. arXiv preprint arXiv:2203. /024, 2022.
[70] K. Chen, Y. Lee, and H. Soh. Multi-modal mutual information (mummi) training for robust self- supervised deep reinforcement learning. In 202/ IEEE International Conference on Robotics and Automation (ICRA), pages 4274-4280. IEEE, 2021.
[71] S. Nair A. Rajeswaran, V. Kumar, C. Finn, and A. Gupta. R3m: A universal visual representation for robot manipulation. arXiv preprint arXiv:2203. 126011 2022.
[72] A. Stooke, K. Lee, P. Abbeel, and M. Laskin. Decoupling representation learning from reinforcement learning. In International Conference on Machine Learning, pages 9870-9879. PMLR, 2021.
[73] S. Parisi, A. Rajeswaran, S. Purushwalkam, and A. Gupta. The unsurprising effectiveness of pre-trained vision models for control. arXiv preprint arXiv:2203.035800 2022.
13

---

[74] D. Yarats, A. Zhang, I. Kostrikov, B. Amos, J. Pineau, and R. Fergus. Improving sample efficiency in model-free reinforcement learning from images. arXiv preprint arXiv:9910.01741, 2019.
[75] A. Zhang, R. McAllister, R. Calandra, Y. Gal, and Levine. Learning invariant representations for reinforcement learning without reconstruction. arXiv preprint arXiv:2006. 10742, 2020.
[76] K. Zakka, A. Zeng. Florence, J. Tompson, J. Bohg, and D. Dwibedi. Xirl: Cross-embodiment inverse reinforcement learning. In Conference on Robot Learning, pages 537-546. PMLR, 2022.
[77] T. Xiao, I. Radosavovic, T. Darrell, and J. Malik. Masked visual pre-training for motor control.
arXiv preprint arXiv:2203.06175 2022.
[78] Y. Seo, K. Lee, S. James, and P. Abbeel. Reinforcement learning with action-free pre-training from videos. arXiv preprint 2rXiv:2203, 13880, 2022.
[79] T. Lesort, N. Díaz-Rodríguuz, J.-F. Goudou, and D. Filliat. State representation learning for control: An overview. Neural Nenworks, 108:379-392, 2018.
[80] M. Niemeyer and A. Geiger. Giraffe: Representing scenes as compositional generative neural feature fields. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2021.
[81] K. Stelzner, K. Kersting, and A. R. Kosiorek. Decomposing 3d scenes into objects via unsuper- vised volume segmentation. arXiv preprint arXiv:2 148, 2021.
[82] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-
778. 2016.
[83] S. Saito, Z. Huang. R. Natsume, S, Morishima, A. Kanazawa, and H. Li. Pifu: Pixel-aligned im- plicit function for high-resolution clothed human digitization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2304-2314, 2019.
[84] R. Hadsell, S. Chopra, and Y. LeCun. Dimensionality reduction by learning invariant mapping. In 2006 IEEE Computer Sociery Conference on Computer Vision and Pattern Recognition (CVPR (06), volume 2, pages 1735-1742. IEEE, 2006.
[85] A. Van den Oord, Y. Li, and O. Vinyals. Representation learning with contrastive predictive coding. arXiv e-prints, pages arXiv-1807, 2018.
[86] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1707.06347 2017.
[87] D. Driess, J.-S. Ha, M. Toussaint, and R. Tedrake. Learning models as functionals of signed- distance fields for manipulation planning. In Conference on Robot Learning (CoRL), 2021.
[88] K. He, G. Gkioxari, P. Dollár, and R. Girshick. Mask r-cnn. In Proceedings of the IEEE
international conference on computer vision, pages 2961-2969, 2017.
[89] A. Raffin, A. Hill, M. Ernestus, A. Gleave, A. Kanervisto, and N. Dormann. Stable baselines3. https:/ :/g gi thub. com/DLR DDR-RM httbleblle-- wewaabl-bbaeeeeeeee baselines3, 2019.
14

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as URL)? [Yes] See website. (b) Did you specify all the training details (e.g., data splits. hyperparameters, how they were chosen)? [Yes] See appendix.
(c) Did you report error bars (e-g., with respect the random seed after running experi- ments multiple times)? [Yes] See plots in main paper.
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See appendix.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] In the paper and in the code.
(b) Did you mention the license of the assets? [Yes] In the code.
(c) Did you include any assets either in the supplemental material or as URL? [Yes] See website.
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [Yes] In the code.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
15


Reviews:
[{"review_id": "PKBVx583a4a", "reviewer": "Reviewer_WpRR", "summary": "The authors proposed the first learned state representation through NeRF supervision for reinforcement learning purposes.\nThe proposed method shows that an offline trained latent-conditioned NeRF comes with a learned state representation that can be beneficial for training an RL policy online more sample efficiently.\nThe final results on three simulated environments are impressive compared to other learned representations and a key-points representation selected by human experts.", "questions": "* Any thoughts on how the proposed method can be generalized to real-world environments?\n", "limitations": "I didn't find a clear section discussing the limitation.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\n* the first learned state representation through NeRF supervision for reinforcement learning purposes.\n* the sample efficiency has greatly improved compared to other learned representations and a key-points representation selected by human experts.\n\nCons:\n* Only simplified simulation environments are used rather than real-world environments rendered by NeRF.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "MwEcXPyQO0j", "reviewer": "Reviewer_iBq4", "summary": "The paper presents a method that uses a Neural Radiance Field representation as a state space for a reinforcement learning algorithm.  This approach is evaluated in tasks where multiple views of the scene are available, and where the 3D information present in the NeRF representation is useful for determining shape and pose of objects.  The paper explores settings where the NeRF represents the scene globally, and also where a compositional architecture allows representing each object with its own radiance field, and shows improved performance over several baselines.\n\n## update after author response\n\nThank you to the authors for a diligent and thorough response. I have raised my score. ", "questions": "- I would have liked to see more discussion of the sensitivity of the RL performance to, say, the data available to train the NeRF model.  What sort of coverage of the scene by input views is required?\n- I am surprised at the poor performance of the keypoint method.  I would guess that appropriately chosen 3D keypoint data should represent an upper bound of possible performance for the NeRF method.  What sort of limitation does the keypoint method have here that the NeRF method outperforms it?  In the door case, for example, what information besides the keypoints would the NeRF represent?   ", "limitations": "The authors are responsible about pointing out the assumptions they make and limitations of the presented approach, and discuss these assumptions and limitations in an up-front way.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Originality\nThe major contribution of the paper is using NeRFs as a representation specifically in RL, and although this is not a surprising combination of methods, it is the first paper I am aware of that uses RL on top of the NeRF representation.  The paper cites many similar approaches which use NeRFs in dynamics models which are then used for planning (though I believe the authors may want to additionally cite [Li, Li, Sitzmann, Agrawal, and Torralba. 3D Neural Scene Representations for Visuomotor Control. CoRL 2021]).  The substitution of RL for planning is a small contribution.\n\n### Quality\nThe paper does well to address different methods for using NeRFs as a representation, both global and compositional, and has a thorough list of baselines for comparison.  The focus is on highlighting the advantages of using the 3D aware representations that come from a NeRF.  However, the tasks are specifically designed to increase the likelihood that this 3D information is useful, and discussion of how sensitive the approach is to those assumptions is not addressed.\n\n### Clarity\nThe paper is well written and the method is described with enough background to give a self-contained understanding of the approach.  There are some issues and minor typos:\n- Line 139 supevision -> supervision\n- Line 188 then -> them\n- In description of the CURL baseline, the text mentions carefully choosing views, but it is not clear how this is done.\n\n### Significance\nThis paper presents a small but interesting contribution.  As the authors mention, it is unclear what realistic situations will actually provide multiple viewpoints of the type given in the simulations here, but this is a first glimpse to suggest that NeRFs can help.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "BUBaGEbaQyv", "reviewer": "Reviewer_aWEL", "summary": "The paper propose a Neural Radiance Fields-based RL (NeRF-RL) method, which makes use of supervision from NeRF to learn state representations describing the objects in the 3D scene. Several experiments involving robotic object manipulations were conducted to demonstrate the superior performance of NeRF-RL over existing methods.", "questions": "Q1. Does NeRF-RL limit to 3D scenes? Can NeRF-RL be used to solve 2D RL tasks such as Atari games? I think it is necessary to discuss whether and how NeRF-RL applies to general RL tasks.\n\nQ2. NeRF-RL requires plenty of offline RL data to train the NeRF encoder before the policy training process. However, offline RL data are difficult to obtain for realistic 3D environments.\n\nQ3. As mentioned before, all the experimental scenes are simple and clear. Can  NeRF-RL generalize well to complicated 3D scenes with a large number of objects? \n\nQ4. Does NeRF require more offline RL data and larger neural network to learn state representations than baseline methods?", "limitations": "See the questions.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "###########Strengths###########\n\nThe paper is well written, and motivated. I believe that the experiments are roughly solid and reproducible.\n\n###########Weaknesses###########\n\nThe paper is lack of adequate analysis and discussion about the limitation of NeRF-RL. The experimental scenes are simple and clear, and have only a small number of objects. I concern about whether NeRF-RL can generalize well to complicated 3D scenes, such as VizDoom and Minecraft. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept