Toward Effective Deep Reinforcement Learning for 3D Robotic Manipulation: End-to-End Learning from Multimodal Raw Sensory Data
Authors: Samyeul Noh
Hyun Myung
Venue: Submitted to ICLR 2023 (2023)
Keywords: deep reinforcement learning
robotic manipulation
end-to-end learning
multimodal representation learning

Abstract: Sample-efficient reinforcement learning (RL) methods capable of learning directly from raw sensory data without the use of human-crafted representations would open up real-world applications in robotics and control. Recent advances in visual RL have shown that learning a latent representation together with existing RL algorithms closes the gap between state-based and image-based training. However, image-based training is still significantly sample-inefficient with respect to learning in 3D continuous control problems (for example, robotic manipulation) compared to state-based training. In this study, we propose an effective model-free off-policy RL method for 3D robotic manipulation that can be trained in an end-to-end manner from multimodal raw sensory data obtained from a vision camera and a robot's joint encoders, without the need for human-crafted representations. Notably, our method is capable of learning a latent multimodal representation and a policy in an efficient, joint, and end-to-end manner from multimodal raw sensory data. Our method, which we dub MERL: Multimodal End-to-end Reinforcement Learning, results in a simple but effective approach capable of significantly outperforming both current state-of-the-art visual RL and state-based RL methods with respect to sample efficiency, learning performance, and training stability in relation to 3D robotic manipulation tasks from DeepMind Control.

Under review as a conference paper at ICLR 2023
TOWARD EFFECTIVE DEEP REINFORCEMENT LEARN- ING FOR 3D ROBOTIC MANIPULATION: MULTIMODAL END-TO-ENT REINFORCEMENT LEARNING FROM VI- SUAL AND PROPRIOCEPTIVE FEEDBACK
Anonymous authors Paper under double-blind review
ABSTRACT
Sample-efficient reinforcement learning (RL) methods capable of learning directly from raw sensory data without the use of human-craftee representations would open up real-world applications in robotics and control. Recent advances in visual RL have shown that learning latent representation together with existing RL algorithms closes the gap between state-based and image-based training. However, image-based training still significantly sample-inefficien with respect learning in 3D continuous control problems (for example, robotic manipulation) compared to state-based training. In this study, we propose an effective model-free off- off-policy RL method for 3D robotic manipulation that can be trained in an end-to-end manner from multimodal raw sensory data obtained from a vision camera and a robot's s joint encoders. Most notably, our method is capable of learning a latent multimodal representation and policy in an efficient, joint, and end-to-end manner from multimodal raw sensory data, without the need for human human-crafted representations or prior expert demonstrations. Our method, which we dub MERL: Multimodal End-to-end Reinforcement Learning, results in simple but effective approach capable of significantly outperforming both current state-of-the-ar visual RI. and state-based RL methods with respect to sample efficiency, learning performance, and training stability in relation to 3D robotic manipulation tasks from DeepMind Control.
INTRODUCTION
Deep reinforcement learning (deep RL), the effective combination of RL and deep learning, has allowed RL methods to attain remarkable results across wide range of domains, including board
and video games under discrete action space (Mnih et al. 2015 Silver al. 2017 Vinyals et al. 2019) and robotics and control under continuous action space (Levine et al. 2016 Zhu et al. 2020 Kalashnikov al. 2021 [barz al., 2021 Kroemer al.) 2021). Many deep RL. studies use human-crafted representations, for it is commonly known that state-based training operating on a coordinate state significantly more sample-efficient than raw sensory data-based training (for example, image-based training). However, the use of human-crafted representations poses several major limitations and issues for 3D robotic manipulation: (a) human-crafted representations cannot perfectly represent the robot environment; (b) in the case of real-world applications, a separate module for environmental perception is required obtain the environment state; and (c) state-based training based on human-crafted representations is not capable of using a neural network architecture repeatedly for different tasks, even for small variations in the environment such as changes in the number, size, shape of objects.
Over the last three years, the RL community has made significant headway on these limitations and issues by significantly improving sample efficiency in image-based training (Hafner et al. 2019a b 2020; 2022; Lee et al.] 2020al Srinivas et al., 2020; Yarats et al.) 2020 2021a ble). A key insight of such studies is that the learning of better oow-dimeniional representations from image pixels is
achieved through an autoencoder (Yarats et al. 2021c), variational inference (Hafner et al. 2019a
2020
2022
[ee et al. 2020a), contrastive learning (Srinivas al. 2020 Yarats et al. 2021b).

---

Under review as a conference paper at ICLR 2023
2021a), which in turn has helped to improve sample efficiency significantly. Recently, some visual RL studies have solved 3D continuous control problems such as quadruped and humanoid locomotion tasks from the DeepMind Control (DMC) suite, which in turn has helped to bridge the gap between state-based and image-based training (Hafner et al. 20200 Yarats et a (021a). Despite such significant progress in visual RI, mage-based training still notably sample-inefficie with respect to learning in 3D continuous control problems (for example, robotic manipulation) compared to state-based training.
data augmentation Yarats et al., 2020
In this paper, we propose an effective deep RL method for 3D robotic manipulation, which we dub MERL: Multimodal End-to-end Reinforcement Learning. MERL simple but effective model-free free RL method that can be trained in an end-to-end manner from two different types of raw sensory data (RGB images and proprioception) having multimodality that differs in dimensionality and value range. Most notably, MERI. is capable of learning a latent multimodal representation and a policy in an efficient, joint, and end-to-end manner from multimodal raw sensory data, without the need for state-of-the-art visual RL and state-based RL methods, MERL provides significant improvements in sample efficiency, learning performance, and training stability in relation to five 3D robotic manipulation tasks from DMC (jaco- fjaco-reach-eee rew---racchhpae duplo. jaco-mone noo--mov--bool box. jaco-lift-- jactoo-o ruco-push--ae pet--push------ooee box-with- obstacle, and jaco-pick-and-stace of the three complex 3D humanoid locomotion tasks from DMC (humanoid-stane
representations or prior expert demonstrations. Compared to current
(Tunyasuvunakool et al. (020). In addition, MERL solves each humanoid-wald and humanoid-rum) within 5M environment steps, whereas the state-of-the-art visual RL, DrQ-v2 (Yarats et al.] 202 solves the same tasks within 30M environment steps. To the best of our knowledge, MERL is the first model-free off-policy method not only to learn latent multimodal representation and a policy in an efficient, joint, and end-to-end manner from multimodal raw sensory data, but also to show new state-of of-the-art performance by significantly outperforming both current state-of-the-art visual RL and state-based RL methods with respect to sample efficiency, learning performance, and training stability.
The main contributions of the paper can be summarized as follows: (1) the introduction of an end- to-end approach learning directly from multimodal raw sensory data for the efficient learning of policy for use in the field of 3D robotic manipulation; (2) a demonstration of the fact that the approach significantly outperforms current state-of-the-art visual RL. and state- based RL methods with respect to sample efficiency, learning performance, and training stability in relation to five 3D robotic manipulation tasks from DMC; (3) demonstration of the superiority of the approach through the performance of complex 3D humanoid locomotion tasks from DMC within 5M environment steps; and (4) the provision of a deep RL. method capable of learning a latent multimodal representation and a policy an efficient, joint, and end-to-end manner from multimodal raw sensory data, without the need for human- human-crafted crafted representations or prior expert demonstrations.
RELATED WORK
2.1 REINFORCEMENT LEARNING FOR 3D ROBOTIC MANIPULATION
Deep RL has seen widespread success across a variety of domains, including board and video games and robotics and control (Mnih al. 2015 Silver et al.] 2017 Vinyalls et al., 2019 Levine et al.) 2016 Zhu et al. 2020: Kalashnikov et al. 2021 Ibarz et al. 2021 Kroemer et al.) (021). In recent years, number of deep RL methods have been successfully applied 3D robotic manipulation tasks ranging from 'easy' (for example, reach-targer) to 'hard' (for example, assembly). Of these methods, some require ennineeeeeee components for perception, state estimation, and low-level control, for they learn from human-crafted representations (for example, (Yamada et al. 2020 Lee et al. 2019b 2021b Nam et al. 2022)) ) and such are commonly referred to as state-based RI.
Over the last three years, visual RL that learns directly from image pixels has been greatly advanced and has seen significant improvements in sample efficiency in 3D continuous control problems, which in turn has helped to bridge the gap between baste and image-based training (Hafner et al. 2019a b. 2020 2022, Srinivas et al., 2020; Yarats et al. 2020: 2021a b c) Wu et al., 2022). A key insight of such studies is that the learning of better low-dimensional representations from image pixels is achieved through various techniques such as an autoencoder, variational inference, contrastive learning, data augmentation. Currently, DrQ-v2 Yarats et al. 2021a) shows of-te-of-the-art
2

---

Under review as a conference paper at ICLR 2023
performance in visual RL studies by solving complex 3D locomotion tasks from the DMC suite (Tassa et al. 2018) such as humanoid locomotion,
previouuu previously unattained by model-free visual RL.
In this study, propose MERL, a deep RL method for 3D robotic manipulation capable of learning a latent multimodal representation and a policy in an efficient, joint, and end-to-end manner from multimodal raw sensory data, including RGB images and proprioception, without the need for human- crafted representations or prior expert demonstrations. Here, the RGB images and proprioception obtained from fixed vision camera and robot's joint encoders, respectively. Experimental results on 3D robotic manipulation tasks from DMC (Tunyasuvunakool et al. (020) show that MERL significantly improves sample efficiency compared state-based RL and DrQ-v2 Yarats al. 2021a), the current state-of oftth-rft visual RL.
2.2 REPRESENTATION LEARNING FOR MULTIMODAL INPUTS
The complementary nature of heterogeneous sensor modalities, such as vision, audio, language, haptic, range, and proprioceptive data, has previously been explored with respect to perception and
decision-making. For instance, many studies have explored the correlation between such different modalities, including the correlation between visual and auditory data in relation to speech recognition
(Yang et al., 2017) Afouras et al. 2018)) or sound source localization (Tian et al. 2018 Owens &
Elros 2018): ; the correlation between visual and haptic data for grasping (Calandra et 2018 Narita
& Kroemer 2021) or manipulation (Lee et al. 2019a 2020b 202 the correlation between visual
and tactile data for object tracking Yuu & Rodriguez, 2018 Lambert et al. (019) or shape completion
[Wang et al. 2018); and the correlation between visual and ranging data for robot navigation (Liu jet al. 2017; Cai et al. 2020). While many of these studies have contributed to improvements in perception performance through multimodal representation learning, in this study, we are interested in improving the performance RL through multimodal representation learning; in particular, the efficient joint learning of a latent multimodal representation and policy.
In addition, some neuroscience studies have proved that the interdependence and concurrency of different sensory inputs aid perception and manipulation (Edelman 1987 Lacey & Sathian 2016; et al., (017). Accordingly, we use two heterogeneous sensory data: RGB images and proprioception (for example, joint angles and velocities) efficiently learn policy for 3D robotic manipulation.
3 BACKGROUND
3.1 END-TO-END REINFORCEMENT LEARNING FROM MULTIMODAL RaW SENSORY DATA
We formulate continuous control problems on the basis of multimodal raw sensory data (that is. RGB image and proprioception) as an infinite-horizn Markov decision process (MDP) (Bellman) 1957). In such setting, we stack three consecutive prior RGB images properly approximate the environment's 's underlying state (Mnih et al.] 2013). Accordingly, such an MDP can be described as { Oinaage OFOP } denotes the state space, including the three
a tuple (O, A, P, R. 7, Po). where O = { consecutive high-dimensional RGB images, Oimag, and the low-dimensional proprioception vector, OPuP; A denotes the action space; P : O A (OO) denotes the transition dynamics that define a probability distribution over the next state given the current state and action; R: O x A (0, 1] denotes the reward function that maps the current state and action to a reward; E [0, 1) denotes discount factor; and Po A(O) denotes the probability distribution of the initial state On- The goal is to find a policy, O + ((A), that maximizes the expected discounted sum of rewards, E2 D r/r,), where og ~ Po. and Vi we have a, x(- ot), oc+1 p(- Oz, az), and r,
R(or, ..).
3.2 DEEP DETERMINISTIC POLICY GRADIENT
Deep Deterministic Policy Gradient (DDPG) Lillicrap et al. 2013)
an off-policy actor-critic RL algorithm for continuous control that concurrently learns function, Qe. and deterministic policy, Tg. For this, DDPG uses Q-learning (Watkins& & Dayan 1992) to learn Qo by minimizing the one-step Bellman residual J₂ (D) = E (0. (Q (o, a) - r, -Q 7Q=(0,11 To (T((0(1))) (0)41)))21 The
policy ze is learned by employing Deterministic Policy Gradient (DPG) (Silvrr et al. 2014 and
3

---

Under review as a conference paper at ICLR 2023
maximizing (D( = Eo,DD (o(o))) where To(o;) approximates argmax, Qa Qo(or, a). Here, 'D is replay buffer of environment transitions and Φ is an exponential moving average of the weights. DDPG is amenable to incorporate n-step returns when estimating temporal difference (TD) error beyond single step. In practice, n-step returns allow for faster reward propagation and have been previously used in policy gradient and Q-learning methods (Hessel et al. 2018 Barth-Maron et al. 2021.
2018
Li & Faisal
3.3
SOFT ACTOR-CRITIC
Soft actor-critic (SAC) (Haarnoja al., 2018a b) is off-policy actor-critic RL algorithm for continuous control problems that concurrently learns a Q-function, Qs a stochastic policy, Ta, and temperature. a, the basis of maximum- maximum-entrropy framework. For this, SAC performs soft policy evaluation and improvement steps at each iteration, with the goal of maximizing a trade-off between expected return and entropy using the following 7-discounted
objective:
a((,,)))> R(o,, ax) + a ( (. oz))], where a is the temperature that balances between optimizing for the reward and for the stochasticity of the policy.
3.4 IMAGE AUGMENTATION IN REINFORCEMENT LEARNING
Image augmentation techniques have been commonly used in computer vision research not only to avoid overfitting, but also to achieve state-of of wiftthe-ar art performance (Shorten & Khoshgoftaar (019)). Recently, in visual RI, DrQ-v2 Yarats et al. 2021a), which is improvement DrQ Yaratss et al.) 2020), provides a new state-of-the-an performance in relation to 3D locomotion tasks from the DMC suite (Tassa et 2018 by adding image augmentation in the form of random shifts. The use of such image augmentation techniques is now an essential factor in achieving a new state-of-the-ar
performance in image-based training (Srinivas et al., 2020 Schwarzer al. 2020 Yarats et al. 2020 2021a Hansen & Wang] (021).
4 MERL: MULTIMODAL END-TO-END REINFORCEMENT LEARNING
In this section, we describe MERL,, an effective deep RI method for 3D robotic manipulation; specifically, a model-free off-policy actor-critic RL algorithm that can be trained in an end-to-end manner from multimodal raw sensory data ttigh-dimensional RGB images and a low-dimensional proprioception vector). Figure T Tillustrates our method (MERL), which learns a latent multimodal representation and policy in an efficient, joint, and end-to-end manner from multimodal raw sensory data, without the need for human-crafie representations or prior expert demonstrations. Here, MERL uses image augmentation technique and a convolutional neural network (CNN) to encode the images; a multi-layer perception (MLP) to encode the proprioception; an MI. and layer normalization with scaling under a decoupled architecture learn the latent multimodal representation from the encoded visual and proprioceptive representations; and actor-critic networks with layer normalization applied to learn the policy.
4.1 MULTIMODAL REPRESENTATION LEARNING FOR 3D ROBOTIC MANIPULATION
Replay Buffer Unlike in visual RL such DrQ-v2 state-hased RL such as DDPG and SAC, our method stores transition (or, az Tt, 0t+1), including multimodal observations, in the replay buffer, for we use two different types of raw sensory data (RGB images and proprioception). Regarding the transition, o, denotes a multimodal observation (three consecutive prior RGB images, o, imuge and proprioception vector, o!"'P) at time (; a, denotes the robot's 's action (for example, joint torques) at time t; 7, denotes the reward at time t; "tt denotes a discount factor at time t: and ot+1 denotes a image
multimodal observation (three consecutive prior RGB images, 0,+1 and a proprioception vector, D at time t ++1 1. Regarding the proprioception vector, it can include the joint positions, velocities, or torques of the robotic arm and hand the position or rotation of the robot's end-effector, or all of the above.
Image Augmentation A deep neural network is a powerful tool with which to learn a latent representation from high-dimensional data; however, such tool requires substantial amount of
4

---

Under review as a conference paper at ICLR 2023
reprosentation
butter
oProp
mage sugmentation
gimsage
image encoder
image
muttimodal
fusion
cnticc
Berttie
BGB images
forward
v(()))
(())
multimodal
reprosettation
s(a
proprioception
encoder
propp
multmodaa
Tusion
actor
Luctor
peopeioception
Figure I: Our method (MERL) learns a latent multimodal representation and policy in an efficient, joint, and end-to-end manner from multimodal raw sensory data (high-dimensional RGB images and a low-dimensional proprioception vector), without the need for human-craftee representations or prior expert demonstrations, A new state-of-the-at performance comes from careful configuration of multimodal representation learning combined with augmantedd RL.
training data. The computer vision community has made great headway regarding this challenge by using various image augmentation techniques (Shorten & Khoshgoftaar 2019). In this study, apply random shift image augmentation technique to image observations of the environment, as in DrQ-v2. Specifically, we pad each side of x4 84 image with four repeating boundary pixels. Next, we replace each repeated pixel value with the average of the four nearest pixel values (that is, we apply bilinear interpolation to the shifted image). Finally, we select random 84 x crop, yielding the original image shifted by 44 pixels.
Image Encoder The augmented images, gimage, obtained by means of the random shift augmenta- tion technique, are embedded into a low-dimensional latent vector, y image by means of a CNN encoder. Regarding the CNN encoder, we use the same architecture in DrQ Yarats al. 2020), which was first introduced in SAC+AE (Yrrats et al. (021c). This is due to the fact that DrQ has successfully shown range of encoders of differing architecture and capacity perform equally well on continu- ous control problems in the case where a random shift augmentation technique is applied to original image observations. The encoding process can be concisely formulated as image fş (aug(o (mige) where fs denotes the image encoder, aug denotes the random shift augmentation, and image denotes the original image observation.
simple MLP. The encod- ing process can be concisely summarized as prop 9c (OPuP), where ge denotes the proprioception encoder and "op denotes the original proprioception observation. Note here that the best perfor- mance is achieved when the proprioception encoder is set as the identity encoder. This due to the fact that the proprioception is already well-encoded by itself (see Appendix C.I for more details).
Proprioception Encoder Regarding the proprioception encoder,
Multimodal Fusion MERL uses different types raw sensory data as input: the RGB images from a fixed vision camera and the proprioception from the joint encoders of the robot arm and hand. The heterogeneous nature of this data requires to use domain-specific encoders obtain the unique characteristics of each modality, which we then fuse into single latent multimodal representation vector, mm, of dimension d.
For visual feedback, we stack three consecutive prior RGB images, apply the random shift augmenta- tion technique to the stacked images, normalize the resulting augmented images, and use a CNN to encode the normalized images into latent representation vector, pimage For proprioception, an MLP to encode the proprioception into a latent representation vector, prop, For multimodal fusion, an MLP and layer normalization with scaling to produce diw-dimensional scaled latent vector, of which the range is (-1, 1) for each modality. Each scaled latent vector is concatenated to produce single latent multimodal representation vector, mm of dimension d. The multimodal fusion
use
5

---

Under review as a conference paper at ICLR 2023
process can be succinetly summarized pm ha rPPP) where rimage fe (aug( ( m
denotes the encoded visual representation from the RGB images and rrmp = 9ę (o denotes the encoded proprioceptive representation from the proprioception. Note here that apply decoupled architecture to multimodal fusion to improve the performance of actor-critic learning; that is, the decoupled architecture uses two different latent multimodal representation vectors, ymm_ and pimn_ to train the actor and critic networks separately. As result, each of the representations are learned to better optimize the losses of the actor and critic, respectively.
4.2 MODEL--RRE ACTOR-CRITIC REINFOR LEARNING
Although SAC has been the de facto off-policy RL algorithm for many RL methods over the past few years, it is prone to suffering from policy entropy collapse. Recently, DrQ-v2 showed that using DDPG instead of SAC as learning algorithm leads to better performance (that is, more robustness and more stability) relation to the tasks from the DMC suite, including 3D continuous control problems. For this reason, we opt for DDPG policy from a latent multimodal representation. As in DrQ-v2, we also augment DDPG with n-step returns to estimate TD error: the augmentation leads faster learning progress by accelerating reward propagation. Here, we opt for augmenting DDPG with 3-step returns. For the augmentation, sample a mini-batch of transitions, addition, we use clipped double Q-learning to reduce over-estimation bias in the target value. For the learning, we train two Q-functions, Q², and Qøz by minimizing the following two losses:
backbone actor-critic RL algorithm to learn
(0t, at, Fes++n-1 Ot+nn). from the replay buffer, DD In
Lo( Veritie C, D) = (o,,, Fr.taa 1. [1Q (rt (rimm a.)) - -y)
Vk E (1,2), (1)
where TD target, y, is defined as follows:
min Qš (r.mm au-n), k=1.2 TTH +
where r. r_mm_ hymm (au(au,(o, (
(c((0)))) (o..or)) Titn heam (fe(aug(o) image )) gc (o ( )),
Here, E represents exploration noise sampled from clip(A((O, o²), -c, c), which is similar to TD3 (Fujimoto et al. 2018); Φ and de represent the exponential moving averages of the weights for the Q target networks; and Meritic represents the multimodal fusion weights for the critic. The final version of MERL uses decoupled multimodal fusion networks for the actor and critic, hy and he respectively. We note here that we use the most recent weights of & for the image encoder, for the proprioception encoder, and 1 for the multimodal fusion, to encode Og and Octn- Finally, we train the deterministic actor, no, using DPG by maximizing the expected returns, as follows:
a, = (T( + e, and atta = ( (ortt (+tt))
man
(0, Vactor; D) EO,ND
min Qs (r-mimss an) 1=1,2
22)
where r_mmm. he (fe (aug(o) immage)) 96 (o, )) and a; To(rrmm_)) + Here, vlactor represents the multimodal fusion weights for the actor. We note here that we prevent the actor's gradients from updating the image and proprioception encoders, for SAC+AE found that this in fact hinders the agent's 's performance.
EXPERIMENTS
In this section, we provide an empirical evaluation of MERL in relation to five 3D robotic manipulation tasks from DMC (jaco-reach-dupg) duplo, faco-move-bar box. jaco-lift-bor jaco-push- moush-box-with-aae wwwtt-wsttccheeee and juco-pick-and-stee
(Tunyasuvunakool et al. 2020). We first present a comparison to prior methods, including current state-of-the-an visual RL and state-based RI. methods, with respect to sample efficiency, learning performance, and training stability. We then present an ablation study that guided the final version of MERL.
5.1 SETUP
Environment Setup We consider an environment for 3D robotic manipulation that provides multimodal raw sensory data observation. In this context, consider learning directly from

---

Under review as a conference paper at ICLR 2023
jaco Reach Duplo
jaco Move Box
jaco Lift Box
200
200
Episode Nefur
Epiode Naturn
-50
200
200
ogent Di0i
FORT
Environment (x:19)
Environment
Environment
Jaco Push Box with Obstacle
Jaco Pick and
200
300
335
Eplode Raturn
350
100
Episade
poPn Stat
6.0
0.5
1.0
3.5
2.0
2.5
3.0
0.0
0.5
1.0
1.5
2.0
3.0
Environment Stops (x:17))
Environment Steps (-10P)
Figure 2: We compare MERL to both current state-of-the-art visual RL and state-based RL methods in relation to five 3D robotic manipulation tasks from DMC (jaco-recc--dulla, jaco-move-bee jaco-lift-hox, powuss-boowwiihhbsssteee and paco-pick-and-seee MERL demonstrates superior sample efficiency, significantly outperforms leading model-free RL baselines, and shows more stable training performance.
multimodal observations, including RGB images and proprioception. Here, the RGB images represent stacks of three consecutive RGB images of size 84x84, 84, stacked along the channel dimension to enable inference of dynamic information such as velocity and acceleration, and the proprioception represents joint angles and velocities for the robotic arm and hand. The action space bounded by 1-1
Task Setup We design a set of five 3D robotic manipulation tasks from DMC (juco-rachh-duple jaco-move-box. bacobbo....aaoijjpp. jaco-lift-ox fitt-box,jacc--aee bush--oo--wtt--bbeeeesee different types of raw sensory data (that is, RGB images obtained from a fixed vision camera and proprioception obtained from robot's joint encoders) observation, where task success requires joint reasoning over visual and proprioceptive feedback. In addition, for each task, we randomize the configuration of the initial positions of the robot and the box at the beginning of each episode, during both training and testing, to enhance the robustness and generalization of the model.
and jaco-pick-and-stane
to include two
Reward Design We adopt a staged, structured, and nulti-component reward function to guide the RL algorithm. which simplifies the challenge of exploration and leads to effective policy learning (Lee et al.) 20206) Yu et al.) 2020). The reward function, R, is combination of reaching reward, pushing reward, vertical reward, floating reward, and lifting reward, subsets thereof for simpler tasks that only include reaching or pushing (see Appendix B. 2 for more details), With this design, the reward is bounded by [0, 1] per timestep.
5.2 COMPARISON TO PRIOR METHODS
Baselines We compare our method to DrQ-v2 Yarats et al.) 2021a), which currently provides the best performance for 3D continuous control problems within model-free visual RL methods, with respect to sample efficiency, learning performance. and training stability. We also compare it to the state-based RL algorithms DDPG (Fujimoto et al. 2018) and SAC (Haamoja et al.) (2018)), which provide upper-bound performance with respect to sample efficiency for many RI. studies.
Evaluation We present our experimental results for five 3D robotic manipulation tasks from DMC (jaco-reach-duple jaco-move-box, jaco-lift-box jaco-push- wwwpppuo-s--iiiiipeeee www.-oootwoosstaceeceoooooo and jaco tick-pick-and- stack), in Figure Our empirical study in Figure reveals that MERL considerably outperforms both the state-based RI. algorithms (DDPG and SAC) and the state-of-the-art visual RI. algorithm
7

---

Under review as a conference paper at ICLR 2023
(DrQ-v2) with respect to sample efficiency, learning performance, and training stability. This suggests that a coordinate state used in state- based RL is insufficient to represent the robot environment and that visual RL requires significantly more data learn a latent representation, both of which in turn suggest that learning from multimodal raw sensory data is required to achieve the best performance with respect to sample efficiency and learning performance in relation to 3D robotic manipulation. Notably, MERL's advantage is more pronounced in the case of more difficult 3D robotic manipulation tasks (for example, jaco-move-box, baccoo-- jaco-lift-boxx
wwwp--uoo---i--ooheeeeee and miick-and-stedd where exploration is more challenging. Most notably, to the best of our knowledge, MERL is the first model-free RL method for 3D robotic manipulation capable of learning latent multimodal representation and policy in an efficient, joint, and end-to-end manner from multimodal raw sensory data, without the need for human-crafted representations or prior expert demonstrations, while showing a new state-of-the-art performance.
5.3 ABLATION STUDY
We conduct an ablation study that leads to the final version of MERL. Our findings are summarized in Figure 3 and detailed below.
Decoupled Multimodal Fusion We examine decision to use decoupled architecture for multimodal fusion compared to using a conventional shared architecture primarily used in multimodal representation learning (Lee al. 2019a 2020b). We hypothesize that the performance of actor critic learning could be improved by feeding multiple representations into the actor and critic separately and learning these representations in such a manner so as better optimize the losses of the actor and critic, respectively. We conduct some experiments on two different architectures for multimodal fusion: decoupled and shared. As shown in Figure 3a| the decoupled architecture provides better performance in relation to the three 3D robotic manipulation tasks from DMC jjaco-reach-dug duplo, jaco-move-bor and jaco-lift- lift-box) compared to the conventional shared architecture. This suggests that the decoupled architecture for multimodal fusion has a greater impact on actor-critic learning compared to the conventional shared architecture. The greater impact may be due to the fact that in the decoupled architecture latent multimodal representations fed into the actor are separately learned to maximize the learning performance of the actor, and likewise in the case of the critic. To the best of our knowledge. MERL is the first RL method apply decoupled architecture to critic learning. We also found that, in the case of the shared architecture for multimodal fusion, updating the multimodal fusion with the actor's gradients greatly hinders the agent's 's performance.
Layer Normalization We examine the efficacy of layer normalization (Ba et al. (016)) commonly used in transformer architectures /Vaswani et al. 2017: Wang et al. 2019). We hypothesize that the application of layer normalization our multimodal fusion will play a crucial role in controlling the gradient scales, as in a transformer architecture, and will lead to favorable sample efficiency. We conduct some experiments for two different cases: multimodal fusion with and without layer normal- ization applied. As shown in Figure 3b] the former case produces significantly better performance in relation to the three 3D robotic manipulation tasks from DMC (jaco-reach-duple jaco-move-bon and jaco-lift-box) than the latter case. This suggests that any increase in speed or stability of the RL process is due to the use of normalized multimodal representations (via layer normalization) in actor- critic networks. In light of this, we further proceed to study whether the use of layer normalization in other baselines would have a similar effect on performance; indeed, we found that the use of layer normalization in RL algorithms is crucial to the study of RI. (see Appendix E. Note that the final version of MERL uses layer normalization in both multimodal fusion and actor critic networks (see Appendix C.I In addition, we perform scaling via tanh before feeding the representation into the critic network in order to match the latent multimodal representation and action to the same scale. We note that the use of layer normalization in RL algorithms has received little attention in the RL community.
Multimodal Representation Dimensions We investigate the required level of compactness of the latent multimodal representation for 3D robotic manipulation by changing the dimensionality of the representation. We hypothesize that more compact representation may increase the tractability of RL yet capture less information for 3D robotic manipulation tasks. We perform several experiments at different representation dimensions: dim 64, 1 128, and 256. As shown in Figure 3cl dim 128
8

---

Under review as a conference paper at ICLR 2023
jeco Reach Duplo
jaco Move Box
Jaco Lift Box
200
202
Eppcade Rutum
1:0
Epccode
100
203
00
6-4
0.5
100
2.0
2.5
Environmmnt Steps (-:1P)
Ensironmant Steps (-UP))
Environment Steps (xor))
(a) Ablation study on decoupled multimodal fusion
Jeco Reach Duplo
Move Box
Jaco Lift Box
202
Epioooe Retur
Edicode Esktodie Ruturn
1.0
323
100
100
000
30-
30
0-
Layelbere
1.0
Environment miment Steps (-11P)
Erxtro mantt Steps [-IF))
Ensironment Stops (xor)
(b) Ablation study on layer normalization
Jeco Reach Duplo
Jaco Move Box
Jaco Lift Box
Eppcodo Return
Eppcoe
002
40
0.
0.2
66
0.2
0.5
1.5
100
0.5
0.5
1.0
2.0
25
30
Environment Steps 1x10Ps
Enxiro ment Stops (=1)
Envinonment Steps (vcof)
(c) Ablation study multimodal representation dimensions
Jaco Reach Duplo
Jaco Move Box
Jaco Lift Box
303
Spdodd Ret.um
150
353
500
100
303
Envtronment Steps 1-10P1
Enstro mmett Steps Extof)
Enviromment Steps (Coor)
(d) Ablation study on exploration noise
Figure 3: An ablation study that led us the final version of MERL: (a) observe that the decoupled architecture for multimodal fusion (red) provides better performance compared to the conventional shared architecture (blue); (b) we observe that the use of layer normalization in multimodal fusion (red) results in significant performance gain in sample efficiency; (c) we observe that dim 128 for multimodal representation (red) provides the best performance for the given 3D robotic manipulation tasks; and (d) we observe that the fixed stochastic exploration (red) provides better performance for 3D robotic manipulation compared to the scheduled exploration (blue) used in DrQ-v2.
provides the best performance in relation to the three 3D robotic manipulation tasks from DMC (juco-reach-uuplo, jaco-move-box, and jaco-lifl-box). In the case of dim = 64, less information for the tasks was captured, and in the case of dim = 256, the learning performance was degraded owing to an increase in the latent state space used in policy learning.
Exploration Noise We investigate the efficacy of the scheduled exploration noise used in DrQ-v2, where the scheduled exploration noise has different levels of exploration at different stages of learning. We hypothesize that scheduled exploration noise will be more helpful to improve sample efficiency
9

---

Under review as a conference paper at ICLR 2023
by having more stochastic exploration the beginning of training compared fixed exploration noise. We perform some experiments at two different levels of exploration noise: decaying and fixed. As shown in Figure 3d contrary to the results of the DrQ-v2 study (Yarats al., 202la). having a fixed level of exploration noise provides better learning performance in relation to the three 31) robotic manipulation tasks from DMC (jaco-reach-duplo, jaco-move-box and jaco-lift-box) than having scheduled level of exploration noise. This suggests that the data accumulated through having stochastic exploration at the beginning of training degrades the learning performance, especially in the case of the task jaco-lift-box Accordingly, we need to set different levels of exploration noise depending on the characteristics of given task. For example, for task that requires lot of stochastic exploration at the beginning of training, such as humanoid locomotion, scheduled exploration noise is more effective, whereas, in the case of 3D robotic manipulation tasks, a fixed stochastic exploration is more effective.
6 CONCLUSION
In this study, we have proposed MERI., conceptually simple but effective model-free off-policy RL method for 3D robotic manipulation capable of learning a latent multimodal representation and policy in an efficient, joint, and end-to-end manner from multimodal raw sensory data (RGB images and proprioception) without the need for maman-crafted representations or prior expert demonstrations. Our experimental results show that MERL significantly outperforms both current state-of-the-art visual RL and state-based RL methods with respect to sample efficiency, learning performance, and training stability in relation to 3D continuous control problems, including robotic manipulation and locomotion tasks from DMC. Most notably, MERL is able to solve the primitive skills reach, push, move, lift, and pick-and-place with respect the chosen 3D robotic manipulation tasks from DMC, within 1M environment steps. Also, MERL is able to solve each of the three complex 3D humanoid locomotion tasks from DMC (humanoid-stann rum) these tasks being among the most difficult of 3D continuous control problems 5M environment steps and without the need for human-crafted representations or prior expert demonstrations. Such experimental results demonstrate that MERL is one of the most effective methods for solving 3D continuous control problems, including robotic manipulation and locomotion tasks, with respect to learning a policy in end-to-end manner, without the need for crrffed representations or prior expert demonstrations. the first effective RL method for 3D robotic manipulation capable of learning latent multimodal representation and a policy in an efficient, joint, and end-to-end manner from multimodal raw sensory data, without the need for human-crafted representations showing new state-of-the-arr performance. Furthermore, to the best of our knowledge, MERL is the first RL method to apply a decoupled architecture actor-critic critic learning. We hope this work will serve as a guideline for future multimodal end-to-end end RL research.
humanoid-walk, and humanoid- within
addition, to the best of our knowledge, MERL is
prior expert demonstrations, while
In future work, we will seek to use more sensors (for example, depth and haptic sensors) of differing modalities, which will lead to more dexterous manipulation. and to test and validate MERI. in real-world robotic environment through sim-to-real. In addition, we hope to explore the use of transformer architectures and object-centric learning to obtain a better representation.
10

---

Under review as a conference paper at ICLR 2023
REFERENCES
Triantafyllos Afouras, Joon Son Chung, Andrew Senior, Oriol Vinyals, and Andrew Zisserman. Deep audio-visual speech recognition. IEEE transactions on pattern analysis and machine intelligence. 2018.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprini arXiv:1607.06450, 2016.
Gabriel Barth-Maron, Matthew W Hoffman, David Budden, Will Dabney, Dan Horgan, Dhruva Tb, Alistair Muldal, Nicolas Heess, and Timothy Lillicrap. Distributed distributional deterministic policy gradients. arXiv preprint 180i 0.6677. 2018.
Richard Bellman. markovian decision process. Journal of mathematics and mechanics, pp. 679-684, 1957.
Jeannette Bohg, Karol Hausman, Bharath Sankaran, Oliver Brock, Danica Kragic, Stefan Schaal, and Gaurav S Sukhatme. Interactive perception: Leveraging action perception and perception in action. IEEE Transactions on Robotics, 33(6):1273 1291, 2017.
Peide Cai, Sukai Wang, Yuxiang Sun, and Ming Liu. Probabilistic end-to-end vehicle navigation in complex dynamic environments with multimodal sensor fusion. IEEE Robotics and Automation Letters, 5(3):4218-42)) 4224, 2020.
Roberto Calandra, Andrew Owens, Dinesh Jayaraman, Justin Lin, Wenzhen Yuan, Jitendra Malik, Edward Adelson, and Sergey Levine. More than a feeling: Learning to grasp and regrasp using vision and touch. IEEE Robolics and Automation Letters, 3(4):3330-33077 2018 8.
Gerald M Edelman. Neural Darwinism: The theory of neuronal group selection. Basic books, 1987.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor- critic methods. In International conference on machine learning, pp. 1587-1596. PMLR, 2018.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pp. 1861-1870 PMIR, 2018a.
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and applications. arXiv preprint arXiv: 259905, 2018b.
Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. arXiv preprint arXiv:1912.01603 2019a.
Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In International conference on machine learning, pp. 2555-2565, PMLR, 2019b.
Danijar Hafner, Timothy Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with discrete world models. arXiv preprint arXiv:2010.02199, 2020.
Danijar Hafner, Kuang-Huei Lee, Ian Fischer, and Pieter Abbeel. Deep hierarchical planning from pixels. arXiv preprint arXiv:2206.044 04114, 2022.
Nicklas Hansen and Xiaolong Wang. Generalization in reinforcement learning by soft data aug- mentation. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pp. 13611-13617 IEEE, 2021.
Mattco Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski. Will Dabney, Dan Horgan, Bilal Piot. Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In Thirty-second AAAI conference artificial intelligence, 2018.
Julian Ibarz, Jie Tan, Chelsea Finn, Mrinal Kalakrishnan, Peter Pastor, and Sergey Levine. How to train your robot with deep reinforcement learning: lessons we have learned. The International Journal of Robotics Research, 40(4-5):688-721, 2021.

---

Under review as a conference paper at ICLR 2023
Dmitry Kalashnikov, Jacob Varley, Yevgen Chebotar, Benjamin Swanson, Rico Jonschkowski, Chelsea Finn, Sergey Levine, and Karol Hausman. Mt-opt: Continuous multi-task robotic rein- forcement learning at scale. arXiv preprint arXhv:110..08212 2021.
Oliver Kroemer, Scott Niekum, and George Konidaris. A review robot learning for manipulation: Challenges, representations, and algorithms. The Journal of Machine Learning Research, 22(1): 1395-1476, 2021.
Simon Lacey and Krish Sathian. Crossmodal and multisensory interactions between vision and touch. In Scholarpedia of Touch, pp. 301-315. Springer, 2016.
Alexander Sasha Lambert, Mustafa Mukadam, Balakumar Sundaralingam, Nathan Ratliff, Byron Boots, and Dieter Fox. Joint inference of kinematic and force trajectories with visuo-tactile sensing. In 2019 International Conference Robotics and Automation (ICRA), pp. 3165-3171. IEEE, 2019.
Alex X Lee, Anusha Nagabandi, Picter Abbeel. and Sergey Levine. Stochastic latent actor-critic: Deep reinforcement learning with a latent variable model. Advances in Neural Information Processing Systems, 33:741-752, 2020a.
Michelle A Lee, Yuke Zhu, Krishnan Srinivasan, Parth Shah, Silvio Savarese, Li Fei-Fei, Animesh Garg, and Jeannette Bohg. Making sense of vision and touch: Self-supervised learning of multimodal representations for contact-rich tasks. In 2019 International Conference on Robotics and Automation (ICRA), pp. 8943-8950. IEEE, 2019a.
Michelle A Lee, Yuke Zhu, Peter Zachares, Matthew Tan, Krishnan Srinivasan, Silvio Savarese, Li Fei-Fei, Animesh Garg, and Jeannette Bohg. Making sense of vision and touch: Learning multimodal representations for contact-rich tasks. IEEE Transactions on Robotics, 36(3):882-596, 2020b.
Michelle A Lee, Matthew Tan, Yuke Zhu, and Jeannette Bohg. Detect, reject, correct: Crossmodal compensation of corrupted sensors. In 2021 IEEE International Conference on Roboties and Automation (ICRA), pp. 909-916. IEEE, 2021a.
Youngwoon Lee, Jingyun Yang, and Joseph J Lim. Learning to coordinate manipulation skills via skill behavior diversification. In International conference on learning representations, 2019b.
Youngwoon Lee, Edward S Hu, and Joseph J Lim. Ikea furniture assembly environment for long- horizon complex manipulation tasks. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pp. 6343-6349. IEEE, 2021b.
Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuomotor policies. The Journal of Machine Learning Research, 17(1):1334-133, 2016.
Luchen Li and A Aldo Faisal. Bayesian distributional policy gradients. Proceedings of the AAAI Conference Artificial Intelligence, volume 35, pp. 8429 8437, 2021.
Timothy Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint aroiv:/500.0220 02971, 2015.
Guan- Horng Liu, Avinash Siravuru, Sai Prabhakar, Manuela Veloso, and George Kantor. Learning end-to-end multimodal sensor policies for autonomous navigation. In Conference on Robot Learning, pp. 249-261. PMLR, 2017.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint 132.5213 2013.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller. Andreas Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 51 518/7540::52-5333 533,, 2015.
12

---

Under review as a conference paper at ICLR 2023
Taewook Nam, Shao-Hua Sun, Karl Pertsch, Sung Ju Hwang, and Joseph J Lim. Skill-based retf-reiinorrement learning. arXiv preprint arXiv:2204.11822 2022.
Tetsuya Narita and Oliver Kroemer. Policy blending and recombination for multimodal contact-rich tasks. IEEE Robotics and Automation Letters, 6(2):2721-2728, 2021.
Andrew Owens and Alexei A Efros. Audio-visual scene analysis with self-supervised multisensory features. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 631-648, 2018.
Andrew M Saxe, James L McClelland, and Surya Ganguli. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. arXiv preprini arXiv: 1312.6120, 2013.
Max Schwarzer, Ankesh Anand, Rishab Goel, R Devon Hjelm, Aaron Courville, and Philip Bach- man. Data-efficient reinforcement learning with self-predictive representations. arXiv preprini arXiv:2007.05929, 2020.
Connor Shorten and Taghi M Khoshgoftaar. A survey image data augmentation for deep learning. Journal of big data, 6(1):1-48, 2019.
David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deterministic policy gradient algorithms. International conference on machine learning, pp. 387-395, PMLR, 2014.
David Silver, Julian Schrittwieser Karen Simonyan, loannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature, 550(7676)3344359) 2017.
Aravind Srinivas, Michael Laskin, and Pieter Abbeel. Curl: Contrastive unsupervised representations for reinforcement learning. arXiv preprint arXiv:2004.0411 041160 2020.
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden. Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint arXiv: vrXivv/801.coo (801100690, 2018
Yapeng Tian, Jing Shi. Bochen Li, Zhiyao Duan, and Chenliang Xu. Audio-visual event localization in unconstrained videos. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 247-263, 2018.
Saran Tunyasuvunakool, Alistair Muldal, Yotam Doron, Siqi Liu, Steven Bohez, Josh Merel, Tom Erez, Timothy Lillicrap, Nicolas Heess, and Yuval Tassa. dm_ mon_ootool: Software and tasks for continuous control. Software Impacts, 6: 100022, nov 2020. doi: 10. 10.10166j /016//.simma.20200 100022.
/ttttps://do..rgg/ 1ttp:/000.11112F_.11 simpa..iipp...............m..... 2020 100022
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Oriol Vinyals, Igor Babuschkin, Wojciech M Czameeki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David Choi, Richard Powell, Timo Ewalds, Petko Georgiev, al. Grandmaster level in starcraft ii using multi-agent reinforeement learning. Nature, 575(7782):350-333 3544444,, 2019.
Qiang Wang, Bei Li, Tong Xiao, Jingbo Zhu, Changliang Li, Derek Wong, and Lidia S Chao. Learning deep transformer models for machine translation. arXiv preprint 1xX.v:906.01788 2019.
Shaoxiong Wang. Jiajun Wu, Xingyuan Sun, Wenzhen Yuan, William T Freeman, Joshua B Tenen- baum, and Edward H Adelson. 3d shape perception from monocular vision, touch, and shape priors. In 2018 JEFF/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 1606-1613. IEFF, 2018
Christopher JCH Watkins and Peter Dayan. Q-learning. Machine learning, 8(3):279-292, 1992.
13

---

Under review as a conference paper at ICLR 2023
Philipp Wu, Alejandro Escontrela, Danijar Hafner, Ken Goldberg, and Pieter Abbeel. Daydreamer: World models for physical robot learning. arXiv preprint arXiv:2206.144 2022.
Jun Yamada, Youngwoon Lee, Gautam Salhotra, Karl Pertsch, Max Pflueger, Gaurav S Sukhatme, Joseph J Lim, and Peter Englert. Motion planner augmented reinforcement learning for robot manipulation in obstructed environments. arXiv preprint arXiv:2010. 1940, 2020.
Xitong Yang, Palghat Ramesh, Radha Chitta, Sriganesh Madhvanath, Edgar A Bernal, and Jiebo Luo. Deep multimodal representation learning from temporal data. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5447-5455, 2017.
Denis Yarats, Ilya Kostrikov, and Rob Fergus. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. In International Conference on Learning Representations, 2020.
Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Mastering visual continuous control: Improved data-augmented reinforcement learning. arXiv preprint arXiv:2107.09644 2021a.
Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Reinforcement learning with prototypical representations. In International Conference on Machine Learning, pp. 11920-11931. PMLR, 2021b.
Denis Yarats, Amy Zhang, Ilya Kostrikov, Brandon Amos, Joelle Pineau, and Rob Fergus. Improving sample efficiency in model-free reinforcement learning from images. In Proceedings of the AAAI Conference Artificial Intelligence, volume 35, pp- 10674-10681, 2021c..
Kuan-Ting Yu and Alberto Rodriguez. Realtime state estimation with tactile and visual sensing for inserting suction-held object. In 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 1628-1635. IEEE, 2018
Tianhe Yu, Deirdre Quillen, Zhampeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta- world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on robot learning. pp. 1094-1100. PMLR, 2020.
Henry Zhu, Justin Yu, Abhishek Gupta, Dhruv Shah, Kristian Hartikainen, Avi Singh, Vikash Kumar, and Sergey Levine. The ingredients of real-world robotic reinforcement learning, arXiv preprint 1rXiv:2004.125700 2020.
14

---

Under review as a conference paper at ICLR 2023
A ALGORITHM DETAILS
Algorithm MERL:
Multimodal end-to-end deep RL
Parametric networks: Image encoder fg. proprioception encoder 9ç, multimodal fusion for critic Neme multimodal fusion for actor / actor ko, critic Qs
Hyper-parameters: Training steps mini-batch size B, learning rate a, target update rate т, standard deviation o, clip value
Image augmentations: Random shift aug for t +- I. do
a, - The (f image )0 (, + and NN0, o²)
0++1 ~ P(. ot, 0+)
D (o.. a, R(o,, at), 1 UD
UPDATECRITIC(D)
UPDATEACTOR(D)
end for
procedure UPDATECRITIC(2)
{(o,, a2, Te, D
mmme Tttt &- herim Laugau image )), g6 (0()))))))) hymm (fe (fg(aug(O image ))
(o(
r, mm_a TTtt ho (fe (aug(o, o, image (0, mmop) ho (aug(a )) 9 (0( of4m))
image
ar- & TT(r + and e clip(N (3), -c,
Compute Los, K.s and venns
Ç f { VV Arrme E.S + Fonseen Ç - Ç - a VV (f R.5 + Los.wenn....
Veritic + v'rittic a omme &ii + &.c
Φx Ob aV 6e Cram Wk [1,2]
δe + - -1 + Thk Vk E 21
end procedure
procedure UPDATEACTOR(2)
{(o,)} D
rumum himen (fe (aug(, (magge) 9) (o.To))
rt mma hirm (Se (aug(o, image immgee
()
at t To(mmmaa) + and ~ clip(NV(0002, -c,
Compute Lo.vane
Vactor < 4/acturr aV Wamm L. abanor
4- 0 - a VoLo, Mann
end procedure
15

---

Under review as a conference paper at ICLR 2023
B TASK DETAILS
In this section, we describe details of the five 3D robotic manipulation tasks from DMC (jaco-reach- duplo, jaco-move- boxve jaco-lift-bo saco-push-bor sethh-oo-witthbbtteeee and jaco-pick-and-stiee with respect to task descriptions, reward design, and task visualizations.
B.1 TASK DESCRIPTIONS
We design a set of five 3D robotic manipulation tasks from DMC (jaco-recch-duplo, jaco-move- mico-lift-bxx, sero--psh--bo--wih--ooe
(Tunyasuvunakod al. (020) to include two different types of raw sensory data (that is, RGB images obtained from a fixed vision camera and proprioception obtained from robot's 's joint encoders) as observation, where task success requires joint reasoning over visual and proprioceptive feedback. For each task, we randomize the configuration the initial positions of the robot and the box at the beginning of each episode, during both training and testing, to enhance the robustness and generalization of the model.
and jaco-pick-and-stan
In the case of the task jaco-reach- reporc-uu--upl,, the robot (more precisely, the center point of the robot's end-effector) is required to reach duplo randomly placed on a workspace. The task is considered successful when the robot reaches within 5 cm of the duplo.
In the case of the task jaco-move-ber the robot is required to move box randomly placed on workspace to specific target position. The task considered successful when the box reaches within 1 cm of the target position.
In the case of the task jaco-lift-bar lif--box,, the robot is required to lift a box randomly placed on workspace to specific target position. The task is considered successful when the box reaches within cm of the target position.
In the case of the task jaco-push--ue push-box-withh box-with-obstacke the robot is required push box randomly placed on a workspace to a specific target position obscured by an obstacle. The task considered successful when the box reaches within I cm of the target position.
In the case of the task iico-pick-and-stern workspace and stack it top of cylinder. The task is considered successful when the box reaches within 1 cm of the cylinder's top center position.
the robot is required to lift a box randomly placed on a
B.2 REWARD DESIGN
We adopt staged, structured, and multi-component reward function guide the RL algorithm. which simplifies the challenge of exploration and leads to effective policy learning (Lee et al. 2020b Yu et al.] 2020). The reward function, R, is combination of a reaching reward, pushing reward, vertical reward, floating reward, and lifting reward, or subsets thereof for simpler tasks that only include reaching or pushing. With this design, the reward is bounded by [0, 1] per timestep.
Jaco Reach Duplo
The reward function for task jaco-reach-duple is defined as follows:
It
Freaching
(reaching)
(3)
Jaco Move Box
The reward function for jaco is defined follows:
Treaching
(reaching)
T2
(4)
Treaching + Tpuuhiig
if dist Poti) Ereaching
(pushing)
Jaco Lift Box
The reward function for task jaco-lift-box is defined as follows:
Treaching
T Evertical
(reaching)
Treaching
Evertical
+
Ploating
if dist (ist(ppcppp Pobj) <
Ereaching
(floating)
(5)
Treaching
Vvertical
Flloating
Piiting
if 2abj > Elosting
(lifting)
16

---

Under review as a conference paper at ICLR 2023
Jaco Push Box with Obstacle as follows:
The reward function for task jacco-pusi-bbx--witto-ottteeee is defined
Freaching
Evertical
(reaching)
To
Treaching
lvertical + pposhing
if dist Poni) <
Ereaching
(pushing)
(6)
Jaco Pick and Stack
The reward function for task suco-pick-and-steee is defined follows:
Treaching
Vveriical
(reaching)
Treaching
Vertical
+ Titoating
if dist (Picp: Pobj) <
E reachhng
(floating)
Treaching
Vertical
Viooatin
Phiting
if Zubj > E Closting
(lifting)
where Freaching
N tolerancceddist(Prg Patj); Sreaching) Pppushing" =
N tolerance(distt (dans
Pobj): Epushing), Evertiaal
N tolerance cosdist( poosisst(Puuuu: (hundd U.z). 1 Eventiea), ''ooating =
and
Surre
Miitinng tolerance dist Puarget); Slifting): Piep represents the position of the end-effector's center point, Poyj represents the position of the object (here, the box's center point), Purget represents the target position. Zobj represents the z-axis value of the object's 's center point, 2targer represents the 2-axis value of the target position, Uhand represents a unit vector vertical to the robot hand, and u.z represents a unit vector [0, 0, -1]. The number of stages is represented by N. Here, we adopted the same function tolerance as in DMC (Tunyasuvunakool et al. 2020).
B.3 TASK VISUALIZATIONS
Figures 4 andd provide visualizations for behaviors generated by MERL in relation to the five 3D) robotic manipulation tasks from DMC jjco-reach-duplo, jaco-move-box. jaco-lift-box, jac--p--o box-with-obstacle, and jaco-pick-ond-stack) The visualization of behaviors learned by MERL during the training procedure is shown in Figure 4 We note here that MERL solves all five of the 3D robotic manipulation tasks within 1M environment steps. This demonstrates that MERL can solve the primitive skills reach, push, move, lift, and pick-and-plan with respect to the chosen 3D robotic manipulation tasks from DMC, within 1M environment steps. Figure F iillustrates the visualization of successful trajectories generated by MERL..
17

---

Under review as a conference paper at ICLR 2023
jaco Duplo
joco Move
jaco
200
Eptodd fauurn
Epiodd Meturn
250
150
150
300
0.0
0.5
1.0
1.5
2.5
0.5
3.5
2.5
3.0
1.0
1.5
2.5
3.0
Environment Staps (x:0P)
Environment Stops (x10f)
Environment Steps (x10f)
Jaco Push Box with Obstacle
Jac Pick and Stack
240
Egiodd Retum
150
Epioods Retuum
123
C20 State
S4C Stars
Steps
Steps
Figure 4: We visualize the behaviors learned by MERL during the training procedure in relation to
the five 3D robotic manipulation tasks from DMC (jaco-reach-dupp) duplo. nacc-move-box.pie jo. bicc-ppsh-bbx--withhoe and andici--nd--tccl
18

---

Under review as a conference paper at ICLR 2023
time elapse
Reach Duplo
Move Box
Lift
Push Box with Obs.
Jaco
Pick Pick Stack
Figure 5: We visualize the successful trajectories generated by MERL relation to the five 3D robotic manipulation tasks from DMC (jaco-recchdduplo, jaloo---,,,, jaco-pussh box-with-obstacle, and jaco-pick-and-staaee
19

---

Under review as a conference paper at ICLR 2023
C IMPLEMENTATION DETAILS
In this section, we describe our implementation details for MERL. We use PyTorch as a deep learning tool and DMC and MuloCo for our simulation. We conduct our experiments a workstation with an Intel 19-9980XE CPU and Nvidia Quadro RTX 8000 GPU. During training, we ran five random seeds for reliable comparison; that is, the experimental results are averaged over five different seeds.
C.I NETWORK ARCHITECTURES
For all networks, we initialize the weight matrix of the convolutional and fully-connected layers with an orthogonal initialization (Saxe et al. 2013) and set the bias to be zero.
Image Encoder Network The image encoder network is modeled four convolutional layers with 3x3 kernels and 32 channels, as in SAC+AB Yarats et al. (021c). An ReLU activation is applied after each convolutional layer. We use stride 1 everywhere, except for the first convolutional layer, which has stride 2. Here, we note that only the critic optimizer is allowed to update the image encoder network weights (that is, we prevent the actor's gradients from updating the image encoder network weights).
Proprioception Encoder Network The proprioception encoder network modeled as a 2-layer MLP with ReLU activations after each layer. Here, the final version of our method uses the pro- prioception encoder as the identity encoder. This is because as shown in Figure 0 contrary to the results in (Lee et al. 2019a; 2020b), the case where the proprioception encoder is set as the identity encoder provides better performance in relation to the three 3D robotic manipulation tasks from DMC (jaco-reach-dun) duplo, jaco-move-bon and jaco-lift- liift-boxx,, compared to the case where the proprioception encoder is set as an MI. encoder. This suggests that the proprioception itself is enough to learn a latent multimodal representation and a policy in an efficient, joint, and end-to-end manner, without the need for additional encodings; that is, is already well-encoded. Note that only the critic optimizer is allowed to update the proprioception encoder network weights.
jaco Duplo
jaco Move
jaco
200
Epaade Ratum
150
Ep Ree.um
Epecade Rattum
300
50-
30
PPSP 176
PTiPASS 124
3.5
0.5
1.5
25
0.5
25
32
Environment Steps (cto7)
Environment Steps (=:0))
Envtronment Steps (otod))
Figure 6: An additional ablation study that led us to the final version of MERL in relation to the proprioception encoder. We observe that the proprioception encoder set as the identity encoder (red) provides the best performance for the given 3D robotic manipulation tasks compared to MLP encoders (blue, green, and yellow).
Multimodal Fusion Network The encoded visual and proprioceptive representations are fed into multimodal fusion to learn a latent multimodal representation. The multimodal fusion network is modeled single fully-connected layer normalized by LayerNorm (Ba et al. 2016). After LayerNorm, the tanh nonlinearity is applied to the output of the fully-connected layer, which serves to match the latent multimodal representation and action the same scale. Finally, each output is concatenated to produce a single latent multimodal representation vector of dimension d. By means of the decoupled architecture, the weights of each multimodal fusion network for actor and critic are updated by the gradients of actor and critic, respectively.
Actor and Critic Networks As in TD3 (Fujimoto et al. 2018 we use clipped double Q-learning for the critic network. where each Q-function is parametrized as 3-layer (in the case of the task jaco-reach-dupe or 6-yyyer (in the case of the tasks jaco-move boo.,ooo.o.oaaaao jaco-lift-bor jaco jaco-pucee pero--psh-bor with-obstacle, and suco-pick-and-stan ML. with LayerNorm and ReLU activations after each layer
20

---

Under review as a conference paper at ICLR 2023
except the last. The actor network is also modeled 3-layer (in the of the task jaco-reacke duplo) or 6-layer (in the case the tasks jaco-move--on beooveebo.jeeee bow.boo.....a....oo... and jaco--icckand--teieeeeeeeeee
jaco-push-box MLP with LayerNorm and ReLU activations after each layer except the last, which outputs the mean and covariance for the diagonal Gaussian that represents the policy. The hidden dimension is set to 1024 for both the critic and the actor.
C.2 HYPER- PARME-ERRA
We provide comprehensive overview of the hyper-parameters used in the case of the five 3D robotic manipulation tasks from DMC (jaco-recc--uplo, jaco-move-hox, jaco-lift-box, jacio--oo.accousssiiei jaco-push-ber obstacle, and faco-pick-and-staie stack) in Table
Table 1: An overview of hyper-paramcters.
Parameter
Setting
Replay buffer capacity
10"
Mini-batch size
256
Frame stack
Seed frames
4000
Exploration steps
2000
Action repeat
Discount factor
0.99
Optimizer
Adam
Learning rate Soft-update rate n-step returns
10-4 4
0.01
Exploration std. dev. Exploration std. dev. clip
0.2
0.3
Hidden dim. for actor-critic
1024
Latent dim. for multimodal representation
128
21

---

Under review as a conference paper at ICLR 2023
D DECOUPLED VS SHARED ARCHITECTURE FOR MULTIMODAL FUSION
We provide a more detailed description of the shared architecture for multimodal fusion to clarify the difference between the decoupled and shared architecture for multimodal fusion. As shown in Figure 7 in the shared architecture, a single representation is used to train both the actor network and the critic network tandem. In contrast, the proposed decoupled architecture (see Figure I uses multiple representations (latent multimodal) to train the actor and critic networks separately. As a result, each of the representations can be learned better optimize the losses the actor and critic, respectively. More specifically, the weights of each multimodal fusion network for the actor and critic are updated by the gradients of the actor and critic, respectively. Consequently, as shown in the ablation studies, the proposed architecture leads to remarkable performance gain in sample efficiency. To the best our knowledge, MERL is the first RL method to apply a decoupled architecture to actor-critic learning.
multiodal regresentation
Q .)
replay buffer
prop
image
image augmentation
Imaae
image encoder
Image
multimodal fusion
crèic
Certtie
RGB images
forward
((Q)
((r)
z(a I>-mmy
propriocsption
encoder
prop
actor
Lactor
progeiocaption
Figure 7: We provide the shared architecture for multimodal fusion. In the shared architecture, a single representation is used train both the actor network and the critic network in tandem. Note that, in the case of the shared architecture for multimodal fusion, updating the multimodal fusion with the actor's gradients greatly hinders the agent's performance.
22

---

Under review as a conference paper at ICLR 2023
E IMPORTANCE OF LAYER NORMALIZATION IN THE STUDY OF RL
As shown in the ablation studies, we found that the use of layer normalization multimodal fusion plays crucial role in controlling the gradient scales and leads a favorable sample efficiency. In light of this, we further study whether the use of layer normalization in other baselines (DrQ-v2 and DDPG State) would have similar effect on performance. To this end, we perform additional experiments for two different cases: original baselines with and without layer normalization applied. As shown in Figure g the former case produces better performance relation to the five 3D robotic manipulation jaco-move-bon
and than the latter case. This suggests that the use of layer normalization in RI. algorithms is crucial to the study of RL. We note that the use of layer normalization in RL algorithms has received little attention in the RL community.
tasks from DMC (jaco-recch-duplee
jaco-lift-box,
sico-pick-and-stiie
Reach Duplo
Move Box
Lift Box
250
300
Eptoodo Return
Return
150
1:0
100
100
agent
agent
Sune
CoOno State
COPO State
Environment
Push Box with Obstacle
Pick Stack
300
Eploode Return
-00
Eplonde
CCPO State
DOPO Stet
0.0
0.5
3.5
2.5
3.0
0.0
0.5
3.0
2.0
2.5
3.0
Enstronment Steps (=:)
Environment Steps (x:10f)
(a) Comparison of DDPG State compared to DDPG State with layer normalization applied.
Duplo
250
175
200
1:0
Epioode Ratum
Epicode
100
100
0-
3.0
2.0
3.0
Environment Steps (x:1ff)
Enstronment Steps 1=
Environment Steps E-:399y
Push Box with Obstacle
Pick and Stack
340
100
Epioode Returm
agort
ogent
20
3.5
0.
3.0
2.5
Ensfronment Steps (=11
Environment
(b) Comparison of DrQ-v2 compared to DrQ-v2 with layer normalization applied.
Figure& 8: We compare the original baselines those with layer normalization applied in relation to five 3D robotic manipulation tasks from DMC (jaco-reach-duple raco-move-box jaco-lift dift-boxx, box, jaco- mush-box-with-obstee and maco-pick-and-stard We observe that the use of layer normalization in RL algorithms is crucial to the study of RL.
23

---

Under review as a conference paper at ICLR 2023
F EXPERIMENTAL RESULTS FOR HUMANOID LOCOMOTION TASKS
We provide additional experimental results in relation to three complex 3D humanoid locomotion tasks (humanoid-stani humanoid-walk, and humanoid-run)- these tasks being among the most difficult of 3D continuous control problems - to clarify the superiority of MERL compared to DrQ-v2. As shown in Figure 9 MERL significantly outperforms DrQ-v2 with respect to sample efficiency. Most notably, MERL solves each of the three complex 3D humanoid locomotion tasks within 5M environment steps, whereas DrQ-v2 requires 30M environment steps to solve the same tasks (see DrQ-v2 paper (Yarats et al. 202 a ). The experimental results demonstrate that MERL. Seeeeeaalii one of the most effective RL methods for solving 3D continuous control problems, including robotic manipulation and locomotion tasks, with respect to learning policy in an end-to-end manner, without the need for human-craftin representations or prior expert demonstrations.
Humanoid Walk
Humanoid Stand
Humanoid Run
ager
200
toloode
6:0
Edtooda Ratum
4:0
Epieode
320
Enseronment Staps
Stepp [1F)
Environment Steps [=10F)
Figure 9: We compare MERL to DrQ-v2 relation to three complex 3D humanoid locomotion tasks from DMC (humanoid-stand, humanoid-walk, and humanoid-rum). MERL demonstrates its superiority compared to DrQ-v2, currently a state-of-the-arr visual RL.
F.1 TASK VISUALIZATIONS
Figures 10 and provide visualizations for behaviors generated by MERL relation to three complex 3D humanoid locomotion tasks from DMC (humanoid-stand, humanoid-walk and humanoid- run). The visualization of behaviors learned by MERL during the training procedure shown in Figure T0 We note that MERL solves all three of the complex 3D humanoid locomotion tasks within 5M environment steps. Figure 11 illustrate the visualization of successful trajectories generated by MERL.
Humanoid
Humanoid
Humanoid
1000
800
Epkoode Reeurn
Epioede Retum
600
Eplodd Return
250
300
400
400
200
Environment Staps [=:1P))
Environment Staps [x:10f)
Environment Steps [=:1F)
Figure 10: We visualize the behaviors learned by MERL during the training procedure in relation to three complex 3D humanoid locomotion tasks from DMC (humanoid-stanne humanoid-wald and humanoid-run).
24

---

Under review as a conference paper at ICLR 2023
time elapse
Humanoid Stand
Humanoid Walk
Humanoid Rum
Figure 11: We visualize the successful trajectories generated by MERL in relation to three complex
humanoid locomotion tasks from DMC (humanoid saumanoid-stand, humanoid-wall and numanoid-run)
25

---

Under review as a conference paper at ICLR 2023
G COMPARISON WITH MORE VISUAL RL BASELINES
We provide the additional experimental results obtained when comparing MERL to other visual
RL baselines (pixel-to-action approaches) in relation to three 3D locomotion tasks from DMC
(quadruped-walk, quadruped-run and humanoid-stand)) The experimental results (Figure I2 show
that MERL significantly outperforms all the baselines with respect to sample efficiency, learning performance, and training stability.
Quadruped
Quadruped
Humeroid
1060
800
tpledde Naturn
splande Neturn
Epleude Natum
800
<00
3.0
25
3.0
2.0
35
300
3.0
2.5
1.0
2.0
2.5
3.0
3.0
0.5
1.0
1.3
2.0
300
Enn ronmentt Stops (x:17
ERN ronment Staps (x:1
Envinnmeent Steps (x:1
Figure 12: We compare MERL to other visual RL baselines (DrQ-v2 Yarats et al. 2021a) CURL
(Srinivas et al. 2020), and SAC+AE Yaarats al. (021ch) in relation three 3D locomotion tasks
from DMC (quadruped walk, guadruped-runt and humanoid-stand) MERL demonstrates superior sample efficiency, significantly outperforms other visual RL baselines, and shows more stable training performance.
26


Reviews:
[{"review_id": "q2Evvi1Ojjz", "reviewer": "Reviewer_c2N6", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents an algorithm called Multimodal End-to-end Reinforcement Learning (MERL) that integrates visual and proprioceptive observations in learning model-free reinforcement learning policies. Specifically, the method builds on SAC and DrQv2, and adds encoders to handle both proprioceptive and image encoders into the policy and value function architectures. The method is evaluated on simulated 3D robotic manipulation tasks from DMC (Jaco tasks), and is compared to SAC from state and DrQv2. The authors also perform an ablation analysis of their method, in terms of how multimodal fusion is performed, the size of the multimodal representation, use of layer normalization, and exploration noise. ", "summary_of_the_review": "Overall, the paper is clear, detailed, and well-executed, but I think that in its current form, there are three main shortcomings: 1) only proprioception + vision are tested, 2) the ablation studies don’t completely demonstrate why MERL is helpful (see layer norm comment above), and 3) there are a limited number of tasks and only from the DMC suite.", "strength_and_weaknesses": "Strengths: \n- The proposed method is conceptually simple and addresses an important problem of how to perform multimodal fusion for policy learning. \n- The empirical results on the Jaco tasks are strong.\n- The paper includes detailed motivation and description of the design decisions for the method, which is quite important for model-free deep RL methods.\n- The paper is well-written and clear.\n\nWeaknesses:\n- Overall, I feel that the paper is too narrow in its scope for the claims that it makes. While the method is proposed to be a general multimodal framework, it is only demonstrated in a single setting with proprioceptive and image data, rather than consider other modalities like touch, sound, etc.\n- Furthermore, the evaluation is only performed on three DMC tasks, and there are many specific design decisions that are made that could impact performance on these tasks in particular.\n- According to the ablation study, the performance improvements of the method seem to be largely attributable to the use of layer normalization in the architecture. It would be helpful to understand if layer norm only helps MERL or if the other baselines also improve on this task with the use of layer norm.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "W9wOGgz2Wo", "reviewer": "Reviewer_tRt9", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper introduces MERL, Multimodal End-to-end Reinforcement Learning, a framework that combines multimodal (namely vision and proprioception) representation learning and RL.", "summary_of_the_review": "The paper is well written, but lacks novelty.", "strength_and_weaknesses": "The paper is well written, clear and structured in a nice way. The method is introduced in a clear way, as well as the experimental setup.\nSome quite important pieces of the approach adopted are not described in the main paper, such as the formulation of the reward stages used to train the agents, which actually can play a very important role.\nThe empirical evaluation includes interesting ablations of the method, examining the impact of various design choices.\nSome important information are missing or unclear. What did you mean with the sentence \"Unlike in ... state-based RL such as DDPG and SAC, our method stores transition ... in the replay buffer,\"? Both DDPG and SAC use a replay buffer.\nMy main concern is that, in my opinion (unless I missed something important from the paper), there is no novelty. Vision+proprioception based agents have been trained successfully in several robotics domains, including manipulation (eg https://openreview.net/forum?id=U0Q8CrtBJxJ, http://proceedings.mlr.press/v87/kalashnikov18a.html, etc). The two modalities considered, that is proprioception and vision, have been explored in several works, and most recent works look at other challenging modalities in the context of multimodal representations especially in robot manipulation, in particular using tactile or audio signals, for example, in addition to vision and proprioception.\nOther considerations, like the observation that having gradients from the agents going through the representation module during training hinders the performance, are well-known as well.\nWhat differentiate your approach for existing vision+proprioception RL approaches, especially in the domain of robot manipulation?\nTraining directly from vision is in fact a challenging task, in particular it can be very data-inefficient; how did you approach this problem and what made your approach overcome it?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "Cu3UGkcISH", "reviewer": "Reviewer_gi8i", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents a method to end-to-end learn manipulation tasks from raw multi-modal sensory data. Multimodality refers to the use of both propriocetion and RGB-images. The proposed RL method is model-free and off-policy. It relies on image augmentation to learn a multimodal representations fed to an actor and a critic network. The proposed methodology is tested on three robot manipulation tasks from the Deepmind Control suite. Comparisons with state-of-the-art alternatives (i.e. DrQ-v2, DDPG from state and SAC from state). Remarkably the proposed approach does better than all other approaches, including DDPG from state. Ablations studies show: (1) the importance of decoupled representations for the actor and the critic networks, (2) the importance of choosing the representation dimensions, (3) the importance of layer normalization and (4) the minor benefits of scheduled exploration noise. ", "summary_of_the_review": "I think the paper is relatively well executed and the published results are a significant step with respect to the state of the art. However, nothing really new is presented in the paper to the point that it becomes surprising to notice that these significant improvements could be achieved with minor adjustments to the network architecture (see ablation studies). Also, the paper is limited to 3 simple tasks which limits a lot the soundness of the proposed approach.\n\nI usually try to be more verbose and provide some useful feedback in my reviews but I don't have much to say on this specific paper. My final score is \"below threshold\" but I am struggling with finding the right evaluation; it would be great if the community could have the code and test the approach on more numerous and more complicated manipulation tasks. ", "strength_and_weaknesses": "The paper present results which are beyond the state of the art. However, presented results are limited to three relatively simple tasks for which a staged reward was needed. The proposed approach is relatively standard and the major theoretical contribution seems to be just a marginal improvement with respect to existing alternatives.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "CE76aD45EWx", "reviewer": "Reviewer_iSe9", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a new reinforcement learning architecture for learning robot behaviors from pixels.", "summary_of_the_review": "Solid contribution, however validated against a single baseline, on a narrow set of evaluations, only in simulation. To make this paper truly convincing, I would want to see 1) a broader palette of pixel-to-actions tasks being explored or 2) justification as to why only manipulation ought to be in scope, or 3) real-world results.", "strength_and_weaknesses": "Strenghs:\n- very important and broad problem to be tackling\n- well-motivated, general approach\n- detailed description of the algorithm, including hyperparameters\n- excellent set of ablations\n\nWeaknesses:\n- Some of the prior art is somewhat misrepresented. Many of the works cited in the section which ends with 'Such methods are commonly referred to as state-based RL.' are not state-based, but learn directly end-to-end policies from pixels (Kalashnikov et al, Ibarz et al for instance). [ADDRESSED]\n- The paper compares against a single baseline (DrQ-v2), but only on a narrow subset of the tasks that DrQ-v2 was able to perform. In particular, being able to train humanoid from pixels was one of the strong claims from DrQ-v2, and experiments in this setting are notably absent from this paper, and (somewhat conveniently) excluded by making this a manipulation-centric paper. This makes this reviewer very suspicious that maybe the authors tried their approach on humanoid, it didn't work well, and they maybe opted to not show these results. If that were the case, I would insist on these results to be incorporated in the paper, so that a reader would have a complete picture of the behavior of the approach. If there are other reasons why focussing on the three Jaco tasks made sense, I would love to see it being made explicit in the paper. [ADDRESSED]\n- The paper does a good job at contrasting the architecture choices made against DrQ-v2. It would have been interesting to 1) further contrast with other pixel-to-action approaches that have been explored in the past, and 2) try to provide some intuition as to why this alternative architecture was interesting to consider beyond merely the empirical results. [ADDRESSED]\n- No real-world results. I am very suspicious of any result which only evaluates algorithms on 'simulated pixels'. There is little evidence for a strong correlation between relative performance in these settings and real-world settings. [Not addressed, but OK to not have in scope of this paper.]\n- No discussion of systems aspects: training step and inference speed, memory in contrast to other approaches. [Not addressed, but OK to not have in scope of this paper.]\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject