Deep Reinforcement Learning for Industrial Insertion Tasks
with Visual Inputs and Natural Rewards

Gerrit Schoettler * 1 Ashvin Nair * 2 Jianlan Luo 2 Shikhar Bahl 2Juan Aparicio Ojea 1 Eugen Solowjow 1 Sergey Levine 2

Abstract
Connector insertion and many other tasks commonly found in modern manufacturing settings
involve complex contact dynamics and friction.
Since it is difficult to capture related physical effects with first-order modeling, traditional control
methodologies often result in brittle and inaccurate controllers, which have to be manually tuned.
Reinforcement learning (RL) methods have been
demonstrated to be capable of learning controllers
in such environments from autonomous interaction with the environment, but running RL algorithms in the real world poses sample efficiency
and safety challenges. Moreover, in practical realworld settings we cannot assume access to perfectstate information or dense reward signals. In thispaper, we consider a variety of difficult industrial
insertion tasks with visual inputs and different
natural reward specifications, namely sparse rewards and goal images. We show that methods
that combine RL with prior information, such asclassical controllers or demonstrations , can solvethese tasks directly by real-world interaction.

1. Electric Connector Plug Insertion Tasks
In this work, we empirically evaluate learning methods on aset of electric connector assembly tasks, pictured in Fig. 1.
Connector plug insertions are difficult for two reasons. First,
the robot must be very precise in lining up the plug with
its socket. As we show in our experiments, errors as small
as ±1 mm can lead to consistent failure. Second, there issignificant friction when the connector plug touches thesocket, and the robot must learn to apply sufficient force
in order to insert the plug. Image sequences of successful
insertions are shown in Fig. 2, where it is also possible tosee details of the gripper setup that we used to ensure a fully
automated training process. In our experiments, we use a 7*1
Equal contribution.
Siemens Corporation, Berkeley,
USA 2 Berkeley AI Research, University of California, Berkeley, Computer Science. Correspondence to: Ashvin Nair<anair17@berkeley.edu>.

degrees of freedom Sawyer robot with end-effector control,
meaning that the action signal ut can be interpreted as therelative end-effector movement in Cartesian coordinates.
To comprehensively evaluate connector assembly tasks, werepeat our experiments on a variety of connectors. Eachconnector offers a different challenge in terms of required
precision and force to overcome friction. We chose to benchmark the controllers performance on the insertion of a USB
connector, a U-Sub connector, and a waterproof Model-E
connector manufactured by MISUMI. All the explored use
cases were part of the IROS 2017 Robotic Grasping andManipulation Competition (Falco et al., 2018), included aspart of a task board developed by NIST to benchmark the
performance of assembly robots.
1.1. Adapters
In the following we describe the used adapters, USB, DSub, and Model-E. The observed difficulty of the insertion
increases in that order.
1.1.1. USB
The USB connector is a ubiquitous, widely-used connector
and offers a challenging insertion task. Because the adapter
becomes smoother and therefore easier to insert over time
due to wear and tear, we periodically replace the adapter.
Of the three tested adapters, the USB adapter is the easiest.

Figure 1. We train an agent directly in the real world to solve
connector insertion tasks that involve contacts and tight tolerancesfrom convenient reward signals such as pixel distance to a goal
image or a sparse electrical signal.

Deep Reinforcement Learning for Industrial Insertion Tasks with Visual Inputs and Natural Rewards

This is the exact true reward for the task of connecting acable, and can be naturally obtained in many manufacturingsystems. As state, the robot is given the Cartesian coordinates of the end-effector xt and the vertical force fz that is
acting on the end-effector. As we could only automaticallydetect the USB connection thus far, we only include theUSB adapter for the sparse experiments.
1.2.3. D ENSE
Figure 2. The three image sequences show rollouts from learned
policies that successfully complete the insertion tasks.

1.1.2. D-S UB
Inserting this adapter requires aligning several pins correctly, and is therefore more sensitive than inserting theUSB adapter. It also requires more downward force due toa tighter fit.
1.1.3. M ODEL -E
This adapter is the most difficult of the three tested connectors as it contains several edges and grooves to align andrequires significant downward force to successfully insert
the part.

In this experiment, the robot receives a manually shaped
reward based on the distance to the target. We use the reward
functionβ
rt = −α · kxt − x∗ k1 −− ϕ · fz , (2)(kxt − x∗ k2 + ε)
where 0 < ε  1. The hyperparameters are set to α = 100,β = 0.002, and ϕ = 0.1. When an insertion is indicatedthrough a distance measurement, the sign of the force term
flips, so that ϕ = −0.1 when the connector is inserted. This
rewards the agent for pressing down after an insertion andshowed to improve the learning process.

2. Methods
To solve the connector insertion tasks, we consider andevaluate a variety of reinforcement learning algorithms.

1.2. Experimental Settings
We consider three settings in our experiments in order toevaluate how plausible it is to solve these tasks with moreconvenient state representations and reward functions andto evaluate the performance of different algorithms changes
as the setting is modified.
1.2.1. V ISUAL
In this experiment, we evaluate whether the RL algorithms
can learn to perform the connector assembly tasks fromvision without having access to state information. The stateprovided to the learned policy is a 32 × 32 grayscale image,such as shown in Fig. 4. For goal specification, we use
a goal image, avoiding the need for state information tocompute rewards. The reward is the pixelwise L1 distanceto the given goal image. Being able to learn from such a
setup is compelling as it does not require any extra state
estimation and many tasks can be specified easily by a goal
image.

2.1. Preliminaries
In a Markov decision process (MDP), an agent at every
time step is at state st ∈ S, takes actions ut ∈ U, receives
a reward rt ∈ R, and the state evolves according to environment transition dynamics p(st+1 |st , ut ). The goal ofreinforcement learning is to choose actions ut ∼ π(ut |st )PH
to maximize the expected returns E[ t=0 γ t rt ] where H isthe horizon and γ is a discount factor. The policy π(ut |st )is often chosen to be an expressive parametric function approximator, such as a neural network, as we use in thiswork.
2.2. Efficient Off-Policy Reinforcement Learning
One class of RL methods additionally estimates the expecteddiscounted return after taking action u from state s, the Qvalue Q(s, u). Q-values can be recursively defined with the
Bellman equation:
Q(st , ut ) = Est+1 [rt + γ max Q(st+1 , ut+1 )]
ut+1

1.2.2. E LECTRICAL (S PARSE )In this experiment, the reward is obtained by directly measuring whether the connection is alive and transmitting:(
1, if insertion signal detectedr=(1)
0, else.

(3)

and learned from off-policy transitions (st , ut , rt , st+1 ). Because we are interested in sample-efficient real-world learning, we use such RL algorithms that can take advantage ofoff-policy data.
For control with continuous actions, computing the required
maximum in the Bellman equation is difficult. ContinuousDeep Reinforcement Learning for Industrial Insertion Tasks with Visual Inputs and Natural Rewards
USB

Model-E Connector

0.025

Final Distance [m]

0.06

0.03

0.04

0.010

0.02

0.02

0.01

0.005
0.000
0.0K

0.5K

1.0K

1.5K

Timesteps

2.0K

2.5K

0.00
0K

Residual RL (SAC)
Residual RL (TD3)
RL + LfD
SAC

0.04

0.020
0.015

D-Sub Connector

0.05

1K

2K

3K

Timesteps

4K

5K

6K

0.00
0.0K

0.5K

1.0K

1.5K

Timesteps

2.0K

2.5K

Figure 3. Resulting final mean distance during the vision-based training. The comparison includes residual RL, learning from demonstrations and pure RL, represented by SAC. Only residual RL with SAC manages to deal with the high-dimensional input and consistently
solves all tasks after the given amount of training. The deterministic policies learn to move downwards, but often get stuck in the beginningof the insertion and fail to recover from unsuccessful attempts.

control algorithms such as deep deterministic policy gradients (DDPG) (Lillicrap et al., 2016) additionally learn a
policy πθ (ut |st ) to approximately choose the maximizing
action. In this paper we specifically consider two related
reinforcement learning algorithms that lend themselves well
to real-world learning as they are sample efficient, stable,
and require little hyperparameter tuning.
2.2.1. T WIN D ELAYED D EEP D ETERMINISTIC P OLICY
G RADIENTS (TD3)
Like DDPG, TD3 optimizes a deterministic policy (Fujimoto et al., 2018) but uses two Q-function approximators to
reduce value overestimation (Van Hasselt et al., 2016) anddelayed policy updates to stabilize training.

πH of our experiments. The P-controller operates in Cartesian space and calculates the current control action by
πH (st ) = −kp · (xt − x∗ ),(5)where x∗ denotes the commanded goal location. As control
gains we use kp = [ 1, 1, 0.3 ]. This P-controller quickly
centers the end-effector above the goal position and reaches
the goal after about 10 time steps when starting from thereset positions, which is located about 5cm above the goal.

2.2.2. S OFT ACTOR C RITIC (SAC)
SAC is an off-policy value-based reinforcement learning
method based on the maximum entropy reinforcement learning framework with a stochastic policy (Haarnoja et al.,2018).
We used the implementation of these RL algorithms publicly
available at rlkit (Pong et al., 2018).
2.3. Residual Reinforcement Learning
Instead of randomly exploring from scratch, we can injectprior information into an RL algorithm in order to speed up
the training process, as well as to minimize unsafe exploration behavior. In residual RL, actions ut are chosen byadditively combining a fixed policy πH (st ) with a parametric policy πθ (ut |st ):ut = πH (st ) + πθ (st ).

(4)

The parameters θ can be learned using any RL algorithm.
In this work, we evaluate both SAC and TD3, explained in
the previous section.
A simple P-controller serves as the hand-designed controller

Figure 4. Successful insertion on the Model-E connector task. The
32 × 32 grayscale images are the only observations that the imagebased reinforcement learning algorithm receives.

2.4. Learning from Demonstrations
Another method to incorporate prior information is to use
demonstrations from an expert policy to guide exploration
during RL. We first collected demonstrations with a keyboard controller. Then, we add a behavior cloning loss while
performing RL that pushes the policy towards the demonstrator actions, as previously considered in (Nair et al., 2018).
Instead of DDPG, the underlying algorithm RL algorithm
used is TD3.

Deep Reinforcement Learning for Industrial Insertion Tasks with Visual Inputs and Natural Rewards

D-Sub connector
Human Controller
SAC
Dense
Residual RL
ImagesPerfect Goal
100%
16%
100%
100%

Noisy Goal
44%
0%
60%
64%

Model-E Connector
Human Controller
SAC
Dense
Residual RL
ImagesPerfect Goal
52%
0%
100%
100%

Noisy Goal
24%
0%
76%
76%

Figure 5. Performance evaluation on the D-Sub connector. We report the average success rate out of 25 rollouts of the trained policies.

We evaluate the industrial applicability of the residual RL approach on a variety of connector insertion tasks that are performed on a real robot, using easy-to-obtain reward signals.
In this section, we consider two types of natural rewards
which are intuitive to humans: an image directly specifying
a goal and a binary sparse reward indicating success. For
both cases, we report success rates on tasks they solve. We
aim to answer the following questions: (1) Can such trained
policies provide comparable performance to policies that aretrained with densely-shaped rewards? (2) Are these trained
policies robust to light variations and noise?
3.1. Vision-based Learning
For the vision-based learning experiments, we use only raw
image observations and L1 distance in image space as thegoal. Sample images that the robot received are shown in
Fig. 4. We evaluate this type of reward on all three tasks. In
our experiments, we use grayscale images converted from
RGB ones, this simplification reduces input space; but isgood enough for our experiments.
3.2. Learning from Sparse Rewards
The applicability of a sparse reward function is exploredon an insertion of the USB connector. The binary insertion
signal is used as the metric for success. This experiment ismost applicable to electronic manufacturing settings where
the electrical connection between connectors can be directly
measured.
3.3. Robustness
For safe and reliable future usage, it is required that theinsertion controller is robust against small measurement orcalibration errors that can occur when disassembling andreassembling a mechanical system. In this experiment, small
goal perturbations are introduced in order to uncover therequired setup precision of our algorithms.
3.4. Exploration Comparison
One advantage of using reinforcement learning is the exploratory behavior that allows the controller to adapt fromnew experiences, unlike a deterministic control law. The

two RL algorithms we consider in this paper, SAC and TD3,
explore differently. SAC maintains a stochastic policy, andthe algorithm also adapts the stochasticity through training.
TD3 has a deterministic policy, but uses another noise process (in our case Gaussian) to inject exploratory behavior
during training time. We compare the two algorithms, aswell as when they are used in conjunction with residual RL,
in order to evaluate the different exploration schemes.

1.0

USB Insertion Task, Sparse Reward, SAC vs. TD3

0.8

Success Rate

3. Experiments

0.6
0.4

Residual RL (SAC)
Residual RL (TD3)
SAC
TD3

0.2
0.0

0K

1K

2K

3K

4K

5K

Timesteps

6K

7K

8K

Figure 6. Comparison of two RL algorithms, SAC and TD3, on the
USB insertion task with sparse rewards. Using residual RL, bothalgorithms can solve the task. Without residual RL, TD3 struggles
to solve the task while SAC eventually does solve it to the samelevel of performance as the residual RL methods. We believe thedifference may be due to the adaptive noise in SAC.

4. Results
In order to evaluate our experiments with dense and visionbased rewards, we analyze the achieved final distance tothe goal throughout the training process. Policies trainedwith sparse rewards are compared based on their successrate because their training objective does not include theminimization of the distance to the goal. We report thesuccess rate of all final policies and compare their robustness
towards measurement noise in the goal location.
4.1. Vision-based Learning
The results of the vision-based experiment are shown in
Fig. 3. Our experiments show that a successful and consistent vision-based insertion policy can be learned from
relatively few samples using residual RL with SAC. This

Deep Reinforcement Learning for Industrial Insertion Tasks with Visual Inputs and Natural Rewards

result suggests that goal-specification through images is apractical way to solve these types of industrial tasks.
Interestingly, during training with pure RL, the policy would
sometimes learn to “hack” the reward signal by moving
down in the image in front of or behind the socket. In contrast, the stabilizing human-engineered controller in residual RL provides sufficient horizontal control to prevent thisand it also transforms the 3-dimensional task into a quasi
1-dimensional problem for the reinforcement learning algorithm, which explains the very good results obtained withresidual RL in conjunction with vision-based rewards.

Table 1. Test-time performance on the USB insertion task. Noised
is added in form of ±1 mm perturbations of the goal location. We
report the average success rate out of 25 rollouts.

USB

Perfect Goal

Noisy Goal

Human Controller

100%

60%

SAC

16%

8%

RL + LfD

100%

32%

Dense

100%

84%

Sparse, SAC

88%

84%

Sparse, TD3

100%

36%

Images100%

80%4.2. Learning From Sparse Rewards
Residual RL
In this experiment, we compare several methods on the USB
insertion task with sparse rewards. The results are reported
in Fig. 7.
USB Insertion Task, Sparse Reward

1.0

than SAC. However, TD3 is significantly less robust, asshown in Tab. 1. Furthermore, we found that the outputtedaction of TD3 approaches the extreme values at the edgeof the allowed action space. This suggests it finds a local
minimum, which performs well, but may not be robust andTD3 cannot improve beyond that policy.

Success Rate

0.8
0.6
0.4
Residual RL (SAC)
RL + LfD
SAC

0.2
0.0

0K

1K

2K

3K

4K

5K

Timesteps

6K

7K

8K

Figure 7. Learning curves for solving the USB insertion task witha sparse reward are shown. In this experiment, ground truth stateis given as observations to the agent. Residual RL and RL withlearning from demonstrations both solve the task relatively quickly,
while RL alone (SAC) takes about twice as long to solve the task
at the same level of performance.

4.3. Robustness
In previous set of experiments, the goal locations were
known exactly. In this case, the hand-engineered controller
performs well. However, once noise is induced to the goal
location, the deterministic P-controller does not solve thetask anymore. After training on perfect observations, a goal
perturbation is created artificially and the controllers aretested under this condition. All results of our robustness
evaluations are listed in Tab. 1, Tab. ??, and Tab. ??.
4.4. Exploration Comparison
A comparison of TD3 and SAC is made in Fig. 6. When
combined with residual RL, they perform comparably.
When considering RL alone, TD3 learns the task faster5. Discussion and Future WorkIn this paper we studied residual RL with natural rewards
and demonstrated that this approach can solve complex industrial assembly tasks with tight tolerances, e. g. connector
plug insertions. We introduced vision inputs to the residual
RL formulation, which increases the algorithm’s usefulness
for a wide range of industrial applications. Compared toprevious work (Johannink et al., 2019), which uses dense reward signals, we showed that we can learn insertion policies
only from sparse binary rewards or even purely from goal
images. We conducted a series of experiments for various
connector type assemblies and could demonstrate the feasibility of our method, even under challenging conditions
such as noisy goals and complex connector geometries. Our
study motivates the application of residual RL to industrial
automation tasks, where reward shaping is not feasible, butsparse rewards or image goals can often be provided.
Future work will include more complex environments focusing on multi-stage assembly tasks through vision. This
would pose a challenge to the goal-based policies as the
background would be visually more complex. Moreover,multi-step tasks involve adapting to previous mistakes or
inaccuracies, which could be difficult, however, in theory
should be able to be handled by RL. Extending the presented
approach to multi-stage assembly tasks will pave the roadto a higher robot autonomy in flexible manufacturing.

Deep Reinforcement Learning for Industrial Insertion Tasks with Visual Inputs and Natural Rewards

Appendix
A. State-Based Training
A.1. Dense Reward Connector Plug Insertion
After evaluating the tasks in the above settings, we further
evaluate with full state information with a dense and carefully shaped reward signal, given in Eq. 2, that incorporates
distance to the goal and force information. Evaluating inthis setting gives us an “oracle” that can be compared tothe previous experiments in order to understand how muchof a challenge sparse or image rewards pose for various
algorithms.
A.2. Dense Reward Connector Plug Insertion
The results of the experiment with dense rewards are shownin Fig. 8.
Here, the same conclusions of residual RL outperforming
pure RL hold. Due to the shaped reward, pure RL makes
more initial progress, but cannot overcome the friction required to fully insert the plugs. It appears that the handdesigned reward function does not incentivise the full insertion enough, as we were able to obtain better results on the
USB insertion with sparse rewards.

B. Related Work
Learning has been applied previously in a variety of robotics
contexts. Different forms of learning have enabled autonomous driving (Pomerleau, 1989), biped locomotion(Nakanishi et al., 2004), block stacking (Deisenroth et al.,2011), grasping (Pinto & Gupta, 2016), and navigation(Giusti et al., 2015; Pathak et al., 2018). Among these methods, many involve reinforcement learning, where an agent
learns to perform a task by maximizing a reward signal. Reinforcement learning algorithms have been developed andapplied to teach robots to perform tasks such as balancing arobot (Deisenroth & Rasmussen, 2011), playing ping-pong(Peters et al., 2010) and baseball (Peters & Schaal, 2008).
The use of large function approximators, such as neuralnetworks, in RL has further broadened the generality of RL(Mnih et al., 2013). Such techniques, called “deep” RL,
have further allowed robots to perform fine-grained manipulation tasks from vision (Levine et al., 2016), open doors(Gu et al., 2016), score a hockey puck (Chebotar et al.,2017), and grasp objects (Kalashnikov et al., 2018). In thiswork, we further explore solving real-world robotics tasks
using RL.
Many RL algorithms introduce prior information about the

specific task to be solved through various means such as
reward shaping (Ng et al., 1999), incorporating a trajectory
planner (Thomas et al., 2018; Eruhimov & Meeussen, 2011;
Mayton et al., 2010), learning classifiers between goals andnon-goals (Ho & Ermon, 2016; Pinto & Gupta, 2016; Levine
et al., 2017). These methods require access to various goalstates to build a robust classifier, which might be difficult
to collect in assembly as there is often only one goal image
possible. Reward shaping can become arbitrarily difficult as
the complexity of the task increases. For complex assembly
tasks, trajectory planners require a host of information about
objects and geometries which can be difficult to provide.
Mainly, previous work on incorporating prior information
has focused on using demonstrations either to initialize apolicy (Peters & Schaal, 2008; Kober & Peter, 2008), inferreward functions using inverse reinforcement learning (Finn
et al., 2016; Abbeel & Ng, 2004; Ziebart et al., 2008; Rhinehart & Kitani, 2017; Fu et al., 2018) or to improve the policy
throughout the learning procedure (Hester et al., 2018; Nairet al., 2018; Rajeswaran et al., 2018; Večerı́k et al., 2017).
These methods require multiple demonstrations, which canbe difficult to collect, especially for assembly tasks. Morerecently, manually specifying a policy and learning the residual task has been proposed (Johannink et al., 2019; Silver
et al., 2018). In this work we evaluate both residual RL andcombining RL with learning from demonstrations (LfD).
Previous work has also tackled high precision assembly
tasks, especially insertion-type tasks. One line of work
focuses on obtaining high dimensional observations, including geometry, forces, joint positions and velocities (Li et al.,2014; Tamar et al., 2017; Inoue et al., 2017; Luo et al., 2019),
but this information is not easily procured, increasing complexity of the experiments and the supervision required tocollect the data. Other work relies on external trajectory
planning or very high precision control (Inoue et al., 2017;
Tamar et al., 2017), but this can be brittle to error in othercomponents of the system, such as perception. We showhow our method not only solves insertion tasks with muchless information about the environment, it also does so undernoisy conditions.

References
Abbeel, P. and Ng, A. Y. Apprenticeship learning via inverse
reinforcement learning. In International Conference on
Machine Learning (ICML), pp. 1, 2004.
Chebotar, Y., Hausman, K., Zhang, M., Sukhatme, G.,
Schaal, S., and Levine, S. Combining Model-Based andDeep Reinforcement Learning for Industrial Insertion Tasks with Visual Inputs and Natural Rewards
USB

Model-E Connector

Final Distance [m]

0.03
0.04

Residual RL (SAC)
SAC

0.015

0.02

0.010

0.02

0.00
0K

D-Sub Connector

0.020

0.01

2K

4K

Timesteps

6K

8K

0.00
0K

0.005
1K

2K

Timesteps

3K

0K

2K

4K

6K

Timesteps

8K

10K

Figure 8. Plots of the final mean distance to the goal during the state-based training. Final distances greater than 0.01 m indicate
unsuccessful insertions. Here, the residual RL approach performs noticeably better than pure RL and is often able to solve the task during
the exploration in the early stages of the training.

Model-Free Updates for Trajectory-Centric Reinforcement Learning. In International Conference on Machine
Learning (ICML), 2017.
Deisenroth, M. P. and Rasmussen, C. E. PILCO: A modelbased and data-efficient approach to policy search. In
International Conference on Machine Learning (ICML),
pp. 465–472, 2011.
Deisenroth, M. P., Rasmussen, C. E., and Fox, D. Learning
to Control a Low-Cost Manipulator using Data-Efficient
Reinforcement Learning. Robotics: Science and Systems(RSS), VII:57–64, 2011.
Eruhimov, V. and Meeussen, W. Outlet detection andpose estimation for robot continuous operation. In 2011
IEEE/RSJ International Conference on Intelligent Robots
and Systems, pp. 2941–2946, 2011.

L. M., Ciresan, D. C., He, F.-L., Rodriguez, J. P., Fontana,F., Faessler, M., Forster, C., Schmidhuber, J. J., Caro,
G. D., Scaramuzza, D., and Gambardella, L. M. A Machine Learning Approach to Visual Perception of Forest
Trails for Mobile Robots. In IEEE Robotics and Automation Letters., volume 1, pp. 2377–3766, 2015. ISBN
9781467380256. doi: 10.1109/LRA.2015.2509024.
Gu, S., Lillicrap, T., Sutskever, I., and Levine, S. Continuous
Deep Q-Learning with Model-based Acceleration. In
International Conference on Machine Learning (ICML),
2016. ISBN 3405062780. doi: 10.3390/robotics2030122.
Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S. Soft
Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor. In International Conference on Machine Learning, 2018.

Falco, J., Sun, Y., and Roa, M. Robotic grasping and manipulation competition: Competitor feedback and lessons
learned. In Sun, Y. and Falco, J. (eds.), Robotic Graspingand Manipulation, pp. 180–189, Cham, 2018. Springer
International Publishing. ISBN 978-3-319-94568-2.

Hester, T., Vecerik, M., Pietquin, O., Lanctot, M., Schaul,T., Piot, B., Horgan, D., Quan, J., Sendonaris, A., DulacArnold, G., Osband, I., Agapiou, J., Leibo, J. Z., andGruslys, A. Learning from Demonstrations for RealWorld Reinforcement Learning. In AAAI Conferenceon Artificial Intelligence, 2018.

Finn, C., Levine, S., and Abbeel, P. Guided Cost Learning:
Deep Inverse Optimal Control via Policy Optimization. In
International Conference on Machine Learning (ICML),
2016.

Ho, J. and Ermon, S. Generative Adversarial Imitation
Learning. In Advances in Neural Information Processing
Systems (NIPS), 2016.

Fu, J., Singh, A., Ghosh, D., Yang, L., and Levine, S. Variational inverse control with events: A general frameworkfor data-driven reward definition. In Advances in Neural Information Processing Systems 31, pp. 8538–8547,
2018.
Fujimoto, S., van Hoof, H., and Meger, D. Addressing
Function Approximation Error in Actor-Critic Methods.
arXiv preprint arXiv:1802.09477, 2018.
Giusti, A., Guzzi, J. J., Cirean, D. C., He, F.-L., Rodrı́guez,
J. P., Fontana, F., Faessler, M., Forster, C., Schmidhuber, J. J., Caro, G. D., Scaramuzza, D., Gambardella,Inoue, T., De Magistris, G., Munawar, A., Yokoya, T., andTachibana, R. Deep reinforcement learning for high precision assembly tasks. In Intelligent Robots and Systems(IROS), 2017 IEEE/RSJ International Conference on, pp.
819–825. IEEE, 2017.
Johannink, T., Bahl, S., Nair, A., Luo, J., Kumar, A.,
Loskyll, M., Aparicio Ojea, J., Solowjow, E., and Levine,
S. Residual Reinforcement Learning for Robot Control.
In IEEE International Conference on Robotics and Automation (ICRA), 2019.
Kalashnikov, D., Irpan, A., Pastor, P., Ibarz, J., Herzog,
A., Jang, E., Quillen, D., Holly, E., Kalakrishnan, M.,

Deep Reinforcement Learning for Industrial Insertion Tasks with Visual Inputs and Natural Rewards

Vanhoucke, V., and Levine, S. QT-Opt: Scalable Deep
Reinforcement Learning for Vision-Based Robotic Manipulation. In Conference on Robot Learning (CoRL),
2018.
Kober, J. and Peter, J. Policy search for motor primitives inrobotics. In Advances in Neural Information Processing
Systems (NIPS), volume 97, pp. 83–117, 2008. ISBN1099401052236. doi: 10.1007/978-3-319-03194-1 4.
Levine, S., Finn, C., Darrell, T., and Abbeel, P. End-toEnd Training of Deep Visuomotor Policies. Journal of
Machine Learning Research (JMLR), 17(1):1334–1373,
2016. ISSN 15337928. doi: 10.1007/s13398-014-0173-7.
2.
Levine, S., Pastor, P., Krizhevsky, A., and Quillen, D. Learning Hand-Eye Coordination for Robotic Grasping with
Deep Learning and Large-Scale Data Collection. International Journal of Robotics Research, 2017.
Li, R., Platt, R., Yuan, W., Ten Pas, A., Roscup, N., Srinivasan, M. A., and Adelson, E. Localization and Manipulation of Small Parts Using GelSight Tactile Sensing.
In International Conference on Intelligent Robots andSystems (IROS), 2014.
Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T.,
Tassa, Y., Silver, D., and Wierstra, D. Continuous control with deep reinforcement learning. In International
Conference on Learning Representations (ICLR), 2016.
ISBN 0-7803-3213-X. doi: 10.1613/jair.301.
Luo, J., Solowjow, E., Wen, C., Aparicio Ojea, J., Agogino,A., Tamar, A., and P, A. Reinforcement learning on
variable impedance controller for high-precision robotic
assembly. In Robotics and Automation (ICRA), 2019
IEEE International Conference on. IEEE, 2019.
Mayton, B., LeGrand, L., and Smith, J. R. Robot, feedthyself: Plugging in to unmodified electrical outlets bysensing emitted ac electric fields. pp. 715–722, 2010.
Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A.,
Antonoglou, I., Wierstra, D., and Riedmiller, M. Playing Atari with Deep Reinforcement Learning. In NIPS
Workshop on Deep Learning, pp. 1–9, 2013. ISBN 14764687 (Electronic) 0028-0836 (Linking). doi: 10.1038/
nature14236.
Nair, A., Mcgrew, B., Andrychowicz, M., Zaremba, W., andAbbeel, P. Overcoming Exploration in Reinforcement
Learning with Demonstrations. In IEEE International
Conference on Robotics and Automation (ICRA), 2018.
Nakanishi, J., Morimoto, J., Endo, G., Cheng, G., Schaal, S.,
and Kawato, M. Learning from demonstration and adaptation of biped locomotion. In Robotics and Autonomous

Systems, volume 47, pp. 79–91, 2004. ISBN 0921-8890.
doi: 10.1016/j.robot.2004.03.003.
Ng, A. Y., Harada, D., and Russell, S. Policy invarianceunder reward transformations: Theory and application to
reward shaping. In International Conference on Machine
Learning (ICML), 1999.
Pathak, D., Mahmoudieh, P., Luo, G., Agrawal, P., Chen,
D., Shentu, Y., Shelhamer, E., Malik, J., Efros, A. A., andDarrell, T. Zero-Shot Visual Imitation. In International
Conference on Learning Representations (ICLR), 2018.
Peters, J. and Schaal, S. Reinforcement learning of motor
skills with policy gradients. Neural Networks, 21(4):682–
697, 2008. ISSN 08936080. doi: 10.1016/j.neunet.2008.
02.003.
Peters, J., Mülling, K., and Altün, Y. Relative Entropy Policy
Search. In AAAI Conference on Artificial Intelligence, pp.
1607–1612, 2010.
Pinto, L. and Gupta, A. Supersizing Self-supervision: Learning to Grasp from 50K Tries and 700 Robot Hours. IEEE
International Conference on Robotics and Automation(ICRA), 2016.
Pomerleau, D. A. Alvinn: An autonomous land vehicle ina neural network. In Advances in Neural Information
Processing Systems (NIPS), pp. 305–313, 1989. ISBN
1-558-60015-9.
Pong, V., Gu, S., Dalal, M., and Levine, S. Temporal Difference Models: Model-Free Deep RL For Model-BasedControl. In International Conference on Learning Representations (ICLR), 2018.
Rajeswaran, A., Kumar, V., Gupta, A., Schulman, J.,
Todorov, E., and Levine, S. Learning Complex Dexterous Manipulation with Deep Reinforcement Learningand Demonstrations. In Robotics: Science and Systems,
2018.
Rhinehart, N. and Kitani, K. M. First-person activity forecasting with online inverse reinforcement learning. In
The IEEE International Conference on Computer Vision(ICCV), Oct 2017.
Silver, T., Allen, K., Tenenbaum, J., and Kaelbling, L. Residual Policy Learning. dec 2018.
Tamar, A., Thomas, G., Zhang, T., Levine, S., and Abbeel,
P. Learning from the hindsight plan episodic mpc improvement. In 2017 IEEE International Conference on
Robotics and Automation (ICRA), pp. 336–343, 2017.
Thomas, G., Chien, M., Tamar, A., Ojea, J. A., and Abbeel,
P. Learning Robotic Assembly from CAD. In IEEE

Deep Reinforcement Learning for Industrial Insertion Tasks with Visual Inputs and Natural Rewards

International Conference on Robotics and Automation(ICRA), 2018.
Van Hasselt, H., Guez, A., and Silver, D. Deep Reinforcement Learning with Double Q-learning. In Association
for the Advancement of Artificial Intelligence (AAAI),
2016. ISBN 1509.06461v3.
Večerı́k, M., Hester, T., Scholz, J., Wang, F., Pietquin, O.,
Piot, B., Heess, N., Rothörl, T., Lampe, T., and Riedmiller,
M. Leveraging Demonstrations for Deep Reinforcement
Learning on Robotics Problems with Sparse Rewards.
CoRR, abs/1707.0, 2017.
Ziebart, B. D., Maas, A., Bagnell, J. A., and Dey, A. K.
Maximum Entropy Inverse Reinforcement Learning. In
AAAI Conference on Artificial Intelligence, pp. 1433–
1438, 2008. ISBN 9781577353683 (ISBN).