IEEE ROBOTICS AND AUTOMATION LETTERS. PREPRINT VERSION. ACCEPTED JUNE, 2023

1

Learning Adaptive Policies for Autonomous Excavation
Under Various Soil Conditions by Adversarial Domain Sampling
Takayuki Osa1,2 , Naoto Osajima3 , Masanori Aizawa4 , Tatsuya Harada1,2

Abstract—Excavation is a frequent task in construction. In thiscontext, automation is expected to reduce hazard risks and laborintensive work. To this end, recent studies have investigated using
reinforcement learning (RL) to automate construction machines.
One of the challenges in applying RL to excavation tasks concerns
obtaining skills adaptable to various conditions. When the conditions of soils differ, the optimal plans for efficiently excavating
the target area will significantly differ. In existing meta-learning
methods, the domain parameters are often uniformly sampled;
this implicitly assumes that the difficulty of the task does notchange significantly for different domain parameters.
In this study, we empirically show that uniformly sampling thedomain parameters is insufficient when the task difficulty varies
according to the task parameters. Correspondingly, we develop
a framework for learning a policy that can be generalized tovarious domain parameters in excavation tasks. We propose twotechniques for improving the performance of an RL method in
our problem setting: adversarial domain sampling and domain
parameter estimation with a sensitivity-aware importance weight.
In the proposed adversarial domain sampling technique, thedomain parameters leading to low expected Q-values are actively
sampled during the training phase. In addition, we propose
a technique for training a domain parameter estimator basedon the sensitivity of the Q-function to the domain parameter.
The proposed techniques improve the performance of the RL
method for our excavation task. We empirically show that
our approach outperforms existing meta-learning and domain
adaptation methods for excavation tasks.
Index Terms—Robotics and Automation in Construction, Reinforcement Learning, Deep Learning Methods

I. INTRODUCTIONEXCAVATION is one of the most frequent tasks in construction, and automating such excavation is expected toreduce hazard risks and labor-intensive work. To this end, theautomation of excavation has been studied for decades [1]–[4].
Recent studies have employed reinforcement learning (RL) to
obtain policies for controlling an excavator [5]–[7]. In RL, theoptimal policy that maximizes the expected return is obtained
through autonomous trials and errors [8]. Based on recent
success of RL in robotics, RL is considered as a promising
approach for automating excavation.
*This work was done as a joint research by the University of Tokyo andKomatsu Ltd.
Manuscript received:February 7, 2023; Revised May, 10, 2023; Accepted
June 18, 2023.
This paper was recommended for publication by Editor Hyungpil Moon
upon evaluation of the Associate Editor and Reviewers’ comments.
1 T. Osa and T. Harada are with the University of Tokyo, Tokyo, 113-8656,
Japan. {osa, harada}@mi.t.u-tokyo.ac.jp
2 T. Osa and T. Harada are with RIKEN Center for Advanced Intelligence
Project, Chuo-ku, Tokyo, 103-0027, Japan
3 N. Osajima is with Kyushu Institute of Technology, Kitakyushu, Fukuoka,
808-0135, Japan
4 M. Aizawa is with Komatsu Ltd., kanagawa, 254-0913, Japan

Policy for excavation

Domain 1

State, reward

Domain 2
ActionActively collect samples in challenging domains

・・・
Domain 3

Fig. 1. To obtain an adaptive policy that can be generalized to various
condition, a policy is trained more frequently on more challenging domains.

When the conditions of soil are different, the optimal plansfor efficiently excavating the target area will be significantly
different. When considering the problem of learning a policythat can work across a set of various domains, the taskdifficulties can vary among a given domain distribution. The
problem of learning a policy applicable to various domains isoften formulated as multitask RL [9] or meta RL [10], which
help obtain a policy that works in or rapidly adapts to various
domains. However, in existing meta-RL and multitask-RL
methods, the domain parameters are often uniformly sampledduring the training phase, thereby implicitly assuming that thedifficulty of the task does not change significantly for different
domain parameters. However, we have observed variance inthe task difficulty within the range of domain parameters for
excavation tasks, and policies trained by existing methods
have not performed well for domains where a task was moredifficult than in other domains.
To address this issue, we develop a framework for learning a
policy that can be generalized to various domain parameters in
excavation tasks even if there is variance in the task difficulty.
We propose two techniques for improving the performanceof an RL method in our problem setting: adversarial domain
sampling, and domain parameter estimation with a sensitivityaware importance weight. In the proposed domain sampling
method, the domain parameters leading to low expected Qvalues are sampled during the training phase. This approach
can be viewed as a type of adversarial training where theagent attempts to maximize the expected return and the domain
sampler attempts to minimize the expected return. In addition,
we also propose a method for training a model to estimate thedomain parameters. In our excavation task, it is necessary toestimate the domain parameters to adjust the policy behavior.
To this end, we propose an importance weight incorporating
the sensitivity of the Q-values to the domain parameters. Our
experimental results show that the proposed method enables
us to obtain a policy that works in various domains (including challenging ones). Moreover, our approach outperforms

2IEEE ROBOTICS AND AUTOMATION LETTERS. PREPRINT VERSION. ACCEPTED JUNE, 2023

existing methods on an excavation task.
The remainder of the paper is structured as follows. Section III describes the related work. Section II introduces the
RL formulation and related approach. Section IV describes the
proposed method.. Section V explains the experimental results,
and Section VI describes the conclusions.
II. P RELIMINARY
A. Reinforcement Learning
We consider a Markov decision process defined by a tuple(S, A, P, r, γ, d), where S is the state space, A is the action
space, P(st+1 |st , at ) is the transition probability density,r(s, a) is the reward function, γ is the discount factor,
and d(s0 ) is the probability density of the initial state. A
policy π(a|s) : S × A 7→ R is defined as the conditional
probability density over the actions given the states. A returnis defined
P∞ as the sum of the discounted rewards over time
Rt = k=0 γ k rt+k . The goal of RL is to obtain a policy thatmaximizes the expected return E[R|π].
B. Meta-Learning
We consider a policy πθ parameterized with a vector θ.
The objective function for meta-learning in general can be
expressed as follows:
L(θ) = ET ∼p(T ) [E[R|πθ , T ]] ,(1)

where T represents a task, and p(T ) is the task distribution
to which the meta-policy must adapt. In meta-RL, a metapolicy is obtained by maximizing the expected return acrosstasks with respect to a given task distribution. In meta-RL
methods, such as PEARL, a policy conditioned on the latentvariable is typically trained in the training phase [11]–[13].
In the test phase, the model was adapted to a task sampledfrom the task distribution; however, the task descriptions
were unknown. Thus, latent variables or task descriptions
are typically estimated during the testing phase. However,in risk-aware RL [14], maximizing the expectation does notnecessarily improve the worst-case performance. As shownin our experiment, a policy trained with an existing meta-RL
method may not work for some of the tasks even if the average
performance across the tasks is acceptable.
C. Problem Setting in This Study
In our problem setting, we observe a depth image of thelandscape as a state s. An action a represents a single trajectory for excavation. We assume that the domain parameter
zdefines the characteristic of the soil in the target excavation
area and that the transition probability p(s′ |s, a; z) varies
according the domain parameter z. For example, the soil canbe hard and crumbly in certain domains while soft and sticky
in other domains. Thus, when the domain parameter z differs,
even if we take the same action under the same state, the next
state s′ can be significantly different. Our goal is to obtaina policy for planning a trajectory to efficiently excavate thetarget area under various values of the domain parameter.
To this end, we train a model conditioned on the domain

parameters that provides different excavation strategies fordifferent values of the domain parameters. We assume that thevalues of the domain parameters are fixed during an episode.
In this study, we assume that the domain parameter z is
known during the training phase. However, in the test phase,the domain parameter z is unknown and therefore needs tobe estimated. Considering the case where a policy is trainedin a simulation and tested in the real world, we think thatour problem setting is reasonable. Our problem setting is anintermediate problem between multitask RL and meta-RL. In
multitask RL, the task ID or task description is provided, andmultiple tasks are solved simultaneously. In meta-RL, unlike
multitask RL, the task ID/description is not provided during
both the training and test phases. Meta-RL often learns a policy
conditioned on the latent variable [11]; thus, the latent variablecorresponding to the test task needs to be estimated online in
the test phase.
D. QT-Opt
We briefly introduce QT-Opt [15], a variant of Deep Qlearning for tasks with a continuous action space. In QTOpt, the optimal Q-function is approximated by applying the
optimal Bellman operator to the approximated Q-function.
Given an approximated Q-function Qw (s, a) parameterizedwith a vector w, the Q-function is updated by minimizing an
objective function as follows:X2w=∥yi − Qw (si , ai )∥(2)(si ,ai )∈B

where B is a batch of samples selected from the replay buffer.
The target value yi is computed as follows:yi = ri + γ max
Qw′ (s′ , a′ ).
′
a

(3)

In QT-Opt, the maximum Q-values are approximated using
the cross-entropy method (CEM) [16]. Unlike actor-critic
methods [17], [18], a policy is not explicitly modeled in
QT-Opt. In this study, we extend QT-Opt to a meta-learning
setting.
III. R ELATED W ORK
Autonomous excavation approaches can be classified
into three categories: 1) optimization-based methods [19],2)imitation-learning-based (IM-based) methods [1], [4], and
3) RL-based methods [5]–[7]. Applying an optimization-based
approach, such as that reported in [19], allows us to realizea robust controller.. However, the method presented in [19]is applied to two-dimensional trajectories, and extending the
optimization-based approach to a three-dimensional space is
not trivial. Using RL for autonomous excavation can be advantageous as it requires less expert knowledge; furthermore,
it allows a policy for planning an excavation trajectory to beestablished even if the dynamics of the simulator are a blackbox [5]. In IM-based methods, data from human operators are
acquired and a controller is trained in a supervised learning
manner. However, compared with RL, IM-based methods
involve time-consuming and costly data acquisition processes.

IV. P ROPOSED M ETHOD
A. Meta-QT-Opt
In this study, our aim is to obtain a policy that maximizes
the expected return across various domains. Thus, our problem
can be formulated as follows:
max Ez∼p(z) Eπ,p(s′ |s,a;z) [Qπ (s, a, z)] ,π(4)where z is the domain parameter defining the transition
probability. p(z) is the prior distribution of z and defines thedomain distribution. The objective function in (4) can be seen
as a variant of the meta-learning objective in (1).
We extend the concept of QT-Opt [15], a variant of Qlearning [24], [25]. In a previous study [5], QT-Opt outperformed TD3 [18] and SAC [17] on the excavation task. In
our framework, we approximate the Q-function conditioned

FC(400)FC(500)concatenation

Batch norm

Batch norm

Conv2d(32,4,2)

Conv2d(16,4,2)

State

Batch norm

ActionConv2d(32,4,2)

FC(500)

Domain parameters

Depth image (65, 84, 1)

Recently, Pascal et al. applied RL to a 2D autonomous excavation and demonstrated that the obtained policy is applicableto actual excavator [7]. Lu et al. proposed to a framework
for excavating a pile of solid objects [20]. In their method, aconvolutional neural network predicted the excavation successfor a given image. Although the authors showed how their
method could be effectively applied to a real-world system,
their focus was on excavating solid objects and did not address
how to generalize to various soil conditions. Our study is built
upon a previous study in [5], which investigated an RL method
for learning to plan a trajectory for a depth image. However,the study in [5] did not address how to deal with various soil
conditions.
In the field of RL, methods have been investigated for obtaining a policy applicable to various conditions. One prevalentapproach is domain randomization, in which a policy is trained
under various conditions [21]. Another approach is metalearning [10], [11]. Given a task distribution, the goal of metalearning is to obtain a policy that can rapidly adapt to a given
test domain [10]. The fundamental difference between domain
randomization and meta-learning is that although domain randomization obtains a single robust policy for various domains,in meta-learning, the policy adapted to the test domain will
be different for each domain in a few-shot adaptation. Our
problem setting is close to those of [12], [13] in the sense thata model for estimating the domain parameters can be trainedin a supervised-learning manner. Nevertheless, the focus ofthese studies was on how to generalize the locomotion policies
for the various terrains; they did not investigate the effects ofthe variance of task difficulties among a set of target tasks.
In previous studies, learning methods were often evaluated inproblem settings where the task difficulty did not significantly
change among the given task distribution.
In this study, we show that it is necessary to employ amethod that explicitly incorporates the task difficulty in our
problem setting. Our approach can be seen as a type ofadversarial training, i.e., the max-min game, where the agent istrying to maximize the expected return and the domain sampler
is trying to minimize the expected return. The concept of ourapproach is similar to those of previous studies on min-max
games in adversarial training [22], [23].

3FC(500)

OSA et al.: LEARNING ADAPTIVE POLICIES FOR AUTONOMOUS EXCAVATION

Fig. 2. Neural network architecture for modeling Qw (s, a, z).

on the domain parameter z with a function Qw (s, a, z)
parameterized with a vector w. We update Qw (s, a, z) by
minimizing an objective function expressed as follows:X2∥yi − Qw (si , ai , z i )∥(5)
L(w) =(si ,ai ,s′i ,z i )∈B

where B represents a batch of samples selected from the replay
buffer. The target value yi is computed as follows:yi = ri + γ max
min Qw′j (s′i , a′ , z i ).
′
a

j=1,2

(6)

Here, the value of maxa′ Qw′j (s′ , a′ , z i ) is approximatedusing the CEM. When selecting an action a for a given states, the action is determined as follows:a∗ = arg max min Qwj (s, a, z).
a

j=1,2

(7)

In the standard CEM, N samples are randomly generated using
the sampling distribution at each iteration and the sampling
distribution is fit to the best M samples. As in [5], we use
the sample with the highest score as the output of the CEM
instead of the mean of the best M samples. The neural networkarchitecture in our implementation is shown in Fig. 2.
B. Adversarial Domain Sampling
In our excavation task, there is a variance in the taskdifficulty in a given task distribution. Thus, a uniform sampling of the domain parameter does not result in satisfactory
performance from the policy. To address this issue, we sample
a domain parameter that minimizes the expected Q-value
during the training phase. In the proposed method, the domain
parameter z is determined as follows:Xz ∗ = arg min
max min Qwj (s, a, z),
(8)
z

s∈B

a

j=1,2

where B is a batch of samples selected from the replay buffer.
This technique enables us to perform trials and errors underdomain parameters for which the current policy does notwork well. Therefore, this technique allows us to improve theworst case performance. Considering a case where a policy
π generates an action by following (7), our approach can beseen as a way of approximately solving a problem expressed
as follows:
max min Ez∼p̃(z) E(s,a)∼π,p(s′ |s,a;z) [Qπ (s, a, z)] .
π

p̃(z)

(9)

C. Training of Domain Parameter Estimator with SensitivityAware Importance Weight
In our problem setting, the value of the domain parameterz is known during the training phase, but unknown in the test
phase. Thus, we need to estimate the domain parameter z fromthe observed state-action pairs in the test phase. To this end,
it is necessary to obtain a model for estimating the domain
parameter. By denoting the model for estimating the domain
parameter by f ϕ (si , ai , s′i ) as parameterized by a vector ϕ,
the model can be trained by minimizing the mean squared
error as follows:X2(10)
z i − f ϕ (si , ai , s′i ) 2 ,L(ϕ) =(si ,ai ,s′i ,z i )∈B′

where B ′ is a batch of samples selected from the latest N
samples in the replay buffer. As the policy π changes over the
training phase, the distribution over the transitions (s, a, s′ )in the replay buffer differs from that induced by the current
policy. To mitigate the covariate shift between the training
and test data distributions, we select a batch of samples fromthe latest M samples instead of the entire replay buffer whenminimizing the loss in (10). In our implementation, we set
M = 10, 000.
In our preliminary experiment, we found that the estimationof the domain parameter was the bottleneck for achieving satisfactory performance. Thus, we propose training the domain
parameter estimator f ϕ (si , ai , s′i ) by using an importanceweight incorporating the sensitivity of the Q-value to thedomain parameter. In our method, the domain parameter
estimator is trained by minimizing the objective function as
follows:
X
2
L(ϕ) =
wi z i − f ϕ (si , ai , s′i ) 2 ,(11)(si ,ai ,si ,z i )∈B′

FC(400)FC(500)concatenation

Batch norm

Batch norm

Conv2d(32,4,2)+
Next stateConv2d(32,4,2)

State

Batch norm

ActionConv2d(16,4,2)

This formulation implies that our approach can be viewed asa type of adversarial training with the max-min game [23],i.e., the agent attempts to maximize the expected return andthe domain sampler attempts to minimize the expected return.
The proposed approach can also be regarded as generating an
automatic curriculum [26] based on the current estimate of the
Q-values.
In practice, the domain parameter z is either
continuousor discrete, and analytically computingP
minz s∈B maxa Qw (s, a, z) is not tractable. Thus,
we uniformly generate L samples of the domain parameterz
P and select the value leading to the minimum value ofs∈B maxa Qw (s, a, z). To maintain the balance between
exploration and exploitation during the training phase, we
use this strategy in an ϵ-greedy-like fashion; we sample
the domain parameter z uniformly with the probability ϵ;otherwise, the domain parameter z is determined based on(8). In our implementation, the domain parameter is sampled
in the beginning of an episode and fixed until the end of theepisode.

FC(500)IEEE ROBOTICS AND AUTOMATION LETTERS. PREPRINT VERSION. ACCEPTED JUNE, 2023

Depth image (65, 84, 2)

4Domain parameter

Fig. 3. Neural network architecture for fϕ (s, a, s′ ).

The corresponding calculation is as follows:
wi = min(wmax , max(w̄i , wmin ),w̄i = P∇z Qw (s, a, z)|s=si ,a=ai ,z=zi(sj ,aj ,z j )∈B(12)2∇z Qw (s, a, z)|s=sj ,a=aj ,z=zj

.
2

(13)
Herein, wmax and wmin are the upper and lower bound forclipping the importance weight, respectively. In our method,
we use the L2-norm of the gradient of the Q-value with respect
to the domain parameter ∥∇z Qw (s, a, z)∥2 as a metric forquantifying the sensitivity of the Q-value to the domain
parameter. When the norm of the gradient ∇z Qw (s, a, z) islarge at a point, the Q-value is considered to be sensitive to thevalue of the domain parameter z at that point and accurately
estimating the value of z is essential. We can incorporate
this observation using the importance weight in (13). As inother techniques with importance weights [27], we clip theimportance weight. In our implementation, we use wmin = 0.5
and wmax = 3.0.
D. Proposed Algorithm
The proposed training procedure is summarized in Algorithm 1. The domain parameter z is sampled in the beginningof the episode and is known to the RL agent in the training
phase. In the beginning of the training, the domain parameters
are uniformly sampled. The adversarial domain samplingstrategy described in Section IV-B is used after Nadapt time
steps. The model to estimate the domain parameter is trainedin a supervised manner using the importance weight proposedin (13).
The procedure for deploying the policy trained by Algorithm 1 is summarized in Algorithm 2. The domain parameterz is sampled in the beginning of the episode; however, its true
value is unknown to the RL agent. The domain parameteris estimated from an observed transition (s, a, s′ ) using thedomain parameter estimator f ϕ (s, a, s′ ). Subsequently, theaction that maximizes the Q-value Qw (s, a, z) is estimatedusing the CEM.
V. E XPERIMENTS
A. Simulation setup
We used a 3D excavation simulator developed by KomatsuLtd., as described in [5]. In the simulation, each action
represented a single trajectory of the excavator bucket. We
assumed that a controller for the excavator arm was provided.

OSA et al.: LEARNING ADAPTIVE POLICIES FOR AUTONOMOUS EXCAVATION

Algorithm 1 Meta-QT-Opt with Adversarial Domain Sampling and Sensitivity-Aware Importance Weight
Input: ϵa for the ϵ-greedy action exploration, ϵd for the ϵgreedy exploration of the domain parameter, time steps toactivate the adversarial domain sampling Nadapt
Initialize the experience replay buffer D, the parameters forthe domain parameter estimator ϕ, and the parameters forcritics and target critics wi , w′i for i = 1, 2.
Set the number of samples in D as N = 0
for each episode doGenerate random value x ∈ [0, 1]
if x < ϵd and N > Nadapt thenSample the domain parameter randomly
else
Select the domain parameter using the strategy in (8)end iffor t = 0 to T do
generate random value xa ∈ [0, 1]
if xa < ϵa thenSelect action randomly
else
Select action that maximizes Qw (s, a, z)
end ifObserve reward r and new state s′Store tuple (s, a, s′ , r) in D
N ←N +1
Sample mini-batch B from D
Update the critics by minimizing LQ (wi ) in (5)Update the target critics by w′i ← (1 − τ )w′i + τ wi
Sample mini-batch B ′ from the latest N samples
Update ϕ by minimizing L(ϕ) in (11)end forend for
Algorithm 2 Meta-QT-Opt Meta-Testing
Input: Domain distribution for testing ptest (z)for each episode do
Sample domain parameter z ∼ ptest (z)for t = 0 to T do
if t = 0 thenSet ẑ = 0
else
Estimate the domain parameter ẑ = f ϕ (s, a, s′ )end ifSelect action: a∗ = arg max Qw (s, a, ẑ)
Observe reward r and new state s′
end forend forThis policy generated a trajectory for a specified controller.
The bucket trajectory was approximated as an arc, and theaction corresponded to the bucket trajectory parameter. The arc
trajectory was parameterized with a three-dimensional vector,
which was used as the action. The state was provided by adepth image capturing the landscape in front of the excavator.
The number of observations was 65 × 84, and we used only

5

The color indicates the height

70

50

30

10

Position of
The area outside thethe excavator
sensor range is filledwith zero(a) Example of the initial state0(b) Example of the goal stateFig. 4. Initial and goal states of the excavation task. An initial state is
randomly generated in the beginning of the episode. The state is represented
as a depth map of the landscape.
TABLE IC HARACTERISTICS OF S OILS IN T EST D OMAINS
Domain nameHardness
Crumbliness
Test Domain 1
Hard
Solid
Test Domain 2
Hard
Crumbly
Test Domain 3
Soft
Solid
Test Domain 4
Soft
Crumbly

the depth information. An example of a depth image is shownin Fig. 4. The range of the sensor was presented in sector
form, and the shape of the state was arranged in a rectangle
by filling the blank region with zeros. The reward was basedon the amount of soil removed by the previous action. For
example, if the amount of soil removed by the previous action
is 0.5 m3 , then the reward is 0.5. The episode was terminatedwhen the amount of soil removed by the previous action
was less than a predefined threshold, i.e., when the bucketof the excavator was almost empty. The dimensionality of thedomain parameters was nine. In our simulation, to represent
the stochastic behavior of soil, the form of the remaining soil
was collapsed stochastically. Parameters that define the degreeand probability of soil collapse were included in the domain
parameters.
To evaluate the performance of the trained policy, we usedfour sets of typical domain parameters within their respective ranges. The characteristics of the four test domains are
summarized in Table I. In general, when the soil is hard,an excavator cannot dig deeply. When the soils are crumbly,
the remaining soil in the target area can easily collapse andtends to be relatively smooth. When the soil is hard, theexcavator bucket cannot dig deeply. Therefore, the amountof soil that can be removed by a single action is less when
the soil is harder. Consequently, when the soil is harder,
more processes are required to completely remove the target
amount of soil. When more processes are required to complete
the task, the estimation of the Q-function becomes morechallenging because more time steps must be considered toaccurately predict the return. Additionally, the soil collapses
stochastically. If the probability of soil collapse is high, thenthe prediction of return becomes uncertain, which can result in
more challenging tasks. In addition to these four test domains,
we evaluated the average return over various domains by
uniformly sampling the domain parameters.
To demonstrate the differences in the task difficulty, Fig. 5

6IEEE ROBOTICS AND AUTOMATION LETTERS. PREPRINT VERSION. ACCEPTED JUNE, 2023

35
30

Return25
20
15
test domain 1
test domain 2
test domain 3
test domain 4

10
5
0

0

50

100
150
Steps[x1000]

200

250

Fig. 5. The learning curve of QT-Opt when a policy is directly trained oneach test domain. Results on four test domains are shown, respectively. Thereis a variance of the task difficulty among these four domains.

depicts the learning curves of QT-Opt on each test domain.
In Fig. 5, a policy is trained on a single domain, unlike inmeta-RL and multitask RL. As shown, learning a policy on
test domain 2 requires more iterations of trial-and-error than in
the other test domains. This result demonstrates the variancein the task difficulty in a given task distribution. Although
the variance of task difficulty was insignificant, it significantly
affected our task, as will be discussed later.
As a baseline, we used the variant of QT-Opt proposedin [5], which employed conservative adversarial training to
reduce the overestimation of the Q-values. We evaluated avariant of QT-Opt in [5] with domain randomization. Below,
we refer to this baseline method as QT-Opt with domain
randomization. In addition, we evaluated a simplified versionof PEARL proposed in [11]. In PEARL, the Q-function
conditioned on the latent variable Qw (s, a, z) is estimated,
and a latent-conditioned policy π(a|s, z) is learned using SAC
as a base RL algorithm. Although the original PEARL learns
the latent variable in an unsupervised manner, we consideredwith the domain parameters as the latent variable in PEARL,
and the posterior distribution was trained in a supervisedlearning manner. Thus, we refer to this variant of PEARL
as “simple-PEARL.”
B. Results
Fig. 6 shows the learning curves for the baseline methods
and proposed method. We report the average return over fiverandom seeds with 30 test episodes. In Fig. 6, the shaded
area represents the 90 % confidence region. Notably, policies
trained with the baseline methods do not work in test domain 2even if the average performance across the tasks is acceptable.
In QT-Opt with domain randomization, simple-PEARL, andmeta-QT-Opt, the performance on test domain 2 evidently
does not improve appropriately during the training. This resultshows that uniform domain sampling, as often used in previous
studies, is not effective when there is significant variance inthe task difficulty among a given task distribution. In contrast, when using the proposed adversarial domain samplingtechnique, meta-QT-Opt significantly outperformed baseline
methods. Our proposed technique successfully addresses theabove issue by actively sampling the domains where theexpected return is low. Interestingly, our adversarial domain
sampling improves the overall performance across various

domains as well as the performance on the most challenging
domain. In addition, the performance was further improved using a sensitivity-aware importance weight to train the posterior
distribution. As shown in Fig. 6(f), the mean squared errors ofthe domain parameter estimated by the proposed method withand without importance sampling did not differ significantly.
The proposed importance-sampling method aims to assign ahigher weight to the estimation accuracy at the point where
the Q-function is more sensitive to domain parameters. Thus,
using importance weighting does not significantly affect theoverall estimation error. However, as shown in Table II, theproposed importance sampling technique improved the policy
performance.
The performances of the baseline methods deteriorated as
the training progressed. A recent study [28] showed that
Q-learning-based algorithms typically result in performance
deterioration, which is attributable to unsatisfactory featurelearning in Q-learning based on bootstrapping. Based on the
results, the proposed algorithm can mitigate the abovementioned issue; however, further investigation is required. Table IIsummarizes the performances of the baseline and proposed
methods after training. The proposed method significantly
outperforms the baseline methods across the test domains.
The difference between the baseline methods and proposed
method is especially evident on test domain 2. Although thepolicies trained by the baseline methods do not work wellon test domain 2, the proposed method successfully trains a
policy that works on test domain 2 as well as on the otherdomains.
Fig. 7 visualizes the Q-function with different valuesof the domain parameter. For the visualization, we show
maxa3 Qw (s, a, z) as the action is three-dimensional in our
excavation task. We show a depth image used as state s in
Fig. 7 (a). The plots of maxa3 Qw (s, a, z) with the valuesof the domain parameter that correspond to the domain 1,
domain 2, domain 3, and domain 4 are shown in Fig. 7(b)(e), respectively. Different scales are used in Fig. 7(b)-(e)to visualize the landscape of the Q-function. As shown, theoutputs of the trained model of Qw (s, a, z) change according
to the value of the domain parameter. The dense red area
corresponds to the distribution of near optimal actions in Fig. 7
and the location of the dense red area is significantly different
between test domaion 2 and the other test domains. This result
implies that different strategies were obtained for different
domains using the proposed method.
Fig. 8 shows the values of the domain parameter sampled during the training with the proposed adaptive samplingtechnique. For the visualization, the values of the domain
parameters are normalized to the range [−1, 1]. Among thedomain parameters, z4 , z5 , z6 , and z7 are discrete variables.
In this study, the domain parameter was sampled uniformly
in the beginning of the training and the adversarial domain
sampling was activated after 50,000 time steps, i.e., after
approximately 800 episodes in Fig. 8. In the top row of Fig. 8,
the values near z0 = −1 and z2 = 1 are intensively sampled
after the adversarial sampling strategy is activated,implyingthat these parameters significantly affect the task difficulty.
In test domain 2, which we found was the most challenging

30

25

25

20

20

20

15

Return

30

25

15

15

10

10

10

5

5

5

0

0

100

200
Steps[x1000]

300

400

0

25

20

20
Return

30

25

10

5

5

0

0

100

200
Steps[x1000]

300

400

300

100

200
Steps[x1000]

300

0

100

200
Steps[x1000]

300

400

(c) Meta-QT-Opt with uniform domain
sampling.

rand. domain
test doamin 1
test domain 2
test domain 3
test domain 4
0

0

400

15

10

0

200
Steps[x1000](b) Simple-PEARL.

30

15

100

400

MSE of domain parameter estimation0(a) QT-Opt with domain randomization.

Return7

30

ReturnReturnOSA et al.: LEARNING ADAPTIVE POLICIES FOR AUTONOMOUS EXCAVATION

12

10

8

6

4

2

0

with importance sampling
without importance sampling
0

50

100

150

200

250

300

350

400

Steps[x1000](d) Meta-QT-Opt with adversarial domain (e) Meta-QT-Opt with adversarial domain (f) Accuracy of domain parameter estimasampling.
sampling + importance weight (IW).
tion.
Fig. 6. Learning curves of the proposed and baseline methods. (a)-(e) show the average returns of each method during training. The proposed method,
Meta-QT-Opt + adaptive sampling + IW, achieved the best performance in all of the test domains. (f) shows the accuracy of domain parameter estimation.
TABLE IIP ERFORMANCE ON TEST DOMAINS AFTER 400,000 TRAINING STEPS .
Domain name

QT-Optwith
Domain rand.

Simple-PEARL

Meta-QT-Opt

Test domain 1
Test domain 2
Test domain 3
Test domain 4
Random Domain20.8±5.1
6.4±1.4
20.4±5.7
17.5±6.0
19.1±5.918.2±1.3
10.2±2.3
18.5±2.5
16.5±2.819.4± 1.4

20.9± 10.3
6.8±3.919.3±8.916.2±7.0
17.4± 7.5

Meta-QT-Opt+ adv. sampling
22.0±1.2
17.5±2.3
23.0±1.0
20.9±1.622.3±1.7

Meta-QT-Opt+ adv. sampling+ IW
22.3±1.6
19.5±1.0
23.5±1.822.7±1.4
23.1±1.2

domain, the values of these domain parameters are z0 = −1
and z2 = 1. This result indicates that the proposed method
actively samples the domain parameters leading to difficult
tasks. Therefore, we conclude that our approach successfully
improves the worst case performance.

one must address the remaining challenges. For example,
the robustness of the policy against observational noise mustbe guaranteed. In reality, depth images obtained from depth
sensors typically contain non-negligible noise. We will addressthese remaining issues in our future studies.

VI. CONCLUSIONSACKNOWLEDGMENT

In this study, we investigated a learning framework forplanning a trajectory for automating an excavator in various
domains. Although prior works on meta-RL and multitask
RL often do not pay attention to the variance in the taskdifficulty among a given task distribution, we empiricallyshowed that it is important to consider the variance in thetask difficulty. We proposed two techniques for improving
the performance of an RL method in our problem setting:adversarial domain sampling and domain parameter estimation
with a sensitivity-aware importance weight. Our adversarial
domain sampling technique can be regarded as a type of
the adversarial training that can help improve the worst-caseperformance. We empirically showed that the proposed method
significantly outperformed baseline methods on the excavation
task. To use the proposed framework in practical applications,This work was partially supported by JST AIP Acceleration Research JPMJCR20U3, Moonshot R&D Grant Number
JPMJPS2011, CREST Grant Number JPMJCR2015, JSPS
KAKENHI Grant Number JP19H01115, JP19K20370.
R EFERENCES[1] X. Shi, P. Lever, and F.-Y. Wang, “Experimental robotic excavation withfuzzy logic and neural networks,” in Proceedings of IEEE International
Conference on Robotics and Automation, vol. 1, 1996, pp. 957–962
vol.1.
[2] A. Stentz, J. Bares, S. Singh, and P. Rowe, “Robotic excavator forautonomous truck loading,” Autonomous robots, vol. 7, pp. 175–186, 1999.
[3] G. J. Maeda, D. C. Rye, and S. P. N. Singh, “Iterative autonomous
excavation,” Field and Service Robotics, pp. 369–382, 2014.
[4] R. Fukui, T. Niho, M. Nakao, and M. Uetake, “Imitation-based control
of automated ore excavator: improvement of autonomous excavation
database quality using clustering andassociation analysis processes,”
Advanced Robotics, vol. 31, no. 11, pp. 595–606, 2017.

8

IEEE ROBOTICS AND AUTOMATION LETTERS. PREPRINT VERSION. ACCEPTED JUNE, 2023

(a) Example of the state(b) Q-function (top) and(c) Q-function (top) and(d) Q-function (top) and(e) Q-function (top) andthe resulting action (bottom) the resulting action (bottom) the resulting action (bottom) the resulting action (bottom)
for test domain 1
for test domain 2
for test domain 3
for test domain 4

Fig. 7. Visualization of Q-function and action corresponding to state in (a) with different domain parameters.

Fig. 8. Visualization of domain parameters sampled using the proposed
adversarial domain sampling.
[5] T. Osa and M. Aizawa, “Deep reinforcement learning with adversarial
training for automated excavation using depth images,” IEEE Access,vol. 10, pp. 4523–4535, 2022.
[6] B. J. Hodel, “Learning to operate an excavator via policy optimization,”
Procedia Computer Science, vol. 140, pp. 376–382, 2018.
[7] P. Egli, D. Gaschen, S. Kerscher, D. Jud, and M. Hutter, “Soiladaptive excavation using reinforcement learning,” IEEE Robotics andAutomation Letters, pp. 1–8, 2022.
[8] R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction,
2nd ed. MIT Press, 2018.
[9] D. Kalashnikov, J. Varley, Y. Chebotar, B. Swanson, R. Jonschkowski,
C. Finn, S. Levine, and K. Hausman, “Scaling up multi-task robotic
reinforcement learning,” in Proceedings of the 5th Conference on Robot
Learning, ser. 164, 2022, pp. 557–575.
[10] C. Finn, P. Abbeel, and S. Levine, “Model-agnostic meta-learning forfast adaptation of deep networks,” in Proceedings of the International
Conference on Machine Learning (ICML), vol. 70, 2017, pp. 1126–1135.
[11] K. Rakelly, A. Zhou, C. Finn, S. Levine, and D. Quillen, “Efficient offpolicy meta-reinforcement learning via probabilistic context variables,”in Proceedings of the International Conference on Machine Learning(ICML), vol. 97, 2019, pp. 5331–5340.
[12] A. Kumar, Z. Fu, D. Pathak, and J. Malik, “Rma: Rapid motor adaptation
for legged robots,” in Procedings of Robotics: Science and Systems,
2021.
[13] A. Agarwal, A. Kumar, J. Malik, and D. Pathak, “Legged locomotion
in challenging terrains using egocentric vision,” in Proceedings of theConference on Robot Learning (CoRL), 2022.
[14] C. Bodnar, A. Li, K. Hausman, P. Pastor, and M. Kalakrishnan, “Quantile
qt-opt for risk-awarevision-based robotic grasping,” in Robotics andScience and Systems, 2020.
[15] D. Kalashnikov, A. Irpan, P. Pastor, J. Ibarz, A. Herzog, E. Jang,
D. Quillen, E. Holly, M. Kalakrishnan, V. Vanhoucke, and S. Levine,“Qt-opt: Scalable deep reinforcement learning for vision-based robotic
manipulation,” Proceedings of the Conference on Robot Learning, 2018.

[16] P. T. de Boer, D. P. Kroese, S. Mannor, and R. Y. Rubinstein, “A tutorial
on the cross-entropy method,” Annals of Operations Research, vol. 134,
pp. 19–67, 2005.
[17] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine, “Soft actor-critic: Offpolicy maximum entropy deep reinforcement learning with a stochastic
actor,” in Proceedings of the International Conference on Machine
Learning (ICML), 2018.
[18] S. Fujimoto, H. van Hoof, and D. Meger, “Addressing function
approximation error in actor-critic methods,” in Proceedings of theInternational Conference on Machine Learning, ser. Proceedingsof Machine Learning Research, J. Dy and A. Krause, Eds.,
vol. 80.
PMLR, 2018, pp. 1587–1596. [Online]. Available:
http://proceedings.mlr.press/v80/fujimoto18a.html[19] Y. Yang, P. Long, X. Song, J. Pan, and L. Zhang, “Optimization-based
framework for excavation trajectory generation,” IEEE Robotics andAutomation Letters, vol. 6, no. 2, pp. 1479–1486, 2021.
[20] Q. Lu and L. Zhang, “Excavation learning for rigid objects in clutter,”IEEE Robotics and Automation Letters, vol. 6, no. 4, pp. 7373–7380,
2021.
[21] J. Tobin, R. Fong, A. Ray, J. Schneider, W. Zaremba, and P. Abbeel,“Domain randomization for transferring deep neural networks from simulation to the real world,” in Proceedings of the IEEE/RSJ International
Conference on Intelligent Robots and Systems (IROS), 2017.
[22] J. Bose, G. Gidel, H. Berard, A. Cianflone, P. Vincent, S. Lacoste-Julien,and W. Hamilton, “Adversarial example games,” in Advances in Neural
Information Processing Systems (NeurIPS), 2020.
[23] J. Wang, T. Zhang, S. Liu, P.-Y. Chen, J. Xu, M. Fardad, and B. Li,“Adversarial attack generation empowered by min-max optimization,” inAdvances in Neural Information Processing Systems (NeurIPS), 2021.
[24] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G.
Bellemare, A. Graves, M. Riedmiller, and A. K. Fidjeland, “Humanlevel control through deep reinforcement learning,” Nature, vol. 518,
no. 7540, pp. 529–533, 2015.
[25] C. J. C. H. Watkins and P. Dayan, “Q-learning,” Machine Learning,
vol. 8, pp. 279–292, 1992.
[26] S. Sukhbaatar, Z. Lin, I. Kostrikov, G. Synnaeve, A. Szlam, and
R. Fergus, “Intrinsic motivation and automatic curricula via asymmetricself-play,” in Proceedings of the International Conference on Learning
Representations (ICLR), 2018.
[27] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,” arXiv, 2017, arXiv:1707.06347.
[28] A. Kumar, R. Agarwal, D. Ghosh, and S. Levine, “Implicit underparameterization inhibits data-efficient deep reinforcement learning,” inProceedings of the International Conference on Learning Representations (ICLR), 2021.