Object-Aware Regularization for Addressing Causal Confusion in Imitation Learning
Authors: Jongjin Park
Younggyo Seo
Chang Liu
Li Zhao
Tao Qin
Jinwoo Shin
Tie-Yan Liu
Venue: NeurIPS 2021 Poster (2021)
Keywords: imitation learning
behavioral cloning
causal confusion
regularization

Abstract: Behavioral cloning has proven to be effective for learning sequential decision-making policies from expert demonstrations. However, behavioral cloning often suffers from the causal confusion problem where a policy relies on the noticeable effect of expert actions due to the strong correlation but not the cause we desire. This paper presents Object-aware REgularizatiOn (OREO), a simple technique that regularizes an imitation policy in an object-aware manner. Our main idea is to encourage a policy to uniformly attend to all semantic objects, in order to prevent the policy from exploiting nuisance variables strongly correlated with expert actions. To this end, we introduce a two-stage approach: (a) we extract semantic objects from images by utilizing discrete codes from a vector-quantized variational autoencoder, and (b) we randomly drop the units that share the same discrete code together, i.e., masking out semantic objects. Our experiments demonstrate that OREO significantly improves the performance of behavioral cloning, outperforming various other regularization and causality-based methods on a variety of Atari environments and a self-driving CARLA environment. We also show that our method even outperforms inverse reinforcement learning methods trained with a considerable amount of environment interaction.

Object-Aware Regularization for Addressing Causal Confusion in Imitation Learning
Jongjin Park
Younggyo Seo 1+
Chang Liu?
Li Zhao?
Tao Qin²
Jinwoo Shin
Tic-Yan Yan Liu
Korea Advanced Institute of Science and Technology
Microsoff Research Asia
Abstract
Behavioral cloning has proven to be effective for learning sequential decision- making policies from expert demonstrations, However, behavioral cloning often suffers from the causal confusion problem where policy relies on the noticeable effect of expert actions due to the strong correlation but not the cause This paper presents Object-aware REgularizatiOn (OREO), a simple technique imitation policy in object-aware manner. Our main idea is
desire.
that regularizes to encourage a policy to uniformly attend to all semantic objects, in order to pre- vent the policy from exploiting nuisance variables strongly correlated with expert actions. To this end, we introduce a two-stage approach: (a) we extract semantic objects from images by utilizing discrete codes from a vector-quantized variational autoencoder, and (b) we randomly drop the units that share the same discrete code together, i.e., masking OREO significantly improves the performance of behavioral cloming, outperform- ing various other regularization and causality-base methods on variety of Atari environments and self- self-driving CARLA environment. We also show that our method even outperforms inverse reinforcement learning methods trained with a considerable amount of environment interaction.
semantic objects. Our experiments demonstrate that
1 Introduction
Imitation learning (IL) holds the promise of learning skills behaviors directly from expert demon- strations, effectively reducing the need for costly and dangerous environment interaction [21, 45]. Its simplest and effective form is behavioral cloning (BC), which learns policy by solving supervised learning problem over state-action pairs from expert demonstrations. While being simple, BC has been successful in a wide range tasks [4, 7, 31, 33] with careful designs. However, it has been recently evidenced that BC often suffers from the causal confusion problem, where the policy relies on nuisance variables strongly correlated with expert actions, instead of the true causes [11, 12, 54].
For example, when we train BC policy on the Atari Pong environment (see Figure la), we observe that policy relies on nuisance variables in images (i.e.. scores) for predicting expert actions, instead of learning the underlying fundamental rule of the environment that experts would have used for making decisions. In particular, Table Ic shows that the policy trained using images with scores struggles to generalize to images with scores masked out (see Figure lb). However, the policy trained with masked images could generalize original images with scores, which shows that successfully learned the rule of the environment. This implies that learning the policy that can identify the true cause of expert actions is important for stable performance deployment time, where nuisance correlates usually do not hold as in expert demonstrations.
"qqual contribution, in alphabetical order. (jongjin. park,, younggyo. seeeggyo.seeeeeeee.oo Skaisttttt.aa This work was done while the author was an intern Microsoft Research Asia
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Setup
Scores
Train
Eval
Original
3.1 1.
Original
Masked
-15.61 9.2
Original
15.9 0.4
Masked
Masked
16.6 0.6
(a) Original
(b) Masked
(c) Performance of behavioral cloning
Figure I: Atari Pong environment with (a) original images and (b) images where scores are masked out. (c) Performance of behavioral cloning (BC) policy trained Original and Masked environments, averaged over four runs. We observe that the policy trained with original images suffers in both environments, which shows that the policy exploits score information for predicting expert actions, instead of learning the underlying fundamental rule of the environment.
In order to address this causal confusion problem, one can consider causal discovery approaches to deduce the cause-effect relationships from observational data [26, 48]. However, it is difficult to apply these approaches to domains with high-ddmeesiionl inputs, as (i) causal discovery from observational data is impossible in general without certain conditions" [38], and (ii) these domains usually do not satisfy the assumption that inputs are structured into random variables connected by causal graph, e.g., objects in images [29, 46]. To address these limitations, de Haan et al. [12] recently proposed method that learns a policy on top of disentangled representations from a 88 3-VAE encoder [19] with random masking, and infers optimal causal mask during the environment interaction by querying interactive experts [43] or environment returns. However, given that environment interaction could be dangerous and incur additional costs, we argue that it is important to develop method for learning the policy robust to causal confusion problem without such a costly environment interaction.
In this paper, we present OREO: Object-aware REgularizatiOn, a new regularization technique that addresses the causal confusion problem in imitation learning without environment interaction. The key idea of our method is to regularize a policy to attend uniformly to all semantic objects in images, in order to prevent the policy from exploiting nuisance correlates for predicting expert actions. To this end, we propose to extract semantic objects from raw images by utilizing vector cettor-quanized variational autoencoder (VQ-VAF) [35]. In our experiments, we discover that the units of a feature map corresponding to the objects with similar semantics, e.g., backgrounds, scores, and characters, are mapped into the same or similar discrete codes (see Figure 3). Based upon this observation, propose regularize the policy by randomly dropping units that share the same discrete code together throughout training. Namely, our method randomly masks out semantically similar objects, which allows object-aware regularization of the policy.
We highlight the main contributions of this paper below:
We present ORFO, simple and effective regularization method for addressing the causal confusion problem, and support the effectiveness of OREO with extensive experiments.
We show that OREO significantly improves the performance of behavioral cloning on confounded Atari environments [5, 12], outperforming various other regularization methods [13, 15, 55, 50] and causality- based methods [12, 47].
We show that OREO even outperforms inverse reinforcement learning methods trained with considerable amount of environment interaction [8, 20).
2
Related work
Imitation learning. Imitation learning (II.) aims to solve complex tasks where learning pol- icy from scratch is difficult or even impossible, by lcarning useful skills or behaviors from expert demonstrations [2. 18, 25, 37, 39, 40, 56]. There are two main approaches for IL: inverse reinforce- ment learning (IRL) methods that find a cost function under which the expert is uniquely optimal [8, 20, 34, 44, 591, and behavioral cloning methods that formulate the IL problem
supervised learning problem that predicts expert actions from states [3, 4, 7, 31, 33, 40J. Our work employs
de Haan et al. [12] showed that causal discovery methods that depend on faithfulness condition [38] are not applicable to imitation learning setup, the condition does not hold in environments with nuisance correlates.

---

Policy
Convolutional Encoder
Random Masking
Objects
Cg Co
Cd
000
VQ-VAE
Encoder
Codebook
Figure 2: Overview of OREO. We first train a VQ- (Q-V model that encodes images into discrete codes from a codebook, where each discrete (prototype) representation represents different semantic objects in images. We then regularize policy by randomly dropping units that share the same discrete code together, i.e., random objects, throughout training.
behavioral cloning as it exhibits the benefit of avoiding costly and dangerous environment interaction, which is crucial for applying imitation learning to real-world scenarios.
Distributional shift and the causal confusion problem. Despite its simplicity, BC is known to suffer from the distributional shift, where the state distribution induced by a policy gets different from the training distribution on which the policy was trained. Several approaches have been proposed for learning the policy robust to distributional shift, including interactive IL methods that query experts [42, 43, 51], and regularization techniques [4, 7J. Recently, it has been evidenced that distributional shift leads to the causal confusion problem [4, 12, 54] where a policy exploits the nuisance correlates in states for predicting expert actions. To address this problem, Bansal et al. [4] proposed to randomly drop previous samples from a sequence of samples, and Wen et [54] proposed an adversarial training scheme of removing information related to previous actions. The work closest to ours de Haan et al. [12], which learns policy with randomly masked disentangled representations and infers the best mask through during environment interaction. Our approach differs in that we regularize the policy to be robust to the causal confusion problem, without any environment interaction.
Causal discovery from observational data. Causal discovery aims to discover causal relations among variables by utilizing observational data [38]. Most prior approaches assume that inputs are structured as disentangled variables [6. 16, 26, 36, 48, 47], which often does not hold in domains with high-dimensional inputs, i.e., images. While Lopez-Paz et al. [29] demonstrated the possibility of observational causal discovery from high- high-dimensional images, combining causal models and representation learning in such domains still remains an open problem [46]. Hence, we instead explore the approach of regularizing a policy that operates on high-dimensional states.
3 Method
3.1
Preliminaries
We consider the standard imitation learning (IL) framework where an agent learns to solve target task from expert demonstrations. Specifically, IL is typically defined in the context of a discrete- time Markov decision process (MDP) [52] without an peplicitly-definee reward function, which is defined as a tuple A. p. 7, Pu). Here, S is the state space, A is the action space, p(s' |s, a) is the transition dynamics, Po is the initial state distribution, and 7 E [o. 1) is the discount factor. The goal mapping from states to actions, using set of expert demonstrations
of II is to learn a policy D [(8;, a.)),N In our problem setup, an agent cannot interact with the environment, hence should learn the policy by using only expert demonstrations.
3

---

10%
Figure 3: Visualization the discrete codes from a VQ-VAE model trained on 8 confounded Atari environments, where previous actions augmented to the images as nuisance variables following the setup in de Haan et al. [12]. The considered environments are Frostbite, Pong. Qbert, Gopher, KungFuMaster, BattleZone, Krull, and Boxing (from left to right, top to bottom). The odd columns show images from environments, and even columns show the corresponding quantized feature maps, respectively. The discrete codes visualized in ID using t-SNE [30]. We observe that the units with similar semantics (e.g., the paddles in Pong environment and the carrots in Gopher environment) exhibit similar colors, i.e., mapped into the same or similar discrete codes.
Behavioral cloning. Behavioral cloning (BC) reduces an imitation learning problem the supervised learning problem training a policy that imitates expert actions. Specifically, we introduce a policy that maps state 8t action dg and a convolutional encoder that maps state s, low-dimensional feature map. Then π and are learned by minimizing the negative log-likelihood of expert actions from demonstrations as follows:
Stc ot) log π (se)
(1)
where и is modeled as multinomial distribution over actions to handle discrete action spaces.
Vector quantized variational autoencoder. The VQ- VAE [35] model consists of an encoder g that compresses images into discrete latent representations, and decoder that reconstructs images from these discrete representations. Both encoder and decoder share codebook C' of prototype vectors which are also learned throughout training. Formally, given a state 8,, the encoder y encodes 8, into feature map h, E RLXD that consists of series of L latent vectors h4 E RDD, , {1,2, ..., L). Then h4 = g(s)) is quantized to discrete representations e E L based on the distance of latent vectors he, to the prototype vectors in the codebook C {ex K follows:
e = (eg(t.1)) (g(t.))) ... (eg(t,))) where q(1, i) = argmin |[,y, - exl122
kE[K]
where [K] the set (1, ... K}. Then the decoder learns to reconstruct s; from discrete represen- tations The VQ- VAE is trained by minimizing the following objective:
LvQvaE(st) = ||st d(ex)l1 + |sg [ha] - eel + B- ||sg [ee] h4 ?
(3)
Leecva
Loodabook
Cromnt
where the operator sg refers to a stop- gradient operator, Lrocon is a reconstruction loss for learning representations useful for reconstructing images, Leodebook is codebook loss bring codebook representations closer to corresponding encoder outputs h, and Ccomst. is a commitment loss weighted by prevent encoder outputs from fluctuating frequently between different representations.
3.2 OREO: Object-aware regularization for behavioral cloning
In this section, we present OREO: Object-aware REgularizatiOn that regularizes policy in an object- aware manner to address the causal confusion problem. Our main idea to encourage the policy to uniformly attend to all semantic objects in images, in order to prevent the policy from exploiting nuisance variables strongly correlated with expert actions. this end, we introduce a two-stage approach: we first train a VQ-VAE model that encodes images into discrete codes, then learn the policy with our regularization scheme of randomly dropping units that share the same discrete codes (see Figure 2 and Algorithm for the overview and pseudocode of OREO, respectively).

---

Algorithm Object-aware regularization (OREO)
Initialize parameters of encoder decoder d, codebook C, policy
while not converged do
VQ- VAE TRAINING
Sample {s: D.
Update parameters of g. d. C' by minimizing Ei=1 LyqVAE($i) end while
Initialize encoder f with parameters of
while not converged do
I/ UPDATE POLICY VIA BEHAVIORAL CLONING
Sample (+;; 0g /==1 D
Get random masks {M, in (4)
Update parameters of f, by minimizing E CORED 0s M,) (5)
end while
Extracting semantic objects. To regularize a policy in object-aware manner, propose
utilize discrete representations from a VQ-VAE model trained by optimizing the objective in (3) with images from expert demonstrations D. Our motivation comes from the observation that the units of feature map corresponding to similar objects are mapped into similar discrete codes (see Figure 3). Then, in order to extract semantic objects from images and utilize them for regularizing the policy, we propose to randomly drop the units of a feature map that share the same discrete code together throughout training. Formally, for each state 8t, we sample K binary random variables my E {0, 1}. k = 1, 1,2, ..., K from Bernoulli distribution with probability 1 where p is the drop probability.
Then, we construct mask M, by utilizing the discrete representations e (2) as follows:
M (M """ , where ((, i) argmin | ex ||2-
kE[K]
By considering units of feature map with the same discrete code, we remark that our method can effectively extract semantic objects from high-dimensional images.
Behavioral cloning with OREO. Now we propose to utilize our object-aware masking scheme for the regularization of a policy. To this end, we first initialize convolutional encoder f with the parameters of VQ-VAE encoder g. We empirically find that employing f as our backbone encoder for instead of fixed encoder is more effective, it allows encoder to learn useful information for predicting actions. Then, we train the policy æ by minimizing the following objective:
LasED (8t, 0t. M,)
log æ (s)) O M))
(5)
where denotes elementwise product, and the mask M, is shared across all channels in feature map (+x). Here, our intuition is that our object-aware regularization scheme should be useful for enforcing the policy not exploit specific objects strongly correlated with expert actions, as the policy should utilize all semantic objects throughout training. Additionally, following Srivastava et al. [50], scale the masked features by a factor of 1/(1 p) during the training ensure the scale of the expected output with masked features to match the scale of outputs at test time.
Experiments
In this section, we designed our experiments to answer the following questions:
How does OREO compare other regularization schemes that randomly drop units from feature map [15, 50], data augmentation schemes [13, 55], and causality-based methods [12. 47] (see Table 1)?
How does OREO compare to inverse reinforcement learning methods that learn a policy with environment interaction [8, 20] (see Figure 5)?
Why is regularization necessary for addressing the causal confusion problem (see Figure 6a), and why is OREO effective for addressing this problem (see Figure 8)?
Can OREO improve BC using various sizes of expert demonstrations (see Figure 7)?
Can OREO also address the causal confusion problem when inputs are high-dimensional, complex real-world images (see Table 3)?
5

---

Table 1: Performance of policies trained on various confounded Atari environments without environ- ment interaction. OREO achieves the best score on 15 out 27 environments, and the best median and mean report the mean of returns averaged over eight runs. We provide standard deviations in Appendix I. CCIL denotes the results without environment interaction.
score (HNS) over all environments, The results for each environment
Environment
BC
Dropout
DropBlock
Cutout
RandomShift
CCIL
CRLR
OREO
Alien
954.1
1003.8
926.4
973.3
806.5
820.0
82.5
1056.2
Amidar
95.8
89.4
110.1
118.7
98.0
74.9
12.0
105.7
Assault
793.8
820.4
815.0
687.6
828.9
683.3
0.0
840.9
Asterix
292.2
313.8
345.4
212.4
135.5
643.2
650.0
180.8
BankHeist
442.1
485.7
508.4
486.1
367.2
653.5
0.0
493.9
BattleZone
11921.2
12457.5
12025.0
11107.5
9180.0
6370.0
1468.8
12700.0
Boxing
18.8
20.3
32.2
20.5
38.3
34.8
-43.0
36.4
Breakout
5.7
5.4
4.8
2.0
0.5
0.0
4.2
ChopperCommand
874.2
921.4
919.4
1016.1
936.4
760.6
1077.2
977.4
CrazyClimber
45372.9
39501.6
38345.6
44523.2
41924.0
22616.8
112.5
55523.4
DemonAttace
157.2
180.5
167.8
173.1
241.8
171.3
0.0
224.5
Enduro
241.4
250.4
341.8
119.6
316.4
143.1
3.9
522.8
Frecway
32.3
32.4
32.7
32.5
33.0
33.1
21.4
32.7
Frostbite
116.3
124.5
128.2
139,4
121.6
53.3
80.0
129,9
Gopher
1713.9
1819.1
1818.2
1481.0
1995.0
1404.5
0.0
2515.0
Hero
11923.1
14109.7
14711.4
14896.6
12816.0
6567.8
346.2
15219.8
Jamesbond
419.0
451.0
473.8
381.8
428.4
387.2
502.8
Kangaroo
2781.5
2912.9
3217.1
2824.0
1923.9
1670.5
122.8
3700.2
Krull
3634.3
3892.1
3832.1
3656.4
3788.7
3090.8
0.1
4051.6
KungFuMaster
15074.8
14452.1
15753.0
11405.6
13389.9
13394.9
0.0
18065.6
MsPacman
1432.9
1733.1
1446.4
1711.0
1223.5
1084.2
105.3
1898.4
Pong
3,2
10.2
11.5
6.8
-0.1
-2.7
-21.0
14.2
PyivateFye
2681.8
2599.
2720.6
2670.6
3969.2
305.3
-1000.0
3124.9
Qbert
5438.4
6469.0
6140.3
5748.6
3921.4
5138.0
125.0
6966.4
RoadRunner
18381.5
21470.9
22265.4
12417.1
16210.0
11834.1
1022.9
24644.2
Seaquest
454.4
471.3
486.8
330.1
1016.8
271.2
172.5
753.1
UpNDown
4221.1
4147.1
4789.2
4159.6
3880.2
2631.1
20.0
4577.9
Median HNS
44.1%
47.4%
49.8%
42.0%
47.6%
36.2%
-1.5%
51.2%
Mean HNS
73.2%
79.0%
91.7%
69.5%
88.1%
71.7%
-4.9%
105.6%
Environments and datasets. We evaluate OREO on 27 Atari environments [5], which are lected by following prior works [12, 49]. Follow- ing de Haan et al. [12], we consider confounded Atari environments, where images are augmented with previous actions (see Figure 4). We utilize a single frame as an input to a policy, to focus on the causal confusion problem from nuisance correlates the current s In our experiments, report two evaluation metries: average
(a) Pong
(b) Enduro
(c) Seaquest
Figure Confounded Atari environments with previous actions (white number in lower left).
from environments and Hamman-normalieer score (HNS =
Agent Agent.zan Random ).
following Mnih et al.
Human Randomyere
[32]. For expert demonstrations, utilize DQN Replay dataset [i]. As this dataset consists of 50M
transitions of each environment collected during the training of DQN agent [32], we use the last N trajectories expert demonstrations. We preprocess input images to grayscale images of 84 x 84 by utilizing Dopamine library [9]. We provide more details in Appendix B.
Implementation. We use single Nvidia P100 GPU and CPU cores for each training run. The training time for OREO is 6 hours on the dataset of size 50000, compared hours for BC, which is because OREO additionally trains VQ- VAE model. As for hyperparameter selection, we use the default hyperparameters from previous similar works [35, 50], i.e., drop probability codebook size of K = 512, and a commitment cost of 8 = 0.25. We use the same hyperparameters across all environments. We report the results over runs unless specified. Source code and more details on implementation are available in Appendix A and B, respectively.
= 0.5,
*We refer to Wen et al. [54] for the discussion on the causal confusion problem from stacking states. While mainly focus the single frame setup, OREO is also effective multiple frame setup (see Appendix F).

---

CranyClnbe
Jammationd
Kangaoo
000
03330
.300
140
Eqpende Pat .m
400-
-0:0
500
2030
GAIL DRIL
20003
100
OREO (Ours)
10
15
10
20
Envirommnn Stepu (+10P)
Envrorment Stapa (x10%)
Envérammnnt Stupa (+10%)
KungFu Master
Pong
FitvateBye
20003
Epacde Bemum
Eplocdo Rutumn
3033
2030
10000
1030
2003
Envrrommnt Stops (x10f)
Environmam Steps (+10%)
Envronment Btops (x10f)
Figure 5: We compare OREO inverse reinforcement learning methods that require environment interaction for learning a policy, on confounded Atari environments. OREO outperforms baseline methods in most cases, even without using any interaction with environments. The solid line and shaded regions represent the mean and standard deviation, respectively, across eight runs.
Baselines. We consider BC as the most basic baseline method. To evaluate the effectiveness of our object-aware regularization scheme, we compare regularization techniques that drop the randomly sampled units (i.e., Dropout [50]) or randomly sampled blocks (i.e., DropBlock (15]) from the feature map of convolutional encoder. We also compare to data augmentation schemes, i.e., Cutout [13] that randomly masks out a square patch from images, and RandomShift [55] that randomly shifts pixels of images for regularization. We also consider the method of de Haan et al. [12] that learns a policy on top of disentangled representations from a 3- 3-V [19] (i.c., CCIL), and an observational causal inference method that estimates the causal contribution of each variable by confounder balancing (i.e., CRLR [47]). We provide the details for baselines in Appendix B and H.
Comparative evaluation. Table shows the performance of various methods that learn a policy without environment interaction. OREO significantly improves the performance of BC in most environments, outperforming other regularization techniques. In particular, OREO achieves the mean HNS of 105.6%, while the second-best method, i.e., DropBlock, achieves 91.7%. This demonstrates that our object-aware regularization scheme is indeed effective for addressing the causal confusion problem (see Figure 8 for qualitative experimental results). We found that CCIL without environment interaction does not exhibit strong performance in most environments, possibly due to the difficulty of learning disentangled representations from nigh-dimmesiooal images [28]. We also provide experimental results for CCIL with environment interaction in Appendix D, where the performance slightly improves but overall trends similar. We observe that CRLR underperforms in most envirouments, which shows the difficulty of causal inference from sigh-dimessional images. We emphasize that OREO also outperforms other baselines in original Atari environments, which implies that our method is also effective for addressing the causal confusion that naturally occurs (see Figure 1). We provide experimental results for the original setup in Appendix C.
Comparison with inverse reinforcement learning methods. To demonstrate that OREO can exhibit strong performance without environment interaction, we compare our method to inverse reinforcement learning (IRL) methods that first learn a reward function using expert demonstrations, and train a policy with environment interaction using learned reward function. Specifically, we consider GAIL [20]. a method that learns reward function by discriminating expert states from on-policy states during environment interaction; and DRIL [8], one of the strongest IRL methods that utilizes the disagreement between ensemble policies as a reward function. As shown in Figure 5, we observe that OREO exhibits superior performance to GAIL and DRIL on most confounded Atari environments, which are trained with 20M environment steps following the setup in Brantley et al. [8]. While IRL methods might outperform OREO asymptotically with more environment interaction, this result demonstrates that OREO indeed allows for achieving strong performance without interaction. We also found that GAIL exhibits almost zero performance in most environments, which similar to the observation of previous works that GAIL suffers in environments with high-dimensional image inputs [8, 12, 41]. We remark that OREO can also be applied to IRI. methods (see Appendix E for relevant experimental results of DRIL + OREO on confounded Atari environments).
7

---

Ep-sodo Rotum
840
Hanes 0e
0.0
(BC)
0.2
Validation Accuracy
40 60
100
0.25
0.5
0.75
128
512
Traiiin
(a) Effects of validation accuracy
Drop probability
Codebook size
Figure 6: (a) Score and validation accuracy on confounded Pong environment, where validation accuracy is not aligned with the score at test time, which necessitates the use of regularization for addressing the causal confusion problem. We visualize the performance of OREO over 8 confounded Atari environments with varying (b) the drop probability of each code from codebook and (c) codebook size. Boxplots are drawn using mean human- normalzzd scores obtained from eight runs.
Why is regularization necessary in confounded environments? A simple and widely used ap- proach to address the overfitting problem in supervised learning model selection with validation dataset. To see how this works in our setup, we first introduce validation dataset consisting of 5 expert demonstrations on confounded Pong environment, and visualize the scores and validation accuracies measured by policy learned with BC in Figure 6a. We observe that this simple scheme not helpful for confounded Atari environments, i.c., validation accuracy is not aligned with the score test time, because the distribution of the validation dataset could be significantly different from the distribution induced by learned policy. As the evaluation of the policy environments impossible, this result implies that regularizing the policy
during training could be dangerous necessary for successful imitation learning in confounded environments.
Effects of hyperparameters. We investigate the effect of two major hyperparameters, i.c., p (0.25, 0.5, 0.75) for the drop probability in (4), and K E {64, 128, 512} for the codebook size in (2). Figure 6b shows that the performance improves p increases, which implies that more strong regularization is effective for addressing the causal confusion problem on confounded Atari environments. Figure 6с shows that too small large codebook size could be harmful to the performance. We remark that our experiments used default hyperparameters' = 0.5 and K 512 for reporting the results, the performance of OREO could be further improved with more tuning.
Effects of expert demonstration size. To investi- gate the effectiveness of OREO with various sizes of expert demonstrations, we evaluate the performance of OREO with varying number of expert demon- strations N E {5. 10, 20, 35, 50). Specifically, we report the mean HNS over 8 confounded Atari en- vironments, which are randomly selected due to the high computation cost of running experiments for all environments. As shown in Figure 7, OREO con-
Human-nomalizess Score
0.6
BC
Dropout
sistently improves the performance of BC across a wide range of dataset sizes. As for the comparison with other baselines, OREO achieves superior per- formance to Dropout and DropBlock except for the extreme case N 5, which is because learning a VQ-VAE model with a limited number of data could be unstable. We also observe that DropBlock and OREO consistently outperform Dropout, which sup- ports our intuition that dropping individual units from feature map is not sufficient for effective regulariza- tion to address the causal confusion problem.
DropBlock
OREO (Ours)
36
Expert Demonstrations
Figure 7: Mean human-normalize normalized score over 8 confounded Atari environments with vary- ing number of expert demonstrations. The solid line and shaded regions represent the mean and standard deviation, respectively, across four runs.

---

Table Performance of policies trained on various confounded Atari environments without environ- ment interaction. (Q-VAE + BC learns BC policy on top of fixed VQ- VAE representations. The results for each environment report the mean and standard deviation of returns over eight runs.
Environment
BC
VQ-VAE BC
VQ-VAE Dropout
VQ- VAE DropBlock
OREO
BankHeist
442.1:1 207
358.8 8.1 258
491.1 = 28.9
488.01 49.7
493.91: 17.6
Enduro
241.4+
154.6+ 107
57.1 126
111.2 1644
522.8+ 29.1
KungeeMaster
15074.8+ 275.5
11055.++
13323.0+
14861.1+ 19615
18065.6+
Pong
3.2:1: 0.7
3.6
10.4
13.6
14.2
PrivatcEye
2681.8= 270.2
2255.8- 560.5
390,2 3009
746.8 527.8
3124.9:: 349.6
RoadRunner
18381,5= 1919,9
5783.2+ 403.6
6633.8 7163
7771 843.6
24644.2++ 2236.1
Seaquest
454.4+t 59.5
55. 2676.9
344.9
325.6 28.2
396.6 36.8
753.1 63.6
UpNDown
4221. 214.5
268.9
3310.81: S362
4073.9 7609
4577.9 9.1 307.6
Median HNS
62.7%
47.9%
45.3%
53.2 29.
72.9
Mean HNS
70.8%
41.3%
45.7%
53.0%
100.1 %
Confounded
Original
BC
0003
OREO (Ours)
Figure 8: We visualize the spatial attention map from convolutional encoder trained with BC and OREO in confounded (left) and original (right) Enduro environment. We observe that the encoder trained with OREO attends to important objects from images (e.g.. approaching cars), while the encoder trained with BC only attends the small region around a car.
Contribution of separate convolutional encoder. In order verify the effect of introducing an additional convolutional encoder f instead of learning a policy on top of the fixed VQ-VAE encoder in (5), we provide the experimental results that evaluate VQ-VAE BC, where BC policy is learned on top of fixed VQ-VAE representations in Table 2. We first observe that the performance of VQ-VAE VAE + BC performs worse than vanilla BC, which is because fixed VQ-VAE representations learned by reconstruction objective (3) do not contain fine-grained features required for imitating expert actions. Instead, one can that OREO significantly improves the performance of BC and outperforms all baselines based on fixed VQ-VAE representations, achieving the mean HNS of 100.1% 1% compared to 53.0% of VQ- VAE + DropBlock. This shows that OREO and BC, but a carefully designed method to exploit the discrete codes from VQ- IQ-VAE for object-aware regularization address the causal confusion problem.
the naïve combination of VQ-VAE
How does OREO improve the performance of BC? To understand how OREO improves the performance of BC, we visualize spatial attention maps from the last convolutional layer of a policy encoder in Figure 8. Specifically, following prior works [24, 57], we compute spatial attention map by averaging the absolute values of feature map along the channel dimension. We then apply 2 2-dimensional spatial softmax and multiply the upscaled attention map with images for visualization. We observe that the activations from the encoder trained with OREO are capturing all important objects of the environment (e.g., car that an agent controls and approaching cars), while the activations from BC are missing information of approaching cars by only focusing on the small region around and a scoreboard. This shows that regularization scheme that encourages policy to uniformly attend to all semantic objects allows for learning the policy that attends to important objects.
Effectiveness on real-world applications. To further demonstrate the effectiveness of OREO on real-world applications where inputs are high-dimensional, complex images, we additionally consider a self-driving CARLA environment [14]. Specifically, we train a conditional imitation learning policy

---

Table 3: Performance of policies trained on 150 expert demonstrations from the CARLA driving dataset, under weather condition of daytime. The results for each environment report the mean and standard deviation of success rates over four runs. OREO achieves the best success rate on all tasks.
Task
BC
Dropout
DropBlock
OREO
Straight
75.0
82.0=
74.0 3.5
87.0
One turn
43.0+t 9.1
59.0= 3.3
53.0 5.2
70.0 7.2
Navigation
16.9
30.4 10.7
21.7+ 9.2
35.7+
Navigation w/ dynamic obstacles
18.0 4.5
26.0= 6.0
19.0 5.2
30.0 4.5
[10] using 150 expert demonstrations from the dataset [53] consisting of 200 x 88 3 real-world images under a weather condition of daytime. Table 3 shows the average success rate of OREO and baseline methods on four CARLA benchmark tasks, i.c., Straight, One turn, Navigation, and Navigation with dynamies obstacles, where each task consists of 25 different navigation routes. The results show that OREO improves the performance of BC and outperforms other regularization methods, which implies that our object-aware regularization can also be effective more complex real-world applications.
5 Discussion
In this paper, we present OREO, a simple regularization method to address the causal confusion problem in imitation learning. OREO regularizes policy in an object-aware manner, by randomly dropping the units of a feature map that share the same discrete codes from a VQ- VAE model. Our experimental results demonstrate that OREO improves the performance of behavioral cloning without costly environment interaction, which is crucial for safe and successful imitation learning.
Limitations. One limitation of our method that it is only designed to regularize a policy when inputs are images, and not applicable to state-based environments. However, we still believe that OREO can be a practical solution to the causal confusion problem in various image-based applications. e.g., video games [32], self-driving [7], and robotic manipulation [23]. Another limitation is that we do not deduce the cause-effect relations for addressing the causal confusion problem, but instead regularize the policy to prevent it from exploiting nuisance correlates. However, given that it is an open problem to infer the structured disentangled variables and discover the causal relations among the variables [46], we believe encouraging the policy to attend to all semantic objects is reasonable and promising direction for addressing this problem.
Potential negative impacts. Real-world applications of behavioral cloning, e.g., autonomous driv- ing [4], require a large amount of data that often contain sensitive information, therefore raising privacy concerns. As our method is built upon variational autoencoder, it could be exposed to privacy violation attacks that infer training data information, such as model inversion [58], and mem- bership inference [17]. For example, the facial information of pedestrians may be reconstructed via membership inference attack. To address this vulnerability to privacy violation attacks, differentially private variational autoencoder would be required for real applications. In addition, pre-training VQ-VAE requires additional computing resources, which might lead to the increased energy cost for learning imitation learning policies. Also, behavioral cloning policy will imitate whatever demonstrations one specifies. If some bad actions are included in expert demonstrations, the policy would perform dangerous actions to users. For these reasons, addition to developing algorithms for better performance, it is also important to consider safe adaptation.
Acknowledgments and Disclosure of Funding
This research was supported by Microsoft Research Asia, Institute of Information & communications Technology Planning Evaluation (IITP) grant funded by the Korea government (MSIT) (No.2019- 0-00075, Artificial Intelligence Graduate School Program(KAISTT Science, ICT), Korea, under the High-Potential Individuals Global Training Program) 202-0-- 01649) supervised by the IIT (Institute for Information & Communications Technology Planning &
and the MSIT (Ministry of
10

---

Evaluation. We would like to thank Kimin Lee, Sangwoo Mo, Seonghyeon Park, Sihyun Yu, and anonymous reviewers for providing helpful feedbacks and suggestions in improving our paper.
References
[1] Agarwal, Rishabh, Schuurmans, Dale, and Norouzi, Mohammad. An optimistic perspective on offline reinforcement learning. In International Conference on Machine Learning, 2020.
[2] Aytar, Yusuf, Pfaff, Tobias, Budden, David, Paine, Tom Le. Wang, Ziyu, and de Freitas, Nando. Playing hard exploration games by watching youtube. In Advances in Neural Information Processing Systems, 2018.
[3] Bain, Michael and Sammut, Claude. A framework for behavioural cloning. In Machine Intelligence 15, 1996.
[4] Bansal, Mayank, Krizhevsky, Alex, and Ogale, Abhijit. Chauffeurnet: Learning to drive by imitating the best and synthesizing the worst. In Proceedings of Robotics: Science and Systems, 2019.
[5] Bellemare, Marc G, Naddaf, Yavar, Veness, Joel, and Bowling, Michael. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253- 279, 2013.
[6] Bengio, Yoshua, Deleu, Tristan, Rahaman, Nasim, Ke, Rosemary, Lachapelle, Sébastien, Bilaniuk, Olexa, Goyal, Anirudh, and Pal, Christopher. A meta-transfer objective for learning disentangle causal mechanisms. International Conference on Learning Representations, 2020.
[7] Bojarski, Mariusz, Del Testa, Davide, Dworakowski, Daniel, Firner, Bernhard, Flepp, Beat, Goyal, Prasoon, Jackel, Lawrence D, Monfort, Mathew, Muller, Urs, Zhang, Jiakai, et al. End end learning for self-driving cars. arXiv preprint
2016.
[8] Brantley, Kiante, Sun, Wen, and Henaff, Mikael. Disagreemeent-rgguaaiidd imitation learning. In International Conference on Learning Representations, 2020.
[9] Castro, Pablo Samuel, Moitra, Subhodeep, Gelada, Carles, Kumar, Saurabh, and Bellemare, Mare G. Dopamine: research framework for deep reinforcement learning. arXiv preprint arXiv.1812.06110. 2018.
[10] Codevilla, Felipe, Müller, Matthias, López, Antonio, Koltun, Vladlen, and Dosovitskiy, Alexey. End-to-end driving via conditional imitation learning. In IEEE International Conference on Robotics and Automation (ICRA). pp. 4693-4700. IEEE, 2018.
[11] Codevilla, Felipe, Santana, Eder, López, Antonio M, and Gaidon, Adrien. Exploring the limitations of behavior cloning for autonomous driving. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019.
[12] de Haan, Pim, Jayaraman, Dinesh, and Levine, Sergey. Causal confusion in imitation learning. In Advances in Neural Information Processing Systems, 2019.
[13] DeVries, Terrance and Taylor, Graham W. Improved regularization of convolutional neural networks with cutout. arXiv preprint arXiv: 04552, 2017.
[14] Dosovitskiy, Alexey, Ros, German, Codevilla, Felipe, Lopez, Antonio, and Koltun, Vladlen. Carla: An open urban driving simulator. In Conference on robot learning, pp. 1-16. PMLR, 2017.
[15] Ghiasi, Golnaz, Lin. Tsung- Yi, and Le, Quoc V. Dropblock: A regularization method for convolutional networks. In Advances in Neural Information Processing Systems, 2018.
[16] Goyal, Anirudh, Lamb, Alex, Hoffmann, Jordan, Sodhani, Shagun, Levine, Sergey, Ben- gio, Yoshua, and Schölkopf, Bernhard. Recurrent independent mechanisms, arXiv preprint arXiv:1909.10893 2019.

---

[17] Hayes, Jamie, Melis, Luca, Danezis, George, and De Cristofaro, Emiliano. Logan: Membership inference attacks against generative models. In Proceedings on Privacy Enhancing Technologies, 2017.
[18] Hester, Todd, Vecerik, Matej. Pietquin, Olivier, Lanctot, Marc, Schaul, Tom, Piot, Bilal, Horgan, Dan, Quan, John, Sendonaris, Andrew, Osband, lan, et al. Deep q-learming from demonstrations. In Proceedings of the AAAI Conference Artificial Intelligence, 2018.
[19] Higgins, Irina, Matthey, Loic, Pal, Arka, Burgess, Christopher, Glorot, Xavier, Botvinick, Matthew, Mohamed, Shakir, and Lerchner, Alexander. beta-vae: Learning basic visual con- cepts with a constrained variational framework. In International Conference on Learning Representations, 2017.
[20] Ho, Jonathan and Ermon, Stefano. Generative adversarial imitation learning. In Advances in Neural Information Processing Systems, 2016.
[21] Hussein, Ahmed, Gaber, Mohamed Medhat, Elyan, Eyad, and Jayne, Chrisina. Imitation learning: A survey of learning methods. ACM Computing Surveys (CSUR), 50(2):1-35, 2017.
[22] Kim, Jaekyeom, Kim, Minjung, Woo, Dongyeon, and Kim, Gunhee. Drop-bottleneck: Learning discrete compressed representation for noise-robust exploration. In International Conference on Learning Representations, 2021.
[23] Kober, Jens and Peters, Jan. Policy search for motor primitives in robotics. Machine learning. 84(1-2):171-203, 2011.
[24] Laskin, Michael, Lee, Kimin, Stooke, Adam, Pinto, Lerrel, Abbeel, Pieter, and Srinivas, Aravind. Reinforcement learning with augmented data. In Advances in Neural Information Processing Systems, 2020.
[25] Le, Hoang M, Yue, Yisong, Carr, Peter, and Lucey, Patrick. Coordinated multi-agent imitation learning. In International Conference on Machine Learning, 2017.
[26] Le, Thuc Duy, Hoang, Tao, Li, Jiuyong, Liu, Lin, Liu, Huawen, and Hu, Shu. A fast pc algorithm for high dimensional causal discovery with multi-core pcs. JEEE/ACM Transactions on Computational Biology and Bioinformatics, 16(5):1483-1495 2016.
[27] Liu, Rosanne, Lehman, Joel, Molino, Piero, Such, Felipe Petroski, Frank, Eric, Sergeev, Alex, and Yosinski, Jason. intriguing failing of convolutional neural networks and the coordcony solution. In Advances in Neural Information Processing Systems, 2018.
[28] Locatello, Francesco, Bauer, Stefan, Lucic, Mario, Ractsch, Gunnar, Gelly, Sylvain, Schölkopf, Bernhard, and Bachem, Olivier. Challenging common assumptions in the unsupervised learning of disentangled representations. In International Conference on Machine Learning, 2019.
[29] Lopez-Paz, David, Nishihara, Robert, Chintala, Soumith, Scholkopf, Bernhard, and Bottou, Léon. Discovering causal signals in images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017.
[30] Maaten, Laurens van der and Hinton, Geoffrey. Visualizing data using t-sne. Journal of Machine Learning Research, 9(Nov):2599-2605 2008
[31] Mahler, Jeffrey and Goldberg, Ken. Learning deep policies for robot bin picking by simulating robust grasping sequences. In Conference on Robot Learning, 2017.
[32] Mnih, Volodymyr, Kavukcuoglu, Koray, Silver, David, Rusu, Andrei A, Veness, Joel, Bellemare, Mare G, Graves, Alex, Riedmiller, Martin, Fidjeland, Andreas K, Ostrovski, Georg, al. Human-level control through deep reinforcement learning. Nature, 51807550);559-533, 2015.
[33] Muller, Urs, Ben, Jan, Cosatto, Eric, Flepp, Beat, and Cun, Yann L. Off-road obstacle avoidance through end-to-end learning. In Advances Neural Information Processing Systems, 2006.
[34] Ng, Andrew Y. Russell, Stuart J, et Algorithms for inverse reinforcement learning. In International Conference Machine Learning, 2000.
12

---

[35] Oord, Aaron van den, Vinyals, Oriol, and Kavukcuoglu, Koray. Neural discrete representation learning. In Advances in Neural Information Processing Systems, 2017.
[36] Parascandolo, Giambattista, Kilbertus, Niki, Rojas-Carulla, Mateo, and Schölkopf, Bernhard. Learning independent causal mechanisms. In International Conference on Machine Learning, 2018.
[37] Pathak, Deepak, Mahmoudieh, Parsa, Luo, Guanghao, Agrawal, Pulkit, Chen, Dian, Shentu, Yide, Shelhamer, Evan, Malik, Jitendra, Efros, Alexei A, and Darrell, Trevor. Zero-shot visual imitation. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition Workshops, 2018.
[38] Pearl, Judea. Causality, Cambridge University Press, 2009.
[39] Pohlen, Tobias, Piot, Bilal, Hester, Todd, Azar, Mohammad Gheshlaghi. Horgan, Dan, Budden, David, Barth-Maron, Gabriel, Van Hasselt, Hado, Quan, John, Večerík, Mel. et al. Observe and look further: Achieving consistent performance on atari. arXiv preprint arXiv: arXiv: 1199, 2018.
[40] Pomerleau, Dean. Alvinn: An autonomous land vehicle in neural network. In Advances in Neural Information Processing Systems, 1989.
[41] Reddy, Siddharth, Dragan, Anca D, and Levine, Sergey, Sqil: Imitation learning via reinforce- ment learning with sparse rewards. In International Conference Learning Representations, 2020.
[42] Ross, Stephane and Bagnell, Andrew. Reinforcement and imitation learning via interactive no-regret learning. In Advances in Neural Information Processing Systems, 2014.
[43] Ross, Stéphane, Gordon, Geoffrey, and Bagnell, Drew. A reduction imitation learning and structured prediction to no-regret online learning. In International Conference Artificial Intelligence and Statistics, 2011.
[44] Russell, Stuart. Learning agents for uncertain environments. In Annual Conference on Compu- tational Learning Theory, 1998.
[45] Schaal, Stefan et al. Learning from demonstration. In Advances in Neural Information Processing Systems, 1997.
[46] Schölkopf, Bernhard. Causality for machine learning. arXiv preprint arXiv:1911.10500 2019.
[47] Shen, Zheyan, Cui, Peng. Kuang, Kun, Li, Bo, and Chen, Peixuan. Causally regularized learning with agnostic data selection bias. In ACM international conference on Multimedia, 2018.
[48] Spirtes, Peter, Glymour, Clark N, Scheines, Richard, and Heckerman, David. Causation, prediction, and search. MIT Press, 2000.
[49] Srinivas, Aravind, Laskin, Michael, and Abbeel, Pieter. Curl:
Contrastive unsupervised representations for reinforcement learning. In International Conference on Machine Learning, 2020.
[50] Srivastava, Nitish, Hinton, Geoffrey, Krizhevsky, Alex, Sutskever, Ilya, and Salakhutdinov, Ruslan. Dropout: simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research, 15(1):192-11988 2014.
[51] Sun, Wen, Venkatraman, Arun, Gordon, Geoffrey J. Boots, Byron, and Bagnell, J Andrew. Deeply aggrevated: Differentiable imitation learning for sequential prediction. In International Conference on Machine Learning, 2017.
[52] Sutton, Richard S and Barto, Andrew G. Reinforcement learning: An introduction. MIT Press, 2018.
13

---

[53] Tai, Lei, Yun, Peng, Chen, Yuying. Liu, Congcong, Ye, Haoyang, and Liu, Ming. Visual- based autonomous driving deployment from stochastic and uncertainty-awarc perspective. In IEEF/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 2622-2628. IEEE, 2019.
[54] Wen, Chuan, Lin, Jierui, Darrell, Trevor, Jayaraman, Dinesh, and Gao, Yang. Fighting copycat agents in behavioral cloning from observation histories. In Advances in Neural Information Processing Systems, 2020.
[55] Yarats, Denis, Kostrikov, Iya, and Fergus, Rob. Image augmentation is all you need: Regu- larizing deep reinforcement learning from pixels. In International Conference Learning Representations, 2021.
[56] Ye, Gu and Alterovitz, Ron. Guided motion planning. In Robotics Research, pp. 291-307. Springer, 2017.
[57] Zagoruyko, Sergey and Komodakis, Nikos. Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer. In International Conference on Learning Representations, 2017.
[58] Zhang, Yuheng, Jia, Ruoxi, Pei, Hengzhi, Wang. Wenxiao, Li, Bo, and Song. Dawn. The secret revealer: Generative model-inversion attacks against deep neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020.
[59] Ziebart, Brian D, Maas, Andrew I., Bagnell, J Andrew, and Dey, Anind K. Maximum entropy inverse reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence. 2008.
14


Reviews:
[{"review_id": "kRL9sQWJEBq", "reviewer": "Reviewer_qCdW", "summary": "The paper considers a recently identified problem in imitation learning, that the trained policy learns relationships to nuisance variables, which do not have a casual relationship to the main goal of the action. \n\nFor instance, when learning pong, the learned policy becomes dependent on the score display and the learned policy is difficult to transfer to an environment where the score display does not exist. On the other hand, a policy learned on an environment where the score display was masked out transfers better to an environment where it is present. \n\nThe authors are proposing a technique to regularize the imitation learning process by masking out certain objects from the environment. They achieve this by training a VQ-VAE, and assuming that units of a feature map corresponding to the same objects are mapped to similar discrete codes. The authors then randomly drop the units of a feature map that share the same discrete code during training. \n\nThe authors test the proposed approach on a number of games from the Atari environment using the \"confounded input model\", where the action from the previous time step is encoded into the current input. Previous work had shown that such a confounded input, although it contains more information than the original form of the input, is in practice harder to learn. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper describes a regularization approach from the general class of approaches initiated by dropout, for the specific problem of nuisance variables in imitation learning. The approach is based on randomly dropping certain objects from the environment, with the objects being identified by the fact that they map to similar discrete codes when encoded by a VQ-VAE. \n\nThe paper shows improvement in the performance of the Atari games considered. \n\nThere is a certain degree of concern about the significance of the proposed approach. The approach does not strictly speaking identify causal relationships. It seems that the primary impact is that it prevents the policy to focus too much on one particular variable. It is also not necessarily clear why training on the confounded input changes the learned policy, and why the proposed regularization improves it (eg Figure 8). While I am aware that this technique of confusing the input was not proposed in this paper, its practical significance is unclear. Would this problem appear in a real world setting? Would the proposed approach work with real world images?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper correctly identifies that the proposed work only focuses on the nuisance features on the current state, and does not actually infer the causal relationships. \n\nThe authors also discuss some hypothetical negative impacts about self-driving cars and imitation learning, however these are hypothetical and not specific to the contributions of this paper. ", "review_text": "", "extra_scores": {"ethical_concerns": "No ethical concerns had been found with this paper.\n", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2 hrs"}, "extra_text": {}}, {"review_id": "dgtzTrQ0rrV", "reviewer": "Reviewer_8tr4", "summary": "In this paper, the authors propose a novel imitation learning method, regularized by object-aware encoding and masking during the training phase. The improvements over the performance of behavioral cloning baselines on Atari games indicate the effectiveness of the proposed framework.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- The proposed method is easy to follow. The overall writing looks well-organized to me. Both big picture and implementation details are discussed intensively. I believe the readers could catch up with the most critical observations in an easy way. \n- The illustrations are helpful for the readers to realize the main motivations and contributions of the proposed method. One possible suggestion is to highlight the masked area in figure 1. It takes me a long time to realize where the masked area is.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The proposed OREO method heavily utilizes VQ-VAE to disentangle the raw RGB input into discrete codes for each semantic object. So the accuracy of such a grouping could be crucial. Even though VQ-VAE works well on Atari games, the real-world application, such as autonomous driving simulation CARLA, typically has much more complicated scenes with higher resolution. I wonder if the proposed OREO could still help when VQ-VAE generated discrete codebook becomes fuzzier. Also, as mentioned in section 5, the application of learning from a sequence of observations could be interesting to study.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "KqH38qaJPJO", "reviewer": "Reviewer_LhiA", "summary": "This paper introduces a policy regularization method that randomly masks objects which share discrete codes inferred from VQ-VAE, to prevent learners from focusing on non-essential parts of expert actions. OREO, the suggested model, is trained without environment interaction but learns how to evenly attend to all objects in confounded Atari games. Consequently, OREO aims to solve the causal confusion problem which is an innate weakness of behavior cloning from observation.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well written, but it has limited novelty. \n1. (originality) OREO can be considered as BC with VQ-VAE, with no other contributions.\nIt has a better performance compared to CCIL, but the poor performance of CCIL seems due to insufficiency of beta-VAE in a high-dimensional environment, and the better results of OREO are likely to result from the ability of VQ-VAE that infers well discrete latent representations in a high-dimensional environment.\n2. (significance) Also, if the authors wanted to claim the suggestion of the dropping technique as a contribution, I think the paper should have included the baseline like DropOut and DropBlock within VQ-VAE architecture, rather than pure convolutional encoder feature maps. To elaborate, the authors suggest the dropping of feature map units that share the same discrete codes, but I think dropping feature map units of random discrete codes or random contiguous discrete codes like in DropOut and DropBlock would be better baselines for the dropping technique.\n\n#### Minor Questions\n1. (clarity/ quality) Is there any reason you choose DRIL and GAIL as IRL baselines? GAIL has already been mentioned in CCIL as “GAIL fails to converge to above-chance performance on any of the (confounded) Atari environments.” I wonder what insights that the evaluation result of DRIL and GAIL each can give to the readers.\n2. (clarity) Please explain more about Figure 3. For people who don’t have prior knowledge of Atari games, it is hard to know the intention and meaning of even column images.  \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, the authors pointed out agreeable limitations and potential negative societal impacts. \nHowever, it would be interesting to give the difference with CCIL in terms of constructing a causal model(graph), because CCIL proposed the way to deduce the cause-effect relations as a causal graph while OREO did not. I think the dropping method does not lessen the need to know cause-effect relation in causal confusion problems, rather the method can provide a better way to utilize the inferred cause-effect relation.  \n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "10 hours"}, "extra_text": {}}, {"review_id": "BNlR-jzq3gX", "reviewer": "Reviewer_jmzw", "summary": "In this paper, the causal confusion problem in imitation learning (IL) is considered. To tackle the confusion that resulted from not knowing the causal relationship, a two-stage approach is proposed. At the first stage, a VQ-VAE model is trained to generate semantic codes for different objects in the state space. At the second stage, a dropout-based approach is adopted to decouple the semantic confusion among the objects. The contributions of the paper are:\n>- Propose a practical approach to solve the causal confusion problem for IL;\n>- Conduct extensive experiments under Atari to verify the performance of the proposed approach.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originally: Good\n\nAddressing the causal confusion problem for IL receives significant attention in recent years. The paper proposes a novel and practical approach to deal with this problem. \n\nQuality: Overall good\n\nThe paper proposes an interesting approach, which is quite practical in my view. The experiments show significant improvement over existing baselines. While I still think the following limitations exist:\n>- The proposed approach seems to work only in vision-based tasks. \n>- The effectiveness of the proposed approach seems to rely heavily on the performance of the VQ-VAE model. I am curious to see how the proposed approach is robust to the quality of the semantic codes learned by VQ-VAE.\n>- The experiments include only Atari tasks.\n>- Only behavior cloning is discussed. While I think applying the approach to other IL methods is also possible. I am curious to see whether this is true.\n\nClarity: The paper is overall well-written. Both the technical and the experimental parts are clear to be understood.\n\nSignificance: To my understanding, the paper provides an interesting approach to address the causal confusion problem in IL, which may be of interest to researchers in this area. I suggest releasing the code to make the results easier to reproduce by other researchers.\n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "They are adequately addressed.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This submission targets the problem of "causal confusion" in imitation learning, where spurious correlates in expert data hurt environmental performance. The approach proposed is simple: train a vector-quantized VAE representation, and apply dropout over it during imitation learning. The motivation is: VQ-VAEs can learn to disentangle objects in images, and this can help coherently omit semantic information. 

The key strengths of this submission are the simplicity of the proposed approach, the breadth of experimental evaluation, and the strong empirical results. In particular, the evaluation uses 27 Atari environments, aside from new rebuttal experiments in simple CARLA settings. These may become valuable contributions to studying this problem. 

This submission is very much borderline, and I only lightly lean towards acceptance. My main reasons for hesitation are:
- Like reviewers have also commented, the paper relies on learning not just good quality discrete codes, but disentangled *objects* through VQ-VAEs (or at least, it is motivated in this way). While this works well enough on Atari settings such as in the submission, I'm not convinced that it will in more complex settings. 
- The CARLA settings tested in new rebuttal experiments ("straight", and "one turn") are simple, and not the standard town driving environments commonly used in the imitation learning literature. 
- Dropout has previously been proposed as an approach to learn robust imitation policies that can tackle these problems: see [1]. This means that the proposed approach is effectively a combination of VQ-VAE with an existing technique 
- Aside from new somewhat preliminary rebuttal experiments on 2 Atari environments in the "stacked inputs" setting, all experiments are performed in the more contrived setting in which past actions are overlaid on the input images. Other works in this literature have now moved away from this assumption, such as [1, 2].

[1] "Chauffeurnet: Learning to drive by imitating the best and synthesizing the worst."
[2] "Fighting Copycat Agents in Behavioral Cloning from Observation Histories"
Decision: Accept (Poster)