Fight fire with fire: countering bad shortcuts in imitation learning with good shortcuts
Authors: Chuan Wen
Jianing Qian
Jierui Lin
Dinesh Jayaraman
Yang Gao
Venue: ICLR 2022 Submitted (2022)

Abstract: When operating in partially observed settings, it is important for a control policy to fuse information from a history of observations.  However, a naive implementation of this approach has been observed repeatedly to fail for imitation-learned policies, often in surprising ways, and to the point of sometimes performing worse than when using instantaneous observations alone. We observe that behavioral cloning policies acting on single observations and observation histories each have their strengths and drawbacks, and combining them optimally could achieve the best of both worlds. Motivated by this, we propose a simple model combination approach inspired by human decision making:  we first compute a coarse action based on the instantaneous observation, and then refine it into a final action using historical information. Our experiments show that this outperforms all baselines on CARLA autonomous driving from images and various MuJoCo continuous control tasks.


Under review as a conference paper at ICLR 2022
FIGHT FIRE WITH FIRE: COUNTERING BAD SHORTCUTS IN IMITATION LEARNING WITH GOOD SHORTCUTS
Anonymous authors Paper under double-blind review
ABSTRACT
When operating in partially observed settings, it is important for control policy to fuse information from history of observations. However, a naive implementation of this approach has been observed repeatedly to fail for initation-leamed policies, often in surprising ways, and to the point of sometimes performing worse than when using instantancous observations alone. We observe that behavioral cloning policies acting on single observations and observation histories each have their strengths and drawbacks, and combining them optimally could achieve the best of both worlds. Motivated by this, we propose a simple model combination approach inspired by human decision making: the instantaneous observation, and then refine it into final action using historical information. Our experiments show that this outperforms all baselines on CARLA autonomous driving from images and various MuJoCo continuous control tasks.
first compute coarse action based
Input Image
BC-SO
BC-OH
Our model
Figure 1: Attention maps of baseline imitation methods and our method, depicted on the CARLA driving task. Behavioral cloning from single observation (BC-SO) attends to the appropriate visual cues in the scene (the traffic light and pedestrian), but performs suboptimally due to lack of informa- tion. Behavioral cloning from observation history (BC-OH) has access to all required information, but manifests the "copycat problem"; ; it relies excessively extrapolating past actions, and fails to attend to important visual cues. Our "coarse-to-fie imitator combines the advantages of each method, and outperforms both.
INTRODUCTION
Learning complex decision behaviors such driving is challenging task for machine learning models. If given expert demonstrations, imitation learning reduces the complex policy learning to supervised learning by mimicking the expert's behavior, which is more sample efficient and has more human-like behavior than the reinforcement learning methods. Many existing imitation learning methods employ the behavioral cloning (BC) strategy to directly regress the mapping from the observations to the expert actions. Comparing with other families imitation learning such as GAIL (Ho & Ermon 2016 Zhang al. 2020 and DAGGER (Ross al. 2011 Spencer al. 2021). behavioral cloning does not have access to online interaction or queryable experts, resulting in severe distributional shift due the compounding errors during online testing.
We are interested in a specific type of distributional shift the POMDP setting copycat prob- lem (Wen al. 2020). Because of the partial observation, it is common take the observation

---

Under review as a conference paper at ICLR 2022
history along with the current observation the input of the imitation policy to compensate for the missing information. However, many previous works find that behavioral cloning from observation history performs worse than expected (Muller et al. 2006 Wang et al., 2019: Bansal et al.) 2019 de Haan et al. 2019). Wenn et al. 2020; 2021) notice that the imitator tends to cheat by simply copying the previous actions rather than taking actions according to the observation and they call it copycat problem. This phenomenon has been reported in several papers in different forms, such as past motion cheating (Bansal al. 2019), inertia problem (Codevilla al. 2019), causal confusion (de Haan et al. 2019) and etc. All of them are essentially the same: the deep neural networks tend to take the incorrect shortcut from the previous action implied in the observation history. which is much simpler than mining real decision logic from massive amounts of data.
This problem further illustrated in Figure T In this scenario, behavioral cloning from single observation (BC-SO) correctly attends to the traffic light and pedestrian. However, without access to the current speed, it cannot take the accurate brake value to slow down in time. In contrast, behavioral cloning from observation history (BC-OH) pays much less attention to the pedestrian and even ignores the traffic light. Instead, it extrapolates the previous actions, which results in the running red light and collision. In summary, BC-SO has a hard time predicting accurate enough actions but is copycat-free, while with access to more information, BC-OH is able to output more fine-grained actions but suffers from the copycat problem.
00
coarse model
a
stop gradient
OF
Qz-1
policy model
ag
0--2
Figure 2: Our approach.
Since both BC-SO and BC-OH have their strengths and drawbacks, it is natural to think about how to combine them in an optimal way that achieves the best of both worlds. Motivated by human decision-making we propose a coarse-to-fine imitation learning approach, as shown in Figure 2
We demonstrate our method in four environments, ranging from autonomous driving robotics control. Our method outperforms other methods (Bansal et al. [219 Wen et al. [2020) 2021 Ross (et al. alt 201T). Furthermore, we conduct extensive ablation studies to verify our hypothesis for why our approach works.
RELATED WORK
Imitation Learning. Imitation learning is a powerful technique to learn complex decision and control behaviors from observed expert demonstrations. We mainly focus on the widely used behavioral cloning paradigm which directly learns a mapping from observations to expert actions. Behavioral cloning usually suffers from distributional shift because the small error between imitator and expert trajectories will compound along the time and finally leads to encountering unfamiliar states not covered by demonstrations. Several existing works have been proposed resolve this issue, but most of them either require environmental interactions (Ho & Ermon 2016)) Brantley et af., 2020 Dadashi et al. 22000) or queriable online experts (Ross et al.) 2001 Laskey et al. 2017b; Sun et al. 2018 Spencer et al. 2021). Solving the distributional shift problem under the offline setting is still appealing. In this paper, we focus on resolving a specific aspect of distributional shift copycat problem (Wen et al. 2020), caused by the nature of MDP that the actions are highly correlated along the time. We will discuss the works related to copycat problem Section 3.2
Shorteut Learning. With the rapid development and widespread use of deep neural networks in academia and industry, more and more limitations of DNN have come into focus. In computer vision and NLP, etc., researchers have found that DNN usually tends to attend to some easy but incorrect clues to make the prediction, such as object texture (Geirhos et al. 2019a). image background (Beery et al. 2018) and word length (Niven & Kao| 2019: McCoy et al. 2019). Geirhos et al.) (2020) summarize this phenomenon as shortcut learning, i.e. the DNNs prefer to learn the easier solution (shortcut) rather than taking more effort to learn the intended solution. We regard the copycat problem as a specific instance of shortcut learning problem in imitation learning, and propose a simpler and more desirable shorteut to alleviate the copycat shortcut.
2

---

Under review as a conference paper at ICLR 2022
3 PRELIMINARIES
We study behavioral cloning algorithms in a partially observed Markov decision process (POMDP), In POMDP, the environment provides observation Ot, which only partially represents the actual state because of noise or sensor occlusion, and reward 7t- To deal with the missing or noisy information, is common take the previous few observations (0z-11 H, H 1, 2....) into account (Murphy 20000) Bansal et al., 2019 Wen et al. 2020; 2021), which forms the observation history o [ '",, "r The final goal of policy learning
train model Th which takes in o, and outputs the action a to
maximize the return, i.c. the sum of the rewards R = Σ
3.1 BEHAVIORAL CLONING
Behavioral cloning (BC) assumes that we are given offline demonstrations D {(ō,, ) collected by an expert and no reward values are provided. The best way to maximize the test return is to just mimic the expert and behave as similar to the expert as possible during online testing. So behavioral cloning reduces policy learning to a simple supervised learning paradigm with training dataset D. Specifically, BC methods usually minimize the following behavioral cloning loss, where (.,,)) is usually mean square error (MSE) negative log likelihood (NLL) loss:
f arg min L( (ō.))
Time
?
Observation
Control
Figure 3: An example of the copycat problem in autonomous driving scenario. The copycat policy learns to cheat by extrapolating from the previous action. Therefore, when the traffic light turns green, the copycat policy ignores the traffic light and copies the previous action, i.e. staying stationary. On the contrary, the human driver gets the "startup" concept by observing the traffic light and takes the accurate throttle according to the speed.
3.2 COPYCAT PROBLEM
In partially observable settings, it is common practice in control theory (Ang et al. 2005 Welch et al. 1995) and reinforcement learning (Mnih et al. 2015 Schulman et al. 2017 to use historical observations to learn more accurate actions. Similarly, people would expect BC with observation history o, (BC-OH) to perform better than the model trained from single observation o, alone (BC- SO). In practice. however, BC-OH could fail unexpectedly. In particular, BC-OH could have lower offline validation loss than BC-SO but performs worse during open-loop testing. As described in Figure 3 Wen al. (2020) coined the term "copycat problem" to describe this phenomenon: due to the nature of the MDP. I.c. the expert actions are temporally correlated, it is common that the imitator tends to take a shorteut and outputs decisions based on previous actions inferred from historical observations. Many recent works (Wang et al. 2219 de Haan et al.) 2019 Wen et al., 20200 Muller ctt al., 2006 Bansal et al. 2019; Codevilla et al. 2019) report the same findings and try to address this discrepancy in various ways. dd Haan al. 2019) purposes to learn policy for each possible causal graphs, then perform target interventions to search for the correct one. [Bansal et al.) (2019) introduces random dropout the observation history, Codevilla et al. (2019) introduces speed prediction regularization module and |Wen et al. (022 treats as a data imbalance problem and up-weights the training loss of changepoints.
3

---

Under review as a conference paper at ICLR 2022
4 METHODOLOGY
Previous works address this copycat problem either from a causality (de Haan et al. 2019) or data distribution (Bansal et al., 2019 Codevilla al. 2019; Wen et 2021) point-of-view. Instead we take much simpler view by analyzing the strengths and drawbacks of BC-SO and BC-OH. We propose to utilize a model combination approach to get the best of both models. Motivated by human decision making, we propose simple but effective combination approach by predicting coarse action based on the instantaneous observation (BC-SO), and then refine it into a final action using historical information (BC-OH).
O2
Hky
6
(a) BC-SO policy
(b) BC-OH policy
(c) Our policy
Figure 4: The comparison of BC-SO. BC-OH and our model. BC-SO and BC-OH learn the mapping from the current observation o and the observation history o, respectively to the expert action ae Our model combines them together to train a coarse action model g. from o, to d, and policy model xy from o to a,. We extract the coarse action Ç from %e (we simply set the action output of To as ()) and then feed it into Ty by concatenating it with the intermediate feature of m..
4.1 COMPARISON BETWEEN BC-SO AND BC-OH
As shown in Figure (aa and Figure 4bb BC-SO and BC-OH take input observations of different time horizons. In a partially observed setting, BC-SO does not have access to historical observations to infer its current action. Thus, it is often hard for BC-SO to output smooth actions. However, this also makes BC-SO free from the copycat problem. As shown in Figure [ BC-SO learns focus on the traffic light and pedestrian. Therefore, BC-SO excels at making coarse decisions during testing, like starting the vehicle in front traffic lights or avoiding the pedestrians. Its limitation is that with only the current observation as input, it is hard for BC-SO to reason about its current speed. So has a hard time driving smoothly on the road, which leads to collisions due to high speed at times.
In contrast, BC-OH takes the observation history input, the output action is smoother than BC-SO for most of the time. However, as shown in Figure BC-OH fails to focus on the traffic light pedestrians during time. The inclusion of historical information leads to the copycat problem that in these critical scenarios, BC-OH fails to focus on important visual cues but predicts actions solely by extrapolating the previous actions. We observe that, comparing with BC-SO, BC-OH can easily infer its speed from observation history and adjust throttle when the speed is too high. But as illustrated in Figure BC-OH often fails to start the car because infers the previous actions from the historical observations and then simply copies it.
Both models have strengths and drawbacks and combining them optimally could take the best of them and outperform either individual model alone. According to the model combination approaches in machine learning (Wolpert) 1992 Breiman] 1996, Freund Schapire) 1997), our solution motivated by how to combine BC-SO and BC-OH to predict the precise actions according to the observation history while avoiding the copycat problem.
4.2 COARSE-TO-FINE IMITATION LEARNING
How might we combine BC-SO and BC-OH to enable relying on historical observations but still avoid the copycat problem? Motivated by human decision making, propose coarse-to-fine imitation learning approach, as shown Figure 4(c) Previous researches have found that there exists coarse action (usually called option decision in psychology literature) in human's 's control process (Klein et al. 1995) Johnson & Raab] 2003 Keller & Ho 1988) Ward et al. (2IIT). This coarse action is representation that plays the implicit explicit psychological role of candidate for action.
4

---

Under review as a conference paper at ICLR 2022
Furthermore, it is generated instantancously from direct perception, without elaborate cognitive processing (Raab & Johnson 2007| Raab et al. 2009). Then, humans refine this coarse action into final actions based on more deliberative cognitive processes involving memory and knowledge.
Inspired by this, we design a coarsc-to-fin imitator, shown in Figure 4(c) At test time, our model operates in two stages: analogous to BC-SO, our "coarse model" Th computes coarse action & based on the instantaneous observation alone, i.e. &. = (o,).. Then, a second "policy model" Fe refines Se into a final action dz, using additional information from observation histories, i.e. a, ( Ct).
How are these trained? We jointly train the coarse model TΦ and policy model Ia end-to-end by BC losses in Eq. 2
arg min (o)) ax)
@ 0- arg min L( (ōr, a.)
In detail, we concatenate the &e with the intermediate feature of my and feed them into the following layers together. To avoid the historical information leaking into TT. isolate the two models with stop- -radddddnntt. The imitator will take advantage of the coarse action and focus on refining it, which avoids taking completely wrong actions by extrapolating from the previous action. The coarse-to-fine process is essentially mimicking the human control process. Recall the vehicle starting example in Section 3.2 The coarse model T takes in the current observation where the traffic light turns green, so it outputs coarse action & representing that the vehicle should decide to start up, i.e. the throttle value should be larger than O. And then Te gets the coarse action and takes an accurate throttle value according to the current speed. With such architecture, our imitation policy will not be affected by the copycat problem and the performance improves a lot, which will be shown in Section
4.3
WHY SHOULD COARSE-TO-FIN IMITATION WORK?
Note that in the scheme described above, the coarse action & is inferred from the instantancous observation oz, which in turn is contained inside the observation history oz. Thus, by the data processing inequality, Ç. can not add any new information to the policy model To (- G)). Without the Se input, however, the policy model Ty reduces to a plain BC-OH policy. This begs the question: why should coarse-to-fine imitation produce any better results than BC-OH? Yet, as we will show in our experiments, it does so consistently across many environments, and often by dramatic margins! We argue that this may be understood from the perspective of optimization difficulties, which lie at the heart of the copycat problem (de Haan et al. 2019 Wen et al. 2020 2021).
has previously been shown that widely used neural network training algorithms prefer simple solutions (Valle-Perez et al. 2019, Shah et al. 2020 Huh et al. 202T). While such simple solutions are often preferable, simple solutions exploiting spurious correlations in training data can fail catastrophically under distributional shift (Geirhos al. 2020; 2019b Beery et al. 2018). Copycat BC-OH solutions are an example of such a shortcut: they tend to rely solefy on information about previous actions contained in the observation history, since this "simple" solution suffices to achieve low behavioral cloning losses, due to temporal correlations in expert training data.
Previous solutions aim to counter this shortcut-learning tendency in the imitation network, such as by upweighting changepoints to preferentially penalize copycats (Wen et al. 2022), or by adversarially removing past action information in the observation history (Wen al." 2020j. We take a different track. To compete with the copycat shortcut above, we provide a second viable "BC-SO shortcut": simply copy the action produced by BC-SO. Recall that BC-SO produces meaningful, but coarse solutions that are responsive environmental observations (unaffected by past actions), but crippled by a lack of historical information. As such, directly copying the BC-SO solution suffices to produce quite low BC losses. Further, the BC-SO shorteut is arguably even simpler than the copycat shortcut which involves first recovering past action information from the observation history and then copying it.
Finally, we argue that the BC-SO shortcut is also relatively desirable compared to the copycat shortcut. First, since BC-SO does necessarily attend to important elements of the current observation, our final policy Ty cannot casily ignore this information. Second, with training, ma may learn to correctly use the additional historical information in o to improve further over the BC-SO shorteut when needed.
5

---

Under review as a conference paper at ICLR 2022
5 EXPERIMENTS
In our experiments, we aim to first verify that our model can resolve the copycat problem and improve over BC-OH as well as previously proposed solutions. Next we will verify the following key hypotheses about why and how our approach works: H1 (simplicity of BC-SO shortcut); To effectively counter the copycat shortcut, it is important to set up the BC-SO shortcut to be as "simple" as possible; H2 (combination correctness): It is not sufficient combine BC-SO and BC-OH in other ways. Providing the BC-SO shortcut to BC-OH is important; H3 (causality correctness): After training, the final coarse-to-fing imitator learns the correct causal relation and is able effectively switch between the BC-SO shortcut solution and utilizing historical information.
5.1 EXPERIMENT SETUP
We evaluate our method on driving simulator CARLA (Dosovitskiy al. (017). and three OpenAI Gym MuJoCo (Todorov et al. 2012 Brockman et al. 2016) robotics control environments.
CARLA. CARLA is a photorealistic urban autonomous driving simulator (Dosovitskiy et al. 2017
and is commonly adopted testhed for imitation learning (Codevilla et 2018 2019; Chen et al.
2020, Wen et al.) 2021 Prakash et al.) (02T). We use the CARLATOR dataset (Codevilla al. 2019) to train all methods, containing 100 hours of driving demonstrations. We evaluate all methods in the hardest CARLA benchmark, Nocrash-Dense, which has the most number of pedestrians and heaviest traffic. As discussed in Wen al. (021). to construct pure POMDP,
only use the sequential frames as input without the current speed. We train all the methods three times from different random initializations account for the high variance (Codevilla et al., 2019). We report the mean and standard deviation of four metries: %success, %progress, #collision and #timeout. The success the number of episodes that are fully completed out of 100 pre-designed evaluation routes. The %progress refers to the fraction of the distance traveled by the agent towards a goal location. The #collision counts the times that the agent crashes into other vehicles, pedestrians, and obstacles. And the #timeout counts the times that the agent fails to reach the destination despite no collision within the specified time, usually caused by unsuccessful starts, wrong routes or traffic jams.
We train all approaches with the CILRS (Codevilla et al. 2019) backbone. We set the length of observation history to H = and stack the sequential frames along the channel dimension as the model input like what Bansal et al. (2019); Wen et al. (2221) do. For our method, the whole model consists of two modules, a coarse network Te with on as input and one policy network To with o, as input. And we use late-fusion by concatenating the output of Tg. i.c the decision Ço, with the feature of the penultimate fully-connection layer of Ty through stop-gradient layer. We train both modules end-to-end. See Supp for the architecture and training details.
Mujoco. We evaluate our method in three standard OpenAl Gym MuJoCo continuous control environments: Hopper, Ant and HalfCheetah. We generate expert data from PPO (Schulman ct al. 2017) policy (10k samples for Ant and Walker2I,, and 20k for Hopper). To simulate partially observations, We add Gaussian noise N(O, x2) with 0.2 to joint velocities. We stack H 2 frames to form the observation history. We train all methods three times with different random initializations and report the final evaluation rewards with their mean and standard deviation. See Supp for network architecture and training details.
We extensively compare our method with baseline methods: BC-SO, BC-OH, Bighting-Copyeee Agents (FCA) (Wen et al. 2020), KeyFrame (Wen et al. 2021). History-Dropout (Bansal et al. 2019), Average-Ensemll and DAGGER (Ross et al. 2011). See Supp for the description and implementation of each method.
5.2 IMITATION RESULTS
CARLA. The evaluation results on Nocrash-Dent benchmark are shown in Table We can see that according to Øsuccess, %progress and #collision, BC-OH performs significantly better than BC-SO because BC-SO does not have access to the additional information provided by the observation history. The most common failure case of BC-SO is that the car is accelerating when going straight because it cannot infer the current speed from a single observation, contributing to an extremely high #collision. This does not happen BC-OH because it can easily adjust its speed according to the

---

Under review as a conference paper at ICLR 2022
Table 1: CARLA Nocrash-Dense results.
METHOD
&SUCCESS (1)
&PrOGRESS (1)
#COLLISION ())
#TIMEOUT (L)
BC-S0
13.1 = 1.8
40.8 3.0
76.4 #
11.1 # 2.9
BC-OH
34.1 = 7.5
62.2 + 9.4
30.2 + 7.9
36.1 # 14.5
OURS
49.3 = 3.6
75.0 + 1.6
39.4 5.0
12.0 + 1.9
FCA
31,2 = 5.2
66.5 + 4.1
34.4 8.1
35.3
9.6 9.6
KEYFRAME
41.9 6.2
7..2
33.9 6.6
24.8 + 7.9
HISTORY-DROPOL
35.6 = 3.5
67.0
45.3 3.5
20.3 = 5.6
AVERAGE-ENEEMBLE
41.7 =
71.5 3.2
43.7
1
15.0 + 0.8
DAGGER 150K
42.7 A 5.7
71.3 a 1.9
35.0
4 3.6
23.0 = 7.1
Table MuJoCo rewards.
METHOD
HOPPER
ANT
HALFCHEETAH
BC-SO
589 94
4198 433
489 # 77
BC-OH
628 99
2922 & 1266
639 4 121
OURS
1124 135
4798 304
1448 74
FCA
831 108
3727 926
1148 81
KEYFRAME
696 28
2930 1321
1062 127
HISTORY-DROPOUT
539
4069 517
1215 70
AGE: ANSEMBBBB
4659 396
729 + 50
DAGGER 8K PPO EXPERT
2383 294
4097+ 418
1842 10
3445
5566
1941
dynamic information. However, the #timeout of BC-OH is the highest among all methods, which is mainly caused by unsuccessful start: When the car is stationary, either stops front of traffic light or pedestrians, BC-OH often remains stationary until timeout, which is typical copycat phenomenon. In contrast, the #timeout BC-SO is pretty low, most of which are caused by traffic jams and wrong routes, indicating that BC-SO free from the copycat problem.
Our method outperforms all the observation-he cenettiion-iittry---eeeeeeeeee history- based baselines on Ssuccess and %progress, even including DAGGER which benefits from the 150K extra online expert data. Especially, our method significantly reduces the #timeout to 12.0, very close to BC-SO. And very few of these timeout cases are caused by copycat-related issues.
MuJoCo. The evaluation results on Hopper, Ant, and HalfCheetah are shown in Table 2 We can see that although BC-OH has historical observations as input, only performs slightly better than BC-SO on Hopper and HalfCheetah, while performing worse than BC-SO on Ant, due to the copycat problem. Our method easily beats all the naive behavioral cloning baselines. We also compare our method with previous methods that intend to address the copycat problem. Our method outperforms all three of them (FCA, KeyFrame and History-Dropout) in all environments. On both Hopper and HalfCheetah, none of the aoove-mentioned offline methods beat DAGGER, which intends to be oracle method with access to online samples.
5.3 ABLATION STUDY
To evaluate the effect of each module, we conduct ablation study experiments on CARLA about 1) different fusion stages: late-fusion (ours), middle-fusion early-fusion (see Supp for the archi- tecture details); 2) coarse action selections: use the final action of т the intermediate features, e.g. Restnet Layer2's feature or Resnet output, as G; 3) ways to combine BC-SO and BC-OH: use BC-SO as T and BC-OH as (ours), BC-OH as T& and BC-OH ne, BC-OH ng and BC-SO as næ; 4) stop-gradient: with or without stop- goopggradeen;; 5) end-to-end vs. two-stage training: train the no and тy end-to-end two-stage: first train Ty and use its checkpoint when training тT. The results are shown in Table 3 and we provide detailed analysis for each ablation respectively in the following paragraphs.
7

---

Under review as a conference paper at ICLR 2022
Table 3: CARLA ablation study results.
METHOD
&SUCCESS (1)
&PROGRESS (1)
#COLLISION (J)
#TIMEOUT ())
OURS
49.3 3.6
75.0 1.6
39.4 5.0
12.0 1.9
EARLY-FUSION
51.4 4.0
75.9 = 2.1
37.4 5.1
12.7 2.6
MIDDLE-FUSION
49.6 1.7
74.8 1.9
37.3 4.0
13.3 3.0
RESNET LAYER2 AS RESNET OUTPUT Çe
39.8 4.0
69.4 1.5
32.6 1.5
28.1 4.4
44.8 5.2
71.9 4.7
40.0 5.7
15.9 3.4
BC-OH AS Wi. BC-OH
37.1 3.4
66.9 2.8
34.1 3.4
28.9 6.0
BC-OH AS We. BC-SO AS me
31.7 = 5.7
62.2
28.3 9.0
40.0 = 14.7
W/O GRADEET W/O END-TO-END
44.4 5.5
71.4 3.0
39.2 4.7
17.1 2.1
47.3 a 3.8
71.2 a 1.2
43.4 4 5.5
We can see that given the coarse action Çr. it makes difference where to inject it into (Ssuccess ranges from 49.3 to 51.4 and other metrics are similar too), indicating that & provides a simpler shortcut than inferring and copying the previous action from the observation history. Wherever inject Çe, the neural network prefers to adopt it directly as the decision rather than expending greater effort take the copyeat shorteut.
we use the intermediate features (Resnet Layer2 and Resnet output) of the decision model as opposed the output action of TT. then %success drops from 49.3 to 39.8 and 44.8 respectively. Moreover, comparing the %ssccess and #timeout, we find that the shallower the feature, the lower Jsuccess and the higher #timeout, indicating that using shallower features coarse action is more likely to suffer from the copycat problem. Intuitively, it is harder for тτ to extract the coarse action from shallower features than from later ones and thus कe has less incentive take the coarse action shortcut compared with the original copycat shortcut. Combining with previous observation in different fusion stages ablation, this experiment clarifies that "simplicity" HI (simplicity of BC- SO shorteut) focuses more what is the input rather than where to input to the policy ma and indeed, the simpler the better.
If we use different combinations of BC-SO and BC-OH, such as BC-OH as Te. BC-SO as BC- OH as both n and my. the imitator performs similarly BC-OH (Ssuccess ranges from 37), indicating that introducing historical information into the coarse model will lead to copycat problem, which verifies H2 (combination correctness), and also verifies that our method's performance not due to higher model capacity.
Moreover, we find that the agent performs worse if we remove noo-gradient (%success drops from 49.3 to 44.4), which is mainly due to timeout (increasing from 12.0 to 17.1). The increasing #timeout shows that our method without stop-gradient still suffers from the copycat problem due to historical information leakage during propppropagttion. Such variant without stop-gradient also demonstrates H2 (combination correctness).
And training Ti and ne stage-by-stage, the performance also deteriorates (Esuccess drops from 49.3 to 47.3). We can see that its #timeout is even fewer than ours but it gets significantly higher #collision (#collision increases from 39.4 to 43.4), indicating that the agent trained stage-by-stage behaves more like BC-SO, i.e. suffering from less copycat but failing to brake in time. We hypothesize that if we use a pretrained coarse model To at the beginning of Ry training, :Tg will prefer to just copy the & as its output rather than learn from scratch to take accurate actions according to both the coarse action and observation history, which can also viewed as overly relying on the "simpler" solution, i.c. shorteut learning and sheds light the importance of H2 (combination correctness).
5.4 ANALYSIS THROUGH CAUSAL INTERVENTION
Because there two inputs of policy network Te. we utilize the intervention technique in causal inference literature (Rubin) 1974 1978 Pearl 1995) to study the causal effect of the coarse action and observation history, which
argument.
Intervention on the coarse action G. To study the causal effect of the coarse action our model, conduct causal intervention experiments &. by setting its value manually and keeping all other
8

---

Under review as a conference paper at ICLR 2022
starting (g(()((,) o. (d 0. 75) = 73
avoiding pedestrian (d, 0.00) -0.666 68
traffic light
Me (do(() 0. 0.00)
-0.46
(g(do((e)) 0. 0.75) = 0.41
My (do(()) 0.75) 0.50
Figure 5: Examples of scenarios with high causal effects. The To (d((())) 0.00/0.75) means that we manually set the value of Ç. to 0.00 and 0.75 respectively, fix the ot and then get the output of the TT. The larger the difference between Ta (do(G)) = 0.00) and Na (do())) 0.75), the stronger the causal effect of G..
factors the same. For easier interpretability, we only intervene on the acceleration dimension, i.e. setting throttle to and 0.75 (the highest value expert demonstrations), which can be denoted by do- calculus do(C;) 0/0.75. 75. The causal effect is defined by C' = (g (do(G) = 0) -y (do(G)) 0.75)), where we omit another input variable o, for simplicity. Through studying the causal effect of S along the trajectories, we find that Çe tends to have very high causal effect at some critical moments such as the examples in Figure 5 which verifies H3 (causality correctness). Especially, the causal effect of Ç. is high when the car is starting, but it decreases after the car is started because this time is necessary to refine its actions according to observation history, which is what we expected.
Intervention on the observation history ã.. Similarly, to study the causal effect of the observation history ã, we intervene it by repeating the current frame H times, i.e. do(o)) ) [ot, (z, which creates counterfactual stationary cases, i.c. the previous action 0 and we denote it as x xd)(()) 0). Recall the example in Section 3.2] to investigate what factors the agents use to determine whether to move forward or stop, we count the percentage of model outputs that change from accelerate to stop after we intervene on the input sequence, i.e.
N (spped > 0, (ā.) > 0, (do( = 0)
N(speed 0, (
We count this metric for BC-OH and our model on the same dataset. There are 66.43% samples changing from accelerate stop in BC- OH such as the example in Figure 6 even though there is no signal other
stop in the scene, e.g. vehicles, pedestrians, red lights obstacles. This illustrates that surprisingly in more than half of the cases, BC-OH is making decisions only according the previous action and ignores the current scene, which is causally incorrect. In the meanwhile, there is only 27.89% our model, indicating that our model learns correct causal relation and significantly alleviates copycat issue, which also verifies H3 (causality correctness).
BCOHI: 0N 0.75
do(o,): 0.05 do 0.75
Ours: 0y 0.7
Figure 6: An example of inter- vention on o,
CONCLUSION
In this paper, we propose simple but effective model combination approach resolve the copycat problem. Our method takes the best of BC-SO and BC-OH to predict the precise actions according to observation history while avoiding the copycat problem. We verify our method on autonomous driving and robotics control environments. Extensive carefully designed ablation and analysis experiments verify that coarse-to- to-fine imitation works by providing an alternate more desirable shortcut to the imitator during training, which supplants the problematic copycat shortcut. Our method outperforms the existing methods and significantly alleviates the copycat problem.
REPRODUCIBILITY STATEMENT
All our experiments can be easily reproduced according to the implementation and training details discussed in Section 5.1 and Section 8.2 The data collection process of CARLA is described in Section 8.2 and Mujoco's s is in Section 8.3. Especially, the CARLA100 is public dataset released by Codevilla et al. 2019). We will release our code and dataset if our paper is accepted.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Kiam Heong Ang, Gregory Chong, and Yun Li. Pid control system analysis, design, and technology. IEEE transactions on control systems technology, 13(4)5559-576, 2005.
Mayank Bansal, Alex Krizhevsky, and Abhijit Ogale. ChauffeurNet: Learning to Drive by Imitating the Best and Synthesizing the Worst. Robotics: Science & Systems (RSS), art. arXiv:1812.0300 2019.
Sara Beery, Grant Van Horn, and Pietro Perona. Recognition in terra incognita. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 456-473, 2018.
Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski, Bernhard Firner, Beat Flepp, Prasoon Goyal, Lawrence D. Jackel, Mathew Monfort, Urs Muller, Jiakai Zhang. Xin Zhang, Jake Zhao, and Karol Zieba. End to end learning for self-driving cars. CoRR, abs/1604.07316. 2016.
Kiante Brantley, Wen Sun, and Mikael Henaff. Disagrerment-Reulariiee Imitation Learning. International Conference in Learning Representations, pp. 1-19, 2020.
Leo Breiman. Bagging predictors. Machine learning, (A(2):123-140) 1996.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym, 2016.
Dian Chen, Brady Zhou, Vladlen Koltun, and Philipp Krähenbühl. Learning by cheating. In Conference on Robot Learning, pp. 66-75. PMLR, 2020.
Felipe Codevilla, Matthias Miiller, Antonio López. Vladlen Koltun, and Alexey Dosovitskiy. End-to- end driving via conditional imitation learning. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pp. 1-9. IEEE, 2018.
Felipe Codevilla, Eder Santana, Antonio López, and Adrien Gaidon. Exploring the limitations of behavior cloning for autonomous driving. In Proceedings of the IEEE International Conference on Computer Vision. pp. 9329 9329-9338, 2019.
Robert Dadashi, Leonard Hussenot, Matthiu Geist, and Olivier Pietquin. Primal wasserstein imitation learning. In International Conference Learning Representations, 2020.
Pim de Haan, Dinesh Jayaraman, and Sergey Levine. Causal confusion in imitation learning. Advances in Neural Information Processing Systems, 32: 1698-11709, 2019.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 JEEE conference on computer vision and pattern recognition, pp.
Ieee, 2009.
Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. CARLA: An open urban driving simulator. In Proceedings of the 1st Annual Conference on Robot Learning, pp. 1-16, 2017.
Yoav Freund and Robert E Schapire. A application to boosting. Journal of computer and system sciences, 55(1):119-1) 1997.
generalization of on-line learning and an
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A. Wichmann, and Wieland Brendel. Imagenet- trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness. In /CLR. OpenReview.net, 2019a.
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A. Wichmann, and Wieland Brendel. Imagenet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness. In International Conference on Learning Representations, 2019b.
Robert Geirhos, Bethge, and Felix Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(1 1):665-673, 2020.
Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias
10

---

Under review as a conference paper at ICLR 2022
Alessandro Giusti, Jérôme Guzzi, Dan C Ciresan, Fang-Lin He, Juan P Rodriguez, Flavio Fontana, Matthias Faessler, Christian Forster, Jürgen Schmidhuber, Gianni Di Caro, et al. A machine learning approach to visual perception of forest trails for mobile robots. IEEE Robotics and Automation Letters, 1(2):661-677 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. Advances neural information processing systems, 29:4565-4555 2016.
Minyoung Huh, Hossein Mobahi. Richard Zhang, Pulkit Agrawal, and Phillip Isola. The low-rank simplicity bias in deep networks. arXiv, 2021.
Joseph G Johnson and Markus Raab. Take the first: Option-generation and resulting choices. Organizational behavior and human decision processes, 91(2):215-229, 2003.
L Robin Keller and Joanna L Ho. Decision problem structuring: Generating options. IEEE Transac- tions on Systems, Man, and Cybernetics, 18(5):115-728, 1988.
Gary Klein, Steve Wolf, Laura Militello, and Caroline Zsambok. Characteristics of skilled option generation in chess. Organizational behavior and human decision processes, 62(1):63-69, 1995.
Michael Laskey, Anca Dragan, Jonathan Lee, Ken Goldberg, and Roy Fox. Dart: Optimizing noise injection in imitation learning. In Conference on Robot Learning (CoRL), volume 2, pp. 12, 70.
Michael Laskey, Jonathan Lee, Roy Fox, Anca Dragan, and Ken Goldberg. Dart: Noise injection for robust imitation learning. In Conference on Robot Learning, pp. 143-156, 2017b.
Tom McCoy, Ellie Pavlick, and Tal Linzen. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, 2019.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 518(7540):529-5 533, 2015.
Urs Muller, Jan Ben, Eric Cosatto, Beat Flepp, and Yann Cun. Off-road obstacle avoidance through end-to-end learning. In Advances in neural information processing systems, pp. 739-746. Citeseer, 2006.
Kevin P Murphy, A survey of pomdp solution techniques. environment, 2:X3, 2000.
Timothy Niven and Hung-Yu Kao. Probing neural network comprehension of natural language arguments. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4658-4664. Association for Computational Linguistics,
IiaattiiinnnLLnuiniiiiiiii 2019.
Judea Pearl. Causal diagrams for empirical research. Biometrika, 82(4):669-688, 1995.
Aditya Prakash, Kashyap Chitta, and Andreas Geiger. Multi-modal fusion transformer for end-to-end autonomous driving. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7077-7087, June 2021.
Markus Raab and Joseph G Johnson. Expertise-based differences in search and option- strategies. Journal of Experimental Psychology: Applied, 13(3):158, 2007.
Markus Raab, Rita F de Oliveira, and Thomas Heinen. How do people perceive and generate options? Progress in brain research, 174:49-59, 2009.
Stéphane Ross, Geoffrey J. Gordon, and J. Andrew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. Journal of Machine Learning Research, 15: 627-635, 2011. ISSN 15324435.
11

---

Under review as a conference paper at ICLR 2022
Donald B Rubin. Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of educational Psychology, 66(5):688, 1974.
Donald B Rubin. Bayesian inference for causal effects: The role of randomization. The Annals of statistics, pp. 34-58, 1978.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint vrXiv:/70700634) 2017.
Harshay Shah, Kaustav Tamuly, Aditi Raghunathan, Prateek Jain, and Praneeth Netrapalli. The pitfalls of simplicity bias neural networks. In Advances in Neural Information Processing Systems, volume 33, pp. 9583-9585. Curran Associates, Inc., 2020.
Jonathan Spencer, Sanjiban Choudhury, Arun Venkatraman, Brian Ziebart, and J Andrew Bag- nell. Feedback in imitation learning: The three regimes of covariate shift. arXiv preprint 2tti::200200287... 2021.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1):1229-1958 2014.
Wen Sun, J. Andrew Bagnell, and Byron Boots. Truncated horizon policy search: Deep combination of reinforcement and imitation. In International Conference Learning Representations, 2018.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033. IEEE, 2012.
Guillermo Valle-Perez, Chico Q. Camargo, and Ard A. Louis. Deep learning generalizes because the parameter-function map is biased towards simple functions. In International Conference on Learn- ing Representations, 2019. URL /tttps::/ net/ forum www.oormmiddrrrggoaaaaoa
Dequan Wang, Coline Devin, Qi-Zhi Cai, Philipp Krähenbüühl, and Trevor Darrell. Monocular plan view networks for autonomous driving. In IROS, 2019.
Paul Ward, Joel Suss, David W Eccles, A Mark Williams, and Kevin R Harris. Skill-based differences in option generation in a complex task: A verbal protocol analysis. Cognitive processing, 12(3): 289-300, 2011.
Greg Welch, Gary Bishop, al. An introduction to the kalman filter. 1995.
Chuan Wen, Jierui Lin, Trevor Darrell, Dinesh Jayaraman, and Yang Gao. Fighting copycat agents in behavioral cloning from observation histories. In Advances in Neural Information Processing Systems, volume 33, 2020.
Chuan Wen, Jierui Lin, Jianing Qian, Yang Gao, and Dinesh Jayaraman. Keyframe-focure focused visual imitation learning. In Proceedings of the 38th International Conference on Machine Learning. Proceedings of Machine Learning Research. PMLR, 18-24 Jul 2021.
David H Wolpert. Stacked generalization. Neural networks, 5(2):241-25)) 1992.
Xin Zhang, Yanhua Li, Ziming Zhang, and Zhi-Li Zhang. f-gail: Learning f-divergence for generative adversarial imitation learning. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 12805-12815 Curran Associates, Inc., 2020.
12

---

Under review as a conference paper at ICLR 2022
8 APPENDIX
8.1 BASELINES
BC-SO&BC-OH. As introduced in Section 3.2 BC-SO and BC-OH are naive behavioral cloning methods from single observation and observation history respectively.
(FCA). Wen et al. (2020) proposed to remove the unique information about the previous actions dt-1a-H from the feature extracted from the observation history to prevent the agent to copy the 1t-1ttH, based on adversarial learning.
Ceantiig-Cooycat-Appe
KeyFrame, Wen al. (022)) analyzed the copycat problem in terms of the imbalance data distribu- tion and proposed a re- weightng method to up-weight the demonstration keyframes corresponding to expert action changepoints.
History-Dropout. To address the copycat issue, Bansal et al. (2019) introduced a dropout on the observation history to randomly erase the channels of historical frames. We implement it by applying a Dropout layer (Srivastava et al. 2014) on the historical observations.
Everage-Ensemtle is commonly used model combination approach. We implement it by averaging the outputs of BC-SO and BC-OH at test time.
Everage-Ensemile
DAGGER. DAGGER (Ross al. 2011) is widely used technique to address the distributional shift issue in behavioral cloning and is thought of as the oracle of imitation learning through online query.
8.2 ADDITIONAL DETAILS ON CARLA EXPERIMENTS
In Section we briefly introduce the experiment setup of CARLA. More details are introduced below.
The CARLA100 dataset (Codevilla et al. 2019) is collected by a PID controller. During collecting. 10% expert actions perturbed by noise (Laseyy et al. 2017a). We use three cameras: a forward-facing one and two lateral cameras facing 30 degrees away towards left or right (Bojarski al. 2016). Both noise injection and multiple cameras are common data augmentation techniques to alleviate distributional shift in autonomous driving.
Data Collection.
speed prediction module
H
predicted speed
perception module
"p
image input BC-SO:
Resner34
conditional module
BC-OHL 02
H
action
Rc
command
Figure 7:
The conditional imitation learning architecture we used as our backbone. The input of BC-SO is the current observation o, and the input of BC-OH the observation history o
Architectures. We use the backbone in conditional imitation learning framework CILRS (Codevilla al., 2019) as our backbone. The only difference is that our model does not have the input speed (to create pure POMDP (Wen et al.) 2021)). As shown in Figure 7 BC-SO and BC-OH use the same architecture with different inputs, o, and o. Illustrated in Figure g our method integrates them together by concatenating the output of coarse model (BC-SO) with the features of the penultimate FC layer of policy model (BC-OH). Moreover, the architectures of early fusion model and middle fusion model mentioned in Section are shown in Figure 9 In particular, early fusion means concatenate the coarse action Ç with the input images, and middle fusion means that we concatenate it with the output feature of Resnet.
13

---

Under review as a conference paper at ICLR 2022
perception module
conditional module
H
B
action
O2
Resnet34
sg
<t
action
Resnet34
Figure 8: The architecture of our method. For simplicity, we omit the speed prediction module and the input command when drawing the figures, and sg means stop-gradient.
perception module
conditional
percepticn module
conditional
Kesnetti
action Ue
Kounet
action
#g
4g
4
(a) early fusion
(b) middle fusion
Figure 9: The architectures of the early fusion model and middle fusion we study in Section 5.3
Training Details. We use the L1 loss to train all the models. We use Adam optimizer, set the initial learning rate to 2x 10 and decay the learning rate 0.1 whenever the loss value longer decreases for 5000 gradient steps. We set the minibatch size to 160 and train all the models until convergence (the learning rate equal to 10-T). Furthermore, our training process. We utilize the noise injection (Laskey et al. 2017a) and multi-camera data augmentation (Bojarski al., 2016 Giusti al.) 2015) to alleviate the distribution shift in offline imitation learning. All the models use the speed regularization (Codevilla et al., (019) to address the copycat problem (also called inertia problem in their paper) to some extent. And we use ImageNet pretrained ResNet34 (Deng et al.) 2009 He et al., (0166) as the perception module to get better initialization (Codevilla al. 2019) and the weighted control loss to balance models' attention to each action dimension. Furthermore, different from the previous works (Codevilla et al. [0188 2019) Wen al. 2021), where the positive acceleration value means applying throttle and the negative one means braking. We empirically find that using acceleration as output modality is better than predicting throttle and brake separately among all baselines (except FCA) and our method.
apply several commonly used techniques to
woo-dimensional action space 1] for steering and acceleration,
8.3 ADDITIONAL DETAILS ON MUJOCO EXPERIMENTS
Data Collection. We first train a RL expert with PPO (Schuiman et al. 2017) and use to generate expert demonstration by rolling out in the environment. Specifically, we collect 10k samples for Ant and Walker2D, and 20k for Hopper based on imitation difficulty.
Architectures. We follow simple design for network architectures, shown in Figure 10 coarse action and policy model, we use three-layer MLP network. We concatenate the output action from the coarse action model the output of policy model, and input that to another fully-connected layer for the final output action.
For both the
Training Details. We MSE loss and Adam optimizer to train all models. We use learning rate of le-4 for both HalfCheetah and Ant, and le-5 for Hopper and use linear learning rate decay. For each environment, train for 1000 epochs until convergence. We set the minibatch size 64. We
14

---

Under review as a conference paper at ICLR 2022
coarse
action
model
action
Dr
at
sg
action
Qt
policy
model
Figure 10: The architecture of our method for MuloCo experiments, where sg means stop-gradient
train each method for three times, and report the mean and standard deviation of evaluation rewards for the last three evaluation steps.
15


Reviews:
[{"review_id": "o26guNWMCx", "reviewer": "Reviewer_Dq7x", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- Good thorough ablation study to test the combination of two BC methods.\n- Good range of simulation based continuous tasks.\n\nWeakness:\n- The paper is difficult to follow and I encourage the authors to improve the writing of the paper. Right now it is not in a shape to be published at a top-notch conference (ICLR).\n- The first two lines of the abstract is not clear to me. It will be great if you could re-write the abstract to have a better understanding of the paper.\n- The idea presented in the paper seems incremental and has less novelty.\n- There is no algorithmic complexity of the proposed method in the paper. I highly encourage the authors to present their method in algorithmic form and show its complexity after combining.\n- There are a lot of redundant information presented in the paper which could be removed (example problem with BC-SO and BC-OH presented in Intro, Related Work and Preliminaries) and make space for important stuff suggested in above suggestion. \n- From section 4.2, the authors have switched the language of BC-single observation to instantaneous observation. Is there a particular reason for change in the language? If not, it would be good to be consistent  which helps the readers to follow easily. \n- There is a typo in Section 3.2: Copycat Problem, fourth last line, purposes -> proposes", "summary_of_the_paper": "The paper addresses a behavioral cloning (BC) distributional shift problem in a POMDP setting. The paper proposes to combine strengths/merits of BC with single observation and BC with historical observation. The author performs an ablation study to verify their proposed method. Also, they test the proposed method on four continuous control task ranging from autonomous driving (CARLA simulation) tasks to Mujoco based tasks.", "summary_of_the_review": "The ideas presented in the paper seems incremental and the paper is not well written to be accepted at ICLR.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "mzShqtZVCcu", "reviewer": "Reviewer_AJ5m", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper describes a simple solution to a common problem (copycat behavior) in BC. Contributions in this area are important, timely, and will be applicable to many areas. The provided evidence is mostly empirical with a few qualitative arguments about why the approach should be superior to state-of-the-art.\n\nThe approach is compared to a number of baselines which is commendable. Unfortunately, it is hard to confirm the results since the code has not been shared at the time of submission.\n\n\n- While the paper states that both models are trained jointly, I would assume that the BC-SO policy could be trained independently. Once training is complete, the BC-OH policy can then be trained with outputs from BC-SO. It might be worthwhile to state this a bit more clearly since training jointly usually implies that one model propagates through the other, which is not the case here due to the gradient stop.\n\n- What is the imitation loss used in the experiments? Does the choice of losses have an influence on the effectiveness of the presented approach?\n\n- I would imagine that the presented technique depends on the available capacity of the networks. I.e. if the capacity of BC-OH is lower, it will rely more quickly on the output of BC-OS. Could you provide an ablation?\n\n- Much of the explanation about why this approach should work better is fairly philosophical, hard to follow, and quite a few statements made are speculation or questionable.\n\n- To me, the most convincing argument is that the optimization surface becomes easier to train. In principle, I could imagine this to be similar to how resnet architectures benefit from a certain structure. The paper would improve by visualizing this benefit. Please take a look at [1] on how this could look like.\n\n- Most experiments are not statistically significant. 3 seeds are not enough. I would also recommend changing the notation from simply stating the standard deviation to showing the 95% confidence interval. Simply state the bounds of the T-test: /mu +- t * std/sqrt(n) For 3 seeds you would have to multiply each standard deviation with 4.303/sqrt(3)=2.48, for 5 seeds with 2.776/sqrt(5) and for 10 seeds with 2.262/sqrt(10). With the current factor of 2.48, many of the confidence intervals would be overlapping, and the statistical significance of the provided results is not ensured. Since the results seem to be fairly high in variance I would recommend running 10 seeds.\n\n\n[1] Li, H., Xu, Z., Taylor, G., Studer, C., & Goldstein, T. (2017). Visualizing the loss landscape of neural nets. arXiv preprint arXiv:1712.09913.", "summary_of_the_paper": "The paper proposes to combine behavioral cloning (BC) from single observations with observations from an observation history to address the copycat problem of behavioral cloning on observation histories alone. The approach first learns a coarse policy that predicts actions from a single observation like it is usually done in single observation BC. They then feed the input into another policy that also receives an observation history which then outputs the combined action.\nThe approach is evaluated on CARLA autonomous driving from images and various MuJoCo continuous control tasks.\n", "summary_of_the_review": "I believe that the paper is interesting (although the approach is extremely simple) and could provide insights into how architectures and engineered biases can be beneficial for behavioral cloning. Unfortunately, the presented results are not statistically significant. Additionally, I believe that the paper could do better in substantiating the claims about why this approach should perform better. This could either be done through loss function visualizations, mathematical arguments (bottlenecks and compression might be applicable), or other quantitative reasoning. I am happy to change my view if these improvements are provided.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "dKrq2Yj1ZHl", "reviewer": "Reviewer_ESVr", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "#### **Strengths**\n* Clear to read and their method is succinctly illustrated (Figure 4)\n\n* Motivations from human-decision making and various works in cognitive psychology\n\n* Improvements over previous methods\n\n* Ablation on design choices in their method\n\n#### **Weaknesses**\n* I would like to see more insight on why their method increases performance by such a drastic margin. Section 4.3 and Table 3 provide some hints, but I am still not completely sure where the improvements came from. If the copycat problem stems from optimization difficulties and networks converging to the simple solution, why wouldn't the refinement model simply copy the coarse action?\n\n* In the ablations, using BC-OH $\\pi_\\phi$ (coarse) and BC-SO as $\\pi_\\theta$ (refinement) leads to a significant drop in performance, performing even worse than the BC-OH baseline in Table 1. I fail to see why this would happen since $\\pi_\\theta$ could simply copy over the coarse action and would like to see a bit more elaboration from the authors.\n\n#### **Suggestions**\n\n* Elaborating on section 5.4, to see if the refinement model actually refines the prediction, an interesting experiment would be to actually quantify the difference between the coarse action and the final action in each model in the ablation.\n\n* The main results and ablations are performed in CARLA so MuJoCo results can moved to supplementary\n\n* I am familiar with the CARLA NoCrash benchmark and know the dense setting in particularly is prone to a lot of noise in evaluation (traffic jams which cause early termination or timeout). I would like to see the community migrate towards more robust experiment testbeds in CARLA (the more recent online leaderboard and provided routes).\n", "summary_of_the_paper": "This work proposes a technique for training imitation learning policies that use temporal information. They show policies trained with their method are less \"causally confused\" (copycat problem) compared to other behavior cloning methods. In their approach, they decompose a policy into two - the first policy only takes in the current time step's observation (reactive) and outputs an action, along with a intermediate feature (their best performing model just uses the action as the feature). The second policy (refinement stage) takes in the feature, along with the history of observations, and outputs the final action. They evaluate their method in an autonomous driving simulator (CARLA NoCrash benchmark) and MuJoCo environments.", "summary_of_the_review": "Overall this work addresses a well known problem (copycat problem) and builds towards training better imitation learning policies with temporal information. I enjoyed that the method is simple and effective on the NoCrash benchmark, but I find the justification and analysis a bit incomplete in its current state to be recommended for acceptance.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "bjJZQnevVHZ", "reviewer": "Reviewer_Lt5X", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "[Strength:]\nOverall, this paper starts with the \"copycat\" problem in imitation learning when using historical observations and proposed an ensemble approach to balance between a single observation-based predictor and a history of observations based predictor. As per my knowledge, this combination is novel. Experimental results also demonstrate its effectiveness.\n\n[Weakness:]\nThe authors did an analysis empirically on why the proposed approach can address the \"copycat\" problem, however, I feel it still lacks thorough studies regarding what exactly is the \"copycat\" problem either in a distributional perspective or a decision making perspective. As a result, I think it is insufficient to recognize the authors' novel contributions.\n", "summary_of_the_paper": "This paper presents a combined approach that combines two predictors with one based on a single observation (BC-SO) and the other based on a history of observations (BC-OH). The intuition behind this approach is that BC-OH has the \"copycat\" problem which basically means the predictor is trying to simply extrapolate the previous action instead of using data to do some decision makings, while BC-SO does not have such a problem but suffers from the partial observability issue. The authors also evaluate their method based on the CARLA and MuJoCo continuous control tasks. Results clearly show the effectiveness of this combined approach.", "summary_of_the_review": "The authors proposed a combined approach to address the \"copycat\" problem in imitation learning when learning with a history of observations. This approach is new, the problem is interesting but the paper lacks satisfactory investigation of the problem itself and how the proposed method can solve the problem.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers unisono do not accept the paper, because it is (a) not well-written; (b) experimentally not convincing, but addresses a nice problem.  I suggest that the authors address the issues in a subsequent paper, and resubmit to one of the main conferences.
Decision: Reject