Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper addresses a behavioral cloning (BC) distributional shift problem in a POMDP setting. The paper proposes to combine strengths/merits of BC with single observation and BC with historical observation. The author performs an ablation study to verify their proposed method. Also, they test the proposed method on four continuous control task ranging from autonomous driving (CARLA simulation) tasks to Mujoco based tasks.
main_review: Strengths:
- Good thorough ablation study to test the combination of two BC methods.
- Good range of simulation based continuous tasks.

Weakness:
- The paper is difficult to follow and I encourage the authors to improve the writing of the paper. Right now it is not in a shape to be published at a top-notch conference (ICLR).
- The first two lines of the abstract is not clear to me. It will be great if you could re-write the abstract to have a better understanding of the paper.
- The idea presented in the paper seems incremental and has less novelty.
- There is no algorithmic complexity of the proposed method in the paper. I highly encourage the authors to present their method in algorithmic form and show its complexity after combining.
- There are a lot of redundant information presented in the paper which could be removed (example problem with BC-SO and BC-OH presented in Intro, Related Work and Preliminaries) and make space for important stuff suggested in above suggestion. 
- From section 4.2, the authors have switched the language of BC-single observation to instantaneous observation. Is there a particular reason for change in the language? If not, it would be good to be consistent  which helps the readers to follow easily. 
- There is a typo in Section 3.2: Copycat Problem, fourth last line, purposes -> proposes
summary_of_the_review: The ideas presented in the paper seems incremental and the paper is not well written to be accepted at ICLR.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes to combine behavioral cloning (BC) from single observations with observations from an observation history to address the copycat problem of behavioral cloning on observation histories alone. The approach first learns a coarse policy that predicts actions from a single observation like it is usually done in single observation BC. They then feed the input into another policy that also receives an observation history which then outputs the combined action.
The approach is evaluated on CARLA autonomous driving from images and various MuJoCo continuous control tasks.

main_review: The paper describes a simple solution to a common problem (copycat behavior) in BC. Contributions in this area are important, timely, and will be applicable to many areas. The provided evidence is mostly empirical with a few qualitative arguments about why the approach should be superior to state-of-the-art.

The approach is compared to a number of baselines which is commendable. Unfortunately, it is hard to confirm the results since the code has not been shared at the time of submission.


- While the paper states that both models are trained jointly, I would assume that the BC-SO policy could be trained independently. Once training is complete, the BC-OH policy can then be trained with outputs from BC-SO. It might be worthwhile to state this a bit more clearly since training jointly usually implies that one model propagates through the other, which is not the case here due to the gradient stop.

- What is the imitation loss used in the experiments? Does the choice of losses have an influence on the effectiveness of the presented approach?

- I would imagine that the presented technique depends on the available capacity of the networks. I.e. if the capacity of BC-OH is lower, it will rely more quickly on the output of BC-OS. Could you provide an ablation?

- Much of the explanation about why this approach should work better is fairly philosophical, hard to follow, and quite a few statements made are speculation or questionable.

- To me, the most convincing argument is that the optimization surface becomes easier to train. In principle, I could imagine this to be similar to how resnet architectures benefit from a certain structure. The paper would improve by visualizing this benefit. Please take a look at [1] on how this could look like.

- Most experiments are not statistically significant. 3 seeds are not enough. I would also recommend changing the notation from simply stating the standard deviation to showing the 95% confidence interval. Simply state the bounds of the T-test: /mu +- t * std/sqrt(n) For 3 seeds you would have to multiply each standard deviation with 4.303/sqrt(3)=2.48, for 5 seeds with 2.776/sqrt(5) and for 10 seeds with 2.262/sqrt(10). With the current factor of 2.48, many of the confidence intervals would be overlapping, and the statistical significance of the provided results is not ensured. Since the results seem to be fairly high in variance I would recommend running 10 seeds.


[1] Li, H., Xu, Z., Taylor, G., Studer, C., & Goldstein, T. (2017). Visualizing the loss landscape of neural nets. arXiv preprint arXiv:1712.09913.
summary_of_the_review: I believe that the paper is interesting (although the approach is extremely simple) and could provide insights into how architectures and engineered biases can be beneficial for behavioral cloning. Unfortunately, the presented results are not statistically significant. Additionally, I believe that the paper could do better in substantiating the claims about why this approach should perform better. This could either be done through loss function visualizations, mathematical arguments (bottlenecks and compression might be applicable), or other quantitative reasoning. I am happy to change my view if these improvements are provided.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work proposes a technique for training imitation learning policies that use temporal information. They show policies trained with their method are less "causally confused" (copycat problem) compared to other behavior cloning methods. In their approach, they decompose a policy into two - the first policy only takes in the current time step's observation (reactive) and outputs an action, along with a intermediate feature (their best performing model just uses the action as the feature). The second policy (refinement stage) takes in the feature, along with the history of observations, and outputs the final action. They evaluate their method in an autonomous driving simulator (CARLA NoCrash benchmark) and MuJoCo environments.
main_review: #### **Strengths**
* Clear to read and their method is succinctly illustrated (Figure 4)

* Motivations from human-decision making and various works in cognitive psychology

* Improvements over previous methods

* Ablation on design choices in their method

#### **Weaknesses**
* I would like to see more insight on why their method increases performance by such a drastic margin. Section 4.3 and Table 3 provide some hints, but I am still not completely sure where the improvements came from. If the copycat problem stems from optimization difficulties and networks converging to the simple solution, why wouldn't the refinement model simply copy the coarse action?

* In the ablations, using BC-OH $\pi_\phi$ (coarse) and BC-SO as $\pi_\theta$ (refinement) leads to a significant drop in performance, performing even worse than the BC-OH baseline in Table 1. I fail to see why this would happen since $\pi_\theta$ could simply copy over the coarse action and would like to see a bit more elaboration from the authors.

#### **Suggestions**

* Elaborating on section 5.4, to see if the refinement model actually refines the prediction, an interesting experiment would be to actually quantify the difference between the coarse action and the final action in each model in the ablation.

* The main results and ablations are performed in CARLA so MuJoCo results can moved to supplementary

* I am familiar with the CARLA NoCrash benchmark and know the dense setting in particularly is prone to a lot of noise in evaluation (traffic jams which cause early termination or timeout). I would like to see the community migrate towards more robust experiment testbeds in CARLA (the more recent online leaderboard and provided routes).

summary_of_the_review: Overall this work addresses a well known problem (copycat problem) and builds towards training better imitation learning policies with temporal information. I enjoyed that the method is simple and effective on the NoCrash benchmark, but I find the justification and analysis a bit incomplete in its current state to be recommended for acceptance.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper presents a combined approach that combines two predictors with one based on a single observation (BC-SO) and the other based on a history of observations (BC-OH). The intuition behind this approach is that BC-OH has the "copycat" problem which basically means the predictor is trying to simply extrapolate the previous action instead of using data to do some decision makings, while BC-SO does not have such a problem but suffers from the partial observability issue. The authors also evaluate their method based on the CARLA and MuJoCo continuous control tasks. Results clearly show the effectiveness of this combined approach.
main_review: [Strength:]
Overall, this paper starts with the "copycat" problem in imitation learning when using historical observations and proposed an ensemble approach to balance between a single observation-based predictor and a history of observations based predictor. As per my knowledge, this combination is novel. Experimental results also demonstrate its effectiveness.

[Weakness:]
The authors did an analysis empirically on why the proposed approach can address the "copycat" problem, however, I feel it still lacks thorough studies regarding what exactly is the "copycat" problem either in a distributional perspective or a decision making perspective. As a result, I think it is insufficient to recognize the authors' novel contributions.

summary_of_the_review: The authors proposed a combined approach to address the "copycat" problem in imitation learning when learning with a history of observations. This approach is new, the problem is interesting but the paper lacks satisfactory investigation of the problem itself and how the proposed method can solve the problem.
review_title: null

Meta-review: The reviewers unisono do not accept the paper, because it is (a) not well-written; (b) experimentally not convincing, but addresses a nice problem.  I suggest that the authors address the issues in a subsequent paper, and resubmit to one of the main conferences.
Decision: Reject