Venue: CoRL 2021 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The authors present a new behavior cloning technique that allows for an implicit modeling of the task to improve the adherence of learned policies to demonstrations, thus improving performance. The authors achieve this by making a change to the policy representation such that it minimizes an energy function on the state and possible actions in the action space, and picks the lowest-energy action. The resulting policies are better at fitting diverse demonstrations.
strengths_and_weaknesses: ### Strengths:

S1) The paper is well structured and gets straight to the point about the innovation being discussed, which I appreciate.

S2) While the proposed innovation seems like a relatively minor iteration over existing methods, the utility of the change in service of better demonstration modeling is clear.

S3) I really like how a number of tasks of varying complexity are explored, especially the simpler problems which do well to establish the behavior of the proposed algorithm without results being obfuscated by complex dynamics.

### Weaknesses:

W1) Data representation could be improved - the number of seeds used for the data in each of the tables is not consistent, nor is the providing of variance information, which is only provided in Table 6.

W2) While I can see that the proposed algorithm is better at behavior cloning and cloning multiple value functions simultaneously, it is not clear to me that discontinuity, which this algorithm makes strides towards eradicating, is strictly bad. It seems to me that continuity could be useful at times, if disconnected demonstrations need to be connected by a single cohesive policy - I'm not saying I necessarily disagree with the authors on the utility of discontinuity, but it would help to have it better justified - particularly in cases where same states might command different actions depending on which demonstration is being cloned, which could potentially result in anomalous behavior in sequence.


### General notes:

This paper gave me some interesting things to think of - like what if we just trained RL actor-critic critic networks and took actions based on the argmax of the critic - though instability of value learning could be a problem there.

On the subject of existing imitation learning algorithms, papers such as DeepMimic and AMP: Adversarial Motion Priors may also bear mentioning as relevant to current state of the art - though they aren't quite behavior cloning so I don't hold that against the authors.

I will admit that I did not have time to verify the theory presented in Section 5 so it is possible I may have missed something there.


summary_of_recommendation: As mentioned, the data representation could use work on consistency. 
issues: Please address data representation consistency
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper proposes to perform Behavioral Cloning (BC) using implicit Energy Based Models (EBMs), and demonstrate the utility of the approach when compared to explicit models using simple feed-forward evaluation for inference and prediction. The authors compare to reasonable baselines on a large set of experiments, ranging from simple simulated environments to real-robot tasks. Although EBMs have been used extensively for on-policy RL problems,  the proposed approach applies strictly to the off-policy setting for imitation learning. A detailed analysis of the EBM modeling power is provided, along with an intriguing theoretical discussion of properties for function approximation.
strengths_and_weaknesses: Strengths:

- The paper is well written and generally easy to follow.
- The idea is interesting and the results seem very promising.
- The analysis is quite extensive and detailed, and did a good job at highlighting useful properties of EBMs and justifying their utility for Imitiation Learning problems. 
- A comprehensive set of experiments were conducted, which is to be commended. The toy problems help elucidate the benefits of using implicit models, while the simulated benchmark and real-robot demonstrate utility over baselines.
- The video was well made and informative.

Weaknesses / Questions:

- Extrapolation/Interpolation:  This property of the implicit approach is demonstrated on the simple 2D regression problems (Figure 3), and presumably should help in out-of-distribution modeling and generalization. However, this is not demonstrated for the higher-dimensional simulated/real robot experiments, as only performance metrics (such as success rates) are reported. 
- Generalization: Related to the above point, it is unclear how well this method generalizes to new instances of the same task, or whether it is only over-fitting to data.  Given the model’s capacity to fit individual datum and “exhibit nearest-neighbor-like behavior” (line 71), combined with a large number of demonstrations (O(10^3)) for the high-dimensional tasks, there is concern that apparent generalization is actually a case of overfitting to densely-sampled data in action/Cartesian space. Showing that this is not the case, and that the method actually generalizes for high-dimensional tasks, could include tests with of different initial ‘piles’ or clusters of particles in the simulated Bi-manual sweeping experiment, for example. 
- Regularization and Overfitting:  If generalization requires additional regularization terms in the loss function (such as  increasing entropy of the distribution), how might this affect the results compared to the explicit models?
- Video: It seems like the 2D experiments were not run to convergence for the Implicit model in many cases. Should we expect the EBM to collapse to the training data completely?
- Sample and  computational complexity: Given that the performance of the EBM requires additional sampling-based optimization for inference, how does the number of samples (or number of iterations for Langevin) scale with dimension and performance? One might presume this would be a bottleneck for high-dimensional problems (ex. joint space control of a quadruped robot). 
- Real-world Experiments: the visual feedback policy operates at 5 Hz, but implicit model inference is 33 Hz (30ms). What is causing the bottleneck here? I find this surprising, but I may be missing something in the implementation description. 

summary_of_recommendation: I find the work to be interesting and novel, and the scale of the experimental work is impressive. However, I have doubts regarding how well the approach generalizes. This is particularly the case in the manipulation experiments which use a large number of expert demonstrations in a relatively low-dimensional setting. As part of this, I am concerned that the model may be over-fitting to the data in many instances. If this is the case, then a discussion on appropriate methods for regularization may be necessary. 

If the authors can demonstrate that this is in fact not the case, through convincing experiment and analysis (simulated results are fine), then I will happily eat crow and change my decision to a Weak Accept, at least. I would also like to have the other points in the review addressed.
issues: Please address the issues listed in the 'Weaknesses / Questions' portion of this review.
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: This paper proposes implict behavioral cloning, a modification to behavioral cloning that learns an implicit policy model (a scalar-valued function that accepts both state and action as inputs, and that must be minimized over actions to select an action) as opposed to the usual explicit policy model (a function that directly returns an action by taking a state as an input). The effectiveness of implicit policy models over explicit policy models is demonstrated over several toy function approximation problems, simulated domains, and physical real robot tasks. The paper also presents theoretical results that prove the effectiveness of implicit models in learning discontinuous, set-valued functions. 
strengths_and_weaknesses: Strengths

The paper is very well-written and easy to follow. The empirical results are interesting and convincing. The toy function approximation results showcase the ability of implicit models to deal with discontinuous functions, multi-valued functions, and extrapolate favorably, when compared to implicit models. The results in simulated domains and real world tasks show that there can be substantial benfits to learning implicit models, especially when there are discontinuous or multimodal behaviors in the dataset.

The video results are extremely engaging (although narration would make it even better). The visualizations provided for the toy function regression examples and the multimodality and robustness qualitative real robot results were impressive.

Weaknesses

It would be nice to include a more thorough discussion of the limitations of implicit models compared to explicit models, especially in the context of real world visuomotor learning. There is some discussion in the appendix, but including some discussion in the main text could be useful to provide a more complete picture. For example, can implciit models only be run at 5 hz? What kind of slow down is there in policy execution times? Does it depend on the method used for optimizing the implicit function at evaluation time and the exact parametrization chosen for the implicit function (i.e. is there hope for reducing the slowdown)?

It could also be interesting to further evaluate the generalization capabilities of implicit models compared to explicit models directly on simulated or physical robot tasks, similar to the experiment presented in Figure 4, where implicit models can exhibit better generalization in lower data regimes. For example, can the implicit model exhibit better spatial generalization to learn a lifting task with less data? (see https://arxiv.org/abs/2103.00375 for an example lifting task). Even showing that implicit models can use less data than explicit models and achieve the same quality of performance as explicit models that use the full dataset on the proposed tasks would be interesting.

A few clarifying questions follow:
- How were policy checkpoints chosen for evaluation when training the explicit and implicit BC models? At least for explicit models, there can be significant variation in performance from epoch to epoch because a surrogate objective is being optimized instead of the true objective of interest (task performance).
- Were hyperparameters such as learning rate kept the same for implicit vs. explicit, or tuned for each method? Since the optimization procedure is different for implicit models, optimization hyperparameters should probably be tuned independently.
- How many demonstrations were collected and used for each real robot task?
summary_of_recommendation: This paper proposes an alternative policy parametrization for behavioral cloning that is well-motivated by toy examples and demonstrates impressive empirical performance in both simulated and real domains.
issues: Address the comments and questions in the weaknesses section above.
review_title: null

Meta-review: === comments before the discussion ===

This paper presents the advantages of energy-based models in the context of imitation learning. The proposed method is evaluated using D4RL benchmark tasks and real-robot experiments.
Reviewers agree that the paper is well-structured and the findings reported in the paper are interesting. However, reviewers requested some clarifications and discussions to improve the paper. Please answer the questions and address the concerns raised by the reviewers such as sample complexity/scalability, experimental details, and limitations of the proposed method.

=== comments after the discussion ===

This paper provides interesting results, and authors addressed the concerns raised by reviewers during the discussion period. The area chair recommends the acceptance of the paper. 

Decision: Accept (Poster)