Implicit Behavioral Cloning
Authors: Pete Florence
Corey Lynch
Andy Zeng
Oscar A Ramirez
Ayzaan Wahid
Laura Downs
Adrian Wong
Johnny Lee
Igor Mordatch
Jonathan Tompson
Venue: CoRL2021 Poster (2021)
Keywords: Implicit Models
Energy-Based Models
Imitation Learning

Abstract: We find that across a wide range of robot policy learning scenarios, treating supervised policy learning with an implicit model generally performs better, on average, than commonly used explicit models.  We present extensive experiments on this finding, and we provide both intuitive insight and theoretical arguments distinguishing the properties of implicit models compared to their explicit counterparts, particularly with respect to approximating complex, potentially discontinuous and multi-valued (set-valued) functions. On robotic policy learning tasks we show that implicit behavior-cloning policies with energy-based models (EBM) often outperform common explicit (Mean Square Error, or Mixture Density) behavior-cloning policies, including on tasks with high-dimensional action spaces and visual image inputs. We find these policies provide competitive results or outperform state-of-the-art offline reinforcement learning methods on the challenging human-expert tasks from the D4RL benchmark suite, despite using no reward information. In the real world, robots with implicit policies can learn complex and remarkably subtle behaviors on contact-rich tasks from human demonstrations, including tasks with high combinatorial complexity and tasks requiring 1mm precision. 


Reviews:
[{"review_id": "MPpJ7yzappw", "reviewer": "Reviewer_BwVg", "summary": "The paper proposes to perform Behavioral Cloning (BC) using implicit Energy Based Models (EBMs), and demonstrate the utility of the approach when compared to explicit models using simple feed-forward evaluation for inference and prediction. The authors compare to reasonable baselines on a large set of experiments, ranging from simple simulated environments to real-robot tasks. Although EBMs have been used extensively for on-policy RL problems,  the proposed approach applies strictly to the off-policy setting for imitation learning. A detailed analysis of the EBM modeling power is provided, along with an intriguing theoretical discussion of properties for function approximation.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- The paper is well written and generally easy to follow.\n- The idea is interesting and the results seem very promising.\n- The analysis is quite extensive and detailed, and did a good job at highlighting useful properties of EBMs and justifying their utility for Imitiation Learning problems. \n- A comprehensive set of experiments were conducted, which is to be commended. The toy problems help elucidate the benefits of using implicit models, while the simulated benchmark and real-robot demonstrate utility over baselines.\n- The video was well made and informative.\n\nWeaknesses / Questions:\n\n- Extrapolation/Interpolation:  This property of the implicit approach is demonstrated on the simple 2D regression problems (Figure 3), and presumably should help in out-of-distribution modeling and generalization. However, this is not demonstrated for the higher-dimensional simulated/real robot experiments, as only performance metrics (such as success rates) are reported. \n- Generalization: Related to the above point, it is unclear how well this method generalizes to new instances of the same task, or whether it is only over-fitting to data.  Given the model’s capacity to fit individual datum and “exhibit nearest-neighbor-like behavior” (line 71), combined with a large number of demonstrations (O(10^3)) for the high-dimensional tasks, there is concern that apparent generalization is actually a case of overfitting to densely-sampled data in action/Cartesian space. Showing that this is not the case, and that the method actually generalizes for high-dimensional tasks, could include tests with of different initial ‘piles’ or clusters of particles in the simulated Bi-manual sweeping experiment, for example. \n- Regularization and Overfitting:  If generalization requires additional regularization terms in the loss function (such as  increasing entropy of the distribution), how might this affect the results compared to the explicit models?\n- Video: It seems like the 2D experiments were not run to convergence for the Implicit model in many cases. Should we expect the EBM to collapse to the training data completely?\n- Sample and  computational complexity: Given that the performance of the EBM requires additional sampling-based optimization for inference, how does the number of samples (or number of iterations for Langevin) scale with dimension and performance? One might presume this would be a bottleneck for high-dimensional problems (ex. joint space control of a quadruped robot). \n- Real-world Experiments: the visual feedback policy operates at 5 Hz, but implicit model inference is 33 Hz (30ms). What is causing the bottleneck here? I find this surprising, but I may be missing something in the implementation description. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I find the work to be interesting and novel, and the scale of the experimental work is impressive. However, I have doubts regarding how well the approach generalizes. This is particularly the case in the manipulation experiments which use a large number of expert demonstrations in a relatively low-dimensional setting. As part of this, I am concerned that the model may be over-fitting to the data in many instances. If this is the case, then a discussion on appropriate methods for regularization may be necessary. \n\nIf the authors can demonstrate that this is in fact not the case, through convincing experiment and analysis (simulated results are fine), then I will happily eat crow and change my decision to a Weak Accept, at least. I would also like to have the other points in the review addressed.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "Please address the issues listed in the 'Weaknesses / Questions' portion of this review.", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {}}, {"review_id": "6skoInP5JG", "reviewer": "Reviewer_o4TR", "summary": "This paper proposes implict behavioral cloning, a modification to behavioral cloning that learns an implicit policy model (a scalar-valued function that accepts both state and action as inputs, and that must be minimized over actions to select an action) as opposed to the usual explicit policy model (a function that directly returns an action by taking a state as an input). The effectiveness of implicit policy models over explicit policy models is demonstrated over several toy function approximation problems, simulated domains, and physical real robot tasks. The paper also presents theoretical results that prove the effectiveness of implicit models in learning discontinuous, set-valued functions. ", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\nThe paper is very well-written and easy to follow. The empirical results are interesting and convincing. The toy function approximation results showcase the ability of implicit models to deal with discontinuous functions, multi-valued functions, and extrapolate favorably, when compared to implicit models. The results in simulated domains and real world tasks show that there can be substantial benfits to learning implicit models, especially when there are discontinuous or multimodal behaviors in the dataset.\n\nThe video results are extremely engaging (although narration would make it even better). The visualizations provided for the toy function regression examples and the multimodality and robustness qualitative real robot results were impressive.\n\nWeaknesses\n\nIt would be nice to include a more thorough discussion of the limitations of implicit models compared to explicit models, especially in the context of real world visuomotor learning. There is some discussion in the appendix, but including some discussion in the main text could be useful to provide a more complete picture. For example, can implciit models only be run at 5 hz? What kind of slow down is there in policy execution times? Does it depend on the method used for optimizing the implicit function at evaluation time and the exact parametrization chosen for the implicit function (i.e. is there hope for reducing the slowdown)?\n\nIt could also be interesting to further evaluate the generalization capabilities of implicit models compared to explicit models directly on simulated or physical robot tasks, similar to the experiment presented in Figure 4, where implicit models can exhibit better generalization in lower data regimes. For example, can the implicit model exhibit better spatial generalization to learn a lifting task with less data? (see https://arxiv.org/abs/2103.00375 for an example lifting task). Even showing that implicit models can use less data than explicit models and achieve the same quality of performance as explicit models that use the full dataset on the proposed tasks would be interesting.\n\nA few clarifying questions follow:\n- How were policy checkpoints chosen for evaluation when training the explicit and implicit BC models? At least for explicit models, there can be significant variation in performance from epoch to epoch because a surrogate objective is being optimized instead of the true objective of interest (task performance).\n- Were hyperparameters such as learning rate kept the same for implicit vs. explicit, or tuned for each method? Since the optimization procedure is different for implicit models, optimization hyperparameters should probably be tuned independently.\n- How many demonstrations were collected and used for each real robot task?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This paper proposes an alternative policy parametrization for behavioral cloning that is well-motivated by toy examples and demonstrates impressive empirical performance in both simulated and real domains.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "Address the comments and questions in the weaknesses section above.", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {}}, {"review_id": "4nPWy5IV83", "reviewer": "Reviewer_iNFj", "summary": "The authors present a new behavior cloning technique that allows for an implicit modeling of the task to improve the adherence of learned policies to demonstrations, thus improving performance. The authors achieve this by making a change to the policy representation such that it minimizes an energy function on the state and possible actions in the action space, and picks the lowest-energy action. The resulting policies are better at fitting diverse demonstrations.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths:\n\nS1) The paper is well structured and gets straight to the point about the innovation being discussed, which I appreciate.\n\nS2) While the proposed innovation seems like a relatively minor iteration over existing methods, the utility of the change in service of better demonstration modeling is clear.\n\nS3) I really like how a number of tasks of varying complexity are explored, especially the simpler problems which do well to establish the behavior of the proposed algorithm without results being obfuscated by complex dynamics.\n\n### Weaknesses:\n\nW1) Data representation could be improved - the number of seeds used for the data in each of the tables is not consistent, nor is the providing of variance information, which is only provided in Table 6.\n\nW2) While I can see that the proposed algorithm is better at behavior cloning and cloning multiple value functions simultaneously, it is not clear to me that discontinuity, which this algorithm makes strides towards eradicating, is strictly bad. It seems to me that continuity could be useful at times, if disconnected demonstrations need to be connected by a single cohesive policy - I'm not saying I necessarily disagree with the authors on the utility of discontinuity, but it would help to have it better justified - particularly in cases where same states might command different actions depending on which demonstration is being cloned, which could potentially result in anomalous behavior in sequence.\n\n\n### General notes:\n\nThis paper gave me some interesting things to think of - like what if we just trained RL actor-critic critic networks and took actions based on the argmax of the critic - though instability of value learning could be a problem there.\n\nOn the subject of existing imitation learning algorithms, papers such as DeepMimic and AMP: Adversarial Motion Priors may also bear mentioning as relevant to current state of the art - though they aren't quite behavior cloning so I don't hold that against the authors.\n\nI will admit that I did not have time to verify the theory presented in Section 5 so it is possible I may have missed something there.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "As mentioned, the data representation could use work on consistency. ", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "Please address data representation consistency", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {}}]

Author rebuttal:

Meta-review: === comments before the discussion ===

This paper presents the advantages of energy-based models in the context of imitation learning. The proposed method is evaluated using D4RL benchmark tasks and real-robot experiments.
Reviewers agree that the paper is well-structured and the findings reported in the paper are interesting. However, reviewers requested some clarifications and discussions to improve the paper. Please answer the questions and address the concerns raised by the reviewers such as sample complexity/scalability, experimental details, and limitations of the proposed method.

=== comments after the discussion ===

This paper provides interesting results, and authors addressed the concerns raised by reviewers during the discussion period. The area chair recommends the acceptance of the paper. 

Decision: Accept (Poster)