Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The goal of this paper is to train models that imitate human behavior using limited samples. A human reward function is given but human behavior may be suboptimal according to this reward function and therefore we cannot use the standard assumption, Boltzmann rationality. The authors propose an imitation learning algorithm that uses the known reward function to form a prior over human policies, the Boltzmann policy distribution (BPD). A posterior over policies is then inferred from human behavior. This approach is empirically more data efficient than behaviorally cloning human behavior.

main_review: Overall, I recommend accepting this paper because it provides a theoretically clean analysis of a novel insight that is relevant to important problems. The key insight is that one can use the Boltzmann distribution over policies as a prior, to make behavioral cloning more data efficient. 

Significance:

The proposed approach is computationally expensive, but it makes sense to use it when compute is abundant but human demonstrations are scarce or expensive. This is likely to be a common situation in the future as human time stays equally expensive while computation budgets grow. Hopefully, paper’s insights can spur more computationally efficient approaches.

The paper provides insights relevant mainly to two important issues. Firstly, imitation learning from systematically suboptimal human behavior. Secondly, human-AI coordination, which benefits from predicting human behavior and is increasingly relevant in areas such as robotics and autonomous driving. 


Experiments:

The authors empirically demonstrate clear advantages over Boltzmann rationality. They also show that their method better predicts human behavior, and that it is more data-efficient than behavioral cloning.

Aside from a toy environment, the evaluation is done in Overcooked, a complex and interesting environment. Overcooked is fittingly a key benchmark for human-AI collaboration which is a goal of this paper.

The paper is overall very well written and illustrated.


Concerns:

1) The authors left it unclear why BPC should be more data efficient than behavioral cloning (BC). While BPC needs less samples empirically, this could just be because the BPC was allowed to learn online during each episode whereas BC was not. In fact, the BC baseline is left underspecified. Is the proposed method a special case of BC that uses a specific prior (the BPC)? For a fair comparison, the authors should use the same models, inference algorithms, online learning, etc for BC and the proposed method. Arguably, the only difference should be the prior (BPC) or any other aspect that the authors claim is novel.

2) One of the motivations of the paper is to understand human preferences, but the proposed method assumes that the human reward function is already known. This seems contradictory or at best requires explanation. I’m referring to the first sentence in the paper here. If the motivation is not to understand human preferences, I’d suggest to clarify the actual motivations throughout the introduction. The other motivations (predicting human actions and collaborating with humans) make more sense to me so you could simply remove ‘understanding human preferences’ to partially address my comment. However, I also think these motivations deserve more than one sentence of elaboration. Additionally, the conclusion says that the the BPD could be useful for reward learning, even though it assumes that the reward is known. There’s also explanation needed here.

3) A further limitation that should be discussed is that the BPC assumes the human policy to be close to optimal with high probability. If the human is systematically irrational they may achieve a much lower reward than what is optimal. Consequently, the BPC may assign very low probability to the true human policy.





———————— Detailed comments ————————


The experiments don’t seem to answer if the BPD prior is actually needed for good performance. It might be enough to do online inference with a different prior (e.g. uniform)?

“Understanding human preferences, predicting human actions, and collaborating with humans all require models of human behavior. “; “However, Boltzmann rationality fails to capture human behavior that is systematically suboptimal.”: some of these key claims currently lack support in the introduction. They’ve been supported e.g. in arXiv:1712.05812 and arXiv:1512.05832.

Figure 2 lacks a bracket after R(s,a). It also lacks the labels b) and c).

The introduction was a bit long, e.g. the last paragraph does not seem essential.

The paper repeatedly says that BPD is hard to compute because policies lie in a (high-dimensional) continuous space. It’s unclear why continuity would be a problem here, and I suspect this is not true. Indeed a high-dimensional discrete space seems more difficult for optimization and inference.

summary_of_the_review: Overall, I recommend accepting this paper because it provides a theoretically clean analysis of a novel insight that is relevant to important problems. The key insight is that one can use the Boltzmann distribution over policies as a prior, to make behavioral cloning more data efficient. The experiments and presentation are overall convincing. I have concerns about the behavioral cloning baseline and the relevance to learning preferences.


UPDATE: Although we were not able to resolve some confusion during the discussion period, my substantive comments have been addressed so I have updated my score now.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes a new model BPD to account for the suboptimality of human behavior, and provide an approximation inference method for BPD. The authors illustrate BPD through a simple simulation and compare BPD with existing methods on the overcooked game. They show that their model can match the performance of the data-extensive BC method and outperform BR. They further evaluate their model on human-AI collaboration and show that their model can match or even outperform the human-aware RL model.
main_review: The idea is pretty interesting and the approximation makes it possible to train a rather implausible model BPD. The paper is well written. I find it easy and clear to read through, without the need to go back and forth. The hyperparameters and the setup of the experiments are clearly provided. It would be good if the code can be shared if accepted.

Though the model is only tested on one complicated task and more experiments are always better, I find the experiments in the paper serve their purpose to illustrate the effectiveness of the model. I also find the simulation very useful for understanding the model, and as a sanity check to make sure after approximation the model is doing something as expected.

The intuition of the model is sort of based on “people are consistent over time”. It makes sense to me, but I’m wondering if there are any references on this. The better performance of BPD compared to BR sort of supports this, but it’s not clear whether it is the consistency of choice that actually benefits BPD. BPD is clearly a much larger and complicated model.

It would be good to put the size of the training set for BPD compared to BC. Is the BPD only directly trained/adapted on the testing data? It would be good to briefly discuss how easy/hard to train BPD.
summary_of_the_review: I’m overall positive about this paper. The idea is interesting and the experiments illustrate the effectiveness of the new model. The model has the potentials to be applied and further adapted to other settings.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper addresses the problem of modeling human behavior in a case where their deviation from the optimal behavior is consistent over time, rather than independent (systematic suboptimality). They claim that systematic suboptimality can be modeled by predicting policies rather than trajectories. To this end, they introduce Boltzmann policy distribution (BPD), as an alternative to Boltzman rationality, as a prior over human policies. BPD enables adaption to human policies over time. While in Boltzmann rationality past behavior cannot be used to predict future behavior (since current action is independent of all the previous actions), the paper claims that since humans are consistent one can use past actions to predict future ones which leads to introducing BPD. They consider that human is sampling over policies rather than trajectories which results in the previous actions and states inducing a posterior over policies. By taking the expectation over the posterior they predict the next action. They further use deep generative models in order to sample from the BPD and minimize the KL divergence between the two distributions (P_BPD and the distribution induced by the generative network) to ensure the predicted distribution is close to the BPD.

Through experiments, they demonstrate cases where Boltzmann rationality is not able to predict the behavior of a suboptimal human while BPD can adapt to the behavior over time and predict the correct next action. They also show that their method is able to predict human behavior as well as the baseline while using fewer data.
main_review: Strengths:
- Very well written.
- Contains many insightful examples and figures.
- Most of the statements are very well supported

summary_of_the_review: The paper addresses and motivates an interesting problem and further solves modifies the previously known Boltzmann rationality, introducing Boltmann Policy Distribution, in order to adapt to the human behaviour over time. The technical details of the approach are well written and supported and experiments are well designed to give illustrations of the cases where the Boltmann rationality will fail while BPD manages to predict human behaviour. I believe the paper is a good paper and authors have taken time to write it in an elegant way and design insightful examples to motivate their problem and show the advantages of their method.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: 
* This paper proposes an interesting approach to model and predict human behavior.

* The approach is called Boltzmann policy distribution (BPD). It improves the Boltzmann rational model in that BPD considers the systematic suboptimality in human behavior.
    * Systematic suboptimality means that the human could be consistent in producing suboptimal behavior. Hence, to capture systematic suboptimality, it is necessary to combine the human's reward function (optimal behavior) and trajectory data (deviation from optimality) in human modeling and prediction.

* This approach predict human policies, rather than trajectories, so that it can capture the (systematically suboptimal) human behavior that is reflected in the human action choices over time.

* Approach detail: the approach follows GAN (generative adversarial networks):
    * The goal is to compute the human BPD policy `= \integral \pi(a|s) * p_PBD(\pi | s1,a1,...,s_{t-1},a_{t-1})` (Eq5).
        * `\pi(a|s)` is approximated as a **generator**, `f_\theta(s,z)` (Eq.6).
        * `p_PBD(\pi | s1,a1,...,s_{t-1},a_{t-1})` is approximated as sampled human trajectories, `q_\theta(z | s1,a1,...,s_{t-1},a_{t-1})` (Eq.6).
    * The base measure, `p_base(\pi)`,  is defined as the optimal human behavior based on the known human reward function with suboptimality parameter \beta (Eq4).
    * The **discriminator**, `d`, used to distinguish between the policy generated by the base measure, `p_base(\pi)`, and the policy generated by the sampled human trajectories, `q_\theta(\pi)`.
    * The networks can be seen as: `\pi(s) -> d(\pi) -> z -> f_\theta(s,z) -> \pi(s) reconstructed`.
    * The training for the generator, `f_\theta(s,z)`, is in Eq8,9 and the training for the discriminator, `d`, is in Eq10.


* Results
    * Apple picking gridworld with simulated human data
        * Cross entropy: BPD < Boltzmann rational models
    * Overcooked (prediction only) with real human-with-human play data
        * Cross entropy: BPD = behavior cloning < random < PPO+self-play, Boltzmann rational models
    * Human-AI collaboration (prediction and control in Overcooked) with "human proxy" policy from previous work
        * Mean return: human-aware RL policy (prev work) > policy based on Boltzmann rationality, self-play policy, behavior cloning policy
        * Mean return: policy based on BPD > policy based on Boltzmann rationality, self-play policy, behavior cloning policy
        * Mean return: policy based on BPD > Human-aware RL policy (prev work) in 2 out of 3 cases.

main_review: 
* Pros
    * Nice writing and figures.
    * Very clear difference between Boltzmann rational model and the proposed method.
    * Real world (Overcooked game) dataset evaluation.
    * The authors not only evaluated prediction power of the proposed method in Overcooked, but also combine the proposed method with RL agent to do the Overcooked game.

* Majors
    * I think (please correct me if I am wrong :) the proposed approach is very much similar to GAIL (`Ho, J., & Ermon, S. (2016). Generative adversarial imitation learning. Advances in neural information processing systems, 29, 4565-4573.`)
        * The generator, `f_\theta` in this work = `\pi_\theta` in Alg.1 in GAIL.
        * The discriminator, `d` in this work = `D_w` in Alg.1 in GAIL.
        * The sampled path, `q_\theta = {s1,a1,…,s_{t-1},a_{t-1}}` in this work = `\tau_i ~ \pi_\theta_i`, in Alg.1 in GAIL.
        * The base measure, `p_base` in this work = the expert policy, `\tau_E ~ \pi_E`, in Alg.1 in GAIL.
        * In my opinion, this work is similar to GAIL. The only differences are the followings:
            * This work captures the suboptimality in the human behavior in the base measure, while GAIL assumes that the human is optimal.
              * This work's reward is implicitly hidden in the base measure, while GAIL's reward can be interpreted as `D_w`.
        * If this is true, it would be great if the authors could add some text to explain the relation between the proposed method and GAIL. This could be formally and/or empirically comparison.


* Minor:
    * The comparison between the proposed method and behavior cloning is a bit unfair, since the proposed method has the knowledge about the human reward, while the behavior cloning does not.
    * Page 2: `Conditioned on the (known) goal of getting to the office, observing the person choose route A gives no information about how the person will act in the future.`
      * This sentence makes sense assuming that the human reward function is correct and fixed. I think if the human reward can be updated, then observing that the human choosing route A could provide information. It would be great to clarify this here.
    * Page 5: `We discuss in Section 3 how to approximate this inference problem.`
      * Section 3 is a typo?
    * Page 6: Eq9: `p_unif` should be `p_base`?

summary_of_the_review: * I think it would be great if the connection between the proposed method and GAIL could be stated more clearly (if there are any). And empirical comparison with GAIL could be helpful, too. This is why I currently give 2 for the `2: The contributions are only marginally significant or novel.`

# Post-rebuttal
Thank you for your clarification! I have adjusted my score.
review_title: null

Meta-review: The paper presents a new approach to learning human behavior by observing a small number of interactions. To this end, it proposes a Bayesian learning framework where a Boltzmann-type prior over human policies, based on an available reward function, governs default behavior. The prior is updated by observing actual trajectories taken by (human) agents, in principle. In practice, a full-fledged implementation using Gaussian priors and features from a neural architecture is proposed and shown to be effective in practical benchmarks. 

The reviewers are all positive about the paper's contributions. One remaining concern is that the effect of the quality of the prior (Boltzmann vs. other type vs. features designed in a different way) on the learning process is not explored to a significant depth. Yet, the results and approach proposed in the paper are valuable to merit acceptance.
Decision: Accept (Poster)