Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper presents a learning from demonstration approach to generate diverse policies representing the different preferences of the expert. The authors seek to learn a GMM in the latent space, and match the state-visitation frequency from state observations only. Experiments on simulated reaching tasks with robot arm and car driving suggest the feasibility of the approach.
strength_and_weaknesses: - Learning from demonstrations is an active area of research, and of interest to the wider community. Encoding and generating diverse behaviors from demonstrations are important from both what to imitate and how to imitate perspective.

- Problem formulation of the paper is similar to apprenticeship learning (or inverse reinforcement learning) under different preferences of the expert. In the problem formulation, the reward function needs to be parameterized in the preferences that the expert demonstrations are looking to encode. Eq.3 objective of matching state-visitation frequency under different preferences of the expert has also been well-studied in discrete spaces. Moreover, ,ean and mode seeking LfD is well-known in literature.

- In continuous spaces, the hidden markov model/hidden semi-markov model is better suited to encode the temporal sequence in the state observations, as compared to the GMM (see [1]). Subsequently, Viterbi decoding can be used for generating diverse demonstrations.

- There are no comparisons with other behavior cloning approaches to establish the merits of the approach. Moreover, there is no numerical quantification between expert demonstrations and generated demonstrations to quantify the performance.

[1] Sequential robot imitation learning from observations, IJRR, 2021

clarity,_quality,_novelty_and_reproducibility: The paper is well-written and easy to follow. Novelty is limited. Experimental set-ups are too simple to understand the limitations of the approach.
summary_of_the_review: This paper presents a learning from demonstration approach to encode and decode diverse expert demonstrations. The problem is of interest to a wider community. Consolidating the problem formulation further, quantifying the performance in the experiments and comparison with other approaches will improve the quality of the paper.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This study built a parameterized solution space to help model driving behaviors for planning, which considering the effect of individual differences of driving demonstrations. 
strength_and_weaknesses: Strength: writing and novelty
Weakness: Reproducibility and application potential 

clarity,_quality,_novelty_and_reproducibility: This description of work is relatively clear. The technological and practical novelty are not good enough (mainly due to these are not detailed well by the authors). Reproducibility is unknown because the codes are not open.
summary_of_the_review: 1. How about planning efficiency? Planning time consumption is vital to driving safety, and generally, an improper solution space could be a huge burden for computation speed. Some comparisons with some classic planning methods may be helpful.
2. How to define the feasibility of solution space? Because there are always some detrimental demonstrations existing. Maybe a not suitable space can cause an undesirable outcome. The authors can make this part clearer. 
3. For a Post-hoc solution search (Fig. 10), a set of trajectories close to the demonstration are obtained, but the relative extent is not discussed. A comparison between similar methods can make your contributions outstanding.
4. The authors proposed a method to find a potential solution from demonstrations. However, the authors should elaborate the application values. For example, learning from demonstrations in auto-driving technologies usually serves for human-like driving or driving safety. However, the treatment proposed in this study seemingly discusses a new approach without telling us what its meaning is. Following the descriptions of this study, the advantage is to find a solution through learning from demonstrations, but there have been many ways that can complete this. Maybe application potential and corresponding validations can help to make your contributions more valuable.
5. Reproducibility is unknown because the codes are not open.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes a method that learns a set of policies that can cover all modalities of the demonstration dataset. The proposed method first models the state distribution of the demonstrations using a Gaussian Mixture Model (GMM). In this paper, a trajectory is represented with indices of the top $d_z$ mixture components that best describe state distributions of the trajectory. Then, this paper searches for policies that best represent each of the trajectory representations using an evolutionary algorithm, which randomly samples policies, rollouts the policies, encodes resulting trajectories into trajectory representations, stores the best policy for each trajectory representation, and iterates this process. Since each trajectory representation holds a policy that generates the corresponding trajectory, this method can achieve a set of policies that generate multi-modal trajectories. The experiments show that the proposed method can generate bimodal trajectories. 
strength_and_weaknesses: ### Strengths

* This paper tackles an important problem of learning policies that perform multi-modal behaviors.

* The experiments demonstrate that the proposed method can acquire policies that correspond to diverse behaviors.


### Weaknesses

* The proposed method uses a Gaussian Mixture Model to approximate the state distribution. This may work on a small state space but would not scale to a high-dimensional state space, which is more practical. Moreover, this does not entail temporal information of the trajectory, which can limit its applicability to complex behaviors (e.g. loop).

* There are some alternative approaches to achieve multi-modal behaviors without need for the simulator and task reward [a,b]. Discussion about comparisons to [a,b] would be great.

* The tested environments and tasks are very simple -- 2D path planning with bimodal trajectories.

* The experiments tested on datasets with only two modes. It would be great to verify the proposed method with more modes.

* Instead of comparing with vanilla VAE, a comparison to VQ-VAE or VAE with a discrete latent space could be more appropriate since a quantized embedding in the proposed method can be the main source of generating multi-modal trajectories.

* The paper claims that the proposed projection function provides the interpretable encoding value. However, the encodings shown in the paper are interpretable mainly because the experiments are done in 2D path planning. The proposed encoding will not be interpretable with the high-dimensional observations.

* The proposed method assumes access to a simulator and a task reward, which limits its scalability to many applications.




[a] Shafiullah et al. Behavior Transformer: Cloning k modes with one stone. NeurIPS 2022

[b] Wang et al. Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning. arXiv 2022
clarity,_quality,_novelty_and_reproducibility: * The paper is well organized but some parts of the proposed approach is not easy to understand. As the proposed method consists of multiple training stages and components, (1) an illustration of the entire algorithm and (2) a better overview of the proposed method could be very helpful to improve the readability.

* The experiments well describe how the proposed method can generate a set of diverse behaviors. However, the experiments are done in too simple environments (2D or 4D observation and action spaces) and the demonstrations include only two modes.

* This paper tackles an important problem of learning diverse behaviors from data and the proposed method looks novel.
summary_of_the_review: This paper tackles an important problem of learning multi-modal behaviors from demonstrations. However, the proposed approach does not seem to scale and the evaluation is done on too simple tasks to show its impact and practical use. Thus, I am recommending rejecting this paper.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a novel algorithm to learn diverse behavior styles from demonstrations. It uses a projection function to encode demonstration trajectories into latent spaces and then uses MAP-ELITES to find a set of possible policies. Experiment results how the proposed model is able to learn a better coverage of different behaviors.
strength_and_weaknesses: Strength:
The approach or the combination of approaches is novel and the experiments clearly show its capability to learn from diverse demonstrations.

Weaknesses:
Would be better to have more ablation studies on the algorithm design choices, e.g. compare some different projection methods, as well as different generative models.

Would make the results more convincing by comparing with more than one baseline, especially other approaches that are designed for multimodal behaviors (e.g. VAE with multimodal latent space distributions).
clarity,_quality,_novelty_and_reproducibility: The overall clarity, quality, and novelty of the paper are good. The paper also gives experiment details to reproduce the results.
summary_of_the_review: The paper proposes a novel framework and the experiment results clearly show its capability to learn diverse behavior styles.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Modeling observations with GMMs, this paper uses those to create a latent space, which is then used to create policies with MAP-ELITES.

The methods proposed in the paper are incremental, and not supported by a comparative experimental section.  Furthermore, the experiments are all for low-dimensional systems---a necessary side effect of the use of GMMs.

Applicability of the proposal method over existing ones is not clarified.
justification_for_why_not_higher_score: Main argument against the paper: the use of GMMs leads to a non-scaling solution.
justification_for_why_not_lower_score: n/a
Decision: Reject