Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 8: Accept
review_content: The authors propose a method to diversify samples generated from a VAE. The method is based on determinantal point processes in the latent space and relies on specifying a kernel in the sample space and a quality metric in the latent space. 

Increasing diversity of samples from VAE-like models is an important and common problem and the authors present a reasonably generic method for solving it. The experiments are sensible and show clear improvement over a reasonable selection of baselines. One improvement I would suggest is adding a more explicit discussion of how the proposed method avoids generating overly diverse, very improbable trajectories. It is also somewhat suspicious that the trajectories shown in Figure 3 are terminated so early.
review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: This paper presents a novel approach for forecasting object trajectories (e.g., predicted paths of vehicles) forcing diversity of outputs. The authors adopt determinantal point processes (DPPs) to capture the diversity and propose a diversity sampling function (DSF) which consists of a neural network. Its trainable parameter (i.e., \gamma) maps the past trajectory into a set of latent codes and they are decoded to feasible trajectories by a pre-trained conditional variational autoencoder (cVAE). The DSF is trained by maximizing the diversity of the output trajectory. But, since the standard log-likelihood function can be singular, they additionally present an objective function for diversity by maximizing the expected cardinality, which admits replicated outputs. In experiments, the proposed method finds more diverse paths than other competitors under both for synthetic 2D objects and human motions.

This paper is well-written and easy to understand. The contribution can be important as it performs better than other generative networks that are not forcing diversity. Unlike them, the proposed method can capture the diversity trajectories, which requires for safety-critical applications. 

Main concerns:

1. The DPP kernel consists of a similarity (equation (8)) and a quality score (equation (9)). However, it is unnatural that the similarity is defined in the data space (x) and the quality is defined in latent space (z). A more naive approach is to define the DPP kernel as a function of x or z. Is there any specific reason to define those scores are defined in different space?

2. To maximize the expected cardinality, it is enough that the eigenvalues of L(\gamma) become large. Does the network find the trivial solution? E.g., all eigenvalues are the same as a very large value. In addition, the proposed diverse loss is not a new approach. The maximum induced cardinality of a DPP was proposed by [1] and its various properties were also studied therein. 

3. In a work of [1], a method to optimize the induced cardinality (a similar to diverse loss in this paper) is proposed. And it would be great to compare the maximum induced cardinality (also can be approximated by the greedy algorithm) to the MAP for inference of diverse trajectory.

4. It is also possible to apply DPP MAP inference to a set of latent codes generated from the encoder of cVAE. Then, the decoder can map the diverse latent variables to feasible data trajectories. Are these outputs also diverse? or does the diversity of latent space reflect the diversity in the data trajectory space? 

5. Computing the gradient of the proposed diverse loss is expensive since it is the trace of an inverse of a parameterized matrix. How long does it take to learn the proposed DSF compared to other methods? 

6. To avoid that the determinants become zero, a popular choice is to shift all eigenvalues with a small amount (this can be done by adding eps * identity matrix to the kernel matrix). Did the authors investigate other practical diverse losses?

7. It seems to be possible to train the parameters of the kernel matrix, i.e., k in equation (8) and \omega in equation (9). Did the author try to learn those parameters?

Overall, this work proposes an approach combined cVAE with a DPP for forecasting diverse trajectories and the empirical results are promising as it outperforms other methods. But, its novelty is incremental and competitors are not actually the models capturing diversity. I vote for a weak acceptance but depending on clarifications on the above concerns in an author response, I would be willing to increase the score.

Minor comments:

1 . Please specify the network architecture of DSF and details on the parameter \gamma.


[1] Gillenwater, Jennifer A., et al. "Maximizing induced cardinality under a determinantal point process." Advances in Neural Information Processing Systems. 2018.
review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: Caveat: I am very familiar with DPPs, but unfamiliar with the literature of trajectory forecasting, autonomous driving, etc.

Summary: This work introduces a generative model for diverse sequences based on a DPP, with the goal of providing likely yet non-overlapping possible future trajectories to models that require such information for safety concerns (eg, autonomous driving). Rather than using the DPPs negative log-likelihood as a measure of the trajectory set's diversity, the authors use the DPP's expected sample size as a proxy for a diversity metric. Experimentally, the authors show on two tasks that this approach generates more diverse, relevant trajectories than several competing baselines.

Recommendation: I recommend this paper be accepted, with the caveat that I am not well equipped to evaluate the experimental contribution of this paper, which in my opinion is the most important part of this work. I would have liked to see more extensive experiments, in particular (a) other baselines (DPP NLL as a diversity loss, different DPP kernels) and (b) experiments on larger datasets.

High level comments and questions:

- When generating the trajectory set (Alg. 3), your algorithm may generate sets of variable size (as the DPP NLL is log-submodular but not necessarily increasing). Did you also consider generating sets of fixed size, or of a minimal size? 

- Previous work has also looked at using DPPs to alleviate mode collapse in GANs (Elfeki et al., ICML 2019). Could you comment on how you expect such a method would perform on your chosen tasks? 

- The intuition that the DPP negative log-likelihood will overwhelmingly penalize subsets that contain a few similar trajectories seems very reasonable. However, I would have liked to see that intuition verified through an experiment that used the NLL as a diversity loss.

- The training sets for both experiments seem fairly small (1100 and 9400 training examples). Could you comment on this choice, and on whether you expect your experimental results to generalize to much larger datasets?

- Your motivating example is autonomous vehicles and safety concerns; with that example in mind, could you comment on how to select the radius R (or, equivalently, \rho) in Eq. 9? It seems like this choice would have significant downstream implications on the trade-off between accuracy and robustness to unpredictability.

- The idea of using the expected subset size as a metric for diversity is compelling. The authors may want to take a look at (Gillenwater et al., NeurIPS'18), which uses a similar metric as a proxy for user engagement. 

Minor comments: 
- after Eq. 2: please consider defining the acronym ELBO before using it.
- Eq. 8: if you are using an exponentiated quadratic, I believe the distance should be squared.
- Eq. 10: consider citing the relevant proposition from (Kulesza & Taskar).
 
review_title: Official Blind Review #3

Meta-review: The paper proposes an approach for forecasting diverse object trajectories using determinantal point processes (DPP). Past trajectory is mapped to a latent code and a conditional VAE is used to generate the future trajectories. Instead of using log-likelihood of DPP, the propose method optimizes expected cardinality as a measure for diversity. While there are some concerns about the core method being incremental in novelty over some existing DPP based methods, the context of the paper is different from these papers (ie, diverse trajectories in continuous space) and reviewers have appreciated the empirical improvements over the baselines, in particular over DPP-NLL and DPP-MAP in latent space. 
Decision: Accept (Poster)