Venue: CoRL 2021 Conference

Reviews:

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: One of the problems in trajectory prediction is that the predicted trajectories do not abide by its context, i.e., they can go out of drivable areas. To solve this issue, the authors proposed an additional "unlikelihood" loss function which discourages the likelihood term to put lower density near negative trajectories. The authors show the improvement on nuScenes dataset where the loss is added to Trajectron++, and Argoverse dataset where the loss is added to Gaussian LaneGCN. The ablation suggests that this approach performs better than an adhoc post-processing approach that removes violation.
strengths_and_weaknesses: Strengths
1. I like the key idea is based on the fact that validating trajectories is easier than generating them. So, we can first classify good and bad trajectories and use those to shape the output distribution. 
2. The idea of using "unlikelihood" seems to be novel, and this paper can inspire future avenues in exploiting negative trajectories.
3. The ablation and the results are convincing.

Weaknesses
1. The loss function used in the experiment is not clear to me. For example in eq 5, what does p_theta(Y|X,z) looks like? Is it a gaussian with learned mean and variant? 
2. How does metrics reflect the multimodal nature of the trajectories? L2 measurement alone does not capture how accurate we model Y|X when they are multimodal. 
3. Figure 1 is not very clear in demonstrating the issue the paper is trying address. The uncertainty is only slightly off the track. Also, it seems like the high uncertainty is more of a problem in this figure. Also it would be nice to compare the result of this method before and after in in figure 1.
4. How sensitive is the result to the hyper-parameter? Is the result overfit to the validation data -- as the paper mentioned that the validation data are used for testing?

Others:
1. There might be similar approaches in ML+motion planning to avoid constraint violation. 

summary_of_recommendation: The paper provides a simple trick that can be added generative models for motion forecasting. The paper provides convincing results that it can improve over the state-of-the-arts. However, I still have slight concern over the hyper-parameter tuning and test settings.
issues: I'd like to see the authors address the 3 points mentioned above in the weakness section. 
Some word choices: geometrical nearness --> geometrical proximity
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: This paper's contrition is to use map information to mitigate false positives in motions forecasting tasks, by minimizing the predictive likelihood of motions that are known to violate known lane directions in a HD map.
strengths_and_weaknesses: Strengths:
 - Motivation
 - Clarity of writing
 - Experiments and multiple real datasets used

Weaknesses:
 - Context checker is not learned, and some of the closest related works were not cited. While the learned part of this work is the original predictive model, and the context checker provides a superior loss function, as the authors demonstrate, the rules are handcrafted (e.g. binary 90 degree rule on lines 156-157), which makes this difficult to immediately extend to anything else, like cyclists, pedestrians, etc. While the point of this paper is not necessarily this, but to open research up to unlikelihood training in motion forecasting, other methods have used learning based approaches to mitigate false positives for autonomous vehicle motion in ways that are less hand-crafted and more widely applicable (see R2P2 algorithm, cited below [A]). In addition, the Beelines paper (cited below [B]) is very similar by using map information to alter the loss function for motion forecasting, and should be discussed in the context of this work.
summary_of_recommendation: I recommend weak-accept in light of the "strengths and weaknesses" discussion above. However, willing to consider strong-accept based on author responses. This is an important area that the community should be looking into more, and comparing against related works would help strengthen this paper.
issues: Line 42 "we are the first to propose unlikelihood training for continuous space of trajectories": See R2P2 [A], e.g. fig 6, noting H(q,p) term to penalize bad samples. Might be worth referencing this work and discussing the differences.

Lines 44-46: "inject contextual information as a learning signal" seemed a little vague - could this be clarified?

Grammar line 64: "while Tang and Salakhutdinov [2], Salzmann et al. [1] uses"

Line 63 and 66: small preference against using reference numbers as nouns to begin a sentence.

Line 78-79, could probability delete "in contrast to continuous predictions that represent the future trajectories in our setting" since next sentence says this too.

Line 92: perhaps clarify different between x and X? X is random and x is not?

Line 101: should "Y_{i}" be "y_{i,gt}" (line 92) or "Y_{i,gt}" (line 113) or "\bold{Y}_{i,gt}" (Fig1 caption) or "\bold{y}_{i,gt}" (line 171). Should some of these mean the same thing and should be following consistent notation? 

Line 102-103: change "between the predicted distribution and ground truth distribution" --> "between the ground truth distribution and the predicted distribution"

Left quotations on lines 105, 137, 138, 283

I think the connection to REINFORECE in lines 138-142 is tenuous, consider removing.

Line 151: "map-based checker": this is related to the Beelines [B] paper, worth citing and comparing to in the related work, since Beelines also contributes map information to alter the typical ML loss for forecasting in a way that avoids the more obvious-to-humans out-of-bounds errors.

Line 157-158: "The trajectories that fail to pass the test are identified as negative trajectories Y i,neg": I wonder if there's a future work here based on a continuum of being more and more negative, the more the vehicle heading differs from the lane's heading, in addition to binary pass/fail tests.

Equation 4: it's interesting the log_sigma cancelled out on line 174 and now doesn't feature in the sigma gradient term, effectively turning this into an L2 learning task with an initially-set hyperparameter of sigma. I.e. the two Gaussians have in effect lost their "complexity terms" that regularize sigma. I assume one can expect non-equal number of positives and negatives sample in general, so this affect would disappear, unless the positive and negative batches are *averaged* and not *summed*? Possibly worth commenting on this somewhere lines 177-186, if space limitation allow for (not big priority).

Lines 187-194: interesting observation! Nice explanation.

Table 1: what is the "Context-Violation-Rate" of the ground truth trajectories for reference? (presumably something non-zero, but close to zero? Unless the "disable rule" in lines 160-161 ensure it'd be zero exactly?)

Lines 235-237: nice explanation.

Line 284: "and leads to a better prediction" - interesting! Can the standard error be added into Table 5 that shows this, analogous to Table 4 etc?

I like the combination of likely and unlikely trajectories in equation 2, especially since even if there's an error in the context checker, significant likelihood of an other-wise valid motion can override such errors. But this really depends on how is gamma set: how should gamma be set? Edit I see on lines 259 it's scheduled 0 to 1, is there anything special about the number 1 in this context, or could it be anything? (do positive and negative sample matter equally for the purpose of accurate prediction? Could you have got better scores if this was schedule up to a maximum of 0.1 or 10?). Additional experiments not necessarily required, more after an explanation and clarification in the text.

[A]
R2P2: A reparameterized pushforward policy for diverse, precise generative path forecasting
by Nicholas Rhinehart, Kris Kitani, and Paul Vernaza
https://openaccess.thecvf.com/content_ECCV_2018/papers/Nicholas_Rhinehart_R2P2_A_ReparameteRized_ECCV_2018_paper.pdf

[B]
Beelines: Motion Prediction Metrics for Self-Driving Safety and Comfort
by Skanda Shridhar, Yuhang Ma, Tara Stentz, Zhengdi Shen, Galen Clark Haynes, Neil Traft
at ICRA 2021, https://arxiv.org/abs/2011.00393
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The work introduces the application of 'unlikelihood' loss to augmenting standard training losses for trajectory learning. The work supports the notion of applying (soft) constraint to the training objective by analysing contextual information to inform what should or should not be penalized.
strengths_and_weaknesses: - The idea is very simple but seems to be effective
- It addresses a seemingly obvious but in my mind absolutely necessary encoding of violations in the application domain
- The paper is well written and the ideas come across clearly
summary_of_recommendation: Overall I think the paper presents a simple but very practically useful application of the unlikelihood loss for trajectory learning.
issues: Comments and Questions
- Connection with MAP: In the abstract the contribution is stated as "Orthogonally, we propose a new objective, unlikelihood training, which forces predicted trajectories that conflict with contextual information to be assigned a lower probability". While the term "unlikelihood" in this context makes perfect sense (and is adopted from prior work - welleck 2019), in my mind what it *really* seems to be in the Bayesian sense is that it is simply a _hyperprior_ in precisely the same sense as in maximum a posteriori (MAP) estimation.  I see that the hyperprior is, in the unliklihood sense, simply another distribution imbued by the "contextual information". Could the authors provide any commentary on a link between the two?

- Unlikelihood data
The authors state "However, these are not given in the dataset. To solve this issue, we approximate the samples by directly drawing a set of trajectories from the predicted distribution and select the trajectories that violate the contextual information out by a context checker." This brings to mind the idea of an adversarial game being played between the model and the checker. Although here the checker is assumed to be a kind of oracle. Would there be any possiblity for encoding uncertainty over the checker itself? Furthermore, it would seem the method is highly dependent on sample complexity. I.e. if the complete set of "invalid" trajectories is sampled uninformatively, then it would seem that the loss would practically contribute not much (since there are no negative samples to learn from). This problem would be exacerbated (potentialy exponentially in higher dimensions). Have the authors considered how such a problem may manifest in the given problems and also how to alleviate such problems (perhaps some kind of informative sampling regime).

- Figure 3. I find the diagram somewhat visually uninformative. This could be improved in many ways: highlighting areas of major differences as given in the caption. At the very least the diagram could be helped with a legend that identifies the various components of interest to leave the reader wondering less (e.g. are there any signficant differences between the blue, green and red distributions? - this is unclear from the photo)
review_title: null

Meta-review: Summary: 
This paper presents a method to improve the quality of agent predictions by incorporating a contextual loss term that motivates unrealistic predictions (e.g. violate lane direction) to be assigned lower likelihoods. The method is validated on two benchmark scene sets. 

Clarity: All reviewers commend the paper for its clear presentation. The paper flowed well and presented a well-motivated, well-defined piece of technical work. 

Significance: Reviewers strongly agree that this paper pushes an area of work that could be extremely impactful to the community. 

Quality: The experimental results are overall convincing. In fact, reviewers nXVp and XGcM both note that the proposed method seems simple, however, provides convincing improvements to algorithm performance. This gives the method a high likelihood of being adopted by the research community. 

Originality: There was mixed feedback about the novelty of the proposed approach, and the paper could improve in articulating its contribution. (See “Opportunities for improvement” for more details).

Pros: 
This paper presents a method that is philosophically compelling in principle and whose utility is backed up by comprehensive simulation experiments. The paper is also very well-written; readers are able to grasp the main ideas right away and see those main ideas come through in the experimental section. 

Main opportunities for improvement: 
There are two main areas that can be improved to further solidify this paper. 

The first is orienting the proposed work better in current literature. Reviewers mention a number of related papers and methods; this paper should explicitly describes similarities and differences between itself and these existing techniques. Reviewers Fbgh and nXVo provide specific examples of missing related works.  

The second is to refine the experimental results section. The revision should expand on 1. implementation details the should be clarified and 2. further analyses and explanations of the existing results. All reviewers, and in particular reviewer Fbgh, make specific suggestions for what needs to be addressed.

Thank you for considering our feedback, and we look forward to seeing the updated paper. 

================ Final Decision

All reviewers recommended accept. I decided to recommend oral presentation because after revisions, reviewer Fbgh updated their score to strong accept as well, the paper has strong scores for impact (4s across the board), extremely strong scores for technical quality and clarity across the board, and positive originality scores. In general, the paper seems to be very well received. 
Decision: Accept (Oral)