Venue: CoRL 2021 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper proposes a learning-based model to generate adversarial pedestrian generation for autonomous vehicle testing. The problem is formulated as learning a pedestrian placement model such that the expected number of collisions between the pedestrian and the vehicle is maximized. REINFORCE algorithm is used to train the model by considering the pedestrian and the vehicle behavioral policies as unknown dynamics, i.e. treating the pedestrian and the vehicle as black-box models. The model learns the pedestrian initial spatial distribution based on the scene semantic, distance to the vehicle, occlusion space, and occupancy map.  The model is trained using simulated data collected from CARLA. The results show that the proposed model can result in a higher number of collisions compared to random initialization and other baselines. The sensitivity of the pedestrian placement model to the pedestrian behavior model is also studied by considering several different behavioral policies. The result shows low sensitivity in most cases.
strengths_and_weaknesses: As the main strengths, the paper targets the problem of autonomous vehicle evaluation, which is an important and less studied subject in autonomous driving research. The introduction provides a convincing motivation behind the work, and the paper is generally well-written, detailed, and convincing. The supplementary materials also provide further details about the model architecture, input/output, rewards, and the algorithm implementaion. The paper seems to be an initial work that can potentially encourage further research on this subject. 
The main weakness is that the approach is still limited to a single pedestrian placement as opposed to considering a variable number of pedestrians or a flow of pedestrians. The placement model is interesting, but a true test may require a combination of both pedestrian initialization and behavioral model. Since there is still a dependency on the pedestrian and vehicle behavioral policies, there is a concern that the placement model becomes dependent on those policies, contrary to being a part of an independent evaluation/test system for autonomous vehicels. Lastly, the abstract and conclusion do not seem to converge to a single problem. Abstract mentions generating maximally challenging scenarios for autonomous vehicle testing as the main problem. The paper however concludes that the pedestrian initial distribution and behavior can be learned, but how to evaluate an autonomous vehicle is still not addressed in the paper. 
summary_of_recommendation: The paper is an initial work on adversarial pedestrian generation for autonomous vehicle testing. Certainly, not everything can be considered in a single paper and there is room for improvement. The main concern is that the paper is too focused on a single pedestrian initiation/behavior problem, which could also be addressed through some other techniques (e.g. model predictive scheme or some heuristic approaches) as opposed to considering a more comprehensive test generation concerning pedestrian-vehicle interactions.
issues: - The idea of generating challenging scenarios is interesting, but the challenging scenario seems to be narrowed down to adversarial trajectory crossing scenarios or maximizing the number of collisions. This might be a limiting assumption as it does not consider other aspects of autonomous vehicle testing. For example, a vehicle might be able to avoid a collision at the cost of sharp, sudden maneuvers. Or a vehicle might go off-road or collide with other vehicles in reaction to a pedestrian motion. In short, how to define a challenging scenario and how to evaluate an autonomous vehicle based on the test result?
- The problem formulation is convincing but the use case is still not completely clear. If the goal is to generate test cases, the result may need to be compared with manually generated test cases either heuristically or by filtering randomly generated scenarios. If the goal is to generate collision-prone smart agents then how it is better than a model predictive system that is designed to collide with another agent. Since the dynamics of a vehicle can be approximated or assumed to be known, a simple model predictive scheme might also be able to generate good initial/behavior distribution. 
- The distance unit is not clear in Table 1 and 2. Is it in centimeter or meter?
- It's recommended to add a list of notations and abbreviations as there are too many abbreviations in the paper.
- Line 218, type: mdoel >> model
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper considers the online/offline construction of a dataset of challenging scenarios that could support/enhance the training of collision avoidance policies for autonomous vehicles in urban scenarios involving realistic pedestrians. In contrast to existing approaches that either construct such training scenarios out of simplified (constant-velocity motion model) or "suicidal" (collision-seeking) pedestrians, this work proposed a novel learning based method to initialize pedestrians and to train a richer set of pedestrian motion models, while still keeping the level of collision (i.e., the overall difficulty of the challenge for the AVs) high. Results are shown in a variety of realistic urban scenes, using different initialization methods and pedestrian policies. There, authors even show that, under certain assumptions, collision-avoiding pedestrians can still lead to the most collisions with AVs when initialized at the right places in a carefully arranged world.
strengths_and_weaknesses: First of all, this paper is well written, and pleasant to read. I did notice a few typos or grammatical mistakes here and there (see list below), but nothing major. However, it took me multiple pages until I finally started to understand the actual premise of the paper, and I believe this is the main shortcoming of this work for now, but which can be addressed.

In my opinion, the need for non-suicidal, yet non-constant-velocity pedestrian models is currently weak in the first half of the paper. After my initial read of the abstract and introduction, I believe that I agreed with this premise (on which the entire paper relies), but would not have been able to articulate why or provide any form of example of such a scenario (i.e., non-trivial, non-suicidal pedestrian having a higher likelihood of collision with an AV).
Then, at the end of Section 3 (line 108-109), I found this sentence that finally mentions that an example of such a non-trivial scenario will be provided later in the experiments section. However, I believe this point really needs to be made clear much earlier in the text. In particular, as a reader, I believe that I would have benefited from a lot more intuition as to what class of examples of this type exist, as the paper heavily relies on this premise. For now, the paper reads a little like "Please believe us that this very interesting, non-trivial, yet counterintuitive statement is true; we will describe a method to fix it, and then at the full end provide you with one line of a table that shows that we were right (and a few sentences in the text)," and I worry this structure might not be best in supporting this premise that carries the entire paper.
Again, I can believe authors are indeed correct with their premise, which would make the work valuable and nuanced and interesting, but as a reader I would have loved to be taken in through this journey, rather than remain a spectator/believer until the "big reveal" (which also needs to be a much bigger/stronger reveal, if this is the main meat of the paper). I believe this is mostly a writing exercise, but one that will be crucial in bringing this paper to the necessary level to be understood/enjoyed by the community and be ready for publication. That is my one major concern with this paper in its current form.


Other than that, I only have a few minor comments, as the rest of the paper is very clear and articulates the method and all the assumptions made very clearly and convincingly:

1. Table 2: many of these values seem to be extremely close to each other, and I worry that these results are not statistically significant. Can authors present any form of analysis, or at least average error/standard deviations to help the reader form a more complete opinion of these results? These additional information should likely be discussed in the text as well.

2. List of typos/grammatical mistakes I caught (please perform a full spell/grammar check anyway, I am not claiming to have caught all of them of course):
	- line 154: "the \mu" --> \mu
	- line 154: 3m^{-1} --> should this rather be 3ms^{-1}? 3 1/m does not look like a velocity.
	- line 181: "encure" --> ensure?
	- line 221: "to fraction" --> to the fraction
	- line 243: could be instead be --> could instead be
	- line 253: the scenes --> the scenes'
	- line 318: mdoel --> model
	- line 321: coo-operative --> cooperative

3. The conclusion currently is very short and only summative, and therefore reads as extremely weak. It provides little over a redundant summary of the work, and no intuition or opening to future works, deeper reflection, etc. I worry it was put together rather hastily (as seen from the big typos in it), and believe it could be rewritten to better use this space to conclude the paper (key insight/take home message, deeper reflection, future works).
summary_of_recommendation: In summary, this work proposes a new method to create challenging pedestrian-vehicle scenarios for AV training using more realistic/nuanced, learned pedestrian motion models. The main premise of the paper is interesting and thought-provoking, but really needs to be expanded upon (provide intuition/example) earlier in the text, and again highlighted more thoroughly in the results section. Provided that this is addressed, I am in favor of accepting this work for presentation at CoRL 2021.
issues: Issues were listed above in my main review (major and minor ones).
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper proposes a reinforcement-learning based approach for learning a distribution over initial pedestrian locations which are challenging for a nearby autonomous vehicle. The core idea is to separate learning the distribution of initial pedestrian locations that lead to collisions from the human behavior model. This approach is compared in the CARLA simulatior with alternative methods such as randomly generating pedestrian locations or generating them in occluded spaces.  

strengths_and_weaknesses: Strengths:
- The problem of interest in this paper -- generating challenging simulated scenarios for autonomous vehicle testing -- is critically important and underexplored
- The idea of characterizing a distribution over initial pedestrian states which would induce a challenging scenario for the AV is interesting

Weaknesses:
- The proposed approach of decoupling the unsafe initial state distribution from the human behavior model requires more justification. Furthermore, there are too many confounding variables in the design of the proposed approach which make the utility of the overall pipeline difficult to evaluate. 
- It is unclear how realistic the proposed approach is for modern autonomous vehicles. More details provided below in the detailed comments section. 
- The clarity of the writing (both from a technical and also narrative perspective) could be improved. 

Detailed comments:
 
Section 1:
- Figure 1. Why does the AV input not include the history of the agent’s movement/poses (if they are not occluded)? This seems key since a core ability of the AV is re-planning and updating its predictions conditioned on the history of the human. 
- Figure 1 caption states that Scenarios #3 and #4 are illegal, but that seems like a misnomer. They aren’t illegal, but rather infeasible (under the laws of physics) for a pedestrian to “simply appear from nowhere into the line of sight of an AV”.
- “We have strong reason to believe that the optimal pedestrian distribution \mu is not easy to model…” This sentiment would be strengthened with a more substantive, quantitative definition of “easy to model”. For example, does this refer to the difficulty in choosing the correct features which influence the pedestrian distribution? Does this refer to the dimensionality of the distribution? 
- “... the pedestrian location distribution \mu is a learnable stochastic policy” The term policy canonically refers to a mapping from states to a (distribution over) actions. With this definition, it is unclear how a distribution over initial states is a policy. Clarifying this use of language would improve this sentence. 
-  “This allows us to model \mu as a convolutional neural network \mu_\Theta” The use of a neural network to model the initial distribution is not well motivated by the preceding paragraphs. For example, alternative methods for computing initial conditions from which safety-critical scenarios can happen include robust-control methods like Hamilton-Jacobi reachability, which compute challenging initial states via the solution to a dynamic programming problem. In the stochastic realm of approaches, the distribution over initial states could be parameterized by a mixture of Gaussians instead of a neural network, perhaps even enabling more sample efficiency. Discussing why a neural network model is essential for representing \mu would strengthen the motivation and technical approach of this work. 
- In the introduction, works which model adversarial (i.e., “suicidal”) pedestrians are referenced and this work seeks to differentiate itself from these works. However, in the second to last paragraph of Section 1, the pedestrian behavior “can be adversarial to the car or the pedestal location distribution \mu, or even not directly cooperative to either side.” This appears to be modelling an adversarial setting, which is precisely what the introduction was claiming was an undesirable modelling paradigm. Clarifying this is essential for understanding the difference between the proposed method’s philosophy and existing work. 

Section 2:
- “When the objective J is maximized…. Then both pi and mu become adversarial to the car’s objective J_\rho”. Notationally this sentence could be made more consistent: the car’s objective is referred to as J^p (not J_\rho) throughout the rest of the text. The reason behind why this statement is true is unclear due to the fact that J is not described, the relation between J and J_\rho is not established, and it is unclear where the adversarial nature of pi and mu come into play. 
- “This leads to the pedestrian only behaving suicidal.” Grammatical reworking of this sentence: “This leads to the pedestrian behaving in a suicidal manner.”
- “To allow for both, we separate nu explicitly into the initial pedestrian locations … and the pedestrian behavior pi.” This is one of the core ideas behind this paper. However, it is not clear why this decoupling should work. More specifically, the same initial pedestrian locations could (a) cause collisions with the AV or (b) not cause collisions with the AV under different pedestrian behavior models and different autonomous vehicle planning/control architectures. For example, a human standing in front of a car is doomed for collision if the human moves towards the car and the car continues forward. But, a human standing in front of a car isn’t doomed for collision if the car reverses and the human runs away. This implies that the distribution over “dangerous” initial pedestrian locations is conditioned on the human policy as well as the autonomous vehicle policy, i.e., mu(S, O, D, P, pi_human(s, g), pi_AV(s)) where pi_human is the human’s policy and pi_AV is the autonomous vehicle’s planner/policy. The justification behind why decoupling the pedestrian behavior and the dangerous initial locations is a reasonable modelling assumption should be improved to increase the overall quality / clarity of the technical contribution. 

Section 3:
- “...pi is constant or independent of E[I].” This sentence is missing a period at the end. 
- “If the pedestrian behavior model \pi … then the third equation will maximize…” There are only two equations stated in Section 3, so it is unclear what third equation this sentence is referring to. 
- p(s^\rho_t) is the goal-driven car model, but it does not appear to be goal-dependent like the pedestrian model. Since an initial pedestrian state’s safety depends not only on where the pedestrian is trying to move but also where the AV is trying to move, it appears critical that the car’s model also captures how the AV behaves differently as a function of it’s own goals. Furthermore, it appears that the car model only depends on the state of the ego vehicle, s^p_t. Thus, it is not possible for the autonomous vehicle to react to / predict the pedestrian since it never has access to the pedestrian’s state, which is an unrealistic modelling assumption of an AV autonomy stack. 
- The external world dynamics are denoted by p(s_t+1 | s_t) but this appears to be an autonomous system without any input. Since the text states that these dynamics also include the AV and the pedestrian and any other traffic participants, then it must also include the actions of all other agents too: i.e., p(s_t+1 | s_t, a_pedestrian, a_AV, a_other_traffic_participants).
- There is a notational inconsistency in equation (3) and (4) when referring to a^p_t and a^c_t -- I believe these should be a^pi and a^\rho respectively.  
- In equation (4) is the policy pi(a^p_t | s_t) the same pedestrian behavior model pi(s^pi, g^pi)? The notational inconsistency makes it unclear to understand what this part of the equation represents. A similar question can be asked of the quantity p(a^c_t | s_t). 

Section 3.3:
- “The car model is intentionally simple… The car is a policy gradient model…” Since the safety of a pedestrian’s initial condition and the car’s policy are inextricably intertwined, the safety assessment of the initial conditions only holds under an appropriate choice of the car model. Realistic autonomous vehicle stacks are incredibly complicated. To assess the impact / utility of the proposed framework, a more realistic car model should be discussed or, ideally, evaluated.

Section 3.4: 
- “It should be noted that the CARLA … is trained to avoid collisions on a dataset where cars have much lower average speed…. This provides sufficient constraints on the collision avoidance abilities of \pi..” This claim is not well justified. Since there is a model mismatch between the simulated human’s policy with respect to other cars, then the human policy may not appear as collision-avoiding as claimed. For example, if the human policy only had to avoid slow-moving cars, then the human policy may not run out of the way of an on-coming faster vehicle, the way a truly cooperative collision-avoiding agent would. This is a confounding variable in the setup of the simulator, and should be either removed as a confound (by using an appropriate pedestrian policy that can truly react to the car in a cooperative way) or it’s effects in the construction of distribution over unsafe initial states should be further discussed / controlled for.

Section 4:
- There is a typo in Table 1 -- OP should be PO. 
- The text references a distribution P\mu-D but this is not described in Table 1. Perhaps this is a typo on the far-right distribution P-D?
- “... m ~ Po(2)” does Po refer to probability distribution? And what does the argument of 2 mean? Please clarify. 
Section 5:
- Typo “mdoel” should be “model” and “coo-operative” should be “cooperative”.  

summary_of_recommendation: The problem of interest in this paper is very important. However, as currently written, the proposed approach and results are too preliminary and difficult to assess. This paper would be better contextualized as a Blue Sky paper focused on (1) clearly outlining what it takes to do “good” simulation / stress testing of safety-critical scenarios and (2) reworking the proposed approach as a first step towards the simulated, automated stress testing of AV safety systems. Clearly describing (1) is something that is still lacking in the literature, and could be a contribution on its own; after this, concrete instantiations of AV safety testing frameworks -- like perhaps the one proposed -- can be better contextualized. 

================
After the discussion period, I have decided to increase my score to "weak accept". 
issues: - Please provide further justification for the proposed idea of decoupling the initial states from the human behavior policy (and the autonomous vehicle model). See the detailed comments above. 
- Please clarify the writing and notational consistency so the proposed method can be more clearly understood.
review_title: null

Meta-review: Summary: This work proposes formulating the problem of safety validation for autonomous vehicles as generating appropriate initial positions for agents to compose difficult, potentially adversarial, scenes. It proposes using a reinforcement learning approach to learn a distribution for the initial position of a pedestrian that maximizes collisions to create challenging scenarios for an autonomous vehicle. The proposed method is validated within the CARLA simulation environment. 

Clarity: The paper clearly presents an important and under-examined problem and all reviewers seemed to understand the proposed approach after reading the full paper. However, all reviewers suggest ways the paper could be reorganized to more quickly help readers grasp the core ideas. 

Significance/Originality: Reviewers had mixed feedback on the impact and originality of the work. It seems like some reviewers were bought into the problem formulation whereas others wanted to see a more experimental basis. 

Quality: All reviewers agreed that the paper has room for improvement in strengthening its premises (see “opportunities for improvement” below for more details). 

Pros: 
The paper addresses a highly critical, interesting, and under-addressed area in robotics. It was clear that all reviewers agreed this problem space is extremely important and of high interest. The paper also presents a framing of the problem that is interesting and likely to spark new discussions. 

Main opportunities for improvement: 
It seems like the paper has three main components - 1. a proposed framing of the safety validation problem as generating initial positions for simulated agents, 2. a decision to use RL to solve this problem, and 3. experimental results with a single pedestrian in a scene. 

A major contribution of the paper is that 1 is creative and somewhat unexpected, making this paper rather unique. However, at the same time, the paper has a higher burden of convincing others to adopt that same view of the validation problem. The paper would benefit from strengthening this - specifically, consider reviewer UAuu’s feedback on how to make the “point” of the paper more crisp and upfront and reviewer s5dQ’s feedback on assumptions that need to be clarified and better supported. Reviewers 7Zap and s5dQ also both comment that the results presented involve significant simplifications (e.g. single agent, simple car model). It would be worth paying special attention to clearly delineating the scope of the problem this work attempts to solve and why that problem is “big enough”. 

Regarding 2, reviewers 7Zap and s5dQ question why the proposed RL approach is needed to solve the problem given the simplifying assumptions mentioned above formulate a problem that is arguably solvable by non-ML methods. Given that using ML is a focal point for CoRL, it is necessary to clarify this point in the rebuttal. 

Finally, reviewers expressed uncertainty about whether or not the technical work is large enough to be a publishable unit or is a still ultimately a “first step”. I would recommend that the rebuttal also spend time justifying why the scope of the paper represents a complete unit of research. 

Thank you for considering our feedback, and we look forward to seeing the updated paper. 

============= Final Decision

Thank you to the authors and reviewers for the extensive and productive discussion. After discussion, the authors have agreed to incorporate key parts of the reviewers' feedback, in particular reviewer s5dQ's concerns, in the final version of the paper, and all reviewers have achieved consensus that the paper would be a useful contribution to the conference. That being said, I think reviewer s5dQ's comments about the paper's shortcomings are valid and accepting as a poster is more appropriate. 
Decision: Accept (Poster)