Transcendental Idealism of Planner: Evaluating Perception from Planning Perspective for Autonomous Driving
Authors: Weixin Li
Xiaodong Yang
Venue: Submitted to ICLR 2023 (2023)
Keywords: Autonomous Driving
Utility Maximisation
Hilbert Space
Planning
Perception

Abstract: Evaluating the performance of perception module in autonomous driving is one of the most critical tasks in developing these complex intelligent systems. While module-level unit test methodologies adopted from traditional computer vision tasks are viable to a certain extent, it still remains far less explored to evaluate how changes in a perception module can impact the planning of an autonomous vehicle in a consistent and holistic manner. In this work, we propose a principled framework that provides a coherent and systematic understanding of how perception modules affect the planning of an autonomous vehicle that actually controls the vehicle. Specifically, planning of an autonomous vehicle is formulated as an expected utility maximisation problem, where all input signals from upstream modules jointly provide a world state description, and the planner aims to find the optimal action to execute by finding the solution to maximise the expected utility determined by both the world state and the action. We show that, under some mild conditions, the objective function can be represented as an inner product between the world state description and the utility function in a Hilbert space. This geometric interpretation enables a novel way to formulate, analyse and evaluate the impact of noise in world state estimation on the solution to the problem, and leads to a universal quantitative metric for such purpose. The whole framework resembles the idea of transcendental idealism in the classical philosophy literature, which gives the name to our approach.

Under review as a conference paper at ICLR 2023
TRANSCENDENTAL IDEALISM OF PLANNER: EVALUATING PERCEPTION FROM THE PLANNING PERSPECTIVE FOR AUTONOMOUS DRIVING
Anonymous authors Paper under double-blind review
ABSTRACT
Evaluating the performance of perception module in autonomous driving is one of the most critical tasks in developing these complex intelligent systems. While module- unit test methodologies adopted from traditional computer vision tasks are viable certain extent, still remains far less explored evaluate how changes in a perception module can impact the planning of an autonomous vehicle in a consistent and holistic manner. this work, we propose principled framework that provides a coherent and systematic understanding of how perception modules affect the planning of an autonomous vehicle that actually controls the vehicle. Specifically, planning of an autonomous vehicle is formulated maximisation problem, where all input signals from upstream modules jointly provide world state description, and the planner aims find the optimal action to execute by finding the solution to maximise the expected utility determined by both the world state and the action. We show that, under some mild conditions, the objective function can be represented as an inner product between the world state description and the utility function in a Hilbert space. This geometric interpretation enables novel way to analyse the impact of noise in world state estimation on the solution the problem, and leads universal quantitative metrie for such purpose. The whole framework resembles the idea of transcendental idealism the classical philosophy literature, which gives the name to our approach.
expected utility
INTRODUCTION
Autonomous driving has recently risen as a fast-advancing realm in both industry and academia, and receives surge of interest from engincering and scientific communities (Yurtsever al. 2020 Sun
et al. 2020). As an intricate system, an autonomous driving vehicle consists of numerous hardware components and interactive onboard modules. As one such component, the onboard perception module serves as the major source of real-time characterisation of the dynamic environment an autonomous vehicle (AV) navigates through.
To evaluate and improve the perception module, conventional perception tasks (such as detection, segmentation, tracking) have been well defined and corresponding performance mcasurements are established in computer vision to benchmark performance of perception algorithms (Lin et al. 2014). Despite their great success in driving the development of advanced perceptual information processing modules, almost all such metrics exclusively focus on the perception-levé performance in a deployment-agnoste fashion, for instance, how close detected object is to the ground truth, while ignoring the actual impact of the result to the entire AV system. Indeed, not all perception errors translate the same to the planning of an AV. Obviously, miss detecting a vehicle front of an AV far more serious than one behind far away. This problem is further compounded by the heterogeneity of perception errors that share little semantics in common ("How dose an error of 5m/s in velocity compare that of size 25% larger?"), where intuitive manual engineering is widely used (Caesar let al. 2020). Although these issues are typically addressed by integrating road test in the real world. the process is extremely costly and time- cone-con (Wachenfeld and Winner 2016 Asljung et al. 2017). In result, tools in great demand to effectively and efficiently measure the impact of perception to the whole autonomous driving system before deployment on road. Unfortunately. these solutions still remain far less explored in the research literature.

---

Under review as a conference paper at ICLR 2023
(8)
(b)
(c)
(d)
ground truth behavior based on ground truth perception output: behavior based on perception output
Figure I: Illustration of behaviour change v.s. driving cost (best viewed colour). The change in AV hehaviour due to perception error is not always correlated to the cost of consequence. In (a) the AV has to circumvent the erroneously perceived cone by making a large detour. While for (b) the AV only needs make a slight detour to the right, yet it inevitably hits the cone. In this case, although the behaviour change is far less than that of (a), the consequence is significantly worse ("hitting an object" v.s. "making a large detour"). ) In (c) the consequence of either way is indifferent the AV in moving forward, yet the change in behaviour is considerable in terms of spatiotemporal motion. As for (d), if there are two falsely detected cones on both sides, which are close enough to the AV when passing by despite no collision, the AV still decides to maintain the same motion as in the ground truth case. Therefore, the final behaviour of the AV does not change given the perception error, but the cost of passing by two close objects already changes the planning process, which will be missed by the metrics that only look at the AV behaviour or planning result.
Most recently, the community starts to approach this problem with some initial efforts (Sun et al.) 2020: Philion et al. 2020; [vanovic and Pavone 2021, Deng et al. 2021). Despite some success, these preliminary solutions only exploit certain aspects of the problem, either implicitly relying on weak correlation between behaviour change and driving cost (Philion et al.) 2020). inferring the holistic cost via local properties (Ivanovic and Pavone 2021). or coarse levels (Sun et al., 2020). In this work, we propose principled and universal framework to quantify how noise in perception input affects the AV planning. This is achieved by explicitly analysing the process of AV planning, in the context of expected utility maximisation (Osborne and Rubinstein 1994), and evaluating the change of utility values critical to the AV reasoning subject to input perception errors. Under some mild conditions (Section 3.3 we show that this planning process can be formulated as an optimisation problem with linear objective function in Hilbert space, where utility to optimise the inner product of an action-wise utility function and the world state distribution represented by perception. This geometric interpretation reveals many natural and insightful properties of the problem, for example, any input error can be decomposed into two parts: one that does not affect the utility comparison plaaning-invvaian error) and the other one that directly changes the planning problem (planning-critical error). Based on this novel insight, we derive a metric that quantify how perception error changes the planning process.
We want to emphasise the necessity of understanding impacts of perception errors on an autonomous driving system via the process of planning. rather than purely from the final result (i.c., the AV behaviour, or the trajectory output from the planning module), as proposed by previous works (Philion al. 2020). This results from the fact that, the final planning result does not necessarily reflect how evaluate the situation, reason with the environment, and assess the costs of actions. In fact, the correlation between behaviour change and the actual consequence weak, or even negative in many common cases, as illustrated in Figure T Actually, most works implicitly or explicitly integrate some priori knowledge of consequences of perception errors into metric design. The complexity of such impact on autonomous driving, however, is far beyond hand-crafted rules, defeating their purposes despite tremendous amounts of manual efforts, e.g., Deng et al.) (202T) assumes that severity of an error should be weighted proportional to the reciprocal of its cubed Manhattan distance to the AV, regardless of its position relative to the AV (in front or behind the AV). contrast, we make little such presumption and fully rely on the planning process to infer the error consequence in fully transparent way, which enables our solution to capture many critical cases. In this regard, the core principle of our design resembles the idea in the philosophical system of transcendental idealism, proposed by Immanuel Kant in his classical work Critique of Pure Reason Kant 1998). which argues
e.g..Degg
2

---

Under review as a conference paper at ICLR 2023
that, due to the limitation of the observer's sensibility, cognition of external objects is processed never as they are in themselves, but via the cognitive faculties and subject to the interpretation the observer's experience. For the same reason, the consequence of environment mistepresentation for an AV due to perception errors is naturally reflected via the change in its planning (the core component of AV that interprets its environment) and measured by the extra loss incurred, which gives the name to our framework: transcendental idealism of planner (TIP).
LITERATURE REVIEW
Planning for Autonomous Vehicles. We consider the behavioural decision and motion planning as the planning process, which generates the vehicle behaviour execute by the controller given the observation up to the planning time. There is a rich literature to address these fundamental problems in autonomous driving, which can be roughly categorised into canonical module-based and data- driven methods. The former relies on explicit modelling of target accomplishment in optimisation frameworks and seeks the optimal solution as the result (Schwarting al., 2018). The latter, on the other hand, aims to directly map raw sensor data into the AV behaviour or final vehicle control signals by leveraging the approximation power of deep learning and massive data (Bojarski et al. 2016, Grigorescu et al. 22000), which has attracted increasing attention recently. In this work, we aim to explore the internal mechanism of a planner to gain some insights of the impacts on planning from perception noise, and specifically focus the module-based planning with explicit target achieving process. Extension of results this work to data-driven planning is left for future work.
Behaviour based Metrics for Upstream Modules. Recent works aimed to assess the performance of perception from the autonomous driving system viewpoint mostly approach the problem in heuristic ways. Considering black-box planning models, Philion et al. (2020) implicitly hypothesise that driving consequences of perception errors are directly correlated to the change in an AV spatiotemporal trajectories planned, and propose the planning KL-divergence (PKL) measure the impact. While intuitive, it fails to incorporate the context of environment and does not preciscly reflect the real cost of input noise in many common traffic scenarios. To deal with the specific problem of object representation, [eng et al. (021) study how object shapes can affect autonomous driving and devise the support distance error (SDE) to quantify such effect. In a very recent work, Ivanovic and Pavone (2021) start looking into the planning process and employs sensitivity as probe of input signal's contribution to AV behaviour. This, however, implicitly leverages local-only properties of differentiable cost functions to infer global results. In comparison, our proposed approach captures the big picture of planning process and applies to far more general cases.
AV PLANNING AS EXPECTED UTILITY MAXIMISATION
To evaluate the performance of perception module from the AV planning perspective in a principled manner, start by introducing the preliminary basics, and then review the expected utility maximi- sation (EUM) as the optimal AV action framework. After that, the interpretation of EUM in a Hilbert space is presented, based on which our metric for perception is derived.
3.1 PRELIMINARY
We first present the mathematical basics to facilitate the following theoretical analysis. Unless otherwise specified explicitly, all notations follow the standardised one in Goodfellow al. (2016). A probability space (.FF is defined by a sample space T. an event space F (a o-algebra on b), and Borel probability measure P on F. A random variable X & + R (d E N) is induced from (2,FP)) PP with distribution function Fx (x). When absolutely continuous, Fx (x) = f fx (t) dt, where (x) is the probability density function (PDF). T.2 denotes the space of square- integrable functions, and is a Lebesgue measure accordingly. A Hilbert space H = (T, (: 1) is defined complete space T with inner- inner-product ( and induced norm norm|l.llze || Let S H be subspace of Hilbert space H. S- {x E H (ır, v) Vyy 5) is the orthogonal complement of S (i.e., the set of all vectors orthogonal to S). The linear span of a set S is span(S). ny == u/|o|| is an element of unit length normed vector space by normalising element v.
3.2 AUTONOMOUS VEHICLES AS RATIONAL AGENTS
An AV is an intelligent agent that aims to accomplish some predefined goals in an interactive and uncertain environment. For this, AV is constantly faced with the problem of planning in the
3

---

Under review as a conference paper at ICLR 2023
dynamic environment, and the quality of planning determines how well the goals can be achieved. By the classical EUM theory (Osborne and Rubinsteim, 1994). at any given time t, an agent aims to achieve the maximum expected reward, defined by the utility function U, via execution of the optimal action of such that
a arg ae E [U.
(1)
where Da, is the set of all feasible AV actions at time t; E S, random variable with distribution function Fs, (s), is the world state at time in the world state space S; and
EU a) = E [U(S,, a)] = Sees
(2)
Intuitively, the utility function encodes the goal or reward the AV is supposed to achieve, for example, to reach destination in time, to minimise likelihood of collision with other objects, and to avoid sharp change in motion, and Fs,(s, (s) captures uncertainty about the stochastic environment given all prior world knowledge and historical observations up to t, which are estimated by modules like localisation and perception. Architectures of many modern AV planners still follow this classical framework (Paden et al. 2016 Buchler et al. 2009 Fan et al. 2018)
3.3 EXPECTED UTILITY MAXIMISATION IN THE HILBERT SPACE
To gain some insights into the expected utility of ① and how input noise is consumed by the planning process, we introduce an interpretation in the Hilbert space to leverage geometric tools available from linear algebra. We first establish the conditions under which probability measure be embedded into a Hilbert space, followed by the interpretation of EUM from a geometric perspective in Section 4 For brevity, all proofs are left in Appendix G
Theorem 1 (Probability Measure Embeddings in Hilbert Space). Let {X, d be compact metric space with d as the metric function, p be a Borel probability measure on X, and X be random variable on X with distribution function Fx (z). If Fx (z) is absolutely continuous and the density function fx square mmuareiintegralle i.e., fx L2 then there exists a unique element Up H such that
Ex [o(z)] (pp:9) Vg E H.
(3)
where element Pp denotes the embedding of probability measure p in the Hilbert space H (I2, (,,)),, with the inner product given by
(g = J2 (()d)))))(())).
The critical condition of Fx (xx) being absolutely continuous with a -quare-integrable density function fx in Theorem D actually general and includes many popular distributions as special cases (see the discussion in Appendix F.. Theorem establishes mapping from probability measures of continuous random variables to H. Additionally, the mapping is also injective by the following result.
Theorem 2 (Injection of Probability Measure Embeddings). Let p and q be two Borel probability measures defined on a compact metric space {X, with absolutely continuous distribution functions, then p q almost everywhere if and only if Pbp Pg. where Pa and Pa are the embeddings of p and in H. rexpectively.
similar result for mixed distributions also available in the appendix (Theorem 4 Under the mild conditions in the aforementioned results, the expected utility maximisation of I can be rewritten as
a' = arg max Ex-p(s) [Uss, a)] arg max Ua) H. (s, H. (5)
Given the injective correspondence between p and Mp established above, we can leverage many tools in algebra (such as inner product, orthogonality, projection, and subspace) to analyse the impact of perception result on AV planning via the EUM in H. denoted "planning utility Hilbert space", where the topological structure is exclusively determined by its inner product.
4 PERCEPTION EVALUATION VIA AV PLANNING
In this section, we derive the extra cost of planning incurred by perception errors through the theoretical foundation established in Section 3 While the actual world state characterisation p(s) consists of signals from modules other than perception (e.g., localisation), for brevity we assume that the perception module is the only source for world state estimation in the following discussion.
Referred unique class of functions that are equal almost everywhere.
4

---

Under review as a conference paper at ICLR 2023
A
E(g)
((p)
P
Arl
Ha
A
T
U
H Va.
(b)
Figure (a) Illustration of EUM H. AU UA_ defines the behaviour direction; & represents the preference score; MP and My are the embeddings of the ground truth and noisy perception result, respectively; Au is the perception error, which is decomposed into the planning-crin meniing-rritical error (PCE) Apy. and the planning-ine invariaat error (PIE) Ap ; and the shaded area corresponds to Ha. (b) A toy example of PCE and PIE (best viewed colour). The AV is moving forward on a road width 6m, in front, the belief of its position is distribution
where there is axis). The ground truth distribution is 21- a uniform distribution with support [-3, - while the perception believes its location (distributed q) is 1 The AV has two action options: 1) to move forward (a', the red arrow), and the utility function is U1 (z) -10 1=e[-1,11 with heing the position of the cone (only large loss for collision with the cone); 2) to make a hard brake (a, the grey arrow), and the utility function regardless of the cone position). In this case, the AU and Ay are illustrated in the top right, while the decomposition of PIE Au and PCE Auy are on the bottom right. Note that, Auj is of the same shape AU (up negative constant), and (AU, Ap =0
line across the road (the
constant U2
-5 (loss of hard braking is identical
4.1 BREAKDOWN OF PERCEPTION ERRORS
Consider a general case where the candidate action set is D, {a;}, and each action is associated with distinct utility function Uss, 0:) E such that
[|U(s, a:) - U(s a,) H 0 + az # aj: Va,, aj E D..
Let a" be the optimal action per EUM of (5) given the ground truth distribution of world state p(s). For specific f a', (a" ,q) = U.. - Ua. and the planning half-space in H. is
Ha (fl AU(a" ,a) 0, H).
(6)
Given the actual perception result (()), a" is preferred over by EUM if and only if na E Ha. i.e.,
E(g:a", (Mg. (a", a)) EU(g. EU a) > 0,
with E(g (3) denoting the a-3 preference score given which exclusively decides the result of EUM. As illustrated Figure E(a). a). the final planning is made correctly if and only if
Mg (aEDa/(aa)) Ha.
(8)
When there is an error in perception q(s) (i.c., || - 0), the preference of may be affected, i.e., (gg a" a) + E(p; a", a), so the preference between a" and by EUM. To understand how the difference Aμ 1y Pp affects the result of EUM, further decompose A into two
orthogonal components:
A My 14p
Au1 AHLL
where
Au1 = (A,,, nAU) = (AHAUU) AU TAUR
(10)
is the projection of Ap onto unit vector nau (denoted behaviour direction) parallel to U.. - Un. and
AuL span (AUU))
(11)
5

---

Under review as a conference paper at ICLR 2023
Algorithm 1: TIP Score Evaluation
Input :An observation sequence from perception (({s))==-.) and its ground truth p (sx }o= Output: score of g( (((+)]m-.))
Get the candidate trajectory set Da.p, and the optimal action a" E Da.p from the planner with the ground truth perception input (({=))==-))
Get the candidate trajectory set Da.q from the planner with the noisy perception input (({=))==-r)
Define the reference action set Da == Da.p U Da..
foreach E Dr do
Compute four estimates via finite-size samples: n. (t), ,,"), (a)), E = - a)) EUs -6 .a),
EU = - U(44)
EU, 3
Σ U(s)),a)
where {s4 Fi=1 { are n wid. observations from ((((s)i==+))) and similar for {sp Ji=1
10,
Compute Ae(a" a.q.p) EU EU2- - EU3 + EU,
end
Compute and output the result I flq.p; U, Da) = min.eP, AE(a", a;g.p)
is the projection of Ap onto the orthogonal complement of the subspace spanned by the behaviour direction. In the presence of perception error A,u, as illustrated in Figure a). the change in preference score of (7) is only determined by Apy:
AE(a", 0; p) - E(g; , a) E(pp; a", a) = (Ap, AU) = (Aul + AHL, AU) (A AU). (12)
For this reason, we denote Auj as the planning-critice invariant error (PIE). This observation reveals a pivotal fact: not all errors in world state estimation
error (PCE), and Dμ1 as the planning-
or perception are of equivalent impact on the AV planning, and the subspace span( { contains all errors that would not affect EUM at all. A toy example of PCE and PIE is shown in Figure b)).
Given this interpretation, we define the maximum reduction in the preference score of a" over any candidate actions as the measure of impact from the perception error on AV planning:
I (q, p; U,D.J = min.eD, Ag(a", a; q.p..
(13)
4.2 EVALUATION OF PERCEPTION ERROR IMPACT BY TIP
In practice, combining 7 and 12). the evaluation of perception error impact can be reduced to the calculation of four expected utilities:
Ag(a", aa; q.P.) = (e))) [U(s, a")] Ep(e) [U(s, ad]] Egto) [U(s, a)] + Ep(s) [U(s, a)] (14)
Computing these expectations in analytical forms typically requires strong assumptions on the forms of both utility and distribution functions, which substantially limits the flexibility and representation capacity. To allow for maximum representation flexibility, we resort to numerical methods and estimate the expected utilities from finite-size samples of world states, and show that the solution is both statistically consistent and uniformly efficient under the mild conditions in Theorem 3 Specifically, for a fixed action a, given an i.i.d. sample of the utilities (U(S, })} with S; drawn from the state distribution ps(s), unbiased estimator of the expected utility based on U-statistics is
EU, = E
(15)
Under many common practical conditions, we show that fast convergence rate via uniform bound of the estimator of (15) can be achieved by the following observation.
Theorem 3 (Exponential Convergence Rate). If there exists an M E IR such that U(S, a) < M almost surely, then
Pr( (U >s) < 2e Ws > 0, L min (M² Var(U(S, + Me/3).
(16)
The exponential convergence rate at O provided by Theorem 3 is significant in the sense that depends on (i) neither the dimensionality of the original state space S (i.e.. the curse of dimensionality not invoked), nor (ii) the distribution (s()) and utility functions (i.e., USS a) and (s(s) can take any arbitrary forms). To facilitate the understanding of our approach, the pseudo code for evaluating TIP is provided in Algorithm[] which sketches the basic routine to compute TIP score of a perception input sequence o( (sx 10 )0=-)) from = 1 = 0 for planning /. = 0.

---

Under review as a conference paper at ICLR 2023
NOS SDE SDE-APD
0.
0.2
NDS SDD-APO
TIP/PKL
False positive
missed deteection ralle
location [m]
NLS
PAL (AV-))
TIP (43-11
-40
location detection at the axis tm)
0.3
O5 0.3
0.
Figure 4: Metrics for AVs of different driving styles. A miss detected stationary obstacle is placed in front of AV at 2=0 moving at 14m/s along the + direction. AV-1 ("jerk-averse") is optimised for driving comfort with braking capped -4m/s", while AV-2 ("collision- averse") is for safety and can brake as much as -6m/s". Both SDE-APD and PKL consider closer miss detec- tion is worse than further ones. TIP, however, reveals that AV-1 regards one at the worst, since the collision is inevitable even the obstacles at 20m and 25m are detected; yet miss detection at 30m leads to collision that could have been avoided otherwise. In contrast, other metrics provide this fine resolution at this level. When miss detection happens behind the AV, both TIP and PKL ignore its impact unlike the other two. Note the symmetry of NDS and SED-API SED-APD in both directions of the x-axis.
0.
02 04
NDS
ADE-APD
PKL
TIP
yow Volocity ertor (m/s)
omor sta (m)
Figure Comparison of metrics on different cases of synthetic noise. The left vertical axes are for NDS and SDE-APD, and the right are for PKL and TIP. Note that, NDS saturates in several cases once the noise reaches some levels, and sensitivity of TIP to various types and levels of noise is generally more consistent than the other three (SDE-APD com- puted by 1RE-APD@t=ls for velocity noise).
5 EMPIRICAL STUDY
In this section, evaluate how the proposed TIP works in practice via extensive experiments both synthetic and real data. Due to the space constraint, more details are left in the appendix.
5.1 BASIC SETTINGS
All AVs used in the experiment are based on the same type regular passenger vehicles. The planner deployed in the autonomous driving system is derived from the popular open-sourced project of Apollo (Fan et al., 2018). It consists various sub-modules of routing, object motion prediction, cost generation. path finder, and trajectory optimisation. At each planning time instant, these sub- modules (except the trajectory optimisation) analyse the environment, input history. and establish the target utility function for final trajectory optimisation, i.e., the objective utility function U(; s) is first created with the perception input as (part of) syper-parameters, The path finder then provides multiple initial paths as candidates for path-wise trajectory optimisation, and the final choice is determined by utility decider. The goals the planner strives to achieve include motion smoothness, traffic rule compliance, safety, progress to the destination, etc. The planner has been extensively verified via rigorous road tests in major cities with millions of population (See Appendix B for more details).
experiments are implemented in scenarios as the standard protocol in autonomous driving (Ried) maier et al. 2020). Scenarios used are collected from real world road tests (see more details Appendix Dj. We consider the planning problem particular frame scenario time, and evaluate the utility of an action (a spatiotemporal trajectory the AV executes) for the next three seconds, following the basic setup (Philion et al., 2020). For comparison, three baselines are adopted from the spectrum of perception metrics: 1) at the conventional end, nuScenes dataset score (NDS) (Caesar et al.) (2220) combines several traditional scoring results for 3D object detection into a single performance measure, SDE average precision distance-weighted (SDE-APD) (Deng) et al. 2021) focuses more on support distance errors near the AV in an ego-centric fashion, and 3) PKL (Philion et al. 2020) serves the representative for AV behaviour-driven metries.
5.2
RESULTS ON SYNTHETIC DATA
In the first set of experiments, aim gain understanding of various metrics in reaction to common types of perception noises. A dataset is synthesised from our curated road test scenarios by adding controlled noise to the 3DD object ground truth of vehicles, to enable clear observation
7

---

Under review as a conference paper at ICLR 2023
-100
-50
NDS SDE-APD
-10
TIP -115.42
TiP PKL
PKL .0.002
-15
PKL
155 1200
0.7
-20
ND511) SDF-APD(1)
PIL
15
training epoch
TIP
Figure 5: Comparison of metrics on real data (best viewed colour). Left: Metrics on different checkpoints during training. Middle: Scatter plot of impacts of perception noise measured by TIP and PKI. Note the number of data points close to the z-axis (PKI.= 0), which correspond to critical errors in planning due the perception noise captured by TIP yet missed by PKL. since the AV behaviours are similar with and without the noi sc. Right: First one is the ground truth with corresponding AV behaviour (the spatio-temporal trajectory represented by the green tube with the z-axis the temporal dimension); second one shows that a false positive (pointed to by the red arrow) causes an outrageous planning error with a jerk of - while the typical limit is around -1.0m/s (Wang al. 2018), despite a mild change in behaviour by PKL. PKL and TIP of this case are highlighted by the red circle in the seatter plot (the middle figure). See details Figure Saneeaand and Figure of Appendix C.
the sensitivity of metrics to specific types of perception errors. For this, 1000 Ss-long scenarios are assembled, with the number of objects per scene between 30 and 500, where AVs are moving on an average speed at least 5m/s. The ground truth is annotated by professionally trained human operators. All objects in the scenario are labeled with location, heading, category, and bounding box from 3D point clouds recorded from LiDARs on AVs during road tests.
In total, six types of errors are considered. The false positives are tested by adding "ghost" vehicles scattered within a 70m-by-30m box centred at the AV, with motion properties randomly perturbed from the AV. The miss detection is created by removing objects from the ground truth randomly with a certain probability (i.e., miss detection rate). Other noises involving location, yaw, velocity and size are sampled from zero-mean Gaussian distributions with different variances and added to corresponding properties of objects in ground truth. The comparisons are shown in Figure 3 While all metrics are negatively correlated with all six types of perception noises, NDS saturates in some types (e.g., velocity) due to its design. SDE-APD, also involving manual engineering, exhibits varying sensitivity at different noise levels (especially for the velocity, as the default matching threshold 0.2m is easily overwhelmed by speed noise larger than Im/s. Selectivity of TIP tends to be more consistent than PKI., the sense that, while both may predict results at similar dynamic ranges when the noise is mild, the former indicates larger loss when input perception errors intensify in most cases.
We further investigate behaviour of TIP in some individual cases. In a typical miss detection scenario, we remove stationary vehicle in front of or behind moving AVs. As shown in Figure 4 outcomes rendered by TIP change with different planner settings, and it predicts the miss detection of the worst loss at the border of collision events when the accdident could have been avoided by small margin if the obstacle is successfully detected. This reveals the superior resolution of TIP in identifying critical events from the planning perspective that would have been missed by other baselines. The case also shows that NDS and SDE-APD fail to distinguish errors at both sides of the AV, due to their spatial or directional homogeneity by design.
5.3 RESULTS ON REAL DATA
In the second set of experiments, we study the results from the real perception module deployed on our AVs, which exemplified by a 3D object detection model that predicts class, location, heading, velocity and size of objects from LiDAR point clouds. TIP is independent of the specific detector and can be applied to various methods (Lang et al. 2019 Yin et al. 2021 Shi et al., (020). We adopt an end-to-end multi-view fusion (MVF) based model to synergise the birds-eye and perspective views of point clouds (Zhou et al. 2019). The model is trained on 780K LiDAR sweeps using annotations of vehicle, pedestrian and cyclist with detection range [-67.2m, 124.8m] x [-51.2mm 51.2m].
typical challenge in developing the perception model is to determine how much training is needed to reach satisfactory level of performance. Conventional solutions require variety of heterogeneous
8

---

Under review as a conference paper at ICLR 2023
Table 1:
Comparison of different detectors across metrics (1).
Table 2:
Subjective evaluation.
Detector
NDS
SDE-APD
PKI.
TIP
Metric
v.s. TTP
MVF+ + Tracking
0.730
0.843
-8.1
-10.5
NDS
17.6% / 82.4%
MVF (PN-T)
0.693
0.852
-9.2
-11.7
SDE-APD
34.5% / 65.5%
5F-MVF (PN-T)
0.744
0.878
-7.9
-9.1
PKL
38.8% / 61.2%
metrics to measure different aspects of the algorithm, including mean average precision (mAP) for detection and mean squared errors predicted motion properties. Recently, some unified metries like NDS (Caesar et al., 2020) are also proposed by manual engineering. These metrics hardly predict outcome of the driving quality improvement of a perception model change, and in most cases the conclusion can only be made from large-scale real road tests, which is almost infeasible for such purpose Waachenfeld and Winner, 2016 Asljung et al. 2017).
We evaluate the performance of our 3D object detection model on the same benchmark as in Sec- tion 5.2 and compare the model output against the ground truth. The model is trained for 15 epochs, and results are reported in the left of Figure Not surprisingly, NDS tends to increase as the model training progresses and the final checkpoint models usually achieve the best performance since NDS combines the errors that are aligned with the loss functions optimised during training. When evaluated with the AV involved, however, the observation is not quite similar. SDE-APD implies that the training, without the AV context, seems to struggle with improving results on close-by objects as the losses are dominated by large number of far-away yet more challenging objects. From either behaviour or planning perspectives, both TIP and PKL indicate that the last checkpoint model is not among the best possible models during training. Instead, models somewhere in the middle of the training can provide better autonomous driving performance. Actually, neither TIP nor PKL is improved significantly beyond the 7th epoch, suggesting that early termination of training may be even more beneficial to driving quality. More importantly, we notice that TIP disagree with PKL on scenarios across models of top performance, where there are quite some critical cases identified by TIP yet missed by PKL. The difference is illustrated in the middle of Figure 5 by the scatter plot of randomly sampled scenarios, where the PKL values are almost zero while the TIP scores non-trivial for quite a number of cases, suggesting the drastic impact of the perception errors on the AV planning process despite similar AV behaviour outputs with or without these errors (related individual examples are discussed in Appendix C.I).
To compare other 3D detectors for offline applications (e.g.. auto labelling (Qi et al. 2021), I), we implement two offline models with PillarNet (Shi et al. 2002) enhanced with transformer modules as the basic detector. The first one, denoted MVF (PN-T), uses the point cloud only from the current frame for prediction. The second one, denoted 5F-MVF (PN-T), leverages consecutive frames around the current one to predict. Results are reported in Table Both offline models, with far less restriction on resources, have better performance by SDE-APD. MVF (PN-T), however, cannot produce precise velocity out of observation from only one frame, which leads to inferior performance by other three metrics (MVF + tracking is the onboard model discussed above). 5F-MVF (PN-T) delivers overall best results across all metrics, despite the marginal gap computed by PKL.
To further justify the soundness of the proposed approach on the scenario level, we also implement a set of subjective evaluation similar to that in (Philion et al. 2020). We collect 258 pairs of scenarios with actual perception noises and check weather TIP, PKL or NDS disagree on the relative severity (i.e., one believes the perception error in scenario A is worse than that in scenario B while the other one thinks alternatively). These scenario pairs are compared and rated by 10 randomly selected human drivers to decide which is worse from the human perspective. The result reported in Table suggests that human drivers side with TIP over other three baselines.
6 CONCLUSION
In this work, we propose a principled framework to evaluate perception from the perspective of planning for autonomous driving. Our approach explicitly exploits the properties of module-based planners and effectively identifies perception noises that may cause large planning change in the context of expected utility maximisation. Extensive experiments on both synthetic and real data confirm that our approach is capable of distinguishing perception errors that would not be separated by conventional metrics those only exclusively focusing on AV behaviours.

---

Under review as a conference paper at ICLR 2023
7 ETHICS STATEMENT
Autonomous driving directly involves interaction with human beings, animals and other assets in the real world. Any performance measure tools for autonomous driving may not identify or cover all possible failure cases of AVs that may lead to negative or even catastrophic consequences. TIP is not an exception, despite the principle to precisely reflect the potential loss planning by design. It is critical that any researchers and practitioners of TIP should still implement standard and comprehensive safety protocols to ensure legitimate compliance to appropriate laws and rules, to minimise the likelihood of any potential negative impacts.
REPRODUCIBILITY STATEMENT
The submission includes three major technical parts: theory, implementation, and empirical study.
The main results of theoretical analysis and treatments presented in Section 3.3 and Section 4.2
The detailed explanation and complete proofs are presented in Appendix [ and Appendix G The computation of TIP is provided as pseudo-code
in Algorithm [T] For the empirical study. the planner is derived from the open-sourced Apollo with configuration fine-tuned on real road test data as depicted in Appendix [ NDS is computed via the open-sourced implementation) SDE-APD) is
implemented in house according to the original paper (Deng et al. (2021). PKL is implemented following the open-sourced project project
REFERENCES
E. Yurtsever, J. Lambert, A. Carballo, and K. Takeda, "A Survey of Autonomous Driving: Common Practices and Emerging Technologies," IEEE Access, 2020.
P. Sun, H. Kretzschmar, X. Dotiwalla, A. Chouard, V. Patnaik, P. Tsui, J. Guo, Y. Zhou, Y. Chai, B. Caine, V. Vasudevan, W. Han, J. Ngiam, H. Zhao, A. Timofeev, S. Ettinger, M. Krivokon, A. Gao, A. Joshi, S. Zhao, S. Cheng, Y. Zhang, J. Shlens, Z. Chen, and D. Anguelov, "Scalabiliity in Perception for Autonomous Driving: Waymo Open Dataset," in CVPR, 2020.
T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollar, and L. Zitnick, "Microsoft COCO: Common Objects in Context," in ECCV, 2014.
H. Caesar, V. Bankiti, A. Lang, S. Vora, V. E. Liong, Q. Xu, A. Krishnan, Y. Pan, G. Baldan, and O. Beijbom, "nuScenes: A Multimodal Dataset for Autonomous Driving," in CVPR, 2020.
W. Wachenfeld and H. Winner, The Release of Autonomous Vehicles. Berlin, Heidelherg: Springer Berlin Heidelberg, 2016, pp. 449.
D. Åsljung, J. Nilsson, and J. Fredriksson, "Using extreme value theory for vehicle level safety validation and implications for autonomous vehicles." IEEE Transactions on Intelligent Vehicles, vol. 2, no. 4, pp. 288-297, 2017.
Philion, A. Kar, and S. Fidler, "Learning to evaluate perception models using planner-centric metrics," in Proceedings the (CVPR), June 2020.
the IEEE/CVF Conference Computer Vision and Pattern Recognition
B. Ivanovic and M. Pavone, "Injecting Planning-Awareness into Prediction and Detection Evaluation," arXiv, 2021.
B. Deng, C. R. Qi. M. Najibi, T. Funkhouser, Y. Zhou, and D. Anguelov, "Revisiting 3d object detection from egocentric perspective," in Advances in Neural Information Processing Systems, A. Beygelzimer, Y. Dauphin, Liang, and J. W. Vaughan, Eds., 2021.
M. Osborne and A. Rubinstein, Course in Game Theory. MIT Press, 1994.
I. Kant, Critique of Pure Reason, ser. The Cambridge Edition of the Works of Immanuel Kant, P. Guyer and A. W. Wood, Eds. Cambridge University Press, 1998.
10

---

Under review as a conference paper at ICLR 2023
W. Schwarting, J. Alonso-Mora and D. Rus, "Planning and poccision-making for autonomous vehicles," Annual Review of Control, Robotics, and Autonomous Systems, vol. 1, no. 1, pp. 187-210, 2018.
M. Bojarski, D. D. Testa, D. Dworakowski, B. Firner, B. Flepp, P. Goyal, L.. D. Jackel, M. Monfort, U. Muller, J. Zhang, X. Zhang, J. Zhao, and K. Zieba, "End to End Learning for Self-Driving Cars," arXiv, 2016.
S. Grigoreseu, B. Trasnea, T. Cocias, and G. Macesanu, "A survey of deep learning techniques for autonomous driving," Journal of Field Robotics, vol. 37. no. 3, pp. 362 362-386, 2020.
I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning- MIT Press, 2016.
B. Paden, M. Cap, Z. Yong, D. Yershov, and E. Frazzoli, "A Survey of Motion Planning and Control Techniques for Self-driving Urban Vehicles," arXiv, 2016.
M. Buchler, K. lagnemma, and S. Singh, The DARPA urban challenge: autonomous vehicles in city traffic. springer, 2009, vol. 56.
H. Fan, F. Zhu, C. Liu, L. Zhang, L. Zhuang, D. Li, W. Zhu, J. Hu, H. Li, and Q. Kong, "Baidu apollo em motion planner," arXiv preprini arXiv: 08048, 2018.
S. Riedmaier, T. Ponn, D. Ludwig, B. Schick, and F. Diermeyer, "Survey on scenario scenario-based safety assessment of automated vehicles," IEEE Access, vol. 8, pp. 87 456-87 477, 2020.
Y. Wang, J.-R. Chardonnet, and F. Merienne, "Speed profile optimization for enhanced passenger comfort: An optimal control approach," 2018 21st International Conference on Intelligent Transportation Systems (ITSC), 2018, pp. 723-728.
A. Lang. S. Vora, H. Caesar, L. Zhou, J. Yang, and O. Beijbom, "PointPillars: Fast Encoders for Object Detection from Point Clouds," in CVPR, 2019.
T. Yin, X. Zhou, and P. Krahenbuhl, "Center-based 3D Object Detection and Tracking," in CVPR, 2021.
S. Shi, C. Guo, L. Jiang, Z. Wang, J. Shi, X. Wang, and H. Li, "PV-RCNN: Point-Voxel Feature Set Abstraction for 3D Object Detection," CVPR, 2020.
Y. Zhou, P. Sun, Y. Zhang, D. Anguelov, J. Gao, T. Ouyang, J. Guo, J. Ngiam, and V. Vasudevan, "End-to-End Multi- Multi-View Fusion for 3D Object Detection in LiDAR Point Clouds," in CoRL, 2019.
C. R. Qi, Y. Zhou, M. Najibi, P. Sun, K. Vo, B. Deng, and D. Anguelov, "Offboard 3D Object Detection from Point Cloud Sequences," arXiv, 2021.
G. Shi, R. Li, and C. Ma, "Pillamet: Real-time and high-performance pillar-based 3d object detection," ECCV, 2022.
Russell and Norvig, Artificial Intelligence: A Modern Approach, 3rd ed. Press. 2009.
USA: Prentice Hall
von Neumann and O. Morgenstern, Theory of Games and Economic Behavior. Princeton, NJ, USA: Princeton University Press, 1944.
A. Berlinet and C. Thomas-Agnan, Reproducing Kernel Hilbert Spaces in Probability and Statistics. Springer US, 2011.
A. Smola, A. Gretton, I.. Song, and B. Schölkopf, "A Hilbert space embedding for distributions," ser. Lecture Notes in Computer Science, 2007, pp. 13-31.
W. Rudin, Principles of Mathematical Analysis, 3rd ed. New York: McGraw-Hill Book Co., 1976.
R. M. Dudley, Real Analysis and Probability, 2nd ed., ser. Cambridge Studies Advanced Mathe- matics. Cambridge University Press, 2002.
K. Chung, Course in Probability Theory. Elsevier Science, 2000.
W. Hoeffding, "Probability inequalities for sums of bounded random variables." Journal of the American Statistical Association, vol. 58, no. 301, pp. 13-30, 1963.
S. N. Bernstein, The Theory of Probabilities.
Moscow: Gastehizdat Publishing House, 1946.
11

---

Under review as a conference paper at ICLR 2023
Appendix
Table of Contents
Additional Literature Review
13
B Autonomous Vehicle Planner
13
C More Comparisons to Related Metrics
14
C.I Comparison with PKL
14
Comparison with IPA
15
D) Scenario Collection
16
E Breakdown of TIP
17
Examples and Non- Non-Examples of Square-Integrable Density Functions
17
Proofs of Theorems in the Main Text
20
Notations
20
Embedding Probability Mcasures H
20
G.3
Injection of Probability Measure Embeddings in H
20
G.4
Approximation of Expectation for Discrete/Mixed Distributions in H
21
G.5
Uniform Convergence Rate of Expected Utility Estimators
22
12

---

Under review as a conference paper at ICLR 2023
A ADDITIONAL LITERATURE REVIEW
Expected Utility Maximisation. Almost all solutions proposed so far in the artificial intelligence realm develop intelligent agents have followed the rational-action strategy as opposed to the human-like behaviour strategy, i.e., the optimal decision made by rational agent should optimise some achievement measurement on consequences of its decision, which is, ideally, well aligned with the predefined high-level goals (Russell and Norvig, 2009). Among many possible formulations of the outcome measurement, the expected utility (EU) hypothesis one of the most popular frameworks in the decision theory (von Neumann and Morgenstern 1944). While first introduced to characterise human behaviours in microeconomics, the idea also finds great success in many other domains, including the optimal decision for artificial intelligence systems (Osborne and Rubinstein 1994).
Embedding Probability Measures in Hilbert Space. Embedding probability measures into a Hilbert space has been explored in the literature for kernel methods (Berlinet and Thomas-Agnan Smola 2 et al. (007). These methodologies exclusively focus on the reproducing kernel Hilbert space, which is spanned by Mercer kernel functions. However, the implicit requirement function continuity may restrict its applicability to our problem, where either utility and probability density functions may be discontinuous, In this work, we consider less restrictive Hilbert space where only square-integrabiiity is needed and more flexible functions are possible.
B AUTONOMOUS VEHICLE PLANNER
We start by introducing the autonomous vehicle planner, which provides the fundamental toolkit for the proposed evaluation framework. Our planner is designed to control an Level 4 AV running in urban areas of major modern cities. Its modulized architecture is similar to many popular utility-based planners (c.g.. (Fan et al. ), which consists four major components as illustrated in Figure 6
The predictor infers the motion information 8m in the futurr (i.e., 0) for all dynamic road objects from perception input history (i.e., /. < 0) up to the planning time (i.e., = 0).
The action proposer analyses the current environment at the planning time from (1) the perception input, (2) future object motion input, and (3) other inputs (e.g.. localisation. traffic lights, semantic maps, routing path, etc.), and proposes various sets of behaviours (e.g.. "go straight" and "lane change") for the AV with an initial feasible spatiotemporal trajectory for each set.
The trajectory optimiser takes results of above components as input, and finds the optimal spatiotemporal trajectory for each behaviour set by numerically solving an optimisation problem with the initial feasible trajectory from the proposer as the starting point.
The optimal trajectories from all behaviour sets are then submitted to the action decider, which assemblies all information to evaluate the utilities of different candidate actions (with corresponding optimal spatiotemporal trajectories), and makes the final decision on the a".
Similar to many popular architectures (c.g., Fan et al. 2018 our planner utility function U(a, s) is of the general form
U(s,a) Σ, X,U,(s, + Us()) + Ua)
where A are the (static) coefficients, the atomic element function Ug depending on both and s characterises the "compatibility" of action a and scenario s, (s(s) depicts the current environment, and (a) evaluates the quality of the action. These terms can be categorized into the following groups
The smooth motion group encourages motion without abrupt change in acceleration, and penalises large jerks (i.e., the derivative of acceleration).
The safety distance to road obstacles group is designed to keep the AV away from other road objects to minimise the collision likelihoods. This distance is defined as 1 distance between the AV spatiotemporal sweeping contour and a foreign object on the road.
The legal motion satisfaction group is designed to enforce the AV to strictly follow all applicable traffic rules when in motion. For instance, the cost for crossing solid yellow lines
13

---

Under review as a conference paper at ICLR 2023
is significant that such behaviour is prohibited unless collision cannot be avoided otherwise. Some other legal options also come at certain prices too to discourage high-risk behaviour (c.g., lane changes in crowded scenes).
The progress to the destination group aims guide the AV to achieve the goal in the big picture and reach the final destination.
The aforemmntioned planner deployed onboard our AVs have gone through rigorous road test in urban areas of major cities with millions population. Results of more than 10,000 miles on average tested every week indicate that the planner achieves 111.3 miles per intervention (MPI), which suggests that the planner used in this work reasonable and validated one.
C MORE COMPARISONS TO RELATED METRICS
In comparison to the other advanced metrics (PKL. (Philion et al. 2020p, IPA (Ivanovic and Pavone 202Tp, SDE (Deng et al., (02T)) that are recently proposed for more effective perception evaluation, approach provides universal and principled solution to evaluate the impact of perception noise from the perspective of the planning process of an AV.
C.I COMPARISON WITH PKL
Here we provide more empirical results to better understand the difference between the proposed TIP and PKL (planning KL-divergence) (Philion al. 2020
Results on Synthetic Data
Figure 7 demonstrates scatter plot for scene-wise TIP and PKL results the synthetic data generated as described in Section 5.2 with 6 false positives per scene. It is observed that some results very close either or y-axis (the top right corner), suggesting that TIP and PKL deviate in determining if a perception eтror (i.e., false positive) is crucial to planning on these cases. A typical scenario of such disagreement is shown in Figure [ where the behaviour of the AV does not change significantly with ground-truth or noisy perception inputs (PKL -0.248), yet the planning process has changed quite lot (TIP = -61.654) due the affinity false positive objects that drastically change the planning cost to close objects. In this case, TIP is capable of detecting serious perception errors that PKL fails to identify.
Results on Real Data
the real data, we also have similar observations, and demonstrate the actual scene for one such scenario in Figure As shown in this case, false detection of vehicle in front of the AV does not change the behaviour considerably (PKL -0.802), while the significant planning cost change is reflected by TIP with value -15.42. More individual examples shown in Figure 10
Overall, on both synthetic and real data, we show that the proposed TIP can efficiently and effectively capture perception emrors critical to the planning that may be missed by PKL. This confirms motivation to exploit the actual AV planning process, as opposed to the AV behaviour (i.e., the result of planning), to gain insights into the impact of input perception error the whole AV system.
Trajectory Optimiser
Predictor
Trajectory
Trajectory
Da
a
Decider
Action Proposer
Perception Input
Trajectory
Misc. Input
Figure 6: Diagram of the major components in our planner which is used for computing the proposed perception evaluation metric TIP.
14

---

Under review as a conference paper at ICLR 2023
-20
-40
PKL
-60
-80
-100
-120
-300
-250
-200
-150
-100
-50
TIP
Figure 7: Result on the false positive synthetic data. The data points (downsampled for clarity) close to r-axis (PKL= 0)) correspond to the cases where the AV behaviours under ground truth and noisy perception inputs are identical. The data points close to y-axis (TIP = 0) correspond to cases where the AV planning preference between the optimal action and others are identical under ground truth and noisy perception inputs. Note the number of cases where TIP disagrees with PKL on the impact AV planning.
C.2 COMPARISON WITH IPA
IPA (injecting planning awareness) is recently developed in (Ivanovic and Pavone 2021 to encode the planning error based on the hypothesis that the impact of object location error is proportional to the gradient magnitude of the planning cost functions involving the AV-object distance. This solution however requires differentiability of the planning cost functions, while our approach does not and thus more applicable to the Level 4 AVs that are typically structured individual components including perception, prediction, planning, etc. Even more serious is that fails to account for all cases since the local properties (gradients) do not always reflect the global ones (overall losses). To illustrate this, consider a scenario, where the cost of AV being close to an object is 1/d. Now assume that there are the following two cases of object location errors.
modularized pipeline of
Case one: The ground truth distance of an object to the AV is 1m, and the noisy distance estimated by perception is 0.9m. Per the metric IPA defined in (Ivanovie and Pavone 2021 the result is
(1/d) dd
Ad = 1 1 1 0.9 0.1,
d=1
while the actual cost difference is
11 1/0.9 1-
0.9
Case two: The ground truth distance of an object to the AV 2m, and the noisy distance estimated by perception is 2.5m. Per the metric IPA defined in (Ivanovie and Pavone 2021 the result is
d
(1/d)
Ad 0.25 x 2.5 2.0 0.125,
dd
d=2
while the actual cost difference is
2.5 = 0.5 - 1/2.5 11222-55 = 0.1.
Obviously, the metric IPA of case two is larger than case one, while the actual error in planning cost the other way, as Taylor series up to first order terms adopted by IPA cannot precisely delineate the cost function value change large range input variation.
15

---

Under review as a conference paper at ICLR 2023
Figure 8: Illustration of AV behaviours in ground truth and synthetic scenes with false positives. The green tube represents the spatiotemporal trajectory of the AV with the z-axis as the temporal dimension (same for the rest). Bold solid lines are boundary of driving areas (e.g., curbs, vegetarian zoom dividers), while light solid lines are centre lines of vehicle lanes with dashed lines as the lane boundaries. Road objects are marked with 3D bounding boxes in green. Sub-figures in the first (second) row are birds-eye view (side view) of the scene, and sub-figures in the left (right) column correspond ground truth (noisy) perception input (same for the rest). In this case, AV intends to move forward under the ground truth perception input (the left column); in the presence of perception input noise (the right column), the AV behaviour remains almost unchanged (PKL = -0.248), since two false positive vehicles (pointed by red arrows) its both sides force the AV to keep moving straight, yet the close-to-object cost (safety distance to road obstacles) has changed considerably during planning, and is reflected by the score of TIP -61.654.
D SCENARIO COLLECTION
The scenarios used in this work are curated from AV road test in real world from public roads in urban areas of megacities, e.g., central business districts, populated residential communities, major commercial areas, etc. Each scenario is a 10s-long excerpt extracted from a continuous interval of road test, which consists of all raw data recordings (LiDAR point clouds, camera images, positioning signals, etc.) from the road test during the interval, and 2) the portion of offline generated high-definition (HD) and birds-eye view (BEV) raster maps that cover the field of perception during the interval. The duration of road test ranges from tens of minutes to several hours, and covers various times of both weekdays and weekends from early morning till late night during a period more than one year, providing a rich blending in weather condition (e.g., sunny, cloudy, rainy, and snowy), traffic intensity (e.g., rush hours on highways and crowded streets during holidays), road participant diversity (e.g., private cars, cyclists, pedestrians, and emergency vehicles), and so forth. The scenarios are selected from non-tre con-trivial situations (i.c., those with few traffic participants are filtered out) with balance in AV motion speed, diversity of traffic participants, weather, geographical locations,
16

---

Under review as a conference paper at ICLR 2023
G
P PRAD
P PRNE
Plado I G PRND
I6L
417
PRIE
Figure 9: Illustration of AV behaviours in reaction to ground truth and actual noisy perception inputs. Under the ground truth perception input (the left column), the AV is clear to move forward with soft braking to keep distance to another vehicle ("82") in front. Given the noisy perception input (the right column), however, the AV has to hard brake to avoid potential collision with the false positive vehicle close to it in front (marked by the red arrow). In either cases, since the AV speed slow and is braking (either soft or hard), the difference behaviour is insignificant (PKL- -0.802), yet the consequence of the false positive is by means trivial: the false positive causes hard brake and virtual collision (between the behaviours under ground truth perception input and false positive), which is precisely captured by the proposed TIP (TIP -15.422). The kinematic motion for the ground truth scenario (bottom left) is = -0 0.36m//2, j = -0.72m/s² and for the noisy scenario (bottom right) is a -0.36m/s²,
-6mss//.. Note how sharp the braking changes in presence of the noisy perception (jerk: -0. -0.72m/s v.s. 4m..../s²). Clearly, this is critical error from the system perspective.
E BREAKDOWN OF TIP
In order to facilitate understanding of the TIP evaluation process, we have illustrated the evaluation steps in Figure [1 where typical false negative case is used for the analysis.
F EXAMPLES AND NON-EXAMPLE OF SQUARE-INTEGRABLE DENSITY FUNCTIONS
Theorem T in the main text requires iquare-ineerrability of a density function, which includes many popular cases that may be used for constructing the utility function for planning.
Example 1 (Bounded PDFs). If both the support and range of the PDF fxx of an random variable bounded, then f() is square-integrable, e.g., uniform distributions.
17

---

Under review as a conference paper at ICLR 2023
Figure 10: Illustration of more AV behaviours in reaction to ground truth and actual noisy perception inputs. Two more outrageous perception errors are shown where an object location is improperly perceived such that it is superimposed with the AV. The ground truth is shown the left and the actual noisy perception is on the right. For the top case: TIP = 132.44, PKL = 0.0, acceleration 0.03m/s², jerk j 0.09m/s² (for both ground truth and actual noisy perception scenarios). For the bottom case: TIP -75.0, PKL = 0.04, = -0.35m/s2 j = -2.43m/s² (for the ground truth scenario), and 00.35m/s
m (for the actual noisy perception scenario).
Example (Parametric PDFs). PDFs of many popular parametric statistical models square- integrable, e.g., (sub-) Mau-)GGasssin, (sub-)Laplace, Gamma (including exponential, Erlang, and x² distributions),
Example 3 (Mixture Models of Countable Components with Square-Integrable PDFs). The PDF of a mixture model is of the form:
f(x) =
as >0, 0, E,0=11
(17)
where f(xx) is the PDF of the i-th component out of the countable set (f(x)), f(x) of 17 is square-integrable vi, fo L2 and M sup,llfillx +00 as
[15(0))2 do / Σ
(o) 4
Σ
anaz (sir
(18)
bd
Σ
a la < M² < +00.
A variety of mixture models are included such as Gaussian mixture models and mixtures of Gamma distributions.
18

---

Under review as a conference paper at ICLR 2023
p
q
22
2.20
PRNR
*
a
Ep(s) [U(s,a")] 5.83
Eg(s) [U(s,a")] --0.8
a
Ep(s) [U(s,a)] - 183.0
(((s) [U(s,a)] 55..3
Figure 11: Breakdown of TIP for typical miss detection scenario. A stationary vehicle 30m in front of the AV is missed by the onboard object detector. Four scenes are: 1) (top left) ground truth behaviour (a") in ground truth environment (p) with EU -5.83 (hard brake avoid collision); 2) (top right) ground truth behaviour (a*) in noisy perceived environment (q) with EU -80.8 (hard brake without any obstacles in front); 3) (bottom left) candidate behaviour (a) in ground truth environment (p) with EU -18.00 (move forward and collide with the object in front); 4) (bottom right) candidate behaviour (a) noisy perceived environment (q) with EU -58.3 (move forward). Planned (for AV) or predicted object motion in scene is rendered as coloured spatiotemporal trajectory (3D tube with z-axis time), e.g., each tube consists of locations the corresponding object time (t is the coordinate of the 7- or vertical axis). Note that in the lower left scene the candidate AV behaviour (move forward a constant speed) is evaluated against the ground truth environment (p), which collides with the object the end of 3s. Corresponding utilities shown under all cases, and TIP this case 199.7 according to the calculation of expected utilities in (5)
On the other hand, since et and e2 norms are not necessarily equivalent infinite-dimmnnioonal spaces, there are indeed some density functions (xr) Lt with infinite e norm.
Non-Exampa I (Square-Unintegrabl PDFs). Let the distribution Fx of random variable X be
x E (-c0, 0,
Fx =
Ja x
x E [0, a]
1,
x E (a, +00)
where a >> 0 is the parameter; and the density function is then
(0,
f(x)
2/a
otherwise
where (x:) is not square-integrable since x increases too fast 0.
19

---

Under review as a conference paper at ICLR 2023
G PROOFS OF THEOREMS IN THE MAIN TEXT
G. NOTATIONS
Besides the notations in Section 3.1 few more introduced follows. A unit step function is W(x c) = to], E R. ( p) denotes the space of absolutely integrable functions.
G.2 EMBEDDING PROBABILITY MEASURES
PROOF (Theorem 1). Since Fx (I) is absolutely continuous, there exists density function fx(x) 1 such that
d
Fx (a) = /x(x)
(19)
dx
almost everywhere. Since fx(x) I2 let M [|xx|l too, Agg H. have
Ex [g(x)]
g(x) dF))
(20)
J. g(1)) (()))()))(d)))
(21)
J. [=(z)]|[(x)) p(dx)
(22)
Milgll
(23)
where (33 follows from the Baachy-Schwarz inequality (Rudin) 1976 Theorem 11.35). Thus, the linear functional Ex [:] is bounded on H. and
Ex [9(x)]
d(x) fx(=)g(=)p(dzz) (fx.9)u Ag H,
where μ4 == fx E 7l is the embedding of the probability measure in H. Now assume that there exists another element y' H such that
Ex [9(x)] = ()' Vg E H.
Since Mp - ye' E H. we have
= (Mp pL, .pp - n') ||p - H 12
= (Mp: Hp - 11) - {pl, Mp - (1')
Ex [Mp 1r) -Ex [Mp 11)
0.
Therefore, the embedding Up for probability measure p in H. is unique equivalence class of the functions that equal almost everywhere.
G.3 INJECTION OF PROBABILITY MEASURE EMBEDDINGS IN H
To prove the injection of probability measure embedding in Theorem 2, a preliminary result of (Dudley 2002 Lemma 9.3.2) first introduced.
Lemma If (X, d.) is metric space, p and q are two probability measures on X, then Ez-p(a) a)) [g] =
Bxng(1) [9] Vi E Ct(XX)) if and only if p = q, where Cx(x) is the space of all bounded continuous functions on X.
PROOF (Theorem 2). Now prove this theorem in the following directions.
Necessity. Since the embedding of a probability measure is unique in H, it is easy to see that μlp = Pl if p = 9.
Sufficiency. Note that, by Weierstrass extreme value theorem Rudiin 1976 Theorem 4.16), any real continuous function C(X) the compact space X is bounded, i.e., Vg E C(X). EM R such that g[x))| < M, Vx E X. It follows that C(X) 122(2') since
[loco)] p(dx)) < M² X < +00.
20

---

Under review as a conference paper at ICLR 2023
Now if Up My almost everywhere, we have
Bp [g(x)] By [s(x)] (pp: 9) (Mg 0) = - (
Mr 9)
(24)
< 14p - Mu 10l11l191112 - 0, Vg C(X).
(25)
Thus p q by Lemma
G.4 APPROXIMATION OF EXPECTATION FOR DISCRETE/MIXED DISTRIBUTIONS IN H.
While Theorem I in the main text only addresses the continuous distributions, a similar result can be found given point-wise continuity conditions for general distributions, which can be decomposed into absolutely continuous and discrete parts (Chung 2000).
Theorem 4 (Approximation of Mixed Distribution). Let (ac(x) be an absolutely continuous distribu- tion function with density,fure function fx(x); Fa(æ) Σ, (r-(ar a discrete distribution function of point mass at countable set {a,} such that b; > and Σ, Fx +x) (1- (1- mixed distribution function with X E (0, as the convex combination coefficient. If fx (z) is
square-integruble, and g(x) E 1 is uniformly continuous al {az), then there exists a sequence of
{Mpan H such that
lim (pp.n: Exx [g(z)]
(26)
We start by considering simple discrete case by the following lemma. Lemma 2. Let (x) = ((x- a) be discrete distribution function with point mass at E X. If g(x) E L is continuous at a, then there exists a sequence of (Kp.)} C H such that
lim (pp.. 9)m Ex [g(z)]
(27)
PROOF (Lemma YE > O, since g(x) continuous a, there exists a a radius r >> 0 such that
g(a) - g(x) & g(a) +E, Va e B(a,r)
with positive measure V p(B(a, > where B(a,r) C X is neighbourhood of around Define
he(x) v (eEBBar)) H.
We have
g(a) - e <(hEr9)# < g(a) +e.
Thus,
lim (h =g(a)= =Ex [g(z)] n
Lemma implies that the expected value of function continuous the point mass of delta distribution can be approximated by an inner product in H with any arbitrary precision. PROOF (Theorem 3). Note that
+ Ex [g(x)]] = > f g(x)) (F))))))()))))))))))) + (1-11) A) E, beg(a.)
Since Fac(x) is absolutely continuous, by Theorem 1, there exists a f E such that
/ g(()) (u,g)u vh H.
(28)
On the other hand, Ve > 0, since g(x) uniformly continuous at {at), there exists radius such that Vi,
g(a.) - < g(x) < g(az) + Vo E B(a,,r)
2(b)) } itself, however, is Cauchy sequence, thus it has limit.
21

---

Under review as a conference paper at ICLR 2023
and V p(B(at, > 0 (translation invariance of Lebesgue measures in R*). Define
he(x) = v Σ by beloEB(orr) E H.
We have
Σ, bo(a)) - Σ big(ai) - E, by < (he < g(a)) + E, b/= Σ, beg(as) +
Thus
lim (N2,1) big(ai). n=330
(29)
Combining
28 and
29
leads to
Ap (1 Ahh >hg,9), = a(z)ddFcc()) (1- -1 big(a)) = Ex [o(x)]
lim
G.5 UNIFORM CONVERGENCE RATE EXPECTED UTILITY ESTIMATORS
PROOF (Theorem 3). Assume that (X,)= and independent and X, E [a, by] almost surely. Let
X = E,
Per Hoeffding's inequality (Hoeffding 1963 Theorem 2), for any 0,
2n²=² -i
Pr X
exp
(30)
V (b; 0. 0;)
By symmetry, it also holds true that, for any O.
2n²²²
Pr X < - <
exp
(31)
ii (h; - az)
Combining one-side inequalities of (30) and leads to
2n²g²
nE²
Pr
Pr( - E [ >
exxp
< 2exp
0,
(32)
Σ (b: az)²
2M2
where M = sup (aa1l,". ... |anl -1b1l (bbl)))
On the other hand, Bernstein inequality (Bernstein, 1946) also provides improved revision of Chebyshev's ' inequality by incorporating both almost-sure bound and variance bound:
Pr -
-
[ > E
2exp
nE²
Ve > 0.
(33)
2Var(X) + 2M
The proof is completed by setting X; USS a) and taking the lowest bound of the tail probability of - E X
32 and
33 for
22


Reviews:
[{"review_id": "zucQmsqzXg", "reviewer": "Reviewer_34kZ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new method to evaluate the perception results in autonomous vehicles based on its influences on the following planning module. With theoretical guarantees, the impact of the perception error can be decomposed into two orthogonal complements that are planning-critical and planning-invariant. Experiments on both synthetic and real datasets demonstrate the efficacy of the proposed evaluation method.\n\n\n\n----\n_Post-rebuttal_:\nThanks for the response.\nIt is glad to see the detail of Q1 and the updated contents in RS.\nFor Q2 and Q3, the authors have provided more explanations and discussions while admitting what is (and is not) included in the current work, which is OK to the reviewer.\nTherefore, the reviewer keeps the original positive score (6).", "summary_of_the_review": "Overall this is a solid paper on an interesting research topic with thorough discussions. \nThe reviewer is not familiar with all aspects and may skip some details.", "strength_and_weaknesses": "###### Pros\n- The topic of evaluating perceptions from the planning perspective is new and in great demand.\n- The proposed method is well theoretically supported, and its implementation is not complicated.\n- Adequate comparisons and case studies to other evaluation methods are provided, which fully demonstrate the proposed method's merits.\n\n###### Cons\n- It seems that the definition of the utility function is critical for this evaluation. The authors could discuss more how the choice and the computation of utility functions affect the usage of this evaluation method.\n- This method assumes the action set is finite, which may not be satisfied by all planners.\n- The definition in Eq. (13) takes the maximum reduction considering all actions. This is the worst-case analysis and is essential for AV, but would it be too conservative (so that the evaluation results may be too far from that of the final chosen action) in practice?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "yMKtL3mVV4l", "reviewer": "Reviewer_NfXU", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a method for evaluating the autonomy perception module from the perspective of a planner. This is an important topic and starts to receive increasing attention in recent years, yet still underexplored. Specifically, the paper proposes to formulate the expected utility maximization objective as an Hilbert inner product between the world state and the utility function. \n\nThe proposed approach is evaluated using synthetic and real data. Results show that the proposed method demonstrates higher sensitivity to perception noise, compared to PKL and NDS. \n", "summary_of_the_review": "I am on the fence and would like to see authors' feedback to make the final decision. ", "strength_and_weaknesses": "Strength: \n1. The paper is well written and the approach is well motivated.\n2. Results show that the proposed method has higher sensitivity in detecting perception noise, compared to PKL and NDS.\n\nWeakness:\n1. It is unclear to me how different methods’ scores are calibrated. Therefore, the proposed method might be just analyzed in the scale it is designed for. Moreover, should we calibrate the same scoring system across different evaluation scenarios? For the proposed TIP, what does a score evaluated from scenario A indicate when compared to a lower score evaluated from scenario B? \n2. The proposed TIP is for evaluating perception quality but the paper only shows results of added perception noise to one particular detector. I would like to see how TIP evaluates and compares different object detectors. \n3. As shown in Figure 4, the proposed TIP does not yield a monotonic score with respect to the distance. Will this cause confusion in practice? \n4. The paper fails to compare with SDE (Deng et al) quantitatively. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "JYCb2mTo1rk", "reviewer": "Reviewer_dqEU", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a method, called transcendental idealism of planner (TIP), to evaluate the end-to-end performance of a perception model in terms of how it affects AV planning. The authors formulate the planning of an autonomous vehicle as an expected utility maximization (EUM) problem. They further show that the objective function can be represented as an inner product between the world state description and the utility function in a Hilbert space, from which they derive the TIP metric.\n\nThe authors compared their proposed TIP metric against the traditional perception metrics, including NDS and PKL, on both synthetic and real datasets. They had the human labelers label the scenarios where the two metrics disagree, and the result shows that their TIP metric better captures the severity of perception errors.", "summary_of_the_review": "Evaluating the end-to-end perception performance in terms how it affects AV planning has significant practical value. The proposed method is shown to better capture the severity of perception errors compared to the existing methods.", "strength_and_weaknesses": "--- Strengths\n\n- Evaluating the end-to-end perception performance in terms how it affects AV planning has significant practical value. The proposed method is shown to better capture the severity of perception errors compared to the existing methods.\n\n- The paper also provides theoretical reasoning for their proposed method in addition to the empirical results.\n\n\n--- Weaknesses\n\n- It is not clear whether this metric is generic to different planner architectures.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject