On-Road Object Importance Estimation: A New Dataset and A Model with Multi-Fold Top-Down Guidance
Authors: Zhixiong Nan
Yilong Chen
Tianfei Zhou
Tao Xiang
Venue: NeurIPS 2024 poster (2024)
Keywords: Object importance estimation
Autonomous vehicles
Primary area: robotics

Abstract: This paper addresses the problem of on-road object importance estimation, which utilizes video sequences captured from the driver's perspective as the input. Although this problem is significant for safer and smarter driving systems, the exploration of this problem remains limited. On one hand, publicly-available large-scale datasets are scarce in the community. To address this dilemma, this paper contributes a new large-scale dataset named Traffic Object Importance (TOI). On the other hand, existing methods often only consider either bottom-up feature or single-fold guidance, leading to limitations in handling highly dynamic and diverse traffic scenarios. Different from existing methods, this paper proposes a model that integrates multi-fold top-down guidance with the bottom-up feature. Specifically, three kinds of top-down guidance factors (i.e., driver intention, semantic context, and traffic rule) are integrated into our model. These factors are important for object importance estimation, but none of the existing methods simultaneously consider them. To our knowledge, this paper proposes the first on-road object importance estimation model that fuses multi-fold top-down guidance factors with bottom-up feature. Extensive experiments demonstrate that our model outperforms state-of-the-art methods by large margins, achieving 23.1% Average Precision (AP) improvement compared with the recently proposed model (i.e., Goal).

On-Road Object Importance Estimation: A New Dataset and A Model with Multi-Fold Top-Down Guidance
Zhixiong Nan', Yilong Chen', Tianfei Zhou +2, and Tao Xiang
Coolleee of Computer Science, Chongqing University, Chongqing, China. School of Computer Science and Technology, Beijing Institute Technology, Beijing. China. nanzx0cqu. edu. en, chenyilongâstu. equ.edu.e. en, tffzou@bit.edii edu. txiangecqu. edu.
Abstract
This paper addresses the problem of on-road object importance estimation, which utilizes video sequences captured from the driver's perspective as the input. Al- though this problem is significant for safer and smarter driving systems, the explo- ration of this problem remains limited. On one hand, publicly-available large-scale datasets are scarce in the community. To address this dilemma, this paper con- tributes new large-scale dataset named Traffic Object Importance (TOI). On the other hand, existing methods often only consider either bottom-up feature or single-fold guidance, leading limitations in handling highly dynamic and diverse traffic scenarios. Different from existing methods, this paper proposes a model that integrates multi-fold top-down guidance with the bottom-up feature. Specifically, three kinds of top-down guidance factors (i.e., driver intention, semantic context, and traffic rule) are integrated into our model. These factors are important for object importance estimation, but none of the existing methods simultaneously consider them. To our knowledge, this paper proposes the first on-road object importance estimation model that fuses multi-fold top-down guidance factors with bottom-up feature. Extensive experiments demonstrate that our model outperforms state-of-the-an art methods by large margins, achieving 23.1% 11 Average Precision (AP) improvement compared with the recently proposed model (i.e., Goal).
Introduction
According to the World Health Statistics WHO (34), road traffic injuries account for significant 29% of all injury deaths, with nearly 1.3 million people losing their lives in traffic accidents annually. Accurately estimating the importance of on-road objects can pave the way for safer (e.g., e. automatic emergency braking (45; 39)) and smarter driving systems (32; 25; 38; 49; 4; 36; 11), potentially preventing numerous accidents.
Although on-road object importance estimation presents significant research value, it has not been widely explored. One main reason the scarcity of publicly-available large-scale datasets in community. Specific for the on-road object importance estimation task, the only one publicly- available dataset is Ohn-Bar al. (33), which contains 3, 187 frames, 8 scenes, and 16,076 object importance annotations. Such small-scale dataset supports to train small and less complex models.
"Corresponding author.
This work is supported by Chongging Natural Science Foundation Innovation and Development Joint Fund (CTTB202NSCQ-L2X0000))
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

However, traffic scenes are dynamic and diverse, asking for complex models to handle various traffic situations. Some researchers have recognized this dilemma and propose some datasets (8; 21; 46). Unfortunately, these dataset are not publicly-available fundamentally addressed. To fundamentally address this dilemma and push forward the advancement of on-road object importance study, this paper will release large-scale dataset (named as TOI, Traffic Object Importance) containing 9,858 frames, 28 scenes, and 44, object importance annotations. Compared to Ohn-Bar (33), TOI achieves a 3.1-fold increase in frames count, a 3.5-fold increase in scene count, and a 2.7-fold increase in object count.
thereby the dilemma has not been
From the perspective of methodology, some methods have been proposed (21; 8; 50; 33). However, these methods are relatively simple, exhibiting the low performance when confronting to challenging traffic scenarios. This motivates us to think about a question: why can not existing methods perform well? The potential reason is that existing on-road importance estimation methods underestimate the complexity of traffic scenarios, individual bottom-up mechanism (50) (assuming important objects are the objects with salient color, texture, size, etc.) or simple top-down guidance mechanism (8; 21) (fusing the bottom-up information with certain type of top-down information such as semantics, ego-car trajectory, driving task (27), etc.) can hardly address dynamic and diverse scenarios.
(a) Driver Intention
(c) Traffic Rule
important object
driving path
uomp object
Semantic Context
unimpertant object
Important
drivable areas
Important object
Figure 1: The crucial factors considered by human drivers when estimating on-road object importance.
Therefore, a smarter model is needed. Inspired by the fact that a human driver can accurately estimate object importance in challenging situations, this paper attempts to design a model by analysing the human reasoning mechanism during estimating object importance. To this end, the primary question is "what essentially crucial factors are considered when a human driver is estimating the importance of objects?"' Firstly, the attributes (e.g., size, color, and texture) of the object is considered. For example, when a truck with the big size and a car with the small size simultaneously appear in front of the ego-car, the truck is more important since it imposes bigger impact on the driving. Secondly, the driver intention is considered. The objects that will riskly collide with the ego-car intention driving path or the objects locating on the ego-car intention driving path present high importance, as shown in Fig. la. Thirdly, overall semantic context of the whole traffic scene is considered. A human usually pay more attention on the objects in drivable areas rather than the objects in undrivable areas. As shown in Fig. 1b, the person riding a bicycle in undrivable areas is unimportant. Fourthly, traffic rule considered. Most of traffic participants obey the traffic rule, thus the traffic rule is an critical factor for human to estimate object importance. For example, as shown in Fig. Ic, when there exists lane marking between the oncoming car and the ego-car, the human driver may consider the oncoming car as unimportant. In contrast, if there is no lane marking between them, the importance of the oncoming car significantly increases. The traffic rule is crucial for object importance estimation. However, none of existing methods utilizes traffic rule to estimate on-road object importance.
Based on the above observations, we propose a model with multi-fold top-down guidance including driver intention, semantic context, and traffic rule. As far as we know, it is the first on-road object importance estimation model that fuses multi-fold top-down guidance factors with the bottom-up feature. The proposed model consists of two kinds of pathways (i.e., bottom-up pathway and top- down pathway). Bottom-up pathway and top-down pathway are fused to estimate object importance. Specifically, in the top-down pathway, the top-down guidance factors of driver intention and semantic context are involved in the proposed Driver Intention and Semantics Guidance (DISG) module, and traffic rule is modeled in the proposed Traffic Rule Guidance (TRG) module. In the bottom-up pathway, Object Feature Extraction (OFE) module is proposed to extract object features in both spatial and temporal dimensions.
2

---

A series of comparison and ablation studies are conducted on a public dataset (33) and our TOI dataset. The comparison experiment results show that our model has a solid advantage over the baselines. The ablation study results validate the effectiveness of our proposed interactive bottom-up & top-down fusion framework and multi-fole top-down guidance modules (i.e., DISG and TRG).
The contributions of this paper are as follows. 1) This paper contributes a new large-scale dataset, which will be publicly released. This dataset is almost three times larger than the current publicly available public dataset (33). 2) This paper contributes an object importance estimation model. As far as we know, it is the first on-road object importance estimation model that integrates multi-fold tow-down guidance factors with the bottom-up feature. 3) The traffic rule is crucial for object importance estimation. However, none of existing methods utilizes traffic rule to estimate on- on-road object importance. This paper considers the effect of traffic rule on object importance and successfully models this abstract concept by proposing an adaptive object-lane interaction mechanism.
2 Related Works
On-Road Object Importance Estimation Related Datasets. Currently, the primary dilemma of research on the on-road object importance estimation is the lack of sufficient data. Among existing datasets relevant to autonomous driving perception tasks, only few meet the data requirement of providing images from the driver's s first-person perspective while also including object importance level labels. Ohn-Bar et al. (33) are the first define the problem of on-road object importance estimation, and they propose a small-scale publicly available dataset, which contains 8 scenes. Gao er al. (8) and Li al. (21) have significantly increased the number of scenes. However, their datasets are not publicly available, thus the contributions to the community is limited. Datasets (19; 41; 48) are with detailed annotations such ego's reaction and road topology. They have great potential to advance the development of on-road object importance estimation. However, currently, they lack importance
object importance level labels and cannot be directly applied this task.
Currently, on-road object importance estimation methods can be divided into two categories: 1) methods solely utilizing bottom-up feature; 2) methods utilizing single-fold top-down guidance.
On-Road Object Importance Estimation Related Methods.
The methods solely utilizing bottom-up feature focus on the visual attributes of the objects. The bottom-up processing method initially introduced in (44), and Itti al. (17) propose one of the first bottom-up mechanism based models. Following this, many researchers are inspired by this concept (43; 15; 18; 35). Zhang et al. (50) introduce a model, which solely relies on RGB clips for object importance estimation. This model employs graph convolutions to characterize the interactions among on-road objects. Nitta et al. (30) develop a model that extracts temporal features from optical flow images to infer the states of moving objects. The optical flow images are also used in Malla et al. (26) to assess the states of moving objects. The bottom-up methods can also be found in the works (33; 52; 47; 23; 14; 24). Although the bottom-up feature is crucial for importance estimation, the methods solely rely on bottom-up feature can not function well in the complex scenarios.
The importance of an object is influenced by many factors such as driver intention, which cannot be fully utilized through bottom-up methods. However, current methods are relatively simple and relies on single-fold guidance. Niu et al. (31) utilize a Transformer with shared weights to identify high-risk objects and generated semantic warning sentences. Li et al. (21) investigate the impact of driver intention, employing the action and trajectory data of the ego-car as additional supervisory signals in auxiliary tasks to enhance model performance. Gao et al. (8) utilize the driver's 's goal to estimate object importance. A cause-effect problem was formulated in (20), which introduced model to estimate the risky object by considering its potential impact on the driver's behavior. Tang er al. (42) provide a more comprehensive understanding of how driver intentions under different tasks affect the driver attention. Single-fold top-down guidance can also be found in the works of attention prediction task (7; 16; 22; 6; 1; 29; 5; 28). However, none of these methods utilizes multi-fold top-down guidance factors to estimate the on-road object importance.
3 A New Dataset: TOI
We thoroughly review existing datasets for on-road object importance estimation as well as the datasets for the related tasks, and provide a summary in Tab. 1. The datasets for the related tasks (e.g., risk assessment (37; 48; 19), accident anticipation (41), and situation awareness (40)) do not include object importance labels, making them unsuitable for object importance estimation task. Most datasets (e.g., (21; 8)) for object importance estimation are not publicly available. The only
3

---

Table I: Comparison between the TOI and State-of-the-art Datasets. 'Impo.' represents the object importance annotation.
Datasel
Tusk
Public Impo
GPS/MU Lidar 3D-Labelk
Aatra-Iaformation
Objects Frames Scenes
FPS
Year
HDD (37)
risk assessment
30
2018
1361 bonda (48)
risk useesmmnt
X
X
1,361
2020
RiskRench (19)
aesessment
x
X
X
6,916
2024
NIDB (41)
accident anticipation
<
X
X
X
X
499,500
4,995
2018
A-SASS (46)
situation awareness
x
/
X
10
30
2022
ROAD (40)
situation awareness
<
x
560,000
122,000
22
2023
Ohn-Bar (33)
on-road object importance estimation
16,076
3,187
2017
Goal (8)
on-road object importance estimation
X
244,980
743
30
2019
(21
on-road object importance estimation
X
2022
TOI
on on- object imporiance estimation
44,120
9,858
28
2024
publicly available dataset is Ohn-Bar (33), but it small scale dataset. In response the scarcity of publicly-availabe large-scale datasets for on-road object importance estimation, we contribute a scage dataset named TOI (Traffic Object Importance). TOI is built by re-annotating the authoritative KITTI (9) dataset. While there are many datasets (e.g., nuScenes (2; 40; 37)) that be used for object importance annotations, we select KITTI dataset for the following reason: KITTI is the worldwide benchmark in the field of autonomous driving. In addition, KITTI is collected in diverse real traffic scenes including rural areas and on highways with rich date formats, making the dateset friendly for various tasks.
Annotation Procedure. The criterion of object importance might be ambiguous as different drivers usually hold different opinions on the importance judgment. Currently, object-level importance labels are annotated without checking mechanism. Although multiple annotators perform the annotations, the annotations finished by the certain annotator are not checked by others, leading to the unreliable and ambiguous annotations. To generate reliable annotations, we adopt two mechanisms: the double- chebli-checking annotation mechanism and the triple-discussing annotation mechanism.
The double-checking annotation mechanism operates as follows. Initially, the first annotator (an experienced driver) labels the object importance at every 10 frames. To guarantee the reliability of annotations, the first annotator only selects one object as the important object at each time of observing the whole 10 frames. The annotation for these 10 frames is finished until all important objects are annotated, then the annotator moves to the next set of 10 frames for annotation. Subsequently, the annotation results are checked by the second annotator (who is also an experienced driver). When the second annotator finds disputed annotation, the first and second annotators discuss together reach an agreement. If they are unable to reach an agreement, the triple-discussie annotation mechanism activated. In this case, the third annotator is invited to discuss the final annotations.
Statistics and Comparison. Totally, 9,858 image frames are annotated, generating 44,120 objects with the importance or unimportance annotations, among which 5,052 objects are annotated important. The annotated data are randomly split into training/testing datasets with a ratio of 8,121 : 1,737. The comparison between TOI and existing on-road object importance estimation datasets and similar task datasets are presented in Tab. 1. Compared to the publicly available Ohn-Bar (33) dataset, TOI represents the significant advantages in following three aspects. Frame quantity: TOI exhibits substantial increase in the number of frames, with 9,858 frames compared to 3,187 frames in the Ohn-Bar dataset. Object quantity: the number of annotated objects is 44, 120 compared to 16,076 in the Ohn-Bar Ban-Bar dataset. Scene diversity: while Ohn-Bar contains only 8 scenes, TOI covers 28 scenes. Compared to Goal (8) dataset, TOI has rich annotations such as Lidar and 3D tracklet labels. The abundance of multimodal annotations enables TOI to support the research on on-road object importance estimation using multimodal learning methods the future. Though Goal (8) presents the advantage in terms of frame number and scene diversity, it is not publicly available. Compared to Li dataset (21). TOI offers the frame rate of 10 FPS. This high temporal resolution is critical for capturing the dynamic changes of on-road object importance.
Annotation Challenges. Compared datasets for other tasks, TOI may not be considered large- scale, it is relatively large-scale compared to existing publicly available datasets for on-road object importance estimation. However, annotating object importance at this scale is challenging. Each annotation requires multiple annotators and undergoes rigorous checking to achieve satisfactory results. In addition, to generate reliable annotations, only one object is annotated at each time observing the whole video sequence, a complete re-observation of the whole video sequence is required for the annotation of the next object. Moreover, object importance is affected by multiple factors, which imposes difficultiess on the annotation.
4

---

Approach
4.1 Overview
Driver Intention and Semonricr Guidonce
Objjet-intention-e Interaation
Semantic Feature
Objjct-lattnninn- Semmatics Interacion
Object Iuportence Estimation
semantic guiding feature.
Estraction
CA
Tap-Down
driver Infention
agj poraze
object sgatial feature
guidiigg
Ubject Preture
Bottow-Up
Extraction
Object-lane interaction weightting
Top-Dhoen
Object-lane interaction
Adapetre
Feature Indtrationn
Objuct-Lane
Interaction
Truffic
Figure 2: The overview of multi-fold top-down guidance aware model.
Consider traffic scenario with N on-road objects in T time steps, the goal of this work is to estimate on-road object importance (A) at the final time step (i.e., = 7) using the video sequence ( (V) captured from the driving perspective over T time step and ego-car velocity information (E) the first time step (i.e., & = 1), which is formulated as:
A = WVVVV, F),
(I)
where N represents an on-road object importance estimation network, V= and A=(A,)
In order to effectively fuse multi-fold top-down guidance factors (i.e., semantic context, driver intention, and traffic rule) with bottom-up object visual feature, we propose multi-fold top-down guidance aware model, the overview of which is illustrated in Fig. 2. Our model is composed of four key modules: Object Feature Extraction (OFE) module detailed in & 4.2, Driver Intention and Semantics Guidance (DISG) module described in $ 4.3, Traffic Rule Guidance (TRG) module explained in ş 4.4, and Object Importance Estimation module introduced $ 4.5.
Firstly, OFE extracts object spatial feature fo, and object temporal feature fo.: from V. Then, DISG takes E, V, and fa.s as inputs, and outputs senee-iiteniionsemaane fori-s- Meanwhile, in TRG, the lane feature f: and fo,t are processed by adaptive object-lane interaction mechanism produce the object-lane interaction feature fort- Finally, foriss and fo-t are used to estimate object importance A.
interaction feature
4.2 Object Feature Extraction
The goal of Object Feature Extraction (OFE) module is extract object features in both temporal and spatial dimensions. The input of OFE is a RGB video sequence V E RRTx3xWxx and the outputs object temporal feature fo,t and object spatial feature fo,--
To begin with, OFE takes and M (M denote optical flow images derived from V) as inputs to extract the object visual feature fo E RXXXXCXW's xn' (reflecting the appearance of the object) and the object motion feature f.. E IXXTxCXW'XH" (reflecting the movement of the object). This procedure is formulated as:
fo (oi(Nv(V)), fm = Roi(NA (M)),
where Ny and NM represent the ResNet18 (12), Roi denotes the ROI pooling (10), C represents the number of channels, W' and H' denote the width and height obtained through ROI pooling.
Subsequently, object spatial feature Sa,. Nx2Cx' xH' obtained based on fr and fm. The goal of fo, is to focus the spatial information of objects. Therefore, average pooling is applicd on the time dimension (i.e. the dimension of T)) of fo and fms and self-attention mechanism is utilized to emphasize the spatial information (i.e., the dimensions of W and H'), which is denoted as follows:
fo. Nuksa (Concat (Arncat(Avg(aaa Avg(f..)
where Avg denotes average pooling, Concat concatenation. Nmhsa represents the multi-head self-attention mechanism, and it has the same meaning in the following parts.
5

---

Meanwhile, object temporal feature fo, E R NNC' is also extracted based on fo and fm. The goal of fo.t is focus on the temporal feature of objects. Therefore, the dimensions of f. and fm are firstly reshaped from N x T x C' x W x H' to N x T x (C' x W' x H'). and then two LSTT networks applied to extract the temporal feature. Finally, subsequently fuse fo, with f. RNC" it is necessary to transform the channel number of fo,: hence a Linear layer is required. This procedure is formulated as:
fo. Linear(NNmhsa (Nonaaaaaasaaaanaaaaaaaa (Concat(Nistmml (fot, (istmm(mmm)) (fm))).
where Nistm is the LSTM network (13), which outputs features for T time steps, and fo.t is computed by indexing the information at the T-th time step.
4.3 Driver Intention and Semantics Guidance
The driver intention and semantic context significantly affect the on-road object importance in top-down manner, thus we propose the Driver Intention and Semanties Guidance (DISG) module. DISG consists of two components, namely semantic feature extraction and object-intention- semantics interaction. The former extracts the semantic guiding feature (i.e.. fs in Eq. (5)) from driving scenarios, and the latter uses f. and intention guiding mask (i.e., m in Eq. (7)) to guide the refinement of fo.s. In the following part, we will sequentially introduce the calculation processes for semanlic guiding feature and intention guiding mask.
Semantic guiding feature. The goal of fa E IR 1x20xx HH' is to guide the model to be aware of the semantic context of on-road objects, which is extracted by:
f. NG(G),
(5)
where NG denotes ResNet18 backbone network and G represents semantic segmentation maps obtained from Vr.
Intention guiding mask. The goal of m E R xH' is making the model be aware of the region that consistent with the driver intention. However, difficult to realize since the driver intention an abstract concept and the limited information regarding the driver intention is known. Additionally, it is not reasonable to assume the driver intention is known in advance. Therefore, we use three common intention behaviors in driving to reflect the driver intention (i.e., turning left, going straight, and turning right). Intention behaviors are formulated as the corresponding intention guiding masks:
(6)
where my M. and m, are manually engineered masks to emphasize the information in the right, center, and left regions of the images in the video sequence, respectively. Their sizes are aligned with the size of f.. a and b represent pre-determined low and high values, respectively. We note that my representing turning left behavior is allocated with higher value at the right part, which is inspired by the finding of Tang et al. (42) demonstrating that when a car is turning left, the driver pays more attention the right side. Before proposing this predefined intention guiding mask, design a learnable mask with random initialization to make the model automatically learn the mask to reflect intention behaviors. However, this strategy does not work. The potential reason is that the intention is abstract to learn.
To automatically select the of corresponding driving behavior, we design the following logic based on the angular velocity E of ego-car:
mi, if E > 8
m
mr, if - m. otherwise
where 3 represents the driver turning threshold.
After obtaining fa and m. DISG module fuses them and uses the fused feature to guide the refinement of fo.. This procedure is implemented by the objjeciintntioon-eeantii interaction component. interaction is to fuse f. and m, and generate the
The first task of object-intentionsemaann intention-semanties interaction feature fi-e E Rlx2Cx X xH which is denoted as follow:
fr. f
(8)

---

where the operator x makes the model pay more attention to the semantic context in the driver intention regions.
The second task of objettiintention-seaaniis interaction to refine fa.s by interacting with fi-s. which is formulated as follow:
fo-i-s = Nomho (fo,s: fi-)) + Jo.s:
(9)
where Nmhon denotes the multi-hean cross-attention mechanism, fa,s serves the query while fi-s serves the key and value, and fo-i-s RRNx2CXW'xHF
4.4 Traffic Rule Guidance
On-road object importance is also closely related with traffic rule, but is often overlooked in previous works. To effectively leverage the traffic rule, we propose the Traffic Rule Guidance (TRG) module. which consists of two components; lane feature extraction and adaptive object-lane interaction.
Lane feature extraction is to make the preparation for adaptive object-lane interaction. In detail, a linear transformation and an activation are applied on lane information L:
fi Relu L)),
(10)
where L are the coordinates of lane marking points, which are derived from Vr via a lane marking detector, Relu is the rectified linear unit activation, and fi E NxC"
Adaptive object-lane interaction is the core of TRG, and it contains two steps: object-lane interaction and object-lane interaction weighting. the first step, lane feature fi and fo, are fused through multi-head cross-attention mechanism and residual mechanism, which can be denoted as:
for Nihea (ft, fo,)) fo.ts
(11)
where for E IR NCC denotes initial object-lane interaction feature, and fi serves as the query while fort serves the key and value,
Factually, for has considered the traffic rule factor by modeling the relation between lane markings and on-road objects. However, the influence of lane markings on-road object importance estimation might not be universally-effeettvv in all scenarios, thus we propose the object-lann lane interaction weighting mechanism to realize adaptive object-lane interaction.
The goal of object-lane interaction weighting is to adaptively penalize the cases in which object-lane relation is weak (e.g., static roadside cars weakly interacts with lane markings). To this end, MLP network is applied on f to compute a score p. and this score is then used to compute the corresponding penalizing coefficient Pa. which is denoted as:
p = Sigmoid( ( (f
(12)
1,
if < 0.5
Pc.
(13)
if 2 0.5
where a is a very small value.
Based on Po. the object-lane interaction feature for NxC' obtained by weighting fmr in Eq. (11), which is denoted as:
fo- = for x Pe-
(14)
We note that object-lane interaction weighting is the of adaptive object-lane interaction, which is significant for object importance estimation (30.4% improvement on AP).
4.5 Object Importance Estimation
Taking forr= in Eq. (9) and for in Eq. (14) the inputs, object importance A E R is estimated. This procedure formulated as:
A = Softtoo (Noima (Linear(LLLLaariii + fo-t))
(15)
where A signifies the importance for each object. The Linear layer transforms the dimensions of forrze from N 20 x W' x H' to N x C' so that it can be added to fo-t-
7

---

Experiments
Metrics. For performance evaluation, classical metrics chosen: Average Precision (AP) and Fl Score (FI), AP is computed by calculating the area under the precision-recal curve at various thresholds, thus is compressive metric to indicate both the precision and recall of model. Fl is computed by precision and recall at a fixed threshold, thus it indicates the balance between precision and recall. Both metrics follow the principle that higher values indicate better performance.
Loss Function. The loss function defined as follow:
(( A) BCELoss(A, A) + FocalLoss(A, (A....) A),
(16)
where A is the predicted object importance, A is the ground-truth.
5.1 Comparison Experiment
Baselines. Our model is compared with seven models. Ohn-Bar (33), Goal (8), Zhang (50), and Li (21) are representative works for on-road object importance estimation, In addition, considering that the salient object detection indicates important objects to the certain extent, three recently- proposed salient object detection models, namely MENet (23), A2S (52), and PGNet (47), are also selected as baselines.
Table 2: Quantitative comparison with baselines on TOI and Ohn-Bar (33) datasets. The 'Video' signifies the usage of RGB video sequence, 'Velocity' denotes the incorporation of vehicle velocity information, and '3D-Ojject' indicates the utilization of 3D object properties information.
Inputs
TOI
Ohn-Bar
Speed (ms/clip)
Method
Video
Velocity
3D-Object
APf
F11
Acct
AP1
FIf
Acct
Yolo
5
9
14
25
48
34
278
Obn-Bar (33) 2R12017
19
28
74
39
14
62
100
Goal (8) 2CRA22199
50
49
90
52
26
65
155
Zhang (50) ICRA22000
16
91
28
10
54
200
Li (21) ICRA22022
23
91
41
0
64
202
MENNet (23) CVPR
26
76
12
62
122
A2S (52) CVPR'2025
78
18
120
PGNet CVPR
84
27
58
130
115
Quantitative Comparison. Tab, 2 shows the comparison results on TOI and Ohn-Bar (33) datasets. We can observe that our model outperforms all seven baselines. On the Ohn- Bar (33) dataset, our model achieves 23.1 % and 96.3% performance improvements on AP and Fl metrics compared with the second-best result, respectively. On the TOI dataset, compared with the second-best result on AP and F1 metrics, our model achieves 20.0% (i.e., (60-50)/50) and 10.2% performance improvements, respectively. The results of MENet (23), A2S (52), and PGNet (47) are not reported on AP metric due to the lack of confidence scores in salient object detection outputs, making it impossible to calculate precision-recat curves for different thresholds. The Fl results for Zhang (50) and Li (21) are both because they predict all objects as unimportant.
5.2 Ablation Studies
Top-down and Bottom-up Framework. To validate the effectiveness of model that inter- actively integrates multi-fold top-down guidance mechanisms with bottom-up features, we con- duct four experiments: #1: only the bottom-up module is enabled: #2: the bottom-up module is combined with TRG module; #3: the bottom-up module is combined with DISG module; #4: the bottom-up module is combined with both TRG module Table 3: Ablation study on top- and DISG module. The experimental results are reported in Tab. 3. down and bottom-up framework. Compared with #1, #2 achieves 55% and 40% performance im- provements on AP and F1, respectively. Similarly, #3 obtains 160% and 56% performance improvements on AP and F1, respec- tively. When both modules are enabled (#4), our model exhibits the best performance. These results validate both TRG and DISG modules are effective.
Method TRG DISG APt FI
#1
25
#2
31
35
84
60 54

---

Driver Intention and Semantics Guidance (DISG). To verify the effectiveness of semantic context guidance and driver intention guidance, we conduct three experiments: #1: the model without seman-
tic guiding feature and intention guiding mask; #2: the model only with semantic guiding feature;
#3: the model only with intention guiding mask; #4: the model Table Ablation study on DISG.
with both semantic guiding feature and intention guiding mask.
Method Seman. Intent. AP1 Flf
The results are shown in Tab. 4. Compared to #1, #2 yields 58.1% 1% and 37.1% performance improvements on AP and Fl metrics, respectively. This enhancement is attributed to the usage of the semantic guiding feature, which enables the model to learn the semantic relation between objects and the whole scene. Meanwhile, #3 achieves 15.4% and 1 11.4%
#2
4
49
48
60 54
performance improvements on AP and Fl metrics, respectively. The effectiveness of our intention guiding mask accounts for this advancement. #4 exhibits the best performance, demonstrating the effectiveness of proposed semantic guiding feature and intention guiding mask.
Traffic Rule Guidance (TRG). To analyze the effects of object-lane interaction and object-lane interaction weighting, we conduct three experiments: #1: the model without object-lane inter- action and object-lane interaction weighting: #2: only the object-lane interaction is enabled;
#3: both the object-lane interaction and the object-lane interaction Table 5: Ablation study on TRG. weighting are enabled. The corresponding results are summarized in Tab. 5. Compared to #1, #3 obtains 15.4% and 38.5% im- provements on the AP and Fl metrics, respectively. These results demonstrate the significance of both object-lane interaction and the object-lane interaction weighting mechanisms. The reason is explainable. When lane information is not utilized, the implicit traffic rule conveyed by the lane is not used. The absence of the traffic rule results in a reduced ability of the model.
Method Interac. Weight APT
FI1
A1
52
39
#2
46
#3
60 54
It comes as a surprise that the individual usage of object-lane interaction (#2) leads to the perfor- mance decreasing on the AP metric compared to #1. This is due to the fact that not all on-road objects are influenced by lanes. Without object-lane interaction weighting, individual object-lane interaction generates a uniform object-lane interaction feature, which could not adaptively extend to diverse scenarios. This result potentially proves the significance of our object-lane interaction weighting, which enables the model to adaptively disable the object-lane interaction feature when object importance weakly rely on object-lane interaction (e.g., static cars on the roadside).
To further analyze object-lane interaction weighting, we visualize its output (i.e., Pe in Eq. (13)). Some examples are illustrated in Fig. 3 where objects with blue masks are penalized (i.e., object-lane inter- action is disabled, pa=a) and objects with yellow masks are not penalized (i.e., object-lane interaction is enabled, p.=1). In Fig. 3a and Fig. 3b, the object-lane interaction weighting penalizes the cars on both sides of the road. The results make sense since the static cars on roadsides are factually not interacting with lanes. In Fig. 3c and Fig. 3d, oncoming cars from the opposite direction and the car on the current lane are not penalized, since these cars are interacting with lanes. We note the yellow mask do not signal the important object. Instead, it indicates that the object-lane interaction is enabled.
Figure 3: Visualization of object-lane interaction weighting.
6 Conclusion
On-road object importance estimation significant for various applications in the fields of assisted driving and autonomous driving. The dilemmas of current research are two fold: 1) the scarcity of large-scale publicly available datasets hinder the development of on-road object importance estimation, and 2) existing methods are relatively simple to handle complex and diverse traffic scenarios. In response to the dilemmas, this paper contributes new dataset and proposes a model with multi-fold top-down guidance. A large range of experiments demonstrate the superiority of our proposed model. The main conclusion that building up the model that comprehensively considers multi-fold top-down guidance (e.g., driver intention, semantic context, and traffic rule) and bottom-up feature (e.g., size, distance, and speed) is a promising way to remarkably push forward the study of on-road object importance estimation.

---

References
Amadori, P.V. Fischer, T, Demiris, Y: Hammerdrive: A task-aware driving visual atten- tion model. IEEE Transactions on Intelligent Transportation Systems 23(6). 5573-5585 (2022).
https//doi..rr//10
[2] Caesar; H., Bankiti, V., Lang, A.H., Vora, S., Liong, V.E., Xu, Q. Krishnan, A., Pan, Y., Baldan, G., Beijbom, O.: nuscenes: A multimodal dataset for autonomous driving. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (2020)
[3] Chen, L.C.,, Papandreou, G., Kokkinos, Murphy, K., Yuille, A.I.: Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEFE Transactions on Pattern Analysis and Machine Intelligence 40(4), 834-848 22018. https://di.orr// 100. 109/TPAM..2017.200
[4] Chen, L., Li, Y., Huang, C., Xing. Y., Tian, D., Li, L., Hu, Z., Teng, S., Lv, C., Wang, J., Cao, D., Zheng, N., Wang. F.Y.: Milestones in autonomous driving and intelligent vehicles -part i: control, computing system design, communication, map. testing, and human behaviors. IEEE Transactions on Systems, Man, and Cybernetics: Systems 53(9), 5831-5847 (2023). https://doi.org/ wtt.:///.0..1100//MM222002.. 109/TMMC.20233
[5] Chen, Y., Nan. Z., Xiang. T.: Fblnet: Feedback loop network for driver attention prediction. In: IEEE/CVF International Conference on Computer Vision. pp. 13325-1333 13334 (2023)
[6] Deng, T.. Yang, K., Li, Y., Yan, H.; Where does the driver look? top-down-based saliency detection in traffic driving environment. IEEE Transactions on Intelligent Transportation Systems 17(7), 2051-2062 (2016). https://doi.org 1 200TTTTS.001..255.. orr///0....///11
[7] Fang, J., Yan, D., Qiao, J., Xue, J., Yu, H.: Dada: Driver attention prediction driving acci-
dent scenarios. IEEE Transactions on Intelligent Transportation Systems 23(6), 4959-4977 (2022). https://doi.orr/IU 1 109/TITS, TTT.TS..220.00446677
[8] Gao, M., Tawari, A., Martin, S.: Goal-oriented object importance estimation in on-road driving videos. In: Proceedings the IEEE International Conference on Roboties and Automation, pp. 5509 -595-5515 (2019). https://coii.or//10.. 10.1 WWWICRAA201998799997
[9] Geiger, A., Lenz, P., Urtasun, R.: Are we ready for autonomous driving? the kitti vision benchmark suite. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. (2012). https://doi.orge 1 1ttp//0.111///PPPP/ (09/CVP..201..224400
[10] Girshick, R.: Fast r-cnn. In: Proceedings of the IEEE International Conference Computer Vision, pp.
1440-1448 (2015). http:///oi.orr/100 109/
[11] Guo, J., Kurup, U., Shah, M.: safe to drive? an overview factors, metrics, and datasets for driveability assessment in autonomous driving. IEEE Transactions on Intelligent Transportation Systems 21(8), 3135-3151 (2020). https://doi.or//10 10.1 109/TTTS.2019 209.2222042
[12] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 770-778 (2016). http:://doi.rr//10 109/CVPR.20166
[13] Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Computation 98). 1735-1780 (1997). https://doi.or/ org.10.... 1 w62//eco..ooooooooooooooooooooeo 19t.e/.....999.99 1997.9.8.177
[14] Hong, F.T., Li, W.H.. Zheng, W.S.: Learning to detect important people in unlabelled images for semi- supervised important people detection. In: Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition. pp. 4145-4153 (2020). https://dii.or
[15] Hu, Z., Lv, C., Hang, P., Huang, C., Xing, Y.: Data-driven estimation driver attention using calibration- free eye gaze and scene features. IEEE Transactions on Industrial Electronics 69(2), 1800-1808 (2022). https://doi.org/1i
2021....002.3055000
[16] Hu, Z., Zhang, Y., Li, Q., Lv, C.: A novel heterogencous network for modeling driver attention with lulti-level visual content. IEEE Transactions on Intelligent Transportation Systems 23(12), 24343-24354 (2022). https///doi.org/1... wttp.//g//..1111/1TT22022222.22. 1009/TTSS.2022.3200000
[17] Itti. L.. Koch, C. Niebur, E.: A model of saliency- based visual attention for rapid scene anal- ysis. IEEE Transactions on Pattern Analysis and Machine Intelligence 20(11), 1254 1259 (1998). http://0o..oog//0119944447555550
10

---

[18] Kruthiventi, S.S.S., Ayush, K., Babu. R.V.: Deepfix: A fully convolutional neural network for
predicting human eye fixations. IEEE Transactions on Image Processing 26(9), 4446-4456 (2017).
https://ooi.org/1000 https//dd..og210010//72200212200 109/71120100.2700000000
[19] Kung. C.H., Yang. C.C., Pao, P.Y., Lu, S.W., Chen, P.L., Lu, H.C., Chen, Y.T.: Riskbench: A scenario-based benchmark for risk identification. arXiv preprint arXiv:23 (2023)
[20] Li, C., Chan, S.H., Chen, Y.T.: Droid: Driver-centric risk object identification. IEEE
Transactions on Pattern Analysis and Machine Intelligence 45(11), 13683- 13698 (2023). https://doi.org/10.. (09/TPAAM22023..22255
[21] Li, Gang, H., Ma, H., Tomizuka, M., Choi, C.: Important object identification with semi-supervised learning for autonomous driving. In: Proceedings of the IEEE International Conference on Robotics and Automation. pp. 2913 2919 (2022). https://doi.org/100 109/ICRA446399,22222 20229981332
[22] Li, J., Zhang. D., Meng. B., Chen, R., Tang, J., Wang, Y.: Enhancement of target feature regions and intention driven visual attention selection in traffic scenes. In: Proceedings of the IEEE Intelligent Vehicles Symposium. pp. 404 410 (2022). https://oi.orr//10 11000010000111191100002222222220
[23] Li, L., Han, J., Zhang, N., Liu, N., Khan, S., Cholakkal, H., Anwer, R.M., Khan, F.S.: Discrimina- tive co-saliency and background mining transformer for co-salient object detection. In: Proceedings of the IEEE/CVF Conferenee Computer Vision and Pattern Recognition. pp. 7247-7256 (2023). htpss//oo..o//100.119vcv9..... 109/CV955229 20232200002000000
[24] Li, W.H., Hong. F.T., Zheng. W.S.: Learning to learn relation for important people detection in still images. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4998-999999 5006 (2019). https://doi.coo org/or0///// 1 109/CVPR.200 201......0190055555 2019.00514
[25] Liu. Y., Zhang. J. Li. Y.. Hansen, P.. Wang, J.: compnn- collaborative interaction design of intelligent vehicle- case study of hmi of adaptive cruise control. In: Proceedings of the International Conference on Human-Computer Interaction, pp. 296-314 (2021)
[26] Malla, S. Choi, C.. Dwivedi, I., Hee Choi, J., Li, J.: Drama: Joint risk localization and captioning in driving. In: Proceedings of the TEFF/CVF Winter Conference on Applications of Computer Vision. pp. 1043-1052 (2023). https://doi.or//1 109/ 2023.001
[27] Nan, Z., Jiang, J., Gao, X., Zhou, S., Zuo w., Wei, P., Zheng, N.: Predicting driven attention via integrating bottom-up stimulus and top-down guidance. IEEE Transactions Image Processing 30, 8293-8305 (2021)
[28] Nan, Z., Shu, T., Gong, R., Wang. S., Wei, P.. Zhu, S.C., Zheng, N.: Learning to infer human attention in daily activities. Pattern Recognition 103, 107314 (2020)
[29] Nan, Z., Xiang, T:: Third-person view attention prediction in natural scenarios with weak information dependency and human-scene interaction mechanism. IEEE Transactions on Circuits and Systems for Video Technology 34(8), 6762-67777 667- (2024)
[30] Nitta, Y., Isogawa, M., Yonetani, R., Sugimoto, M.: Importance rank-Icarning of ohjects in urban seenes for assisting visually impaired people. IEEE Access 11, 62932-62941 (2023). http:///bbooo///00.0111ACCCCCCC (09/AACCESS.0223.33338
[31] Niu, Y., Ding, M., Zhang, Y., Ohtani, K., Takeda, K.: Auditory and visual warning information generation of the risk object in driving scenes hased on weakly supervised learning. In: Proceedings of the IEEE Intelligent Vehicles Symposium. pp. 1572-1577 (2022). http::/doi.org/10 109//V 2.22999977.222228778888
[32] Ohn Bar, E., Trivedi, M.M.: Looking at humans in the age of self-driving and highly automated vehicles. IEEE Transactions on Intelligent Vehicles 1(1), 90-104 (2016). http:://oi.org// 10. 109/TTV.2016.551 1067
[33] Ohn-Bar Bar, E., Trivedi, M.M.: Are all objects equal? deep spatio-temporal importance prediction in driving vidcos. Pattern Recognition 64, 425 436 (2017)
[34] Organization, W.H.: World health statistics 2023 (2023)
[35] Pal, A., Mondal, S. Christensen, H.I.: "looking at the right stuff" - guided semantic-gazz gaze for autonomous driving. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 118001 1889 (2020). https://doi.org// 10.1 109/CVPR4200022020000 2020.01199
[36] Qiu, Y.: Human-centered hmi design for level 3 automated driving takeover process. In: Proceedings of the IEEE International Conference on Intelligent Computing and Human- Computer Interaction, pp. 43 54 (2023). https://doi.org/10. http://0011011001111111110 222.CCC11187712222.1212 22....20....2....

---

[37] Ramanishka, V., Chen, Y.T., Misu, T., Saenko, K.: Toward driving scene understanding: A dataset for learning driver behavior and causal reasoning. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (2018)
[38] Sharma, N., Garg. R.D.: Real-time iot-based connected vehicle infrastructure for intelligent trans- portation safety, IEEE Transactions on Intelligent Transportation Systems 24(8), 8339-8347 8347 (2023). https//doi.org/1000 109/TTT...111.......1 109.110T.....2..2522777777
[39] Sidorenko, G.,
Thunberg, J.. Sjöberg. K., Fedorov, A., Vinel. A.: Safety of automatic emer- gency braking in platooning. IEEE Transactions on Vehicular Technology 71(3), 2319 2332 (2022). huts:://oii.ogg 100 109/TVT.2021111 3138939
[40] Singh, G., Akrigg. S., Maio. M.D., Fontana, V., Alitappeh, R.J., Khan, S., Saha. S., Jeddisaravi. K., Yousefi, F., Culley, J., Nicholson, T., Omokeowa, J., Grazioso, S., Bradley, A., Gironimo, G.D., Cuzzolin, F: Road: The road event awareness datasct for autonomous driving. IEEE Transactions Pattern Analysis and Machine Intelligence 45(1), 1036-1054 (2023). https://doi. http:://ooooog/1...
[41] Suzuki, T., Kataoka, H., Aoki, Y., Satoh, Y:: Anticipating traffic accidents with adaptive loss and large-scale incident db. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3521-3529 (2018). https://doi.orgg hrgp://ooooorg/1... / 109/CVPR.2018.00777
[42] Tang, X., Yu, Su, Modeling driver's visual fixation behavior using white-box repre- sentations. IEEE Transactions on Intelligent Transportation Systems 23(9), 15434-15449 (2022). https://doi.rr// 1
[43] Tian, H., Deng, T., Yan, H.: Driving as well as on a sunny day? predicting driver's fixation in rainy weather conditions via a dual-branch visual model. IEEE/CAÁ Journal of Automatica Sinica 9(7), 1335-1338 (2022). orgp 1
[44] Treisman, A.M., Gelade, G.: A feature-integration theory of attention. Cognitive psychology 12(1), 97-136 (1980)
[45] Wan, J., Li, X., Dai, H.N., Kusiak, A., Martinez-Gareía, M., Li, D.: Artifcilliitelligenne--eiiee cus- tomized manufacturing factory: Key technologies, applications, and challenges. Proceedings of the IEEE 109(4), 377-398 (2021). https://ooi.orr/10.. 10//PPRC.22000003448800
[46] Wu, T., Sachdeva, E., Akash, K., Wu, X., Misu, T., Ortiz, J.: Toward adaptive situational awareness support system for urban driving. In: 2022 IEEE Intelligent Vehicles Symposium, pp. 1073-1080 (2022)
[47] Xie, C., Xia, C., Ma, M., 7hao, 7., Chen, X., Li, J.: Pyramid grafting network for one-stage high resolution saliency detection. In: Proceedings of the IREE/CVF Conference Computer Vision and Pattern Recognition. pp. 11707-11716 (2022). https://doi.org org/ 110.///VPR2228..222....111
[48] Yu, S.Y., Malawade, A. Muthirayan, D., Khargonekar, P.P., Faruque, M.A.A.: Scene-graph augmented data-driven risk assessment of autonomous vehicle decisions. IEEE Transactions on Intelligent Transporta- tion Systems 23(7), 7941-7951 (2022). https://doi.org/100 109/TTSS 1.7734584
[49] Yurtsever, E., Lambert, J., Carballo, A.. Takeda, K.: A survey of autonomous driv- ing: Common practices and emerging technologies. IEEE Access 8, 58443-58469 (2020). https://do..rr/100 1 109/ACCESS STT.EESS..000.9900 2920.2983149
[50] Zhang, 7., Tawari, A., Martin, S., Crandall, D.: Interaction graphs for object importance estimation in on-road driving videos. In: Proceedings of the IEEE International Conference on Robotics and Automation. 200.0....44045222.99999977
pp. 8920-8922 (2020). https://doi.ooo http:://doi.or//100 1rg/10.111
[51] Zheng, T., Huang, Y., Liu, Y., Tang, W., Yang. Z., Cai, D., He, X.: Clmett: Cross layer refinement network for lane detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattemn Recognition. pp. 888 897 (2022). https:///oii org/ 1000///5...001//2228888820000000 20222000080022009977
[52] Zhou, H. Qiao, B., Yang. L., Lai. J., Xie, X.: Texture-guided saliency distilling for unsupervised salient object detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 7257-722 7267 (2023). https://doi.or////// wtt.::oo.......001////77772000.0
12

---

Appendix
This appendix provides specialized terms explanation, additional experimental results, limitations, and experimental details, which are organized follows:
A Explanation of Specialized Terms;
B Additional Experimental Results;
C Limitations;
D Experimental Details.
A Explanation of Specialized Terms
the low-level information extracted directly from the input images or video frames using backbone networks.
Bottom-up feature:
2) Top-down guidance: the high-level information including semantic understanding, prior knowl- edge, specific goals, etc.
3) Ego-car: the capturing video sequences that are used the input of the model.
Intention driving path: the path from the ego-car current position to the intention destination.
5) Intention behaviors: the actions that the driver intends to perform based on their goals (e.g., turning left, going straight, and turning right).
B Additional Experimental Results
B.1 Qualitative Comparison
GT
DV1
walting
constrained byn
Ons
Goal
2:1-1
Zhung Ohn-Bar
Figure 4: Qualitative comparison with baselines (i.e., Goal (8), Ohn- (33), and Zhang (50)). Red boxes represent important objects and green boxes denote unimportant objects.
Four scenarios (a)-(d) are illustrated in Fig. 4. In regular scenarios such as Fig. 4a, most methods function well. However, in complex scenarios, our method exhibits significant superiority. For example, in Fig. 4b, the white car on the opposite lane poses no immediate threat since should obey the traffic rule to not drive across the solid lane marking. Baselines falsely predict it as an important object as they neglect the influence of traffic rule on object importance, while our method provides the correct estimation by considering the object-lane interaction relation in TRG. In Fig. 4c, the ego-car turning left, and the vehicle on the right side important because its driving path will risky collide with the intention driving path of ego-car. Our method successfully predicts the important object under guidance of driver intention, while other methods fail. In Fig. 4d, pedestrian waiting to cross the road. Her intention walking path collides with ego-car's 's intention driving path, thus the pedestrian is important. Our model correctly classifies her as important thanks the consideration of bottom-up feature and top-down guidance.
13

---

B.2 Ablation Study on Object Feature Extraction (OFE)
To validate the rationality of OFE, we conduct three experiments. #1: only object spatial fea- ture is used; #2: only object temporal feature is used; #3: both object features are used. The results in Tab. 6 indicate that both spatial and temporal feature Table 6: Ablation study on OFE. of objects serve as valid foundations for accurately evaluating object importance. Removing either of them will lead the performance decreasing. validating the reasonableness of OFF module. The reasons self-evident, traffic scenarios highly dynamic and diverse, thus object temporal feature, which reflects motions and behaviors, is significant for object importance estimation. At the same time, object spatial feature conveys rich information such as size, distance, and orientation, thus it is also significant for object importance estimation.
Method
Spat. Temp. APT FIf
#1
35
#2
14
15
B.3 Hyperparameter Selection
We perform the experiments hyperparameter selection, and the results are reported in Tab. 7. In the hyperparameter selection for parameters a and we choose the optimal values, a = 1, 1.5. as the hyperparameters for our model. In the experiments for selecting the hyperparameter observed that the impact of a is minimal across the three tested values, indicating that our method robust.
is
Table 7: Ablation studies on hyperparameter selection.
Parameter
AP
Flf
Parameter
AP1
FIf
1,b 2.5
51
41
0.1
57
45
1, 2
a 0.01
55
52
1,b 1.5
a 0.001
60
54
= 1,b =
49
48
C
Limitations
Figure 5: Failure examples. Top row is GT and bottom row object importance estimation.
Poor lane marking detection results will limit the performance of our model. In the experiments, we find two kinds of failures. First, as shown in Fig. 5a, lane markings of current lane are falsely detected, thus the white on the right side is supposed to locate in front of ego-car, leading to the false estimation for the white car. Second, as shown Fig. 5b, the ego-car moves at slow speed and lane markings are detected, thus the model estimates the in front of the ego-car as an unimportant parking car. Additionally, considering the effect of lane markings on on-road object importance is small but important step towards modeling the traffic rule for this task. In the future, more traffic rule needs to be considered.
Currently, our model only considers the effect of three types of driver intentions (i.e., turning left, turning right, going straight) on object importance estimation. However, real-world driving scenarios much more complex. In future work, fine- greinerrind intentions need to be modeled.
14

---

D Experimental Details
D.1 Model Structure Details
Specific details of our model components are introduced in Tab. 8.
Table 8: Network architecture of our model. For Region of Interest pooling layer (ROI pooling), we list the output shape. For average pooling (Avg pooling), we list pooling scale. For Long Short-Term Memory network (LSTT),, we list input and output dimension and layer number. For multi-head self- attention layer (MHSA), we list the hidden size and the head number. For multi-head cross-attention layer (MHCA), we list the hidden size and the head number. For linear layer (Linear), we list the input and output dimension. For Layer Normalization layer (LN), list the channel dimension. For multi-layer perceptron network (MLP), we list the input channel dimension and each hidden channel dimension. Note that we use different background colors in the table distinguish between different modules in our model, where orange represents the OFE module, blue represents the DISG module, green represents the TRG module, and gray represents the OIE module.
Layer
Details
1-17
ResNer18(The last FC layer is removed)
18-34
ResNet18(The last FC layer is removed)
35
ROI pooling I0)
Avg pooling(16)
37
ROI pooling(10) (10)
38
Avg pooling(16)
39-40
LSTM(512 x x0x 10, 256, 2)
41
MHSA 8)
42-43
LSTM(512x 10x 10, 256, 2)
44
MHSAA 8)
45
Lincar(512, 256), LN(256), ReLU
46
MHCAA(1024 8)
47
Linear x 10x 10, 256), LN(256), ReLU
48
Lincar(20x4 256), LN(256), ReLU
49
MHCA(256, 8) MLP(256, (1)). Sigmoid
50
51-52
MLP(256. [128, 2)). Softmax
D.2 Implementation Details
Before the training and inference stages, we utilize CLRNet (51) model with backbone of ResNet101
and pretrained on the CULane dataset to get the L in Eq. (10), and the G in Eq. (5) are generated by applying DeepLabv3 (3). We use OpenCV libary to get the M in Eq. (2). During the training and inference stages, we resize V and M in Eq. (2), and G in Eq. (5) to 320 x 320 (f.e., W x H=320 320). We set the W/=10, H0=10, C=512, and C'=256. The a and b in Eq. (6) are set as 1 and 1.5, respectively. Then, the 3 in Eq. (7) is set as 2.2 since the ego-car undergoes steering when the angular velocity is around + 2.2 based on the statistical analysis of the IMU data. In addition, the size of m,, m,, and m, is 10 10 (excluding the channel dimension), which is targeted to align with the size of f. (Eq. (8)) The length of video clip is set as 16. We use SDG optimizer with a weight decay of 5e- and momentum of 0.9. We set the batch size as 8, and use the cosine learning strategy with an initial learning rate of le 44 Our model implementation is based on Py PyTorch, and experiments are conducted using an NVIDIA RTX3090 GPU,
D.3 Object Bounding Boxes Are Assumed Known
We assume the ground truth of object bounding boxes are given. The purpose is to focus on the on-road object importance estimation while minimizing the influencing factors from the upstream object detection task. This setup is consistent with the existing on-road object importance estimation method (21). Additionally, this setup is reasonable since current object detection models are highly advanced and capable of detecting most objects on the road.
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The contribution 1) claimed is reflected in 3; The contribution 2) claimed is reflected in $ 5.2; The contribution 3) we claimed is reflected in 5.2.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We dicuss the limitations of our work in C.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
16

---

Justification:
This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We clearly describe our model 4, and the model details can be found in $ D.1. The dataset proposed is fully described in 3.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
17

---

Answer: [Yes]
Justification: This paper contributes new dataset and a model, the dataset will be publicly released attpss//gitae com/CQU-aee ADHRI-Lab/TI
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (uttps://nips.. ec/
oolicyyyyyy) for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: halliii/ggiee//ee
/ /nips...
ssiisnin ssissssliccyyyy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
The experimental details is specified in $ D.2 and B.3.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Error bars are not reported because it would be too computationally expensive.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
18

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The computer resources we used are provided in D.2.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https /ttprr////e/...eeeeeee cc cwtppcccp//ctteeeeeeee cttte//tetiiiieeeeiiiieeeeeeeeee
Answer: [Yes]
Justification: The research we conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our research on-road object importance estimation focuses only on improving the accuracy of importance estimation for road safety purposes. As such, it does not have direct societal impacts beyond the scope of enhancing these technical capabilities.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
19

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: Our paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [No]
Justification: We are unable to find the license for the dataset and code we used, but they are all open source, and cite them in the original paper correctly.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
20

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The dataset proposed is well documented and is the documentation provided alongside the assets.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/codeemoodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
21


Reviews:
[{"review_id": "RvO3e7ptG9", "reviewer": "Reviewer_xrYx", "summary": "This work addresses the issue of estimating the importance of on-road objects using video sequences from a driver’s perspective, a critical task for enhancing driving safety. The authors introduce the Traffic Object Importance (TOI) dataset, which is significantly larger and more diverse than existing datasets, and propose a novel model that integrates multi-fold top-down guidance factors—driver intention, semantic context, and traffic rules—with bottom-up features for more accurate importance estimation. Experimental results demonstrate that the proposed model significantly outperforms state-of-the-art methods in on-road object importance estimation.", "questions": "1. How many annotators were involved in the annotation procedure for the dataset? Can the authors provide detailed examples of their double-checking and triple-discussing annotation mechanisms?\n\n2. Were the annotators familiar with German traffic rules, given that the dataset was collected in Germany (KITTI dataset)? How was the expertise of the annotators in relation to German traffic laws ensured and validated?\n\n3. Have the authors considered building the first benchmark using their dataset with existing object detection methods, such as YOLO? What were the performance outcomes of these existing methods when applied to the new task?\n\n4. Can the authors provide more detailed statistics about the dataset, such as the number of important objects in different categories? How do these statistics compare to other datasets in the same domain?", "limitations": "yes the authors mentioned it in the appendix", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The introduction of the Traffic Object Importance (TOI) dataset, which is significantly larger and more diverse than existing datasets, provides a robust foundation for training and evaluating models in on-road object importance estimation, thereby addressing a major limitation in the field.\n\n2.  The proposed model effectively integrates multi-fold top-down guidance factors—driver intention, semantic context, and traffic rules—with bottom-up features, which showed good performance for the TOI task.", "weaknesses": "1. Lack of description of the annotation details. \nHow many annotators are involved in the annotation procedure? It would be good if the authors can provide some annotation procedure samples regarding the double-checking annotation mechanism and the triple-discussing annotation mechanism.\n\n2. It seems this annotation will be varied according to different traffic rules. Since KITTI is collected in Germany, the annotators should be familiar to germany traffic rules. However the authors did not mention this information in their submission, thereby the label quality is doubtful.\n\n3. The authors are encouraged to build up the first benchmark based on the proposed dataset by using various existing object detection methods, e.g., Yolo, with the proposed head or simpler head. It is interesting to see how the existing object detectors work on this new task.\n\n4. More statistics of the dataset are encouraged to be given, e.g., the number of important object of different categories, etc.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# To Reviewer xrYx\n\nThank you very much for your positive comments on our proposed dataset addressing a major limitation in the field and our model showing the good performance.\n\n> ***Q1:**\nLack of description of the annotation details. How many annotators are involved in the annotation procedure?*\n\n**A1:**\nThe details of annotating a sequence is shown in Fig. R2 of the rebuttal PDF file. We also illustrate the annotation process in Fig. R1 of the rebuttal PDF file, from which we can observe the annotation process involves three types of annotators, namely the first, second, and third annotator. Six experienced drivers are recruited as volunteers. It is worth noting that every volunteer is able to act as the role of the first, second, or third annotator.\n\n--- \n\n> ***Q2:**\nIt would be good if the authors can provide some annotation procedure samples regarding the double-checking annotation mechanism and the triple-discussing annotation mechanism.*\n\n**A2:**\nTo illustrate the double-checking and triple-discussing annotation mechanisms, we provide a schematic showing our annotation process in Fig. R1 of the rebuttal PDF file. To better explain double-checking and triple-discussing mechanisms, an example is provided, please see the image (with a van and a cyclist inside) in the box \"discuss together\" in Fig. R1. The first annotator annotates both van and cyclist as important objects, considering both objects are along the driving path. When the second annotator is checking the annotation, claiming that the annotation for the cyclist is reasonable but the annotation for the van is disputed, since van is far from the driver (though it is along the driving path) and the driver will only pay attention to the cyclist. At this time, the first and second annotators discuss together to convince each other. If they can not reach an agreement, the triple-discussing mechanism is activated, and the third annotator join the discussion. The third annotator holds the idea that the van is also a potential object that might affect the driving safety, thus the final annotation is that both van and cyclist are important objects.\n\n---\n\n> ***Q3:**\nIt seems this annotation will be varied according to different traffic rules. Since KITTI is collected in Germany, the annotators should be familiar to germany traffic rules. However the authors did not mention this information in their submission, thereby the label quality is doubtful.* \n\n**A3:**\nDuring our annotation process, we focus primarily on universal traffic rules (e.g., vehicles should not drive across solid lanes). Therefore, the traffic rule imposes little effect on annotations. Factually, we find the main effect comes from that many objects simultaneously exist in an image. Thus, to guarantee the reliability of annotations, the first annotator only annotates one object at each time of observing the whole sequence. The annotation for a sequence is finished until all objects are annotated, as illustrated in Fig. R2 of the rebuttal PDF file. In addition, in existing datasets, although multiple annotators perform the annotations, it is not mentioned that the initial annotations are checked by other annotators. Differently, the double-checking and triple-discussing annotation mechanisms are introduced to further guarantee the quality of our annotations. We hope our explanation alleviates your concern.\n\n---\n\n> ***Q4:**\nThe authors are encouraged to build up the first benchmark based on the proposed dataset by using various existing object detection methods, e.g., Yolo, with the proposed head or simpler head. It is interesting to see how the existing object detectors work on this new task.*\n\n**A4:**\nWe have conducted the experiment that uses YOLO to build up a benchmark. To make YOLO suitable for our task, we adjust the output dimensions of the final fully connected layer in YOLO detection head to 2 to indicate important and unimportant. The experimental results are reported in Tab. R1 of the rebuttal PDF file.\n\n---\n\n> ***Q5:**\nCan the authors provide more detailed statistics about the dataset, such as the number of important objects in different categories? How do these statistics compare to other datasets in the same domain?*\n\n**A5:**\nWe present more statistics about the dataset in Fig. R4 of the rebuttal PDF file, including the statistics on different driving intentions and the number of important objects in different object categories. Unfortunately, due to the missing information in other datasets, we could not provide the corresponding information of other datasets.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WwItFq6FX2", "reviewer": "Reviewer_8N7U", "summary": "This paper presents a novel dataset for on-road object importance estimation. More data about which objects are important for self-driving is included and is promised to be released. Moreover, a novel method that integrates driven intention, semantic context, and traffic rule is devised to tackle the related problem. The paper is well-written.", "questions": "(1) Regarding the task, my major concern is the definition of importance. It is shown that surrounding objects that follow the traffic rules are not considered as important. Only the objects ahead of the car or have an intersection with the ego-car's direction are important. I doubt whether this is strictly appropriate. For example, if a pedestrian walking along the road, he/she will not be considered as important. However, what if this pedestrian suddenly steps into the road ahead, potential collisions would happen. Therefore, I think a nearby walking pedestrian should be considered as important or at least recognized into a third category like \"needs care\". I wonder how the authors solve this problem in the dataset.\n(2) Regarding the driver's intention, it is indeed difficult to define appropriately. The authors have mentioned this in the paper, but the strategy introduced to accommodate this is still not clear to me. The authors mentioned learning the intention values based on driving behaviors, but how do we know the driving behaviors? Are these behaviors (e.g. turning left) already provided in the dataset? \n(3) More visualization about the labels and method comparisons are better to be presented for more clarity.", "limitations": "The authors have mentioned limitations.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "A new dataset is introduced with rich data and labels. The presented method is novel and shown to be effective for the studied problem. Details about the dataset and the method are comprehensive and technically sound. Results are also promising.", "weaknesses": "Some of the concepts lack sufficient details to explain. See questions below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# To Reviewer 8N7U\n\nThank you very much for your positive comments and we appreciate your thoughtful feedback and suggestions.\n\n> ***Q1:**\nRegarding the task, my major concern is the definition of importance. It is shown that surrounding objects that follow the traffic rules are not considered as important. Only the objects ahead of the car or have an intersection with the ego-car's direction are important. I doubt whether this is strictly appropriate. For example, if a pedestrian walking along the road, he/she will not be considered as important. However, what if this pedestrian suddenly steps into the road ahead, potential collisions would happen. Therefore, I think a nearby walking pedestrian should be considered as important or at least recognized into a third category like \"needs care\". I wonder how the authors solve this problem in the dataset.*\n\n**A1:**\nOur core idea to define the object importance is whether an object affects the safe driving. With this core idea in the mind, we summarize four types of importance evaluation guidelines, including object attribute, driver intention, traffic semantic context, and traffic rule, as detailed in Line50-63 in the original version. Considering the certain individual guideline might not be strictly applicable for some traffic scenarios, we introduce the double-checking and triple-discussing annotation mechanisms to comprehensively make use of four mutually-dependent guidelines to handle disputed and complex scenarios. Your above mentioned scenario is complex. In this scenario, object attribute (e.g., the distance and object category) is the dominant guideline. The closer an object is to the ego-car, the more important it is. In addition, pedestrian is a special object category, the behavior of a pedestrian is more random than a vehicle, and traffic rule imposes fewer constraints on a pedestrian than that on a vehicle. Moreover, the pedestrian is a crucial category in traffic scenes. Under the similar condition, the pedestrian is more important than other categories. Therefore, a nearby walking pedestrian should be annotated as important.\n\n---\n\n> ***Q2:**\nRegarding the driver's intention, it is indeed difficult to define appropriately. The authors have mentioned this in the paper, but the strategy introduced to accommodate this is still not clear to me. The authors mentioned learning the intention values based on driving behaviors, but how do we know the driving behaviors? Are these behaviors (e.g. turning left) already provided in the dataset?*\n\n**A2:**\nActually, the annotations of driving behaviors are not provided. However, the KITTI dataset contains detailed IMU/GPS data, and we can determine whether the vehicle is turning left, turning right, or going straight based on the vehicle's lateral angular velocity derived from the IMU data.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BtKYEriAyA", "reviewer": "Reviewer_Yrjv", "summary": "This paper collects a new large-scale dataset and proposes a novel method that integrates multi-fold top-down guidance with the bottom feature to address the problem of on-road object importance estimation. Specifically, the dataset is almost three times larger than the current publicly dataset for on-road object importance. In addition, this paper considers an adaptive mechanism for object-lane interaction, effectively modeling the impact of traffic rules on object importance. Experiments on several benchmarks validate the effectiveness of the proposed method.", "questions": "(1) Can the proposed method be applied to surrounding view images? I suggest that the authors should consider the application on the current perception pipeline for vision-based autonomous driving pipeline.\n\n(2) I suggest that the authors should analyze the latency of the proposed method, which determines whether the method can be integrated into the practical driving scenarios.", "limitations": "The proposed method only considers the effect of three types of driver intentions on object importance estimation, which is not sufficient for complex driving scenarios. I carefully checked the paper and found no potential negative societal impact.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper makes several key contributions and demonstrates strengths for on-road object importance estimation\n\n(1) This paper introduces a novel, extensive dataset, set to be released to the public, which is nearly three times the size of the current largest public dataset. \n\n(2) The method is well-motivated and straightforward. It estimates the importance of objects on the road, integrating various top-down guidance factors with bottom-up features, marking the first of its kind.\n\n(3) The proposed method addresses the pivotal role of traffic rules in estimating object importance, an aspect previously overlooked by existing methods. It successfully encapsulates this concept through an innovative, adaptive mechanism for object-lane interaction.", "weaknesses": "This paper has also two weaknesses: \n\n(1) The paper does not provide a detailed discussion on the computational efficiency of the proposed method, which is crucial for real driving scenarios. Moreover, it is recommended to compare the model parameters and latency with other methods.\n\n(2) Another concern lies in the practicality of the method. This method and the proposed dataset are both for single-camera scenarios, but in real autonomous driving scenarios, surrounding view is a more widely used type and a safer option. Will the proposed method also work well in the surrounding view?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# To Reviewer Yrjv\n\nThank you very much for your positive comments and we appreciate your thoughtful feedback and suggestions.\n\n> ***Q1:**\nThe paper does not provide a detailed discussion on the computational efficiency of the proposed method, which is crucial for real driving scenarios. Moreover, it is recommended to compare the model parameters and latency with other methods.*\n\n**A1:**\nAccording to your suggestion, we compute the latency and parameters of our method and other methods, as reported in Tab. R1 of the rebuttal PDF file. Our model asks for relatively large parameters (i.e., 173M) because multi-fold top-down guidances are involved in the model. However, 173M parameters only occupy a fraction of storage space. Therefore, the parameters will not hinder the application of our method. In comparison, the latency of a model largely affects its deployment on practical platforms. We can observe that, the latency of our method is only longer than that of Ohn-Bar. We note that Ohn-Bar is an early model using relatively simple VGG backbone, thus presenting the shortest latency.\n\n---\n\n> ***Q2:**\nAnother concern lies in the practicality of the method. This method and the proposed dataset are both for single-camera scenarios, but in real autonomous driving scenarios, surrounding view is a more widely used type and a safer option. Will the proposed method also work well in the surrounding view? Can the proposed method be applied to surrounding view images? I suggest that the authors should consider the application on the current perception pipeline for vision-based autonomous driving pipeline.*\n\n**A2:**\nThanks for this insightful comment. Surrounding view images in autonomous driving can be broadly categorized into three types: front view, side view, and back view. Our method might not be applicable for the side view images, since DISG (Driver Intention and Semantics Guidance) module makes use of the intention guiding mask, while this mask is mainly designed for the front view. Our method should be suitable for back view images, since they are quite similar to the front view images. Thank you again for this comment, which inspire us to extend the model to be applicable for all surrounding views in the future.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "V6EMXkI2kv", "reviewer": "Reviewer_mvxo", "summary": "1.This paper this contributes a new large-scale dataset named Traffic Object Importance (TOI) to addresses the problem of on-road object importance estimation, which utilizes video sequences captured from the driver’s perspective as the input.\n2.The author also proposes a model that integrates multi-fold top-down guidance with the bottom-up feature.", "questions": "1.I consider whether 16 frames constitute interval sampling or continuous sampling, and how many types of intentional behaviors can be expressed using 16 frames in the paper.\n\n2.The author may can add another evaluation metric for the experiment.", "limitations": "Yes.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1.This paper describes in great detail the specialized methodology and the structure of the models.\n\n2.The scarcity of large-scale publicly available datasets hinder the development of on-road object importance estimation.\n\n3. This paper considers the effect of traffic rule on object importance and successfully models this abstract concept by proposing an adaptive object-lane interaction mechanism.", "weaknesses": "1.In page 3 the author mentions that the traffic rule is crucial for object importance and focus on the traffic line rules , but the influence of traffic rules is varied, such as signalization. Therefore, in page 4 of table 1, the author is able to provide statistics on the scenario categories of TOI dataset and the traffic rule constraints within the dataset in experiment. \n\n2.In page 6, the author uses three common intention behaviors in driving to reflect the driver intention (i.e., turning left, going straight, and turning right). Since the video clip length is set at 16 frames, it is important to clarify if each of the three intentions corresponds to individual frames with the 16-frame clip cut during the training and testing phases, or if multiple intentions are present within the 16 frames. The authors should further elaborate and provide the proportion of each intention in the dataset.\n\n3.Insufficient evaluation of indicators in the experimental section. The author may add another evaluation indicator.\n\n4.The section three can include a schematic diagram of the annotation process for the dataset.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# To Reviewer mvxo\n\nThank you very much for your positive comments and we appreciate your thoughtful feedback and suggestions.\n\n> ***Q1:**\nIn page 3 the author mentions that the traffic rule is crucial for object importance and focus on the traffic line rules, but the influence of traffic rules is varied, such as signalization. Therefore, in page 4 of table 1, the author is able to provide statistics on the scenario categories of TOI dataset and the traffic rule constraints within the dataset in experiment.*\n\n**A1:**\nThanks for this kind suggestion. We have added scenario categories and traffic rules to Table. 1 in our revised manuscript.\n\n---\n\n> ***Q2:**\nIn page 6, the author uses three common intention behaviors in driving to reflect the driver intention (i.e., turning left, going straight, and turning right). Since the video clip length is set at 16 frames, it is important to clarify if each of the three intentions corresponds to individual frames with the 16-frame clip cut during the training and testing phases, or if multiple intentions are present within the 16 frames. I consider whether 16 frames constitute interval sampling or continuous sampling, and how many types of intentional behaviors can be expressed using 16 frames in the paper. The authors should further elaborate and provide the proportion of each intention in the dataset.*\n\n**A2:**\nWe apologize for any confusion. In both training and testing phases, 16 frames inputted to the model are continuous images. 16 frames occupy 1.6s in the temporal dimension, which is actually in a short time. Therefore, they share the same kind of intention. According to your suggestion, as shown in Fig. R4 of the rebuttal PDF file, we provide the proportion of each intention in the whole dataset, and we also provide the statistics of driving intentions in each scene.\n\n---\n\n> ***Q3:**\nThe author may can add another evaluation metric for the experiment.*\n\n**A3:** \nThank you very much for this suggestion. According to your suggestion, we add another evaluation metric (i.e., accuracy), as shown in Tab. R1 of the rebuttal PDF file.\n\n---\n\n> ***Q4:**\nThe section three can include a schematic diagram of the annotation process for the dataset.*\n\n**A4:**\nThank you very much for this suggestion. We illustrate the annotation process, as shown in Fig. R1 of the rebuttal PDF file.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
# General Response
We thank reviewers for their valuable feedback. We are encouraged by the reviewers’ positive comments on our work. Specifically, they find our model novel (8N7U) and effective (xrYx, 8N7U), our idea well-motivated (Yrjv), our proposed dataset sound (8N7U), our paper detailed (8N7U, mvxo) and well-written (8N7U). Reviewer xrYx remarks that *"provides a robust foundation for training and evaluating models in on-road object importance estimation, thereby addressing a major limitation in the field"*.
Reviewer Yrjv comments that *“The proposed method addresses the pivotal role of traffic rules in estimating object importance”*.
 
We will provide detailed responses to reviewers' questions and feedback, hoping to address any confusion and concern.

Meta-review: ## Summary 

The Traffic Object Importance (TOI) dataset is introduced to estimate the importance of on-road objects from a driver's perspective, a crucial task for enhancing driving safety. The dataset is larger and more diverse than existing ones, and a novel model is proposed that integrates driver intention, semantic context, and traffic rules with bottom-up features for more accurate estimation. Experimental results show that the proposed model significantly outperforms state-of-the-art methods in on-road object importance estimation. The paper also introduces an adaptive mechanism for object-lane interaction.

## Strengths
-  Introduces a larger and diverse dataset for on-road object importance estimation.
- Integrates top-down guidance factors (driver intention, semantic context, traffic rules) with bottom-up features.
- Shows promising performance for the TOI task.
- Addresses the pivotal role of traffic rules in estimating object importance.
- Proposes an adaptive object-lane interaction mechanism.

## Weaknesses

- Lack of detailed annotation details, including the number of annotators involved and the annotation procedure samples.
- Annotation may vary according to different traffic rules, which could affect label quality.
- Definition of importance is unclear, with only objects ahead of the car or with an intersection with the ego-car's direction considered important.
- Driver's intention definition is difficult to define, with unclear strategies for learning intention values based on driving behaviors.
- More statistics of the dataset are needed, such as the number of important objects of different categories.

## Conclusion

The paper proposed (i) a dataset that is larger and more diverse than existing ones, and (ii)  a novel model  that integrates driver intention, semantic context, and traffic rules with bottom-up features for more accurate estimation. Therefore, the recommendation is the acceptance of the paper but it should include a benchmark with the current object detectors.
Decision: Accept (poster)