On-Road Object Importance Estimation: A New Dataset and A Model with Multi-Fold Top-Down Guidance

Authors: zhixiong  nan
yilong  chen
tianfei  zhou
tao  xiang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-12 04:44:36

Abstract: This paper addresses the problem of on-road object importance estimation, which utilizes video sequences captured from the driver's perspective as the input. Although this problem is significant for safer and smarter driving systems, the exploration of this problem remains limited. On one hand, publicly-available large-scale datasets are scarce in the community. To address this dilemma, this paper contributes a new large-scale dataset named Traffic Object Importance (TOI). On the other hand, existing methods often only consider either bottom-up feature or single-fold guidance, leading to limitations in handling highly dynamic and diverse traffic scenarios. Different from existing methods, this paper proposes a model that integrates multi-fold top-down guidance with the bottom-up feature. Specifically, three kinds of top-down guidance factors (i.e., driver intention, semantic context, and traffic rule) are integrated into our model. These factors are important for object importance estimation, but none of the existing methods simultaneously consider them. To our knowledge, this paper proposes the first on-road object importance estimation model that fuses multi-fold top-down guidance factors with bottom-up feature. Extensive experiments demonstrate that our model outperforms state-of-the-art methods by large margins, achieving 23.1% Average Precision (AP) improvement compared with the recently proposed model (i.e., Goal).

Main paper: 
 introduction: 
 According to the World Health Statistics of WHO (34) , road traffic injuries account for a significant 29% of all injury deaths, with nearly 1.3 million people losing their lives in traffic accidents annually. Accurately estimating the importance of on-road objects can pave the way for safer (e.g., automatic emergency braking (45; 39) ) and smarter driving systems (32; 25; 38; 49; 4; 36; 11) , potentially preventing numerous accidents.Although on-road object importance estimation presents significant research value, it has not been widely explored. One main reason is the scarcity of publicly-available large-scale datasets in the community. Specific for the on-road object importance estimation task, the only one publiclyavailable dataset is Ohn-Bar et al. (33) , which contains 3,187 frames, 8 scenes, and 16,076 object importance annotations. Such small-scale dataset supports to train small and less complex models. However, traffic scenes are dynamic and diverse, asking for complex models to handle various traffic situations. Some researchers have recognized this dilemma and propose some datasets (8; 21; 46) . Unfortunately, these dataset are not publicly-available, thereby the dilemma has not been fundamentally addressed. To fundamentally address this dilemma and push forward the advancement of on-road object importance study, this paper will release a large-scale dataset (named as TOI, Traffic Object Importance) containing 9,858 frames, 28 scenes, and 44,120 object importance annotations. Compared to Ohn-Bar (33) , TOI achieves a 3.1-fold increase in frames count, a 3.5-fold increase in scene count, and a 2.7-fold increase in object count.From the perspective of methodology, some methods have been proposed (21; 8; 50; 33) . However, these methods are relatively simple, exhibiting the low performance when confronting to challenging traffic scenarios. This motivates us to think about a question: why can not existing methods perform well? The potential reason is that existing on-road importance estimation methods underestimate the complexity of traffic scenarios, individual bottom-up mechanism (50) (assuming important objects are the objects with salient color, texture, size, etc.) or simple top-down guidance mechanism (8; 21) (fusing the bottom-up information with a certain type of top-down information such as semantics, ego-car trajectory, driving task (27) , etc.) can hardly address dynamic and diverse scenarios. Therefore, a smarter model is needed. Inspired by the fact that a human driver can accurately estimate object importance in challenging situations, this paper attempts to design a model by analysing the human reasoning mechanism during estimating object importance. To this end, the primary question is "what essentially crucial factors are considered when a human driver is estimating the importance of objects?". Firstly, the attributes (e.g., size, color, and texture) of the object is considered. For example, when a truck with the big size and a car with the small size simultaneously appear in front of the ego-car, the truck is more important since it imposes bigger impact on the driving. Secondly, the driver intention is considered. The objects that will riskly collide with the ego-car intention driving path or the objects locating on the ego-car intention driving path present high importance, as shown in Fig. 1a . Thirdly, overall semantic context of the whole traffic scene is considered. A human usually pay more attention on the objects in drivable areas rather than the objects in undrivable areas. As shown in Fig. 1b , the person riding a bicycle in undrivable areas is unimportant. Fourthly, traffic rule is considered. Most of traffic participants obey the traffic rule, thus the traffic rule is an critical factor for a human to estimate object importance. For example, as shown in Fig. 1c , when there exists a lane marking between the oncoming car and the ego-car, the human driver may consider the oncoming car as unimportant. In contrast, if there is no lane marking between them, the importance of the oncoming car significantly increases. The traffic rule is crucial for object importance estimation. However, none of existing methods utilizes traffic rule to estimate on-road object importance.Based on the above observations, we propose a model with multi-fold top-down guidance including driver intention, semantic context, and traffic rule. As far as we know, it is the first on-road object importance estimation model that fuses multi-fold top-down guidance factors with the bottom-up feature. The proposed model consists of two kinds of pathways (i.e., bottom-up pathway and topdown pathway). Bottom-up pathway and top-down pathway are fused to estimate object importance. Specifically, in the top-down pathway, the top-down guidance factors of driver intention and semantic context are involved in the proposed Driver Intention and Semantics Guidance (DISG) module, and traffic rule is modeled in the proposed Traffic Rule Guidance (TRG) module. In the bottom-up pathway, Object Feature Extraction (OFE) module is proposed to extract object features in both spatial and temporal dimensions.A series of comparison and ablation studies are conducted on a public dataset (33) and our TOI dataset. The comparison experiment results show that our model has a solid advantage over the baselines. The ablation study results validate the effectiveness of our proposed interactive bottom-up & top-down fusion framework and multi-fold top-down guidance modules (i.e., DISG and TRG).The contributions of this paper are as follows. 1) This paper contributes a new large-scale dataset, which will be publicly released. This dataset is almost three times larger than the current publicly available public dataset (33) . 2) This paper contributes an object importance estimation model. As far as we know, it is the first on-road object importance estimation model that integrates multi-fold top-down guidance factors with the bottom-up feature. 3) The traffic rule is crucial for object importance estimation. However, none of existing methods utilizes traffic rule to estimate on-road object importance. This paper considers the effect of traffic rule on object importance and successfully models this abstract concept by proposing an adaptive object-lane interaction mechanism. 
related works: 
 On-Road Object Importance Estimation Related Datasets. Currently, the primary dilemma of research on the on-road object importance estimation is the lack of sufficient data. Among existing datasets relevant to autonomous driving perception tasks, only a few meet the data requirement of providing images from the driver's first-person perspective while also including object importance level labels. Ohn-Bar et al. (33) are the first to define the problem of on-road object importance estimation, and they propose a small-scale publicly available dataset, which contains 8 scenes. Gao et al. (8) and Li et al. (21) have significantly increased the number of scenes. However, their datasets are not publicly available, thus the contributions to the community is limited. Datasets (19; 41; 48) are with detailed annotations such as ego's reaction and road topology. They have great potential to advance the development of on-road object importance estimation. However, currently, they lack object importance level labels and cannot be directly applied to this task.On-Road Object Importance Estimation Related Methods. Currently, on-road object importance estimation methods can be divided into two categories: 1) methods solely utilizing bottom-up feature; 2) methods utilizing single-fold top-down guidance.The methods solely utilizing bottom-up feature focus on the visual attributes of the objects. The bottom-up processing method is initially introduced in (44), and Itti et al. (17) propose one of the first bottom-up mechanism based models. Following this, many researchers are inspired by this concept (43; 15; 18; 35) . Zhang et al. (50) introduce a model, which solely relies on RGB clips for object importance estimation. This model employs graph convolutions to characterize the interactions among on-road objects. Nitta et al. (30) develop a model that extracts temporal features from optical flow images to infer the states of moving objects. The optical flow images are also used in Malla et al. (26) to assess the states of moving objects. The bottom-up methods can also be found in the works (33; 52; 47; 23; 14; 24) . Although the bottom-up feature is crucial for importance estimation, the methods solely rely on bottom-up feature can not function well in the complex scenarios.The importance of an object is influenced by many factors such as driver intention, which cannot be fully utilized through bottom-up methods. However, current methods are relatively simple and relies on single-fold guidance. Niu et al. (31) utilize a Transformer with shared weights to identify high-risk objects and generated semantic warning sentences. Li et al. (21) investigate the impact of driver intention, employing the action and trajectory data of the ego-car as additional supervisory signals in auxiliary tasks to enhance model performance. Gao et al. (8) utilize the driver's goal to estimate object importance. A cause-effect problem was formulated in (20) , which introduced a model to estimate the risky object by considering its potential impact on the driver's behavior. Tang et al. (42) provide a more comprehensive understanding of how driver intentions under different tasks affect the driver attention. Single-fold top-down guidance can also be found in the works of attention prediction task (7; 16; 22; 6; 1; 29; 5; 28) . However, none of these methods utilizes multi-fold top-down guidance factors to estimate the on-road object importance. 
a new dataset: toi: 
 We thoroughly review existing datasets for on-road object importance estimation as well as the datasets for the related tasks, and provide a summary in Tab. 1. The datasets for the related tasks (e.g., risk assessment (37; 48; 19) , accident anticipation (41) , and situation awareness (40) ) do not include object importance labels, making them unsuitable for object importance estimation task. Most datasets (e.g., (21; 8) ) for object importance estimation are not publicly available. The only publicly available dataset is Ohn-Bar (33) , but it is a small scale dataset. In response to the scarcity of publicly-available large-scale datasets for on-road object importance estimation, we contribute The double-checking annotation mechanism operates as follows. Initially, the first annotator (an experienced driver) labels the object importance at every 10 frames. To guarantee the reliability of annotations, the first annotator only selects one object as the important object at each time of observing the whole 10 frames. The annotation for these 10 frames is finished until all important objects are annotated, then the annotator moves to the next set of 10 frames for annotation. Subsequently, the annotation results are checked by the second annotator (who is also an experienced driver). When the second annotator finds a disputed annotation, the first and second annotators discuss together to reach an agreement. If they are unable to reach an agreement, the triple-discussion annotation mechanism is activated. In this case, the third annotator is invited to discuss the final annotations.Statistics and Comparison. Totally, 9,858 image frames are annotated, generating 44,120 objects with the importance or unimportance annotations, among which 5,052 objects are annotated as important.The annotated data are randomly split into training/testing datasets with a ratio of 8,121 : 1,737. The comparison between TOI and existing on-road object importance estimation datasets and similar task datasets are presented in Tab. 1. Compared to the publicly available Ohn-Bar (33) dataset, TOI represents the significant advantages in following three aspects. Frame quantity: TOI exhibits a substantial increase in the number of frames, with 9,858 frames compared to 3,187 frames in the Ohn-Bar dataset. Object quantity: the number of annotated objects is 44,120 compared to 16,076 in the Ohn-Bar dataset. Scene diversity: while Ohn-Bar contains only 8 scenes, TOI covers 28 scenes. Compared to Goal (8) dataset, TOI has rich annotations such as Lidar and 3D tracklet labels. The abundance of multimodal annotations enables TOI to support the research on on-road object importance estimation using multimodal learning methods in the future. Though Goal (8) presents the advantage in terms of frame number and scene diversity, it is not publicly available. Compared to Li dataset (21) , TOI offers the frame rate of 10 FPS. This high temporal resolution is critical for capturing the dynamic changes of on-road object importance.Annotation Challenges. Compared to datasets for other tasks, TOI may not be considered largescale, it is relatively large-scale compared to existing publicly available datasets for on-road object importance estimation. However, annotating object importance at this scale is challenging. Each annotation requires multiple annotators and undergoes rigorous checking to achieve satisfactory results. In addition, to generate reliable annotations, only one object is annotated at each time observing the whole video sequence, a complete re-observation of the whole video sequence is required for the annotation of the next object. Moreover, object importance is affected by multiple factors, which imposes difficulties on the annotation. Consider a traffic scenario with N on-road objects in T time steps, the goal of this work is to estimate on-road object importance (A) at the final time step (i.e., t = T ) using the video sequence (V ) captured from the driving perspective over T time step and ego-car velocity information (E) at the first time step (i.e., t = 1), which is formulated as:EQUATIONwhere N represents an on-road object importance estimation network, V ={V t } T t=1 , and A={A i } N i=1 . In order to effectively fuse multi-fold top-down guidance factors (i.e., semantic context, driver intention, and traffic rule) with bottom-up object visual feature, we propose a multi-fold top-down guidance aware model, the overview of which is illustrated in Fig. Firstly, OFE extracts object spatial feature f o,s and object temporal feature f o,t from V . Then, DISG takes E, V , and f o,s as inputs, and outputs object-intention-semantics interaction feature f o-i-s . Meanwhile, in TRG, the lane feature f l and f o,t are processed by adaptive object-lane interaction mechanism to produce the object-lane interaction feature f o-l . Finally, f o-i-s and f o-l are used to estimate object importance A. 
object feature extraction: 
 The goal of Object Feature Extraction (OFE) module is to extract object features in both temporal and spatial dimensions. The input of OFE is a RGB video sequence V ∈ R T ×3×W ×H , and the outputs are object temporal feature f o,t and object spatial feature f o,s .To begin with, OFE takes V and M (M denote optical flow images derived from V ) as inputs to extract the object visual featuref v ∈ R N ×T ×C×W ′ ×H ′(reflecting the appearance of the object) and the object motion feature f m ∈ R N ×T ×C×W ′ ×H ′ (reflecting the movement of the object). This procedure is formulated as:EQUATIONwhere N V and N M represent the two ResNet18 (12) , Roi denotes the ROI pooling (10) , C represents the number of channels, W ′ and H ′ denote the width and height obtained through ROI pooling.Subsequently, object spatial feature f o,s ∈ R N ×2C×W ′ ×H ′ is obtained based on f v and f m . The goal of f o,s is to focus on the spatial information of objects. Therefore, an average pooling is applied on the time dimension (i.e., the dimension of T ) of f v and f m , and a self-attention mechanism is utilized to emphasize the spatial information (i.e., the dimensions of W ′ and H ′ ), which is denoted as follows:EQUATIONwhere Avg denotes average pooling, Concat is concatenation. N mhsa represents the multi-head self-attention mechanism, and it has the same meaning in the following parts.Meanwhile, object temporal feature f o,t ∈ R N ×C ′ is also extracted based on f v and f m . The goal of f o,t is to focus on the temporal feature of objects. Therefore, the dimensions of f v and f m are firstly reshaped fromN × T × C × W ′ × H ′ to N × T × (C × W ′ × H ′ ), and then two LSTM networks are applied to extract the temporal feature. Finally, to subsequently fuse f o,t with f l ∈ R N ×C ′ , it is necessary to transform the channel number of f o,t , hence a Linear layer is required. This procedure is formulated as:EQUATIONwhere N lstm is the LSTM network (13) , which outputs features for T time steps, and f o,t is computed by indexing the information at the T -th time step.The driver intention and semantic context significantly affect the on-road object importance in a top-down manner, thus we propose the Driver Intention and Semantics Guidance (DISG) module. DISG consists of two components, namely semantic feature extraction and object-intentionsemantics interaction. The former extracts the semantic guiding feature (i.e., f s in Eq. ( 5)) from driving scenarios, and the latter uses f s and intention guiding mask (i.e., m in Eq. ( 7)) to guide the refinement of f o,s . In the following part, we will sequentially introduce the calculation processes for semantic guiding feature and intention guiding mask.Semantic guiding feature. The goal of f s ∈ R 1×2C×W ′ ×H ′ is to guide the model to be aware of the semantic context of on-road objects, which is extracted by:EQUATIONwhere N G denotes a ResNet18 backbone network and G represents semantic segmentation maps obtained from V T .Intention guiding mask. The goal of m ∈ R W ′ ×H ′ is making the model be aware of the region that is consistent with the driver intention. However, it is difficult to realize since the driver intention is an abstract concept and the limited information regarding the driver intention is known. Additionally, it is not reasonable to assume the driver intention is known in advance. Therefore, we use three common intention behaviors in driving to reflect the driver intention (i.e., turning left, going straight, and turning right). Intention behaviors are formulated as the corresponding intention guiding masks: where m l , m s , and m r are manually engineered masks to emphasize the information in the right, center, and left regions of the images in the video sequence, respectively. Their sizes are aligned with the size of f s . a and b represent pre-determined low and high values, respectively. We note that m l representing turning left behavior is allocated with higher value at the right part, which is inspired by the finding of Tang et al. (42) demonstrating that when a car is turning left, the driver pays more attention to the right side. Before proposing this predefined intention guiding mask, we design a learnable mask with random initialization to make the model automatically learn the mask to reflect intention behaviors. However, this strategy does not work. The potential reason is that the intention is abstract to learn.To automatically select the m of corresponding driving behavior, we design the following logic based on the angular velocity E of ego-car:EQUATIONwhere β represents the driver turning threshold.After obtaining f s and m, DISG module fuses them and uses the fused feature to guide the refinement of f o,s . This procedure is implemented by the object-intention-semantics interaction component. The first task of object-intention-semantics interaction is to fuse f s and m, and generate the intention-semantics interaction feature f i-s ∈ R 1×2C×W ×H , which is denoted as follow:EQUATION)where the operator × makes the model pay more attention to the semantic context in the driver intention regions.The second task of object-intention-semantics interaction is to refine f o,s by interacting with f i-s , which is formulated as follow:EQUATIONwhere N mhca denotes the multi-head cross-attention mechanism, f o,s serves as the query while f i-s serves as the key and value, andf o-i-s ∈ R N ×2C×W ′ ×H ′ .On-road object importance is also closely related with traffic rule, but it is often overlooked in previous works. To effectively leverage the traffic rule, we propose the Traffic Rule Guidance (TRG) module, which consists of two components: lane feature extraction and adaptive object-lane interaction.Lane feature extraction is to make the preparation for adaptive object-lane interaction. In detail, a linear transformation and an activation are applied on lane information L:EQUATION)where L are the coordinates of lane marking points, which are derived from V T via a lane marking detector, Relu is the rectified linear unit activation, andf l ∈ R N ×C ′ .Adaptive object-lane interaction is the core of TRG, and it contains two steps: object-lane interaction and object-lane interaction weighting. In the first step, lane feature f l and f o,t are fused through a multi-head cross-attention mechanism and a residual mechanism, which can be denoted as:EQUATIONwhere f m o-l ∈ R N ×C ′ denotes initial object-lane interaction feature, and f l serves as the query while f o,t serves as the key and value, Factually, f m o-l has considered the traffic rule factor by modeling the relation between lane markings and on-road objects. However, the influence of lane markings on on-road object importance estimation might not be universally-effective in all scenarios, thus we propose the object-lane interaction weighting mechanism to realize adaptive object-lane interaction.The goal of object-lane interaction weighting is to adaptively penalize the cases in which object-lane relation is weak (e.g., static roadside cars weakly interacts with lane markings). To this end, a MLP network is applied on f m o-l to compute a score p, and this score is then used to compute the corresponding penalizing coefficient p c , which is denoted as:EQUATIONEQUATION)where α is a very small value.Based on p c , the object-lane interaction feature f o-l ∈ R N ×C ′ is obtained by weighting f m o-l in Eq. ( 11), which is denoted as:EQUATIONWe note that object-lane interaction weighting is the core of adaptive object-lane interaction, which is significant for object importance estimation (30.4% improvement on AP).Taking f o-i-s in Eq. ( 9) and f o-l in Eq. ( 14) as the inputs, object importance A ∈ R N is estimated. This procedure is formulated as:EQUATIONwhere A signifies the importance for each object. The Linear layer transforms the dimensions off o-i-s from N × 2C × W ′ × H ′ to N × C ′ so that it can be added to f o-l . 
experiments: 
 Metrics. For performance evaluation, two classical metrics are chosen: Average Precision (AP) and F1 Score (F1). AP is computed by calculating the area under the precision-recall curve at various thresholds, thus it is a compressive metric to indicate both the precision and recall of a model. F1 is computed by precision and recall at a fixed threshold, thus it indicates the balance between precision and recall. Both metrics follow the principle that higher values indicate better performance.Loss Function. The loss function is defined as follow:EQUATION)where A is the predicted object importance, Â is the ground-truth.Baselines. Our model is compared with seven models. Ohn-Bar (33) , Goal (8), Zhang (50) , and Li ( 21) are representative works for on-road object importance estimation. In addition, considering that the salient object detection indicates important objects to the certain extent, three recentlyproposed salient object detection models, namely MENet (23), A2S (52) , and PGNet (47) , are also selected as baselines. 21) are both 0 because they predict all objects as unimportant.Top-down and Bottom-up Framework. To validate the effectiveness of our model that interactively integrates multi-fold top-down guidance mechanisms with bottom-up features, we conduct four experiments: #1: only the bottom-up module is enabled; #2: the bottom-up module is combined with TRG module; #3: the bottom-up module is combined with DISG module; Traffic Rule Guidance (TRG). To analyze the effects of object-lane interaction and object-lane interaction weighting, we conduct three experiments: #1: the model without object-lane interaction and object-lane interaction weighting; #2: only the object-lane interaction is enabled; #3: both the object-lane interaction and the object-lane interaction weighting are enabled. The corresponding results are summarized in Tab. 5. Compared to #1, #3 obtains 15.4% and 38.5% improvements on the AP and F1 metrics, respectively. These results demonstrate the significance of both object-lane interaction and the object-lane interaction weighting mechanisms. The reason is explainable. When lane information is not utilized, the implicit traffic rule conveyed by the lane is not used. The absence of the traffic rule results in a reduced ability of the model.It comes as a surprise that the individual usage of object-lane interaction (#2) leads to the performance decreasing on the AP metric compared to #1. This is due to the fact that not all on-road objects are influenced by lanes. Without object-lane interaction weighting, individual object-lane interaction generates a uniform object-lane interaction feature, which could not adaptively extend to diverse scenarios. This result potentially proves the significance of our object-lane interaction weighting, which enables the model to adaptively disable the object-lane interaction feature when object importance weakly rely on object-lane interaction (e.g., static cars on the roadside).To further analyze object-lane interaction weighting, we visualize its output (i.e., p c in Eq. ( 13)). Some examples are illustrated in Fig. 3 where objects with blue masks are penalized (i.e., object-lane interaction is disabled, p c =α) and objects with yellow masks are not penalized (i.e., object-lane interaction is enabled, p c =1). In Fig. 3a and Fig. 3b , the object-lane interaction weighting penalizes the cars on both sides of the road. The results make sense since the static cars on roadsides are factually not interacting with lanes. In Fig. 3c and Fig. 3d , oncoming cars from the opposite direction and the car on the current lane are not penalized, since these cars are interacting with lanes. We note the yellow mask do not signal the important object. Instead, it indicates that the object-lane interaction is enabled. 
conclusion: 
 On-road object importance estimation is significant for various applications in the fields of assisted driving and autonomous driving. The dilemmas of current research are two fold: 1) the scarcity of large-scale publicly available datasets hinder the development of on-road object importance estimation, and 2) existing methods are relatively simple to handle complex and diverse traffic scenarios. In response to the dilemmas, this paper contributes a new dataset and proposes a model with multi-fold top-down guidance. A large range of experiments demonstrate the superiority of our proposed model. The main conclusion is that building up the model that comprehensively considers multi-fold top-down guidance (e.g., driver intention, semantic context, and traffic rule) and bottom-up feature (e.g., size, distance, and speed) is a promising way to remarkably push forward the study of on-road object importance estimation. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: Error bars are not reported because it would be too computationally expensive.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: Our research on-road object importance estimation focuses only on improving the accuracy of importance estimation for road safety purposes. As such, it does not have direct societal impacts beyond the scope of enhancing these technical capabilities. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: Our paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [No] Justification: We are unable to find the license for the dataset and code we used, but they are all open source, and we cite them in the original paper correctly.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: To validate the rationality of OFE, we conduct three experiments, #1: only object spatial feature is used; #2: only object temporal feature is used; #3: both object features are used. The results in Tab. 6 indicate that both spatial and temporal feature of objects serve as valid foundations for accurately evaluating object importance. Removing either of them will lead to the performance decreasing, validating the reasonableness of OFE module. The reasons are self-evident, traffic scenarios are highly dynamic and diverse, thus object temporal feature, which reflects motions and behaviors, is significant for object importance estimation. At the same time, object spatial feature conveys rich information such as size, distance, and orientation, thus it is also significant for object importance estimation.We perform the experiments on hyperparameter selection, and the results are reported in Tab. 7. In the hyperparameter selection for parameters a and b, we choose the optimal values, a = 1, b = 1.5, as the hyperparameters for our model. In the experiments for selecting the hyperparameter α, it is observed that the impact of α is minimal across the three tested values, indicating that our method is robust.Table 7 : Ablation studies on hyperparameter selection. Poor lane marking detection results will limit the performance of our model. In the experiments, we find two kinds of failures. First, as shown in Fig. 5a , lane markings of current lane are falsely detected, thus the white car on the right side is supposed to locate in front of ego-car, leading to the false estimation for the white car. Second, as shown in Fig. 5b , the ego-car moves at a slow speed and lane markings are no detected, thus the model estimates the car in front of the ego-car as an unimportant parking car. Additionally, considering the effect of lane markings on on-road object importance is a small but important step towards modeling the traffic rule for this task. In the future, more traffic rule needs to be considered.Currently, our model only considers the effect of three types of driver intentions (i.e., turning left, turning right, going straight) on object importance estimation. However, real-world driving scenarios are much more complex. In future work, fine-grained intentions need to be modeled.Specific details of our model components are introduced in Tab. 8.Table 8 : Network architecture of our model. For Region of Interest pooling layer (ROI pooling), we list the output shape. For average pooling (Avg pooling), we list pooling scale. For Long Short-Term Memory network (LSTM), we list input and output dimension and layer number. For multi-head selfattention layer (MHSA), we list the hidden size and the head number. For multi-head cross-attention layer (MHCA), we list the hidden size and the head number. For linear layer (Linear), we list the input and output dimension. For Layer Normalization layer (LN), we list the channel dimension. For multi-layer perceptron network (MLP), we list the input channel dimension and each hidden channel dimension. Note that we use different background colors in the table to distinguish between different modules in our model, where orange represents the OFE module, blue represents the DISG module, green represents the TRG module, and gray represents the OIE module. Before the training and inference stages, we utilize CLRNet (51) model with backbone of ResNet101 and pretrained on the CULane dataset to get the L in Eq. (10) , and the G in Eq. ( 5) are generated by applying DeepLabv3 (3). We use OpenCV libary to get the M in Eq. ( 2). During the training and inference stages, we resize V and M in Eq. ( 2), and G in Eq. ( 5) to 320×320 (i.e., W ×H=320×320).We set the W ′ =10, H ′ =10, C=512, and C ′ =256. The a and b in Eq. ( 6) are set as 1 and 1.5, respectively. Then, the β in Eq. ( 7) is set as 2.2 since the ego-car undergoes steering when the angular velocity is around ± 2.2 based on the statistical analysis of the IMU data. In addition, the size of m l , m s , and m r is 10 × 10 (excluding the channel dimension), which is targeted to align with the size of f s (Eq. ( 8)). The length of video clip is set as 16. We use SDG optimizer with a weight decay of 5e -4 and a momentum of 0.9. We set the batch size as 8, and use the cosine learning strategy with an initial learning rate of 1e -4 . Our model implementation is based on PyTorch, and experiments are conducted using an NVIDIA RTX3090 GPU.We assume the ground truth of object bounding boxes are given. The purpose is to focus on the on-road object importance estimation while minimizing the influencing factors from the upstream object detection task. This setup is consistent with the existing on-road object importance estimation method (21) . Additionally, this setup is reasonable since current object detection models are highly advanced and capable of detecting most objects on the road.Justification: This paper contributes a new dataset and a model, which will be publicly released in https://github.com/CQU-ADHRI-Lab/TOI Guidelines:• The answer NA means that paper does not include experiments requiring code. Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 


Decision: ## Summary 

The Traffic Object Importance (TOI) dataset is introduced to estimate the importance of on-road objects from a driver's perspective, a crucial task for enhancing driving safety. The dataset is larger and more diverse than existing ones, and a novel model is proposed that integrates driver intention, semantic context, and traffic rules with bottom-up features for more accurate estimation. Experimental results show that the proposed model significantly outperforms state-of-the-art methods in on-road object importance estimation. The paper also introduces an adaptive mechanism for object-lane interaction.

## Strengths
-  Introduces a larger and diverse dataset for on-road object importance estimation.
- Integrates top-down guidance factors (driver intention, semantic context, traffic rules) with bottom-up features.
- Shows promising performance for the TOI task.
- Addresses the pivotal role of traffic rules in estimating object importance.
- Proposes an adaptive object-lane interaction mechanism.

## Weaknesses

- Lack of detailed annotation details, including the number of annotators involved and the annotation procedure samples.
- Annotation may vary according to different traffic rules, which could affect label quality.
- Definition of importance is unclear, with only objects ahead of the car or with an intersection with the ego-car's direction considered important.
- Driver's intention definition is difficult to define, with unclear strategies for learning intention values based on driving behaviors.
- More statistics of the dataset are needed, such as the number of important objects of different categories.

## Conclusion

The paper proposed (i) a dataset that is larger and more diverse than existing ones, and (ii)  a novel model  that integrates driver intention, semantic context, and traffic rules with bottom-up features for more accurate estimation. Therefore, the recommendation is the acceptance of the paper but it should include a benchmark with the current object detectors.

Reviews: review: paper_summary: paper_summary: This work addresses the issue of estimating the importance of on-road objects using video sequences from a driver’s perspective, a critical task for enhancing driving safety. The authors introduce the Traffic Object Importance (TOI) dataset, which is significantly larger and more diverse than existing datasets, and propose a novel model that integrates multi-fold top-down guidance factors—driver intention, semantic context, and traffic rules—with bottom-up features for more accurate importance estimation. Experimental results demonstrate that the proposed model significantly outperforms state-of-the-art methods in on-road object importance estimation.
strength_weakness: strength_weakness: 1. Lack of description of the annotation details. 
How many annotators are involved in the annotation procedure? It would be good if the authors can provide some annotation procedure samples regarding the double-checking annotation mechanism and the triple-discussing annotation mechanism.

2. It seems this annotation will be varied according to different traffic rules. Since KITTI is collected in Germany, the annotators should be familiar to germany traffic rules. However the authors did not mention this information in their submission, thereby the label quality is doubtful.

3. The authors are encouraged to build up the first benchmark based on the proposed dataset by using various existing object detection methods, e.g., Yolo, with the proposed head or simpler head. It is interesting to see how the existing object detectors work on this new task.

4. More statistics of the dataset are encouraged to be given, e.g., the number of important object of different categories, etc.
questions: questions: 1. How many annotators were involved in the annotation procedure for the dataset? Can the authors provide detailed examples of their double-checking and triple-discussing annotation mechanisms?

2. Were the annotators familiar with German traffic rules, given that the dataset was collected in Germany (KITTI dataset)? How was the expertise of the annotators in relation to German traffic laws ensured and validated?

3. Have the authors considered building the first benchmark using their dataset with existing object detection methods, such as YOLO? What were the performance outcomes of these existing methods when applied to the new task?

4. Can the authors provide more detailed statistics about the dataset, such as the number of important objects in different categories? How do these statistics compare to other datasets in the same domain?
limitations: limitations: yes the authors mentioned it in the appendix
paper_summary: paper_summary: This paper presents a novel dataset for on-road object importance estimation. More data about which objects are important for self-driving is included and is promised to be released. Moreover, a novel method that integrates driven intention, semantic context, and traffic rule is devised to tackle the related problem. The paper is well-written.
strength_weakness: strength_weakness: Some of the concepts lack sufficient details to explain. See questions below.
questions: questions: (1) Regarding the task, my major concern is the definition of importance. It is shown that surrounding objects that follow the traffic rules are not considered as important. Only the objects ahead of the car or have an intersection with the ego-car's direction are important. I doubt whether this is strictly appropriate. For example, if a pedestrian walking along the road, he/she will not be considered as important. However, what if this pedestrian suddenly steps into the road ahead, potential collisions would happen. Therefore, I think a nearby walking pedestrian should be considered as important or at least recognized into a third category like "needs care". I wonder how the authors solve this problem in the dataset.
(2) Regarding the driver's intention, it is indeed difficult to define appropriately. The authors have mentioned this in the paper, but the strategy introduced to accommodate this is still not clear to me. The authors mentioned learning the intention values based on driving behaviors, but how do we know the driving behaviors? Are these behaviors (e.g. turning left) already provided in the dataset? 
(3) More visualization about the labels and method comparisons are better to be presented for more clarity.
limitations: limitations: The authors have mentioned limitations.
paper_summary: paper_summary: This paper collects a new large-scale dataset and proposes a novel method that integrates multi-fold top-down guidance with the bottom feature to address the problem of on-road object importance estimation. Specifically, the dataset is almost three times larger than the current publicly dataset for on-road object importance. In addition, this paper considers an adaptive mechanism for object-lane interaction, effectively modeling the impact of traffic rules on object importance. Experiments on several benchmarks validate the effectiveness of the proposed method.
strength_weakness: strength_weakness: This paper has also two weaknesses: 

(1) The paper does not provide a detailed discussion on the computational efficiency of the proposed method, which is crucial for real driving scenarios. Moreover, it is recommended to compare the model parameters and latency with other methods.

(2) Another concern lies in the practicality of the method. This method and the proposed dataset are both for single-camera scenarios, but in real autonomous driving scenarios, surrounding view is a more widely used type and a safer option. Will the proposed method also work well in the surrounding view?
questions: questions: (1) Can the proposed method be applied to surrounding view images? I suggest that the authors should consider the application on the current perception pipeline for vision-based autonomous driving pipeline.

(2) I suggest that the authors should analyze the latency of the proposed method, which determines whether the method can be integrated into the practical driving scenarios.
limitations: limitations: The proposed method only considers the effect of three types of driver intentions on object importance estimation, which is not sufficient for complex driving scenarios. I carefully checked the paper and found no potential negative societal impact.
paper_summary: paper_summary: 1.This paper this contributes a new large-scale dataset named Traffic Object Importance (TOI) to addresses the problem of on-road object importance estimation, which utilizes video sequences captured from the driver’s perspective as the input.
2.The author also proposes a model that integrates multi-fold top-down guidance with the bottom-up feature.
strength_weakness: strength_weakness: 1.In page 3 the author mentions that the traffic rule is crucial for object importance and focus on the traffic line rules , but the influence of traffic rules is varied, such as signalization. Therefore, in page 4 of table 1, the author is able to provide statistics on the scenario categories of TOI dataset and the traffic rule constraints within the dataset in experiment. 

2.In page 6, the author uses three common intention behaviors in driving to reflect the driver intention (i.e., turning left, going straight, and turning right). Since the video clip length is set at 16 frames, it is important to clarify if each of the three intentions corresponds to individual frames with the 16-frame clip cut during the training and testing phases, or if multiple intentions are present within the 16 frames. The authors should further elaborate and provide the proportion of each intention in the dataset.

3.Insufficient evaluation of indicators in the experimental section. The author may add another evaluation indicator.

4.The section three can include a schematic diagram of the annotation process for the dataset.
questions: questions: 1.I consider whether 16 frames constitute interval sampling or continuous sampling, and how many types of intentional behaviors can be expressed using 16 frames in the paper.

2.The author may can add another evaluation metric for the experiment.
limitations: limitations: Yes.

Comments: title: Post rebuttal
Response to the authors
comment: Thanks for the rebuttal. Most of my concerns are well-addressed therefore I tend to keep my positive rating.
In fact, I think I do not get too much new information from the authors' response to my question 1, though this would not affect my original rating. Overall this is an interesting paper. I only suggest that the authors present more discussions on the annotation process, not only on how annotations are obtained but also on the strengths and weaknesses of the applied annotation policy. Regarding the answer to my question 2, I can get the idea and I also suggest adding more related discussions in the paper.
Thank you for your response. My concerns are mostly solved. I would lile to improve my score to 6.