R4D: Utilizing Reference Objects for Long-Range Distance Estimation

1 INTRODUCTION Estimating the distances of objects from the vehicle is crucial for several autonomous driving tasks , including lane changing , route planning , speed adjustment , collision avoidance , to name a few . Although existing methods and datasets focus on short-range objects , knowing the distance of long-range objects — objects beyond a typical LiDAR range of∼80 meters ( as shown in Figure 1 ) — is necessary for freeway driving , heavy-duty truck driving , and wet road driving . Based on the US Department of Transportation ( Blanco & Hankey , 2005 ) , on rural highways with a standard speed limit of 65 miles/h , it takes ∼145 meters for a passenger vehicle to come to a complete stop in an emergency , greatly exceeding the typical LiDAR sensing range . The required stopping distance grows significantly with a heavy load truck or in bad road conditions such as snow , ice , or rain . For example , the stopping distance increases from 145 meters to 183 meters and 278 meters for trucking and wet road driving , respec- tively ( Administration , 2016 ; Blanco & Hankey , 2005 ) . In addition , given that harsh and sudden breaking on freeways is unsafe , it remains critical to estimate the distance of objects beyond the minimum required stopping distance in order to provide enough time for a gradual slow-down or lane change . Therefore , to allow sufficient time for an appropriate reaction and to ensure safety , autonomous driving systems are required to estimate the distance to long-range objects . We name this critical but underexplored task as Long-Range Distance Estimation . Concretely , given the short-range LiDAR signal and the camera image , the distances of long-range objects ( beyond LiDAR range ) are expected as output . Following conventions , the distance is measured between the camera and the object center along the camera ’ s optical axis ( Gökçe et al. , 2015 ; Haseeb et al. , 2018 ; Zhu & Fang , 2019 ) . We discuss more about making design choices for our task in the related work . For this new task , we present pseudo long-range KITTI dataset and anonymous internal long-range dataset . Since KITTI ( Geiger et al. , 2013 ) do not provide ground-truth distance for long-range objects , the pseudo long-range KITTI dataset is a derived dataset by removing LiDAR points beyond 40 meters and considering the objects beyond 40 meters as target . More importantly , we build a large scale dataset with annotated real long-range objects ( from 80 to 300 meters ) . In summary , with different definition of “ long-range ” ( either beyond 40 or 80 meters ) , both datasets include LiDAR points , camera images , and distance labels for long-range objects . Neither LiDAR nor camera alone solves this long-range distance estimation problem to the desired accuracy . Most existing LiDAR technologies do not meet long-range sensing requirements . According to the Waymo and KITTI self-driving car datasets , the maximum LiDAR range is only around 80 meters , which falls short of the range required for the aforementioned scenarios . Even though some advanced LiDAR systems claim to achieve a longer sensing range , e.g. , Waymo ’ s 5th generation LiDAR system ( Jeyachandran , 2020 ) and Velodyne Alpha PrimeTM ( VelodyneLidar , 2021 ) which reach up to 300 meters , LiDAR points are sparse at long-range and , thus , more likely to be occluded . Therefore , LiDAR alone is not enough to cover all safety-critical self-driving car applications . Cameras , on the other hand , sense objects at a longer range and capture rich semantic information such as object appearance , geometry , and contextual hints . However , they do not , in and of themselves , provide depth information . Classical geometry-based algorithms could estimate the distance of canonical objects ( e.g. , sedans , trucks ) based on their pixel sizes in the camera image ( Criminisi et al. , 2000 ; Tuohy et al. , 2010 ; Gökçe et al. , 2015 ; Haseeb et al. , 2018 ; Qi et al. , 2019 ) . However , these approaches yield inaccurate results on long-range objects due to errors in size estimation . Appearance-based methods ( Song et al. , 2020 ; Zhu & Fang , 2019 ) give unsatisfactory results on long-range objects . Such methods rely on a single appearance cue to estimate the distance , overlooking context or other relevant signals in the scene . At long-range , objects are visually small thus produce less informative appearance features . Although neither LiDAR nor camera alone can solve this problem , these two signals provide complementary cues for long-range distance estimation . In this paper , we propose R4D , a method to utilize reference objects with known and accurate distances for long-range distance estimation ( R4D ) . The main motivation behind our approach lies in theories of human perception from cognitive science : humans often estimate the distance of an object relative to other reference points or objects ( Granrud et al. , 1984 ) . Specifically , we train a model to localize a long-range object ( target ) given references with known distances . We represent the target object and references as a graph . As shown in Figure 2 , we define objects as nodes , with edges connecting the target to the reference objects . The reference information is propagated to the long-range target object by extracting target-reference ( Tar-Ref ) embeddings . R4D then feeds all target-reference embeddings to an attention module which fuses information from different references , by weighing their relative importance and combining them into one distance prediction . Experiments on the pseudo long-range KITTI and anonymous internal long-range datasets show that R4D significantly improves the distance estimation and achieves state-of-the-art long-range localization results . For example , on the anonymous internal long-range dataset , with R4D , the distances of 8.9 % more vehicles ( from 53.4 % to 62.3 % ) are predicted with a relative distance error below 10 % . By conducting experiments on images captured at different times of the day , e.g. , train on daytime and test on nighttime images , R4D also shows stronger robustness against domain changes . To summarize , our contributions are three-fold . ( 1 ) We propose a critical but underexplored task Long-Range Distance Estimation . ( 2 ) We present two datasets , pseudo long-range KITTI dataset and anonymous internal long-range dataset . To facilitate future research , the datasets will be made publicly available . ( 3 ) We develop R4D , the first framework to accurately estimate the distance of long-range objects by using references with known distances . 2 RELATED WORK . In this section , we discuss the related tasks and the detailed design choices of our proposed task . Monocular distance estimation . Estimating the distance of objects from an RGB image ( i.e. , monocular distance estimation ) is a popular computer vision topic . More than a decade ago , researchers designed geometry-based algorithms to estimate object distances . For example , Tuohy et al . ( 2010 ) use inverted perspective mapping ( IPM ) to convert an image to the bird ’ s eye view for estimating distances . Learning-based methods were later proposed ( Qi et al. , 2019 ; Song et al. , 2020 ) . For example , SVR ( Gökçe et al. , 2015 ) and DisNet ( Haseeb et al. , 2018 ) take the pixel height and width of an object as input and estimate object distance by using support vector regression ( Drucker et al. , 1997 ) and multi-layer perceptron ( MLP ) . However , such methods are subject to errors in object size estimation ( Zhu & Fang , 2019 ) , which can be even more pronounced for longrange objects . Zhu & Fang ( 2019 ) developed an end-to-end distance estimator with a Convolutional Neural Network ( CNN ) feature extractor . Specifically , given an RGB image and object bounding boxes , the model extracts per-object embeddings with a CNN and ROI pooling operator ( Girshick , 2015 ) , and then predicts a distance for each object . As previously mentioned , this method relies heavily on object appearance , which is a less informative cue for long-range objects . Another related research topic is monocular per-pixel depth estimation . Eigen et al . ( 2014 ) ; Garg et al . ( 2016 ) ; Godard et al . ( 2019 ) ; Lee & Kim ( 2019 ) ; Liu et al . ( 2015 ) ; Shu et al . ( 2020 ) propose to predict dense depth maps given RGB images . For example , Lee & Kim ( 2019 ) generate multiresolution relative depth maps to construct the final depth map . However , these methods can be resource-intensive and , therefore , difficult to incorporate within a latency-sensitive system such as autonomous driving ( Zhu & Fang , 2019 ) . In addition , it is non-trivial to translate a depth map into per-object distance estimates due to occlusions , looseness of bounding boxes , etc . Long-range depth sensing . In addition , other approaches such as stereo cameras and Radar have been used to sense objects at long-range . Stereo camera systems simulate human binocular vision and use multiple cameras to create stereo-pairs and perceive depth ( Khamis et al. , 2018 ; Poggi et al. , 2019 ) . However , stereo vision systems have a few limitations : they are difficult to calibrate , are sensitive to vibrations ( Zhang et al. , 2020 ) , and require a special hardware setup . In contrast , R4D does not need special hardware and works with existing sensors present on most autonomous vehicles , such as standalone camera and LiDAR . Radar , on the other hand , uses radio waves to determine the range , angle , and velocity of objects . However , due to its limited angular resolution ( Scheiner et al. , 2020 ) , it is difficult for Radar to accurately localize long-range objects . Monocular 3D object detection . Monocular 3D object detection aims to detect all objects in an image by predicting a 3D bounding box for each of them , e.g. , MonoPair ( Chen et al. , 2020 ) . Besides distance ( from the target object to the autonomous vehicle ) , the monocular 3D object detector also predicts object size . For short-range vehicles , both object distance and object size are critical for selfdriving tasks , such as route planning , collision avoidance , and lane changing . However , for longrange vehicles , the distance is significantly larger than object size ( e.g. , 300 meters vs. 2 meters ) , thus the object size is less critical compared to the distance under the real-world scenario . Moreover , predicting the object size increases the latency and the complexity of the model . Therefore , our task focuses on estimating the distance of long-range objects . 3 METHODOLOGY Fused Embedding ( 1⨉C ) Motivated by the observation that humans estimate the distance of an object relative to other references ( Granrud et al. , 1984 ) , we propose R4D , a method utilizing Reference objects For long-range Distance estimation . References can be any combination of objects or points with known and accurate distances to the autonomous vehicle , such as LiDAR detections , objects detected by other sensors , map features , etc . We represent the target object and references as a graph shown in the “ Raw Inputs ” dashed box in Figure 2 . The target object and its references are nodes in this graph . Between a pair of target and reference objects , an edge is built to encode their pairwise relationship . The detailed architecture is illustrated in Figure 3 . To model the target-reference pairwise relationship , we propose to extract union embeddings and geo-distance embeddings which encode visual and geometry relationships , respectively ( Section 3.1 ) . Then , inspired by the intuition that reference objects are not equally important , we introduce an attention module in Sec- tion 3.2 to selectively aggregate pairwise relationships . Finally , R4D is trained with an auxiliary supervision : the relative distances between the target and its references , as explained in Section 3.3 . In our setup , and without loss of generality , we use a monocular camera as the main sensor for detecting long-range target objects , and adopt LiDAR detected short-range objects as references . It is worthy noting that R4D is not specifically designed for LiDAR or monocular images and is readily extended to other sensors and references .

Summary: This paper proposes to utilize camera images to further increase the perception capabilities for self-driving vehicles, especially in long-range settings. The paper builds on top of the assumption that there exist objects with accurate distance information. Then, pairwise relationships can be established between such objects and the anchoring objects. By formulating such relationships as a graph, distances can be better estimated for long-range objects.