R4D: Utilizing Reference Objects for Long-Range Distance Estimation

1 INTRODUCTION Estimating the distances of objects from the vehicle is crucial for several autonomous driving tasks , including lane changing , route planning , speed adjustment , collision avoidance , to name a few . Although existing methods and datasets focus on short-range objects , knowing the distance of long-range objects — objects beyond a typical LiDAR range of∼80 meters ( as shown in Figure 1 ) — is necessary for freeway driving , heavy-duty truck driving , and wet road driving . Based on the US Department of Transportation ( Blanco & Hankey , 2005 ) , on rural highways with a standard speed limit of 65 miles/h , it takes ∼145 meters for a passenger vehicle to come to a complete stop in an emergency , greatly exceeding the typical LiDAR sensing range . The required stopping distance grows significantly with a heavy load truck or in bad road conditions such as snow , ice , or rain . For example , the stopping distance increases from 145 meters to 183 meters and 278 meters for trucking and wet road driving , respec- tively ( Administration , 2016 ; Blanco & Hankey , 2005 ) . In addition , given that harsh and sudden breaking on freeways is unsafe , it remains critical to estimate the distance of objects beyond the minimum required stopping distance in order to provide enough time for a gradual slow-down or lane change . Therefore , to allow sufficient time for an appropriate reaction and to ensure safety , autonomous driving systems are required to estimate the distance to long-range objects . We name this critical but underexplored task as Long-Range Distance Estimation . Concretely , given the short-range LiDAR signal and the camera image , the distances of long-range objects ( beyond LiDAR range ) are expected as output . Following conventions , the distance is measured between the camera and the object center along the camera ’ s optical axis ( Gökçe et al. , 2015 ; Haseeb et al. , 2018 ; Zhu & Fang , 2019 ) . We discuss more about making design choices for our task in the related work . For this new task , we present pseudo long-range KITTI dataset and anonymous internal long-range dataset . Since KITTI ( Geiger et al. , 2013 ) do not provide ground-truth distance for long-range objects , the pseudo long-range KITTI dataset is a derived dataset by removing LiDAR points beyond 40 meters and considering the objects beyond 40 meters as target . More importantly , we build a large scale dataset with annotated real long-range objects ( from 80 to 300 meters ) . In summary , with different definition of “ long-range ” ( either beyond 40 or 80 meters ) , both datasets include LiDAR points , camera images , and distance labels for long-range objects . Neither LiDAR nor camera alone solves this long-range distance estimation problem to the desired accuracy . Most existing LiDAR technologies do not meet long-range sensing requirements . According to the Waymo and KITTI self-driving car datasets , the maximum LiDAR range is only around 80 meters , which falls short of the range required for the aforementioned scenarios . Even though some advanced LiDAR systems claim to achieve a longer sensing range , e.g. , Waymo ’ s 5th generation LiDAR system ( Jeyachandran , 2020 ) and Velodyne Alpha PrimeTM ( VelodyneLidar , 2021 ) which reach up to 300 meters , LiDAR points are sparse at long-range and , thus , more likely to be occluded . Therefore , LiDAR alone is not enough to cover all safety-critical self-driving car applications . Cameras , on the other hand , sense objects at a longer range and capture rich semantic information such as object appearance , geometry , and contextual hints . However , they do not , in and of themselves , provide depth information . Classical geometry-based algorithms could estimate the distance of canonical objects ( e.g. , sedans , trucks ) based on their pixel sizes in the camera image ( Criminisi et al. , 2000 ; Tuohy et al. , 2010 ; Gökçe et al. , 2015 ; Haseeb et al. , 2018 ; Qi et al. , 2019 ) . However , these approaches yield inaccurate results on long-range objects due to errors in size estimation . Appearance-based methods ( Song et al. , 2020 ; Zhu & Fang , 2019 ) give unsatisfactory results on long-range objects . Such methods rely on a single appearance cue to estimate the distance , overlooking context or other relevant signals in the scene . At long-range , objects are visually small thus produce less informative appearance features . Although neither LiDAR nor camera alone can solve this problem , these two signals provide complementary cues for long-range distance estimation . In this paper , we propose R4D , a method to utilize reference objects with known and accurate distances for long-range distance estimation ( R4D ) . The main motivation behind our approach lies in theories of human perception from cognitive science : humans often estimate the distance of an object relative to other reference points or objects ( Granrud et al. , 1984 ) . Specifically , we train a model to localize a long-range object ( target ) given references with known distances . We represent the target object and references as a graph . As shown in Figure 2 , we define objects as nodes , with edges connecting the target to the reference objects . The reference information is propagated to the long-range target object by extracting target-reference ( Tar-Ref ) embeddings . R4D then feeds all target-reference embeddings to an attention module which fuses information from different references , by weighing their relative importance and combining them into one distance prediction . Experiments on the pseudo long-range KITTI and anonymous internal long-range datasets show that R4D significantly improves the distance estimation and achieves state-of-the-art long-range localization results . For example , on the anonymous internal long-range dataset , with R4D , the distances of 8.9 % more vehicles ( from 53.4 % to 62.3 % ) are predicted with a relative distance error below 10 % . By conducting experiments on images captured at different times of the day , e.g. , train on daytime and test on nighttime images , R4D also shows stronger robustness against domain changes . To summarize , our contributions are three-fold . ( 1 ) We propose a critical but underexplored task Long-Range Distance Estimation . ( 2 ) We present two datasets , pseudo long-range KITTI dataset and anonymous internal long-range dataset . To facilitate future research , the datasets will be made publicly available . ( 3 ) We develop R4D , the first framework to accurately estimate the distance of long-range objects by using references with known distances . 2 RELATED WORK . In this section , we discuss the related tasks and the detailed design choices of our proposed task . Monocular distance estimation . Estimating the distance of objects from an RGB image ( i.e. , monocular distance estimation ) is a popular computer vision topic . More than a decade ago , researchers designed geometry-based algorithms to estimate object distances . For example , Tuohy et al . ( 2010 ) use inverted perspective mapping ( IPM ) to convert an image to the bird ’ s eye view for estimating distances . Learning-based methods were later proposed ( Qi et al. , 2019 ; Song et al. , 2020 ) . For example , SVR ( Gökçe et al. , 2015 ) and DisNet ( Haseeb et al. , 2018 ) take the pixel height and width of an object as input and estimate object distance by using support vector regression ( Drucker et al. , 1997 ) and multi-layer perceptron ( MLP ) . However , such methods are subject to errors in object size estimation ( Zhu & Fang , 2019 ) , which can be even more pronounced for longrange objects . Zhu & Fang ( 2019 ) developed an end-to-end distance estimator with a Convolutional Neural Network ( CNN ) feature extractor . Specifically , given an RGB image and object bounding boxes , the model extracts per-object embeddings with a CNN and ROI pooling operator ( Girshick , 2015 ) , and then predicts a distance for each object . As previously mentioned , this method relies heavily on object appearance , which is a less informative cue for long-range objects . Another related research topic is monocular per-pixel depth estimation . Eigen et al . ( 2014 ) ; Garg et al . ( 2016 ) ; Godard et al . ( 2019 ) ; Lee & Kim ( 2019 ) ; Liu et al . ( 2015 ) ; Shu et al . ( 2020 ) propose to predict dense depth maps given RGB images . For example , Lee & Kim ( 2019 ) generate multiresolution relative depth maps to construct the final depth map . However , these methods can be resource-intensive and , therefore , difficult to incorporate within a latency-sensitive system such as autonomous driving ( Zhu & Fang , 2019 ) . In addition , it is non-trivial to translate a depth map into per-object distance estimates due to occlusions , looseness of bounding boxes , etc . Long-range depth sensing . In addition , other approaches such as stereo cameras and Radar have been used to sense objects at long-range . Stereo camera systems simulate human binocular vision and use multiple cameras to create stereo-pairs and perceive depth ( Khamis et al. , 2018 ; Poggi et al. , 2019 ) . However , stereo vision systems have a few limitations : they are difficult to calibrate , are sensitive to vibrations ( Zhang et al. , 2020 ) , and require a special hardware setup . In contrast , R4D does not need special hardware and works with existing sensors present on most autonomous vehicles , such as standalone camera and LiDAR . Radar , on the other hand , uses radio waves to determine the range , angle , and velocity of objects . However , due to its limited angular resolution ( Scheiner et al. , 2020 ) , it is difficult for Radar to accurately localize long-range objects . Monocular 3D object detection . Monocular 3D object detection aims to detect all objects in an image by predicting a 3D bounding box for each of them , e.g. , MonoPair ( Chen et al. , 2020 ) . Besides distance ( from the target object to the autonomous vehicle ) , the monocular 3D object detector also predicts object size . For short-range vehicles , both object distance and object size are critical for selfdriving tasks , such as route planning , collision avoidance , and lane changing . However , for longrange vehicles , the distance is significantly larger than object size ( e.g. , 300 meters vs. 2 meters ) , thus the object size is less critical compared to the distance under the real-world scenario . Moreover , predicting the object size increases the latency and the complexity of the model . Therefore , our task focuses on estimating the distance of long-range objects . 3 METHODOLOGY Fused Embedding ( 1⨉C ) Motivated by the observation that humans estimate the distance of an object relative to other references ( Granrud et al. , 1984 ) , we propose R4D , a method utilizing Reference objects For long-range Distance estimation . References can be any combination of objects or points with known and accurate distances to the autonomous vehicle , such as LiDAR detections , objects detected by other sensors , map features , etc . We represent the target object and references as a graph shown in the “ Raw Inputs ” dashed box in Figure 2 . The target object and its references are nodes in this graph . Between a pair of target and reference objects , an edge is built to encode their pairwise relationship . The detailed architecture is illustrated in Figure 3 . To model the target-reference pairwise relationship , we propose to extract union embeddings and geo-distance embeddings which encode visual and geometry relationships , respectively ( Section 3.1 ) . Then , inspired by the intuition that reference objects are not equally important , we introduce an attention module in Sec- tion 3.2 to selectively aggregate pairwise relationships . Finally , R4D is trained with an auxiliary supervision : the relative distances between the target and its references , as explained in Section 3.3 . In our setup , and without loss of generality , we use a monocular camera as the main sensor for detecting long-range target objects , and adopt LiDAR detected short-range objects as references . It is worthy noting that R4D is not specifically designed for LiDAR or monocular images and is readily extended to other sensors and references .

Summary: The paper proposes a method for the task of long-range object distance estimation. It proposes a framework to predict long-range distance by modeling pairwise relationship between short-range objects and long-range objects. To evaluate the method, two datasets are also introduced. Experimental results are mainly compared to the baseline Zhu & Fang (2019) and several distance metrics are evaluated.