Monocular 3D Lane Line Detection in Autonomous Driving — A Review
By: Patrick Langechuan Liu
Tags: Autonomous Cars, Autonomous Vehicles, Machine Learning, Computer Vision, Deep Learning

2D LaneNet

Before we dive into the 3D lane line detection algorithms, one important 2D lane line detection algorithm is to revisit is LaneNet (Towards End-to-End Lane Detection: an Instance Segmentation Approach, IV 2018). Its 2D lane line detection performance has been surpassed by many new algorithms, but it was quite innovative at the time and many of its ideas formed the foundation of 3D lane line detection.

Its contribution to the 2D lane line perception is the proposal of a segment-then-cluster approach for lane line semantic segmentation — we will talk about this idea again later in Semi-local 3D LaneNet. More interestingly, it also uses a lightweight network called H-Net to predict a homographic transformation (in the form of a 3x3 matrix H) directly from the perspective image. The homographic transformation matrix H is essentially IPM which projects sampled lane line points to BEV, and it is used to optimize a lane fitting postprocessing. The basic assumption here is that the lanes should be perfectly described by a 3rd order polynomial in BEV space.

The overall pipeline of LaneNet (source: LaneNet)

Another implicit assumption adopted by LaneNet is that the lane lines are on a flat road. For a non-flat road with slopes, there is not a single best IPM that describes the transformation between the perspective image and the BEV image, and LaneNet still approximates the road-camera model using a homographic transformation. Then the question is — what is the best transformation to describe a non-flat road? You may argue that the best transformation should accurately map the horizon (meeting points between the road and sky in camera image) to infinity depth, or you may argue that the best one should project the lane line closest to ego car to parallel lines in 3D space. LaneNet defines the best transformation as one mapping that minimizes reprojection errors of fitted curves.

3D LaneNet

The pioneering work that popularized the field of monocular 3D lane line detection is 3D-LaneNet (ICCV 2019) from General Motors’ research center in Israel. 3D LaneNet does not need fragile assumptions such as the flat ground assumption and it only assumes zero camera roll wrt to the local road surface. Like 2D LaneNet, it also estimates the homographic transformation between the 2D perspective image and the 3D local road plane. Different from LaneNet which predicts the homographic matrix directly, 3D LaneNet predicts the camera height and pitch which uniquely determine the homography. These two parameters are learned in a supervised manner.

3D LaneNet network architecture (source: 3D-LaneNet)

The network architecture is a dual-pathway backbone that converts from image to BEV space. This actually reminds me of the PoseNet structure in the self-supervised depth learning Sfm-learner (CVPR 2017), which predicts 6 DoF ego-motion and then uses it to warp neighboring images.

Anchor-based representation of 3D lane lines

Instead of predicting the depth of the lane line points directly, it first predicts the pitch/height of the camera and thus constructs a road projection plane. The road projection plane is defined according to the camera mounting pitch angle 𝜑_cam and height h_cam. Then 3D lane lines are represented relative to that plane. Concretely, a lane line point is defined by its projection point (x, y) on the plane and the elevation Δz.

Road projection plane and anchor design (source: 3D-LaneNet and Semi-Local 3D LaneNet)

3D LaneNet uses an anchor-based lane prediction head. The anchor associated with a groundtruth lane is the one closest to the x-coordinate of the lane at Y_ref=20 m.

The total number of predictions is #anchor*(2*K+1) per lane line type. K (K=6) is the number of predefined y position per lane line. Each of the K points predicts two numbers, deviation from anchor dx and elevation z. Each anchor also has one conf score to predict the existence of a lane line.

Note that due to this anchor design, 3D LaneNet can only handle normal topologies with largely parallel lane lines to the ego car motion direction. It cannot handle, for example, perpendicular lane lines to the ego car.

Semi-local 3D LaneNet (3D LaneNet+)

Semi-local 3D LaneNet builds on 3D-LaneNet and adds two more contributions, the ability to handle more complex topologies, and uncertainty prediction. The uncertainty estimation is quite standard aleatoric uncertainty estimation, and we won’t go into details here. The paper is republished in NeurIPS 2020 workshop as 3D-LaneNet+.

The majority of the work is based on 3D LaneNet. It also has the dual-pathway backbone, prediction of camera height and roll, and has the last feature map in BEV space. The main difference lies in a more flexible lane line representation that allows modeling of more complex lane topologies including splits, merges, and lanes perpendicular to the vehicle travel direction.

Semi-local representation of 3D Lane Line

Semi-local representation also leverages the road projection plane as a reference and formulates the 3D lane line wrt its BEV projection onto the road projection plane. Yet instead of associating each lane to a predefined anchor, semi-local 3D LaneNet proposes a compact semi-local representation. Essentially it breaks the BEV image (projection of 3D structures to road projection plane) into non-overlapping grids called image tiles. The assumption is that each image tile can only have one lane line going through it, and the cropped lane segment in each tile is simple enough to be parameterized as a 2 DoF line segment (distance to tile center and orientation). Then the next step is to learn a globally consistent embedding for each lane to cluster the small lane segments into full curves.

Network architecture of Semi-local 3D LaneNet (source: Semi-local 3D LaneNet)

This detect-then-cluster approach is inspired by the 2D LaneNet. Semi-local 3D LaneNet uses push-pull loss to train the embedding on image tiles and has much less computation burden than semantic segmentation from the original 2D LaneNet. During inference, the clustering is done by the mode-seeking algorithm mean-shift to find the center of each cluster and then set a threshold to get cluster members.

Gen-LaneNet

Gen-LaneNet (ECCV 2020) builds on the standard practices of 3D LaneNet and proposes a two-stage method for 3D lane line detection. It proposes to perform 2D lane line detection first, and then lift the 2D detection results to 3D using a separate network called 3D-GeoNet.

Decoupling 3D structure prediction from 2D lane line detection is motivated by the fact that encoding of 3D geometry is rather independent of image features. This decoupling makes the module more flexible and scalable. It can immediately benefit from the ever-evolving 2D lane line detection algorithms in the first stage. More importantly, it allows the second stage to be trained with a synthetic dataset alone, and even a non-photorealistic simulation will do as the dependency on image features have been removed.

In 3D LaneNet there is no guarantee that the projection of prediction matches the image and lacks the 2D-3D consistency. In Gen-LaneNet, 3D are lifted from 2D and thus this consistency is inherent to the pipeline.

The deliberate removal of image features in the second stage is similar to predicting distance with bbox info only in Deep Object Distance Estimator, and with skeleton only in MonoLoco. A similar strategy to alleviate the sim2real domain gap can be also found in many early studies in monocular BEV semantic segmentation.

Network architecture for Gen-LaneNet (source: Gen-LaneNet)

Gen-LaneNet also pointed out one major drawback of 3D-LaneNet that the top view projection does not align with the IPM transformed features in the presence of a non-zero slope. In other words, IPM assumes a flat ground, and when this assumption breaks with non-zero slope, the IPM transformed view is not real top view (BEV). Rather, the IPM transformed features is a warped BEV view, referred to as virtual top view in this paper. This virtual top view is obtained by raytracing each lane line point and project to the road projection plane (orange points in the chart below). 3D lane line groundtruths are preprocessed to virtual top view to aligns with IPM transformed features.

The virtual top view and real top view in 3D Lane Geometry (source: Gen-LaneNet)

The concepts of real top view and virtual top view are not very straightforward to grasp. To give a more concrete example, an uphill 3D lane with zero yaw will project its two parallel lane lines to exactly parallel targets in the real top view, but they will appear diverging as we go uphill in the virtual top view. This diverging view is actually in alignment with the IPM transformed features from the 3D LaneNet’s dual-pathway backbone.

Mathematically, in the above diagram, we have the following equation, where h is the camera height, and z is the height deviation (above is positive) from the road projection plane the x-o-y plane).

Suppose we have two parallel lane lines in 3D, so the lane width Δx is constant. In the real top view, they will still appear parallel as the lane width is still Δx. In the virtual top view, the lane width becomes Δx̄ = Δx*h/(h-z), if the z becomes larger for an uphill road (z < h), the lane width becomes wider and appear divergent.

Gen-LaneNet still uses an anchor-based representation and thus suffers from the same drawbacks as 3D LaneNet. A much better way maybe is to combine the advantages of both methods: use the virtual top view and decoupled pipeline of Gen-LaneNet and the semi-local tile representation of Semi-local 3D LaneNet.

Datasets

One of the main reasons for the limited studies in the 3D lane one detection field is the lack of relevant datasets and benchmarks. There are mainly three data sources to build a 3D lane line dataset: synthetic data, depth measurements (with lidars or possibly stereo cameras), and temporal sensor fusion.

3D LaneNet and Semi-Local 3D LaneNet used the simulated environment Blender to generate a large amount of synthetic data with known 3D groundtruth. Similarly, Gen-LaneNet leveraged the simulator inside the Apollo project and generated the Apollo 3D Synthetic lane line dataset.

3D LaneNet and Semi-Local 3D LaneNet also collected 2.5 hrs of real data with lidar detector and semi-manual annotation, but only to validate the idea. Collecting multi-sensor data of calibrated and synchronized camera and lidar data is not quite scalable. In addition, lidar-based dataset essentially can only go up to ~50 meters as it is lidar to reliably detect lane lines beyond that.

Another more scalable way to get real 3D lane line data is similar to what described in MonoLayout (WACV 2020). It uses self-generated ground truth by aggregating results throughout a video (so-called temporal sensor fusion), given that accurate ego-motion information can be obtained. The aggregated groundtruth can be truncated according to a preset distance threshold and projected back to single image frames. The same idea applies to lidar data as well if we want to see beyond the abovementioned limit of 50 meters.

Takeaways

Predicting camera extrinsic parameters to perform a homography transformation (IPM) on feature maps seems to be the standard practice.

seems to be the standard practice. The virtual top view addresses the misalignment between the transformed feature map and the generated groundtruth.

addresses the misalignment between the transformed feature map and the generated groundtruth. The anchor-based representation will fail due to complex lane line topologies (roundabout, vertical lane lines, etc) in city driving scenarios. Prediction on semi-local tile followed by clustering seems to be a much more flexible way to handle complex geometries.

will fail due to complex lane line topologies (roundabout, vertical lane lines, etc) in city driving scenarios. Prediction on seems to be a much more flexible way to handle complex geometries. Synthetic dataset and sim2real is the way to go to bootstrap the 3D lane line detection system, especially given the open-sourced Apollo dataset. The scalable method to build 3D lane line datasets from temporally aggregated single image perception results is still underexplored. I am looking forward to more work in this direction in the future.

is still underexplored. I am looking forward to more work in this direction in the future. For future work, it may be better to use the virtual top view and decoupled pipeline of Gen-LaneNet and the semi-local tile representation of Semi-local 3D LaneNet.

References