# Survey of Deep Learning-Based Methods for FMCW Radar Odometry and Ego-Localization
## 5 Deep Learning Approaches
### Fusion and Cascading Architectures
#### 5.3.2 Lidar Fusion  
Over recent years, many authors have implemented deep generative models to generate specific synthetic data from custom input data, where the generation of radar to lidar is frequently performed, as expensive lidar data can be exchanged for cheap radars with a generative model. The generative model not only performs translation between sensors, such as lidar to radar [128] or radar to lidar [101], but also translations between lidar representations in different weather conditions [129]. Based on generative modelling, [PERSON] et al. [101] introduce a GAN to convert a spinning radar scan into a synthetic lidar map. Instead of sharing polar features, this network generates a synthetic lidar BEV image from a radar scan. The synthetic lidar output aims to remove the clutter in the radar scan while generating features similar to a lidar scan. The synthetic lidar scan is then used by the authors along with a prior map for odometry and ego-localization in a particle filter localization scheme. The odometry is estimated using point-to-plane ICP between consecutive lidar scans. Localization is performed using a metric based on number of matched points to the map. [PERSON] et al. [102] perform odometry on a prior lidar map with a known position using a differentiable Kalman-Filter approach shown in Figure 9. A cartesian 2D radar scan and a lidar submap are fed through several U-Net architectures to extract two separate feature embeddings from each input sensor. The lidar embedding is rotated by a range of angles \(\delta\theta\) and translated by a range of possible solution factors \(\Delta x\) and \(\Delta y\). The difference tensor between the radar embedding and each transformed lidar embedding is split up into patches and each patch is fed through another network to obtain a patch based difference. This difference is averaged and its softmin is taken to finally generate marginal probability distributions for each solution candidate, \(\delta x\), \(\delta y\) and \(\delta\theta\). The model is trained with three different loss functions. The first is a cross-entropy loss of the found marginal probability distributions using a one-hot encoding of the ground truth parameters. The second loss is the mean squared error between the ground truth parameters and the resulting estimated parameters from the marginal distributions and corresponding offset. Finally, the last loss is based on a differentiable Kalman-Filter to balance between the likelihood and a lower uncertainty in the posterior. In a similar fashion to the approaches presented in [101; 102], [PERSON] et al. [103] utilize deep learning to localize a radar scan on a lidar map. Specifically, a U-Net network is employed to extract a mask from the radar scan, similar to the methodology by [85]. Concurrently, a point cloud is extracted from the radar scan using the BFAR detector [130], with features from the mask sampled at the extracted points serving as weight values. A significant part of this method relies on a differential ICP backend, which matches the extracted points with a reference lidar map. [PERSON] et al. [131] implement a GAN network for moving object detection in lidar data. Although radar data is not used directly by the deep learning module, the Doppler velocity of the radar scan acts as a verification tool to ensure that dynamic objects have been accurately determined. Additionally, lidar data are utilized to filter ghost objects in the radar data. [PERSON] et al. [104] conduct joint place recognition of radar and lidar scans by employing a shared encoder-decoder architecture for both the polar representation of a radar scan and a lidar submap. The output features are transformed into a BEV representation, and a 2D FFT is applied to generate a final vector descriptor. The network is trained using a joint training scheme, optimizing the loss with different sensors for each query sample (anchor, positive, negative). Similar to the work by [PERSON] et al. [104], the work by [PERSON] et al.