## 4 Semantic Information Extraction  
### Virtual LiDAR Data Generation  
Conventionally, LiDAR can accurately measure the distance information of objects in the environment relative to the robot itself, and the virtual radar proposed in this work obtains similar distance measurement information through the secondary processing of semantic segmentation results, simulating the detection results of LiDAR sensors. Since  
Figure 1: Semantic-structure-aware global orientation estimation framework.  
most of the area in the original image is the ground and objects on the ground, we considered using IPM to convert the original image to a top view and synthesized it to obtain a surveillance-synthesized image. The process of IPM transformation can be described as  
\[\left[\begin{array}{c}x_{vch}\\ y_{vch}\\ 1\end{array}\right]=\lambda\left[\begin{array}{cc}\mathbf{R}_{c}&\mathbf{t}_{c} \end{array}\right]^{-1}_{\text{col:1,2,4}}\pi_{c}^{-1}\left(\left[\begin{array} []{c}u_{fish}\\ v_{fish}\\ 1\end{array}\right]\right), \tag{1}\]  
where \(\left[u_{fish},v_{fish}\right]^{T}\) denotes the pixel coordinates of the fish-eye camera image, \(\pi_{c}(\cdot)\) denotes the projection model with its inverse projection model as \(\pi_{c}^{-1}(\cdot)\), \(\left[\begin{array}{cc}\mathbf{R}_{c}&\mathbf{t}_{c}\end{array}\right]\) denotes the external parameters of the camera, i.e., the homogeneous transformation matrix from the robot coordinate system to the camera coordinate system, and \(\left[\begin{array}{c}\mid_{\text{col:}k}\end{array}\right]\) denotes the \(k\)-th column of the matrix. \(\lambda\) denotes the scaling factor, which can be obtained by calibrating the correspondence between the transformed image and the actual scene size, and \(\left[\begin{array}{cc}x_{v\text{e}h}&y_{vch}\end{array}\right]^{T}\) denotes the position of the point in the final robot coordinate system.  
After IPM transformation, a virtual camera directly above the robot and shot vertically downward can be further constructed, which can map the points under the obtained robot coordinate system to obtain the final synthesized surround view image as  
\[\left[\begin{array}{c}u_{ipm}\\ v_{ipm}\\ 1\end{array}\right]=\mathbf{K}_{ipm}\left[\begin{array}{c}x_{vch}\\ y_{vch}\\ 1\end{array}\right], \tag{2}\]  
where \(\left[\begin{array}{cc}u_{ipm}&v_{ipm}\end{array}\right]^{T}\) denotes the pixel coordinate of the synthesized surround view image and \(\mathbf{K}_{ipm}\) denotes the internal parameter matrix for the virtual camera.  
To obtain the segmentation results of free spaces, this paper trained a semantic segmentation network model to distinguish between passable space and non-passable space. Then, through the morphological processing at the image level, the segmentation results can be further modified to obtain a more ideal segmentation effect, and the boundaries of the passable area can be further extracted. Since the scale transformation coefficient between the pixel distance of the image and the actual distance can also be obtained at the same time during the calibration process of surround view synthesizing, the pixel distance between the point on the passable area boundary in the image and the pixel distance of the image center can be directly converted into the actual distance under the robot coordinate system or virtual radar sensor coordinate system. Then, according to the scanning method using LiDAR, all boundary points are sampled at fixed angular intervals, and the boundary points in the same angle window can be represented by the closest point so that the final virtual radar measurement data can be obtained. The process of virtual LiDAR data generation is shown in Figure 2.  
Figure 2: The process of virtual LiDAR data generation.