# Tightly-Coupled GNSS/Vision Using a Sky-Pointing Camera for Vehicle Navigation in Urban Areas
## 4 Camera-Based Non-Line-Of-Sight Effect Mitigation
### Image Segmentation-Based NLOS Mitigation Algorithm
To avoid these kinds of issues, the developed algorithm includes four steps:  
* Image filtering: Given that the images captured using a sky-pointing camera are highly corrupted by bright (e.g., sun light) and dark (e.g., buildings or clouds) objects/structures, we adopted a sequential application of open-close filters denoted Alternate Sequential Filter (ASF) [60]. Indeed, we observed that when the noise is wide-spread over an image, using a single open-close filter with a large structuring element leads to segmentation errors (bright objects tend to be lost and the difference between the sky segment and the other structures in the image becomes hard to observe). ASF provides efficient results since it alternates openings and closings proceeding in an incremental way from small to a given size of the structuring element \(m\), \(m\geq 1\)[60]. Consider \(\Omega_{m}\) and \(\varsigma_{m}\) the morphological opening and closing of size \(m\), respectively. The ASF is a sequential combination of \(\Omega_{m}\) and \(\varsigma_{m}\) such as \(\gamma_{m}=\Omega_{m\varsigma_{m}}\) is a morphological filter. Thus, we have: \[ASF_{m}=\gamma_{m}\ldots\gamma_{2}\gamma_{1}\] (8) For illustration, if \(m=2\), we have \(ASF_{2}(I)=\Omega_{2}[\varsigma_{2}(\Omega_{1}[\varsigma_{1}(I)])]\), where \(I\) is the image to filter. The result obtained by using the filter defined in Equation (8) is a less noisy image than the original image. The different portions of the image are more obvious after this step. However, the output of the ASF still provide edges within the sky areas. Since our approach uses edges to  
Figure 5: Image segmentation using the Otsu methods. (**a**) Original; (**b**) Otsu; (**c**) Improved Otsu [14]. categorize sky and non-sky, it is important to remove such edges. For this reason, the levelling algorithm [61] is used along with ASF to find a good trade-off between good identification of edges between sky and buildings and suppression of edges within the same structures (sky and other objects in the image). * Colour space conversion: once the image is filtered, we determine that the perceptual brightness (luminance) of the image is enough to accurately distinguish the main partitions contained in the image since it depicts sharp details. For this reason, the RGB (Red, Green and Blue) image is converted to the Luv colour space. The luminance channel L is then extracted for further processing. * Edge detection: the luminance channel extracted from the filtered image is smooth and suitable for edge detection with limited errors. The edge detection here consists of finding discontinuity in the luminance of the pixels within the image. The well-known canny edge detector [59], which consists of smoothing the image with a Gaussian filter, computing horizontal and vertical gradients, computing the magnitude of the gradient, performing non-maximal suppression and performing hysteresis thresholding, is used in this paper. * Flood-fill algorithm application: At this point, the decision should be made on which edges mark the limit between sky and non-sky areas. The flood-fill step is initialized by assuming the pixel at the centre of the image as belonging to the sky category. Then, the pixels from the centre of the image are filled until we reach an edge. In other words, we used the output of the edge detector algorithm as a mask to stop filling at edges. This process is illustrated in Figure 6.  
Figure 7 shows the steps of the image segmentation algorithm we implemented. The original image is a relatively complex image containing sun light, clouds, reflective building facades, etc. The second box represents a smoother (filtered) image. The small bright spots observed on the building facades are filtered out, and structures look sharper and less influenced by the sun. The third box depicts the luminance channel of the filtered image represented in the Luv colour space. Dominant segments are kept, and lighting effects decrease, while less dominant segments look smoother.