# Tightly-Coupled GNSS/Vision Using a Sky-Pointing Camera for Vehicle Navigation in Urban Areas
## 4 Camera-Based Non-Line-Of-Sight Effect Mitigation
### Image Segmentation-Based NLOS Mitigation Algorithm  
Image segmentation is a process that consists of partitioning the image into two or more portions called segments. The resulting output image is a simplified version of the input image that depicts \"meaningful\" regions or objects only. The NLOS mitigation method used in this paper consists of detecting and rejecting the obstructed satellites that are the principal sources of the multipath and NLOS errors. Our method uses a sky-pointing camera to acquire the GNSS receiver antenna's surrounding environment [14; 31; 58]. The captured images need to be segmented into sky and non-sky. The tracked satellites are then reprojected onto the segmented image. Finally, the satellites that lie on the non-sky areas in the image are removed before computing the final position solution. The concept has been presented in our previous work [14] where we evaluated various segmentation algorithms in different weather conditions. The steps of this method are depicted in Figure 3.  
However, segmenting an image into sky and non-sky is not an easy task due to lighting variations, weather conditions and the facades of buildings (e.g., buildings with glass). Figure 4A,B,D, shows few challenging images to segment (in sunny and cloudy weather conditions). Figure 4: Example of complex downtown images taken using an upward-facing camera. Correct image segmentation can be affected by the environment illumination, building facades (reflections observed on the glasses facades). Left (**A**,**B**): sunny weather condition; right (**C**,**D**): cloudy weather condition. Figure 3: NLOS effect mitigation algorithm steps: “Data Acquisition” box: synchronized GNSS receiver and monocular system (down) and an example of the captured image (top); “Image Segmentation” box: the blue area in the segmented image represents the sky and the black area is the non-sky area (obstacle); “Satellite Visibility” box: the green circles are the non-obstructed satellites, and the red rectangles are the obstructed satellites that will be ignored. Using conventional image segmentation algorithms, most of them initially developed for different purposes than outdoor navigation (e.g., biomedical applications), will produce poor results. Figure 5 shows the Otsu method segmentation applied to binarize an image. Figure 5 shows the Otsu method without pre-processing. Figure 5 shows the Otsu method with pre-processing that we introduced in [14]. It can clearly be seen that accurately segmenting this image is challenging, and segmentation errors are observed both in Figure 5 and Figure 5. Surrounding buildings and sky are reflected and visible on the buildings with glass facades. Partitioning such images in a way that pixels belonging to sky and buildings share the same label with regard to features such as colour or texture is generally difficult to achieve. In this work, we introduce an improved segmentation approach tailored for vision-based outdoor vehicular navigation applications. The algorithm consists of using edges in order to partition an image into sky and non-sky segments. Objects (buildings, sky, trees, etc.) in an image visually occupy regions. These regions are separated by using algorithms such as the canny edge detector [59]. The centre of the image is assumed to be a sky segment. This assumption makes sense, since in normal conditions, a sky-pointing camera mounted on the top of a land vehicle driven on the road will not be inclined in a such way that the centre of the image is obstructed by a building. From the middle of the image, the following pixels are \"filled\" as sky until we reach an edge. Then, anything beyond the edge is non-sky. However, as shown in Figure 4, clouds (or sun in some other cases (not shown)) can cause the sky area to appear non-homogeneous. The canny edge detector in this case will detect edges within sky area and thus leads to segmentation errors.