# Tightly-Coupled GNSS/Vision Using a Sky-Pointing Camera for Vehicle Navigation in Urban Areas
## 1 Introduction
An improved visual odometry and a robust visual localization using an omnidirectional camera are presented in [20; 21], respectively. In the development of their improved omnidirectional visual odometry, the authors in [20; 21] adapted the epipolar constraint to the omnidirectional camera sensor. They also improved the matching process by propagating the current uncertainty of the system. Overall, their method has proven to be robust because their adaptive matching process reduces false positives. Another \"stand-alone\" solution available in the literature consists of using map data as presented in [22; 23]. In order to facilitate the segmentation of the camera images into sky and non-sky, [PERSON] and [PERSON] [23] use an infra-red camera, while a visual spectral camera is used in [22]. Moreover, [PERSON] and [PERSON] [22] proposed an improved image segmentation algorithm tailored for sky-pointing camera-based navigation. Practically, both methods consist of determining the vehicle position by matching skylines obtained from a 3D building model with the skylines observed with a camera mounted on the top of the vehicle. Using the skyline as a fingerprint reduces the vehicle localization problem to a place recognition problem. This approach could be useful in cases where it is difficult to obtain the position solution from any type of sensor since it can provide the user position even in the absence of GNSS. Readers can refer to the mentioned references for more details. Integration between inertial measurement units (IMU) and cameras has also been extensively studied. A method of estimating robot attitude angles by fusing IMU data with a monocular vision system data is proposed in [24]. A six degrees of freedom (6-DOF) inertial sensor was fused with a low cost camera for an autonomous mobile robot navigation system using an extended Kalman filter. The proposed method has been shown to be effective in terms of computation load and attitude estimation accuracy. More than two sensors can also be integrated. In [25], GNSS, INS and Ultra-Wideband (UWB) were integrated in a tightly coupled way using a robust Kalman filter that supported Vehicle to Infrastructure (V2I) communication. Furthermore, a monocular camera, IMU and GNSS were integrated for vehicle navigation in harsh environments in [26; 27]. In [2], the navigation performance of GNSS/INS/vision integration was evaluated via simulated data, where the number of the observed satellites was continually decreased from three to one GPS satellites. As expected, the position error increased as the number of satellites decreased. Finally, GNSS/vision integration has been presented in [28; 29]. The vision system was combined with a differential GPS (DGPS) in [28]. The presented work focused on evaluating the heading obtained from the combined solution for micro-Unmanned Aerial Vehicle (UAV) navigation. Both simulated and field experiments showed that the DGPS/vision approach outperformed the fused onboard data (autopilot, magnetometer). A deeply-coupled GNSS/vision approach for vehicle navigation was introduced in [29]. The GNSS/vision system was able to provide continuous navigation in challenging environments that outperformed the GNSS/INS system. The research proposed in this paper fits in this category with the contrast that this work uses only line-of-sight satellites selected by segmenting an upward-facing camera image into sky and non-sky areas. Moreover, our tightly-coupled implementation takes care of the scale factor issue that affects monocular vision systems. It should be noted that the above-mentioned approaches do not represent an exhaustive list of sensor fusion approaches that exist in the literature. An autonomous car such as Faraday's FF 91 fuses more than 30 sensors including cameras, radars and ultrasonic sensors. This list of approaches is given for illustration of what exists in the literature pertaining to the integration of optical sensors with various other types of sensors.