# TAKEN: A Traffic Knowledge-Based Navigation System for Connected and Autonomous Vehicles
## 3 TAKEN-Traffic Knowledge-Based Navigation for CAVs
### Traffic Knowledge Generation-Analysis Module
#### 3.1.2 Visual Perception  
For any agents or objects on the road, objects and event identifications are required. Cameras are very important for an SDC because the data generated by such an instrument are very close to what humans can perceive. Therefore, different types of computations can be performed on such data to obtain valuable knowledge. Cameras are generally characterised by metrics such as field of view, resolution, dynamic range, focal length (\(f\)), depth of field and frame rate. Cameras are used in tasks such as depth estimation, segmentation, object detection and localisation, lane estimation, etc.  
In the TAKEN system, depth information is obtained using stereo cameras with the assumption that the setup has two cameras whose focal lengths are equal and optical axes are parallel to each other. Figure 3A shows how the stereo camera is set up in an ideal case.  
Figure 3B is the bird's eye view of the stereo system. 'Z' is the depth and our goal is to calculate 'Z' along with the sector formed by the object of interest. \"\(X\)-axis\" represents the virtual screen in the 3D space. 'O' is the object in the real world. 'O_I' and 'O_r' are the images formed on the virtual screens of left and right RGB cameras respectively. Triangles formed by 127 and 139 are similar. Therefore,  
\[\frac{Z}{f}=\frac{X}{X_{L}} \tag{5}\]  
Similarly, triangles formed by 458 and 469 are similar and  
\[\frac{Z}{f}=\frac{X-b}{X_{R}} \tag{6}\]  
Figure 3: Stereo system setup showing the stereo camera setup in (**A**) and a bird’s eye view of stereo system in (**B**).  
Figure 2: Handling cases of sensor failure.  
Figure 4 depicts the aerial view of different scenarios encountered by the self-driving agent. Figure 4A represents the case wherein overtaking is possible by the self-driving agent. Figure 4B, on the other hand, highlights a situation in which the self-driving agent has decided to overtake but encounters another vehicle approaching it and renders overtaking risky. Figure 4C shows a scenario in which a pedestrian obstructs the self-driving agent's drive path. The green-coloured line represents the path that the car must take to reach the destination. The blue and red curves are the consequence of low-level decision making, red meaning invalid/risky path and blue depicting a safe path. The red-coloured car is the self-driving agent and others are non-players (i.e., other actors in the scene). The equivalent scenarios from the driving agent's perspective as in the Carla simulator are shown in Figure 5.  
We can calculate disparity as follows:  
\[\text{Disparity, }d=X_{L}-X_{R} \tag{7}\]  
where, \(X_{L}=P_{l}-P_{0}\) ; \(X_{R}=P_{r}-P_{0}\); and P = (U,V) represents a pixel in the pixel coordinate system (see Figure 3).  
Therefore, we can compute depth as:  
\[Z*X_{L}=f*X\ \ \&\ \ Z*X_{R}=(f*X)-(f*b) \tag{8}\]  
\[Z=\frac{f*b}{X_{L}-X_{R}}=\frac{f*b}{d} \tag{9}\]  
Figure 4: Three example scenarios (**A**–**C**) with their aerial views.  
Figure 5: Three example scenarios (**A**–**C**) self-driving agent’s perspective.  
Equation (9) gives us the depth of an object identified on the virtual screens. This is used to estimate the sector of impact with respect to an object detected in the image and is computed using Equation (10). The sector of impact is calculated as a function of camera intrinsic metrics and the angle formed by the two extreme points of the detected object. Thresholding this information can help us estimate the state of static and dynamic objects in the vehicle's vicinity.  
\[pedal\_position=F(cos^{-1}(\frac{\vec{P}\vec{Q}}{|P|.|Q|}),camera\ intrinsics) \tag{10}\]  
where P and Q are vectors of the form [x,y] such that (x,y) is the point in the image.  
This is used by the behavioural model in the process of decision making. In order to detect objects in the scene, the work trains the YOLOv3 [10; 12] model on the custom dataset containing labelled synthetic images of both static and non-static objects extracted from the Carla simulator.