# Autonomous Driving Assistance with Dynamic Objects Using Traffic Surveillance Cameras
## 3 Problem Formulation  
The traffic surveillance camera image used as input information in this study has a high resolution in the horizontal (u) and vertical (v) directions to detect long distance objects, whereas in visualization precision, aerial and general maps are used. Therefore, in order to map the exact position of an object tracked in an input image, a transformation matrix capable of explaining the differences (translate, rotation, and scale) between coordinate systems is required. We assumed that our camera follows the general pinhole camera model [27]. In this model, the perspective projection of a 3D point into the 2D image point can be represented as follows:  
\[\lambda\ P_{c}=[K|C][R|T]P_{w} \tag{1}\]  
where,  
\[[K|C]= \begin{bmatrix}f_{x}&s&c_{x}\\ 0&f_{y}&c_{y}\\ 0&0&1\end{bmatrix} \]  
\[[R|T]= \begin{bmatrix}r_{11}&r_{12}&r_{13}&t_{x}\\ r_{21}&r_{22}&r_{23}&t_{y}\\ r_{31}&r_{32}&r_{33}&t_{z}\end{bmatrix} \]  
\[R=R_{z}(\phi)\ R_{x}(\theta)= \begin{bmatrix}cos\phi&-cos\phi sin\phi&sin\phi \\ sin\phi&cos\phi cos\phi&-sin\phi cos\phi\\ 0&sin\theta&cos\phi\end{bmatrix} \]\([K|C]\) is the matrix of the camera's intrinsic parameters, i.e., focal length, skew, and the optical center of the camera. \([R|T]\) is the matrix of the camera's extrinsic parameters, the \(3\times 3\) rotation matrix and the \(3\times 1\) translation matrix of the camera, respectively. R is referred to as a pan-tilt camera model. The camera calibration problem is the problem of estimating \([K|C]\) and \([R|T]\). \([K|C]\) depends only on the intrinsic parameters of the camera. \([R|T]\) is the extrinsic parameters of the position of the cameras (camera translation positions \(t_{x},t_{y}\), \(t_{z}\) and rotational angles \(\theta\) and \(\phi\)), and projective depth (\(\lambda\)). We assume that the traffic surveillance cameras have zero pixel skew and an equal aspect ratio [28,29] and that the image center is the same as the optical center. The road section was also assumed to be flat and straight. It is a pan-tilt camera model. We rewrote \([K|C]=diag(f,\,f,\,1)\) and \(R=R_{z}(\phi),\,\,R_{x}(\theta)\). Figure 3 shows the calibration procedure. For this purpose, as shown in Figures 4 and 5, invariable feature points (lanes, road markings, crosswalks, etc.) were defined in the traffic surveillance camera image of the target location for object location information visualization, and road information was measured and produced in the world coordinates with precision on the map. The map was built in high definition for automated vehicles. Corresponding points were required to match the coordinates. The corresponding points on the map with image feature points and precision were performed manually to maintain consistency. Here, when a feature point for a partial region of an image is used, the feature point defined in the image should be defined to cover as wide a range as possible because the region without the feature point may be distorted and the location of the object may be incorrectly mapped. In this study, after defining 25 invariant feature points for each image to calculate the transformation relationship between coordinate systems, the position of the object was transformed and visualized using the transformation equation. To check the transformation relationship between the two coordinate systems with different dimensions in the traffic surveillance camera image and visualization, a transformation matrix was constructed using a point on the world coordinate system that matches a specific point on the traffic surveillance camera image together with the intrinsic and extrinsic parameters. A method of calculating and extracting a large number of invariant feature points from an image and mapping the corresponding feature points in the world coordinate system to calculate a transformation matrix such as a homography matrix  
Figure 4: Road line and critical path in the HD map and aerial image. Figure 3: Data processing flowchart with estimated parameters.