## 4 Proposed Moving Object Detection Method  
In this section, the overall process of the proposed method is explained in detail. First, since the two sensors are used simultaneously, we applied an extended Kalman filter [34] to calibrate the IMU sensor and the camera. The two sensors are combined in one platform; the geometric relation between the two sensors is shown in Figure 3. \([G]\) stands for the global coordinate, \([l]\) stands for the IMU coordinate, and \([C]\) stands for the camera coordinate. The movement of global coordinate (\(T_{G}\)) is obtained from the IMU sensor, and this can be applied to the image point (\(P\)) with the calibration information of the two sensors (\(T_{C}\)). In the consecutive image sequences, the feature points are obtained using a Harris corner detector. The pixels with a value above the minimum threshold are considered interesting points. Next, the SIFT descriptor [28] is applied to find the matching relation between the two consecutive images. Many  
Figure 3: Geometric relation between camera and IMU (Inertial Measurement Unit) sensor. Figure 2: The proposed background classification method. kinds of detectors and descriptors have been studied [35]. The performance of moving object detection varies from changes of these variables. We did several tests to choose these variables, finally deciding on the appropriate parameters that suited our configuration. An example of feature points and their matching relations is displayed in Figure 4. In the example, the cars are stopped at a red traffic light, except for the taxi on the right which is going to make a right turn. Using IMU data, the transformation matrix of consecutive images can be calculated. If the movement is smaller than the threshold, the situation is defined as \"static camera\" case, and the moving feature point is simply classified by the Euclidean distance since there is no egomotion. Otherwise, if the movement exceed the threshold, the situation is defined as \"motion camera\" case, and the proposed method using epipolar line is applied to classify the feature points. We use the IMU sensor to calculate the initial fundamental matrix. The fundamental matrix (\(F\)) can be calculated in the stereo vision system by the following equation [32]. \[F=K^{\prime-T}RK^{T}[KR^{T}t]_{X}, \tag{1}\]  
where \(R\) and \(t\) are the rotation and translation matrix from the two cameras, and \(K^{\prime}\) and \(K\) are the intrinsic parameters of the two cameras. Since in this case the two images are captured with a single camera, the two parameters are the same (\(K^{\prime}=K\)). The rotation and translation data in the consecutive images are obtained from the IMU sensor. If the obtained IMU data is referred as \(A_{k},~{}A_{k-1},~{}b_{k},~{}b_{k-1}\) from the frame \(k\) and frame \(k-1\), the following equations can calculate the rotation \(R\) and translation t of the consecutive images. \[A_{k}\times A_{k-1}^{-1}=R, \tag{2}\]  
\[b_{n}-b_{n-1}=t. \tag{3}\]  
The fundamental matrix and feature points are used to draw an epipolar line on the consecutive images. Figure 5 shows the epipolar lines drawn using the IMU sensor on the three example images. The equation below defines a mapping between feature points and epipolar lines. \[\overline{p_{r}}^{T}F\overline{p_{l}}=0, \tag{4}\]  
\[e_{r}=F\overline{p_{l}}, \tag{5}\]  
\[e_{l}=F^{T}\overline{p_{r}}, \tag{6}\]  
where \(\overline{p_{l}}\) and \(\overline{p_{r}}\) are the feature points from the left and right images. \(e_{r}\) is the right epipolar line, and \(e_{l}\) is the left epipolar line. Figure 3:0 Feature points and matching relations example. If the feature point is moved only by the camera movement, the corresponding feature point of the consecutive image might be placed on the epipolar line. However, if the feature point is the moving object, the feature point may not be on the line. Therefore, the orthogonal distance of the epipolar line and the feature point can be used to classify the moving object.