Motion Interpretation for In-Car Vision Systems

In this paper we consider a single camera, attached to a moving vehicle. Based on the observations, we want to estimate the time-to-contact of vehicles around us. We propose three different approaches for the estimation of this parameter. The measurement input for these approaches respectively consists of the horizontal position in the image plane of a point observed on a vehicle, the vertical position and the observed vehicle width. This measurement data can robustly be obtained in practice. It is possible to automatically select the appmach that provides the most accurate result for a given situation. The influence of disturbances in the observations in practice is suppressed by means of tempomlfiltering. Due to the similar structure of the three approaches, the same temporalfiltering technique can be applied. The practical value of the algorithms is illustrated by means of real video data. 1 I n t r o d u c t i o n For in-car intelligence systems we are interested in how the behaviour of other vehicles on the road limits our driving possibilities. In this paper we consider the application where a single camera is attached to a moving vehicle, typically driving on a highway. Based on observations with the camera, we want to determine the motion of other vehicles around us, relative to our own motion. In this paper we focus on the motion component, parallel to our own motion. In literature, this parameter is usually referred to as the timcto-contact. For many applications the timeto-contact is an important parameter. It expresses the ”nearness” of an object relative to the observer. The camera provides us with information about induced motion components in the image plane of the camera. This motion is the projection of the real world motion of a point on the moving image plane. In order to understand the observed motion of an object point (e.g. located on a vehicle), one needs to know which part of the induced motion is caused by the ego-motion of the camera and which part is the result of relative object motion. In other words: to estimate the observed motion caused by relative motion of a vehicle, the image data should implicitly or explicitly be stabilized. In the following section we briefly address the issue of camera stabilization. Section 3 presents algorithms for the estimation of the motion parameter time-tocontact. In section 4 we show several experimental results based on a real video data. The paper is concluded with a discussion in section 5. 2 Camera s t ab i l i za t ion In this section we briefly summarize our camera stabilization method. A more detailed description can be found in [7]. We start with the definition of 2 coordinate systems, attached to the nodal point of the camera: the real world ceordinate system (X,Y,Z) and the observer co-ordinate system ( X , Y, Z). Both are illustrated in figure 1. The %axis points in the motion direction of our own vehicle. The Z-axis of the observer frame corresponds with the viewingdirection of the camera. In this paper, the position of a point in the image plane of the camera is expressed by the co-ordinate (rZ,ry) (: observer co-ordinate ( T = , T ~ , F ) , with F the focal length of the camera). The motion of our vehicle is predominantly translational. Two aspects of the ego-motion of the camera are important for interpretation of the motion observed in the image plane: (i) the (average) angle between driving direction of the vehicle and the viewing direction of the camera and (ii) the (small) camera rotations around the mean viewing direction. Our estimation techniques employ data observed in the background. In order to be.robust to the large U.S. Government work not protected by US. copyright 135 diversity in appearance of the background only moderate assumptions are made on the appearance of this data. The dynamical characteristics of the mean viewing direction and the inter-frame rotations differ considerably, asking for different estimation techniques. The mean viewing direction can be estimated once in a while. It can be derived directly from the position 'in the image plane' of the so called Focusof-Expansion (FOE). Points observed in the background seem to emerge from this point. Every time our own vehicle is moving in a straight lane, the observed motion of backround points in the image plane is (approximately) along straight lines. These lines virtually intersection in the FOE. We apply least squares estimation techniques to estimate the position of the FOE. The estimation of the (small) inter-frame rotations of the camera can only be based on a small number of images making the solution more sensitive for inaccuracies in its input. Therefore, we base our method on the estimation of the optical flow field of background points, and not on only a small number of tracks. The redundancy in the estimated motion field is exploited to obtain a robust estimation of the inter-frame rotations. An iteratively re-weighted orthogonal least squares technique is used. The information about the viewing direction and the inter-kame camera rotations makes accurate stabilization of the video stream possible. With this information we can transform the observations into the situation where the camera is looking straight ahead (i.e. observer co-ordinate frame aligned with the real world co-ordinate frame). This justifies the simplified camera setup assumed in the next section. 3 Motion interpretation 3.1 The time-to-contact Both our own motion and that of other vehicles is mainly in the %direction. The relative motion of a vehicle, measured in this direction, defines whether a vehicle approaches, retreats or stays at the same distance from the camera. The time-t-contact is defined in this direction. This parameter, denoted by r(t) , expresses how much time elapses until a vehicle reaches us. In case of an approaching vehicle, T is positive. In case of retreating vehicles, T is negative. T is plus or minus infinity when a vehicle stays a t approximately the same distance. Different approaches to estimate T have been described in literature. Examples of general estimation approaches for T are optical flow based approaches [lO][S], feature (point/line) based approaches [2][4][9] and closed contour based approaches [3]. However, the kind of image features these general approaches rely on and/or the sensitive for camera rotations make them impractical for our application. Three alternative approaches have been proposed for our application. These approaches are based on: 1. the vehicle width ([12][5]) 2. the vertical position of the vehicle relative to the 3. the horizontal position of the vehicle relative to FOE ([11[51[61). the FOE, when assuming parallel motion ((61). The first approach can be seen as a variation on the closed contour approaches. When the perspective projection model is applied, the following equation for s(t) holds ([12][5]): Equation 1 only provides us with an approximation for T in case of deformation. In 171 we show that deformation can be neglected for our application. This approach based on the vehicle width is rather insensitive for camera rotations. Major advantage over the general closed contour approach is that the vehicle width is more easily determined than a closed contour. Draw back of this approach is its sensitivity for object occlusion. Instead of the vehicle width, also the distance between headlights, the width of the license plate, etc. can be used. Notice that for practical reasons the vehicle height is less suited for our application. First of all the upper and lower border of a vehicle are generally more difficult to detect compared with the side borders. Furthermore, whereas camera pitch is more severe than yaw, estimation of T based on vehicle height will be less accurate. The second approach can be seen as a feature based approach combined with the assumption of planar motion. Knowledge about the position of the FOE together with the planar motion assumption provides enough information to relate the vertical position of a point in the image plane to its distance to the camera. Consider a camera, positioned at height H above the road surface and with viewing direction aligned to the driving direction. When a vehicle is observed above image line T y ( t ) in the image plane. we can estimate T using the following relation: Advantage of this approach is that. it is relatively easy to accurately determine the height a t which the vehicle is observed. Furthermore, this approach is