Abstract:
A method to equip a vehicle to perform object detection and tracking and a surround view camera system to perform the object detection and tracking involve two or more cameras arranged respectively at two or more locations of the vehicle. The cameras capture images within a field of view of the two or more cameras. A processing system obtains the images from the two or more cameras and performs image processing to detect and track objects in the field of view of the two or more cameras.

Description:
CROSS REFERENCE TO RELATED APPLICATION 
       [0001]    This application claims the benefit of priority of U.S. Provisional Application No. 62/324,602 filed Apr. 19, 2016, the disclosure of which is incorporated herein by reference in its entirety. 
     
    
     INTRODUCTION 
       [0002]    The subject disclosure relates to a surround view camera system for object detection and tracking. 
         [0003]    Cameras are increasingly used in vehicles (e.g., automobiles, construction equipment, farm equipment, automated manufacturing facilities) for automation and safety systems. Surround-view or rear cameras provide images that facilitate an enhanced view during parking, for example. Forward-looking cameras are used alone or in combination with other sensors (e.g., radar, lidar) to detect and track objects and enable semi-autonomous driving, for example. However, the field of view of the forward-looking camera is insufficient in many scenarios. For example, in a parking lot, in which other vehicles or pedestrians may be approaching from any direction, the forward-looking camera system cannot detect a potential threat of collision. As another example, when an adjacent vehicle changes lanes without allowing sufficient space, that vehicle may not be detected by a forward-looking camera system. Accordingly, it is desirable to provide a surround view camera system for object detection and tracking. 
       SUMMARY 
       [0004]    In one exemplary embodiment, a surround view camera system in a vehicle includes two or more cameras arranged respectively at two or more locations of the vehicle. The two or more cameras capture images within a field of view of the two or more cameras. The system also includes a processing system to obtain the images from the two or more cameras and perform image processing to detect and track objects in the field of view of the two or more cameras. 
         [0005]    In addition to one or more of the features described herein, the processing system performing image processing includes the processing system pre-processing each of the images individually including de-warping each of the images. 
         [0006]    In addition to one or more of the features described herein, the processing system being configured to perform image processing includes the processing system being configured to perform visual recognition techniques to detect the objects in each of the images in which the objects appear. 
         [0007]    In addition to one or more of the features described herein, the processing system being configured to perform image processing includes the processing system being configured to perform inter-image detection to detect the objects based on overlapping areas in the images obtained by the two or more cameras. 
         [0008]    In addition to one or more of the features described herein, the processing system being configured to perform image processing includes the processing system being configured to perform temporal detection on a frame-by-frame basis to track movement of the objects. 
         [0009]    In addition to one or more of the features described herein, the processing system is further configured to obtain vehicle dynamics information about the vehicle. 
         [0010]    In addition to one or more of the features described herein, the processing system is further configured to obtain information from other sensors of the vehicle, the other sensors including a radar system, a lidar system, or an ultrasonic sensor system. 
         [0011]    In addition to one or more of the features described herein, the processing system is further configured to output information about the locations of the objects in the field of view of the two or more cameras in a vehicle coordinate system. 
         [0012]    In addition to one or more of the features described herein, the processing system is further configured to present the objects in the field of view of the two or more cameras in a three-dimensional bounding box (BBOX). 
         [0013]    In addition to one or more of the features described herein, the processing system is further configured to provide information about the objects in the field of view of the two or more cameras to a controller in the vehicle, the controller being configured to control safety and autonomous systems of the vehicle. 
         [0014]    In another exemplary embodiment, a method of equipping a vehicle to perform object detection and tracking with a surround view camera system includes arranging two or more cameras at respective two or more locations of the vehicle. The two or more cameras capture images within a field of view of the two or more cameras. The method also includes a processing system obtaining the images from the two or more cameras and performing image processing to detect and track objects in the field of view of the two or more cameras. 
         [0015]    In addition to one or more of the features described herein, the performing the image processing includes pre-processing each of the images individually, the pre-processing including de-warping each of the images. 
         [0016]    In addition to one or more of the features described herein, the performing the image processing includes performing visual recognition techniques to detect the objects in each of the images in which the objects appear. 
         [0017]    In addition to one or more of the features described herein, the performing the image processing includes performing inter-image detection to detect the objects based on overlapping areas in the images obtained by the two or more cameras. 
         [0018]    In addition to one or more of the features described herein, the performing the image processing includes performing temporal detection on a frame-by-frame basis to track movement of the objects. 
         [0019]    In addition to one or more of the features described herein, the performing the image processing includes obtaining vehicle dynamics information about the vehicle. 
         [0020]    In addition to one or more of the features described herein, the performing the image processing includes obtaining information from other sensors of the vehicle, the other sensors including a radar system, a lidar system, or an ultrasonic sensor system. 
         [0021]    In addition to one or more of the features described herein, the method includes the processing system outputting information about the locations of the objects in the field of view of the two or more cameras in a vehicle coordinate system. 
         [0022]    In addition to one or more of the features described herein, the method includes the processing system presenting the objects in the field of view of the two or more cameras in a three-dimensional bounding box (BBOX). 
         [0023]    In addition to one or more of the features described herein, the method includes the processing system providing information about the objects in the field of view of the two or more cameras to a controller in the vehicle and the controller controlling safety and autonomous systems of the vehicle. 
         [0024]    The above features and advantages, and other features and advantages of the disclosure are readily apparent from the following detailed description when taken in connection with the accompanying drawings. 
     
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         [0025]    Other features, advantages and details appear, by way of example only, in the following detailed description, the detailed description referring to the drawings in which: 
           [0026]      FIG. 1  depicts an exemplary embodiment of a surround view camera system according to one or more embodiments; 
           [0027]      FIG. 2  shows exemplary scenarios in which the surround view camera system facilitates detection and tracking of objects according to one or more embodiments; 
           [0028]      FIG. 3  is a process flow of a method of performing object detection and tracking with a surround view camera system according to one or more embodiments; 
           [0029]      FIG. 4  illustrates an exemplary output of the surround view camera system according to one or more embodiments; 
           [0030]      FIG. 5  depicts two exemplary outputs of the surround view camera system according to one of more embodiments; and 
           [0031]      FIG. 6  illustrates another exemplary output of the surround view camera system according to one or more embodiments. 
       
    
    
     DETAILED DESCRIPTION 
       [0032]    The following description is merely exemplary in nature and is not intended to limit the present disclosure, its application or uses. It should be understood that throughout the drawings, corresponding reference numerals indicate like or corresponding parts and features. 
         [0033]    As previously noted, forward-looking camera systems have been used for object detection. The information obtained about objects in front of the vehicle may be used for adaptive cruise control (ACC), automatic emergency braking (AEB), or forward collision warning (FCW), for example. To address other scenarios and to enhance automated systems, information is desirable about objects in proximity to the vehicle that are not necessarily only in front of the vehicle. While surround view cameras provide images around the vehicle, these camera images have not been used for object detection and tracking. Embodiments of the systems and methods detailed herein relate to a surround view camera system for object detection and tracking. As detailed, the surround view camera system is not simply an extension of the processing used in the forward-looking camera system to multiple cameras disposed around the vehicle. Instead, the multiple views can provide enhanced information that cannot be obtained with a single camera image. For example, images from each of the different views are pre-processed, overlapping images are resolved, and images in the different views are used to filter false alarms or adjust detection thresholds. 
         [0034]      FIG. 1  depicts an exemplary embodiment of a surround view camera system  100  according to one or more embodiments. The vehicle  101  shown in  FIG. 1  is an automobile  102 . The surround view camera system  100  includes four cameras  140   a  through  140   d  (generally referred to as  140 ) in the exemplary embodiment shown in  FIG. 1 . Camera  140   a  captures images on the passenger side of the vehicle  101 , and camera  140   c  captures images on the driver side of the vehicle  101 . Camera  140   b  captures images from the front of the vehicle  101 , and camera  140   d  captures images at the rear of the vehicle  101 . In alternate embodiments, fewer or more cameras  140  may be used and can be arranged in other parts of the vehicle  101 . 
         [0035]    The images from the different cameras  140  are sent to the processing system  110  of the surround view camera system  100  for processing. The communication between the cameras  140  and processing system  110  may be over wires that are routed around the vehicle  101  or may be wireless. The processing system  110  may include an application specific integrated circuit (ASIC), an electronic circuit, a processor (shared, dedicated, or group) and memory that executes one or more software or firmware programs, a combinational logic circuit, and/or other suitable components that provide the described functionality. A controller  120  of the vehicle  101  is shown in  FIG. 1 . This controller  120  may be separate from and coupled to the processing system  110 , or, in alternate embodiments, the functionality described for the processing system  110  may be performed by components of the controller  120 . The controller  120  can include or communicate with systems such as the systems that perform ACC, AEB. FCW, and other safety and autonomous driving functions. Additional known sensors  130  (e.g., radar, lidar, ultrasonic sensors) may be incorporated in the vehicle  101  and may be used in the processing of information from the cameras  140 . 
         [0036]    The cameras  140  may include extreme wide angle lenses such that the images obtained by the cameras  140  are distorted (i.e., fisheye images). The extreme wide angle lenses have an ultra-wide field of view and, thus, provide images that facilitate  360  degree coverage around the vehicle  101  with the four cameras  140  shown in  FIG. 1 . The raw images obtained with the extreme wide angle lenses also require pre-processing of the images to unwarp the image distortion or fisheye effect, as further discussed with reference to  FIG. 3 . The pre-processing may also include image enhancement and virtual camera view synthesis. 
         [0037]      FIG. 2  shows exemplary scenarios  210   a  through  210   d  in which the surround view camera system  100  facilitates detection and tracking of objects  220  according to one or more embodiments. Scenario  210   a  shows an object  220 , another vehicle, in a side blind zone of the vehicle  101  that includes the surround view camera system  100 . The field of view (FOV)  201  of the surround view camera system  100  is indicated and shows that a portion of the object  220  is within the FOV  201 . Thus, even if the object  220  is not visible in the side mirror, for example, the surround view camera system  100  will detect the object  220 . 
         [0038]    In scenario  210   b , an object  220 , which is another vehicle, cuts into the lane of the vehicle  101 . A forward-looking camera system may only see the object  220  when it is in the position shown in  FIG. 2 . For example, a typical forward-looking camera system used for ACC has a 50 degree field of view, which is insufficient to capture the object  220 . The object  220  may be in the A-pillar blind spot of the driver and in a blind spot of the forward-looking camera system until it cuts into the lane of the vehicle  101 . According to the one or more embodiments described herein, the surround view camera system  100  can detect and track the object  220 . That is, the object  220  that is cutting into the lane of the vehicle  101  in scenario  210   b  would be detected when it is approaching the vehicle  101  (in the position shown in scenario  210   a ) or when it is on the side of the vehicle  101  (between the positions shown in scenarios  210   a  and  210   b ). By detecting and tracking the object  220  while it is in the FOV  201 , the surround view camera system  100  can better-prepare the driver or automated systems of the vehicle  101  for the cut-in shown in scenario  210   b.    
         [0039]    Scenarios  210   c  and  210   d  show several objects  220  that are in the FOV  201  at various positions relative to the vehicle  101 . The forward-looking camera system would only detect some of the objects  220  shown within the FOV  201  of the surround view camera system  100 . As further discussed with reference to  FIG. 3 , the multiple cameras  140  of the surround view camera system  100  also facilitate identification of false alarms and detection of low-resolution objects  220  based on the several views. 
         [0040]      FIG. 3  is a process flow of a method of performing object detection and tracking with a surround view camera system  100  according to one or more embodiments. At block  310 , the processes include obtaining images from the surround view cameras  140  by the processing system  110 , which may be separate from or part of the controller  120 . Pre-processing the images, at block  320 , can include a number of image processing operations based on the types of images that are obtained. For example, when the cameras  140  have an ultra-wide field of view and provide fisheye images, the pre-processing includes de-warping. Camera  140  calibration parameters can be used for this known procedure. Pre-processing may also include other known procedures such as smoothing and image enhancement. 
         [0041]    Obtaining vehicle information, at block  330 , includes obtaining motion information, for example, what can aid in tracking of objects  220 . Exemplary vehicle information includes speed, angle of motion, acceleration, or information from the global positioning system (GPS) receiver. This information may be provided to the processing system  110  through the controller  120  or directly from other vehicle systems that obtain information about vehicle dynamics. According to alternate or additional embodiments, the vehicle information obtained at block  330  can also include data from other sensors  130  (e.g., radar, lidar) mounted on the vehicle  101 . 
         [0042]    At block  340 , operations are performed to detect and track objects  220  based on the images obtained by the cameras  140 . These operations include known image processing, computer vision, and machine learning operations and may be performed by a deep learning neural network, for example. Known algorithms and processes that may be used as part of block  340  include a deep learning method, for example, a deep convolution neural network (DCNN), or other computer vision methods, such as deformable part models (DPM), along with other visual recognition techniques. The processing at block  340  facilitates organizing and outputting detection and tracking information at block  350 . 
         [0043]    The processing at block  340  includes performing individual frame detection at block  343 . This process may use the known DPM algorithm, for example, to perform detection of objects  220  within each of the individual frames obtained by each of the cameras  140 . Performing inter-image detection, at block  345 , is also part of the processing at block  340 . The inter-image detection operation involves associating and matching objects  220  that are captured by more than one camera  140  of the surround view camera system  100 . Essentially, the position of an object  220  can be triangulated based on the images from two or more cameras  140 . The process facilitates resolving overlapping images by filtering false alarms or adjusting detection thresholds, for example. 
         [0044]    According to the exemplary arrangement shown in  FIG. 1 , the camera  140   a  on the passenger side of the vehicle  101  has an overlapping area in its image field with the camera  140   d  that is located at the rear of the vehicle  101 . If, for example, the processing at block  343  detects an object  220  in a frame obtained by camera  140   a  in the overlapping area but does not detect that same object  220  in a frame of an obtained by camera  140   d , then the detection threshold is reduced for processing of the frame from the camera  140   d  (at block  343 ). If the object  220  is still not detected, then the detection of the object  220  in the frame from camera  140   a  may be deemed as a false alarm. On the other hand, if both cameras  140  detect the object  220  and match their detections to determine that they detected the same object  220 , the object location can be estimated from a triangulation technique based on the two (or more) cameras  140 . This is one example of the inter-image detection processing (at block  345 ) to resolve objects  220  based on images obtained by the different cameras  140  of the surround view camera system  100 . 
         [0045]    Performing temporal detection, at block  347 , is also part of the processing at block  340 . The position of an object  220  that is detected (according to block  343  or, additionally,  345 ) is tracked in time based on its location from one frame to the next. While the temporal tracking (at block  347 ) relies on detection at block  343  or  345 , the temporal tracking (at block  347 ) may enhance the detection at block  343  or  345 , as well. For example, an object  220  that may otherwise be dismissed as a false alarm may instead be determined to have moved out of an overlapping area of coverage of two cameras  140  based on the temporal detection at block  347 . The temporal detection (at block  347 ) facilitates determining the movement of an object  220  relative to the vehicle  101 . For example, a determination of whether an object  220  is moving toward or away from the vehicle  101  can affect information provided to other vehicle systems (e.g., ACC, AEB) through the controller  120 . That is, an object  220  moving away from the vehicle  101  may not be used to trigger the AEB system while an object  220  moving toward the vehicle  101  may trigger the AEB system. 
         [0046]    As the discussion indicates and as shown in  FIG. 3 , the processing at blocks  343 ,  345 , and  347  is inter-related and can be iterative. In addition, the processing at block  340  can use vehicle information obtained at block  330 . As previously noted, the vehicle information can include information about the dynamics of the vehicle  101  and can additionally include data from other sensors  130 . Some or all of this additional information can be used to resolve objects  220  in any of the processes associated with block  340 . For example, tracking an object  220  using the temporal detection (at block  347 ) can be aided by range information to the object  220  provided by the radar or lidar systems. As another example, information about the speed or trajectory of the vehicle  101  can facilitate enhanced detection of the relative movement of an object  220 . 
         [0047]      FIG. 4  illustrates an exemplary output  410  of the surround view camera system  100  according to one or more embodiments. The output  410  is a stitched-together image of four images  420   a ,  420   b ,  420   c , and  420   d  that correspond with the exemplary camera  140  positions shown in  FIG. 1 . Thus, for example, image  420   b  is an image obtained by camera  140   b  at the front of the vehicle  101 . Objects  220  are indicated within the images  420   a  through  420   d  by bounding boxes as shown, for example. This output  410  may be displayed for the driver and may also be provided to an advance driver assistance system (ADAS) to provide driver alerts or enhanced information, for example. 
         [0048]      FIG. 5  depicts two exemplary outputs  510   a ,  510   b  of the surround view camera system  100  according to one of more embodiments. The outputs  510   a ,  510   b  are both on the vehicle coordinate system. Objects  220  around the vehicle  101  and the trajectory of each of the objects  220  are shown. The projection of information about the objects  220  to the vehicle coordinate system facilitates ease of communication with other vehicle systems by providing a common frame of reference. The outputs  510   a ,  510   b  can be displayed to the driver in addition to being provided to the controller  120  for coordination with other vehicle systems. 
         [0049]    The output  510   a  shows a top-down view that shows the vehicle  101  and five different objects  220  around the vehicle  101 . The angle of each object  220  relative to the vehicle  101  is shown and indicates the direction of travel of each object  220 . The output  510   b  also shows a top-down view of the vehicle  101  and five objects  220  around the vehicle  101 . The objects  220  may be color-coded or coded by pattern, as shown in  FIG. 5 . The coding may indicate direction of travel, relative speed, confidence level in detection of the object  220 , or another characteristic. For example, object  220   a  may be a stationary object while objects  220   b  are moving away from the vehicle  101  (in opposite directions relative to each other) and objects  220   c  are moving toward the vehicle  101  (in opposite directions relative to each other). Alternately, objects  220   b  may be moving in a same direction as the vehicle  101  and objects  220   c  may be moving in an opposite direction as the vehicle  101 . According to yet another embodiment, objects  220   b  may be slower-moving than objects  220   c.    
         [0050]      FIG. 6  illustrates another exemplary output of the surround view camera system  100  according to one or more embodiments. A three-dimensional bounding box (BBOX) is used to indicate each object  220  that is detected by the surround view camera system  100 . Color or pattern coding may be used to indicate additional information about the objects  220 . For example, objects  220   a  may have been detected by one of the side cameras  140   a ,  140   c  ( FIG. 1 ), while objects  220   b  may have been detected by a front or rear camera  140   b ,  140   d.    
         [0051]    While the above disclosure has been described with reference to exemplary embodiments, it will be understood by those skilled in the art that various changes may be made and equivalents may be substituted for elements thereof without departing from its scope. In addition, many modifications may be made to adapt a particular situation or material to the teachings of the disclosure without departing from the essential scope thereof. Therefore, it is intended that the present disclosure not be limited to the particular embodiments disclosed, but will include all embodiments falling within the scope thereof.