Abstract:
A multi-sensor system includes multiple sensors that are integrated onto the same substrate forming a unitary multi-sensor platform that provides a known consistent physical relationship between the multiple sensors. A processor can also be integrated onto the substrate so that data from the multiple sensors can be processed locally by the multi-sensor system.

Description:
BACKGROUND OF THE INVENTION  
         [0001]    Different sensors can be used in vehicles to identify objects and possible collision conditions. For example, there may be an optical sensor, such as a camera, mounted to the roof of the vehicle. Another Infrared (IR) sensor may be mounted in the front grill of the vehicle. A third inertial sensor may be located in yet another location in the central portion of the vehicle. Data from these different sensors is correlated together to identify and track objects that may come within a certain vicinity of the vehicle.  
           [0002]    The measurements from the different sensors must be translated to a common reference point before the different data can be accurately correlated. This translation is difficult because the sensors are positioned in different locations on the vehicle. For example, the sensor located inside the front bumper of the vehicle may move in one direction during a collision while the sensor located on the top of the vehicle roof may move in a different direction.  
           [0003]    One of the sensors may also experience vibrations at a different time than the other sensor. For example, the front bumper sensor may experience a vertical or horizontal movement when the vehicle runs over an obstacle before any movements or vibrations are experienced by the roof sensor. This different movements of sensors relative to each other make is very difficult to accurately determine the precise position and orientation of the sensors when the sensor readings are taken. This makes it difficult to translate the data into common reference coordinates.  
           [0004]    The present invention addresses this and other problems associated with the prior art.  
         SUMMARY OF THE INVENTION  
         [0005]    A multi-sensor system includes multiple sensors that are integrated onto the same substrate forming a unitary multi-sensor platform that provides a known consistent physical relationship between the multiple sensors. A processor can also be integrated onto the substrate so that data from the multiple sensors can be processed locally by the multi-sensor system.  
           [0006]    The foregoing and other objects, features and advantages of the invention will become more readily apparent from the following detailed description of a preferred embodiment of the invention which proceeds with reference to the accompanying drawings. 
       
    
    
     BRIEF DESCRIPTION OF THE DRAWINGS  
       [0007]    [0007]FIG. 1 is a block diagram of a multi-sensor system.  
         [0008]    [0008]FIG. 2 is a block diagram of an alternate embodiment of the multi-sensor system that includes an on-board processor.  
         [0009]    [0009]FIG. 3 is a flow diagram showing how the processor in FIG. 2 operates.  
         [0010]    [0010]FIG. 4 is detailed diagram showing how different elements in the multi-sensor system are electrically connected together.  
         [0011]    [0011]FIG. 5 is a diagram showing how different multi-sensor systems operate together to track objects. 
     
    
     DETAILED DESCRIPTION  
       [0012]    [0012]FIG. 1 shows a multi-sensor system  12  that includes different sensors  16  and  18  that are both integrally attached to or integrally formed into the substrate  14 . Because the two sensors  16  and  18  are integrated onto the same substrate  14 , any forces experienced by sensor  16  are also experienced by sensor  18 . One type of material that is used for substrate  14  is invar. Invar is a rigid metal that has been cured with respect to temperature so that its dimensions do not change with fluxuations in temperature. Any rigid material that is resilient to expansion or contraction with temperature changes can be used.  
         [0013]    Locating the sensors  16  and  18  on the same substrate  14  simplifies the cost of sensor manufacturing and installation. For example, the two sensors  16  can be assembled onto the substrate  14  in a factory prior to being installed on a vehicle. If the two sensors  16  and  18  were not mounted on the same substrate  14 , then each sensor would have to be separately mounted on the vehicle and then calibrated to a known alignment with respect to each other. Even if the two sensors were installed correctly, changes in the shape of the vehicle due to wear, temperature, etc. over time could change the initial alignment between the two sensors.  
         [0014]    Premounting or prefabricating the sensors  16  and  18  on the substrate  14  prior to installation on a vehicle, prevents these alignment errors. Only the substrate  14  of the multi-sensor system  12  has to be mounted to the vehicle, not the individual sensors  16  and  18 . This allows the relative position  20  and alignment between the two sensors  16  and  18  to remain the same regardless of how the substrate  14  is mounted on the vehicle. Wiring is also simplified since only one wiring harness has to be run through the vehicle to the multi-sensor system  12 .  
         [0015]    In one example, the sensor  16  senses an area  24  and the sensor  18  senses an area  22  that are both coincident. One of the sensors may have a wider field of view than the other sensor. There can also be more than two sensors on substrate  14  and any active or passive sensor that provides object detection or vehicle force measurements can be mounted onto substrate  14 . Some examples of sensors include ultrasonic, Infra-Red (IR), video, radar, and lidar sensors.  
         [0016]    Depending on the substrate  14  and the types of sensors, different mounting techniques can be used. The sensors may be separate components that are glued or bolted onto the substrate  14 . If the multi-sensor system  12  is an integrated circuit, then the sensors  16  and  18  may be integrally fabricated onto a silicon or alternative temperature resilent substrate  14  using known deposition processes.  
         [0017]    In one example, sensor  14  is a radar or lidar sensor and sensor  18  is a camera. Combining a video camera sensor with a radar and/or lidar sensor on the substrate  14  provides several advantages. The camera sensor  18  provides good angle resolution and object identification. The radar or lidar sensor  16  on the other hand is very effective in identifying range information.  
         [0018]    Combining the camera video sensor  18  with the radar or lidar sensor  16  on the same substrate  14  allows more effective correlation of camera angle and identification data with radar or lidar range information. For example, the radar sensor  14  may only be able to measure angle of an object to within one-half a degree. Because of the limited angle accuracy of the radar angle readings, it may not be possible to determine from the radar reading along if an oncoming vehicle is coming from the same lane of traffic or from an opposite lane of traffic.  
         [0019]    The video sensor  18  may be able to accurately determine the angle of an object to within one-tenth or one-one hundredth of a degree. By correlating the radar information with the camera information, the location of an on-coming vehicle can be determined more accurately.  
         [0020]    Do to vibration differences and possible inaccuracies in sensor alignment, it may not be possible, within fractional degrees of accuracy, to correlate information with separately mounted sensors. In other words, if the camera angle varies within plus or minus one degree with respect to the radar angle, then the camera data may not be able to refine the radar measurements.  
         [0021]    By mounting the camera sensor  18  and the radar sensor  16  to the same substrate  14 , the relative position and alignment between the two sensors remains essentially the same regardless of physical effects on the vehicle. Thus, the camera data can be correlated with radar data to within fractions of a degree of accuracy.  
         [0022]    In another example, a first sensor may detect one object out in front of the vehicle. A second sensor located somewhere else on the vehicle may detect two different objects in front of the vehicle. Because of vibrations in different parts of the vehicle, a central processor may not be able to determine which of the two objects detected by the second sensor is associated with the object detected by the first sensor. With the multi-sensor system  12 , measurement errors caused by this vehicle vibration is cancelled since the two sensors  16  and  18  effectively experience the same amount of vibration at the same time.  
         [0023]    [0023]FIG. 2 shows an alternative embodiment where a processor  26  is mounted to the substrate  14 . Again the processor  26  can be a standalone component that is rigidly attached to substrate  14 . Alternatively, the processor  26  is a portion of the same integrated circuit that also contains the circuitry for sensors  16  and  18 . The processor  26  can perform signal processing tasks for both sensor  18  and sensor  16  and can also handle communication and diagnostics tasks. Tracks for identified objects are sent over connection  28  to other multi-sensor systems in the vehicle or to a vehicle control system as shown later in FIG. 5.  
         [0024]    In previous multi-sensor applications, each sensor was required to send all data back to the same central processing system. This takes additional time and circuitry to send all of the data over a bus. By mounting the processor  26  in the multi-sensor system  12 , data from both sensor  16  and sensor  18  can be processed locally requiring fewer reports to be sent over connection  28 .  
         [0025]    Referring to FIG. 3, the processor  26  in FIG. 2 receives radar reports from the first sensor  16  in block  34 . The processor  26  receives image reports from the second sensor  18  in block  36 . The processor  26  correlates the different reports in block  38 . Since the relative position of the two sensors  16  and  18  are the same and possibly coincident, the processor  26  does not have to perform as many calculations transforming sensor measurements into common body coordinates for the vehicle.  
         [0026]    The correlation may include first determining if the reports actually identify an object in block  40 . The processor  26  can verify or refine object detection information from one of the sensors with the message reports received from the other sensor. If both sensors do not verify detection of the same object within some degree of certainty, then the processor system  26  may discard the message reports or continue to analyze additional reports in block  40 .  
         [0027]    When an object is detected in block  40 , the processor  26  only has to send one report in block  42  representing the information obtained from both sensor  16  and sensor  18 . This reduces the total amount of data that has to be sent either to a central controller or another multi-sensor system in block  42 .  
         [0028]    [0028]FIG. 4 shows in further detail the different devices that may be integrated on the multi-sensor substrate  14 . Camera optics  50  and radar transmit/receive modules  52  are each connected to a Central Processing Unit (CPU)  54  and a digital signal processor  56 . A memory  58  is used to store sensor data, signal processing applications and other operating system functions. The CPU  54  is also used for conducting distributed sensor fusion as described in further detail below.  
       Distributed Sensor Fusion  
       [0029]    Referring to FIG. 5, different multi-sensor systems  12 A- 12 D are used for monitoring different zones around a vehicle  60 . For example, system  12 A monitors zone  1 , system  12 B monitors zone  2 , system  12 C monitors zone  3  and system  12 D monitors zone  4 . The CPU  54  and digital signal processor  56  (FIG. 4) in each multi-sensor system  12 A- 12 D in combination with the camera and radar sensors identify and track objects autonomously, without having to communicate with a central controller  68  in vehicle  60 .  
         [0030]    Whenever an object is detected, identified and tracked, a track file is created for that object in memory  58  (FIG. 4). If the object moves to another zone around the vehicle  60 , the multi-sensor system for the zone where the object was previously detected only has to send the track files to the other multi-sensor system associated with the overlapping region.  
         [0031]    For example, a bicycle  65  may be initially detected by multi-sensor system  12 A at location  64 A in zone  1 . The multi-sensor system  12 A creates a track file containing position, speed, acceleration, range, angle, heading, etc. for the bike  65 .  
         [0032]    As the vehicle  60  moves, or the bike  65  moves, or both, the bike  65  may move into a new position  64 B in an overlapping region  66  between zone  1  and zone  2 . The multi-sensor system  12 A upon detecting the bike  65  in the overlapping region  66  sends the latest track file for the bike  65  to multi-sensor system  12 B over bus  62 . This allows the multi-sensor system  12 B to start actively tracking bike  65  using the track information received from multi-sensor system  12 A.  
         [0033]    The multi-sensor system  12 A only has to send a few of the latest track files for the common area  66  over connection  64  to multi-sensor  12 B in order for system  12 B to maintain a track on bike  65 . The track files can be exchanged between any of the multi-sensor systems  12 A- 12 D. When there are two multi-sensor systems that have overlapping tracks for the same object, the track file with the greatest confidence of accuracy is used for vehicle warning, security, and control operations. There are known algorithms that calculate track files and calculate a degree of confidence in the track file calculations. Therefore, describing these algorithms will not be discussed in further detail.  
         [0034]    There may be vibrational effects on the different multi-sensor systems  12 A- 12 D. This however does not effect the track calculations generated by the individual multi-sensor systems  12 A- 12 D. The only compensation for any vibration may be when the track files are translated into body coordinates when a possible control decision is made by the central controller  68 .  
         [0035]    The connection  62  can a CAN bus, wireless 802.11 link or any other type of wired or wireless link. The system described above can use dedicated processor systems, micro controllers, programmable logic devices, or microprocessors that perform some or all of the operations. Some of the operations described above may be implemented in software and other operations may be implemented in hardware.  
         [0036]    For the sake of convenience, the operations are described as various interconnected functional blocks or distinct software modules. This is not necessary, however, and there may be cases where these functional blocks or modules are equivalently aggregated into a single logic device, program or operation with unclear boundaries. In any event, the functional blocks and software modules or features of the flexible interface can be implemented by themselves, or in combination with other operations in either hardware or software.  
         [0037]    Having described and illustrated the principles of the invention in a preferred embodiment thereof, it should be apparent that the invention may be modified in arrangement and detail without departing from such principles. I claim all modifications and variation coming within the spirit and scope of the following claims.