Patent Description:
It has been observed that camera based vision systems used on automated vehicles often have difficulty detecting the state of a traffic-signal (e.g. red, yellow, green) when the traffic-signal is moving due to, for example, wind. <CIT> and <CIT> both disclose a traffic light recognition device and method.

In accordance with one embodiment, a traffic-light-detection system that visually determines a light-state of a traffic-light proximate to an automated vehicle is provided. Said system (<NUM>) comprises:.

Further features and advantages will appear more clearly on a reading of the following detailed description of the preferred embodiment, which is given by way of non-limiting example only and with reference to the accompanying drawings.

<FIG> illustrates a non-limiting example of a traffic-light-detection system <NUM>, hereafter referred to as the system <NUM>. In general, the system <NUM> visually determines a light-state <NUM> (e.g. red, green yellow) of a traffic-light <NUM> proximate to, e.g. in front of, within fifty meters (<NUM>) for example, an automated vehicle, e.g. a host-vehicle <NUM>. Advantageously, the system <NUM> is able to determine the light-state <NUM> when the traffic-light <NUM> is moving, e.g. swinging from a cable <NUM> (<FIG>) that spans an intersection <NUM>. When the traffic-light <NUM> is moving, the light emitted by the traffic-light <NUM> appears to vary from the perspective of the host-vehicle <NUM> as the direction of the bore-site of light emitted by the traffic-light <NUM> varies. The apparent variation in apparent-intensity and/or size of the illuminated light (e.g. red, green yellow) on the traffic-light <NUM> may make it difficult for prior examples of visual traffic light detection systems to determine the light-state <NUM> of the traffic-light.

As used herein, the term automated vehicle may apply to instances when the host-vehicle <NUM> is being operated in an automated-mode <NUM>, i.e. a fully autonomous mode, where a human-operator (not shown) of the host-vehicle <NUM> may do little more than designate a destination in order to operate the host-vehicle <NUM>. However, full automation is not a requirement. It is contemplated that the teachings presented herein are useful when the host-vehicle <NUM> is operated in a manual-mode <NUM> where the degree or level of automation may be little more than providing an audible or visual warning to the human-operator who is generally in control of the steering, accelerator, and brakes of the host-vehicle <NUM>. For example, the system <NUM> may merely warn the human-operator as needed to, for example, avoid 'running' a red-light, i.e. traveling through an intersection <NUM> (<FIG>) when the traffic-light <NUM> indicates that the host-vehicle <NUM> should stop.

In an example , the system <NUM> includes a camera <NUM> on a host-vehicle <NUM>. The camera <NUM> may be a video-camera or a camera capable of taking periodically timed images. Whatever type is used, the camera <NUM> needs to be capable to render a series-of-images <NUM> of the traffic-light <NUM>. Those in the art will recognize that there are a wide variety of commercially available cameras that are suitable for this application. If the camera <NUM> is used to determine a motion-pattern <NUM> of the traffic-light <NUM> if/when the traffic-light <NUM> is moving, it is preferable that the camera <NUM> has or is characterized by a minimum frame-rate, ten frames-per-second (<NUM> fps) for example. As used herein, the motion-pattern <NUM> may be a characterization of the type of motion exhibited by the traffic-light <NUM>. For example, the motion-pattern <NUM> may be characterized as swinging forward-and-backward relative to the host-vehicle <NUM>, side-to-side (i.e. sideways or left-and-right) relative to the host-vehicle <NUM>, swinging diagonally relative to the host-vehicle <NUM>, oscillatory-rotating about a vertical-axis of the traffic-light <NUM>, oscillating vertically along the vertical-axis (bouncing), or any combination thereof. The details of how the motion-pattern <NUM> is determined and how the series-of-images <NUM> are analyzed will be explained in more detail later.

The system <NUM> according to the invention further includes a radar <NUM> on the host-vehicle <NUM>. A variety of radar-devices are commercially available for automotive applications that would be suitable to emit a radar-signal toward the traffic-light <NUM> and detect instances of radar-returns <NUM> reflected by, and/or returning from, the traffic-light <NUM>. If the radar <NUM> is used to determine a motion-pattern <NUM> of the traffic-light <NUM> when the traffic-light <NUM> is moving, it is preferable that the radar <NUM> has or is characterized by a minimum frame-rate, ten frames-per-second (<NUM> fps) for example.

The system <NUM> includes a controller <NUM> in communication with the camera <NUM>, and the radar <NUM> if the radar <NUM> is provided or included in the configuration of the system <NUM>. The controller <NUM> may include a processor (not specifically shown) such as a microprocessor or other control circuitry such as analog and/or digital control circuitry including an application specific integrated circuit (ASIC) for processing data as should be evident to those in the art. The controller <NUM> may include memory (not specifically shown), including non-volatile memory, such as electrically erasable programmable read-only memory (EEPROM) for storing one or more routines, thresholds, and captured data. The one or more routines may be executed by the processor to perform steps for determining the motion-pattern <NUM> and the light-state <NUM> based on signals received by the controller <NUM> from the camera <NUM> and optionally the radar <NUM> as described herein.

If, in one example not according to the invention, the system <NUM> does not include the radar <NUM>, then the controller <NUM> may be configured to determine a motion-pattern <NUM> of the traffic-light <NUM> based on the series-of-images <NUM>, and then select a preferred-image <NUM> from the series-of-images <NUM>. If the traffic-light is not moving, then likely any or all of the images in the series-of-images <NUM> could be analyzed to determine the light-state <NUM>. Indeed, it is recognized that basing the determination of the light-state <NUM> on multiple images increases the confidence-level of the determination of the light-state <NUM>. However, when the traffic-light <NUM> is moving, the position of the illuminated light (red, green, yellow) in each image of the series-of-images <NUM> likely changes which can make it difficult to have a high confidence-level.

The preferred-image <NUM> from the series-of-images <NUM> corresponds to one or more of the images that shows a light-source <NUM> (<FIG>) of the traffic-light <NUM> as being most directed at the camera <NUM> when/while the motion-pattern <NUM> indicates that the traffic-light <NUM> is moving. That is, the preferred-image <NUM> is selected from the series-of-images <NUM> as being the one or more images that is/are most likely to indicate the light-state <NUM> with high-confidence. It is recognized that if the motion-pattern <NUM> is relatively periodic, i.e. relatively predictable, then there may be multiple instances of the preferred-image <NUM> present in the series-of-images <NUM> that are temporally spaced apart in accordance with the periodicity of motion or oscillation of the traffic-light <NUM>.

<FIG> illustrates a non-limiting example of an intersection <NUM> equipped with a traffic-light <NUM> that is suspended above the intersection <NUM> by a cable <NUM> that is attached to poles at each end of the cable <NUM>. In this non-limiting example the traffic-light <NUM> is moving or swinging with a motion-pattern <NUM> that can be characterized as swinging diagonally relative to a view-perspective of the host-vehicle <NUM>.

<FIG> are non-limiting examples of images from the series-of-images <NUM> that may have been taken by the camera <NUM> from the perspective illustrated in <FIG> while the traffic-light was moving in accordance with the motion-pattern <NUM> illustrated in <FIG>. <FIG> may be characterized as an instance of the preferred-image <NUM> because the light-source <NUM>, the green light in this example, is well-directed toward the camera <NUM>, i.e. is pointed almost directly at the camera <NUM>. By contrast, <FIG> may be characterized as an instance of a discarded-image <NUM> from the series-of-images <NUM> because the light-source is not well-directed toward the camera <NUM>, i.e. is pointed away from the camera <NUM>. If the motion-pattern <NUM> is relatively periodic, then there may be multiple instances of the preferred-image <NUM> shown in <FIG> present in the series-of-images <NUM> that are temporally spaced apart and repeat or reoccur on a periodic basis. It follows that the multiple instances of the preferred-image <NUM> shown in <FIG> may be grouped together for image-processing to determine the light-state <NUM> based on multiple instances of the preferred-image <NUM> and thereby determine the light-state <NUM> with high-confidence.

As previously suggested, the system <NUM> according to the invention includes the radar <NUM>, so it follows that the controller <NUM> is in communication with the camera <NUM> and the radar <NUM>. In this radar-included embodiment of the system <NUM>, the controller <NUM> is configured to determine the motion-pattern <NUM> of the traffic-light <NUM> based on the radar-returns <NUM>, and then select the preferred-image <NUM> from the series-of-images <NUM> based on the motion-pattern <NUM> indicated by the radar <NUM> rather than what might be indicated by the camera <NUM>. Alternatively, it is contemplated that information from both the camera <NUM> and the radar <NUM> may be combined to determine which images of the series-of-images <NUM> is the preferred-image <NUM> or are the preferred-images. The radar-returns <NUM> may be analyzed using a variety of techniques known to those in the radar arts to determine the motion-pattern <NUM> of the traffic-light <NUM>.

Based on the radar-returns <NUM>, it is determined when a mid-point of a periodic motion occurs, and then designate the image from the series-of-images <NUM> that temporally coincides with the mid-point as the preferred-image <NUM>. In view of the motion-pattern <NUM> suggested in <FIG>, the image that corresponds to the mid-point is likely that shown in <FIG>, which corresponds to the position of the traffic-light shown in <FIG>. That is, it is believed that the mid-point is where the light-source <NUM> of the traffic-light <NUM> characterized as being most directed at the camera <NUM>. In contrast, an end-point of the motion-pattern <NUM>, i.e. a point of maximum deflection away from the mid-point, may correspond to <FIG>. It is also contemplated that the series-of-images <NUM> would likely include one or more images that is/are the opposite of <FIG>. That is, if <FIG> is characterized as the traffic-light <NUM> swinging toward and leftward relative to the camera <NUM>, then there is expected to be an opposite image in the series-of-images that shows the traffic-light <NUM> as swinging away-from and rightward relative to the camera <NUM>.

Several examples not according to the invention of how images of the traffic-light <NUM> can be subjected to image-processing to help determine the motion-pattern <NUM> and the light-state <NUM> will now be discussed. One option is to determine an apparent-size <NUM> of the light-source <NUM> in the series-of-images by, for example, counting the number of camera-pixels that detect the light-source <NUM>. It is recognized that counting the number of camera-pixels that detect the light-source <NUM> in each image of the series-of-images <NUM> will include tracking the position of where the light-source <NUM> appears in each of the images. Several algorithms for tracking a moving object in a series-of-images are known and could be used here. The preferred-image <NUM> is then selected based on which of the images shows or renders the apparent-size <NUM> characterized as largest in the series-of-images <NUM>. That is, the preferred-image <NUM> is the image that has the greatest number of camera-pixels that detect the light-source <NUM>. In this example it may not be necessary to determine the particular type of motion, e.g. forward-and-backward, side-to-side, diagonal, oscillatory-rotating, bouncing. The effect of variation of the apparent-size <NUM> due to swinging if the traffic-light <NUM> is evident by examining <FIG>.

Another option not according to the invention is for the controller <NUM> to be configured to determine an apparent-intensity <NUM> of the light-source <NUM> in the series-of-images <NUM>. When the traffic-light <NUM> is characterized as being most directed at the camera <NUM>, it is expected that the apparent-intensity <NUM> of the light-source will be the greatest. The preferred-image is then selected for having the apparent-intensity <NUM> characterized as greatest in images of the series-of-images <NUM>. Here again, it may not be necessary to determine the particular type of motion, e.g. forward-and-backward, side-to-side, diagonal, oscillatory-rotating, bouncing.

Another option not according to the invention is for the controller <NUM> to be configured to define a bounding-box <NUM> about each image of the traffic-light <NUM> in the series-of-images <NUM>, and determine if the traffic-light <NUM> is moving and/or the motion-pattern <NUM> based on changes in a box-size of the bounding-box <NUM> over the series-of-images. As used herein, the box-size may consist of, or include, a box-height, a box-width, a box-area, or any combination thereof. For example, if the motion is forward-and-backward relative to the camera <NUM>, the box-height would vary, but the box-width may be substantially unchanged. If the motion-pattern <NUM> is determine to be forward-and-backward, the selection of the preferred-image <NUM> may be relatively critical to making a reliable determination of the light-state <NUM> because of the substantial variation in the apparent-intensity <NUM>. In contrast, side-to-side motion may cause greater variation in box-width and box-area when compared to variation in box-height. However, since the apparent-intensity <NUM> may not change substantially, the selection which image or images are the preferred-image <NUM> may not be critical to making a reliable determination of the light-state <NUM>.

It is contemplated that the controller <NUM> may be advantageously further configured to determine when the motion-pattern <NUM> is such that an other-light <NUM> (<FIG>) may be periodically revealed to the camera <NUM>. As shown in <FIG>, the swinging of the traffic-light <NUM> causes the red-light that is not directed to the host-vehicle <NUM> to be revealed to the camera <NUM>. Alternatively, the other-light <NUM> could be a street-light (not shown) or advertisement-light (not shown) that is hidden behind the traffic-signal in <FIG>. Knowledge that the motion-pattern <NUM> is such that the other-light <NUM> may be periodically revealed to the camera <NUM> may help to prevent confusion about the light-state <NUM>. For example, if the system <NUM> detects both the light-source <NUM> as a relatively constant intensity green-light and the other-light <NUM> as a flashing red-light, the vehicle-operation block <NUM> (<FIG>) may erroneously elect to apply the brakes rather than continue through the intersection <NUM>.

It is also contemplated that the controller <NUM> may be advantageously further configured to determine a motion-period <NUM> of the motion-pattern <NUM>, where the motion-period <NUM> may correspond to the period of oscillation of the swinging-motion of the traffic-light <NUM>. Given the motion-period <NUM>, the controller <NUM> can predict which of future-images <NUM> rendered by the camera <NUM> will be selected as the preferred-image <NUM>. That is, rather than wait until a large number of images in the series-of-images <NUM> have been rendered and analyzed to select the preferred-image <NUM>, the motion-period <NUM> can be used to predict when the next instance of an image likely to be an instance of the preferred-image <NUM> will occur. This will allow the system <NUM> to more quickly and confidently determine that the light-state <NUM> has recently changed.

The system <NUM> may also include, or have access to via wireless communications, a digital-map <NUM> that indicates traffic-light-positions of various instances of the traffic-light <NUM> that the host-vehicle <NUM> may encounter.

Accordingly, a traffic-light-detection system (the system <NUM>), a controller <NUM> for the system <NUM>, and a method of operating the system <NUM> is provided. The system <NUM> provides for more reliable and quicker determination of the light-state <NUM> of a traffic-light <NUM> when the traffic-light <NUM> is swinging or otherwise moving relative to a host-vehicle <NUM> equipped with the system <NUM>.

Claim 1:
A traffic-light-detection system (<NUM>) that uses radar and imagery to determine a light state (<NUM>) of a traffic light (<NUM>) proximate to a host vehicle (<NUM>), said system (<NUM>) comprising:
a camera (<NUM>) on the host vehicle (<NUM>), said camera (<NUM>) configured to render a series of images (<NUM>) of a traffic light (<NUM>) proximate to the host vehicle (<NUM>);
a radar (<NUM>) on the host vehicle (<NUM>), said radar (<NUM>) configured to detect radar returns (<NUM>) reflected by the traffic light (<NUM>); and
a controller (<NUM>) in communication with the camera (<NUM>) and the radar (<NUM>), and configured to:
determine a motion pattern (<NUM>) of the traffic light (<NUM>) based on the radar returns (<NUM>) that indicates that the traffic light (<NUM>) is moving with a periodic motion;
select, based on the motion pattern (<NUM>) and from the series of images (<NUM>), a preferred image (<NUM>) that shows a light source (<NUM>) of the traffic light (<NUM>) when the traffic light (<NUM>) is at or near a mid-point of the periodic motion and the light source (<NUM>) is most directed at the camera (<NUM>); and
determine the light state (<NUM>) of the traffic light (<NUM>) based on the preferred image (<NUM>).