Systems and methods for mitigating light-emitting diode (LED) imaging artifacts in an imaging system of a vehicle

This disclosure is generally directed to mitigating light-emitting diode (LED) imaging artifacts in an imaging system of a vehicle. In an example embodiment, the imaging system includes a first camera that operates under control of a first pulse trigger sequence, and a second camera that operates under control of a second pulse trigger sequence. The second pulse trigger sequence has a temporal offset with respect to the first pulse trigger sequence. The first camera captures an image of a light source, such as a traffic light containing LEDs. This image may contain an LED imaging artifact indicating that the traffic light is off. The second camera also captures an image of the light source. The temporal offset of the second pulse trigger sequence may eliminate the LED imaging artifact in the second image. A controller may compare the two images and determine that the traffic light is actually on.

FIELD OF THE DISCLOSURE

This disclosure generally relates to operations associated with a vehicle and more particularly relates to mitigating LED imaging artifacts in images captured by an imaging system of a vehicle or other device, such as a smartphone or the like.

BACKGROUND

Light-emitting diodes (LEDs) offer several advantages in comparison to incandescent bulbs and have therefore been incorporated into a variety of lighting fixtures, such as streetlights, traffic lights, and billboards. The light intensity emitted by the light-emitting diodes in most of these fixtures is controllable by powering the light-emitting diodes with a voltage having a pulse waveform. The duty cycle and/or the pulse repetition frequency of the pulse waveform can be varied to in order to vary the amount of light emitted by each LED.

The pulse repetition frequency is typically selected such that human eyes do not notice a flickering of the LED when the LED toggles between on and off states. However, a digital imaging camera mounted on a vehicle may, in at least some cases, produce images having imaging artifacts that are characterized as LED flicker. In some cases, a vehicle controller of the vehicle may interpret the LED flicker of a red traffic light as the traffic light being off when the red light is on. The vehicle controller may also present an image of the flickering red traffic light on a display screen in the vehicle, which can be annoying to the driver.

Some conventional approaches to solving the problem of LED flicker in vehicles have focused on the use of hardware solutions, such as LOFIC (lateral overflow integration capacitor), image processing solutions, such as chopping (adding components to filter the image signal), and image capture strategies, such as split pixel imaging that captures an image of a scene simultaneously instead of sequentially.

DETAILED DESCRIPTION

Overview

In terms of a general overview, certain embodiments described in this disclosure are directed to systems and methods for mitigating light-emitting diode (LED) imaging artifacts in an imaging system of a vehicle. In an example embodiment, the imaging system includes a first camera that operates under control of a first pulse trigger sequence. The first pulse trigger sequence is a part of a first pulse trigger mode of operation. A second camera of the imaging system operates under control of a second pulse trigger sequence. The second pulse trigger sequence, which is a part of a second pulse trigger mode of operation, has a temporal offset with respect to the first pulse trigger sequence. The first camera captures an image of a light source. In an example scenario, the light source is a traffic light containing light-emitting diodes that are toggled on and off in rapid succession when the traffic light is in an on state. The image captured by the first camera may contain an LED imaging artifact erroneously indicating that the traffic light is off. The second camera also captures an image of the light source. The temporal offset of the second pulse trigger sequence may eliminate the LED imaging artifact in the image captured by the second camera. A controller may compare the two images and determine that the traffic light is actually on.

Illustrative Embodiments

The disclosure will be described more fully hereinafter with reference to the accompanying drawings, in which example embodiments of the disclosure are shown. This disclosure may, however, be embodied in many different forms and should not be construed as limited to the example embodiments set forth herein. It will be apparent to persons skilled in the relevant art that various changes in form and detail can be made to various embodiments without departing from the spirit and scope of the present disclosure. Thus, the breadth and scope of the present disclosure should not be limited by any of the above-described example embodiments but should be defined only in accordance with the following claims and their equivalents. The description below has been presented for the purposes of illustration and is not intended to be exhaustive or to be limited to the precise form disclosed. It should be understood that alternate implementations may be used in any combination desired to form additional hybrid implementations of the present disclosure. For example, any of the functionality described with respect to a particular device or component may be performed by another device or component. Furthermore, while specific device characteristics have been described, embodiments of the disclosure may relate to numerous other device characteristics. Further, although embodiments have been described in language specific to structural features and/or methodological acts, it is to be understood that the disclosure is not necessarily limited to the specific features or acts described. Rather, the specific features and acts are disclosed as illustrative forms of implementing the embodiments.

Certain words and phrases are used herein solely for convenience and such words and terms should be interpreted as referring to various objects and actions that are generally understood in various forms and equivalencies by persons of ordinary skill in the art. It should be understood that the examples provided below are directed at traffic lights for explaining principles in accordance with the disclosure. However, these principles can be equally applicable to objects other than traffic lights that incorporate LEDs and lighting elements (bulbs, for example) that are toggled on and off. A few examples of such objects may include household bulbs, streetlights, commercial lighting, traffic lights, and billboards. It should also be understood that the word “example” as used herein is intended to be non-exclusionary and non-limiting in nature.

FIG. 1shows an example vehicle115that includes an imaging system for capturing images of various types of light sources. Although illustrated as a sedan, the vehicle115may take the form of another passenger or commercial automobile, such as, for example, a truck, a high-performance vehicle, a crossover vehicle, a van, a rickshaw, a jeepney, a minivan, a taxi, a bus, etc., and may be configured and/or programmed to include various types of automotive drive systems. Example drive systems can include various types of internal combustion engines (ICEs) powertrains having a gasoline, diesel, or natural gas-powered combustion engine with conventional drive components, such as a transmission, a drive shaft, a differential, etc. In some instances, the imaging system may be independent of the vehicle. For example, imaging system may be part of a phone/camera of a mobile device that is brought into a vehicle. That is, the systems and methods described herein can apply to portable camera devices, such as mobile devices, smartphones, wearables, etc.

In some cases, the vehicle115may be an electric vehicle (EV) that includes a battery EV (BEV) drive system, a hybrid electric vehicle (HEV) having an independent onboard powerplant, or a plug-in hybrid electric vehicle (PHEV) that includes a HEV powertrain connectable to an external power source and/or includes a parallel or series hybrid powertrain having a combustion engine powerplant and one or more EV drive systems. HEVs may further include battery and/or supercapacitor banks for power storage, flywheel power storage systems, or other power generation and storage infrastructure.

In some other cases, the vehicle115may be a fuel cell vehicle (FCV) that converts liquid or solid fuel to usable power using a fuel cell, (e.g., a hydrogen fuel cell vehicle (HFCV) powertrain, etc.) and/or any combination of these drive systems and components.

It must be understood that irrespective of the type of engine, fuel, or energy source used, the vehicle115in accordance with disclosure can be any vehicle having two or more cameras arranged to perform the various functions described herein. Such vehicles can include any vehicle conforming to any of six levels of driving automation defined by the Society of Automotive Engineers (SAE). The six levels of driving automation range from Level 0 (fully manual) to Level 5 (fully autonomous). These levels have been adopted by the U.S. Department of Transportation. Level 0 (L0) vehicles are manually controlled vehicles having no driving related automation. Level 1 (L1) vehicles incorporate some features, such as cruise control, but a human driver retains control of most driving and maneuvering operations. Level 2 (L2) vehicles are partially automated with certain driving operations, such as steering, braking, and lane control being controlled by a vehicle computer. The driver retains some level of control of the vehicle and may override certain operations executed by the vehicle computer. Level 3 (L3) vehicles provide conditional driving automation but are smarter in terms of having an ability to sense a driving environment and certain driving situations. Level 4 (L4) vehicles can operate in a self-driving mode and include features where the vehicle computer takes control during certain types of equipment failures. The level of human intervention is very low. Level 5 (L5) vehicles are fully autonomous vehicles that do not involve human participation.

The vehicle115may include various components, such as, for example, a vehicle computer120, an imaging system controller125, a camera130, and a wireless communication system (not shown). The imaging system controller125and the camera130are two example components of an imaging system of the vehicle115that is used to capture images. The vehicle computer120may perform various functions, such as controlling engine operations (fuel injection, speed control, emissions control, braking, etc.), managing climate controls (air conditioning, heating etc.), activating airbags, and issuing warnings (check engine light, bulb failure, low tire pressure, vehicle in blind spot, etc.). In some cases, the vehicle computer120may include more than one computer.

In an example implementation in accordance with the disclosure, the imaging system controller125may be configured to support wireless communications with a server computer145and/or a cloud storage system140via a network135. The network135may be, and/or include, the Internet, a private network, public network or other configuration that operates using any one or more known communication protocols, such as, for example, transmission control protocol/Internet protocol (TCP/IP), Bluetooth®, BLE®, Wi-Fi based on the Institute of Electrical and Electronics Engineers (IEEE) standard 802.11, UWB, and cellular technologies, such as Time Division Multiple Access (TDMA), Code Division Multiple Access (CDMA), High-Speed Packet Access (HSPDA), Long-Term Evolution (LTE), Global System for Mobile Communications (GSM), and Fifth Generation (5G), to name a few examples.

The server computer145and/or cloud storage system140may provide to the imaging system controller125, various types of information that may be pertinent to mitigating LED imaging artifacts in accordance with the disclosure. Some examples of such information can include timing characteristics of light sources, such as traffic lights and road signs.

The vehicle computer120may be installed in an engine compartment of the vehicle115(or elsewhere in the vehicle115) and communicatively coupled to the imaging system controller125. In some implementations, the imaging system controller125may be a part of the vehicle computer120.

The camera130may be mounted upon any component of the vehicle115, such as, for example, the hood of the vehicle115, a dashboard of the vehicle115, or a roof of the vehicle115(as shown). The camera130may be a digital camera that captures digital images at an image capture rate that is controllable by the imaging system controller125. In this example embodiment, the camera130is generally oriented to capture images of objects around the vehicle115. Images and/or video generated by the camera130may be displayed on a display screen of an infotainment system provided in the vehicle115for viewing by an occupant of the vehicle115and/or provided to the imaging system controller125for generating images and/or information that may be upon the display screen of the infotainment system. When the vehicle115is an autonomous vehicle, the imaging system controller125may evaluate the images and/or video for purposes of automatically controlling various operations of the vehicle.

The images captured by the camera130typically include driving-related objects, such as, for example, traffic lights, road signs, pedestrians, obstacles, and crosswalks. LED imaging artifact mitigation procedures and systems in accordance with the disclosure particularly involve imaging of various light sources, such as, for example, a traffic light, a road sign, a pedestrian sign, and/or a light mounted on a second vehicle (a brake light, for example). The imaging system controller125may evaluate an image of a traffic light, such as, for example, a traffic light105to determine whether a red warning light106of the traffic light105is on. The red warning light106may include LEDs that are toggled on and off when the red warning light106is on at which time, the imaging system controller125may cooperate with the vehicle computer120to stop the vehicle115. On the other hand, when the red warning light106is off and a green light107of the traffic light105is on, the imaging system controller125may cooperate with the vehicle computer120to move the vehicle115forward.

The imaging system controller125may also evaluate an image of another traffic light110in order to determine if a green light is on in the traffic light110indicating that it is safe for the vehicle115to move forwards and make a right turn.

In another scenario, the imaging system controller125may evaluate an image of a road sign providing a driving warning, such as, for example, “Watch out for accident 3.1415926 miles ahead.” The words may be displayed by LEDs incorporated into the road sign, and the imaging system controller125may mitigate any LED artifacts if present in images of the road sign that are captured by the camera130.

In yet another scenario, the imaging system controller125may evaluate an image of a billboard incorporating LEDs, mitigate any LED artifacts if present in the image, and generate a customized message that is displayed on the display screen of the infotainment system in the vehicle115.

FIG. 2illustrates a scenario in which the imaging system provided in the vehicle115generates an LED imaging artifact. In this scenario, the red warning light106of the traffic light105is currently on (as indicated by an illumination status205shown inFIG. 2). The red warning light106contains a set of LEDs. The set of LEDs includes an example LED207. The LEDs in the red warning light106are activated upon application of power to the red warning light106. The applied power is illustrated in the form of a pulsed waveform210that causes each of the LEDs in the red warning light106to toggle between an on state and an off state. The toggling rate is typically indiscernible to a human driver because the human eye cannot perceive very rapid light changes in the LEDs and/or the human brain adapts to such light changes. However, a machine-vision apparatus, such as a camera used in a vehicle, may register the flickering and draw erroneous conclusions in some cases with respect to a traffic light.

The pulsed waveform210contains a series of pulses that go to a high level to cause the LEDs to turn on, and drop to a low level to turn off the LEDs. Various characteristics of the pulsed waveform210such as, for example, a duty cycle, phase, and/or a pulse repetition frequency (prf), can be tailored in order to control the light intensity produced by the LEDs. For example, increasing a pulse width of each of the pulses in the pulsed waveform210increases the average voltage level of the pulsed waveform210, thereby causing the LEDs to shine brighter, and vice-versa.

In the example scenario illustrated inFIG. 2, the LEDs of the red warning light106are turned on, for example, at a timing instant “t1” and a timing instant “t3” and are turned off at a timing instant “t2.” The pulses of the pulsed waveform210persist for a time period245corresponding to a period of time over which the red warning light106is on, and disappear when power is disconnected from the red warning light106, such as, for example, when power is applied to a green light107of the traffic light105instead.

The imaging system controller125may configure the camera130to capture images of the traffic light105and/or the traffic light110based on various camera settings. In an exemplary camera setting in accordance with the disclosure, the imaging system controller125configures the camera130to operate in an image capture mode wherein a pulse trigger sequence215is used to trigger a sequence of image capture cycles. Each image capture cycle may be initiated by a transition edge of a respective pulse in the pulse trigger sequence215. For example, an image capture cycle218is triggered by a rising edge of a pulse217of the pulse trigger sequence215. Other image capture cycles are sequentially triggered by the rising edges of other sequentially occurring pulses in the pulse trigger sequence215. Each image capture cycle includes an exposure period and a readout period. The exposure period may be varied by the imaging system controller125either automatically (depending on parameters such as, for example, camera hardware and/or ambient light conditions) or under control of a software program. In an example implementation, the imaging system controller125may execute a software program to tailor the exposure period of the image capture cycles in order to mitigate imaging artifacts in accordance with an embodiment of the disclosure.

Various characteristics of the pulse trigger sequence215such as, for example, the duty cycle and the pulse repetition frequency (prf), can be varied by the imaging system controller125in order to control the exposure periods. In the example scenario illustrated inFIG. 2, the camera shutter is open at each of the timing instants “t1,” “t2,” and “t3.” At these timing instants, light associated with an image to be captured enters the lens of the camera130and is incident upon an image sensor contained inside the camera130. The captured image in this example, includes the traffic light105. The status of the images captured by the camera130at the timing instants “t1,” “t2,” and “t3” are indicated by the imaging status220. At timing instant “t1” the pulsed waveform210is in a high state thus causing the LEDs of the red warning light106to be in an on state. The image capture sequence216at timing instant “t1” corresponds to an exposure period and the imaging sensor in the camera130captures an image225. Consequently, as shown by the imaging status220, the image225captured by the camera130at the timing instant “t1” includes the red warning light106in an illuminated condition (all LEDs including the LED207are in an on state).

Similarly, at timing instant “t3” the pulsed waveform210is in a high state thus causing the LEDs of the red warning light106to be in an on state. The image capture sequence216at timing instant “t3” corresponds to an exposure period and the imaging sensor in the camera130captures an image235. Consequently, as shown by the imaging status220, the image235captured by the camera130at the timing instant “t3” includes the red warning light106in an illuminated condition (all LEDs including the LED207are in an on state).

However, at timing instant “t2” the pulsed waveform210is in a low state whereby the LEDs of the red warning light106are off. The image capture sequence216at timing instant “t2” corresponds to an exposure period and the imaging sensor in the camera130captures an image230of the red warning light106in an off state. The imaging status220shows the image230captured by the camera130at the timing instant “t2” includes the red warning light106in a non-illuminated condition (all LEDs including the LED207are in an off state).

In reality, the red warning light106is still in an illuminated condition (all LEDs including the LED207are merely off momentarily as a result of the rapid toggling caused by the pulsed waveform210). The image230represents an LED imaging artifact because the image does not accurately reflect the status of the traffic light105as would be perceived by a human driver observing the traffic light105.

It is desirable in accordance with the disclosure for the imaging system controller125to evaluate the image230in comparison with other images, such as, for example, the image225and the image235. In one example embodiment, an illumination condition of the red warning light106is determined by comparing a first status of the red warning light106in the image230and a second status of the red warning light106in the image235. In another example embodiment, an illumination condition of the LED207may be determined by comparing a first status of the LED207in the image230and a second status of the LED207in the image235. In yet another example embodiment, an LED imaging artifact may be identified by a majority polling procedure applied to a preselected number of images, such as, for example, one image that is different in comparison to 10 other neighboring images over a time frame. The neighboring images may precede and/or succeed the image associated with the LED imaging artifact.

A time frame for evaluating the various images using any of the procedures indicated above can correspond, for example, to the time period245, or can correspond to any other time period during which an adequate number of images have been captured. The time frame can also be selected on the basis of various factors, such as, for example, one or more image capture settings on the camera130, a frame repetition rate of the pulse trigger sequence215, and/or an expected duration for the red warning light106to stay on. In some cases, if not known, the frame repetition rate of the pulsed waveform may be estimated by the imaging system controller125via communications with other devices that are connected to the electrical grid. Furthermore, in some cases, the camera130can be a rolling shutter camera and individual images captured by the rolling shutter camera may be evaluated in order to detect any banding features, if present, inside the image frame. Such banding features may suggest the occurrence of flicker at higher rates than the image frame row exposure duration and row to row temporal offset during readout.

At timing instant “t4” the pulsed waveform210remains at a steady low state because the red warning light106has been turned off and power has been applied to another light, such as the green light107. At this time, the camera130captures an image240that accurately reflects the red warning light106in an unlit condition.

In some cases, the camera may utilize a rolling shutter mode of operation wherein each row of the camera's exposure is read out sequentially, which leads to a temporal offset in each row as the exposure of the first row is slightly delayed compared to the next row and so on. It must be understood that the shape of the image capture sequence216shown inFIG. 2is merely for purposes of illustration. In some other cases, the exposure period may be illustrated by a different shape, such as, for example, a parallelogram. A vertical dimension of the parallelogram may be indicated as a temporal-row axis and a row position of a light source can indicate whether the light source is on or off. Furthermore, a light source may be either on, dimmer, or off, based on the duration of the exposure relative to the duration of light produced by the light source. In some implementations, the time period245can represent one frame in a repetitive frame format of operation. In some cases, no time gap may be present between frames and in some other cases an inter-frame time gap may be provided. The inter-frame time gap may be adjusted to affect a smaller readout period. The camera105may also be operated at a frame rate that is not constant and the exposure periods adjusted so as to produce a higher frame rate.

FIG. 3shows the vehicle115equipped with an imaging system that includes two cameras for capturing images of various types of light sources in accordance with an embodiment of the disclosure. In this embodiment, the vehicle115may include various components, such as, for example, the vehicle computer120, the imaging system controller125, the camera130, and the wireless communication system. The imaging system of the vehicle can include various components, such as, for example, the camera130and the imaging system controller125and further include a second camera305. In other embodiments, the imaging system can include more than two cameras. The field of view of each of multiple cameras, may be arranged to overlap so as to provide a desired cumulative field of view. Some or all of the cameras may also include objects such as wide-angle lenses and filters.

FIG. 4illustrates a scenario wherein the camera130and the camera305shown inFIG. 3are employed for identifying and mitigating LED imaging artifacts in accordance with the disclosure. More than two cameras can be employed in other applications in accordance with the disclosure. It must be understood that elements indicated inFIG. 4by the same reference numerals as used inFIG. 2are identical to each other. For example, the pulsed waveform210, the pulse trigger sequence215, and the image capture sequence216shown inFIG. 4are identical to the pulsed waveform210, the pulse trigger sequence215, and the image capture sequence216shown inFIG. 2.

The pulse trigger sequence215is used for triggering the image capture sequence216in the first camera130. The pulse trigger sequence405is used for triggering the image capture sequence406in the second camera305. The pulse trigger sequence405and/or the image capture sequence406may be identical to, similar to, or different than, the pulse trigger sequence215and/or the image capture sequence216, respectively. For example, a pulse width “e2” of each pulse in the pulse trigger sequence405can be identical to, similar to, or different than, a pulse width “e1” of each pulse in the pulse trigger sequence215.

However, a temporal offset is provided between the pulse trigger sequence215and the pulse trigger sequence405in accordance with the disclosure. The temporal offset is illustrated inFIG. 4by a temporal offset “d1” between a leading edge of the first pulse in the pulse trigger sequence215and a leading edge of the first pulse in the pulse trigger sequence405. The temporal offset may be provided in various ways. In one example implementation, the temporal offset can be provided by employing a different pulse repetition frequency (prf) and/or a duty cycle for the pulse trigger sequence405in comparison to the pulse repetition frequency (prf) and/or duty cycle employed for the pulse trigger sequence215. In another example implementation, the temporal offset can be provided by using a delay element (a D-flipflop, for example) to apply a signal delay upon the pulse trigger sequence405with respect to the pulse trigger sequence215. In yet another example implementation, the temporal offset “d1”

A time period of the temporal offset “d1” to be applied may be determined in various ways. In one example application, the temporal offset “d1” may be a fraction of the time period245. In another example application, the temporal offset “d1” may be calculated on the basis of a desired amount of delay in various aspects of a rolling shutter mode of operation such as, for example, row readout times, a number of rows between an image that is present in adjacent frames, and trigger delay between start of frame captures in multiple cameras.

In some cases, the imaging system controller125may determine the temporal offset “d1” by wirelessly fetching from the server computer145and/or the cloud storage system140, some timing parameters associated with the pulsed waveform210. For example, the imaging system controller125may obtain from the server computer145and/or the cloud storage system140, timing sequence information related to red, amber, and green lights of the traffic light105. Specifically, in this example we may determine the alternating current (AC) grid frequency and offset at that location to estimate the grid connected traffic light105timing sequence information. In another example, a vehicle manufacturer may utilize a PWM controller of a specific timing settings wherein the temporal offset may not be known a priori but the frequency may be estimated on the basis of vehicle model recognition from the image captured.

In an example application, the temporal offset “d1” is a fixed offset that is time-invariant. In another example application, the temporal offset “d1” may be varied periodically and/or intermittently over time. In some cases, a first temporal offset may be changed to a second temporal offset based on identifying a timing pattern of the LED207when the LED207toggles between an on state and an off state.

The status of the images captured by the first camera130at the timing instants “t1,” “t2,” and “t3” are described above and shown as the image225, image230, and image235. The status of the images captured by the second camera305at timing instants “t5,” “t6,” and “t7” are shown by an image415, an image420, and an image425.

At timing instant “t5” the pulsed waveform210is in a high state thus causing the LEDs of the red warning light106to be in an on state. The image capture sequence406at timing instant “t5” corresponds to an exposure period and the imaging sensor in the second camera305captures an image415. The image415captured by the camera305at the timing instant “t5” includes the red warning light106in an illuminated condition (all LEDs including the LED207are in an on state).

At timing instant “t6” the pulsed waveform210is in a high state thus causing the LEDs of the red warning light106to be in an on state. The image capture sequence406at timing instant “t6” corresponds to an exposure period and the imaging sensor in the camera305captures an image420. The image420captured by the camera305at the timing instant “t6” includes the red warning light106in an illuminated condition (all LEDs including the LED207are in an on state).

At timing instant “t7” the pulsed waveform210is in a high state thus causing the LEDs of the red warning light106to be in an on state. The image capture sequence406at timing instant “t7” corresponds to an exposure period and the imaging sensor in the camera305captures an image425. The image425captured by the camera305at the timing instant “t7” includes the red warning light106in an illuminated condition (all LEDs including the LED207are in an on state).

In this example, the temporal offset applied to the pulse trigger sequence405has resulted in the image420indicating that the red warning light106is in an illuminated condition even though the preceding image230captured by the first camera130indicates the red warning light106is in an unlit condition.

It is desirable in accordance with the disclosure for the imaging system controller125to evaluate the image230in comparison with other images, such as the image415and/or the image420. In one example embodiment, an illumination condition of the LED207is based on comparing a first status of the LED207in the image230and a second status of the LED207in the image235. In another example embodiment, an illumination condition of the red warning light106is based on comparing a first status of the red warning light106in the image230and a second status of the red warning light106in the image235. In yet another example embodiment, an LED imaging artifact may be identified by a majority polling procedure applied to a preselected number of images, such as, for example, one image that is different in comparison to 10 other neighboring images over a time frame. The neighboring images may precede and/or succeed the image associated with the LED imaging artifact.

A time frame for evaluating the various images using any of the procedures indicated above can correspond, for example, to the time period245, or can correspond to any other time period during which an adequate number of images have been captured. The time frame can also be selected on the basis of various factors, such as, for example, one or more image capture settings on the camera130and/or the camera305, a frame repetition rate of the pulse trigger sequence215, a frame repetition rate of the pulse trigger sequence405, and/or an expected duration for the red warning light106to stay on. In some cases, if not known, the characteristics of these types of signals may be determined based on information obtained from other devices that are connected to the electrical grid.

In some implementations, individual images may be evaluated to identify whether banding features are present. Such banding features may suggest the occurrence of flicker at higher rates than exposure affecting intensity in rolling shutter in each camera image at each time of imaging.

FIG. 5shows some example components that may be included in the vehicle115. The example components can include the camera130, the camera305, the vehicle computer120, an infotainment system510, a wireless communication system520, and the imaging system controller125. The various components can be communicatively coupled to each other via one or more buses, such as an example bus511. The bus511may be implemented using various wired and/or wireless technologies. For example, the bus511can be a vehicle bus that uses a controller area network (CAN) bus protocol, a Media Oriented Systems Transport (MOST) bus protocol, and/or a CAN flexible data (CAN-FD) bus protocol. Some or all portions of the bus511may also be implemented using wireless technologies, such as Bluetooth®, Ultra-Wideband, Wi-Fi, Zigbee®, or near-field-communications (NFC). For example, the bus511may include a Bluetooth® communication link that allows the imaging system controller125to wirelessly communicate with the camera130and/or the camera305.

In an example implementation, the infotainment system510includes a display515that may be configured to display various types of information provided by the imaging system controller125. In some implementations, the display515may include a graphical user interface (GUI) (or a human machine interface (HMI)) that may be used to accept input from an occupant of the vehicle115, and also to display items, such as messages, icons, and/or soft keys.

The wireless communication system520can include various wireless communication nodes. In one example implementation, some or all of the wireless communication nodes can include a Bluetooth® low energy module (BLEM) and/or a Bluetooth® low energy antenna module (BLEAM).

The imaging system controller125may include a processor530, a communication system535, and a memory540. The communication system535can include one or more wireless transceivers (BLEAMs, for example) that may be used for various purposes, such as, for example, to allow the imaging system controller125to transmit commands to the camera130and the camera305and to receive images captured by the camera130and the camera305.

The memory540, which is one example of a non-transitory computer-readable medium, may be used to store an operating system (OS)560, a database550, and various code modules, such as an imaging artifact mitigating module545. The code modules are provided in the form of computer-executable instructions that can be executed by the processor530for performing various operations in accordance with the disclosure.

The imaging artifact mitigating module545may be executed by the processor530for performing various operations in accordance with the disclosure, such as, for example, identifying imaging artifacts in images captured by the camera130and/or the camera305, and for determining an illumination condition of the traffic light105.

In an example implementation, the database550may be used to store timing information related to various light sources, such as the traffic light105, and information fetched from the server computer145and/or cloud storage system140.

FIG. 6shows a flowchart600of an example method in accordance with the disclosure for mitigating imaging artifacts in images captured by an imaging system of a vehicle, such as the vehicle115. The flowchart600illustrates a sequence of operations that can be implemented in hardware, software, or a combination thereof. In the context of software, the operations represent computer-executable instructions stored on one or more non-transitory computer-readable media, such as the memory540, that, when executed by one or more processors, such as the processor530, perform the recited operations. Generally, computer-executable instructions include routines, programs, objects, components, data structures, and the like that perform particular functions or implement particular abstract data types. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described operations may be carried out in a different order, omitted, combined in any order, and/or carried out in parallel. Some or all of the operations described in the flowchart600may be carried out by using the imaging system controller125and the communication system535. The description below may make reference to certain components and objects shown inFIGS. 1-4, but it should be understood that this is done for purposes of explaining certain aspects of the disclosure and that the description is equally applicable to many other embodiments.

At block605, an image scene may be identified. The identification may be made based on various factors, such as, for example, to assist the vehicle115move along a road. In this case, the imaging system controller125may communicate with the camera130and/or the camera305to capture images of various objects as the vehicle115moves along the road. For example, the camera130and/or the camera305may be activated to capture a set of images and/or a video clip of an image scene ahead of the vehicle115.

At block610, the imaging system controller125may begin evaluating the images to characterize various light sources that may be present in the images and/or artifacts indicating LED flicker during rolling shutter image capture or within a single image that indicates the presence of off states.

At block615, the evaluation performed by the imaging system controller125may involve characterizing various light sources as unvarying light sources (for example, a street lamp that stays on continuously) or light sources that turn on and off at a rapid rate (such as, for example, the traffic light105). Characterizing may further involve identifying light sources that are relevant to the vehicle115(such as the traffic light105) versus light sources that are deemed irrelevant (such as a light that illuminates a poster located beside the road).

At block620, evaluation performed by the imaging system controller125may involve identifying a possibility of having to deal with imaging artifacts that may be present in one or more light sources (such as the traffic light105, which contains LEDs that toggle on and off in the manner described above and may produce imaging artifacts).

At block625a determination may be made whether to employ two or more cameras for mitigating imaging artifacts. Mitigating operations may be terminated if deemed unnecessary, such as, for example, to set a longer exposure during an overcast day or to reduce an exposure during a very sunny day. In some cases, a longer exposure setting may lead to more cycles of LED on-off cycles being captured. If mitigating operations are deemed necessary, at block630, a determination may be made whether the light source (such as the traffic light105) is within a field-of-view of two or more cameras mounted on the vehicle115(such as the camera130and the camera305).

In some cases, further operations may be terminated if the light source is not within the field-of-view of the two or more cameras. In some other cases, where a light source is within field-of-view of a single camera but not within field-of-view of the two or more cameras, mitigating operations described above with respect to images captured by a single camera, such as the camera130, may be carried out. Such mitigation operations may include using a longer duration exposure to capture a potential light source at one or more on cycles while accepting undesirable saturation of the sensor. Another example may be to perform sub-window capture of the detected light source to increase the image capture rate to improve the probability of capturing the on cycle of any suspected light source. In other cases, the single camera may detect image frames with on light source states and image frames with off light source states that may require the use of filtering via hidden Markov model, use of sequential Monte Carlo methods, exact Kalman filter, particle filter, etc. These mitigation strategies may further inform sensor fusion strategies and planning logic. In some cases, this may result in feature deactivation, such as deactivation of traffic sign recognition visual alerts in a display.

If the light source is within the field-of-view of the two or more cameras, at block635, optimal settings for the two or more cameras (exposure times, temporal offset, etc. for example) can be determined. This operation may include feature-homography (via operations such as ransac, transform, correction for exposure/brightness global difference, and image subtract) followed by subtraction to identify anomalies. A confidence threshold may be used in some cases to evaluate the anomalies.

In some applications, a determination may be based on a relative, numerically-weighted comparison using characteristics that may include current exposure time, current weather patterns, or other atmospheric conditions (e.g., a sunny day vs. a cloudy day), and other factors. For example, during an overcast day, a nominal exposure time for a camera may be substantially longer in duration as compared with a bright sunny day. A relatively long duration image exposure may increase a probability associated with obtaining a correct light state for a light source in the captured images. By comparison, on a sunny day, the exposure time may be relatively short such that the image is not over-exposed and light-saturated.

At block640, the imaging system controller125may identify any adverse effects caused by the settings determined in block635. For example, the imaging system controller125may identify that a particular exposure setting may lead to excessive imaging artifacts being present in the images captured by the camera130and/or the camera305.

At block645, the imaging system controller125may determine a first pulsed trigger mode of operation for the first camera130. The first pulsed trigger mode of operation involves the application of the pulse trigger sequence215and the image capture sequence216to the camera130. The imaging system controller125may also determine a second pulsed trigger mode of operation for the second camera305. The second pulsed trigger mode of operation involves the application of the pulse trigger sequence405and the image capture sequence406to the camera305.

Various aspects related to the first pulsed trigger mode of operation and/or the second pulsed trigger mode of operation may involve identifying and accepting a trade-off between performance and ability of other perception algorithms to detect the true state of light sources. For example, changes to exposure settings may negatively impact individual image quality resulting in reduction in computer vision performance in object detection. In other cases, the cameras exposure may not be temporal aligned, thereby resulting in errors associated with relative object motion in the image frame, e.g. optical flow, creating additional errors in depth estimation. The negative effects of operating the camera105and/or the camera305under non-ideal exposure settings may be expressed in terms of some arbitrary cost scale, e.g.1to11, based on engineering knowledge for a range of ideal to non-ideal exposure settings that is predicted to obtain the most temporal coverage of light pulses. This process may further analyze the exposure settings associated with rolling shutter cameras, e.g. CMOS, wherein individual rows of the camera may be exposed at differing times relative to other rows. A compounding procedure may be used in some cases, wherein camera rows may cover differing field of views, e.g. forward-looking camera and surround view cameras, so row100of one camera and row500of another camera may provide images of the same light source.

At block650, the pulse trigger sequence215and the image capture sequence216to the camera130are applied to the camera130and/or the pulse trigger sequence405and the image capture sequence406are applied to the camera305. The camera130and/or the camera305are then operated for capturing images, such as the images shown inFIG. 2and/orFIG. 4.

At block655, the imaging system controller125may evaluate the captured images with respect to homography. Such evaluation may include evaluating an image where there is a temporal delay and/or more complex homography (unaligned cameras) in two images. A homography approach may be used to ensure matching individual pixels even under relative motion. The homography approach may also be applied over multiple images (captured by a single camera or by multiple cameras) to track a given light source location in pixel space over time to identify on/off states over time. The pixel space may further be brought into 3D space by techniques such as Simultaneous Localization and Mapping (SLAM), combined stereo, and temporal stereo structure from motion (SFM). The imaging system controller125may also evaluate a quality of various types of homographic matches.

At block660, the imaging system controller125may evaluate the captured images to identify LED imaging artifacts in pixel space and/or 3D space.

At block665, the imaging system controller125may evaluate the captured images to assess various illumination parameters, such as, for example, illumination parameters associated with the red warning light106. Assessing such parameters may be based on ensuring that information provided by the pertaining red warning light106(stop indication) is discernible.

At block670, the imaging system controller125may determine a confidence value of information derived from images captured of one or more light sources over various time frames, such as, for example, various image capture frames and/or the time period245(shown inFIG. 2andFIG. 4) that corresponds to a duration of the red warning light106being illuminated.

At block675, the imaging system controller125may compare the confidence level against a confidence threshold.

At step680, responsive to determining that the confidence threshold is met, the imaging system controller125may incorporate image-related information into one or more perception algorithms for purposes, such as overriding a present determination (for example, a most recent image frame evaluation result), as well as informing sensor fusion and planning logic in the imaging system controller125. For example, if the most recent set of image frames captured by the camera(s) indicate that a camera is off on most frames but the confidence level determination made in accordance with the disclosure (for example, <0.4) indicates otherwise (e.g. the light source on/off state is predicted as on and the pulse characteristics predicted indicate that the exposure time period and light source time periods do not overlap) then the imaging system controller125may inform the planning algorithm of a software program that the light is presently on (e.g. traffic light state is a red light). In such cases, the software program may predict the path and motion plan of other vehicles on the basis of a red light state and actuate a display in the vehicle115accordingly (for example, display a a traffic light icon indicating a red light on the display screen of an infotainment system), provide a braking warning, and/or actuate braking while in a autonomous/semi-autonomous operation. In another example, the confidence level may be high, (for example, >0.7) thereby indicating that the image frame state, e.g. all light signals are off, matches that of the true state, a result derived by using a perception algorithm may remain unaltered and thus inform other algorithms/logic, e.g. planning logic, resulting in a different set of vehicle actuation, e.g. deactivation of low speed stop and go semi-autonomous vehicle operations on the basis of a traffic condition that is out of the scope of the features safe operation conditions as defined by the manufacturer.

At step685, the imaging system controller125may incorporate image-related information into one or more planning prediction algorithms, e.g. prediction of other vehicle path and motion plans, and planning algorithms for purposes, such as motion, path, evaluating, or strategizing.

At step690, the imaging system controller125may cooperate with the vehicle computer120to execute one or more vehicle control actions. For example, the vehicle115may continue moving forward through the traffic light105or stop at the traffic light105based on the illumination status of the red warning light106and/or the green light107.

Implementations of the systems, apparatuses, devices, and methods disclosed herein may comprise or utilize one or more devices that include hardware, such as, for example, one or more processors and system memory, as discussed herein. An implementation of the devices, systems, and methods disclosed herein may communicate over a computer network. A “network” is defined as one or more data links that enable the transport of electronic data between computer systems and/or modules and/or other electronic devices. When information is transferred or provided over a network or another communications connection (either hardwired, wireless, or any combination of hardwired or wireless) to a computer, the computer properly views the connection as a transmission medium. Transmission media can include a network and/or data links, which can be used to carry desired program code means in the form of computer-executable instructions or data structures and which can be accessed by a general purpose or special purpose computer. Combinations of the above should also be included within the scope of non-transitory computer-readable media.

A memory device, such as the memory540, can include any one memory element or a combination of volatile memory elements (e.g., random access memory (RAM, such as DRAM, SRAM, SDRAM, etc.)) and non-volatile memory elements (e.g., ROM, hard drive, tape, CDROM, etc.). Moreover, the memory device may incorporate electronic, magnetic, optical, and/or other types of storage media. In the context of this document, a “non-transitory computer-readable medium” can be, for example but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device. More specific examples (a non-exhaustive list) of the computer-readable medium would include the following: a portable computer diskette (magnetic), a random-access memory (RAM) (electronic), a read-only memory (ROM) (electronic), an erasable programmable read-only memory (EPROM, EEPROM, or Flash memory) (electronic), and a portable compact disc read-only memory (CD ROM) (optical). Note that the computer-readable medium could even be paper or another suitable medium upon which the program is printed, since the program can be electronically captured, for instance, via optical scanning of the paper or other medium, then compiled, interpreted or otherwise processed in a suitable manner if necessary, and then stored in a computer memory.