Systems and methods for haziness detection

A method performed by an electronic device is described. The method includes determining a haziness confidence level based on multiple modalities. The method also includes determining whether to perform an action based on the haziness confidence level. The method may include performing the action, including performing haziness reduction based on the haziness confidence level.

TECHNICAL FIELD

The present disclosure relates generally to electronic devices. More specifically, the present disclosure relates to systems and methods for haziness detection.

BACKGROUND

In the last several decades, the use of electronic devices has become more common. In particular, advances in electronic technology have reduced the cost of increasingly complex and useful electronic devices. Cost reduction and consumer demand have proliferated the use of electronic devices such that they are practically ubiquitous in modern society. As the use of electronic devices has expanded, so has the demand for new and improved features of electronic devices. More specifically, electronic devices that perform new functions and/or that perform functions faster, more efficiently or with higher quality are often sought after.

In some cases, captured images may suffer from degraded quality. For example, environmental scene conditions including atmospheric effects such as haze and glare can reduce image quality. As can be observed from this discussion, systems and methods that improve image usage may be beneficial.

SUMMARY

A method performed by an electronic device is described. The method includes determining a haziness confidence level based on multiple modalities. The method also includes determining whether to perform an action based on the haziness confidence level. The modalities may provide image modality information and location modality information, direction modality information, time of day modality information, local weather report modality information, remote device images modality information, an indoor/outdoor indicator, user override indicator, and/or temperature modality information.

The method may include performing the action. Performing the action may include performing haziness reduction based on the haziness confidence level. Performing haziness reduction may include adjusting an auto white balance, an auto focus, an auto exposure, a color, a sharpness, and/or a local tone mapping curve. Performing the action may include performing at least one non-image processing action.

Determining the haziness confidence level may be based on a two-stage classification. The two-stage classification may include performing a first feature extraction based on a camera input signal to produce a first set of one or more extracted features, performing a first classification based on the first set of extracted features and one or more of the modalities, and determining that the first classification indicates that haziness is detected. The two-stage classification may also include performing a second feature extraction based on the camera input signal and one or more of the modalities to produce a second set of one or more extracted features, and performing a second classification based on the second set of extracted features and one or more of the modalities.

The first set of extracted features may include a dark channel feature, a gradient of intensity feature, and/or a blurriness/sharpness feature. The second set of extracted features may include a dark channel feature, a magnitude of gradient feature, a phase of gradient feature, and/or a gradient spatial information feature.

The haziness confidence level may be obtained by a detector. The detector may obtain the haziness confidence level based on one or more of the modalities.

The haziness confidence level may be obtained by multiple detectors. Each detector may obtain a modality haziness confidence level based on one or more of the modalities. The modality haziness confidence levels may be combined to form the haziness confidence level.

An electronic device is also described. The electronic device includes a processor configured to determine a haziness confidence level based on multiple modalities and to determine whether to perform an action based on the haziness confidence level.

An apparatus is also described. The apparatus includes means for determining a haziness confidence level based on multiple modalities. The apparatus also includes means for determining whether to perform an action based on the haziness confidence level.

A computer-program product is also described. The computer-program product includes a non-transitory computer-readable medium with instructions. The instructions include code for causing an electronic device to determine a haziness confidence level based on multiple modalities. The instructions also include code for causing the electronic device to determine whether to perform an action based on the haziness confidence level.

DETAILED DESCRIPTION

Some configurations of the systems and methods disclosed herein may relate to haziness detection and haziness reduction using information from multiple modalities. As used herein, the terms “haze” and derivatives thereof (e.g., “hazy,” “haziness,” etc.) may refer to environmental scene conditions including, for example, the appearance of airborne particles such as haze, fog, steam, water vapor, air pollution (e.g., smog), rain, snow, dust, and/or smoke in a captured image. Automatic haziness detection and/or haziness reduction (e.g., de-hazing) can be very complicated, especially when the haziness detection and/or haziness reduction is based only on the camera optical input. Some reasons for this difficulty may include a large variation in lighting conditions, the color of the haziness, the location of the hazy areas and the degree of the haziness. Furthermore, there may be a large variation in scene composition (such as different subjects being within the scene). Reliable and fast haziness detection and/or haziness reduction (on electronic devices with cameras, for example) can greatly restore the visibility of the objects within a scene and improve the perception of camera users. Moreover, haziness detection and/or haziness reduction may improve computer vision processing, such as object detection, object tracking, object recognition, etc. For example, haziness detection and/or haziness reduction may improve performance for advanced driver assistance systems (ADAS).

Information from multiple modalities may be utilized to improve haziness detection. A modality may be a means for obtaining information (e.g., sensing) regarding surroundings of an electronic device or of a scene. In some configurations, multiple modalities may be readily available in electronic devices (e.g., cameras, video camcorders, digital cameras, cellular phones, smart phones, computers, televisions, automobiles, personal cameras, action cameras, surveillance cameras, mounted cameras, connected cameras, robots, drones, smart applications, healthcare equipment, set-top boxes, etc.). One or more modalities (in addition to the camera input, for example) may provide information that may be utilized to improve the accuracy of haziness detection and/or haziness reduction. Examples of such information include global positioning system (GPS) location (e.g., city versus countryside, mountain area, costal area, etc.), orientation from a compass sensor, time of day, local weather report (e.g., forecast), remote device images (e.g., satellite images, connected camera images, personal/action camera images, etc.), ambient light, temperature(s) (e.g., at one location and/or a temperature difference between two locations or environments), barometric pressure, humidity, time of year (e.g., calendar day), etc. Another example of such information may be pictures taken at the same location posted in an online service (e.g., map service, social network, etc.), which may be used as a reference to detect haziness and/or to reduce the haziness.

Various configurations are now described with reference to the Figures, where like reference numbers may indicate functionally similar elements. The systems and methods as generally described and illustrated in the Figures herein could be arranged and designed in a wide variety of different configurations. Thus, the following more detailed description of several configurations, as represented in the Figures, is not intended to limit scope, as claimed, but is merely representative of the systems and methods.

FIG. 1is a block diagram illustrating one example of an electronic device102in which systems and methods for haziness detection based on multiple modalities may be implemented. Examples of the electronic device102include smartphones, cellular phones, computers (e.g., desktop computers, laptop computers, servers, etc.), tablet devices, media players, televisions, gaming consoles, set-top boxes, personal digital assistants (PDAs), robots, aircraft, unmanned aerial vehicles (UAVs), automobiles, etc. The electronic device102may include one or more components or elements. One or more of the components or elements may be implemented in hardware (e.g., circuitry) or a combination of hardware and software (e.g., a processor with instructions). In some configurations, the electronic device102may include a processor104, a memory106, a display108, an image sensor110, an optical system112, an image signal processor132, a communication interface114, a clock116and/or one or more sensors118. The processor104may be coupled to (e.g., in electronic communication with) the memory106, display108, image sensor110, optical system112, image signal processor132, communication interface114, clock116and/or one or more sensors118. It should be noted that although communication between the image sensor110and/or optical system112may pass via the image signal processor132, the processor104may additionally or alternatively communicate directly with the image sensor110and/or optical system112in some configurations.

The communication interface114may enable the electronic device102to communicate with one or more other electronic devices. For example, the communication interface114may provide an interface for wired and/or wireless communications. In some configurations, the communication interface114may be coupled to one or more antennas120for transmitting and/or receiving radio frequency (RF) signals. Additionally or alternatively, the communication interface114may enable one or more kinds of wireline (e.g., Universal Serial Bus (USB), Ethernet, etc.) communication.

The electronic device102may obtain one or more images (e.g., digital images, image frames, video, etc.). For example, the electronic device102may include the image sensor110and the optical system112(e.g., lenses) that focuses images of objects that are located within the field of view of the optical system112onto the image sensor110. In some configurations, the image sensor110may capture the one or more images. The optical system112may be coupled to and/or controlled by the image signal processor132and/or the processor104. Additionally or alternatively, the electronic device102may request and/or receive the one or more images from another device (e.g., an external image sensor coupled to the electronic device102, a network server, traffic camera, drop camera, automobile camera, web camera, etc.). In some configurations, the electronic device102may request and/or receive the one or more images via the communication interface114. For example, the electronic device102may or may not include a camera (e.g., an image sensor110and/or optical system112) and may receive images from a remote device.

In some configurations, the electronic device102may include an image data buffer (not shown). The image data buffer may buffer (e.g., store) image data from the image sensor110. The buffered image data may be provided to the image signal processor132and/or to the processor104.

In some configurations, the electronic device102may include an image signal processor (ISP)132. The image signal processor132may receive image data from the image sensor110(e.g., raw sensor data and/or pre-processed sensor data). The image signal processor132may perform one or more operations on the image data. For example, the image signal processor132may perform decompanding, local tone mapping (LTM), filtering, scaling and/or cropping, etc. The image signal processor132may provide the resulting image data to the processor104. For example, the image data may be image modality information.

In some configurations, the electronic device102may include a camera software application and a display108. When the camera application is running, images of objects that are located within the field of view of the optical system112may be recorded by the image sensor110. The images that are being recorded by the image sensor110may be presented on the display108. In some configurations, these images may be displayed in rapid succession at a relatively high frame rate so that, at any given moment in time, the objects that are located within the field of view of the optical system112are presented on the display108. The one or more images obtained by the electronic device102may be one or more video frames and/or one or more still images. The terms video frame and digital image may be used interchangeably herein.

The processor104may include and/or implement an automatic scene detector122, a haziness detector124and/or a haziness reducer126. The automatic scene detector122may control one or more aspects of image capture and/or processing. For example, the automatic scene detector122may control auto white balance, auto focus and/or auto exposure. For instance, the automatic scene detector122may control optical system112(e.g., lens) focus, image sensor110gain, image sensor110exposure time, etc. In some configurations, the automatic scene detector122may control auto white balance, auto focus and/or auto exposure for a remote electronic device (e.g., a remote camera, network device, etc.) by sending information to the remote device via the communication interface.

The haziness detector124may perform haziness detection based on multiple modalities130to determine a haziness confidence level. Each of the modalities130may be a means for obtaining information (e.g., sensing) regarding the surroundings of the electronic device102. For example, the image sensor110and optical system112(which may be referred to as a camera) may be one example of a modality130for capturing image information regarding the surroundings of the electronic device102.

The communication interface114may be a modality130for requesting and/or receiving information regarding the surroundings of the electronic device102(and/or of a remote electronic device). For example, the communication interface114may request and/or receive information regarding the location (e.g., latitude and longitude information, other location information, etc.) of an electronic device, the current time of day, a local weather report and/or forecast, remote device images (e.g., satellite images, connected camera images, personal/action camera images, etc.), current temperature, air pollution index, etc. In some configurations, multiple communication interfaces114may be implemented and/or utilized. For example, one communication interface114may be a global positioning system (GPS) receiver, another communication interface114may be a cellular (e.g., 3G, Long Term Evolution (LTE), CDMA, etc.) communication interface114, and yet another communication interface114may be a wireless local area network (WLAN) interface (e.g., 802.11 interface).

The clock116may be a modality130for telling time of day. For example, the clock116may be utilized in addition to or alternatively from the communication interface114for determining a time of day.

The one or more sensors118may be modalities130for obtaining one or more types of information. Examples of the sensor(s)118include temperature sensors, barometric sensors, humidity sensors, accelerometers (which may be used in inertial navigation, for example), ambient light sensors, direction sensors (e.g., compass sensors), etc.

The one or more modalities130may provide information to the haziness detector124. Examples of modality information include image information (e.g., one or more images, frames, etc.), remote device image information, location information, direction information, time of day, weather report and/or forecast information, satellite information (e.g., satellite images, Doppler radar, temperature, etc.), light information (which may be utilized to determine whether the electronic device102in indoors or outdoors, for example), temperature, air pollution index, etc. In some configurations, the electronic device102may additionally or alternatively request and/or receive one or more kinds of modality information from a remote electronic device. For example, a remote electronic device may include one or more of the aforementioned modalities (e.g., sensors) and may provide corresponding modality information to the electronic device102via the communication interface114.

The haziness detector124may determine a haziness confidence level based on multiple modalities130. For example, the haziness detector124may perform haziness detection based on the multiple modalities to determine a haziness confidence level. A haziness confidence level may be a measure of confidence that the captured image includes haziness and/or that the electronic device102surroundings are hazy. For example, the haziness detector124may extract one or more features from an image. The one or more features from the image and one or more kinds of (additional) modality information may be utilized to determine the haziness confidence level. In some configurations, the haziness detector124may employ a two-stage classification to determine the haziness confidence level. An example of two-stage classification is given in connection withFIG. 4.

In some configurations, the haziness detector124may employ a support vector machine (SVM) and/or a neural network classifier to determine the haziness confidence level. For example, the support vector machine classifier may be a pre-trained classifier that separates vectors (e.g., the extracted feature(s) and one or more types of additional modality information) that indicate haziness from vectors that do not indicate haziness. The classifier may be referred to as a decision boundary. For instance, the classifier may be a hyperplane that divides vectors indicating haziness from vectors that do not indicate haziness. The haziness confidence level may be reflected by the distance between the vector and the hyperplane. For instance, if a vector is on the side of the classifier that indicates haziness and is near the classifier, the haziness detector124may indicate that haziness is detected with a low haziness confidence level. In another instance, if a vector is on the side of the classifier that indicates haziness and is far from the classifier, the haziness detector124may indicate that haziness is detected with a high haziness confidence level.

In some configurations, the classifier (e.g., decision boundary) may be pre-trained. For example, the classifier (e.g., SVM boundary) may be trained using pre-labeled hazy scene and non-hazy scene images (e.g., through a gradient descent method). For instance, a training data set may be utilized to train an SVM classifier and/or a neural network classifier. In some configurations, each sample of the training data set may include a data point from each modality (e.g., a 2D image, GPS, time of day, day of the year, weather report/forecast for that location at the time the image is taken, air pollution index for that day, etc., and a binary annotation to indicate whether the image is hazy or not). The training data set may be utilized to generate the support vector in SVM and/or weights/bias for each layer in the neural network through back-propagation.

In some configurations, determining the haziness confidence level may be accomplished as follows. In an SVM approach, the haziness confidence level (e.g., hazy score) may be computed as the inner product of the input data (e.g., the image or extracted features and/or one or more kinds of modality information) and the supporting vectors. For example, the haziness detector124may compute an inner product of the supporting vectors and a vector based on the input data (e.g., (2D) image, GPS, time of day and/or day of the year, etc.). In a neural network approach, the haziness detector124may put the input data through multiple layers with combinations of weights and biases. The output from the final layer may be the haziness confidence level (e.g., hazy score).

The processor104may determine whether to perform an action based on the haziness confidence level. For example, the processor104may determine to perform an action in a case that the haziness confidence level satisfies one or more criteria (e.g., in a case that the haziness confidence level is greater than or equal to a particular level (e.g., a haziness threshold level)).

Examples of actions based on the haziness confidence level may include image processing (e.g., haziness reduction) and non-haziness reduction actions. In some configurations, the electronic device102(e.g., processor104) may only perform image processing based on the haziness confidence level. In other configurations, the electronic device102may only perform one or more non-image processing actions (e.g., non-haziness reduction action(s)). In yet other configurations, the electronic device102may perform one or more image processing actions in combination with one or more non-image processing actions.

For example, the electronic device102(e.g., processor104) process the image based on the haziness confidence level. In some configurations, processing the image may include performing haziness reduction. For example, the processor104may include and/or implement a haziness reducer126. The haziness reducer126may not perform haziness reduction if the haziness confidence level is low (e.g., below a haziness threshold). If the haziness confidence level is high enough (e.g., meets or exceeds the haziness threshold), the processor104may perform haziness reduction. In some configurations, the degree of haziness reduction performed may be determined based on the haziness confidence level. For example, greater haziness reduction may be performed for a greater haziness confidence level. Reducing haziness may include adjusting local tone mapping (LTM), auto white balance, auto focus, auto exposure, color correction, and/or sharpness.

It should be noted that haziness reduction may be performed on the current image (e.g., frame) and/or on a subsequent image (e.g., frame). For example, an electronic device102may perform haziness detection on a frame. Haziness reduction may be performed on that frame by adjusting, for example, color, exposure, and/or sharpness. Additionally or alternatively, haziness reduction may be performed by adjusting auto white balance, auto focus, and/or auto exposure (in addition to or alternatively from adjusting color, exposure and/or sharpness) for a subsequent frame.

Examples of non-image processing actions may include one or more of activating fog lights, activating a notification light, activating a notification tone, activating a heating and/or cooling system (e.g., heating, ventilating and air conditioning (HVAC) system) for haziness (e.g., fogginess, condensation, etc.) on a windshield, and displaying one or more objects (e.g., lane marker(s), vehicle marker(s), road edge marker(s), barrier marker(s), etc., on a heads-up display, windshield projector or other display, for example), etc. One or more non-image processing actions may be performed by an ADAS and/or in conjunction with an ADAS in some configurations. Another example of non-haziness reduction actions may include sending haziness information (e.g., haziness confidence level) to a remote device. For instance, the electronic device102may be a server that receives image data (and optionally additional modality information) from a remote device. The electronic device102may determine haziness information and send the haziness information to the remote device. In some configurations, the remote device may optionally perform haziness reduction based on the haziness information.

The memory106may store instructions and/or data. The processor104may access (e.g., read from and/or write to) the memory106. Examples of instructions and/or data that may be stored by the memory106may include image data, modality information, haziness confidence level(s), automatic scene detector instructions, haziness detector instructions, haziness reducer instructions, etc.

In some configurations, the electronic device102may present a user interface128on the display108. For example, the user interface128may enable a user to interact with the electronic device102. In some configurations, the user interface128may enable a user to indicate whether haziness is present (in the environment and/or in a captured image, for example).

In some configurations, the display108may be a touchscreen that receives input from physical touch (by a finger, stylus or other tool, for example). For instance, the touchscreen may be an input interface that receives touch input indicating whether haziness is present or not. Additionally or alternatively, the electronic device102may include or be coupled to another input interface. For example, the electronic device102may include a camera facing a user and may detect user gestures (e.g., hand gestures, arm gestures, eye tracking, eyelid blink, etc.) for indicating whether haziness is present. In another example, the electronic device102may be coupled to a mouse and may detect a mouse click indicating whether haziness is present. Thus, whether haziness is present may be indicated in any suitable way (e.g., a touch input, a mouse click, a recognized gesture, etc.). Accordingly, the modalities130may include means for receiving user input in some configurations. For example, the sensor(s)118may include a touchscreen display108(and/or the user interface128) for receiving user input indicating haziness (or no haziness). Additionally or alternatively, the modalities130may include an input interface (e.g., input port for an input device such as a mouse, a touch pad, a camera, a microphone, etc.) for receiving user input indicating haziness (or no haziness).

It should be noted that no user input may be necessary in some configurations. For example, the electronic device102may automatically detect haziness and/or reduce haziness in the one or more images.

FIG. 2is a block diagram illustrating an example of one configuration of an electronic device202in which systems and methods for haziness detection and/or haziness reduction may be implemented. The electronic device202ofFIG. 2may be one example of the electronic device102ofFIG. 1. The electronic device202may include a camera lens246that obtains images (e.g., photographs and/or video). Additionally or alternatively, the electronic device202may obtain images from a remote device.

The electronic device202may include one or more haziness detectors224. Each haziness detector224may determine a haziness confidence level of a scene viewed by the camera lens246based on one or more modalities. Each of the modalities may produce modality information248. Examples of modality information248include the location250aof the electronic device202(and/or the location of the scene being photographed, for example), the direction250bof the electronic device202(e.g., which direction the electronic device202is pointing), the time of day250c, a local weather report250d(e.g., forecast), one or more remote device images250e, an indoor/outdoor indicator250f, a user override indicator250g, and a temperature250h. It should be noted that image modality information, which may be obtained from an image sensor and/or optical system (e.g., camera lens246) may also be included in the modality information248.

Each of the modalities may be readily available to the electronic device202(e.g., as mobile phones). For example, the location250amay be obtained using GPS functionality on the electronic device202to determine whether the electronic device202is located in a city, a countryside, a mountain area or a coastal area, etc. The location250amay be utilized to determine an altitude in some configurations. The direction250bmay be based on information from a compass (e.g., orientation) sensor to determine the direction of the camera (e.g., towards the ocean or away from the ocean). Other modality information248may also be readily obtained by the electronic device202. For example, pictures taken at the same location (e.g., location250a) that are posted on a map web service and/or a social network may be used as references to detect haziness (e.g., haze, fog, smoke, etc.) and to perform haziness reduction (e.g., locations with previous instances of haze are more likely to currently have haze). Each of the modalities may be more or less useful (e.g., more or less indicative of haziness), depending on the specific location (e.g., location250a) of the electronic device202.

The time of day250cmay be obtained from a clock on the electronic device202and/or by requesting the time from a remote electronic device (e.g., a global navigation satellite system (GNSS), a location server, a base station, access point, a web server, etc.). The time of day250cmay be utilized in combination with one or more of the other modality information248. For example, the time of day250cmay be indicative of haziness in combination with the location250aand the direction250bof the electronic device202. For example, an image captured by the electronic device202may be likely to contain haze when the electronic device202is pointed at a coastal scene (e.g., near a pier or beach) in the morning hours during the winter in a coastal area.

The local weather report250dmay be obtained via a communication interface. For example, the electronic device202may request a local weather report250dfrom a remote electronic device (e.g., web server). The local weather report250dmay indicate local weather conditions (e.g., fog, smog, haze, wind, rain, snow, cloudiness, visibility, clarity, etc.), temperature, humidity, barometric pressure, etc., that may indicate a likelihood of haziness in the area. The local weather report250dmay be utilized in combination with one or of the other modality information248.

The remote device images250emay be obtained via a communication interface. The remote device images250emay include image information (e.g., one or more images) received from one or more remote devices. For example, the electronic device202may obtain the remote device image(s)250efrom one or more network devices (e.g., web servers), one or more satellites, one or more vehicles (e.g., one or more interior and/or exterior vehicle camera(s)), infrastructure (e.g., traffic camera(s), security camera(s), etc.), one or more connected cameras (e.g., drop cams), one or more personal/action cameras, etc. Some remote devices may not have image signal processing capabilities sufficient for haziness detection and/or haziness reduction. Additionally or alternatively, the electronic device202may not include an integrated camera in some configurations and may operate to detect haziness and/or reduce haziness in an image captured by a remote device (instead of in an image captured from an integrated camera, for example).

In one example, the electronic device202may request satellite images from a remote electronic device (e.g., web server). For instance, the satellite images may be provided by a weather service, mapping service, or some other web service. In other configurations, the electronic device202may receive the remote device images250edirectly from a satellite. The remote device images250emay indicate local weather conditions (e.g., fog, smog, haze, cloudiness, precipitation, etc.). Additionally or alternatively, the remote device images250emay provide a color of the surroundings when haze is not present. Accordingly, the remote device images250emay indicate a likelihood of haziness in the area. For instance, cloud cover over an area may indicate an increased likelihood of haziness.

In some configurations, the remote device images250emay be utilized in combination with one or more of the other modality information248. For example, satellite images may be utilized in combination with an image captured from the camera lens246. The color of the satellite image(s) may be compared with the color from the captured image. If the captured image has a dull or muted color in comparison with the satellite images, this may indicate an increase likelihood of haziness.

The indoor/outdoor indicator250fmay be obtained via an ambient light sensor. For example, the electronic device202may check the brightness of the ambient light provided by an ambient light sensor. Brightness over a brightness threshold may indicate that the electronic device202is outdoors. Additionally or alternatively, the indoor/outdoor indicator250fmay be derived from the location250a. For example, the location information250amay indicate whether the electronic device202is likely in a structure or outdoors. Accordingly, the indoor/outdoor indicator250fmay indicate a likelihood of haziness in the area (as haziness may be more likely to occur outdoors, for example). The indoor/outdoor indicator250fmay be utilized in combination with one or of the other modality information248. For example, the indoor/outdoor indicator250fmay be utilized in combination with the location250a, the direction250band the local weather report250d. For instance, if the indoor/outdoor indicator250findicates that the electronic device202is indoors, but the location250aindicates that the electronic device202is near the edge of the building, the direction250bindicates that the electronic device202camera is pointing outwards from the building and the local weather report250dindicates fog, this may be a scenario in which a user is taking a picture out of a window with an increased likelihood of haziness. In other scenarios, however, if the indoor/outdoor indicator250findicates that the electronic device202is indoors, there may be a decreased likelihood of haziness.

The user override indicator250gmay be obtained via a user interface. For example, the electronic device202may detect a touch or click event on a user interface object (e.g., button, text, etc.) that indicates haziness or no haziness. For example, the electronic device202may receive a user input that indicates haziness or no haziness. The user override indicator250gmay indicate that there is haziness or not in an image. The user override indicator250gmay be utilized in combination with one or more of the other modality information248. For example, even if the electronic device202determines (from multiple modalities) that no haziness is detected (and/or a low haziness confidence level), but the user override indicator250gindicates that there is haziness, the haziness detector224may be overridden to detect haziness. In some configurations, user override indicators250gmay be utilized to update the classifier training.

The temperature250hmay be obtained via a temperature sensor (e.g., heat sensor) and/or via a communication interface. For example, the electronic device202may obtain a temperature reading from a temperature sensor included in the electronic device202and/or may request and/or receive a temperature reading from a remote electronic device (e.g., web server). For instance, the temperature250hmay be part of a local weather report250din some configurations. The temperature250hmay indicate the temperature of the electronic device202and/or an outdoor temperature. Accordingly, the temperature250hmay indicate a likelihood of haziness in the area (as haziness may tend to occur in certain temperature ranges, for example). In some configurations, a temperature differential may be utilized. For example, haziness may be more likely to occur when a temperature of the electronic device202is at a particular differential from the outdoor temperature. For instance, if the electronic device202is in a car with a temperature at a certain differential from the outdoor temperature, haziness (e.g., condensation, frost, etc.) may be likely to occur on the windshield. The temperature250hmay be utilized in combination with one or more of the other modality information248. For example, the temperature250hmay be utilized in combination with the location250a, where certain locations (e.g., near water) may exhibit increased likelihood for haziness at certain temperatures.

Certain environmental scene conditions (e.g., weather, lighting, etc.) may cause features in the image to appear less clear. For example, a vehicle may be traveling through fog or through rain. In addition, a window (e.g., a windshield) through which the image sensor captures the image may be foggy due to a difference between a temperature inside of the vehicle and a temperature outside of the vehicle, or may be otherwise unclear due to condensate or frost. When features in an image are difficult to discern (e.g., the image is “hazy”), the image may indicate an environmental scene condition (e.g., rain, fog, a foggy windshield, smog, haze etc.). In configurations where the systems and methods are used in conjunction with a vehicle assistance application (e.g., advanced driver assistance systems (ADAS) and/or an augmented display), a hazy image may be an indicator that an operator of the vehicle may not be able to see features (e.g., traffic signs, traffic lanes, pedestrians, other vehicles, trees, etc.) of the scene outside of the vehicle. The haziness detector(s)224may detect hazy images from the image sensor.

In some implementations, the haziness detector(s)224may determine whether a window (e.g., a windshield) that the image sensor faces is foggy or whether the camera lens246of the image sensor is wet or foggy. To illustrate, the image sensor may be located within a cabin, such as a passenger cabin of the vehicle, and may be positioned to capture a scene through a window of the vehicle. The haziness detector(s)224may receive the sensor data that is associated with and/or corresponds to multiple temperatures250h, such as a first temperature inside the vehicle and a second temperature outside of the vehicle. The haziness detector(s)224may be configured to determine a temperature difference between the first temperature and the second temperature. When the temperature difference indicates that a difference between the first temperature (inside the vehicle) exceeds the temperature (outside the vehicle), the haziness detector(s)224may determine that the image captured by the image sensor is hazy in part because the window that the image sensor faces is foggy. In response to a determination that the window is foggy, the haziness detector(s)224may perform image processing (e.g., haziness reduction) based on the haziness confidence level. In some configurations, the electronic device202initiates an action, such as activating a heating, ventilation, or air conditioning (HVAC) system and/or a windshield wiper, as illustrative, non-limiting examples.

In another example, the haziness detector(s)224may determine that haziness is in a scene outside of a vehicle (not condensation on a window or foggy windshield, for instance). In this case, the haziness detection may initiate one or more actions. These actions may be different from one or more actions initiated for a foggy windshield. For haziness due to the scene outside of a vehicle, for example, the action(s) may include dehazing in a heads up display and/or ADAS functionality (e.g., breaking, slowing, lane marking, audible alerts, etc.).

It should be noted that although some modality information248is illustrated inFIG. 2, fewer, more, and/or different kinds of modality information248may be utilized. For example, barometric pressure, day of the year, humidity and/or other kinds of modality information248may be utilized. For instance, barometric pressure may be obtained from a barometric pressure sensor on the electronic device202and/or from the communication interface (e.g., from a remote web server, as part of a local weather report, etc.). The day of the year may be obtained from a calendar application on the electronic device202and/or from the communication interface (e.g., from a remote web server.). The humidity may be obtained from a humidity sensor on the electronic device202and/or from the communication interface (e.g., from a remote web server, as part of a local weather report, etc.). Another example of modality information248may be weather almanac information, which may be obtained from the communication interface (e.g., from a remote web server.). Another example of modality information248may be air pollution information (e.g., an air pollution index), which may be obtained from the communication interface (e.g., from a remote web server.). One or more of these modality information248may have a bearing on the likelihood of haziness occurring. Accordingly, the one or more haziness detectors224may utilize multiple modalities to determine a haziness confidence level.

In some configurations, the electronic device202may perform haziness reduction by a haziness reducer226based on multiple modalities. The haziness reducer226may provide for adjustments to color240, exposure242, and/or sharpness244, thereby removing or reducing the effect of haze on a particular scene and/or photograph. It should be noted that the haziness reducer226, the automatic scene detector222or both may adjust contrast in order to perform haziness reduction. The results of the haziness detector224can also be used to adjust the settings within an automatic scene detector222(e.g., 3A module). For example, the automatic scene detector222may include settings for and/or control auto white balance234, auto focus236and auto exposure238. It should be noted that the electronic device202may additionally or alternatively perform haziness reduction by controlling (e.g., adjusting) auto white balance234, auto focus236, and/or auto exposure238.

FIG. 3is a block diagram illustrating another example of a configuration of an electronic device302in which systems and methods for haziness detection and/or haziness reduction may be implemented. The electronic device302may include a camera lens346. The camera lens346may be one example of the camera lens246described in connection withFIG. 2. The camera lens346may focus light on the image sensor310. The image sensor310may be one example of the image sensor110described in connection withFIG. 1. The image sensor310may provide a raw camera signal352to an image signal processor332. The image signal processor332may be one example of the image signal processor132described in connection withFIG. 1. The image signal processor332may be a specialized digital signal processor (DSP) that is used for image processing by the electronic device302. The image signal processor332may convert the raw camera signal352into a camera signal354. In some configurations, the image signal processor332or a separate resizer may resize an image (e.g., resize a red-green-blue (RGB) plane). For example, the image signal processor332or a separate resizer may resize an image or frame to Quarter Video Graphics Array (QVGA). Resizing may provide comparable accuracy and less computation. Resizing may be performed before feature extraction (e.g., feature extraction by the haziness detector324). The camera signal354may be provided to a haziness detector324. The haziness detector324may be one example of the haziness detector124described in connection withFIG. 1.

In some configurations, an electronic device (e.g., electronic device102,202,302) may include an image signal processor (e.g., image signal processor132,332), but may not include a camera (e.g., may not include a camera lens246,346, image sensor110and/or optical system112). In these configurations, the electronic device (e.g., electronic device102,202,302) may receive image data (e.g., a camera signal) from a remote device (e.g., from a separate camera). For example, an electronic device (e.g., electronic device102,202,302) may receive image data from a remote electronic device (e.g., a network server, a separate camera coupled to the electronic device, etc.). Accordingly, haziness detection may be performed on the electronic device. Haziness reduction may be performed on the electronic device and/or in coordination with a remote electronic device. For example, the electronic device302may send one or more adjustments to a remote image sensor and/or camera lens (e.g., adjustments to lens focus, image sensor gain, image sensor exposure time, etc.).

The haziness detector324may determine whether haziness is detected within the scene viewed by the camera lens346. More specifically, the haziness detector324may perform haziness detection based on multiple modalities to determine a haziness confidence level356. This may be accomplished as described in connection with one or more ofFIGS. 1 and 2. More examples of haziness detection are given in connection with one or more ofFIGS. 4-9. For instance, the haziness detector324may employ two-stage classification to reduce computational complexity in some configurations. In some examples, the classifier(s) may employ linear support vector machine (SVM), where it may not be necessary to save all of the support vectors in the model. The haziness confidence level356may reflect the relative confidence that the haziness detector324has detected haziness within the scene viewed by the camera lens346. For example, a low haziness confidence level356may indicate that any haziness in the scene is minimal and/or unlikely. In cases with a low haziness confidence level356, haziness reduction processing may be reduced and/or not performed. A high haziness confidence level356may indicate that the scene includes significant haziness and/or that haziness is likely. In a case with a high haziness confidence level356, increased haziness reduction may be employed to remove and/or reduce haziness from the scene.

The haziness confidence level356may be provided to a haziness reducer326. The haziness reducer326may reduce the amount of haziness in a scene viewed by the camera lens346. For example, the haziness reducer326may adjust color, exposure, and/or sharpness settings to reduce and/or remove the perceived haziness. A more specific example of haziness reduction is given in connection with one or more ofFIGS. 9-10. The haziness confidence level356may also be provided to an automatic scene detector322(e.g., 3A module). The automatic scene detector322(e.g., 3A module) may adjust the auto white balance, the auto focus and/or the auto exposure based on the haziness confidence level356to reduce/remove haziness from the scene viewed by the camera lens346.

FIG. 4is a block diagram of an example of one configuration of a haziness detector424. The haziness detector424ofFIG. 4may be one configuration of one or more of the haziness detectors124,224,324described in connection with one or more ofFIGS. 1-3. The haziness detector424may receive a camera signal454. The camera signal454may be a signal received via a camera lens, image sensor and/or image signal processor. Alternatively, the camera signal454may be received from a remote device (e.g., a remote camera). In some configurations, the camera signal454may be processed using an image signal processor prior to providing the camera signal454to the haziness detector424.

In some configurations, the haziness detector424may be based on a two-stage classification. It should be noted that fewer or more stages may be used for classification (e.g., a multi-stage classification system). A first stage of the classification system may include a first feature extractor458(image based, for example) and a first classifier462. A second stage of the classification system may include a second feature extractor460(image and modality based, for example) and a second classifier464. In some configurations, one or more of the classifiers462,464may be based on support vector machines (SVMs).

The first feature extractor458may perform feature extraction. In some configurations, the first feature extractor458may use less computation power than the second feature extractor460. The first feature extractor458may receive the camera signal454. The first feature extractor458may then extract one or more first extracted features466from the camera signal454. Examples of features that may be extracted include color features (such as a dark channel feature470) and spatial features (such as the gradient of intensity472, a blurriness/sharpness indicator474, and/or a magnitude/phase of the gradient). The first feature extractor458may provide the one or more first extracted features466to the first classifier462. The first classifier462may also receive one or more kinds of modality information448. As discussed above, the modality information448may be additional information related to haziness that is/are available to the electronic device (via the one or more sensors, via the Internet, etc.).

Based on the extracted features and the modality information448, the first classifier462may output a haze/no haze decision468to the second feature extractor460. If the first classifier462outputs a no haze decision468, then no additional computing may be performed regarding haziness, as haziness is not detected. However, if the first classifier462outputs a haze decision468, then at least some haziness has been detected and the second feature extractor460may extract one or more second extracted features469.

In some configurations, the second feature extractor460may employ more computationally intense feature extraction to obtain more specific extracted features from the camera signal454. For example, the second feature extractor460may obtain one or more second extracted features469from the camera signal454that are based on both the image and one or more additional kinds of modality information448. Examples of features extracted by the second feature extractor460include a dark channel feature476, the magnitude of the gradient478, the phase of the gradient480, and/or gradient spatial information482. Thus, the second feature extractor460may provide more advanced extracted features than the first feature extractor458in some configurations. This approach may allow saving computational resources in cases where no haze is detected with less demanding extraction.

The second feature extractor460may provide one or more of the second extracted features469to the second classifier464. Based on the one or more extracted features and one or more kinds of modality information448, the second classifier464may output a haziness confidence level456. The haziness confidence level456may reflect the confidence that haziness is included within the camera signal454.

In some configurations, an SVM classifier in each stage may be trained on the extracted features from a training data set to create a decision boundary in the high dimensional feature space to differentiate hazy scenes from non-hazy scenes. In detection, the SVM may be applied to the same set of features. The SVM may evaluate whether the input image falls to one side of the decision boundary or the other. A suitable operating point of each SVM classifier may be selected to achieve improved (e.g., optimal) precision/efficiency tradeoff. For example, a linear-kernel SVM may be used for both the first stage and the second stage (in a two-stage configuration). A binary decision may be made at the first stage based on one operating point. The second stage may output a continuous haziness confidence level456.

FIG. 5is a flow diagram illustrating one configuration of a method500for haziness detection based on multiple modalities. The method500may be performed by an electronic device (e.g., one or more of electronic devices102,202,302). The electronic device may be configured with a haziness detector (e.g., haziness detector(s)124,224,324,424).

The electronic device may optionally obtain502a camera input signal (e.g., one or more images). For example, the camera input signal may be obtained via a lens, an image sensor, and/or an image signal processor as described above in connection with one or more ofFIGS. 1-4. Alternatively, the camera input signal may be obtained from a remote device (e.g., a remote camera coupled to the electronic device, a network server in communication with the electronic device, etc.). This may be accomplished as described in connection with one or more ofFIGS. 1-4.

The electronic device may determine504a haziness confidence level based on multiple modalities. For example, the electronic device may perform haziness detection based on multiple modalities to determine a haziness confidence level. This may be accomplished as described above in connection with one or more ofFIGS. 1-4. For example, the electronic device may extract one or more features from one or more images (e.g., from the camera input signal) and classify the image based on the feature(s) and one or more (additional) kinds of modality information. In some configurations, haziness detection may be performed based on a two-stage classification.

The electronic device may determine506whether to perform an action based on the haziness confidence level. This may be accomplished as described in connection with one or more ofFIGS. 1-3. For example, the electronic device may determine506to perform an action in a case that the haziness confidence level satisfies one or more criteria (e.g., in a case that the haziness confidence level is greater than or equal to a haziness threshold level). Examples of actions based on the haziness confidence level may include image processing action(s) (e.g., haziness reduction) and non-image processing action(s). For example, the electronic device may perform haziness reduction based on the haziness confidence level as described in connection with one or more ofFIGS. 1-3. Additionally or alternatively, the electronic device may perform non-image processing action(s) based on the haziness confidence level as described in connection withFIG. 1.

FIG. 6is a flow diagram illustrating a more specific configuration of a method600for haziness detection based on multiple modalities. The method600may be performed by an electronic device (e.g., one or more of electronic devices102,202,302). The electronic device may be configured with a haziness detector (e.g., haziness detector(s)124,224,324,424).

The electronic device may obtain602a frame (e.g., image) of a camera input signal. This may be accomplished as described above in connection with one or more ofFIGS. 1-4. For example, the electronic device may obtain602one or more images via a lens, an image sensor, and/or an image signal processor.

The electronic device may perform604a first (e.g., basic) feature extraction on the frame. This may be accomplished as described above in connection withFIG. 4. For example, the electronic device may determine a dark channel feature of the frame, a gradient of intensity of the frame, and/or a blurriness/sharpness indicator of the frame.

The electronic device may perform606a first classification based on the first extracted features and one or more modalities. This may be accomplished as described above in connection withFIG. 4. In some configurations, the first classification may use a support vector machine (SVM) to determine whether haziness is detected in the frame or not.

The electronic device may determine608whether the first classification indicates that haziness is detected in the frame. This may be accomplished as described above in connection withFIG. 4. For example, if a vector (based on the first extracted features and/or one or more kinds of modality information) is located on one side of the decision boundary, the first classification may indicate that haze is detected. If the vector falls on the other side of the decision boundary, however, haziness may not be detected. If haziness is not detected, then operation may end610.

If haziness is detected in the frame, the electronic device may perform612a second (e.g., more intensive) feature extraction on the frame. This may be accomplished as described above in connection withFIG. 4. In some configurations, the second feature extraction may include more computationally intense feature extractions, such as gradient spatial information, gradient phase information, gradient magnitude information, and/or additional dark channel features. Unlike the first feature extraction in some configurations, for example, the second feature extraction may be based on one or more modalities. For example, the second feature extraction may be based on the location of the electronic device, the direction the electronic device is facing, the time of day, etc.

The electronic device may perform614a second classification based on the second extracted features and one or more modalities to obtain a haziness confidence level. This may be accomplished as described above in connection withFIG. 4. In some configurations, the second classification may use a support vector machine (SVM) to determine a haziness confidence level that haziness is detected in the frame. The second classification may be based on the second extracted features and one or more modalities.

The electronic device may optionally perform616haziness reduction based on the haziness confidence level. This may be accomplished as described above in connection with one or more ofFIGS. 1-3. For example, the electronic device may adjust the auto white balance, the auto focus, the auto exposure, the color, the sharpness, and/or the local contrast to reduce/remove the visible haziness. Thus, the haziness reduction may include signal processing and adjusting one or more physical settings of the camera.

FIG. 7is a block diagram illustrating another example of a configuration of a haziness detector724. The haziness detector724may receive a camera input750aand one or more additional types of modality information: location750b, direction750c, time of day750d, local weather report750e(e.g., forecast), remote device images750f, an indoor/outdoor indicator750g, a user override indicator750h, a temperature750i, an air pollution index750j, and/or a day of the year750k. Based on the camera input750aand the one or more additional types of modality information750b-k, the haziness detector724may determine a haziness confidence level756. For example, the haziness detector724may utilize the camera input750aand one or more additional types of modality information750b-kto directly determine a single haziness confidence level.

FIG. 8is a block diagram illustrating an example of a configuration of multiple haziness detectors824a-hoperating in parallel. The haziness detectors824a-hmay be an example of one or more of the haziness detectors124,224,324,424described in connection with one or more ofFIGS. 1-4. Each haziness detector824may receive the camera input850iand one or more types of modality information850a-850h. For example, Haziness Detector A824amay receive the location modality information850a. The Haziness Detector A824amay output a location modality haziness confidence level884a(e.g., a haziness confidence level based on the camera input and the location modality information).

As another example, Haziness Detector B824bmay receive the direction modality information850b. Haziness Detector B824bmay output a direction modality haziness confidence level884b.

Haziness Detector C824cmay receive the time of day modality information850cand may output a time of day modality haziness confidence level884c. Haziness Detector D824dmay receive the local weather report (e.g., forecast) modality information850dand may output the local weather forecast modality haziness confidence level884d.

Haziness Detector E824emay receive the remote device images modality information850eand may output a remote device images modality haziness confidence level884e. Haziness Detector F824fmay receive an indoor/outdoor indicator modality information850fand may output an indoor/outdoor modality haziness confidence level884f.

Haziness Detector G824amay receive user override indicator modality information850gand may output a user override modality haziness confidence level884g. Haziness Detector H824hmay receive temperature modality information850hand may output a temperature modality haziness confidence level884h.

Each of the obtained modality haziness confidence levels884a-hmay then be combined to obtain an overall haziness confidence level856. For example, each of the modality haziness confidence levels884a-hmay be provided to a confidence level combiner886. The confidence level combiner886may be implemented in one or more of the electronic devices102,202,302(e.g., in a processor104), for example. In some configurations, the confidence level combiner886may combine the modality haziness confidence levels884a-hby averaging the modality haziness confidence levels884a-h. In some configurations, the confidence level combiner886may determine a weighted average of the modality haziness confidence levels884a-hto produce the haziness confidence level856. Having multiple haziness detectors824a-hrunning in parallel may provide an advantage of reduced computation time in obtaining the haziness confidence level856.

It should be noted that fewer or more haziness detectors824may be implemented. For example, a haziness detector for a day of the year modality information may be implemented. Additionally or alternatively, a haziness detector for an air pollution index may be implemented. The result(s) from one or more of the detectors824may be combined as described above.

FIG. 9is a block diagram illustrating one example of one configuration of an image signal processor932. The image signal processor932may be one example of one or more of the image signal processors132,332described in connection with one or more ofFIGS. 1 and 3. Some objectives to be met during haziness reduction may include enhancing image processing and/or accuracy of associated applications (e.g., ADAS functionality, object detection, object tracking, object recognition, auto focus, depth mapping, and/or 3D modeling, etc. For example, one or more of the approaches for haziness reduction may achieve consistency during preview frames and between snapshots with no flickering, no obvious artifacts (color, noise, and/or artificial contours), and/or enhancing ADAS image quality and detection accuracy. In some configurations, local tone mapping (LTM) hardware in a camera image signal processor pipeline may be controlled to reduce haziness.

An image sensor910may capture image data. The image sensor910described in connection withFIG. 9may be an example of one or more of the image sensors110,310described in connection with one or more ofFIGS. 1 and 3. The image data may be provided to an optional video front end (VFE)988. The video front end988may perform one or more pre-processing operations. For example, the video front end988may generate an image based on the image sensor output(s). The video front end988may provide the pre-processed image data to the image signal processor932and to a statistics collector996.

The image signal processor932may include an optional decompander990, a local tone mapper (LTM)992, and/or an optional scaler/cropper994. It should be noted that more or fewer processing elements may be implemented in the image signal processor932.

The decompander990may perform a decompanding operation on the pre-processed image data. It should be noted that the decompander990may be included in (e.g., implemented by) the image signal processor932and/or the video front end988.

The local tone mapper992may perform local tone mapping on the image data. For example, local tone mapping may include mapping a set of colors to a different set of colors. For instance, a first dynamic range (e.g., a higher dynamic range) of input image data may be mapped to a second dynamic range (e.g., a lower dynamic range). In some configurations, the second dynamic range may correspond to the dynamic range of a display (e.g., display108). Local tone mapping may be performed in an effort to reduce contrast in an input image into a range that is displayable and to preserve color. In some configurations, the local tone mapper992may apply a local tone mapping curve to the image data. In particular, the local tone mapper992may boost local contrast (e.g., adjust intensity values of pixels to increase differences between neighboring pixels potentially rendering objects in the image clearer) in high intensity regions of the image data. Additional detail regarding local tone mapping is given in connection withFIG. 10.

The scaler/cropper994may scale and/or crop the image data. The image signal processor932may accordingly produce a (processed) camera signal954. The camera signal954may be provided to a haziness detector924.

The haziness detector924may be one example of one or more of the haziness detectors124,224,324,424,724,824described in connection with one or more ofFIGS. 1-4 and 7-8. The haziness detector924may determine a haziness confidence level956as described above. The haziness confidence level956may be provided to the local tone mapper992. The local tone mapper992may control (e.g., adjust) the local tone mapping in order to perform haziness reduction. Controlling the local tone mapping may include modifying one or more LTM curves to boost local contrast in high intensity regions. For example, the local tone mapper992may increase the depth of an LTM curve for higher haziness confidence levels956and/or decrease the depth of the LTM curve for lower haziness confidence levels956. Examples of LTM curves are provided in connection withFIG. 10.

It should be noted that the statistics collector996may collect statistics998of the pre-processed image data. The statistics998may be provided to the haziness detector924. In some configurations, the haziness detector924may utilize the statistics998as an input vector that may be used to train the classifier.

FIG. 10includes several graphs1001a-fillustrating various aspects of local tone mapping (LTM) functions (e.g., curves). For example, the graphs1001a-fmay relate to input and output gray-scale levels. Input and output gray-scale levels may optionally be quantized with a different number of bits (e.g., 256 or 1024). The respective vertical axes1003,1073of Graph A1001aand Graph D100dmay represent adjustment values applied to the illuminance. The respective vertical axes1007,1077of Graph B1001band Graph E1001emay represent adjustment amounts applied to the illuminance. The horizontal axes1005,1009,1075,1079of Graphs A-B1001a-band D-E1001d-emay represent illuminance values. Graph C1001cand Graph F1001fmay represent final mapping curves from input illuminance values (on respective horizontal axes1013,1083) to output illuminance values (on respective vertical axes1011,1081).

LTM functions may be utilized by an image signal processor132,332(e.g., the local tone mapper992) to perform haziness reduction, for example. A digital image (e.g., image data from an image sensor, raw camera signal, pre-processed image data, etc.) may be composed of pixels. Each pixel may have an intensity value in one or more channels (e.g., red, green, blue, etc.). The intensity value may have a particular range (from 0-255, for example). For each pixel in an image, an image signal processor132,332(e.g., local tone mapper992) may adjust the pixel's intensity in one or more of the channels based on intensities of neighboring pixels. Graph A1001aillustrates one example of an LTM master curve. The LTM master curve illustrates adjustments applied to the pixel's intensity (along the vertical axis1003) for different average intensities of neighboring pixels (along the horizontal axis1005). As illustrated, image signal processor132,332may increase an intensity of a pixel in a low intensity area and may decrease an intensity of a pixel in a high intensity area. For example, the dip around 200 in the LTM master curves may be used to reduce the intensity level and increase the contrast around the illuminance level of 200. Graph B1001billustrates one example of a master curve scale factor and Graph C1001cillustrates one example of mask rectification.

Graph D1001dillustrates one example of an LTM shift curve. An LTM shift curve shows how an intensity of a pixel is adjusted (along the vertical axis1073) based on a difference (along the horizontal axis1075) between the intensity and an average intensity of neighboring pixels. Graph E1001eillustrates one example of a shift curve scale factor and Graph F1001fillustrates one example of an LTM principle curve.

One or more of an LTM master curve and an LTM shift curve may be controlled based on a haziness confidence level. For example, adjustments to pixel intensities may be increased as a haziness confidence level increases. For instance, reducing haziness may include increasing the depth of one or more LTM curves (as a haziness confidence level increases, for example).

FIG. 11is a graph illustrating a haziness reduction decision. In particular, the graph illustrates a haziness confidence level1115over time1117(in frames (at 30 frames per second (fps), for example)). Specifically, one plot illustrates a haziness confidence level from a previous frame1119and a smoothed haziness confidence level1121. For example, an electronic device may smooth the haziness confidence level over time1117. As illustrated inFIG. 11, a haziness reduction decision1123may occur when the haziness confidence level1115is high enough (e.g., greater than or equal to a haziness confidence threshold). The plots inFIG. 11may illustrate the haziness reduction decision taken from an ADAS application with resized images.

FIG. 12Ais an example of an air pollution index map1235. In some configurations, an electronic device (e.g., electronic device102,202,302) may request and/or receive an air pollution index for one or more locations from a remote device (e.g., remote server). For example, an electronic device may request and/or receive an air pollution index for an area in which the electronic device is located. The air pollution index may be one kind of modality information that may be utilized to determine a haziness confidence value as described herein. The air pollution index map1235illustrates one example of a map with air pollution indices.

FIG. 12Bis a diagram that illustrates approaches for performing haziness detection based on one kind of modality information (air pollution index). For example, the block diagram illustrates one approach for determining an air pollution index haziness confidence level1233, where the air pollution index haziness confidence level1233is a normalized air pollution index. The air pollution index haziness confidence level1233may be one example of a modality haziness confidence level as described in connection withFIG. 8.

In this example, the air pollution index1225(which may be retrieved from a remote server, for example) may be provided to a normalizer1227. In particular, the air pollution index map1235provides examples of air pollution indices that may be obtained by an electronic device. The normalizer1227may normalize the air pollution index1225to produce a raw air pollution index haziness level1229. In some configurations, raw air pollution index haziness level1229may be determined in accordance with the following equation:

Another approach to determining the air pollution index haziness confidence level may be a machine learning approach: using regression to generate a one-dimensional function that links the air pollution index to an air pollution index haziness confidence level (e.g., visual haze score). Data labeling and air pollution indices may be recorded in order to train the regressor. This approach is illustrated in connection with the regression graph1245. The regression graph1245illustrates a haze score1237over the air pollution index1239. In particular, the regression graph1245illustrates a haze score regression1241and a linear haze score1243.

FIG. 13illustrates examples of images before haziness reduction1347and after haziness reduction1349in accordance with the systems and methods disclosed herein. Specifically, the top image is an example of an image before haziness reduction1347, with a visibility of approximately 50 meters (m). The lower image illustrates an example of an image after haziness reduction1349, which illustrates a significant improvement in visibility. Some configurations of the systems and methods disclosed herein may be applied to traffic sign and/or lane detection. For example, an automobile may be implemented with the systems and methods disclosed herein. In hazy scenarios, the systems and methods disclosed herein may provide haziness detection and haziness reduction. The de-hazed images may enable an electronic device (e.g., an automobile) to detect and/or track street signs and/or lane stripes with greater accuracy.

FIG. 14illustrates certain components that may be included within an electronic device1402configured to implement various configurations of the systems and methods disclosed herein. The electronic device1402may be an access terminal, a mobile station, a user equipment (UE), a base station, an access point, a broadcast transmitter, a node B, an evolved node B, etc. The electronic device1402may be implemented in accordance with one or more of the electronic devices102,202,302described herein. The electronic device1402includes a processor1469. The processor1469may be a general purpose single- or multi-chip microprocessor (e.g., an ARM), a special purpose microprocessor (e.g., a digital signal processor (DSP)), a microcontroller, a programmable gate array, etc. The processor1469may be referred to as a central processing unit (CPU). Although just a single processor1469is shown in the electronic device1402, in an alternative configuration, a combination of processors (e.g., an ARM and DSP) could be used.

The electronic device1402also includes memory1451. The memory1451may be any electronic component capable of storing electronic information. The memory1451may be embodied as random access memory (RAM), read-only memory (ROM), magnetic disk storage media, optical storage media, flash memory devices in RAM, on-board memory included with the processor, EPROM memory, EEPROM memory, registers, and so forth, including combinations thereof.

Data1453aand instructions1455amay be stored in the memory1451. The instructions1455amay be executable by the processor1469to implement one or more of the methods500,600disclosed herein. Executing the instructions1455amay involve the use of the data1453athat is stored in the memory1451. When the processor1469executes the instructions1455a, various portions of the instructions1455bmay be loaded onto the processor1469, and various pieces of data1453bmay be loaded onto the processor1469.

The electronic device1402may also include a transmitter1457and a receiver1459to allow transmission and reception of signals to and from the electronic device1402. The transmitter1457and receiver1459may be collectively referred to as a transceiver1461. Multiple antennas1463a-bmay be electrically coupled to the transceiver1461. The electronic device1402may also include (not shown) multiple transmitters, multiple receivers, multiple transceivers and/or additional antennas.

The electronic device1402may include a digital signal processor (DSP)1465. The electronic device1402may also include a communications interface1467. The communications interface1467may enable the electronic device/wireless to device1402to communicate with one or more other devices and/or users. For example, the communications interface1467may include one or more wired and/or wireless interfaces for inter-device communication. In some configurations, the transceiver1461may be included in the communications interface1467. Additionally or alternatively, the communications interface1467may include one or more other input/output interfaces (e.g., touch screens, mouse ports, etc.).

The functions described herein may be implemented in software or firmware being executed by hardware. The functions may be stored as one or more instructions on a computer-readable medium. The terms “computer-readable medium” or “computer-program product” refers to any tangible storage medium that can be accessed by a computer or a processor. By way of example, and not limitation, a computer-readable medium may comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer. Disk and disc, as used herein, includes compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk and Blu-ray® disc where disks usually reproduce data magnetically, while discs reproduce data optically with lasers. It should be noted that a computer-readable medium may be tangible and non-transitory. The term “computer-program product” refers to a computing device or processor in combination with code or instructions (e.g., a “program”) that may be executed, processed or computed by the computing device or processor. As used herein, the term “code” may refer to software, instructions, code or data that is/are executable by a computing device or processor.