Patent Description:
Advanced driver assistance systems (ADAS) may enhance safety and convenience for drivers by implementing sensors provided inside or outside a vehicle. The ADAS may assist a driver by detecting objects and alerting the driver of hazardous road conditions. In addition, a vehicle camera may function as the eyes of a vehicle. A vehicle camera may recognize road condition information including, for example, road signs, lanes, other vehicles, and the like, while a vehicle is traveling and generate data with the recognized road condition information, thereby assisting an ADAS with operating smoothly. While the vehicle is traveling, images may be obtained in various environments. However, in a poor driving environment (for example, in case of occurrence of an issue such as a loss of detailed information of an image due to brightness saturation and locally degraded brightness), a typical ADAS and/or vehicle camera may not effectively perform an additional image reprocessing technology for the stable provision of road condition information.

In one general aspect, an image processing method includes the method steps of claim <NUM>.

The transformed transmission maps and the haze-free images may be based on a size of a single mask of the source transmission map.

The filters may include: a first filter configured to restore a texture; and a second filter configured to suppress a halo artifact.

The first filter may include a multi-directional kernel-based filter, and the second filter may include a guided filter.

The generating of the output image may include: determining an edge map by detecting an edge in the input image; and blending the first haze-free image and the second haze-free image based on the determined edge map.

The blending may include blending the first haze-free image and the second haze-free image by assigning a greater weight to an edge region in the second haze-free image than a weight assigned to the edge region in the first haze-free image, wherein the edge region is determined based on the edge map.

The blending may include: generating first transformed haze-free images of different scales by performing a pyramid transformation on the first haze-free image; generating second transformed haze-free images of different scales by performing a pyramid transformation on the second haze-free image; and generating blended intermediate images by blending the first transformed haze-free images and the second transformed haze-free images based on the edge map.

The generating of the output image may include generating the output image by performing an inverted pyramid transformation on the blended intermediate images.

The determining of the edge map may include determining the edge map by performing a morphological operation on the detected edge of the input image.

A non-transitory computer-readable storage medium may store instructions that, when executed by a processor, configure the processor to perform the method.

In another general aspect, an image processing apparatus includes the features of claim <NUM>.

The filters may include: a multi-directional kernel-based filter configured to restore a texture; and a guided filter configured to suppress a halo artifact.

For the determining of the blending weight, the processor may be configured to: detect an edge in the input image; and determine the blending weight by performing a morphological operation on the detected edge.

The apparatus may include: a camera configured to generate the input image; and a control system configured to control a vehicle based on a generated control instruction, wherein the processor may be configured to generate the control instruction based on the generated output image, and the apparatus is a vehicle control apparatus.

The apparatus may include a memory storing instructions that, when executed by the processor, configure the processor to perform the determining of the source transmission map, the determining of the transformed transmission maps, the generating of the haze-free images, and the generating of the output image.

In another general aspect, a vehicle control apparatus includes: a camera configured to generate an input image of surroundings of a vehicle; a processor configured to perform the method of claim <NUM>, and generate a control instruction for traveling of the vehicle based on the generated output image; and a control system configured to control the vehicle based on the generated control instruction.

In another general aspect, an image processing method includes: determining a source transmission map, based on a dark channel map of an input image generated based on a mask of a predetermined size; determining first and second transformed transmission maps by respectively applying a multi-directional kernel-based filter and a guided filter to the source transmission map; generating first and second haze-free images by removing haze from the input image based respectively on the first and second transformed transmission maps; and generating an output image by blending the first and second haze-free images based on an edge map of the input image.

Throughout the drawings and the detailed description, unless otherwise described or provided, the same reference numerals refer to the same elements, features, and structures.

Also, descriptions of features that are known may be omitted for increased clarity and conciseness.

The terminology used herein is for the purpose of describing particular examples only, and is not to be used to limit the disclosure. As used herein, the terms "include," "comprise," and "have" specify the presence of stated features, numbers, operations, elements, components, and/or combinations thereof, but do not preclude the presence or addition of one or more other features, numbers, operations, elements, components, and/or combinations thereof. The term used in the embodiments such as "unit", etc., indicates a unit for processing at least one function or operation, and where the unit is hardware or a combination of hardware and software. The use of the term "may" herein with respect to an example or embodiment (for example, as to what an example or embodiment may include or implement) means that at least one example or embodiment exists where such a feature is included or implemented, while all examples are not limited thereto.

In addition, terms such as first, second, A, B, (a), (b), and the like may be used herein to describe components. Each of these terminologies is not used to define an essence, order, or sequence of a corresponding component but used merely to distinguish the corresponding component from other component(s). Although terms of "first" or "second" are used herein to describe various members, components, regions, layers, or sections, these members, components, regions, layers, or sections are not to be limited by these terms.

Likewise, expressions, for example, "between" and "immediately between" and "adjacent to" and "immediately adjacent to" may also be construed as described in the foregoing.

Unless otherwise defined, all terms, including technical and scientific terms, used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains consistent with and after an understanding of the present disclosure. Terms, such as those defined in commonly used dictionaries, are to be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and the present disclosure, and are not to be interpreted in an idealized or overly formal sense unless expressly so defined herein.

<FIG> illustrates an example of removing haze (e.g., haze, fog, mist, and/or other similar atmospheric phenomena) from an input image. Referring to <FIG>, an image processing apparatus <NUM> may obtain an input image <NUM>. For example, the image processing apparatus <NUM> may receive the input image <NUM> from a camera (e.g., a camera included in the image processing apparatus <NUM> that captures the input image <NUM>). The input image <NUM> may be a video image including a plurality of frames, or a still image corresponding to a single frame. The image processing apparatus <NUM> may generate an output image <NUM> by performing image processing on the input image <NUM>. For example, the input image <NUM> may include a haze component, and the image processing may include removing the haze component. Thus, the input image <NUM> may also be referred to as a haze image.

In an example, the image processing apparatus <NUM> may remove haze from the input image <NUM> based on a dark channel prior (DCP) method. The DCP method may remove haze from the input image <NUM> using a dark channel map corresponding to the input image <NUM>. The dark channel map may include a dark channel value of each pixel of the input image <NUM>. For example, a dark channel value of one pixel may be determined to be a smallest value among pixel values of channels of the pixel and neighboring pixels. Hereinafter, a non-limiting example of deriving the dark channel map from an input image (e.g., the input image <NUM>) will be described in further detail below with reference to <FIG>.

In an example, the image processing apparatus <NUM> may use a single mask of a single size to perform the DCP method. The mask may refer to a window used to derive the dark channel value, and the mask may correspond to the dark channel map. However, when a typical image processing apparatus generates a single haze-free image using the single mask of the single size, a blocking artifact or edge degradation may occur. To prevent such artifacts, masks of multiple sizes may be used. However, even when the typical image processing apparatus uses the masks of the multiple sizes to perform the DCP method, the prevention of the blocking artifact may be limited, and there may be a lack of stability in blending intermediate outputs.

In contrast, for example, as illustrated in <FIG>, the image processing apparatus <NUM> of one or more embodiments may generate a source transmission map <NUM> based on the input image <NUM> using a single mask of a single size, generate a plurality of transformed transmission maps <NUM> by performing post-processing on the source transmission map <NUM> using various filters, and generate the output image <NUM> by blending a plurality of haze-free images <NUM> generated based on the transformed transmission maps <NUM>. In this example, the image processing apparatus <NUM> of one or more embodiments may perform a DCP method that is robust against edge information and prevents a blocking artifact by determining and selecting an effective combination of the filters and adjusting a blending weight to be determined as suitable for characteristics of the filters.

For example, the image processing apparatus <NUM> may derive the dark channel map corresponding to the input image <NUM> and determine the source transmission map <NUM> based on the dark channel map. Here, an inverse relationship may be established between the dark channel map and a transmission map. A dark channel value of the dark channel map may correspond to an amount of haze of the input image <NUM>. A pixel including a greater number of haze components may have a greater dark channel value. A transmission value may refer to a rate at which light reflected by an object reaches a viewer through the atmosphere. Thus, a pixel of the input image <NUM> including a greater number of haze components may have a smaller transmission value in a corresponding pixel of the source transmission map <NUM>. This is because a greater amount of haze may scatter a greater amount of light.

The image processing apparatus <NUM> may determine the transformed transmission maps <NUM> by applying the filters of different characteristics to the source transmission map <NUM>. In such an example, a combination of filters having complementary characteristics may be selected as the filters to be applied from the filters of different characteristics. For example, a filter of a type specialized or configured to restore a texture (e.g., a texture of a region of the input image <NUM> degraded, or not included, in the source transmission map <NUM>) and a filter of a type specialized or configured to suppress a halo artifact (e.g., a halo artifact generated in the source transmission map <NUM>) may be selected and used to determine the transformed transmission maps <NUM>. For convenience of description, the former filter (e.g., configured to restore the texture) may be referred to herein as a first filter, and the latter filter (e.g., configured to suppress the halo artifact) may be referred to herein as a second filter. In addition, the first filter may reduce a blocking artifact, and the second filter may reduce a blocking artifact and naturally express an edge. For example, the first filter may include a multi-directional kernel-based filter, and the second filter may include a guided filter. Hereinafter, a non-limiting example of deriving transformed transmission maps (e.g., the transformed transmission maps <NUM>) will be described in further detail below with reference to <FIG>.

The image processing apparatus <NUM> may generate the haze-free images <NUM> by removing haze from the input image <NUM> based respectively on the transformed transmission maps <NUM>. For example, the image processing apparatus <NUM> may determine a first transformed transmission map among the transformed transmission maps <NUM> by applying the first filter to the source transmission map <NUM>, and generate a first haze-free image among the haze-free images <NUM> based on the input image <NUM> and the first transformed transmission map. In addition, the image processing apparatus <NUM> may determine a second transformed transmission map among the transformed transmission maps <NUM> by applying the second filter to the source transmission map <NUM>, and generate a second haze-free image among the haze-free images <NUM> based on the input image <NUM> and the second transformed transmission map. The transformed transmission maps <NUM> and the haze-free images <NUM> may be generated based on the source transmission map <NUM> through the single mask of the single size, and thus may be based on a size of the single mask.

The image processing apparatus <NUM> may generate the output image <NUM> by blending the haze-free images <NUM>. The output image <NUM> may also be referred to herein as a blended image. The image processing apparatus <NUM> may determine a suitable blending weight based on the characteristics of the filters and/or features of the haze-free images <NUM>. For example, when the first filter is specialized or configured to restore a texture, the first haze-free image generated using the first filter may have a great weight for a flat region (e.g., a region corresponding to a flat surface of the input image <NUM>). However, the first filter may have a limit in suppressing a halo artifact in a strong edge region. In addition, when the second filter is specialized or configured to suppress a halo artifact, the second haze-free image generated using the second filter may have a great weight for such strong edge region. Thus, a multi-directional kernel-based technology (e.g., included in the first filter) which is effective in restoring an amount of information in a flat region, and a guided filter-based technology (e.g., included in the second filter) which is effective in removing a halo effect in a high-frequency region may be variably used depending on a region. In such a case, the image processing apparatus <NUM> may stably blend the haze-free images <NUM> using a blending method such as sub-band blending.

The output image <NUM> may be used for various image analyses. When haze is removed from the input image <NUM>, an amount of information to be used for the analyses may be improved. For example, the output image <NUM> may be used as data for an advanced driver assistance systems (ADAS). In such a case, the image processing apparatus <NUM> may be provided in a front camera module of a vehicle. In this example, the ADAS may recognize road condition information (including, for example, signs, lanes, and other vehicles) during traveling of the vehicle using the output image <NUM>, and control the vehicle based on the recognized road condition information. In addition or in another example, the output image <NUM> may be used for a computer vision field including, for example, object detection, separation, and recognition through, for example, a surveillance camera and a commercially used camera (e.g., a smartphone camera, a mirrorless camera, and a digital single-lens reflex [DSLR] camera), and the like.

<FIG> is illustrates an example of determining a source transmission map from an input image. Referring to <FIG>, an image processing apparatus (e.g., the image processing apparatus <NUM>) may generate a dark channel map <NUM> corresponding to an input image <NUM> and determine a source transmission map <NUM> based on the generated dark channel map <NUM>. When there is no haze, most of light reflected by an object may be sensed by a camera. However, when there is haze, a portion of light reflected by an object may be scattered by the haze, and ambient light may be reflected by the haze in a direction of the camera. Thus, the portion of the light reflected by the object and a portion of the ambient light may be sensed by the camera. Based on this, the influence of haze on an input image may be modeled as represented by Equation <NUM> below, for example.

In Equation <NUM>, I(x) denotes an input image, and x denotes a pixel index and is defined as a two-dimensional (2D) vector. J(x) denotes an image without being affected by haze. J(x) may also be referred to as a haze-removed image or a haze-free image. In addition, J(x) may be referred to as a radiance. t(x) denotes a medium transmission, which indicates a rate at which light reflected by an object reaches a camera through the air. t(x) may also be referred to as a transmission map. A denotes air-light.

Equation <NUM> may be transformed into Equation <NUM> below, for example. Referring to Equation <NUM>, when I(x) is given, J(x) may be obtained by estimating t(x) and A.

t(x) may be estimated based on a dark channel. For example, the image processing apparatus <NUM> may estimate t(x) for each region of an image by estimating rough distance information through the dark channel. When a distance between an object and a camera increases, the influence of haze increases and the brightness of an image may increase accordingly, and the image processing apparatus <NUM> may estimate the distance information. The dark channel may be estimated as represented by Equation <NUM> below, for example.

In Equation <NUM>, D(x) denotes a dark channel. I denotes an input image and C denotes channels (e.g., three channels of R, G, and B) of the input image. Ω(x) denotes a pixel region corresponding to a mask. y denotes pixels included in the mask. According to Equation <NUM> above, a dark channel D(x) of a pixel x may be determined to be a smallest value among pixel values of the channels of the pixels in the mask including the pixel x at a reference position (e.g., center). For example, a size of the mask may be set as <NUM>×<NUM>, and an operation associated with Equation <NUM> may be performed repeatedly for each pixel position of each image. D(x) may have a value between <NUM> and <NUM>. By inverting D(x) as represented by Equation <NUM> below, for example, t(x) may be obtained.

In Equation <NUM>, D(x) denotes a dark channel, and t(x) denotes a medium transmission. D(x) may correspond to the dark channel map <NUM>, and t(x) may correspond to the source transmission map <NUM>. The dark channel map <NUM> corresponding to the input image <NUM> may be determined based on Equation <NUM>. The source transmission map <NUM> corresponding to the dark channel map <NUM> may be determined based on Equation <NUM>. Subsequently, transformed transmission maps (e.g., the transformed transmission maps <NUM> of <FIG>) corresponding to the source transmission map <NUM> may be determined, and then haze-free images (e.g., the haze-free images <NUM> of <FIG>) may be generated by applying the transformed transmission maps to Equation <NUM> above. Here, A may be estimated through a known method. Hereinafter, a non-limiting example of generating haze-free images (e.g., the haze-free images <NUM>) based on a source transmission map (e.g., either of the source transmission maps <NUM> and <NUM>) will be further described in detail below with reference to <FIG>.

<FIG> illustrates an example of generating haze-free images from a source transmission map. Referring to <FIG>, an image processing apparatus (e.g., the image processing apparatus <NUM>) may generate a first transformed transmission map <NUM> by applying a first filter <NUM> to a source transmission map <NUM>. For example, the first filter <NUM> may be a filter of a type specialized or configured to restore a texture. For example, the first filter <NUM> may be a multi-directional kernel-based filter. In an example, the first transformed transmission map <NUM> may be generated based on Equation <NUM> below.

In Equation <NUM>, t̂ denotes a transmission map in which edge information is restored and may correspond to the first transformed transmission map <NUM>. t denotes a transmission map before the restoration and may correspond to the source transmission map <NUM>. Dj denotes a multi-directional kernel and Wj denotes a weight of each kernel, in which j denotes the number of directions. For example, j = <NUM> (including, for example, upper left, top, upper right, left, middle, right, lower right, bottom, and lower left). λ denotes a regularizing parameter.

In an example, a small weight may be assigned to a relatively large edge region (e.g., a blocking artifact) and a great weight may be assigned to a relatively small edge region (e.g., where significant texture information is to be restored, such as a flat region), using a Gaussian function. For example, in a case in which an edge is present in a transmission map, whether the edge is an artifact generated by a mask or is edge information of an edge of an input image may be determined. In this example, when the edge is determined to be the artifact, the edge may be removed, and when the edge is determined to be the edge information, the edge may not be removed. Thus, a blocking artifact may be removed, and edge information of a haze image may be restored. Due to a characteristic of the first filter <NUM>, detailed edge information may be effectively restored in a flat region. However, a halo effect may not be removed in an edge region having a similar intensity to an edge by the blocking artifact.

The image processing apparatus may also generate a second transformed transmission map <NUM> by applying a second filter <NUM> to the source transmission map <NUM>. The second filter <NUM> may have a characteristic that complements the first filter <NUM>. For example, the second filter <NUM> may be a filter of a type specialized or configured to suppress a halo artifact. For example, the second filter <NUM> may be a guided filter. In an example, the second transformed transmission map <NUM> may be generated based on Equation <NUM> below.

In Equation <NUM>, <MAT> denotes a transmission map in which edge information is restored and may correspond to the second transformed transmission map <NUM>. I denotes a haze image and may correspond to an input image <NUM>. a and b corresponding to a mask denote constants corresponding to respective regions in the haze image. For example, a size of the mask may be <NUM>*<NUM>. The mask described here may be different from that used when generating a dark channel. To distinguish these masks, the mask used for applying the guided filter may be referred to as a guided filter mask, and the mask used for deriving the dark channel may be referred to as a dark channel mask.

A guided kernel-based filter may estimate a linear regression parameter based on a local distribution of pixel values of a haze image using a linear regression method. The constants a and b having certain values for each mask may be linear regression parameters, which may be derived through linear regression as represented by Equation <NUM> below, for example.

In Equation <NUM>, k denotes a mask index and may correspond to a center pixel of a mask. wk denotes a kth mask. i denotes a pixel index of a pixel included in wk. Ii denotes a haze image, and ε denotes a regulating parameter. ti denotes a transmission map before restoration and may correspond to the source transmission map <NUM>. ak and bk may be respectively represented by Equations <NUM> and <NUM> below, for example. <MAT> <MAT>.

In Equations <NUM> and <NUM>, p may correspond to the source transmission map <NUM> from which a noise effect is removed. p denotes a mean of p. µ and σk<NUM> denote a mean of a haze image and a variance of the haze image, respectively. |w| denotes the number of pixels corresponding to a mask. For example, p may be derived by applying a blur effect to t. When constants a and b are determined based on a similarity to the haze image, information of the haze image may be effectively used for a region having great edge information and applied to a transmission map. Thus, a halo effect may be effectively reduced in the region having the great edge information.

The image processing apparatus may determine the transformed transmission maps <NUM> and <NUM> by post-processing the source transmission map <NUM> using the two methods described above, respectively, and derive haze-free images <NUM> and <NUM> corresponding to the transformed transmission maps <NUM> and <NUM> based on Equation <NUM> above. For example, the image processing apparatus may apply the input image <NUM> as I(x) to Equation <NUM> and apply the first transformed transmission map <NUM> as t(x) to Equation <NUM>, and thus obtain a first haze-free image <NUM> corresponding to J(x). In addition, the image processing apparatus may similarly apply the input image <NUM> and the second transformed transmission map <NUM> to Equation <NUM>, and thus obtain a second haze-free image <NUM>.

<FIG> illustrates an example haze-free image generated through a multi-directional kernel-based filter (e.g., a first filter). <FIG> illustrates an example haze-free image generated through a guided filter (e.g., a second filter). Referring to <FIG>, haze-free images <NUM> and <NUM> respectively corresponding to input images <NUM> and <NUM> may be generated based on a multi-directional kernel-based filter and the respective input images <NUM> and <NUM>. The input image <NUM> may include a strong edge region <NUM> and the input image <NUM> may include a flat region <NUM>. As shown in the haze-free image <NUM>, a texture of a flat region <NUM>, corresponding to the flat region <NUM>, is effectively restored through the multi-directional kernel-based filter. However, as shown in the haze-free image <NUM>, that there still is a halo artifact in a strong edge region <NUM> corresponding to the strong edge region <NUM>.

Referring to <FIG>, haze-free images <NUM> and <NUM> respectively corresponding to input images <NUM> and <NUM> may be generated based on a guided filter and the respective input images <NUM> and <NUM>. The input images <NUM> and <NUM> may further respectively correspond to the input images <NUM> and <NUM> of <FIG>. The input image <NUM> may include a strong edge region <NUM>, and the input image <NUM> may include a flat region <NUM>. As shown in the haze-free image <NUM>, a halo artifact of a strong edge region <NUM>, corresponding to the strong edge region <NUM>, is effectively removed through the guided filter. However, as shown in the haze-free image <NUM>, texture information of a flat region <NUM>, corresponding to the flat region <NUM>, is lost.

Thus, by using an image generated through the multi-directional kernel-based filter for texture information and using an image generated through the guided filter for a strong edge, the image processing apparatus <NUM> of one or more embodiments may generate an output image that is robust against both the cases. Hereinafter, a non-limiting example of generating an output image (e.g., the output image <NUM>) by blending images will be described in detail with reference to <FIG>.

<FIG> illustrates an example of generating an output image using haze-free images. Referring to <FIG>, an image processing apparatus (e.g., the image processing apparatus <NUM>) may generate an output image <NUM> by blending a first haze-free image <NUM> and a second haze-free image <NUM>. In such a case, the image processing apparatus may blend the first haze-free image <NUM> and the second haze-free image <NUM> by applying a weight <NUM>. The weight <NUM> may also be referred to herein as a blending weight. The haze-free images <NUM> and <NUM> may have respective strong points in different regions, and it may thus be effective to blend the haze-free images <NUM> and <NUM> by assigning the weight <NUM> according to their respective strong points. For example, the weights <NUM> may include a weight assigned.

In an example, the image processing apparatus may determine a blending weight based on edges in an input image and blend haze-free images (e.g., the haze-free images <NUM> and <NUM>) based on the determined blending weight. For example, the first haze-free image <NUM> may be generated based on a first filter specialized or configured to restore a texture, and the second haze-free image <NUM> may be generated based on a second filter specialized or configured to suppress a halo artifact. In this example, the first haze-free image <NUM> may be used to generate the output image <NUM> for a flat region, and the second haze-free image <NUM> may be used to generate the output image <NUM> for a strong edge region.

In the example, to generate the output image <NUM>, the image processing apparatus may determine an edge map by detecting an edge in the input image and blend the first haze-free image <NUM> and the second haze-free image <NUM> based on the determined edge map. For example, such edge detection may include Canny edge detection. The edge map may specify a strong edge region in a corresponding image. The first haze-free image <NUM> may have a strong point in restoring a texture of a flat region based on a characteristic of the first filter, and the second haze-free image <NUM> may have a strong point in suppressing a halo artifact in a strong edge region based on a characteristic of the second filter. That is, for example, more information of the texture of the flat region may be restored in the first haze-free image <NUM> than in the second haze-free image <NUM>, and more information of the halo artifact in the strong edge region may be suppressed in the second haze-free image <NUM> than in the first haze-free image <NUM>. Thus, the image processing apparatus may blend the first haze-free image <NUM> and the second haze-free image <NUM> by assigning a great weight to an edge region in the second haze-free image <NUM> corresponding to the edge map (e.g., a greater weight than a weight assigned to a corresponding edge region in the first haze-free image <NUM>), and blending the first haze-free image <NUM> and the second haze-free image <NUM> based on the assigned great weight (and the assigned lesser weight).

In the example, the image processing apparatus may determine the edge map by detecting an edge in the input image and performing a morphological operation on the detected edge. For example, through the morphological operation, a narrow region corresponding to an edge line may be expanded to a wide region having a wider width than the edge line. In a case in which there is an edge in the second haze-free image <NUM>, it may be effective to use the edge by a unit of a wide region expanded from a line in lieu of a unit a narrow line, in the process of blending. Thus, through the morphological operation, the image processing apparatus may expand a width of the edge. The morphological operation may be represented by Equation <NUM> below, for example.

In Equation <NUM>, E'(x) denotes an edge map obtained after the morphological operation is applied, and E(x) denotes an edge map before the application of the morphological operation, y denotes a position of a pixel corresponding to a mask Ω. For example, a size of the mask may be <NUM>*<NUM>. In a case in which there is a region determined to be an edge inside the mask, an entire area of the mask may be substituted by a value that is same as the edge. Through such a process described above, an edge region may be expanded, and thus the image processing apparatus may maximize an effect of blending.

<FIG> illustrates an example of sub-band blending. In a case of image blending, there may be an unnatural result according to a blending method. For example, referring to <FIG>, an edge map <NUM> may be generated through a morphological operation. In this example, when blending a first haze-free image <NUM> and a second haze-free image <NUM> using the generated edge map <NUM>, an unnatural blending result may occur in a region in which a size of the edge map <NUM> changes drastically. In an example, through a blending method such as sub-band blending, the image processing apparatus may naturally blend images. The sub-band blending may refer to a method of blending images by classifying the images by frequency.

In an example, an image processing apparatus (e.g., the image processing apparatus <NUM>) may perform Laplacian pyramid blending as the sub-band blending. The Laplacian pyramid blending may be represented by Equation <NUM> below, for example.

In Equation <NUM>, L denotes Laplacian pyramid blending for which three images R<NUM>, R<NUM>, and E' may be used as an input. R<NUM> and R<NUM> are a radiance obtained using a multi-directional kernel-based filter and a radiance obtained using a guided filter, respectively, and may correspond to the first haze-free image <NUM> and the second haze-free image <NUM>, respectively. E' may correspond to the edge map <NUM> and be generated through a morphological operation. R' denotes a blended radiance and may correspond to an output image <NUM>.

Referring to <FIG>, the image processing apparatus may generate first transformed haze-free images S11, S12, and S13 of various scales by performing a pyramid transform on the first haze-free image <NUM>. For example, the pyramid transform may include providing a blur effect to an image, downsampling a resultant image obtained from the blur effect, upsampling a resultant image obtained from the downsampling, and obtaining a Laplacian image corresponding to a difference between an initial image and a final resultant image. The first transformed haze-free images S11, S12, and S13 may correspond to Laplacian images generated by such a pyramid transform. The pyramid transform may also be performed on the second haze-free image <NUM>, and second transformed haze-free images S21, S22, and S23 of various scales may thus be generated.

The image processing apparatus may generate blended images S1, S2, and S3 by blending the first transformed haze-free images S11, S12, and S13 and the second transformed haze-free images S21, S22, and S23 based on the edge map <NUM>. The image processing apparatus may then generate the output image <NUM> by performing an inverted pyramid transform on the generated blended images S1, S2, and S3. Here, the blended images S1, S2, and S3 may also be referred to herein as intermediate blended images S1, S2, and S3.

<FIG> illustrates a comparison between simple blending and sub-band blending. Referring to <FIG>, an image <NUM> corresponds to a result of simple blending, and an image <NUM> corresponds to a result of Laplacian pyramid blending. As shown in the image <NUM>, there is an unnatural blending result such as blocks or boundaries around an edge. However, as shown in the image <NUM>, surroundings of the edge are expressed naturally. Such Laplacian sub-band blending of one or more embodiments may use multiple scales and variably apply a weight to a low-frequency region and a high-frequency region, and thus obtain a stable blending result, compared to a typical method of simply applying the weight to an original scale.

<FIG> illustrates an example of an image processing method for haze removal. Referring to <FIG>, an image processing apparatus may determine a source transmission map based on a dark channel map of an input image in operation <NUM>, determine transformed transmission maps by applying filters of different characteristics to the determined source transmission map in operation <NUM>, generate haze-free images by removing haze from the input image based respectively on the transformed transmission maps in operation <NUM>, and generate an output image by blending the generated haze-free images in operation <NUM>. In an example, a more detailed description of the image processing method described above with reference to <FIG> includes the operations described above with reference to <FIG> and what is to be described hereinafter with reference to <FIG>. Thus, a repeated and detailed description will be omitted here for brevity.

<FIG> illustrates an example of an image processing apparatus for haze removal. Referring to <FIG>, an image processing apparatus <NUM> may include a processor <NUM> (e.g., one or more processors) and a memory <NUM> (e.g., one or more memories). The image processing apparatus <NUM> may generate an output image by removing haze from an input image. The image processing apparatus <NUM> may generate various radiances using various filters and generate the output image by blending the generated radiances. The input image may be a haze image, and the output image may be a blended image. For example, the output image may be used as data for ADAS. In an example, the image processing apparatus <NUM> corresponds to, or is included in, the image processing apparatus <NUM>.

The memory <NUM> may be connected to the processor <NUM>, and store instructions executable by the processor <NUM> and data to be processed by the processor <NUM> or data processed by the processor <NUM>. The memory <NUM> may include a non-transitory computer-readable medium, for example, a high-speed random-access memory (RAM) and/or nonvolatile computer-readable storage medium (e.g., at least one disk storage device, a flash memory device, or other nonvolatile solid-state memory devices).

The processor <NUM> may execute instructions to perform one or more, or all, of the methods and operations described above with reference to <FIG>. For example, the processor <NUM> may determine a source transmission map based on a dark channel map of an input image, determine transformed transmission maps by applying filters of different characteristics to the determined source transmission map, generate haze-free images by removing haze from the input image based respectively on the transformed transmission maps, and generate an output image by blending the generated haze-free images.

<FIG> illustrates an example of a subsequent method after haze removal. As described above, when an output image is generated after haze is removed from an input image, the generated output image may be used for various image analyses. The output image will be hereinafter described as being used as data for ADAS. Operations <NUM> through <NUM> to be described hereinafter with reference to <FIG> may be performed by a vehicle control apparatus. The vehicle control apparatus may be provided in a vehicle and configured to control various functions for the vehicle including, for example, traveling-related functions such as acceleration, steering, and braking, and additional functions such as door opening and closing, window opening and closing, and airbag deployment.

Referring to <FIG>, in operation <NUM>, the vehicle control apparatus may generate a target image by removing haze from a haze image. The target image may correspond to an output image. In operation <NUM>, the vehicle control apparatus may recognize a lane and an object in the target image. As an amount of information of the target image is improved by the haze removal, a recognition rate of the lane and the object may be improved. In operation <NUM>, the vehicle control apparatus may control the vehicle based on a result of the recognizing. For example, the vehicle control apparatus may control the traveling-related functions and/or the additional functions of the vehicle that are described above. Based on such a recognition result, the vehicle control apparatus may perform, for example, adaptive cruise control (ACC), autonomous emergency braking (AEB), blind spot detection (BSD), lane change assistance (LCA), and the like.

<FIG> illustrates an example of an electronic apparatus. Referring to <FIG>, an electronic apparatus <NUM> may include a processor <NUM> (e.g., one or more processors), a memory <NUM> (e.g., one or more memories), a camera <NUM>, a sensor <NUM>, a control system <NUM>, a storage device <NUM>, an input device <NUM>, an output device <NUM>, and a network interface <NUM>. The processor <NUM>, the memory <NUM>, the camera <NUM>, the sensor <NUM>, the control system <NUM>, the storage device <NUM>, the input device <NUM>, the output device <NUM>, and the network interface <NUM> may communicate with one another through a communication bus <NUM>. In an example, the electronic apparatus <NUM> corresponds to, or is included in, either of the image processing apparatus <NUM> and the image processing apparatus <NUM>.

The electronic apparatus <NUM> may generate an output image by removing haze from an input image and perform subsequent operations based on the output image. For example, the electronic apparatus <NUM> may correspond to the vehicle control apparatus described above with reference to <FIG>, and perform subsequent operations for a vehicle, for example, traveling function control and additional function control. For example, the electronic apparatus <NUM> may include the image processing apparatus <NUM> of <FIG> in terms of structure and/or function. For example, the electronic apparatus <NUM> may be embodied as at least a portion of a mobile device (e.g., a mobile phone, a smartphone, a personal digital assistant (PDA), a netbook, a tablet computer, and a laptop computer), a wearable device (e.g., a smart watch, a smart band, and smart eyeglasses), a computing device (e.g., a desktop and a server), a security device (e.g., a door lock), and a vehicle (e.g., a smart vehicle).

The processor <NUM> may execute functions and instructions in the electronic apparatus <NUM>. For example, the processor <NUM> may process instructions stored in the memory <NUM> and/or the storage device <NUM>. In an example, the processor <NUM> may generate an output image by removing haze from an input image, and generate a control instruction on traveling of a vehicle based on the generated output image. In addition, the processor <NUM> may perform one or more, or all, of the methods or operations described above with reference to <FIG>.

The memory <NUM> may store data for haze removal. The memory <NUM> may include a non-transitory computer-readable storage medium or device. The memory <NUM> may store instructions to be executed by the processor <NUM>, and store related information while software or an application is being executed by the electronic apparatus <NUM>.

The camera <NUM> may capture a video image and/or a still image (or a photograph). For example, the camera <NUM> may be installed in a vehicle and configured to capture an image of the surroundings of the vehicle in a preset direction (e.g., front, side, rear, upward, and downward), and generate an image associated with traveling of the vehicle. In an example, the camera <NUM> may provide a three-dimensional (3D) image including depth information of objects.

The sensor <NUM> may sense visual, auditory, and tactile information associated with the electronic apparatus <NUM>. For example, the sensor <NUM> may include an ultrasonic sensor, a radio detection and ranging (radar) sensor, and a light detection and ranging (lidar) sensor. The control system <NUM> may control a vehicle based on a control instruction of the processor <NUM>. For example, the control system <NUM> may physically control various functions for the vehicle that include traveling functions such as acceleration, steering, and braking, and additional functions such as door opening and closing, window opening and closing, and airbag deployment.

The storage device <NUM> may include a non-transitory computer-readable storage medium or device. In an example, the storage device <NUM> may store a greater amount of information for a longer period of time compared to the memory <NUM>. For example, the storage device <NUM> may include a magnetic hard disk, an optical disc, a flash memory, a floppy disk, or a nonvolatile memory of another type that is well-known in a related technical field.

The input device <NUM> may receive an input from a user through a traditional input method using a keyboard and a mouse, and a new input method using a touch input, a voice input, and an image input. The input device <NUM> may include, for example, a keyboard, a mouse, a touchscreen, a microphone, and other devices that may detect the input from the user and transmit the detected input to the electronic apparatus <NUM>.

The output device <NUM> may provide an output of the electronic apparatus <NUM> to a user through a visual, auditory, or tactile channel. The output device <NUM> may include, for example, a display, a touchscreen, a speaker, a vibration generator, and other devices that may provide the user with the output. The network device <NUM> may communicate with an external device through a wired or wireless network.

The image processing apparatuses, the vehicle control apparatuses, the electronic apparatuses, processors, memories, cameras, sensors, control systems, storage devices, input devices, output devices, network interfaces, image processing apparatus <NUM>, image processing apparatus <NUM>, processor <NUM>, memory <NUM>, electronic apparatus <NUM>, bus <NUM>, processor <NUM>, memory <NUM>, camera <NUM>, sensor <NUM>, control system <NUM>, storage device <NUM>, input device <NUM>, output device <NUM>, network interface <NUM>, and other apparatuses, devices, units, modules, and components described herein with respect to <FIG> are implemented by or representative of hardware components. Examples of hardware components that may be used to perform the operations described in this application where appropriate include controllers, sensors, generators, drivers, memories, comparators, arithmetic logic units, adders, subtractors, multipliers, dividers, integrators, and any other electronic components configured to perform the operations described in this application. In other examples, one or more of the hardware components that perform the operations described in this application are implemented by computing hardware, for example, by one or more processors or computers. A processor or computer may be implemented by one or more processing elements, such as an array of logic gates, a controller and an arithmetic logic unit, a digital signal processor, a microcomputer, a programmable logic controller, a field-programmable gate array, a programmable logic array, a microprocessor, or any other device or combination of devices that is configured to respond to and execute instructions in a defined manner to achieve a desired result. In one example, a processor or computer includes, or is connected to, one or more memories storing instructions or software that are executed by the processor or computer. Hardware components implemented by a processor or computer may execute instructions or software, such as an operating system (OS) and one or more software applications that run on the OS, to perform the operations described in this application. The hardware components may also access, manipulate, process, create, and store data in response to execution of the instructions or software. For simplicity, the singular term "processor" or "computer" may be used in the description of the examples described in this application, but in other examples multiple processors or computers may be used, or a processor or computer may include multiple processing elements, or multiple types of processing elements, or both. For example, a single hardware component or two or more hardware components may be implemented by a single processor, or two or more processors, or a processor and a controller. One or more hardware components may be implemented by one or more processors, or a processor and a controller, and one or more other hardware components may be implemented by one or more other processors, or another processor and another controller. One or more processors, or a processor and a controller, may implement a single hardware component, or two or more hardware components. A hardware component may have any one or more of different processing configurations, examples of which include a single processor, independent processors, parallel processors, single-instruction single-data (SISD) multiprocessing, single-instruction multiple-data (SIMD) multiprocessing, multiple-instruction single-data (MISD) multiprocessing, and multiple-instruction multiple-data (MIMD) multiprocessing.

Instructions or software to or software to control computing hardware, for example, one or more processors or computers, to implement the hardware components and perform the methods as described above may be written as computer programs, code segments, instructions or any combination thereof, for individually or collectively instructing or configuring the one or more processors or computers to operate as a machine or special-purpose computer to perform the operations that are performed by the hardware components and the methods as described above. The instructions or software may be written using any programming language based on the block diagrams and the flow charts illustrated in the drawings and the corresponding descriptions used herein, which disclose algorithms for performing the operations that are performed by the hardware components and the methods as described above.

Claim 1:
An image processing method, comprising:
determining (<NUM>) a source transmission map (<NUM>; <NUM>; <NUM>) based on a dark channel map (<NUM>) of an input image (<NUM>; <NUM>; <NUM>; <NUM>; <NUM>; <NUM>; <NUM>);
determining (<NUM>) transformed transmission maps (<NUM>; <NUM>, <NUM>) by applying different filters (<NUM>, <NUM>) to the determined source transmission map;
generating (<NUM>) haze-free images (<NUM>; <NUM>, <NUM>; <NUM>; <NUM>; <NUM>; <NUM>; <NUM>, <NUM>; <NUM>, <NUM>) by removing haze from the input image based respectively on the determined transformed transmission maps; and
generating (<NUM>) an output image (<NUM>; <NUM>; <NUM>; <NUM>; <NUM>) by blending the generated haze-free images based on a blending weight (<NUM>),
the method being characterized in that the blending weight is determined based on edges in the input image,
and in that the generated haze-free images comprise:
a first haze-free image (<NUM>; <NUM>; <NUM>) for determining a flat region of the output image, wherein the first haze-free image comprises the flat region including a restored texture; and
a second haze-free image (<NUM>; <NUM>; <NUM>) for determining a strong edge region of the output image, wherein the second haze-free image comprises the strong edge region including a suppressed halo artifact.