Patent Description:
Images may be obtained through various devices. For example, an image may be captured by using a built-in camera of a smartphone, and a captured image may be corrected.

Algorithms for correcting an image may vary. As a representative example, a method of determining one scene that may represent an entire image and correcting an overall tone and/or color of an image to emphasize the scene may be used.

However, objects having various characteristics may be in one image. When uniform correction is performed on an entire image without distinguishing the characteristics of objects in the image, it is difficult to obtain an evenly corrected image.

<CIT> describes systems and methods of face and skin sensitive image enhancement. A face map that includes for each pixel of an input image a respective face probability value indicating a degree to which the pixel corresponds to a human face is calculated. A skin map that includes for each pixel of the input image a respective skin probability value indicating a degree to which the pixel correspond to human skin is ascertained. The input image is enhanced with an enhancement level that varies pixel-by-pixel in accordance with the respective face probability values and the respective skin probability values.

<CIT> describes a method and apparatus for enhancing a digital photographic image that uses a combination of global and content-specific operations. Image data is analyzed to detect one or more content items in the image to be separately enhanced. Then a content-specific enhancement operation is performed on data representing at least one detected content item, separately from the remainder of the image data.

<CIT> describes an image correction method that includes: obtaining information regarding at least one of brightness and colors of pixels constituting an input image, the input image comprising a plurality of regions classified according to whether the at least one of the brightness and the colors of the pixels are substantially uniformly distributed in a corresponding region; determining a weight with respect to at least one pixel based on the obtained information; and correcting the input image with respect to the at least one pixel based on the determined weight.

<CIT> describes how sub-regions within a face image are identified to be enhanced by applying a localized smoothing kernel to luminance data corresponding to the sub-regions of the face image. An enhanced face image is generated including an enhanced version of the face that includes certain original pixels in combination with pixels corresponding to the one or more enhanced sub-regions of the face.

According to one aspect of the invention there is provided an image processing system according to claim <NUM>.

According to a second aspect of the invention, a system-on-chip (SoC) for generating an enhanced image by correcting a first image is provided according to claim <NUM>.

Example embodiments will be more clearly understood from the following detailed description taken in conjunction with the accompanying drawings in which:.

<FIG> is a block diagram showing an image processing system according to an example embodiment.

Referring to <FIG>, an image processing system <NUM> includes a plurality of circuits for outputting an enhanced image by correcting an image image_h (first image). The image image_h may be generated by an image signal processor (ISP). For example, the image image_h may correspond to a still image or one frame of a moving picture including a plurality of frames (a video).

The image processing system <NUM> includes an image processing circuit <NUM>, a segmentation circuit <NUM>, and may include an upscaling circuit <NUM>. The image processing circuit <NUM>, the segmentation circuit <NUM>, and the upscaling circuit <NUM> may each be embodied by hardware, software (or firmware), or a combination of hardware and software to implement the inventive concepts. Also, although not shown, the image processing system <NUM> may further include an encoder for encoding an enhanced image.

The segmentation circuit <NUM> may receive an image and perform a segmentation operation of dividing the image into a plurality of regions by using a trained neural network model <NUM>. The segmentation circuit <NUM> infers a class corresponding to each of a plurality of pixels in the image. The segmentation circuit <NUM> forms a segmentation map by inferring a class corresponding to each pixel. According to an embodiment, pixels corresponding to the same class inference information may form a region.

Also, the segmentation circuit <NUM> calculates the confidence of the inferred class for each pixel. The confidence is formed in the form of a confidence map. According to an embodiment, the size of a segmentation map and the size of a confidence map may each be the same as the size of the image.

According to an embodiment, the segmentation circuit <NUM> may receive a low-resolution image image_l, which is a reduced image of an image image_h to be corrected. Restated, the low-resolution image may be a lower resolution version of the first image. The low-resolution image image_l may have a smaller size than the image image_h. The segmentation circuit <NUM> may generate a low-resolution segmentation map and a low-resolution confidence map for the low-resolution image image_l by using the neural network model <NUM>. In this case, the size of the low-resolution segmentation map and the size of the low-resolution confidence map may each be the same as the size of the low-resolution image image_l. Hereinafter, the neural network model <NUM> will be described in greater detail with reference to <FIG>.

The upscaling circuit <NUM> may receive the low-resolution segmentation map and the low-resolution confidence map from the segmentation circuit <NUM> and generate a segmentation map and a confidence map each having the same size as the image image_h by interpolating and enlarging each in correspondence to the size of the image image_h.

The image processing circuit <NUM> may receive the image image_h, perform a series of processes for improving the image quality of the image image_h or correcting the image image_h, and generate an enhanced image as a result thereof. The image processing circuit <NUM> may receive the image image_h and the segmentation map and the confidence map for the image image_h.

The image processing circuit <NUM> includes a tuning circuit <NUM> and at least one correcting circuit <NUM>.

The tuning circuit <NUM> determines correction effects to be applied to each pixel of the image image_h and the intensity of the correction effects based on the segmentation map and the confidence map. Hereinafter, for convenience of explanation, the intensity of a correction effect may be referred to as a correction value. To apply at least one correction effect to each pixel, the tuning circuit <NUM> generates correction maps corresponding to respective correction effects and provide the correction maps to the at least one correcting circuit <NUM>.

The at least one correcting circuit <NUM> may receive the correction maps from the tuning circuit <NUM> and apply correction effects to the respective pixels of the image image_h. According to an embodiment, the at least one correcting circuit <NUM> may apply a correction effect to a plurality of pixels to which the same correction effect is to be applied.

According to an embodiment, the at least one correcting circuit <NUM> may include a denoise circuit <NUM>, a color correction circuit <NUM>, and a sharpen circuit <NUM>. The denoise circuit <NUM> may reduce noise of at least one pixel, the color correction circuit <NUM> may adjust the color value of at least one pixel, and the sharpen circuit <NUM> may increase the sharpness of at least one pixel. The at least one correcting circuit <NUM> may further include a circuit providing various types of correction effects, and the at least one correcting circuit <NUM> may each be implemented by hardware, software (or firmware), or a combination of hardware and software.

According to an embodiment, the image processing circuit <NUM> may perform correction region-by-region on regions each including pixels that are classified into the same class. For example, to apply a first correction effect on a first region including pixels that are classified into a first class, the image processing circuit <NUM> may use at least one of the denoise circuit <NUM> that reduces the noise of the first region, the color correction circuit <NUM> that adjusts the color value of the first region, and the sharpen circuit <NUM> that increases the sharpness of the first region. At this time, the image processing circuit <NUM> may use a confidence map to adjust the intensity of a correction effect applied to a first pixel or the first region including the first pixel.

The image processing circuit <NUM> according to example embodiments may determine and apply a correction effect to be applied to each of a plurality of pixels in the image image_h. In other words, by applying correction effects pixel-by-pixel, fine correction may be performed, and thus, an image with improved image quality may be obtained.

Also, by performing segmentation and correction pixel-by-pixel on the image image_h in which pixels of different classes are complexly arranged, accurate corrections corresponding to the respective pixels may be performed as compared to corrections region-by-region, thereby improving user satisfaction.

<FIG> is a block diagram showing a neural network model according to an example embodiment.

Referring to <FIG>, the segmentation circuit <NUM> may receive the low-resolution image image_l, which is generated by reducing the image image_h, from an ISP. The segmentation circuit <NUM> may input the low-resolution image image_l to the neural network model <NUM> and obtain a low-resolution segmentation map map_seg_l and a low-resolution confidence map map_conf_l as outputs. The segmentation circuit <NUM> may provide the low-resolution segmentation map map_seg_l and the low-resolution confidence map map_conf_l to the upscaling circuit <NUM>.

The low-resolution segmentation map map_seg_l may include class inference information for each pixel of the low-resolution image image_l. The low-resolution confidence map map_conf_l may include confidence of class inference information for each pixel of the low-resolution image image_l. For example, a second pixel of the low-resolution segmentation map map_seg_l and a third pixel of the low-resolution confidence map map_conf_l may each correspond to the first pixel of the low-resolution image image_l. In this case, the second pixel of the low-resolution segmentation map map_seg_l may indicate class inference information for the first pixel, and the third pixel of the low-resolution confidence map map_conf_l may indicate the probability and/or confidence of the corresponding class inference information.

The segmentation circuit <NUM> includes the neural network model <NUM>. The neural network model <NUM> has been trained. The neural network model <NUM> may learn relationships between a plurality of classes defined to have different correction effects and pixels. In this case, the classes may have different weights for at least one of a denoise effect, a color correction effect, and a sharpening effect.

The neural network model <NUM> may be trained off-line. According to an embodiment, the neural network model <NUM> may be generated by being trained in a training device, e.g., a server that trains a neural network based on a large amount of learning data. The neural network model <NUM> uses an arbitrary pixel and a class labeled in correspondence to the arbitrary pixel as training data. For example, a training image and classes labeled for respective pixels of the training image may be used as training data.

Hereinafter, in the present specification, descriptions will be given under an assumption that parameters (e.g., network topology, bias, weight, etc.) of the neural network model <NUM> are already determined through training. However, the inventive concepts are not limited thereto.

For example, the neural network model <NUM> may include at least one of various types of neural network models like a convolution neural network (CNN), a region with convolution neural network (R-CNN), a region proposal network (RPN), a recurrent neural network (RNN), a stacking-based deep neural network (S-DNN), a state-space dynamic neural network (S-SDNN), a deconvolution network, a deep belief network (DBN), a restricted Boltzmann machine (RBM), a fully convolutional network, a long short-term memory (LSTM) network, and a classification network.

<FIG> is a block diagram showing an upscaling circuit according to an example embodiment.

Referring to <FIG>, the upscaling circuit <NUM> may generate a segmentation map map_seg_h and a confidence map map_conf_h by enlarging and/or interpolating the low-resolution segmentation map map_seg_l and the low-resolution confidence map map_conf_l. The size of each of the segmentation map map_seg_h and the confidence map map_conf_h generated by the upscaling circuit <NUM> may be the same as the size of the image image_h in <FIG>.

The upscaling circuit <NUM> may use various algorithms to perform scaling and/or interpolation. For example, the upscaling circuit <NUM> may convolve a low quality or low-resolution image through an interpolation kernel and resample a convolved image in a new grid. For example, the upscaling circuit <NUM> may use a linear interpolation filter and may apply a joint bilateral filter to keep edges sharp. However, the inventive concepts are not limited thereto.

The upscaling circuit <NUM> may provide the segmentation map map_seg_h and the confidence map map_conf_h to an image processing circuit (e.g., <NUM> of <FIG>).

<FIG> is a diagram showing examples of a segmentation map and a confidence map according to an example embodiment.

Referring to <FIG> and <FIG> together, the segmentation map map_seg_h may include a plurality of pixels px, and each pixel px may include class inference information info_class. In other words, each pixel px of the segmentation map map_seg_h may include class inference information info_class of a corresponding pixel of the image image_h. According to an embodiment, the class inference information info_class may include n (n is a natural number; e.g., n is <NUM>) bits, but the inventive concepts are not limited thereto. For example, when the type of class inference information info_class output from a neural network model increases, the number of bits representing the class inference information info_class may increase. Restated, each pixel of the segmentation map may include a value with n bits.

The confidence map map_conf_h may include a plurality of pixels px, and each pixel px may include confidence conf. Each pixel px of the confidence map map_seg_h may correspond to a pixel of the segmentation map map_seg_h and a pixel of the image image_h. Each pixel px of the confidence map map_conf_h may include confidence conf (or reliability) of the class inference information info_class of a corresponding pixel of the image image_h. According to an example embodiment, the confidence conf may include m (m is a natural number greater than n; e.g., m is <NUM>) bits, but the inventive concepts are not limited thereto. Restated, each pixel of the confidence map may include a value with m bits, where a value of m is greater than a value of n.

<FIG> is a diagram showing an example of a segmentation map according to an example embodiment.

Referring to <FIG>, the position of each pixel in the segmentation map map_seg_h may correspond to the position of a corresponding pixel in the image image_h, and the value of each pixel in the segmentation map map_seg_h may indicate a class to which a corresponding pixel of the image image_h is inferred.

For example, in the image image_h, a portion corresponding to a branch may be inferred as a first class class1, a portion corresponding to the sky may be inferred as a second class class2, a portion corresponding to the human hair may be inferred as a third class class3, a portion corresponding to the human face may be inferred as a fourth class class4, and a portion corresponding to the human skin may be inferred as a fifth class class5.

According to an example embodiment, pixels of the image image_h may be segmented considering correction effects to be applied to the respective pixels of the image image_h. In other words, pixels to which different correction effects need to be applied may be segmented into different classes.

Also, according to an example embodiment, because segmentation is performed pixel-by-pixel, the accuracy of segmentation may be improved even in a detailed and complex image image_h including objects like hair and tree branches.

<FIG> is a diagram showing an example of an operation of a tuning circuit according to an example embodiment.

Referring to <FIG>, the tuning circuit <NUM> may receive the image image_h from the outside and may receive the segmentation map map_seg_h and the confidence map map_conf_h from the upscaling circuit <NUM> of <FIG>. For example, the image image_h may be generated within an image processing circuit (e.g., <NUM> of <FIG>) through an image sensor.

The tuning circuit <NUM> may include a configuration table <NUM>, a select circuit <NUM>, and a mix circuit <NUM>. The select circuit <NUM> and the mix circuit <NUM> may each be implemented by hardware, software (or firmware), or a combination of hardware and software.

The configuration table <NUM> may include class inference information that each of the pixels of the segmentation map map_seg_h indicates and correction effects to be applied according to confidence information that each pixel of the confidence map map_conf_h indicates. For example, the configuration table <NUM> may include correction values indicating the intensity of correction effects.

According to an embodiment, a pixel of the segmentation map map_seg_h corresponding to an arbitrary pixel of the image image_h may represent a first class, and a pixel of the confidence map map_conf_h corresponding to the arbitrary pixel of the image image_h may represent a first value. In this case, when the first class has the first value and is inferred, the configuration table <NUM> may include correction effects to be applied to the arbitrary pixel and a correction value to be applied to the arbitrary pixel. For example, when an arbitrary pixel is determined as the first class having a first confidence, the configuration table <NUM> may apply a first correction effect to the arbitrary pixel as much as a first correction value and apply a second correction effect to the arbitrary pixel as much as a second correction value.

In the present specification, although it is described that the configuration table <NUM> is determined in advance in correspondence to at least one correcting circuit, the inventive concepts are not limited thereto, and information included in the configuration table <NUM> may be changed.

The select circuit <NUM> may determine the class of each pixel of the image image_h based on the segmentation map map_seg_h, the confidence map map_conf_h, and a pre-set threshold value. For example, the select circuit <NUM> may determine the class of each pixel of the image image_h based on the class inference information and the confidence for each pixel. In this case, the select circuit <NUM> may include class information and a pre-set threshold value. The class information may include types of classes to which each pixel may be classified into.

A case in which an arbitrary pixel is inferred as a first class according to the segmentation map map_seg_h and inference confidence of the arbitrary pixel corresponds to a first value according to the confidence map map_conf_h will be described as an example. A threshold needed to be determined as the first class may be a first threshold. The select circuit <NUM> may compare the first threshold value for determining the arbitrary pixel as the first class according to an inference result with a first value. The select circuit <NUM> may determine the class of the arbitrary pixel as the first class if the first value exceeds (or is greater than or equal to) the first threshold value.

The class of each of a plurality of pixels in the image image_h may be determined by the select circuit <NUM>. Even when any two pixels are of the same class, correction effects to be applied thereto may be different according to the confidence of each of the two pixels. In this case, the mix circuit <NUM> may determine the intensity of correction effects to be applied to each of the pixels, that is, correction values, in consideration of confidence.

The mix circuit <NUM> may generate first to third correction maps tmap1, tmap2, and tmap3 corresponding to respective correction effects by taking into account the confidence map map_conf_h based on the classes of respective pixels determined by the select circuit <NUM>. The size of each of the first to third correction maps tmap1, tmap2, and tmap3 may be the same as the size of the image image_h. Pixels of the first to third correction maps tmap1, tmap2, and tmap3 may represent correction values to be applied to the respective pixels in the image image_h.

According to an embodiment, the mix circuit <NUM> may generate a first correction map tmap1 including correction values for the denoise effect, a second correction map tmap2 including correction values for the color correction effect, and a third correction map tmap3 including correction values for the sharpening effect. The mix circuit <NUM> may provide the first to third correction maps tmap1, tmap2, and tmap3 to the denoise circuit <NUM>, the color correction circuit <NUM>, and the sharpen circuit <NUM>, respectively.

Although it is shown in <FIG> that three correction maps, that is, the first to third correction maps tmap1, tmap2, and tmap3, are generated, the inventive concepts are not limited thereto, and the mix circuit <NUM> may generate as many correction maps as the number of the correcting circuits (e.g. the denoise circuit <NUM>, the color correction circuit <NUM>, and the sharpen circuit <NUM>). That is, differing numbers and types of correcting circuits may be implemented, and therefore different numbers and types of correction maps may be utilized.

<FIG> is a diagram showing an example of class information according to an example embodiment.

Referring to <FIG>, class information info_class may be included in a tuning circuit (e.g., <NUM> of <FIG>). The class information info_class may include a plurality of classes into which pixels of an image may be classified.

Classes and/or subclasses may be generated based on correction effects. In other words, pixels to which the same correction effect is to be applied may be classified into the same class, and pixels to which different correction effects are to be applied may be classified into different classes. However, the inventive concepts are not limited thereto, and classes and/or subclasses may be generated based on characteristics of pixels, e.g., color information.

According to an embodiment, one class may include at least one subclass. According to an embodiment, the class information info_class may include first to seventh classes class1 to class7, and a fourth class class4 may include first to third subclasses subclass1 to subclass3. For example, the first to seventh classes class1 to class7 may correspond to a face class, a skin class, a sky class, a detail class, an eye class, an eyebrow class, and a hair class, respectively. In this case, the detail class may include a grass subclass, a sand subclass, and a branch subclass. However, the types and numbers of classes and subclasses are not limited thereto and may be changed. Accordingly, any number of classes and subclasses may be applied, and each class may have varying numbers of subclasses or no subclasses. The classes and subclasses may be applied in the segmentation map. Each pixel of the segmentation map may include least one of a face class, a skin class, a sky class, a detail class, an eye class, an eyebrow class, and a hair class.

<FIG> is a diagram showing an example of a part of a setting value table according to an example embodiment.

Referring to <FIG> and <FIG> together, the configuration table <NUM> may include correction values according to confidence for a plurality of correction effects. Hereinafter, description will be given based on an example of one class, but the description may be similarly applied to other classes.

For example, the select circuit <NUM> of the tuning circuit <NUM> may determine an arbitrary pixel in the image image_h as the first class class1. At this time, the mix circuit <NUM> may determine a correction value according to the confidence of the arbitrary pixel by referring to the confidence map map_conf_h for the image image_h.

For example, when the confidence of the first class class1 is <NUM>, the mix circuit <NUM> may determine to apply a denoise effect by d2, a color correction effect by c2, and a sharpening effect by s2 to the arbitrary pixel. Therefore, in the first correction map tmap1 for the denoise effect, a pixel value corresponding to the arbitrary pixel may be determined as d2, in the second correction map tmap2 for the color correction effect, a pixel value corresponding to the arbitrary pixel may be determined as c2, and, in the third correction map tmap3 for the sharpening effect, a pixel value corresponding to the arbitrary pixel may be determined as s2. In other words, the mix circuit <NUM> generates the first correction map tmap1 including d2, the second correction map tmap2 including c2, and the third correction map tmap3 including s2.

According to an example embodiment of the present disclosure, because correction effects and the correction values are determined pixel-by-pixel, even in the image image_h including complex shapes, correction according to the characteristics of each pixel may be performed. Therefore, it may be helpful to improve the image quality of the image image_h including complex shapes.

<FIG> is a flowchart showing an operation of an image processing system according to an example embodiment.

Referring to <FIG>, <FIG>, and <FIG> together, an operation of receiving an image image_h may be performed (operation S110). For example, the image processing system <NUM> may obtain the image image_h from an image sensor.

An operation of generating a low-resolution image image_l based on the received image image_h may be performed (operation S120).

By using the trained neural network model <NUM>, a low-resolution segmentation map map_seg_l and a low-resolution confidence map map_conf_l for the low-resolution image image_l may be generated (operation S130). According to an embodiment, the neural network model <NUM> may be trained by using a training image and correct answer classes labeled to respective pixels of the training image as training data. At this time, the correct answer classes may correspond to correction effects to be applied to the respective pixels of the training image. According to an embodiment, classes to which the respective pixels of the low-resolution image image_l belong may be inferred by the neural network model <NUM>, and confidence of an inferred class may be calculated for each pixel. In some embodiments, the neural network model <NUM> may include organizational and processing models, such as convolutional neural networks (CNN), deconvolutional neural networks, recurrent neural networks (RNN) optionally including long short-term memory (LSTM) units and/or gated recurrent units (GRU), stacked neural networks (SNN), state-space dynamic neural networks (SSDNN), deep belief networks (DBN), generative adversarial networks (GANs), and/or restricted Boltzmann machines (RBM).

By enlarging the low-resolution segmentation map map_seg_l and the low-resolution confidence map map_conf_l, the segmentation map map_seg_h and the confidence map map_conf_h for the image image_h may be generated (operation S140).

Based on the segmentation map map_seg_h and the confidence map map_conf_h, correction effects to be applied to each pixel of the image image_h and the intensity of the correction effects, that is, a correction value, may be determined (operation S150). According to an embodiment, the first to third correction maps tmap1, tmap2, and tmap3 may be generated for the respective correction effects. The first to third correction maps tmap1, tmap2, and tmap3 may include correction values and may each have the same size as the image image_h.

An enhanced image may be generated by applying correction effects corresponding to the correction values to the respective pixels (operation S160). According to an embodiment, the denoise circuit <NUM>, the color correction circuit <NUM>, and the sharpen circuit <NUM> may apply correction effects to the respective pixels of the image image_h based on the first to third correction maps tmap1, tmap2, and tmap3. In this case, the enhanced image may correspond to the image image_h with an improved image quality. The enhanced image may be stored in a memory, transmitted to another device, displayed on a display or otherwise transferred, stored or used.

Thereafter, the enhanced image is provided to an encoder and an operation of compressing the enhanced image in various formats may be further performed.

<FIG> is a block diagram showing an image processing system according to an example embodiment. Since the image processing system <NUM> of <FIG> is similar to the image processing system <NUM> of <FIG>, descriptions identical to those already given above will be omitted. For example, the image processing circuit <NUM> may be substantially the same as the image processing circuit <NUM>. The at least one correcting circuit <NUM> may be substantially the same as the at least one correcting circuit <NUM>. The denoise circuit <NUM> may be substantially the same as the denoise circuit <NUM>. The color correction circuit <NUM> may be substantially the same as the color correction circuit <NUM>. The sharpen circuit <NUM> may be substantially the same as the sharpen circuit <NUM>. The tuning circuit <NUM> may be substantially the same as the tuning circuit <NUM>.

Unlike the image processing system <NUM> of <FIG>, in the image processing system <NUM> of <FIG>, the upscaling circuit (<NUM> of <FIG>) may be omitted. Therefore, a segmentation circuit <NUM> receives an image image_h and generate a segmentation map map_seg_h and a confidence map map_conf_h for the image image_h by using a trained neural network model <NUM>.

An image processing circuit <NUM> is referred to as a first circuit or as a first portion of the first circuit, and the segmentation circuit <NUM> is referred to as a second circuit. The image processing system <NUM> of <FIG> may further include the upscaling circuit (<NUM> of <FIG>; e.g., a third circuit or a second portion of the first circuit) as compared to the image processing system <NUM> of <FIG>.

The second circuit generates first class inference information for each pixel of the image image_h and a first confidence for the first class inference information by using the trained neural network model <NUM>. The first circuit determines a correction value for each pixel of the image image_h based on the first class inference information and the first confidence and apply correction effects corresponding to the correction values to the respective pixel of the image image_h, thereby generating an enhanced image. Here, when a third circuit is further included as in the image processing system <NUM> of <FIG>, the second circuit may receive a low-resolution image and generate second class inference information and a second confidence for each pixel of the low-resolution image by using the neural network model <NUM>. The third circuit may generate the first class inference information and the first confidence based on the second class inference information and the second confidence. The first circuit may be configured to generate the first class inference information and the first confidence via the third circuit.

The segmentation circuit <NUM> provides the segmentation map map_seg_h and the confidence map map_conf_h to a tuning circuit <NUM> of the image processing circuit <NUM>. The image processing circuit <NUM> performs pixel-by-pixel correction operations on the image image_h based on the segmentation map map_seg_h and the confidence map map_conf_h.

Referring to <FIG> and <FIG> together, an operation of receiving an image image_h may be performed (operation S210). For example, the image processing system <NUM> may obtain the image image_h from an image sensor. The image processing circuit <NUM> and the segmentation circuit <NUM> may receive the image image _h.

The segmentation map map_seg_h and the confidence map map_conf_h for the image image_h are generated by using the trained neural network model <NUM> (operation S220). The segmentation circuit <NUM> infers a class to which each pixel of the image image_h belongs by using the neural network model <NUM> and calculates the confidence of the class inferred for each pixel.

Based on the segmentation map map_seg_h and the confidence map map_conf_h, correction effects to be applied to each pixel of the image image_h and the intensity of the correction effects, that is, a correction value, is determined (operation S230). An enhanced image is generated by applying correction effects corresponding to the correction values to the respective pixels (operation S240).

Thereafter, the enhanced image is provided to an encoder and an operation of compressing the enhanced image in various formats may be further performed by the encoder.

<FIG> is a block diagram showing an image processing system according to an example embodiment. Since the image processing system <NUM> of <FIG> is similar to the image processing system <NUM> of <FIG>, descriptions identical to those already given above will be omitted. For example, the image processing circuit <NUM> may be substantially the same as the image processing circuit <NUM>. The denoise circuit <NUM> may be substantially the same as the denoise circuit <NUM>. The color correction circuit <NUM> may be substantially the same as the color correction circuit <NUM>. The sharpen circuit <NUM> may be substantially the same as the sharpen circuit <NUM>. The tuning circuit <NUM> may be substantially the same as the tuning circuit <NUM>.

Unlike the image processing system <NUM> of <FIG>, the image processing system <NUM> of <FIG> may further include a scene detection circuit <NUM>. The scene detection circuit <NUM> according to an example embodiment may determine a scene representing the image image_h. For example, the scene representing the image image_h may be determined as food, a landscape, persons, etc. The scene detection circuit <NUM> (otherwise known as a color based segmentation circuit) may use various algorithms to determine the scene representing the image image_h. For example, a feature point extraction algorithm, a face recognition algorithm, a color-based segmentation algorithm, etc. may be used.

The scene detection circuit <NUM> may generate correction effects and correction values according to a determined scene. Information including the correction effects and correction values generated by the scene detection circuit <NUM> may be referred to as sub-correction information. The scene detection circuit <NUM> may provide sub-correction information (map_seg_h1) to the image processing circuit <NUM>. The image processing circuit <NUM> may apply correction effects to the image image_h based on the sub-correction information. In addition, the scene detection circuit <NUM> may be implemented by hardware, software (or firmware), or a combination of hardware and software for carrying out the technical spirit. In this case, the scene detection circuit <NUM> may be included in an ISP.

As described above with reference to <FIG>, a segmentation map map_seg_h2 and a confidence map map_conf_h may be generated through a segmentation circuit <NUM> and an upscaling circuit <NUM>. The confidence map map_conf_h may include a probability and/or a confidence for each pixel of the segmentation map map_seg_h2.

The image processing circuit <NUM> may receive the sub-correction information map_seg_h1 from the scene detection circuit <NUM> and may receive the segmentation map map_seg_h2 and the confidence map map_conf_h from the upscaling circuit <NUM>.

The image processing circuit <NUM> may correct the image image_h based on the sub-correction information map_seg_h1 and correct the image image_h based on the segmentation map map_seg_h2 and the confidence map map_conf_h. The two correction operations may be performed in parallel or sequentially. An enhanced image is produced as a result of the correction operations.

A tuning circuit <NUM> determines a class for each pixel of the image image_h based on the segmentation map map_seg_h and the confidence map map_conf_h and determines correction effects and correction values to be applied to each pixel. The tuning circuit <NUM> generates a correction map including correction effects and correction values.

At least one of correcting circuits <NUM>, <NUM>, and <NUM> may correct the image image_h based on the sub-correction information. Next, the image image_h may be corrected based on the correction map generated by the tuning circuit <NUM>. For example, the at least one of the correcting circuits <NUM>, <NUM>, and <NUM> may correct the entire image image_h by using the sub-correction information and correct each pixel of the image image_h based on the correction map.

Referring to <FIG> and <FIG> together, an operation of receiving an image image_h may be performed (operation S310). A low-resolution image image_l may be generated based on the image image_h (operation S315). By using the trained neural network model <NUM>, a low-resolution segmentation map map_seg_l and a low-resolution confidence map map_conf_l for the low-resolution image image_l may be generated (operation S320). By enlarging the low-resolution segmentation map map_seg_l and the low-resolution confidence map map_conf_l, the segmentation map map_seg_h2 and the confidence map map_conf_h for the image image_h may be generated (operation S325). Based on the segmentation map map_seg_h2 and the confidence map map_conf_h, correction effects to be applied to each pixel of the image image_h and correction values may be determined (operation S330). Correction effects corresponding to the correction values may be applied to each pixel (operation S335).

A scene representing the image image_h may be determined (operation S345). Based on a determined scene, sub-correction information map_seg_h1 for the image image_h may be generated (operation S350). Based on the sub-correction information, correction effects may be applied to the entire image image_h (operation S355).

Operations S345 to S355 may be performed in parallel or sequentially with operations S315 to S335.

<FIG> is a block diagram showing a system according to an example embodiment.

Referring to <FIG>, a system <NUM> may be implemented as a handheld device like a mobile phone, a smartphone, a tablet computer, a personal digital assistant (PDA), an enterprise digital assistant (EDA), a digital still camera, a digital video camera, a portable multimedia player (PMP), a personal navigation device (PDN), a handheld game console, or an e-book.

The system <NUM> may include a system-on-chip (SoC) <NUM> and a memory device <NUM>. The SoC <NUM> may include a central processing unit (CPU) <NUM>, a graphics processing unit (GPU) <NUM>, a neural processing unit (NPU) <NUM>, an image signal processor (ISP) <NUM>, a memory interface (MIF) <NUM>, a clock management unit (CMU) <NUM>, and a power management unit (PMU) <NUM>. The CPU <NUM>, the GPU <NUM>, the NPU <NUM>, and the ISP <NUM> may be referred to as master (or primary) IP devices, and the MIF <NUM> may be referred to as a slave (or secondary) IP device.

At least one of the CPU <NUM>, the GPU <NUM>, the NPU <NUM>, and the ISP <NUM> may include the image processing systems <NUM>, <NUM>, or <NUM> described above with reference to <FIG>. The components of the image processing system <NUM>, <NUM>, or <NUM> may be implemented in the same IP device or at least one component may be implemented in another IP device. According to an embodiment, the GPU <NUM> may include an upscaling circuit <NUM>, the NPU <NUM> may include a segmentation circuit <NUM>, and the ISP <NUM> may include an image processing circuit <NUM>.

For example, the ISP <NUM> may generate an image to be corrected, generate a low-resolution image for the image to be corrected, and provide the low-resolution image to the NPU <NUM>. The segmentation circuit <NUM> of the NPU <NUM> may generate a low-resolution segmentation map and a low-resolution confidence map and provide the low-resolution segmentation map and the low-resolution confidence map to the GPU <NUM>. The upscaling circuit <NUM> of the GPU <NUM> may generate a segmentation map and a confidence map and may provide the segmentation map and the confidence map back to the ISP <NUM>. The image processing circuit <NUM> of the ISP <NUM> may generate an enhanced image by performing pixel-by-pixel corrections on the image based on the segmentation map and the confidence map.

The CPU <NUM> may process or execute instructions and/or data stored in the memory device <NUM> in response to a clock signal generated by the CMU <NUM>.

The GPU <NUM> may obtain image data stored in the memory device <NUM> in response to the clock signal generated by the CMU <NUM>. The GPU <NUM> may generate data for an image to be output through a display device (not shown) from image data provided by the MIF <NUM> or may encode the image data.

The NPU <NUM> may refer to any device for executing a machine learning model. The NPU <NUM> may be a hardware block designed to execute a machine learning model. The machine learning model may be a model based on an artificial neural network, a decision tree, a support vector machine, a regression analysis, a Bayesian network, a genetic algorithm, etc. As a nonlimiting example, an artificial neural network may include a CNN, an R-CNN, an RPN, an RNN, an S-DNN, an S-SDNN, a deconvolution network, a DBN, an RBM, a fully convolutional network, an LSTM network, and a classification network. The machine learning model may be trained on an external system and input or stored in the system <NUM>. Alternatively, or in addition, the machine learning model may be trained at least in part by the system <NUM>.

The ISP <NUM> may perform a signal processing operation on raw data received from an image sensor (not shown) located outside the SoC <NUM> and generate digital data having improved image quality. The image to be enhanced (e.g. the received raw data) may be received from an image sensor, storage (e.g. memory device <NUM>) or from an external device or network.

The MIF <NUM> may provide an interface for the memory device <NUM> located outside the SoC <NUM>. The memory device <NUM> may be dynamic random access memory (DRAM), phase-change random access memory (PRAM), resistive random access memory (ReRAM), or flash memory.

The CMU <NUM> may generate a clock signal and provide the clock signal to components of the SoC <NUM>. The CMU <NUM> may include a clock generator like a phase locked loop (PLL), a delayed locked loop (DLL), and a crystal. The PMU <NUM> may convert external power into internal power and supply power to the components of the SoC <NUM> from the internal power.

The SoC <NUM> may further include a volatile memory. The volatile memory may be implemented as DRAM, SRAM, etc. The volatile memory may store various programs and data for the operations of the image processing circuit <NUM>, the segmentation circuit <NUM>, and the upscaling circuit <NUM> and store data generated by the image processing circuit <NUM>, the segmentation circuit <NUM>, and the upscaling circuit <NUM>. According to an embodiment, the volatile memory may store a low-resolution segmentation map and a low-resolution confidence map or may store a segmentation map and a confidence map.

According to an example embodiment, each of at least one correcting circuit of the image processing circuit <NUM> may access the volatile memory through direct memory access (DMA). To this end, the SoC <NUM> may further include an access device like a DMA controller, a memory DMA (MDMA), a peripheral DMA (PDMA), a remote DMA (RDMA), a smart DMA (SDMA), etc..

<FIG> is a block diagram of an electronic device according to an example embodiment.

Referring to <FIG>, an electronic device <NUM> according to an example embodiment may include an image sensor <NUM>, an ISP <NUM>, a display device <NUM>, an application processor (AP) <NUM>, a working memory <NUM>, a storage <NUM>, a user interface <NUM>, and a wireless transceiver <NUM>, wherein the ISP <NUM> may be implemented as a separate integrated circuit from the AP <NUM>.

According to an example embodiment, the image processing system <NUM>, <NUM>, or <NUM> described above with reference to <FIG> may be implemented on the ISP <NUM> and/or the AP <NUM>. For example, an image processing circuit may be implemented on the ISP <NUM>, and the remaining components may be implemented on the AP <NUM>.

The image sensor <NUM> may generate image data, such as raw image data, based on a received optical signal, and provide binary data to the ISP <NUM>. The AP <NUM> controls the overall operation of the electronic device <NUM> and may be implemented as a system-on-chip (SoC) that drives an application program, an operating system, etc. The AP <NUM> may control the operation of the ISP <NUM> and provide converted image data generated by the ISP <NUM> to the display device <NUM> or store the converted image data in the storage <NUM>.

The working memory <NUM> may store programs and/or data processed or executed by the AP <NUM>. The storage <NUM> may be implemented with a non-volatile memory device like a NAND flash or a resistive memory. For example, the storage <NUM> may be provided as a memory card (a multimedia card (MMC), an embedded multimedia card (eMMC), a secure digital (SD) card, a micro SD card, etc.). The storage <NUM> may store data and/or programs regarding an execution algorithm that controls an image processing operation of the ISP <NUM>, and, when an image processing operation is performed, the data and/or the programs may be loaded to the working memory <NUM>.

The user interface <NUM> may be implemented with various devices capable of receiving user inputs, e.g., a keyboard, a curtain key panel, a touch panel, a fingerprint sensor, a microphone, etc. The user interface <NUM> may receive a user input and provide a signal corresponding to the received user input to the AP <NUM>. The wireless transceiver <NUM> may include a modem <NUM>, a transceiver <NUM>, and an antenna <NUM>.

Additionally, the image processing system <NUM> and/or the components included therein may include and/or be included in processing circuitry such as hardware including logic circuits; a hardware/software combination such as a processor executing software; or a combination thereof. For example, the processing circuity may include, but is not limited to, a central processing unit (CPU), a memory controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), and programmable logic unit, a microprocessor, application-specific integrated circuit (ASIC), etc..

Claim 1:
An image processing system comprising:
a segmentation circuit (<NUM>) configured to generate a segmentation map comprising pixel-by-pixel class inference information of a first image and a confidence map comprising confidence of the class inference information using a trained neural network model (<NUM>), wherein the neural network model (<NUM>) has been trained by using an arbitrary pixel and a correct answer class labeled in correspondence to the arbitrary pixel; and
an image processing circuit comprising:
a tuning circuit (<NUM>) configured to receive the segmentation map and the confidence map, determine classes of respective pixels of the first image, correction effects for each pixel of the first image, and correction values indicating intensity of the correction effects based on the segmentation map and the confidence map, and generate a correction map based on the classes and the correction values of the respective pixels; and
at least one correcting circuit (<NUM>) configured to generate an enhanced image by applying correction effects according to the correction values to the respective pixels of the first image based on the correction map.