Patent Description:
White balance ("WB") refers to low-level computer vision tasks applied to all camera images. WB is performed to represent scene objects with the same colors even when the scene objects are photographed under different lighting conditions. Conceptually, WB is performed to normalize a lighting effect to be added to a captured image of a scene so that it may appear that images of all objects are captured under ideal "white light". WB is one of initial color manipulation operations applied to an unprocessed raw-RGB image from a sensor by a camera's onboard integrated signal processor ("ISP"). After WB is performed, additional color rendering operations are applied by the ISP to additionally process the raw-RGB image by final standard RGB (sRGB) encoding.

WB aims to normalize the lighting effect to be added to a scene but in some cases, the ISP incorporates aesthetic considerations into color rendering, based on photographic preferences. The photographic preferences are not always based on the white-light assumption and may vary according to various factors such as cultural preference and scene content.

Most digital cameras provide an option for controlling a WB setting during capturing of an image (or shooting). However, once the WB setting is selected and an image is completely processed by final sRGB by the ISP, it is difficult to perform WB editing without access to an unprocessed original raw-RGB image. This problem worsens when the WB setting is incorrect, thereby causing a strong color cast to occur in a final sRGB image.

A function of editing WB in a sRGB image is useful not only in photography tasks but also in computer vision applications such as object recognition, scene understanding and color augmentation.

Recent studies show that an image captured with an incorrect WB setting produces an effect similar to an untargeted adversarial attack on deep neural network ("DNN") models.

To understand difficulties with WB editing of sRGB images, it is useful to review a method of performing a WB by a camera. WB includes two operations performed in tandem by the ISP: (<NUM>) estimating a response of a camera sensor to scene illumination in the form of raw-RGB vector; and (<NUM>) dividing R, G and B color channels of colors of a raw-RGB image by a corresponding channel response of the raw-RGB vector.

A first operation of estimating an illumination vector includes an auto white balance ("AWB") process of a camera. In addition to the AWB, a user may manually select one WB presetting from among WB presettings, for which a raw-RGB vector for each camera is determined by a camera manufacturer, when most cameras are used. The presettings correspond to general scene illumination (e.g., daylight, shade, and incandescent light).

When a raw-RGB vector of scene illumination is defined, linear scaling is individually and independently applied to each color channel so as to normalize illumination. Such scaling work is performed using a 3x3 diagonal matrix. Actually, a white-balance (or white-balanced) raw-RGB image is additionally processed by a large number of ISP operations for each non-linear camera for rendering a final image in an output reference color space, i.e., a sRGB color space. Such nonlinear work makes it difficult to use traditional diagonal correction for correcting an image rendered with a strong color cast due to camera WB errors.

To perform correct post-capture WB editing, a rendered sRGB value should be appropriately inverted to obtain a corresponding unprocessed raw-RGB value and be thereafter rendered again (or re-rendered). This can be achieved by accurate radiometric calibration methods of calculating metadata for color de-rendering. A recent study has proposed a method of directly correcting a sRGB image captured with an incorrect WB setting. This study has introduced exemplar-based framework using a large-scale data set including more than <NUM>,<NUM> sRGB images rendered by software camera pipeline due to the incorrect WB setting. Each sRGB image includes a corresponding sRGB image rendered with a correct WB setting. When an input image is given, a data set is searched for to detect similar images by using a k-nearest neighbor ("KNN") strategy and a mapping function for a corresponding correct WB image is calculated. This study shows that aforementioned color mapping calculated from exemplars is effective in correcting the input image. Afterward, in this study, a KNN idea was expanded to map a correct WB image to erroneous views for image augmentation for training a deep neural network.

<CIT> describes an image capture device for AWB. The image capture device includes a camera, a primary sensor for sensing environmental conditions of the image capture device and a plurality of supplementary sensors for sensing the environmental conditions of the image capture device, an AWB unit for performing an AWB operation according to the environmental conditions, and a controller for controlling the camera, the primary sensor, the plurality of supplementary sensors and the AWB unit.

<CIT> describes an image white balance processing method, device, storage medium, and mobile terminal, which obtains an original image to be processed, inputs the original image into a pre-trained white balance coefficient matrix determination model or a white balance processing model, and determines a target image corresponding to the original image according to the white balance coefficient matrix determination model or the output result of the white balance processing model.

Methods and apparatuses according to embodiments of the disclosure relate to deep white-balancing editing.

There is provided mobile device according to the claims.

The above-described aspects, other aspects and features of embodiments of the disclosure will become more apparent from the following description taken with the accompanying drawings.

Embodiments of the disclosure provide a method and an apparatus for deep white-balancing editing.

Specifically, a deep learning approach for realistic editing of white balance (WB) of a sRGB image will be introduced herein. An image from a sensor, which is rendered by sRGB color space encoding by an integrated signal processor (ISP), is captured by a camera. Rendering performed by the ISP starts with a WB process used to remove a color cast in scene illumination. Thereafter, the ISP applies a series of nonlinear color manipulation operations to improve visual quality of a final sRGB image. A recent study showed that a sRGB image rendered with incorrect WB cannot be easily corrected by nonlinear rendering of an ISP. This study proposed a k-nearest neighbor (KNN) solution based on tens of thousands of image pairs. However, an ISP may erroneously correct WB of an image at an initial stage of a process, and this error may propagate in the entire process even in the KNN solution.

In the above-described embodiments of the disclosure, this problem may be fixed using a DNN architecture trained by an end-to-end method to learn exact WB. The DNN architecture maps an input image to two WB settings corresponding to indoor illumination and outdoor illumination. The above-described embodiments of the disclosure are more exact than a KNN approach method in terms of correcting an incorrect WB setting and provide a user with freedom of editing WB of a sRGB image to other illumination settings.

The above-described embodiments of the disclosure provide a deep learning framework permitting realistic post-capture WB editing of a sRGB image. The framework includes a single encoder network combined with three decoder networks targeted to the following WB settings: (<NUM>) a "correct" AWB setting; (<NUM>) an indoor WB setting; and (<NUM>) an outdoor WB setting. A first decoder may edit a sRGB image with incorrectly controlled WB to have correct WB. The framework is useful for post-capture WB correction. Additional indoor and outdoor decoders provide users with a function of blending two outputs of the decoders to create a wide range of WB appearances. This function supports photo editing work for controlling aesthetic WB properties of an image. The framework is easy to be generalized with respect to images other than training data and allows a latest result to be obtained with respect to the work described above.

The disclosure permits a variety of modifications and a plurality of examples.

A detailed description of the above-described embodiments of the disclosure will be omitted herein when it is determined that the essence of the disclosure would be obscure due to a detailed description of relevant technologies. Numbers (e.g., first, second, etc.) used herein are identifier codes for distinguishing one element from another.

It will be understood by those of ordinary skill in the art that when elements are referred to as being "connected" or "coupled" to each other, the elements are directly connected or coupled to each other or connected to each other with another element therebetween unless stated otherwise.

Each element described herein may additionally perform part of or all of functions of other elements, as well as a function thereof, or a major function of each element may be performed entirely by another element.

In the present specification, the term "image" may be understood to mean a still image, a moving picture including a series of still images (or frames), or a video. The image may be a two-dimensional (2D) or three-dimensional (3D) image.

In the present specification, the term "neural network" refers to a representative example of an artificial intelligence (AI) model but the above-described embodiments are not limited to an Al model using an algorithm.

In the present specification, the term "parameter" or "neural network parameter" may be understood to mean a value used during operation of each layer of a neural network and may include, for example, a weight used when an input value is applied to an operation expression. Here, the parameter may be expressed in a matrix form. The parameter is a value set as a result of training and may updated through training data when needed.

<FIG> is a block diagram of an apparatus <NUM> for deep white-balancing editing according to embodiments of the disclosure.

Referring to <FIG>, the apparatus <NUM> includes an encoder <NUM>, a first WB decoder <NUM>, and a second WB decoder <NUM>.

The apparatus <NUM> may be embodied as an electronic device (e.g., a mobile device) and/or a server.

The encoder <NUM> is configured to obtain (or receive) an input image (e.g., a sRGB image) with an original WB corrected by image signal processing by an ISP of a camera. The encoder <NUM> is also configured to obtain and transmit an intermediate representation of the obtained input image, and the intermediate representation has the original WB that is not corrected by image signal processing. The encoder <NUM> includes, for example, a neural network such as a convolutional network (CNN) or a deep neural network (DNN) to be described with reference to <FIG> below.

The first WB decoder <NUM> is configured to obtain and transmit a first output image with first WB different from the original WB, based on the obtained intermediate representation. The first WB decoder <NUM> may include, for example, a neural network such as a CNN or a DNN, as described with reference to <FIG> below. For example, the first WB may be auto white balance (AWB).

The second WB decoder <NUM> is configured to obtain and transmit a second output image with second WB different from the original WB and the first WB, based on the obtained intermediate representation. The second WB decoder <NUM> may include, for example, a neural network such as a CNN or a DNN, as described with reference to <FIG> below. The second WB may be, for example, shade WB correlated to <NUM> (K) or incandescent WB correlated to <NUM> color temperatures.

Specifically, when an input sRGB image IWB(in) rendered with WB setting WB(in) by an ISP of an unknown camera is considered, colors of the input sRGB image IWB(in) are edited so that the input sRGB image IWB(in) may appear to be re-rendered with target WB setting WB(t).

When an unprocessed original raw-RGB image is available, the work described above may be accurately performed. When unprocessed original raw-RGB values are reconstructed, the WB setting WB(in) may be changed to the target WB setting WB(t) and thereafter the sRGB image may be re-rendered by a software-based ISP to restore to an original sRGB color space. This process may be described by the following equation: <MAT>.

The process aims to model a function of G·F to produce the final sRGB image IWB(t). First, how the functions G and F cooperate with each other to produce the final sRGB image IWB(t) is analyzed. From Equation <NUM> above, the function G receives an intermediate representation, renders the intermediate representation by sRGB color space encoding to have a target WB setting WB(t), and the function F transforms an input sRGB image IWB(in) into an intermediate representation (i.e., a raw-RGB image D with a captured WB setting).

Due to nonlinearity applied to a rendering chain of an ISP, the function G may be regarded as a hybrid function including a set of sub-functions, and each of the sub-functions renders the intermediate representation with a certain WB setting.

The process aims to produce the final sRGB image IWB(t) with the target WB setting WB(t) rather than reconstructing/re-rendering an original raw-RGB value. Therefore, the function of G·F may be modeled with a encoder/decoder scheme. Decoders g<NUM> and g<NUM>,. (i.e., the first and second WB decoders <NUM> and <NUM>) produce final SRGB images with different WB settings, whereas an encoder f (i.e., the encoder <NUM> of <FIG>) transmits the input sRGB image IWB(in) as a latent representation. Similar to Equation (<NUM>), the framework may be formulated as follows: <MAT>.

f:IWB(in)→Z,gt:Z→ ÎWB(t), and Z may denote an intermediate representation (i.e., a latent representation) of an input original sRGB image IWB(in).

As in a case of Equation <NUM> above, the process aims to change a function gt into a new function gy targeted to another WB setting WBy, so that the functions f and gt may be independent of each other without requiring a change of the function f.

In embodiments of the disclosure, the following three different WB settings are targeted:.

Therefore, three different types of decoders gA, gT and gS for producing output images corresponding to the AWB, the incandescent WB, and the shade WB are generated.

The incandescent WB and the shade WB are selected according to color properties. This can be understood when considering illumination in terms of correlation color temperature. For example, the incandescent and shade WB settings are related to color temperatures <NUM> and <NUM>, respectively. A wide range of illumination color temperatures is considered as a range of pleasing illuminations. In addition, a wide color temperature range between the incandescent WB and the shade WB permits an approximate value of an image within color temperatures within this range through interpolation. The details of such an interpolation process will be described in <FIG> below. In an AWB mode, a correlated color temperature varies according to an illumination condition of an input image and thus is not fixed.

Referring back to <FIG>, an overview of the DNN architecture is illustrated. A U-Net architecture may be used together with a multi-scale skip connection <NUM> between the encoder <NUM> and the first and second WB decoders <NUM> and <NUM>. A framework includes two main units. A first main unit is the encoder <NUM>, which is a <NUM>-level encoder unit for extracting a multi-scale latent representation (intermediate representation) of an input image. A second main unit includes at least two <NUM>-level decoders (e.g., the first and second WB decoders <NUM> and <NUM>). The main units have different bottle necks and different upsampling components. Each convolutional layer has <NUM> channels at a first level of the encoder <NUM> and a last layer of each of the first and second WB decoders <NUM> and <NUM>. The number of channels of each convolutional layer is double the number of channels at each subsequent or previous level. (For example, the third level of the encoder <NUM> has <NUM> channels for each convolution layer).

<FIG> is a block diagram of the encoder <NUM> of the apparatus <NUM> for deep white-balancing editing of <FIG>.

Referring to <FIG>, the encoder <NUM> may include a first level <NUM>, a second level <NUM>, and a third level <NUM>. An input image may be transmitted through each layer of the first level <NUM>, the second level <NUM>, and the third level <NUM> to obtain an intermediate representation output from the first WB decoder <NUM> and/or the second WB decoder <NUM>.

The first level <NUM> may include 3x3 convolutional layers 205a and 205b with a stride <NUM> and a padding <NUM>, and rectified linear unit (ReLU) layers 205c and 205d. The ReLU layer 205c may be located between the convolutional layers 205a and 205b, and the convolutional layer 205b may be located between the ReLU layers 205c and 205d. For example, the convolutional layers 205a and 205b and the ReLU layers 205c and 205d may each have a size of 128x128x24.

The second level <NUM> may include 2x2 max-pooling layers 210a and 210b with a stride <NUM>, 3x3 convolutional layers 210c and 210d with a stride <NUM> and a padding <NUM>, and ReLU layers 210e and 210f. The ReLU layer 210e may be located between the convolutional layers 210c and 210d, the convolutional layer 210d may be located between the ReLU layers 210e and 210f, and the convolutional layers 210c and 210d and the ReLU layers 210e and 210f may be located between the max-pooling layers 210a and 210b. For example, the convolutional layers 210c and 210d and the ReLU layers 210e and 210f may each have a size of 64x64x48, the max-pooling layer 210a may have a size of 64x64x24, and the max-pooling layer 210b may have a size of 32x32x48.

The third level <NUM> may include 3x3 convolutional layers 215a and 215b with a stride <NUM> and a padding <NUM>, ReLU layers 215c and 215d, and a 2x2 max-pooling layer 215e with a stride <NUM>. The ReLU layer 215c may be located between the convolutional layers 215a and 215b, the convolutional layer 215b may be located between the ReLU layers 215c and 215d, and the max-pooling layer 215e may be located behind the convolutional layers 215a and 215b and the ReLU layers 215c and 215d. For example, the convolutional layers 215a and 215b and the ReLU layers 215c and 215d may each have a size of 16x16x192, and the max-pooling layer 215e have a size of 8x8x192.

<FIG> is a block diagram of the first WB decoder <NUM> of the apparatus <NUM> for deep white-balancing editing of <FIG>.

Referring to <FIG>, the first Wb decoder <NUM> includes a first level <NUM>, a second level <NUM>, a third level <NUM>, and a fourth level <NUM>. An intermediate representation may be transmitted through each layer of the first level <NUM>, the second level <NUM>, the third level <NUM>, and the fourth level <NUM> to obtain a first output image.

The first level <NUM> may include ReLU layers 305a and 305b, and 3x3 convolutional layers 305c and 305d with a stride <NUM> and a padding <NUM>. The convolutional layer 305c may be located between the ReLU layers 305a and 305b, and the ReLU layer 305b may be located between the convolutional layers 305c and 305d. For example, the ReLU layers 305a and 305b and the convolutional layers 305c and 305d may each have a size of 8x8x384.

The second level <NUM> may include a 2x2 upsampling layer 310a, 3x3 convolutional layers 310b, 310c and 310d with a stride <NUM> and a padding <NUM>, a depth concatenation layer 310e, and ReLU layers 310f and <NUM>. Layers of the second level <NUM> may be the upsampling layer 310a, the convolutional layer 310b, the depth concatenation layer 310e, the ReLU layer 310f, the convolutional layer 310c, the ReLU layer <NUM>, and the convolutional layer 310d that are arranged in a direction from an input to an output. For example, the upsampling layer 310a, the convolutional layers 310b, 310c, and 310d, and the ReLU layers 310f and <NUM> may each have a size of 16x16x192, and the depth concatenation layer 310e has a size of 16x16x384.

The third level <NUM> may include a 2x2 upsampling layer 315a, 3x3 convolutional layers 315b, 315c and 315d with a stride <NUM> and a padding <NUM>, a depth concatenation layer 315e, and ReLU layers 315f and <NUM>. Layers of the third level <NUM> may be the upsampling layer 315a, the convolutional layer 315b, the depth concatenation layer 315e, the ReLU layer 315f, the convolutional layer 315c, the ReLU layer <NUM>, and the convolutional layer 315d that are arranged in a direction from an input to an output. For example, the upsampling layer 315a, the convolutional layers 315b, 315c, and 315d, and the ReLU layers 315f and <NUM> may each have a size of 64x64x48, and the depth concatenation layer 315e has a size of 64x64x96.

The fourth level <NUM> may include a 2x2 upsampling layer 320a, 3x3 convolutional layers 320b, 320c and 320d with a stride <NUM> and a padding <NUM>, a depth concatenation layer 320e, ReLU layers 320f and <NUM>, and a 1x1 convolutional layer <NUM> with a stride <NUM> and a padding <NUM>. Layers of the fourth level <NUM> may be the upsampling layer 320a, the convolutional layer 320b, the depth concatenation layer 320e, the ReLU layer 320f, the convolutional layer 320c, the ReLU layer <NUM>, the convolutional layer 320d, and the convolutional layer <NUM> that are arranged in a direction from an input to an output. For example, the upsampling layer 320a, the convolutional layers 320b, 320c, and 320d, and the ReLU layers 320f and <NUM> may each have a size of 128x128x24, the depth concatenation layer 320e may have a size of 128x128x48, and the convolutional layer <NUM> may have a size of 128x128x3.

<FIG> is a block diagram of the second WB decoder <NUM> of the apparatus <NUM> for deep white-balancing editing of <FIG>.

Referring to <FIG>, the second WB decoder <NUM> includes a first level <NUM>, a second level <NUM>, a third level <NUM>, and a fourth level <NUM>. An intermediate representation may be transmitted through each layer of the first level <NUM>, the second level <NUM>, the third level <NUM>, and the fourth level <NUM> to obtain a second output image.

The first level <NUM> may include ReLU layers 405a and 405b and 3x3 convolutional layers 405c and 405d with a stride <NUM> and a padding <NUM>. The convolutional layer 405c may be located between the ReLU layers 405a and 405b, and the ReLU layer 405b may be located between the convolutional layers 405c and 405d. For example, the ReLU layers 405a and 405b and the convolutional layers 405c and 405d may each have a size of 8x8x384.

The second level <NUM> may include a 2x2 upsampling layer 410a, 3x3 convolutional layers 410b, 410c and 410d with a stride <NUM> and a padding <NUM>, a depth concatenation layer 410e, and ReLU layers 410f and <NUM>. Layers of the second level <NUM> may be the upsampling layer 410a, the convolutional layer 410b, the depth concatenation layer 410e, the ReLU layer 410f, the convolutional layer 410c, the ReLU layer <NUM>, and the convolutional layer 410d that are arranged in a direction from an input to an output. For example, the upsampling layer 410a, the convolutional layers 410b, 410c, and 410d, and the ReLU layers 410f and <NUM> may each have a size of 16x16x192, and the depth concatenation layer 410e has a size of 16x16x384.

The third level <NUM> may include a 2x2 upsampling layer 415a, 3x3 convolutional layers 415b, 415c and 415d with a stride <NUM> and a padding <NUM>, a depth concatenation layer 415e, and ReLU layers 415f and <NUM>. Layers of the third level <NUM> may be the upsampling layer 415a, the convolutional layer 415b, the depth concatenation layer 415e, the ReLU layer 415f, the convolutional layer 415c, the ReLU layer <NUM>, and the convolutional layer 415d that are arranged in a direction from an input to an output. For example, the upsampling layer 415a, the convolutional layers 415b, 415c, and 415d, and the ReLU layers 415f and <NUM> may each have a size of 64x64x48, and the depth concatenation layer 415e has a size of 64x64x96.

The fourth level <NUM> may include a 2x2 upsampling layer 420a, 3x3 convolutional layers 420b, 420c and 420d with a stride <NUM> and a padding <NUM>, a depth concatenation layer 420e, ReLU layers 420f and <NUM>, and a 1x1 convolutional layer <NUM> with a stride <NUM> and a padding <NUM>. Layers of the fourth level <NUM> may be the upsampling layer 420a, the convolutional layer 420b, the depth concatenation layer 420e, the ReLU layer 420f, the convolutional layer 420c, the ReLU layer <NUM>, the convolutional layer 420d, and the convolutional layer <NUM> that are arranged in a direction from an input to an output. For example, the upsampling layer 420a, the convolutional layers 420b, 420c, and 420d, and the ReLU layers 420f and <NUM> may each have a size of 128x128x24, the depth concatenation layer 420e may have a size of 128x128x48, and the convolutional layer <NUM> may have a size of 128x128x3.

Referring to <FIG>, a skip connection 120a may connect the first level <NUM> of the encoder <NUM> to the fourth level <NUM> of the first WB decoder <NUM> and/or the fourth level <NUM> of the second WB decoder <NUM>. A skip connection 120b may connect the second level <NUM> of the encoder <NUM> to the third level <NUM> of the first WB decoder <NUM> and/or the third level <NUM> of the second WB decoder <NUM>. A skip connection 120c may connect the third level <NUM> of the encoder <NUM> to the second level <NUM> of the first WB decoder <NUM> and/or the second level <NUM> of the second WB decoder <NUM>.

<FIG> is a flowchart of a deep white-balancing editing method <NUM> according to embodiments of the disclosure.

The method <NUM> may be performed by at least one processor using the apparatus <NUM> for deep white-balancing editing of <FIG>.

Referring to <FIG>, in operation <NUM>, the method <NUM> includes obtaining an input image with original WB corrected by image signal processing.

In operation <NUM>, the method <NUM> includes obtaining an intermediate representation of the obtained input image by using a first neural network, in which the intermediate representation has the original WB that is not corrected by image signal processing.

In operation <NUM>, the method <NUM> includes obtaining a first output image with first WB different from the original WB by using a second neural network, based on the obtained intermediate representation.

In operation <NUM>, the method <NUM> includes obtaining a second output image with second WB different from the original WB and the first WB by using a third neural network, based on the obtained intermediate representation.

The method <NUM> may further include displaying a slider for selecting one of a plurality of WBs, and displaying a resultant image with a WB selected from among the plurality of WBs by using the displayed slider and the obtained first output image, the obtained second output image, a blended image of the obtained first and second output images, or a combination thereof, based on a user input for selecting one of the plurality of WBs.

The at least one processor, the first neural network, and the second neural network may be implemented in a server, and the method <NUM> may further include receiving an input image for which an original WB is corrected from an ISP of an electronic device and transmitting the obtained first output image by image signal processing by the ISP.

<FIG> is a flowchart of a training (or learning) method <NUM> for deep white-balancing editing, according to embodiments of the disclosure.

The method <NUM> may be performed by at least one processor that trains the apparatus <NUM> for deep white-balancing editing of <FIG>.

Referring to <FIG>, in operation <NUM>, the method <NUM> includes obtaining a rendered WB data set. The rendered WB data set is used to train and verify the apparatus <NUM> for deep white-balancing editing. This data set may include about <NUM>,<NUM> sRGB images rendered with various types of camera models and various WB settings, including a shade setting and an incandescent setting. For each image, there is a corresponding ground-truth image rendered with a correct WB setting (i.e., considered as a correct AWB result). This data set includes two subsets, i.e., a training set (set <NUM>) and a test set (set <NUM>). Three-fold cross-validation using the set <NUM> is performed to train and verify the apparatus100. In particular, <NUM>,<NUM> training images are randomly selected from among two folds, and <NUM>,<NUM> verification images are randomly selected from among remaining folds. For each training image, three ground-truth images are rendered together with (i) correct WB (represented by AWB), (ii) shade WB, and (iii) incandescent WB. Three exemplars are trained while changing training and verification folds.

In operation <NUM>, the method <NUM> includes augmenting the obtained rendered WB data set. In detail, additional <NUM>,<NUM> raw-RGB images of the same scene included in the rendered WB data set are rendered but a training image is augmented using a certain color temperature. In each epoch, four 128x128 patches are randomly selected from among each training image and a corresponding ground-truth image for each decoder. Geometric augmentation (rotation and flipping) is applicable as additional data augmentation to the selected patches to avoid overfitting.

In operation <NUM>, the method <NUM> may include processing the augmented rendered WB data set by using the apparatus <NUM> to obtain reconstructed images (patches) corresponding to the correct WB, the shade WB, and the incandescent WB.

In operation <NUM>, the method <NUM> includes obtaining a loss between the reconstructed images corresponding to the correct WB, the shade WB, and the incandescent WB and the ground-truth image (patch). For example, the apparatus <NUM> is trained to minimize a squared L2-norm loss function between the reconstructed patches and the ground-truth patch.

h and w denote a width and height of a patch, respectively, and p denotes an index of each pixel of each trained patch P and ground-truth camera-rendered patch C. i∈ {A, T, S} denotes three target WB settings. Alternatively, an L1-norm loss function may be used to train the apparatus <NUM>.

In operation <NUM>, the method <NUM> includes updating a parameter of a neural network in the apparatus <NUM> (e.g., the encoder <NUM>, the first WB decoder <NUM>, and the second WB decoder <NUM>) to minimize the obtained loss. For example, after a weight of a convolutional layer of the apparatus <NUM> is initialized, a training process may be repeatedly performed <NUM>,<NUM> times using an adaptive moment estimation optimizer with a gradient moving average attenuation rate β<NUM>=<NUM> and a squared gradient moving average attenuation rate β<NUM>=<NUM>. A learning rate <NUM>-<NUM> may be used and reduced by <NUM> per <NUM> epochs. An L2 normalization ratio may be set, for example, to <NUM>-<NUM>, and <NUM> training patches may be deployed as a mini batch size for iteration. <FIG> is a diagram of a color mapping process in a deep white-balancing editing method according to embodiments of the disclosure.

A DNN model according to an embodiment of the disclosure is a fully-convolutional network, in which a <NUM>-level encoder/decoder with a 2x2 max-pooling layer and an upsampling layer is used and thus is subject to a restriction that dimensions have to be multiples of <NUM><NUM> and is capable of processing an input image to have original dimension.

However, the sizes of all of input images may be adjusted to a size of up to) <NUM> pixels to ensure a consistent execution time for input images with all sizes. The DNN model is applied to such an image with an adjusted size. Thereafter, a color mapping function between the input image with the adjusted size and an output image is calculated and applied to a full-size input image.

Referring to <FIG>, a color mapping process is performed to produce IWB(t) from a resolution (i.e., hxwx3 pixels) of an original input image with respect to an input image IWB(in) (h × w pixels) and a corresponding output image ÎWB(t)(h'xw' pixels, and in this case, h'≤h and w'≤w) produced by an encoder <NUM> and one of decoders (i.e., a decoder <NUM>). A polynomial function based on a mapping matrix ρ for globally mapping colors of a down-sampled input image IWB(in)↓ to colors of the produced output image ÎWB(i) may be employed. For example, the mapping matrix ρ may be calculated by a closed-form solution, as follows: <MAT>.

S=h(r(IWB(in)↓)),Y=r(IWB(t)),r:I(h×w×<NUM>)→I(hw×<NUM>) may be a reshape function of constructing I(hw!<NUM>) from I(h×w×<NUM>), and h:I(hw×<NUM>)→I(hw ×n) may be a polynomial kernel function of mapping RGB vectors of an image to a higher n-dimensional space. For example, <NUM>-dimensional polynomial mapping may be used.

When the mapping matrix ρ is calculated, a final result of the same input image may be calculated by the following equation: <MAT>.

<FIG> is a flowchart of a deep white-balancing editing method <NUM> according to other embodiments of the disclosure.

In operation <NUM>, the method <NUM> includes obtaining an input image with original WB corrected by image signal processing.

In operation <NUM>, the method <NUM> includes downsampling the obtained input image.

In operation <NUM>, the method <NUM> includes obtaining a downsampled intermediate representation of the downsampled input image by using a first neural network, in which the intermediate representation has the original WB that is not corrected by image signal processing.

In operation <NUM>, the method <NUM> includes obtaining a downsampled output image with first WB different from the original WB by using a second neural network, based on the obtained downsampled intermediate representation.

In operation <NUM>, the method <NUM> includes obtaining a first output image with the first WB different from the original WB by applying color mapping to the obtained downsampled output image.

<FIG> is a diagram of a user interface <NUM> for deep white-balancing editing according to embodiments of the disclosure.

The user interface <NUM> displays an input image <NUM> and a slider <NUM>, so that a user may select an image from among output images generated based on three available WB settings such as an AWB setting, a shade WB setting, and an incandescent WB setting. For example, a first output image <NUM> with incandescent WB (<NUM>) or a second output image <NUM> with shade WB (<NUM>) may be produced and displayed on the user interface <NUM> as shown in <FIG>. The slider <NUM> may be in a range <NUM> of Kelvins, e.g., <NUM> to <NUM>, and a user may select a Kelvin value within the range <NUM> by sliding the slider <NUM> to the selected value. Based on a user input for selecting the value, the user interface <NUM> may display an output image corresponding to the selected value.

Using the shade WB setting and the incandescent WB setting and the value selected from the range <NUM> by the user, the user may additionally edit the input image <NUM> to be a third output image <NUM> with a certain WB (e.g., <NUM>) other than the shade WB or the incandescent lamp WB in terms of color temperature, as shown in <FIG>. To produce an effect of setting a new target WB corresponding to a color temperature t that is not produced by one of decoders (e.g., the first WB decoder <NUM>), interpolation may be performed between the first and second output images <NUM> and <NUM> respectively produced using the incandescent and shade WB settings. The interpolation may be described by the following equation: <MAT>.

IWB(T) and IWB(S) may respectively denote the first and second output images <NUM> and <NUM> respectively produced with the incandescent and shade WB settings, and b may denote an interpolation ratio given by <MAT>.

The user interface <NUM> may be implemented in an electronic device, e.g., a user application installed in a mobile device. The user application may be configured to edit a WB of an image captured by a camera of an electronic device and processed by an image signal processor of the electronic device. The user application may be part of a gallery editing service.

<FIG> is a block diagram of a computer system <NUM> according to embodiments of the disclosure.

As shown in <FIG>, the computer system <NUM> may include a processor <NUM>, a memory <NUM>, an input/output interface <NUM>, and a display <NUM>.

The processor <NUM> may perform overall control of the apparatus <NUM> for deep white balancing editing of <FIG>, and execute one or more programs stored in the memory <NUM>. The processor <NUM> is embodied as hardware, firmware, or a combination of hardware and software. The processor <NUM> may be a central processing unit (CPU), a graphics processing unit (GPU), an acceleration processing unit (APU), a microprocessor, a microcontroller, a digital signal processor (DSP), a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), or another type of processing component. In some embodiments of the disclosure, the processor <NUM> includes one or more processors programmed to perform functions.

The processor <NUM> according to embodiments of the disclosure may perform one or a combination of functions by using the encoder <NUM>, the first WB decoder <NUM>, and the second WB decoder <NUM> described above with reference to <FIG>.

The memory <NUM> may include a hard disc (e.g., a magnetic disc, an optical disc, a magneto-optical disc and/or a solid-state disc), a compact disc (CD), a digital versatile disc (DVD), a floppy disk, a cartridge, a magnetic tape and/or other types of non-transitory computer-readable media, as well as a corresponding drive. Alternatively, the memory <NUM> may include random access memory (RAM), read-only memory (ROM), and/or other types of dynamic or static storage devices (for example, a flash memory, a magnetic memory, and/or an optical memory).

The memory <NUM> may store various types of data, a program, or an application for driving and controlling the apparatus <NUM>. The program stored in the memory <NUM> may include one or more instructions. The program, including one or more instructions or an application, stored in the memory <NUM> may be executed by the processor <NUM>.

The input/output interface <NUM> allows the computer system <NUM> to communicate with other devices, such as other electronic devices and other servers, through a wired connection, a wireless connection, or a combination thereof. For example, when the apparatus <NUM> and the computer system <NUM> are implemented in a server, the processor <NUM> may receive an input image with an original WB corrected by image signal processing by an image signal processor of an electronic device from the image signal processor through the input/output interface <NUM>. The processor <NUM> may transmit an output image with a WB different from the original WB to an electronic device through the input/output interface <NUM>.

The display <NUM> may obtain data from, for example, the processor <NUM>, and display the obtained data. The display <NUM> may include, for example, a touch screen, a television, a computer monitor, or a combination thereof.

Embodiments of the disclosure provide a deep learning framework for editing a WB. The framework exactly corrects a WB of an image with an incorrect WB. The framework also allows a user to freely edit a WB of a sRGB image with a different illumination setting. The framework includes a single encoder and a multi-decoder. A multi-decoder model is trained to produce several WB settings by an end-to-end manner. The framework may achieve a most advanced result of WB correction and manipulation and produce a more efficient compilation result than those of previous tasks for WB correction and manipulation.

<FIG> is a block diagram of a mobile device <NUM> according to an embodiment of the disclosure.

Referring to <FIG>, the mobile device <NUM> may include a camera module <NUM> and a processor <NUM>. Embodiments of the disclosure are not limited thereto, and the mobile device <NUM> may further include a display <NUM>, a user interface module <NUM>, an illuminance sensing module <NUM>, a GPS sensing module <NUM>, a network information obtaining module <NUM>, and a storage module <NUM>.

The camera module <NUM> may include a lens module <NUM>, an image sensing module <NUM>, and an image signal processor <NUM>.

The lens module <NUM> may include at least one lens to collect light reflected from a subject and transmit the light to the image sensing module <NUM>. According to an embodiment of the disclosure, the camera module <NUM> may include a plurality of lens modules. In this case, the plurality of lens modules may have the same lens attribute (e.g., an angle of view, a focal length, auto focus, F number or optical zoom). Alternatively, at least one lens module may have at least one lens attribute different from lens attributes of the other lens module. For example, the lens module <NUM> may include at least one lens of a wide-angle lens, a super-wide-angle lens, or a telephoto lens.

The image sensing module <NUM> may include at least one image sensor to convert light collected by and transmitted from the lens module <NUM> into an electrical signal. In addition, the image sensing module <NUM> may obtain an image corresponding to a subject by using the electrical signal. The image obtained through the image sensing module <NUM> will be described as raw image data herein. For example, at least one image sensor may be a charge-coupled device (CCD) sensor or a complementary metal-oxide semiconductor (CMOS) sensor and the image sensing module <NUM> may include a color filter array configured with a Bayer pattern and a corresponding pixel array including at least one CCD sensor and a CMOS sensor. However, embodiments of the disclosure are not limited thereto, and the image sensing module <NUM> may be embodied in various forms. For example, the image sensing module <NUM> may be embodied as a Foveon sensor. Raw image data may be understood to mean image data obtained from the image sensing module <NUM> without being image-processed by the image signal processor <NUM>, and may be, for example, color filter array data such as raw Bayer pattern data.

The image signal processor <NUM> may image-process the raw image data and output data with a certain image format. The image format may be understood to mean an image format with a certain color space that an output device (e.g., a display device or a printer device) may represent to output an image or store the image before the image is output. For example, the image format may be an RGB-based format, a CMYK-based format, a YCbCR-based format, or the like. Examples of the RGB-based format include a sRGB format, an Adobe RGB format, a Prophoto RGB format, etc., and examples of the CMYK-based format include a SWOP CMYK format, etc..

For example, the image signal processor <NUM> may process raw image data and output sRGB image data.

The image signal processor <NUM> may be implemented by hardware. However, the image signal processor <NUM> is not limited thereto and may be implemented by a combination of hardware and software. In this case, the image signal processor <NUM> may be embodied as a software module in the processor <NUM> outside the camera module <NUM>. Alternatively, the image signal processor <NUM> may be embodied as hardware separated from the camera module <NUM> and the processor <NUM>. The image signal processor <NUM> may perform a function of processing raw image data in conjunction with the processor <NUM> outside the camera module <NUM>. In another embodiment of the disclosure, the image signal processor <NUM> may be embodied as a system-on-chip (SoC) in the camera module <NUM>.

The image signal processor <NUM> or the processor <NUM> may include a demosaicking module <NUM>, a noise reduction module <NUM>, a white-balancing control module <NUM>, a color space conversion module <NUM>, a color manipulation module <NUM>, and an image output module <NUM>.

The demosaicking module <NUM> may perform demosaicking on raw image data. For example, the demosaicking module <NUM> may perform RGB demosaicking on the raw image data to obtain RGB image data. Demosaicking may be a type of color interpolation.

The noise reduction module <NUM> may perform a noise reduction process on RGB data to obtain RGB image data in which noise is reduced.

The white-balancing control module <NUM> may receive first image data and output second image data with a controlled WB. The first image data may be original RGB image data and the second image data may be RGB data with the controlled WB. In this case, the first image data may be sRGB image data obtained while passing through at least one of the demosaicking module <NUM> or the noise reduction module <NUM>.

Alternatively, the first image data may be original color filter array data, and the second image data may be color filter array data with a controlled WB.

According to an embodiment of the disclosure, the white-balancing control module <NUM> may input first image data to at least one Al model among a plurality of Al models and obtain at least one piece of second image data with a controlled WB. In this case, the plurality of Al models may consist of a neural network and may each include an encoder model and a decoder model. The plurality of Al models may include different decoder models. The plurality of Al models may include the same encoder model.

For example, the white-balancing control module <NUM> may include the encoder <NUM> (corresponding to an encoder model) and the first WB decoder <NUM> and the second WB decoder <NUM> (corresponding to decoder models) of <FIG>. However, it will be understood by those of ordinary skill in the art that the white-balancing control module <NUM> is not limited to the first WB decoder <NUM> and the second WB decoder <NUM> and may further include a third WB decoder, a fourth WB decoder, etc. The first WB decoder and the second WB decoder may be selectively used by a processor or a video signal processor, and data processed by the encoder is processed input data.

The encoder and a plurality of WB decoders may be configured as a neural network and trained using an original sRGB image or a sRGB image with a changed WB as training data.

In addition, the encoder and the plurality of WB decoders may be trained using an original raw image and a raw image with a changed WB as training data.

Here, "original" is used as an expression corresponding to "changed WB", and is not limited to an image on which WB control is not performed by the image signal processor <NUM> or the like with respect to original raw image data obtained by the image sensing module <NUM> and may be understood to mean an image on which WB control is performed by the image signal processor <NUM> with respect to the original raw image data.

In this case, the white-balancing control module <NUM> may be regarded as an original WB control module when the image on which WB control is not performed by the image signal processor <NUM> or the like with respect to the original raw image data obtained by the image sensing module <NUM> is input as input data to the white-balancing control module <NUM>, and may be regarded as a post-WB control module when the image on which WB control is performed by the image signal processor <NUM> or the like with respect to the original raw image data obtained by the image sensing module <NUM> is input as input data to the white-balancing control module <NUM>. The above description of WBs with reference to the drawings before <FIG> relates to a post-WB control module but it will be understood by those of ordinary skill in the art that embodiments of the disclosure are not limited thereto and the above description may also apply to the original WB module.

"changed WB" may be understood to mean that a WB is changed to have a WB attribute (e.g., the AWB, the incandescent WB, or the shade WB) that is expected to be an output of a certain WB decoder.

The encoder may be configured as a first neural network, and receive sRGB image data as input data and output raw image data corresponding to the sRGB image data as output data.

In an embodiment of the disclosure, the encoding may receive raw image data as input data and output an intermediate representation corresponding to the raw image data as output data.

Each of the plurality of decoders may be configured as a second neural network having a structure different from that of the first neural network, and receive raw image data output from the encoder as input data. The encoder may receive, as input data, original raw image data obtained through the image sensing module <NUM>. In this case, an Al model composed of only decoders may be mounted in the mobile device <NUM> to process original raw image data obtained through the image sensing module <NUM>. That is, during training of the Al model, the decoders and an encoder may be trained through one training process but when the trained Al model is applied to the mobile device <NUM>, only the decoders may be selected and stored in the storage module <NUM> of the mobile device <NUM>.

In an embodiment of the disclosure, each of the decoders may receive an intermediate representation output from the encoder as input data. The decoders convert the input data to correspond to a predetermined WB attribute and output a result of converting the input data.

When the decoders receive raw image data as input data, each of the decoders converts the raw image data according to a WB attribute corresponding thereto to produce and output sRGB image data to which the WB attribute is reflected.

In an embodiment of the disclosure, in order to interface with a user, the mobile device may convert raw image data into sRGB image data, display the sRGB image data, and receive a user desired WB setting value through a displayed screen, but in order to reobtain an image to which the desired WB setting value is applied, the white-balancing control module <NUM> may obtain raw image data corresponding to the displayed sRGB image through the encoder and obtain sRGB image data with a changed WB through a decoder corresponding to the user's desired WB setting.

In another embodiment of the disclosure, based on information about a shooting condition obtained through the sensor modules <NUM>, one of the plurality of Al models may be selected and an image to which a WB setting value corresponding to the shooting condition may be obtained by the selected Al model. In this case, the Al model may be an Al configured only with decoders and receive original raw image obtained through the image sensing module <NUM> as input data.

When the decoders receive, as input data, an intermediate representation corresponding to input raw image data, the decoders generate raw image data converted according to a WB attribute corresponding to each of the decoders and output. That is, when a user interface is not needed, in order to obtain again an image to which a user's desired WB setting value is applied, the white-balancing control module <NUM> may obtain an intermediate representation corresponding to raw image data through the encoder and obtain raw image data with a changed WB through a decoder corresponding to the user's desired WB setting.

A weight and a bias of the first neural network and at least one second neural network may be trained and set in advance. In an embodiment of the disclosure, the processor <NUM> or the image signal processor <NUM> may receive the weight and the bias of the first neural network and the at least one second neural network, which are trained in advance and stored, from a separate training device, and control the first neural network and the at least one second neural network stored in the storage module <NUM> to be set with the received at least one weight and bias.

The white-balancing control module <NUM> may determine an Al model corresponding to a shooting condition among the plurality of Al models stored in the storage model <NUM>, based on information about the shooting condition obtained through the sensor module <NUM>, and may input first image data to the determined Al model so as to obtain second image data with a WB controlled to correspond to the shooting condition. This will be described in detail after describing the sensor module <NUM> below.

The color spatial conversion module <NUM> may obtain image data with a converted color spatial by performing color space conversion on input image data. For example, the color space conversion module <NUM> may perform CIE-XYZ image data by performing color spatial conversion on RGB image data.

The color manipulation module <NUM> may obtain color-manipulated image data by performing color manipulation on image data with a converted color space. Color manipulation may include gamma correction and may be referred to as photo-finishing.

The image output module <NUM> may output image data by performing mapping on the color-manipulated image data. For example, the output image data may be SRGB image data.

The sensor module <NUM> may obtain information about a shooting condition. For example, the sensor module <NUM> may include an illuminance sensing module <NUM>, a GPS sensing module <NUM>, and a network information obtaining module <NUM>.

The illuminance sensing module <NUM> may obtain illuminance sensor information. The illuminance sensor information may represent a lux sensed by the illuminance sensor.

The GPS sensing module <NUM> may obtain GPS sensor information. The GPS sensor information may represent at least one of a longitude, latitude, or altitude obtained by the GPS sensor. The GPS sensor information may be obtained and updated periodically or aperiodically in response to a request.

The network information obtaining module <NUM> may obtain network status information. The network status information may include, but is not limited to, at least one of received signal strength indication (RSSI) information of at least one scanned access point (AP) or service set identifier (SSID) information of the scanned at least one at least one AP.

The white-balancing control module <NUM> may determine an Al model corresponding to a shooting condition among a plurality of Al models, for WB control, stored in the storage model <NUM>, based on information about the shooting condition obtained through the sensor module <NUM>, and may input first image data to the determined Al model so as to obtain second image data with a WB controlled to correspond to the shooting condition.

For example, the white-balancing control module <NUM> may identify whether the mobile device <NUM> is located indoors or outdoors, based on at least one of GPS sensor information, illuminance sensor information, or network status information. The white-balancing control module <NUM> may determine an Al model corresponding to a result of the identification result, based on the result of the identification. For example, there may be a tungsten/incandescent WB setting (or a fluorescent WB setting) representing indoor lighting, a shade WB setting representing outdoor lighting, and the like, and the white-balancing control module <NUM> may identify whether the mobile device <NUM> is located indoors or outdoors and determine an Al model corresponding to a WB setting related to a result of the identification among various types of WB settings, based on the result of the identification.

Referring to the drawings before <FIG>, although it is described above that output image data with a certain WB is obtained using a WB decoder corresponding to a certain WB setting (e.g., the AWB setting, the incandescent WB setting or the shade WB setting), embodiments of the disclosure are not limited thereto, and it will be understood by those of ordinary skill in the art that in the embodiments of the drawings before <FIG>, second image data with a WB controlled to correspond to the shooting condition may be obtained from first image data, based on information about a shooting condition (indoor/outdoor).

Generally, the AWB setting is an accurate WB setting, but an output image with a more appropriate WB setting may be obtained using a WB decoder corresponding to a WB setting determined based on the information about the shooting condition (indoor/outdoor). However, embodiments of the disclosure are not limited thereto, and an output image with the AWB setting and an output image with a WB setting corresponding to the shooting condition among an indoor WB setting and an outdoor WB setting may be simultaneously obtained and displayed on a display, so that a user may select one of the output images.

In an embodiment of the disclosure, the processor <NUM> may determine a WB setting corresponding to information obtained using information about a shooting condition obtained through at least one sensor module <NUM>. The processor <NUM> may determine an Al model corresponding to the determined WB setting among the plurality of Al models stored in the storage module <NUM>. The processor <NUM> may control the image signal processor <NUM> to process CFA image data obtained from the image sensing module <NUM> by loading the determined Al model to the image signal processor <NUM>. More specifically, the processor <NUM> may control data about the Al model determined among the plurality of Al models stored in the storage module <NUM> to be loaded to a memory (not shown) in the processor <NUM>. The image signal processor <NUM> may process the CFA image data received from the image sensing module <NUM> by using the data about the Al model loaded to the memory. The image signal processor <NUM> may obtain RGB image data by processing the CFA image data by the Al model.

In an embodiment of the disclosure, a storage module (not shown) in the image signal processor <NUM> may store a plurality of Al models. Here, the storage module differs from the memory in the image signal processor <NUM>. That is, the storage module may be a flash memory or DRAM, and the memory may be SRAM. In this case, the image signal processor <NUM> receives data about a WB setting corresponding to information, which is obtained from the processor <NUM> through at least one sensor module <NUM>, by using the information. The image signal processor <NUM> may determine an Al model to be processed by using the received data about the WB setting. The image signal processor <NUM> may control the determined Al model to be loaded to a memory (not shown) in the image signal processor <NUM>. The image signal processor <NUM> may process CFA image data received from the image sensing module <NUM> by using data about the Al model loaded to the memory. The image signal processor <NUM> may obtain RGB image data by processing the CFA image data by the Al model.

The white-balancing control module <NUM> may identify that a mobile device is located indoors when GPS sensor information is not updated, and identify that the mobile device is located outdoors when GPS sensor information is updated.

The white-balancing control module <NUM> may identify that the mobile device is located outdoors when a sensor value (e.g., a lux value) of illuminance sensor information is greater than or equal to a first value. In this case, the white-balancing control module <NUM> may identify whether the sun rises, based on current time information, and an estimated value according to the position of the sun corresponding to the current time may be set to the first value. Because there is only an artificial illumination at the time when the sun is not expected to rise, the mobile device may identify as being located indoors regardless of the sensor value of the illuminance sensor information.

The white-balancing control module <NUM> may identify that the mobile device is located indoors when the sensor value (e.g., a lux value) of the illuminance sensor information is less than or equal to a second value.

The white-balancing control module <NUM> may identify that the mobile device is located indoors when an average RSSI of scanned APs is greater than or equal to a first value, and identify that the mobile device is located outdoors when the average RSSI of the scanned APs is less than a second value.

Otherwise, the white-balancing control module <NUM> may identify that the module is located outdoors, when among the scanned APs, the number of APs, the RSSI of which is less than or equal to the second value, is greater than or equal to a second number or the number of APs, which is greater than or equal to the first value, is less than a first number.

The white-balancing control module <NUM> may identify that the module is located indoors when among the scanned APs, the number of APs, the RSSI of which is greater than or equal to the first value, is greater than or equal to the first number.

The white-balancing control module <NUM> may identify that the module is located outdoors, when among the scanned APs, the number of APs, the RSSI of which is less an or equal to the second value, is greater than or equal to a second number or the number of APs, which is greater than or equal to the first value, is less than the first number.

Only APs with an SSID, the part of which corresponds to a character string, may be considered when the condition described above is satisfied. However, embodiments of the disclosure are not limited thereto, and whether a final condition is satisfied may be determined by differentiating between a group of corresponding APs and other groups of APs and assigning different weights to results of determining whether the groups each satisfy an intermediate condition.

An embodiment in which the white-balancing control module <NUM> identifies whether the mobile device <NUM> is located indoors or outdoors, based on at least one of GPS sensor information, illuminance sensor information, or network status information, will be described in detail with reference to <FIG> below.

The display <NUM> may display various types of image data. In particular, the processor <NUM> controls the display <NUM> to display data with a certain image format output through the image signal processor <NUM> or the like.

The user interface module <NUM> may receive various types of user inputs. The display <NUM> and the user interface module <NUM> may be embodied together as a touch screen that includes a touch sensor and a display panel (e.g., an LCD or an AMOLED). The processor <NUM> may control the camera module <NUM> to be driven, based on a user input received through the user interface module <NUM>. That is, the processor <NUM> may control the display panel to display a user interface (UI) icon for starting to capture an image and an UI icon for setting an image capture condition. When a user input that is input through the touch sensor is received, the processor <NUM> may control the image sensing module <NUM> to convert light collected through the lens module <NUM> in the camera module <NUM> into an electrical signal.

The storage module <NUM> may display various types of image data. In particular, the processor <NUM> controls the storage module <NUM> to store data with a certain image format output through the image signal processor <NUM> or the like. The image format stored in the storage module <NUM> may be different from an image format output to the display <NUM>. For example, the processor <NUM> may perform loss/lossless compression on at least one output image obtained from the white-balancing control module <NUM> according to a certain compression standard (e.g., the JPEG standard), and control the storage module <NUM> to store the compressed output image.

The storage module <NUM> may also store information about an Al model used in the white-balancing module <NUM>.

<FIG> is a diagram for describing identifying whether the mobile device is located indoors or outdoors, based on various types of information obtained by a mobile device, and performing WB control, based on a result of the identification.

Referring to <FIG>, a mobile device <NUM> may obtain illuminance sensor information by sensing light from an indoor lamp <NUM> (e.g., fluorescent light or incandescent light) by using an illuminance sensor <NUM>. The mobile device <NUM> may identify that the mobile device <NUM> is located indoors when a value of the illuminance sensor information is less than or equal to a first value. This is because in general, an illumination value indoors is considerably less than an illumination value outdoors.

The mobile device <NUM> may obtain GPS sensor information from a GPS satellite <NUM> using a GPS sensor. When the GPS sensor information is not updated, the mobile device <NUM> may identify that the mobile device <NUM> is located indoors. This is because generally, it is difficult to receive GPS sensor information when the mobile device <NUM> is located indoors.

The mobile device <NUM> may scan APs <NUM> therearound to obtain a list of at least one scanned AP. The mobile device <NUM> may identify the number of scanned APs <NUM>, based on the number of SSIDs of the scanned APs <NUM>, and identify signal strength of each of the scanned APs <NUM>, based on RSSIs <NUM> of the scanned APs <NUM>.

The mobile device <NUM> may identify that the mobile device <NUM> is located indoors when an average value of the RSSIs <NUM> of the scanned APs <NUM> is greater than or equal to the first value. This is because generally, when the mobile device <NUM> is located indoors, the APs <NUM> are located close to the mobile device <NUM> and thus the average value of the RSSIs <NUM> of the APs <NUM> is small.

Alternatively, the mobile device <NUM> may identify that the mobile device <NUM> is located indoors when the number of APs <NUM>, the RSSI <NUM> of which is greater than or equal to the first value, is greater than or equal to a first number. Generally, this is because when the mobile device <NUM> is located indoors, many APs are located close to the mobile device <NUM> and thus the number of APs, the RSSI <NUM> of which is large indoors, is greater than the number of APs, the RSSI <NUM> of which is large outdoors.

The mobile device <NUM> may obtain the illuminance sensor information by sensing light from the sun <NUM> by using the illuminance sensor <NUM>. The mobile device <NUM> may identify that the mobile device <NUM> is located indoors when a value of the illuminance sensor information is greater than or equal to a second value. This is because in general, an illumination value outdoors is considerably greater than an illumination value indoors.

The mobile device <NUM> may obtain GPS sensor information from the GPS satellite <NUM> by using a GPS sensor. When the GPS sensor information is updated, the mobile device <NUM> may identify that the mobile device <NUM> is located outdoors. This is because generally, the GPS sensor information is more easily received when the mobile device <NUM> is located outdoors than when the mobile device <NUM> is located indoors and thus may be updated in response to a request in most cases.

The mobile device <NUM> may scan the APs <NUM> around the mobile device <NUM>. The mobile device <NUM> may identify the number of scanned APs <NUM>, based on the number of SSIDs of the scanned APs <NUM>, and identify signal strength of each of the scanned APs <NUM>, based on RSSIs <NUM> of the scanned APs <NUM>.

The mobile device <NUM> may identify that the mobile device <NUM> is located outdoors when an average value of the RSSIs <NUM> of the scanned APs <NUM> is less than a second value. This is because generally, when the mobile device <NUM> is located outdoors, the average RSSI of the APs <NUM> is small due to a large distance to the mobile device <NUM> from the APs <NUM> or due to various types of obstacles.

Alternatively, the mobile device <NUM> may identify that the mobile device <NUM> is located outdoors when the number of APs <NUM>, the RSSI <NUM> of which is less than or equal to the second value, is greater than or equal to a second number. This is because generally, when the mobile device <NUM> is located outdoors, the number of APs <NUM> with a small RSSI outdoors is less than the number of APs <NUM> with a small RSSI indoors due to a large distance to the mobile device <NUM> from the APs <NUM> or due to various types of obstacles.

However, embodiments of the disclosure are not limited thereto, and the mobile device <NUM> may identify that the mobile device <NUM> is located outdoors when the number of APs <NUM>, the RSSI <NUM> of which is greater than or equal to the first value, is less than the first number. This is because generally, when the mobile device <NUM> is located outdoors, the number of APs <NUM> close to the mobile device <NUM> outdoors is less than the number of APs <NUM> close to the mobile device <NUM> indoors.

However, embodiments of the disclosure are not limited thereto, and the mobile device <NUM> may identify whether the mobile device <NUM> is located indoors or outdoors by various techniques, based on at least one of the illumination sensor information, the GPS sensor information or the network status information.

The mobile device <NUM> may perform white balancing control, based on an identification result. For example, there may be the tungsten/incandescent WB setting (or a fluorescent WB setting) representing indoor lighting, the shade WB setting representing outdoor lighting, and the like, and the mobile device <NUM> may identify whether the mobile device <NUM> is located indoors or outdoors, determine an Al model related to a WB setting among a plurality of Al models related to a plurality of WB settings, based on a result of the identification, and obtain an output image by using the determined Al model.

Claim 1:
A mobile device comprising:
a camera module (<NUM>) including a lens module (<NUM>) and an image sensing module (<NUM>) configured to obtain first image data of an input image with an original white balance corrected by image signal processing by an ISP of the camera module from light collected through the lens module:
a sensor module (<NUM>) configured to obtain information about a shooting condition;
a storage module (<NUM>) storing information about a plurality of artificial intelligence, Al, models for white-balance control;
an image signal processor (<NUM>) configured to obtain second image data by processing at least one of the stored Al models; and
at least one application processor (<NUM>),
wherein the at least one application processor is configured to:
determine an Al model corresponding to the shooting condition among the plurality of Al models, for white-balance control, stored in the storage module, based on the information about the shooting condition obtained by the sensor module; and
control the determined Al model to be loaded into the image signal processor, and
the image signal processor is further configured to input the first image data to the determined Al model so as to obtain the second image data with a white balance controlled to correspond to the shooting condition, and
wherein the plurality of Al models each comprise an encoder model (<NUM>) for obtaining , based on the first image data, an intermediate representation of the input image with the original white balance that is not corrected by the image signal processing, and a decoder model (<NUM>) for obtaining the second image data of a first output image with a first white balance different from the original white balance, based on the intermediate representation, and the plurality of Al models each comprise different decoder models, each decoder model having a predetermined WB attribute corresponding thereto.