Patent Description:
Embodiments of the invention relate to a device with image processing capability for enhancing picture quality.

Modern devices with image display capabilities typically perform image enhancement operations when displaying images. For example, a television may enhance images or videos to be displayed on a screen, and a smartphone may enhance images or videos captured by or displayed on the smartphone. However, a conventional device typically performs image enhancement operations based on algorithms or formulations pre-configured by the device manufacturer. There is limited flexibility in adjusting the algorithms or formulations once the device is in use by a consumer. Thus, there is a need for improving the design of an image processing device to allow more flexibility in picture quality adjustment.

<CIT> discloses to use face detection to provide for automatic enhancement of appearances of an image based on knowledge of human faces in the image and to output the enhanced image through an output device or a display device.

<NPL>, discloses a framework for automatic photo enhancement that attempts to take local and global image semantics into account.

<CIT> discloses an image enhancement system that includes an image quality component for assigning an image quality to the images, based on image quality parameters of input digital images. An image categorizing component is provided for assigning a semantic class to an input image based on image content. A model maps assigned image quality and assigned semantic class to candidate aesthetic enhancements, whereby for at least some of the input images, the enhancement is applied in a mode which is dependent on the assigned semantic class.

<NPL>, discloses a system for automatic image enhancement so that the appearance of a user-specified image becomes preferable to the user. The system has two components: a preference estimator and a parameter optimizer. The preference estimator learns the user's preference from an example image provided by the user. Then the parameter optimizer determines the best parameter set for the image enhancement functions by using the trained preference estimator.

<CIT> discloses using a machine learning framework to detect a set of one or more attributes of an input image and outputting an output image comprising a modified version of the input image, wherein the input image is modified by modifying at least a subset of the detected set of attributes.

Document <NPL>" discloses that in order to incrementally learn a boosted cascade, all strong classifier of the cascade must be incrementally learned sequentially, and online training samples need to be filtered in terms of the updated strong classifiers. The same document further discloses that in order to demonstrate the efficacy of this incremental learning algorithm, three face image sets are employed: <NUM> frontal faces with sun glasses, <NUM> frontal faces with scarves and <NUM> half-profile faces in extreme lightening conditions. The same document further discloses that in order to improve the detection rate, a small part of missed faces are manually labeled, and <NUM> online training samples are generated from each face.

Document <NPL>" discloses assigning, by a user, a new object class when a detector fails to detect any object in an input image, generating labeled images for the new object class and re-training a model with old and new object classes.

The present invention is illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings in which like references indicate similar elements. It should be noted that different references to "an" or "one" embodiment in this disclosure are not necessarily to the same embodiment, and such references mean at least one.

In the following description, numerous specific details are set forth. In other instances, well-known circuits, structures and techniques have not been shown in detail in order not to obscure the understanding of this description. It will be appreciated, however, by one skilled in the art, that the invention may be practiced without such specific details. Those of ordinary skill in the art, with the included descriptions, will be able to implement appropriate functionality without undue experimentation.

A device including an image processing circuit is described herein. A user may view images (e.g., a video) on a display panel coupled to the image processing circuit. The image processing circuit generates a training database containing the images that are labeled automatically and/or manually. The image processing circuit further uses the training database to re-train one or more models, based on which one or more attributes of the images are identified. A picture quality (PQ) engine enhances the quality of output images by changing certain image values associated with the identified attributes. If a user is not satisfied with the quality of output images shown on the display panel, the user may provide feedback to the device to help re-train the models that were used for generating the identified attributes. Thus, users can tailor the training database and the models according to their viewing experiences and preferences, and, as a result, the device provides flexibility in image quality adjustment.

<FIG> is a block diagram illustrating an image processing circuit <NUM> performing picture quality enhancement according to one embodiment. The image processing circuit <NUM> may be part of a device, also referred to as an edge device, such as a television, a smartphone, a computing device, a network-connected device, a gaming device, an entertainment device, an Internet-of-things (IoT) device, or any device capable of processing and displaying images and/or videos. The images to be processed may be captured by the same device, or by a different source and then downloaded, streamed, transferred, or otherwise accessible to the device. Preferably, the image processing circuit <NUM> may include an artificial intelligence (AI) processor <NUM> coupled to a picture quality (PQ) engine <NUM>. The AI processor <NUM> can be trained to infer representative characteristics in input images, and the PQ engine <NUM> can enhance the picture quality of the input images based on the representative characteristics.

The image processing circuit <NUM> includes an input port <NUM> for receiving an input image <NUM> and an output port <NUM> for outputting an output image <NUM>, which, in this example, is the processed image of the input image <NUM>. The output image <NUM> is sent to a display panel <NUM> for display. For ease of description, an input image and its corresponding output image are provided as an example. It is understood that the following description is applicable when the image processing circuit <NUM> receives an image sequence (e.g., a video) as input and generates a corresponding image sequence as output.

The image processing circuit <NUM> further includes a control module <NUM> which sends control signals (shown in dotted lines) to control and manage on-device training and inference operations. The control module <NUM> triggers training operations performed by a training engine <NUM> to train or re-train models <NUM> with labeled images from a training database <NUM>. The control module <NUM> also triggers inference operations performed by an attribute identification engine <NUM> to identify attributes (i.e., representative characteristics) in the input image <NUM>. Preferably, the attribute identification engine <NUM> may identify the attributes by inference and/or measurement based on one or more models <NUM>. The attribute identified by the attribute identification engine <NUM> may be a type (e.g., a scene type or an object type), statistic information, or a feature in the image content. For example, the attributes may include a scene type, types of objects in a scene, contrast information (e.g., histogram or statistics), luminance information (e.g., histogram or statistics), edge directions and strength, noise and degree of blur, segmentation information, motion information, etc. Preferably, the attribute may be identified using a machine-learning or deep-learning algorithm.

Preferably, the image processing circuit <NUM> may be implemented in a system-on-a-chip (SoC). Preferably, the image processing circuit <NUM> may be implemented in more than one chip in the same electronic device.

Preferably, the attribute identification engine <NUM> may identify multiple attributes from an image (e.g., a scene type as well as contrast information), which are collectively referred to as an attribute set of the image. The attribute identification engine <NUM> may further generate a confidence level of an identified attribute; e.g., <NUM>% confidence for the nature scene type. A high confidence level (e.g., when the confidence level exceeds a threshold) indicates that the identified attribute has a correspondingly high probability to be correctly identified. The attribute identification engine <NUM> sends the attribute set to the PQ engine <NUM> and a data collection module <NUM>.

The PQ engine <NUM> performs image enhancement operations on the input image <NUM> using image processing algorithms based on the attribute set of the input image <NUM>. Different algorithms may be used for different attributes; e.g., an algorithm for noise reduction, another algorithm for a nature scene, and yet another algorithm for a scene type of food. Preferably, the PQ engine <NUM> may perform one or more of the following operations: de-noising, scaling, contrast adjustment, color adjustment, and sharpness adjustment. For example, the PQ engine <NUM> may increase the warmth of the image color in a food scene, increase the sharpness in a blurry image, and de-noise in a noisy image. The output of the PQ engine <NUM> is the output image <NUM>, which is sent to the data collection module <NUM> and the output port <NUM>.

The data collection module <NUM> receives the output image <NUM> from the PQ engine <NUM>, and also receives the input image <NUM> and the attribute set of the input image <NUM> from the attribute identification engine <NUM>. Preferably, one or more identified attributes in the attribute set may be attached with respective confidence levels.

The data collection module <NUM> is a part of the image processing circuit <NUM> which provides labeled images to the training database <NUM>. In a manual labeling approach, the input image <NUM> is labeled by a user. In an automatic labeling approach, the input image <NUM> is automatically labeled with identified attributes of high confidence levels. The automatic labeling and the manual labeling will be described with reference to <FIG>, respectively.

The control module <NUM> may trigger the training engine <NUM> to perform training operations to train and/or re-train models <NUM> with the labeled images from the training database <NUM>. The training operations may be performed periodically or based on events. For example, the training operations may start when the image processing circuit <NUM> enters a sleep state or an idle state. For an edge device with limited processing resources (e.g., a smart TV, a smartphone, an IoT device, etc.), the models <NUM> may be initially trained on a server such as a cloud server, and re-trained on the edge device by the training engine <NUM> based on images or videos viewed on the edge device. The training operations change the weights or parameters in the models <NUM>, such as filter weights in an image filter, kernel weights in a neural network kernel, thresholds, etc. Preferably, the training operations may be performed by machine learning, deep learning, or other types of learning operations.

<FIG> is a block diagram illustrating automatic labeling according to one embodiment. In <FIG>, only the inference part of the image processing circuit <NUM> is illustrated. In this embodiment, the data collection module <NUM> includes a confidence check circuit <NUM>, which compares the confidence level of an identified attribute of an input image with a threshold. Different attributes may be compared with different thresholds. The data collection module <NUM> labels the input image with the identified attribute that exceeds its confidence threshold, and updates the training database <NUM> with the labeled image for on-device training operations. For example, an input image may be identified to have the following attributes: an urban street scene type, object types of cars, people, and buildings, and <NUM>% contrast. If the urban street scene type is identified to have a high confidence level (i.e., exceeding its confidence threshold), the data collection module <NUM> is to label the input image as having the urban street scene type. The data collection module <NUM> may further label the input image with other attributes having confidence levels exceeding their respective confidence thresholds. The labeled image is then stored in the training database <NUM>.

<FIG> is a block diagram illustrating manual labeling according to one embodiment. In <FIG>, only the inference part of the image processing circuit <NUM> is illustrated. In this embodiment, the data collection module <NUM> is coupled to a user interface <NUM> to receive a user's feedback on a displayed image. The user interface <NUM> may provide a graphical user interface (GUI) on a display panel, a voice-controlled user interface to receive a user's voice commands, or other means for receiving user's input regarding a displayed image. Via the user interface <NUM>, a user may indicate that one or more of the images have poor picture quality and request to start a manual labeling process. Referring also to <FIG>, the output image <NUM> is an example of a displayed image. In the description hereinafter, the terms "displayed image" and "output image" are used interchangeably.

Preferably, the manual labeling may be performed on demand by a user. A user may mark a displayed image as having poor picture quality; e.g., by selecting a button, and the marking action triggers the start of a manual labeling process. Alternatively, a user may request to start a manual labeling process at any time regarding any image attribute. The image processing circuit <NUM>, in response, requests the user to label the displayed image or the corresponding input image with a correct value or type of an attribute, where the "correctness" may be determined from the user's perspective. Preferably, the user interface <NUM> may present the user with a number of selectable values or types to replace the device-identified attribute. Using the scene type as an example, the user interface <NUM> may present the user with options such as "people", "food", "nature", "landmark" to select as the scene type attribute for an image. The user may select one of the presented types (e.g., people) to indicate the correct scene type attribute for the image. Preferably, the user may add a new label such as "animals" to indicate the correct scene type attribute for the image.

To improve the training accuracy, the data collection module <NUM> may retrieve, from the training database <NUM>, multiple sample images that are similar to the user-labeled image with respect to an attribute of interest. Preferably, the data collection module <NUM> includes a sample select circuit <NUM>, which selects sample images from the training database <NUM> and provides the selected sample images to the user. Each of the selected sample images has a confidence level exceeding a predetermined threshold with respect to the attribute of interest. For example, the sample image may be displayed on the display panel along with a list of selectable values or types of an attribute of interest. A user may label a sample image by selecting a value or type from the list. Preferably, a user may add a new value or a new type to the attribute of interest. Using the above example in which scene type is the attribute of interest, each sample image may be presented with a list of people", "food", "nature", "landmark" for the user to select. The user may select from the list. Alternatively, the user may add "animals" to the list as a new option for the scene type. The manual labeling process ends when the user labels all of the sample images provided by the sample select circuit <NUM>.

<FIG> illustrates an example of an output image with an incorrectly identified scene type attribute according to one embodiment. The display panel <NUM> may be coupled to the output port <NUM> of the image processing circuit <NUM> (<FIG>). A user may mark an image <NUM> of a person as having poor picture quality; e.g., by selecting a button or commanding by voice or other means when the image <NUM> is displayed. In response, the image processing circuit <NUM> presents the image <NUM> with a list of scene types for the scene type attribute. The image processing circuit <NUM> may also present values, types, or information of other attributes of the image <NUM> to the user in prior or subsequent lists. For the scene type attribute, the display panel <NUM> shows the image <NUM> with the automatically generated confidence level for each scene type: e.g., people (<NUM>), vehicle (<NUM>), tree (<NUM>), drink (<NUM>), and house (<NUM>), where each number in the parenthesis indicates the confidence level of the corresponding scene type in the image. According to the confidence levels, the attribute identification engine <NUM> identifies a tree scene type. The tree scene type in this example is the identified attribute. Based on this identified attribute, the PQ engine <NUM> may apply an algorithm for enhancing a tree scene to generate an output image.

However, a user can determine, from the image <NUM>, that the correct scene type attribute should be "people. " For this image <NUM>, the people scene type is a user-identified attribute that is different from the device-identified attribute of a tree scene type. In this example, the user may select the "people" tab to change the device-identified attribute for the image <NUM>. The user-identified scene type of people becomes a label of the image <NUM>.

After the user labels the image <NUM> with a corrected attribute, the image processing circuit <NUM> presents the user with a number of sample images that were previously identified as the people scene type. The user may label these sample images with respect to the scene type to indicate whether or not they were correctly identified as containing the people scene type. <FIG> illustrates one of the sample images <NUM> as an example. The data collection module <NUM> stores the user-labeled images, including the user-labeled image <NUM> and the user-labeled sample images <NUM>, into the training database <NUM>.

The training engine <NUM> uses the labeled images from the training database <NUM> to re-train the models <NUM>. The models <NUM> may have been trained to detect a feature (e.g., edge directions and strength, segmentation information, motion, etc.) in an image or an image sequence, classify the image content, measure a condition of an image (e.g., contrast, sharpness, brightness, luminance, noise, etc.), etc. The models <NUM> may be described by mathematical formulations or representations. The models <NUM> may initially be installed in the image processing circuit <NUM> and can be re-trained, or refined, with labeled images to learn from the user's image viewing experience on the device.

<FIG> is a flow diagram illustrating a method <NUM> for image enhancement according to one embodiment. Preferably, the image enhancement operations described herein include generating training data and re-training a model. The method <NUM> may be performed, for example, by the image processing circuit <NUM> of <FIG> and/or the device <NUM> of <FIG>. It is understood that <FIG> and <FIG> are for illustrative purposes only; other image processing devices may perform the method <NUM>.

The method <NUM> begins at step <NUM> with the device identifying an attribute from an input image based on a model stored in the device. At step <NUM>, the device generates an output image for display by enhancing the input image based on the identified attribute. At step <NUM>, the device generates a labeled image based on the input image labeled with the identified attribute. At step <NUM>, the device adds the labeled image to a training database stored in the device. At step <NUM>, the device re-trains the model using the training database. Preferably, the model may be re-trained on the device.

<FIG> illustrates an example of a device <NUM> according to one embodiment. The device <NUM> may include the image processing circuit <NUM> of <FIG>, which performs the aforementioned image enhancement operations. The device <NUM> includes processing hardware <NUM>. Preferably, the processing hardware <NUM> may include one or more processors, such as central processing units (CPUs), graphics processing units (GPUs), digital processing units (DSPs), multimedia processors, and other general-purpose and/or special-purpose processing circuitry. Preferably, the processing hardware <NUM> may include hardware circuitry including but not limited to: the attribute identification engine <NUM>, the PQ engine <NUM>, the data collection module <NUM>, and the training engine <NUM> in <FIG>. Additionally or alternatively, the processing hardware <NUM> may include an artificial intelligence (AI) processor <NUM>, which may be an example of the AI processor <NUM> in <FIG>. Referring back to <FIG>, preferably, the training engine <NUM> and/or the attribute identification engine <NUM> may be part of the AI processor <NUM>. Preferably, the AI processor <NUM> may be part of a GPU. Preferably, the AI processor <NUM> may include a hardware accelerator, such as a convolution neural network (CNN) accelerator <NUM>.

The CNN accelerator <NUM> includes hardware components specialized for accelerating neural network operations by convolutional operations, fully-connected operations, activation, pooling, normalization, element-wise mathematical computations, etc. Preferably, the CNN accelerator <NUM> includes multiple compute units and memory (e.g., Static Random Access Memory (SRAM)), where each compute unit further includes multipliers and adder circuits, among others, for performing mathematical operations such as multiply-and-accumulate (MAC) operations to accelerate the convolution, activation, pooling, normalization, and other neural network operations. The CNN accelerator <NUM> may perform fixed and floating-point neural network operations. In connection with the picture quality enhancement described herein, the CNN accelerator <NUM> may perform training and inference operations described in connection with <FIG>.

The device <NUM> further includes a memory and storage hardware <NUM> coupled to the processing hardware <NUM>. The memory and storage hardware <NUM> may include memory devices such as dynamic random access memory (DRAM), SRAM, flash memory, and other non-transitory machine-readable storage medium; e.g., volatile or non-volatile memory devices. The memory and storage hardware <NUM> may further include storage devices, for example, any type of solid-state or magnetic storage device. Preferably, the memory and storage hardware <NUM> may store the models <NUM> and the training database <NUM> of <FIG>. Preferably, the memory and storage hardware <NUM> may store instructions which, when executed by the processing hardware <NUM>, cause the processing hardware <NUM> to perform the aforementioned image enhancement operations, such as the method <NUM> of <FIG>.

The device <NUM> may also include a display panel <NUM> to display information such as images, videos, messages, Web pages, games, texts, and other types of text, image, and video data. The images may be labeled by a user via a user interface, such as a keyboard, a touchpad, a touch screen, a mouse, a touch screen, etc. The device <NUM> may also include audio hardware <NUM>, such as a microphone and a speaker, for receiving and generating sounds. The audio hardware <NUM> may also provide a user interface for sending and receiving voice commands.

Preferably, the device <NUM> may also include a network interface <NUM> to connect to a wired and/or wireless network for transmitting and/or receiving voice, digital data and/or media signals. It is understood the embodiment of <FIG> is simplified for illustration purposes. Additional hardware components may be included.

The operations of the flow diagram of <FIG> have been described with reference to the exemplary embodiments of <FIG> and <FIG>. However, it should be understood that the operations of the flow diagram of <FIG> can be performed by embodiments of the invention other than the embodiments of <FIG> and <FIG>, and the embodiments of <FIG> and <FIG> can perform operations different than those discussed with reference to the flow diagram. While the flow diagram of <FIG> shows a particular order of operations performed by certain embodiments of the invention, it should be understood that such order is exemplary (e.g., alternative embodiments may perform the operations in a different order, combine certain operations, overlap certain operations, etc.).

Claim 1:
A device (<NUM>) comprising:
a user interface (<NUM>); and
an image processing circuit (<NUM>), comprising:
a memory (<NUM>);
an attribute identification engine (<NUM>) to identify an attribute from an input image (<NUM>) based on a model (<NUM>) stored in the memory (<NUM>); and
a picture quality, in the following also referred to as PQ, engine (<NUM>) to generate an output image (<NUM>) for display by enhancing the input image (<NUM>) based on the identified attribute;
characterized in that the image processing circuit (<NUM>) further comprises:
a data collection module (<NUM>) to generate a labeled image based on the input image (<NUM>) labeled with the identified attribute, and to add the labeled image to a training database (<NUM>) stored in the memory (<NUM>); and
a training engine (<NUM>) to re-train the model (<NUM>) using the training database (<NUM>);
wherein the data collection module (<NUM>) is further operative to:
receive, via the user interface (<NUM>), a user-identified attribute which changes the identified attribute for the input image (<NUM>);
generate the labeled image based on the input image (<NUM>) labeled with the user-identified attribute;
retrieve a plurality of sample images from the training database (<NUM>), each sample image having a confidence level exceeding a predetermined threshold with respect to the user-identified attribute; and
provide each sample image for the user to label to thereby generate labeled images for the training database (<NUM>).