Patent ID: 12217390

DETAILED DESCRIPTION OF EXAMPLE EMBODIMENTS

According to a first aspect of the present disclosure, a device for correcting an image includes a memory storing one or more instructions; and a processor configured to execute the one or more instructions stored in the memory, wherein the processor, by executing the one or more instructions, is further configured to obtain an image including a plurality of objects, identify the plurality of objects in the image based on a result of using one or more neural networks, determine a plurality of correction filters respectively corresponding to the plurality of identified objects, and correct the plurality of objects in the image, respectively, by using the plurality of determined correction filters.

The processor, by executing the one or more instructions, may be further configured to identify the plurality of objects and determine the plurality of correction filters, based on a training result of using a predetermined plurality of image attributes used for identifying the plurality of objects and a predetermined plurality of display attributes of objects used for determining the plurality of correction filters.

Further, the plurality of image attributes and the display attributes of the plurality of objects may be dynamically selected by the neural network.

Also, the processor, by executing one or more instructions, may obtain category information of each of the plurality of objects in the image in response to an input of the image into the neural network, and obtain information about a plurality of correction filters respectively corresponding to the plurality of objects in response to an input of the category information of each of the plurality of objects to the neural network.

The neural network may also include a plurality of layers for identifying the plurality of objects, and the processor, by executing the one or more instructions, may detect image attributes included in the image from the image using the plurality of layers for identifying the plurality of objects, and determine a category corresponding to each of the plurality of objects based on the detected image attributes.

The neural network may also include a plurality of layers for determining a plurality of correction filters, and the processor, by executing the one or more instructions, may detect display attributes of the object respectively corresponding to the plurality of objects using the plurality of layers for determining the plurality of correction filters, and determine the plurality of correction filters respectively corresponding to the plurality of objects based on the detected display attributes of the object.

The processor, by executing the one or more instructions, may be further configured to learn a criterion for identifying the plurality of objects and determining the plurality of correction filters, by using a first network for determining a category corresponding to each of the plurality of objects in the neural network and a second network for determining the plurality of correction filters respectively corresponding to the plurality of objects.

The device may further include a user interface configured to receive a user input, wherein the processor, by executing the one or more instructions, is further configured to control a display to display the image on a screen of the device, control the user interface to receive the user input that touches one of the plurality of objects in the displayed image, and obtain a correction filter corresponding to an object selected from the plurality of objects when location information touched by the user input is input to the neural network.

The processor, by executing one or more instructions, may also recommend a determined correction filter.

The processor, by executing the one or more instructions, may also control a display to display a preview image representing a corrected image using the recommended correction filter.

According to a second aspect of the present disclosure, a method of correcting an image includes obtaining an image including a plurality of objects; identifying the plurality of objects in the image; determining a plurality of correction filters respectively corresponding to the plurality of identified objects; and correcting the plurality of objects in the image, respectively, by using the plurality of determined correction filters, wherein the identifying of the plurality of objects and the determining of the plurality of correction filters include identifying the plurality of objects and determining the plurality of correction filters based on a result of using one or more neural networks.

According to a third aspect of the present disclosure, a computer-readable recording medium having recorded thereon a program for performing the method of the second aspect in a computer may be provided.

Terms used in this specification will now be briefly described before describing embodiments in detail.

Although the terms used in the following description are selected, as much as possible, from general terms that are widely used at present while taking into consideration the functions obtained in accordance with the embodiments, these terms may be replaced by other terms based on intentions of one of ordinary skill in the art, customs, emergence of new technologies, or the like. In a particular case, terms that are arbitrarily selected by the applicant may be used. In this case, the meanings of these terms may be described in corresponding parts of the embodiments. Therefore, it is noted that the terms used herein is construed based on practical meanings thereof and the whole content of this specification, rather than being simply construed based on names of the terms.

Terms including ordinals such as first, second, etc. may be used to describe various elements, but the elements are not limited by terms. Terms are used only for the purpose of distinguishing one component from another. For example, without departing from the scope of the disclosure, a first component may be referred to as a second component, and similarly, a second component may also be referred to as a first component. As used herein, the term “and/or” includes any and all combinations of one or more of the associated listed items.

It will be understood that the terms “comprises”, “comprising”, “includes” and/or “including”, when used herein, specify the presence of stated elements, but do not preclude the presence or addition of one or more other elements. The term “part” or “module” is used to denote an entity for performing at least one function or operation, and may be embodied as, but is not limited to, a software element or a hardware element such as a field-programmable gate array (FPGA) or an application-specific integrated circuit (ASIC). A “part” or “module” may be configured to exist in an addressable storage medium or to operate one or more processors. Thus, for example, the “part” or “module” includes elements (e.g., software elements, object-oriented software elements, class elements, and task elements), processes, functions, attributes, procedures, subroutines, segments of program code, drivers, firmware, micro-codes, circuits, data, database, data structures, tables, arrays, and parameters. Functions provided in elements and “parts” or “modules” may be combined to a smaller number of elements and “parts” or “modules” or may be divided into a larger number of elements and “parts” or “modules”.

Reference will now be made in detail to embodiments, examples of which are illustrated in the accompanying drawings, wherein like reference numerals refer to like elements throughout. In this regard, the present embodiments may have different forms and should not be construed as being limited to the descriptions set forth herein. In the following description, for clarity, parts or elements that are not related to the embodiments are omitted.

FIG.1is a schematic diagram illustrating an example in which a device10corrects an image according to an embodiment.

Referring toFIG.1, the device10according to an embodiment may correct the image using a neural network12. Here, the neural network12may be a set of algorithms that identify and/or determine objects in an image, using results of statistical machine training to extract and use various attributes within the image. Further, the neural network12may be implemented as software or engine for executing the set of algorithms described above. The neural network12embodied in software, engine, or the like may be executed by a processor in the device10or by a processor in a server (not shown). The neural network may identify objects in an image by abstracting various attributes included in the image input to the neural network12. In this case, abstracting the attributes in the image may be detecting the attributes from the image and determining a key attribute among the detected attributes.

According to an embodiment, the device10may use the neural network12to identify objects in an original image11and determine or recommend a correction filter corresponding to each of the identified objects. Also, the device10may correct the original image11using the determined or recommended correction filter.

According to an embodiment, the device10may identify the objects in the image by abstracting various attributes within the image from the original image11input to the neural network12, and obtain information about the correction filter corresponding to each of the identified objects. The device10may also apply the obtained correction filter to each of the identified objects using the neural network12to obtain a resultant image13.

Meanwhile, the device10may be a smart phone, a tablet PC, a PC, a smart TV, a mobile phone, a personal digital assistant (PDA), a laptop, a media player, a micro server, a global positioning system (GPS) device, an e-book terminal, a digital broadcast terminal, a navigation device, a kiosk, an MP3 player, a digital camera, a consumer electronics, and another mobile or non-mobile computing device but is not limited thereto. Also, the device10may be a wearable device, such as a watch, a pair of glasses, a hair band, and a ring having a communication function and a data processing function.

FIG.2is a flowchart illustrating a method, performed by the device10, of correcting an image using a neural network according to an embodiment.

In operation S210, the device10obtains the image including a plurality of objects.

The device10according to an embodiment may obtain the image stored in the device10. For example, the device10may execute a photo album application or the like to obtain a previously stored image.

According to another embodiment, the device10may receive the image from an external server or an external device. Here, the external server or the external device may include a social network server, a cloud server, a web server, a content providing server, etc. For example, when at least one of a social network service (SNS) application, a web application, a video playback application, and a search application is executed in the device10, the device10may access the external server or the external device to receive the image. According to another example, the device10may receive the image using a short-range wireless network from the external device located around the device10.

According to another embodiment, the device10may obtain a new image through a camera provided in the device10. For example, when a camera application or the like is executed in the device10, the device10may detect an object located around the device10and obtain a preview image. The preview image is an image provided to guide capturing of an object in the camera application or the like, and may be obtained in the device10prior to capturing. Also, the device10may obtain an image (hereinafter, referred to as a captured image) of the object around the device10as a capturing request is input from a user while obtaining the preview image.

On the other hand, the preview image is an image obtained through the camera of the device10in real time, and may have a resolution different from that of the captured image. For example, the resolution of the preview image may be set to HD (High Definition), while the resolution of the captured image may be set to UHD (Ultra High Definition). A method, performed by the device10, of obtaining the image through the camera application will be described in detail later with reference toFIGS.14to16.

In operation S220, the device10identifies a plurality of objects in the image. The device10may use the neural network12to identify the objects in the image from the image. For example, the device10may obtain output data such as a location (e.g., coordinates of pixels corresponding to the object) of each of the objects included in the image and/or a category of each of the objects, etc. as a result of applying the image to the neural network12as input data.

Specifically, the device10may use the neural network12to detect predetermined image attributes in the image and to determine, based on the detected image attributes, the location of each of the objects included in the image and/or the category of each of the objects. Here, an image attribute may represent an image intrinsic feature that may be applied as an input parameter of a neural network for identifying an object. For example, the image attribute may include a color, an edge, a polygon, a saturation, a brightness, a color temperature, a blur, a sharpness, a contrast, and the like, but the image attribute is not limited thereto.

Further, in operation S230, the device10determines a plurality of correction filters respectively corresponding to the plurality of identified objects.

The device10according to an embodiment may determine the correction filters respectively corresponding to the identified objects based on a result of analyzing a display attribute of each of the identified objects using the neural network12described above.

Specifically, the device10may detect display attributes of the object from an image region corresponding to the identified objects based on the result of using the neural network12, and based on the detected display attributes of the object to determine corresponding correction filters. Here, the display attribute of the object may be an attribute of an object detected from the image to determine the correction filter corresponding to the object, and may include at least one of a color, an edge, a polygon, a saturation, a brightness, a color temperature, a blur, a sharpness, and a contrast.

On the other hand, that the device10determines the correction filter using the neural network12may be that the device100may obtain a type of a correction filter to be applied to each of the objects in the image and/or a parameter to be input to the correction filter. To this end, the device10may further train the neural network12such that the neural network12described above determines the correction filter corresponding to each of the identified objects. For example, the device10may train the neural network12by repeatedly performing an operation of analyzing and/or evaluating a result of supervised learning and/or unsupervised learning on the display attribute of the object corresponding to the category of the object in the neural network12.

On the other hand, the correction filter may be a hardware component and/or a software module included in the device10for changing the display attribute of the objects in the image. For example, the correction filter may include a color correction filter, a UV filter, a color temperature conversion filter, a sharpness enhancement filter, a blur filter, and the like.

However, this is only an example, and according to another example, the device10may determine the correction filter corresponding to each of the identified objects, based on information about a predetermined correction filter for each category of the object. For example, when the object included in the image is identified as a ‘dog’, the device10may determine a first correction filter as a correction filter for the ‘dog’, according to the information about the predetermined correction filter.

In operation S240, the device10uses the plurality of determined correction filters to correct each of the plurality of objects in the image. The device10may apply different correction filters with respect to each of the objects in the image based on locations of the objects in the image output from the neural network12and information relating to the correction filter corresponding to each of the objects.

For example, the device10may obtain a location of a first object in the image output from the neural network12, a location of a second object, and information about correction filters corresponding to the first object (and/or parameters input to the correction filters), and correction filters corresponding to the second object. Here, the information about the correction filter may include information about at least one of a type of the correction filter, a type of a parameter applied to the correction filter, and a value of the parameter. Accordingly, the device10may apply correction filters corresponding to the first object to a region corresponding to the first object in the image, and apply correction filters corresponding to the second object to a region corresponding to the second object in the image. At this time, the correction filters corresponding to the first and second objects may be different from each other. Also, even when the correction filters corresponding to the first and second objects are the same, different parameters may be input to the correction filters. As such, the device10may perform an appropriate correction on each of the objects included in the image by correcting the image region corresponding to each of the objects in the image.

Also, the device10may display an image on which a correction operation is completed on a screen of the device10.

The device10according to an embodiment may identify each of the objects included in the image and determine the correction filter suitable for each of the identified objects to perform correction, thereby further improving quality of the image compared to uniformly applying the correction filter to the entire image. In particular, the device10may apply a specific filter determined to be appropriate for a specific object to the specific object, rather than to all objects in the image, thereby improving the quality of the image without changing attributes of the other object.

FIG.3is a diagram illustrating a configuration of the device10according to an embodiment.

Referring toFIG.3, the device10includes a display310, a memory320, and a processor330.

The display310may display information processed in the processor330under the control of the processor330. The display310may output image data processed by the processor330to a screen of the device10. Also, the display310may output various graphic user interfaces (GUIs) to the screen of the device10under the control of the processor330.

The memory320may store various data, programs, or applications for driving and controlling the device10. The programs stored in the memory320may include one or more instructions. The programs (one or more instructions) or the applications stored in the memory320may be executed by the processor330.

The memory320according to an embodiment may include one or more instructions constituting a neural network. The memory320may also include one or more instructions to control the neural network. The neural network may include a plurality of layers including one or more instructions for identifying and/or determining objects in an image from an input image, and a plurality of layers including one or more instructions for determining a correction filter corresponding to each of identified objects from the input image. A detailed structure of the memory320will be described with reference toFIG.31.

The processor330may execute an operating system (OS) and various applications stored in the memory320. The processor330may include one or more processors including a single core, a dual core, a triple core, a quad core, and multiple cores thereof. Also, for example, the processor330may be implemented as a main processor (not shown) and a sub processor (not shown) operating in a sleep mode.

The processor330according to an embodiment obtains an image. For example, the processor330may read an image stored in the memory320by executing a photo album application or the like. Alternatively, the processor330may receive an image from an external server (for example, an SNS server, a cloud server, a content providing server, or the like) through a communicator (not shown). Alternatively, the processor330may obtain a preview image and a captured image through a camera (not shown).

The processor330according to an embodiment may use the instructions constituting the neural network stored in the memory320to identify objects in the image and determine correction filters respectively corresponding to the objects. Thereby, the processor330may perform image correction that is most suitable for each of the objects in the image.

On the other hand, a method, performed by the processor330, of identifying an object in the image and determining a correction filter with respect to the identified object may correspond to operations S220to S230described above with reference toFIG.2. Also, the processor330may control the memory320to store the corrected image.

FIG.4is a diagram showing a configuration of a device400according to another embodiment.

Referring toFIG.4, the device400includes a first processor410, a display420, a memory430, and a second processor440. However, all illustrated components are not indispensable components. The device400may be implemented by more components than the illustrated components, and the device400may be implemented by fewer components than the illustrated components. For example, the device400may further include a camera (not shown) and a communicator (not shown) in addition to the first processor410, the display420, the memory430, and the second processor440.

The first processor410may control execution of at least one application installed in the device400and may perform graphical processing on an image obtained by the device400. The first processor410may be implemented as a system on chip (SoC) in which functions of a central processing unit (CPU), a graphics processing unit (GPU), a communication chip, a sensor, etc. are integrated. Also, the first processor410may be described as an application processor (AP) within the specification.

The display420and the memory430may correspond to the display310and the memory320described above with reference toFIG.3respectively.

The second processor440may use the neural network12to identify objects in the image from the image. Also, the second processor440according to an embodiment may use the neural network12to determine correction filters respectively corresponding to the identified objects.

Meanwhile, the second processor440may be manufactured as a dedicated hardware chip for artificial intelligence (AI) that performs functions of object identification and correction filter determination using the neural network12.

According to various embodiments of the present disclosure, functions performed by the first processor410may be correspondingly performed by the applications stored in the memory430and performing various functions, and functions performed by the second processor440may be correspondingly performed by an OS of the device400.

For example, when a camera application generates an image and transmits the image to the OS, the OS may perform a series of processes to identify the objects included in the image and determine and apply correction filters to each of the identified objects.

According to an embodiment, the first processor410and the second processor440may cooperate with each another such that the processor330described above with reference toFIG.3may perform a series of processes to identify a plurality of objects included in the image and apply a correction filter to each of the identified objects. This will be described later in detail with reference toFIGS.5to20.

FIG.5is a flowchart illustrating a method, performed by the first processor410and the second processor440, of correcting an object in an image obtained upon capturing, according to an embodiment.

In operation S510, the first processor410may display a preview image through the display420.

The first processor410may display the preview image including at least one object sensed by the device400through the display420as a camera application is executed based on a user input. However, this is an embodiment only, and the first processor410may display the preview image through the display420even when another application accompanying a capturing function is executed in addition to the camera application. Here, also, a resolution of the preview image may be determined according to a resolution set on the display420of the device400.

In operation S520, the first processor410may capture at least one object through a camera (not shown) to obtain an image.

The first processor410may capture the image including the at least one object sensed at the time when a capturing request is received through the camera (not shown) when the capturing request is received. The captured image may be an image encoded according to a resolution predetermined in the camera.

In operation S530, the first processor410may store the obtained image through the memory430.

In operation S540, the first processor410may request the second processor440to identify the at least one object included in the stored image.

For example, the first processor410may transmit an object identification request signal including an image stored in the second processor440to the second processor440. According to another example, the first processor410may transmit an object identification request signal including an identifier of a location of the memory430in which the image is stored to the second processor440.

In operation S550, the second processor440may use a neural network to identify at least one object included in the stored image. Here, operation S550may correspond to operation S220described above with reference toFIG.2.

In operation S560, the second processor440may determine a correction filter to apply to each of the identified at least one object. Here, operation S560may correspond to operation S230described above with reference toFIG.2.

In operation S570, the second processor440may transmit to the first processor410correction filter information determined for each of the identified at least one object.

The correction filter information according to an embodiment may include information about location information (for example, pixel coordinates) in which the identified object is displayed on the image, a type of a correction filter corresponding to the identified object, and a parameter value in applying the correction filter, etc.

In operation S580, the first processor410may apply the correction filter to each of the at least one object included in the image. Here, operation S580may correspond to operation S240described above with reference toFIG.2.

In operation S590, the first processor410may store the image to which the correction filter is applied through the memory430.

According to an embodiment, the first processor410may store an image before the correction filter is applied in a specific field region set in the format of the image to which the correction filter is applied. For example, when the image to which the correction filter is applied is stored in the JPEG file format, the first processor410may store the image before the correction filter is applied in a maker note region of a header of the image to which the correction filter is applied. According to another example, the first processor410may store in a maker note a link that may be connected with a region of the memory320where image before the correction filter is applied is stored or identification information of the region of the memory320.

Meanwhile, according to another embodiment, the first processor410may remove the image before the correction filter is applied from the memory430as the image to which the correction filter is applied is stored.

In operation S595, the first processor410may display the image to which the correction filter is applied through the display420. For example, the first processor410may display the image to which the correction filter is applied through the display420, as a photo album application displaying a captured or stored image based on a user input is executed.

FIG.6is a diagram illustrating information included in an object identification request600transmitted from the first processor410to the second processor420for identification of an object included in an image according to an embodiment.

An interface for transmitting/receiving data (e.g., an image to which a correction filter is applied) between the first processor410and the second processor420may be defined for the object identification request600.

For example, an application program interface (API) function having data to be input to a neural network as a transfer value may be defined. In this case, when the first processor410calls the API function and inputs an image before correction and a request purpose as the data transfer value, the API function may transfer the age before correction and the request purpose as the data to be input to the neural network to the second processor420.

Referring toFIG.6, the object identification request600may include an image610and information620about the request purpose. In the present embodiment, the object identification request600includes the image610. However, this is an example only, and an identifier of a location of a memory in which the image610is stored may be included in the object identification request600.

The information620about the request purpose may include a type622of the request purpose indicating a request target and information about a response condition624. In the present embodiment, the type622of the request purpose may be set to object recognition, and the response condition624may be set such that a maximum number of recognized objects is limited to 10.

FIG.7is a flowchart for explaining a method, performed by the first processor410and the second processor440, of correcting an object in an image obtained upon capturing, according to another embodiment.

InFIG.7, operations S710to S750may correspond to operations S510to S550described above with reference toFIG.5respectively.

On the other hand, in operation S760, the second processor440may transmit information about at least one identified object to the first processor410.

According to an embodiment, the second processor440may transmit to the first processor410information including a location of the identified object in the image and identification information of the object corresponding to the location. For example, the second processor440may transmit to the first processor410information informing that a ‘dog’ object is included in a region between [x1, y1] and [x2, y2] in the image.

In operation S770, the first processor410may determine a correction filter to apply to each of the at least one identified object, based on the information about the at least one identified object. Here, operation S770may correspond to operation S560described above with reference toFIG.5.

Also, operations S780and s790may correspond to operations S580and S590described above with reference toFIG.5respectively.

In operation S795, the first processor410may display an image to which the correction filter is applied through the display420. For example, the first processor410may display the image to which the correction filter is applied through the display420, as a photo album application displaying a captured or stored image based on the user input is executed.

FIG.8is a flowchart illustrating a method, performed by the device10, of recommending a correction filter that may be applied to each of objects included in an image, according to an embodiment.

In operation S810, the device10may recommend, for each of the objects in the image, determined correction filters according to a training result of using the neural network12. For example, the device10may display a list of the correction filters respectively corresponding to the objects in the image determined using the neural network12in a region associated with a corresponding object on the image.

Alternatively, the device10may display correction filters for correcting an object corresponding to a location touched by a user input in the image on a region associated with a selected object on the image. For example, the device10may display a list of correction filters corresponding to at least one selected object in response to a user input touching at least one object in the image.

In operation S820, the device10may display a preview image representing a corrected image using the recommended correction filters.

According to an embodiment, the device10may display at least one of a first preview image (e.g., a preview image in which a first correction filter is applied to a ‘dog’ object ofFIG.11, a second correction filter is applied to a ‘flower’ object, a third correction filter is applied to a ‘pizza’ object, and the second correction filter is applied to a ‘waffle’ object) in which at least one of the corresponding recommended correction filters is applied to all the objects included in the image, a second preview image (e.g., a preview image in which the first correction filter is applied to the ‘dog’ object ofFIG.11) in which at least one of the corresponding recommended correction filters is applied to at least one of the objects included in the image, and a third preview image in which all the corresponding recommended correction filters are applied to all the objects.

At this time, the second preview image is that at least one of preferred correction filters is applied at least one object preferred by a user among the objects included in the image, based on a previous correction history of the device10.

In operation S830, the device10may store the corrected image in response to a user input. The device10may store the corrected image corresponding to a selected preview image, in response to a user input selecting one of the first through third preview images. Further, the device10may store information about the correction filter applied to the corrected image together with or in association with the corrected image, and may store the corrected image and an original image in association with each other.

For example, the device10may store at least one of information about a correction filter applied to a specific field region set in a format of the corrected image, the original image, and an address where the original image is stored.

On the other hand, operation S820may be omitted. In this case, the device10may store an image to which the correction filters selected by the user input are applied, in response to the user input for the correction filters recommended in operation S810.

FIG.9is a flowchart illustrating a method, performed by the first processor410and the second processor440, of recommending a correction filter that may be applied to each of objects included in an image, according to an embodiment.

InFIG.9, operations S905to S925may correspond to operations S510to S550described above with reference toFIG.5respectively.

Meanwhile, in operation S930, the second processor440may select at least one or more correction filter candidates to apply to an identified object. For example, the second processor440may select at least one correction filter that is determined to be suitable to apply to the identified object, according to a result of analyzing a display attribute of the identified object using a neural network. Here, suitability of the correction filter for the identified object may be calculated as a probability value through the neural network. The second processor440may select at least one or more correction filters whose probability values are equal to or greater than a predetermined threshold value as the correction filter candidates.

In operation S935, the second processor440may display the correction filter candidate to be applied to each of at least one object included in the image through a display.

For example, the second processor440may display a correction filter candidate list on a location (e.g., within a predetermined distance range) adjacent to the identified object. Also, the second processor440may display a corrected preview image in which the correction filter is applied to each of the at least one object included in the image.

In operation S940, the second processor440may determine a correction filter to apply to the identified object. For example, the second processor440may select, from the at least one or more correction filter candidates, a correction filter selected by the user as the correction filter to apply to the identified object.

Operations S945to S960may correspond to operations S570to S595described above with reference toFIG.5.

FIG.10is a flowchart illustrating a method, performed by the first processor410and the second processor440, of recommending a correction filter that may be applied to each of objects included in an image, according to an embodiment.

InFIG.10, operations S1005to S1030may correspond to operations S905to S930described above with reference toFIG.9.

On the other hand, in operation S1035, the second processor440may transmit information about the at least one correction filter candidate to the first processor410.

The information about the correction filter candidate according to an embodiment may include location information (for example, pixel coordinates) indicating an identified object on the image, and information about a type of a correction filter candidate corresponding to the identified object.

In operation S1040, the first processor410may display a correction filter candidate to be applied to each of at least one or more objects included in the image through a display.

For example, the first processor410may display a correction filter candidate list on a location (e.g., within a predetermined distance range) adjacent to the identified object. Also, the first processor410may display a correction preview image in which a correction filter is applied to each of the at least one or more objects included in the image.

In operation S1045, the first processor410may determine a correction filter to apply to the identified object. For example, the first processor410may select, from the at least one correction filter candidate, a correction filter selected by a user as the correction filter to apply to the identified object.

Operations S1050and S1060may correspond to operations S950and S960described above with reference toFIG.9.

FIG.11is an example in which the device10recommends correction filters respectively corresponding to objects in an image1110.

Referring toFIG.11, the device10may display correction filter lists1111to1114respectively corresponding to the objects in the image1110, in order to recommend objects identified based on a result of using the neural network12and correction filters respectively corresponding to the identified objects.

At this time, the device10may display an indicator (shown by an arrow inFIG.11) indicating the identified object corresponding to the correction filter lists1111to1114. The device10may also display information indicating a recognition result of each of the identified objects together with the indicator. When the recognition result is wrong in view of a user, the device10may provide an input interface for receiving a user command for re-recognition. For example, when the identified object or identified object recognition result information is touched, the device10may provide a re-recognition result through re-recognition.

According to an embodiment, the device10may receive a user input with respect to at least one of the correction filter lists1111through1114. The device10may perform image correction corresponding to the selected correction filter lists1111to1114in response to the user input. For example, in response to a user input touching the correction filter list1111corresponding to a ‘dog’ in the image1110, the device10may apply a first correction filter and/or a third correction filter to an image region1120on which the ‘dog’ is displayed, thereby generating and/or storing a corrected image.

The device10according to an embodiment may select and apply any one of a plurality of filters determined to be appropriate for the identified object based on a user input, thereby providing information about an appropriate correction filter to an object as well as generating an image in which a quality improvement effect of the image desired by the user is maximized.

FIG.12is an example in which the device10displays preview images1203through1205.

Referring toFIG.12, the device10may display the preview images1203through1205obtained as a result of applying correction filters determined based on a result using of the neural network12together with an uncorrected image1201. Therefore, a user may easily identify an image change before and after correction.

For example, the device10may display a first preview image1203in which correction filters corresponding to objects included in the image1201are applied to the image1201, a second preview image1204in which correction filters are applied to an object (e.g. a ‘dog’) preferred by the user according to a previous correction history, and a third preview image1205in which all correction filters are applied to all the objects included in the image1201.

On the other hand, the device10may receive a user input that slides a first region1202on which a plurality of preview images are displayed so as to display the preview images. The device10may display a preview image in which different objects in the image1201are corrected, in response to the user input that slides the first region1202.

According to an embodiment, the device10may emphasize (e.g., an object flicker, a boundary enhancement of the object) and display an object to which the correction filters are applied in the preview images1203through1205.

According to an embodiment, the device10may change a size at which the preview images1203through1205are displayed on a screen, in response to a user input1206dragging and dropping a boundary of the preview images1203through1205.

FIG.13is a diagram for explaining a method, performed by the first processor410and the second processor440, of applying a correction filter to an object included in a preview image in real time, according to an embodiment.

In operation S1310, the first processor410may obtain the preview image through a camera (not shown).

The first processor410may obtain the preview image including at least one object sensed by the device400through the camera (not shown) as a camera application is executed. Here, the first processor410may obtain the preview image in real time according to a predetermined frame rate. For example, when the frame rate is set to 30 fps, the first processor410may obtain 30 preview images per second.

On the other hand, when a user requests a correction of the preview image, the first processor410may change the frame rate in consideration of a time required in a series of correction processes to be described later. For example, the first processor410may change the frame rate from 30 fps to 24 fps.

In operation S1320, the first processor410may request the second processor440to identify at least one object included in the preview image.

For example, the first processor410may transmit an object identification request signal including the preview image to the second processor440to the second processor440.

In operation S1330, the second processor440may use a neural network to identify the at least one object included in the preview image. Here, operation S1330may correspond to a method of identifying at least one object included in an image in operation S220described above with reference toFIG.2.

In operation S1340, the second processor440may determine the correction filter to apply to each of the identified at least one object.

Meanwhile, according to another example, the second processor440may determine at least one correction filter candidate to apply to each of the identified at least one object.

In operation S1350, the second processor440may transmit to the first processor410correction filter information determined for each of the identified at least one object.

On the other hand, according to another example, in operation S1340, when the at least one correction filter candidate is determined, the second processor440may transmit information about the at least one correction filter candidate to the first processor410.

In operation S1360, the first processor410may apply the correction filter to the at least one object included in the preview image based on the received correction filter information.

On the other hand, according to another example, when the first processor410receives the information about the at least one correction filter candidate from the second processor440, the first processor410may select any one of the at least one correction filter candidate, based on a user input.

In operation S1370, the first processor410may display the preview image to which the correction filter is applied through a display. The first processor410according to an embodiment may display an image to which the correction filter is applied according to the predetermined frame rate in real time.

FIG.14is a diagram for explaining a method, performed by the first processor410and the second processor440, of applying a correction filter to an object included in a preview image in real time, according to another embodiment.

InFIG.14, operations S1410to S1430may correspond to operations S1310to S1330described above with reference toFIG.13.

On the other hand, in operation S1440, the second processor440may transmit information about an identified at least one object to the first processor410.

The information about the identified at least one object according to an embodiment may include information about a location of the object in an image and identification information of the object.

In operation S1450, the first processor410may determine a correction filter to apply to each identified at least one object.

Meanwhile, according to another example, the second processor440may determine at least one correction filter candidate to apply to each identified at least one object.

Operations S1460and S1470may correspond to operations S1360and S1370described above with reference toFIG.13.

FIG.15is a diagram for explaining a method, performed by the device10, of applying a correction filter to each of a plurality of objects1511,1512,1513, and1514included in a preview image1510in real time.

Referring toFIG.15, the device10may identify the plurality of preview images1511,1512,1513, and1514included in the preview image1510sensed by the device10as an application for obtaining the preview image1510in real time, such as a camera application installed in the device10is executed. For example, the device10may identify a plurality of objects included in an image that is visible on a screen before capturing a moving image or taking a photograph. Also, the device10may apply the correction filter to a region corresponding to each of the plurality of objects1511,1512,1513, and1514identified from the preview image1510to display the plurality of objects1511,1512,1513, and1514on the screen. The device10may apply the correction filter corresponding to each of the plurality of identified objects1511,1512,1513and1514, thereby improving the quality of an image obtained in real time.

Also, the device10may store a preview image1520to which the correction filter is applied when a user input to request capturing a moving image or taking a photograph is received.

Meanwhile, in the present embodiment, the method of applying the correction filter to each of the plurality of objects1511,1512,1513, and1514included in the preview image1510in real time may correspond to a method described above with reference toFIG.2.

FIG.16is a diagram for explaining a method, performed by the first processor410and the second processor440, of applying a correction filter to an object included in an image stored in the memory430, according to an embodiment.

In operation S1610, the first processor410may display the image stored in the memory430through a display.

The first processor410may receive a user input to request execution of a photo album application. As the user input is received, the first processor410may display a list of at least one image stored in the memory430through the display. Also, as a specific image is selected from the list of displayed at least one image, the first processor410may display the selected image.

In operation S1620, the first processor410may request the second processor440to identify the at least one object included in the stored image.

For example, the first processor410may transmit an object identification request signal including the stored image to the second processor440to the second processor440. According to another example, the first processor410may transmit an object identification request signal including an identifier for a location of the memory430in which the image is stored to the second processor440.

On the other hand, operations S1630to S1660may correspond to operations S550to S580described above with reference toFIG.5respectively.

In operation S1670, the first processor410may store the image to which the correction filter is applied through the memory430.

In operation S1680, the first processor410may display the image to which the correction filter is applied through the display.

For example, the first processor410may display the image to which a correction filter is applied using a photo album application that is being executed.

According to another example, the first processor410may display the image to which the correction filter is applied using an image editing application when a user input to request execution of the image editing application is received. For example, the first processor410may display the image to which the correction filter is applied, which is stored in the memory430, on an image list provided in the image editing application.

FIG.17is a diagram for explaining a method, performed by the first processor410and the second processor440, of applying a correction filter to an object included in an image stored in the memory430, according to another embodiment.

InFIG.17, operations S1710to S1730may correspond to operations S1610to S1630described above with reference toFIG.16.

In operation S1740, the second processor440may transmit information about identified at least one object to the first processor410. According to an embodiment, the second processor440may transmit information including a location of the identified object in the image and identity information of the object corresponding to the location to the first processor410.

In operation S1750, the first processor410may determine the correction filter with respect to each identified at least one object. On the other hand, operation S1750may correspond to a method of determining a correction filter in operation S230described above with reference toFIG.2.

On the other hand, operations S1760to S1780may correspond to operations S1660to S1680described above with reference toFIG.16.

FIG.18is a flowchart illustrating a method, performed by the device10, of correcting an object in an image in response to a user input, according to an embodiment.

Referring toFIG.18, in operation S1810, the device10may display the image. The device10may display an uncorrected image.

In operation S1820, the device10may receive the user input that touches one of objects in the image. However, a method of selecting one of the objects in the image is not limited to a touch. For example, when the device10provides a separate input apparatus, a user may select one of the objects in the image using the provided input apparatus.

In operation S1830, the device10may determine correction filters corresponding to the selected object in the input image as location information of the image touched by the user input and the image are input to the neural network12. Also, the device10may identify the object in the image corresponding to the location information and determine a correction filter corresponding to the identified object, based on a result of using the neural network12.

Alternatively, the device10may extract image information of a partial region corresponding to a location touched by the user from the image and input only image information of the extracted image region to the neural network12. Here, the image information may be image data related to at least partial region of the image.

According to an embodiment, the device10may identify the object selected by the user and apply the correction filter corresponding to the identified object, thereby selectively correcting an object wanted by the user, without analyzing the entire image using the neural network12, and thus an effect of improving the image quality may be maximized while reducing consumption of resources (for example, computing resources) used for object identification.

FIG.19is a diagram illustrating information included in an object identification request1900transmitted from the first processor410to the second processor440in an image for identification of an object selected based on a user input, according to an embodiment.

The first processor410according to an embodiment may display the image through the display420as at least one application including a photo album application, a camera application, an image editing application, and a web application is executed. The first processor410may receive a user input that selects an object in the displayed image. Also, the first processor410may transmit the identification request1900for the selected object to the second processor440.

An interface for transmitting/receiving data (e.g., an image before applying a correction filter) between the first processor410and the second processor420may be defined for the object identification request1900.

For example, an application program interface (API) function having data to be input to a neural network as a transfer value may be defined. In this case, when the first processor410calls the API function and inputs an image before correction and a request purpose as the data transfer value, the API function may transfer the image before correction and the request purpose to the second processor420as the data to be input to the neural network.

Referring toFIG.19, the identification request1900for the selected object may include information1910about a correction target and information1920about the request purpose. The information1910about the correction target may include information about an image1912and a region1914of a selected object in the image1912. Also, the information1920about the request purpose may include information indicating that the request purpose is object recognition.

FIG.20is an example in which the device10corrects an object in an image in response to a user input.

Referring toFIG.20, the device10may input an image2010displayed on a screen and first location information (i.e., a location {x1, y1} or a predetermined region such as {xn, yn} from {x1, y1}) touched by a user input to the neural network12. The device10may obtain, from the neural network12, information about an object corresponding to the first location information and a first correction filter to be applied to the object, a parameter value P1 to be input to the first correction filter, a second correction filter, and a parameter value P2 to be input to the second correction filter.

The device10may correct the image2010based on the values obtained from the neural network12. Thus, the device10may perform image correction only on a ‘dog’ touched by the user input in the image2010. As image correction is performed, the device10may display a corrected image2020in which a region corresponding to the ‘dog’ is corrected through a display.

FIG.21is a flowchart illustrating a method, performed by the device10, of determining a correction filter with respect to an object related to a user among a plurality of objects identified based on user information.

In operation S2110, the device10may identify the plurality of objects in an image using a neural network.

Meanwhile, operation S2110may correspond to operation S230described above with reference toFIG.2.

In operation S2120, the device10may select the object related to the user among the plurality of identified objects based on the previously stored user information.

Here, the user information represents information that may identify the user and the object related to the user. For example, an image of the user, an image of a family or friend of the user, an image of a user's pet, etc. may be included in the user information. The device10may also obtain the user information through a memory of the device10or through at least one application installed in the device10. For example, the device10may obtain the user information through a photo album application or obtain the user information stored in an external server of an SNS application.

The device10may compare the plurality of identified objects with the user information to select the object related to the user among the plurality of identified objects. For example, when a plurality of persons are included in the image, the device10may compare the user information with the plurality of persons to select the user and a friend of the user from the plurality of persons.

In operation S2130, the device10may determine a correction filter corresponding to the selected object.

On the other hand, the method performed by the device10of determining the correction filter corresponding to the selected object may correspond to a method described above with reference to S230ofFIG.2. The device10according to an embodiment may perform correction only on the object related to the user without correcting all the objects included in the image, thereby more efficiently using resources such as a time and memory required for correction.

FIG.22is a diagram illustrating information included in an object identification request2200transmitted from the first processor410to the second processor440for identification of an object related to a user in an image, according to an embodiment.

An interface for transmitting/receiving data (e.g., an image before applying a correction filter) between the first processor410and the second processor420may be defined for the object identification request2200.

For example, an application program interface (API) function having data to be input to a neural network as a transfer value may be defined. In this case, when the first processor410calls the API function and inputs an image before correction and a request purpose as the data transfer value, the API function may transfer the image before correction and the request purpose to the second processor420as the data to be input to the neural network.

Referring toFIG.22, the object identification request2200may include an image2210and information2220about the request purpose. In the embodiment, the object identification request2200includes the image2210. However, this is a merely embodiment, and an identifier for a location of a memory in which the image2210is stored may be stored in the object identification request2200.

Also, the information2220about the request purpose may include a type2222of the request purpose indicating a request target and information about a response condition2224. In the embodiment, the type2222of the request purpose may be set to object recognition, and the response condition2224may be set to recognize the object related to the user and be limited to 10 as the maximum number of recognized objects.

FIG.23is a diagram illustrating a method, performed by the device10, of determining a correction filter for an object related to a user among a plurality of objects identified based on user information.

Referring toFIG.23, the device10may identify a plurality of objects2311,2312, and2313in an image2310using a neural network. Specifically, the device10may identify the cherry tree2311, the person 12312and the person 22313included in the image2310using the neural network.

The device10according to an embodiment may obtain previously stored user information from a server associated with a memory or an application. The device10may, for example, obtain a plurality of photos stored in a photo book application.

The device10may compare the identified person 12312and person 22313with a plurality of previously obtained photos as the plurality of persons2312and2313are included in the image2310. The device10may determine that the person 12312is a user of the device10and the person 22313is a person who is not related to the user as a result of comparison.

Accordingly, the device10may select the cherry tree2311and the person 12312from among the plurality of objects2311,2312, and2313. The device10may determine a correction filter corresponding to each of the selected cherry tree2311and the person 12312and correct the cherry tree2311and the person 12312included in the image2310using the determined correction filter.

FIG.24is a flowchart illustrating a method, performed by the device10, of determining a correction filter corresponding to an object based on history information about a correction filter of an object.

In operation S2410, the device10may identify a plurality of objects in an image using a neural network.

On the other hand, operation S2410may correspond to operation S230described above with reference toFIG.2.

In operation S2420, the device10may obtain the history information indicating the correction filter set by a user with respect to each of the plurality of objects before a correction of the image.

The device10may store information of the correction filter set with respect to each of the plurality of objects identified from at least one image obtained before the correction of the image. For example, the device10may store a correction filter corresponding to a first object as a first correction filter when the user selects the first correction filter for the first object.

The device10may also store information about the number of times of setting for each type of correction filters when there is a history in which the user has set various types of correction filters with respect to the first object.

In operation S2430, the device10may determine the correction filter corresponding to each of the plurality of objects based on the obtained history information.

For example, when the device10has selected the correction filter selected by the user with respect to the first object in the history information as the first correction filter, the device10may determine the correction filter corresponding to the first object among the plurality of objects as the first correction filter.

Also, when there is a history in which the user has set various types of correction filters with respect to the first object, the device10may weight the number of times of settings for each correction filter based on the history to set the correction filter corresponding to the first object.

The device10according to an embodiment may set the correction filter with respect to the object by reflecting a user's taste or the like using the history information indicating the correction filter set by the user with respect to the object.

FIG.25is a flowchart illustrating a method, performed by the device10, of determining a correction filter corresponding to an object based on history information about a correction filter of an object.

Referring toFIG.25, the device10may obtain history information2520in an image2510to determine the correction filter corresponding to each of a plurality of identified objects based on a neural network. Here, the history information2520may be generated from a correction result of at least one image that has been processed before an image correction. Specifically, the history information2520may include information of a correction filter set by a user with respect to each of the plurality of objects included in the at least one image processed before the image correction.

For example, the history information2520may include information indicating that a first correction filter, a second correction filter, and a third correction filter are set to animal2511, plant2514, and food2512and2513, respectively. Accordingly, the device10may determine correction filters of the animal2511, the food2512and2513, and the plant2514included in the image2510as the first correction filter, the third correction filter, and a second correction filter.

FIG.26is a diagram for illustrating a method, performed by the device10, of applying a correction filter to each of a plurality of objects2611and2612included in an image2610using a subject of content including the image2610.

Referring toFIG.26, the device10may identify the subject of the content in which the image2610is included. Here, the content may include moving images, photos, and the like, but a type of the content is not limited to the above-described examples.

The device10may use the image2610or metadata of the content to identify a title of the content, a channel providing the content, or the like. Also, the device10may identify the subject of the content based on at least one of the identified title of the content and channel providing the content. For example, the subject of the content may include sports, food, and beauty, but this is merely an example, and the subject of the content is not limited to the example described above. In an embodiment ofFIG.26, the device10may identify the subject of the content as a football game moving image including the image2610.

Meanwhile, the device10may identify the plurality of objects2611and2612included in the image2610using the neural network. For example, the device10may verify that the athlete2611and the billboard2612are included in the image2610.

The device10may determine the correction filter applied to each of the plurality of objects2611and2612based on the identified subject of the content and plurality of objects2611and2612. For example, the device10may select a correction filter A that clarifies an edge of the athlete2611as the image2610is identified as a scene of the football game moving image. Also, according to another example, the device10may select a correction filter B that increases the contrast of the billboard2612.

The device10may generate a corrected image2620by correcting the athlete2611and the billboard2612included in the image2610respectively using the selected correction filter A and the selected correction filter B.

FIG.27is a detailed diagram of the processor330ofFIG.3according to an embodiment.

Referring toFIG.27, the processor330ofFIG.3may include a controller2701and a graphic processing unit2702.

The controller2701may perform an operation for controlling the graphic processing unit2702. The controller2701may include one or more cores (not shown) and a connection path (e.g., a bus) for transmitting and receiving signals with the graphics processing unit2702and/or other components.

The graphic processing unit2702may include an operator2703and a renderer2704. The operator2703may include a plurality of processors that perform operations in parallel. According to an embodiment, the operator2703may process one or more instructions included in layers in a neural network module in parallel, under the control of the controller2701. This will be described later in more detail with reference toFIG.31. Also, the operator2703may provide a processing result (i.e., correction filter information corresponding to each of objects in an image) to the controller2710.

Also, the operator2703may correct attribute values such as coordinate values, shapes, sizes, colors, etc. by which the respective objects are to be displayed according to a layout of a screen. The renderer2704may output a result corrected by the operator2703to a display region of the display310.

Meanwhile, the processor330may further include a random access memory (RAM) (not shown) and a read-only memory (ROM) temporarily and/or permanently storing signals (or data) processed inside the processor330. The processor330may also be implemented as a system-on-chip (SoC) including at least one of the controller2701, the graphic processing unit2702, the RAM, and the ROM described above.

FIG.28is a detailed diagram of the processor330ofFIG.3according to another embodiment.

Referring toFIG.28, the processor330according to some embodiments may include a data training unit2810and a data recognizer2820.

The data training unit2810may obtain data to be used for training and apply the obtained data to a data recognition model that will be described later, thereby learning a determination criterion for identification of an object or determination of a correction filter regarding the object included in an image.

The data training unit2810may learn to have the determination criterion for the data recognition model to identify the object included in the image. For example, the data training unit2810may learn an image attribute used to identify the object and a category according to a type of the object.

The data training unit2810may also train a display attribute of the object such that the data recognition model has the determination criterion for determining a correction filter suitable for the object.

The data recognizer2820may identify the object based on the data or determine the correction filter of the object included in the image. For example, the data recognizer2820may use a training result regarding the image attribute to identify the object included in the image.

According to another example, the data recognizer2820may use a training result regarding the display attribute of the object to determine the correction filter of the identified object. Further, the data recognizer2820may use a result value output by the data recognition model having the obtained data as an input value to update the data recognition model.

At least one of the data training unit2810and the data recognizer2820may be manufactured in the form of at least one hardware chip and mounted on the device10. For example, at least one of the data training unit2810and the data recognizer2820may be manufactured in the form of a dedicated hardware chip for artificial intelligence (AI) or may be manufactured in a part of an existing general-purpose processor (e.g. a CPU or an application processor) or a graphic-only processor (e.g., a GPU) and mounted on the various devices10described above.

In this case, the data training unit2810and the data recognizer2820may be mounted on one device10, or may be mounted on separate devices. For example, one of the data training unit2810and the data recognizer2820may be included in the device10, and the other may be included in a server. The data training unit2810and the data recognizer2820may provide model information constructed by the data training unit2810to the data recognizer2820by wired or wirelessly. Data input to the data recognizer2820may be provided to the data training unit2810as additional training data.

Meanwhile, at least one of the data training unit2810and the data recognizer2820may be implemented as a software module. When at least one of the data training unit2810and the data recognizer2820is implemented as a software module (or a program module including an instruction), the software module may be a stored in non-transitory computer-readable media. Also, in this case, at least one software module may be provided by an operating system (OS) or by a predetermined application. Alternatively, some of the at least one software module may be provided by an operating system (OS), and the others may be provided by a predetermined application.

FIG.29is a block diagram of the data training unit2810according to an embodiment.

Referring toFIG.29, the data training unit2810according to an embodiment may include a data obtainer2910, a preprocessor2920, a training data selector2930, a model training unit2940, and a model evaluator2950. However, this is only an embodiment, and the data training unit2810may include some of the above-described components or may further include other components in addition to the above-described components.

The data obtainer2910may obtain data necessary for identification of an object or determination of a correction filter. For example, the data obtainer2910may obtain data by sensing a peripheral situation of the device10. According to another example, the data obtainer2910may obtain data from an external server such as a social network server, a cloud server, or a content providing server.

The data obtainer2910may obtain at least one of an image and a moving image. Here, the moving image may include a plurality of images. For example, the data obtainer2910may receive the moving image through a camera of the device10including the data training unit2810or an external camera (e.g. a CCTV, a black box, or the like) capable of communicating with the device10including the data training unit2810. Here, the camera may include one or more image sensors (e.g., a front sensor or a rear sensor), a lens, an image signal processor (ISP), or a flash (e.g., LED or xenon lamp, etc.)

The preprocessor2920may preprocess obtained data such that the obtained data may be used for training for identification of the object or determination of the correction filter. The preprocessor2920may process the obtained data to a predetermined format such that the model training unit2940, which will be described later, may use the data obtained for training for identification of the object or determination of the correction filter.

For example, the preprocessor2920may divide an input image into a plurality of images and may detect an R attribute, a G attribute, and a B attribute from each of the divided images. Also, the preprocessor2920may determine a representative attribute value with respect to attributes detected for each region of a predetermined size, from each of the divided images. The representative attribute value may include a maximum attribute value, a minimum attribute value, and an average attribute value.

The training data selector2930may select data required for training from the preprocessed data. The selected data may be provided to the model training unit2940. The training data selector2930may select data necessary for training from among the preprocessed data according to a predetermined selection criterion for identification of the object or determination of the correction filter. The training data selector2930may also select data according to a predetermined selection criterion by training by the model training unit2940, which will be described later.

For example, the training data selector2930may determine, based on the preprocessed data, types of image attributes having a relatively high correlation (for example, a high density of a probability distribution), the number, a level, etc. as data included in a criterion for identifying an object.

The model training unit2940may learn to have a determination criterion as to how a data recognition model determines identification of the object or determination of the correction filter based on training data. Also, the model training unit2940may learn a selection criterion as to which training data should be used for identification of the object or determination of the correction filter.

For example, the model training unit2940may learn to have a first criterion as a determination criterion for identifying objects in a training image. Here, the first criterion may include types, number or levels of image attributes used by the device10to identify an object in the training image from the training image, using the neural network.

According to another example, the model training unit2940may use a neural network to learn to have a second criterion as a criterion for determining the correction filter corresponding to each of the objects in the training image. Here, the second criterion may include types, number or levels of display attributes of the object, etc., that the device10uses to determine the correction filters respectively corresponding to the objects in the training image using the neural network.

Also, the model training unit2940may train the data recognition model used for identification of the object or determination of the correction filter using the training data. In this case, the data recognition model may be a previously constructed model. For example, the data recognition model may be a previously constructed model by receiving basic training data (e.g., a sample image, etc.).

The data recognition model may be constructed considering an application field of a recognition model, the purpose of training, or the computer performance of a device. The data recognition model may be, for example, a model based on a neural network. For example, a model such as Deep Neural Network (DNN), Recurrent Neural Network (RNN), or Bidirectional Recurrent Deep Neural Network (BRDNN) may be used as the data recognition model, but is not limited thereto.

According to various embodiments, when a plurality of previously constructed data recognition models are present, the model training unit2940may determine a data recognition model having a large relation between input training data and basic training data as a data recognition model to learn. In this case, the basic training data may be previously classified according to a type of data, and the data recognition model may be previously constructed for each type of data. For example, the basic training data may be previously classified by various criteria such as a region where the training data is generated, a time at which the training data is generated, a size of the training data, a genre of the training data, a creator of the training data, a type of an object in the training data, etc.

Also, the model training unit2940may train the data recognition model by using, for example, a training algorithm including an error back-propagation method or a gradient descent method.

Also, the model training unit2940may train the data recognition model through, for example, supervised learning using, as an input value, training data for training of the determination criterion. Also, the model training unit2940train the data recognition model through, for example, unsupervised learning that finds the determination criteria for identification of the object or determination of the correction filter by self-learning using the data necessary for identification of the object or determination of the correction filter without any supervising.

Further, the model training unit2940may train the data recognition model through, for example, reinforcement learning using feedback on whether a result of identification of the object or determination of the correction filter is correct based on training.

Further, when the data recognition model is trained, the model training unit2940may store the trained data recognition model. In this case, the model training unit2940may store the trained data recognition model in a memory of the device10including the data training unit2810. Alternatively, the model training unit2940may store the trained data recognition model in a memory of the device10including the data recognizer2820that will be described later. Alternatively, the model training unit2940may store the trained data recognition model in a memory of a server connected to the device10over a wired or wireless network.

In this case, the memory in which the trained data recognition model is stored may also store, for example, commands or data associated with at least one other component of the device10. The memory may also store software and/or program. The program may include, for example, kernel, middleware, an application programming interface (API), and/or an application program (or application), etc.

The model evaluator2950may input evaluation data to the data recognition model, and when a recognition result output from the evaluation data does not satisfy a predetermined criterion, the model evaluator2950may cause the model training unit2940to learn again. In this case, the evaluation data may be predetermined data for evaluating the data recognition model. Here, the evaluation data may include a matching ratio between an object identified based on the data recognition model and an actual object, the image quality of the image after the correction filter is applied, etc.

For example, when the number or the ratio of evaluation data whose recognition result is not correct among recognition results of the trained data recognition model with respect to the evaluation data exceeds a predetermined threshold value, the model evaluator2950may evaluate the trained data recognition model not to satisfy the predetermined criterion. For example, in case where the predetermined criterion is defined as a ratio of 2%, when the trained data recognition model outputs an incorrect recognition result with respect to the evaluation data exceeding 20 among a total of 1000 evaluation data, the model evaluator2950may evaluate the trained data recognition model not to be suitable.

On the other hand, when a plurality of trained data recognition models are present, the model evaluator2950may evaluate whether each of trained moving image recognition models satisfies a predetermined criterion, and may determine a model satisfying the predetermined criterion as a final data recognition model. In this case, when a plurality of models satisfying the predetermined criterion are present, the model evaluator2950may determine any one or a predetermined number of models previously set in the order of higher evaluation scores as the final data recognition model.

Meanwhile, at least one of the data obtainer2910, the preprocessor2920, the training data selector2930, the model training unit2940, and the model evaluator2950in the data training unit2810may be manufactured in the form of at least one hardware chip and mounted on the device10. For example, at least one of the data obtainer2910, the preprocessor2920, the training data selector2930, the model training unit2940, and the model evaluator2950may be manufactured in the form of a dedicated hardware chip for artificial intelligence (AI) or may be manufactured in a part of an existing general-purpose processor (e.g. a CPU or an application processor) or a graphic-only processor (e.g., a GPU) and mounted on the various devices10described above.

In this case, the data obtainer2910, the preprocessor2920, the training data selector2930, the model training unit2940, and the model evaluator2950may be mounted on one device10, or may be mounted on the separate devices10. For example, some of the data obtainer2910, the preprocessor2920, the training data selector2930, the model training unit2940, and the model evaluator2950may be included in the device10, and the others may be included in a server.

Also, at least one of the data the data obtainer2910, the preprocessor2920, the training data selector2930, the model training unit2940, and the model evaluator2950may be implemented as a software module. When at least one of the data obtainer2910, the preprocessor2920, the training data selector2930, the model training unit2940, and the model evaluator2950is implemented as a software module (or a program module including an instruction), the software module may be a stored in non-transitory computer-readable media. Also, in this case, at least one software module may be provided by an operating system (OS) or by a predetermined application. Alternatively, some of the at least one software module may be provided by an operating system (OS), and the others may be provided by a predetermined application.

FIG.30is a block diagram of the data recognizer2820according to an embodiment.

Referring toFIG.30, the data recognizer2820according to an embodiment may include a data obtainer3010, a preprocessor3020, a recognition data selector3030, a recognition result provider3040, and a model updater3050. However, this is only an embodiment, and the data recognizer2820may include some of the above-described components or may further include other components in addition to the above-described components.

The data obtainer3010may obtain data necessary for identification of an object or determination of a correction filter. The preprocessor3020may preprocess the obtained data such that the obtained data may be used for identification of the object or determination of the correction filter. The preprocessor3020may process the obtained data to a predetermined format such that the recognition result provider3040, which will be described later, may use the obtained data for identification of the object or determination of the correction filter.

The recognition data selector3030may select recognition data necessary for identification of the object or determination of the correction filter among the preprocessed data. The selected recognition data may be provided to the recognition result provider3040. The recognition data selector3030may select partly or wholly the preprocessed recognition data according to a predetermined selection criterion for identification of the object or determination of the correction filter.

The recognition result provider3040may apply the selected data to a data recognition model to determine a situation. The recognition result provider3040may provide a recognition result according to a recognition purpose of data. The recognition result provider3040may apply the selected recognition data to the data recognition model by using the recognition data selected by the recognition data selecting unit3030as an input value. Also, the recognition result may be determined by the data recognition model.

The recognition result provider3040may provide identification information of at least one object included in an image. For example, the recognition result provider3040may provide information about a category to which the identified object belongs, a name of the identified object, location information of the object, and the like.

Also, the recognition result provider3040may provide correction filter information suitable for the identified object as the object included in the image is identified. For example, the recognition result provider3040may provide information indicating that correction filters suitable for an object identified as a dog is a first correction filter, a second correction filter, and a sixth correction filter in the form of text or image, etc. The recognition result provider3040may apply the correction filter determined to be suitable for the identified object to a region including the object in the image to provide a corrected image.

The model updater3050may control the data recognition model to be updated based on evaluation of the recognition result provided by the recognition result provider3040. For example, the model updater3050may provide the model training unit2440with the recognition result provided by the recognition result provider3040to control the model training unit2440to update the data recognition model.

Meanwhile, at least one of the data obtainer3010, the preprocessor3020, the recognition data selector3030, the recognition result provider3040, and the model updater3050in the data recognizer2820may be manufactured in the form of at least one hardware chip and mounted on the device10. For example, at least one of the data obtainer3010, the preprocessor3020, the recognition data selector3030, the recognition result provider3040, and the model updater3050may be manufactured in the form of a dedicated hardware chip for artificial intelligence (AI) or may be manufactured in a part of an existing general-purpose processor (e.g. a CPU or an application processor) or a graphic-only processor (e.g., a GPU) and mounted on the various devices10described above.

In this case, the data obtainer3010, the preprocessor3020, the recognition data selector3030, the recognition result provider3040, and the model updater3050may be mounted on one device10, or may be mounted on the separate devices10. For example, some of the data obtainer3010, the preprocessor3020, the recognition data selector3030, the recognition result provider3040, and the model updater3050may be included in the device10, and the others may be included in a server.

Also, at least one of the data obtainer3010, the preprocessor3020, the recognition data selector3030, the recognition result provider3040, and the model updater3050may be implemented as a software module. When at least one of the data obtainer3010, the preprocessor3020, the recognition data selector3030, the recognition result provider3040, and the model updater3050is implemented as a software module (or a program module including an instruction), the software module may be a stored in non-transitory computer-readable media. Also, in this case, at least one software module may be provided by an operating system (OS) or by a predetermined application. Alternatively, some of the at least one software module may be provided by an operating system (OS), and the others may be provided by a predetermined application.

FIG.31is a diagram for explaining a software configuration stored in the memory320ofFIG.3.

Referring toFIG.31, the memory320may store programs (one or more instructions) for processing and control of the processor330. The programs stored in the memory320may be classified into a plurality of modules according to functions. According to an embodiment, the memory320may include a neural network module3110, a correction filter module3120, an application module3130, and a touch module3140.

The neural network module3110may include N layers3111through3114and a pulley connected layer3115. Each of the first through Nth layers3111through3114may include one or more instructions that detect at least one image attribute and/or a display attribute of an object from an image and abstract the detected at least one image attribute and/or display attribute of the object.

For example, each of the first through Nth layers3111through3114may include a convolutional layer including one or more instructions that detect the image attribute and/or the display attribute of the object from the image, and/or a pooling layer including one or more instructions that extract a representative value from the detected image attribute and/or display attribute of the object.

Also, the pulley connected layer3115may one or more instructions that to identify objects in the image and determine a correction filter corresponding to each of the identified objects, by using the image attribute and/or display attributes of the object detected from the N layers3111through3114.

At this time, according to an embodiment, the memory320may include a first pulley connected layer that uses image attributes to identify the objects in the image, and a second pulley connected layer that uses the display attributes of the object to determine the correction filter corresponding to each of the objects.

The neural network module3110may also store training result data (e.g., a probability distribution (correlation) between image attributes and categories of objects and a probability distribution (correlation) between display attributes of objects and categories of objects).

The correction filter module3120may include one or more instructions for correcting the image. The processor330may use the one or more instructions stored in the correction filter module3120to perform image correction corresponding to each of the objects in the image. Alternatively, the processor330may use the one or more instructions stored in the correction filter module3120to transfer a signal to correct each of the objects in the image to a correction filter component (not shown).

The application module3130may include one or more instructions, images, sound, etc., that the processor330uses to execute an application. For example, the application module3130may include various program modules such as a photo album application module, a messenger application module, an SNS application module, and the like.

The touch module3140may include one or more instructions that sense a touch event of a user input through the display310and transmit a signal corresponding to the sensed event to the processor330. According to an embodiment, the touch module3140may be configured as separate hardware including a controller.

Although various program modules are shown inFIG.31, some of the program modules may be omitted, modified or added depending on a type and characteristics of the device10. For example, the memory320may further include a base module that stores signals transmitted from the respective hardware included in the device10to the processor330and/or signals transmitted from the processor330to the respective hardware, and a network module for supporting a network connection between the device10and an external device.

FIG.32is an example of the neural network12that a device uses to correct an image.

Referring toFIG.32, the neural network12may include a first network3210that includes first through third layers3211,3213, and3215for identifying objects from an image, and a second network3220that includes fourth through sixth layers3221,3223, and3225for determining information related to a correction filter corresponding to each of the identified objects.

According to an embodiment, the device10may detect at least one image attribute from the image using the first through third layers3211,3213, and3215. At this time, the image attributes detected in the respective layers may be different from each other, but are not limited thereto.

For example, the device10may use the first through third layers3211,3213, and3215to detect at least one of a color, an edge, a polygon, a saturation, a brightness, a color temperature, blur, sharpness, and contrast from the image. In this case, the device10may differentiate a detection level of at least one of the color, edge, polygon, saturation, brightness, color temperature, blur, sharpness, and contrast detected from each layer.

The device10may combine (or connect) result images (or result vectors) using the first through third layers3211,3213, and3215of the first network3210to identify the objects in the image using a previously stored training result. Also, the device10may verify the identified objects.

According to an embodiment, the device10may use the first network3210to detect a display attribute of the object corresponding to each of the identified objects. Also, the device10may determine, for each of the objects in the image, based on a result of using the second network3220, determine at least one correction filter according to the display attribute of the object corresponding to each of the identified objects. Also, the device10may verify the determined correction filter.

According to an embodiment, the device10may select the image attribute or the display attribute of the object detected using the neural network12in association with each other. For example, based on the result of using the neural network12, the device10may select an image attribute or a display attribute of the object to be detected in a next layer according to a result image (or a result vector) of a previous layer.

On the other hand, the device10may abstract the image using the first to sixth layers3211to3225. For example, the device10divides the image into regions of a predetermined size and extract a maximum value (or a minimum value or an average value) of an image attribute (or a display attribute of the object) detected from the divided regions to identify and/or determine objects (or correction filters respectively corresponding to the objects) in the image. This will be described with reference toFIG.33.

Meanwhile, inFIG.32, each of the first and second networks3210and3220include three layers, but is not limited thereto. For example, each network may include three or less or more layers.

Also, inFIG.32, the device10divides the layers included in the first and second networks3210and3220, but is not limited thereto. According to an embodiment, the device10may identify the objects in the image from the image using the first layer3211, the second layer3213and the third layer3215, and determine the correction filter corresponding to each of the identified objects using the fourth layer3221, the fifth layer3223and the sixth layer3225.

FIG.33is a detailed diagram of the first layer3211of the first network3210ofFIG.32.

Referring toFIG.33, the first layer3211includes a convolutional layer3310and a pooling layer3320according to a deep convolutional neural network method.

According to an embodiment, the convolutional layer3310may be used by the device10to detect at least one predetermined image attribute from at least a partial region3302of an input image3301, using the first layer3211. At this time, the convolutional layer3310may be configured to divide the input image3301into a plurality of images. For example, the device10may divide the input image3301into three images3311,3312, and3313using the convolutional layer3310and detect an R attribute, a G attribute, and a B attribute from the images3311,3312, and3313respectively.

According to an embodiment, the device10may use the convolutional layer3310to give a weighted value to the attribute detected from the image. The device10may also perform non-linear transform according to a sigmoid algorithm on the attribute detected from the image using the convolutional layer3310.

According to an embodiment, the pooling layer3320may be used by the device10to determine a representative attribute value indicating a predetermined L×L size region3331(e.g., a 5×5 pixel size, etc.) among detected image attributes while moving the L×L size region3331in each of the images3311,3312, and3313. For example, the pooling layer3320may be configured to extract a maximum attribute value or a minimum attribute value or an average attribute value in the L×L size region3331. This may allow the device10to abstract the image while reducing a size of an image3332that is input to the next layer (e.g., the second layer3213).

Meanwhile, inFIG.33, an example of the first layer3211ofFIG.32is described, but the example ofFIG.33may be applied to other layers constituting the neural network12.

Also, according to an embodiment, the second layer3213or the fifth layer3223constituting the neural network12may not include a pooling layer. In this case, the device10may transmit a resultant image of a convolutional layer included in the second layer3213and the fifth layer3223to a next layer.

FIG.34is a detailed diagram of the first network3210ofFIG.32.

Referring toFIG.34, the first network3210ofFIG.32may further include a full connection layer3410that identifies and/or determines an object in an image based on attributes detected from the image. The full connection layer3410may correspond to the pulley connected layer3115ofFIG.31.

According to an embodiment, the device10may use the full connection layer3410to connect (or combine) image attributes detected using the first to third layers3211,3213and3215in the first network3210. Also, the device10may use the full connection layer3410to determine a category of an object probabilistically similar to the connected image attributes.

For example, the device10may use the full connection layer3410to obtain a result value that an object in an input image ofFIG.3is most probabilistically similar to a ‘dog’ and a location of the object in the image.

Meanwhile, the examples described inFIGS.33and34may also be applied to the second network3220. Accordingly, the second network3220may further include a full connection layer, and may be connected to the fourth to sixth layers3221,3223, and3225included in the second network3220.

Alternatively, the neural network12may include one full connection layer. In this case, the full connection layer may be connected to the first to sixth layers3211to3225of the first and second networks3210and3220. The device10may dynamically adjust connection between the full connection layer and the first through sixth layers3211through3225to identify objects in the image from the input image and determine a correction filter corresponding to the identified objects.

FIG.35is a flowchart illustrating a method, performed by the device10, of learning criteria for identifying an object and determining a correction filter by using a first network and a second network.

Referring toFIG.35, in operation S3510, the device10may input a training image and a category of each of objects in the training image to the neural network12. The device10may input the training image and the category of each of the objects in the training image to the neural network12based on a supervising learning method.

For example, the device10may input a label indicating a category of a first object in the training image to the neural network12along with the training image. Further, the device10may further input a location of the first object in the training image to the neural network12.

On the other hand, according to an embodiment, the device10may input a training image generated by a user of the device10to the neural network12. The training image generated by the user may include a captured image, a corrected image, or an image received from an external server, etc., in response to a user input by the device10. The device10may train the neural network12more appropriately to the user by inputting the training image generated by the user to the neural network12.

However, this is merely an embodiment. According to another example, the device10may train the neural network12by repeatedly performing an operation of analyzing and/or evaluating not only supervised learning but also unsupervised leaning (or autonomous learning or active learning) on an image attribute for each object in the neural network12.

In operation S3520, the device10may use the first network in the neural network12to learn to have a first criterion as a determination criterion for identifying objects in the training image. Here, the first criterion may include a type, number or a level, etc. of image attributes used by the device10to identify the object in the training image from the training image, using the neural network12.

The device10may learn to have the first criterion by analyzing correlation between the image attributes detected from the training image and the objects in the learning image.

For example, the device10may use the first network to analyze correlation between the first object and image attributes detected from X training images including the first object. The device10may use the first network to analyze correlation between the image attributes and the first object using a probability distribution between each of the image attributes detected from the X training images and a category corresponding to the first object.

The device10may determine, as the first criterion, the type, the number or the level, etc. of image attributes with a relatively high correlation (i.e., a high density of the probability distribution) with respect to the category corresponding to the first object.

Also, the device10may remove, from the first criterion, the type, the number or the level, etc. of image attributes with a relatively low correlation (i.e., a low density of the probability distribution) with respect to the category corresponding to the first object.

Specifically, the device10may use the first network to determine the first object included in the X training images and a brightness, an edge, a color, a sharpness, and the like among the image attributes extracted from the X training images and derive a brightness value, a shape of the edge, a set value of the color, a sharpness value, and the like to obtain a probability distribution of a face category to which the first object belongs and the derived image attributes. Accordingly, the device10may determine a brightness value, a shape of an edge, a set value of a color, a sharpness value, etc., with a relatively high correlation to an image including the face category.

The device10may use the first network to remove a type of an image attribute with a relatively low correlation to the image including the face category. Also, the device10may use the first network to determine a shape of an edge, a set value of a color, etc. with the relatively low correlation to the image including the face category.

The device10may sequentially apply a hierarchical network in the neural network12to remove criterions with a low correlation to the image, thereby reducing an amount of computation required to identify objects in the image.

In operation S3530, the device10may use the second network in the neural network12to learn to have a second criterion as a determination criterion for determining the correction filter corresponding to each of the objects in the training image. Here, the second criterion may include a type, number or a level, etc. of display attributes used by the device10to determine the correction filter corresponding to each of the objects in the training image, using the neural network12.

The device10may learn to have the second criterion by analyzing correlation between the display attributes of the object detected from an image region corresponding to the objects included in the training image and the objects.

For example, the device10may use the second network to a probability distribution between the first object and display attributes detected from a partial region of the training image corresponding to the first object. The device10may use the obtained to probability distribution to analyze correlation between the display attributes of the object and the category corresponding to the first object.

The device10may determine, as the second criterion, the type, the number or the level, etc. of display attributes with a relatively high correlation (i.e., a high density of the probability distribution) with respect to the category corresponding to the first object.

Also, the device10may remove, from the second criterion, the type, the number or the level, etc. of display attributes with a relatively low correlation (i.e., a low density of the probability distribution) with respect to the category corresponding to the first object.

Specifically, when the category corresponding to the first object corresponds to a face, the device10may derive set values such as a brightness, an edge, a color, a sharpness, and the like among the display attributes of the object. The device10may use the second network to obtain a probability distribution between the face category and the set values such as a brightness value, a shape of the edge, a set value of the color, a sharpness value, and the like. Accordingly, the device10may determine the correction filter having a brightness value, a shape of an edge, a set value of a color, a sharpness value, etc., with a relatively high correlation to the face category.

Also, the device10may use the second network to remove a type of a display attribute of the object with a relatively low correlation to the face category. The device10may also use the second network to determine a correction filter having a brightness value, a shape of an edge, a set value of a color, a sharpness value, etc. with a relatively low correlation to the face category.

FIG.36is an example in which the device10learns the neural network12.

Referring toFIG.36, the device10may input training images3610in which categories of objects are determined to the neural network12.

The device10according to an embodiment may use a first network3620of the neural network12to detect image attributes from the training images3610and obtain a probability distribution3621between the detected image attributes and the categories of the objects included in the training images3610.

According to an embodiment, the device10may normalize (or abstract) the obtained probability distribution3621using the first network3620.

The device10may learn to have a first criterion3662by analyzing correlation between image attributes of the training images3610and the categories of the objects included in the training images3610using the first network3620.

According to an embodiment, the device10may use a second network3630of the neural network12to detect display attributes of the object from a partial region of the training images3610corresponding to the objects. Also, the device10may obtain a probability distribution3631between the detected display attributes of the object and the categories of the objects included in the training images3610.

According to an embodiment, the device10may normalize (or abstract) the obtained probability distribution3631using the second network3630.

The device10may learn to have a second criterion3632by analyzing correlation between the display attributes of the object corresponding to the objects included in the training images3610and the categories of the objects included in the training images3610using the second network3630.

FIG.37is a diagram for explaining a method, performed by the device10, of selecting an image attribute to identify a plurality of objects3711and3712included in an image3710.

Referring toFIG.37, the device10may train neural networks3740and1550using different data sets3720and3730according to types of the plurality of objects3711and3712to be identified in the image3710. The device10may also differently train the first neural network3740for identifying an object3711corresponding to a background among the plurality of objects3711and3712and a second neural network3750for identifying an object3712other than the background.

Specifically, the device10may train the first neural network3740including at least one layer using the data set3720relating to the background such as mountain, sea, river, lake, and the like. Here, each of the plurality of layers included in the first neural network3740may be trained to detect an image attribute suitable for identifying the background.

Also, the device10may train the second neural network3750including at least one layer using the data set3730relating to an object such as dogs, cats, people, etc. other than the background. Here, each of the plurality of layers included in the second neural network3750may be trained to detect an image attribute suitable for identifying the object other than the background.

However, this is merely an embodiment, and a neural network for identifying an object corresponding to a background and an object other than the background among a plurality of objects may be trained as one neural network.

The device10may identify the plurality of objects3711and3712as forest and cat respectively as a result of identifying the plurality of objects3711and3712included in the image3710using the first neural network3740and the second neural network3750, respectively.

FIG.38is a detailed diagram showing a configuration of a device3800according to an embodiment.

Referring toFIG.38, the device3800according to an embodiment may include a user inputter3810, a sensing unit3840, a communicator3850, and an A/V inputter3860, in addition to an outputter3820, a memory3870, and a processor3830corresponding to the display310, the memory320and the processor330ofFIG.3respectively.

The user inputter3810means a means for a user to input data for controlling the device3800. For example, the user inputter3810may include a key pad, a dome switch, a touch pad (a contact capacitance type, a pressure resistive type, an infrared ray detection type, a surface ultrasonic wave conduction type, an integral tension measurement type, a piezo effect type, etc.), a jog wheel, a jog switch, and the like, but is not limited thereto.

According to an embodiment, the user inputter3810may receive a user input to touch one of objects in an image displayed on a screen using the touch pad.

The outputter3820may output an audio signal, a video signal, or a vibration signal, and may include a display3821, a sound outputter3822, and a vibration motor3823.

The display3821may display and output information processed in the device3800. When the display3821and the touch pad are configured as a touch screen in a layer structure, the display3821may be used as an input device in addition to as an output device. The display3821may include at least one of a liquid crystal display, a thin film transistor-liquid crystal display, an organic light-emitting diode display, a flexible display, a three-dimensional (3D) display, an electrophoretic display, and a transparent display.

The sound outputter3822may output audio data received from the communicator3850or stored in the memory3870. The vibration motor3823may output a vibration signal.

The processor3830typically controls the overall operation of the device3800and the signal flow between the internal components of the device3800and performs a function of processing the data. For example, the processor3830may generally control the user inputter3810, the outputter3820, the sensing unit3840, a communicator3850, the A/V inputter3860, etc., by executing programs (one or more instructions) stored in the memory3870.

According to an embodiment, to perform the above described function of the device3800, the processor3830may use a neural network to identify objects in an image from the image and determine correction filters respectively corresponding to the identified objects, thereby controlling the components of the device10to correct each of the objects in the image. The processor3830corresponds to the processor330ofFIG.3, and thus a detailed description thereof is omitted.

The sensing unit3840may sense a state of the device3800or a state around the device3800and may transmit sensed information to the controller1300. The sensing unit3840may include at least one among a geomagnetic sensor3841, an acceleration sensor3842, a temperature/humidity sensor3843, an infrared sensor3844, a gyroscope sensor3845, a location sensor (e.g. a GPS)3846, a pressure sensor3847, a proximity sensor3848, and an RGB (illuminance) sensor3849, but is not limited thereto. Functions of respective sensors may be intuitively inferred by one of ordinary skill in the art and thus, detailed descriptions thereof will be omitted.

The communicator3850may include one or more components for communicating with an external server (e.g., an SNS server, a cloud server, a content providing server, etc.) and other external devices. For example, the communicator3850may include a short range wireless communicator3851, a mobile communicator3852, and a broadcast receiver3853.

The short range wireless communicator3851includes a Bluetooth communicator, a Bluetooth low energy (BLE) communicator, a near field communicator, a WLAN (WiFi) communicator, a Zigbee communicator, an infrared data association (IrDA) communicator, a WFD (Wi-Fi Direct) communicator, an UWB (ultra wideband) communicator, an Ant+communicator, and the like, but is not limited thereto.

The mobile communicator3852may transceive wireless signals to and from at least one of a base station, an external terminal, and a server on a mobile communication network. The wireless signals herein may include various types of data per transceiving audio call signals, video communication call signals or text/multimedia messages.

The broadcast receiver3853may receive broadcasting signals and/or information related to broadcasting from the outside via broadcasting channels. Depending on a type of implementation, the device3800may not include the broadcast receiver3853.

The A/V inputter3860is to input audio signals or video signals, and may include a camera3861, a microphone3862, etc.

The camera3861may capture an image in a camera recognition range. The image captured by the camera3861may be image processed by the processor3830and displayed and output through the display3821.

The memory3870may store programs (one or more instructions) for processing and controlling of the processor3830and store data input to and output from the device3800.

The programs stored in the memory3870may be classified into a plurality of modules per function and may be, for example, the UI module3871, a touch screen module3872, a notification module3873, etc.

The UI module3871may provide specialized UI, graphical UI (GUI), etc., which are linked to the device3800per application. The touch screen module3872may sense a user's touch gesture on the touch screen and transmit information about the touch gesture to the processor3830. According to an embodiment, the touch screen module3872may recognize and analyze touch code. The touch screen module3872may be configured as separate hardware including a controller.

Various sensors may be arranged inside or near the touch screen for detecting the touch on the touch screen or a close touch. A tactile sensor is an example of a sensor for detecting the touch on the touch screen. The tactile sensor may sense the touch of a particular object at a level of human feeling or at a higher level than that. The tactile sensor may detect various information such as roughness of a contact surface, hardness of a contact material, and temperature of a contact point.

Also, the proximity sensor is another example of sensors for detecting the touch on the touch screen.

The proximity sensor is a sensor which detects an existence of an object approaching a certain detection surface or an object in the vicinity, without a mechanical contact, via an electromagnetic force or infrared rays. Examples of the proximity sensors are a transparence-type photoelectric sensor, a direct reflection-type photoelectric sensor, a mirror reflection-type photoelectric sensor, a high-frequency oscillation-type proximity sensor, a capacitance-type proximity sensor, a magnet-type proximity sensor, and an infrared ray proximity sensor. Various touch gestures of the user may include a tap, a touch and hold, a double tap, a drag, a fanning, a flick, a drag and drop, a swipe, etc.

The notification module3873may generate a signal to notify an event occurrence of the device3800. An example of an event occurred in the device3800may include call signal reception, message reception, key signal input, schedule notification, etc. The notification module3873may output the notification signal in a video signal-type via the display3821or in an audio signal-type via the sound outputter3822, or in a vibration signal-type via the vibration motor3823.

The memory3870may include at least one type of storage media such as a flash memory, a hard disk, a multimedia micro-card, a card type memory (for example, secure digital (SD) or extreme digital (XD) memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), a magnetic memory, a magnetic disk, and an optical disk.

Meanwhile, the configuration of the devices10,400, and3800shown inFIGS.1to38is an embodiment, and each component of the devices10,400, and3800may be integrated, added, or omitted according to the specification of the devices10,400, and3800. That is, two or more components may be combined into one component, or one component may be divided into two or more components when necessary. Also, a function performed in each configuration (or module) is intended to describe the embodiments, and a specific operation or device thereof does not limit the scope of the present disclosure.

FIG.39is a schematic diagram illustrating an example in which the device10corrects an image3910according to another embodiment.

Referring toFIG.39, according to another embodiment, the device10may correct the image3910using the neural network22provided in the server20.

According to another embodiment, the device10may transmit the image3910to the server20and receive a corrected image3920. The server20may use the neural network22to identify objects in the image3910received from the device10and determine correction filters respectively corresponding to the identified objects. The server20may also use the determined correction filters to correct each of the objects in the image3910and then transmit the corrected image3920to the device10.

Alternatively, the server20may transmit information about the determined correction filters to the device10using the neural network22. In this case, the device10may use the information about the correction filters received from the server20to correct each of the objects in the image3910, thereby generating the corrected image3920.

Meanwhile, the server20may be a computing device that provides services to client devices and may be, for example, a PC, a laptop, a mobile phone, a micro server, a global positioning system (GPS), an e-book terminal, consumer electronics, an electronic device in a vehicle, another mobile or non-mobile computing device, but is not limited thereto. The server20may include all types of devices having a communication function and a data processing function.

Also, the device10may be communicatively coupled to the server20over a network. In this case, the network may include a local region network (LAN), a wide region network (WAN), a value added network (VAN), a mobile radio communication network, a satellite communication network, and a mutual combination thereof, may be a data communication network in a comprehensive sense that enables the device10and the server20to communicate smoothly with each other, and may include a wired Internet, a wireless Internet, and a mobile wireless communication network.

FIG.40is a flowchart illustrating a method, performed by the device10and the server20, of applying a correction filter to at least one object included in an image, according to an embodiment.

In operation S4010, the device10obtains the image including a plurality of objects. For example, the device10may read a stored image as a photo album application, an image editing application, and the like are executed in the device10. Further, according to another example, the device100may download an image from an external server (e.g., an SNS server, a cloud server, a content providing server, or the like), or may capture an image using a camera.

In operation S4020, the device10transmits the obtained image to the server20including the neural network22.

In operation S4030, the server20may identify the objects in the image and determine correction filters respectively corresponding to the identified objects, based on a training result of using the neural network22.

The server20may detect at least one image attribute from the image by inputting the image to the neural network22and may determine a location of each of the objects in the image and/or a category of each of the objects based on the detected image attributes. The server20may also use the neural network22to detect display attributes of the object from an image region corresponding to the identified objects and determine correction filters respectively corresponding to the objects based on the detected display attributes of the object. With regard to a structure and function of the neural network22, the embodiments shown inFIGS.31to35may be applied, and thus detailed descriptions thereof are omitted.

In operation S4040, the device10receives, from the server20, information about the identified objects from the image and information about the correction filters respectively corresponding to the objects.

In operation S4050, the device10corrects each of the plurality of objects in the image, based on the information received from the server20. The device10may correct the image using the correction filters respectively corresponding to the objects in the image.

For example, the device10may apply a first correction filter to a first object in the image and a second correction filter to a second object in the image. Alternatively, the device10may apply the first correction filter to the first and second objects in the image. In this case, parameters of the first correction filter applied to each of the first and second objects in the image may be different.

FIG.41is a flowchart for explaining a method, performed by the device10and the server20, of correcting an object in an image obtained upon capturing, according to an embodiment.

In operation S4110, the device10may display a preview image through the display420.

The device10may display the preview image including at least one object sensed by the device400through the display420as a camera application is executed.

In operation S4120, the device10may capture at least one object through a camera (not shown) to obtain an image.

The device10may capture the at least one object detected at a time when a capturing request is received through the camera (not shown) when the capturing request is received.

In operation S4130, the device10may store the obtained image through the memory430.

In operation S4140, the device10may request the server20to identify the at least one object included in the stored image.

For example, the device10may transmit an object identification request signal including the stored image to the server20. According to another example, the device10may transmit an object identification request signal including an identifier of a location of the memory430in which the image is stored to the server20.

In operation S4150, the server20may use a neural network to identify the at least one object included in the stored image.

On the other hand, operation S4140may correspond to a method of identifying at least one object included in an image of operation S220described above with reference toFIG.2.

In operation S4160, the server20may transmit information about the identified at least one object to the device10.

In operation S4170, the device10may determine a correction filter to apply to each identified at least one object included in the image.

On the other hand, operation S4170may correspond to a method of determining a correction filter with respect to each of objects of operation S230described above with reference toFIG.2.

In operation S4180, the device10may apply the correction filter to each of the at least one object included in the image. Here, operation S4180may correspond to operation S240described above with reference toFIG.2.

In operation S4190, the device10may store the image to which the correction filter is applied through the memory430.

In operation S4195, the device10may display the image to which the correction filter is applied through the display420. For example, the device10may display the image to which the correction filter is applied, through the display420, as the photo album application that displays images captured or stored based on a user input is executed.

FIG.42is a flowchart for explaining a method, performed by the device10and the server20, of correcting an object in an image obtained upon capturing, according to another embodiment.

In the embodiment, a first processor and a second processor included in the device10may correspond to the first processor410and the second processor420described above with reference toFIG.4, respectively.

In operation S4205, the first processor may display a preview image through a display. The first processor may display the preview image including at least one object sensed by the device10through the display as a camera application is executed.

In operation S4210, the first processor may capture the at least one object through a camera (not shown) to obtain the image.

In operation S4215, the first processor may store the obtained image through a memory.

In operation S4220, the first processor may request the server20to identify the at least one object included in the stored image.

In operation S4225, the server20may use a neural network to identify the at least one object included in the stored image.

On the other hand, operation S4225may correspond to a method of identifying at least one object included in an image of operation S220described above with reference toFIG.2.

In operation S4230, the server20may transmit information about the identified at least one object to the second processor.

In operation S4235, the second processor may use the neural network to determine a correction filter to apply to each of the identified at least one object in the image.

On the other hand, operation S4235may correspond to a method of determining a correction filter to an identified object of operation S230described above with reference toFIG.2.

In operation S4240, the second processor may transmit information about the determined correction filter with respect to each of the identified at least one object to the first processor.

In operation S4245, the first processor may apply the correction filter determined with respect to each of the identified at least one object in the image.

In operation S4250, the first processor may store the image to which the correction filter is applied through the memory.

In operation S4255, the first processor may display the image to which the correction filter is applied through the display420. For example, the first processor may display the image to which the correction filter is applied through the display420, as a photo album application that displays images captured or stored based on a user input is executed.

FIG.43is a flowchart for explaining a method, performed by the device10, the first server30, and the second server40, of correcting an object in an image obtained upon capturing, according to another embodiment.

In the embodiment, a first processor and a second processor included in the device10may correspond to the first processor410and the second processor420described above with reference toFIG.4, respectively.

On the other hand, operations S4305to S4315may correspond to operations S4205to S4315described above with reference toFIG.42.

In operation S4320, the first processor may request the first server30to identify at least one object included in a stored image.

In operation S4325, the first server30may use a neural network to identify the at least one object included in the stored image.

On the other hand, operation S4325may correspond to a method of identifying at least one object included in an image of operation S220described above with reference toFIG.2.

In operation S4330, the first server30may transmit information about the identified at least one object to the second server40.

In operation S4335, the second server40may use the neural network to determine a correction filter to apply to each of the identified at least one object in the image.

In operation S4340, the second server40may transmit information about the correction filter determined with respect to each of the identified at least one object to the first processor.

Operations S4345to S4355may also correspond to operations S4245to S4255described above with reference toFIG.42.

FIG.44is a flowchart for explaining a method, performed by the device10and the server20, of correcting an object in a preview image, according to an embodiment.

In the embodiment, a first processor and a second processor included in the device10may correspond to the first processor410and the second processor420described above with reference toFIG.4, respectively.

In operation S4410, the first processor may obtain the preview image through a camera. The first processor may obtain the preview image in real time through the camera as a camera application is executed.

In operation S4420, the first processor may transmit an identification request with respect to at least one object included in the preview image to the second processor.

In operation S4430, the second processor may use a neural network to identify the at least one object included in the preview image.

In operation S4440, the second processor may transmit information about the identified at least one object to the server20.

In operation S4450, the server20may use the neural network to determine a correction filter to apply to each of the identified at least one object in the image.

In operation S4460, the server20may transmit information about the correction filter determined with respect to each of the identified at least one object to the first processor.

In operation S4470, the first processor may apply the determined correction filter to each of the identified at least one object in the image.

In operation S4480, the first processor may display the image to which the correction filter is applied through a display.

FIG.45is a flowchart illustrating a method, performed by the device10and the server20, of correcting an object in a stored image, according to an embodiment.

In the embodiment, a first processor and a second processor included in the device10may correspond to the first processor410and the second processor420described above with reference toFIG.4, respectively.

In operation S4510, the first processor may display a stored image through a display. For example, the first processor may display the stored image through the display as a photo album application, an image editing application, or the like is executed.

Also, operations S4520through S4570may correspond to operations S4420through S4470described above with reference toFIG.44respectively.

In operation S4580, the first processor may store the image to which a correction filter is applied through a memory.

In operation S4590, the first processor may display the image to which the correction filter is applied through the display. For example, the first processor may display the image to which the correction filter is applied through the display, as the photo album application that displays images captured or stored based on a user input is executed.

FIG.46is a flowchart illustrating a method, performed by the device10and the server20, of correcting an object in a stored image, according to another embodiment.

In the embodiment, a first processor and a second processor included in the device10may correspond to the first processor410and the second processor420described above with reference toFIG.4, respectively.

On the other hand, operation S4610may correspond to operation S4510described above with reference toFIG.45.

In operation S4620, the first processor may transmit an identification request for at least one object included in a displayed image to the server20.

In operation S4630, the server20may use a neural network to identify at least one object included in the image.

In operation S4640, the server20may use the neural network to determine a correction filter to apply to each of the identified at least one object in the image.

In operation S4650, the server20may transmit information about the correction filter determined with respect to each of the identified at least one object to the first processor.

Operations S4660to S4680may also correspond to operations S4570to S4590described above with reference toFIG.45.

FIG.47is a flowchart illustrating a method, performed by the device10, the first server30, and the second server40, of correcting an object in a stored image, according to an embodiment.

In operation S4710, the device10may display the stored image through a display. For example, the device10may display the stored image through the display as a photo album application, an image editing application, or the like is executed.

In operation S4720, the device10may transmit an identification request with respect to at least one object included in the displayed image to the first server30.

In operation S4730, the first server30may use a neural network to identify at least one object included in a preview image.

In operation S4740, the first server30may transmit information about the identified at least one object to the second server40.

In operation S4750, the second server40may use the neural network to determine a correction filter to apply to each of the identified at least one object in the image.

In operation S4760, the second server40may transmit information about the correction filter determined with respect to each of the identified at least one object to the device10.

Operations S4770to S4790may correspond to operations S4660to S4680described above with reference toFIG.46.

FIG.48is a flowchart illustrating a method, performed by the device10, the cloud server50, and the server20, of correcting an object in an image stored in the cloud server50, according to an embodiment.

In operation S4805, the device10may display the image through a display. For example, the device10may display the image through the display as a camera application, a photo album application, an image editing application, or the like is executed.

In operation S4810, the device10may transmit the displayed image to the cloud server50. For example, the device10may transmit the image to the cloud server50to store the image in the cloud server50.

In operation S4815, the cloud server50may store the image received from the device10.

In operation S4820, the device10may transmit an identification request for at least one object included in the image stored in the cloud server50to the server20.

In operation S4825, the server20may request the cloud server50for an image for identification of the object.

In operation S4830, the cloud server50may transmit the stored image to the server20at a request of the server20.

In operation S4835, the server20may use a neural network to identify at least one object included in a preview image.

In operation S4840, the server20may use the neural network to determine a correction filter to apply to each of the identified at least one object in the image.

In operation S4845, the server20may transmit information about the correction filter determined with respect to each of the identified at least one object to the device10.

Also, operations S4850to S4860may correspond to operations S4770to S4790with reference toFIG.47.

FIG.49is a flowchart for explaining a method, performed by the device10, the first server30, and the second server40, of correcting an object in an image stored in the cloud server50, according to an embodiment.

On the other hand, operations S4905to S4915may correspond to operations S4805to S4815described above with reference toFIG.48.

In operation S4920, the device10may transmit an identification request for the at least one object included in the image stored in the cloud server50to the first server30.

In operation S4925, the first server30may request the cloud server50for an image for identification of the object.

In operation S4930, the cloud server50may transmit the stored image to the server20at a request of the first server30.

In operation S4935, the first server30may use a neural network to identify at least one object included in a preview image.

In operation S4940, the first server30may transmit information about the identified at least one object to the second server40.

In operation S4945, the second server40may use the neural network to determine a correction filter to apply to each of the identified at least one object in the image.

In operation S4950, the second server40may transmit information about the correction filter determined with respect to each of the identified at least one object to the device10.

Also, operations S4955to S4965may correspond to operations S4850to S4860described above with reference toFIG.48.

FIG.50is a diagram showing a configuration of the server20according to another embodiment.

Referring toFIG.50, the server20may include a communicator5010, a controller5020, and a memory5030.

The communicator5010may include at least one component to allow the server20to communicate with client devices such as the device10under the control of the controller5020. The controller5020may transmit and receive data to and from the device10connected through the communicator5010.

The communicator5010may include one of, for example, wireless LAN (e.g., Wi-Fi), Bluetooth, and wired Ethernet (Ethernet). Also, the communicator5010may include a combination of wireless LAN, Bluetooth, and wired Ethernet.

The controller5020controls the overall operation of the server20and the signal flow between the internal components of the server20and performs processing of data. The controller5020may include one or more processors implemented as one or more cores (not shown) and may include a RAM (not shown) and a ROM (not shown) used as a storage region storing signals or data input to the controller5020.

The controller5020may execute one or more programs stored in the memory5030. According to an embodiment, the controller5020may read a result of using a neural network from the memory5030, identify objects in an image received from the device10through the communicator5010, and determine a correction filter corresponding to each of the identified objects.

Specifically, the controller5020may detect predetermined image attributes in the image according to the result of using the neural network, and, based on the detected image attributes, determine a location of each of the objects in the image and/or a category corresponding to each of the objects.

Further, the controller5020may detect display attributes of the object from an image region corresponding to the identified objects, based on the result of using the neural network, and determine correction filters respectively corresponding to the objects based on the detected display attributes of the object. For example, the controller5020may determine a type of the correction filter to be applied to each of the objects in the image and/or a parameter to be input to the correction filter using the neural network.

The controller5020may control the communicator5010to transmit information (e.g., location information in the image and category information) about the objects in the image obtained using the neural network, and correction filter information (e.g. the type of the correction filter and parameter information) corresponding to each of the objects to the device10.

The memory5030may store various data, program, or application for driving and controlling the server20. The program stored in the memory5030may include one or more instructions. The program (one or more instructions) or the application stored in the memory5030may be executed by the controller5020.

According to an embodiment, the memory5030may include one or more programs that constitute a plurality of layers in the neural network and a neural network module (not shown) in which the result of using the neural network is stored. In this regard, the embodiment ofFIG.21may be applied, and thus a detailed description thereof will be omitted.

The memory5030may also include an application module (not shown) storing one or more programs for executing applications and application data, a communication module (not shown) storing one or more programs for communicating with the outside, and communication data, but is not limited thereto.

According to an embodiment, the memory5030may store the result of using the neural network for each client device.

On the other hand, when the server20further includes a correction filter (for example, a correction filter program module or a hardware component) capable of correcting the image, the controller5020may control the memory5030or a hardware component (not shown) to correct each of the objects in the image using the determined correction filters, and control the communicator5010to transmit the corrected image to the device10.

Also, all of the components shown inFIG.50are not indispensable components of the server20. The server20may be implemented by more components than the components shown inFIG.50, and the server20may be implemented by fewer components than those shown inFIG.50. For example, the server20may further include a user inputter (not shown) receiving a user input, and a display (not shown) displaying and outputting information processed by the server20.

FIG.51is a diagram showing an example in which the device10and the server20learn and recognize data in association with each other, according to some embodiments.

Referring toFIG.51, the server20may learn a criterion for identifying an object included in an image. For example, the server20may learn an attribute of the object used to identify the object and an attribute of the object. Also, the server20may learn a display attribute that is a basis for determining a correction filter suitable for the object. The server20may obtain data to be used for training and apply the obtained data to a data recognition model that will be described later, thereby learning the criterion for identifying the object or determining the correction filter.

Meanwhile, the data obtainer5110, the preprocessor5120, the training data selector5130, the model training unit5140, and the model evaluator5150of the server20may perform functions of the data obtainer2910, the preprocessor2920, the training data selector2930, the model training unit2940, and the model evaluator2950shown inFIG.29, respectively.

The data obtainer5110may obtain data necessary for identification of the object or determination of the correction filter. The preprocessor5120may preprocess the obtained data such that the obtained data may be used for training for identification of the object or determination of the correction filter. The training data selector5130may select data necessary for training from the preprocessed data. The selected data may be provided to the model training unit5140.

The model training unit5140may learn a criterion as to which image attribute or display attribute to use in the input image to identify the object or determine the correction filter and how to determine a type of object or the correction filter using an image attribute or a display attribute. The model training unit5140may obtain data to be used for training and apply the obtained data to a data recognition model that will be described later to learn the criterion for identifying the object or determining the correction filter. The model evaluator5150may input evaluation data to the data recognition model and allow the model training unit5140to learn again when a recognition result output from the evaluation data does not satisfy a predetermined criterion.

The server20may provide the generated recognition model to the device10. In this case, the device10may identify the object or determine the correction filter using the received recognition model.

On the other hand, according to another embodiment, the server20may apply the data received from the device10to the generated data recognition model to identify the object or determine the correction filter. For example, the device10may transmit data selected by the recognition data selector3030to the server20, apply the data selected by the server20to the recognition model to identify the object or determine the correction filter.

The server20may also provide the device10with information about the object identified by or the type of the correction filter determined by the server20. Accordingly, the device10may receive from the server20identification information of the object included in the image or the information about the type of correction filter.

Meanwhile, the above-described embodiments of the present disclosure may be embodied in a general-purpose digital computer that may be embodied as a program that may be executed by a computer and operates the program using a computer-readable recording medium.

The computer-readable recording medium may be a magnetic storage medium such as a ROM, a floppy disk, a hard disk, etc., an optical reading medium (e.g., a CD-ROM or a DVD) and a carrier wave (e.g., transmission via the Internet).

The embodiments may be implemented in a software program that includes instructions stored on a computer-readable storage medium.

The computer may include the device10according to the embodiments, which is a device capable of calling stored instructions from a storage medium and operating according to the embodiments in accordance with the called instructions.

The computer-readable storage medium may be provided in the form of a non-transitory storage medium. Here, ‘non-transitory’ means that the storage medium does not include a signal and is tangible, but does not distinguish whether data is stored semi-permanently or temporarily on the storage medium.

Further, the control method according to the embodiments may be provided in a computer program product. The computer program product may be traded between a seller and a buyer as a product.

The computer program product may include a software program and a computer-readable storage medium having stored thereon the software program. For example, the computer program product may include a product (e.g., a downloadable application) in the form of S/W program that is electronically distributed through a manufacturer of the device10or an electronic marketplace (e.g. Google Play Store and App Store). For electronic distribution, at least a part of the S/W program may be stored on a storage medium or may be generated temporarily. In this case, the storage medium may be a storage medium of a server of the manufacturer, a server of the electronic market, or a relay temporarily storing the SW program.

The computer program product may include a server storage medium of a server or a storage medium of the device10in a system including the server and the device10. Alternatively, when a third device (e.g., a smart phone) in communication with the server or the device10is present, the computer program product may include a storage medium of the third device. Alternatively, the computer program product may include S/W program itself transmitted from the server to the device10or the third device, or transmitted from the third device to the device10.

In this case, one of the server, the device10and the third device may execute the computer program product to perform the method according to the embodiments. Alternatively, two or more of the server, the device10and the third device may execute the computer program product to distribute the method according to the embodiments.

For example, the server (e.g., a cloud server or an artificial intelligence server, etc.) may execute the computer program product stored in the server to control the device10in communication with the server to perform the method according to the embodiments.

As another example, the third device may execute the computer program product to control the device10in communication with the third device to perform the method according to the embodiment. When the third device executes the computer program product, the third device may download the computer program product from the server and execute the downloaded computer program product. Alternatively, the third device may execute the provided computer program product provided in a preloaded manner to perform the method according to the embodiments.

While embodiments of the present disclosure have been particularly shown and described with reference to the accompanying drawings, it will be understood by those of ordinary skill in the art that various changes in form and details may be made therein without departing from the spirit and scope of the disclosure as defined by the appended claims. Thus, the embodiments described above should be considered in descriptive sense only and not for purposes of limitation.