Patent Description:
Recently, photos are taken through electronic apparatuses such as smartphones as well as cameras. An electronic apparatus not only has a photo taking function but also provides various filters to improve the quality of photos. For example, an electronic apparatus provides a technology of analyzing an image and proposing an optimal filter set corresponding to a screen type. In other words, based on the type of a currently captured screen being "restaurant", an electronic apparatus may correct the captured image using a set of filters corresponding to the restaurant, and based on the type of a currently captured screen being "person", the electronic apparatus may correct the captured image using a set of filters corresponding to the person.

However, in the case of a prior art electronic apparatus, it often does not understand the overall configuration of a screen based on determining a set of filters for correcting an image, ending up providing a set of filters corresponding to a type that a user does not want.

In addition, since the prior art electronic apparatus determines the type of screen based on a single object, based on multiple objects being included in a captured image, it often finds it difficult to determine the type of screen. Accordingly, in order to prevent misrecognition, in many cases, the electronic apparatus determines "No detect" in which a set of filters are not provided.

Further, based on an area of the screen obtained during image capturing being narrow, it may be difficult to determine the type of screen due to the limited angle of view.

Therefore, a method of accurately determining the type of screen included in an image and providing a set of filters corresponding to the type of screen is required.

<CIT>D1 discloses context-aware image filtering for mobile cameras. An image processing application executing on a processing device detects an object in an image that is in a field of view of a camera associated with the processing device. The image processing application further determines, based on the object detected in the image, a recommended filter for the image from a list of filters available for filtering the image. The image processing application causes a display device to display a user interface that includes a contextual user interface control. The contextual user interface control indicates that a context-aware image filtering mode is available for the image. Responsive to receiving a selection of the contextual user interface control, the image processing application enables the context-aware image filtering mode by automatically applying the recommended filter to the image to generate a filtered image and causes the display device to display the filtered image.

<CIT>discloses a method by which an electronic apparatus provides recommendation information related to photography. The method includes identifying a subject included in a preview image recognized by a first camera, obtaining information of the identified subject, obtaining information related to light in surroundings of the identified subject, determining a recommended photographing composition based on the information of the identified subject and the information related to the light in the surroundings of the subject, and providing information about the recommended photographing composition. The electronic apparatus obtains a first image (e.g., a maximum viewing angle image) having a viewing angle greater than that of the preview image by using a second camera different from the first camera that obtains the preview image.

<CIT>discloses a system for gaze and gesture detection in unconstrained environments. The system may include an image capturing device configured to generate a <NUM>-degree image of an unconstrained environment comprising an individual and an object of interest. The system may further include a depth sensor configured to generate a three dimensional depth map of the unconstrained environment and a knowledge base. A processor may be included that is configured to generate an environment map from the <NUM>-degree image and the depth map and determine a direction of attention from the individual in the <NUM>-degree image. The processor may be further configured to generate a directionality vector representative of the direction of attention and project the directionality vector from the individual on the environment map. The processor may also be configured to detect an intersection of the directionality vector with the object of interest and search the knowledge base for the identity of the object of interest using data relating to the object of interest.

<CIT> discloses a posture analysis device comprising: a skeleton extraction unit that acquires skeleton data including feature point data indicating joint positions of a person appearing in the image data by image recognition using the image data as input; a posture model storage unit in which a posture label is associated with each piece of skeleton data; and a posture estimation unit that determines the posture of the person appearing in the image data from the posture label predetermined in the posture model based on the skeleton data acquired by the skeleton extraction unit.

The present disclosure is to provide an electronic apparatus that may identify a type of screen more accurately based on a relationship between objects in an image captured using a plurality of lenses and correct the captured image based on a set of filters corresponding to the identified type of screen and a controlling method thereof.

An electronic apparatus according to an embodiment includes a camera including a first lens and a second lens capable of obtaining an image having an wider angle of view different than the first lens, a display, a memory, and a processor configured to provide a first image obtained using the first lens to the display as a live view, obtain a second image using the second lens while providing the first image as a live view, obtain information regarding the second image and information regarding at least one object included in the second image using at least one neural network model, identify a screen type of the second image based on the information regarding the second image and the information regarding at least one object included in the second image, identify a set of filters corresponding to the screen type of the second image, and correct the first image provided as the live view based on the identified set of filters.

The information regarding the second image includes depth map information corresponding to the second image and saliency information corresponding to the second image, and the processor is configured to obtain the depth map information by inputting the second image to a first neural network model, and obtain the saliency information by inputting the second image to a second neural network model.

The information regarding at least one object included in the second image may include at least one of type information of the at least one object, three-dimensional location information of the at least one object, information regarding an area where the at least one object is located, or posture information of the at least one object, and the processor may be configured to obtain segmentation information in which the at least one object and a background included in the second image are segmented and type information of the at least one object by inputting the second image to a third neural network model, obtain the three-dimensional location information of the at least one object and the information regarding an area where the at least one object is located based on the segmentation information and the depth map information, and obtain the posture information of the at least one object by inputting the information regarding the at least one object included in the segmentation information to a fourth neural network model.

The processor may be configured to obtain relationship information between the at least one object by inputting the three-dimensional location information of the object and the information regarding an area where the object is located to a fifth neural network model.

The processor may be configured to obtain heat map information corresponding to corrected saliency information by inputting the relationship information, the posture information of the at least one object, focus information regarding the second image, and the saliency information to a sixth neural network model.

The processor may be configured to identify a screen type of the second image based on the second image, the obtained heat map information and the type information of the at least one object.

The memory includes a plurality of sets of filters corresponding to a plurality of screen types, and the processor is configured to identify a set of filters corresponding to the screen type of the second image from among the plurality of sets of filters.

The processor may be configured to control the display to provide information regarding the screen type together on a live view including the corrected first image.

The second lens is capable of obtaining an image having a wider angle of view than the first lens.

A controlling method of an electronic apparatus comprising a camera including a first lens and a second lens capable of obtaining an image having a wider angle of view than the first lens according to an embodiment includes providing a first image obtained using the first lens as a live view, obtaining a second image using the second lens while providing the first image as a live view, obtaining information regarding the second image and information regarding at least one object included in the second image using at least one neural network model, identifying a screen type of the second image based on the information regarding the second image and the information regarding at least one object included in the second image, identifying a set of filters corresponding to the screen type of the second image, and correcting the first image provided as the live view based on the identified set of filters.

The information regarding the second image may include depth map information corresponding to the second image and saliency information corresponding to the second image, and the obtaining information regarding the at least one object may include obtaining the depth map information by inputting the second image to a first neural network model, and obtaining the saliency information by inputting the second image to a second neural network model.

The information regarding at least one object included in the second image may include at least one of type information of the at least one object, three-dimensional location information of the at least one object, information regarding an area where the at least one object is located, or posture information of the at least one object, and the obtaining information regarding the at least one object may include obtaining segmentation information in which the at least one object and a background included in the second image are segmented and type information of the at least one object by inputting the second image to a third neural network model, obtaining the three-dimensional location information of the at least one object and the information regarding an area where the at least one object is located based on the segmentation information and the depth map information, and obtaining the posture information of the at least one object by inputting the information regarding the at least one object included in the segmentation information to a fourth neural network model.

The identifying a screen type of the second image may include obtaining relationship information between the at least one object by inputting the three-dimensional location information of the object and the information regarding an area where the object is located to a fifth neural network model.

The identifying a screen type of the second image may include obtaining heat map information corresponding to corrected saliency information by inputting the relationship information, the posture information of the at least one object, focus information regarding the second image, and the saliency information to a sixth neural network model.

The identifying a screen type of the second image may include identifying a screen type of the second image based on the second image, the obtained heat map information and the type information of the at least one object.

The electronic apparatus stores a plurality of sets of filters corresponding to a plurality of screen types, and the identifying a set of filters may include identifying a set of filters corresponding to the screen type of the second image from among the plurality of sets of filters.

The method may include providing information regarding the screen type together on a live view including the corrected first image.

According to the above-described various embodiments, an electronic apparatus may identify the type of screen of a currently captured screen more accurately to provide an accurate screen filter effect on the currently captured image.

The present embodiments may be variously modified and have various embodiments, and specific embodiments of the present disclosure will be shown in the drawings and described in detail in the detailed description. However, it is to be understood that the present disclosure is not limited to the specific embodiments, and includes all modifications, equivalents, and alternatives as long as they fall into the scope of the appended claims. In connection with the description of the drawings, like reference numerals may be used for like components.

In describing the present disclosure, when it is decided that a detailed description for the known art related to the present disclosure may obscure the gist of the present disclosure, the detailed description will be omitted.

In addition, the following embodiments may be modified in many different forms, and the scope of the technical teaching of the present disclosure is not limited to the following embodiments. Rather, these embodiments are provided so that the disclosure will be more through and complete and the technical teaching of the present disclosure is fully conveyed to those skilled in the art.

Terms used in this disclosure are used only to describe specific embodiments, and are not intended to limit the scope of rights. Singular expressions include plural expressions unless the context clearly indicates otherwise.

In the disclosure, an expression "have," "may have, "include," or "may include" indicates existence of a corresponding feature (for example, a numerical value, a function, an operation, or a component such as a part), and does not exclude existence of an additional feature.

In the disclosure, an expression "A or B," "at least one of A and/or B," or "one or more of A and/or B," may include all possible combinations of items enumerated together. For example, "A or B," "at least one of A and B," or "at least one of A or B" may indicate all of <NUM>) a case where at least one A is included, <NUM>) a case where at least one B is included, or <NUM>) a case where both of at least one A and at least one B are included.

Expressions "first" or "second" used in the disclosure may indicate various components regardless of a sequence and/or importance of the components, will be used only to distinguish one component from the other components, and do not limit the corresponding components.

When it is mentioned that any component (for example, a first component) is (operatively or communicatively) coupled to or is connected to another component (for example, a second component), it is to be understood that any component is directly coupled to another component or may be coupled to another component through the other component (for example, a third component).

On the other hand, when it is mentioned that any component (for example, a first component) is "directly coupled" or "directly connected" to another component (for example, a second component), it is to be understood that the other component (for example, a third component) is not present between any component and another component.

An expression "~configured (or set) to" used in the disclosure may be replaced by an expression "~suitable for," "~having the capacity to," "~designed to," "~adapted to," "~made to," or "~capable of" depending on a situation. A term "~configured (or set) to" may not necessarily mean "specifically designed to" in hardware.

Instead, in some situations, an expression "~apparatus configured to" may mean that the apparatus may "do" together with other apparatuses or components. For example, a "sub-processor configured (or set) to perform A, B, and C" may mean a dedicated processor (for example, an embedded processor) for performing the corresponding operations or a generic-purpose processor (for example, a central processing unit (CPU) or an application processor) that may perform the corresponding operations by executing one or more software programs stored in a memory device.

In the disclosure, the term "module" or "unit" performs at least one function or operation, and may be embodied as hardware, software, or a combination thereof. A plurality of "modules" or a plurality of "units" may be integrated into at least one module to be implemented as one processor, except a "module" or "unit" which is described as embodied as particular hardware.

Meanwhile, various components and areas in the drawings are schematically drawn. Therefore, the technical spirit of the present disclosure is not limited by the relative size or spacing drawn in the accompanying drawings.

Meanwhile, the electronic apparatus according to an embodiment may include at least one of smartphones, tablet personal computers (PCs), desktop PCs, laptop PCs, or wearable devices. Here, the wearable device may include at least one of an accessory type of a device (e.g., a timepiece, a ring, a bracelet, an anklet, a necklace, glasses, a contact lens, or a head-mounted-device (HMD)), one-piece fabric or clothes type of a circuit (e.g., electronic clothes), a body-attached type of a circuit (e.g., a skin pad or a tattoo), or a bio-implantable type of a circuit.

According to some embodiments, the electronic apparatus may include at least one of televisions (TVs), digital video desk (DVD) players, audios, refrigerators, air-conditioners, cleaners, ovens, microwave ovens, washing machines, air cleaners, set-top boxes, home automation control panels, security control panels, media boxes (e.g., Samsung HomeSync™, Apple TV™, or Google TV™), game consoles (e.g., Xbox™ or PlayStation™), electronic dictionaries, electronic keys, camcorders, electronic picture frames, or the like.

Hereinafter, embodiments according to the present disclosure will be described in detail so that those skilled in the art can easily implement the present disclosure with reference to accompanying drawings.

Hereinafter, the present disclosure will be described in greater detail with reference to the drawings. <FIG> is a block diagram illustrating configuration of an electronic apparatus according to an embodiment. An electronic apparatus <NUM> includes a memory <NUM>, a camera <NUM>, a display <NUM>, and a processor <NUM>. In this case, the electronic apparatus <NUM> may be implemented as a smartphone. However, the electronic apparatus according to an embodiment is not limited to a specific type of device, and it may be implemented as various types of electronic apparatuses <NUM> such as a tablet PC, a digital camera, etc..

The memory <NUM> may store data used by a module for correcting an image according to a screen type of the image to perform various operations. Modules for correcting an image may include an image preprocessing module <NUM>, a screen element detection module <NUM>, a screen analysis module <NUM>, an image correction module <NUM>, and a live view providing module <NUM>. In addition, the memory <NUM> may detect a screen element included in the image, and store a plurality of neural network models to determine a screen type based on the detected element.

Meanwhile, the memory <NUM> may include a non-volatile memory capable of maintaining stored information even if power supply is interrupted, and a volatile memory which uses a continuous power supply in order to maintain stored information. Data for performing various operations by a module for correcting an image according to a screen type of the image may be stored in a non-volatile memory. In addition, a plurality of neural network models may also be stored in the memory in order to detect a screen element included in the image and determine a screen type based on the detected element. Further, the memory <NUM> may store a plurality of filter sets corresponding to a plurality of screen types.

In addition, the memory <NUM> may include at least one buffer that temporarily stores a plurality of image frames obtained through each of a plurality of lenses included in the camera <NUM>.

The camera <NUM> includes a plurality of lenses that are different from each other (e.g., a first lens <NUM>, a second lens <NUM>). Here, the fact that the plurality of lenses are different from each other may include a case in which the field of view (FOV) of each of the plurality of lenses is different from each other and a case in which the positions of each of the plurality of lenses are different from each other. For example, as illustrated in <FIG>, the camera <NUM> of the electronic apparatus <NUM> may include a telephoto lens <NUM>, a wide angle lens <NUM> and an ultra wide angle lens <NUM>, which are disposed on the back of the electronic apparatus <NUM>, and it may also include a three dimensional depth lens <NUM>. In addition to the telephoto lens <NUM>, the wide angle lens <NUM> and the ultra wide angle lens <NUM> disposed on the back of the electronic apparatus <NUM>, a telephoto lens (not illustrated) disposed on the front of the electronic apparatus <NUM> may be further included. In other words, there is no particular limit to the number and type of lenses according to the present disclosure. In this case, the telephoto lens <NUM> has a wider angle of view than a ultra telephoto lens, a standard lens has a wider angle of view than the telephoto lens <NUM>, the wide angle lens <NUM> has a wider angle of view than the standard lens, and the ultra wide angle lens <NUM> has a wider angle of view than the wide angle lens <NUM>. For example, the angle of view of the ultra telephoto lens may be <NUM> degrees to <NUM> degrees, the angle of view of the telephoto lens <NUM> may be <NUM> degrees to <NUM> degrees, the angle of view pf the standard lens may be <NUM> degrees, the angle of view of the wide angle lens <NUM> may be <NUM> degrees to <NUM> degrees, and the angle of view of the ultra wide angle lens <NUM> may be <NUM> degrees to <NUM> degrees.

As the angle of view of the lens is wide, an image frame obtained through the lens may include a relatively wide range of scenes, whereas the size of an object included in the image frame may be relatively small and an exaggeration of perspective may occur. Meanwhile, as the angle of view of the lens narrows, the image frame obtained through the lens may enlarge the size of the object and include the enlarged object, whereas only a relatively narrow range of scenes may be included.

However, for convenience of description, a case in which the camera <NUM> of the electronic apparatus <NUM> includes two lenses, the first lens <NUM> (e.g., a wide angle lens) and the second lens <NUM> (e.g., a ultra wide angle lens) will be mainly described.

In addition, the camera <NUM> may further include an Image Signal Processor (ISP) for processing signals obtained through a plurality of lenses in addition to the plurality of lenses.

The display <NUM> displays a live view of an image captured through the camera <NUM> (particularly, the first lens <NUM>). In addition, the display <NUM> may further provide information on the screen type on the live view including the corrected first image.

Meanwhile, the display <NUM> may be implemented as a Liquid Crystal Display Panel (LCD), an Organic Light Emitting Diodes (OLED), etc., and the display <NUM> may also be implemented as a flexible display, a transparent display, etc. in some cases. However, the display <NUM> according to an embodiment is not limited to a specific type.

The processor <NUM> is electrically connected to the memory <NUM> to control the overall functions and operations of the electronic apparatus <NUM>.

Based on a camera application being executed or a user command for correcting image according to the type of screen is input, the processor <NUM> may load data for a module to perform various operations for correcting the image according to the screen type stored in a memory onto a volatile memory. Subsequently, the processor <NUM> may detect a screen element, and load a plurality of neural network models for determining the screen type according to the detected screen element onto the volatile memory. The processor <NUM> may perform various operations through various modules and neural network models based on the data loaded onto the volatile memory. Here, loading means an operation of loading and storing data stored in a non-volatile memory onto the volatile memory so that the processor <NUM> can access it.

In particular, based on a camera application being executed by a first user command, the processor <NUM> may obtain at least one image through the camera <NUM>.

In particular, the processor <NUM> obtains a first image through the first lens <NUM> included in the camera <NUM>. The processor <NUM> may provide the first image obtained through the first lens <NUM> as a live view image through a live view providing module <NUM>. In this case, the live view image is an image that is output to the display <NUM> immediately after the image input to a sensor included in the camera <NUM> is processed, and a user may adjust the composition, focus, exposure, etc. of the screen using the live view image.

While the first image obtained through the first lens <NUM> is provided as the live view image, the processor <NUM> may obtain a second image through the second lens <NUM> included in the camera <NUM>. In this case, the second lens <NUM> is a lens capable of obtaining an image with a wider angle of view than the first lens <NUM>. For example, the first lens <NUM> may be implemented as the wide angle lens <NUM>, and the second lens <NUM> may be implemented as the ultra wide angle lens <NUM>.

The processor <NUM> may perform preprocessing of the second lens obtained through the second lens using an image preprocessing module <NUM>. In this case, the image preprocessing module <NUM> may resize the second image and normalize the brightness and contrast of the second image.

The processor <NUM> may detect a screen element included in the preprocessed second image using a screen element detection module <NUM>. Specifically, the screen element detection module <NUM> may obtain information on the second image and information on at least one object included in the second image using at least one neural network model. In this case, the information on the second image may include depth map information corresponding to the second image, saliency information corresponding to the second image, and focus information of the second image. In addition, the information on at least one object included in the second image may include type information of the at least one object, three-dimensional position information of the at least one object, information on an area where the at least one object is located, and posture information of the at least one object.

A method in which the screen element detection module <NUM> obtains information on an image and information on an object using at least one module and a neural network model will be described with reference to <FIG>. The screen element detection module <NUM> may include a screen element segmentation module <NUM>, a depth information acquisition module <NUM>, a saliency information acquisition module <NUM>, and an object posture acquisition module <NUM> as illustrated in <FIG>.

The screen element segmentation module <NUM> may obtain segmentation information in which elements included in the second image <NUM> obtained through the second lens are segmented using a neural network model trained to segment screen elements. Here, the elements included in the second image <NUM> may include at least one object element and background element included in the second image <NUM>. In particular, the screen element segmentation module <NUM> may segment screen elements through panoptic segmentation. Panoptic segmentation may obtain information on background elements (e.g., background area information, background type information, etc.) through semantic segmentation, and obtain information on each of at least one object element (e.g., location information of at least one object, area information of at least one object, type information of at least one object, etc.) through instance segmentation.

The depth information acquisition module <NUM> obtains depth information corresponding to the second image <NUM> by inputting the second image <NUM> to a neural network model trained to obtain depth information of an image. In this case, the depth information indicates depth information of elements included in the second image <NUM>, and may be expressed in gray scale.

The saliency information acquisition module <NUM> obtains saliency information corresponding to the second image <NUM> by inputting the second image <NUM> to a neural network model trained to obtain saliency information corresponding to an image. In this case, the saliency information means information indicating a degree of salience compared to the surroundings of pixels included in the image. In this case, the electronic apparatus <NUM> may obtain heat map information of the image as illustrated in <FIG> based on the obtained saliency information.

In addition, the object posture acquisition module <NUM> may obtain posture information of at least one object included in the second image <NUM> by inputting information on at least one object included in segmentation information to a neural network model trained to obtain posture information of an object. In this case, the posture information of at least one object may include not only posture information of objects but also gaze information of the objects.

In addition, the screen element detection module <NUM> may obtain three dimensional location information of at least one object and information on an area where the at least one object is located based on screen element segmentation information (particularly, object area information, object location information, etc.) and depth map information.

Through the above-described method, the screen element detection module <NUM> may obtain second image information/object information <NUM>. Specifically, the screen element detection module <NUM> may obtain saliency information corresponding to the second image <NUM> and focus information of the second image <NUM> as information regarding an image, and obtain type information of an object, posture information of an object, three dimensional location information of an object, and information on an area where an object is located as information regarding an object.

Referring back to <FIG>, the processor <NUM> may obtain information regarding a screen type using a screen analysis module <NUM>. Specifically, the screen analysis module <NUM> may obtain information regarding a screen type by analyzing the screen based on image information and object information obtained through the screen element detection module <NUM>.

A method in which the screen analysis module <NUM> obtains information regarding a screen type using at least one module and a neural network model will be described with reference to <FIG>. The screen analysis module <NUM> may include an object relationship analysis module <NUM>, a heat map acquisition module <NUM> and a screen classification module <NUM>. In this case, the screen analysis module <NUM> may further include a screen configuration element DB <NUM>, but this is only an example. The screen configuration element DB <NUM> may exist as another component (e.g., the memory <NUM>) inside the electronic apparatus <NUM> or may exist outside the electronic apparatus <NUM>.

The object relationship analysis module <NUM> may obtain information regarding a relationship between objects included in the second image by inputting object information to a neural network model trained to obtain information regarding a relationship between objects. In this case, the relationship between objects may include information about an inclusive relationship between object, information regarding a distribution between objects classified into the same class, and the like.

The heat map acquisition module <NUM> may correct saliency information by inputting information regarding a relationship between objects, posture information of at least one object, focus information regarding the second image, and saliency information to a neural network model trained to correct saliency information. In other words, the heat map acquisition module <NUM> may determine the importance of the objects based on the relationship information between objects output from the object relationship analysis module <NUM>, the posture information of objects, the focus information, etc., and may correct the saliency information based on the importance of the objects. In other words, the saliency information may be corrected so that the saliency values of objects determined to be of high importance are increased and the saliency values objects determined to be of low importance are decreased. For example, the heat map acquisition module <NUM> may obtain a heat map <NUM> based on the saliency information obtained by the saliency information acquisition module <NUM> as illustrated in the left side of <FIG>. However, the heat map acquisition module <NUM> may correct the saliency information based on the information regarding a relationship between objects, the posture information of at least one object, and the focus information regarding the second image, and obtain a heat map <NUM>, as illustrated in the right side of <FIG>, based on the corrected saliency information. In other words, considering that food is placed in the central area, people's postures or gazes are not directed toward the camera, and the focus of the image is on the food, the food object is determined to be more important than the human object. Thus, as illustrated in the left side of <FIG>, the heat map <NUM> as illustrated in the left side of <FIG> may be corrected to the heat map <NUM> as illustrated in the right side of <FIG>.

The screen classification module <NUM> may obtain information <NUM> regarding a screen type of the second image based on the second image, corrected heat map information and type information of at least one object. In particular, the screen classification module <NUM> may identify the screen type of the second image using the screen configuration element DB <NUM>. In this case, the screen configuration element DB <NUM> is a database for storing a general relationship between screen configuration elements extracted from a large-scale image data set, and may have an upper and lower hierarchical structure as illustrated in <FIG>. The numbers shown in each element illustrated in <FIG> may be numbers indicating the probability that a higher class would exist in the corresponding element.

For example, the screen classification module <NUM> may determine a food object which is an object having a high heat value, as an important object based on the corrected heat map information <NUM> illustrated in the right side of <FIG> and the object type information, and determine that the type of the current screen is a restaurant based on information regarding the food object that is determined to be an important object and the screen configuration element DB <NUM>.

Hereinafter, various embodiments in which the screen classification module <NUM> determines a screen type will be described with reference to <FIG>.

<FIG> is an image in which a picture of an elephant exist in a monitor. Existing object detection-based technology classified the screen type as an animal due to an elephant present in the central area of the screen, but the electronic apparatus <NUM> according to an embodiment may determine a relationship between the elephant and the monitor and classify the screen type as indoor or an office by determining that a desk and the like exist.

<FIG> is an image of a landscape including a plurality of people. Existing object detection-based technology may classify the screen type as a portrait due to people placed on the screen, but the electronic apparatus <NUM> according to an embodiment may determine posture information (i.e., gaze information) of the people, determine a relationship between the people and the landscape, and determine that the landscape is located in the central area to classify the screen type as a scenery.

<FIG> is an image of a restaurant including a plurality of people. Existing object detection-based technology classified the screen type as a portrait due to the people placed on the screen, but the electronic apparatus <NUM> according to an embodiment may determine posture information of the people, determine a relationship between the people, food and household items, and determine that the food and the household items are located in the central area to classify the screen type as a restaurant.

<FIG> is an image of making a hamburger. Existing object detection-based technology classified the screen type as a restaurant due to bread and hamburgers placed in the screen, but the electronic apparatus <NUM> according to an embodiment may determine a relationship between people, and classify the screen type as food.

<FIG> is an image including people. Existing object detection-based technology classified the screen type as a restaurant due to food placed in the screen, but the electronic apparatus <NUM> according to an embodiment may determine posture information and gaze information of the people and a relationship between the people and the food and classify the screen type as a portrait.

As illustrated in <FIG>, conventionally, screen types have been classified simply through representative objects present in an image, but screen types can be classified more accurately by classifying them after analyzing a relationship between objects as shown in an embodiment of the present disclosure.

Referring back to <FIG>, the processor <NUM> may correct the first image provided as a live view through the image correction module <NUM>. Specifically, the image correction module <NUM> may identify a set of filters for correcting the first image based on information regarding a screen type obtained through the screen analysis module <NUM>, and correct the first image provided as a live view using the identified filter set.

According to an embodiment, the image correction module <NUM> may obtain a set of filters corresponding to a screen type obtained through the screen analysis module <NUM> from among a plurality of filter sets stored in the memory <NUM>. Specifically, as illustrated in <FIG>, the memory <NUM> stores a set of filters for correcting a plurality of image quality factors (e.g., saturation, brightness, white balance (WB), gamma correction, etc.) corresponding to each screen type. In addition, the image correction module <NUM> may obtain a set of filters corresponding to a screen type obtained through the screen analysis module <NUM> from among a plurality of filter sets stored in the memory <NUM>. The image correction module <NUM> may correct the first image provided as a live view based on the obtained filter set.

According to another embodiment, the image correction module <NUM> may obtain filter set information from an external device (or an external server). Specifically, the image correction module <NUM> may transmit information regarding a screen type obtained through the screen analysis module <NUM> to an external device, and obtain filter set information corresponding to the screen type from the external device. The image correction module <NUM> may correct the first image provided as a live view based on the obtained filter set.

Meanwhile, the processor <NUM> may control the display <NUM> to provide information regarding a screen type of the second image together with a live view while providing the corrected first image as a live view. For example, the processor <NUM> may control the display <NUM> to provide information of "restaurant mode" together with the corrected first image on the live view screen.

<FIG> is a view provided to explain a method of correcting a live view to correspond to a screen type according to an embodiment.

As illustrated in <FIG>, the first lens <NUM> (a lens selected by a user, a wide angle lens) may obtain raw data using light received through the outside, and output the obtained raw data to an ISP <NUM>. The ISP <NUM> may obtain the first image by processing the obtained raw data, and the electronic apparatus <NUM> may provide the obtained first image as a live view screen (<NUM>).

While the first image obtained through the first lens <NUM> is provided as a live view screen, the second lens <NUM> (an ultra wide angle lens) may also obtain raw data using light received through the outside, and output the obtained raw data to the ISP <NUM>. The ISP <NUM> may obtain the second image by processing the obtained raw data.

As illustrated in <FIG> and <FIG>, the electronic apparatus <NUM> may detect a screen element using at least one neural network model (<NUM>). In this case, the electronic apparatus <NUM> may output information regarding an image and information regarding an object as information regarding a screen element.

As described in <FIG> and <FIG>, the electronic apparatus <NUM> may analyze a screen based on information regarding an image and information regarding an object (<NUM>). In this case, the electronic apparatus <NUM> may obtain information regarding a screen type by determining a relationship between objects based on the information regarding an image and the information regarding an object.

The electronic apparatus <NUM> may select a set of filters corresponding to the screen type from among a plurality of filter sets stored in a filter set DB <NUM> (<NUM>). The electronic apparatus <NUM> may output and store the first image by correcting the first image provided as a live view screen using the selected filter set (<NUM>). In other words, the electronic apparatus <NUM> may correct the first image provided as a live view screen in real time, and based on a photographing command being input, the first image which is corrected by applying a set of filters to the first image photographed at a time based on the photographing command being input may be stored in the memory <NUM> (particularly, a gallery folder in the memory <NUM>).

<FIG> is a view provided to explain a method of correcting a live view to correspond to a screen type according to another embodiment.

In <FIG>, unlike <FIG>, raw data obtained through the second lens <NUM> need not be input to the ISP <NUM>, and a screen element may be detected using the raw data (<NUM>). In other words, as illustrated in <FIG>, as a screen element is detected by directly using the raw data, power consumption required for processing the ISP <NUM> can be reduced, and heat generation can also be reduced.

In the above-described embodiments, it is described that a screen type for the entire screen is determined and the screen is corrected through a set of filters corresponding to the screen type, but they are only exemplary embodiments. The screen may be divided into a plurality of areas, and the screen may be corrected using a plurality of filter sets corresponding to each of the plurality of divided areas. For example, as illustrated in <FIG>, the electronic apparatus <NUM> may segment an image into a first area <NUM>, a second area <NUM> and a third area <NUM>. The electronic apparatus may identify a first filter set for correcting a person object corresponding to the segmented first area <NUM>, a second filter set for correcting a background corresponding to the segmented second area <NUM> and a third filter set for correcting a food object corresponding to the segmented third area <NUM>. Subsequently, the electronic apparatus <NUM> may correct the first area <NUM> using the first filter set, correct the second area <NUM> using the second filter set and correct the third area <NUM> using the third filter set.

<FIG> is a flowchart provided to explain a controlling method of an electronic apparatus according to an embodiment.

The electronic apparatus <NUM> provides the first image obtained using the first lens as a live view (S910). In this case, the first lens is a lens selected by the user, and may be, for example, a wide angle lens.

The electronic apparatus <NUM> obtains the second image using the second lens while providing the first image as a live view (S920). In this case, the second lens is a lens capable of obtaining an image with a wider angle of view than the first lens, and may be, for example, an ultra wide angle lens.

The electronic apparatus <NUM> obtains information regarding the second image using at least one neural network model and information regarding at least one object included in the second image (S930). In this case, the information regarding the second image include depth map information corresponding to the second image, saliency information corresponding to the second image, focus information of the second image, etc., and particularly, the electronic apparatus <NUM> may obtain the depth map information by inputting the second image to the first neural network model, and obtain the saliency information by inputting the second image to the second neural network model. In addition, the information regarding at least one object included in the second image may include type information of the at least one object, three dimensional location information of the at least one object, information regarding an area where the at least one object is located and posture information of the at least one object. In particular, the electronic apparatus <NUM> may obtain segmentation information in which at least one object and a background included in the second image are segmented by inputting the second image to the third neural network model and type information of the at least one object. In addition, the electronic apparatus <NUM> may obtain the three dimensional location information of the at least one object and the information regarding an area where the at least one object is located based on the segmentation information and the depth map information. Further, the electronic apparatus <NUM> may obtain the posture information of the at least one object by inputting information regarding at least one object included in the segmentation information to the fourth neural network model.

The electronic apparatus <NUM> identifies a screen type of the second image based on the information regarding the second image and the information regarding at least one object included in the second image (S940). Specifically, the electronic apparatus <NUM> may obtain relationship information between at least one object by inputting three dimensional location information of an object and information regarding an area where the object is located to the fifth neural network model. The electronic apparatus <NUM> may obtain corrected saliency information by inputting the relation information, the posture information of at least one object, the focus information regarding the second image, and the saliency information to the sixth neural network model, and obtain heat map information based on the obtained saliency information. The electronic apparatus <NUM> may identify a screen type of the second image based on the second image, the heat map information and the type information of at least one object.

The electronic apparatus <NUM> identifies a set of filters corresponding to the screen type of the second image (S950). Specifically, the electronic apparatus <NUM> may identify a filter set corresponding to the screen type from among a plurality of filter sets stored in the electronic apparatus <NUM>.

The electronic apparatus <NUM> corrects the first image based on the identified filter set and provide the same as a live view (S960). In this case, the electronic apparatus <NUM> may provide information regarding the screen type on the live view while providing the corrected first image as the live view.

<FIG> is a block diagram provided to explain configuration of an electronic apparatus according to an embodiment. As illustrated in <FIG>, an electronic apparatus <NUM> according to an embodiment may include a display <NUM>, a speaker <NUM>, a camera <NUM>, a memory <NUM>, a communication interface <NUM>, an input interface <NUM>, a sensor <NUM>, and a processor <NUM>. However, such configuration is only an example, and new configuration may be added or some configuration may be omitted. Meanwhile, since the display <NUM>, the camera <NUM>, the memory <NUM>, and the processor <NUM> are the same as the display <NUM>, the camera <NUM>, the memory <NUM>, and the processor <NUM> described in <FIG>, overlapping descriptions will be omitted.

The speaker <NUM> may output a voice message. In particular, the speaker <NUM> may provide a guide message including information regarding a screen type in the form of a voice message. In this case, the speaker <NUM> may be included in the electronic apparatus <NUM>, but this is only an example. The speaker <NUM> may be electrically connected to the electronic apparatus <NUM> and may be located outside the electronic apparatus <NUM>.

The communication interface <NUM> includes a circuit, and may perform communication with an external device. Specifically, the processor <NUM> may receive various data or information from a connected external device through the communication interface <NUM>, and transmit various data or information to the external device.

The communication interface <NUM> may include at least one of a WiFi module, a Bluetooth module, a wireless communication module, or an NFC module. Specifically, each of the WiFi module and the Bluetooth module may perform communication in a WiFi method and a Bluetooth method, respectively. Based on the Wi-Fi module or the Bluetooth module being used, various types of connection information such as a service set identifier (SSID) may be transmitted or received, communication may be established using the various connection information, and thereafter various types of information may be transmitted or received.

In addition, the wireless communication module may perform communication according to various communication standards such as IEEE, Zigbee, 3rd Generation (<NUM>), 3rd Generation Partnership Project (3GPP), Long Term Evolution (LTE), 5th Generation (<NUM>), etc. The NFC module may perform communication in a Near Field Communication (NFC) that uses a <NUM> band among various radio frequency-identification (RF-ID) frequency bands such as <NUM>, <NUM>, <NUM>, <NUM> to <NUM>, and <NUM>.

In particular, in various embodiments according to the present disclosure, the communication interface <NUM> may receive various kinds of information such as data related to a neural network model <NUM> from an external device. In addition, the communication interface <NUM> may transmit information regarding a screen type to an external device and receive information regarding a set of filters corresponding to the screen type from the external device.

The input interface <NUM> includes a circuit, and the processor <NUM> may receive a user command for controlling the operation of the electronic apparatus <NUM> through the input interface1060. Specifically, the input interface <NUM> may be implemented in a form in which it is included in the display <NUM>, but this is only an example. The input interface <NUM> may consist of a button, a microphone, a remote control receiver (not illustrated) and the like.

In particular, in various embodiments according to the present disclosure, the input interface1060 may receive various user commands such as a user command for executing a camera application, a user command for photographing an image, a user command for correcting a live view screen with a set of filters corresponding to a current screen type, and the like.

The sensor <NUM> may obtain various information regarding the electronic apparatus <NUM>. In particular, the sensor <NUM> may include a GPS capable of obtaining location information of the electronic apparatus <NUM>, and it may include various sensors such as a biometric sensor (e.g., a heartbeat sensor, a PPG sensor, etc.), a motion sensor for detecting a motion of the electronic apparatus <NUM>, and the like.

The processor <NUM> may control the electronic apparatus <NUM> according to at least one instruction stored in the memory <NUM>. In particular, the processor <NUM> may provide the first image obtained using the first lens <NUM> to the display <NUM> as a live view, obtain the second image using the second lens <NUM> while providing the first image as a live view, obtain information regarding the second image using at least one neural network model and information regarding at least one object included in the second image, identify a screen type of the second image based on the information regarding at least one object included in the second image, identify a set of filters corresponding to the screen type of the second image, and correct the first image provided as a live view based on the identified filter set.

Meanwhile, in the above-described embodiment, it is described that the electronic apparatus <NUM> identifies a screen type based on a relationship between objects in an image photographed using a plurality of lenses and the photographed image is corrected based on a set of filters corresponding to the identified screen type, but this is only an example. The currently captured image can be corrected in association with an external server, which will be described in detail with reference to <FIG>.

Firstly, the electronic apparatus <NUM> may provide the first image obtained using the first lens as a live view (S1110).

Subsequently, the electronic apparatus <NUM> may obtain the second image using the second lens while providing the first image as a live view (S1120). In this case, the second lens may be a lens capable of obtaining an image with a wider angle of view than the first lens.

The electronic apparatus <NUM> may transmit the obtained second image to a server <NUM> (S1130). In this case, as illustrated in <FIG>, the second image may be an image processed through the ISP <NUM>, but this is only an example. The second image may be raw data as described in <FIG>.

The server <NUM> may obtain information regarding the second image and information regarding at least one object included in the second image using at least one neural network model (S1140).

The server <NUM> may identify a screen type of the second image based on the information regarding the second image and the information regarding at least one object included in the second image (S1150).

The server <NUM> may identify a set of filters corresponding to the screen type of the second image (S1160). Specifically, the server <NUM> may identify a set of filters corresponding to the screen type of the second image from among a plurality of filter sets stored in the server <NUM>.

The server <NUM> may transmit the identified filter set to the electronic apparatus <NUM> (S1170).

The electronic apparatus <NUM> may correct the first image based on the filter set transmitted from the server <NUM> and provide the same as a live view (S1180). In this case, the electronic apparatus <NUM> may provide information regarding the screen type on the live view while providing the corrected first image as the live view, and may store the corrected first image captured at a time based on a photographing command being input.

According to another embodiment not falling into the scope of the claims, the electronic apparatus <NUM> may transmit both the first image and the second image to the server <NUM>, and the server <NUM> may identify filter set information based on the second image and correct the first image based on the identified filter set information. In addition, the server <NUM> may transmit the corrected first image to the electronic apparatus <NUM>, and the electronic apparatus <NUM> may immediately provide the corrected first image received from the server <NUM>.

Meanwhile, the function related to a neural network model as in the above description may be performed through a memory and a processor. The processor may consist of one or more processors. In this case, the one or more processors may be general-purpose processors such as a CPU and an AP, graphics-only processors such as a graphics processing unit (GPU) and a vision processing unit (VPU), or AI-only processors, such as a neural processing unit (NPU). One or more processors control input data to be processed according to predefined operation rules or AI models stored in a non-volatile memory and a volatile memory. The predefined operation rules or AI models are characterized in that they are created through learning.

Here, being created through learning means that a predefined operation rule or an AI model having desired characteristics is created by applying a learning algorithm to a plurality of learning data. Such learning may be performed in a device itself in which AI according to the disclosure is performed, or may be performed through a separate server/system.

An AI model may include a plurality of neural network layers. Each of the plurality of neural network layers has a plurality of weight values, and a neural network operation is performed through an operation result of a previous layer and an operation of a plurality of weight values. Examples of neural network layers include a convolutional neural network (CNN), a deep neural network (DNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural networks (BRDNN), a Generative Adversarial Networks (GAN), and deep Q-networks, and the like, and the neural networks in the disclosure are not limited to the above examples except for the cases specified.

A learning algorithm is a method of training a predetermined target device (e.g., a robot) using a plurality of learning data so that the predetermined target device can make a decision or make a prediction by itself. Examples of learning algorithms include supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but are not limited to the above examples except for the cases specified in the present disclosure.

Here, the term 'non-transitory storage medium' refers to a tangible device and should be understood to not include a signal (e.g., an electromagnetic wave) but is not intended to distinguish between a case in which data is semi-permanently stored in the storage medium and a case in which data is temporarily stored in the storage medium. For example, the 'non-transitory storage medium' may include a buffer in which data is temporarily stored.

In an embodiment, methods according to various embodiments as set forth herein may be provided by being included in a computer program product. The computer program product may be traded as a product between a seller and a purchaser. The computer program product may be distributed in the form of a storage medium (e.g., compact disc read only memory (CD-ROM)) that is readable by devices, may be distributed through an application store (e.g., Play Store™) or directly between two user devices (e.g., smartphones), or may be distributed online (e.g., by downloading or uploading). In the case of an online distribution, at least part of the computer program product (e.g., a downloadable application) may be at least temporarily stored in a machine-readable storage medium such as a server of the manufacturer, a server of an application store, or a memory of a relay server or may be temporarily generated.

Each component (e.g., a module or a program) of various embodiments of the disclosure as described above may include a single entity or a plurality of entities, and some of the sub-components described above may be omitted or other sub-components may be further included in various embodiments. Alternatively or additionally, some components (e.g., modules or programs) may be integrated into one entity to perform functions, which are performed by the components prior to the integration, in the same or similar manner.

Operations performed by a module, a program, or another component according to various embodiments may be performed in a sequential, parallel, iterative, or heuristic manner, or at least some of the operations may be performed in a different order or omitted, or other operations may be added.

Meanwhile, the term "part" or "module" used herein includes a unit configured as hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A "part" or "module" may be understood as an integral component or a minimum unit for performing one or more functions or part of the minimum unit. For example, a module may be configured as an application-specific integrated circuit (ASIC).

Various embodiments of the disclosure may be implemented by software including instructions stored in a machine (e.g., a computer) readable storage medium. The machine is a device capable of calling an instruction stored in a storage medium and operating according to the called instruction and may include an electronic apparatus (e.g., the electronic apparatus <NUM>) according to various embodiments set forth herein.

Based on the instruction being executed by a processor, a function corresponding to the instruction may be performed directly by the processor or under control of the processor. The instructions may include code generated or executed by a compiler or an interpreter.

Claim 1:
An electronic apparatus (<NUM>, <NUM>) comprising:
a camera (<NUM>, <NUM>) including a first lens (<NUM>) and a second lens (<NUM>) capable of obtaining an image having a wider angle of view than the first lens (<NUM>);
a display (<NUM>, <NUM>);
a memory (<NUM>, <NUM>); and
a processor (<NUM>) configured to:
provide a first image obtained using the first lens (<NUM>) to the display (<NUM>, <NUM>) as a live view;
obtain a second image using the second lens (<NUM>) while providing the first image as a live view, wherein the second image includes a relatively wider range of a scene than the first image;
obtain information regarding the second image and information regarding at least one object included in the second image using at least one neural network model;
identify a screen type of the second image based on the information regarding the second image and the information regarding at least one object included in the second image, wherein the screen type indicates a classification of the scene included in the second image;
identify a set of filters corresponding to the screen type of the second image; and
correct the first image provided as the live view based on the identified set of filters,
wherein the memory (<NUM>, <NUM>) includes a plurality of sets of filters corresponding to a plurality of screen types,
wherein the processor (<NUM>) is configured to identify the set of filters corresponding to the screen type of the second image from among the plurality of sets of filters,
wherein the information regarding the second image comprises depth map information corresponding to the second image and saliency information corresponding to the second image; and
wherein the processor is configured to obtain the depth map information by inputting the second image to a first neural network model, and obtain the saliency information by inputting the second image to a second neural network model.