Patent Description:
In recent years, artificial intelligence (AI) systems have been used in various fields. An AI system is a system in which a machine learns, judges, and iteratively improves analysis and decision making, unlike an existing rule-based smart system. As the use of AI systems increases, for example, an accuracy, a recognition rate and understanding or anticipation of a user's taste may be correspondingly increased. As such, existing rule-based smart systems are gradually being replaced by deep learning-based AI systems.

AI technology is composed of machine learning, for example deep learning, and elementary technologies that utilize machine learning.

Machine learning is an algorithmic technology that is capable of classifying or learning characteristics of input data. Element technology is a technology that simulates functions, such as recognition and judgment of a human brain, using machine learning algorithms, such as deep learning. Machine learning is composed of technical fields such as linguistic understanding, visual understanding, reasoning, prediction, knowledge representation, motion control, or the like.

Various fields implementing AI technology may include the following. Linguistic understanding is a technology for recognizing, applying, and/or processing human language or characters and includes natural language processing, machine translation, dialogue system, question and answer, voice recognition or synthesis, and the like. Visual understanding is a technique for recognizing and processing objects as human vision, including object recognition, object tracking, image search, human recognition, scene understanding, spatial understanding, image enhancement, and the like. Inference prediction is a technique for judging and logically inferring and predicting information, including knowledge-based and probability-based inference, optimization prediction, preference-based planning, recommendation, or the like. Knowledge representation is a technology for automating human experience information into knowledge data, including knowledge building (data generation or classification), knowledge management (data utilization), or the like. Motion control is a technique for controlling the autonomous running of the vehicle and the motion of the robot, including motion control (navigation, collision, driving), operation control (behavior control), or the like.

For efficient management of information and various user experiences, electronic apparatuses have provided a function to generate tag information for an image. Tag information is data for an image and a kind of metadata.

As for the related-art tag information generation method, it is general that the tag information is generated uniformly only with information that may be identified in the same manner by anyone from an image, and a unique thinking, feeling, or the like, of a user associated with an image is not reflected as tag information.

<CIT> discusses a method of processing audio signals recorded during display of image data from a media file on a display device. <CIT> discusses a method by which a document image is analyzed for the purposes of establishing a searchable data structure. <CIT> discusses displaying a normalized object.

The disclosure provides an electronic apparatus capable of generating tag information based on a user voice and a control method thereof.

Hereinafter, embodiments of the disclosure will be described with reference to the accompanying drawings. The embodiments according to the invention are described with reference to <FIG>, <FIG> and <FIG>. The other embodiments are not encompassed by the wording of the claims but are considered as useful for understanding the invention.

However, it should be understood that the disclosure is not limited to the specific embodiments described hereinafter, but includes various modifications, equivalents, and/or alternatives of the embodiments of the present disclosure. In relation to explanation of the drawings, similar drawing reference numerals may be used for similar constituent elements.

In this specification, the expressions "have," "may have," "include," or "may include" or the like represent presence of a corresponding feature (for example: components such as numbers, functions, operations, or parts) and does not exclude the presence of additional feature.

In the description, the term "A or B", "at least one of A or/and B", or "one or more of A or/and B" may include all possible combinations of the items that are enumerated together. For example, the term "at least one of A or/and B" means (<NUM>) including at least one A, (<NUM>) including at least one B, or (<NUM>) including both at least one A and at least one B.

As used herein, the terms "first," "second," or the like may denote various components, regardless of order and/or importance, and may be used to distinguish one component from another, and does not limit the components. For example, a first user device and a second user device may indicate different user devices regardless of a sequence or importance thereof. For example, the first component may be named the second component and the second component may also be similarly named the first component, without departing from the scope of the disclosure.

Terms such as "module," "unit," "part," and so on are used to refer to an element that performs at least one function or operation, and such element may be implemented as hardware or software, or a combination of hardware and software. Further, except for when each of a plurality of "modules," "units," "parts," and the like needs to be realized in an individual hardware, the components may be integrated in at least one module or chip and be realized in at least one processor.

If it is described that a certain element (e.g., first element) is "operatively or communicatively coupled with/to" or is "connected to" another element (e.g., second element), it should be understood that the certain element may be connected to the other element directly or through still another element (e.g., third element. On the other hand, if it is described that a certain element (e.g., first element) is "directly coupled to" or "directly connected to" another element (e.g., second element), it may be understood that there is no element (e.g., third element) between the certain element and the another element.

Also, the expression "configured to" used in the disclosure may be interchangeably used with other expressions such as "suitable for," "having the capacity to," "designed to," "adapted to," "made to," and "capable of," depending on cases. Meanwhile, the term "configured to" does not necessarily mean that a device is "specifically designed to" in terms of hardware. Instead, under some circumstances, the expression "a device configured to" may mean that the device "is capable of" performing an operation together with another device or component. For example, the phrase "a processor configured to perform A, B, and C" may mean a dedicated processor (e.g., an embedded processor) for performing the corresponding operations, or a generic-purpose processor (e.g., a central processing unit (CPU) or an application processor) that can perform the corresponding operations by executing one or more software programs stored in a memory device.

Terms used in the disclosure may be used to describe specific embodiments rather than restricting the scope of other embodiments. Terms used in the disclosure including technical and scientific terms may have the same meanings as those that are generally understood by those skilled in the art to which the disclosure pertains. Terms defined in a general dictionary among terms used in the disclosure may be interpreted as meanings that are the same as or similar to meanings within a context of the related art, and are not interpreted as ideal or excessively formal meanings unless clearly defined in the disclosure. In some cases, terms may not be interpreted to exclude embodiments of the disclosure even where they may be defined in the disclosure.

The electronic apparatus according to various embodiments may include at least one of a smartphone, a tablet personal computer (PC), a mobile phone, a video phone, an e-book reader, a desktop PC, a laptop computer, a netbook computer, a workstation, a server, a personal digital assistant (PDA), a portable multimedia player (PMP), a moving picture experts group phase <NUM> or phase <NUM> (MPEG-<NUM> or MPEG-<NUM>) audio layer-<NUM> (MP3) player, a mobile medical device, a camera, or a wearable device. According to various embodiments, the wearable device may be of the type of accessory (e.g., a watch, a ring, a bracelet, a bracelet, a necklace, glasses, a contact lens or a head-mounted-device (HMD)), a fabric or clothing (e.g., an electronic garment), a body attachment (e.g., a skin pad or tattoo), or a bio-implantable (e.g., implantable circuit).

In another embodiment, the electronic apparatus may be a home appliance. Home appliances may include at least one of, for example, televisions (TVs), digital video disc (DVD) players, audio, refrigerators, air-conditioners, vacuum cleaners, ovens, microwaves, washing machines, air cleaners, set-top box, home automation control panel, a security control panel, a TV box such as Samsung HomeSync ™, Apple TV ™ or Google TV ™) a game console (e.g., Xbox ™, PlayStation ™), an e-dictionary, an electronic key, a camcorder, or an electronic photo frame.

In another embodiment, the electronic apparatus may include at least one of a variety of medical devices such as various portable medical measurement devices (a blood glucose meter, a heart rate meter, a blood pressure meter or a temperature measuring device, a magnetic resonance angiography (MRA), a magnetic resonance imaging (MRI), computed tomography (CT), a photographing device, or an ultrasonic device, etc.), a navigator, global navigation satellite system (GNSS), an event data recorder (EDR), a flight data recorder (FDR), automotive infotainment devices, marine electronic equipment (e.g., marine navigation devices, gyro compasses, etc.), avionics, security devices, head units for the vehicle, industrial or home robots, automatic teller's machine (ATM), point of sales (POS) of a store, or internet of things (e.g., light bulbs, various sensors, electricity or gas meters, sprinkler devices, fire alarms, thermostats, street lights, a toaster, a fitness equipment, a hot water tank, a heater, a boiler, etc.).

In another embodiment, the electronic apparatus may include at least one of a piece of furniture or a part of a building / structure, an electronic board, an electronic signature receiving device, a projector, or various measuring devices (e.g., water, electricity, gas, or radio wave measuring instruments, etc.). In various embodiments, the electronic apparatus may be a combination of one or more of the various devices described above. The electronic apparatus according to some embodiments may be a flexible electronic apparatus. Further, the electronic apparatus according to the embodiment of the disclosure is not limited to the above-described devices, and may include a new electronic apparatus according to technological advancement.

The disclosure relates to an art for tagging information in an image, and more particularly, to generating tag information for an object included in an image more easily based on a voice (or speech). Tag information refers to data for image as a kind of metadata.

<FIG> is a use diagram of an electronic apparatus <NUM> to tag information to an image according to an embodiment.

Referring to <FIG>, the electronic apparatus <NUM> may display an image <NUM> that includes at least one object, and information included in the user voice may be tagged to the object included in the image <NUM>. Here, the object may be any of a person, an object, or the like, which is present in a distinguishable manner from others. The object may be referred to as an entity.

The user may give a voice describing about the image <NUM> while the image <NUM> is being displayed in the electronic apparatus <NUM>. The user voice describing the image <NUM> may be input through a microphone provided in the electronic apparatus <NUM>. Alternatively, the user voice may be input through an external microphone electrically connected to the electronic apparatus <NUM>.

An object associated with a voice of the objects included in the image <NUM>, that is, a target object to be tagged with information, may be identified. The voice may include at least one description for identifying a target object to be tagged with information. For example, the voice may include at least one of a description of an appearance of the target object, a description of a gender of the target object, a description of a color of the target object, a description of a position of the target object, a description of a category of the target object, a description of a name of the target object, or the like. According to an embodiment, the target object in an image may be identified by inputting a voice and an image to an artificial intelligence (AI) model trained by an AI algorithm.

In an example of <FIG>, the target object <NUM> may be identified in the image <NUM> based on a voice that "the left is my son, Junseo.

When an object related to a voice is identified, the electronic apparatus <NUM> may display a user interface (UI) element notifying that the identified object is the target object to be tagged. For example, as illustrated in <FIG>, the electronic apparatus <NUM> may display a rectangular UI element surrounding the target object <NUM> to be tagged.

Tag information <NUM> for the target object <NUM> may be obtained based on the voice. In <FIG>, the words included in the voice, "son" and "Junseo", are tagged to the target object <NUM>. According to an embodiment, a voice is input to an AI model trained by an AI algorithm to obtain a keyword from the voice, and the obtained keyword may be tagged to the target object. The electronic apparatus <NUM> may display the tagged keywords with the image <NUM>.

The keyword of the input user voice subsequently input may be additionally tagged to the target object <NUM>. Referring to the embodiment of <FIG>, "<FIG> years" and "little rascal" are additionally tagged. The tag information <NUM> may be displayed with the image <NUM>. As illustrated in <FIG>, a keyword of the voice subsequently input is added as the tag information and may be displayed together with the keyword previously displayed.

<FIG> is a block diagram illustrating a configuration of the electronic apparatus <NUM> according to an embodiment.

Referring to <FIG>, the electronic apparatus <NUM> includes a display <NUM>, a memory <NUM>, a microphone <NUM>, and a processor <NUM>. According to an implementation type, some configurations may be omitted, and some hardware / software configurations which are not illustrated but are obvious to those skilled in the art may be additionally included in the electronic apparatus <NUM>.

The display <NUM> may include, for example, a liquid crystal display (LCD), light-emitting diode (LED) display, an organic light-emitting diode (OLED), a display (for example, active-matrix organic light-emitting diode (AMOLED), passive-matrix OLED (PMOLED)), or a microelectromechanical systems (MEMS) display, or an electronic paper display.

The display <NUM> may display various images, for example, images including texts, images, videos, icons, or symbols.

According to an embodiment, the display <NUM> may include a touch screen and may receive a touch, a gesture, proximity, or hovering input using, for example, an electronic pen or a part of the user's body.

The memory <NUM> may, for example, include an embedded memory or an external memory. The embedded memory may, for example, include at least one of a volatile memory (e.g. dynamic random access memory (DRAM)), a static RAM, a synchronous dynamic RAM (SDRAM), a non-volatile memory (e.g. one time programmable read only memory (OTPROM)), a programmable ROM (PROM), an erasable and programmable ROM (EPROM), an electrically erasable and programmable ROM (EEPROM), a mask ROM, a flash ROM, a flash memory (e.g. NAND flash or NOR flash, etc.), a hard drive, and a solid state drive (SSD).

The external memory may further include a flash drive, such as a compact flash (CF), a secure digital (SD), a micro secure digital (Micro-SD), a mini secure digital (Mini-SD), an extreme digital (xD), a multi-media card (MMC), or a memory stick. The external memory may be connected to the electronic apparatus <NUM> functionally and/or physically through various interfaces.

The memory <NUM> is accessed by the processor <NUM>, and read / write / modify / delete / update of data by the processor <NUM> may be performed. The term memory in this document refers to the memory <NUM> (not shown), a ROM in the processor <NUM>, a RAM, or a memory card (for example, a micro SD card and a memory stick) provided on the electronic apparatus <NUM>. The memory <NUM> may store computer executable instructions. In the memory <NUM>, a program, data, or the like, for configuring various screens to be displayed in a display region of the display <NUM> may be stored.

According to an embodiment, the memory <NUM> may be implemented with not only a storage medium inside the electronic apparatus <NUM> but also a web server, or the like, through network.

The microphone <NUM> is a structure for receiving sound. The microphone <NUM> may convert the received sound into an electrical signal. The microphone <NUM> may be implemented integrally with or separated from the electronic apparatus <NUM>. The separated microphone <NUM> may be electrically connected to the electronic apparatus <NUM>.

The processor <NUM> is a configuration for controlling the overall operation of the electronic apparatus <NUM>. The processor <NUM> may be implemented, for example, as a central processing unit (CPU), an application specific integrated circuit (ASIC), a system on chip (SoC), a MICOM, or the like. The processor <NUM> may operate an operating system or an application program to control a plurality of hardware or software components connected to the processor <NUM>, and may perform various data processing and operations. According to one embodiment, the processor <NUM> may further include a graphics processing unit (GPU) and / or an image signal processor. The processor <NUM> may execute the computer executable instructions to perform the functions according to various embodiments of the disclosure.

The processor <NUM> may perform various operations in association with the display <NUM>, the memory <NUM>, and the microphone <NUM>, by executing a computer-executable instruction stored in the memory <NUM>.

For example, the processor <NUM> may control the display <NUM> to display an image including at least one object. The processor <NUM> may identify an object related to a voice among at least one object included in an image displayed in the display <NUM> and generate tag information for the identified object, by inputting a user voice received through the microphone <NUM> to a model trained by the AI algorithm. The processor <NUM> may provide the generated tag information.

For example, referring to <FIG>, when user voice is input through the microphone <NUM> of the electronic apparatus <NUM> while the image <NUM> is displayed through the display <NUM> of the electronic apparatus <NUM>, the processor <NUM> may input voice to the AI model trained by the AI algorithm to identify the object <NUM> associated with the voice, generate tag information <NUM> for the object, and provide the tag information <NUM> via the display <NUM>.

The AI model trained by the AI algorithm may be stored in the memory <NUM>. The AI model trained by the AI algorithm may be stored in a server external to the electronic apparatus <NUM> so that when the electronic apparatus <NUM> transmits the image and user voice (voice data) to the server, the external server may identify the object related to the voice in the image and transmit the result to the electronic apparatus <NUM>.

According to an embodiment, a content of the voice of the user from a point designated by the user may be tagged to an object in the image. That is, the tagging function of the electronic apparatus <NUM> may be initiated at a point of time desired by the user. A variety of ways may be used to initiate the tagging function.

<FIG> and <FIG> are views illustrating various embodiments related to executing a tagging function.

Referring to <FIG>, the electronic apparatus <NUM> according to one embodiment of may provide a UI element <NUM> for initiating a tagging function for the image <NUM>, and may initiate a tagging function when the UI element <NUM> is selected. A tagging operation may be performed based on the input voice after the tagging function is initiated. The UI element <NUM> may be selected via a variety of input methods. For example, if the electronic apparatus <NUM> is an apparatus including a touch screen, the user may initiate the tagging function by touching the UI element <NUM>.

<FIG> is a view illustrating various embodiments related to executing a tagging function based on voice recognition.

The electronic apparatus <NUM> may include an AI agent for providing a voice recognition server. The AI agent is a dedicated program for providing AI-based service (for example, voice recognition service, assistant service, translation service, search service, or the like), and may be executed by an existing general-purpose processor or a separate AI-dedicated processor.

For example, a tagging function may be initiated when a user utters a word (e.g., a "Bixby") to initiate a voice recognition function and utters a voice (e.g., "Execute a tagging function") requesting a tagging function, a tagging function may be initiated. In this example, a UI <NUM> indicating that the tagging function for the image <NUM> has been initiated may be displayed. For example, as shown in <FIG>, the UI <NUM> may include a guide phrase that guides a user to speak information to be tagged to an object in the image <NUM>. After the tagging function is initiated by the voice, a tagging operation may be performed based on the input voice.

As described above, according to an embodiment in which a tagging operation is performed based on the voice that is input after the tagging function is initiated, there is an effect that only the words desired by a user may be tagged to an image.

According to an embodiment, a tag may be deleted during tagging via voice. For example, a UI element to delete a keyword of the voice included in the tag information may be displayed. This will be described with reference to <FIG>.

Referring to <FIG>, the electronic apparatus <NUM> may provide a UI element <NUM> for deleting by tagged words. When the user selects the UI element <NUM>, the tag may be deleted. For example, when the UI element <NUM> displayed beside "son" is selected, the "son" may be deleted from the tag information.

According to another embodiment, it is possible to modify and edit tagged words. For example, if the electronic apparatus <NUM> has a touch screen, if the user touches, for example, "son" among the tag information displayed on the touch screen, "son" may be in a modifiable state. A cursor and a soft keyboard may be displayed in a modifiable state, and the user may modify the content using a soft keyboard. It is possible to modify, delete, and also enter a new tag content through the soft keyboard.

According to the embodiments described above, the content a user does not want to be tagged during speaking may be excluded, and there is an effect that even if voice is not correctly recognized, a tag may be modified.

When tagging is terminated, the user may terminate a tagging function. According to the disclosure, the tagging function may be terminated by various methods. <FIG> illustrates an embodiment of terminating the tagging function.

Referring to <FIG>, the electronic apparatus <NUM> may provide a UI <NUM> asking whether to terminate the tagging. For example, the electronic apparatus <NUM> may determine that the user is intended to finish the tagging and display a UI <NUM> if the user voice is not entered for a predetermined period of time during tagging. In response to the displayed UI <NUM>, if the user speaks a consented voice, such as, for example, "Yes," the electronic apparatus <NUM> may recognize this and terminate the tagging function. The electronic apparatus <NUM> may match the object <NUM> in the image <NUM> with the tag information <NUM> and store the same in the memory <NUM>.

The tagging function may be terminated in a variety of ways. According to another embodiment, when a user speaks a voice intended for terminating tagging, such as "describing a photo ended", " the electronic apparatus <NUM> may recognize it and terminate the tagging function. According to another embodiment, the electronic apparatus <NUM> may display an icon for terminating the tagging function along with the image <NUM> and may terminate the tagging function if the icon is selected.

According to an embodiment, if there are a plurality of objects with which the user wishes to tag information in an image, the user may distinguish the objects for tagging. For example, if a plurality of objects associated with a voice input from an image are identified, the electronic apparatus <NUM> may obtain tagging information for each of the plurality of objects based on the input voice. This embodiment will be described with reference to <FIG>.

<FIG> is a diagram illustrating an embodiment of tagging information to a plurality of objects according to an embodiment.

Referring to <FIG>, when tagging for Junseo is completed, the user may perform tagging for Yunseo in the image <NUM>.

The electronic apparatus <NUM> may identify an object corresponding to a trigger voice when the trigger voice for object conversion is input while tagging for the first object in the image <NUM> is performed, and perform new tagging on the identified object. The trigger voice may include, for example, at least one of a description of the appearance of the target object, a description of the gender of the target object, a description of the color of the target object, a description of the position of the target object, a description of the category of the target object, a description of the name of the target object, or the like. As illustrated in <FIG>, a description of the position of an object, such as "right", may be a trigger voice to switch the object to be tagged.

The electronic apparatus <NUM> may display a UI element indicating that the new object is to be tagged if a new object <NUM> is identified by inputting a trigger voice for object conversion. For example, as shown in <FIG>, a rectangular UI element surrounding the identified object <NUM> may be displayed. When the new object <NUM> is identified and user voice is input, tag information <NUM> for the new object <NUM> may be generated based on the input voice.

According to the embodiment, while the user speaks naturally, tag information for each of several objects in the image may be generated.

Identifying the target object for tagging based on the voice is described in the embodiments above, but according to still another embodiment, the target object may be identified by a user manipulation. <FIG> illustrates the embodiment.

Referring to <FIG>, when the electronic apparatus <NUM> has a touch screen, when a user touch with respect to the displayed image <NUM> is detected through the touch screen, the electronic apparatus <NUM> may identify an object located where the touch is detected as a target object for tagging. When the target object for tagging is identified and the user voice is input, tag information <NUM> for the target object <NUM> for tagging may be generated based on the input voice.

Identifying an object based on user manipulation, as described above, may be used as an auxiliary means when identification of an object based on voice is not successfully performed. For example, the electronic apparatus <NUM> may display a guide UI, such as "please touch an object to be tagged", if the electronic apparatus <NUM> fails to identify the object based on voice.

According to an embodiment, tag information may be obtained with reference to pre-generated tag information. For example, the electronic apparatus <NUM> may identify the first object associated with a voice in an image and obtain the tag information for the first object with reference to the pre-generated tag information for a second object included in the image. An embodiment of the present invention will be described with reference to <FIG>.

Referring to <FIG>, if pre-generated tag information <NUM> exists for the first object <NUM> among the objects in the image <NUM>, the pre-generated tag information <NUM> may be referred to, when the tag information for the second object <NUM> is generated based on the user voice. The tag information <NUM> may be stored in the memory <NUM>.

As shown in <FIG>, when a user voice of "older than Junseo by four years" is input, the electronic apparatus <NUM> may input the voice and tag information <NUM> to a trained AI model using an AI algorithm to obtain information from the pre-generated tag information <NUM> that Junseo is five years old and the target object <NUM> to be tagged is four years older than Junseo, and thus may tag "nine years" to a target object <NUM> for tagging based on the obtained information.

<FIG> is a diagram illustrating an embodiment according to the invention associated with generating new tag information with reference to pre-generated tag information.

Referring to <FIG>, when there is the pre-generated tag information <NUM> for the first object <NUM> among the objects inside the image <NUM>, the pre-generated tag information <NUM> may be referred to, when the tag information for a second object <NUM> is generated based on the user voice.

As illustrated in <FIG>, when the user voice, "that water gun Yunseo is playing with," the electronic apparatus <NUM> may extract tag information <NUM> including "Yunseo" among the pre-generated tag information and generate new tag information <NUM> including the extracted tag information <NUM>.

The generated tag information may be used in various ways. For example, when a user utters a voice including an intention to purchase a specific object, the electronic apparatus <NUM> may display a web page for purchasing the object by referring to tag information corresponding to the object. For example, in the example of <FIG>, when the user utters a voice saying, "Junseo is dying to have the water gun Yunseo is playing with. I need to order one for Junseo too," the electronic apparatus <NUM> may refer to a named entity from the tag information <NUM> generated for the water gun and display a website that sells a pink water gun by referring to the named entity.

According to an embodiment, the electronic apparatus <NUM> inputs an image to a first AI model to obtain information on at least one object in the image, and identify an object related to the voice among the obtained at least one object based on information on the at least one object and a word included in the voice. The voice is input to a second AI model to obtain tag information including a keyword of the voice.

The tag information may include tag information obtained by analyzing an image as well as tag information obtained based on the voice. That is, for example, the tag information may include information about the identified object among information about at least one object obtained by inputting the image to the first AI model as well as the keyword of the obtained voice by inputting the voice to the second artificial intelligence model. The first AI model is a model for visual understanding, and the second AI model is a model for linguistic understanding.

<FIG> is a diagram illustrating tag information according to an embodiment.

The tag information for the object in the image may include information about the object obtained by analyzing the image. Referring to <FIG>, a "named entity" item is information obtained through image analysis. Various conventional image analysis techniques may be used to obtain information about an object. For example, an AI model may be used to analyze an image to recognize a category of an object (human, animal, thing, landscape, etc.), and it is possible to recognize an appearance, color, gender, age, face, or the like, of the object. Through such image analysis, information on the object may be obtained and included in the tag information.

The tag information may include information on an object obtained by inputting the voice input from the user to the AI model. Referring to <FIG>, "attribute" item refers to information obtained through the voice.

The information obtained through the voice is reflected with a user's unique thinking, feeling, or the like, and has a property as more personalized information as compared to the information obtained through image analysis. According to an embodiment, tag information may be generated by including information obtained through image analysis and information obtained through voice, so that more various kinds of tag information may be utilized.

The tag information includes an image in which the target object for tagging is identified. Referring to <FIG>, "image" item is an image in which the object to be tagged is identified.

The tag information <NUM>, <NUM>, and <NUM> generated as shown above may be stored in the memory <NUM>. The generated tag information <NUM> and <NUM> may be stored in an external server and shared with other users.

<FIG> is a diagram illustrating tag information generating according to an embodiment.

Referring to <FIG>, a user voice is input to a voice engine <NUM>, and an image including at least one object is input to an image engine <NUM>. The voice engine <NUM> may perform processing on the input voice using a model for linguistic understanding trained using an AI algorithm. The image engine <NUM> may perform processing on the image using a model for visual understanding trained by using an AI algorithm.

A voice input to an automatic voice recognition module (ASR) <NUM> is converted to a plain text. A natural linguistic understanding (NLU) module <NUM> receiving the converted text may perform several analyses for making the system understand the converted text.

Based on the converted text using the ASR module <NUM>, the NLU module <NUM> may perform a natural linguistic understanding process to grasp the intention of the user utterance. In the natural linguistic understanding process, morpheme analysis, syntactic analysis, dialogue act, main action, keyword analysis, or the like, may be performed. Specifically, the NLU module <NUM> may perform a morpheme analysis to divide a text into a unit of a morpheme that is the smallest semantic unit to analyze which word class each morpheme has. By the morpheme analysis, information of a word class such as noun, verb, adjective, postposition, or the like, may be identified. The NLU module <NUM> may perform syntactic analysis processing. The syntactic analysis is performed by dividing a user's utterance with a specific reference, such as a noun phrase, a verb phrase, an adjective phrase, etc., and analyzing whether there is any relationship between each divided chunk. Through this syntactic analysis, subject, object, qualifiers of the user utterance may be grasped.

Keywords of the voice obtained as a result of analyzing the voice from the NLU module <NUM> may be included as "attribute" item from tag information.

The image engine <NUM> includes an object recognition engine <NUM>, and the object recognition engine <NUM> may identify which things appear in the image. For example, the image engine <NUM> may use the feature vector technique to identify what the object is. The feature vector technique is a technique in which various features of an object, such as color, shape, movement, etc., are combined to recognize what is the object. For example, if an image including a male child is input, the object recognition engine may define a male child through recognition of physique, ratio, facial feature, or the like, of the object.

The object recognition engine <NUM> may provide information (for example, "male child") on the object as a result of processing an image and this may be included in the tag information as the "named entity" item.

An agent <NUM> may be an AI agent. The AI agent is a dedicated program for providing an AI-based service (e.g., a voice recognition service, an assistant service, a translation service, a search service, etc.), and may be executed by an existing general purpose processor (e.g., a central processing unit (CPU)) or a separate AI-only processor (e.g., graphics processing unit (GPU), etc.). The agent <NUM> may be mounted on the electronic apparatus <NUM> or mounted on the external server of the electronic apparatus <NUM>.

For an entity (for example, what is "that", call "that person") among the user utterance which is not understandable, the agent <NUM> may distinguish by utilizing a dialogue task ontology <NUM>.

A tag generator <NUM> may generate tag information based on information obtained through the voice engine <NUM> and the image engine <NUM> and store the information in database <NUM>.

In the above-described embodiment, at least one of the voice engine <NUM>, the object recognition engine <NUM>, the agent <NUM>, the dialogue task ontology <NUM>, and the tag generator <NUM> may be implemented as a software module or in a form of at least one hardware chip and mounted to the electronic apparatus <NUM>, or may be alternatively mounted on an external server of the electronic apparatus <NUM>. When the voice engine <NUM>, the object recognition engine <NUM>, and the agent <NUM> are implemented in a software module (or a program module including an instruction), the software module may be stored in a computer readable non-transitory computer readable media. In this example, the software module may be provided by an operating system (OS), or provided by a predetermined application. Alternatively, some of the software modules may be provided by the OS, and some of which may be provided by predetermined applications.

<FIG> is a diagram illustrating generating tag information according to an embodiment.

When a user (A) says "the left is my son Junseo" while looking at an image displayed on the electronic apparatus <NUM>, the object recognition engine <NUM> may identify objects in the photo, obtain feature information (e.g., gender, age, etc.) of the object through the image analysis, and obtain information on the object position in the image, the relationship of the object, the gender, the name, or the like. The tag information <NUM> may be generated based on the recognition information for the image and the recognition information for the voice. The tag information <NUM> may include various information as an attribute, such as, for example, relation, gender, age, color, or the like, depending on the recognition information.

Referring to <FIG>, information may be added to the tag information <NUM> based on an additionally input voice. In this example, abstract information (age: child) grasped by image recognition may be supplemented as detailed in formation (age: <NUM>) through recognition information for the voice.

Database including tag information may be generated for each user, and a group including various users may share tag information of each other. For example, referring to <FIG>, a first user's database <NUM> and a second user's database <NUM> may be generated. The first user's database <NUM> and the second user's database <NUM> may be stored in a cloud server. Databases for tag information may be shared between designated user groups. The agent <NUM> may provide various services based on the database for shared tag information. For example, users belonging to a group may have a dialogue using tag information when having a dialogue. An embodiment will be described with reference to <FIG>.

<FIG> illustrates a chatting screen displayed on the electronic apparatus <NUM> according to an embodiment. A UI <NUM> including information on the daughter of Jaein may be displayed to a user of the electronic apparatus <NUM> who is not aware of the daughter of a dialogue partner Jaein.

The agent <NUM> may include the NLU module <NUM>, a tagging action planner <NUM>, and a natural language generation (NLG) module <NUM>. The agent <NUM> may be an AI agent. The AI agent is a dedicated program for providing an AI-based service (e.g., a voice recognition service, an assistant service, a translation service, a search service, or the like), and may be executed by an existing general purpose processor (e.g. CPU) or a separate AI-only processor (e.g., GPU). The agent <NUM> may be mounted on the electronic apparatus <NUM> or mounted on the external server of the electronic apparatus <NUM>.

The NLU module <NUM> performs a natural linguistic understanding process on the text input to the chat screen, and based on the result, the tagging action planner <NUM> may obtain tagging information <NUM> included in the data base <NUM> of Jaein. The NLG module <NUM> is a module for making the result data in a natural language form, and may generate information in a natural language form based on the tagging information <NUM>, and the UI <NUM> including the generated information may be displayed in the electronic apparatus <NUM>.

<FIG> is a diagram illustrating tag information usage according to another embodiment.

Referring to <FIG>, when a user is interested in a specific portion of an object having tag information, a portion of the object may be enlarged and displayed, and the portion may be separately managed as an image of interest of the user. For example, as shown in <FIG>, if the user voicees "Look at the facial expression of Junseo. He is so cute," natural language analysis is performed through the NLU module <NUM>, and the tag information <NUM> corresponding to "Junseo" is searched (①). The object <NUM> in the image is identified from the tag information <NUM> (②), and the face is detected as the relevant area in the image from the word "facial expression" obtained in the NLU module <NUM> (③④), and the object recognition engine <NUM> detects a face area 11a in the object <NUM> (⑥). The detected area may be enlarged and displayed in the electronic apparatus <NUM> and added to the tag information <NUM> (⑦⑧). The electronic apparatus <NUM> may separately store the detected area in the memory <NUM> as an image of interest.

<FIG> is a view illustrating an embodiment of generating tag information for a food in a refrigerator.

The electronic apparatus <NUM> may be implemented as a refrigerator including a display. Images <NUM> and <NUM> shown in <FIG> is an image of the inside of the electronic apparatus <NUM> embodied as a refrigerator and may be displayed through a display provided on a front surface of the refrigerator. Referring to <FIG>, the NLU module <NUM> may identify the meaning of the "what is input now" during the utterance of the user (Jaein), and the object recognition engine <NUM> may compare the current image <NUM> with the previous image <NUM> before the event that the refrigerator door is opened and closed to identify newly added objects <NUM> and <NUM>. Based on the information obtained by analyzing the user's utterance by the NLU module <NUM>, the object recognition engine <NUM> may recognize the position of the objects <NUM> and <NUM> in the current image <NUM>. Tag information 1721a and 1722b for each of the objects <NUM>, <NUM> may be generated based on information about what the objects <NUM>, <NUM> are (e.g., food type), when the objects <NUM>, <NUM> are created (e.g., create time).

As described above, tag information for foods included in an image of the electronic apparatus <NUM> embodied in a refrigerator may be generated, and the generated tag information may be stored in a database of a user (Jaein). The tag information may then be provided to Sera who is allowed to share the tag information. The embodiment will be described with reference to <FIG>.

Referring to <FIG>, looking at the image <NUM> capturing the inside of a refrigerator, if another user (Sera) utters a voice, "which young radish kimchi is made by Jaein?", the NLU module <NUM> may analyze the meaning, and the tagging action planner <NUM> may obtain information on a position of the object based on the voice recognition result and the tag information 1722b of the database of Jaein. Based on the obtained information, the agent <NUM> may output a voice guide, such as "let me show you a position of storage in a refrigerator" along with an image <NUM> including a graphical object (a rectangle surrounding the object) for identifying the position of the object. The agent may be mounted on the electronic apparatus <NUM> embodied in a refrigerator, mounted on a mobile terminal device (e.g., a smartphone), or another separate agent.

The embodiments described above may be implemented by using artificial intelligence (AI) systems. The AI systems are computer systems that implement human-level intelligence in which the machine learns, determines and becomes smart, and the more the AI systems are used, the more the recognition rate is improved.

AI technology may be composed of a technology that learns a model using an AI algorithm that classifies / learns the characteristics of input data by itself and a technology that simulates functions such as recognition and determination of human brain using trained model.

Examples of the trained model include a model for linguistic understanding for recognizing human language / characters, a model for visual understanding for recognizing objects as human vision, a model for inference / prediction for judging and logically inferring and predicting information, and a model for knowledge representation for processing human experience information into knowledge data, or the like.

According to an embodiment, human language and characters may be recognized, applied, and processed using the linguistic understanding model. Relevant functions include natural language processing, machine translation, dialogue system, question / answer, voice recognition / synthesis, or the like.

According to an embodiment, using visual understanding model, recognition of an object in an image, object tracking, image search, human recognition, scene understanding, spatial understanding, or the like, may be performed.

According to an embodiment, using inference / prediction model, knowledge / probability based inference, optimization prediction, preference-based planning, and recommendation, or the like, may be performed.

According to an embodiment, using knowledge representation model, knowledge building (data generation / classification) and knowledge management (data utilization), or the like, may be performed.

According to an embodiment, a new model may be generated and used based on two or more of the various models above.

The learned (or trained) model is a learned determination model based on an artificial intelligence (AI) algorithm, for example, it may be a model based on a neural network. The learned AI model may include a plurality of weighted network nodes that may be designed to simulate the human brain structure on a computer and simulate a neuron of a human neural network. The plurality of network nodes may each establish a connection relationship so that the neurons simulate the synaptic activity of the neurons sending and receiving signals through the synapse. Also, the learned AI model may include, for example, a neural network model or a deep learning model developed from a neural network model. In the deep learning model, a plurality of network nodes are located at different depths (or layers), and may transmit and receive data according to a convolution connection relationship. Examples of learned AI models include, but are not limited to, Deep Neural Network (DNN), Recurrent Neural Network (RNN), and Bidirectional Recurrent Deep Neural Network (BRDNN).

Various embodiments may use a personal assistant program that is an AI-dedicated program (or AI agent). The personal assistant program is a dedicated program to provide AI-based service and may be executed by an existing general-purpose processor (e.g., CPU) or separate AI-exclusive processor (e.g., GPU, or the like).

When a predetermined user input (e.g., an icon touch corresponding to a personal assistant chatbot, a user voice including a preset word, or the like) is input or a button (e.g., a button for executing an AI agent) provided in the electronic apparatus <NUM> is pressed, an AI agent mounted on the electronic apparatus <NUM> may be operate (or executed). The AI agent may transmit information on the voice of a user and information on an image including the target object to be tagged to an external server, and may provide tagging information based on the data received from the external server.

The AI agent may be previously executed before a predetermined user input is detected or a button provided in the electronic apparatus <NUM> is selected. In this example, when the tagging function is executed, the AI agent of the electronic apparatus <NUM> may obtain tagging information based on the image and the user voice. The AI agent may be in a standby state before a predetermined user input is detected or a button provided in the electronic apparatus <NUM> is selected. Here, the standby state is a state in which a predefined user input is detected to control the start of operation of the AI agent. When the tagging function is executed while the AI agent is in a standby state, the electronic apparatus <NUM> may operate the AI agent and obtain tagging information based on the image and the user voice.

The above-described embodiments may be implemented using an AI model in a server external to the electronic apparatus <NUM>, or the electronic apparatus <NUM> may directly use the AI model. In this example, the AI agent mounted on the electronic apparatus <NUM> may control the AI model to perform the operations of the above-described embodiments.

<FIG> is a block diagram illustrating a processor for learning and using a recognition model according to an embodiment.

Referring to <FIG>, a processor <NUM> may include at least one of a learning unit <NUM> and an analysis unit <NUM>.

The learning unit <NUM> may generate or train an artificial intelligence model using learning data.

For example, the learning unit <NUM> may generate, train, or update a model for linguistic understanding by using human language as learning data.

As a still another example, the learning unit <NUM> may generate, train, or update a visual understanding model using an image including at least one object as learning data.

As another example, the learning unit <NUM> may generate, train, or update a model for object recognition that recognizes (or identifies) an object referred to by a human language refers an image using an image including a human language and at least one object as learning data. The model for object recognition may be generated, trained or updated separately, or may be generated, trained or updated based on a model for linguistic understanding and a model for visual understanding.

The analysis unit <NUM> may perform linguistic understanding, visual understanding, object recognition, or the like, by using the predetermined data as input data to a trained model.

For example, the analysis unit <NUM> may identify an intention of a user appearing in the voice by using the voice as input data to the linguistic understanding model.

When functionally divided, the linguistic understanding model may include the ASR module, NLU module, and NLG module. The ASR module, NLU module, and NLG module have been described above and thus will not be further described.

As another example, the analysis unit <NUM> may obtain (or estimate, infer) information about the at least one object using an image including at least one object as input data in a model for visual understanding. For example, if the image <NUM> shown in <FIG> is used as input data in a model for visual understanding, two of the objects that are classified into humans are detected, and that one is a boy and one is a girl may be obtained.

As another example, the analysis unit <NUM> may recognize an object which the user voice in the image refers to with an image including user voice and at least one object as input data for a model for object recognition. Alternatively, the analysis unit <NUM> may recognize an object which the user voice in the image refers to using information obtained through the model for linguistic understanding and the model for visual understanding as input data.

For example, at least a portion of the learning unit <NUM> and at least a portion of the analysis unit <NUM> may be implemented as software modules or at least one hardware chip form and mounted in the electronic apparatus <NUM>. For example, at least one of the learning unit <NUM> and the analysis unit <NUM> may be manufactured in the form of an exclusive-use hardware chip for artificial intelligence (AI), or a conventional general purpose processor (e.g., a CPU or an application processor) or a graphics-only processor (e.g., a GPU) and may be mounted on various electronic apparatuses as described above. Herein, the exclusive-use hardware chip for artificial intelligence is a dedicated processor for probability calculation, and it has higher parallel processing performance than existing general purpose processor, so it can quickly process computation tasks in artificial intelligence such as machine learning. When the learning unit <NUM> and the analysis unit <NUM> are implemented as a software module (or a program module including an instruction), the software module may be stored in a computer-readable non-transitory computer readable media. In this case, the software module may be provided by an operating system (OS) or by a predetermined application. Alternatively, some of the software modules may be provided by an O/S, and some of the software modules may be provided by a predetermined application.

The learning unit <NUM> and the analysis unit <NUM> may be mounted on one electronic apparatus, or may be mounted on separate electronic apparatuses, respectively. For example, the processor <NUM> of <FIG> may be a processor <NUM> of <FIG>. For example, one of the learning unit <NUM> and the analysis unit <NUM> may be implemented in the electronic apparatus <NUM>, and the other one may be implemented in an external server. In addition, the learning unit <NUM> and the analysis unit <NUM> may provide the model information constructed by the learning unit <NUM> to the analysis unit <NUM> via wired or wireless communication, and provide data which is input to the analysis unit <NUM> to the learning unit <NUM> as additional data.

<FIG> is a block diagram illustrating a learning unit <NUM> according to an embodiment.

Referring to <FIG>, the learning unit <NUM> according to some embodiments may implement a learning data acquisition unit <NUM>-<NUM> and a model learning unit <NUM>-<NUM>. The learning unit <NUM> may further selectively implement at least one of a learning data preprocessor <NUM>-<NUM>, a learning data selection unit <NUM>-<NUM>, and a model evaluation unit <NUM>-<NUM>.

The learning data acquisition unit <NUM>-<NUM> may obtain learning data for training models for linguistic understanding, visual understanding, object recognition, or the like. The learning data may be data collected or tested by the learning unit <NUM> or a manufacturer of the learning unit <NUM>. For example, the learning data may include an image that includes voice, at least one object.

The model learning unit <NUM>-<NUM> may use the learning data to train a model to have a reference about how to understand, recognize, determine, infer, or the like, the input data. For example, the model learning unit <NUM>-<NUM> may extract a feature from the input image, project the feature into a vector space, and index the information of the object in each vector to construct a model for visual understanding.

The model learning unit <NUM>-<NUM> may train a model through supervised learning of at least a part of the learning data. Alternatively, the model learning unit <NUM>-<NUM> may learn, for example, by itself using learning data without specific guidance to make the artificial intelligence model learn through unsupervised learning which detects a criterion for determination of a situation. Also, the model learning unit <NUM>-<NUM> may train the model through reinforcement learning using, for example, feedback on whether the result of providing the response according to learning is correct. The model learning unit <NUM>-<NUM> may also make the model learn using, for example, a learning algorithm including an error back-propagation method or a gradient descent.

In addition, the model learning unit <NUM>-<NUM> may learn a selection criterion about which learning data should be used for estimating a target of recognition using input data.

The model learning unit <NUM>-<NUM> may determine a model having a great relevance between the input learning data and the basic learning data as a model to be learned when there are a plurality of models previously constructed. In this case, the basic learning data may be pre-classified according to the type of data, and the model may be pre-constructed for each type of data. For example, the basic learning data may be pre-classified into various criteria such as a region in which the learning data is generated, the time at which the learning data is generated, the size of the learning data, the genre of the learning data, the creator of the learning data, the type of the object in the learning data, or the like.

When the model is trained, the model learning unit <NUM>-<NUM> may store the trained model. For example, the model learning unit <NUM>-<NUM> may store the trained model in the memory <NUM> of the electronic apparatus <NUM>. Alternatively, the model learning unit <NUM>-<NUM> may store the trained model in a memory of a server connected to the electronic apparatus <NUM> via a wired or wireless network.

The learning unit <NUM> may further implement a learning data preprocessor <NUM>-<NUM> and a learning data selection unit <NUM>-<NUM> to improve the processing capability of the model or to save resources or time required for generation of the model.

The learning data preprocessor <NUM>-<NUM> may preprocess obtained data so that the data obtained in the learning for determining the situation may be used. The learning data preprocessor <NUM>-<NUM> may process the obtained data into a predetermined format so that the model learning unit <NUM>-<NUM> may use the obtained data for learning for the determination of a situation.

The learning data selection unit <NUM>-<NUM> may select data required for learning from the data obtained by the learning data acquisition unit <NUM>-<NUM> or the data preprocessed by the learning data preprocessor <NUM>-<NUM>. The selected learning data may be provided to the model learning unit <NUM>-<NUM>. The learning data selection unit <NUM>-<NUM> may select learning data necessary for learning from the obtained or preprocessed data in accordance with a predetermined selection criterion. The learning data selection unit <NUM>-<NUM> may also select learning data according to a predetermined selection criterion by learning by the model learning unit <NUM>-<NUM>.

The learning unit <NUM> may further include the model evaluation unit <NUM>-<NUM> to improve a processing ability of the model.

The model evaluation unit <NUM>-<NUM> may input evaluation data to the model, and if the result which is output from the evaluation data does not satisfy a predetermined criterion, the model evaluation unit <NUM>-<NUM> may make the model learning unit <NUM>-<NUM> learn again. In this example, the evaluation data may be predefined data to evaluate the model.

For example, the model evaluation unit <NUM>-<NUM> may evaluate, among the analysis results of the trained model for the evaluation data, that the analysis result does not satisfy a predetermined criterion when the number or ratio of the evaluation data of which analysis result is not correct exceeds a preset threshold.

When there are a plurality of trained models, the model evaluation unit <NUM>-<NUM> may evaluate whether each trained model satisfies a predetermined criterion, and determine the model which satisfies a predetermined criterion as a final model. Here, when there are a plurality of models that satisfy a predetermined criterion, the model evaluation unit <NUM>-<NUM> may determine one or a predetermined number of models which are set in an order of higher evaluation score as a final model.

Referring to <FIG>, the analysis unit <NUM> according to some embodiments may include the input data acquisition unit <NUM>-<NUM> and the analysis result provision unit <NUM>-<NUM>. The analysis unit <NUM> may further include at least one of a data preprocessor <NUM>-<NUM>, a data selection unit <NUM>-<NUM>, and a model update unit <NUM>-<NUM> in a selective manner.

The data acquisition unit <NUM>-<NUM> may obtain data necessary for analysis. The analysis result provision unit <NUM>-<NUM> may provide a result which is obtained by inputting the data obtained from the data acquisition unit <NUM>-<NUM> to the trained model. The analysis result provision unit <NUM>-<NUM> may provide an analysis result in accordance with a purpose of data analysis. The analysis result provision unit <NUM>-<NUM> applies the data selected by the data preprocessor <NUM>-<NUM> or the data selection unit <NUM>-<NUM> to be described later to the model as an input value to obtain the response result. The response result may be determined by the model.

In an embodiment, the analysis result provision unit <NUM>-<NUM> may obtain (or estimate) information for each object by applying an image including at least one object obtained from the data acquisition unit <NUM>-<NUM> to a trained model for visual understanding.

In another embodiment, the analysis result provision unit <NUM>-<NUM> may apply an image including at least one object and the voice obtained from the data acquisition unit <NUM>-<NUM> to a trained model for object recognition to recognize (or identify) an object indicated by the voice.

As another embodiment, the analysis result provision unit <NUM>-<NUM> may determine a user's intention corresponding to the voice by applying voice (voice data) obtained from the data acquisition unit <NUM>-<NUM> to a trained model for linguistic understanding.

The analysis unit <NUM> may further include the data preprocessor <NUM>-<NUM> and the data selection unit <NUM>-<NUM> in order to improve an analysis result of the model or save resources or time to provide the analysis result.

The input data preprocessor <NUM>-<NUM> may preprocess the obtained data so that the obtained data may be used to determine a situation. The data preprocessor <NUM>-<NUM> may process the obtained data into the pre-defined format by the analysis result provision unit <NUM>-<NUM>.

The data selection unit <NUM>-<NUM> may select data required for determining a situation from the data acquired by the data acquisition unit <NUM>-<NUM> or the data preprocessed by the input data preprocessor <NUM>-<NUM>. The selected data may be provided to the analysis result provision unit <NUM>-<NUM>. The data selection unit <NUM>-<NUM> can select some or all of the obtained or preprocessed data according to a predetermined selection criterion for determining a situation. The data selection unit <NUM>-<NUM> can also select data according to a predetermined selection criterion by learning by the model learning unit <NUM>-<NUM>.

The model update unit <NUM>-<NUM> may control the updating of the model based on the evaluation of the analysis result provided by the analysis result provision unit <NUM>-<NUM>. For example, the model update unit <NUM>-<NUM> may provide the analysis result provided by the analysis result provision unit <NUM>-<NUM> to the model learning unit <NUM>-<NUM> so that the model learning unit <NUM>-<NUM> may ask for further learning or updating the model.

<FIG> is a diagram illustrating an embodiment in which the learning unit <NUM> and the analysis unit <NUM> are implemented in different devices.

Referring to <FIG>, an external server <NUM> may include the learning unit <NUM>, and the electronic apparatus <NUM> may include the analysis unit <NUM>. The electronic apparatus <NUM> and the server <NUM> may communicate with each other on the network.

The analysis result provision unit <NUM>-<NUM> of the electronic apparatus <NUM> applies the data selected by the data selection unit <NUM>-<NUM> to the model generated by the server <NUM> to obtain the analysis result. The analysis result provision unit <NUM>-<NUM> of the electronic apparatus <NUM> may receive the model generated by the server <NUM> from the server <NUM>, determine a user's intention in the user voice input through the microphone <NUM> using the received model, obtain object information from an image displayed on the display <NUM>, or identify an object related to user voice in the image.

<FIG> is a block diagram illustrating a detailed configuration of the electronic apparatus <NUM> according to an embodiment.

Referring to <FIG>, the electronic apparatus <NUM> may include the display <NUM>, the memory <NUM>, the microphone <NUM>, the processor <NUM>, a touch sensor <NUM>, a communicator <NUM>, a global positioning system (GPS) receiver <NUM>, a motion sensor <NUM>, a button <NUM>, a camera <NUM>, and an audio outputter <NUM>. Some of the configurations may be omitted, depending on the embodiment, and appropriate hardware/software configurations as would be apparent to those skilled in the art, although not shown, may be further included in the electronic apparatus <NUM>.

The display <NUM> is configured to display an image. The display <NUM> may be the same as the display <NUM> of <FIG>. The display <NUM> may be implemented as a touch screen along with the touch sensor <NUM> capable of sensing a touch operation of a user.

The touch sensor <NUM> may sense a touch operation of a user or a pen. The touch sensor <NUM> includes a touch sensor, and the touch sensor may be implemented by electrostatic type or resistive type. The electrostatic type uses a dielectric coated on the surface of the display layer to sense micro electricity which is excited to the human body when a part of the body of the user is touched on the surface of the display layer, thereby calculating touch coordinates. The resistive type includes two electrode plates so that when a user touches a screen, upper and lower plates of the touched point are contacted to sense flow of current, thereby calculating touch coordinates. As described above, the touch sensor may be implemented in various types.

When the pen is used as an input means, if the pen is a pen including a coil therein, the touch sensor <NUM> may include a magnetic field detecting sensor capable of sensing a magnetic field changed by a coil inside the pen. Accordingly, the proximity operation, that is, hovering, may be detected as well as the touch operation.

The touch sensor <NUM> plays a role as an inputter, may receive a selection of an object from a user, and may receive writing of a user or a pen.

The processor <NUM> may determine the type of touch operation based on a sensing signal sensed by the touch sensor of the touch sensor <NUM>. The touch operation may be of a variety of operations, such as simple touch, tap, touch and hold, move, flick, drag and drop, pinch-in, pinch-out, and the like. The processor <NUM> may control various configurations of the electronic apparatus <NUM> according to a user touch sensed by the touch sensor <NUM>.

When the display <NUM> and the touch sensor <NUM> are implemented as a touch screen, for example, as illustrated in <FIG>, touch for an icon <NUM> for executing a tagging function may be sensed, and as another example, as illustrated in <FIG>, a user touch for selecting a target object to be tagged may be sensed.

The memory <NUM> may be the same as the memory <NUM> of <FIG>. The memory <NUM> may store computer executable instructions and the processor <NUM> may perform various functions of the electronic apparatus <NUM> by executing computer executable instructions.

According to an embodiment, the memory <NUM> may store an AI model trained by an AI algorithm. The AI model may include a model for linguistic understanding, a model for visual understanding, and a model for object recognition. The AI model has been described above and will not be further described.

The microphone <NUM> is configured to receive user voice or other sound to convert the sound to a digital signal. The processor <NUM> may use user voice input through the microphone <NUM> to a tagging function.

The communicator <NUM> is configured to communicate with various types of external devices in accordance with various types communication methods. The communicator <NUM> may include various communication chips such as a Wi-Fi chip <NUM>, a Bluetooth chip <NUM>, a near field communication (NFC) chip <NUM>, a wireless communication chip <NUM>.

The Wi-Fi chip <NUM>, the Bluetooth chip <NUM>, and the NFC chip <NUM> perform communication by Wi-Fi method, Bluetooth method, and NFC method, respectively. The NFC chip <NUM> may communicate by the NFC method using a <NUM> band among various RF-ID frequency bands such as <NUM>, <NUM>, <NUM>, <NUM>~<NUM>, <NUM>, or the like. When using the Wi-Fi chip <NUM> or the Bluetooth chip <NUM>, various connection information such as service set identifier (SSID) may be transmitted and received for communication connection and then various information may be transmitted and received. The wireless communication chip <NUM> may communicate according to various communication specifications such as IEEE, Zigbee, <NUM>rd generation (<NUM>), <NUM>rd generation partnership project (3GPP), long term evolution (LTE), or the like.

In addition to the communication methods as described above, the communicator <NUM> may perform communication with various external devices through at least one communication method among infrared data association (IrDA), radio frequency identification (RFID), ultra-wideband (UWB), WirelessDisplay (WiDi), wirelessHD (WiHD), wireless home digital interface (WHDI), Miracast, wireless gigabit alliance (Wigig), Wi-Fi direct, AirPlay, Z-wave, internet protocol version <NUM>(IPv4) over low-power wireless personal area networks (4LoWPAN), general packet radio service (GPRS), weightless, digital living network alliance (DLNA), ANT +, digital enhanced cordless telecommunications (DECT), wireless local area network (WLAN), global system for mobile communications (GSM), universal mobile telecommunications system (UMTS), wireless broadband (WiBRO), or the like.

According to an embodiment, the communicator <NUM> may communicate with an external server. The external server may store a model for visual understanding, a model for linguistic understanding, a model for object recognition, or the like, and the processor <NUM> may control the communicator <NUM> to transmit the image and the voice to the external server when the user voice is input through the microphone <NUM> while the image including at least one object is displayed on the display <NUM>.

According to an embodiment, when the tag information for the object included in an image is generated in an external server, the external server may transmit tag information to the electronic apparatus <NUM> and receive tag information from the external server through the communicator <NUM>.

According to another embodiment, the electronic apparatus <NUM> itself may generate tag information. In this example, the processor <NUM> may control the communicator <NUM> to transmit the image and voice to the external server where the AI model for image analysis and voice recognition is stored, and if the information on the object in the image from the external server and the voice recognition result are received through the communicator <NUM>, the processor <NUM> may identify the object related to the voice in the image based on the received information, and generate tag information for the identified object.

The motion sensor <NUM> may include a plurality of motion detecting sensors <NUM>-<NUM> to <NUM>-m.

The plurality of motion detecting sensors <NUM>-<NUM> to <NUM>-m are sensors for sensing a rotation state of the electronic apparatus <NUM>, the position of the user, and the like. The sensor for sensing the rotation state may be a geomagnetic sensor, an acceleration sensor, a gyro sensor, or the like. The acceleration sensor outputs a sensing value corresponding to the gravitational acceleration which changes according to the slope of a device to which the sensor is attached. The gyro sensor is a sensor that detects the angular velocity by measuring the force of the Coriolis force acting in that velocity direction when rotational movement occurs. The geomagnetic sensor is a sensor for sensing an azimuth angle. Sensors for sensing the location of a user include an image sensor, an infrared sensor, an ultrasonic sensor, and a proximity sensor.

The camera <NUM> is configured to capture a subject for photography to generate a photographed image. According to an embodiment, tag information for an object included in the image photographed and generated using the camera <NUM> may be generated.

The GPS receiver <NUM> is configured to receive a GPS signal from a global positioning system (GPS) satellite and calculate a current position of the electronic apparatus <NUM>. When the tagging function is executed, the processor <NUM> may calculate the current position using the GPS signal received by the GPS signal receiver <NUM>, and generate information about the current position as tag information for the image.

The button <NUM> may be various types of buttons such as a mechanical button, a touch pad, a wheel, and the like formed in an arbitrary area such as a front surface portion, a side surface portion, and a back surface portion of the main body of the electronic apparatus <NUM>. The button <NUM> may be a power button to turn on or turn off power.

The processor <NUM> may perform a control operation according to the user voice input through the microphone <NUM>, the user motion detected by the motion sensor <NUM>, the user motion captured through the camera <NUM>, or the like. That is, the electronic apparatus <NUM> may operate in a motion control mode or a voice control mode, in addition to a general mode controlled by a user's touch or a button operation. When operating in the motion control mode, the processor <NUM> may track the motion change of the user and perform a corresponding control operation. When operating in the voice control mode, the processor <NUM> may operate in a voice recognition mode that performs a control operation according to user voice input through the microphone <NUM>.

The audio outputter <NUM> is configured to output audio and may include a speaker and / or an earphone terminal. When a microphone is provided in an earphone, a user voice may be received through the earphone terminal.

The processor <NUM> may control overall operations of the electronic apparatus <NUM>. For example, the processor <NUM> may control overall operations of the electronic apparatus <NUM> using various modules stored in the memory <NUM>. The processor <NUM> may be the same as the processor <NUM> of <FIG>.

The processor <NUM> includes at least one of a random access memory (RAM) <NUM>, a read-only memory (ROM) <NUM>, a graphics processing unit (GPU) <NUM>, a central processing unit (CPU) <NUM>, a first to nth interfaces <NUM>-<NUM>~<NUM>-n, and a bus <NUM>. The RAM <NUM>, the ROM <NUM>, the GPU <NUM>, the CPU <NUM>, the first to nth interfaces <NUM>-<NUM> to <NUM>-n, or the like, may be interconnected through the bus <NUM>.

The ROM <NUM> stores one or more instructions for booting the system and the like. When the turn-on instruction for the electronic apparatus <NUM> is input and power is supplied, the CPU <NUM> copies the OS stored in the memory <NUM> to the RAM <NUM> according to the stored one or more instructions in the ROM <NUM>, and executes the OS to boot the system. When the booting is completed, the CPU <NUM> copies various application programs stored in the memory <NUM> to the RAM <NUM>, executes the application program copied to the RAM <NUM>, and performs various operations. The CPU <NUM> may perform various operations using various program modules, data, or the like, stored in the memory <NUM>.

When booting of the electronic apparatus <NUM> is completed, the GPU <NUM> may display an image. The GPU <NUM> may generate a screen including various objects such as icons, images, text, and the like, using a calculator and a renderer. Here, a calculator may calculate an attribute value such as a coordinate value, a shape, a size, and a color to be displayed by each object according to the layout of the screen. A renderer may generate display screens of various layouts including objects based on the attribute value calculated by the calculator. A screen generated by the renderer is provided to the display <NUM> and is displayed in a display region.

The first to nth interface <NUM>-<NUM> to <NUM>-n are connected to the various elements <NUM>-<NUM>, <NUM>-<NUM> described above. One of the interfaces may be a network interface connected to an external device through the network.

The processor <NUM> controls the display <NUM> to display an image including at least one object, input the voice received through the microphone <NUM> to the AI model trained by the AI algorithm, identify an object related to the voice of the at least one object included in the image, obtain tag information for the identified object, and provide the obtained tag information. For example, as shown in <FIG>, processor <NUM> may provide tag information <NUM> via the display <NUM>.

The processor <NUM> inputs the displayed image to a first AI model to obtain information on at least one object included in the image, and identify an object related to the voice of the at least one object based on information on the at least one object and a word included in the voice received through the microphone <NUM>. The processor <NUM> inputs voice received via the microphone <NUM> to a second AI model to obtain tag information including the keyword of the voice. The first AI model and the second AI model may be stored in the memory <NUM>.

The processor <NUM> may control the display <NUM> to display a keyword of the voice along with an image. For example, as illustrated in <FIG>, the tag information <NUM> including a keyword of the voice may be displayed on the image <NUM>.

The processor <NUM> may control the display <NUM> to display the keyword of the voice subsequently input along with the keyword which is displayed previously. For example, referring to <FIG>, the keyword "<NUM> years" of "he is only <NUM> years old" that is subsequently input after "the left is my son Junseo" may be displayed along with the keywords "son" and "Junseo" which are previously displayed.

The processor <NUM> may control the display <NUM> to display a UI element to display a keyword of the voice included as the tag information and simultaneously delete each word. For example, the processor <NUM> may control the display <NUM> to display a UI element <NUM> as illustrated in <FIG>.

The processor <NUM> may identify a first object associated with the received voice in an image and obtain tag information for the first object with reference to the generated tag information for a second object included in the image. For example, as described with reference to <FIG>, new tag information may be generated with reference to the generated tag information.

When the object associated with the received voice is identified from the image, the processor <NUM> may control the display <NUM> to display a UI element indicating that the identified object is the target object to be tagged. For example, as illustrated in <FIG>, a rectangular UI element surrounding the target object <NUM> to be tagged as illustrated in <FIG> may be displayed.

When a plurality of objects associated with the voice are identified in the image, the processor <NUM> may obtain tagging information for each of the plurality of objects on the basis of the voice. For example, as shown in <FIG>, the processor <NUM> may obtain tag information for each of the first object (Junseo) and the second object (Yunseo) in the image based on the voice.

The processor <NUM> may store the obtained tag information associated with the image in the memory <NUM>.

<FIG> and <FIG> are flowcharts of a network system using an AI model according to various embodiments.

Referring to <FIG> and <FIG>, the network system using the AI model may include first elements <NUM>, <NUM> and second elements <NUM>, <NUM>.

The first elements <NUM>, <NUM> may be the electronic apparatus <NUM>. The second element <NUM>, <NUM> may be a server in which the AI model is stored. Alternatively, the first element <NUM>, <NUM> may be a general-purpose processor, and the second element <NUM>, <NUM> may be an AI-only processor. Alternatively, the first element <NUM>, <NUM> may be at least one application, and the second element <NUM>, <NUM> may be an operating system (OS). That is, the second element <NUM>, <NUM> may be an element that is more integrated, dedicated, with less delay, with better performance, or with larger resources and may be an element that may process more processing required for generating, updating, or applying a model more quickly and effectively than the first element <NUM>, <NUM>.

An interface for transmitting or receiving data between the first element <NUM>, <NUM> and the second element <NUM>, <NUM> may be defined.

For example, an application program interface (API) that has learning data to be applied to the model as a factor value (or an intermediate value or a transfer value) may be defined. The API may be defined as a set of sub-routines or functions that may be called for any processing of another protocol (e.g., a protocol defined in the external server of the electronic apparatus <NUM>) from any one protocol (e.g., the protocol defined in the electronic apparatus <NUM>). That is, an environment in which the operation of another protocol may be performed in one protocol through the API may be provided.

Referring to <FIG>, the first element <NUM> may display an image including at least one object in operation S2001.

While an image including at least one object is being displayed, the first element <NUM> may receive a user voice in operation S2003.

When the user voice is input, the first element <NUM> may transmit the image and the voice to the second element <NUM> in operation S2005.

The second element <NUM> may identify an object associated with the voice in the image based on the received voice and image in operation S2007.

The second element <NUM> may input a model for visual understanding of the image to obtain information about the included object in the image. For example, information about a position, a type of an object, a color of an object, a size of an object, a name of an object, a gender of an object, or the like, may be obtained as information on the object.

The second element <NUM> may input the voice into a model for linguistic understanding to obtain information about the user intent which the voice indicates. For example, the information about a user intention indicating whether a voice is a request for an action, whether a speaker requests a value of a certain variable to a listener (WH-Question), or whether the speaker requests an answer of YES/NO to a listener (YN-Question), or the speaker informs the listener, or the like, information indicating an action desired by the voice, information indicating a keyword included in the voice may be obtained.

The second element <NUM> may identify an object associated with the voice in the image based on information about the object and information about the user intention which the voice indicates. For example, the second element <NUM> may determine a word to identify a target object to be tagged among the keywords included in the voice and identify an object associated with the voice based on the determined word. For example, a word for identifying an object may be associated with at least one of a description of the appearance of the object, a description of the gender of the object, a description of the color of the object, a description of the position of the object, a description of the object's category, a description of the object's name, or the like. For example, a word for identifying an object in the voice "the left is my son, Junseo" may be determined to be "left", based on which an object located in the left within the image may be identified as an object associated with the voice.

The second element <NUM> may generate tag information for the identified object in operation S2009. According to one embodiment, the second element <NUM> may determine a word to describe an object of the keywords of voice and include the determined word to generate tag information. For example, a word for describing an object in the voice "the left is my son, Junseo" may be determined to be "son" and "Junseo," and based on it, tag information that includes the texts "son" and "Junseo" for the object located in the left in the image may be generated.

The second element <NUM> may transmit the generated tag information to the first element <NUM>. The first element <NUM> may provide tag information in operation S2013. For example, as shown in <FIG>, the tag information <NUM> may be provided around the target object <NUM> to be tagged.

<FIG> is a flowchart of a network system using a recognition model according to another embodiment.

Referring to <FIG>, the first element <NUM> may display an image including at least one object in operation S2101.

While an image including at least one object is being displayed, the first element <NUM> may receive the user voice in operation S2103.

When the user voice is input, the first element <NUM> may transmit the image and the voice to the second element <NUM> in operation S2105.

The second element <NUM> may obtain information on the object included in the received image in operation S2107.

The second element <NUM> may input a model for visual understanding of the image to obtain information about the included object in the image. For example, information about the type of the object, the color of the object, the size of the object, the name of the object, the gender of the object, or the like, may be obtained as information about the object.

The second element <NUM> may recognize the received voice in operation S2109.

The second element <NUM> may input the voice into a model for linguistic understanding to recognize the voice. For example, the second element <NUM> may identify the information about a user intention indicating whether a voice is a request for an action, whether a speaker requests a value of a certain variable to a listener (WH-Question), or whether the speaker requests an answer of YES/NO to a listener (YN-Question), or the speaker informs the listener, or the like, determine information indicating an action desired by the voice and information indicating a keyword included in the voice.

The second element <NUM> may transmit the obtained object information and the voice recognition result to the first element <NUM> in operation S2109.

The first element <NUM> may identify the object associated with the voice from the image based on the received information on the object and the voice recognition result in operation S2111.

The voice recognition result received from the second element <NUM> may include information about the keyword of voice. The first element <NUM> may determine a word for identifying an object of the keywords of voice and identify an object in the image having information corresponding to the determined word.

The first element <NUM> may generate the tag information for the identified object in operation S2113.

The first element <NUM> may determine a word for describing the identified object among the keywords of the voice and generate the tag information including the determined word.

The first element <NUM> may provide the generated tag information in operation S2115. For example, the tag information may be displayed through the display of the first element <NUM>.

<FIG> is a flowchart illustrating a method of controlling an electronic apparatus according to an embodiment. The flowchart shown in <FIG> may be configured with operations that are processed in the electronic apparatus <NUM> described herein. Accordingly, the content described with respect to the electronic apparatus <NUM> may also be applied to the flowchart shown in <FIG>, even though omitted below.

Referring to <FIG>, the electronic apparatus <NUM> displays an image including at least one object in operation S2210. This image may be pre-stored in the memory <NUM> of the electronic apparatus <NUM>. For example, the image has been photographed through the camera <NUM>.

The electronic apparatus <NUM> receives the voice in operation S2220. The voice may be received through the microphone <NUM> provided in the electronic apparatus <NUM>. The electronic apparatus <NUM> may remove an ambient noise, or the like, from the voice received through the microphone.

The electronic apparatus <NUM> inputs the voice to an AI model trained by an AI algorithm to identify an object related to the voice of at least one object included in the image and obtain tag information for the identified object in operation S2230. The AI model may be stored in the electronic apparatus <NUM>. Alternatively, the AI model may be stored in a server external to the electronic apparatus <NUM>, and the electronic apparatus <NUM> may provide the voice to an external server.

According to the invention, the electronic apparatus <NUM> inputs the image to the first AI model to obtain information about the at least one object, and identify an object related to the voice of the at least one object based on information on the at least one object and a word included in the voice in operation S2230.

In this example, the electronic apparatus <NUM> inputs the voice to the second AI model to obtain tag information including a keyword of the voice. The tag information may further include information on the identified object among information about the at least one object obtained by inputting the image to the first AI model.

According to another embodiment, in step S2230, the electronic apparatus <NUM> may identify the first object associated with the voice and obtain tag information for the first object with reference to the generated tag information for a second object included in the image.

According to another embodiment, the electronic apparatus <NUM> may obtain tagging information for each of the plurality of objects based on the voice when a plurality of objects associated with the voice in the image are identified in operation S2230.

The electronic apparatus <NUM> provides the obtained tag information in operation S2240.

According to an embodiment, in step S2240, a keyword of voice may be displayed together with an image. In this example, the electronic apparatus <NUM> may display the keywords of the input voice together with the previously displayed keywords. The electronic apparatus <NUM> may also display a UI element for deleting a keyword of voice from the tag information.

The electronic apparatus <NUM> may store the tag information associated with the image. For example, the memory <NUM> of the electronic apparatus <NUM> may store tag information. Alternatively, tag information may be stored in an external server. In this example, the tag information may be shared with other users.

According to the various embodiments described above, tag information of an image may be generated by only describing an image, and in particular, tag information may be generated for an identified object by identifying an object to be tagged in the voice of a user, thereby improving the convenience of a user using a tagging function.

The various embodiments described above may be implemented in a non-transitory computer-readable recording medium, which is readable by computer or a device similar to computer using software, hardware, or the combination of software and hardware. By hardware implementation, the embodiments of the disclosure may be implemented using at least one of application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), processors, controllers, microcontrollers, microprocessors, or electric units for performing other functions. In some cases, embodiments described herein may be implemented by the processor itself. According to a software implementation, embodiments such as the procedures and functions described herein may be implemented with separate software modules. Each of the above-described software modules may perform one or more of the functions and operations described herein.

In accordance with the embodiment of the present disclosure, the various embodiments described above may be practiced with other computer-readable media including instructions stored on a machine-readable storage medium. The device may include an electronic device (e.g., electronic device <NUM>) in accordance with the disclosed embodiments as an apparatus that is operable to invoke stored instructions from the storage medium and act upon the called instructions. When an instruction is executed by a processor, the processor may perform the function corresponding to the instruction, either directly, or using other components under the control of the processor. The instructions may include code generated or executed by the compiler or interpreter. Here, "non-transitory" means that the storage medium does not include a signal and is tangible, but does not distinguish whether data is permanently or temporarily stored in a storage medium.

Also, in accordance with one embodiment of the present disclosure, a method according to various embodiments described above may be provided in a computer program product. A computer program product may be traded between a seller and a purchaser as a commodity. A computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read only memory (CD-ROM)) or distributed online through an application store (e.g. PlayStore ™). In the case of on-line distribution, at least a portion of the computer program product may be stored temporarily, or at least temporarily, in a storage medium such as a manufacturer's server, a server of an application store, or a memory of a relay server.

Further, each of the components (for example, modules or programs) according to the above-described various embodiments may be composed of one or a plurality of entities, and some subcomponents of the above-mentioned subcomponents may be omitted, or other subcomponents may be further included in various embodiments. Alternatively or additionally, some components (e.g., modules or programs) may be integrated into one entity to perform the same or similar functions performed by each respective component prior to integration. Operations performed by a module, program, or other component, in accordance with various embodiments, may be performed sequentially, in a parallel, repetitive, or heuristically manner, or at least some operations may be performed in a different order.

Claim 1:
A control method of an electronic apparatus, the method comprising:
displaying an image including at least one object;
obtaining a user voice;
obtaining information on the at least one object by inputting the image to a first AI model for visual understanding, and identifying a target object among the at least one object based on the information obtained and a word included in the user voice;
obtaining tag information on the target object including a keyword of the user voice by inputting the user voice to a second AI model for linguistic understanding; and
providing the obtained tag information.