Text input device and method therefor

Electronic device includes display, microphone, and processor configured to activate voice input function based on user input, display graphic representation for indicating that the voice input function is activated, provide, on the display, a text display area for displaying text inputted by a plurality of user input methods and a keyboard input interface for receiving a user keyboard input, the plurality of user input methods including user keyboard input method and user voice input method, receive, via the keyboard input interface, the user keyboard input corresponding to a first text, display the first text in the text display area based on receiving the user keyboard input, receive user voice input corresponding to a second text while the keyboard input interface is provided and the voice input function is activated, and display the second text next to the first text in the text display area based on the user voice input.

BACKGROUND

The present disclosure relates to a device and method for inputting a text in response to a voice input.

2. Description of Related Art

With the development of information technology (IT), electronic devices, such as smartphones, which perform various functions in combination have been developed, and electronic devices having a voice recognition function have been released in order to improve operability. A voice recognition function has an advantage that enables a device to be easily controlled by recognizing a user's voice without contacting a separate operation button or a touch module.

According to a voice recognition function, for example, a portable terminal such as a smartphone may execute a call function, write a text message, and easily set various functions such as pathfinding, Internet search, or alarm setting, without pressing a separate button.

There is a need for a text input device and method that provide a voice input interface capable of efficiently recognizing a user voice input.

SUMMARY

A text input device according to an embodiment of the present disclosure includes: a voice input interface configured to recognize a voice input; a script input interface configured to recognize a script input; and a processor configured to determine whether a voice activation condition indicating whether a user's voice input is started is satisfied, activate the voice input interface based on a result of the determining, and simultaneously provide the activated voice input interface and the script input interface.

A text input method according to an embodiment of the present disclosure includes: providing a script input interface configured to recognize a script input; determining whether a voice activation condition indicating whether a user's voice input is started is satisfied; and activating a voice input interface configured to recognize a voice input, based on a result of the determining, and simultaneously providing the activated voice input interface and the script input interface.

A computer-readable recording medium according to an embodiment of the present disclosure may be a computer-readable recording medium having recorded thereon a program for executing the above-described method.

According to an embodiment, a text input device and method that provide a voice input interface recognizing a user voice input may be provided.

DETAILED DESCRIPTION

A text input device according to an embodiment of the present disclosure includes: a voice input interface configured to recognize a voice input; a script input interface configured to recognize a script input; and a processor configured to determine whether a voice activation condition indicating whether a user's voice input is started is satisfied, activate the voice input interface based on a result of the determining, and simultaneously provide the activated voice input interface and the script input interface.

Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings so that the present disclosure may be easily carried out by those of ordinary skill in the art. However, the present disclosure may be embodied in many different forms and is not limited to the embodiments described herein. Also, in order to clearly describe the present disclosure, parts having no relation to the description are omitted, and like reference numerals are assigned to like elements throughout the specification.

Some embodiments of the present disclosure may be represented by functional block configurations and various processing operations. All or part of such functional blocks may be implemented by any number of hardware and/or software elements configured to perform particular functions. For example, the functional blocks of the present disclosure may be implemented as one or more microprocessors or may be implemented as circuit configurations for certain functions. Also, for example, the functional blocks of the present disclosure may be implemented with various programming or scripting languages. The functional blocks may be implemented as algorithms that are executed on one or more processors. Also, the present disclosure may employ any conventional techniques for electronic environment configuration, signal processing, and/or data processing.

Also, the connecting lines or connecting members illustrated in the drawings are intended to represent exemplary functional connections and/or physical or logical connections between elements. In a real device, connections between elements may be represented by various functional connections, physical connections, or circuit connections that are replaceable or added.

Also, the terms “-er/or” and “module” as used herein represent a unit for processing at least one function or operation, which may be implemented by hardware, software, or a combination thereof. The terms “-er/or” and “module” may be implemented by a program that is stored in an addressable storage medium and is executable by a processor.

For example, the terms “-er/or” and “module” may refer to elements such as software elements, object-oriented software elements, class elements, and task elements, and may include processes, functions, attributes, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuits, data, a database, data structures, tables, arrays, and variables.

The embodiments of the present disclosure, which are described in this specification and drawings, are merely presented as specific examples so as to easily explain the technical contents of the present disclosure and help the understanding of the present disclosure and are not intended to limit the scope of the present disclosure. That is, it will be obvious to those of ordinary skill in the art that other modifications based on the technical idea of the present disclosure may be made. Also, the respective embodiments may be operated in combination with each other as necessary. For example, some portions of one embodiment of the present disclosure and some portions of another embodiment of the present disclosure may be combined with each other so that the device may be operated.

FIG. 1is a diagram of a text input device according to an embodiment.

Referring toFIG. 1, a text input device100includes a processor110, a script input interface120, and a voice input interface130. The processor110controls overall operations of the text input device100.

A user10may provide a voice input or a script input to the text input device100. The voice input is a user's voice including information, for example, text information. The script input is a user's action including information, for example, text information. The user's action may include a contact with the device.

The text input device100may receive the script input by using the script input interface120and may receive the voice input by using the voice input interface130. The script input interface120may include a keyboard user interface (UI) on which characters, numbers, and symbols are displayed. For UIs included in the script input interface120, characters may be different for each type of language. For example, a UI for Korean input and a UI for English input may exist separately. The text input device100may simultaneously provide the script input interface120and the voice input interface130.

The processor110may determine whether a voice activation condition and a voice deactivation condition are satisfied. The voice activation condition may be a result of determining the intention of the user10to provide the voice input. The voice deactivation condition may be a result of determining the intention of the user10to end providing the voice input. The processor110may automatically activate or deactivate the voice input interface130based on whether the determined voice activation condition or voice deactivation condition is satisfied.

In an embodiment, the processor110may determine whether the voice activation condition is satisfied, based on sound information received by using a microphone. The processor110according to the embodiment may activate the microphone based on a previously determined condition. For example, the processor110may activate the microphone when the script input is detected. The processor110may determine whether user voice information is included in the sound information received by using the microphone, and may determine whether the voice activation condition is satisfied, based on a result of the determining.

In an embodiment, the processor110may determine whether the voice activation condition is satisfied, based on a slope of the text input device100. The slope of the text input device100may be measured by using an accelerometer or a gyroscope. The processor110may determine whether the voice activation condition is satisfied by comparing the slope of the text input device100with a threshold value. Alternatively, the processor110may determine whether the voice activation condition is satisfied, based on whether the slope of the text input device100is a slope suitable for user face recognition. However, the operation of determining the voice activation condition based on the slope of the text input device100is not limited to the above-described embodiments.

In an embodiment, the processor110may determine whether the voice activation condition is satisfied, based on a distance from the user's face to the text input device100. The processor110according to the embodiment may determine the distance from the user's face to the text input device100based on image information received by using a camera. The processor110may determine whether the voice activation condition is satisfied by comparing the aforementioned distance with a threshold value. Alternatively, the processor110may determine whether the voice activation condition is satisfied, based on whether the aforementioned distance is a distance suitable for user face recognition. However, the operation, performed by the processor110, of determining the voice activation condition based on the distance is not limited to the above-described embodiments.

In an embodiment, the processor110may determine whether the voice activation condition is satisfied, based on a determination of a user's lip movement. The processor110according to the embodiment may activate a camera on the user side, for example, a front camera, based on a previously determined condition. For example, the processor110may activate the camera when the script input is detected. The processor110may determine the user's lip movement based on image information received by using a camera. For example, the processor110may determine whether the voice activation condition is satisfied, based on whether the user's lip movement included in the received image information corresponds to the voice input.

In an embodiment, the processor110may determine whether the voice activation condition is satisfied, based on context information. The context information may include at least one of surrounding environment information of the text input device100, state information of the text input device100, user state information, user's usage history information of the text input device100, and user schedule information, but the present disclosure is not limited thereto.

The surrounding environment information of the text input device100refers to environment information within a certain radius from the text input device100, and may include weather information, temperature information, humidity information, illumination information, noise information, sound information, time information, and the like. For example, when the surrounding environment is a crowded area, the text input device100may determine that the voice activation condition is not satisfied. Alternatively, when a current time is not a late time, the text input device100may satisfy the voice activation condition with a higher frequency. However, the surrounding environment information included in the context information is not limited thereto.

The state information of the text input device100may include mode information of the text input device100(e.g., a sound mode, a vibration mode, a silent mode, a power saving mode, a blocking mode, a multi-window mode, an automatic rotation mode, etc.), location information of the text input device100, time information, communication module activation information (e.g., Wi-Fi ON/Bluetooth OFF/GPS ON/NFC ON, etc.), network connection state information of the text input device100, information about an application executed in the text input device100(e.g., application identification information, application type, application use time, application use cycle, etc.), and the like. For example, the text input device100may determine whether the voice activation condition is satisfied, based on a posture, an angle, a relative position, and the like of the text input device100when the user starts a conversation. However, the state information of the text input device included in the context information is not limited thereto.

The user state information is information about a user's characteristics, movement, and life pattern, and the like, and may include information about a user's gender, a user's walking state, exercising state, driving state, and sleeping state, a user's mood state, and the like. For example, when a gesture that the user uses frequently during a voice input is recognized, the text input device100may satisfy the voice activation condition with a higher frequency. However, the user state information included in the context information is not limited thereto.

The user's usage history information of the text input device100is information about the history of the user using the text input device100, and may include execution history of the application, the history of functions executed in the application, a user's call history, a user's text history, and the frequency of words included in the text information. For example, the text input device may satisfy the voice activation condition with a higher frequency during a text conversation with a specific user. However, the usage history information included in the context information is not limited thereto.

In determining whether the voice activation condition is satisfied, which context information is to be used may be determined according to learning based on a preset criterion. For example, supervised learning that uses a certain voice input and certain context information as input values and unsupervised learning that discovers a pattern of a voice activation condition by self-learning a type of context information necessary for determining a voice activation condition without separate supervision may be used to determine the voice activation condition. Also, for example, reinforcement learning that uses a feedback on a result of grasping the user's intention according to learning is correct may be used to determine the voice activation condition.

In an embodiment, the processor110may determine whether the voice activation condition is satisfied by combining the above-described embodiments in parallel or in combination.

The processor110may activate the voice input interface130for recognizing the voice input, based on a result of determining whether the voice activation condition is satisfied. The processor110may simultaneously provide the voice input interface130and the script input interface120.

The processor110generates voice text information based on the voice input recognized by using the voice input interface130. Also, the processor110generates script text information based on the script input recognized by using the script input interface120. The processor110may generate text information by combining the voice text information with the script text information.

The processor110may control a display (not illustrated) of the text input device100such that the generated text information is displayed thereon. In order to display the text information, the processor110may display the script text information and the voice text information in the order of generation.

The processor110may determine whether the voice deactivation condition is satisfied. The voice deactivation condition may be a result of determining the intention of the user10to end providing the voice input.

In an embodiment, the processor110may determine whether the voice deactivation condition is satisfied, based on context information. Alternatively, the processor110may determine whether the voice deactivation condition is satisfied, based on sound information received by using a microphone, a slope of the text input device100, a distance from a user's face, and a determination of a user's lip movement. For example, the processor110may determine whether the voice deactivation condition is satisfied, based on whether the slope of the text input device100is a slope unsuitable for user face recognition. Also, the processor110may determine the voice deactivation condition based on an element used to determine whether the voice activation condition is satisfied. However, the operation, performed by the processor110, of determining the voice deactivation condition is not limited to the above-described embodiments.

The processor110may deactivate the voice input interface130based on a result of determining whether the voice deactivation condition is satisfied. The text input device100may simultaneously provide the script input interface120and the voice input interface130and control whether to activate the voice input interface130automatically or according to the user's selection, based on the voice activation condition or the voice deactivation condition. Thus, the text input operation may be efficiently performed.

Examples of the text input device100may include smartphones, tablet personal computers (PCs), PCs, smart televisions (TVs), mobile phones, personal digital assistants (PDAs), laptop computers, media players, micro servers, global positioning system (GPS) devices, e-book terminals, digital broadcasting terminals, navigations, kiosk, MP3 players, digital cameras, home appliances, and other mobile or non-mobile computing devices, but the present disclosure is not limited thereto. Also, examples of the text input device100may include wearable devices, such as watches, glasses, hair bands, or rings, which have a communication function and a data processing function. However, the present disclosure is not limited thereto, and the text input device100may include any types of devices capable of receiving a user input and providing text based thereon.

Also, the text input device100may communicate with a server and other devices (not illustrated) via a certain network in order to use various pieces of context information. In this case, the network may include a local area network (LAN), a wide area network (WAN), a value added network (VAN), a mobile radio communication network, a satellite communication network, or any combinations thereof. The network may be a comprehensive data communication network that enables network configuration entities to efficiently communicate with each other and may include a wired Internet, a wireless Internet, and a mobile wireless communication network. Examples of the wireless communication may include wireless LAN (Wi-Fi), Bluetooth, Bluetooth Low Energy, Zigbee, Wi-Fi Direct (WFD), ultra wideband (UWB), Infrared Data Association (IrDA), and Near Field Communication (NFC), but the present disclosure is not limited thereto.

FIG. 2is a diagram of a text input device according to an embodiment.

Referring toFIG. 2, a text input device200includes a processor210, a script input interface220, a voice input interface230, and a sensor240.

The processor210controls overall operations of the text input device200.

The text input device200may receive a script input by using the script input interface220and may receive a voice input by using the voice input interface230. The text input device200may simultaneously provide the script input interface220and the voice input interface230.

The sensor240includes at least one sensor that receives information from the outside. In an embodiment, the sensor240may include a microphone that receives sound information. In an embodiment, the sensor240may include an accelerometer or a gyroscope that detects movement of the text input device200. In an embodiment, the sensor240may include a camera that receives image information. In an embodiment, the sensor240may include a temperature sensor or a humidity sensor that receives surrounding environment information. However, this is an example, and the type of sensor included in the sensor240is not limited thereto. The sensor240may provide the received information to the processor210. The processor210may determine whether a voice activation condition is satisfied, based on the provided information.

FIG. 3is a diagram of a text input device according to an embodiment.

Referring toFIG. 3, a text input device300includes a processor310, a script input interface320, a voice input interface330, and an auxiliary voice input interface340.

A user30may provide a voice input or a script input to the text input device300. The voice input is a user's voice including information, for example, text information. The script input is a user's action including information, for example, text information. The user's action may include a contact with the device.

The processor310may control the script input interface320and the voice input interface330such that the script input is input by using the script input interface320and the voice input is input by using the voice input interface330.

The processor310may determine whether a voice activation condition and a voice deactivation condition are satisfied. The processor310may automatically activate or deactivate the voice input interface330based on a result of determining whether the voice activation condition or the voice deactivation condition is satisfied.

The processor310may generate recommended voice text information from the voice input recognized by using the voice input interface330, based on context information. The processor310may provide the recommended voice text information to a user by using the auxiliary voice input interface340. The recommended voice text information is text information expected to be input by the user, based on the voice input and the context information.

The context information may include at least one of surrounding environment information of the text input device300, state information of the text input device300, user state information, user's usage history information of the text input device300, and user schedule information, but the present disclosure is not limited thereto.

In generating the recommended voice text information, which context information is to be used may be determined according to learning based on a preset criterion. For example, supervised learning that uses a certain voice input and certain context information as input values and unsupervised learning that discovers a pattern of recommended voice text information by self-learning a type of context information necessary for the recommended voice text information without separate supervision may be used to generate the recommended voice text information. Also, for example, reinforcement learning that uses a feedback on whether a result of grasping the user's intention according to learning is correct may be used to generate the recommended voice text information.

The processor310may receive a user input of selecting one of pieces of the provided recommended voice text information as voice text information by using the auxiliary voice input interface340. Also, the processor310may generate script text information based on the script input recognized by using the script input interface320. The processor310may generate text information by combining the selected voice text information with the script text information.

The processor310may control a display (not illustrated) such that the generated text information is displayed thereon. In order to display the text information, the processor310may display the script text information and the voice text information in the order of generation.

FIG. 4Ais a diagram for describing a method, performed by a text input device400a, of performing a text input operation, according to some embodiments.

Referring toFIG. 4A, the text input device400aincludes a script input interface420aand a voice input interface (not illustrated). A user40may provide a voice input or a script input to the text input device400a.

The text input device400amay receive the script input by using the script input interface420aand may receive the voice input by using the voice input interface. The text input device400amay generate voice text information based on the voice input recognized by using the voice input interface and may generate script text information based on the script input recognized by using the script input interface420a. The text input device400amay generate text information by combining the voice text information with the script text information.

InFIG. 4A, the script input interface420ais displayed in the form of a QWERTY keyboard UI, but this is an example. The form of the script input interface420aaccording to the present embodiment is not limited to the aforementioned keyboard UI. Also, although the voice input interface is not illustrated inFIG. 4A, this is an example. The script input interface420amay be displayed on the text input device400ain the form of a UI.

The text input device400amay provide the script input interface420awhen a text input request is detected. For example, the text input device400amay provide the script input interface420awhen a user input of requesting a UI for text input is detected. Alternatively, the text input device400amay provide the script input interface in response to a request from an application executed in the text input device400a.

As described above, the text input device400amay generate script text information based on the script input recognized by using the script input interface320. The script input may include a text of at least one of characters, numbers, and symbols. The text input device400amay display the generated script text information on the text display area401a.

The text input device400amay determine whether a voice activation condition, which is a result of determining the intention of the user to provide the voice input, is satisfied. In an embodiment, the text input device400amay determine whether the voice activation condition is satisfied, based on a combination of the presence or absence of script input, sound information received by using a microphone, a slope of the text input device400a, a distance from a user's face, and a user's lip movement. However, the method of determining the voice activation condition is not limited to the above-described embodiments.

The text input device400amay provide the voice input interface for recognizing the voice input together with the script input interface420a, based on the determined voice activation condition.

In an embodiment, the text input device400amay display a state of the voice input interface, that is, whether activated or deactivated, while the voice input interface is provided. For example, the text input device400amay change a color of a voice input activation icon402adisplayed on the text input device400a, based on the state of the voice input interface. The voice input activation icon402amay be located on the script input interface420a. However, the method of displaying the state of the voice input interface while the voice input interface is provided is not limited to the above-described embodiments.

As described above, the text input device400amay generate voice text information based on the voice input recognized by using the voice input interface. The text input device400amay display the generated voice text information on the text display area401a.

In an embodiment, when a voice input interface is provided together with the script input interface, the text input device400amay display, on the text display area401a, the voice text information or the script text information generated based on the voice input or the script input.

In an embodiment, the text input device400amay display the voice text information or the script text information on the text display area401ain the order of input. When the inputs are performed at the same time, the text input device400amay determine a display order of voice text information or script text information according to a pre-designated priority flag. The priority flag according to the embodiment may be designated by the user or may be pre-designated and stored in a memory (not illustrated).

FIG. 4Bis a diagram for describing a method, performed by a text input device400b, of performing a text input operation, according to some embodiments.

Referring toFIG. 4B, the text input device400bincludes a script input interface420b, a voice input interface (not illustrated), and an auxiliary voice input interface440b. A user40may provide a voice input or a script input to the text input device400b.

The text input device400bmay receive the script input by using the script input interface420band may receive the voice input by using the voice input interface.

InFIG. 4B, the script input interface420bis displayed in the form of a QWERTY keyboard UI, but this is an example. The form of the script input interface420baccording to the present embodiment is not limited to the aforementioned keyboard UI. Also, although the voice input interface is not illustrated inFIG. 4B, this is an example. The voice input interface may be displayed on the text input device400bin the form of a UI.

The text input device400bprovides the script input interface420bwhen a text input request is detected. When the text input device400bprovides the script input interface420b, the text input device400bmay set a text display area401bthat is a preset area for displaying text information.

The text input device400bmay generate script text information based on the script input recognized by using the script input interface420b. The text input device400bmay display the generated script text information on the text display area401b.

The text input device400bmay determine whether a voice activation condition, which is a result of determining the intention of the user to provide the voice input, is satisfied. In an embodiment, the text input device400bmay determine whether the voice activation condition is satisfied, based on a combination of the presence or absence of script input, sound information received by using a microphone, a slope of the text input device400b, a distance from a user's face, and a user's lip movement. However, the method of determining the voice activation condition is not limited to the above-described embodiments.

The text input device400bmay provide the voice input interface for recognizing the voice input together with the script input interface420b, based on the determined voice activation condition.

In an embodiment, the text input device400bmay display a state of the voice input interface, that is, whether activated or deactivated, while the voice input interface is provided. For example, the text input device400bmay change a color of a voice input activation icon402bdisplayed on the text input device400b, based on the state of the voice input interface. However, the method of displaying the state of the voice input interface while the voice input interface is provided is not limited to the above-described embodiments.

The text input device400bmay generate recommended voice text information from the voice input received by using the voice input interface, based on context information. The recommended voice text information is text information expected to be input by the user, based on the voice input and the context information. For example, when the text input device400bdetermines that text information “text” is included in the voice input, the text input device400bmay generate “texture,” “text,” and “texted” as the recommended voice text information.

The text input device400bmay provide the recommended voice text information to the user by using the auxiliary voice input interface440b. Also, the text input device400bmay receive a user input of selecting one of pieces of the provided recommended voice text information as voice text information by using the auxiliary voice input interface440b.

The text input device400bmay display the selected voice text information on the text display area401b.

More specifically, when a voice input interface is provided together with the script input interface, the text input device400bmay display, on the text display area401b, the voice text information or the script text information generated based on the voice input or the script input.

In an embodiment, the text input device400bmay display the voice text information or the script text information on the text display area401bin the order of input. When the inputs are performed at the same time, the text input device400bmay determine a display order of voice text information or script text information according to a pre-designated priority flag. The priority flag according to the embodiment may be designated by the user or may be pre-designated and stored in a memory (not illustrated).

FIG. 5is a diagram for describing a method, performed by a text input device, of performing a text input operation based on context information, according to some embodiments.

Referring toFIG. 5, text input devices500aand500binclude script input interfaces520aand520b, voice input interfaces (not illustrated), and auxiliary voice input interfaces540aand540b, respectively. A user may provide voice inputs or script inputs to the text input devices500aand500b.

The text input devices500aand500bmay receive the script inputs by using the script input interfaces520aand520b, respectively, and may receive the voice inputs by using the voice input interfaces, respectively. When the text input devices500aand500bprovide the script input interfaces520aand520b, respectively, the text input devices500aand500bmay set text display areas501aand501bthat are a preset area for displaying text information, respectively.

The text input devices500aand500bmay determine whether a voice activation condition, which is a result of determining the intention of the user to provide the voice input, is satisfied. In an embodiment, the text input devices500aand500bmay determine whether the voice activation condition is satisfied, based on a combination of the presence or absence of script input, sound information received by using a microphone, slopes of the text input devices500aand500b, a distance from a user's face, and a user's lip movement. However, the method of determining the voice activation condition is not limited to the above-described embodiments.

The text input devices500aand500bmay provide the voice input interfaces for recognizing the voice inputs together with the script input interfaces520aand520b, based on the determined voice activation condition, respectively.

The text input devices500aand500bmay generate recommended voice text information from the voice inputs received by using the voice input interfaces, based on context information, respectively. The recommended voice text information is text information expected to be input by the user, based on the voice input and the context information.

In an embodiment, the context information includes gender information of the user. In an embodiment, the gender information of the user may be determined from the user's voice input, or may be previously stored or learned and determined.

Referring toFIG. 5, when the gender of the user included in the context information is female, the text input device500amay generate “fiancee,” “husband,” and “boyfriend” as recommended voice text information. When the gender of the user included in the context information is male, the text input device500bmay generate “fiancee,” “wife,” and “girlfriend” as recommended voice text information. Each word disclosed inFIG. 5is an example, and the methods, performed by the text input devices500aand500b, of generating the recommended voice text information based on the gender information are not limited thereto.

FIG. 6is a flowchart for describing a text input method according to an embodiment.

In operation S610, a text input device provides a script input interface for receiving a script input. The script input is a user's action including information, for example, text information. The user's action may include a contact with the device. The script input interface may include a keyboard UI on which characters, numbers, and symbols are displayed. In an embodiment, the text input device may provide the script input interface when a user input of requesting an interface for text input is detected.

In operation S620, the text input device determines whether a voice activation condition is satisfied. The voice activation condition may be a result of determining the intention of the user to provide the voice input. In an embodiment, the text input device may determine whether the voice activation condition is satisfied, based on a combination of the presence or absence of script input, sound information received by using a microphone, a slope of the text input device, a distance from a user's face, and a user's lip movement. Alternatively, when the script input interface is provided, the text input device may automatically determine that the voice activation condition is satisfied. However, the method of determining the voice activation condition is not limited to the above-described embodiments.

In operation S630, the text input device provides a voice input interface for recognizing voice input together with the script input interface, based on the determined voice activation condition. For example, when the user's voice is included in the sound information received by using the microphone and the voice activation condition is satisfied, the text input device may provide the voice input interface together with the script input interface when the user's voice is present. Alternatively, when the script input interface is provided and the voice activation condition is satisfied, the text input device may automatically provide the script input interface and the voice input interface at the same time at any time. That is, when the voice activation condition is satisfied, the text input device may activate the voice input interface to provide the user with a text input using both the voice input and the script input.

FIG. 7is a flowchart for describing a method of determining a voice activation condition, according to an embodiment.

In operation S710, a text input device activates a microphone. The text input device may activate the microphone when a preset condition is satisfied. For example, the text input device may activate the microphone when a script input is present. In an embodiment, when a script input that is input via the script input interface within a certain time is present, the text input device may determine that the script input is present.

In operation S720, the text input device receives sound information by using the microphone. The sound information is auditory information around the text input device, which is received by using the microphone.

In operation S730, the text input device determines whether a user's voice is included in the received sound information. The method, performed by the text input device, of determining whether the user's voice is included may be performed based on various algorithms. Also, the text input device may perform noise cancellation and various signal processing operations in order to determine whether the user's voice is included.

In operation S740, the text input device determines whether the voice activation condition is satisfied, based on whether the user's voice is included. When the user's voice is recognized, the text input device may determine that the voice activation condition has been satisfied, and provide the user with a text input via voice.

FIG. 8is a flowchart for describing a method of determining a voice activation condition, according to an embodiment.

In operation S810, the text input device measures a slope of the text input device. In an embodiment, the text input device may measure the slope of the text input device by using an accelerometer or a gyroscope. In an embodiment, the text input device may perform a slope measuring operation when a preset condition is satisfied. For example, the text input device may measure the slope when a script input is present. In an embodiment, when a script input that is input via the script input interface within a certain time is present, the text input device may determine that the script input is present.

In operation S820, the text input device determines whether the voice activation condition is satisfied, based on the measured slope. In an embodiment, the text input device may determine whether the voice activation condition is satisfied by comparing the measured slope with a threshold value. Alternatively, the text input device may determine whether the voice activation condition is satisfied, based on whether the measured slope is a slope suitable for user face recognition. However, the operation of determining the voice activation condition based on the slope of the text input device is not limited to the above-described embodiments.

FIG. 9is a flowchart for describing a method of determining a voice activation condition, according to an embodiment.

In operation S910, the text input device measures a distance from a user's face to the text input device. In an embodiment, the text input device may measure the aforementioned distance by using a camera on the user side, for example, a front camera. In an embodiment, the text input device may activate the camera when a preset condition is satisfied. For example, the text input device may activate the camera when a script input is present. Alternatively, the text input device may activate the camera when the slope of the text input device satisfies a certain condition. However, the camera activating operation and the distance measuring operation of the text input device are not limited to the above-described embodiments.

In operation S920, the text input device determines whether the voice activation condition is satisfied, based on the measured distance. In an embodiment, the text input device may determine whether the voice activation condition is satisfied by comparing the measured distance with a threshold value. Alternatively, the text input device may determine whether the voice activation condition is satisfied, based on whether the measured distance is a distance suitable for user face recognition. However, the operation, performed by the text input device, of determining the voice activation condition based on the distance is not limited to the above-described embodiments.

FIG. 10is a flowchart for describing a method of determining a voice activation condition, according to an embodiment.

In operation S1010, the text input device activates a camera to receive image information. In an embodiment, the text input device may activate a camera on the user side, for example, a front camera. In an embodiment, the text input device may activate the camera when a preset condition is satisfied. For example, the text input device may activate the camera when a script input is present. Alternatively, the text input device may activate the camera when the slope of the text input device satisfies a certain condition. However, the camera activating operation and the distance measuring operation of the text input device are not limited to the above-described embodiments.

In operation S1020, the text input device determines a user's lip movement based on the received image information. In an embodiment, the text input device may recognize a user's face from the image information.

In operation S1030, the text input device determines whether a voice activation condition is satisfied, based on the determined lip movement. In an embodiment, the text input device may determine whether the voice activation condition is satisfied, based on whether the lip movement is present. Alternatively, the text input device may determine whether the voice activation condition is satisfied, based on whether the determined lip movement includes text information. However, the operation, performed by the text input device, of determining the voice activation condition based on the lip movement is not limited to the above-described embodiments.

FIG. 11is a flowchart for describing a method of determining a voice activation condition, according to an embodiment.

In operation S1110, a text input device measures a slope of the text input device. In an embodiment, the text input device may measure the slope of the text input device by using an accelerometer or a gyroscope. For example, the text input device may perform a slope measuring operation when a preset condition is satisfied.

In operation S1120, the text input device activates a camera to receive image information. In an embodiment, the text input device may receive the image information by using a camera on the user side, for example, a front camera. In an embodiment, the text input device may activate the camera when a preset condition is satisfied. For example, the text input device may activate the camera when a script input is present. Alternatively, the text input device may activate the camera when the slope of the text input device satisfies a certain condition.

In operation S1130, the text input device measures a distance from a user's face to the text input device. In an embodiment, the text input device may measure the aforementioned distance based on the image information received by using the camera.

In operation S1140, the text input device activates a microphone to receive sound information. The text input device may activate the microphone when a preset condition is satisfied. For example, the text input device may activate the microphone together when the camera is activated. Alternatively, the text input device may activate the microphone when a distance between the text input device and the user's face satisfies a certain condition.

In operation S1150, the text input device determines whether a voice activation condition is satisfied. For example, when the user's voice is recognized from the sound information received by using the microphone, the text input device may determine that the voice activation condition has been satisfied. Alternatively, when the user's lip movement is recognized from the image information received by using the camera, the text input device may determine that the voice activation condition has been satisfied. Alternatively, when the user's voice is recognized from the sound information and the user's lip movement is recognized from the image information received by using the camera, the text input device may determine that the voice activation condition has been satisfied. The text input device may determine whether the voice activation condition is satisfied by using sensing operations of various sensors, and provide the user with a text input via voice.

FIG. 12is a flowchart for describing a text input method according to an embodiment.

In operation S1210, a text input device provides a script input interface for recognizing a script input. In an embodiment, the text input device may provide the script input interface when a user input of requesting an interface for text input is detected.

In operation S1220, the text input device determines whether a voice activation condition is satisfied. In an embodiment, the text input device may determine whether the voice activation condition is satisfied, based on a combination of the presence or absence of script input, sound information received by using a microphone, a slope of the text input device, a distance from a user's face, and a user's lip movement.

In operation S1230, the text input device simultaneously provides a voice input interface for recognizing voice input and a script input interface, based on a result of determining whether the voice activation condition is satisfied.

In operation S1240, the text input device determines whether a voice deactivation condition is satisfied. The voice deactivation condition is a result of determining the intention of the user to end providing the voice input. In an embodiment, the text input device may determine whether the voice deactivation condition is satisfied, based on sound information received by using a microphone, a slope of the text input device, a distance from a user's face, a user's lip movement, and the like. For example, the text input device may determine whether the voice deactivation condition is satisfied, based on whether the slope of the text input device is a slope unsuitable for user face recognition. Also, the text input device may determine the voice deactivation condition based on an element used to determine whether the voice activation condition is satisfied.

In operation S1250, the text input device deactivates the voice input interface, based on whether the voice deactivation condition is satisfied. The text input device may simultaneously provide the script input interface and the voice input interface and control whether to activate the voice input interface automatically or according to the user's selection, based on the voice activation condition or the voice deactivation condition. Thus, the text input operation may be efficiently performed.

FIGS. 13 and 14are block diagrams of a first device1000according to some embodiments.

As illustrated inFIG. 13, the first device1000according to some embodiments may include a user inputter1100, an outputter1200, a processor1300, and a communicator1500. However, all elements illustrated inFIG. 13are not essential to the first device1000. The first device1000may include more elements than those illustrated inFIG. 13or may include fewer elements than those illustrated inFIG. 13.

For example, as illustrated inFIG. 14, the first device1000according to some embodiments may further include, in addition to the user inputter1100, the outputter1200, the processor1300, and the communicator1500, a sensor1400, an audio/video (A/V) inputter1600, and a memory1700.

The user inputter1100is a device that allows a user to input data for controlling the first device1000. For example, the user inputter1100may include a key pad, a dome switch, a touch pad (e.g., a touch-type capacitive touch pad, a pressure-type resistive touch pad, an infrared beam sensing type touch pad, a surface acoustic wave type touch pad, an integral strain gauge type touch pad, a piezo effect type touch pad, etc.), a jog wheel, and a jog switch, but the present disclosure is not limited thereto.

The user inputter1100may include a script input interface and a voice input interface. The user inputter1100may receive a voice input or a script input. Also, the user inputter1100may receive a user input of selecting one of pieces of recommended voice text information as voice text information.

The outputter1200may output an audio signal, a video signal, or a vibration signal. The outputter1200may include a display1210, an audio outputter1220, and a vibration motor1230.

The display1210displays information processed by the first device1000. For example, the display1210may display the script input interface for receiving the script input or a voice input activation icon indicating whether the voice input interface is activated. Also, the display1210may display text information and a text display area that is a preset area for displaying the text information.

When the display1210and the touch pad form a layered structure to constitute a touch screen, the display1210may also be used as an input device as well as an output device. The display1210may include at least one selected from liquid crystal display (LCD), thin-film transistor-liquid crystal display (TFT-LCD), organic light-emitting diode (OLED), flexible display, three-dimensional (3D) display, and electrophoretic display. According to the implementation type of the first device1000, the first device1000may include two or more displays1210. The two or more displays1210may be arranged to face each other by using a hinge.

The audio outputter1220outputs audio data that is received from the communicator1500or is stored in the memory1700. Also, the audio outputter1220outputs an audio signal associated with the function performed by the first device1000(e.g., a call signal reception sound, a message reception sound, a notification sound, etc.). The audio outputter1220may include a speaker, a buzzer, and the like.

The vibration motor1230may output a vibration signal. For example, the vibration motor1230may output a vibration signal corresponding to the output of audio data or video data (e.g., a call signal reception sound, a message reception sound, etc.). Also, the vibration motor1230may output a vibration signal when a touch is input to a touch screen.

The processor1300controls overall operations of the first device1000. For example, the processor1300may execute programs stored in the memory1700to control the user inputter1100, the outputter1200, the sensor1400, the communicator1500, and the A/V inputter1600.

Specifically, the processor1300may control the user inputter1100such that a script input is input by using a script input interface and a voice input is input by using a voice input interface.

The processor1300may determine whether a voice activation condition and a voice deactivation condition are satisfied. In an embodiment, the processor1300may determine whether the voice activation condition and the voice deactivation condition are satisfied, based on a combination of the presence or absence of script input, sound information received by using a microphone, a slope of the first device1000, a distance from a user's face, and a user's lip movement. However, the method of determining the voice activation condition is not limited to the above-described embodiments.

The processor1300may automatically activate or deactivate the voice input interface included in the user inputter1100, based on the determined voice activation condition or the determined voice deactivation condition.

Also, the processor1300may generate recommended voice text information from the voice input recognized by using the user inputter1100, based on context information. The processor1300may provide the generated recommended voice text information to the user.

The processor1300may control the user inputter1100to receive a user input of selecting one of pieces of the provided recommended voice text information as voice text information by using the auxiliary voice input interface included in the user inputter1100. Also, the processor1300may generate script text information based on the script input recognized by using the script input interface. The processor1300may generate text information by combining the selected voice text information with the script text information.

The processor1300may control the display1210such that the generated text information is displayed thereon. In order to display the text information, the processor1300may display the script text information and the voice text information in the order of generation.

The sensor1400may sense a state of the first device1000or a state around the first device1000and transmit sensed information to the processor1300.

The sensor1400may include at least one of a geomagnetic sensor1410, an acceleration sensor1420, a temperature/humidity sensor1430, an infrared sensor1440, a gyroscope sensor1450, a position sensor (e.g., GPS)1460, a barometric sensor1470, a proximity sensor1480, and an RGB sensor (illuminance sensor)1490, but the present disclosure is not limited thereto. Because the functions of the respective sensors may be intuitively inferred from their names, detailed descriptions thereof will be omitted.

The communicator1500may include one or more elements for communication between the first device1000and a head mounted display (HMD) device or communication between the first device1000and a server. For example, the communicator1500may include a short-range wireless communicator1510, a mobile communicator1520, and a broadcasting receiver1530.

The short-range wireless communicator1510may include a Bluetooth communicator, a Bluetooth Low Energy (BLE) communicator, a near field communicator, a wireless local area network (WLAN) (Wi-Fi) communicator, a Zigbee communicator, an infrared data association (IrDA) communicator, a Wi-Fi direct (WFD) communicator, an ultra-wideband (UWB) communicator, an Ant+ communicator, and the like, but the present disclosure is not limited thereto.

The mobile communicator1520may transmit and receive a wireless signal with at least one of a base station, an external terminal, or a server on a mobile communication network. Examples of the wireless signal may include a voice call signal, a video call signal, and various types of data according to transmission or reception of text/multimedia messages.

The broadcasting receiver1530may receive a broadcasting signal and/or broadcasting-related information from the outside via a broadcasting channel. The broadcasting channel may include a satellite channel, a ground wave channel, or the like. The first device1000may not include the broadcasting receiver1530according to an implementation example.

Also, the communicator1500may transmit or receive information for using context information with an HMD device, a server, and a peripheral device.

The A/V inputter1600may be provided for input of an audio signal or a video signal. For the input of the audio signal or the video signal, the A/V inputter1600may include a camera1610, a microphone1620, and the like. The camera1610may obtain an image frame, such as a still image or a moving image, through an image sensor in a video call mode or an image capture mode. An image captured by the image sensor may be processed by the processor1300or a separate image processor (not illustrated).

The image frame processed by the camera1610may be stored in the memory1700or may be transmitted to the outside through the communicator1500. Alternatively, the image frame may be used for the operation, performed by the processor1300, of determining the voice activation condition and the voice deactivation condition. The camera1610may include two or more cameras according to a configuration of a terminal.

The microphone1620may receive an external audio signal and process the external audio signal into electrical voice data. For example, the microphone1620may receive an audio signal from an external device or a person. The microphone1620may use various noise cancellation algorithms for cancelling noise generated in a process of receiving an external audio signal. The audio signal received through the microphone1620may be used for the operation, performed by the processor1300, of determining the voice activation condition and the voice deactivation condition.

The memory1700may store programs for processing and control of the processor1300and may store data input to the first device1000or data output from the first device1000.

The memory1700may include at least one type of storage medium selected from among a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (for example, a secure digital (SD) or extreme digital (XD) memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable ROM (EEPROM), programmable ROM (PROM), magnetic memory, magnetic disk, and optical disk.

The programs stored in the memory1700may be classified into a plurality of modules according to functions thereof. For example, the programs may be classified into a UI module1710, a touch screen module1720, and a notification module1730.

The UI module1710may provide a specialized UI or GUI that interoperates with the first device1000according to each application. The touch screen module1720may detect a user's touch gesture on a touch screen and transmit information about the touch gesture to the processor1300. The touch screen module1720according to some embodiments may recognize and analyze a touch code. The touch screen module1720may be implemented as separate hardware including a controller.

Various sensors may be provided inside or near the touch screen so as to detect a touch on the touch screen or hovering above the touch screen. An example of the sensor that detects the touch on the touch screen may be a tactile sensor. The tactile sensor may detect a contact of a specific object at or beyond a sensitivity of a person. The tactile sensor may sense various pieces of information, such as roughness of a contact surface, hardness of a contact object, or a temperature of a contact point.

Also, an example of the sensor that detects the touch on the touch screen may be a proximity sensor.

The proximity sensor refers to a sensor that detects the presence or absence of an object approaching a certain detection surface or an object existing near the sensor by using an electromagnetic force or infrared light, without mechanical contact. Examples of the proximity sensor include a transmission-type photoelectric sensor, a direct reflection-type photoelectric sensor, a mirror reflection-type photoelectric sensor, a high frequency oscillation-type proximity sensor, a capacity-type proximity sensor, a magnetic proximity sensor, and an infrared-type proximity sensor. The user's touch gesture may include a tap, a touch-and-hold, a double-tap, a drag, a panning, a flick, a drag-and-drop, a swipe, and the like.

The notification module1730may output a signal for notifying that an event occurs in the first device1000. Examples of events occurring in the first device1000include a call signal reception, a message reception, a key signal input, and a schedule notification. The notification module1730may output a notification signal through the display1210in the form of a video signal. The notification module1730may output a notification signal through the audio outputter1220in the form of an audio signal. The notification module1730may output a notification signal through the vibration motor1230in the form of a vibration signal.

FIG. 15is a block diagram of the processor1300according to some embodiments.

Referring toFIG. 15, the processor1300according to some embodiments may include a data trainer1310and a data recognizer1320.

According to an embodiment, at least part of the data trainer1310and at least part of the data recognizer1320may be implemented as a software module or may be manufactured in the form of a hardware chip and mounted on a text input device.

The data trainer1310may learn a criterion for determining a voice activation and deactivation conditions and generating recommended voice text information. The data trainer1310may learn a criterion regarding which data is to be used for determining the voice activation and deactivation conditions and generating the recommended voice text information. Also, the data trainer1310may learn a criterion regarding how to determine the intention of the user by using data, how to obtain and provide associated information, and how to determine the voice activation and deactivation conditions and generate the recommended voice text information. The data trainer1310may learn a criterion for determining the voice activation and deactivation conditions and generating the recommended voice text information by obtaining data to be used for learning and applying the obtained data to a data recognition model to be described below.

Types of training data input to the data trainer1310may include voice data, text data, and the like.

The data recognizer1320may determine the voice activation and deactivation conditions or generate the recommended voice text information, based on the data. The data recognizer1320may determine the voice activation and deactivation conditions or generate the recommended voice text information from certain data by using the learned data recognition model. The data recognizer1320may obtain certain data according to a preset criterion obtained by learning and use the data recognition model using the obtained data as an input value. Also, the data recognizer1320may use the data recognition model to determine the voice activation and deactivation conditions or generate the recommended voice text information, based on certain data. Also, a result value output by the data recognition model using the obtained data as the input value may be used to update the data recognition model.

At least one of the data trainer1310and the data recognizer1320may be manufactured in the form of at least one hardware chip and mounted on the first device1000. For example, at least one of the data trainer1310and the data recognizer1320may be manufactured in the form of a dedicated hardware chip for artificial intelligence (AI), or may be manufactured as part of an existing general-purpose processor (e.g., central processing unit (CPU) or application processor) or a dedicated graphics processor (e.g., graphics processing unit (GPU)) and mounted on the first device1000. In this case, the dedicated hardware chip for AI is a dedicated processor specialized for probability calculation. Because the dedicated hardware chip has higher parallel processing performance than that of the existing general-purpose processor, it is possible to quickly process computational operations in an AI field such as machine learning.

The data trainer1310and the data recognizer1320may be mounted on a single text input device, or may be respectively mounted on separate devices. For example, one of the data trainer1310and the data recognizer1320may be included in the single text input device, and the other thereof may be included in the server. Also, model information constructed by the data trainer1310may be provided to the data recognizer1320through wired or wireless communication, and data input to the data recognizer1320may be provided to the data trainer1310as additional training data.

At least one of the data trainer1310and the data recognizer1320may be implemented as at least one software module. When at least one of the data trainer1310and the data recognizer1320is implemented as a software module (or a program module including instructions), the software module may be stored in a non-transitory computer-readable medium. Also, in this case, at least one software module may be provided by an operating system (OS) or may be provided by a certain application. Alternatively, part of at least one software module may be provided by an OS, and the remaining part thereof may be provided by a certain application.

FIG. 16is a block diagram of the data trainer1310according to some embodiments.

Referring toFIG. 16, the data trainer1310according to some embodiments may include a data obtainer1310-1, a pre-processor1310-2, a training data selector1310-3, a model learner1310-4, and a model evaluator1310-5.

The data obtainer1310-1may obtain data necessary for determining a voice activation and deactivation conditions or generating recommended voice text information. The data obtainer1310-1may obtain, for example, a certain user voice and certain context information.

The pre-processor1310-2may pre-process the obtained data such that the obtained data is used for learning for determining the voice activation and deactivation conditions or generating the recommended voice text information. The pre-processor1310-2may process the obtained data into a preset format such that the model learner1310-4to be described below is enabled to use the obtained data for learning for determining the voice activation and deactivation conditions and generating the recommended voice text information.

The training data selector1310-3may select data necessary for learning from among pieces of pre-processed data. The selected data may be provided to the model learner1310-4. The training data selector1310-3may select data necessary for learning from among pieces of pre-processed data according to a preset criterion for determining the voice activation and deactivation conditions and generating the recommended voice text information. Also, the training data selector1310-3may select data according to a preset criterion by learning performed by the model learner1310-4to be described below.

The model learner1310-4may learn a criterion regarding whether to determine the voice activation and deactivation conditions or generate the recommended voice text information, based on training data. Also, the model learner1310-4may learn a criterion regarding which training data is to be used for determining the voice activation and deactivation conditions and generating the recommended voice text information.

Also, the model learner1310-4may learn a criterion regarding which training data is to be used for determining the voice activation and deactivation conditions and generating the recommended voice text information. In this case, the data recognition model may be a pre-constructed model. For example, the data recognition model may be a model that is pre-constructed by receiving basic training data (e.g., sample data, etc.).

The data recognition model may be constructed considering the application field of the recognition model, the purpose of learning, the computer performance of the device, or the like. The data recognition model may be designed to simulate a human brain structure on a computer. The data recognition model may include a plurality of network nodes having weights, which simulate neurons of a human neural network. The network nodes may form a connection relationship such that the neurons simulate the synaptic activity of the neurons that exchange signals through synapses. The data recognition model may include, for example, a neural network model, or a deep learning model developed from the neural network model. In the deep learning model, a plurality of network nodes may be located at different depths (or layers) and exchange data according to a convolutional connection relationship. For example, the data recognition model may include models such as a deep neural network (DNN), a recurrent neural network (RNN), or a bidirectional recurrent deep neural network (BRDNN), but the present disclosure is not limited thereto.

According to various embodiments, when there are a plurality of pre-constructed recognition models, the model learner1310-4may determine, as a data recognition model to be learned, a data recognition model having a large correlation between input training data and basic training data. In this case, the basic training data may be previously classified for each data type, and the data recognition model may be pre-constructed for each data type. For example, the basic training data may be previously classified based on various criteria such as a region in which the training data is generated, a time at which the training data is generated, a size of the training data, a genre of the training data, a creator of the training data, and a type of an object in the training data.

Also, the model learner1310-4may learn the data recognition model by using, for example, a learning algorithm including error back-propagation or gradient descent.

Also, the model learner1310-4may learn the data recognition model through, for example, supervised learning using the training data as the input value. Also, the model learner1310-4may self-learn the type of data necessary for determining the situation without separate supervision, so as to learn the data recognition model through unsupervised learning that finds a criterion for determining the situation. Also, the model learner1310-4may learn the data recognition model through, for example, reinforcement learning using a feedback on whether a result of determining a situation according to learning is correct.

Also, when the data recognition model is learned, the model learner1310-4may store the learned data recognition model. In this case, the model learner1310-4may store the learned data recognition model in the memory of the electronic device including the data recognizer1320. Alternatively, the model learner1310-4may store the learned data recognition model in the memory of the electronic device including the data recognizer1320, which is to be described below. Alternatively, the model learner1310-4may store the learned data recognition model in a memory of a server connected to the electronic device via a wired or wireless network.

In this case, the memory in which the learned data recognition model is stored may also store, for example, commands or data associated with at least one other element of the electronic device. Also, the memory may store software and/or programs. The programs may include, for example, kernel, middleware, application programming interface (API), and/or application programs (or “applications”).

When evaluation data is input to the data recognition model and a recognition result output from the evaluation data does not satisfy a certain criterion, the model evaluator1310-5may cause the model learner1310-4to learn again. In this case, the evaluation data may be data preset for evaluating the data recognition model.

For example, when the number or ratio of evaluation data, an analysis result of which is inaccurate among analysis results of the learned data recognition model for the evaluation data, exceeds a preset threshold value, the model evaluator1310-5may evaluate that the certain criterion is not satisfied. For example, in a case in which the certain criterion is defined as a ratio of 2%, when the learned data recognition model outputs an incorrect recognition result for more than 20 pieces of evaluation data among a total of 1,000 pieces of evaluation data, the model evaluator1310-5may evaluate that the learned data recognition model is unsuitable.

When there are a plurality of learned data recognition models, the model evaluator1310-5may evaluate whether each learned data recognition model satisfies the certain criterion and determine a model satisfying the certain criterion as a final data recognition model. In this case, when there are a plurality of models satisfying the certain criterion, the model evaluator1310-5may determine, as the final data recognition model, any one model or a certain number of models that are preset in the descending order of evaluation scores.

At least one of the data obtainer1310-1, the pre-processor1310-2, the training data selector1310-3, the model learner1310-4, and the model evaluator1310-5in the data trainer1310may be manufactured in the form of a hardware chip and mounted on the electronic device. For example, at least one of the data obtainer1310-1, the pre-processor1310-2, the training data selector1310-3, the model learner1310-4, and the model evaluator1310-5may be manufactured in the form of a dedicated hardware chip for AI, or may be manufactured as part of an existing general-purpose processor (e.g., CPU or application processor) or a dedicated graphics processor (e.g., GPU) and mounted on the above-described various electronic devices.

Also, the data obtainer1310-1, the pre-processor1310-2, the training data selector1310-3, the model learner1310-4, and the model evaluator1310-5may be mounted on the single electronic device or may be respectively mounted on separate electronic devices. For example, some of the data obtainer1310-1, the pre-processor1310-2, the training data selector1310-3, the model learner1310-4, and the model evaluator1310-5may be included in the electronic device, and the others thereof may be included in the server.

Also, at least one of the data obtainer1310-1, the pre-processor1310-2, the training data selector1310-3, the model learner1310-4, and the model evaluator1310-5may be implemented as a software module. When at least one of the data obtainer1310-1, the pre-processor1310-2, the training data selector1310-3, the model learner1310-4, and the model evaluator1310-5is implemented as a software module (or a program module including instructions), the software module may be stored in a non-transitory computer-readable medium. Also, in this case, at least one software module may be provided by an OS or may be provided by a certain application. Alternatively, part of at least one software module may be provided by an OS, and the remaining part thereof may be provided by a certain application.

FIG. 17is a block diagram of the data recognizer1320according to some embodiments.

Referring toFIG. 17, the data recognizer1320according to some embodiments may include a data obtainer1320-1, a pre-processor1320-2, a recognition data selector1320-3, a recognition result provider1320-4, and a model updater1320-5.

The data obtainer1320-1may obtain data necessary for situation determination, and the pre-processor1320-2may pre-process the obtained data so as to enable the data obtained for the situation determination to be used. The pre-processor1320-2may process the obtained data into a preset format such that the recognition result provider1320-4uses the obtained data for situation determination.

The recognition data selector1320-3may select data necessary for the situation determination from among pieces of the pre-processed data. The selected data may be provided to the recognition result provider1320-4. The recognition data selector1320-3may select all or part of the pre-processed data according to a preset criterion for the situation determination. Also, the recognition data selector1320-3may select data according to a preset criterion by learning performed by the model learner1310-4to be described below.

The recognition result provider1320-4may determine the situation by applying the selected data to the data recognition model. The recognition result provider1320-4may provide a recognition result according to a data recognition purpose. The recognition result provider1320-4may apply the selected data to the data recognition model by using the data selected by the recognition data selector1320-3as an input value. Also, the analysis result may be determined by the data recognition model.

The model updater1320-5may update the data recognition model based on the evaluation of the recognition result provided by the recognition result provider1320-4. For example, the model updater1320-5may provide, to the model learner1310-4, the recognition result provided by the recognition result provider1320-4, such that the model learner1310-4updates the data recognition model.

At least one of the data obtainer1320-1, the pre-processor1320-2, the recognition data selector1320-3, the recognition result provider1320-4, and the model updater1320-5in the data recognizer1320may be manufactured in the form of a hardware chip and mounted on the electronic device. For example, at least one of the data obtainer1320-1, the pre-processor1320-2, the recognition data selector1320-3, the recognition result provider1320-4, and the model updater1320-5may be manufactured in the form of a dedicated hardware chip for AI, or may be manufactured as part of an existing general-purpose processor (e.g., CPU or application processor) or a dedicated graphics processor (e.g., GPU) and mounted on the above-described various electronic devices.

Also, the data obtainer1320-1, the pre-processor1320-2, the recognition data selector1320-3, the recognition result provider1320-4, and the model updater1320-5may be mounted on the single electronic device or may be respectively mounted on separate electronic devices. For example, some of the data obtainer1320-1, the pre-processor1320-2, the recognition data selector1320-3, the recognition result provider1320-4, and the model updater1320-5may be included in the electronic device, and the others thereof may be included in the server.

Also, at least one of the data obtainer1320-1, the pre-processor1320-2, the recognition data selector1320-3, the recognition result provider1320-4, and the model updater1320-5may be implemented as a software module. When at least one of the data obtainer1320-1, the pre-processor1320-2, the recognition data selector1320-3, the recognition result provider1320-4, and the model updater1320-5is implemented as a software module (or a program module including instructions), the software module may be stored in a non-transitory computer-readable medium. Also, in this case, at least one software module may be provided by an OS or may be provided by a certain application. Alternatively, part of at least one software module may be provided by an OS, and the remaining part thereof may be provided by a certain application.

FIG. 18is a diagram illustrating an example in which a text input device1000and a server2000interoperate with each other to train and recognize data, according to some embodiments.

Referring toFIG. 18, the server2000may learn a criterion for determining voice activation and deactivation conditions and generating recommended voice text information, and the text input device1000may determine the determination of the voice activation and deactivation conditions and the generation of the recommended voice text information, based on a result of learning performed by the server2000.

In this case, a model learner2340of the server2000may perform the function of the data trainer1310illustrated inFIG. 16. A model learner2340of the server2000may learn a criterion regarding which data is to be used for determining the voice activation and deactivation conditions and generating the recommended voice text information. Also, the model learner2340of the server2000may learn a criterion for determining the voice activation and deactivation conditions and generating the recommended voice text information by using data. The model learner2340may learn a criterion for determining the voice activation and deactivation conditions and generating the recommended voice text information by obtaining data to be used for learning and applying the obtained data to a data recognition model to be described below.

Also, the recognition result provider1320-4of the text input device1000may determine voice activation and deactivation conditions and generate recommended voice text information by applying the data selected by the recognition data selector1320-3to a data recognition model generated by the server2000. For example, the recognition result provider1320-4may transmit the data selected by the recognition data selector1320-3to the server2000, and the server2000may make a request for determining the voice activation and deactivation conditions and generating the recommended voice text information by applying the data selected by the recognition data selector1320-3to the data recognition model. Also, the recognition result provider1320-4may receive, from the server2000, information about the determination of the voice activation and deactivation conditions and the generation of the recommended voice text information, which are determined by the server2000.

Alternatively, the recognition result provider1320-4of the text input device1000may receive, from the server2000, the data recognition model generated by the server2000and may use the received data recognition model to determine the voice activation and deactivation conditions and generate the recommended voice text information. In this case, the recognition result provider1320-4of the text input device1000may determine the voice activation and deactivation conditions and generate the recommended voice text information by applying the data selected by the recognition data selector1320-3to the data recognition model received from the server2000.

Also, the text input device1000and the server2000may effectively distribute and perform operations for the learning of the data recognition model and the data recognition. In this manner, data processing may be efficiently performed so as to provide a service matching a user's intention and a user's privacy may be effectively protected.

Some embodiments of the present disclosure may be implemented as a software program that includes instructions stored in a computer-readable storage medium.

For example, a computer is a device capable of calling the stored instructions from the storage medium and performing the operations of the embodiments according to the called instructions. The computer may include the device according to the embodiments or an external server communicatively connected to the device.

The computer-readable storage medium may be provided in the form of a non-transitory storage medium. The term “non-transitory” means that the storage media does not include a signal or current and is tangible, and does not distinguish between semi-permanent and temporary storage of data in the storage medium. For example, the non-transitory storage medium may include not only a non-transitory readable storage medium such as compact disc (CD), digital versatile disc (DVD), hard disk, Blu-ray disk, universal serial bus (USB), internal memory, memory card, ROM, or RAM, but also a temporary storage medium such as register, cache, or buffer.

Also, the methods according to the embodiments may be provided as a computer program product.

The computer program product may include a software program, a computer-readable storage medium on which the software program is stored, or a product traded between a seller and a buyer.

For example, the computer program product may include a product (e.g., downloadable app) of a software program form that is distributed electronically through a device manufacturer or an electronic market (e.g., Google Play Store, App Store, etc.). For electronic distribution, at least part of the software program may be stored in a storage medium or may be temporarily generated. In this case, the storage medium may be a server of a manufacturer or an electronic market, or a storage medium of a relay server.