ELECTRONIC DEVICE AND OPERATION METHOD THEREOF

An electronic device is provided. The electronic device includes a processor and a memory operatively connected to the processor. The memory may store instructions that, when executed, cause the processor to extract at least one or more utterance records of a user by using a user account included in the electronic device or operatively connected to the electronic device, to analyze the extracted at least one or more utterance records, to generate an utterance set including at least one or more operations based on the analyzed utterance records, to generate at least one or more quick command names corresponding to the utterance set, and to provide response data including the at least one or more quick command names.

BACKGROUND

The disclosure relates to an electronic device and an operating method thereof.

2. Description of Related Art

Nowadays, with the development of artificial intelligence (AI) technology, terminals including AI used for the main purpose of assistant are spread. Furthermore, in addition to a conventional input method using a keyboard or a mouse, electronic devices have recently supported various input methods such as a voice input. For example, the electronic devices such as smart phones or tablet personal computers (PCs) may receive a user voice and then may provide a service that performs an operation corresponding to the received user voice.

The speech recognition service is being developed based on a technology for processing a natural language. The technology for processing a natural language refers to a technology that grasps the intent of a user input (utterance) and generates the result matched with the intent to provide the user with the service.

In the meantime, the voice recognition service provides a quick command function that enables the electronic device to perform various functions based on a specific input of a user. Users naturally give various commands to the AI included in the terminal, or have a conversation with the AI.

Accordingly, there is an increasing need to recommend a quick command and/or a quick command name optimized for users.

SUMMARY

Aspects of the disclosure are to address at least the above-mentioned problems and/or disadvantages and to provide at least the advantages described below. Accordingly, an aspect of the disclosure is to provide a method and device for providing a personalized quick command and/or quick command name.

Another aspect of the disclosure is to provide a method and device for providing a quick command and/or quick command name generated based on user input.

Another aspect of the disclosure is to provide a method and device for providing a quick command and/or quick command name generated based on a usage pattern of a user.

Another aspect of the disclosure is to provide a method and device for providing the user with a consistent user experience by providing a quick command and/or quick command name based on the quick command, even when the user directly adds the quick command.

In accordance with an aspect of the disclosure, an electronic device is provided. The electronic device includes a processor and a memory operatively connected to the processor. The memory may store instructions that, when executed, cause the processor to extract at least one or more utterance records of a user by using a user account included in the electronic device or operatively connected to the electronic device, to analyze the extracted at least one or more utterance records, to generate an utterance set including at least one or more operations based on the analyzed utterance records, to generate at least one or more quick command names corresponding to the utterance set, and to provide response data including the at least one or more quick command names.

In accordance with another aspect of the disclosure, a method performed by an electronic device is provided. The method includes extracting at least one or more utterance records of a user by using a user account included in the electronic device or operatively connected to the electronic device when a process for a memory included in the electronic device or operatively connected to the electronic device is executed, analyzing the extracted at least one or more utterance records, generating an utterance set including at least one or more operations based on the analyzed utterance records, generating at least one or more quick command names corresponding to the utterance set, and providing response data including the at least one or more quick command names.

According to embodiments disclosed in this specification, it is possible to provide a method and device for providing a quick command and/or quick command name generated based on a user's usage pattern.

According to embodiments disclosed in this specification, it is possible to provide a method and device for providing a quick command and/or quick command name generated based on a user input.

According to embodiments disclosed in this specification, it is possible to provide a method and device for providing a quick command and/or quick command name based on the quick command, even when a user directly adds the quick command.

Besides, a variety of effects directly or indirectly understood through the specification may be provided.

DETAILED DESCRIPTION

FIG. 1is a block diagram illustrating an electronic device in a network environment according to an embodiment of the disclosure.

FIG. 2is a block diagram200illustrating the program140according to an embodiment of the disclosure.

Referring toFIG. 2, the program140may include an operating system (OS)142to control one or more resources of the electronic device101, middleware144, or an application146executable in the OS142. The OS142may include, for example, Android™, iOS™, Windows™, Symbian™, Tizen™, or Bada™. At least part of the program140, for example, may be pre-loaded on the electronic device101during manufacture, or may be downloaded from or updated by an external electronic device (e.g., the electronic device102or104, or the server108) during use by a user.

The OS142may control management (e.g., allocating or deallocation) of one or more system resources (e.g., process, memory, or power source) of the electronic device101. The OS142, additionally or alternatively, may include one or more driver programs to drive other hardware devices of the electronic device101, for example, the input module150, the sound output module155, the display module160, the audio module170, the sensor module176, the interface177, the haptic module179, the camera module180, the power management module188, the battery189, the communication module190, the subscriber identification module196, or the antenna module197.

The middleware144may provide various functions to the application146such that a function or information provided from one or more resources of the electronic device101may be used by the application146. The middleware144may include, for example, an application manager201, a window manager203, a multimedia manager205, a resource manager207, a power manager209, a database manager211, a package manager213, a connectivity manager215, a notification manager217, a location manager219, a graphic manager221, a security manager223, a telephony manager225, or a voice recognition manager227.

The application manager201, for example, may manage the life cycle of the application146. The window manager203, for example, may manage one or more graphical user interface (GUI) resources that are used on a screen. The multimedia manager205, for example, may identify one or more formats to be used to play media files, and may encode or decode a corresponding one of the media files using a codec appropriate for a corresponding format selected from the one or more formats. The resource manager207, for example, may manage the source code of the application146or a memory space of the memory130. The power manager209, for example, may manage the capacity, temperature, or power of the battery189, and determine or provide related information to be used for the operation of the electronic device101based at least in part on corresponding information of the capacity, temperature, or power of the battery189. According to an embodiment, the power manager209may interwork with a basic input/output system (BIOS) (not shown) of the electronic device101.

The database manager211, for example, may generate, search, or change a database to be used by the application146. The package manager213, for example, may manage installation or update of an application that is distributed in the form of a package file. The connectivity manager215, for example, may manage a wireless connection or a direct connection between the electronic device101and the external electronic device. The notification manager217, for example, may provide a function to notify a user of an occurrence of a specified event (e.g., an incoming call, message, or alert). The location manager219, for example, may manage locational information on the electronic device101. The graphic manager221, for example, may manage one or more graphic effects to be offered to a user or a user interface related to the one or more graphic effects.

The security manager223, for example, may provide system security or user authentication. The telephony manager225, for example, may manage a voice call function or a video call function provided by the electronic device101. The voice recognition manager227, for example, may transmit a user's voice data to the server108, and receive, from the server108, a command corresponding to a function to be executed on the electronic device101based at least in part on the voice data, or text data converted based at least in part on the voice data. According to an embodiment, the middleware144may dynamically delete some existing components or add new components. According to an embodiment, at least part of the middleware144may be included as part of the OS142or may be implemented as another software separate from the OS142.

The application146may include, for example, a home251, dialer253, short message service (SMS)/multimedia messaging service (MIMS)255, instant message (IM)257, browser259, camera261, alarm263, contact265, voice recognition267, email269, calendar271, media player273, album275, watch277, health279(e.g., for measuring the degree of workout or biometric information, such as blood sugar), or environmental information281(e.g., for measuring air pressure, humidity, or temperature information) application. According to an embodiment, the application146may further include an information exchanging application (not shown) that is capable of supporting information exchange between the electronic device101and the external electronic device. The information exchange application, for example, may include a notification relay application adapted to transfer designated information (e.g., a call, message, or alert) to the external electronic device or a device management application adapted to manage the external electronic device. The notification relay application may transfer notification information corresponding to an occurrence of a specified event (e.g., receipt of an email) at another application (e.g., the email application269) of the electronic device101to the external electronic device. Additionally or alternatively, the notification relay application may receive notification information from the external electronic device and provide the notification information to a user of the electronic device101.

The device management application may control the power (e.g., turn-on or turn-off) or the function (e.g., adjustment of brightness, resolution, or focus) of the external electronic device or some component thereof (e.g., a display device or a camera module of the external electronic device). The device management application, additionally or alternatively, may support installation, delete, or update of an application running on the external electronic device.

FIG. 3is a block diagram illustrating an integrated intelligence system, according to an embodiment of the disclosure.

Referring toFIG. 3, an integrated intelligence system according to an embodiment may include a user terminal301, an intelligence server400, and a service server500.

The user terminal301according to an embodiment may be a terminal device (or an electronic device) capable of connecting to Internet, and may be, for example, a mobile phone, a smartphone, a personal digital assistant (PDA), a notebook computer, a television (TV), a white household appliance, a wearable device, a head mounted display (HID), or a smart speaker.

According to the illustrated embodiment, the user terminal301may include a communication interface390, a microphone370, a speaker355, a display360, a memory330, or a processor320. The listed components may be operatively or electrically connected to one another.

The communication interface390according to an embodiment may be connected to an external device and may be configured to transmit or receive data to or from the external device. The microphone370according to an embodiment may receive a sound (e.g., a user utterance) to convert the sound into an electrical signal. The speaker355according to an embodiment may output the electrical signal as sound (e.g., voice). The display360according to an embodiment may be configured to display an image or a video. The display360according to an embodiment may display the graphical user interface (GUI) of the running app (or an application program).

The memory330according to an embodiment may store a client module331, a software development kit (SDK)333, and a plurality of apps335. The client module331and the SDK333may constitute a framework (or a solution program) for performing general-purposed functions. Furthermore, the client module331or the SDK333may constitute the framework for processing a voice input.

The plurality of apps335may be programs for performing a specified function. According to an embodiment, the plurality of apps335may include a first app335aand/or a second app335b. According to an embodiment, each of the plurality of apps335may include a plurality of actions for performing a specified function. For example, the apps may include an alarm app, a message app, and/or a schedule app. According to an embodiment, the plurality of apps335may be executed by the processor320to sequentially execute at least part of the plurality of actions.

According to an embodiment, the processor320may control overall operations of the user terminal301. For example, the processor320may be electrically connected to the communication interface390, the microphone370, the speaker355, and the display360so as to perform a specified operation. For example, the processor320may include at least one processor.

Moreover, the processor320according to an embodiment may execute the program stored in the memory330so as to perform a specified function. For example, according to an embodiment, the processor320may execute at least one of the client module331or the SDK333so as to perform a following operation for processing a voice input. The processor320may control operations of the plurality of apps335via the SDK333. The following actions described as the actions of the client module331or the SDK333may be the actions performed by the execution of the processor320.

According to an embodiment, the client module331may receive a voice input. For example, the client module331may receive a voice signal corresponding to a user utterance detected through the microphone370. The client module331may transmit the received voice input (e.g., a voice signal) to the intelligence server400. The client module331may transmit state information of the user terminal301to the intelligence server400together with the received voice input. For example, the state information may be execution state information of an app.

According to an embodiment, the client module331may receive a result corresponding to the received voice input. For example, when the intelligence server400is capable of calculating the result corresponding to the received voice input, the client module331may receive the result corresponding to the received voice input. The client module331may display the received result on the display360.

According to an embodiment, the client module331may receive a plan corresponding to the received voice input. The client module331may display, on the display360, a result of executing a plurality of actions of an app depending on the plan. For example, the client module331may sequentially display the result of executing the plurality of actions on a display. For another example, the user terminal301may display only a part of results (e.g., a result of the last action) of executing the plurality of actions, on the display.

According to an embodiment, the client module331may receive a request for obtaining information necessary to calculate the result corresponding to a voice input, from the intelligence server400. According to an embodiment, the client module331may transmit the necessary information to the intelligence server400in response to the request.

According to an embodiment, the client module331may transmit, to the intelligence server400, information about the result of executing a plurality of actions depending on the plan. The intelligence server400may identify that the received voice input is correctly processed, using the result information.

According to an embodiment, the client module331may include a speech recognition module. According to an embodiment, the client module331may recognize a voice input for performing a limited function, via the speech recognition module. For example, the client module331may launch an intelligence app for processing a specific voice input by performing an organic action, in response to a specified voice input (e.g., wake up!).

According to an embodiment, the intelligence server400may receive information associated with a user's voice input from the user terminal301over a communication network. According to an embodiment, the intelligence server400may convert data associated with the received voice input to text data. According to an embodiment, the intelligence server400may generate at least one plan for performing a task corresponding to the user's voice input, based on the text data.

According to an embodiment, the plan may be generated by an artificial intelligent (AI) system. The AI system may be a rule-based system, or may be a neural network-based system (e.g., a feedforward neural network (FNN) and/or a recurrent neural network (RNN)). Alternatively, the AI system may be a combination of the above-described systems or an AI system different from the above-described system. According to an embodiment, the plan may be selected from a set of predefined plans or may be generated in real time in response to a user's request. For example, the AI system may select at least one plan of the plurality of predefined plans.

According to an embodiment, the intelligence server400may transmit a result according to the generated plan to the user terminal301or may transmit the generated plan to the user terminal301. According to an embodiment, the user terminal301may display the result according to the plan, on a display. According to an embodiment, the user terminal301may display a result of executing the action according to the plan, on the display.

The intelligence server400according to an embodiment may include a front end410, a natural language platform420, a capsule database430, an execution engine440, an end user interface450, a management platform460, a big data platform470, or an analytic platform480.

According to an embodiment, the front end410may receive a voice input received from the user terminal300. The front end410may transmit a response corresponding to the voice input to the user terminal301.

According to an embodiment, the natural language platform420may include an automatic speech recognition (ASR) module421, a natural language understanding (NLU) module423, a planner module425, a natural language generator (NLG) module427, and/or a text to speech module (TTS) module429.

According to an embodiment, the ASR module421may convert the voice input received from the user terminal301into text data. According to an embodiment, the NLU module423may grasp the intent of the user, using the text data of the voice input. For example, the NLU module423may grasp the intent of the user by performing syntactic analysis or semantic analysis. According to an embodiment, the NLU module423may grasp the meaning of words extracted from the voice input by using linguistic features (e.g., syntactic elements) such as morphemes or phrases and may determine the intent of the user by matching the grasped meaning of the words to the intent.

According to an embodiment, the planner module425may generate the plan by using a parameter and the intent that is determined by the NLU module423. According to an embodiment, the planner module425may determine a plurality of domains necessary to perform a task, based on the determined intent. The planner module425may determine a plurality of actions included in each of the plurality of domains determined based on the intent. According to an embodiment, the planner module425may determine the parameter necessary to perform the determined plurality of actions or a result value output by the execution of the plurality of actions. The parameter and the result value may be defined as a concept of a specified form (or class). As such, the plan may include the plurality of actions and/or a plurality of concepts, which are determined by the intent of the user. The planner module425may determine the relationship between the plurality of actions and the plurality of concepts stepwise (or hierarchically). For example, the planner module425may determine the execution sequence of the plurality of actions, which are determined based on the user's intent, based on the plurality of concepts. In other words, the planner module425may determine an execution sequence of the plurality of actions, based on the parameters necessary to perform the plurality of actions and the result output by the execution of the plurality of actions. Accordingly, the planner module425may generate a plan including information (e.g., ontology) about the relationship between the plurality of actions and the plurality of concepts. The planner module425may generate the plan, using information stored in the capsule database (DB)430storing a set of relationships between concepts and actions.

According to an embodiment, the NLG module427may change specified information into information in a text form. The information changed to the text form may be in the form of a natural language speech. The TTS module429according to an embodiment may change information in the text form to information in a voice form.

According to an embodiment, all or part of the functions of the natural language platform420may be also implemented in the user terminal301.

The capsule DB430may store information about the relationship between the actions and the plurality of concepts corresponding to a plurality of domains. According to an embodiment, the capsule may include a plurality of action objects (or action information) and concept objects (or concept information) included in the plan. According to an embodiment, the capsule DB430may store the plurality of capsules in a form of a concept action network (CAN). According to an embodiment, the plurality of capsules may be stored in the function registry included in the capsule DB430.

The capsule DB430may include a strategy registry that stores strategy information necessary to determine a plan corresponding to a voice input. When there are a plurality of plans corresponding to the voice input, the strategy information may include reference information for determining one plan. According to an embodiment, the capsule DB430may include a follow-up registry that stores information of the follow-up action for suggesting a follow-up action to the user in a specified context. For example, the follow-up action may include a follow-up utterance. According to an embodiment, the capsule DB430may include a layout registry storing layout information of information output via the user terminal301. According to an embodiment, the capsule DB430may include a vocabulary registry storing vocabulary information included in capsule information. According to an embodiment, the capsule DB430may include a dialog registry storing information about dialog (or interaction) with the user. The capsule DB430may update an object stored via a developer tool. For example, the developer tool may include a function editor for updating an action object or a concept object. The developer tool may include a vocabulary editor for updating a vocabulary. The developer tool may include a strategy editor that generates and registers a strategy for determining the plan. The developer tool may include a dialog editor that creates a dialog with the user. The developer tool may include a follow-up editor capable of activating a follow-up target and editing the follow-up utterance for providing a hint. The follow-up target may be determined based on a target, the user's preference, or an environment condition, which is currently set. The capsule DB430according to an embodiment may be also implemented in the user terminal301.

According to an embodiment, the execution engine440may calculate a result by using the generated plan. The end user interface450may transmit the calculated result to the user terminal301. Accordingly, the user terminal301may receive the result and may provide the user with the received result. According to an embodiment, the management platform460may manage information used by the intelligence server400. According to an embodiment, the big data platform470may collect data of the user. According to an embodiment, the analytic platform480may manage quality of service (QoS) of the intelligence server400. For example, the analytic platform480may manage the component and processing speed (or efficiency) of the intelligence server400.

According to an embodiment, the service server500may provide the user terminal301with a specified service (e.g., ordering food or booking a hotel). According to an embodiment, the service server500may be a server operated by the third party. According to an embodiment, the service server500may provide the intelligence server400with information for generating a plan corresponding to the received voice input. The provided information may be stored in the capsule DB430. Furthermore, the service server500may provide the intelligence server400with result information according to the plan. The service server500may include a plurality of service servers501,502,503, . . . .

In the above-described integrated intelligence system, the user terminal301may provide the user with various intelligent services in response to a user input. The user input may include, for example, an input through a physical button, a touch input, or a voice input.

According to an embodiment, the user terminal301may provide a speech recognition service via an intelligence app (or a speech recognition app) stored therein. In this case, for example, the user terminal301may recognize a user utterance or a voice input, which is received via the microphone, and may provide the user with a service corresponding to the recognized voice input.

According to an embodiment, the user terminal301may perform a specified action, based on the received voice input, independently, or together with the intelligence server and/or the service server. For example, the user terminal301may launch an app corresponding to the received voice input and may perform the specified action via the executed app.

According to an embodiment, when providing a service together with the intelligence server400and/or the service server500, the user terminal301may detect a user utterance by using the microphone370and may generate a signal (or voice data) corresponding to the detected user utterance. The user terminal may transmit the voice data to the intelligence server400by using the communication interface390.

According to an embodiment, the intelligence server400may generate a plan for performing a task corresponding to the voice input or the result of performing an action depending on the plan, as a response to the voice input received from the user terminal301. For example, the plan may include a plurality of actions for performing the task corresponding to the voice input of the user and/or a plurality of concepts associated with the plurality of actions. The concept may define a parameter to be input upon executing the plurality of actions or a result value output by the execution of the plurality of actions. The plan may include relationship information between the plurality of actions and/or the plurality of concepts.

According to an embodiment, the user terminal301may receive the response by using the communication interface390. The user terminal301may output the voice signal generated in the user terminal301to the outside by using the speaker355or may output an image generated in the user terminal301to the outside by using the display360.

FIG. 4is a diagram illustrating a form in which relationship information between a concept and an action is stored in a database, according to an embodiment of the disclosure.

A capsule database (e.g., the capsule DB430) of the intelligence server400may store a capsule in the form of a CAN. The capsule DB may store an action for processing a task corresponding to a user's voice input and a parameter necessary for the action, in a form of CAN.

Referring toFIG. 4, the capsule DB may store a plurality capsules (a capsule A431and a capsule B434) respectively corresponding to a plurality of domains (e.g., applications). According to an embodiment, a single capsule (e.g., the capsule A431) may correspond to a single domain (e.g., a location (geo) or an application). Furthermore, at least one service provider (e.g., CP1432, CP2433, CP3435or CP4436) for performing a function for a domain associated with the capsule may correspond to one capsule. According to an embodiment, the single capsule may include at least one or more actions430aand at least one or more concepts430bfor performing a specified function.

The natural language platform420may generate a plan for performing a task corresponding to the received voice input, using the capsule stored in a capsule database. For example, the planner module425of the natural language platform may generate the plan by using the capsule stored in the capsule database. For example, a plan407may be generated by using actions431aand432aand concepts431band432bof the capsule A431and an action434aand a concept434bof the capsule B434.

FIG. 5is a view illustrating a screen in which a user terminal processes a voice input received through an intelligence app, according to an embodiment of the disclosure.

The user terminal301may execute an intelligence app to process a user input through the intelligence server400.

Referring toFIG. 5, on screen310, when recognizing a specified voice input (e.g., wake up!) or receiving an input via a hardware key (e.g., a dedicated hardware key), the user terminal301may launch an intelligence app for processing a voice input. For example, the user terminal301may launch the intelligence app in a state where a schedule app is executed. According to an embodiment, the user terminal301may display an object (e.g., an icon)311corresponding to the intelligence app, on the display360. According to an embodiment, the user terminal301may receive a voice input by a user utterance. For example, the user terminal301may receive a voice input saying that “let me know the schedule of this week!” According to an embodiment, the user terminal301may display a user interface (UI)313(e.g., an input window) of the intelligence app, in which text data of the received voice input is displayed, on a display.

According to an embodiment, on screen315, the user terminal301may display a result corresponding to the received voice input, on the display. For example, the user terminal301may receive a plan corresponding to the received user input and may display ‘the schedule of this week’ on the display depending on the plan.

In an embodiment, the user terminal301ofFIGS. 3, 4, and 5may correspond to the electronic device101ofFIG. 1. In an embodiment, the intelligence server400ofFIG. 3may correspond to one of the electronic device104and the server108ofFIG. 1. In an embodiment, the processor320ofFIG. 3may correspond to the processor120ofFIG. 1; the display360ofFIG. 3may correspond to the display device (i.e., display module160ofFIG. 1); the speaker355ofFIG. 3may correspond to the sound output device (i.e., output module155ofFIG. 1).

FIG. 6is a block diagram illustrating a structure of an electronic device600, according to an embodiment of the disclosure. For clarity of description, details the same as the above-described details may be briefly described or omitted.

Referring toFIG. 6, the electronic device600may include a processor601(e.g., the processor320ofFIG. 3and/or the processor120ofFIG. 1), a memory602(e.g., the memory130ofFIG. 1), a user interface603, and a communication module604(e.g., the communication module190ofFIG. 1). The user interface603may include a microphone (not illustrated) (e.g., the microphone370ofFIG. 3and/or the input module150ofFIG. 1) and a speaker (not illustrated) (e.g., the speaker355ofFIG. 3and/or the sound output device (i.e., output module155) ofFIG. 1).

The electronic device600may further include at least one of additional components in addition to the components illustrated inFIG. 6. According to an embodiment, the components of the electronic device600may be the same entities or may constitute separate entities.

For example, the electronic device600may include a smartphone, a tablet PC, a wearable device, a home appliance, or a digital camera. According to an embodiment, the processor601may be operatively coupled to the communication module604, the memory602, and the user interface603(a microphone (not illustrated) and a speaker (not illustrated)) to perform overall functions of the electronic device600. For example, the processor601may include one or more processors. For example, the one or more processors may include an image signal processor (ISP), an application processor (AP), or a communication processor (CP).

Furthermore, the processor601may drive a module (e.g., a quick command recommender module710, a sequence DB creator module720, a pattern finder module730, a name recommender module750, an ASR module760, and/or an NLU module770inFIG. 7) by executing instructions stored in the memory602.

The processor601may be operatively connected to the module (e.g., the quick command recommender module710, the sequence DB creator module720, the pattern finder module730, the name recommender module750, the ASR module760, and/or the NLU module770inFIG. 7) to perform overall functions of the electronic device600. In the embodiment disclosed in this specification, it may be understood that an operation performed (or executed) by the module (e.g., the quick command recommender module710, the sequence DB creator module720, the pattern finder module730, the name recommender module750, the ASR module760, and/or the NLU module770inFIG. 7) is an operation performed by the processor601executing instructions stored in the memory602.

In an embodiment, the processor601may include the module (e.g., the quick command recommender module710, the sequence DB creator module720, the pattern finder module730, the name recommender module750, the ASR module760, and/or the NLU module770inFIG. 7). In this case, an operation performed (or executed) by each module (e.g., the quick command recommender module710, the sequence DB creator module720, the pattern finder module730, the name recommender module750, the ASR module760, and/or the NLU module770inFIG. 7) may be implemented as at least part of the processor601.

Several modules described in various embodiments of the disclosure may be implemented by hardware or software.

The memory602may store a database (not illustrated) (e.g., the database740ofFIG. 7) including at least one input data. The memory602may store commands, information, or data associated with operations of components included in the electronic device600. For example, the memory602may store instructions, when executed, that cause the processor601to perform various operations described in the disclosure.

In an embodiment, the electronic device600may receive a user input by using the user interface603. The user input may be an input including a user voice signal (e.g., a user's utterance input).

In an embodiment, the user input may be the user's voice input (e.g., an utterance). When the user input is a voice input, the electronic device600may receive a user input through a microphone (or a voice receiving device) (not illustrated).

In an embodiment, the user input may be a gesture input and/or a touch input. When the user input is a gesture input and/or a touch input, the electronic device600may receive a user input through a sensor (not illustrated).

According to an embodiment, the processor601may include a sound module (not illustrated). The sound module may recognize a user input for executing an operation. For example, the sound module may recognize and receive the voice signal. For example, the sound module recognizing the user input may have a high speech recognition rate because ambient noise is strong.

According to an embodiment, the sound module may be learned to recognize and receive the user input by using an algorithm for recognizing a voice. For example, the algorithm for recognizing the voice may be at least one of a hidden Markov model (HMM) algorithm, an artificial neural network (ANN) algorithm, and a dynamic time warping (DTW) algorithm.

According to an embodiment, the sound module may perform tasks of data refinement, data integration, data reduction, and/or data conversion. The data refinement may include an operation of filling in incomplete data and correcting inconsistent data. The data integration may include an operation of merging various divided databases and files for easy analysis. The data reduction may include an operation of sampling only some of input data or reducing the dimension of data to be analyzed. The data conversion may include an operation of normalizing or grouping data by obtaining an average value of the data. The sound module may process data, thereby preventing meaningless values from being included in data or preventing data quality from being degraded due to unintended variables. Accuracy and timeliness may be increased through the sound module.

In an embodiment, at least one operation among operations of each component described with reference to the electronic device600may be performed (or executed) by an external server (not illustrated) or another electronic device (not illustrated). For example, the processor601may transmit a user input to the external server (not illustrated) or the other electronic device (not illustrated) by using the communication module604.

A processor (not illustrated) included in an external server (not illustrated) or the other electronic device (not illustrated) may receive the user input, may generate response data, and may transmit the response data to the electronic device600.

The processor601may receive the response data corresponding to the user input from the external server (not illustrated) or the other electronic device (not illustrated) through the communication module604. When receiving the response data, the processor601may allow the output device (i.e., user interface603) to output the response data. Alternatively, other devices may be controlled through the communication circuit (i.e., communication module604); alternatively, data may be stored through the communication circuit (i.e., communication module604). The processor601may be composed of at least one or more processors, and may be driven while being physically divided into a main processor performing high-performance processing and an auxiliary processor performing low-power processing. Alternatively, one processor may process data while switching between high performance and low power depending on situations.

Hereinafter, an operation of the processor601will be described in detail.

In an embodiment, the processor601may extract at least one or more utterance records of a user by using a user account included in the electronic device or operatively connected to the electronic device, may analyze the extracted at least one or more utterance records, may generate an utterance set including at least one or more operations based on the analyzed utterance records, may generate at least one or more quick command names corresponding to the utterance set, and may provide response data including the at least one or more quick command names.

In an embodiment, the processor601may receive a voice signal included in a user input by using the sound model operatively connected to the processor and may cause the sound model to be learned by using a learning algorithm.

In an embodiment, the processor601may separate at least one or more utterances, which are included in the utterance records, into at least one or more sequences based on at least one of information about a time of an utterance, which is included in the extracted at least one or more utterance records, or information about a location of the utterance.

In an embodiment, the processor601may separate at least one or more utterances, which are included in the utterance records, into at least one or more sequences based on information about at least one of a goal, a capsule, or a signal of an utterance included in the extracted at least one or more utterance records.

In an embodiment, the processor601may compare utterance reception times of a plurality of utterances included in the extracted at least one or more utterance records and may include the plurality of utterances in an identical sequence when a difference between the utterance reception times is not greater than a specified value.

In an embodiment, the processor601may compare the utterance reception times by using the duration information when an utterance including duration information is included in the extracted at least one or more utterance records, and may include the plurality of utterances in the identical sequence when the difference between the utterance reception times, which is obtained by comparing the utterance reception times by using the duration information, is not greater than the specified value.

In an embodiment, the processor601may model a relational model between the utterance set and the quick command names and may learn generation or recommendation of the quick command names by using the modeled relational model.

In an embodiment, the processor601may perform learning by receiving an utterance included in the utterance set or a natural language (NL) result, which is obtained by analyzing the utterance, as inputs and outputting a quick command name for utterances included in the utterance set as a result by using the relational model.

In an embodiment, the processor601may find an important keyword included in the utterance set and may generate the quick command names for the utterance set by using the important keyword.

In an embodiment, the processor601may embed a word, a phrase, and an entire utterance included in the utterance set and may generate the quick command names for the utterance set by using at least one of a word and a phrase, which have the highest similarity.

FIG. 7is another block diagram illustrating a structure of an electronic device700, according to an embodiment of the disclosure.

For clarity of description, details the same as the above-described details may be briefly described or omitted.

Referring toFIG. 7, the electronic device700may include the quick command recommender module710, the sequence DB creator module720, the pattern finder module730, a database740, the name recommender module750, the ASR module760, and/or the NLU module770. The listed components may be operatively or electrically connected to one another.

The quick command recommender module710may analyze the user's utterance record, and may find a repeated sequential utterance set in the user's utterance record. The utterance record of the user may mean a record uttered by the user of the electronic device700by using a speech recognition system of the electronic device700. For example, the utterance record of the user may mean a record of an utterance uttered by the user by using a voice assistant.

In an embodiment, the quick command recommender module710may generally manage a function of recording an utterance record741in a sequence DB742, finding a pattern, which appears repeatedly in the sequence DB742, processing the pattern in a form of a quick command, and recommending a suitable name as a quick command.

FIG. 7separately illustrates the sequence DB creator module720, the pattern finder module730, and the name recommender module750. However, the quick command recommender module710may include the sequence DB creator module720, the pattern finder module730, and the name recommender module750.

The sequence DB creator module720may convert the utterance record741of the user into a form of the sequence DB742such that the pattern finder module730is capable of analyzing a user utterance pattern. The sequence DB creator module720may separate the utterance record741of the user into at least one or more sequences based on the time and/or location (place) of an utterance. The sequence DB creator module720may use the natural language (NL) result (e.g., a capsule, a goal, or a signal) of an utterance as an element capable of being expressed in one sequence. The NL result may mean a result grasped by the NLU module by using text data of the utterance input. For example, the NL result may mean the result of extracting the capsule, goal, and signal of the utterance input. This will be described in detail with reference toFIGS. 9 and 10.

The sequence DB creator module720may create one sequence by using whether the user has continuously received utterances. In an embodiment, when a difference in reception time between a plurality of utterances is not greater than a specified threshold with respect to the plurality of utterances received at the same location (place), the sequence DB creator module720may include the plurality of utterances received at the same location (place) in one sequence.

In an embodiment, when an utterance including duration information is present in the user's utterance record741, the sequence DB creator module720may compare the utterance with the specified threshold in consideration of the duration information. For example, in the case of a user who frequently employs a 3-minute timer to cook ramen, a pattern indicating that the timer is explicitly terminated after an alarm sounds once may be repeated. In this case, only when the user terminates the alarm while setting the alarm is included in the same sequence, this may be grasped as a pattern. Accordingly, even when the specified threshold is 1 minute, the sequence DB creator module720may receive the user's first utterance of “start a 3-minute timer”, and then may receive the user's second utterance of “stop the timer” after 3 minutes and 30 seconds. In this case, the sequence DB creator module720may generate one sequence for the first utterance and the second utterance, by using information of ‘3 minutes’, which is duration information included in the first utterance, instead of generating the first utterance and the second utterance as separate sequences by the specified threshold. Even when a long interval is present between the reception times of two utterances thus received, the sequence DB creator module720may include two utterances in the same sequence by using the duration information.

The pattern finder module730may find a sequential utterance set, which the user frequently and repeatedly employs, by using the sequence DB742. The pattern finder module730may find the utterance set by using a sequential pattern mining algorithm. For example, the pattern finder module730may find the desired utterance set by using an algorithm such as GSP, PrefixSpan, and/or SPADE.

In an embodiment, the pattern finder module730may calculate the support of a pattern, which is present in the sequence DB742, by using the sequential pattern mining algorithm. The support may mean the number of sequences, in each of which the pattern is present. For example, when three pieces of sequence data of (Music-Play Song, SmartThings-TurnOn, SmartThings-TurnOn), (Music-PlaySong, SmartThings-TurnOn, Setting-Volume), and (Music-PlaySong, Weather, SmartThings-TurnOn) are present in the sequence DB742, a pattern of (Music-PlaySong, SmartThings-TurnOn) may be included in all the three sequences, and thus the support may be calculated as 3. A pattern of (Music-PlaySong, Weather) is present in only one sequence, and thus the support is calculated as 1. A pattern of (SmartThings-TurnOn, Music-PlaySong) is not present in a sequence having the corresponding order, and thus the support is calculated as 0.

In an embodiment, the pattern finder module730may find patterns, of which the number is not less than the support set by the user in the sequence DB742, by using a sequential pattern mining algorithm. For example, when three pieces of sequence data of (Music-PlaySong, SmartThings-TurnOn, SmartThings-TurnOn), (Music-PlaySong, SmartThings-TurnOn, Setting-Volume), and (Music-PlaySong, Weather, SmartThings-TurnOn) are present in the sequence DB742and the support set by the user is 3, the pattern finder module730may find a pattern of (Music-PlaySong, SmartThings-TurnOn).

In an embodiment, when a quick command is generated, the pattern finder module730may have a waiting time between utterances by using average reception time information and duration information between elements in a pattern.

The name recommender module750may recommend and/or create a quick command name for the found utterance set. In an embodiment, the name recommender module750may model the relationship between the quick command name and the utterance set corresponding to the quick command name by using the already-created quick command name and then may recommend a quick command name. For example, the name recommender module750may receive, as inputs, an utterance included in the utterance set and/or an NL result of analyzing the utterance and then may output, as a result (output), names of utterances included in the utterance set. The NL result may include a capsule for performing the utterance, a goal for processing the utterance, and a parameter (signal) included in the utterance.

In an embodiment, the electronic device101may generate the relational model by using AI. The AI system may be a rule-based system, or may be a neural network-based system (e.g., a feedforward neural network (FNN) and/or a recurrent neural network (RNN)). Alternatively, the AI system may be a combination of the above-described systems or an AI system different from the above-described system.

In an embodiment, the name recommender module750may perform learning by using the already-created quick command name and/or the already-created quick command.

In an embodiment, the name recommender module750may find an important keyword of the utterance set, may summarize information about the utterance set, and may recommend and/or generate a quick command name for the utterance set. The important keyword of the utterance set may mean a word or phrase that has a lot of influence on the NL result. For example, the name recommender module750may quantify the degree of influence of each of the words on the NL result with respect to words included in the utterance set and may express the qualified degree by using a number. The name recommender module750may use words with a high number by using the number expressed in each word and then may combine at least one or more quick command names. This will be described in detail with reference toFIG. 11.

In an embodiment, the name recommender module750may generate and/or recommend a quick command name corresponding to the utterance set by using information about an utterance reception time and/or utterance reception location (place) that affected the creation of the utterance set. For example, the name recommender module750may consider that each utterance included in the utterance set included in some of the sequences created through the sequence DB creator module720has similar utterance reception time information and/or similar utterance reception location (place) information. The name recommender module750may generate the quick command name by using the utterance reception time information and/or the utterance reception location information. This will be described in detail with reference toFIG. 12.

In an embodiment, the name recommender module750may find words and/or phrases having a high similarity with words, phrases and/or sentences included in the utterance set and may generate and/or recommend the words and/or phrases as the quick command name. In an embodiment, the name recommender module750may embed words, phrases, and/or entire utterances included in the utterance set, may find a word and/or phrase with the highest similarity in a dictionary, and may generate and/or recommend the word and/or phrase as a quick command name. The embedding may refer to a scheme of expressing string data as a numeric vector. For example, the name recommender module750may find words and/or phrases with high similarity by using word embedding that expresses words included in the utterance set as a dense vector. This will be described in detail with reference toFIG. 13.

In an embodiment, the name recommender module750may recommend the quick command name within a predefined quick command name candidate. For example, the quick command name candidate defined by using user-friendly words and/or phrases such as onomatopoeia, mimetic words and/or magic spells may be stored in the database740of the electronic device or a memory (not shown) operatively connected to the electronic device name. The name recommender module750may recommend a quick command name by using the defined quick command name candidate.

In an embodiment, even when the user directly creates the quick command, the name recommender module750may generate and/or recommend a quick command name. For example, the name recommender module750may determine whether there is an utterance having the same NL result (e.g. a capsule, a goal, or a signal) as the quick command generated by the user and may recommend the quick command name by using the location and time information of the utterance record including the corresponding utterance.

The ASR module760may convert the received user input into text data. For example, the ASR module760may convert the received voice data into text data.

The NLU module770may grasp the user's intent by performing syntactic analysis or semantic analysis. According to an embodiment, the NLU module770may grasp the meaning of words extracted from the voice input by using linguistic features (e.g., syntactic elements) such as morphemes or phrases and may determine the intent of the user by matching the grasped meaning of the words to the intent.

According to an embodiment disclosed in this specification, a user-customized quick command name may be created and/or recommended by creating and/or recommending a quick command name by using the user's utterance record.

FIG. 8is a diagram of a user's utterance record, according to an embodiment of the disclosure.

For clarity of description, details the same as the above-described details may be briefly described or omitted. The utterance record of the user may mean a record uttered by the user of the electronic device700by using the speech recognition system of the electronic device700. For example, the utterance record of the user may mean a record of an utterance uttered by the user by using a voice assistant. The electronic device700may store the user's utterance record in the utterance record741ofFIG. 7.

Referring toFIG. 8, the utterance record of the user may include information about utterance content (utterance)801, an utterance time (time)802, and an utterance place (location)803of a user utterance.

For example, a first utterance810of a user may include information about utterance content811of ‘tell me today's schedule’, an utterance time812of ‘Sep. 3, 2020 7:00 AM’, and an utterance place (location)813of ‘home’.

The first utterance810, a second utterance820, a third utterance830, and a fourth utterance840inFIG. 8may include information about the same utterance places (locations)813,823,833, and843of ‘home’ and information about similar utterance times812,822,832, and842from ‘Sep. 3, 2020 7:00 AM’ to ‘Sep. 3, 2020 7:02 AM’.

A fifth utterance850and a sixth utterance860ofFIG. 8may include information about the same utterance places (locations)853and863of ‘company’ and information about similar utterance times852and862on ‘Sep. 3, 2020 19:04’ and ‘Sep. 3, 2020 19:05’.

A seventh utterance870, an eighth utterance880, and a ninth utterance890inFIG. 8may include information about the same utterance places (locations)873,883, and893of ‘home’ and information about similar utterance times872,882, and892from ‘Sep. 3, 2020 19:45’ to ‘Sep. 3, 2020 19:46’.

The electronic device700may convert the utterance record ofFIG. 8into a form of a sequence DB by using the similarity of information included in the utterance record. For example, a sequence DB creator module (e.g., the sequence DB creator module720ofFIG. 7) may convert the user's utterance record into a form of a sequence DB (e.g., the sequence DB742ofFIG. 7) such that the pattern finder module (e.g., the pattern finder module730ofFIG. 7) is capable of analyzing a user utterance pattern.

FIG. 9is a diagram in which a conversation record of a user is converted into a sequence, according to an embodiment of the disclosure.

For clarity of description, details the same as the above-described details may be briefly described or omitted.

The sequence ofFIG. 9is a diagram in which the utterance record of the user shown inFIG. 8is divided into at least one sequence based on information about a time and/or location (place) of an utterance. This is described together with reference toFIG. 8.

Referring toFIG. 9, the first sequence910ofFIG. 9is obtained as the electronic device700expresses the first utterance810, the second utterance820, the third utterance830, and the fourth utterance840ofFIG. 8as one sequence. The first utterance810, the second utterance820, the third utterance830, and the fourth utterance840inFIG. 8may include information about the same utterance places (locations)813,823,833, and843of ‘home’ and information about similar utterance times812,822,832, and842from ‘Sep. 3, 2020 7:00 AM’ to ‘Sep. 3, 2020 7:02 AM’.

Accordingly, the electronic device700may express the first utterance810, the second utterance820, the third utterance830, and the fourth utterance840inFIG. 8as one sequence. The electronic device700may express the utterance time812of the first utterance, which is a start utterance of the first sequence910, as a start time911of the first sequence.

A second sequence920ofFIG. 9is obtained as the electronic device700expresses the fifth utterance850and the sixth utterance860ofFIG. 8as one sequence. The fifth utterance850and the sixth utterance860ofFIG. 8may include information about the same utterance places (locations)853and863of ‘company’ and information about similar utterance times852and862on ‘Sep. 3, 2020 19:04’ and ‘Sep. 3, 2020 19:05’. Accordingly, the electronic device700may express the fifth utterance850and the sixth utterance860inFIG. 8as one sequence. The electronic device700may express the utterance time852of the fifth utterance, which is a start utterance of the second sequence920, as a start time921of the second sequence.

A third sequence930ofFIG. 9is obtained as the electronic device700expresses the seventh utterance870, the eighth utterance880, and the ninth utterance890ofFIG. 8as one sequence. The seventh utterance870, the eighth utterance880, and the ninth utterance890inFIG. 8may include information about the same utterance places (locations)873,883, and893of ‘home’ and information about similar utterance times872,882, and892from ‘Sep. 3, 2020 19:45’ to ‘Sep. 3, 2020 19:46’. Accordingly, the electronic device700may express the seventh utterance870, the eighth utterance880, and the ninth utterance890inFIG. 8as one sequence. The electronic device700may express the utterance time872of the seventh utterance, which is a start utterance of the third sequence930, as a start time931of the third sequence.

FIG. 10is a diagram of an element table including an NL result of an utterance generated by analyzing a user's utterance record, according to an embodiment of the disclosure.

The element table ofFIG. 10is a diagram of an element table including an NL result of an utterance generated by analyzing an utterance record of a user shown inFIG. 8. This will be described together with reference toFIG. 8. For clarity of description, details the same as the above-described details may be briefly described or omitted.

The electronic device700may analyze an utterance content (utterance) (e.g., the utterance content801inFIG. 8) of the user's utterance record and then may generate an NL result1020of an utterance including information about a capsule for performing the utterance, a goal for processing the utterance, and a parameter (signal) included in the utterance.

For example, the electronic device700may analyze the meaning of ‘tell me today's schedule’, which is the utterance content (utterance) of the first utterance810ofFIG. 8and then may generate a first NL result1021depending on the analyzed meaning. Referring toFIG. 10, the first NL result1021includes (schedule, show schedule, date: today) as a result for a capsule for performing the utterance, a goal for processing the utterance, and a parameter (signal) included in the utterance, respectively.

When the user directly creates a quick command, the electronic device700may determine whether there is an utterance having the same NL result (e.g., an NL result the same as at least one or more of a capsule for performing an utterance, a goal for processing the utterance, and a parameters (signal) included in the utterance) as the quick command created by the user, by using the element table ofFIG. 10.

FIGS. 11 to 13are conceptual diagrams of a method, in which the electronic device700recommends a quick command name, according to an embodiment of the disclosure.

FIG. 11is a conceptual diagram of a method, in which the electronic device700recommends a quick command name by using an important keyword, according to an embodiment of the disclosure.

The electronic device700may find an important keyword of the utterance set, may summarize information about the utterance set, and may recommend and/or generate a quick command name for the utterance set. The important keyword of the utterance set may mean a word or phrase that has a lot of influence on the NL result. For example, the electronic device700may find a word or phrase that has a lot of influence on the NL result and then may generate a quick command name by using the corresponding word or phrase.

Referring toFIG. 11, in a case of utterances1101,1102,1103, and1104, words such as “3 minutes”, “timer”, “brushing teeth song”, and “song” may have a lot of influence on classifying a capsule for performing the actual utterance of each utterance, a goal for processing the utterance, and a parameter (signal) included in the utterance. Accordingly, a quick command name such as “3-minute brushing”1111and “timer brushing teeth song”1112may be created based on these words or phrases.

FIG. 12is a conceptual diagram illustrating a method, in which the electronic device700recommends a quick command name by using information about utterance reception time and/or utterance reception place, according to an embodiment of the disclosure.

Referring toFIG. 12, the electronic device700may generate and/or recommend a quick command name corresponding to the utterance set by using information about an utterance reception time and/or utterance reception location (place) that affected the creation of the utterance set. The conceptual diagram shown inFIG. 12is the same as the description given with reference toFIGS. 8 to 11, and thus are omitted to avoid redundancy.

FIG. 13is a conceptual diagram illustrating a method, in which the electronic device700finds words and/or phrases having a high similarity with words, phrases and/or sentences included in an utterance set and generates and/or recommends the words and/or phrases as the quick command name, according to an embodiment of the disclosure.

The electronic device700may embed words, phrases, and/or entire utterances included in the utterance set, may find a word and/or phrase with the highest similarity in a dictionary, and may generate and/or recommend the word and/or phrase as a quick command name. The embedding may refer to a scheme of expressing string data as a numeric vector. For example, the name recommender module750may find words and/or phrases with high similarity by using word embedding that expresses a word included in the utterance set as a dense vector.

Referring toFIG. 13, for an instruction set as an example, when a first utterance1301and a second utterance1302included in a first utterance set1300are embedded, each of the embedded utterances may have a high similarity value, which uses a similarity scale such as euclidean distance or cosine similarity, with the embedding of words such as “slumber”, “deep sleep”, and “sleep”. The electronic device700may recommend a quick command name of “slumber”1311, “deep sleep”1312, “kooh” (onomatopoeia)1313, “saegeun saegeun” (onomatopoeia)1314, and/or “sleep”1315for the first utterance set1300.

Hereinafter, a method performed by the electronic device101according to an embodiment disclosed in the specification will be described with reference toFIGS. 14 and 15.

FIG. 14is a flowchart of a method, in which an electronic device recommends a quick command name, according to an embodiment of the disclosure.

According to an embodiment, it may be understood that the process illustrated inFIG. 14is performed by the processor (e.g., the processor120ofFIG. 1) of an electronic device (e.g., the electronic device101ofFIG. 1) by executing instructions stored in a memory (e.g., the memory130ofFIG. 1).

Referring toFIG. 14, in a method1400, in operation1401, the electronic device101may extract at least one or more utterance records of a user. The utterance record of the user may mean a record uttered by the user of the electronic device101by using a speech recognition system of the electronic device101. For example, the utterance record of the user may mean a record of an utterance uttered by the user by using a voice assistant. The user's utterance record may mean data stored in the storage included in the electronic device101or operatively connected to the electronic device101. The electronic device101may extract at least one or more conversation records of a user by using a user account included in the electronic device101or operatively connected to the electronic device101.

In an embodiment, the electronic device101may extract the utterance record of the user in response to a user input. The user input may include a touch input, a gesture input, and/or a voice input. In an embodiment, the electronic device101may receive the user input by using a user interface.

In an embodiment, the user input may be the user's voice input (e.g., an utterance). When the user input is a voice input, the electronic device101may receive the user input through a microphone (or a voice receiving device) included in the electronic device or operatively connected to the electronic device.

In an embodiment, the user input may be a gesture input and/or a touch input. When the user input is a gesture input and/or a touch input, the electronic device101may receive the user input through a sensor included in the electronic device or operatively connected to the electronic device.

In an embodiment, the electronic device101may identify input data matched with the received user input. For example, when the user input is a voice input (e.g., an utterance), the electronic device101may convert the received user input into text data. In an embodiment, the electronic device101may process data of the received voice input of the user. For example, the electronic device101may perform data refinement, data integration, data reduction, and/or data conversion on the received voice input data of the user. The electronic device101may improve the quality of data by processing the data.

In operation1403, the electronic device101may analyze the user's utterance record and may convert the utterance record into a sequence form.

In an embodiment, the electronic device101may separate the utterance record of the user into at least one or more sequences based on the time and/or location (place) of an utterance. The electronic device101may use the NL result (e.g., a capsule, a goal, or a signal) of an utterance as an element capable of being expressed in one sequence.

The electronic device101may create one sequence by using whether the user has continuously received utterances. In an embodiment, when a difference in reception time between a plurality of utterances is not greater than a specified threshold with respect to the plurality of utterances received at the same location (place), the electronic device101may include the plurality of utterances received at the same location (place) in one sequence.

In an embodiment, when an utterance including duration information is present in the user's utterance record, the electronic device101may compare the utterance with the specified threshold in consideration of the duration information. For example, in the case of a user who frequently employs a 3-minute timer to cook ramen, a pattern indicating that the timer is explicitly terminated after an alarm sounds once may be repeated. In this case, only when the user terminates the alarm while setting the alarm is included in the same sequence, this may be grasped as a pattern. Accordingly, even when the specified threshold is 1 minute, the electronic device101may receive the user's first utterance of “start a 3-minute timer”, and then may receive the user's second utterance of “stop the timer” after 3 minutes and 30 seconds. In this case, the electronic device101may generate one sequence for the first utterance and the second utterance, by using information of ‘3 minutes’, which is duration information included in the first utterance, instead of generating the first utterance and the second utterance as separate sequences by the specified threshold. Even when a long interval is present between the reception times of two utterances thus received, the electronic device101may include two utterances in the same sequence by using the duration information.

In operation1405, the electronic device101may discover an utterance set. The electronic device101may find a sequential utterance set, which the user frequently and repeatedly employs, by using the sequence. The electronic device101may find the utterance set by using a sequential pattern mining algorithm. For example, the electronic device101may find the desired utterance set by using an algorithm such as GSP, PrefixSpan, and/or SPADE.

In an embodiment, the electronic device101may calculate the support of a pattern that is present in a database (e.g., the sequence DB742ofFIG. 7) included in the electronic device101or operatively connected to the electronic device101. The support may mean the number of sequences, in each of which the pattern is present. For example, when three pieces of sequence data of (Music-PlaySong, SmartThings-TurnOn, SmartThings-TurnOn), (Music-Play Song, SmartThings-TurnOn, Setting-Volume), and (Music-PlaySong, Weather, SmartThings-TurnOn) are present in a database (e.g., the sequence DB742ofFIG. 7) included in the electronic device101or operatively connected to the electronic device101, a pattern of (Music-PlaySong, SmartThings-TurnOn) may be included in all the three sequences, and thus the support may be calculated as 3. A pattern of (Music-PlaySong, Weather) is present in only one sequence, and thus the support is calculated as 1. A pattern of (SmartThings-TurnOn, Music-PlaySong) is not present in a sequence having the corresponding order, and thus the support is calculated as 0.

In an embodiment, the electronic device101may find patterns, of which the number is not less than the support set by the user, in a database (e.g., the sequence DB742ofFIG. 7) included in the electronic device101or operatively connected to the electronic device101. For example, when three pieces of sequence data of (Music-Play Song, SmartThings-TurnOn, SmartThings-TurnOn), (Music-PlaySong, SmartThings-TurnOn, Setting-Volume), and (Music-PlaySong, Weather, SmartThings-TurnOn) are present in a database (e.g., the sequence DB742ofFIG. 7), which is included in the electronic device101or which is operatively connected to the electronic device101, and the support set by the user is 3, the electronic device101may find a pattern of (Music-PlaySong, SmartThings-TurnOn).

In an embodiment, when a quick command is generated, the electronic device101may have a waiting time between utterances by using average reception time information and duration information between elements in a pattern.

In operation1407, the electronic device101may recommend and/or generate a quick command name. In an embodiment, the electronic device101may model the relationship between the quick command name and the utterance set corresponding to the quick command name by using the already-created quick command name and then may recommend a quick command name. For example, the electronic device101may receive, as inputs, an utterance included in the utterance set and/or an NL result of analyzing the utterance and then may output, as a result (output), names of utterances included in the utterance set. The NL result may include a capsule for performing the utterance, a goal for processing the utterance, and a parameter (signal) included in the utterance.

In an embodiment, the electronic device101may generate the relational model by using AI. The AI system may be a rule-based system, or may be a neural network-based system (e.g., a feedforward neural network (FNN) and/or a recurrent neural network (RNN)). Alternatively, the AI system may be a combination of the above-described systems or an AI system different from the above-described system.

In an embodiment, the electronic device101may perform learning by using the already-created quick command name and/or the already-created quick command.

In an embodiment, the electronic device101may find an important keyword of the utterance set, may summarize information about the utterance set, and may recommend and/or generate a quick command name for the utterance set. The important keyword of the utterance set may mean a word or phrase that has a lot of influence on the NL result. For example, the electronic device101may quantify the degree of influence of each of the words on the NL result with respect to words included in the utterance set and may express the qualified degree by using a number. The electronic device101may use words with a high number by using the number expressed in each word and then may combine at least one or more quick command names.

In an embodiment, the electronic device101may generate and/or recommend a quick command name corresponding to the utterance set by using information about an utterance reception time and/or utterance reception location (place) that affected the creation of the utterance set. For example, the electronic device101may consider that each utterance included in the utterance set included in some of sequences stored in a database (e.g., the sequence DB742ofFIG. 7), which is included in the electronic device101or which is operatively connected to the electronic device101, has similar utterance reception time information and/or similar utterance reception location (place) information. The electronic device101may generate the quick command name by using the utterance reception time information and/or the utterance reception location information.

In an embodiment, the electronic device101may find words and/or phrases having a high similarity with words, phrases and/or sentences included in the utterance set and may generate and/or recommend the words and/or phrases as the quick command name. In an embodiment, the electronic device101may embed words, phrases, and/or entire utterances included in the utterance set, may find a word and/or phrase with the highest similarity in a dictionary, and may generate and/or recommend the word and/or phrase as a quick command name. The embedding may refer to a scheme of expressing string data as a numeric vector. For example, the electronic device101may find words and/or phrases with high similarity by using word embedding that expresses words included in the utterance set as a dense vector.

In an embodiment, the electronic device101may recommend the quick command name within a predefined quick command name candidate. For example, the quick command name candidate defined by using user-friendly words and/or phrases such as onomatopoeia, mimetic words and/or magic spells may be stored in a database of the electronic device or a memory (not shown) operatively connected to the electronic device name. The electronic device101may recommend a quick command name by using the defined quick command name candidate.

In an embodiment, even when the user directly creates the quick command, the electronic device101may generate and/or recommend a quick command name. For example, the electronic device101may determine whether there is an utterance having the same NL result (e.g. a capsule, a goal, or a signal) as the quick command generated by the user and may recommend the quick command name by using the location and time information of the utterance record including the corresponding utterance.

In operation1409, the electronic device101may provide response data. The response data may mean data including at least one or more quick command names. The electronic device101may provide a user with the response data including the at least one or more quick command names by using an output device (e.g., a display or a speaker) included in the electronic device101or operatively connected to the electronic device101.

In an embodiment, the electronic device101may convert response data including the quick command name in a form of a text into voice data by using a TTS module. The electronic device101(and/or a processor (not shown)) may output response data converted into voice data through a speaker (not shown).

It is illustrated inFIG. 14that the electronic device101sequentially performs operation1401to operation1409. However, this is only an example. For example, the operations may be performed at the same time. A part of the operations may be performed by the electronic device101and the other parts may be performed by an external device. For example, operation1409may be performed by the electronic device101and operation1401to operation1407may be performed by a server.

FIG. 15is another flowchart of a method, in which an electronic device recommends a quick command name, according to an embodiment of the disclosure.

According to an embodiment, it may be understood that the process illustrated inFIG. 15is performed by the processor (e.g., the processor120ofFIG. 1) of an electronic device (e.g., the electronic device101ofFIG. 1) by executing instructions stored in a memory (e.g., the memory130ofFIG. 1).

Referring toFIG. 15, in a method1500, in operation1501, the electronic device101may receive a user input for requesting generation of a quick command. The user input may include a touch input, a gesture input, and/or a voice input. In an embodiment, the electronic device101may receive the user input by using a user interface.

In an embodiment, the user input may be the user's voice input (e.g., an utterance). When the user input is a voice input, the electronic device101may receive the user input through a microphone (or a voice receiving device) included in the electronic device or operatively connected to the electronic device.

In an embodiment, the user input may be a gesture input and/or a touch input. When the user input is a gesture input and/or a touch input, the electronic device101may receive the user input through a sensor included in the electronic device or operatively connected to the electronic device.

In operation1503, the electronic device101may generate and/or add a quick command in response to the received user input.

In an embodiment, the electronic device101may identify input data matched with the received user input. For example, when the user input is a voice input (e.g., an utterance), the electronic device101may convert the received user input into text data. The electronic device101may generate at least one or more quick commands included in the user input by identifying the converted text and may add the generated quick commands as an utterance record to a database included in the electronic device101or operationally connected to the electronic device101.

In operation1505, the electronic device101may determine whether the quick command name is received. When determining that the quick command name has been received, the electronic device101may perform operation1507.

In operation1507, the electronic device101may provide second response data. The second response data may mean data including the received quick command name and/or the received quick command.

In an embodiment, the electronic device101may store the received quick command and the received quick command name in a database included in the electronic device101or operatively connected to the electronic device101.

In an embodiment, the electronic device101may analyze the received quick command and the received quick command name and then may recommend at least one or more quick commands capable of being included in one utterance set. The electronic device101may include the at least one or more quick commands in the second response data and then may provide the second response data.

On the other hand, when determining that the quick command name has not been received in operation1505, the electronic device101may perform operation1509.

In operation1509, the electronic device101may recommend a quick command name. The electronic device101may generate and/or recommend the quick command name by using the user's utterance record and a sequence, which are stored in a database included in the electronic device101or operatively connected to the electronic device101. This is described in detail with reference toFIG. 14, and thus is omitted to avoid redundancy.

In operation1511, the electronic device101may provide first response data. The first response data may mean data including at least one or more quick command names. The electronic device101may provide a user with the response data including the at least one or more quick command names by using an output device (e.g., a display or a speaker) included in the electronic device101or operatively connected to the electronic device101.

It is illustrated inFIG. 15that the electronic device101sequentially performs operation1501to operation1511. However, this is only an example. For example, the operations may be performed at the same time. A part of the operations may be performed by the electronic device101and the other parts may be performed by an external device. For example, operation1501, operation1507, and operation1511may be performed by the electronic device101, and operation1503, operation1505, and operation1509may be performed by a server.

In an embodiment, a method performed by the electronic device101may include extracting at least one or more utterance records of a user by using a user account included in the electronic device or operatively connected to the electronic device when a process for a memory included in the electronic device or operatively connected to the electronic device is executed, analyzing the extracted at least one or more utterance records, generating an utterance set including at least one or more operations based on the analyzed utterance records, generating at least one or more quick command names corresponding to the utterance set, and providing response data including the at least one or more quick command names.

In an embodiment, the method performed by the electronic device101may further include receiving a voice signal included in a user input by using a sound model included in the electronic device or operatively connected to the electronic device and causing the sound model to be learned by using a learning algorithm.

In an embodiment, the method performed by the electronic device101may further include separating at least one or more utterances, which are included in the utterance records, into at least one or more sequences based on at least one of information about a time of an utterance, which is included in the extracted at least one or more utterance records, or information about a location of the utterance.

In an embodiment, the method performed by the electronic device101may further include separating at least one or more utterances, which are included in the utterance records, into at least one or more sequences based on information about at least one of a goal, a capsule, or a signal of an utterance included in the extracted at least one or more utterance records.

In an embodiment, the method performed by the electronic device101may further include comparing utterance reception times of a plurality of utterances included in the extracted at least one or more utterance records and including the plurality of utterances in an identical sequence when a difference between the utterance reception times is not greater than a specified value.

In an embodiment, the method performed by the electronic device101may further include comparing the utterance reception times by using the duration information when an utterance including duration information is included in the extracted at least one or more utterance records, and including the plurality of utterances in the identical sequence when the difference between the utterance reception times, which is obtained by comparing the utterance reception times by using the duration information, is not greater than the specified value.

In an embodiment, the method performed by the electronic device101may further include modeling a relational model between the utterance set and the quick command names and learning generation or recommendation of the quick command names by using the modeled relational model.

In an embodiment, the method performed by the electronic device101may further include performing learning by receiving an utterance included in the utterance set or an NL result, which is obtained by analyzing the utterance, as inputs and outputting a quick command name for utterances included in the utterance set as a result by using the relational model.

In an embodiment, the method performed by the electronic device101may further include finding an important keyword included in the utterance set and generating the quick command names for the utterance set by using the important keyword.

In an embodiment, the method performed by the electronic device101may further include embedding a word, a phrase, and an entire utterance included in the utterance set and generating the quick command names for the utterance set by using at least one of a word and a phrase, which have the highest similarity.