SYSTEM AND METHOD FOR PREDICTING TASK COMPLETION OF VOICE ASSISTANT FROM ONLINE USER LOGS

A method for predicting a task completion of a voice assistant from online user logs may include obtaining a voice assistant log regarding a user voice input of a user of an electronic device requesting a voice assistant of the electronic device to perform a task; extracting a set of features from the voice assistant log; and identifying a task completion estimation metric that is indicative of a performance of the voice assistant in performing the task, based on the set of features and a trained artificial intelligence (AI) model.

BACKGROUND

The disclosure relates to a system and method for identifying a task completion estimation metric that is indicative of a performance of a voice assistant in performing a requested task, based on a set of features extracted from a voice assistant log and a trained artificial intelligence (AI) model.

2. Description of Related Art

A voice assistant may refer to a software agent that is configured to perform a task based on a user voice input. For example, a user may provide a user voice input to an electronic device requesting a voice assistant of the electronic device to perform a task, and the voice assistant may perform the task based on the user voice input. As an example, the user may provide a user voice input of “call dad,” and the voice assistant may cause the electronic device to call the requested contact.

A user's satisfaction with the voice assistant may vary based on whether the voice assistant performs the task. For example, a user may be satisfied with the voice assistant if the voice assistant performs the task, and the user may be dissatisfied with the voice assistant if the voice assistant is unable to perform the task. Also, the users satisfaction with the voice assistant may vary based on the extent and manner of interaction between the user and the voice assistant. For example, the user may be more satisfied with the voice assistant if the voice assistant performs the task directly in response to a single user voice input, whereas the user may be less satisfied with the voice assistant if the voice assistant performs the task after requiring the user to input multiple user voice inputs.

Identifying a users satisfaction with a voice assistant may be impractical, time consuming, and/or may require a significant amount of processing resources.

SUMMARY

According to an aspect of an example embodiment, a method may include obtaining a voice assistant log regarding a user voice input of a user of an electronic device requesting a voice assistant of the electronic device to perform a task; extracting a set of features from the voice assistant log; and identifying a task completion estimation metric that is indicative of a performance of the voice assistant in performing the task, based on the set of features and a trained artificial intelligence (AI) model.

According to an aspect of an example embodiment, a device may include a memory configured to store instructions; and a processor configured to execute the instructions to: obtain a voice assistant log regarding a user voice input of a user of an electronic device requesting a voice assistant of the electronic device to perform a task; extract a set of features from the voice assistant log; and identify a task completion estimation metric that is indicative of a performance of the voice assistant in performing the task, based on the set of features and a trained artificial intelligence (AI) model.

According to an aspect of an example embodiment, a non-transitory computer-readable medium may store instructions that, when executed by one or more processors of an electronic device, cause the one or more processors to: obtain a voice assistant log regarding a user voice input of a user of an electronic device requesting a voice assistant of the electronic device to perform a task; extract a set of features from the voice assistant log; and identify a task completion estimation metric that is indicative of a performance of the voice assistant in performing the task, based on the set of features and a trained artificial intelligence (AI) model.

The set of features may include a latency value of the user voice input and a response of the voice assistant.

The set of features may include an application associated with the task.

The set of features may include a day of a week of the user voice input.

The set of features may include an hour of a day of the user voice input.

The set of features may include a first bidirectional encoder representations from transformers (BERT) embedding for the user voice input, and a second BERT embedding for a response of the voice assistant.

The set of features may include a sentiment of the user voice input.

The set of features may include a similarity value between the user voice input and a subsequent user voice input.

The set of features may include an identifier of whether the user voice input includes an interrogative word.

The set of features may include a number of stop words of the user voice input.

The set of features may include a number of words of the user voice input.

The trained AI model may be trained using training virtual assistant logs that are paired with known task completion estimation metrics.

The task completion estimation metric may be indicative of a user satisfaction with the voice assistant.

An action may be performed based on the task completion estimation metric.

DETAILED DESCRIPTION

FIG. 1is a diagram of a system for training an AI model, and identifying a task completion estimation metric using the trained AI model according to an example embodiment.

Referring toFIG. 1, according to an embodiment of the present disclosure, an electronic device101is included in a network environment100. The electronic device101may include at least one of a bus110, a processor120, a memory130, an input/output interface150, a display160, a communication interface170, or an event processing module180. In some embodiments, the electronic device101may exclude at least one of the components or may add another component.

The bus110may include a circuit for connecting the components120to180with one another and transferring communications (e.g., control messages and/or data) between the components.

The processor120may include one or more of a central processing unit (CPU), an application processor (AP), or a communication processor (CP). The processor120may perform control on at least one of the other components of the electronic device101, and/or perform an operation or data processing relating to communication.

The memory130may include a volatile and/or non-volatile memory. For example, the memory130may store commands or data related to at least one other component of the electronic device101. According to an embodiment of the present disclosure, the memory130may store software and/or a program140. The program140may include, e.g., a kernel141, middleware143, an application programming interface (API)145, and/or an application program (or “application”)147. At least a portion of the kernel141, middleware143, or API145may be denoted an operating system (OS).

For example, the kernel141may control or manage system resources (e.g., the bus110, processor120, or a memory130) used to perform operations or functions implemented in other programs (e.g., the middleware143, API145, or application program147). The kernel141may provide an interface that allows the middleware143, the API145, or the application147to access the individual components of the electronic device101to control or manage the system resources.

The middleware143may function as a relay to allow the API145or the application147to communicate data with the kernel141, for example. A plurality of applications147may be provided. The middleware143may control work requests received from the applications147, e.g., by allocation the priority of using the system resources of the electronic device101(e.g., the bus110, the processor120, or the memory130) to at least one of the plurality of applications134.

The API145is an interface allowing the application147to control functions provided from the kernel141or the middleware143. For example, the API133may include at least one interface or function (e.g., a command) for filing control, window control, image processing or text control.

The input/output interface150may serve as an interface that may, e.g., transfer commands or data input from a user or other external devices to other component(s) of the electronic device101. Further, the input/output interface150may output commands or data received from other component(s) of the electronic device101to the user or the other external device.

The display160may include, e.g., a liquid crystal display (LCD), a light emitting diode (LED) display, an organic light emitting diode (OLED) display, or a microelectromechanical systems (MEMS) display, or an electronic paper display. The display160may display, e.g., various contents (e.g., text, images, videos, icons, or symbols) to the user. The display160may include a touchscreen and may receive, e.g., a touch, gesture, proximity or hovering input using an electronic pen or a body portion of the user.

For example, the communication interface170may set up communication between the electronic device101and an external electronic device (e.g., a first electronic device102, a second electronic device104, or a server106). For example, the communication interface170may be connected with the network162or164through wireless or wired communication to communicate with the external electronic device.

The first external electronic device102or the second external electronic device104may be a wearable device or an electronic device101—mountable wearable device (e.g., a head mounted display (HMD)). When the electronic device101is mounted in a HMD (e.g., the electronic device102), the electronic device101may detect the mounting in the HMD and operate in a virtual reality mode. When the electronic device101is mounted in the electronic device102(e.g., the HMD), the electronic device101may communicate with the electronic device102through the communication interface170. The electronic device101may be directly connected with the electronic device102to communicate with the electronic device102without involving with a separate network.

The network162may include at least one of communication networks, e.g., a computer network (e.g., local area network (LAN) or wide area network (WAN)), Internet, or a telephone network.

The first and second external electronic devices102and104each may be a device of the same or a different type from the electronic device101. According to an embodiment of the present disclosure, the server106may include a group of one or more servers. According to an embodiment of the present disclosure, all or some of operations executed on the electronic device101may be executed on another or multiple other electronic devices (e.g., the electronic devices102and104or server106). According to an embodiment of the present disclosure, when the electronic device101should perform some function or service automatically or at a request, the electronic device101, instead of executing the function or service on its own or additionally, may request another device (e.g., electronic devices102and104or server106) to perform at least some functions associated therewith. The other electronic device (e.g., electronic devices102and104or server106) may execute the requested functions or additional functions and transfer a result of the execution to the electronic device101. The electronic device101may provide a requested function or service by processing the received result as it is or additionally. To that end, a cloud computing, distributed computing, or client-server computing technique may be used, for example.

AlthoughFIG. 1shows that the electronic device101includes the communication interface170to communicate with the external electronic device104or106via the network162, the electronic device101may be independently operated without a separate communication function, according to an embodiment of the present disclosure.

For example, the event processing server module may include at least one of the components of the event processing module180and perform (or instead perform) at least one of the operations (or functions) conducted by the event processing module180.

The event processing module180may process at least part of information obtained from other elements (e.g., the processor120, the memory130, the input/output interface150, or the communication interface170) and may provide the same to the user in various manners.

Although inFIG. 1the event processing module180is shown to be a module separate from the processor120, at least a portion of the event processing module180may be included or implemented in the processor120or at least one other module, or the overall function of the event processing module180may be included or implemented in the processor120shown or another processor. The event processing module180may perform operations according to embodiments of the present disclosure in interoperation with at least one program140stored in the memory130.

FIG. 2is a diagram of a process of training an AI model, and identifying a task completion estimation metric using the trained AI model according to an example embodiment.

As shown inFIG. 2, a server106may obtain user voice assistant logs210respectively regarding user voice inputs of users of electronic devices101requesting voice assistants of the electronic devices101to perform tasks. The server106may obtain known task completion estimation metrics220that are respectively associated with corresponding voice assistant logs210. Further, the server106may extract a respective set of features230from each of the voice assistant logs210. The server106may train an AI model240using the extracted sets of features230of the voice assistant logs210, and the known task completion estimation metrics220of the voice assistant logs210. In this way, in real-time, the trained AI model240may obtain an extracted set of features250of a user voice assistant log260that is associated with an unknown task completion estimation metric, and identify a task completion estimation metric270that is indicative of a performance of a voice assistant in performing a task.

FIG. 3is a flowchart of a process of identifying a task completion estimation metric using a trained AI model according to an example embodiment.

As shown inFIG. 3, the process may include obtaining a voice assistant log regarding a user voice input of a user of an electronic device requesting a voice assistant of the electronic device to perform a task (operation310). The electronic device101may obtain the voice assistant log. Alternatively, the server106may obtain the voice assistant log.

The electronic device101may include a voice assistant. For example, the voice assistant may be a virtual assistant, an intelligent virtual assistant, an intelligent personal assistant, or the like. The voice assistant may be configured to perform a task using one or more applications of the electronic device101. For example, the voice assistant may call a contact using a phone application, send a message to a contact using a messaging application, identify and output weather information using a weather application, identify and output appointment information using a calendar application, identify and output requested information using a web browsing application, or the like.

The user may provide a user voice input to the electronic device101requesting the voice assistant to perform a task. For example, the user may provide a user voice input of “call dad” requesting the voice assistant to call a father of the user. Based on the user voice input, the voice assistant may output a response to the user voice input and/or may perform the requested task. For example, the voice assistant may output “Okay, calling dad” via a speaker of the electronic device101and may call the father of the user using a telephone application of the electronic device101. As another example, the voice assistant may be unable to perform the requested task based on the user voice input, and may request additional information from the user. For instance, the voice assistant may output “I'm sorry, but I did not hear that Can you please repeat?,” via the speaker of the electronic device101in order to request the user to repeat, or clarify, the user voice input. It should be understood that a particular interaction between a voice assistant and the user may include any type and number of user voice inputs, and any type and number of responses from the voice assistant.

The voice assistant log may be a set of data associated with a particular interaction between a user and a voice assistant. For example, a voice assistant log may include information identifying a user voice input of the user, a requested task, an application associated with the requested task, a time of day of the user voice input, a day of the week of the user voice input, a response of the voice assistant, whether the task was completed, a duration of the particular interaction between the user and the voice assistant, respective time stamps of the user voice input and the response of the voice assistant, or the like.

The electronic device101may generate the voice assistant log based on the user of the electronic device101interacting with the voice assistant, and may obtain the voice assistant log based on generating the voice assistant log. Additionally, the electronic device101may transmit the voice assistant log to the server106. In this case, the server106may obtain the voice assistant log based on receiving the voice assistant log from the electronic device101.

As further shown inFIG. 3, the process may include extracting a set of features from the voice assistant log (operation320).

The electronic device101may extract a set of features from the voice assistant log. Alternatively, the server106may extract the set of features from the voice assistant log.

The set of features may include a latency value of the user voice input and a response of the voice assistant, an application associated with the task, a day of a week of the user voice input, an hour of a day of the user voice input, a bidirectional encoder representations from transformers (BERT) embedding for the user voice input, a BERT embedding for a response of the voice assistant, a sentiment of the user voice input, a sentiment of a subsequent user voice input, a similarity value between the user voice input and the subsequent user voice input, an identifier of whether the user voice input includes an interrogative word (e.g., “who,” “what,” “why,” etc.), a number of stop words (e.g., “a,” “the,” “is,” etc.) of the user voice input, a number of words of the user voice input, a number of user voice inputs of the user during the particular interaction, a number of responses of the voice assistant during the particular interaction, a duration of the particular interaction, whether the task was performed, or the like.

The set of features may include one or more of the foregoing delineated features, and may include any combination or permutation of the foregoing delineated features. Each feature of the set of features may be encoded to generate a feature vector.

As further shown inFIG. 3, the process may include identifying a task completion estimation metric that is indicative of a performance of the voice assistant in performing the task, based on the set of features and a trained AI model (operation330).

The electronic device101may identify the task completion estimation metric, based on the set of features and a trained AI model. For example, the electronic device101may store the trained AI model, input the set of features into the trained AI model, and identify the task completion estimation metric based on an output of the trained AI model. Alternatively, the server106may identify the task completion estimation metric. For example, the server106may store the trained AI model, input the set of features into the trained AI model, and identify the task completion estimation metric based on an output of the trained AI model.

The trained AI model may be configured to obtain a set of features of a voice assistant log as an input, and output the task completion estimation metric. The trained AI model may be a convolution neural network (CNN), a deep neural network (DNN), a support vector machine (SVM), a K-nearest neighbor (KNN), a random forest, a gradient boosting technique, a linear regression technique, a feedforward neural network, a deep Q network, or the like. According to a non-limiting embodiment, the AI model may be a feedforward neural network including a combination of linear layers, dropout layers, and rectified linear unit (ReLU) activations. Further, the loss function may be a binary cross-entropy loss function.

The trained AI model may be trained using supervised learning, unsupervised learning, reinforcement learning, or the like. For example, the trained AI model may be trained using paired sets of features from training voice assistant logs and known task completion estimation metrics. The server106may train the AI model, and provide the trained AI model to the electronic device101.

The trained AI model may obtain the set of features extracted from the voice assistant log, and may identify a task completion estimation metric based on the set of features. The trained AI model may be configured to assign weights to the features of the set of features, and may identify the task completion estimation metric based on assigning the weights to the features.

The trained AI model may identify the task completion estimation metric that is indicative of a performance of the voice assistant in performing the task. For example, the task completion estimation metric may be a score, a value, etc., that is indicative of the performance of the voice assistant in performing the task. As examples, a task completion estimation metric having a low score (e.g., “0”) may be indicative of the voice assistant being unable to perform the task, a task completion estimation metric having a high score (e.g., “1”) may be indicative of the voice assistant performing the task in a highly efficient and seamless manner, and a task completion estimation metric having a medium score (e.g., “0.5”) may be indicative of the voice assistant performing the task albeit in a non-efficient or non-seamless manner.

The task completion estimation metric may be indicative of a user satisfaction with the voice assistant. For example, a task completion estimation metric having a low score (e.g., “0”) may be indicative of the user being dissatisfied with the voice assistant, a task completion estimation metric having a high score (e.g., “1”) may be indicative of the user being highly satisfied with voice assistant, and a task completion estimation metric having a medium score (e.g., “0.5”) may be indicative of the user being indifferent towards the voice assistant.

The task completion estimation metric may be identified in real-time, or substantially in real-time. For example, the task completion estimation metric may be identified within a threshold time frame (e.g., one second, five seconds, ten seconds, etc.) of the end of the particular interaction between the user and the voice assistant, may be identified within a threshold time frame of the input of the user voice input, or the like.

The electronic device101may perform an action based on the task completion estimation metric. For example, the electronic device101transmit the task completion estimation metric to the server106. In this way, task completion estimation metrics from electronic devices101may be aggregated and analyzed by the server106. For example, the server106may provide, to an electronic device101associated with an administrator, information identifying various task completion estimation metrics from electronic devices101such as in the form of a dashboard. The information may include task completion estimation metrics that are aggregated on a region (e.g., city, country, etc.) basis, on a device (e.g., type of smartphone) basis, on an application basis, etc. Accordingly, voice assistants or applications associated with requested tasks that are associated with low task completion estimation metrics may be identified and improved.

The electronic device101may identify whether the task completion estimation metric is less than or equal to a threshold (e.g., “0.5,” “0.4,” 0.3,” etc.), and perform the action based on identifying that the task completion estimation metric is less than or equal to the threshold. In this case, the voice assistant may perform a different, but related, task in an effort to mitigate the user's experience. As an example, the voice assistant may output an option to call another person associated with the requested task, output an option to place a reservation at another restaurant, etc. As another example, the voice assistant may output information that is apologetic. For example, the voice assistant may output an apologetic emoticon, may express an apology, or the like.

In this way, the electronic device101may identify a task completion estimation metric using a trained AI model in real-time, and perform an action based on the identified task completion estimation metric.

AlthoughFIG. 3shows example operations of a process of identifying a task completion estimation metric using a trained AI model, the process may include additional operations, fewer operations, different operations, or differently arranged operations than those depicted inFIG. 3. Additionally, or alternatively, two or more of the operations of the process may be performed in parallel.

FIG. 4is a diagram of voice assistant logs associated with task completion estimation metrics according to an example embodiment. As shown inFIG. 4, a voice assistant log may be associated with an interaction identifier that identifies the particular interaction between a user and a voice assistant, a request identifier that identifies a request to perform a task, a user voice input, a voice assistant response, an application associated with the request, and a task completion estimation metric.

As shown inFIG. 4, the voice assistant logs410and420are associated with high task completion estimation metrics because the voice assistants performed the requested tasks, and did so in a seamless manner. In contrast, the voice assistant logs430and440are associated with low task completion estimation metrics because the voice assistants did not perform the requested tasks, let alone in a seamless manner.

By identifying the respective task completion estimation metrics, the quality and efficiency of voice assistants may be improved. In this way, the example embodiments provide an improvement in the functioning of electronic devices101and an improvement in the utilization of processor and/or memory resources of electronic devices101.