Patent Description:
It is with respect to these and other general considerations that the aspects disclosed herein have been made. Also, although relatively specific problems may be discussed, it should be understood that the examples should not be limited to solving the specific problems identified in the background or elsewhere in this disclosure. <CIT> relates to technologies for emotional modulation of natural language responses which include a computing device that receives natural language requests from a user. The computing device identifies emotional features of the request and estimates an emotional state of the request by comparing the emotional features to a vocal pattern database. The computing device generates a natural language response and modulates the emotional content of the natural language response based on the emotional state of the request and the vocal pattern database. The computing device may modulate the natural language response to mimic the emotional state of the request, or to oppose the emotional state of the request. Possible emotional states include urgency, certainty, and dominance. Possible emotional features include acoustic, prosodic, and linguistic characteristics of the user request. The computing device may update the vocal pattern database based on the user request to adapt to the user. Other embodiments are described and claimed.

It is the object of the present invention to provide an improved method of determining a class of a task based on acoustic features of audio data.

According to the present disclosure, the above and other issues are resolved by determining importance and priority of a task based on acoustic features of audio data associated with the task.

While previous methods of determining a task exist with respect to receiving a voice command, the present disclosure relates to determining importance and urgency of a task based on acoustic features of audio data associated with the task. The task may include initiating a communication (e.g., a telephone call) or an alert and performing a process (e.g., making or updating a calendar appointment and activities). The audio data may include voice data and ambient sounds. In particular, the audio data associated with the task includes background sound, e.g., siren, engine sounds, people talking, and ambient noises. The disclosed technology addresses the issue by analyzing the voice data or the audio data for identifying acoustic features of the voice data or the audio data, classifying the data into classes that characterizes the situation based on a level of urgency and importance. For example, voice data may include a command to perform while acoustic features of the voice data may indicate a level of stress by the speaker and the ambient noise may indicate a level of urgency of the circumstance. A machine learning processing determines importance and urgency of the task by receiving acoustic features of the audio data.

The disclosed technology includes determining audio sub-stream data of received audio input associated with a task command and generating embedding vector data and/or values of acoustic features of respective the audio sub-stream data. The disclosed technology further includes a machine learning (ML) system to determine an importance and urgency of a task based on the embedding vector data and/or the values of acoustic features. The ML system may use a regression model for sequential analysis of the acoustic features or embedding models for parallel processing. The ML model may use a neural network, e.g., a recurring neural network. As a result, the disclosed technology determines levels of urgency and importance of user utterance or audio data efficiently and accurately by focusing on classifying acoustic features of the data.

This Summary is provided to introduce a selection of concepts in a simplified form, which is further described below in the Detailed Description. This Summary is not intended to be used to limit the scope of the claimed subject matter. Additional aspects, features, and/or advantages of examples will be set forth in part in the following description and, in part, will be apparent from the description, or may be learned by practice of the disclosure.

Various aspects of the disclosure are described more fully below with reference to the accompanying drawings, which from a part hereof, and which show specific example aspects. However, different aspects of the disclosure may be implemented in many different ways and should not be construed as limited to the aspects set forth herein; rather, these aspects are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the aspects to those skilled in the art. Aspects may be practiced as methods, systems or devices. Accordingly, aspects may take the form of a hardware implementation, an entirely software implementation or an implementation combining software and hardware aspects. The following detailed description is, therefore, not to be taken in a limiting sense.

Traditional systems generate tasks and determine importance and urgency of a task based on user interactions with devices, e.g., calendar applications and digital assistants, and relationships with other tasks. These systems generate tasks or items in to-do lists as users interact with the devices via graphical user interfaces, voice commands, and other methods as input commands. Some other systems automatically generate tasks based on predetermined set of rules or patterns of activities of the user as determined by the systems.

Systems may prioritize tasks in various ways. In some aspects, the system may prioritize tasks based on commands received from the user to prioritize. In some other aspects, the systems automatically prioritize tasks based on predefined rules of prioritization and pattern matching of activities against rules. Some applications prioritize tasks relative to existing tasks based on context of respective tasks. However, issues arise when users expect the systems automatically determine urgency and priority of tasks when the tasks based on a given command indicate the same or similar contexts. For example, a task with a command based on a verbal input "Call Mom" made by a user at 7pm on Saturday night over two weekends may indicate the same level of priority of the task to call mother. The user, however, may expect one of them to be higher in priority than any other task at the time because the user was spoke "Call Mom" when she was in a car accident at 7pm on the second Saturday night.

Traditional task management relies on receiving a command or an event based on predetermined rules for generating and prioritizing tasks. The invention determines task urgency and priority based on Vocal/Audio/Acoustic features and assigns importance, urgency, and/or priority of tasks. In aspects, tasks may be explicit entries in any To-Do or Task management applications. Additionally, or alternatively, tasks may be implicit entries extracted from emails, or set of actions requested by commands from a voice assistant. Task entries may be captured through multi-modal channels. A machine learning (ML) model uses the acoustic features (not limited to voice in the foreground audio but also the background audio). Acoustic features determine importance of tasks based on the feedback from the model. Further actions categorize the tasks based on the importance and provide the user with an improved execution of the task.

As discussed in more detail below, the present disclosure relates to determining importance and urgency of a task based on acoustic features of audio data associated with the task using a machine learning model. In particular, the acoustic features of audio data may include pitch, volume, rhythm, and the like. Audio data may be associated with a task when the audio data includes a command for the task or the task may be inferred from the audio data. The audio data may include one or more audio sub-streams. Respectively sub-streams may be associated with distinct types of audio data and context.

<FIG> illustrates an overview of an example system for determining importance and urgency of a task based on acoustic features of audio data using a machine learning model in accordance to aspects of the present disclosure. System <NUM> represents a system for machine learning model or a neural network to generate a task and determine importance, urgency, and priority of the task using a model generator and a task manager. System <NUM> includes a client device <NUM>, a task manager <NUM>, and a model generator <NUM>. A user <NUM> interacts with the client device <NUM>. The task manager <NUM> and the model generator <NUM> may be implemented as one or more servers, connected from the client device <NUM> via a network (not shown). Additionally, or alternatively, the task manager <NUM> and the model generator <NUM> may include one or more sets of instructions to, at least in part, execute as applications on the client device <NUM>. The client device <NUM> communicates with the task manager <NUM>. The task manager <NUM> includes an audio receiver <NUM>, an audio sub-stream generator <NUM>, an audio type / feature determiner <NUM>, a storage that stores trained audio type models <NUM>, a task generator <NUM>, a task classifier <NUM>, and a task processor <NUM>. The task manager <NUM> communicates with the model generator <NUM>. The model generator <NUM> generates trained models by training models for determining audio types and task classes. The model generator <NUM> includes an audio type model trainer <NUM>, a task classification model trainer <NUM>, a storage for storing training audio-type data <NUM>, a storage for storing training task class data <NUM>, and a model deployer <NUM>.

The client device <NUM> may be a smart speaker, a smartphone, a personal computing device, or a general computing device providing input and output capabilities. The client device <NUM> includes input parts (e.g., one or more microphone, a video camera, touch sensors, a keyboard, etc.) and output parts (e.g., one or more speakers, a display, etc.) In aspects, the client device <NUM> may communicate over the network (not shown) with the task manager <NUM>.

In aspects, the audio receiver <NUM> may receive audio data <NUM> from audio input parts (e.g., a microphone) in the client device <NUM>. The audio receiver <NUM> receives the audio data <NUM> as a stream of audio signals. An audio sub-stream generator <NUM> generates one or more audio sub-streams based on the received audio data <NUM>. In aspects, the stream of audio data <NUM> may include a sub-stream that represents a foreground sound <NUM> and one or more sub-streams of background sound <NUM> respectively. In aspects, the audio sub-stream generator <NUM> separates the stream of audio data <NUM> into one or more sub-streams of the background sound <NUM> using a technology that is similar to, but not limited to, identifying and canceling background noise data in audio data.

The audio type / feature determiner <NUM> uses a machine learning model or a neural network to determine a type and features of acoustics in the received audio data <NUM>. Audio type or data type (114A and 122A) indicates types of respective sub-streams of the audio data. A type 114A of the foreground sound <NUM> in the audio data <NUM> may be voice data. A type 122A of the background sound <NUM> may be a siren from a police car or an ambulance, for example. In aspects, the audio type / feature determiner <NUM> may use a machine learning (ML) model or a neural network to determine or predict the audio type and features. The ML model or the neural network for determining audio type may use the trained audio type models <NUM>. Acoustic features include pitch, tone, intensity, volume. The foreground voice data may include values for a pitch 116A, a tone 118A, and intensity 120A. The background siren sound may include values for a pitch 116B, a tone 118B, and intensity 120B. The audio type / feature determiner <NUM> determines types of respective sub-streams of the audio data <NUM> using a trained machine learning (ML) model. The storage for the trained audio type models <NUM> stores the trained ML models used by the audio type / feature determiner <NUM>. In aspects, the trained ML model determines a type of audio data based on a set of predetermined rules to analyze the received sub-stream of audio stream data.

In aspects, the audio type / feature determiner <NUM> may determine an audio type and acoustic features of the audio stream data based on analyzing the audio stream data without splitting the audio stream data into audio sub-streams. This way, processing for determining the audio type and the acoustic features becomes less intensive than generating the audio sub-streams and determining the audio type and the acoustic features for respective audio sub-streams. There may be a trade-off of accuracy in determining the audio type and acoustic features based on the audio stream data because analyzing the respective audio sub-streams may enable determining the audio type and the acoustic features more accurately with more detailed analyses than using the audio stream data.

The task generator <NUM> generates a task based on the received audio input and the determined sub-streams of audio data <NUM>. In aspects, the task generator <NUM> generates a task using the foreground sound <NUM> with a data type 114A (e.g., voice in foreground). For example, a task may be to call mother when the foreground sound <NUM> includes a data type 114A of voice and a speech made by the voice is "Call Mom.

The task classifier <NUM> classifies a task into one or more of predefined classes using acoustic features of sub-streams of the received audio data <NUM>. The predefined classes are associated with importance and urgency of the task (e.g., "Important," "Not Important," "Urgent," "Not Urgent," etc.). The task classifier <NUM> may use a machine learning model to classify the task. The machine learning model, which may be stored in the storage for trained models <NUM>, may determine importance and urgency of a given task using values of acoustic features of the audio data. Additionally or alternatively, the task classifier <NUM> may use a neural network with a set of trained parameters to classify the task. The storage for trained task classification models <NUM> includes the set of trained parameters used by the task classifier <NUM> to classify the task. The neural network may use embeddings that at least represent acoustic features of the received audio data <NUM>. In aspects, embeddings represent a set of multi-dimensional vectors or mappings of acoustic features to vectors of continuous numbers. The Audio sub-stream generator <NUM> may convert audio signals of the sub-streams of the received audio data <NUM> into embeddings as input to the neural network in the task classifier <NUM>. The task classifier <NUM> may use the neural network to determine a likelihood of the classifications by determining probability distributions for respective classes associated with importance and urgency of the task. For example, the task classifier <NUM> classifies a task in a class "Important" and not "Not Important" when the probability distribution indicates the class "Important" as more probable than the class "Not Important.

In aspects, the task classifier <NUM> may use a regression data model to determine importance, urgency, and a priority of classes. In particular, the regression data model refers to acoustic features of an audio sub-stream at a preceding time. For example, a task with a background sound of a siren that becomes louder over time may be classified as being important and / or urgent. The task classifier <NUM> may analyze the acoustic feature of the siren background audio sub-stream regressively over time duration of the audio input. In aspects, a pitch and a volume of the audio sub-stream changes over time to determine that the source of the siren (e.g., a fire truck) is coming closer.

The task processor <NUM> processes and executes the generated task at a determined level of importance and urgency. In aspects, the task processor <NUM> interacts with a telephonic application to place a phone call based on the task. Additionally, or alternatively, the task processor <NUM> may interface with external applications including but not limited to calendar applications, task management applications (not shown), and telecommunication applications, and transmit the task according to the determined classes of importance and urgency of the task. When the classes of the task indicate the task as "urgent," the task processor <NUM> may initiate performing the task with urgency, for example. In some other aspects, the task processor <NUM> may cause the calendar application to indicate the task with highlights or an emphasis when the determined class of the task indicates the task as "important.

The model generator <NUM> generates trained audio type models <NUM> and trained task classification models <NUM> and deploys the trained data to the storage for the trained models <NUM> in the task manager <NUM>. The audio type model trainer trains audio type model using a training set of correct audio data and types. The training set may include, for example, a pair of audio stream data with a voice speech and a type "voice. " Another training data may be another pair of audio stream data that includes siren and a type "siren. " The audio type model trainer <NUM> may store trained audio type data in the storage for training audio type data <NUM>.

The task classification model trainer <NUM> trains task classification models using received training data. The training data for task classification may include a set of acoustic features of one or more sub-streams of audio data and a correct classification for importance, urgency, and priority. The classification for priority may include more than two classes of priority for ranking. Trained data for classification may include a set of rules or conditions (i.e., a trained model) of values for acoustic features and a designated class. Additionally or alternatively, the trained data for classification may include a set of trained parameters (or a trained model) for a neural network that receives embeddings of audio data as input and determines one or more classes of the audio data for importance, urgency, and priority.

The model deployer <NUM> deploys the trained audio type data and the trained data for classification to the task manager <NUM> by storing the trained audio type data in the storage for the trained audio type models <NUM> and the trained data for task classification in the storage for trained task classification models <NUM>.

As will be appreciated, the various methods, devices, applications, features, etc., described with respect to <FIG> are not intended to limit the system <NUM> to being performed by the particular applications and features described. Accordingly, additional configurations may be used to practice the methods and systems herein and/or features and applications described may be excluded without departing from the methods and systems disclosed herein.

<FIG> illustrates an example of data structures of determining importance and urgency of a task based on acoustic features of audio data in accordance with aspects of the present disclosure. The data structure <NUM> in the example includes three types of data structures: an audio stream input <NUM>, a set of audio type <NUM> and acoustic features <NUM>, and a task <NUM> and its task class <NUM>.

The audio stream input <NUM> represent audio input data from the client device <NUM>, received by the audio receiver <NUM> of the task manager <NUM>. The audio stream input <NUM> includes multiple audio sub-streams (e.g., audio sub-stream A 212A, audio sub-stream B 212B, and audio sub-stream C 212C, for example). Each audio sub-stream includes a set of audio signals during a time duration of the audio stream input <NUM>. For example, the set of audio signals may include amplitude values for respective audio frequencies. In aspects, each of the audio sub-streams represents an audio data stream of a distinct sound. For example, the distinct sound may be a voice, a siren of a fire engine, people's voices that are indistinguishable, a sound of hand clapping, a gun shooting, and others. Sound processing technologies (e.g., noise canceling, speaker recognition) may identify and isolate various types of audio into distinct audio sub-streams.

The audio type <NUM> includes one or more types of the respective audio sub-streams. For audio sub-stream A 212A, audio type <NUM> includes a foreground voice sound with a speech text of "Call Mom. " Acoustic features <NUM> of the audio sub-stream A 212A include a temp of a value <NUM>, a beat at <NUM>, a rhythm at <NUM>, and a volume at <NUM>. In aspects, a larger value of each acoustic feature indicates a larger magnitude of the feature. Audio sub-stream B 212B translates into an audio type of a siren in background, with acoustic features <NUM> of a tempo at <NUM>, a beat at <NUM>, a step of <NUM>, a rhythm at <NUM>, and a volume at <NUM>. Audio sub-stream C 212C translates into an audio type of people voices (indistinguishable) in background, with acoustic features of a tempo at <NUM>, a beat at <NUM>, a step at <NUM>, a rhythm at <NUM>, and a volume at <NUM>.

Based on the audio type <NUM> of the combination of the audio type <NUM> across the multiple sub-streams of the audio steam input <NUM>, the present invention generates a task <NUM> of, for example, "Call Mom" that is to place a telephone call to Mom as the callee (or a destination of the call). The present invention further determines importance of the task as a task class based on the set of acoustic features <NUM> across the multiple audio sub-streams of the audio stream input <NUM>: "IMPORTANT. " Additionally or alternatively, the present invention determines a class associated with urgency and priority of the task based on acoustic features <NUM>. For example, the task <NUM> to "Call Mom" may be classified as "urgent" when one of the background sound sub-streams includes a siren from a fire engine. The task classifier (an ML or an NN) <NUM> may be trained to determine such that calling mother when the received audio input includes a siren from a fire engine in the background is urgent. In some other aspects, the task <NUM> to "Call Mom" may be "not urgent" when the received audio stream input <NUM> does not include a siren in any of the audio sub-streams. In some other aspects, the classification may take into account a context of the task <NUM> (e.g., making a telephone call to Mom), in addition to the acoustic features.

<FIG> illustrates an example of data structures of determining importance and urgency of a task by extracting speech data and context of audio data in accordance with aspects of the present disclosure. Additionally or alternatively, the present disclosure extracts, from the audio stream input <NUM>, speech data 312A and a sound context 312B. In aspects, the speech data 312A may be based on the foreground sound of the audio stream input <NUM>. The sound context 312B may be based on sound acoustics in the background audio sub-streams. The present invention generates a task <NUM> (e.g., to initiate a phone call, for example) with a command <NUM> to "Call Mom. " Concurrently, the present disclosure determines a task class <NUM> as being IMPORTANT.

In aspects, the present disclosure converts the audio stream input <NUM> into a set of embeddings (i.e., multiple-dimensional vectors). The task classifier (e.g., using a neural network with a trained model) <NUM> classifies the sound context into one or more classes that indicate importance, urgency, and priority of the task. In the example of <FIG>, the task class <NUM> indicates "important. " Additionally or alternatively, the task class <NUM> may indicate other values including "not important," "urgent," "not urgent," and a priority level (e.g., from a value ranging from <NUM> to <NUM> in the order of priority of the task).

<FIG> illustrates an example of data structures of determining importance and urgency of a task in accordance with aspects of the present disclosure. <FIG> illustrates values of respective classifications of audio stream input: task classification <NUM>, task ranking <NUM>, and classification rules <NUM>.

Task classification <NUM> may include classes based on importance <NUM> and urgency <NUM>. Classes of importance <NUM> may be one of "important" and "not important. " Classes of urgency <NUM> may be one of "urgent" and "not urgent.

Task ranking <NUM> may include ranking scores to determine a priority of a task. An urgency score <NUM> takes a value from <NUM> and <NUM> with the <NUM> being the most urgent for the task. An importance score <NUM> takes a value between <NUM> and <NUM> with the <NUM> being the most important task.

Classification rules <NUM> include a set of rules to classify tasks. In aspects, the present disclosure determines a classification based on a set of conditions. As an example, the classification rule <NUM> indicates three acoustic features items as a set of conditions to determine a class. The exemplar classification rule <NUM> indicates that a foreground pitch greater than <NUM>, a foreground volume greater than <NUM>, and a background audio type of "siren. " The rule specifies that the set of conditions translate into a class that indicates "urgent" and "important. " In aspects, the classification rules <NUM> may be used to train a machine learning model for the task classifier <NUM>.

<FIG> illustrates an example of method of determining importance and urgency of a task based on acoustic features of audio data associated with the task using a machine learning model in accordance with aspects of the present disclosure.

A general order of the operations for the method <NUM> is shown in <FIG>. Generally, the method <NUM> begins with start operation <NUM> and ends with end operation <NUM>. The method <NUM> may include more or fewer steps or may arrange the order of the steps differently than those shown in <FIG>. The method <NUM> can be executed as a set of computer-executable instructions executed by a computer system and encoded or stored on a computer readable medium. Further, the method <NUM> can be performed by gates or circuits associated with a processor, an ASIC, an FPGA, a SOC or other hardware device. Hereinafter, the method <NUM> shall be explained with reference to the systems, components, devices, modules, software, data structures, data characteristic representations, signaling diagrams, methods, etc., described in conjunction with <FIG>, <FIG>, <FIG>, <FIG>, <FIG>, <FIG>, and <FIG>.

Following start operation <NUM>, the method <NUM> begins with receive operation <NUM>, which receives audio training data including sub-streams of audio data and acoustic features with correct audio types and task classes. For example, the audio training data may include audio data with a person's voice shouting a phrase "Call Mom" in the foreground audio while there is a siren from a fire engine and people's indistinguishable voices, both in the background audio.

Train operation <NUM> trains a machine learning model using the received audio training data. In particular, the machine learning model for determining an audio type may be trained using the exemplar training data to train a voice shouting "Call Mom. " Another machine learning model may be trained for classifying the task into importance, urgency, and priority. A set of acoustic features of the training data may be used to train the machine learning model to determine a class of the task "Call Mom" to be "important" when the acoustic features of the foreground voice includes a large volume at high pitch (i.e., "shouting or screaming. Additionally the train operation <NUM> train the machine learning model to classify the task "Call Mom" to be "urgent" when the audio input includes a siren by a fire engine in the background. In aspects, the steps from <NUM> to <NUM> are for training models for determining audio types and a class of the task. In some aspects, the train operation <NUM> trains the machine learning model for task classifications using training data where the audio stream data includes a foreground audio type of a speech with a word "Stop" with acoustic features of the foreground audio includes a high pitch, a short bursting rhythm, and a high intensity. The training data may specify a set of classes "important" and "urgent. " Accordingly, use of the trained task classification model may cause the task performer to immediately stop ongoing task (e.g., playing a music) without confirming the command with the user.

Receive operation <NUM> receives audio stream input. In aspects, the audio receiver <NUM> of the task manager <NUM> may receive the audio stream input. The audio stream input may include multiple audio sub-streams, each representing a distinct type of audio data (e.g., a foreground voice, a background siren, etc.).

Generate operation <NUM> generates audio sub-streams based on the received audio stream input. The generate operation <NUM> may split the received audio stream input into sub-streams by identifying distinct types of audio data and splitting the audio stream into a set of sub-streams. In aspects, the generate operation <NUM> uses a technology that is similar to identify noises in a stream of audio or sound used in noise cancellation to identify and generate sub-streams of the audio stream. In some other aspects, the client device may include plurality of audio input device (e.g., directional microphones) to receive audio data and identify distinct sources of the sound based on directions of the audio input devices. Audio stream data may be separated into sub-streams based on audio sources identified by the directional audio input.

In aspects, the generation operation <NUM> splits the received audio stream input to a fixed time length (e.g., <NUM> seconds) segments. Log-mel filter bank (LMFB) features may be used as audio features from the received audio stream input. An input audio waveform may be analyzed with Sparse Fast Fourier Transform (SFFT) points in a fixed window size of samples and a frameshift (e.g., <NUM> SFFT points, a window size of <NUM> samples, and a frameshift of <NUM> samples). In aspects, use of some software packages or libraries for audio analysis (e.g., Librosa, See, <NPL>) may determine LMFB features. The Hidden Markov Toolkit (HTK) formula definition of the Mel scale may be adopted to determine a pitch of the audio stream input. Sampling rates may depend on the model size constraints. For example, there may be two tasks for classification of Urgency and Importance. There may be a final spectrogram which has <NUM>-time bins and the number of frequency bins is <NUM> in both tasks. Log-mel deltas and delta-deltas without padding may also be computed, thereby reducing the number of time samples to <NUM> samples. Accordingly, this results in the final tensor size of 423x128x3. In aspects, the generation operation <NUM> may normalize or scale each feature value into a value between zero and one before providing the speech feature tensors into the Convolutional Neural Network (CNN) classifier.

The input tensors are fed into distinct classifiers that use the fully convolutional neural network (FCNN). For example, FCNN may include <NUM> stacked convolutional layers with small kernel sizes (e.g., VGG architecture, See, <NPL>. Each convolutional layer is followed by a Batch Normalization operation and the rectified linear activation function (e.g., the Rectified Linear Unit (ReLU) activation layer). In the convolutional layers <NUM> to <NUM>, a Dropout may be used to alleviate possible issues with overfitting after the fourth and eighth Rectified Linear Unit (ReLU) layers a <NUM> x <NUM> max-pooling layer is appended. In addition for each output channel attention is applied over each output channel of the last convolutional layer. A global pooling layer, followed by a softmax layer generates the final classification decision for each task.

For example, an audio sub-stream may represent a foreground voice with a speech input of "Call Mom. " Another audio sub-stream may represent a background sound of a siren from a fire engine. Yet another audio sub-stream may represent a background sound of people talking but words spoken by one person in the background are indistinguishable from words spoken by another in the background.

Determine operation <NUM> determines acoustic features of the respective audio sub-streams. The acoustic features may include but not limited to a tempo, a beat, a rhythm, a step, and a volume of the respective audio sub-streams. In some aspects, an audio with a speech voice type may include a pitch and a speed of the speech as acoustic features to determine whether the speaker was shouting, screaming, or talking in a casual manner.

Determine operation <NUM> determines audio types of the respective audio sub-streams. The audio types may include whether an audio sub-stream is a foreground audio or a background audio. The audio types may also include a voice, a siren from a fire engine or a first responder vehicle, indistinguishable voices of a crowd in the background, a crashing sound of objects, and others. An audio stream input may include more than one audio sub-streams. Each audio sub-stream may be associated with an audio type that is distinct from other audio sub-streams. The determine operation <NUM> may determine the audio type using a machine learning model. The machine learning model may have been trained in the train operation <NUM>.

Generation operation <NUM> generates a task based on the audio input. For example, when the audio input includes a foreground audio of a speech voice and the speech text is "Call Mom," the generation operation <NUM> may generate a task for initiating a telephone call to Mom using a predetermined phonebook. In some other aspects, the generation operation <NUM> may generate a task based on the background audio of the audio input. For example, the digital assistant with the client device is operating in a vacant house. The audio input may include a background yet high-volume audio of a siren from a fire engine and a sound of someone banging on the door of the house. The generation operation <NUM> may automatically generate a task to call the owner of the vacant house to notify the owner a possible emergency at the vacant house. The steps from <NUM> to <NUM> correspond to generating a task <NUM>. Additionally, or alternatively, the generation operation <NUM> may receive an existing task for revising a class of the existing task, for example. The generation operation <NUM> may interact with external applications that manage tasks (e.g., calendar applications, to-do list applications, and appointment scheduler applications).

Classify operation <NUM> classifies the task based on its importance, urgency, and priority based on sound acoustics. In aspects, the classify operation <NUM> may determine a class of the task. The classify operation <NUM> may use a learnt machine learning model for classifying a task based on acoustic features. For example, a task for placing a call to Mom may include a "Call Mom" voice speech in the foreground sound with a high volume and high pitch (e.g., screaming) acoustic features along with a background sound of a siren from a fire engine based on the acoustic features. The classify operation <NUM> classifies the task as "IMPORTANT" and "URGENT" based on the combination of acoustic features. In aspects, a combination of a foreground voice sound "Call <NUM>" with acoustic features of the background noise indicating alarms, sirens, explosions, etc., would be more important than the same foreground voice sound without background noises. Classify operation <NUM> classifies the task accordingly by analyzing both the foreground audio (e.g., voice) data and acoustics of the background audio data. In some other aspects, the classifying operation <NUM> classifies the task to stop playing music (e.g., on a smart speaker) as "URGENT" when the foreground voice speech is "STOP THAT" with its acoustic features (e.g., a high pitch, unstable tone, a speed above a predetermined threshold, etc.) of the voice sound indicating a high level of stress by the speaker. This way, the classify operation <NUM> may classify, for example, a voice speech with certain acoustic features as shouting under stress, thus the task is URGENT. A level of stress may be among attributes of speech or voice data as determined based on acoustic features of the voice data.

In aspects, there may be one classifier (i.e., classification Curg task) for urgency (where urgency classes outcome are Furg). Another classifier may classify importance (i.e., Cimp task importance classes outcome labels are Fimp). Results of the two classifiers may be combined using the following equation (<NUM>):
<MAT>.

In aspects, the training of splitting the audio stream input may be based on training data with recommendation of correct answers. For example, there may be <NUM>,<NUM> audio clips for training models marked with labels of classes for "Urgency" and "Importance. " There may be <NUM>,<NUM> test audio clips. Data augmentation techniques may be used to improve models. Stochastic gradient descent (SGD) with a cosine-decay-restart learning rate scheduler may be used to train the models. The maximum and minimum learning rates may be <NUM> and 1e-<NUM>, respectively. In aspects without validation data, the average output of models when learning rate hits around the minimum number may be used. For example, Pyrotch or Keras may be used to implement the CNN-based models for classifications. Upon completing the training, the networks may be used by applying the networks to smaller audio segments (e.g., <NUM> milliseconds) for inference.

In some other aspects, there may be an audio stream without foreground sound but only a set of background sounds. The generation operation <NUM> generated a task to make a telephone call to an owner of a house. The set of background sounds may include a siren from a fire engine and another background sound of someone banging strongly on the door. The classify operation <NUM> may classify the task as "IMPORTANT" and "URGENT. " In some aspects, the classify operation <NUM> may determine a level of priority for performing a task based on acoustic features of the audio stream.

Perform operation <NUM> performs the task at the classified level of importance and/or urgency. The perform operation <NUM> may interact with external servers and devices to perform the tasks based on a level of importance, urgency, and priority as classified by the classify operation <NUM>. For example, the perform operation <NUM> may interact with an external telephony server to initiate a phone call with a determined level of urgency at a timing that reflects a level of importance. The perform operation <NUM> performs a task that is both important and urgent may start performing the task at early as possible, before performing other tasks. In some other aspects, the perform operation <NUM> may interact with an emails sever to transmit emails. In yet some other aspects, the perform operation <NUM> may save the task in a calendar application or a task management application.

<FIG> is example of method of determining importance and urgency of a task based on acoustic features of audio data associated with the task using embedding of the audio data and a neural network in accordance with aspects of the present disclosure.

Following start operation <NUM>, the method <NUM> begins with receive operation <NUM>, which receives training data. The training data include sub-streams of audio data and acoustic features with correct audio types and task classes. For example, the audio training data may include audio data with a person's voice shouting a phrase "Call Mom" in the foreground audio while there is a siren from a fire engine and people's indistinguishable voices, both in the background audio.

Train operation <NUM> trains a task classifier neural network using the received audio training data. In particular, the neural network for determining an audio type may be trained using the exemplar training data to train a voice shouting "Call Mom. " Another neural network may be trained for classifying the task into importance, urgency, and priority. A set of acoustic features of the training data may be used to train the machine learning model to determine a class of the task "Call Mom" to be IMPORTANT when the acoustic features of the foreground voice includes a large volume at high pitch (i.e., "shouting or screaming. Additionally the train operation <NUM> train the neural network to classify the task "Call Mom" to be URGENT when the audio input includes a siren by a fire engine in the background. In aspects, the steps from <NUM> to <NUM> are for training models for determining audio types and classifying tasks.

Generate operation <NUM> generates audio sub-streams based on the received audio stream input. The generate operation <NUM> may split the received audio stream input into sub-streams by identifying distinct types of audio data and splitting the audio stream into a set of sub-streams. For example, an audio sub-stream may represent a foreground voice with a speech input of "Call Mom. " Another audio sub-stream may represent a background sound of a siren from a fire engine. Yet another audio sub-stream may represent a background sound of people talking but words spoken by one person in the background are indistinguishable from words spoken by another in the background.

Generate operation <NUM> generates embeddings of respective audio sub-streams as input to the classifier (i.e., a neural network). In aspects, the embeddings represents multi-dimensional vectors. Each dimension of the vectors associated with an acoustic feature and other parameters that characterize the audio stream and its audio sub-streams.

Generate operation <NUM> generates a task based on the audio input. For example, when the audio input includes a foreground audio of a speech voice and the speech text is "Call Mom," the generation operation <NUM> may generate a task for initiating a telephone call to Mom using a predetermined phonebook. In some other aspects, the generation operation <NUM> may generate a task based on the background audio of the audio input. For example, the digital assistant with the client device is placed in a vacant house. The audio input may include a background yet high-volume audio of a siren from a fire engine and a sound of someone banging on the door of the house. The generation operation <NUM> may automatically generate a task to call the owner of the vacant house to notify the owner a possible emergency at the vacant house. The steps from <NUM> to <NUM> correspond to generating a task <NUM>. Additionally, or alternatively, the generation operation <NUM> may receive an existing task for revising a class of the existing task, for example. The generation operation <NUM> may interact with external applications that manage tasks (e.g., calendar applications, to-do list applications, and appointment scheduler applications).

The classify operation <NUM> classifies the task based on its importance, urgency, and priority based on sound acoustics. In aspects, the classify operation <NUM> may determine a class of the task. The classify operation <NUM> may use a neural network (e.g., a multi-layered, recurring neural network) using the embeddings as input for determining a class of a task based on acoustic features. For example, a task for placing a call to Mom may include a "Call Mom" voice speech in the foreground sound with a high volume and high pitch (e.g., screaming) acoustic features along with a background sound of a siren from a fire engine based on the acoustic features. The classify operation <NUM> classifies the task as "IMPORTANT" and "URGENT" based on the combination of acoustic features. The neural network may use a trained set of parameters as trained by the train operation <NUM>.

Perform operation <NUM> performs the task according to the determined class. The perform operation <NUM> may interact with external servers and devices to perform the tasks based on a level of importance, urgency, and priority as classified by the classify operation <NUM>. For example, the perform operation <NUM> may interact with an external telephony server to initiate a phone call with a determined level of urgency at a timing that reflects a level of importance. The perform operation <NUM> performs a task that is both important and urgent may start performing the task at early as possible, before performing other tasks. In some other aspects, the perform operation <NUM> may interact with an emails sever to transmit emails. In yet some other aspects, the perform operation <NUM> may save the task in a calendar application or a task management application. In aspects, method <NUM> may end with end operation <NUM>. In aspects, the perform operation <NUM> may update a position of the task on a to-do list in a to-do list application according to the class. The position of the task may indicate a ranked level of importance and / or urgency of tasks on the to-do list.

As should be appreciated, operations <NUM>-<NUM> are described for purposes of illustrating the present methods and systems and are not intended to limit the disclosure to a particular sequence of steps, e.g., steps may be performed in different order, additional steps may be performed, and disclosed steps may be excluded without departing from the present disclosure.

<FIG> is a block diagram illustrating physical components (e.g., hardware) of a computing device <NUM> with which aspects of the disclosure may be practiced. The computing device components described below may be suitable for the computing devices described above. In a basic configuration, the computing device <NUM> may include at least one processing unit <NUM> and a system memory <NUM>. Depending on the configuration and type of computing device, the system memory <NUM> may comprise, but is not limited to, volatile storage (e.g., random access memory), non-volatile storage (e.g., read-only memory), flash memory, or any combination of such memories. The system memory <NUM> may include an operating system <NUM> and one or more program tools <NUM> suitable for performing the various aspects disclosed herein such. The operating system <NUM>, for example, may be suitable for controlling the operation of the computing device <NUM>. Furthermore, aspects of the disclosure may be practiced in conjunction with a graphics library, other operating systems, or any other application program and is not limited to any particular application or system. This basic configuration is illustrated in <FIG> by those components within a dashed line <NUM>. The computing device <NUM> may have additional features or functionality. For example, the computing device <NUM> may also include additional data storage devices (removable and/or non-removable) such as, for example, magnetic disks, optical disks, or tape. Such additional storage is illustrated in <FIG> by a removable storage device <NUM> and a non-removable storage device <NUM>.

As stated above, a number of program tools and data files may be stored in the system memory <NUM>. While executing on the at least one processing unit <NUM>, the program tools <NUM> (e.g., an application <NUM>) may perform processes including, but not limited to, the aspects, as described herein. The application <NUM> includes an audio receiver <NUM>, an acoustic type / feature determiner <NUM>, a task generator <NUM>, a task classifier <NUM>, and a task processor <NUM>, as described in more detail with regard to <FIG>. Other program tools that may be used in accordance with aspects of the present disclosure may include electronic mail and contacts applications, task management applications, calendar applications, word processing applications, spreadsheet applications, database applications, slide presentation applications, drawing or computer-aided application programs, etc..

Furthermore, aspects of the disclosure may be practiced in an electrical circuit comprising discrete electronic elements, packaged or integrated electronic chips containing logic gates, a circuit utilizing a microprocessor, or on a single chip containing electronic elements or microprocessors. For example, aspects of the disclosure may be practiced via a system-on-a-chip (SOC) where each or many of the components illustrated in <FIG> may be integrated onto a single integrated circuit. Such an SOC device may include one or more processing units, graphics units, communications units, system virtualization units and various application functionality all of which are integrated (or "burned") onto the chip substrate as a single integrated circuit. When operating via an SOC, the functionality, described herein, with respect to the capability of client to switch protocols may be operated via application-specific logic integrated with other components of the computing device <NUM> on the single integrated circuit (chip). Aspects of the disclosure may also be practiced using other technologies capable of performing logical operations such as, for example, AND, OR, and NOT, including but not limited to mechanical, optical, fluidic, and quantum technologies. In addition, aspects of the disclosure may be practiced within a general purpose computer or in any other circuits or systems.

The computing device <NUM> may also have one or more input device(s) <NUM>, such as a keyboard, a mouse, a pen, a sound or voice input device, a touch or swipe input device, etc. The output device(s) <NUM> such as a display, speakers, a printer, etc. may also be included. The aforementioned devices are examples and others may be used. The computing device <NUM> may include one or more communication connections <NUM> allowing communications with other computing devices <NUM>. Examples of suitable communication connections <NUM> include, but are not limited to, radio frequency (RF) transmitter, receiver, and/or transceiver circuitry; universal serial bus (USB), parallel, and/or serial ports.

Computer storage media may include volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information, such as computer readable instructions, data structures, or program tools.

Communication media may be embodied by computer readable instructions, data structures, program tools, or other data in a modulated data signal, such as a carrier wave or other transport mechanism, and includes any information delivery media.

<FIG> and <FIG> illustrate a computing device or mobile computing device <NUM>, for example, a mobile telephone, a smart phone, wearable computer (such as a smart watch), a tablet computer, a laptop computer, and the like, with which aspects of the disclosure may be practiced. In some aspects, the client utilized by a user may be a mobile computing device. With reference to <FIG>, one aspect of a mobile computing device <NUM> for implementing the aspects is illustrated. In a basic configuration, the mobile computing device <NUM> is a handheld computer having both input elements and output elements. The mobile computing device <NUM> typically includes a display <NUM> and one or more input buttons <NUM> that allow the user to enter information into the mobile computing device <NUM>. The display <NUM> of the mobile computing device <NUM> may also function as an input device (e.g., a touch screen display). If included as an optional input element, a side input element <NUM> allows further user input. The side input element <NUM> may be a rotary switch, a button, or any other type of manual input element. In alternative aspects, mobile computing device <NUM> may incorporate more or less input elements. For example, the display <NUM> may not be a touch screen in some aspects. In yet another alternative aspect, the mobile computing device <NUM> is a portable phone system, such as a cellular phone. The mobile computing device <NUM> may also include an optional keypad <NUM>. Optional keypad <NUM> may be a physical keypad or a "soft" keypad generated on the touch screen display. In various aspects, the output elements include the display <NUM> for showing a graphical user interface (GUI), a visual indicator <NUM> (e.g., a light emitting diode), and/or an audio transducer <NUM> (e.g., a speaker). In some aspects, the mobile computing device <NUM> incorporates a vibration transducer for providing the user with tactile feedback. In yet another aspect, the mobile computing device <NUM> incorporates input and/or output ports, such as an audio input (e.g., a microphone jack), an audio output (e.g., a headphone jack), and a video output (e.g., a HDMI port) for sending signals to or receiving signals from an external device.

<FIG> is a block diagram illustrating the architecture of one aspect of computing device, a server (e.g., a task manager <NUM> and a model generator <NUM> in <FIG>), a mobile computing device, etc. That is, the mobile computing device <NUM> can incorporate a system <NUM> (e.g., a system architecture) to implement some aspects. The system <NUM> can implemented as a "smart phone" capable of running one or more applications (e.g., browser, e-mail, calendaring, contact managers, messaging clients, games, and media clients/players). In some aspects, the system <NUM> is integrated as a computing device, such as an integrated digital assistant (PDA) and wireless phone.

One or more application programs <NUM> may be loaded into the memory <NUM> and run on or in association with the operating system <NUM>. Examples of the application programs include phone dialer programs, e-mail programs, information management (PIM) programs, word processing programs, spreadsheet programs, Internet browser programs, messaging programs, and so forth. The system <NUM> also includes a non-volatile storage area <NUM> within the memory <NUM>. The non-volatile storage area <NUM> may be used to store persistent information that should not be lost if the system <NUM> is powered down. The application programs <NUM> may use and store information in the non-volatile storage area <NUM>, such as e-mail or other messages used by an e-mail application, and the like. A synchronization application (not shown) also resides on the system <NUM> and is programmed to interact with a corresponding synchronization application resident on a host computer to keep the information stored in the non-volatile storage area <NUM> synchronized with corresponding information stored at the host computer. As should be appreciated, other applications may be loaded into the memory <NUM> and run on the mobile computing device <NUM> described herein.

The visual indicator <NUM> (e.g., LED) may be used to provide visual notifications, and/or an audio interface <NUM> may be used for producing audible notifications via the audio transducer <NUM>. In the illustrated configuration, the visual indicator <NUM> is a light emitting diode (LED) and the audio transducer <NUM> is a speaker. These devices may be directly coupled to the power supply <NUM> so that when activated, they remain on for a duration dictated by the notification mechanism even though the processor <NUM> and other components might shut down for conserving battery power. The LED may be programmed to remain on indefinitely until the user takes action to indicate the powered-on status of the device. The audio interface <NUM> is used to provide audible signals to and receive audible signals from the user. For example, in addition to being coupled to the audio transducer <NUM>, the audio interface <NUM> may also be coupled to a microphone to receive audible input, such as to facilitate a telephone conversation. In accordance with aspects of the present disclosure, the microphone may also serve as an audio sensor to facilitate control of notifications, as will be described below. The system <NUM> may further include a video interface <NUM> that enables an operation of an on-board camera <NUM> to record still images, video stream, and the like.

Claim 1:
A computer-implemented method of determining a class of a task based on acoustic features (<NUM>) of audio data (<NUM>) associated with the task, the method comprising:
receiving audio input, wherein the audio input is associated with a task;
generating, based on the received audio input, a plurality of audio sub-streams;
determining one or more acoustic features (<NUM>) of the received audio input corresponding to one or more of the generated plurality of audio sub-streams;
determining, based on the plurality of audio sub-streams, an audio type;
generating, based on the audio type, the task;
determining, based on the one or more acoustic features (<NUM>), a class of the task using a trained machine learning model for classifying tasks, wherein the class of the task is associated with importance and urgency of the task; and
performing the task according to the class of the task.