Patent Description:
<CIT> describes that in certain implementations, follow-up responses may be provided for prior natural language inputs of a user. As an example, a natural language input associated with a user may be received at a computer system. A determination of whether information sufficient for providing an adequate response to the natural language input is currently accessible to the computer system may be effectuated. A first response to the natural language input (that indicates that a follow-up response will be provided) may be provided based on a determination that information sufficient for providing an adequate response to the natural language input is not currently accessible. Information sufficient for providing an adequate response to the natural language input may be received. A second response to the natural language input may then be provided based on the received sufficient information.

<CIT> describes that a query-processing system processes an input audio stream that represents a succession of queries spoken by a user. The query-processing system listens continuously to the input audio stream, parses queries and takes appropriate actions in mid-stream. In some embodiments, the system processes queries in parallel, limited by serial constraints. In some embodiments, the system parses and executes queries while a previous query's execution is still in progress. To accommodate users who tend to speak slowly and express a thought in separate parts, the query-processing system halts the outputting of results corresponding to a previous query if it detects that a new speech utterance modifies the meaning of the previous query.

<CIT> describes a speech recognition terminal device communicable with a speech recognition server includes a request device for requesting the speech recognition server to carry out the speech recognition of an acquired speech command, a prediction device for predicting a present delay time until a result of the requested speech recognition is obtained from the speech recognition server, a determination device for determining a filler word with a time length in accordance with the predicted present delay time, a filler speaking device for outputting the determined filler word during a waiting time until the result of the requested speech recognition is obtained from the speech recognition server, and a response device for responding to the user when the result of the speech recognition is acquired from the speech recognition server.

It is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.

Embodiments of the present disclosure propose method and apparatus for responding in a voice conversation by an electronic conversational agent. A voice input may be received in an audio upstream. In response to the voice input, a primary response and at least one supplementary response may be generated. A primary voice output may be generated based on the primary response. At least one supplementary voice output may be generated based on the at least one supplementary response. The primary voice output and the at least one supplementary voice output may be provided in an audio downstream, wherein the at least one supplementary voice output is provided during a time period adjacent to the primary voice output in the audio downstream.

It should be noted that the above one or more aspects comprise the features hereinafter fully described and particularly pointed out in the claims. The following description and the drawings set forth in detail certain illustrative features of the one or more aspects. These features are only indicative of the various ways in which the principles of various aspects may be employed, and this disclosure is intended to include all such aspects and their equivalents.

The disclosed aspects will hereinafter be described in connection with the appended drawings that are provided to illustrate and not to limit the disclosed aspects.

The present disclosure will now be discussed with reference to several example implementations. It is to be understood that these implementations are discussed only for enabling those skilled in the art to better understand and thus implement the embodiments of the present disclosure, rather than suggesting any limitations on the scope of the present disclosure.

Usually, during a full duplex voice conversation between a chatbot and a user, the chatbot maintains two timely-synchronized audio streams: one continuous audio upstream for receiving the user's voice inputs, and another continuous audio downstream for providing voice outputs generated by the chatbot. Accordingly, the chatbot and the user may have simultaneous and bidirectional communications with each other. Through recognizing voice information in a voice input in the audio upstream in real time, the chatbot may start to prepare voice outputs to be provided in the audio downstream, even when the user has not finished the voice input. The full duplex voice conversation enables the chatbot to generate responses and provide voice outputs more quickly.

After receiving a message input by a user, a traditional chatbot only provides one single response to the message. That is, during a conversation between the chatbot and the user, the chatbot provides responses in a <NUM>:<NUM> parity interaction mode. However, when applying a full duplex voice conversation in practice, its simultaneous and bidirectional fashions of communication sometimes require more complex non-parity interaction modes rather than the <NUM>: <NUM> parity interaction mode. For example, when two humans are talking with each other over a telephone or face to face, their conversation doesn't have to be kept in <NUM>: <NUM> parity. One party may talk more while the other party may listen more and talk less. Also, it is very common for a human to speak a short acknowledge or phrase firstly, and later supplement with more detailed contents after thinking over carefully. It would be helpful to improve user experience of a chatbot if the chatbot may work in a manner similar with the human-to-human conversations as mentioned above.

Embodiments of the present disclosure propose a multi-phrase response mechanism for a chatbot to respond in a voice conversation, wherein the voice conversation may be any types of voice conversation that are based on audio streams, such as, a full duplex voice conversation. Herein, the multi-phrase response mechanism may refer to responding to a single voice input, through generating more than one response, e.g., a primary response and at least one supplementary response, and accordingly providing more than one voice output, e.g., a primary voice output and at least one supplementary voice output. The primary response may be a direct response to the voice input, e.g., an answer to a question in the voice input. While the at least one supplementary response may be an addition or supplementation to the primary response, and intends for filling an idle time period during which the primary response and the primary voice output are prepared or during which no further valid voice input is received, in order to make the voice conversation more smoothly and make the chatbot's behaviors more anthropomorphic. Through the multi-phrase response mechanism, when receiving a voice input from a user in an audio upstream, the chatbot may, in response to the voice input, generate a primary response and at least one supplementary response. The chatbot may provide at least one supplementary voice output corresponding to the at least one supplementary response in an audio downstream before or after providing a primary voice output corresponding to the primary response. The at least one supplementary voice output is provided during a time period adjacent to the primary voice output in the audio downstream.

Through providing the supplementary voice output before the primary voice output, the latency that the user feels may be shortened. Through providing the supplementary voice output after the primary voice output, non-parity experience of the voice conversation may be optimized. Accordingly, the embodiments of the present disclosure could make a voice conversation between the chatbot and the user more natural and closer to human-to-human conversations.

It should be appreciated that the term "voice conversation" used hereinafter refers to any types of voice conversation that are based on audio streams, e.g., audio upstream and/or audio downstream. Although such "voice conversation" is referred to as "full duplex voice conversation" in some paragraphs below, it may be also referred to as various other expressions, e.g., "stream-based voice conversation", "realtime voice conversation", etc., and all these expressions may be used interchangeably. Moreover, in the following discussions, the term "user" may refer to a chatting participant who is consuming an automatic chatting service provided by a chatbot, and may be a real human, or a virtual human, e.g., an electronic chatting participant.

<FIG> illustrates an exemplary application scenario <NUM> of a full duplex voice conversation. In the scenario <NUM>, an electronic conversational agent <NUM>, e.g., a chatbot, is performing a full duplex voice conversation with a user <NUM>. The chatbot <NUM> receives voice inputs from the user <NUM> in a continuous audio upstream, and provides voice outputs in a continuous audio downstream.

The chatbot <NUM> may comprise a continuous speech recognition (CSR) module <NUM>. Speech recognition may take an audio clip as input and a recognized text as output. The CSR module <NUM> may perform continuous speech recognition on the continuous audio upstream in real time, wherein the audio upstream comprises a plurality of audio clips corresponding to voice inputs from the user <NUM>. The CSR module <NUM> may provide two types of text outputs, e.g., intermediate SR result and final SR result. An intermediate SR result represents a partial recognition of a part of a voice input, e.g., a part of a sentence spoken by the user, which is usually yielded after each new syllable, character, word, or phrase, etc. is spoken. A final SR result represents a complete recognition of the voice input after the user <NUM> has finished the voice input, e.g., a whole sentence is spoken. Thus, each sentence spoken by the user <NUM> may yield a plurality of intermediate SR results and one final SR result.

The chatbot <NUM> may comprise a language prediction (LP) module <NUM> which tries to predict a whole text of the user's voice input based on an intermediate SR result from the CSR module <NUM>. For example, the LP module <NUM> may try to complement a part of a sentence with possible remaining parts so as to make the sentence complete, thus enabling the chatbot <NUM> to "start thinking" even before the user <NUM> finishes the whole voice input.

The chatbot <NUM> may comprise a conversation engine (CE) module <NUM>. Traditional conversation engine may be applied for generating a chatbot's text response based on a text input from a user. The CE module <NUM> may take either a language-predicted text sentence from an intermediate SR result, or a whole text sentence from the final SR result as input, and generate corresponding candidates of text response as output.

The chatbot <NUM> may comprise a text-to-speech (TTS) conversion module <NUM> which may convert a text response generated by the CE module <NUM> into a voice output, e.g., at least one audio clip. The audio clip generated by the TTS conversion module <NUM> may be provided as voice output to the user <NUM> in the audio downstream.

In the scenario <NUM>, since the chatbot <NUM> may perform continuous speech recognition on the voice input from the user <NUM>, the chatbot <NUM> may establish a single processing thread for each of the intermediate SR results and the final SR result obtained by the CSR module <NUM>. Each thread calls the LP module <NUM>, the CE module <NUM> and the TTS conversion module <NUM> so as to generate a candidate voice output by this thread. In other words, the chatbot <NUM> maintains one or more threads in parallel during processing the voice input. Considering that the intermediate SR results are obtained earlier than the final SR result, those threads corresponding to the intermediate SR results would also generate candidate voice outputs earlier than the thread corresponding to the final SR result. Thus, if it is determined that a LP result in a certain thread matches the final SR result, i.e., the LP module <NUM> makes a correct prediction based on the intermediate SR result in this thread, a candidate voice output by this thread may be selected as a final voice output to be provided to the user <NUM>. Through this way, the chatbot <NUM> may provide a voice output to the user <NUM> more quickly, without the need of waiting for a processing result of the thread corresponding to the final SR result.

It should be appreciated that all the modules in the chatbot <NUM> as mentioned above are exemplary, and depending on specific application designs and requirements, more or less modules may be included in the chatbot <NUM>. Moreover, various existing techniques or approaches may be adopted for implementing each of the modules in the chatbot <NUM>.

<FIG> illustrates an exemplary processing flow <NUM> for responding in a full duplex voice conversation. The processing flow <NUM> may be based on the application scenario <NUM> shown in <FIG>.

A voice input <NUM> from a user may be occurring in an audio upstream. It is assumed that the user intends to speak a sentence "Tell a joke" in the voice input <NUM>.

At <NUM>, continuous speech recognition (CSR) may be performed on the voice input <NUM> during receiving the voice input <NUM> in the audio upstream.

Assuming that an intermediate SR result <NUM>, i.e., a word "Tell", is firstly obtained by the CSR, then a thread <NUM> may be established for this intermediate SR result <NUM>. In the thread <NUM>, at <NUM>, language prediction (LP) is made based on the intermediate SR result <NUM>. For example, a LP result "Tell me" may be generated based on the intermediate SR result "Tell". Then, at <NUM>, a conversation engine (CE) may be used for generating a text response based on the LP result. For example, a text response "What happened?" may be generated based on the LP result "Tell me". At <NUM>, text-to-speech (TTS) conversion may be performed on the text response obtained at <NUM>, so as to generate a candidate voice output <NUM> which is an audio clip corresponding to the text response "What happened?".

Assuming that an intermediate SR result <NUM>, i.e., a phrase "Tell a", is further obtained by the CSR, then a thread <NUM> may be established for this intermediate SR result <NUM>. In the thread <NUM>, at <NUM>, a LP result "Tell a joke" may be generated based on the intermediate SR result "Tell a". Then, a text response "Here is a funny thing. " may be generated by the CE at <NUM> based on the LP result "Tell a joke". The text response "Here is a funny thing. " is further used for generating a candidate voice output <NUM> through the TTS at <NUM>.

Assuming that a final SR result <NUM>, i.e., a sentence "Tell a joke", is further obtained by the CSR, then a thread <NUM> may be established for this final SR result <NUM>. In the thread <NUM>, a text response "Here is a funny thing. " may be generated by the CE at <NUM> based on the final SR result "Tell a joke". The text response "Here is a funny thing. " is further used for generating a candidate voice output <NUM> through the TTS at <NUM>.

The thread <NUM>, thread <NUM> and thread <NUM> in <FIG> are established sequentially along with the user is speaking more words in the voice input <NUM>. Accordingly, these threads may generate candidate voice outputs in different time points. For example, the candidate voice output <NUM> may be obtained earlier than the candidate voice output <NUM>, and the candidate voice output <NUM> may be obtained earlier than the candidate voice output <NUM>, etc..

In an implementation, after obtaining the final SR result <NUM>, a matching between this final SR result <NUM> and those LP results in other threads may be performed so as to determine which thread is based on a correct prediction of the voice input <NUM>. For example, in the example of <FIG>, it may be determined that the LP result "Tell a joke" obtained at <NUM> in the thread <NUM> matches with the final SR result <NUM>. Accordingly, the candidate voice output <NUM> by the thread <NUM> may be provided as a final voice output to the user in an audio downstream. It can be seen that the candidate voice output <NUM> by the thread <NUM> is the same as the candidate voice output <NUM> by the thread <NUM> in terms of content, but is obtained earlier than the candidate voice output <NUM>. Through providing the candidate voice output <NUM> in the audio downstream, rather than waiting for the generating of the candidate voice output <NUM>, the chatbot may respond to the voice output <NUM> more quickly.

It should be appreciated that although the example of <FIG> involves three exemplary threads established based on words recognized from the voice input <NUM>, more or less threads may be established based on at least one of syllables, characters, phrases, etc. recognized from the voice input <NUM>. Although it is shown in <FIG> that each of the threads finally generates a candidate voice output, it is also possible that not each thread has generated its candidate voice output when a matched LP result is identified from these threads or when a candidate voice output corresponding to the matched LP result is provided in the audio downstream. In this case, those threads having not finished their processes may be stopped or removed immediately.

Exemplary implementations of full duplex voice conversation have been discussed above in connection with <FIG>. It should be appreciated that the embodiments of the present disclosure discussed below are proposed not only for the exemplary implementations of the full duplex voice conversation in <FIG>, but also for any other implementations of voice conversation that are based on audio streams.

<FIG> illustrates an exemplary process <NUM> for implementing a multi-phrase response mechanism in a voice conversation according to an embodiment.

According to the process <NUM>, a chatbot may monitor an audio upstream <NUM> from a user to the chatbot. A voice input <NUM> by the user may be detected by the chatbot from the audio upstream <NUM>. During receiving the voice input <NUM>, the chatbot may apply a multi-phrase response mechanism <NUM>, so as to generate at least one voice output <NUM> which is used for responding to the voice input <NUM>. The at least one voice output <NUM> may comprise a primary voice output and at least one supplementary voice output. The at least one voice output <NUM> may be further provided to the user in an audio downstream <NUM> from the chatbot to the user.

The multi-phrase response mechanism <NUM> in <FIG> may collectively refer to any mechanisms for responding in a voice conversation according to the embodiments of the present disclosure. In an implementation, the multi-phrase response mechanism <NUM> may be a padding response mechanism <NUM>, which will be discussed in details below in connection with <FIG>. In an implementation, the multi-phrase response mechanism <NUM> may be an intent-execution response mechanism <NUM>, which will be discussed in details below in connection with <FIG>. In an implementation, the multi-phrase response mechanism <NUM> may be a collective follow-up response mechanism <NUM>, which will be discussed in details below in connection with <FIG>. In an implementation, the multi-phrase response mechanism <NUM> may be a recursive follow-up response mechanism <NUM>, which will be discussed in details below in connection with <FIG>.

It should be appreciated that although various multi-phrase response mechanisms are discussed below separately, any of these mechanisms may be combined together according to specific application designs and requirements.

In some cases, although humans may listen, think and speak in a very time-efficient manner, when the humans talk with each other during a conversation, it is still impossible to provide every reply immediately after the other party has spoken a sentence. In order to let the other party know that he or she has started thinking about the reply, it is a common practice to speak one or more padding words, e.g., "hmm", "well", "let me think", etc., before giving the final reply. Such conversation practice in human's conversation also plays an important role for the design of a chatbot so as to enable the chatbot to keep the rhythm of a voice conversation between the chatbot and a user. Through responding by padding words as an early acknowledgement during the chatbot is preparing the final response, the chatbot may let the user know that his voice input is not missed, thus avoiding unnecessary repetition of the voice input by the user. Since the padding words are just a few predetermined and fixed expression paradigms contextually independent from content in the user's voice input, it is relatively easy for the chatbot to provide an extra voice output based on the padding words immediately after the user's voice input, thus shortening the latency that the user feels before hearing the final voice output provided by the chatbot.

The embodiments of the present disclosure propose a padding response mechanism, which may be configured for, in response to a user's voice input, providing at least one supplementary voice output before a primary voice output is provided, wherein the at least one supplementary voice output is generated based on at least one padding word, and the primary voice output includes content being as a direct and substantial reply to the user's voice input. Herein, the term "word" may refer to one or more words, phrases, characters, etc..

<FIG> illustrates an exemplary voice conversation <NUM> according to an embodiment. The voice conversation <NUM> may be based on, e.g., the padding response mechanism. The voice conversation <NUM> is schematically shown in a chat window between a chatbot and a user.

The user may provide a voice input <NUM>, e.g., "Hi", in an audio upstream. The chatbot may respond by a voice output <NUM>, e.g., "Morning", in an audio downstream.

The user may further provide a voice input <NUM>, e.g., "Tell a joke", in the audio upstream. During receiving the voice input <NUM>, the chatbot may apply the padding response mechanism. Through the padding response mechanism, the chatbot may firstly provide a supplementary voice output <NUM>, e.g., "Well", and then provide a primary voice output <NUM>, e.g., "Here is a funny thing. The supplementary voice output <NUM> is generated based at least on the padding word "Well", and is provided in the audio downstream during the chatbot is preparing the primary voice output <NUM>.

It should be appreciated that the chat window in <FIG> is exemplary, and depending on specific application requirements, the chat window may be changed in various manners in terms of layout, elements being incorporated, etc. Moreover, although the voice conversation <NUM> is presented in the visual chat window, the visual chat window may also be omitted in some cases that the chatbot is implemented in a smart device without or disabling display functions, e.g., an intelligent speaker without a display screen.

<FIG> illustrates an exemplary method <NUM> for responding in a voice conversation according to an embodiment. The method <NUM> may be an exemplary implementation of the padding response mechanism.

A voice input <NUM> may be received by a chatbot from a user in an audio upstream in the voice conversation between the chatbot and the user.

During receiving the voice input <NUM>, the chatbot may perform continuous speech recognition (CSR) on the voice input <NUM>. Through the CSR, at least one intermediate SR result <NUM> may be obtained, e.g., intermediate SR result <NUM>, intermediate SR result <NUM>, etc..

As mentioned above, for each intermediate SR result, a thread may be established. In each thread, language prediction (LP) may be performed on the corresponding intermediate SR result so as to generate a LP result which may be deemed as a prediction of the voice input <NUM>. Accordingly, at least one LP result <NUM> may be generated in the method <NUM>, e.g., LP result <NUM>, LP result <NUM>, etc. Taking the intermediate SR result <NUM> as an example, a thread <NUM> may be established, in which the LP result <NUM> may be generated through performing the LP on the intermediate SR result <NUM>.

In each thread, conversation engine (CE) may be called for generating a candidate primary response based on the corresponding LP result. Accordingly, at least one candidate primary response <NUM> may be obtained in the method <NUM>, e.g., candidate primary response <NUM>, candidate primary response <NUM>, etc. Taking the thread <NUM> as an example, the candidate primary response <NUM> may be generated by the CE based on the LP result <NUM>.

In each thread, text-to-speech (TTS) conversion may be performed on the corresponding candidate primary response so as to generate a candidate primary voice output. Accordingly, at least one candidate primary voice output <NUM> may be generated in the method <NUM>, e.g., candidate primary voice output <NUM>, candidate primary voice output <NUM>, etc. Taking the thread <NUM> as an example, the candidate primary voice output <NUM> may be generated through performing the TTS conversion on the candidate primary response <NUM>.

Through the CSR, a final SR result <NUM> may be obtained, which is a SR result of the whole voice input <NUM>.

At <NUM>, a matching may be performed between the final SR result <NUM> and the at least one LP result <NUM>, so as to find out a matched LP result <NUM> and accordingly find out a matched thread. The matched LP result <NUM> may indicate a certain thread from which a primary voice output to be provided to the user shall be obtained. In other words, the matched LP result <NUM> may be used for extracting, at <NUM>, a primary voice output from the at least one candidate primary voice output <NUM>. Assuming that the LP result <NUM> in a thread <NUM> is determined as the matched LP result <NUM> and accordingly the thread <NUM> is a matched thread, then the candidate primary voice output <NUM> obtained through the thread <NUM> may be extracted as the final primary voice output to be provided to the user.

At <NUM>, the extracted primary voice output may be provided to the user in an audio downstream.

The method <NUM> may try to provide at least one supplementary voice output which is based on at least one padding word during a time period between a time point of completing the receiving of the voice input <NUM> and a time point of providing the primary voice output.

In one case, if the established threads work efficiently enough, it is possible that at the time point of completing the receiving of the voice input <NUM> or yielding the final SR result <NUM> by the CSR, a candidate primary voice output obtained through a matched thread is already ready for being provided to the user. In such case, any supplementary voice output generated from padding words would be never necessary, because the chatbot may provide the primary voice output directly.

Otherwise, in most cases, since the CE and TTS may take a rather long time to be performed, the matched thread may have not generated the candidate primary voice output yet at the time point of completing the receiving of the voice input <NUM> or yielding the final SR result <NUM> by the CSR, and thus there exists a chance to provide a supplementary voice output during an end-to-end latency which is between the time point of completing the receiving of the voice input <NUM> and the time point of providing the primary voice output. The longer the end-to-end latency could be expected by the chatbot, the more likely a supplementary voice output may be added.

In an implementation, the end-to-end latency may be calculated as CE processing time plus TTS processing time minus LP time gain, wherein the LP time gain refers to a time difference between the time point of completing the receiving of the voice input <NUM> and the time point of predicting a LP result through the LP. Expected CE processing time, i.e., estimation of CE processing time, depends on specific implementation of the CE, e.g., rule-based CE, search-based CE, generative approach-based CE, mixed model-based CE, etc. Expected TTS processing time, i.e., estimation of TTS processing time, depends on a length of text response that the CE yields.

According to the method <NUM>, when the matched LP result <NUM> is found, the matched thread is also determined. Accordingly, it may be determined at <NUM> that whether criteria for providing a supplementary voice output which is based on at least one padding word <NUM> are met. The criteria may be established based in part on the end-to-end latency caused by the matched thread and a duration of output of the at least one padding word <NUM>, wherein the duration of output of the at least one padding word <NUM> may refer to a duration of a voice output generated based on the at least one padding word <NUM> in an assumption that it is to generate the voice output based on the at least one padding word <NUM>.

If the CE in the matched thread has generated a candidate primary response by the time point that the CSR yields the final SR result <NUM>, then the end-to-end latency may be calculated as follow: <MAT> where Texpected denotes the end-to-end latency caused by the matched thread, <MAT> denotes actual CE processing time in the matched thread, <MAT> denotes an actual length of the candidate primary response generated by the CE based on the LP result in the matched thread, <MAT> denotes expected TTS processing time in the matched thread, and <MAT> denotes an actual LP time gain in the matched thread.

Otherwise, if the CE in the matched thread has not generated a candidate primary response yet by the time point that the CSR yields the final SR result <NUM>, an estimation of the CE processing time as well as an estimation of a length of the candidate primary response to be generated are also necessary, and then the end-to-end latency may be calculated as follow: <MAT> where Texpected denotes the end-to-end latency caused by the matched thread, <MAT> denotes expected CE processing time for the CE to generate a candidate primary response based on a LP result in the matched thread, <MAT> denotes an expected length of the candidate primary response to be generated by the CE in the matched thread, <MAT> denotes expected TTS processing time in the matched thread, and <MAT> denotes an actual LP time gain in the matched thread.

In an implementation, according to the criteria, if the duration of output of the at least one padding word <NUM> is within a duration of the end-to-end latency with a predetermined probability, then it may be decided that a supplementary voice output which is based on the at least one padding word <NUM> can be provided. The predetermined probability may be any previously-set value, e.g., <NUM>%, <NUM>%, <NUM>%, etc. Assuming that the end-to-end latency conforms with, e.g., a normal distribution N(λ, σ), and the duration of output of the at least one padding word <NUM> is K milliseconds, i.e., an audio clip generated based on the at least one padding word <NUM> lasts K milliseconds, then the decision may be evaluated as follow: <MAT>.

It should be appreciated that the at least one padding word <NUM> may be determined in various approaches. For example, the at least one padding word <NUM> may be selected from a set of predetermined padding words randomly, selected from the set of predetermined padding words under a consideration of whether the criteria may be met, etc..

If it is determined at <NUM> that the criteria are met, a supplementary response <NUM> may be generated based on the at least one padding word <NUM> in various approaches. For example, the supplementary response <NUM> may be generated from the at least one padding word <NUM> directly, generated through combining the at least one padding word <NUM> with any other words, etc. Then, a supplementary voice output <NUM> may be generated through TTS based on the supplementary response <NUM>. At <NUM>, the supplementary voice output <NUM> may be provided to the user in the audio downstream. It should be appreciated that, since the process of generating and providing the supplementary voice output is usually faster than the process of generating and providing the primary voice output, the supplementary voice output may be provided in the audio downstream earlier than the primary voice output. However, during the providing of the supplementary voice output in the audio downstream, if the primary voice output gets ready to be provided, the providing of the supplementary voice output may be interrupted, and instead the providing of the primary voice output may be started in the audio downstream.

If it is determined at <NUM> that the criteria are not met, then the method <NUM> may give up the attempt of providing any supplementary voice output.

It should be appreciated that the method <NUM> is exemplary, and the present disclosure shall cover any additions, deletions, replacements and reordering applied to the steps in the method <NUM>. In an implementation, instead of using padding words in the format of text, pre-recorded or cached padding audio clips may also be adopted by the method <NUM>. In this case, the criteria involved in the decision at <NUM> may be based in part on the duration of a padding audio clip directly, and the padding audio clip may be provided to the user directly as a supplementary voice output, without the need of performing the steps of generating the supplementary response <NUM> and the supplementary voice output <NUM>. In an implementation, although it is shown in <FIG> that the primary voice output is extracted at <NUM> from the at least one candidate primary voice output <NUM>, it is not required that, when the extracting at <NUM> is performed, all the threads have already generated their respective candidate primary voice outputs or even have already generated their respective candidate primary responses. Actually, it is very possible that only few threads including the matched thread have generated candidate primary voice outputs when the extracting at <NUM> is performed, while other threads are still in progress.

<FIG> illustrates an exemplary processing flow <NUM> for responding in a voice conversation according to an embodiment. The processing flow <NUM> is based on the padding response mechanism, and may be deemed as an improvement of the processing flow <NUM> in <FIG>.

The processing flow <NUM> is similar with the processing flow <NUM> in <FIG>, except that a supplementary voice output <NUM> is provided to the user in the audio downstream before the primary voice output <NUM> is provided. Specifically, after obtaining the final SR result <NUM>, the processing flow <NUM> may further determine, according to the method <NUM> in <FIG>, whether criteria for providing a supplementary voice output which is based on a padding word, e.g., "Well", are met. Assuming that the criteria are met, the supplementary voice output <NUM> may be then generated and provided in the audio downstream accordingly.

In some cases, a voice conversation between a chatbot and a user may involve execution of a task. For example, in the voice conversation, the user may request the chatbot to execute a task. Herein, the task may refer to an operation or process that involves multiple-party interactions, complex procedures, etc., and takes longer time to execute as compared to making response in a pure chat scenario. For example, the task may be related to controlling a target smart device, e.g., opening a light, setting an alarm clock, adjusting a target temperature of an air conditioner, etc. For example, the task may be related to obtaining a certain service from a third-party service platform, e.g., calling a Taxi through a car-hailing app, booking a ticket on a ticket agency website, etc..

On one hand, since the execution of the task may take a rather long time, the chatbot may also need a long time to return a result of the execution of the task to the user in the voice conversation. For example, if the user requests to control a target device, the chatbot may need to transfer the request to a server or a cloud service node associated with the target device, the server or cloud service node needs to send a corresponding command to the target device, the target device adjusts its operation status according to the command and returns an execution result to the server or cloud service node, the server or cloud service node returns the execution result to the chatbot, and finally the chatbot may inform the user about the execution result by providing a voice output. The above procedure for controlling the target device is time consuming. Moreover, for example, if the user requests to obtain a certain service from a third-party service platform, since such a task involves interactions with the third-party service platform, it will also cause a rather long time to execute. In any cases, a long-time latency between receiving a request to execute a task from the user and providing an execution result to the user would bring higher risk that the user may get confused whether or not his task request has already been understood and is executed correctly.

On the other hand, it is required that the execution of the task shall be performed only once with respect to each task request. It would be unexpected if the user only request to execute the task once, but the chatbot causes the task to be executed more than one time.

The embodiments of the present disclosure propose an intent-execution response mechanism which may be applied in the application scenario in which a voice conversation between a chatbot and a user involves execution of a task as mentioned above. According to the intent-execution response mechanism, in response to a voice input from the user which requests to execute a task, the chatbot may firstly provide a supplementary voice output which is based on intent of the task to the user as an acknowledgement that the chatbot has understood the task, before providing a primary voice output to the user which indicates a result of executing the task. Moreover, the intent-execution response mechanism may ensure that the task is executed only once through triggering the execution of the task only by a final SR result of the voice input by the user.

<FIG> illustrates an exemplary voice conversation <NUM> according to an embodiment. The voice conversation <NUM> may be based on, e.g., the intent-execution response mechanism. The voice conversation <NUM> is schematically shown in a chat window between a chatbot and a user.

The user may provide a voice input <NUM>, e.g., "Set aircon two degrees higher", in an audio upstream. The voice input <NUM> may indicate that the user wants to request the chatbot to execute a task, e.g., increasing the temperature of the air conditioner by two degrees Celsius.

During receiving the voice input <NUM>, the chatbot may identify, from predictions of the voice input <NUM>, that intent of the task is to increase the temperature of the air conditioner by two degrees Celsius. Based on the intent of the task, the chatbot may generate and provide a supplementary voice output, e.g., "Adjusting the air conditioner".

After completing the receiving of the whole voice input <NUM>, e.g., obtaining a final SR result of the whole voice input <NUM>, the chatbot may identify the intent of the task based on the whole voice input <NUM>. The chatbot may further execute the task based on the intent of the task, e.g., sending a command corresponding to the task to the air conditioner directly, or transferring the request by the user to a server of the air conditioner which is to control the air conditioner as requested, etc..

After the execution of the task is finished, the chatbot may receive a result of the execution of the task from the air conditioner, or the server, etc. The chatbot may further generate a primary voice output <NUM>, e.g., "Your home aircon has been set to <NUM> degrees Celsius", based on the result of the execution of the task, and provide the primary voice output <NUM> in the audio downstream.

It should be appreciated that the chat window in <FIG> is exemplary, and depending on specific application requirements, the chat window may be changed in various manners in terms of layout, elements being incorporated, etc. Moreover, although the voice conversation <NUM> is presented in the visual chat window, the visual chat window may also be omitted in some cases that the chatbot is implemented in a smart device without or disabling display functions.

<FIG> illustrates an exemplary method <NUM> for responding in a voice conversation according to an embodiment. The method <NUM> may be an exemplary implementation of the intent-execution response mechanism.

A voice input <NUM> may be received by a chatbot from a user in an audio upstream in the voice conversation between the chatbot and the user. The voice input <NUM> may include a request for executing a task.

For each intermediate SR result, a thread may be established. In each thread, language prediction (LP) may be performed on the corresponding intermediate SR result so as to generate a LP result which may be deemed as a prediction of the voice input <NUM>. Accordingly, at least one LP result <NUM> may be generated in the method <NUM>, e.g., LP result <NUM>, LP result <NUM>, etc. Taking the intermediate SR result <NUM> as an example, a thread <NUM> may be established, in which the LP result <NUM> may be generated through performing the LP on the intermediate SR result <NUM>.

In each thread, language understanding (LU) may be performed on the corresponding LP result so as to identify intent of the task. Accordingly, a set of task intents <NUM> may be identified in the method <NUM>, e.g., task intent <NUM>, task intent <NUM>, etc. Taking the thread <NUM> as an example, the task intent <NUM> may be generated by the LU based on the LP result <NUM>. The LU may adopt any existing natural language understanding techniques that direct to deriving a semantic representation from a text input, wherein in the method <NUM>, the text input may be a LP result, and the semantic representation may be task intent. It should be appreciated that task intents may be predefined for an object on which tasks are to be executed. For example, assuming that the object is an air conditioner, a list of possible task intents may be predefined, e.g., "turn on air conditioner", "enter into the sleep mode", "increase the temperature by two degrees Celsius", etc. Thus, the purpose of the LU would be mapping a LP result into one of the predefined task intents.

In each thread, a candidate supplementary response may be generated based on the corresponding task intent. Accordingly, at least one candidate supplementary response <NUM> may be obtained in the method <NUM>, e.g., candidate supplementary response <NUM>, candidate supplementary response <NUM>, etc. Taking the thread <NUM> as an example, the candidate supplementary response <NUM> may be generated based on the task intent <NUM>. In an implementation, a candidate supplementary response may be formed by task intent directly. For example, if the identified task intent is "turn on air conditioner", then a candidate supplementary response "turn on air conditioner" may be formed from this identified task intent directly. In an implementation, a candidate supplementary response may be formed through generating a natural language expression relevant to the task intent. For example, if the identified task intent is "increase the temperature by two degrees Celsius", a candidate supplementary response may be generated as, e.g., "Adjusting the air conditioner", "The air conditioner is setting two degrees higher", etc. Various learning-based language models may be adopted here for generating a natural language expression relevant to task intent.

In each thread, text-to-speech (TTS) conversion may be performed on the corresponding candidate supplementary response so as to generate a candidate supplementary voice output. Accordingly, at least one candidate supplementary voice output <NUM> may be generated in the method <NUM>, e.g., candidate supplementary voice output <NUM>, candidate supplementary voice output <NUM>, etc. Taking the thread <NUM> as an example, the candidate supplementary voice output <NUM> may be generated through performing the TTS conversion on the candidate supplementary response <NUM>.

At <NUM>, a matching may be performed between the final SR result <NUM> and the at least one LP result <NUM>, so as to find out a matched LP result <NUM> and accordingly find out a matched thread. The matched LP result <NUM> may indicate a certain thread from which a supplementary voice output to be provided to the user shall be obtained. In other words, the matched LP result <NUM> may be used for extracting, at <NUM>, a supplementary voice output from the at least one candidate supplementary voice output <NUM>. Assuming that the LP result <NUM> in a thread <NUM> is determined as the matched LP result <NUM> and accordingly the thread <NUM> is a matched thread, then the candidate supplementary voice output <NUM> obtained through the thread <NUM> may be extracted as the final supplementary voice output to be provided to the user.

At <NUM>, the extracted supplementary voice output may be provided to the user in an audio downstream.

According to the method <NUM>, when the final SR result <NUM> is obtained, LU may be performed on the final SR result <NUM> so as to identify intent of the task <NUM>. Then, at <NUM>, the chatbot may trigger the execution of the task.

After the execution of the task is finished, a result <NUM> of the execution of the task may be obtained from, e.g., the object on which the task has been executed, a server or cloud service node for the object, etc. The execution result <NUM> may be predefined in various manners, e.g., only indicating whether the task has been executed successfully, describing the task together with whether the execution is successful or not, etc..

A primary response <NUM> may be generated based on the execution result <NUM>. In an implementation, the primary response <NUM> may be formed by the execution result <NUM> directly. For example, if the execution result is "Successful", then a primary response "Successful" may be formed from this execution result directly, if the execution result is "The air conditioner is turned on successfully", then a primary response "The air conditioner is turned on successfully" may be formed from this execution result directly, etc. In an implementation, the primary response may be formed through generating a natural language expression relevant to the execution result. For example, if the execution result is "increase temperature by two degrees Celsius, successful", a primary response may be generated as, e.g., "The temperature has been adjusted successfully", "The air conditioner has been set two degrees higher", etc. Various learning-based language models may be adopted here for generating a natural language expression relevant to an execution result.

A primary voice output <NUM> may be generated based on the primary response <NUM> through TTS conversion, and provided to the user in the audio downstream at <NUM>.

It should be appreciated that, since the process of generating and providing the supplementary voice output is usually faster than the process of executing the task and further generating and providing the primary voice output, the supplementary voice output may be provided in the audio downstream earlier than the primary voice output. However, during providing the supplementary voice output in the audio downstream, if the primary voice output gets ready to be provided, the providing of the supplementary voice output may be interrupted, and instead the providing of the primary voice output may be started in the audio downstream.

It should be appreciated that the method <NUM> is exemplary, and the present disclosure shall cover any additions, deletions, replacements and reordering applied to the steps in the method <NUM>. In an implementation, although it is shown in <FIG> that the supplementary voice output is extracted at <NUM> from the at least one candidate supplementary voice output <NUM>, it is not required that, when the extracting at <NUM> is performed, all the threads have already generated their respective candidate supplementary voice outputs or even have already generated their respective candidate supplementary responses or their respective task intents.

<FIG> illustrates an exemplary processing flow <NUM> for responding in a voice conversation according to an embodiment. The processing flow <NUM> is based on the intent-execution response mechanism.

A voice input <NUM> from a user may be occurring in an audio upstream. It is assumed that the user intends to say a sentence "Open the air cleaner" in the voice input <NUM>, so as to request to execute a task, e.g., turning on the air cleaner.

At <NUM>, continuous speech recognition (CSR) is performed on the voice input <NUM> during receiving the voice input <NUM> in the audio upstream.

Assuming that an intermediate SR result <NUM>, i.e., a word "Open", is firstly obtained by the CSR, then a thread <NUM> may be established for this intermediate SR result <NUM>. In the thread <NUM>, at <NUM>, language prediction (LP) is made based on the intermediate SR result <NUM>. For example, a LP result "Open the door" may be generated based on the intermediate SR result "Open". Then, at <NUM>, a language understanding (LU) may be used for identifying task intent indicated in the LP result, e.g., "Unlock the door". Moreover, a candidate supplementary response, e.g., "The door is opening", may be generated based on the intent of the task "Unlock the door". At <NUM>, a text-to-speech (TTS) conversion may be performed on the candidate supplementary response obtained at <NUM>, so as to generate a candidate supplementary voice output <NUM> which is an audio clip corresponding to the candidate supplementary response "The door is opening".

Assuming that an intermediate SR result <NUM>, i.e., a phrase "Open the air", is further obtained by the CSR, then a thread <NUM> may be established for this intermediate SR result <NUM>. In the thread <NUM>, at <NUM>, a LP result "Open the air cleaner" may be generated based on the intermediate SR result <NUM>. At <NUM>, LU is performed on the LP result "Open the air cleaner" so as to identify task intent, e.g., "Turn on the air cleaner", and accordingly a candidate supplementary response "Air cleaner is turning on" may be generated. The candidate supplementary response "Air cleaner is turning on" is further used for generating a candidate supplementary voice output <NUM> through the TTS at <NUM>.

Assuming that a final SR result <NUM>, i.e., a sentence "Open the air cleaner", is further obtained by the CSR, then a thread <NUM> may be established for this final SR result <NUM>. In the thread <NUM>, LU is performed on the final SR result "Open the air cleaner" so as to identify task intent, e.g., "Turn on the air cleaner". Then, at <NUM>, the chatbot may trigger a task execution (TE) process so as to turn on the air cleaner. Assuming that a result of executing the task indicates that the task has been executed successfully, a primary response "The air cleaner has been opened" may be generated accordingly. The primary response is further used for generating a primary voice output <NUM> through the TTS at <NUM>. Then the primary voice output <NUM> may be provided to the user in an audio downstream.

The thread <NUM>, thread <NUM> and thread <NUM> in <FIG> are established sequentially along with the user is speaking more words in the voice input <NUM>. Accordingly, these threads may generate voice outputs in different time points. Especially, since the thread <NUM> and the thread <NUM> do not involve execution of the task, the candidate supplementary voice outputs <NUM> and <NUM> may be obtained much earlier than the primary voice output <NUM> in the thread <NUM>.

In an implementation, after obtaining the final SR result <NUM>, a matching between this final SR result <NUM> and those LP results in other threads may be performed so as to determine which thread is based on a correct prediction of the voice input <NUM>. For example, in the example of <FIG>, it may be determined that the LP result "Open the air conditioner" obtained at <NUM> in the thread <NUM> matches with the final SR result <NUM>. Accordingly, the candidate supplementary voice output <NUM> by the thread <NUM> may be provided as a final supplementary voice output to the user in the audio downstream. As shown in <FIG>, the candidate supplementary voice output <NUM> is provided earlier than the primary voice output <NUM> in the audio downstream. Through providing the supplementary voice output <NUM> in the audio downstream earlier than the primary voice output <NUM>, it may be avoided that the user gets confused whether or not his task request has already been understood and is executed correctly.

It should be appreciated that although the example of <FIG> involves three exemplary threads established based on words recognized from the voice input <NUM>, more or less threads may be established based on syllables, characters, phrases, etc. recognized from the voice input <NUM>. Although it is shown in <FIG> that each of the threads established for an intermediate SR result obtains a candidate supplementary voice output, it is also possible that not each of these threads has obtained its candidate supplementary voice output when a matched LP result is identified from these threads or when a supplementary voice output corresponding to the matched LP result is provided in the audio downstream. In this case, those threads not finishing their processes may be stopped or removed immediately.

In some cases, during a voice conversation between a chatbot and a user, the user may speak less proactively. For example, after the chatbot provides a voice output to the user, the user may keep silent for a long time period. Under such circumstances, the chatbot may try to speak more so as to keep the conversation going smoothly.

The embodiments of the present disclosure propose a collective follow-up response mechanism which is configured for providing at least one supplementary voice output during a time period between a time point of completing the providing of a primary voice output and a time point of receiving a next voice input by the user. According to the collective follow-up response mechanism, in response to a voice input from the user, the chatbot may generate a primary response and a set of follow-up responses concurrently, wherein the set of follow-up responses are generated collectively at one time. After providing to the user a primary voice output which is based on the primary response, the chatbot may further provide at least one follow-up voice output, as supplementary voice output, which is based on the follow-up responses.

<FIG> illustrates an exemplary voice conversation <NUM> according to an embodiment. The voice conversation <NUM> may be based on, e.g., the collective follow-up response mechanism. The voice conversation <NUM> is schematically shown in a chat window between a chatbot and a user.

The user may provide a voice input <NUM>, e.g., "What's the temperature today?", in an audio upstream.

In response to the voice input <NUM>, the chatbot may generate a primary response "Today's temperature is <NUM> to <NUM> degrees Celsius" and a set of follow-up responses concurrently. The set of follow-up responses may comprise, e.g., "It's sunny today", "The Particulate Matter <NUM> level is <NUM>", "It will rain tomorrow", etc..

The chatbot may firstly provide a primary voice output <NUM> which is generated based on the primary response to the user in an audio downstream. Then, at <NUM>, the chatbot may perform voice activity detection (VAD) to detect whether any further valid voice input occurs in the audio upstream.

If there is no valid voice input within a predetermined duration starting from the primary voice output, e.g., <NUM> seconds, the chatbot may extract a follow-up response, e.g., "It's sunny today", from the set of follow-up responses, and generate a follow-up voice output <NUM> based on the extracted follow-up response. The chatbot may provide the follow-up voice output <NUM> to the user as a supplementary voice output in the audio downstream. Then, at <NUM>, the chatbot may continue the voice activity detection to detect whether any further valid voice input occurs in the audio upstream.

If there is no valid voice input within a predetermined duration starting from the follow-up voice output <NUM>, e.g., <NUM> seconds, the chatbot may extract another follow-up response, e.g., "The Particulate Matter <NUM> level is <NUM>", from the set of follow-up responses, and generate another follow-up voice output <NUM> based on the extracted follow-up response. The chatbot may provide the follow-up voice output <NUM> to the user as another supplementary voice output in the audio downstream. Then, at <NUM>, the chatbot may continue the voice activity detection to detect whether any further valid voice input occurs in the audio upstream.

Assuming that there is a valid voice input detected within a predetermined duration starting from the follow-up voice output <NUM>, for example, a further voice input <NUM> of "Thank you" is received from the user, the chatbot may stop extracting further follow-up responses from the set of follow-up responses and accordingly stop generating further follow-up voice outputs based on the set of follow-up responses.

The chatbot may provide a voice output <NUM> of "My pleasure" so as to respond to the voice input <NUM>.

<FIG> illustrates an exemplary method <NUM> for responding in a voice conversation according to an embodiment. The method <NUM> may be an exemplary implementation of the collective follow-up response mechanism.

During receiving the voice input <NUM>, the chatbot may perform continuous speech recognition (CSR) on the voice input <NUM>. The chatbot may use a conversation engine (CE) to process a correct prediction of the voice input <NUM> which is based on an intermediate SR result or to process a final SR result of the voice input <NUM>. Through the CE, a primary response <NUM> and a set of follow-up responses <NUM> may be generated concurrently, wherein the set of follow-up responses <NUM> may comprise, e.g., follow-up response <NUM>, follow-up response <NUM>, follow-up response <NUM>, etc. The primary response <NUM> may be an optimal direct reply to the voice input <NUM>, which is generated depending on specific implementations of the CE. In an implementation, each of the set of follow-up responses <NUM> may also be a suitable response associated with the voice input <NUM>, but is used as a supplement to the primary response <NUM>. For example, assuming that the voice input <NUM> is a query related to the weather, if the primary response <NUM> involves information of temperature, then the set of follow-up responses <NUM> may intend to provide weather information in other aspects, e.g., wind speed, cloud cover, air quality, etc. In an implementation, the set of follow-up responses <NUM> may be formed from a relative verbose response determined by the CE. For example, assuming that besides generating the primary response <NUM>, the CE also generates a verbose response containing several sentences which is also suitable for responding to the voice input <NUM>, then the CE may split the verbose response into several parts, e.g., sentence by sentence, and each of the several parts may form a follow-up response in the set of follow-up responses <NUM>. In an implementation, instead of being as a response associated with the voice input <NUM>, the set of follow-up responses <NUM> may also comprise responses not associated with the voice input <NUM>, e.g., new topics. In this case, those responses in the set of follow-up responses <NUM> not associated with the voice input <NUM> may also be generated by the CE according to specific implementations of the CE. This is an efficient way to keep the user engage to the conversation.

After generating the primary response <NUM>, the chatbot may perform text-to-speech (TTS) conversion on the primary response <NUM> so as to generate a primary voice output <NUM>. Then, at <NUM>, the primary voice output <NUM> may be provided to the user in an audio downstream.

After providing the primary voice output <NUM>, the chatbot may perform, at <NUM>, voice activity detection (VAD) in the audio upstream within a predetermined duration starting from the primary voice output. The predetermined duration may be defined manually according to various application requirements, e.g., <NUM> seconds, <NUM> seconds, etc. The predetermined duration may start from various time points, e.g., a time point of completing the providing of the primary voice output in the audio downstream, any time point during providing the primary voice output, etc. The VAD directs to detecting any apparent human speech activity in the audio upstream, e.g., valid voice input. Herein, a valid voice input refers to a voice input by the user which has enough volume to be recognized, enough length to comprise any meaningful sentences, phrases, or words, etc. For example, if a voice input detected by the VAD has a too low volume to be recognized, this voice input may be deemed as an invalid voice input. For example, if a voice input detected by the VAD has a too short length, which may be, e.g., coughing, exclamation, etc., this voice input may also be deemed as an invalid voice input.

At <NUM>, it may be determined whether the VAD has detected any valid voice input in the predetermined duration. If yes, i.e., the user provides a new voice input, the method <NUM> may stop trying to provide any supplementary voice output based on the set of follow-up responses <NUM>, and return to the beginning so as to process this new voice input.

If it is determined at <NUM> that the VAD does not detect any valid voice input during the predetermine duration, the method <NUM> may extract a follow-up response from the set of follow-up responses <NUM> at <NUM>. In an implementation, the set of follow-up responses <NUM> may be ranked based on a predetermined rule, and thus a highest ranked follow-up response may be extracted at <NUM>. In an implementation, a follow-up response may be randomly extracted from the set of follow-up responses <NUM>. In any cases, after extracting a follow-up response at <NUM>, the set of follow-up responses <NUM> may be updated by removing the extracted follow-up response so as to avoid the same follow-up response to be extracted again in the following process of the method <NUM>.

The chatbot may perform TTS conversion on the extracted follow-up response so as to generate a follow-up voice output <NUM>. Then, at <NUM>, the follow-up voice output <NUM> may be provided to the user as a supplementary voice output in the audio downstream.

At <NUM>, the chatbot may decide whether the process of providing follow-up voice output shall be terminated. The decision at <NUM> may be based on various termination criteria. In an implementation, the termination criteria at <NUM> may be based on content of the voice input <NUM>. For example, if the user indicates in the voice input <NUM> that he wants to listen three latest news, then it may be determined at <NUM> that whether the number of news that have been provided to the user through the primary voice output and the possible previously-provided follow-up responses reaches the threshold number "three". In an implementation, the termination criteria at <NUM> may be based on a context of the conversation between the chatbot and the user. For example, if the context implies that the user is very interested in sports, then when the user indicates in the voice input <NUM> that he wants to listen to some sports news, a relative high threshold number, e.g., ten, may be set for the termination criteria at <NUM>. In an implementation, the termination criteria at <NUM> may be based on the number of remaining follow-up responses in the set of follow-up responses <NUM>. For example, it may be determined at <NUM> that whether the set of follow-up responses <NUM> does not include any remaining follow-up response.

If it is decided at <NUM> to terminate, the method <NUM> will end at <NUM>. While if it is decided at <NUM> not to terminate, the method <NUM> will return to the VAD at <NUM>. Accordingly, the method <NUM> will continue a process of providing a next follow-up voice output following the steps <NUM> to <NUM> in <FIG>. For example, the chatbot may perform VAD in another predetermined duration starting from the previous voice output, e.g., the follow-up voice output provided previously. Said another predetermined duration may be the same as or different from the predetermined duration previously performing the VAD. Based on the detection result of the VAD, the chatbot may decide whether a next follow-up response shall be extracted and used for generating a next follow-up voice output to be provided to the user. In this way, the method <NUM> will be performed iteratively, until a new valid voice input is detected or the termination criteria are met.

It should be appreciated that the method <NUM> is exemplary, and the present disclosure shall cover any additions, deletions, replacements and reordering applied to the steps in the method <NUM>. For example, although it is shown in <FIG> that a follow-up response can be extracted only when no valid voice input is detected through VAD, it is possible that a follow-up response may be extracted when a certain type of valid voice input is detected through VAD. For example, if the detected valid voice output indicates that the user wants to hear more follow-up voice outputs, e.g., the user says "Continue", then the process of providing a next follow-up voice output may continue, instead of returning to the beginning of the method <NUM>.

<FIG> illustrates an exemplary processing flow <NUM> for responding in a voice conversation according to an embodiment. The processing flow <NUM> is based on the collective follow-up response mechanism.

A voice input <NUM> from a user may be occurring in an audio upstream.

At <NUM>, a conversation engine (CE) may be used for generating a primary response and a set of follow-up responses concurrently based on a correct prediction of the voice input <NUM> which is based on an intermediate SR result, or based on a final SR result of the voice input <NUM>.

Upon the primary response is generated, text-to-speech (TTS) conversion may be performed on the primary response at <NUM> so as to generate a primary voice output <NUM>. Then, the primary voice output <NUM> may be provided to the user in an audio downstream.

In an implementation, voice activity detection (VAD) <NUM> may be continuously performed throughout the processing flow <NUM>.

After providing the primary voice output <NUM>, it may be determined whether the VAD <NUM> detects any valid voice input in the audio upstream within a predetermined duration, W1, starting from the previous voice output, e.g., the primary voice output. For example, the predetermined duration W1 may start from a time point of completing the providing of the primary voice output <NUM> (as shown in <FIG>). However, the predetermined duration W1 may also start from other time points, e.g., any time point during providing the primary voice output, etc..

According to the processing flow <NUM>, no valid voice input is detected within the predetermined duration W1. Then, a follow-up response <NUM> may be extracted from the set of follow-up responses, and generate a follow-up voice output <NUM> through TTS at <NUM>. Upon the follow-up voice output <NUM> is generated, it may be provided to the user as a supplementary voice output in the audio downstream.

After providing the follow-up voice output <NUM>, it may be determined whether the VAD <NUM> detects any valid voice input in the audio upstream within a predetermined duration, W2, starting from the previous voice output, e.g., the follow-up voice output <NUM>. For example, the predetermined duration W2 may start from a time point of completing the providing of the follow-up voice output <NUM> (as shown in <FIG>) or any other time point during providing the follow-up voice output <NUM>, etc..

Assuming that the VAD detects a voice input <NUM> in the audio upstream within the predetermined duration W2, but this voice input <NUM> is further determined as an invalid voice input because of, e.g., low volume, etc. Thus, the processing flow <NUM> may continue to extract a next follow-up response, e.g., follow-up response <NUM> from the set of follow-up responses, and generate a next follow-up voice output <NUM> through TTS at <NUM>. Upon the follow-up voice output <NUM> is generated, it may be provided to the user as a further supplementary voice output in the audio downstream.

After providing the follow-up voice output <NUM>, it may be determined whether the VAD <NUM> detects any valid voice input in the audio upstream within a predetermined duration, W3, starting from the previous voice output, e.g., the follow-up voice output <NUM>. For example, the predetermined duration W3 may start from a time point of completing the providing of the follow-up voice output <NUM> (as shown in <FIG>) or any other time point during providing the follow-up voice output <NUM>, etc..

Assuming that the VAD <NUM> detects a valid voice input <NUM> in the audio upstream within the predetermined duration W3, the processing flow <NUM> may stop generating further follow-up voice outputs based on the set of follow-up responses. For example, follow-up response <NUM> and any subsequent follow-up responses in the set of follow-up responses would not be used for generating follow-up voice outputs and being provided to the user.

The embodiments of the present disclosure further propose a recursive follow-up response mechanism, which is also configured for providing at least one supplementary voice output during a time period between a time point of completing the providing of a primary voice output and a time point of receiving a next voice input by the user. However, as compared to the collective follow-up response mechanism, the recursive follow-up response mechanism would generate a set of follow-up responses recursively through a plurality of iterations after generating a primary response, instead of generating the set of follow-up responses at one time. For example, the recursive follow-up response mechanism may generate the set of follow-up responses one-by one, and a next follow-up response would be generated only when a certain condition is met.

When processing costs and/or latencies of generating follow-up responses are high, the collective follow-up response mechanism which generates all the follow-up responses within one call to CE would be inefficient. For example, if a user speaks a new voice input before all the generated follow-up responses are processed for providing corresponding follow-up voice outputs, those remaining follow-up responses not having been processed shall be discarded. Thus, the high costs of generating these follow-up responses are wasted. In contrast, the recursive follow-up response mechanism generates one follow-up response at one time calling to CE, and only when a certain condition is met after a follow-up voice output has been provided, a next call to CE would be performed so as to generate a next follow-up response. Thus, even if the process of providing follow-up voice outputs is interrupted in the middle by a new voice input, no wasted calls to CE would occur, and accordingly processing resources would be saved.

Moreover, according to the collective follow-up response mechanism, since contents included in follow-up responses are static after these follow-up responses have been generated through one single call to CE, it would be impossible to support dynamic changes of the contents in the generated follow-up responses, e.g., inserting breaking news just happened during the process of providing follow-up voice outputs. On the contrary, since the recursive follow-up response mechanism generates a follow-up response just before providing a corresponding follow-up voice output, there is flexibility to dynamically adjust content included in a next follow-up response to be generated.

Furthermore, it is possible for the recursive follow-up response mechanism to implement more flexible and complex policies on whether or not to go on providing follow-up voice outputs. For example, for each time calling the CE, the chatbot may dynamically re-evaluate whether to continue the process based on voice activity detection (VAD), context of conversation, latest voice input, etc. In some scenarios, the recursive follow-up response mechanism may also support infinite providing of follow-up voice outputs, as long as the re-evaluating is keeping positive.

<FIG> illustrates an exemplary voice conversation <NUM> according to an embodiment. The voice conversation <NUM> may be based on, e.g., the recursive follow-up response mechanism. The voice conversation <NUM> is schematically shown in a chat window between a chatbot and a user.

The user may provide a voice input <NUM>, e.g., "News today", in an audio upstream.

In response to the voice input <NUM>, the chatbot may generate a primary response which includes a piece of news, e.g., "High winds are expected in Hong Kong today due to an approaching typhoon. The State Oceanic Administration had issued alerts for waves and storms", and provide a primary voice input <NUM>, which is generated based on the primary response, to the user in an audio downstream.

Then, at <NUM>, the chatbot may perform voice activity detection to detect whether any further valid voice input occurs in the audio upstream.

Assuming that no valid voice input is detected within a predetermined duration starting from the primary voice output, e.g., <NUM> seconds, the chatbot may further generate a follow-up response which includes another piece of news, e.g., "The 18th Shanghai Cooperation Summit is scheduled for June <NUM> to <NUM> in Qingdao, Shandong Province". The chatbot may generate a follow-up voice output <NUM> based on the follow-up response, and provide the follow-up voice output <NUM> to the user as a supplementary voice output in the audio downstream. Then, at <NUM>, the chatbot may continue the voice activity detection to detect whether any further valid voice input occurs in the audio upstream.

If no valid voice input is detected within a predetermined duration starting from the follow-up voice output <NUM>, e.g., <NUM> seconds, the chatbot may further generate a second follow-up response which includes another piece of news, e.g., "Breaking news! Typhoon Ewiniar has just made landfall in southernmost of Guangdong Province <NUM> minutes before, at <NUM>:<NUM> am. Packing winds of <NUM> per hour". For example, the news included in the second follow-up response may be just obtained by the chatbot during preparing and providing the primary voice output <NUM> or the follow-up voice output <NUM>. The chatbot may generate a follow-up voice output <NUM> based on the second follow-up response, and provide the follow-up voice output <NUM> to the user as a second supplementary voice output in the audio downstream. Then, at <NUM>, the chatbot may continue the voice activity detection to detect whether any further valid voice input occurs in the audio upstream.

Assuming that there is a valid voice input detected within a predetermined duration starting from the follow-up voice output <NUM>, for example, a further voice input <NUM> of "Thank you" is received from the user, the chatbot may stop generating further follow-up responses and accordingly stop generating and providing further follow-up voice outputs.

<FIG> illustrates an exemplary method <NUM> for responding in a voice conversation according to an embodiment. The method <NUM> may be an exemplary implementation of the recursive follow-up response mechanism.

During receiving the voice input <NUM>, the chatbot may perform continuous speech recognition (CSR) on the voice input <NUM>. The chatbot may use a conversation engine (CE) to process a correct prediction of the voice input <NUM> which is based on an intermediate SR result or to process a final SR result of the voice input <NUM>. Through the CE, a primary response <NUM> may be generated, which is an optimal reply to the voice input <NUM> generated depending on specific implementations of the CE.

Along with the primary response <NUM>, the CE may also generate a pseudo input <NUM> based at least on the voice input <NUM>. The pseudo input <NUM> directs to expressing intent of the user reflected in the voice input <NUM>, and may be the same as content of the voice input <NUM>, or semantically relevant to the content of the voice input <NUM>. For example, if the voice input <NUM> is "Give some news", the pseudo input <NUM> may be generated as, e.g., "Give some news", "news today", etc. The pseudo input <NUM> may be later used for generating a follow-up response in the subsequent process of the method <NUM>.

According to the method <NUM>, after providing the primary voice output <NUM>, the chatbot may try to recursively generate one or more follow-up responses and provide corresponding follow-up voice outputs through a plurality of iterations. For a single iteration, the chatbot will generate only one follow-up response and provide one corresponding follow-up voice output.

After providing the primary voice output <NUM>, the chatbot may perform, at <NUM>, voice activity detection (VAD) in the audio upstream within a predetermined duration starting from the previous voice output, e.g., the primary voice output <NUM>. The predetermined duration may start from various time points, e.g., a time point of completing the providing of the primary voice output in the audio downstream, any time point during providing the primary voice output, etc..

At <NUM>, it may be determined whether the VAD has detected any valid voice input in the predetermined duration. If yes, i.e., the user provides a new valid voice input, the method <NUM> may stop trying to provide any supplementary voice output, and return to the beginning so as to process this new voice input.

If it is determined at <NUM> that the VAD does not detect any valid voice input during the predetermined duration, the chatbot may use the CE to generate a follow-up response <NUM> based at least on a pseudo input which is generated along with the previously-generated response, i.e., the pseudo input <NUM> generated along with the primary response <NUM>. In an implementation, the generation of the follow-up response <NUM> may be further based on a context of the conversation. The follow-up response <NUM> is also a suitable response associated with the voice input <NUM>, but is used as a supplement to the primary response <NUM>. For example, assuming that the voice input <NUM> is a query related to the weather, if the primary response <NUM> involves information of temperature, then the follow-up response <NUM> may further provide weather information about, e.g., wind speed, etc..

Along with the follow-up response <NUM>, the CE may also generate a new pseudo input <NUM> based at least on the voice input <NUM>. Similar with the pseudo input <NUM>, the new pseudo input <NUM> may also direct to expressing intent of the user reflected in the voice input <NUM>, and may be the same as the content of the voice input <NUM>, or semantically relevant to the content of the voice input <NUM>. The pseudo input <NUM> may be later used for generating a next follow-up response in the subsequent iteration of the method <NUM>.

The chatbot may perform TTS conversion on the follow-up response <NUM> so as to generate a follow-up voice output <NUM>. Then, at <NUM>, the follow-up voice output <NUM> may be provided to the user as a supplementary voice output in the audio downstream.

At <NUM>, the chatbot may decide whether the process of providing follow-up voice outputs shall be terminated. The decision at <NUM> may be based on various termination criteria. In an implementation, the termination criteria at <NUM> may be based on the content of the voice input <NUM>. For example, if the user indicates in the voice input <NUM> that he wants to listen three latest news, then it may be determined at <NUM> that whether the number of news that have been provided to the user through the primary voice output and the possible previously-provided follow-up responses reaches the threshold number "three". In an implementation, the termination criteria at <NUM> may be based on a context of the conversation. For example, if the context implies that the user is very interested in sports, then when the user indicates in the voice input <NUM> that he wants to listen some sports news, a relative high threshold number, e.g., ten, may be set for the termination criteria at <NUM>.

If it is decided at <NUM> to terminate, the method <NUM> will end at <NUM>. While if it is decided at <NUM> not to terminate, the method <NUM> will return to the VAD at <NUM>. Accordingly, the method <NUM> will continue to perform a next iteration for providing a next follow-up voice output following the steps <NUM> to <NUM> in <FIG>. For example, in the next iteration, the chatbot may perform VAD in another predetermined duration starting from the previous voice output, e.g., the follow-up voice output provided previously. Said another predetermined duration may be the same as or different from the predetermined duration previously performing the VAD. Based on the detection result of the VAD, the chatbot may decide whether a next follow-up response shall be generated based at least on a pseudo input, e.g., the pseudo input <NUM> generated along with the previous follow-up response. If the next follow-up response is generated, it is further used for generating a next follow-up voice output to be provided to the user. Moreover, a new pseudo input may be generated along with said next follow-up response. In this way, the method <NUM> will be performed iteratively, until a new valid voice input is detected or the termination criteria are met.

It should be appreciated that the method <NUM> is exemplary, and the present disclosure shall cover any additions, deletions, replacements and reordering applied to the steps in the method <NUM>. For example, although it is shown in <FIG> that a follow-up response can be generated only when no valid voice input is detected through VAD, it is possible that a follow-up response may be generated when a certain type of valid voice input is detected through VAD. For example, if the detected valid voice output indicates that the user wants to hear more follow-up voice outputs, then the process of providing a next follow-up voice output may continue, instead of returning to the beginning of the method <NUM>.

<FIG> illustrates an exemplary processing flow <NUM> for responding in a voice conversation according to an embodiment. The processing flow <NUM> is based on the recursive follow-up response mechanism.

At <NUM>, a conversation engine (CE) may be used for generating a primary response and a pseudo input based on a correct prediction of the voice input <NUM> which is based on an intermediate SR result, or based on a final SR result of the voice input <NUM>.

After providing the primary voice output <NUM>, it may be determined whether the VAD <NUM> detects any valid voice input in the audio upstream within a predetermined duration, W1, starting from the previous voice output, e.g., the primary voice output. For example, the predetermined duration W1 may start from a time point of completing the providing of the primary voice output <NUM> (as shown in <FIG>), or start from other time points, e.g., any time point during providing the primary voice output <NUM>, etc..

Assuming that the VAD detects a voice input <NUM> in the audio upstream within the predetermined duration W1, but this voice input <NUM> is further determined as an invalid voice input because of, e.g., too short length, etc. Thus, the processing flow <NUM> may continue to perform an iteration for generating and providing a follow-up voice output. For example, at <NUM>, the CE may be used for generating a follow-up response <NUM> based at least on a pseudo input generated along with the previous response, e.g., the pseudo input generated along with the primary response. Meanwhile, the CE may also generate a new pseudo input <NUM>. Upon the follow-up response <NUM> is generated, TTS conversion may be performed on the follow-up response <NUM> at <NUM> so as to generate a follow-up voice output <NUM>. Then, the follow-up voice output <NUM> may be provided to the user as a supplementary voice output in the audio downstream.

Assuming that the VAD does not detect any valid voice input in the audio upstream within the predetermined duration W2, the processing flow <NUM> may continue to perform a next iteration for generating and providing a further follow-up voice output. For example, at <NUM>, the CE may be used for generating a follow-up response <NUM> based at least on a pseudo input generated along with the previous response, e.g., the new pseudo input <NUM> generated along with the follow-up response <NUM>. Meanwhile, the CE may also generate a new pseudo input <NUM>. Upon the follow-up response <NUM> is generated, TTS conversion may be performed on the follow-up response <NUM> at <NUM> so as to generate a follow-up voice output <NUM>. Then, the follow-up voice output <NUM> may be provided to the user as a further supplementary voice output in the audio downstream.

Assuming that the VAD <NUM> detects a valid voice input <NUM> in the audio upstream within the predetermined duration W3, the processing flow <NUM> may stop performing any further iteration for trying to provide further follow-up voice outputs.

It should be appreciated that although a plurality of exemplary multi-phrase response mechanisms have been discussed above separately, any of these mechanisms may be combined together according to specific application designs and requirements. For example, when combining the intent-execution response mechanism with the padding response mechanism, taking the conversation <NUM> in <FIG> as an example, instead of the supplementary voice output <NUM> of "Adjusting the air conditioner", the chatbot may provide a supplementary voice output "Okay. Adjusting the air conditioner" wherein the padding word "Okay" is obtained through the padding response mechanism and the sentence "Adjusting the air conditioner" is obtained through the intent-execution response mechanism. For example, when combining the collective follow-up response mechanism with the padding response mechanism, taking the conversation <NUM> in <FIG> as an example, instead of the primary voice output <NUM> of "Today's temperature is <NUM> to <NUM> degrees Celsius", the chatbot may provide a voice output "Hmm, today's temperature is <NUM> to <NUM> degrees Celsius" wherein the padding word "Hmm" is obtained through the padding response mechanism and the sentence "today's temperature is <NUM> to <NUM> degrees Celsius" is obtained through the collective follow-up response mechanism. Similarly, any other combinations of the multi-phrase response mechanisms would be obvious for those skilled in the art.

When implementing anyone or a combination of the exemplary multi-phrase response mechanisms discussed above in a chatbot, the chatbot may dynamically determine whether to apply a certain multi-phrase response mechanism among the multi-phrase response mechanisms during a voice conversation. For example, the chatbot may keep detecting whether there is a voice input occurred in an audio upstream. When a voice input is detected as occurring in the audio upstream, the chatbot may first provide a supplementary voice output in an audio downstream through the padding response mechanism, and then provide a primary voice output in the audio downstream. Meanwhile, if the voice input is detected to be related to a task, the chatbot may additionally provide a second supplementary voice output through the intent-execution response mechanism after providing the supplementary voice output generated by the padding response mechanism, or provide the second supplementary voice output generated by the intent-execution response mechanism instead of the supplementary voice output generated by the padding response mechanism, and then the chatbot may provide a primary voice output through the intent-execution response mechanism. In any cases, after the primary voice output is provided, the chatbot may keep detecting whether there is a further valid voice input occurred in the audio upstream. If no further valid voice input is detected in the audio upstream within a predetermined time period, the chatbot may further provide a follow-up voice output in the audio downstream through either of the collective follow-up response mechanism or the recursive follow-up response mechanism. Then, the chatbot may continue to detect valid voice input in a second predetermined time period, and determine whether to provide a second follow-up voice output in the audio downstream through either of the collective follow-up response mechanism or the recursive follow-up response mechanism, and so on.

<FIG> illustrates a flowchart of an exemplary method <NUM> for responding in a voice conversation by an electronic conversational agent according to an embodiment.

At <NUM>, a voice input may be received in an audio upstream.

At <NUM>, in response to the voice input, a primary response and at least one supplementary response may be generated.

At <NUM>, a primary voice output may be generated based on the primary response.

At <NUM>, at least one supplementary voice output may be generated based on the at least one supplementary response.

At <NUM>, the primary voice output and the at least one supplementary voice output may be provided in an audio downstream, wherein the at least one supplementary voice output is provided during a time period adjacent to the primary voice output in the audio downstream.

In an implementation, no valid voice input is received during the time period.

In an implementation, the time period is between a time point of completing the receiving of the voice input and a time point of providing the primary voice output.

According to the above implementation, the generating the at least one supplementary response may comprise: generating the at least one supplementary response based on at least one padding word, wherein duration of output of the at least one padding word is within duration of the time period with a predetermined probability. The generating the primary response may comprise: generating a prediction of the voice input during receiving the voice input; and generating the primary response based on the prediction.

According to the above implementation, the voice input is associated with a task, and the generating the at least one supplementary response may comprise: generating a prediction of the voice input during receiving the voice input; identifying intent of the task based on the prediction; and generating the at least one supplementary response based on the intent of the task. The generating the primary response may comprise: identifying the intent of the task based on the voice input after completing the receiving of the voice input; executing the task based on the intent of the task; and generating the primary response based on a result of executing the task.

In an implementation, the time period is between a time point of completing the providing of the primary voice output and a time point of receiving a next voice input.

According to the above implementation, the generating the at least one supplementary response may comprise: generating a set of follow-up responses concurrently with the generating of the primary response. The generating the at least one supplementary voice output may comprise, for each follow-up response in the set of follow-up responses: performing voice activity detection in the audio upstream within a predetermined duration starting from a previous voice output; and if no valid voice input is detected through the voice activity detection, generating a follow-up voice output based on the follow-up response. While if a valid voice input is detected through the voice activity detection, the method may stop generating follow-up voice outputs based on the set of follow-up responses.

According to the above implementation, the generating the at least one supplementary response may comprise: generating a set of follow-up responses recursively through a plurality of iterations after the generating of the primary response. The generating the at least one supplementary response may comprise, for each iteration in the plurality of iterations: performing voice activity detection in the audio upstream within a predetermined duration starting from the previous voice output; if no valid voice input is detected through the voice activity detection, generating a follow-up response based at least on a pseudo input, the pseudo input being generated along with the previous response; and generating a new pseudo input based at least on the voice input. While if a valid voice input is detected through the voice activity detection, the method may stop performing any further iterations.

It should be appreciated that the method <NUM> may further comprise any steps/processes for responding in a voice conversation according to the embodiments of the present disclosure as mentioned above.

<FIG> illustrates an exemplary apparatus <NUM> for responding in a voice conversation according to an embodiment.

The apparatus <NUM> may comprise: a voice input receiving module <NUM>, for receiving a voice input in an audio upstream; a response generating module <NUM>, for generating, in response to the voice input, a primary response and at least one supplementary response; a primary voice output generating module <NUM>, for generating a primary voice output based on the primary response; a supplementary voice output generating module <NUM>, for generating at least one supplementary voice output based on the at least one supplementary response; and a voice output providing module <NUM>, for providing the primary voice output and the at least one supplementary voice output in an audio downstream, wherein the at least one supplementary voice output is provided during a time period adjacent to the primary voice output in the audio downstream.

According to the above implementation, the response generating module <NUM> may be further for: generating the at least one supplementary response based on at least one padding word, wherein duration of output of the at least one padding word is within duration of the time period with a predetermined probability.

According to the above implementation, the voice input is associated with a task, and the response generating module <NUM> may be further for: generating a prediction of the voice input during receiving the voice input; identifying intent of the task based on the prediction; and generating the at least one supplementary response based on the intent of the task.

In an implementation, the time period is between a time point of completing the providing of the primary voice output and a time point of receiving a next voice input. The response generating module <NUM> may be further for generating the at least one supplementary response through: generating a set of follow-up responses concurrently with the generating of the primary response; or generating a set of follow-up responses recursively through a plurality of iterations after the generating of the primary response.

Moreover, the apparatus <NUM> may also comprise any other modules configured for responding in a voice conversation according to the embodiments of the present disclosure as mentioned above.

The apparatus <NUM> may comprise at least one processor <NUM> and a memory <NUM> storing computer-executable instructions. When executing the computer-executable instructions, the at least one processor <NUM> may: receive a voice input in an audio upstream; generate, in response to the voice input, a primary response and at least one supplementary response; generate a primary voice output based on the primary response; generate at least one supplementary voice output based on the at least one supplementary response; and provide the primary voice output and the at least one supplementary voice output in an audio downstream, wherein the at least one supplementary voice output is provided during a time period adjacent to the primary voice output in the audio downstream. The at least one processor <NUM> may be further configured for performing any operations of the methods for responding in a voice conversation according to the embodiments of the present disclosure as mentioned above.

The embodiments of the present disclosure may be embodied in a non-transitory computer-readable medium. The non-transitory computer-readable medium may comprise instructions that, when executed, cause one or more processors to perform any operations of the methods for responding in a voice conversation according to the embodiments of the present disclosure as mentioned above.

It should be appreciated that all the operations in the methods described above are merely exemplary, and the present disclosure is not limited to any operations in the methods or sequence orders of these operations, and should cover all other equivalents under the same or similar concepts.

It should also be appreciated that all the modules in the apparatuses described above may be implemented in various approaches. These modules may be implemented as hardware, software, or a combination thereof. Moreover, any of these modules may be further functionally divided into sub-modules or combined together.

Processors have been described in connection with various apparatuses and methods. These processors may be implemented using electronic hardware, computer software, or any combination thereof. Whether such processors are implemented as hardware or software will depend upon the particular application and overall design constraints imposed on the system. By way of example, a processor, any portion of a processor, or any combination of processors presented in the present disclosure may be implemented with a microprocessor, microcontroller, digital signal processor (DSP), a field-programmable gate array (FPGA), a programmable logic device (PLD), a state machine, gated logic, discrete hardware circuits, and other suitable processing components configured to perform the various functions described throughout the present disclosure. The functionality of a processor, any portion of a processor, or any combination of processors presented in the present disclosure may be implemented with software being executed by a microprocessor, microcontroller, DSP, or other suitable platform.

Software shall be construed broadly to mean instructions, instruction sets, code, code segments, program code, programs, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, threads of execution, procedures, functions, etc. The software may reside on a computer-readable medium. A computer-readable medium may include, by way of example, memory such as a magnetic storage device (e.g., hard disk, floppy disk, magnetic strip), an optical disk, a smart card, a flash memory device, random access memory (RAM), read only memory (ROM), programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), a register, or a removable disk. Although memory is shown separate from the processors in the various aspects presented throughout the present disclosure, the memory may be internal to the processors, e.g., cache or register.

Claim 1:
A method (<NUM>) for responding in a voice conversation by an electronic conversational agent (<NUM>), comprising:
receiving (<NUM>) a voice input (<NUM>; <NUM>; <NUM>; <NUM>; <NUM>; <NUM>; <NUM>; <NUM>; <NUM>; <NUM>) in an audio upstream (<NUM>);
generating candidate voice outputs (<NUM>, <NUM>, <NUM>) using content of the audio upstream;
establishing respective processing threads for the candidate voice outputs, wherein the respective processing threads are processed in parallel;
selecting, from the processing threads , a first candidate voice output (<NUM>, <NUM>, <NUM>);
generating (<NUM>) a primary voice output (<NUM>; <NUM>; <NUM>; <NUM>; <NUM>; <NUM>; <NUM>; <NUM>; <NUM>; <NUM>; <NUM>; <NUM>) based on the first candidate voice output (<NUM>, <NUM>, <NUM>);
generating (<NUM>) at least one supplementary voice output based on a second candidate voice output (<NUM>, <NUM>, <NUM>);
determining (<NUM>) that generation of the primary voice output is not complete; providing (<NUM>) a first supplementary voice output of the at least one supplementary voice output in the audio downstream;
determining (<NUM>) that generation of the primary voice output is complete; and
providing (<NUM>) the primary voice output in the audio downstream, wherein the at least one supplementary voice output is provided during a time period adjacent to the primary voice output in the audio downstream.