Abstract:
Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for speech endpointing are described. In one aspect, a method includes the action of accessing voice query log data that includes voice queries spoken by a particular user. The actions further include based on the voice query log data that includes voice queries spoken by a particular user, determining a pause threshold from the voice query log data that includes voice queries spoken by the particular user. The actions further include receiving, from the particular user, an utterance. The actions further include determining that the particular user has stopped speaking for at least a period of time equal to the pause threshold. The actions further include based on determining that the particular user has stopped speaking for at least a period of time equal to the pause threshold, processing the utterance as a voice query.

Description:
CROSS-REFERENCE TO RELATED APPLICATION 
       [0001]    This application is a continuation of U.S. application Ser. No. 14/923,637, filed Oct. 27, 2015, which claims the benefit of U.S. Provisional Application No. 62/243,463, filed Oct. 19, 2015, the contents of each are incorporated by reference. 
     
    
     TECHNICAL FIELD 
       [0002]    This disclosure generally relates to speech recognition, and one particular implementation relates to endpointing speech. 
       BACKGROUND 
       [0003]    Natural language processing systems typically use endpointers to determine when a user has started and finished speaking. Some traditional endpointers evaluate the duration of pauses between words in determining when an utterance begins or ends. For instance, if a user says “what is &lt;long pause&gt; for dinner,” a traditional endpointer may segment the voice input at the long pause, and may instruct the natural language processing system to attempt to process the incomplete phrase “what is,” instead of the complete phrase “what is for dinner.” If an endpointer designates an incorrect beginning or ending point for a voice input, the result of processing the voice input using the natural language processing system may be inaccurate or undesirable. 
       SUMMARY 
       [0004]    Different users may have different comfort levels with using voice input on their mobile devices. Some users may use the voice input features frequently and be able to form voice queries without hesitating during speaking. Other users may not use the voice input features as often or quickly formulate queries without hesitating. The more experienced user may benefit from a system that begins to process the user&#39;s voice queries after measuring a short pause in the user&#39;s speech. The less experienced user may benefit form a system that begins to process the user&#39;s voice queries after measuring a longer pause in the user&#39;s speech to ensure the less experienced user has finished speaking. 
         [0005]    To determine a pause length to apply to a particular user&#39;s voice queries, a system analyzes the particular user&#39;s previous voice queries. For a particular user, the system looks at voice query frequency, pause length between words of previous voice queries, completeness of previous voice queries, and length of previous voice queries. A user who more frequently speaks voice queries may benefit from a shorter pause length to apply to a user&#39;s voice queries compared to a user who does not speak voice queries as frequently. A user who has a short average pause length between words of previous voice queries may benefit from a shorter pause length than a user who has longer average pause lengths between words. A user who speaks complete voice queries more often may benefit from a shorter pause length than a user who speaks complete queries less frequently. A user who speaks longer voice queries may benefit from a shorter pause length than a user who speaks shorter voice queries. Once the system computes an appropriate pause length to apply to a particular user&#39;s voice queries, the system may generate an endpoint during the particular user&#39;s future utterances using the particular user&#39;s pause length. 
         [0006]    In general, another innovative aspect of the subject matter described in this specification may be implemented in methods that include the actions of accessing voice query log data that includes voice queries spoken by a particular user; based on the voice query log data that includes voice queries spoken by a particular user, determining a pause threshold from the voice query log data that includes voice queries spoken by the particular user; receiving, from the particular user, an utterance; determining that the particular user has stopped speaking for at least a period of time equal to the pause threshold; and based on determining that the particular user has stopped speaking for at least a period of time equal to the pause threshold, processing the utterance as a voice query. 
         [0007]    These and other embodiments can each optionally include one or more of the following features. The action of determining the pause threshold from the voice query log data that includes voice queries spoken by the particular user includes classifying the particular user as an expert user of a speech recognition system or as a novice user of the speech recognition system; and based on classifying the particular user as the expert user of the speech recognition system or as the novice user of the speech recognition system, determining the pause threshold. The voice query log data a timestamp associated with each voice query, data indicating whether each voice query is complete, and speech pause intervals associated with each voice query. The action of determining a pause threshold from the voice query log data that includes voice queries spoken by the particular user includes determining the pause threshold based on the timestamp associated with each voice query, the data indicating whether each voice query is complete, and the speech pause intervals associated with each voice query. 
         [0008]    The actions further include based on the voice query log data, determining an average number of voice queries spoken by the particular user each day. The action of determining the pause threshold is based further on the average number of voice queries spoken by the particular user each day. The actions further include based on the voice query log data, determining an average length of voice queries spoken by the particular user. The action of determining the pause threshold is based further on the average length of voice queries spoken by the particular user. The actions further include based on the voice query log data, determining an average pause interval for voice queries spoken by the particular user. The action of determining the pause threshold is based further on the average pause interval for voice queries spoken by the particular user. 
         [0009]    Other embodiments of this aspect include corresponding systems, apparatus, and computer programs recorded on computer storage devices, each configured to perform the operations of the methods. 
         [0010]    Particular embodiments of the subject matter described in this specification can be implemented so as to realize one or more of the following advantages. A user may use the voice input capabilities of a computing device and speak at a pace that is comfortable for the user. An utterance may be endpointed at the intended end of the utterance, leading to more accurate or desirable natural language processing outputs, and to faster processing by the natural language processing system. 
         [0011]    The details of one or more embodiments of the subject matter described in this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims. 
     
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         [0012]      FIG. 1  is a diagram of example utterances and signals used in determining, for a particular user, whether a user has finished speaking a voice query. 
           [0013]      FIG. 2  is a diagram of an example system that classifies a particular user based on the particular user&#39;s experience with speech input. 
           [0014]      FIG. 3  is a diagram of an example process for classifying a particular user based on the particular user&#39;s experience with speech input and determining whether the particular user has finished speaking a voice query. 
           [0015]      FIG. 4  is a block diagram of computing devices on which the processes described herein, or portions thereof, may be implemented. 
       
    
    
       [0016]    Like reference numbers and designations in the various drawings indicate like elements. 
       DETAILED DESCRIPTION 
       [0017]      FIG. 1  is a diagram  100  of example utterances and signals used in determining, for a particular user, whether a user has finished speaking a voice query. In general, diagram  100  illustrates signals  103 - 118  that are generated or detected by computing device  121  when the computing device  121  is processing an incoming audio input. The computing device  121  receives the audio data corresponding to utterance  124  through a microphone or other audio input device of the computing device  121 , and generates a transcription of the utterance  124  depending on a user profile assigned to the user  127 . 
         [0018]    The utterance timing  130  represents the timing of the various words of the user  127  speaking utterance  124  (in the  FIG. 1 , “Text Mom love you”). The user  127  speaks each word with an increasing pause length between each word. The number of dots between each word is proportional to the pause length between each word. Each dot may represent a particular period of time, such as one hundred milliseconds. The first word  133 , “Text,” is followed by pause  136  of three dots that may correspond to three hundred milliseconds. The second word  139 , “Mom,” is followed by pause  142  of eight dots that may correspond to eight hundred milliseconds. The third word  145 , “love,” is followed by pause  128  of twelve dots that may correspond to 1.2 seconds. The fourth word  151 , “you,” is followed by pause  154 . In  FIG. 1 , pause  154  is twenty dots that may correspond to two seconds. Because word  151  is at the end of utterance  124 , a pause after word  151  may be much longer than pause  154  because the user  127  has stopped speaking. These pauses and the other pauses described below represent a natural period of silence between two words, and not that the user actively stopped speaking. 
         [0019]    In some implementations, the computing device  121  may generate, without factoring in any characteristics of the user  127 , the general endpoint signal  103  and the complete query signal  106 . The complete query signal  106  represents an estimate performed by the computing device  121  that the generated transcription of the utterance  130  represents a complete utterance. The computing device  121  compares the generated transcription to one or more complete utterances that the user  127  and other users have previously spoken. The computing device  121  may compare the generated transcription to the complete utterances after a speech recognizer of computing device  121  has identified a new word. For example, after the user  127  speaks word  133 , a speech recognizer of the computing device  121  generates the transcription “text.” The computing device  121  compares “text” to other complete utterances and determines that “text” is not a complete utterance. After the user  127  speaks word  139 , the speech recognizer generates the transcription “text mom” that the computing device  121  identifies as complete. A similar determination is made after word  151 . After the user  127  speaks word  145 , the speech recognizer generates the transcription “text mom love” that the computing device  121  identifies as incomplete. 
         [0020]    The general endpoint signal  103  represents an estimate performed by the computing device  121  that the user  127  has finished speaking. The computing device  121  may generate the general endpoint signal  103  based on the length of pauses between speech audio. The computing device  121  may generate the general endpoint signal  103  without generating a transcription of the utterance  124 . For example, the computing device  121  may receive audio data corresponding to word  133 . During pause  136 , the computing device  121  measures the time as it elapses during pause  136 . The pause  136  may only last for three hundred milliseconds. If the general endpoint threshold is longer than three hundred milliseconds, such as six hundred milliseconds, then the computing device  121  will not trigger the general endpointer. After the computing device  121  receives audio data corresponding to word  139 , the computing devices  121  measures the time of pause  142 . After six hundred milliseconds of pause  142  has elapsed, the computing device  121  triggers the general endpointer and the general endpoint signal  103  indicates that an endpoint has been reached. With the general endpoint signal  103  indicating an endpoint of the utterance  124  and the complete query signal  106  indicating that the utterance  124  is complete, the computing device generates transcription  157 , “text mom,” for utterance  124 . 
         [0021]    In some implementations, the computing device  121  may factor in the characteristics of the user  127  when identifying an endpoint of the utterance  124 . On one hand, a novice user may speak with longer pauses between words possibly because the novice user may be unfamiliar with what terms may be best to speak to the computing device  121 . On the other hand, an expert user may speak with shorter pauses between words because the expert user may be more comfortable and familiar with the speech input technology of computing device  121 . Accordingly, the computing device  121  may lengthen or shorten the amount of time before it identifies a pause depending on how the computing device  121  categorizes the user. 
         [0022]    The novice pause detector signal  109  illustrates the computing device  121  detecting a pause in audio data corresponding to utterance  124 , where the detected pause length is longer than the pause length the corresponds to the general endpointer. For example, the computing device  121  may detect pauses with a length of one second when the user  127  is classified as a novice user. Applying this pause threshold to utterance  124 , the computing device  121  will not detect novice length pauses during pauses  136  and  124  because those pauses are of length three hundred milliseconds and eight hundred milliseconds, respectively. The computing device  121  does detect novice length pauses during pauses  148  and  154 . As shown in novice pause detector signal  109 , the computing device  121  detects a pause of one second during pause  148  after the user  127  spoke word  145 . The computing device  121  also detects a pause of one second during pause  154  after the user spoke word  151 . 
         [0023]    The computing device  121  determines, based on the novice pause detector signal  109  and the complete query signal  106 , a speech endpoint for the utterance  124  when the computing device classifies the user as a novice. When the computing device  121  detects a pause, such as the pause of the novice pause detector signal  109  during pause  148 , the computing device  121  determines whether the utterance  124  is complete. During pause  148 , the complete query signal  106  indicates that the utterance  124  is not complete. Even though the computing device  121  detected a novice length pause, the utterance  124  is not complete, so the computing device  121  continues processing the audio data of the utterance  124 . During pause  154 , the computing device  121  detects a novice length pause and the complete query signal  106  indicates that the utterance is complete and, therefore, generates an endpoint of the utterance  124  as indicated by the novice endpoint signal  112 . When the user  127  is classified as a novice, the endpoint of the utterance  124  is after word  151 , and the transcription  160  of the utterance  124  is “Text Mom love you.” 
         [0024]    The expert pause detector signal  115  illustrates the computing device  121  detecting a pause in audio data corresponding to utterance  124 , where the detected pause length is shorter than the pause length the corresponds to the general endpointer. For example, the computing device  121  may detect pauses with a length of three hundred milliseconds when the user  127  is classified as an expert user. Applying this pause threshold to utterance  124 , the computing device  121  detects expert length pauses during pauses  136 ,  142 ,  148 , and  154 . Because none of the pauses are less than three hundred milliseconds, all of the pauses in utterance  124  include an expert length pause detection. 
         [0025]    The computing device  121  combines the expert pause detector signal  115  and the complete query signal  106  to determine a speech endpoint for the utterance  124  when the computing device classifies the user as an expert. When the computing device  121  detects a pause, such as the pause of the expert pause detector signal  115  during pause  136 , the computing device  121  determines whether the utterance  124  is complete. During pause  136 , the complete query signal  106  indicates that the utterance  124  is not complete. Even though the computing device  121  detected an expert length pause, the utterance  124  is not complete, so the computing device  121  continues processing the audio data of the utterance  124 . During pause  142 , the computing device  121  detects an expert length pause and the complete query signal  106  indicates that the utterance is complete and, therefore, generates an endpoint of the utterance  124  as indicated by the expert endpoint signal  118 . When the user  127  is classified as an expert, the endpoint of the utterance  124  is after word  139 , and the transcription  163  of the utterance  124  is “Text Mom.” 
         [0026]      FIG. 2  is diagram of an example system  200  that classifies a particular user based on the particular user&#39;s experience with speech input. In some implementations, the system  200  may be included in a computing device that the particular user uses for speech input, such as computing device  121 . In some implementations, the system may be included in a server that processes transcriptions of speech input. 
         [0027]    The system  200  includes voice queries  205 . The voice query log  205  stores the previous voice queries that users provide to the system  200 . The voice query log  205  may include search queries, for example, “cat videos,” and command queries, for example, “call mom.” The voice query log  205  may include for each stored voice query, a timestamp, data indicating the duration of each pause between words, and data indicating whether the voice query is complete or incomplete based on a comparison with other voice queries. 
         [0028]    Query log  210  illustrates the voice queries provided by the user Bob. The voice queries in query log  210  include three voice queries and each includes either a complete indicator “[C]” or an incomplete indicator “[I].” Each voice query includes a timestamp that notes the date and time that Bob spoke the voice query. Each voice query includes data indicating the pause intervals between the spoken words. For example, “cat videos” may include data to indicate that Bob paused two hundred milliseconds between “cat” and “video.” “Call . . . mom” may include data to indicate that Bob paused one second between “call” and “mom.” 
         [0029]    Query log  215  illustrates the voice queries provided by the user Alice. The voice queries in query log  215  include five voice queries and each includes either a complete indicator “[C]” or an incomplete indicator “[I].” Each voice query includes a timestamp that notes the date and time that Alice spoke the voice query. Each voice query includes data indicating the pause intervals between the spoken words. For example, “Text Sally that I&#39;ll be ten minutes late” may include data to indicate that Alice paused one millisecond between “text” and “Sally,” paused three hundred milliseconds between “Sally” and “that,” and paused 1.5 seconds between “that” and “I&#39;ll,” as well as pause intervals between the other words. “Call mom” may include data to indicate that Alice paused three milliseconds between “call” and “mom.” 
         [0030]    The voice query processor  220  processes the voice queries received from the voice query log  205 . The voice query processor  220  generates a voice query experience score for each user. The voice query experience score indicates a level of experience that a particular user has with voice queries. A higher voice query experience score indicates that the particular user may have more experience with speaking voice queries. For example, to generate the voice query experience score for Bob, the voice query processor  220  processes query log  210 . 
         [0031]    The voice query processor  220  includes a query completeness processor  225 . The query completeness processor  225  accesses, for each user, the completeness data for each voice query and adjusts the user&#39;s voice query experience score. If a particular user has more complete voice queries and fewer incomplete voice queries, then the query completeness processor  225  increases the voice query experience score. If a particular user has more incomplete voice queries and fewer complete voice queries, then the query completeness processor  225  decreases the voice query experience score. The query completeness processor  225  may compare a ratio of complete queries to incomplete queries to a completeness ratio threshold in determining whether to increase or decrease the voice query experience score. For example, Bob has one incomplete voice query and two complete queries. Based on that information, the query completeness processor  225  may decrease Bob&#39;s voice query experience score. Alice has no incomplete voice queries and five complete queries. Based on that information, the query completeness processor  225  may increase Alice&#39;s voice query experience score. 
         [0032]    The voice query processor  220  includes a query length processor  230 . The query length processor  230  computes, for each user, a length of each voice query and adjusts the user&#39;s voice query experience score. If a particular user has more long voice queries and fewer short voice queries, then the voice query processor  220  increases the voice query experience score. If a particular user has more short voice queries and fewer long voice queries, then the voice query processor  220  decreases the voice query experience score. The query length processor  230  may use a threshold to determine whether a voice query is long or short. The query length processor  230  may compare a ratio of long queries to short queries to a length ratio threshold in determining whether to increase or decrease the voice query experience score. For example, Bob has one incomplete voice query and two complete queries. Based on that information, the query completeness processor  225  may decrease Bob&#39;s voice query experience score. Alice has no incomplete voice queries and five complete queries. Based on that information, the query completeness processor  225  may increase Alice&#39;s voice query experience score. 
         [0033]    The voice query processor  220  includes a pause interval processor  235 . The pause interval processor  235  computes, for each user, an average pause length between words for the user&#39;s voice queries. The pause interval processor  235  may compare the average pause length for each user to a pause threshold to determine whether to increase or decrease the user&#39;s voice query experience score. An average pause length above the pause threshold decreases the voice query experience score. An average pause length below the pause threshold increases the voice query experience score. For example, Bob may have an average pause length of 1.2 seconds. Alice may have an average pause length of two hundred milliseconds. If the pause threshold is one second, then the pause interval processor  235  increases the voice query experience score for Alice and decreases the voice query experience score for Bob. 
         [0034]    The voice query processor  220  includes a query counter  240 . The query counter  240  computes, for each user, a number of voice queries submitted and adjusts the voice query experience score. If a particular user provides many voice queries, then the voice query processor  220  increases the voice query experience score. If a particular user provides fewer voice queries, then the voice query processor  220  decreases the voice query experience score. The query length processor  230  may use a daily voice query threshold and compare the threshold to an average daily number of voice queries to determine whether a user submits many or few voice queries. For example, the daily voice query threshold is one query per day. Based on query log  210 , Bob submitted less than one query per day. Therefore, the query counter  240  decreases Bob&#39;s voice query experience score. Alice has more than one query per day based on query log  215 . Therefore, the query counter  240  increases Alice&#39;s voice query experience score. 
         [0035]    Each of the processors included in the voice query processor  220  may increase or decrease the user&#39;s voice query experience score an amount that is proportional to a difference between the threshold and the corresponding user&#39;s value. For example, Alice averages 2.5 voice queries per day, and Bob averages 0.125 queries per day. With a daily voice query threshold of one query per day, Alice is 1.5 over the threshold, and Bob is 0.875 below the threshold. The query counter  240  increases Alice&#39;s voice query experience score by some factor multiplied by 1.5 and decreases Bob&#39;s voice query experience score by the same factor multiplied by 0.875. 
         [0036]    The voice query processor  220  provides each user&#39;s voice query experience score to a user profiler  245 . The user profiler  245  assigns a voice query experience profile to each user based the user&#39;s voice query experience score. The voice profiler  245  accesses the profile thresholds  250  to match each user&#39;s voice query experience score to a voice query experience profile. Each voice query experience profile may correspond to a range of voice query experience score. For example, an expert profile may correspond to a range of 80 to 100. A novice profile may correspond to a range of 0 to 20. Other profiles may exist between the voice query experience scores of 20 and 80. In the example in  FIG. 2 , Bob may have a voice query experience score of 18 and Alice may have a voice query experience score of 88. Therefore, Bob is classified as a novice, and Alice is classified as an expert. The user profiler  245  then stores the voice query experience profile for each user in profile storage  255 . Profiles  260  illustrates the profiles of Alice and Bob stored in profile storage  255   
         [0037]    Each voice query experience profile corresponds to a different pause length threshold that the system  200  subsequently uses when generating an endpoint for a future utterance. As shown in  FIG. 1  and described above, the expert profile corresponds to a pause length threshold of three hundred milliseconds that is used to endpoint an utterance. The novice profile corresponds to a pause length threshold of one second. The system  200  may define other profiles such as an intermediate profile with a pause length threshold of six hundred milliseconds. In some implementations, the system  200  may assign a pause length threshold to a user without assigning a profile to the user. The system may generate a voice query experience score and compute a pause length threshold that is inversely proportional to the voice query experience score. 
         [0038]      FIG. 3  is a diagram of an example process  300  for classifying a particular user based on the particular user&#39;s experience with speech input and determining whether the particular user has finished speaking a voice query. The process  300  may be performed by a computing device such as computing device  121  from  FIG. 1  or computing device  200  from  FIG. 2 . The process  300  analyzes a user&#39;s previous voice queries to determine a pause threshold for use in determining when the user has finished speaking future queries. 
         [0039]    The computing device accesses voice query log data ( 310 ). The computing device determines, for a particular user, a pause threshold ( 320 ). The computing device receives, from the particular user, an utterance ( 330 ). The computing device determines that the particular user has stopped speaking for the pause threshold ( 340 ). The computing device processes the utterance as a voice query ( 350 ). 
         [0040]      FIG. 4  is a block diagram of computing devices  400  that may be used to implement the systems and methods described in this document, as either a client or as a server or plurality of servers. Computing device  400  is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. 
         [0041]    Computing device  400  includes a processor  402 , memory  404 , a storage device  406 , a high-speed interface  408  connecting to memory  404  and high-speed expansion ports  410 , and a low speed interface  412  connecting to low speed bus  414  and storage device  406 . Each of the components  402 ,  404 ,  406 ,  408 ,  410 , and  412 , are interconnected using various busses, and may be mounted on a common motherboard or in other manners as appropriate. The processor  402  can process instructions for execution within the computing device  400 , including instructions stored in the memory  404  or on the storage device  406  to display graphical information for a GUI on an external input/output device, such as display  416  coupled to high speed interface  408 . In other implementations, multiple processors and/or multiple buses may be used, as appropriate, along with multiple memories and types of memory. Also, multiple computing devices  400  may be connected, with each device providing portions of the necessary operations, e.g., as a server bank, a group of blade servers, or a multi-processor system. 
         [0042]    The computing device  400  may be implemented in a number of different forms, as shown in the figure. For example, it may be implemented as a standard server  420 , or multiple times in a group of such servers. It may also be implemented as part of a rack server system  424 . In addition, it may be implemented in a personal computer such as a laptop computer  422 . Alternatively, components from computing device  400  may be combined with other components in a mobile device (not shown). Each of such devices may contain one or more of computing device  400  and an entire system may be made up of multiple computing devices  400  communicating with each other. 
         [0043]    Various implementations of the systems and methods described here can be realized in digital electronic circuitry, integrated circuitry, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and/or combinations of such implementations. These various implementations can include implementation in one or more computer programs that are executable and/or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device. 
         [0044]    The systems and techniques described here can be implemented in a computing system that includes a back end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front end component, e.g., a client computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here, or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (“LAN”), a wide area network (“WAN”), and the Internet. 
         [0045]    A number of embodiments have been described. Nevertheless, it will be understood that various modifications may be made without departing from the spirit and scope of the invention. In addition, the logic flows depicted in the figures do not require the particular order shown, or sequential order, to achieve desirable results. In addition, other steps may be provided, or steps may be eliminated, from the described flows, and other components may be added to, or removed from, the described systems. Accordingly, other embodiments are within the scope of the following claims.