Patent Description:
This specification generally describes electronic communications.

Speech synthesis refers to the artificial production of human speech. Speech synthesizers can be implemented in software or hardware components to generate speech output corresponding to a text. For instance, a text-to-speech (TTS) system typically converts normal language text into speech by concatenating pieces of recorded speech that are stored in a database.

<CIT> discloses a device for practicing language wherein an inputted sentence understanding part performs the syntax analysis of the sentence inputted by the learner so as to generate a syntax analysis tree, and information with which the level of the ability for language of the learner can be detected is extracted. The dictionary which is used in such a case and the rule of grammar are stored in a grammar dictionary knowledge. Next, the semantic expression is generated from the understanding part and the syntax tree and transferred to a conversation control part. The control part calculates the synthetic level of the ability for language of the learner and decides the contents of speech by a device in accordance with a conversation control rule stored in a conversation control knowledge, then gives them to an outputted sentence generation part. The generation part receives the decided contents of speech and information on the level of the ability for language, generates the outputted sentence based on an outputted sentence generation knowledge and outputs it to display.

Speech synthesis has become more central to user experience as a greater portion of electronic computing has shifted from desktop to mobile environments. For example, increases in the use of smaller mobile devices without displays have led to increases in the use of text-to-speech (TTS) systems for accessing and using content that is displayed on mobile devices.

This specification discloses improved user interfaces, in particular better computer-to-user communication through improved TTS.

One particular issue with existing TTS systems is that such systems are often unable to adapt to varying language proficiencies of different users. This lack of flexibility often prevents users with limited language proficiencies from understanding complex text-to-speech outputs. For instance, non-native language speakers that use a TTS system can have difficulty understanding a text-to-speech output because of their limited language familiarity. Another issue with existing TTS systems is that a user's instantaneous ability to understand text-to-speech outputs can also vary based on a particular user context. For instance, some user contexts include background noise that can make it more difficult to understand longer or more complex text-to-speech outputs.

In some implementations, a system adjusts the text used for a text-to-speech output based on the language proficiency of a user to increase a likelihood that the user can comprehend the text-to-speech output. For instance, the language proficiency of a user can be inferred from prior user activity and be used to adjust the text-to-speech output to an appropriate complexity that is commensurate with the language proficiency of the user. In some examples, a system obtains multiple candidate text segments that correspond to different levels of language proficiency. The system then selects the candidate text segment that best matches and most closely corresponds to a user's language proficiency and provides a synthesized utterance of the selected text segment for output to the user. In other examples, a system alters the text in a text segment to better correspond to the user's language proficiency prior to generating a text-to-speech output. Various aspects of a text segment can be adjusted, including the vocabulary, sentence structure, length, and so on. The system then provides a synthesized utterance of the altered text segment for output to the user.

For situations in which the systems discussed here collect personal information about users, or may make use of personal information, the users may be provided with an opportunity to control whether programs or features collect personal information, e.g., information about a user's social network, social actions or activities, profession, a user's preferences, or a user's current location, or to control whether and/or how to receive content from the content server that may be more relevant to the user. In addition, certain data may be anonymized in one or more ways before it is stored or used, so that personally identifiable information is removed. For example, a user's identity may be anonymized so that no personally identifiable information can be determined for the user, or a user's geographic location may be generalized where location information is obtained, such as to a city, zip code, or state level, so that a particular location of a user cannot be determined. Thus, the user may have control over how information is collected about him or her and used by a content server.

In one aspect, a computer-implemented method can include: determining, by the one or more computers, a language proficiency of a user of a client device; determining, by the one or more computers, a text segment for output by a text-to-speech module based on the determined language proficiency of the user; generating, by the one or more computers, audio data including a synthesized utterance of the text segment; and providing, by the one or more computers and to the client device, the audio data including the synthesized utterance of the text segment.

Other versions include corresponding systems, configured to perform the actions of the methods encoded on computer storage devices.

One or more implementations can include the following optional features. For example, in some implementations, the client device displays a mobile application that uses a text-to-speech interface.

In some implementations, determining the language proficiency of the user includes inferring a language proficiency of the user based at least on previous queries submitted by the user.

In some implementations, determining the text segment for output by the text-to-speech module includes: identifying multiple text segments as candidates for a text-to-speech output of the user, the multiple text segments having different levels of language complexity; and selecting from among the multiple text segments based at least on the determined language proficiency of the user of the client device.

In some implementations, selecting from among the multiple text segments includes: determining a language complexity score for each of the multiple text segments; and selecting the text segment having the language complexity score that best matches a reference score that describes the language proficiency of the user of the client device.

In some implementations, determining the text segment for output by the text-to-speech module includes: identifying a text segment for a text-to-speech output to the user; computing a complexity score of the text segment for the text-to-speech output; and modifying the text segment for the text-to-speech output to the user based at least on the determined language proficiency of the user and the complexity score of text segment for the text-to-speech output.

In some implementations, modifying the text segment for the text-to-speech output to the user includes: determining an overall complexity score for the user based at least on the determining language proficiency of the user; determining a complexity score for individual portions within the text segment for the text-to-speech output to the user; identifying one or more individual portions within the text segment with complexity scores greater than the overall complexity score for the user; and modifying the one or more individual portions within the text segment to reduce complexity scores below the overall complexity score.

In some implementations, modifying the text segment for the text-to-speech output to the user includes: receiving data indicating a context associated with the user; determining an overall complexity score for the context associated with the user; determining that the complexity score of the text segment exceeds the overall complexity score for the context associated with the user; and modifying the text segment to reduce the complexity score below the overall complexity score for the context associated with the user.

In another aspect not forming part of the invention, a computer program comprises machine readable instructions that when executed by computing apparatus causes it to perform any of the above methods.

Other potential features and advantages will become apparent from the description, the drawings, and the claims.

In the drawings, like reference numbers represent corresponding parts throughout.

<FIG> is a diagram that illustrates examples of processes 100A and 100B for generating text-to-speech outputs based on language proficiency. The processes 100A and 100B are used to generate different text-to-speech outputs for a user 102a with high language proficiency and a user 102b with low language proficiency, respectively, for a text query <NUM>. As depicted, after receiving a query <NUM> on the user devices 106a and 106b, the process 100A generates a high-complexity text-to-speech output 108a for the user 102a whereas the process 100B generates a low-complexity output 108b for the user 102b. In addition, the TTS systems that execute processes 100A and 100B can include a language proficiency estimator <NUM>, a text-to-speech engine <NUM>. In addition, the text-to-speech engine <NUM> can further include a text analyzer <NUM>, a linguistics analyzer <NUM>, and a waveform generator <NUM>.

In general, the content of text that is used to generate a text-to-speech output can be determined according to a language proficiency of a user. In addition, or as an alternative, the text to be used to generate a text-to-speech output can be determined based on a context of the user, for example, the location or activity of a user, background noise present, a current task of the user and so on. Further, the text to be converted to an audible form may be adjusted or determined using other information, such as indications that a user has failed to complete a task or is repeating an action.

In the example, two users, user 102a and user 102b, provide the same query <NUM> on user devices 106a and 106b, respectively, as input to an application, web page, or other search functionality. For instance, the query <NUM> can be a voice query sent to the user devices 106a and 106b to determine a weather forecast for the current day. The query <NUM> is then transmitted to the text-to-speech engine <NUM> to generate a text-to-speech output in response to the query <NUM>.

The language proficiency estimator <NUM> can be a software module within a TTS system that determines a language proficiency score associated with a particular user (e.g., the user 102a or the user 102b) based on user data 108a. The language proficiency score can be an estimate of the user's ability to understand communications in a particular language, in particular, to understand speech in the particular language. One measure of language proficiency is the ability of a user to successfully complete a voice-controlled task. Many types of tasks, such as setting a calendar appointment, looking up directions, and so on, follow a sequence of interactions in which a user and device exchange verbal communication. The rate at which a user successfully completes these task workflows through a voice interface is a strong indicator of the user's language proficiency. For example, a user that completes nine out of ten voice tasks that the user initiates likely has a high language proficiency. On the other hand, a user that fails to complete the majority of voice tasks that the user initiates can be inferred to have a low language proficiency, since the user may not have fully understood the communications from the device or may not have been able to provide appropriate verbal responses. As discussed further below, when a user does not complete workflows that include standard TTS outputs, resulting in a low language proficiency score, the TTS may use adapted, simplified outputs that may increase the ability of the user to understand and complete various tasks.

As shown, the user data 108a can include words used within prior text queries submitted by the user, an indication whether English, or any other language utilized by the TTS system, is the native language of the user, and a set of activities and/or behaviors that are reflective of a user's language comprehension skills. For example, as depicted in <FIG>, a typing speed of the user can be used to determine language fluency of the user in a language. In addition, a language vocabulary complexity score or language proficiency score can be assigned to the user based on associating a pre-determined complexity to words that were used by the user in previous text queries. In another example, the number of misrecognized words in prior queries can also be used to determine the language proficiency score. For instance, a high number of misrecognized words can be used to indicate a low language proficiency. In some implementations, the language proficiency score is determined by looking up a stored score associated with the user, which was determined for the user prior to submission of the query <NUM>.

Although <FIG> depicts the language proficiency estimator <NUM> as a separate component to the TTS engine <NUM>, in some implementations, as depicted in <FIG>, the language proficiency estimator <NUM> can be an integrated software module within the TTS engine <NUM>. In such instances, operations involving the language proficiency estimation can be directly modulated by the TTS engine <NUM>.

In some implementations, the language proficiency score assigned to the user may be based on a particular user context estimated for the user. For instance, as described more particularly with respect to <FIG>, a user context determination can be used to determine context-specific language proficiencies that can cause a user to temporarily have limited language comprehension abilities. For example, if the user context indicates significant background noise or if the user is engaged in a task such as driving, the language proficiency score can be used to indicate that the user's present language comprehension ability is temporarily diminished relative to other user contexts.

In the invention, instead of inferring language proficiency based on previous user activity, the language proficiency score is instead directly provided to the TTS engine <NUM> without the use of the language proficiency estimator <NUM>. A language proficiency score is designated to a user based on user input during a registration process that specifies a user's level of language proficiency. For example, during the registration, the user can provide a selection that specifies the user's skill level, which can then be used to calculate the appropriate language proficiency for the user. In the invention, the user provides demographic information, and may provide further types of information, such as education level, places of residences, etc., that can be used to specify the user's level of language proficiency.

In the examples described above, the language proficiency score can either be a set of discrete values that are adjusted periodically based on recently generated user activity data, or a continuous score that is initially designated during a registration process. In the first instance, the value of the language proficiency score can be biased based on one or more factors that indicate that a user's present language comprehension and proficiency may be attenuated (e.g., a user context indicating significant background noise). In the second instance, the value of the language proficiency score can be preset after an initial calculation and adjusted only after specific milestone events that indicate that a user's language proficiency has increased (e.g., an increase in typing rate or a decrease in correction rate for a given language). In other instances, a combination of these two techniques can be used to variably adjust the text-to-speech output based on a particular text input. In such instances, multiple language proficiency scores that each represent a particular aspect of the user's language skills can be used to determine to how best adjust the text-to-speech output for the user. For example, one language proficiency score can represent a complexity of the user's vocabulary whereas another language proficiency score can be used to represent the user's grammar skills.

The TTS engine <NUM> can use the language proficiency score to generate a text-to-speech output that is adapted to the language proficiency indicated by the user's language proficiency score. In some instances, the TTS engine <NUM> adapts the text-to-speech output based on selecting a particular TTS string from a set of candidate TTS strings for the text query <NUM>. In such instances, the TTS engine <NUM> selects the particular TTS string based on using the language proficiency score of the user to predict a likelihood that each of the candidate TTS strings will accurately be interpreted by the user. More particular descriptions related to these techniques are provided with respect to <FIG>. Alternatively, in other instances, the TTS engine <NUM> can select a baseline TTS string and adjust the structure of the TTS string based on the user's level of language proficiency indicated by the language proficiency score. In such instances, the TTS engine <NUM> can adjust the grammar of the baseline TTS string, provide word substitutions and/or reduce the sentence complexity to generate an adapted TTS string that is more likely to be understood by the user. More particular descriptions related to these techniques are provided with respect to <FIG>.

Referring still to <FIG>, the TTS engine <NUM> may generate different text-to-speech outputs for users 102a and 102b because the language proficiency scores for the users are different. For example, in process 100A, the language proficiency score 106a indicates high English-language proficiency, inferred from the user data 108a indicating that the user 102a has a complex vocabulary, has English as a first language, and has a relatively high word per minute in prior user queries. Based on the value of the language proficiency score 106a, the TTS engine <NUM> generates a high complexity text-to-speech output 108a that includes a complex grammatical structure. As depicted, the text-to-speech output 108a includes an independent clause that describes that today's forecast is sunny, in addition to a subordinate clause that includes additional information about the high temperature and the low temperature of the day.

In the example of process 100B, the language proficiency score 106b indicates low English-language proficiency, inferred from user activity data 108b indicating that the user 102b has a simple vocabulary, has English as a second language, and has previously provided ten incorrect queries. In this example, the TTS engine <NUM> generates a low complexity text-to-speech output 108b that includes a simpler grammatical structure relative to the text-to-speech output 108a. For instance, instead of including multiple clauses within a single sentence, the text-to-speech output 108b includes a single independent clause that conveys the same primary information as the text-to-speech output 108a (e.g., today's forecast being sunny), but does not include additional information related to the high and low temperatures for the day.

The adaptation of text for a TTS output can be performed by various different devices and software modules. For example, a TTS engine of a server system may include functionality to adjust text based on a language proficiency score and then output audio including a synthesized utterance of the adjusted text. As another example, a pre-processing module of a server system may adjust text and pass the adjusted text to a TTS engine for speech synthesis. As another example, a user device may include a TTS engine, or a TTS engine and a text pre-processor, to be able to generate appropriate TTS outputs.

In some implementations, a TTS system can include software modules that are configured to exchange communications with a third-party mobile application of a client device or a web page. For instance, the TTS functionality of the system can be made available to a third-party mobile application through an application package interface (API). The API can include defined set of protocols that an application or web site can use to request TTS audio from a server system that runs the TTS engine <NUM>. In some implementations, the API can make available TTS functionality that runs locally on a user's device. For example, the API may be available to an application or web page through an inter-process communication (IPC), remote procedure call (RPC), or other system call or function. A TTS engine, and associated language proficiency analysis or text preprocessing, may be run locally on the user's device to determine an appropriate text for the user's language proficiency and generate the audio for the synthesized speech also.

For example, the third-party application or web page can use the API to generate a set of voice instructions that are provided to the user based on a task flow of a voice interface of the third-party application or web page. The API can specify that the application or web page should provide text to be converted to speech. In some instances, other information can be provided, such as a user identifier or a language proficiency score.

In implementations where the TTS engine <NUM> exchanges communications with a third-party application using an API, the TTS engine <NUM> can be used to determine whether a text segment from a third-party application should be adjusted prior to generating a text-to-speech output for the text. For example, the API can include computer-implemented protocols that specify conditions within the third-party application that initiate the generation of an adaptive text-to-speech output.

As an example, one API may permit an application to submit multiple different text segments as candidates for a TTS output, where the different text segments correspond to different levels of language proficiency. For example, the candidates can be text segments having equivalent meanings but different complexity levels (e.g., a high complexity response, a medium complexity response, and a low complexity response). The TTS engine <NUM> may then determine the level of language proficiency needed to understand each candidate, determine an appropriate language proficiency score for the user, and select the candidate text that best corresponds to the language proficiency score. The TTS engine <NUM> then provides synthesized audio for the selected text back to the application, e.g., over a network using the API. In some instances, the API can be locally available on the user devices 106a and 106b. In such instances, the API can be accessible over various types of inter-process communication (IPC) or via a system call. For example, the output of the API on the user devices 106a and 106b can be the text-to-speech output of the TTS engine <NUM> since the API operates locally on the user devices 106a and 106b.

In another example, an API can allow the third-party application to provide a single text segment and a value that indicates whether the TTS engine <NUM> is permitted to modify the text segment to generate a text segment with a different complexity. If the app or web page indicates that alteration is permitted, the TTS system <NUM> may make various changes to the text, for example, to reduce the complexity of the text when the language proficiency score suggests that the original text is more complex than the user can understand in a spoken response. In yet other examples, an API allows the third-party application to also provide user data (e.g., prior user queries submitted on the third-party application) along with the text segment such that the TTS engine <NUM> can determine a user context associated with the user and adjust generate a particular text-to-speech output based on the determined user context. Similarly, an API can allow an application to provide context data from a user device (e.g., a global positioning signal, accelerometer data, ambient noise level, etc.) or an indication of a user context to allow the TTS engine <NUM> to adjust the text-to-speech outputs that will ultimately be provided to the user through the third-party application. In some instances, the third party application can also provide the API with data that can be used to determine a language proficiency of the user.

In some implementations, the TTS engine <NUM> can adjust the text-to-speech output for a user query without using a language proficiency of the user or determining a context associated with the user. In such implementations, TTS engine <NUM> can determine that an initial text-to-speech output is too complex for a user based on receiving signals that a user has misunderstood the output (e.g., multiple retries on the same query or task). In response, the TTS engine <NUM> can reduce the complexity of a subsequent text-to-speech response for a retried query or related queries. Thus, when a user fails to successfully complete an action, the TTS engine <NUM> may progressively reduce the amount of detail or language proficiency required to understand the TTS output until it reaches a level that the user understands.

<FIG> is a diagram that illustrates an example of a system <NUM> that adaptively generates a text-to-speech output based on a user context. Briefly, the system <NUM> can include a TTS engine <NUM> that includes a query analyzer <NUM>, a language proficiency estimator <NUM>, an interpolator <NUM>, a linguistics analyzer <NUM>, a re-ranker <NUM>, and a waveform generator <NUM>. The system <NUM> also includes a context repository <NUM> that stores a set of context profiles <NUM>, and a user history manager <NUM> that stores user history data <NUM>. In some instances, the TTS engine <NUM> corresponds to the TTS engine <NUM> as described with respect to <FIG>.

In the example, a user <NUM> initially submits a query <NUM> on a user device <NUM> that includes a request for information related to the user's first meeting for the day. The user device <NUM> can then transmit the query <NUM> and context data <NUM> associated with the user <NUM> to the query analyzer <NUM> and the language proficiency estimator <NUM>, respectively. Other types of TTS outputs that are not responses to queries, e.g., calendar reminders, notifications, task workflows, etc., may be adapted using the same techniques.

The context data <NUM> can include information relating to a particular context associated with the user <NUM> such as time intervals between repeated text queries, global positioning signal (GPS) data indicating a location, speed, or movement pattern associated with the user <NUM>, prior text queries submitted to the TTS engine <NUM> within a particular time period, or other types of background information that can indicate user activity related to the TTS engine <NUM>. In some instances, the context data <NUM> can indicate a type of query <NUM> submitted to the TTS engine <NUM>, such as whether the query <NUM> is a text segment associated with a user action, or an instruction transmitted to the TTS engine <NUM> to generate a text-to-speech output.

After receiving the query <NUM>, the query analyzer <NUM> parses the query <NUM> to identify information that is responsive to the query <NUM>. For example, in some instances where the query <NUM> is a voice query, the query analyzer <NUM> initially generates a transcription of the voice query, and then processes individual words or segments within the query <NUM> to determine information that is responsive to the query <NUM>, for example, by providing the query to a search engine and receive search results. The transcription of the query and the identified information can then be transmitted to the linguistics analyzer <NUM>.

Referring to now to the language proficiency estimator <NUM>, after receiving the context data <NUM>, the language proficiency estimator <NUM> computes a language proficiency for the user <NUM> based on the received context data <NUM> using techniques described with respect to <FIG>. In particular, the language proficiency estimator <NUM> parses through various context profiles <NUM> stored on the repository <NUM>. The context profile <NUM> can be an archived library including related types of information that are associated with a particular user context and can be included within a text-to-speech output. The context profile <NUM> additionally specifies a value, associated with each type of information, which represents an extent to which each type of information is likely to be understood by the user <NUM> when the user <NUM> is presently within a context associated with the context profile <NUM>.

In the example depicted in <FIG>, the context profile <NUM> specifies that the user <NUM> is presently in a context indicating that the user <NUM> is on his/her daily commute to work. In addition, the context profile <NUM> also specifies values for individual words and phrases that are likely to be comprehended by the user <NUM>. For instance, data or time information is associated with a value of "<NUM>" for "SINCE," indicating that the user <NUM> is more likely to understand generalized information associated with a meeting (e.g., time of the next upcoming meeting) <NUM> rather than detailed information associated with a meeting (e.g., a party attending the meeting, or location of the meeting. In this example, the differences of the values indicate differences in the user's ability to understand particular types of information because the user's ability to understand complex or detailed information is diminished.

The value associated with individual words and phrases can determined based on user activity data from previous user sessions where the user <NUM> was previously in the context indicated by the context data <NUM>. For instance, historical user data can be transmitted from the user history manager <NUM>, which retrieves data stored within the query logs <NUM>. In the example, the value for date and time information can be increased based on determining that the user commonly accesses date and time information associated with meetings more frequently than locations of the meetings.

After the language proficiency estimator <NUM> selects a particular context profile <NUM> that corresponds to the received context data <NUM>, the language proficiency estimator <NUM> transmits the selected context profile <NUM> to the interpolator <NUM>. The interpolator <NUM> parses the selected context profile <NUM>, and extracts individual words and phrases included and their associated values. In some instances, the interpolator <NUM> transmits the different types of information and associated values directly to the linguistics analyzer <NUM> for generating a list of text-to-speech output candidates 240a. In such instances, the interpolator <NUM> extracts specific types of information and associated values from the selected context profile <NUM> and transmits them to the linguistics analyzer <NUM>. In other instances, the interpolator <NUM> can also transmit the selected context profile <NUM> to the re-ranker <NUM>.

In some instances, the TTS engine <NUM> can be provided a set of structured data (e.g., fields of a calendar event). In such instances, the interpolator <NUM> can convert the structured data to text at a level that matches the user's proficiency indicated by the context profile <NUM>. For example, the TTS engine <NUM> may access data indicating one or more grammars indicating different levels of detail or complexity to express the information in the structured data, and select an appropriate grammar based on the user's language proficiency score. Similarly, the TTS engine <NUM> can use dictionaries to select words that are appropriate given the language proficiency score.

The linguistics analyzer <NUM> performs processing operations such as normalization on the information included within the query <NUM>. For instance, the query analyzer <NUM> can assign phonetic transcriptions to each word or snippet included within the query <NUM>, and divide the query <NUM> into prosodic units such as phrases, clauses, and sentences using a text-to-phenome conversion. The linguistics analyzer <NUM> also generates a list 240a that includes multiple text-to-speech output candidates that are identified as being responsive to the query <NUM>. In the example, the list 240a includes multiple text-to-speech output candidates with different levels of complexity. For example, the response "At <NUM>:00PM with Mr. John near Dupont Circle" is the most complex response because it identifies a time for the meeting, a location for the meeting, an individual with whom the meeting will take place. In comparison, the response "In three hours" is the least complex because it only identifies a time for the meeting.

The list 240a also includes a baseline rank for the text-to-speech candidates based on the likelihood that each text-to-speech output candidate is likely to be responsive to the query <NUM>. In the example, the list 240a indicates that most complex text-to-speech output candidate is the most likely to be responsive to the query <NUM> because it includes the greatest amount of information that is associated with the content of the query <NUM>.

After the linguistics analyzer generates the list 240a of text-to-speech output candidates, the re-ranker <NUM> generates a list 240b, which includes an adjusted rank for the text-to-speech output candidates based on the received context data <NUM>. For instance, the re-ranker <NUM> can adjust the rank based on the scores associated with particular types of information included within the selected context profile <NUM>.

In the example, the re-ranker <NUM> ranks the simplest text-to-speech output as the highest based on the context profile <NUM> indicating that the user <NUM> is likely to comprehend date and time information within a text-to-speech response but not likely to understand party names or location information within the text-to-speech response given the present context of the user indicating that the user is commuting to work. In this regard, the received context data <NUM> can be used to adjust the selection of a particular text-to-speech output candidate that to increase the likelihood that the user <NUM> will understand the contents of the text-to-speech output 204c of the TTS engine <NUM>.

<FIG> is a diagram that illustrates an example of a system <NUM> for modifying sentence structure within a text-to-speech output. Briefly, a TTS engine <NUM> receives a query <NUM> and a language proficiency profile <NUM> for a user (e.g., the user <NUM>). The TTS engine <NUM> then perform operations <NUM>, <NUM>, and <NUM> to generate an adjusted text-to-speech output 302c that is responsive to the query <NUM>. In some instances, the TTS engine <NUM> corresponds to the TTS engine <NUM> described with respect to <FIG>, or the TTS engine <NUM> described with respect to <FIG>.

In general, the TTS engine <NUM> can modify the sentence structure of a baseline text-to-speech output 306a for the query <NUM> using different types of adjustment techniques. As an example, the TTS engine <NUM> can substitute words or phrases within the baseline text-to-speech output 306a based on determining that a complexity score associated with individual words or phrases is greater than a threshold score indicated by the language complexity profile <NUM> of a user. As another example, the TTS engine <NUM> can rearrange individual sentence clauses such that the overall complexity of baseline text-to-speech output 306a is reduced to a satisfactory level based on the language complexity profile <NUM>. The TTS engine <NUM> can also re-order words, split or combine sentences, and make other changes to adjust the complexity of text.

In more detail, during the operation <NUM>, the TTS engine <NUM> initially generates a baseline text-to-speech output 306a that is responsive to the query <NUM>. The TTS engine <NUM> then parses the baseline text-to-speech output 306a into segments 312a-312c. The TTS engine <NUM> also detects punctuation marks (e.g., commas, periods, semicolons, etc.) that indicate breakpoints between individual segments. The TTS engine <NUM> also computes a complexity score for each of the segments 312a-312c. In some instances, the complexity score can be computed based on the frequency of a particular word within a particular language. Alternative techniques can include computing the complexity score based on the frequency of use by the user, or frequency of occurrence in historical content accessed by the user (e.g., news articles, webpages, etc.). In each of these examples, the complexity score can be used to indicate words that are likely to be comprehended by the user and other words that are unlikely to be comprehended by the user.

In the example, segments 312a and 312b are determined to be relatively complex based on the inclusion of high complex terms such as "FORECAST" and "CONSISTENT," respectively. However, the segment 312c is determined to be relatively simple because the terms included are relatively simple. This determination is represented by the segments 312a and 312b having higher complexity scores (e.g., <NUM>, <NUM>) compared to the complexity score for the segment 312c (e.g., <NUM>).

As described above, the language proficiency profile <NUM> can be used to compute a threshold complexity score that indicates the maximal complexity that is comprehendible by the user. In the example, the threshold complexity score can be computed to be "<NUM>" such that the TTS <NUM> determines that the segments 312a and 312b are unlikely to be comprehended by the user.

After identifying individual segments with associated complexity scores greater than the threshold complexity score indicated by the language proficiency profile <NUM>, during the operation <NUM>, the TTS engine <NUM> substitutes the identified words with alternates that are predicted to be more likely to be understood by the user. As depicted in <FIG>, "FORECAST" can be substituted with "WEATHER," and "CONSISTENT" can be substituted with "CHANGE. " In these examples, segments 314an and 314b represent simpler alternatives with lower complexity scores below the threshold complexity score indicated by the language proficiency profile <NUM>.

In some implementations, TTS engine <NUM> can process word substitutions for high complexity words using a trained skip-gram model that uses unsupervised techniques to determine appropriately complex words to replace highly complex words. In some instances, the TTS engine <NUM> can also use thesaurus or synonym data to process word substitutions for high complex words.

Referring now to operation <NUM>, sentence clauses of a query can be adjusted based on computing complexities associated with particular sentence structures and determining whether is the user will be able to understand the sentence structure based on a language proficiency indicated by the language proficiency profile <NUM>.

In the example, the TTS engine <NUM> determines that the baseline text-to-speech response 306a has a high sentence complexity based on determining that the baseline text-to-speech response 306a includes three sentence clauses (e.g., "today's forecast is sunny," "but not consistent," and "and warm"). In response, the TTS engine <NUM> can generate adjusted sentence portions 316a and 316b, which combine a dependent clause and an independent clause into a single clause that does not include a segmenting punctuation mark. As a result, the adjusted text-to-speech response 306b includes both simpler vocabulary (e.g., "WEATHER," "CHANGE") as well as a simpler sentence structure (e.g., no clause separations), increasing the likelihood that the user will understand the adjusted text-to-speech output 306b. The adjusted text-to-speech output 306b is then generated for output by the TTS engine <NUM> as the output 306c.

In some implementations, the TTS engine <NUM> can perform sentence structure adjustment based on using a user-specific restructuring algorithm that include adjusts the baseline query 302a using weighting factors to avoid particular sentence structures that are identified to be problematic for the user. For example, the user-specific restructuring algorithm can specify an option to down-weights the inclusion of subordinate clauses or up-weights sentence clauses that have simple subject verb object sequences.

<FIG> is a block diagram that illustrates an example of a system <NUM> that adaptively generates text-to-speech outputs based on using clustering techniques. The system <NUM> includes a language proficiency estimator <NUM>, a user similarity determiner <NUM>, a complexity optimizer, and a machine learning system <NUM>.

Briefly, the language proficiency estimator <NUM> receives data from a plurality of users <NUM>. The language proficiency estimator <NUM> then estimates a set of language complexity profiles <NUM> for each of the plurality of users <NUM>, which is then sent to the user similarity determiner <NUM>. The user similarity determiner <NUM> identifies user clusters <NUM> of similar users. The complexity optimizer <NUM> and the machine learning system <NUM> then analyzes the language complexity profiles <NUM> of each user within the user clusters <NUM> and the context data received from the plurality of users <NUM> in order to generate a complexity mapping <NUM>.

In general, the system <NUM> can be used to analyze relationships between active language complexity and passive language complexity for a population of users. Active language complexity refers to detected language input provided by the user (e.g., text queries, voice input, etc.). Passive language complexity refers to a user's ability to understand or comprehend speech signals that are provided to the user. In this regard, the system <NUM> can use the determined relationship between the active language complexity and the passive language complexity for multiple users to determine an appropriate passive language complexity for each individual user where the particular user has the highest likelihood of understanding a text-to-speech output.

The plurality of users <NUM> can be multiple users that use an application associated with a TTS engine (e.g., the TTS engine <NUM>). For instance, the plurality of users <NUM> can be a set of users that use a mobile application that utilizes a TTS engine to provide users with text-to-speech features over a user interface of the mobile application. In such an instance, data from the plurality of users <NUM> (e.g., prior user queries, user selections, etc.) can be tracked by the mobile application and aggregated for analysis by the language proficiency estimator <NUM>.

The language proficiency estimator <NUM> can initially measure passive language complexities for the plurality of users <NUM> using substantially similar techniques as those described previously with respect to <FIG>. The language proficiency estimator <NUM> can then generate the language complexity profiles <NUM>, which includes an individual language complexity profile for each of the plurality of users <NUM>. Each individual language complexity profile includes data indicating the passive language complexity and the active language complexity for each of the plurality of users <NUM>.

The user similarity determiner <NUM> uses the language complexity data included within the set of language proficiency profiles <NUM> to identify similar users within the plurality of users <NUM>. In some instances, the user similarity determiner <NUM> can group users that have similar active language complexities (e.g., similar language inputs, speech queries provided, etc.). In other instances, the user similarity determiner <NUM> can determine similar users by comparing words included in prior user-submitted queries, particular user behaviors on a mobile application, or user locations. The user similarity determiner <NUM> then clusters the similar users to generate the user clusters <NUM>.

In some implementations, the user similarity determiner <NUM> generates the user clusters <NUM> based stored on cluster data <NUM> that include aggregate data for users in specified clusters. For example, the cluster data <NUM> can be grouped by specific parameters (e.g. number of incorrect query responses, etc.) that indicate a passive language complexity associated with the plurality of users <NUM>.

After generating the user clusters <NUM>, the complexity optimizer <NUM> varies the complexity of the language output by a TTS system and measures a user's passive language complexity using a set of parameters that indicate a user's ability to understand language output by the TTS system (e.g., understanding rate, voice action flow completion rate, or answer success rate) to indicate user performance. For instance, the parameters can be used to characterize how well users within each cluster <NUM> understand a given text-to-speech output. In such instances, the complexity optimizer <NUM> can initially provide a low complexity speech signal to the user and recursively provide additional speech signals within a range of complexities.

In some implementations, the complexity optimizer <NUM> can also determine the optimal passive language complexity for various user contexts associated with each user cluster <NUM>. For instance, after measuring the user's language proficiency using the set of parameters, the complexity optimizer <NUM> can then classify the measured data by context data received from the plurality of users <NUM> such that an optimal passive language complexity can be determined for each user context.

After gathering performance data for the range of passive language complexities, the machine learning system <NUM> then determines a particular passive language complexity where the performance parameters indicate that the user's language comprehension is the strongest. For instance, the machine learning system <NUM> aggregates the performance data all users within a particular user cluster <NUM> to determine relationships between the active language complexity, the passive language complexity, and the user context.

The aggregate data for the user cluster <NUM> can then compared to individual data for each user within the user cluster <NUM> to determine an actual language complexity score for each user within the user cluster <NUM>. For instance, as depicted in <FIG>, the complexity mapping <NUM> can represent the relationship between active language complexity and passive language complexity to infer the actual language complexity, which corresponds to the active language complexity mapped to the optimal passive language complexity.

The complexity mapping <NUM> represents relationships between active language complexity, TTS complexity, and passive language complexity for all user clusters within the plurality of user <NUM>, which can then be used to predict the appropriate TTS complexity for a subsequent query by an individual user. For example, as described above, user inputs (e.g., queries, text messages, e-mails, etc.) can be used to group similar users into user clusters <NUM>. For each cluster, the system provides TTS outputs requiring varying levels of language proficiency to understand. The system then assesses the responses received from users, and the rate of task completion for the varied TTS outputs, to determine a level of language complexity that is appropriate for the users in each cluster. The system stores a mapping <NUM> between cluster identifiers and TTS complexity scores corresponding to the identified clusters. The system then uses the complexity mapping <NUM> to determine an appropriate level of complexity for a TTS output for a user. For example, the system identifies a cluster that represents a user's active language proficiency, looks up a corresponding TTS complexity score (e.g., indicating a level of passive language understanding) for the cluster in the mapping <NUM>, and generates a TTS output having a complexity level indicated by the retrieved TTS complexity score.

The actual language complexity determined for a user can then be used to adjust the TTS system using techniques described with respect to <FIG>. In this regard, aggregate language complexity data from a group of similar users (e.g., the user cluster <NUM>) can be used to intelligently adjust the performance of a TTS system with respect to a single user.

<FIG> is a flow diagram that illustrates an example of a process <NUM> for adaptively generating text-to-speech output. Briefly, the process <NUM> can include determining a language proficiency of a user of a client device (<NUM>), determining a text segment for output by a text-to-speech module (<NUM>), generating audio data including a synthesized utterance of the text segment (<NUM>), and providing the audio data to the client device (<NUM>).

In more detail, the process <NUM> can include determining a language proficiency of a user of a client device (<NUM>). For instance, as described with respect to <FIG>, the language proficiency estimator <NUM> can determine a language proficiency for a user using a variety of techniques. In some instances, the language proficiency can represent an assigned score that indicates a level of language proficiency. In other instances, the language proficiency can represent an assigned category from a plurality of categories of language proficiency. In other instances, the language proficiency can be determined based on user input and/or behaviors indicating a proficiency level of the user.

In some implementations, the language proficiency can be inferred from different user signals. For instance, as described with respect to <FIG>, language proficiency can be inferred from vocabulary complexity of user inputs, data entry rate of the user, a number of misrecognized words from a speech input, a number of completed voice actions for different levels of TTS complexity, or a level of complexity of texts viewed by the user (e.g., books, articles, text on webpages, etc.).

The process <NUM> can include determining a text segment for output by a text-to-speech module (<NUM>). For instance, a TTS engine can adjust a baseline text segment based on the determine language proficiency of the user. In some instances, as described with respect to <FIG>, the text segment for output can be adjusted based on a user context associated the with user. In other instances, as described with respect to <FIG>, the text segment for output can also be adjusted by word substitution or sentence restructuring in order to reduce the complexity of the text segment. For example, the adjustment can be based on how rare individual words included in the text segments, the type of verbs used (e.g., compound verbs, or verb tense), the linguistic structure of the text segment (e.g., number of subordinate clauses, amount of separation between related words, degree the that phrases are nested, etc. In other examples, the adjustment can also be based on linguistic measures above with reference measurements for linguistic characteristics (e.g., average separation between subjects and verbs, separation between adjectives and nouns, etc.). In such examples, reference measurements can represent averages, or could include ranges or examples for different complexity levels.

In some implementations, determining the text segment for output can include selecting text segments that have scores that best match reference scores that describe a language proficiency level of the user. In other implementations, individual words or phrases can be scored for complexity, and then the most complex words can be substituted, deleted, or restructured such that overall complexity meets an appropriate level for the user.

The process <NUM> can include generating audio data including a synthesized utterance of the text segment (<NUM>).

The process <NUM> can include providing the audio data to the client device (<NUM>).

<FIG> is a block diagram of computing devices <NUM>, <NUM> that can be used to implement the systems and methods described in this document, as either a client or as a server or plurality of servers. Computing device <NUM> is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. Computing device <NUM> is intended to represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smartphones, and other similar computing devices. Additionally, computing device <NUM> or <NUM> can include Universal Serial Bus (USB) flash drives. The USB flash drives can store operating systems and other applications. The USB flash drives can include input/output components, such as a wireless transmitter or USB connector that can be inserted into a USB port of another computing device.

Each of the components <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, and <NUM>, are interconnected using various busses, and can be mounted on a common motherboard or in other manners as appropriate. In other implementations, multiple processors and/or multiple buses can be used, as appropriate, along with multiple memories and types of memory. Also, multiple computing devices <NUM> can be connected, with each device providing portions of the necessary operations, e.g., as a server bank, a group of blade servers, or a multi-processor system.

The memory <NUM> can also be another form of computer-readable medium, such as a magnetic or optical disk.

In one implementation, the storage device <NUM> can be or contain a computer-readable medium, such as a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid state memory device, or an array of devices, including devices in a storage area network or other configurations. The computer program product can also contain instructions that, when executed, perform one or more methods, such as those described above.

The high speed controller <NUM> manages bandwidth-intensive operations for the computing device <NUM>, while the low speed controller <NUM> manages lower bandwidth intensive operations. In one implementation, the high-speed controller <NUM> is coupled to memory <NUM>, display <NUM>, e.g., through a graphics processor or accelerator, and to high-speed expansion ports <NUM>, which can accept various expansion cards (not shown). The low-speed expansion port, which can include various communication ports, e.g., USB, Bluetooth, Ethernet, wireless Ethernet can be coupled to one or more input/output devices, such as a keyboard, a pointing device, microphone/speaker pair, a scanner, or a networking device such as a switch or router, e.g., through a network adapter. The computing device <NUM> can be implemented in a number of different forms, as shown in the figure. For example, it can be implemented as a standard server <NUM>, or multiple times in a group of such servers. It can also be implemented as part of a rack server system <NUM>. In addition, it can be implemented in a personal computer such as a laptop computer <NUM>. Alternatively, components from computing device <NUM> can be combined with other components in a mobile device (not shown), such as device <NUM>. Each of such devices can contain one or more of computing device <NUM>, <NUM>, and an entire system can be made up of multiple computing devices <NUM>, <NUM> communicating with each other.

The computing device <NUM> can be implemented in a number of different forms, as shown in the figure. For example, it can be implemented as a standard server <NUM>, or multiple times in a group of such servers. It can also be implemented as part of a rack server system <NUM>. In addition, it can be implemented in a personal computer such as a laptop computer <NUM>. Alternatively, components from computing device <NUM> can be combined with other components in a mobile device (not shown), such as device <NUM>. Each of such devices can contain one or more of computing device <NUM>, <NUM>, and an entire system can be made up of multiple computing devices <NUM>, <NUM> communicating with each other.

Computing device <NUM> includes a processor <NUM>, memory <NUM>, and an input/output device such as a display <NUM>, a communication interface <NUM>, and a transceiver <NUM>, among other components. The device <NUM> can also be provided with a storage device, such as a microdrive or other device, to provide additional storage. Each of the components <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, and <NUM>, are interconnected using various buses, and several of the components can be mounted on a common motherboard or in other manners as appropriate.

The processor can be implemented as a chipset of chips that include separate and multiple analog and digital processors. Additionally, the processor can be implemented using any of a number of architectures. For example, the processor <NUM> can be a CISC (Complex Instruction Set Computers) processor, a RISC (Reduced Instruction Set Computer) processor, or a MISC (Minimal Instruction Set Computer) processor. The processor can provide, for example, for coordination of the other components of the device <NUM>, such as control of user interfaces, applications run by device <NUM>, and wireless communication by device <NUM>.

Processor <NUM> can communicate with a user through control interface <NUM> and display interface <NUM> coupled to a display <NUM>. The display <NUM> can be, for example, a TFT (Thin-Film-Transistor Liquid Crystal Display) display or an OLED (Organic Light Emitting Diode) display, or other appropriate display technology. The display interface <NUM> can include appropriate circuitry for driving the display <NUM> to present graphical and other information to a user. The control interface <NUM> can receive commands from a user and convert them for submission to the processor <NUM>. In addition, an external interface <NUM> can be provide in communication with processor <NUM>, so as to enable near area communication of device <NUM> with other devices. External interface <NUM> can provide, for example, for wired communication in some implementations, or for wireless communication in other implementations, and multiple interfaces can also be used.

Expansion memory <NUM> can also be provided and connected to device <NUM> through expansion interface <NUM>, which can include, for example, a SIMM (Single In Line Memory Module) card interface. Such expansion memory <NUM> can provide extra storage space for device <NUM>, or can also store applications or other information for device <NUM>. Specifically, expansion memory <NUM> can include instructions to carry out or supplement the processes described above, and can include secure information also. Thus, for example, expansion memory <NUM> can be provide as a security module for device <NUM>, and can be programmed with instructions that permit secure use of device <NUM>. In addition, secure applications can be provided via the SIMM cards, along with additional information, such as placing identifying information on the SIMM card in a non-hackable manner.

The memory can include, for example, flash memory and/or NVRAM memory, as discussed below. The information carrier is a computer- or machine-readable medium, such as the memory <NUM>, expansion memory <NUM>, or memory on processor <NUM> that can be received, for example, over transceiver <NUM> or external interface <NUM>.

Device <NUM> can communicate wirelessly through communication interface <NUM>, which can include digital signal processing circuitry where necessary. Communication interface <NUM> can provide for communications under various modes or protocols, such as GSM voice calls, SMS, EMS, or MMS messaging, CDMA, TDMA, PDC, WCDMA, CDMA2000, or GPRS, among others. Such communication can occur, for example, through radio-frequency transceiver <NUM>. In addition, short-range communication can occur, such as using a Bluetooth, Wi-Fi, or other such transceiver (not shown). In addition, GPS (Global Positioning System) receiver module <NUM> can provide additional navigation- and location-related wireless data to device <NUM>, which can be used as appropriate by applications running on device <NUM>.

Device <NUM> can also communicate audibly using audio codec <NUM>, which can receive spoken information from a user and convert it to usable digital information. Audio codec <NUM> can likewise generate audible sound for a user, such as through a speaker, e.g., in a handset of device <NUM>. Such sound can include sound from voice telephone calls, can include recorded sound, e.g., voice messages, music files, etc. and can also include sound generated by applications operating on device <NUM>.

The computing device <NUM> can be implemented in a number of different forms, as shown in the figure. For example, it can be implemented as a cellular telephone <NUM>. It can also be implemented as part of a smartphone <NUM>, personal digital assistant, or other similar mobile device.

Various implementations of the systems and methods described here can be realized in digital electronic circuitry, integrated circuitry, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and/or combinations of such implementations. These various implementations can include implementation in one or more computer programs that are executable and/or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

As used herein, the terms "machine-readable medium" "computer-readable medium" refers to any computer program product, apparatus and/or device, e.g., magnetic discs, optical disks, memory, Programmable Logic Devices (PLDs), used to provide machine instructions and/or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal.

Claim 1:
A computer-implemented method when executed on data processing hardware causes the data processing hardware to perform operations comprising:
during a registration process for a client device (106a, 106b, <NUM>):
receiving demographic information for a user (102a, 102b, <NUM>) of the client device (106a, 106b, <NUM>); and
designating a language proficiency to the user (102a, 102b, <NUM>) based on the received demographic information, the language proficiency designated to the user (102a, 102b, <NUM>) comprising one of a first level of language proficiency or a second level of language proficiency different than the first level of language proficiency;
receiving a voice query (<NUM>, <NUM>) input to the client device (106a, 106b, <NUM>) by the user (102a, 102b, <NUM>);
generating audio data comprising a synthesized utterance of a particular text segment responsive to the voice query (<NUM>, <NUM>) and based on the language proficiency designated to the user (102a, 102b, <NUM>), the particular text segment comprising one of:
a first text segment when the language proficiency designated to the user (102a, 102b, <NUM>) comprises the first level of language proficiency, the first text segment comprising primary information responsive to the voice query (<NUM>, <NUM>); or
a second text segment when the language proficiency designated to the user (102a, 102b, <NUM>) comprises the second level of language proficiency, the second text segment comprising additional information responsive to the voice query (<NUM>, <NUM>) that is not included in the first text segment; and
providing the audio data for audible output from the client device (106a, 106b, <NUM>).