Patent ID: 12198671

In the drawings, like reference numbers represent corresponding parts throughout.

DETAILED DESCRIPTION

FIG.1is a diagram that illustrates examples of processes100A and100B for generating text-to-speech outputs based on language proficiency. The processes100A and100B are used to generate different text-to-speech outputs for a user102awith high language proficiency and a user102bwith low language proficiency, respectively, for a text query104. As depicted, after receiving a query104on the user devices106aand106b, the process100A generates a high-complexity text-to-speech output108afor the user102awhereas the process100B generates a low-complexity output108bfor the user102b. In addition, the TTS systems that execute processes100A and100B can include a language proficiency estimator110, a text-to-speech engine120. In addition, the text-to-speech engine120can further include a text analyzer122, a linguistics analyzer124, and a waveform generator126.

In general, the content of text that is used to generate a text-to-speech output can be determined according to a language proficiency of a user. In addition, or as an alternative, the text to be used to generate a text-to-speech output can be determined based on a context of the user, for example, the location or activity of a user, background noise present, a current task of the user and so on. Further, the text to be converted to an audible form may be adjusted or determined using other information, such as indications that a user has failed to complete a task or is repeating an action.

In the example, two users, user102aand user102b, provide the same query104on user devices106aand106b, respectively, as input to an application, web page, or other search functionality. For instance, the query104can be a voice query sent to the user devices106aand106bto determine a weather forecast for the current day. The query104is then transmitted to the text-to-speech engine120to generate a text-to-speech output in response to the query104.

The language proficiency estimator110can be a software module within a TTS system that determines a language proficiency score associated with a particular user (e.g., the user102aor the user102b) based on user data108a. The language proficiency score can be an estimate of the user's ability to understand communications in a particular language, in particular, to understand speech in the particular language. One measure of language proficiency is the ability of a user to successfully complete a voice-controlled task. Many types of tasks, such as setting a calendar appointment, looking up directions, and so on, follow a sequence of interactions in which a user and device exchange verbal communication. The rate at which a user successfully completes these task workflows through a voice interface is a strong indicator of the user's language proficiency. For example, a user that completes nine out of ten voice tasks that the user initiates likely has a high language proficiency. On the other hand, a user that fails to complete the majority of voice tasks that the user initiates can be inferred to have a low language proficiency, since the user may not have fully understood the communications from the device or may not have been able to provide appropriate verbal responses. As discussed further below, when a user does not complete workflows that include standard TTS outputs, resulting in a low language proficiency score, the TTS may use adapted, simplified outputs that may increase the ability of the user to understand and complete various tasks.

As shown, the user data108acan include words used within prior text queries submitted by the user, an indication whether English, or any other language utilized by the TTS system, is the native language of the user, and a set of activities and/or behaviors that are reflective of a user's language comprehension skills. For example, as depicted inFIG.1, a typing speed of the user can be used to determine language fluency of the user in a language. In addition, a language vocabulary complexity score or language proficiency score can be assigned to the user based on associating a pre-determined complexity to words that were used by the user in previous text queries. In another example, the number of misrecognized words in prior queries can also be used to determine the language proficiency score. For instance, a high number of misrecognized words can be used to indicate a low language proficiency. In some implementations, the language proficiency score is determined by looking up a stored score associated with the user, which was determined for the user prior to submission of the query104.

AlthoughFIG.1depicts the language proficiency estimator110as a separate component to the TTS engine120, in some implementations, as depicted inFIG.2, the language proficiency estimator110can be an integrated software module within the TTS engine120. In such instances, operations involving the language proficiency estimation can be directly modulated by the TTS engine120.

In some implementations, the language proficiency score assigned to the user may be based on a particular user context estimated for the user. For instance, as described more particularly with respect toFIG.2, a user context determination can be used to determine context-specific language proficiencies that can cause a user to temporarily have limited language comprehension abilities. For example, if the user context indicates significant background noise or if the user is engaged in a task such as driving, the language proficiency score can be used to indicate that the user's present language comprehension ability is temporarily diminished relative to other user contexts.

In some implementations, instead of inferring language proficiency based on previous user activity, the language proficiency score can instead be directly provided to the TTS engine120without the use of the language proficiency estimator110. For instance, a language proficiency score can be designated to a user based on user input during a registration process that specifies a user's level of language proficiency. For example, during the registration, the user can provide a selection that specifies the user's skill level, which can then be used to calculate the appropriate language proficiency for the user. In other examples, the user can provide other types of information such as demographic information, education level, places of residences, etc., that can be used to specify the user's level of language proficiency.

In the examples described above, the language proficiency score can either be a set of discrete values that are adjusted periodically based on recently generated user activity data, or a continuous score that is initially designated during a registration process. In the first instance, the value of the language proficiency score can be biased based on one or more factors that indicate that a user's present language comprehension and proficiency may be attenuated (e.g., a user context indicating significant background noise). In the second instance, the value of the language proficiency score can be preset after an initial calculation and adjusted only after specific milestone events that indicate that a user's language proficiency has increased (e.g., an increase in typing rate or a decrease in correction rate for a given language). In other instances, a combination of these two techniques can be used to variably adjust the text-to-speech output based on a particular text input. In such instances, multiple language proficiency scores that each represent a particular aspect of the user's language skills can be used to determine to how best adjust the text-to-speech output for the user. For example, one language proficiency score can represent a complexity of the user's vocabulary whereas another language proficiency score can be used to represent the user's grammar skills.

The TTS engine120can use the language proficiency score to generate a text-to-speech output that is adapted to the language proficiency indicated by the user's language proficiency score. In some instances, the TTS engine120adapts the text-to-speech output based on selecting a particular TTS string from a set of candidate TTS strings for the text query104. In such instances, the TTS engine120selects the particular TTS string based on using the language proficiency score of the user to predict a likelihood that each of the candidate TTS strings will accurately be interpreted by the user. More particular descriptions related to these techniques are provided with respect toFIG.2. Alternatively, in other instances, the TTS engine120can select a baseline TTS string and adjust the structure of the TTS string based on the user's level of language proficiency indicated by the language proficiency score. In such instances, the TTS engine120can adjust the grammar of the baseline TTS string, provide word substitutions and/or reduce the sentence complexity to generate an adapted TTS string that is more likely to be understood by the user. More particular descriptions related to these techniques are provided with respect toFIG.3.

Referring still toFIG.1, the TTS engine120may generate different text-to-speech outputs for users102aand102bbecause the language proficiency scores for the users are different. For example, in process100A, the language proficiency score106aindicates high English-language proficiency, inferred from the user data108aindicating that the user102ahas a complex vocabulary, has English as a first language, and has a relatively high word per minute in prior user queries. Based on the value of the language proficiency score106a, the TTS engine120generates a high complexity text-to-speech output108athat includes a complex grammatical structure. As depicted, the text-to-speech output108aincludes an independent clause that describes that today's forecast is sunny, in addition to a subordinate clause that includes additional information about the high temperature and the low temperature of the day.

In the example of process100B, the language proficiency score106bindicates low English-language proficiency, inferred from user activity data108bindicating that the user102bhas a simple vocabulary, has English as a second language, and has previously provided ten incorrect queries. In this example, the TTS engine120generates a low complexity text-to-speech output108bthat includes a simpler grammatical structure relative to the text-to-speech output108a. For instance, instead of including multiple clauses within a single sentence, the text-to-speech output108bincludes a single independent clause that conveys the same primary information as the text-to-speech output108a(e.g., today's forecast being sunny), but does not include additional information related to the high and low temperatures for the day.

The adaptation of text for a TTS output can be performed by various different devices and software modules. For example, a TTS engine of a server system may include functionality to adjust text based on a language proficiency score and then output audio including a synthesized utterance of the adjusted text. As another example, a pre-processing module of a server system may adjust text and pass the adjusted text to a TTS engine for speech synthesis. As another example, a user device may include a TTS engine, or a TTS engine and a text pre-processor, to be able to generate appropriate TTS outputs.

In some implementations, a TTS system can include software modules that are configured to exchange communications with a third-party mobile application of a client device or a web page. For instance, the TTS functionality of the system can be made available to a third-party mobile application through an application package interface (API). The API can include defined set of protocols that an application or web site can use to request TTS audio from a server system that runs the TTS engine120. In some implementations, the API can make available TTS functionality that runs locally on a user's device. For example, the API may be available to an application or web page through an inter-process communication (IPC), remote procedure call (RPC), or other system call or function. A TTS engine, and associated language proficiency analysis or text preprocessing, may be run locally on the user's device to determine an appropriate text for the user's language proficiency and generate the audio for the synthesized speech also.

For example, the third-party application or web page can use the API to generate a set of voice instructions that are provided to the user based on a task flow of a voice interface of the third-party application or web page. The API can specify that the application or web page should provide text to be converted to speech. In some instances, other information can be provided, such as a user identifier or a language proficiency score.

In implementations where the TTS engine120exchanges communications with a third-party application using an API, the TTS engine120can be used to determine whether a text segment from a third-party application should be adjusted prior to generating a text-to-speech output for the text. For example, the API can include computer-implemented protocols that specify conditions within the third-party application that initiate the generation of an adaptive text-to-speech output.

As an example, one API may permit an application to submit multiple different text segments as candidates for a TTS output, where the different text segments correspond to different levels of language proficiency. For example, the candidates can be text segments having equivalent meanings but different complexity levels (e.g., a high complexity response, a medium complexity response, and a low complexity response). The TTS engine120may then determine the level of language proficiency needed to understand each candidate, determine an appropriate language proficiency score for the user, and select the candidate text that best corresponds to the language proficiency score. The TTS engine120then provides synthesized audio for the selected text back to the application, e.g., over a network using the API. In some instances, the API can be locally available on the user devices106aand106b. In such instances, the API can be accessible over various types of inter-process communication (IPC) or via a system call. For example, the output of the API on the user devices106aand106bcan be the text-to-speech output of the TTS engine120since the API operates locally on the user devices106aand106b.

In another example, an API can allow the third-party application to provide a single text segment and a value that indicates whether the TTS engine120is permitted to modify the text segment to generate a text segment with a different complexity. If the app or web page indicates that alteration is permitted, the TTS system120may make various changes to the text, for example, to reduce the complexity of the text when the language proficiency score suggests that the original text is more complex than the user can understand in a spoken response. In yet other examples, an API allows the third-party application to also provide user data (e.g., prior user queries submitted on the third-party application) along with the text segment such that the TTS engine120can determine a user context associated with the user and adjust generate a particular text-to-speech output based on the determined user context. Similarly, an API can allow an application to provide context data from a user device (e.g., a global positioning signal, accelerometer data, ambient noise level, etc.) or an indication of a user context to allow the TTS engine120to adjust the text-to-speech outputs that will ultimately be provided to the user through the third-party application. In some instances, the third party application can also provide the API with data that can be used to determine a language proficiency of the user.

In some implementations, the TTS engine120can adjust the text-to-speech output for a user query without using a language proficiency of the user or determining a context associated with the user. In such implementations, TTS engine120can determine that an initial text-to-speech output is too complex for a user based on receiving signals that a user has misunderstood the output (e.g., multiple retries on the same query or task). In response, the TTS engine120can reduce the complexity of a subsequent text-to-speech response for a retried query or related queries. Thus, when a user fails to successfully complete an action, the TTS engine120may progressively reduce the amount of detail or language proficiency required to understand the TTS output until it reaches a level that the user understands.

FIG.2is a diagram that illustrates an example of a system200that adaptively generates a text-to-speech output based on a user context. Briefly, the system200can include a TTS engine210that includes a query analyzer211, a language proficiency estimator212, an interpolator213, a linguistics analyzer214, a re-ranker215, and a waveform generator216. The system200also includes a context repository220that stores a set of context profiles232, and a user history manager230that stores user history data234. In some instances, the TTS engine210corresponds to the TTS engine120as described with respect toFIG.1.

In the example, a user202initially submits a query204on a user device208that includes a request for information related to the user's first meeting for the day. The user device208can then transmit the query204and context data206associated with the user202to the query analyzer211and the language proficiency estimator212, respectively. Other types of TTS outputs that are not responses to queries, e.g., calendar reminders, notifications, task workflows, etc., may be adapted using the same techniques.

The context data206can include information relating to a particular context associated with the user202such as time intervals between repeated text queries, global positioning signal (GPS) data indicating a location, speed, or movement pattern associated with the user202, prior text queries submitted to the TTS engine210within a particular time period, or other types of background information that can indicate user activity related to the TTS engine210. In some instances, the context data206can indicate a type of query204submitted to the TTS engine210, such as whether the query204is a text segment associated with a user action, or an instruction transmitted to the TTS engine210to generate a text-to-speech output.

After receiving the query204, the query analyzer211parses the query204to identify information that is responsive to the query204. For example, in some instances where the query204is a voice query, the query analyzer211initially generates a transcription of the voice query, and then processes individual words or segments within the query204to determine information that is responsive to the query204, for example, by providing the query to a search engine and receive search results. The transcription of the query and the identified information can then be transmitted to the linguistics analyzer214.204

Referring to now to the language proficiency estimator212, after receiving the context data206, the language proficiency estimator212computes a language proficiency for the user202based on the received context data206using techniques described with respect toFIG.1. In particular, the language proficiency estimator212parses through various context profiles232stored on the repository220. The context profile232can be an archived library including related types of information that are associated with a particular user context and can be included within a text-to-speech output. The context profile232additionally specifies a value, associated with each type of information, which represents an extent to which each type of information is likely to be understood by the user202when the user202is presently within a context associated with the context profile232.

In the example depicted inFIG.2, the context profile232specifies that the user202is presently in a context indicating that the user202is on his/her daily commute to work. In addition, the context profile232also specifies values for individual words and phrases that are likely to be comprehended by the user202. For instance, data or time information is associated with a value of “0.9” for “SINCE,” indicating that the user202is more likely to understand generalized information associated with a meeting (e.g., time of the next upcoming meeting)204rather than detailed information associated with a meeting (e.g., a party attending the meeting, or location of the meeting. In this example, the differences of the values indicate differences in the user's ability to understand particular types of information because the user's ability to understand complex or detailed information is diminished.

The value associated with individual words and phrases can determined based on user activity data from previous user sessions where the user202was previously in the context indicated by the context data206. For instance, historical user data can be transmitted from the user history manager230, which retrieves data stored within the query logs234. In the example, the value for date and time information can be increased based on determining that the user commonly accesses date and time information associated with meetings more frequently than locations of the meetings.

After the language proficiency estimator212selects a particular context profile232that corresponds to the received context data206, the language proficiency estimator212transmits the selected context profile232to the interpolator213. The interpolator213parses the selected context profile232, and extracts individual words and phrases included and their associated values. In some instances, the interpolator213transmits the different types of information and associated values directly to the linguistics analyzer214for generating a list of text-to-speech output candidates240a. In such instances, the interpolator213extracts specific types of information and associated values from the selected context profile232and transmits them to the linguistics analyzer214. In other instances, the interpolator213can also transmit the selected context profile232to the re-ranker215.

In some instances, the TTS engine210can be provided a set of structured data (e.g., fields of a calendar event). In such instances, the interpolator213can convert the structured data to text at a level that matches the user's proficiency indicated by the context profile232. For example, the TTS engine210may access data indicating one or more grammars indicating different levels of detail or complexity to express the information in the structured data, and select an appropriate grammar based on the user's language proficiency score. Similarly, the TTS engine210can use dictionaries to select words that are appropriate given the language proficiency score.

The linguistics analyzer214performs processing operations such as normalization on the information included within the query204. For instance, the query analyzer211can assign phonetic transcriptions to each word or snippet included within the query204, and divide the query204into prosodic units such as phrases, clauses, and sentences using a text-to-phenome conversion. The linguistics analyzer214also generates a list240athat includes multiple text-to-speech output candidates that are identified as being responsive to the query204. In the example, the list240aincludes multiple text-to-speech output candidates with different levels of complexity. For example, the response “At 12:00 PM with Mr. John near Dupont Circle” is the most complex response because it identifies a time for the meeting, a location for the meeting, an individual with whom the meeting will take place. In comparison, the response “In three hours” is the least complex because it only identifies a time for the meeting.

The list240aalso includes a baseline rank for the text-to-speech candidates based on the likelihood that each text-to-speech output candidate is likely to be responsive to the query204. In the example, the list240aindicates that most complex text-to-speech output candidate is the most likely to be responsive to the query204because it includes the greatest amount of information that is associated with the content of the query204.

After the linguistics analyzer generates the list240aof text-to-speech output candidates, the re-ranker215generates a list240b, which includes an adjusted rank for the text-to-speech output candidates based on the received context data206. For instance, the re-ranker215can adjust the rank based on the scores associated with particular types of information included within the selected context profile232.

In the example, the re-ranker215ranks the simplest text-to-speech output as the highest based on the context profile232indicating that the user202is likely to comprehend date and time information within a text-to-speech response but not likely to understand party names or location information within the text-to-speech response given the present context of the user indicating that the user is commuting to work. In this regard, the received context data206can be used to adjust the selection of a particular text-to-speech output candidate that to increase the likelihood that the user202will understand the contents of the text-to-speech output204cof the TTS engine210.

FIG.3is a diagram that illustrates an example of a system300for modifying sentence structure within a text-to-speech output. Briefly, a TTS engine310receives a query302and a language proficiency profile304for a user (e.g., the user202). The TTS engine310then perform operations312,314, and316to generate an adjusted text-to-speech output302cthat is responsive to the query302. In some instances, the TTS engine310corresponds to the TTS engine120described with respect toFIG.1, or the TTS engine210described with respect toFIG.2.

In general, the TTS engine310can modify the sentence structure of a baseline text-to-speech output306afor the query302using different types of adjustment techniques. As an example, the TTS engine310can substitute words or phrases within the baseline text-to-speech output306abased on determining that a complexity score associated with individual words or phrases is greater than a threshold score indicated by the language complexity profile304of a user. As another example, the TTS engine310can rearrange individual sentence clauses such that the overall complexity of baseline text-to-speech output306ais reduced to a satisfactory level based on the language complexity profile304. The TTS engine310can also re-order words, split or combine sentences, and make other changes to adjust the complexity of text.

In more detail, during the operation312, the TTS engine310initially generates a baseline text-to-speech output306athat is responsive to the query302. The TTS engine310then parses the baseline text-to-speech output306ainto segments312a-312c. The TTS engine310also detects punctuation marks (e.g., commas, periods, semicolons, etc.) that indicate breakpoints between individual segments. The TTS engine310also computes a complexity score for each of the segments312a-312c. In some instances, the complexity score can be computed based on the frequency of a particular word within a particular language. Alternative techniques can include computing the complexity score based on the frequency of use by the user, or frequency of occurrence in historical content accessed by the user (e.g., news articles, webpages, etc.). In each of these examples, the complexity score can be used to indicate words that are likely to be comprehended by the user and other words that are unlikely to be comprehended by the user.

In the example, segments312aand312bare determined to be relatively complex based on the inclusion of high complex terms such as “FORECAST” and “CONSISTENT,” respectively. However, the segment312cis determined to be relatively simple because the terms included are relatively simple. This determination is represented by the segments312aand312bhaving higher complexity scores (e.g., 0.83, 0.75) compared to the complexity score for the segment312c(e.g., 0.41).

As described above, the language proficiency profile304can be used to compute a threshold complexity score that indicates the maximal complexity that is comprehendible by the user. In the example, the threshold complexity score can be computed to be “0.7” such that the TTS310determines that the segments312aand312bare unlikely to be comprehended by the user.

After identifying individual segments with associated complexity scores greater than the threshold complexity score indicated by the language proficiency profile304, during the operation314, the TTS engine310substitutes the identified words with alternates that are predicted to be more likely to be understood by the user. As depicted inFIG.3, “FORECAST” can be substituted with “WEATHER,” and “CONSISTENT” can be substituted with “CHANGE.” In these examples, segments314anand314brepresent simpler alternatives with lower complexity scores below the threshold complexity score indicated by the language proficiency profile304.

In some implementations, TTS engine310can process word substitutions for high complexity words using a trained skip-gram model that uses unsupervised techniques to determine appropriately complex words to replace highly complex words. In some instances, the TTS engine310can also use thesaurus or synonym data to process word substitutions for high complex words.

Referring now to operation316, sentence clauses of a query can be adjusted based on computing complexities associated with particular sentence structures and determining whether is the user will be able to understand the sentence structure based on a language proficiency indicated by the language proficiency profile304.

In the example, the TTS engine310determines that the baseline text-to-speech response306ahas a high sentence complexity based on determining that the baseline text-to-speech response306aincludes three sentence clauses (e.g., “today's forecast is sunny,” “but not consistent,” and “and warm”). In response, the TTS engine310can generate adjusted sentence portions316aand316b, which combine a dependent clause and an independent clause into a single clause that does not include a segmenting punctuation mark. As a result, the adjusted text-to-speech response306bincludes both simpler vocabulary (e.g., “WEATHER,” “CHANGE”) as well as a simpler sentence structure (e.g., no clause separations), increasing the likelihood that the user will understand the adjusted text-to-speech output306b. The adjusted text-to-speech output306bis then generated for output by the TTS engine310as the output306c.

In some implementations, the TTS engine310can perform sentence structure adjustment based on using a user-specific restructuring algorithm that include adjusts the baseline query302ausing weighting factors to avoid particular sentence structures that are identified to be problematic for the user. For example, the user-specific restructuring algorithm can specify an option to down-weights the inclusion of subordinate clauses or up-weights sentence clauses that have simple subject verb object sequences.

FIG.4is a block diagram that illustrates an example of a system400that adaptively generates text-to-speech outputs based on using clustering techniques. The system400includes a language proficiency estimator410, a user similarity determiner420, a complexity optimizer, and a machine learning system400.

Briefly, the language proficiency estimator410receives data from a plurality of users402. The language proficiency estimator410then estimates a set of language complexity profiles412for each of the plurality of users402, which is then sent to the user similarity determiner420. The user similarity determiner420identifies user clusters424of similar users. The complexity optimizer430and the machine learning system440then analyzes the language complexity profiles412of each user within the user clusters424and the context data received from the plurality of users402in order to generate a complexity mapping442.

In general, the system400can be used to analyze relationships between active language complexity and passive language complexity for a population of users. Active language complexity refers to detected language input provided by the user (e.g., text queries, voice input, etc.). Passive language complexity refers to a user's ability to understand or comprehend speech signals that are provided to the user. In this regard, the system400can use the determined relationship between the active language complexity and the passive language complexity for multiple users to determine an appropriate passive language complexity for each individual user where the particular user has the highest likelihood of understanding a text-to-speech output.

The plurality of users402can be multiple users that use an application associated with a TTS engine (e.g., the TTS engine120). For instance, the plurality of users402can be a set of users that use a mobile application that utilizes a TTS engine to provide users with text-to-speech features over a user interface of the mobile application. In such an instance, data from the plurality of users402(e.g., prior user queries, user selections, etc.) can be tracked by the mobile application and aggregated for analysis by the language proficiency estimator410.

The language proficiency estimator410can initially measure passive language complexities for the plurality of users402using substantially similar techniques as those described previously with respect toFIG.1. The language proficiency estimator410can then generate the language complexity profiles412, which includes an individual language complexity profile for each of the plurality of users402. Each individual language complexity profile includes data indicating the passive language complexity and the active language complexity for each of the plurality of users402.

The user similarity determiner420uses the language complexity data included within the set of language proficiency profiles412to identify similar users within the plurality of users402. In some instances, the user similarity determiner420can group users that have similar active language complexities (e.g., similar language inputs, speech queries provided, etc.). In other instances, the user similarity determiner420can determine similar users by comparing words included in prior user-submitted queries, particular user behaviors on a mobile application, or user locations. The user similarity determiner420then clusters the similar users to generate the user clusters424.

In some implementations, the user similarity determiner420generates the user clusters424based stored on cluster data422that include aggregate data for users in specified clusters. For example, the cluster data422can be grouped by specific parameters (e.g. number of incorrect query responses, etc.) that indicate a passive language complexity associated with the plurality of users402.

After generating the user clusters424, the complexity optimizer430varies the complexity of the language output by a TTS system and measures a user's passive language complexity using a set of parameters that indicate a user's ability to understand language output by the TTS system (e.g., understanding rate, voice action flow completion rate, or answer success rate) to indicate user performance. For instance, the parameters can be used to characterize how well users within each cluster424understand a given text-to-speech output. In such instances, the complexity optimizer430can initially provide a low complexity speech signal to the user and recursively provide additional speech signals within a range of complexities.

In some implementations, the complexity optimizer430can also determine the optimal passive language complexity for various user contexts associated with each user cluster424. For instance, after measuring the user's language proficiency using the set of parameters, the complexity optimizer430can then classify the measured data by context data received from the plurality of users402such that an optimal passive language complexity can be determined for each user context.

After gathering performance data for the range of passive language complexities, the machine learning system440then determines a particular passive language complexity where the performance parameters indicate that the user's language comprehension is the strongest. For instance, the machine learning system440aggregates the performance data all users within a particular user cluster424to determine relationships between the active language complexity, the passive language complexity, and the user context.

The aggregate data for the user cluster424can then compared to individual data for each user within the user cluster424to determine an actual language complexity score for each user within the user cluster424. For instance, as depicted inFIG.4, the complexity mapping442can represent the relationship between active language complexity and passive language complexity to infer the actual language complexity, which corresponds to the active language complexity mapped to the optimal passive language complexity.

The complexity mapping442represents relationships between active language complexity, TTS complexity, and passive language complexity for all user clusters within the plurality of user402, which can then be used to predict the appropriate TTS complexity for a subsequent query by an individual user. For example, as described above, user inputs (e.g., queries, text messages, e-mails, etc.) can be used to group similar users into user clusters424. For each cluster, the system provides TTS outputs requiring varying levels of language proficiency to understand. The system then assesses the responses received from users, and the rate of task completion for the varied TTS outputs, to determine a level of language complexity that is appropriate for the users in each cluster. The system stores a mapping442between cluster identifiers and TTS complexity scores corresponding to the identified clusters. The system then uses the complexity mapping442to determine an appropriate level of complexity for a TTS output for a user. For example, the system identifies a cluster that represents a user's active language proficiency, looks up a corresponding TTS complexity score (e.g., indicating a level of passive language understanding) for the cluster in the mapping442, and generates a TTS output having a complexity level indicated by the retrieved TTS complexity score.

The actual language complexity determined for a user can then be used to adjust the TTS system using techniques described with respect toFIGS.1-3. In this regard, aggregate language complexity data from a group of similar users (e.g., the user cluster424) can be used to intelligently adjust the performance of a TTS system with respect to a single user.

FIG.5is a flow diagram that illustrates an example of a process500for adaptively generating text-to-speech output. Briefly, the process500can include determining a language proficiency of a user of a client device (510), determining a text segment for output by a text-to-speech module (520), generating audio data including a synthesized utterance of the text segment (530), and providing the audio data to the client device (540).

In more detail, the process500can include determining a language proficiency of a user of a client device (510). For instance, as described with respect toFIG.1, the language proficiency estimator110can determine a language proficiency for a user using a variety of techniques. In some instances, the language proficiency can represent an assigned score that indicates a level of language proficiency. In other instances, the language proficiency can represent an assigned category from a plurality of categories of language proficiency. In other instances, the language proficiency can be determined based on user input and/or behaviors indicating a proficiency level of the user.

In some implementations, the language proficiency can be inferred from different user signals. For instance, as described with respect toFIG.1, language proficiency can be inferred from vocabulary complexity of user inputs, data entry rate of the user, a number of misrecognized words from a speech input, a number of completed voice actions for different levels of TTS complexity, or a level of complexity of texts viewed by the user (e.g., books, articles, text on webpages, etc.).

The process500can include determining a text segment for output by a text-to-speech module (520). For instance, a TTS engine can adjust a baseline text segment based on the determine language proficiency of the user. In some instances, as described with respect toFIG.2, the text segment for output can be adjusted based on a user context associated the with user. In other instances, as described with respect toFIG.3, the text segment for output can also be adjusted by word substitution or sentence restructuring in order to reduce the complexity of the text segment. For example, the adjustment can be based on how rare individual words included in the text segments, the type of verbs used (e.g., compound verbs, or verb tense), the linguistic structure of the text segment (e.g., number of subordinate clauses, amount of separation between related words, degree the that phrases are nested, etc. In other examples, the adjustment can also be based on linguistic measures above with reference measurements for linguistic characteristics (e.g., average separation between subjects and verbs, separation between adjectives and nouns, etc.). In such examples, reference measurements can represent averages, or could include ranges or examples for different complexity levels.

In some implementations, determining the text segment for output can include selecting text segments that have scores that best match reference scores that describe a language proficiency level of the user. In other implementations, individual words or phrases can be scored for complexity, and then the most complex words can be substituted, deleted, or restructured such that overall complexity meets an appropriate level for the user.

The process500can include generating audio data including a synthesized utterance of the text segment (530).

The process500can include providing the audio data to the client device (540).

FIG.6is a block diagram of computing devices600,650that can be used to implement the systems and methods described in this document, as either a client or as a server or plurality of servers. Computing device600is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. Computing device650is intended to represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smartphones, and other similar computing devices. Additionally, computing device600or650can include Universal Serial Bus (USB) flash drives. The USB flash drives can store operating systems and other applications. The USB flash drives can include input/output components, such as a wireless transmitter or USB connector that can be inserted into a USB port of another computing device. The components shown here, their connections and relationships, and their functions, are meant to be exemplary only, and are not meant to limit implementations of the inventions described and/or claimed in this document.

Computing device600includes a processor602, memory604, a storage device606, a high-speed interface608connecting to memory604and high-speed expansion ports610, and a low speed interface612connecting to low speed bus614and storage device606. Each of the components602,604,606,608,610, and612, are interconnected using various busses, and can be mounted on a common motherboard or in other manners as appropriate. The processor602can process instructions for execution within the computing device600, including instructions stored in the memory604or on the storage device606to display graphical information for a GUI on an external input/output device, such as display616coupled to high speed interface608. In other implementations, multiple processors and/or multiple buses can be used, as appropriate, along with multiple memories and types of memory. Also, multiple computing devices600can be connected, with each device providing portions of the necessary operations, e.g., as a server bank, a group of blade servers, or a multi-processor system.

The memory604stores information within the computing device600. In one implementation, the memory604is a volatile memory unit or units. In another implementation, the memory604is a non-volatile memory unit or units. The memory604can also be another form of computer-readable medium, such as a magnetic or optical disk.

The storage device606is capable of providing mass storage for the computing device600. In one implementation, the storage device606can be or contain a computer-readable medium, such as a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid state memory device, or an array of devices, including devices in a storage area network or other configurations. A computer program product can be tangibly embodied in an information carrier. The computer program product can also contain instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer- or machine-readable medium, such as the memory604, the storage device606, or memory on processor602.

The high speed controller608manages bandwidth-intensive operations for the computing device600, while the low speed controller612manages lower bandwidth intensive operations. Such allocation of functions is exemplary only. In one implementation, the high-speed controller608is coupled to memory604, display616, e.g., through a graphics processor or accelerator, and to high-speed expansion ports610, which can accept various expansion cards (not shown). In the implementation, low-speed controller612is coupled to storage device606and low-speed expansion port614. The low-speed expansion port, which can include various communication ports, e.g., USB, Bluetooth, Ethernet, wireless Ethernet can be coupled to one or more input/output devices, such as a keyboard, a pointing device, microphone/speaker pair, a scanner, or a networking device such as a switch or router, e.g., through a network adapter. The computing device600can be implemented in a number of different forms, as shown in the figure. For example, it can be implemented as a standard server620, or multiple times in a group of such servers. It can also be implemented as part of a rack server system624. In addition, it can be implemented in a personal computer such as a laptop computer622. Alternatively, components from computing device600can be combined with other components in a mobile device (not shown), such as device650. Each of such devices can contain one or more of computing device600,650, and an entire system can be made up of multiple computing devices600,650communicating with each other.

The computing device600can be implemented in a number of different forms, as shown in the figure. For example, it can be implemented as a standard server620, or multiple times in a group of such servers. It can also be implemented as part of a rack server system624. In addition, it can be implemented in a personal computer such as a laptop computer622. Alternatively, components from computing device600can be combined with other components in a mobile device (not shown), such as device650. Each of such devices can contain one or more of computing device600,650, and an entire system can be made up of multiple computing devices600,650communicating with each other.

Computing device650includes a processor652, memory664, and an input/output device such as a display654, a communication interface666, and a transceiver668, among other components. The device650can also be provided with a storage device, such as a microdrive or other device, to provide additional storage. Each of the components650,652,664,654,666, and668, are interconnected using various buses, and several of the components can be mounted on a common motherboard or in other manners as appropriate.

The processor652can execute instructions within the computing device650, including instructions stored in the memory664. The processor can be implemented as a chipset of chips that include separate and multiple analog and digital processors. Additionally, the processor can be implemented using any of a number of architectures. For example, the processor610can be a CISC (Complex Instruction Set Computers) processor, a RISC (Reduced Instruction Set Computer) processor, or a MISC (Minimal Instruction Set Computer) processor. The processor can provide, for example, for coordination of the other components of the device650, such as control of user interfaces, applications run by device650, and wireless communication by device650.

Processor652can communicate with a user through control interface658and display interface656coupled to a display654. The display654can be, for example, a TFT (Thin-Film-Transistor Liquid Crystal Display) display or an OLED (Organic Light Emitting Diode) display, or other appropriate display technology. The display interface656can include appropriate circuitry for driving the display654to present graphical and other information to a user. The control interface658can receive commands from a user and convert them for submission to the processor652. In addition, an external interface662can be provide in communication with processor652, so as to enable near area communication of device650with other devices. External interface662can provide, for example, for wired communication in some implementations, or for wireless communication in other implementations, and multiple interfaces can also be used.

The memory664stores information within the computing device650. The memory664can be implemented as one or more of a computer-readable medium or media, a volatile memory unit or units, or a non-volatile memory unit or units. Expansion memory674can also be provided and connected to device650through expansion interface672, which can include, for example, a SIMM (Single In Line Memory Module) card interface. Such expansion memory674can provide extra storage space for device650, or can also store applications or other information for device650. Specifically, expansion memory674can include instructions to carry out or supplement the processes described above, and can include secure information also. Thus, for example, expansion memory674can be provide as a security module for device650, and can be programmed with instructions that permit secure use of device650. In addition, secure applications can be provided via the SIMM cards, along with additional information, such as placing identifying information on the SIMM card in a non-hackable manner.

The memory can include, for example, flash memory and/or NVRAM memory, as discussed below. In one implementation, a computer program product is tangibly embodied in an information carrier. The computer program product contains instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer- or machine-readable medium, such as the memory664, expansion memory674, or memory on processor652that can be received, for example, over transceiver668or external interface662.

Device650can communicate wirelessly through communication interface666, which can include digital signal processing circuitry where necessary. Communication interface666can provide for communications under various modes or protocols, such as GSM voice calls, SMS, EMS, or MMS messaging, CDMA, TDMA, PDC, WCDMA, CDMA2000, or GPRS, among others. Such communication can occur, for example, through radio-frequency transceiver668. In addition, short-range communication can occur, such as using a Bluetooth, Wi-Fi, or other such transceiver (not shown). In addition, GPS (Global Positioning System) receiver module670can provide additional navigation- and location-related wireless data to device650, which can be used as appropriate by applications running on device650.

Device650can also communicate audibly using audio codec660, which can receive spoken information from a user and convert it to usable digital information. Audio codec660can likewise generate audible sound for a user, such as through a speaker, e.g., in a handset of device650. Such sound can include sound from voice telephone calls, can include recorded sound, e.g., voice messages, music files, etc. and can also include sound generated by applications operating on device650.

The computing device650can be implemented in a number of different forms, as shown in the figure. For example, it can be implemented as a cellular telephone480. It can also be implemented as part of a smartphone682, personal digital assistant, or other similar mobile device.

Various implementations of the systems and methods described here can be realized in digital electronic circuitry, integrated circuitry, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and/or combinations of such implementations. These various implementations can include implementation in one or more computer programs that are executable and/or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

These computer programs (also known as programs, software, software applications or code) include machine instructions for a programmable processor, and can be implemented in a high-level procedural and/or object-oriented programming language, and/or in assembly/machine language. As used herein, the terms “machine-readable medium” “computer-readable medium” refers to any computer program product, apparatus and/or device, e.g., magnetic discs, optical disks, memory, Programmable Logic Devices (PLDs), used to provide machine instructions and/or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term “machine-readable signal” refers to any signal used to provide machine instructions and/or data to a programmable processor.

To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input.

The systems and techniques described here can be implemented in a computing system that includes a back end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front end component, e.g., a client computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here, or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (“LAN”), a wide area network (“WAN”), and the Internet.

The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

A number of embodiments have been described. Nevertheless, it will be understood that various modifications can be made without departing from the spirit and scope of the invention. In addition, the logic flows depicted in the figures do not require the particular order shown, or sequential order, to achieve desirable results. In addition, other steps can be provided, or steps can be eliminated, from the described flows, and other components can be added to, or removed from, the described systems. Accordingly, other embodiments are within the scope of the following claims.