Patent Publication Number: US-11650996-B1

Title: Determining query intent and complexity using machine learning

Description:
INTRODUCTION 
     Aspects of the present disclosure relate to determining an intent of a query and a complexity of the query using machine learning to automate responding to the query. 
     Application providers frequently offer various self-help or customer support options in order to improve user experiences of the application. For example, if a user encounters a problem while using the application, the user may submit a query, about the problem. Queries may be received as text from an input device, as audio, as an image or as other methods of input. A human customer support agent may be assigned to the query and respond to it in order to remedy the problem. However, for large applications with a large number of users, human response to all such queries is impractical. Many providers therefore employ automated processes to allow a computing device to make decisions about the query. 
     However, existing systems of automating a response to queries have several drawbacks. First, responding to a query typically involves more than one analysis, and as a result, existing systems separate the task of responding to queries into multiple parts, and rely on serialized systems to deal with the multiple parts. Thus, for existing system to respond to a query, two (or more) separate data processing systems may need to be trained and maintained. Further, during the actual processing of a query, the data may need to be prepared as input for each data processing system separately, increasing the time needed to respond to the query as well as the processing resources used in responding to a single query. For applications, even a small increase in response time for each individual query can lead to significant processing delays not only the customer service or self-help services of an application, but for the application as a whole. 
     Thus, systems and methods are needed to automate responding to user queries that avoid the shortcomings of existing systems. 
     BRIEF SUMMARY 
     Certain embodiments provide a method for determining query intent and complexity based on text input. The method generally includes receiving, from a user device, a text query and preprocessing the text query to generate a query vector. The method further includes providing the query vector to an intent model configured to output a user intent of the text query and providing the query vector to a complexity model configured to output a complexity of the text query. The method further includes receiving the user intent of the text query from the intent model and receiving the complexity of the text query from the complexity model. The method further includes determining, based on the user intent and the complexity of the query, a routing target for the text query and routing the text query to the routing target. 
     Other embodiments include a computing device comprising a processor and memory including instructions that cause the computing device to perform the method determining query intent and complexity based on text input described above. Still other embodiments provide a computer readable medium storing instructions that can cause a computing device to perform the method for determining query intent and complexity based on text input described above. 
     The following description and the related drawings set forth in detail certain illustrative features of one or more embodiments. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       The appended figures depict certain aspects of the one or more embodiments and are therefore not to be considered limiting of the scope of this disclosure. 
         FIG.  1    depicts an example computing environment for automating responses to user queries. 
         FIG.  2 A  depicts an example preprocessor. 
         FIG.  2 B  depicts an example query vector. 
         FIG.  3 A  depicts an example intent system. 
         FIG.  3 B  depicts an example complexity system. 
         FIG.  4    is a conceptual illustration of an example data pipeline used in determining query intent and complexity. 
         FIG.  5    is a flow diagram illustrating an example method for determining query intent and complexity based on text input. 
         FIG.  6    depicts an example message routing server, for use in determining query intent and complexity. 
     
    
    
     To facilitate understanding, identical reference numerals have been used, where possible, to designate identical elements that are common to the drawings. It is contemplated that elements and features of one embodiment may be beneficially incorporated in other embodiments without further recitation. 
     DETAILED DESCRIPTION 
     Aspects of the present disclosure provide apparatuses, methods, processing systems, and computer readable mediums for determining an intent of a query and a complexity of the query using machine learning to automate routing the query. 
     Effective responses to user queries about an application can improve the user experience with the application. Responding to a user query may include determining an intent (or meaning) of the query as well as determining a complexity of the query. The intent of the query can be used to determine a topic related to the query and effectively route the query to a source of information related to the topic, which may then in turn respond to the query. 
     The complexity of the query can also be used to determine if the service the user currently uses may be inadequate for the user&#39;s needs, and if an additional service of the application (e.g., a premium service) may be better suited for the user&#39;s task. The complexity of the query can further be used to determine what sort of agent would be best suited to respond to the query, such as an expert or a non-expert (e.g., a generalist support agent). 
     Existing methods for responding to queries tend to deal with the two tasks (intent and complexity) serially, that is, using output from an intent determination while determining complexity or vice versa. By performing the two tasks in parallel, however, improvements in system performance, and ultimately user experience, may be realized. For example, by providing two machine learning models to perform each task separately, the process of responding to a query can be performed in parallel. Further, input provided to machine learning models is typically preprocessed, which can be a significant operation when the system is handling many queries. If the two machine learning models make use of the same data pipeline, preprocessing can be condensed to a single step, thus reducing processing time for responding to a query and lowering turnaround time between a query being submitted and responded to. Reducing this response time may improve the user experience with the application in addition to improving the efficiency of processing resources used in responding to the query. 
       FIG.  1    depicts an example computing environment  100  for automating responses to user queries. Computing environment  100  includes message routing server  120 , intent system  130 , complexity system  140  and user device  150 , all connected via network  110 . Computing environment  100  also includes information server  162 , agent device  164  and expert device  166 , which are potential routing targets of message routing server  120 . Computing environment  100  also includes application information database  136  connected to intent system  130 . In other examples, application information database  136  may be available to intent system  130  over network  110  rather than being connected to intent system  130 . Further, although shown as separate devices in other examples, the functions of message routing server  120 , intent system  130  and complexity system  140  may be performed by a single computing device, or by a distributed or cloud computing system comprising additional computing devices. Further, the functions of application information database  136 , information server  162 , agent device  164  and expert device  166  may also be performed in the cloud computing system. Further still, in some embodiments user device  150  may comprise part of the cloud computing system operated by a user remotely. 
     Message routing server  120  is a computing device comprising at least a memory and a processor capable of executing various computer-executable instructions, such as software applications or utilities. Message routing server  120  is used to route messages and traffic of computing environment  100 , in this case routing communication from user device  150  based on text query  152 . Message routing server  120  includes preprocessor  122 , query vector  124 , routing module  126 , query intent  134 , query complexity  144 , and service offer  128 . 
     Preprocessor  122  is a software utility, routine, script, application, or similar executing on message routing server  120 , which uses text query  152  to generate a query vector representation of text query  152 . The functions of preprocessor  122  are described in greater detail below with respect to  FIG.  2 A . Thus, query vector  124  is the product of preprocessor  122 . Query vector  124  may be suitable for use by intent model  132  and complexity model  142 . 
     Routing Module  126  is a software routine, routine, script, application, or similar executing on message routing server  120 , which performs routing of text queries, such as text query  152 , to various routing targets of message routing server  120 . Routing module  126  may make use of query intent  134  and query complexity  144  in determining a routing target for text query  152 . 
     Query Intent  134  is a determination of the intent of or associated with text query  152 . Query intent  134  may either be a classification of text query  152  specifying a most likely intent class (or set of most likely intent classes) or may be a set of intent class scores showing a likelihood of association between text query  152  and each intent class. Query intent  134  is generated as output of intent model  132  and transmitted from intent system  130  to message routing server  120 . 
     Query Complexity  144  is a classification of the complexity of text query  152 . Query complexity  144  may comprise a complexity class that identifies the complexity of a query by the proper method of response to the query (e.g., whether a query is best answered by an expert agent or by a non-expert agent). Query complexity  144  may also be a set of numeric complexity class scores showing a likelihood of association between text query  152  and each complexity class. Message routing server  120  uses query complexity  144  to determine a proper routing target for text query  152  (e.g., a query best answered by an expert agent should be routed to an expert agent). Query complexity  144  is generated as output of complexity model  142  and transmitted from complexity system  140  to message routing server  120 . 
     Service offer  128  is a message generated by message routing server  120  to be transmitted to a user of user device  150 . As discussed, message routing server  120  is associated with a software application. The software application may be a web-based data processing application which collects information from users and processes the information, and may include a plurality of different service levels (e.g., a free service level and a plurality of paid service levels). By analyzing query intent  134  and query complexity  144 , message routing server  120  may determine that the service the user is currently using is not sufficient for the user&#39;s needs. For example, if the software application is a tax processing application and text query  152  relates to IRS form 1099MISC, query intent  134  may indicate that the intent of text query  152  is 1099MISC. If processing of form 1099MISC is not offered in a free service level of the software application, service offer  128  may include an offer for the user to purchase a recommended service that includes form 1099MISC processing. 
     Intent system  130  is a computing device used to execute intent model  132 . In some cases, intent system  130  may also be used to train intent model  132 . In this example, intent system  130  is a separate computing device from message routing server  120 , comprising a processor, a memory and storage. Intent model  132  is included in the storage, and can be executed by the processor while resident in the memory. In other embodiments, intent model  132  may be trained by intent system  130  but execute on message routing server  120 . Intent model  132  may be a variety of different machine learning models, including logistic regression models, decision trees, random forests, and deep neural networks. Intent model  132  accepts a vector representing text as input (in this case, query vector  124 ) as input and outputs a determined intent of the text represented by the vector (shown as query intent  134 ). The functions of intent model  132  are described in greater detail below with respect to  FIG.  3 A . 
     Application information database  136  is a database or storage device connected to intent system  130 , which includes application-specific information for a plurality of users. For example, application information database  136  may include tax information for users in support of a tax preparation application offered to users. In some cases, text query  152  may include an identifier of the user that authored text query  152 . If so, intent system  130  may access application information database  136  to obtain application information related to the user, which can be used to further refine query intent  134 . Further, text query  152  may include an application identifier, which identifies the application from which or about which the user is inquiring. 
     Complexity system  140  may be a computing device used to execute complexity model  142 . In some cases, complexity system  140  may also be used to train complexity model  142 . In this example, complexity system  140  is a separate computing device from message routing server  120 , comprising a processor, a memory and storage. Complexity model  142  is included in the storage, and can be executed by the processor while resident in the memory. In other embodiments, complexity model  142  may be trained by complexity system  140  but execute on message routing server  120 . Complexity model  142  is also a machine learning model. Like intent model  132 , complexity model  142  may also be a variety of different machine learning models. Complexity model  142  accepts a vector representing text as input (in this case, query vector  124 ) and outputs a determined complexity of the text represented by the vector (shown as query complexity  144 ). In general, complexity model  142  performs an analysis of query vector  124  to identify a relative complexity level of query vector  124 . The functions of complexity model  142  are described in greater detail below with respect to  FIG.  3 B . 
     User Device  150  may be a computing device comprising at least a memory and a processor capable of executing various computer-executable instructions, such as software applications or utilities. User device  150  is operated by a user of the software application associated with message routing server  120 . A user of user device  150  may generate a text query  152  based on the user&#39;s use of an application operating on user device  150 . 
     Text query  152  may be a question or message about a problem the user is having with an application running on user device  150  (either locally or remotely, such as via web application). Text query  152  may be received at user device  150  in a variety of forms. Text query  152  may be received at user device  150  as text entered via an input device, such as a touchscreen or a keyboard. Text query  152  may also be received at user device  150  as audio recorded by a microphone, and processed using natural language processing to determine a text representation of the audio. Text query  152  may also be received at user device  150  as an image containing text (e.g., an image of a paper document taken by a digital camera or scanned in via a scanner), and processed using optical character recognition (OCR) to determine a text representation of the image. User device  150  transmits text query  152  to message routing server  120  over network  110  as part of resolving text query  152 . 
     Information server  162 , agent device  164 , and expert device  166  are computing devices that may receive messages routed from message routing server  120  and thus may be referred to as routing targets. Information server  162  may be a computing device storing information related to the software application associated with message routing server  120 . Agent device  164  may be a computing device operated by a non-expert (e.g., general) support agent, and expert device  166  may be a computing device operated by an expert (e.g., specialist) support agent. 
     During the course of using the software application, the user of user device  150  may encounter a problem. In order to resolve this problem, the user creates text query  152 . As described above, the user may create text query  152  by submitting a query via text, audio or image to user device  150 . The software application may add an application identifier to text query  152  to assist routing by message routing server  120 . The software application may also include application context within the text query  152 . The application context may be used by message routing server  120  to pinpoint a section of the software application the user was at when the user created text query  152 . Text query  152  is then sent by user device  150  to message routing server  120 . In general, message routing server  120  routes text query  152  by analyzing both the intent (or meaning) and complexity of text query  152 . To determine the intent and complexity of text query  152 , message routing server  120  preprocesses text query  152  and passes the result (query vector  124 ) to intent model  132  via intent system  130  and to complexity model  142  via complexity system  140 . 
     Intent model  132  accepts query vector  124  as input and outputs query intent  134 . Complexity model  142  also accepts query vector  124  as input and outputs query complexity  144 . Query vector  124  may be provided to intent model  132  and complexity model  142  in parallel, that is, without waiting for one model to finish before the other, as each accepts the same input. This parallelism creates time and processing efficiencies. 
     Message routing server  120  may use the application identifier, query intent  134  and query complexity  144  to determine where to route text query  152  (e.g., to one of the routing targets, including information server  162 , agent device  164 , and expert device  166 ). For example, message routing server  120  may use a numeric intent class score to determine an intent of text query  152 . For example, message routing server  120  may apply a threshold to intent class scores for text query  152  (e.g. a score of at least 80 points on a 100 point scale for a given intent class) to determine that text query  152  is associated with a particular intent class. If message routing server  120  determines that text query  152  relates to an intent that may be answered by information contained in a server, message routing server  120  may then use the application identifier to determine a server associated with the application identifier, which in this case is information server  162 , and so route text query  152  to information server  162 . 
     If, however, message routing server  120  determines the topic of text query  152  is not answerable using information server  162 , message routing server  120  may instead route text query  152  to either agent device  164  or expert device  166 . In this case, message routing server  120  may compare complexity class scores for text query  152  to a threshold to determine a proper routing target of text query  152 . For example, message routing server  120  may apply a threshold to establish a likelihood that text query  152  is associated with a particular complexity class. Using this threshold information, message routing server  120  may determine whether text query  152  should be routed to agent device  164  or expert device  166  (e.g., a score of 88 for the complexity class “expert response needed” may lead to message routing server  120  routing text query  152  to expert device  166 ). If multiple complexity class scores exceed the threshold, message routing server  120  may default to routing text query  152  to expert device  166 , or may make a copy of text query  152  and route text query  152  to multiple routing targets. 
     Message routing server  120  may use both query intent  134  and query complexity  144  to further refine a routing decision for queries determined to require an expert response. For example, message routing server  120  may use query complexity  144  to determine that an expert response is needed. Thereafter, message routing server  120  may use query intent  134  to determine a more specific type of expert needed to respond to text query  152 . For example, some text queries may require response by one type of expert (e.g., a tax attorney) while other text queries may require response by a different type of expert (e.g., an accountant). Message routing server  120  may access a database of expert profiles to determine a routing target in this example. The database of expert profiles may indicate an association between each expert agent and a set of intents or topics of the software application. Message routing server  120  may use the database of expert profiles to determine a proper expert agent for a particular text query, based on query intent  134 , and route text query  152  to the proper expert agent. 
     Message routing server  120  may further use query intent  134  to determine a recommended service to offer to the user of user device  150 . By again using a threshold of intent classes, message routing server  120  may determine that query intent  134  indicates that text query  152  relates to a particular intent not covered by a base service level of the software application. Using the application identifier and the application content, message routing server  120  may further determine that text query  152  relates to a topic that cannot be resolved using the user&#39;s current level of service. Message routing server  120  may therefore offer an upgraded service via service offer  128 . 
     Because intent model  132  and complexity model  142  make use of the same input (query vector  124 ), overhead processing time is saved as preprocessing is only performed once. Further, as the two models can be executed in parallel, the time to respond to a particular query can be reduced. For a message routing server that processes many hundreds or thousands of queries, this decrease in processing time results in a significant savings in both processing time for queries and processing resources used in responding to queries. 
       FIG.  2 A  depicts an example preprocessor, such as preprocessor  122  of  FIG.  1   . Preprocessor  122  is a software component or routine executing on a message routing server, such as message routing server  120  of  FIG.  1   . Preprocessor  122  performs preprocessing tasks on a text query to generate a query vector representing the text query. Preprocessor  122  includes stop word module  220 . Stop word module  220  is a software subroutine of preprocessor  122  executed as part of the execution of preprocessor  122 . Stop word module  220  is used to remove stop words from the text query. Stop words may be removed before natural language processing in order to improve processing time and efficiency, because they are common words that convey little meaning, but may complicate searching for phrases. Common stop words include prepositions and articles, such as “the,” “is,” “at,” “which” or “on”. Identifying stop words to remove can be performed by utilizing a pre-defined list of English stop words. 
     Preprocessor  122  also includes stemming module  222 . Stemming module  222  is a software subroutine of preprocessor  122  executed as part of the execution of preprocessor  122 . Stemming module  222  reduces derived words of the text query to roots, such as by performing stemming or lemmatization. Stemming is the process of reducing inflected (or sometimes derived) words to their word stem, base, or root form. Stemming algorithms such as Porter&#39;s Algorithm and others may be applied during stemming. Lemmatization is a more complex approach to determining a stem of a word that involves first determining the part of speech of a word, and applying different normalization rules for each part of speech. In lemmatization, the part of speech is first detected prior to attempting to find the root since for some languages, the stemming rules change depending on a word&#39;s part of speech. In some cases, stemming and lemmatization approaches may be used together in hybrid approaches. 
     Preprocessor  122  also includes tokenization module  224 . Tokenization module  224  is a software subroutine of preprocessor  122  executed as part of the execution of preprocessor  122 . Tokenization module  224  is used to perform tokenization on text query  152 . Tokenization is the process of converting a sequence of characters (such as a word) into a token with an assigned meaning. For example, the word “apple” may be associated with a dictionary ID “123” whereas “orange” may be associated with a dictionary ID “241”. Some methods used to identify tokens in text, such as support text query  152 , include: regular expressions, specific sequences of characters termed a flag, specific separating characters called delimiters, and explicit definition by a dictionary. Special non-alphabetical characters, including punctuation characters, are commonly used to identify tokens because of their natural use in written languages; however, tokenization may ultimately eliminate all the non-alphabetical symbols in the original text. 
     Preprocessor  122  also includes augmentation module  226 . Augmentation module  226  is a software subroutine of preprocessor  122  executed as part of the execution of preprocessor  122 . Augmentation module  226  is used to perform data augmentation on text query  152 . Data augmentation typically involves supplementing a data set by modifying the available data. In natural language processing, data augmentation includes using the thesaurus method to add synonyms of words to the text set, and add alternate versions or alternate spellings of words to the text. For example, data augmentation may prompt preprocessor  122  to add the phrase “can not” and the word “cannot” if the word “can&#39;t” is present in the text. Data augmentation may also involve expanding certain words into more complete definitions. For example, if text query  152  is a question about a tax form, preprocessor  122  may add the full name of the tax form to the text during data augmentation. 
     Preprocessor  122  also includes vectorization module  228 . Vectorization module  228  is a software subroutine of preprocessor  122  executed as part of the execution of preprocessor  122 . In other examples, vectorization module  228  may be a local client of a vectorization service executing remotely. Vectorization module  228  is used to perform vectorization of the text query. Vectorization involves converting letter strings, words, phrases or whole sentences into numerical values of fields of a vector. In some implementations, a tool such as “word2vec” can be used to create vectors representing individual words from text-based data. In word2vec, a vector represents a single word, with the dimensions of the vector representing associations between the single word and other words in a corpus. In this example, the output of vectorization module  228  is a query vector that represents the text query after preprocessing. If the query vector represents a full sentence rather than a single world, vectorization module  228  may generate the query vector by calculating a word vector for each word of the sentence, and then computing the average value for each dimension of the word vectors. 
     Another example of a tool to perform vectorization is TF-IDF (term frequency-inverse document frequency), where each unique word among a collection of sentences is weighted by the word&#39;s occurrence in a single sentence, and inversely weighted by the word&#39;s total occurrence across all sentences. TD-IDF can be used to identify and give a high weight to relevant words (e.g., words that are frequent within a particular sentence but are infrequent across all sentences). 
     As an example of preprocessing, consider the text “My 1095a is not correct.” “My 1095a is not correct” may first be augmented by augmentation module  226  so that the form name “1095a” is replaced with its full name, “Health Insurance Marketplace Statement”. The text is then lower-cased and parsed, the result being the following set of words: [“my”, “health”, “insurance”, “marketplace”, “statement”, “is”, “not”, “correct”]. The English stop words (in this case, “my”, “is” and “not”) are removed by stop word module  220 , which leaves [“health”, “insurance”, “marketplace”, “statement”, “correct”]. Then each word is converted into a vector by vectorization module  228 . Vectorization module  228  may then calculate the average of each vector to create a query vector for the complete text. 
       FIG.  2 B  depicts an example query vector, such as query vector  124  of  FIG.  1   . Query vector  124  is the output of a preprocessor, such as preprocessor  122 . Query vector  124  represents the text of a text query in a numerical vector space. Query vector  124  includes numeric vector  230 . Numeric vector  230  is the numeric data representing text of the text query that query vector  124  represents. In this example, numeric vector  230  is the raw output of the preprocessor. 
     In this example, query vector  124  has been prepared for use in training a machine learning model, and as a result, query vector  124  includes intent class  232  and complexity class  234 . To be used in training, a query vector is labeled with an intent class (for use in training an intent model) and a complexity class (for use in training a complexity model). Intent class  232  identifies a topic associated with a query (e.g., “amend taxes” or “printing”). Complexity class  234  identifies a complexity of the query as discussed above with respect to  FIG.  1   . Complexity class  234 , if indicating the query is best answered by an expert, may further specify a type of expert (e.g., for a tax preparation application, an accountant or a tax attorney). 
     In general, both intent class  232  and complexity class  234  may be created by a human expert. That is, a human expert may analyze a text query and determine, based on the expert&#39;s insight and expertise, whether the text query should be answered by an expert agent or can be answered by a non-expert agent (e.g., a complexity class). The human expert may also determine a topic or intent of the text query (e.g., an intent class). As described in further detail below with respect to  FIG.  3 A , human-created complexity and intent classes can be used in supervised learning or semi-supervised learning. In unsupervised learning, intent class  232  and complexity class  234  may be generated by an unsupervised machine learning model. In such an example, query vector  124  may represent a query vector after analysis by an intent model and a complexity model, rather than a query vector used to train an intent model and a complexity model. 
     Additionally, query vector  124  includes application identifier  236 , which is a data element that identifies a particular software application associated with the query. Application identifier  236  may be used by the intent model to categorize an intent of query vector  124 . In other examples, query vector  124  may also include an application context (not shown). The application context may be used to further refine the intent of query vector  124 . 
       FIG.  3 A  depicts an example intent system, such as intent system  130  of  FIG.  1   . Intent system  130  is a computing device including a processor and memory used to train and execute intent model  132 , which in turn includes learning parameters  310 . Learning parameters  310  are data elements that control the execution of, and thus affect the output of, intent model  132 . Intent model  132  is used to classify vectors representing text queries into various intent classes. Example intent classes may relate to application functionality (such as “printing” or “error screen”) or may relate to various topics covered by the application (such as “1099MISC” or “deductions”). In general, intent model  132  analyzes vectors to identify keywords in the vectors. For example, to identify an intent such as “printing” intent model  132  may determine that a vector is closely associated with other vectors known to intent model  132  to correspond to the intent “printing,” using vector comparison techniques. 
     In different embodiments, intent model  132  may assign a score for the vector for each intent class. For example, intent model  132  may determine, on a point scale, a relevance of the vector to each intent class. For example, a query vector representing the text “My 1095a is not correct” may be assigned a relatively high score (e.g., 90 on a 100 point scale) for the intent class “health insurance” but a relatively score (e.g., below 10 on a 100 point scale) for all other intent classes. As another example, a query vector representing the text “can I file 1099MISC” may be assigned a relatively high score for the intent class “1099MISC.” Intent classes or intent class scores may be used by a message routing server as described above. 
     Intent system  130  also includes training data  320 , output  322 , output optimizer  324  and validation data  326 , which are used to train intent model  132 . Training data  320  is a set of query vectors, such as query vector  124  of  FIG.  2 B . Depending on the type of training used, the set of query vectors may be fully labeled, fully unlabeled or partially labeled. Output  322  is output of intent model  132 , which in this example is a intent classification for the query vectors of training data  320 . Output optimizer  324  is a software routine that compares output  322  to the class labels of training data  320  to determine accuracy. Based on the comparison of output  322  to training data  320 , output optimizer  324  modifies learning parameters  310 . Validation data  326  is an additional set of query vectors, similar to but distinct from training data  320 . 
     Intent system  130  may train intent model  132  using a variety of methods. In general, training a machine learning model can be classified into one of three methods: supervised learning, semi-supervised learning and unsupervised learning. 
     Supervised learning involves use of a set of fully labeled training data (in this example, training data  320 ). In general, to train intent model  132  using supervised learning, input (in this case training data  320 ) is provided to intent model  132 . Intent model  132  then executes, based on learning parameters  310 , and generates output  322 . Output optimizer  324  then compares output  322  to the intent class labels of training data  320 . Output optimizer  324  then modifies learning parameters  310 . Training a machine learning model may be iterative, and may be performed many times until output  322  produced by intent model  132  exceeds an accuracy threshold, or until the improvement in output  322  falls below a threshold for iterative improvement. After training, intent model  132  may be tested by using an additional set of validation data  326  (e.g., data with known labels that was not used for training), to verify training is complete. 
     Unsupervised learning does not require a labeled set of data set in order to train a machine learning model. If run using unsupervised learning, an unsupervised training model may cluster unlabeled data. The unsupervised training model can define the boundary of the clusters so that metrics (e.g., mean squared error from cluster centroid) can be optimized. So, for example, vectors representing text queries could be clustered to determine likeness and a label may be determined for each cluster. 
     In some cases, combining supervised and unsupervised approaches into a semi-supervised approach may improve results. Semi-supervised learning relies on an initially heterogeneous set of training data, with some of the training data being labeled but some (often the majority) being unlabeled. For example, if a cluster of query vectors includes a labeled query vector, the label may be assumed to apply to all of the query vectors in the cluster. Thus, an intent and complexity class may be applied to all of the clustered query vectors based on a single labeled instance in the cluster. 
     Unsupervised techniques combined with natural language processing techniques such as topic modeling (e.g., Latent Dirichlet Allocation (LDA) or non-negative matrix factorization) can establish what makes the classes of the labeled vectors distinct, in semantics, sentiment, or other aspects. As an example, LDA starts with the classes of the labeled vectors, and assumes that each class is associated with unknown vector values. LDA then identifies the vector values for each class based on the co-occurrences of values in vectors of each class. LDA can then be used to compute the probability that a vector belongs to each class. Once these probabilities are established, each of the unlabeled vectors may be classified into one of the classes (e.g., labeled). Because two labels may be applied to each vector, LDA and clustering may be performed twice, once for intent classes and once for complexity classes. This can be an iterative process, where the experts again label a small portion of the vectors that do not fall into any of the existing classes, and the clustering is repeated, until all vectors are labeled. 
       FIG.  3 B  depicts an example complexity system, such as complexity system  140  of  FIG.  1   . Like intent system  130  of  FIG.  3 A , complexity system  140  is a computing device including a processor and memory. Complexity system  140  is used to train and execute complexity model  142 , which in turn includes learning parameters  310 . Complexity model  142  is used to classify the complexity of vectors representing text queries into various complexity classes. Example complexity classes include “expert response needed,” “expert response beneficial” or “expert response not needed.” For example, complexity model  142  may determine that a vector is complex enough to warrant expert response if the vector is closely associated with other vectors within the “expert response needed” complexity class. 
     In different embodiments, complexity model  142  may assign a score for the vector for each complexity class. If assigning a score for the vector, complexity model  142  may determine, on a point scale, a relevance of the vector to a complexity class. For example, a query vector representing the text “can&#39;t print” may be assigned a relatively high score (e.g., 90 on a 100 point scale) for the complexity class “expert response not needed” but a relatively low score (e.g., below 10 on a 100 point scale) for all other complexity classes. On the other hand, a query vector representing the text “can I deduct personal expenses” may be assigned a relatively high score for the complexity class “expert response needed.” Complexity classes or complexity class scores may be used by a message routing server as described above. 
     Complexity system  140  also includes training data  320 , output  322 , output optimizer  324  and validation data  326 . The training of complexity model  142  may be generally similar to the training of intent model  132  as described above with respect to  FIG.  3 A . However, unlike intent model  132 , training complexity model  142  uses the complexity classes of training data  320 , rather than the intent classes of training data  320 . 
       FIG.  4    is a conceptual illustration of an example data pipeline  400  used in determining query intent and complexity. A data pipeline refers to a set of data processing elements as well as the inputs and outputs of those data processing elements. Data pipeline  400  includes text query  152 , preprocessor  122 , query vector  124 , intent model  132 , complexity model  142 , query intent  134 , and query complexity  144 . Data pipeline  400  is used in a method to automatically respond to user queries about a software application. 
     As discussed above with respect to  FIG.  1   , preprocessor  122  accepts text query  152  as input and produces query vector  124  as output. To prepare text query  152  to be used as input, preprocessor  122  may perform several preprocessing and featurization tasks. As described above with respect to  FIG.  2 A , preprocessing may include removing stop words, performing tokenization, and performing data augmentation. 
     After preprocessor  122  generates query vector  124 , query vector  124  may be provided as input to both intent model  132  and complexity model  142 , possibly in parallel. Intent model  132  and complexity model  142 , as discussed above with respect to  FIGS.  3 A and  3 B , are machine learning models. 
     When query vector  124  is provided to intent model  132 , intent model  132  produces query intent  134  as output. Further, when query vector  124  is provided to complexity model  142 , query complexity  144  is produced as output. As discussed above with respect to  FIG.  1   , query intent  134  and query complexity  144  can be used by a message routing server to determine a routing target for text query  152 . Because intent model  132  and complexity model  142  make use of the same data pipeline, the processing time and resources needed to train and execute intent model  132  and complexity model  142  are reduced. In particular, preprocessor  122  creates one vector for use in both models, and similarly only one set of training data is labeled and featurized to train both models. 
       FIG.  5    is a flow diagram illustrating an example method  500  for determining query intent and complexity based on text input. Method  500  may be performed by a message routing server, such as message routing server  120  of  FIG.  1     
     Method  500  begins at  510  where the message routing server receives, from a user device, a text query. In general, the text query is a query asked by a user operating the user device and related to an application the user is using on the user device. The text query may include an indication of which application the user&#39;s query refers to. The text query may be transmitted by the user device to the message routing service over a network connection. For example, if the application is a tax preparation application, the query may be related to tax issues, or may be related to use of the tax preparation application. 
     At  520 , the message routing server preprocesses the text query to generate a query vector, such as by using the techniques described above. 
     At  530 , the message routing server provides the query vector to an intent model configured to output a user intent of the text query. The intent model is a machine learning model that predicts an intent associated with a query vector based on the text query. In some examples the intent model may execute on the message routing server while in other examples the intent model may execute on a separate computing device. 
     At  540 , the message routing server provides the query vector to a complexity model configured to output a complexity of the query vector. The message routing server may provide the query vector to the intent model and the complexity model in parallel, as depicted and described with respect to  FIG.  4   . The complexity model is a machine learning model that predicts a complexity of the query vector, which can be used to estimate the complexity of the text query. In some examples the complexity model may execute on the message routing server while in other examples the complexity model may execute on a separate computing device. 
     As discussed above with respect to  FIG.  4   , the complexity model and the intent model may share a data pipeline. In method  500 , the text query is only preprocessed once before being used as input for both the intent model and the complexity model. In some examples of method  500 , the intent model and the complexity model use the same input vector. 
     At  550 , the message routing server receives the user intent of the text query from the intent model and the complexity of the text query from the complexity model. Because the query vector can be provided to the intent model and the complexity model in parallel, the output from the models can be received by the message routing server in close succession. The user intent and the complexity can then be used to determine a proper routing target by the message routing server. For example, the message routing server can determine, based on the user intent, an topic associated with the text query and the application. If so, the message routing server can then route the text query to a server associated with the topic. 
     In other examples of method  500 , the message routing server determines the text query is associated with an expert complexity class and route the text query to an expert agent. In general, the complexity of the text query may be defined as a complexity class. Query complexities may be classified as described above. The message routing server may route the text query to an expert agent if the text query is too complex to be answered by a non-expert support agent. For example, if the application is a tax preparation application the text query may relate to a question that should be answered by an accountant rather than a customer service representative. In other examples of method  500 , the message routing server may determine the text query is associated with a non-expert complexity class and route the text query to a non-expert agent. 
     At  560 , the message routing server determines, based on the complexity of the text query and the user intent of the text query, a recommended service. In general, the application may have many service tiers available to users. For example, the application may include at least a base service available to all users. In that example, the recommended service may be an additional service of the application other than the base service. The recommended service is typically an upgrade to services of the application currently used by the user. In some examples of method  500 , determining the recommended service includes identifying the recommended service for the user based on a product tier associated with the topic. 
     At  570 , the message routing server transmits, to the user device, a service offer for the recommended service. The service offer may include details on how the user may purchase or otherwise obtain the recommended service. The message routing server typically transmits the service offer to the user device over a network connection. 
     At  580 , the message routing server routes the text query to a routing target, based on the query complexity and the query intent, such as by using techniques described above. A routing target may be any of a group of computing devices available over to the message routing server over the network. For example, routing targets may include computing devices operated by expert agents, computing devices operating by non-expert agents, or information servers containing information for a particular application. In some examples, method  500  further includes determining, based on the user intent, a topic associated with the text query and an application and routing the text query to an information server. 
     In some examples of method  500 , the intent model determines the user intent based in part on tax return information of a user associated with the user device obtained from a tax information database. The message routing service may have access to such a tax return database if, for example, the application associated with the message routing service is a tax preparation application. Tax return information of the user may be stored in the tax information database if the user has used the tax preparation application previously. 
       FIG.  6    depicts an example message routing server  600 , such as message routing server  120  of  FIG.  1   . As shown, the message routing server  600  includes, without limitation, a central processing unit (CPU)  602 , one or more input/output (I/O) device interfaces  604 , which may allow for the connection of various I/O devices  614  (e.g., keyboards, displays, mouse devices, pen input, etc.) to message routing server  600 , network interface  606 , memory  608 , storage  610 , and an interconnect  612 . 
     The CPU  602  may retrieve and execute programming instructions stored in the memory  608 . Similarly, the CPU  602  may retrieve and store application data residing in the memory  608 . The interconnect  612  transmits programming instructions and application data, among the CPU  602 , I/O device interface  604 , network interface  606 , memory  608 , and storage  610 . The CPU  602  is included to be representative of a single CPU, multiple CPUs, a single CPU having multiple processing cores, and the like. The I/O device interface  604  may provide an interface for capturing data from one or more input devices integrated into or connected to the message routing server  600 , such as keyboards, mice, touchscreens, and so on. The memory  608  may represent a random access memory (RAM), while the storage  610  may be a solid state drive, for example. Although shown as a single unit, the storage  610  may be a combination of fixed and/or removable storage devices, such as fixed drives, removable memory cards, network attached storage (NAS), or cloud-based storage. 
     As shown, the memory  608  includes preprocessor  622 , routing module  624 , intent model  626  and complexity model  628 , which are software routines executed based on instructions stored in the storage  610 . Such instructions may be executed by the CPU  602 . 
     As shown, the storage  610  includes query intent  632 , query complexity  634 , query vector  636  and service offer  638 . Preprocessor  622  may generate query vector  636  by preprocessing a text query received from a user device via network interface  606 . Query intent  632  is a classification of the intent of the text query represented by query vector  636 , produced as output by intent model  626 . Query complexity  634  is a classification of the complexity of the text query represented by query vector  624 , produced as output by complexity model  628 . Query intent  632  and query complexity  634  may be used by routing module  624  to generate service offer  638 . 
     The preceding description is provided to enable any person skilled in the art to practice the various embodiments described herein. The examples discussed herein are not limiting of the scope, applicability, or embodiments set forth in the claims. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments. For example, changes may be made in the function and arrangement of elements discussed without departing from the scope of the disclosure. Various examples may omit, substitute, or add various procedures or components as appropriate. For instance, the methods described may be performed in an order different from that described, and various steps may be added, omitted, or combined. Also, features described with respect to some examples may be combined in some other examples. For example, an apparatus may be implemented or a method may be practiced using any number of the aspects set forth herein. In addition, the scope of the disclosure is intended to cover such an apparatus or method that is practiced using other structure, functionality, or structure and functionality in addition to, or other than, the various aspects of the disclosure set forth herein. It should be understood that any aspect of the disclosure disclosed herein may be embodied by one or more elements of a claim. 
     As used herein, the term “determining” encompasses a wide variety of actions. For example, “determining” may include calculating, computing, processing, deriving, investigating, looking up (e.g., looking up in a table, a database or another data structure), ascertaining and the like. Also, “determining” may include receiving (e.g., receiving information), accessing (e.g., accessing data in a memory) and the like. Also, “determining” may include resolving, selecting, choosing, establishing and the like. 
     The methods disclosed herein comprise one or more steps or actions for achieving the methods. The method steps and/or actions may be interchanged with one another without departing from the scope of the claims. In other words, unless a specific order of steps or actions is specified, the order and/or use of specific steps and/or actions may be modified without departing from the scope of the claims. Further, the various operations of methods described above may be performed by any suitable means capable of performing the corresponding functions. The means may include various hardware and/or software component(s) and/or module(s), including, but not limited to a circuit, an application specific integrated circuit (ASIC), or processor. Generally, where there are operations illustrated in figures, those operations may have corresponding counterpart means-plus-function components with similar numbering. 
     The various illustrative logical blocks, modules and circuits described in connection with the present disclosure may be implemented or performed with a general purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device (PLD), discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general-purpose processor may be a microprocessor, but in the alternative, the processor may be any commercially available processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration. 
     A processing system may be implemented with a bus architecture. The bus may include any number of interconnecting buses and bridges depending on the specific application of the processing system and the overall design constraints. The bus may link together various circuits including a processor, machine-readable media, and input/output devices, among others. A user interface (e.g., keypad, display, mouse, joystick, etc.) may also be connected to the bus. The bus may also link various other circuits such as timing sources, peripherals, voltage regulators, power management circuits, and other circuit elements that are well known in the art, and therefore, will not be described any further. The processor may be implemented with one or more general-purpose and/or special-purpose processors. Examples include microprocessors, microcontrollers, DSP processors, and other circuitry that can execute software. Those skilled in the art will recognize how best to implement the described functionality for the processing system depending on the particular application and the overall design constraints imposed on the overall system. 
     If implemented in software, the functions may be stored or transmitted over as one or more instructions or code on a computer-readable medium. Software shall be construed broadly to mean instructions, data, or any combination thereof, whether referred to as software, firmware, middleware, microcode, hardware description language, or otherwise. Computer-readable media include both computer storage media and communication media, such as any medium that facilitates transfer of a computer program from one place to another. The processor may be responsible for managing the bus and general processing, including the execution of software modules stored on the computer-readable storage media. A computer-readable storage medium may be coupled to a processor such that the processor can read information from, and write information to, the storage medium. In the alternative, the storage medium may be integral to the processor. By way of example, the computer-readable media may include a transmission line, a carrier wave modulated by data, and/or a computer readable storage medium with instructions stored thereon separate from the wireless node, all of which may be accessed by the processor through the bus interface. Alternatively, or in addition, the computer-readable media, or any portion thereof, may be integrated into the processor, such as the case may be with cache and/or general register files. Examples of machine-readable storage media may include, by way of example, RAM (Random Access Memory), flash memory, ROM (Read Only Memory), PROM (Programmable Read-Only Memory), EPROM (Erasable Programmable Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), registers, magnetic disks, optical disks, hard drives, or any other suitable storage medium, or any combination thereof. The machine-readable media may be embodied in a computer-program product. 
     A software module may comprise a single instruction, or many instructions, and may be distributed over several different code segments, among different programs, and across multiple storage media. The computer-readable media may comprise a number of software modules. The software modules include instructions that, when executed by an apparatus such as a processor, cause the processing system to perform various functions. The software modules may include a transmission module and a receiving module. Each software module may reside in a single storage device or be distributed across multiple storage devices. By way of example, a software module may be loaded into RAM from a hard drive when a triggering event occurs. During execution of the software module, the processor may load some of the instructions into cache to increase access speed. One or more cache lines may then be loaded into a general register file for execution by the processor. When referring to the functionality of a software module, it will be understood that such functionality is implemented by the processor when executing instructions from that software module. 
     The following claims are not intended to be limited to the embodiments shown herein, but are to be accorded the full scope consistent with the language of the claims. Within a claim, reference to an element in the singular is not intended to mean “one and only one” unless specifically so stated, but rather “one or more.” Unless specifically stated otherwise, the term “some” refers to one or more. No claim element is to be construed under the provisions of 55 U.S.C. § 112(f) unless the element is expressly recited using the phrase “means for” or, in the case of a method claim, the element is recited using the phrase “step for.” All structural and functional equivalents to the elements of the various aspects described throughout this disclosure that are known or later come to be known to those of ordinary skill in the art are expressly incorporated herein by reference and are intended to be encompassed by the claims. Moreover, nothing disclosed herein is intended to be dedicated to the public regardless of whether such disclosure is explicitly recited in the claims.