Patent Publication Number: US-2021192131-A1

Title: Machine learning of response selection to structured data input including momentum classification

Description:
CROSS-REFERENCE TO RELATED APPLICATION 
     This application is a continuation-in-part of U.S. patent application Ser. No. 15/992,851, which is a continuation of U.S. patent application Ser. No. 15/897,885, which is a continuation-in-part of U.S. patent application Ser. No. 15/826,151 (referred to herein as the “&#39;151 application”), which are all incorporated by reference in their entireties. This application also claims priority to U.S. Provisional Patent Application No. 62/956,166, which is incorporated by reference in its entirety. 
    
    
     BACKGROUND OF THE DISCLOSURE 
     Field of the Disclosure 
     The present disclosure relates to supervised learning processing, and more particularly, to a system, method, and computer program product involving machine learning of response selection to structured data input including momentum. 
     Description of the Related Art 
     Supervised learning is a machine learning process that infers a function for determining an output based on training data, and the function is used to map new input data to an output. A natural language processor (NLP) represents one embodiment of a supervised learning processor. In general, a natural language processor (NLP) includes one or more NLP models to generate a prediction about and a response to a human-understandable, natural language (NL) statement. In at least one embodiment, the NL statement may be a statement, such as a query or command, and the NLP interprets the statement in order to provide a response. 
     Humans intuitively decide on how to respond to a conversational statement. For example, if a human is asked by an inquiring individual, “How large is the lot at 123 Pecan?” The human intuitively knows the context of the statement relates to the area of a parcel of real estate at a particular address. The human responder then provides an appropriate response. If the human knows the answer, the human responds to the inquirer with the answer. So, if the lot is ½ acre, strictly the answer to the question is “the lot is ½ acre.” However, humans can intuitively enhance the response. For example, the human may know the history and specific details about the inquirer and provide a more insightful response that enhances the answer with information the human could anticipate that the inquirer would want to know or utilize semantics appropriate to the inquirer. For example, if the human knows the inquirer has children and would like a swimming pool, the human in addition to providing the size of the lot, the human may have insights into the inquirer and augment the answer with information with such insights, such as the presence or absence of a pool and state the particular schools nearby. Thus, in response to, “How large is the lot at 123 Pecan?” rather the human may respond, “The lot is rather large at ½ acre, has a pool, and the nearby schools are highly rated.” Additionally, the human can intuitively gauge the acceptability to the inquirer of the response. 
     However, machines do not have the benefit of human intuition and cannot determine a proper response in the same way as a human. Furthermore, machine responses are often disadvantageously repetitive, ‘mechanical,’ and easily distinguishable from a human response. Additionally, the machine responses are not insightful. 
     SUMMARY OF THE DISCLOSURE 
     In at least one embodiment, a system of one or more computers perform operations to respond to messages, such as textual messages, from user devices with one or more templates that have been ranked using predetermined criteria including a machine learning classification of a momentum of the messages. At least one embodiment includes a method of machine learning for selecting a ranked response to a message, such as a textual message, received from a user device. An embodiment of the method, operating in an electronic, machine learning processing system, includes receiving messages from user devices after transmission of one or more ranked response templates to the user devices, providing the messages to a momentum classifier implemented as a machine learning model, assigning momentum classifications to the messages using the machine learning model, correlating the templates communicated to the user devices with corresponding momentum classifications, operating a ranking engine to rank the selection of response templates in accordance with ranking criteria, where the ranking criteria includes momentum classifications for the response templates; and selecting a highest ranked response template to provide a response to the message received from the user devices. Other embodiments include corresponding computer systems, apparatus, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of the methods. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       The present disclosure may be better understood, and its numerous objects, features and advantages made apparent to those skilled in the art by referencing the accompanying drawings. The use of the same reference number throughout the several figures designates a like or similar element. 
         FIG. 1  depicts an exemplary machine learning system that includes a learning, ranked response selection module. 
         FIG. 2  depicts one embodiment of the machine learning system that includes a learning, ranked response selection module of  FIG. 1 . 
         FIG. 3  depicts an exemplary response ranking and selection process. 
         FIGS. 4 and 5  depict structured data input. 
         FIG. 6  depicts a response template data structure and response template library. 
         FIGS. 7-8  depict selected response template candidates. 
         FIGS. 9-12  depict exemplary response template ranking parameters and results. 
         FIG. 13  depicts an exemplary computer system. 
         FIG. 14  illustrates a response analysis system. 
         FIG. 15  is a flowchart showing exemplary machine learning operations executed in certain embodiments of the disclosed system. 
         FIG. 16  is a table showing exemplary text messages and corresponding momentum scores for classification by a machine learning model. 
         FIG. 17  is a table showing exemplary response criterion and corresponding weighting factors. 
         FIG. 18  depicts an exemplary computer system. 
         FIG. 19  depicts embodiments of a computer implemented natural language processing system. 
     
    
    
     DETAILED DESCRIPTION 
     A machine learning of response selection to structured data input enables a machine to flexibly and responsively actively engage with a response recipient through a device, such as any electronic device connected to a data network. In at least one embodiment, the response selection module improves response selection to the structure data input by initially filtering a library of templates to identify candidate templates that best respond to the input. In at least one embodiment, the response selection module ranks the identified candidate templates to provide the response to the device. The response selection module learns by receiving feedback, such as a linked recipient action result signal. The linked recipient action result signal tracks activity of a recipient that is linked to receipt of a particular response. As a particular response becomes linked more frequently with activity that is considered a success, the probability of selecting the template from which the response was derived increases. To all the response selection module to continue learning, new response templates are introduced to the library of templates. In at least one embodiment, the response selection module selects the new response templates in accordance with a predetermined function to allow the response selection module to gauge success of the new response templates. 
     In at least one embodiment, the response selection module employs multiple mechanisms to provide ongoing improvement of and learning by the response selection module. Ranking the response templates allows the response selection module to continually learn which response to select and provide to recipients to maximize successful outcomes. Additionally, in at least one embodiment, the response selection module accesses additional data sources that can be processed to provide insights that may be helpful in better engaging the recipient and resulting in improved success rates. Furthermore, insights can yield proactive engagement with a past recipient. Additionally, successful insights can be incorporated into response templates for future use with other recipients. The same learning process then allows the response selection module to learn based on such new insights. 
     In at least one embodiment, the response selection module uses particular ranking criteria to improve learning by the response selection module of when and which response templates to select for particular recipients. In at least one embodiment, the ranking criteria determines a conversion rate of each candidate template that takes into account multiple factors that influence the ranking process. In at least one embodiment, the conversion rate is defined as recipient activities associated with a template relative to total impressions of the template. In at least one embodiment, the multiple factors that influence the conversion rate include weighting response templates based on (i) activities and measures of closeness that correlate the activity, the recipient, and the provided response and (ii) weighting particular activities where some activities are perceived as more valuable than others. For example, in the context of selling real estate, if the linked recipient action result feedback signal indicates the recipient sends a reply to a response, the response selection module weights the outcome of this activity. If the linked recipient action result feedback signal indicates the recipient schedules a showing of a house, the response selection module weights the outcome of this activity more heavily. Thus, the response selection module provides technical advantages by employing multiple learning mechanisms to enhance the response selection module to learn, modify responses, and improve performance over time to develop machine intuition. 
     In at least one embodiment, the provided responses include textual inputs, such as text messages, that users send to the electronic, machine learning processing system. In at least one embodiment, the text messages are provided to the input of a machine learning engine, which analyzes the text of the messages and assigns momentum scores to the corresponding response templates. In certain embodiments, the momentum scores correspond to an assessment of the satisfaction of the users as the users proceed to a targeted goal of the processing system, such as the purchase of a home. As an example, momentum scores may be used as predictors of the likelihood that users will to continue engagement with the system when certain response templates are used. In at least one embodiment, momentum scores may be used in the selection of the highest ranked, candidate response templates that are used to communicate with the users. 
       FIG. 1  depicts an exemplary machine learning system  100  that includes a learning, ranked response selection module  101  that provides responses to structured data input. The response selection module  101  can be implemented as a special purpose hardware machine including as data processing system configured to execute instructions to implement the response selection module  101 . The response selection module  101  receives structured data input, utilizes filtering criteria to select candidate response templates from a library of templates, ranks the candidate response templates in accordance with ranking criteria, and sends a response derived from the highest ranked response to device  118 . The ranking criteria is adaptive to feedback, such as the linked recipient action result feedback signal  134  or feedback from a human that monitors the structured data input and selected candidate templates. In at least one embodiment, the ranking criteria adapts to the feedback by modifying parameter values of the ranking criteria that improve template selection and, thus, responses to structured data input. 
     In at least one embodiment, the response selection module  101  can receive structured data input from multiple sources and from multiple types of sources. In at least one embodiment, the structured data input has a natural language processing (NLP) output schema to allow the response selection module  101  to seamlessly interact with natural language processor systems, such as NLP system  100 . In at least one embodiment, the NLP output schema refers to a schema utilized by a NLP system, such as the schema used in the exemplary response prediction input data  500  ( FIG. 5 ). In at least one embodiment, the NLP system  100  interacts with the data processing system  110  and receives a natural language statement submitted by an indirect requestor device(s)  108  as described as the “requestor device(s)” in the U.S. Pat. No. 10,013,654, issued Jul. 3, 2018, and titled “COOPERATIVELY OPERATING A NETWORK OF SUPERVISED LEARNING PROCESSORS TO CONCURRENTLY DISTRIBUTE SUPERVISED LEARNING PROCESSOR TRAINING AND PROVIDE PREDICTIVE RESPONSES TO INPUT DATA” (the &#39;654 patent), which is incorporated herein by reference. The device(s)  108  is referred to as an “indirect” requestor device in  FIG. 1  because the response selection module  101  receives structured data input indirectly from the device(s)  108 . The data processing system  110  utilizes the NL statement  106  and contextual data  112  to generate the input data  102  as, for example, described in the &#39;654 patent. In at least one embodiment, contextual data  112  includes conversation state data and user profile data as described in the &#39;654 patent. User profile data represents factual and behavioral information about the user. The NLP system  100  includes natural language (NL) model(s)  130  that generate the structured data input (SDI)  105  as described in the &#39;654 patent. In this embodiment, the structured data input  105  is response prediction input data also as described in the &#39;654 patent. 
     In at least one embodiment, the response selection module  101  also receives structured data input directly from the direct requestor device(s)  114 . In at least one embodiment, the direct requestor device(s)  114  access the insight/other data  116 , which includes information that may provide greater insight into users of the indirect requestor device(s)  108 . For example, if the insight/other data indicates that one of the users has pets and has searched for houses with particular criteria, the direct requestor device  114  could generate structured data input  115  asking if there are houses that meet the criteria that are within a certain distance from a park and indicate that the response selection module  101  send the response to device  118 , which in this instance represents the device  108 . In at least one embodiment, the response selection module  101  selects a response in accordance with filter criteria derived from the structured data input  115  and informs the user of a house meeting the user&#39;s criteria and enhancing the response with the additional information about a nearby park. 
     The response selection module  101  can also obtain information from external data source(s)  113  that can include additional information that might be relevant to the user of device  118 . In at least one embodiment, the response selection module  101  can utilize the additional information to derive filter criteria to select a response template that may have a higher chance of success in directing the user to a preferred activity. 
     The linked recipient action result feedback signal  134  represents data that correlates to an activity of a user of the device  118  that is linked to a response  115  provided to the device  118  by the response selection module  101 . In at least one embodiment, an external data source(s)  113  receive the feedback signal  134 , and, in at least one embodiment, the response selection module  101  receives and stores the feedback signal  134 . The source (not shown) of the feedback signal  134  can be any device that can transmit data related to the user of device  118 . For example, the source can be an electronic device of a sales person that has information that the user of device  118  performed an activity related to the response  115  and provides this information. The source can be an application that allows the user to communicate the feedback signal  134  directly, such as a direct reply to machine learning system  100  or other action that correlates the response  115  to an activity of the user within a window of time that allows the response selection module  101  to infer that the user&#39;s action was linked to the response  115 . 
     The feedback signal  134  enables the response selection module  101  to learn and improve performance by correlating particular actions with a response  115 . As subsequently described in more detail, when the response selection module  101  correlates the action and responses, in at least one embodiment, the response selection module  101  adjusts template selection ranking criteria accordingly. By adjusting the ranking criteria, the response selection module  101  can improve ranking and selection of response templates. 
       FIG. 2  depicts an exemplary machine learning system  200  that includes learning, ranked response selection module  202 , which represents one embodiment of the response selection module  101 . As previously discussed in conjunction with the response selection module  101 , the response selection module  202  utilizes the data from data sources  203  to select and rank response templates from response template library  208 . The structured data input  204  represents structured data input from any source, such as NLP system  100  and device(s)  114 . 
       FIG. 3  depicts an exemplary response ranking and selection process  300 . In at least one embodiment, the machine learning system  200  operates in accordance with the process  300 . Referring to  FIGS. 2 and 3 , in operation  302  the response selection module  202  receives the structured data input  204 , which is constructed in accordance with a natural language processing output schema. 
       FIGS. 4 and 5  depict exemplary structured data input  400 , which is the response prediction input data from the &#39;151 application. The structured data input  400  includes an incoming message  404  that asks, “How many bedrooms on the first floor?” The contextual data  406  is structured using a natural language processing output schema that includes various parameters such as intents path data  402 , i.e. intents path=property details, rooms, bedrooms, room, count. Normalized entities  408  with entity names “listing” and “which,” and associated entity values of “tx_actris_1234567” and “first floor,” respectively. 
     Referring to  FIGS. 2-5 , operation  302  receives the structured data input  204 . The structured data input  204  includes a statement to which the response selection module  202  selects one or more responses. The incoming message  404 , “How many bedrooms on the first floor?” represents one example of a statement contained in the structured data input  204 . Operation  304  derives template filter criteria utilizing the structured data input  204 . The structured data input  204  has a structured natural language processing output schema that allows the response selection module  302  to identify specific parameters within the structured data input  204  from which the response selection module  202  formulates filtering criteria to identify candidate response templates from the response template library  208  that are responsive to the statement. To effectively function as filtering criteria, the response templates  208  include corresponding parameters and values that can be queried against to identify the candidate response templates. 
       FIG. 6  depicts an exemplary response template data structure  602  and response template library  604 . The response template  602  is a data structure that includes filter parameters  606 . The depicted filter parameters  606  are illustrative and can be replaced with any filter parameters that allow the response selection module  202  to identify candidate response templates. The filter parameters  602  include &lt;INTENT PATH&gt;, &lt;ENTITY_NAME(S)&gt;, and &lt;ENTITY_VALUES&gt; that correspond directly to the same type of parameters in data  402  and  408  in the structured data input  204 . The filter parameters  606  can include additional parameters and values that may not correspond to parameters in structured data input  204 , such as user profile and group profile data in the contextual data  112 , insight parameters in the insight/other data  116 , and external data in external data source(s)  113 . For example, the structured data input  204  may include specific contextual data about the requestor, such as the requestor&#39;s age, preferred commute time range, employment location, pets, children, desired spending range, and desired lot, house size, and number of bathrooms in a home buying context. The external data source(s)  113  can include additional information such as details about a home that can be accessed using a multiple listing service (MLS) number, map distances, school districts, school ratings, etc. 
     The insight/other data  116  can overlap with other data sources but can also include additional information that, for example, may be derived from data from other data sources. Such additional filter parameters are illustratively represented by the &lt;EXTERNAL DATA&gt; and &lt;PROFILE DATA&gt; filter parameters. Accordingly, the filter parameters  602  in the response template can be structured to be responsive to inquiries based on such additional parameters and values. The filter parameters  606  can also include disqualifying data that prevents a response template from being selected. An example disqualifier is if the dynamic content  608  includes mention of an object, such as a pool, and the response should not have such contact because, for example, inclusion of the object in the response  115  could be, for example, misleading. 
     The response template  602  also includes content data structure  608 . The content represents that actual content that can be provided to a device  118 . The content can include static and/or dynamic content fields. Dynamic content refers to content that the response selection module  202  populates with data, and the populated data can change depending on, for example, parameters of the structured data input  204 . For example, the {ANSWER} field can represent dynamic content such as insertion of a particular address when responding to a real estate related statement. Each [OBJECT] can refer to, for example, static content, such as introductory or concluding phrases. Additionally, the content can include any type of content, such as text, photo, video, and hyperlinks. The response template  602  also includes an identifier (ID) to uniquely identify each response template. 
     The response template library  604  serves as a storage repository for the response templates N+1 number of response templates, where N is an integer. In at least one embodiment the response template library  604  is stored in a database and is accessible using database queries, such as structured query language (SQL) queries. 
     Referring to  FIGS. 2-6 , in at least one embodiment, the response selection module  202  includes a template filter  210  that performs operation  304  to derive the template filter criteria by formulating a query to the response template library  604  to identify response templates having specific parameters and values contained in the structured data input  204 . For example, operation  304  utilizes the intent path data  402  of intents path: “property details, rooms, bedrooms, room, count” and the presence of a “listing” entity name to formulate a filter criteria query of select intent_path templates where intent_path=’ property details, rooms, bedrooms, room, count” and required entity_names=“listing.” Operation  306  applies the filter criteria to the template library  604  to identify and select response templates that meet the filter criteria. 
       FIGS. 7-8  depict selected candidate response templates having ID&#39;s 1-5. Each of the candidate response templates meets the filter criteria for ‘property details, rooms, bedrooms, room, count” and required entity_names=“listing. In at least one embodiment, the candidate response templates have additional qualifiers such as a status code of “ok,” which indicates the response templates are available for use and a qualifier that response templates are not for multi-answers. Each candidate response template 1-5 also indicates that specific entity_values are not required to select the response template. Each response template includes static and dynamic content. For example, response template id=1, has content of ““The home at {{{message.fulfillment.normalized_entities.Listing.string}}} has {{#message.fulfillment.answer}}{{{value}}} bedroom{{# is_plural}}s{{/is_plural}}{Umessage.fulfillment.answer}}.” The bracketed fields enable entry of dynamic content that is populated with values from the data sources  203 . For example, from structured data input  400 , the content in template 1 is “The home at 61204 Mesa has 3 bedrooms,” the content in template 2 is “There are 3 bedrooms in the home at 61204 Mesa.” The content can also include dynamic insights if available that are populated from data sources  203 , such as “The home at 61204 Mesa has 3 bedrooms and a pool,” if an insight into the requestor indicates the requestor was previously interested in properties with a pool. 
     A ranking engine  212  performs operation  308  and ranks the candidate response templates using ranking criteria. By ranking the candidate templates, the ranking engine  212  allows the response selection module  202  to select the candidate response template with an estimated highest chance of causing the recipient to engage in an activity considered successful, such as replying to the response or taking certain action, like scheduling a showing of a home for purchase. The particular ranking criteria is a matter of design choice. Exemplary ranking criteria is set forth below:
         for each candidate response template, determining the conversion rate in accordance with equation [1]:       

     
       
         
           
             
               
                 
                   
                     Conversion 
                      
                     
                         
                     
                      
                     Rate 
                   
                   = 
                   
                     
                       Success 
                        
                       
                           
                       
                        
                       Count 
                     
                     
                       Attempt 
                        
                       
                           
                       
                        
                       Count 
                     
                   
                 
               
               
                 
                   [ 
                   1 
                   ] 
                 
               
             
             
               
                 
                   
                     Success 
                      
                     
                         
                     
                      
                     Count 
                   
                   = 
                   
                     
                       ∑ 
                       a 
                     
                      
                     
                       ( 
                       
                         
                           S 
                            
                           
                             C 
                             User 
                           
                         
                         + 
                         
                           S 
                            
                           
                             C 
                             Group 
                           
                         
                         + 
                         
                           SC 
                           All 
                         
                       
                       ) 
                     
                   
                 
               
               
                 
                     
                 
               
             
             
               
                 
                   
                     Attempt 
                      
                     
                         
                     
                      
                     Count 
                   
                   = 
                   
                     
                       ∑ 
                       a 
                     
                      
                     
                       ( 
                       
                         
                           I 
                           User 
                         
                         + 
                         
                           I 
                           Group 
                         
                         + 
                         
                           I 
                           All 
                         
                       
                       ) 
                     
                   
                 
               
               
                 
                     
                 
               
             
             
               
                 
                   
                     S 
                      
                     
                       C 
                       User 
                     
                   
                   = 
                   
                     
                       w 
                       a 
                     
                      
                     
                       w 
                       U 
                     
                      
                     
                        
                       
                         
                           conversions 
                           
                             a 
                             , 
                             U 
                           
                         
                          
                         
                           ( 
                           ∅ 
                           ) 
                         
                       
                        
                     
                   
                 
               
               
                 
                     
                 
               
             
             
               
                 
                   
                     S 
                      
                     
                       C 
                       Group 
                     
                   
                   = 
                   
                     
                       w 
                       a 
                     
                      
                     
                       w 
                       G 
                     
                      
                     
                       
                         ∑ 
                         
                           k 
                           ∈ 
                           x 
                         
                       
                        
                       
                          
                         
                           
                             conversions 
                             
                               a 
                               , 
                               k 
                             
                           
                            
                           
                             ( 
                             ∅ 
                             ) 
                           
                         
                          
                       
                     
                   
                 
               
               
                 
                     
                 
               
             
             
               
                 
                   
                     S 
                      
                     
                       C 
                       All 
                     
                   
                   = 
                   
                     
                       w 
                       a 
                     
                      
                     
                       w 
                       A 
                     
                      
                     
                        
                       
                         
                           conversions 
                           
                             a 
                             , 
                             A 
                           
                         
                          
                         
                           ( 
                           ∅ 
                           ) 
                         
                       
                        
                     
                   
                 
               
               
                 
                     
                 
               
             
             
               
                 
                   
                     Attempt 
                      
                     
                         
                     
                      
                     Count 
                   
                   = 
                   
                     
                       ∑ 
                       a 
                     
                      
                     
                       ( 
                       
                         
                           I 
                           User 
                         
                         + 
                         
                           I 
                           Group 
                         
                         + 
                         
                           I 
                           
                             A 
                              
                             l 
                              
                             l 
                           
                         
                       
                       ) 
                     
                   
                 
               
               
                 
                     
                 
               
             
             
               
                 
                   
                     I 
                     User 
                   
                   = 
                   
                     
                       w 
                       a 
                     
                      
                     
                       w 
                       U 
                     
                      
                     
                        
                       
                         
                           impressions 
                           
                             a 
                             , 
                             U 
                           
                         
                          
                         
                           ( 
                           ∅ 
                           ) 
                         
                       
                        
                     
                   
                 
               
               
                 
                     
                 
               
             
             
               
                 
                   
                     I 
                     Group 
                   
                   = 
                   
                     
                       w 
                       a 
                     
                      
                     
                       w 
                       G 
                     
                      
                     
                       
                         ∑ 
                         
                           k 
                           ∈ 
                           x 
                         
                       
                        
                       
                          
                         
                           
                             impressions 
                             
                               a 
                               , 
                               k 
                             
                           
                            
                           
                             ( 
                             ∅ 
                             ) 
                           
                         
                          
                       
                     
                   
                 
               
               
                 
                     
                 
               
             
             
               
                 
                   
                     I 
                     All 
                   
                   = 
                   
                     
                       w 
                       a 
                     
                      
                     
                       w 
                       A 
                     
                      
                     
                        
                       
                         
                           impressions 
                           
                             a 
                             , 
                             A 
                           
                         
                          
                         
                           ( 
                           ∅ 
                           ) 
                         
                       
                        
                     
                   
                 
               
               
                 
                     
                 
               
             
           
         
       
         
         
           
             wherein: 
             w a  is an activity weight; 
             w U  is a user weight of a user U associated with the structured data input, w G  is a group weight of a group G that includes the user and is a subset of all users of the supervised learning processing system, and w A  is a weight for all users A of the supervised learning processing system; 
             conversion is a conversion associated with the response template for an activity a by a user the user U, the group G, or all users A; and 
             impression is an impression associated with the response template for an activity a by a user the user U, the group G, or all users A. 
           
         
       
    
     In summary, the foregoing ranking criteria determines a conversion rate that is based on conversions and impressions by a user, a group with similar attributes as the user, and all users. Successes for the user are more indicative of future success than successes by the group and all users, and successes for the group are more indicative of future success than successes by the all users. Furthermore, different activities are considered more valuable than others as previously described. By determining the weighted conversions and activities relative to the total number of impressions, the ranking engine  212  determines a conversion rate that provides a measure of performance for each candidate response template. The ranking engine  212  revises the number of impressions and from feedback signal  134  adjusts the conversion data. Furthermore, the weights can be adjusted to further enhance the learning of the ranking engine  212 . 
     In at least one embodiment, the conversion rate is a function of the success values and the attempt values, i.e. conversion rate=f(success, attempt), and not strictly (success/attempt). For example, the conversion rate function can incorporate distributions, such as beta distributions parameterized by success and attempt values, of conversion rates and ranking of the candidate response template can be based on a random sampling of the conversion rate values in the distribution. Utilizing this ‘distribution’ based conversion rate function allows a probability of assigning a higher conversion rate based ranking to candidate response templates that do not have the strictly highest (success/attempt) value. In at least one embodiment, the particular distributions are mathematically shaped to provide a probably frequency of ranking a particular candidate response template with the highest conversion rate. By allowing the ranking engine  212  to distribute the highest conversion rate ranking among candidate response templates, the response selection module  202  learns the effectiveness of different response templates. Additionally, the response selection module  202  can insert new response templates into the collection of candidate response templates to allow the response selection module  202  to learn about the success of the experimental response templates. When adding new response templates that do not have observed conversion and impression values, the conversion rate function can be modified to ensure any number of highest conversion rate values for each candidate response template by, for example, inserting an override factor that forces a high conversion rate of the new candidate response template. The override factor can be, for example, a weight selected from a distribution of weights or a random number that ensures selection of the new candidate response template at some probabilistic frequency. In at least one embodiment, the override factor is determined by an epsilon-greedy function that forces occasional highest ranking and selection of the new candidate response template. 
       FIGS. 9-12  depict exemplary response template ranking parameters and results. Cells G1, M1, and S1 are values of respective weights w U , w G , and w A . Each weight from the user to group to all is an order of magnitude larger, which represents the relative importance of the conversion feedback. Cells E2, K2, and Q2 represent weight w a =1 for a user activity of “Got Response,” which is receiving a reply to the response derived from the particular response template. Cells G2, M2, and S2 represent weight w a =4 for a user activity of “Showing” of a property because showing is more valuable than receiving a response. The ranking results include “Prior” and “Observed” results. “Prior” is inserted as a value that represents estimated performance to initialize the ranking data so that the conversion rates are more meaningful. “Prior” and “Observed” values are added together for purposes of values in Equation [1]. The weighted conversions, weighted impressions, and overall conversion rates are computed in accordance with Equation [1]. 
     The response selection module  202  includes a populating engine  214  that in operation  314  populates any dynamic fields in the candidate response templates and converts the populated content into a response message ready for sending. Population can occur before or after selection of the response template. Operation  312  determines whether the response selection module  202  provides the ranked candidate templates to a human artificial intelligence technician (AIT)  218  to allow the AIT  218  to select the response to send to device  118  or allow the response selector  216  to directly select and send the response to the device  118 . The function to determine the outcome of operation  312  is a matter of design choice. In at least one embodiment, the function of operation  312  relies on a confidence level assigned to the response selection module  202  for specific structured data input and responses. If operation  312  selects the AIT  218 , the response selection module  202  provides the candidate response templates to the AIT  218  with ranking information for selection. Operation  318  provides a response derived from the selected response template to the device  118 . In at least one embodiment, the derived response is the populated content from the selected response template. The selection made by the AIT  218  is fed back to the ranking engine  212  to allow the ranking engine to revise the impression counts for the selected response template and to correlate any feedback signal  134  to the selected response template. If operation  312  does not select the AIT  218  to select the template and send the response, in operation  322  the response selector  216  selects the response template in accordance with selection criteria and provides the response template content to the device  118 . The selection criteria is a matter of design choice. In at least one embodiment, the selection criteria selects the template with the highest conversion rate. However, such criteria can quickly eliminate candidate templates from future selection. So, other selection criteria utilizes a distribution of the conversion rates of the candidate templates to proportionately select candidate templates in accordance with their conversion rates. Additionally, the selection criteria can use an override to select new candidate templates or to force selection of particular candidate templates. Thus, the response selection module  202  is able to continue learning which candidate template is best on a user, group, and all users basis. 
       FIG. 13  depicts an electronic environment  1300  in which certain embodiments of the disclosed system may operate. In this example, a plurality of users execute client applications  1302 ,  1304 , and  1306  at respective devices  1308 ,  1310 , and  1310 . In certain embodiments, the client applications  1302 ,  1304 , and  1306  are the same type of application, such as a real-estate application. In certain embodiments, the client applications  1302 ,  1304 , and  1306  include web browser clients and text messaging applications. 
     In at least one embodiment, the client applications receive template communications  1324  from a template selection system  1322  over a network  1314 , such as the Internet. In certain embodiments, the template selection system  1322  selects templates that are to be sent to the users in the manner described herein. 
     In at least one embodiment, the users employ the client applications to send responses  1326  to a response analysis system  1328 . In certain embodiments, the responses  1326  are in the form of messages, such as textual messages, that may be analyzed by an NLP machine language model of the response analysis system  1328 . Users provide responses  1326  under various circumstances. For example, users may provide textual answers to questions that are posed to the user in a template. As another example, users may provide textual inquiries, requests, or comments that, although not explicitly related to a response to a particular template, may be indicative of the momentum associated with a particular template. In certain embodiments, the response analysis system  1322  analyzes the text of responses received from the client applications to assign momentum scores to the templates associated with the responses. In certain embodiments, the responses are associated with the templates in time (e.g., received within a time period after the transmission of the template to the user), in sequence (e.g., received in a particular sequence of responses after transmission of the template to the user), or in subject matter (e.g., pertaining to the same subject matter as the template transmitted to the user) to the template selection system  1322 . In certain embodiments, the momentum scores are stored along with the templates in a template database store  1324 . 
       FIG. 14  depicts a response analysis system  1400  that may be implemented in certain embodiments of the disclosed system. In the example shown in  FIG. 14 , responses  1402  are provided to a response API  1404 , which provides the text from the responses  1402  to the input of a machine learning model  1406 . In certain embodiments, the machine language model  1406  has been trained using training data  1408 . The training data  1408  includes text messages that have been labeled to train the machine learning model  1406  to assign momentum classifications to the responses  1402 . In certain embodiments, the momentum classifications are numerical momentum scores, which can be used as metrics to calculate conversion rates for the templates. 
     In the illustrated embodiment, the machine language model  1406  outputs momentum classifications  1410  for each of the responses  1402 . In certain embodiments, the machine language model  1406  also provides a confidence score for each of the responses  1402 . The confidence score is generated by the machine language model  1406  and corresponds to the accuracy with which the machine language model  1406  has determined that the corresponding momentum classification is correct. In certain embodiments, a classification aggregator  1412  is employed to aggregate the response classifications associated with each of the templates to generate a momentum score for each template so that the momentum classification assigned to a template takes multiple responses into consideration when determining an overall momentum classification for the template. The classification aggregator  1412  generates the overall momentum classification for the template and correlates the momentum classification with the template for storage in the templates and scores storage  1414 . Certain embodiments retrieve the momentum classification for the template to determine the conversion rate that is assigned to the template during template ranking operations. 
       FIG. 15  depicts a flowchart  1500  showing exemplary operations that may be executed in certain embodiments of the disclosed system. In this example, text responses are received at operation  1502  and assigned a momentum score at operation  1504 . In certain embodiments, the momentum score of the text response is correlated with the corresponding template at operation  1506 . In certain embodiments, the text responses are correlated with the corresponding templates prior to classifying the momentum scores of the text responses. A determination is made at operation  1508  as to whether more text responses are to be classified in correlated with the templates corresponding to the text responses. If no further text responses are to be classified, the momentum scores for the text responses for each template are aggregated at operation  1510  to provide a single momentum score for each template. At operation  1512 , certain embodiments use the aggregated momentum score for each template in determining the overall template conversion score assigned to the template during template ranking operations. 
       FIG. 16  is a table  1600  showing exemplary text messages and corresponding momentum scores that may be classified by a machine learning model. In this example, the momentum scores range from 1 through 5, with a score of 5 representing the highest momentum. In this example, text messages of the type shown at  1602  are classified by the machine learning model with a low momentum score of 1. In certain embodiments, a momentum score of 1 indicates that the user wants to stop using the SERVICE. Text messages of the type shown at  1604  are classified by the machine learning model with a momentum score of 2. In certain embodiments, a momentum score of 2 indicates that the user is having a problematic experience with the SERVICE. Text messages of the type shown at  1606  are classified by the machine learning model with a momentum score of 3. In certain embodiments, a momentum score of 3 corresponds to a neutral or irrelevant comment relating to the SERVICE. Text messages of the type shown at  1608  are classified by the machine learning model with a momentum score of 4. In certain embodiments, a momentum score of 4 indicates that the user is having a productive experience with the SERVICE. Text messages of the type shown at  1610  are classified by the machine learning model with a momentum score of 5. In certain embodiments, a momentum score of 5 indicates that the user is moving toward a major milestone or the end of the user&#39;s journey with the SERVICE. 
     Score components for the templates may be defined as a percentage of users in the values range from 0 (nobody) to 1 (all users) associated with a particular response criterion multiplied by a corresponding weighting factor. In certain embodiments, the score components for the templates are added together to arrive at an overall template conversion score, which may be used, for example, during template ranking operations. As an example, the template conversion score for a system implementing a response analysis system may be determined using the equation: 
       Template Conversion Score=( C   1   *w   1 )+( C   2   *w   2 )+( C   3   *w   3 )+ . . . ( C   n   *w   n ) 
     where
 
C x  is a value between 0 and 1 defined as a percentage of users whose actions meet particular response criterion associated with the template; and
 
w x  is the weighting factor that is assigned to the response criterion.
 
       FIG. 17  depicts a table  1700  showing exemplary response criterion and corresponding weighting factors that may be used in certain embodiments of the disclosed system. In this example, the values for the percentage of users G meeting the stated response criterion are shown in the RESPONSE CRITERION column. In certain embodiments, the values for G in the RESPONSE CRITERION will range from 0 through 1. Examples of weighting factor values w x  that may be used with the values G of the RESPONSE CRITERION are shown in the WEIGHTING FACTOR. As shown in this example, the momentum score is taken into consideration in row  1702  and corresponds to the value for the percentage of users that respond with the message of momentum 4 or 5 within twelve hours of transmission of the template. In this example, the response criterion has a weighting factor of 0.66, that when multiplied by the value for the percentage of users that respond with the message of momentum 4 or 5 within twelve hours of transmission of the template result in a sub-score SS2. A total conversion factor for the template may be obtained by adding the values of SS1 through SS9 together. 
       FIG. 18  is a block diagram illustrating an exemplary network environment in which the machine learning system that includes a learning ranked response selection module response selection module and response analysis system may be practiced. Network  1802  (e.g., a private wide area network (WAN) or the Internet) includes a number of networked computer systems  1804  ( 1 )-(N) that can each implement a specialized machine learning system and the learning, ranked response selection module response selection module. Communication between user requestor devices  1806  ( 1 )-(N) and each specialized machine learning system and the learning, ranked response selection module response selection module  1804  ( 1 )-(N) typically occurs over a network, such as a public switched telephone network or cable network of asynchronous digital subscriber line (ADSL) channels or high-bandwidth trunks, for example, communications channels providing T1 or OC3 service. Requestor devices  1806  ( 1 )-(N) typically access server computer systems  1804  ( 1 )-(N) through a service provider, such as an internet service provider (“ISP”) by executing application specific software, commonly referred to as a browser, on one of requestor devices  1806 ( 1 )-(N). 
     Requestor device  1806 ( 1 )-(N) and/or specialized machine learning systems and the learning, ranked response selection module response selection modules  1804 ( 1 )-(N) may include, for example, computer systems of any appropriate design, including a mainframe, a mini-computer, a personal computer system including notebook computers, a wireless, mobile computing device (including personal digital assistants, smart phones, and tablet computers). These computer systems are typically information handling systems, which are designed to provide computing power to one or more users, either locally or remotely. Such a computer system may also include one or a plurality of input/output (“I/O”) devices coupled to the system processor to perform specialized functions. Tangible, non-transitory memories (also referred to as “storage devices”) such as hard disks, compact disk (“CD”) drives, digital versatile disk (“DVD”) drives, and magneto-optical drives may also be provided, either as an integrated or peripheral device. In at least one embodiment, the machine learning system and the learning, ranked response selection module response selection module can be implemented using code stored in a tangible, non-transient computer readable medium and executed by one or more processors. In at least one embodiment, the machine learning system and the learning, ranked response selection module response selection module can be implemented completely in hardware using, for example, logic circuits and other circuits including field programmable gate arrays. 
     Embodiments of individual machine learning systems  1804 ( 1 )-(N) can be implemented on a computer system such as computer  1900  illustrated in  FIG. 19 . The computer  1900  can be a dedicated computer system or a virtual, emulated system located in, for example, a cloud computing environment. Input user device(s)  1910 , such as a keyboard and/or mouse, are coupled to a bi-directional system bus  1918 . The input user device(s)  1910  are for introducing user input to the computer system and communicating that user input to processor  1913 . The computer system of  FIG. 19  generally also includes a non-transitory video memory  1914 , non-transitory main memory  1915 , and non-transitory mass storage  1909 , all coupled to bi-directional system bus  1918  along with input user device(s)  1910  and processor  1913 . The mass storage  1909  may include both fixed and removable media, such as a hard drive, one or more CDs or DVDs, solid-state memory including flash memory, and other available mass storage technology. Bus  1918  may contain, for example, 32 of 64 address lines for addressing video memory  1914  or main memory  1915 . The system bus  1918  also includes, for example, an n-bit data bus for transferring DATA between and among the components, such as CPU  1913 , main memory  1915 , video memory  1914  and mass storage  1909 , where “n” is, for example, 32 or 64. Alternatively, multiplex data/address lines may be used instead of separate data and address lines. 
     I/O device(s)  1919  may provide connections to peripheral devices, such as a printer, and may also provide a direct connection to remote server computer systems via a telephone link or to the Internet via an ISP. I/O device(s)  1919  may also include a network interface device to provide a direct connection to remote server computer systems via a direct network link to the Internet via a POP (point of presence). Such connection may be made using, for example, wireless techniques, including digital cellular telephone connection, Cellular Digital Packet Data (CDPD) connection, digital satellite data connection or the like. Examples of I/O devices include modems, sound and video devices, and specialized communication devices such as the aforementioned network interface. 
     Computer programs and data are generally stored as instructions and data in a non-transient computer readable medium such as a flash memory, optical memory, magnetic memory, compact disks, digital versatile disks, and any other type of memory. The computer program is loaded from a memory, such as mass storage  1909 , into main memory  1915  for execution. Computer programs may also be in the form of electronic signals modulated in accordance with the computer program and data communication technology when transferred via a network. In at least one embodiment, Java applets or any other technology is used with web pages to allow a user of a web browser to make and submit selections and allow a client computer system to capture the user selection and submit the selection data to a server computer system. 
     The processor  1913 , in one embodiment, is a microprocessor manufactured by Motorola Inc. of Illinois, Intel Corporation of California, or Advanced Micro Devices of California. However, any other suitable single or multiple microprocessors or microcomputers may be utilized. Main memory  1915  is comprised of dynamic random access memory (DRAM). Video memory  1914  is a dual-ported video random access memory. One port of the video memory  1914  is coupled to video amplifier  1916 . The video amplifier  1916  is used to drive the display  1917 . Video amplifier  1916  is well known in the art and may be implemented by any suitable means. This circuitry converts pixel DATA stored in video memory  1914  to a raster signal suitable for use by display  1917 . Display  1917  is a type of monitor suitable for displaying graphic images. The computer system described above is for purposes of example only. 
     Although embodiments have been described in detail, it should be understood that various changes, substitutions, and alterations can be made hereto without departing from the spirit and scope of the invention as defined by the appended claims.