Patent Publication Number: US-11640622-B2

Title: Preservation of scores of the quality of traffic to network sites across clients and over time

Description:
RELATED APPLICATIONS 
     The present application is a continuation of U.S. patent application Ser. No. 14/325,093, entitled “PRESERVATION OF SCORES OF THE QUALITY OF TRAFFIC TO NETWORK SITES ACROSS CLIENTS AND OVER TIME” and filed Jul. 7, 2013, which is a continuation of U.S. Pat. No. 8,775,257 filed Nov. 5, 2010 and issued on Jul. 8, 2014, which claims priority to International Application PCT/US2009/042883 filed May 5, 2009, which claims priority to U.S. Provisional Patent Application No. 61/050,565 filed May 5, 2008. Each of these applications is hereby incorporated by reference in its entirety for all purposes. 
    
    
     TECHNICAL FIELD 
     The disclosed technology relates to assessing the value of traffic associated with network sites. 
     BACKGROUND 
     An increasing number of companies, agencies, individuals, and other parties (collectively “advertisers”) use online advertising to advertise to users of Internet or other network sites or services. An advertiser purchases advertising space from an individual publisher or from an advertising network that distributes advertisements to one or more publishers. A publisher or advertising network may charge the advertiser using one of several methods, including cost-per-click and cost-per-impression. In a cost-per-click system, an advertiser is charged based on the number of times that agents click on its advertisement. An advertiser is not charged when a publisher displays an advertisement to an agent unless the agent clicks on the advertisement. In a cost-per-impression system, an advertiser is charged based on the number of times a publisher displays its advertisement to an agent. 
     Click fraud, or fraudulent clicks on advertisements, is an issue that concerns advertisers and publishers who use cost-per-click and other payment models. Similarly, impression fraud, or displays of advertisements in situations where the advertisements will not make an impression on a human user, is an issue that concerns advertisers and publishers who use cost-per-impression and other payment models. Click or impression fraud can take a number of forms, including clicks on an advertisement by or displays of an advertisement to competitors, web robots, or users with personal or political agendas. In addition, an adware or clickware virus may install itself on a computer and generate clicks on or impressions of advertisements without the computer user&#39;s knowledge. Fraudulent clicks or impressions do not generate revenue or other value for an advertiser; however, the advertiser must pay for the clicks or impressions. Click or impression fraud therefore harms the advertiser by increasing advertising expense, and at the same time harms the publisher by lowering the perceived value of traffic the advertiser receives from the publisher. The need therefore exists for a system that overcomes the above limitations, in addition to providing other benefits. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         FIG.  1    is a block flow diagram of a process that may be used to preserve scores of the quality of traffic to network sites according to one or more embodiments. 
         FIG.  2    is a block flow diagram of a process that may be used to preserve scores of the quality of traffic when performing updates for the same client according to one or more embodiments. 
         FIG.  3    is a table of example data that illustrates how scores may be preserved over time for a given client according to one or more embodiments. 
         FIG.  4    is a block diagram of a representative facility for scoring the quality of network traffic and an environment in which the facility operates. 
         FIG.  5    is a flow diagram of a method of computing the quality of network traffic. 
         FIG.  6    is a flow diagram of a method of computing a correlation between a rule set that is used to assess the quality of traffic and a desired agent action. 
         FIG.  7    is a block diagram of a data structure used to compute the correlation between each rule in the rule set used to assess the quality of traffic to a network site and a desired agent action. 
         FIG.  8    is a flow diagram of a method of scoring an agent action based on a rule set. 
         FIG.  9    is a flow diagram of a method of generating training set data. 
         FIG.  10    is a flow diagram of a method of identifying correlated parameters that characterize traffic associated with network sites. 
         FIG.  11    is a block diagram of a data structure used to identify correlated parameters that characterize traffic associated with network sites, the data structure depicted prior to processing data characterizing the traffic. 
         FIG.  12    is a block diagram of a data structure used to identify correlated parameters that characterize traffic associated with network sites, the data structure depicted after processing data characterizing the traffic. 
     
    
    
     DETAILED DESCRIPTION 
     In order to improve the quality of the traffic that is sent to or received by a network site, a method and/or system for scoring the quality of traffic to network sites may be used. The quality of traffic for a network site may be determined based on a variety of factors, including the amount of click or impression fraud, whether valuable actions (e.g., purchases) are generated, characteristics of the advertiser and/or publisher, and other factors. For example, a method and system for generating non-binary scores of traffic to network sites is described in commonly owned PCT Patent Application Serial No. US07/64454, entitled Scoring Quality of Traffic to Network Sites Using Interrelated Traffic Parameters, filed Mar. 20, 2007, the substance of which is included herein as Appendix A, below. 
     The method and/or system for scoring the quality of traffic to network sites, such as that described in Appendix A, may extract session data, or information identifying an agent&#39;s interactions with a server, from one or more server logs or other data sources obtained from a publisher, advertiser, or third party. In addition, supplemental data may be obtained from external data sources to assist in interpreting the agent&#39;s interactions with the server. A session may be defined as one or more entries in the server log or other data source indicative of an agent&#39;s interaction with a network site. 
     The method and/or system may apply a multi-factor analysis, in the form of a rule set, to the session data. Each rule in the rule set is an expression that receives as input one or more parameters associated with an agent&#39;s session. When the rule is applied to the input parameters, it produces a result that reflects the value of an agent&#39;s actions associated with that agent&#39;s session. Within each rule set, each rule may be weighted differently, such as based on how accurately it predicts desirable agent actions, in order to generate an optimum combination of rules. 
     A result vector is a combination of all rule results for a particular session. The method and system may generate an association table, which has a plurality of rows, each row representing a unique result vector (i.e., combination of rule results). The result vector for each session is mapped to the association table, and additional information, such as whether a transaction associated with the session was fraudulent or non-fraudulent, is recorded. 
     The analysis of the session data identifies agent actions that are desirable to a publisher, advertiser, or third party. Agent actions that are desirable to a publisher, advertiser, or third party include any activity that generates value for the publisher, advertiser, or third party, such as a click, a conversion (e.g., purchase), a submission of a form, bookmarking of the site, a rollover event, an impression, or other activity by the user. The odds of conversion may be defined as the probability that an agent interaction with a network site will result in a desirable agent action. 
     The method and system generates a relative, raw score for each agent action or for an aggregate number of agent actions based on whether the agent action is desired by the publisher, advertiser, or third party. The raw score may be scaled to place it in a form that may be more readily understood and used by advertisers and publishers. For example, a score may be scaled to fall within a standard range, such as a range from 300 to 800. The score may be used to assess the quality of the traffic received by a network site. A lower score is indicative of fraudulent, likely fraudulent, or otherwise non-productive traffic having lithe value, whereas a higher score is indicative of traffic having desirable characteristics and therefore greater value. Further discussion of each of these concepts may be found in Appendix A. 
     Score Preservation 
     A method and system for preserving scores of the quality of traffic to network sites, so that the scores are consistent over time according to one or more embodiments and are comparable across clients according to one or more embodiments, is described. A score may be generated for a client (i.e., a party interested in receiving traffic scores) a single time, or a score may be generated for a client many times over a given time period. For example, a score may be generated for a client on a periodic basis (e.g., once a week, once a month, four times a year), at the client&#39;s request, or when there are changes in data used to assess the quality of traffic. In addition, scores may be generated for multiple clients, whether members of the same or different populations (e.g., industry segments). When multiple scores are generated for a single client or across many clients, it is desirable to preserve the consistency of the scores. Preserving the consistency of scores allows a single client to compare performance over time, regardless of changes to the volume of traffic or changes to the methodology used to analyze the traffic. Preserving the consistency of scores also allows multiple clients to compare performance, even if the clients are in different industries to attract different types of traffic. 
     Once a score has been generated to measure the quality of traffic to a network site, such as in the manner described in Appendix A, it is desirable to preserve the consistency of the score over time. Preserving the consistency of a score means that if the quality of traffic associated with a client&#39;s site is approximately the same during each scoring period, the score generated for each scoring period should also be approximately the same. Without a score preservation technique, scores for different scoring periods may vary dramatically, even if the quality of traffic associated with the site remains the same. For example, scores may vary for different scoring periods when different rules are applied to the traffic, when different traffic data is provided to the scoring system, when the methodology of the scoring system is improved, and/or in other circumstances. Preserving the consistency of a score also means that if the quality of traffic associated with a client&#39;s site increases or decreases from one scoring period to the next, the score preservation technique should properly reflect the increase or decrease in traffic quality over time. That is, a decrease in a client&#39;s score should indicate that traffic quality has decreased, not simply that new rules have been discovered to better detect click fraud. Among other benefits, preserving the consistency of scores over time facilitates an accurate representation of trends in traffic quality for a given client. 
     In addition, it is desirable to maintain the consistency of scores across various clients to allow the quality of traffic to be compared across clients. For example, a similar score for two clients should indicate that the quality of traffic associated with the clients&#39; sites is similar. In contrast, a lower score for a first client as compared to a second client should indicate a lower quality of traffic for the first client than for the second. Without a score preservation technique, scores generated for various clients may differ significantly, even if the quality of traffic is similar. For example, each client may provide different traffic data to the scoring system, different rule sets may be applied to each client (i.e., based on industry segment), different rules may be triggered by each client, and other distinguishing factors may apply. Preserving the consistency of scores across clients allows scores to be compared among diverse clients. Scores may be compared across all clients as a whole, or scores may be compared across clients in a given population. For example, clients may be separated into different populations according to factors such as industry segment, site environment, type of traffic, and/or other factors. 
     The method and system for score preservation described herein may be used for a variety of benefits, including updating scores for a given client (such as during a scheduled periodic update, after adding new rules, and/or after receiving modified data fields from the client), producing scores that are comparable among clients, and using a reference data set to scale scores for a new client (e.g., a search engine) that does not have access to conversion data. Other benefits of the method and system will be apparent to one skilled in the art. 
     Concept 
     The method and system for preserving scores of the quality of traffic to network sites preserves core parameters in order to maintain score distribution. Core parameters are standard measures of an average score (such as a mean or median) and a variance in score (such as a spread or standard deviation). The methodology is herein illustrated using the following core parameters: odds 1 -score (denoted as u), which is an average score of the traffic based on the odds of conversion, and spread (denoted as v), which is the amount by which traffic scores are spread out, or dispersed. In other embodiments, the methodology may use other core parameters such as the mean (denoted as μ) and standard deviation (denoted as σ) of an average score. The methodology is similar in different embodiments, except that raw scores are scaled according to different algorithms in order to produce scaled scores. 
     Consider two data objects A and B from two different sources. For example, data object A may comprise combined advertiser data, while data object B may comprise search engine data. As another example, both data objects may be from the same client, but from two different time periods; that is, data object A may comprise September/October data associated with an advertiser, and data object B may comprise October/December data associated with the same advertiser. 
     In some embodiments, data object A comprises the following:
         A rule set R A  and rule weights used to generate result vectors.   An association table F A  of result vectors.   Known or estimated odds of conversion for each result vector.   Scaled score for each result vector.   Parameters u A  and u B , either known or estimated, such as using equations (1) and (2) below.       

     In some embodiments, data object B comprises the following:
         A rule set R B  overlapping with R A  and rule weights used to generate result vectors.   An association table F B  of result vectors.   Raw (un-scaled) score for each result vector.
 
Estimation of Core Parameters
       

     In some embodiments, core parameters u and v are set according to default values (e.g., u=650 and v=50). In other embodiments, core parameters u and v may be estimated according to equations (1) and (2): 
     
       
         
           
             
               
                 
                   
                     u 
                     ^ 
                   
                   = 
                   
                     
                       
                         
                           Σ 
                           + 
                         
                         ⁢ 
                         
                           n 
                           f 
                         
                         ⁢ 
                         
                           S 
                           f 
                         
                       
                       
                         
                           Σ 
                           + 
                         
                         ⁢ 
                         
                           n 
                           f 
                         
                       
                     
                     · 
                     
                       { 
                       
                         1 
                         - 
                         
                           
                             
                               Σ 
                               + 
                             
                             ⁢ 
                             
                               n 
                               f 
                             
                             ⁢ 
                             
                               L 
                               f 
                             
                             ⁢ 
                             
                               
                                 S 
                                 f 
                               
                               · 
                               
                                 Σ 
                                 + 
                               
                             
                             ⁢ 
                             
                               n 
                               f 
                             
                             ⁢ 
                             
                               L 
                               f 
                             
                           
                           
                             
                               Σ 
                               + 
                             
                             ⁢ 
                             
                               n 
                               f 
                             
                             ⁢ 
                             
                               
                                 L 
                                 f 
                                 2 
                               
                               · 
                               
                                 Σ 
                                 + 
                               
                             
                             ⁢ 
                             
                               n 
                               f 
                             
                             ⁢ 
                             
                               S 
                               f 
                             
                           
                         
                       
                       } 
                     
                     · 
                     
                       
                         { 
                         
                           1 
                           - 
                           
                             
                               
                                 Σ 
                                 + 
                               
                               ⁢ 
                               
                                 n 
                                 f 
                               
                               ⁢ 
                               
                                 L 
                                 f 
                               
                             
                             
                               
                                 Σ 
                                 + 
                               
                               ⁢ 
                               
                                 n 
                                 f 
                               
                               ⁢ 
                               
                                 
                                   L 
                                   f 
                                   2 
                                 
                                 · 
                                 
                                   Σ 
                                   + 
                                 
                               
                               ⁢ 
                               
                                 n 
                                 f 
                               
                             
                           
                         
                         } 
                       
                       
                         - 
                         1 
                       
                     
                   
                 
               
               
                 
                   ( 
                   1 
                   ) 
                 
               
             
             
               
                 
                   
                       
                   
                   ⁢ 
                   
                     
                       v 
                       ^ 
                     
                     = 
                     
                       
                         
                           
                             
                               Σ 
                               + 
                             
                             ⁢ 
                             
                               
                                 n 
                                 f 
                               
                               ⁡ 
                               
                                 ( 
                                 
                                   
                                     S 
                                     f 
                                   
                                   - 
                                   
                                     u 
                                     ^ 
                                   
                                 
                                 ) 
                               
                             
                           
                           
                             
                               Σ 
                               + 
                             
                             ⁢ 
                             
                               n 
                               f 
                             
                             ⁢ 
                             
                               L 
                               f 
                             
                           
                         
                         · 
                         ln 
                       
                       ⁢ 
                       
                           
                       
                       ⁢ 
                       2 
                     
                   
                 
               
               
                 
                   ( 
                   2 
                   ) 
                 
               
             
           
         
       
         
         where f is a result vector, n f  is a number of clicks per result vector, S f  is a scaled score for the result vector, and L f  is the logarithm of an odds of conversion Q f  for the result vector. In addition, Σ +  is equivalent to 
       
    
               ∑     f   ∈     Ω   +         ,         
where Ω is the set of all result vectors, and Ω +  is the subset of all result vectors simultaneously satisfying v f &gt;0 and v f &lt;1. If necessary, smaller result vectors with a similar score may be grouped together to reduce or eliminate the gap between Ω and Ω + .
 
     Equations (1) and (2) are derived from the system:
 
 Q   f =2 (S     f     −u)/v , for  f∈Ω   + 
 
This is a system with many equations (one for each f∈Ω + , unless the result vectors have been grouped into several bins), and only two unknowns. It can be solved as a weighted regression problem, where the weight for each f is n f . Taking the neperian logarithm on each side, and using the notation L f =log Q f , the system becomes:
 
 v·L   f =( S   f   −u )·ln 2, for  f∈Ω   + 
 
     By construction, Q f &gt;0 if f∈Ω + , and thus the logarithm L f  is always defined. The solution is given by equations (1) and (2) above. 
     Estimated Odds of Conversion 
     In some embodiments, the odds of conversion Q f  may be determined according to equation (3): 
                     Q   f     =         v   f         n   f     -     v   f         ·       {       Σ   ⁢           ⁢     v   g         Σ   ⁡     (       n   g     -     v   g       )         }       -   1                 (   3   )               
where each summation is computed over all result vectors g using historical data (such as data accumulated over a few weeks), summarized at the result vector level. Odds of conversion Q f =1 corresponds to an average result vector.
 
Scaled Score
 
     In some embodiments, the scaled score S f  may be determined according to equation (4): 
                     S   f     =     u   +     v   ·       ln   ⁢           ⁢     Q   f         ln   ⁢           ⁢   2                   (   4   )               
Note that when the odds of conversion are neutral (i.e., Q f =1), then S f =u. When the odds of conversion are reduced by a factor of 2, the scaled score S f  decreases by v points.
 
Methodology
 
       FIG.  1    is a flow diagram of a process  100  that may be used to preserve scores of the quality of traffic to network sites. At a block  105 , data object A and data object B (as described above) are received. As described above, data object A includes rule set R A  and data object B includes rule set R B . Once the data objects have been received, at a block  110 , the methodology determines the intersection R AB  of the two rule sets R A  and R B . That is, the subset of rules that apply to both data object A and data object B is determined. The intersection is represented by equation (5):
 
 R   AB   =R   A   ∩R   B   (5)
 
     Typical rules included in the intersection R AB  may include geographic rules, various substrings found in a user agent, time parameters (such as day of the week), blacklisted IP addresses, distribution partner rules (if available in both A and B), query-based rules (such as length of keyword, number of terms, keyword category, keyword blacklist), properties attached to the IP address or IP range (such as white list, anonymous proxy, known robot, AOL, corporate proxy, suspicious domain name), combinations of these rules, and other rules and rule combinations. 
     Once the intersection R AB  of the two rule sets is determined, in some embodiments, the methodology is performed as follows:
         1. At a block  115 , build an association table F A|AB  based on the intersection R AB  applied to data object A.   2. At a block  120 , determine the odds of conversion and an average scaled score for each result vector f∈F A|AB , using weighted averages on result vectors in F A . In some embodiments, weight is based on the number of clicks or transactions.   3. At a block  125 , estimate û A|AB  and {circumflex over (v)} A|AB  using equations (1) and (2) (above) applied to F A|AB .   4. At a block  130 , build an association table F B|AB  based on the intersection R AB  applied to data object B.   5. At a block  135 , retrieve odds of conversion and average scaled score for each result vector f∈F B|AB , from F A|AB . Use the fact that each f∈F B|AB  has an equivalent f′∈F A|AB  with known odds of conversion and average scaled score, by construction. Note that f∈F B|AB  is equivalent to f′∈F A|AB  if an only if the result vectors are associated with the same rule configuration from R AB .   6. At a block  140 , estimate û B|AB , and {circumflex over (v)} B|AB  using equations (1) and (2) (above) applied to F B|AB . Note that, in general, of n f ≠n f′ , even when f∈F B|AB  is equivalent to f′∈F A|AB . This ensures that (û A|AB , {circumflex over (v)} A|AB ) and (û B|AB , {circumflex over (v)} B|AB ) are usually different, unless A=B.   7. At a block  145 , estimate the core parameters û B  and {circumflex over (v)} B  associated with data object B using equations (6) and (7):       

     
       
         
           
             
               
                 
                   
                     
                       u 
                       ^ 
                     
                     B 
                   
                   = 
                   
                     
                       
                         u 
                         ^ 
                       
                       A 
                     
                     · 
                     
                       
                         
                           u 
                           ^ 
                         
                         
                           B 
                           ❘ 
                           AB 
                         
                       
                       
                         
                           u 
                           ^ 
                         
                         
                           A 
                           ❘ 
                           AB 
                         
                       
                     
                   
                 
               
               
                 
                   ( 
                   6 
                   ) 
                 
               
             
             
               
                 
                   
                     
                       v 
                       ^ 
                     
                     B 
                   
                   = 
                   
                     
                       
                         v 
                         ^ 
                       
                       A 
                     
                     · 
                     
                       
                         
                           v 
                           ^ 
                         
                         
                           B 
                           ❘ 
                           AB 
                         
                       
                       
                         
                           v 
                           ^ 
                         
                         
                           A 
                           ❘ 
                           AB 
                         
                       
                     
                   
                 
               
               
                 
                   ( 
                   7 
                   ) 
                 
               
             
           
         
       
         
         
           
             8. At a block  150 , scale the raw scores available in data object B, using û B  and {circumflex over (v)} B  as core parameters.
 
In general, the scores are scaled by applying a subset of rules to each of the datasets, determining a correction factor based on the application of the subset of rules, and then applying the correction factor to the raw scores.
 
Other Considerations
 
           
         
       
    
     If the weights associated with the result vectors from data objects A and B are distributed quite differently, then û B , {circumflex over (v)} B  will be quite different from û A , {circumflex over (v)} A . For example, if data object B contains data that is of lower quality than a reference data set A, û B  should be less than û A . 
     Note that if most of the fraud in data object B does not show up in the smaller common rule subset R AB  (but instead, in more ad hoc rules outside R A ), then û B  will be overestimated. However, in most instances, low quality that is detected in more advanced rules usually impacts a small percentage of transactions, and the low quality generally “transpires” to some extent in the smaller rule set R AB , particularly if the smaller rule set is carefully built. 
     Simplified Procedure when Data Objects A and B are From the Same Client—Re-Scaling Schedule 
     In some embodiments, a score is generated for a given client multiple times over a given time period. For example, periodic updates of a score may be performed on a scheduled basis, when a rule set is modified, and/or in other circumstances. The methodology in these embodiments is simpler, in the sense that it can be done without explicitly identifying a common rule set R AB , particularly if the data has not changed significantly. 
     To preserve the consistency of scores for a given client over time, the methodology maintains the same average score and variance between scoring periods. For example, a score may be generated for a given client during a first scoring period. For each subsequent scoring period, the score is rescaled so that it is consistent with the average score and the variance of the first scoring period. 
     When the methodology is used to preserve the consistency of scores for a client over time (e.g., update a client&#39;s score), in some embodiments, the methodology may use an overlapping time period to further increase score accuracy. For example, scoring periods may have a minimum number of days (e.g., seven) in common. 
       FIG.  2    is a flow diagram of a process  200  that may be used to preserve scores of the quality of traffic when performing updates for the same client:
         1. At a block  205 , determine u A  and v A , the parameters obtained during the last rescaling that occurred (period A), such as eight weeks ago.   2. At a block  210 , estimate u A|AB  and v A|AB  using available scaled scores on weeks −1 and −2. This time period is referred to as the AB period.   3. At a block  215 , modify the rule set (such as by computing new weights for each rule, as part of a rule updating schedule).   4. Set u B|AB =u A|AB  and v B|AB =v A|AB , so that these parameters are the same for period A and period B.   5. Moving forward, at a block  220 , apply the modified rule set in period B (week −2, −1, +0, +1, etc.) to generate raw scores for period B.   6. At a block  225 , scale the raw scores for period B by using the scaling formula applied to the raw scores with u B =u B|AB  and v B =v B|AB .       

     In those embodiments in which the data set changes but does not have backward compatibility, it may not be feasible to use an overlapping window (such as week −1, −2). Instead, consecutive weeks may be used. 
       FIG.  3    illustrates how scores may be preserved over time for a given client in some embodiments. Column  305  lists the days on which a score is generated for the client. Column  310  lists the average score generated for the client on each day according to a previous scoring methodology (referred to in  FIG.  3    as “old scoring”). Column  315  lists the standard deviation associated with the scores of column  310 . In the example illustrated by  FIG.  3   , the previous scoring methodology is applied to days 1 through 28 (note that the previous scoring methodology is also applied to days 29-32 for illustration purposes). On day 29, a new scoring methodology is applied, e.g., new rules may be added to the rule set. Column  320  lists the average score generated for the client according to the new scoring methodology (referred to in  FIG.  3    as “new scoring”). Column  325  lists the standard deviation associated with the scores of  320 . 
     As described above, scores may vary significantly for a given client when the scoring methodology changes, even if the quality of traffic received by the client remains the same. As illustrated by  FIG.  3   , on day 28, the client has an average score of  635  and a corresponding standard deviation of 39. On day 29, if the previous scoring methodology continues to be applied, the client will have an average score of  608  and a standard deviation of 30. However, when the new scoring methodology is applied on day 29, the client&#39;s average score is 453 with a corresponding standard deviation of 19. To generate data that is used to scale the new scores, new scores may be computed backward over a previous time frame.  FIG.  3    illustrates scores computed backward according to the new methodology for the previous 14 days, i.e., days 15 through 28. 
     The overlapping time period in which both old and new scores have been generated—here, days 15 through 28—is used to calibrate the new scores going forward. That is, a linear transformation to be applied to the new scores is determined according to the formulas described above, using the old average score 631 and standard deviation 36 (blocks  340  and  345 ) in the overlapping time period, and the new average score 466 and standard deviation 22 (blocks  350  and  355 ) in the overlapping time period. Column  330  contains the newly calibrated scores. For example, on day 29, when the new scoring methodology is combined with the linear transformation, the client has an average score of 609 and a corresponding standard deviation of 31. In addition, other data may be generated for analysis, including the average score 640 and standard deviation 38 (blocks  360  and  365 ) for a given time period—here, four weeks. 
     Scaling Raw Scores when Odds of Conversion are not Available 
     In some embodiments, odds of conversion are not available. For example, a search engine does not generally have access to data that indicates whether a desirable agent action was ultimately generated at an advertiser&#39;s site. In such embodiments, assume an association table with no conversion metrics and a raw score S* f  for each result vector f. Estimated values {circumflex over (Q)} f  will be obtained for the odds of conversion. 
     The methodology relies on the fact that a good, standard estimator of the odds of conversion, for a given result vector f, is provided by the two-parameter equation (8):
 
 {circumflex over (Q)}   f =exp(α+β S*   f )  (8)
 
     Next, α and β are estimated using two data points. For example, the 50th and 25th percentiles of the raw score distribution, denoted S* 50  and S* 25 , may be used as data points. 
     Since no conversion is available, in some embodiments, educated guesses may be made regarding the odds of conversion Q 50  and Q 25  at S* 50  and S* 25 , respectively. In other embodiments, Q 50  and Q 25  may be obtained using external data. 
     In general, good educated guesses may be Q 50 =1.00 and Q 25 =0.50, assuming raw scores are sorted by quality, in decreasing order. That is, Q 25  corresponds to relatively fraudulent clicks (25th percentile) converting at a rate of about 0.50×average. 
     Let L 50 =ln Q 50  and L 25 =ln Q 25 . Then α and β may be determined by solving the system: 
             {               L   50     =     α   +     β   ⁢           ⁢     S   50   *                       L   25     =     α   +     β   ⁢           ⁢     S   25   *                             
which provides the solution:
 
     
       
         
           
             { 
             
               
                 
                   
                     
                       α 
                       = 
                       
                         
                           ( 
                           
                             
                               
                                 S 
                                 50 
                                 * 
                               
                               ⁢ 
                               
                                 L 
                                 25 
                               
                             
                             - 
                             
                               
                                 S 
                                 25 
                                 * 
                               
                               ⁢ 
                               
                                 L 
                                 50 
                               
                             
                           
                           ) 
                         
                         ⁢ 
                         
                           / 
                         
                         ⁢ 
                         
                           ( 
                           
                             
                               S 
                               50 
                               * 
                             
                             - 
                             
                               S 
                               25 
                               * 
                             
                           
                           ) 
                         
                       
                     
                   
                 
                 
                   
                     
                       
                         β 
                         = 
                         
                           
                             ( 
                             
                               
                                 L 
                                 50 
                               
                               - 
                               
                                 L 
                                 25 
                               
                             
                             ) 
                           
                           ⁢ 
                           
                             / 
                           
                           ⁢ 
                           
                             ( 
                             
                               
                                 S 
                                 50 
                                 * 
                               
                               - 
                               
                                 S 
                                 25 
                                 * 
                               
                             
                             ) 
                           
                         
                       
                       ⁢ 
                       
                           
                       
                     
                   
                 
               
                 
             
           
         
       
     
     Note that if S* 50 =S* 25 , then the solution will not work. Instead, different data points may be used, such as the 75th and 25th percentiles, instead of the 50th and 25th percentiles. However, it should be noted that S* 50 =S* 25  would indicate that the raw score distribution is very poor. 
     In other embodiments, a solution may consist of using more than two data points and performing a regression on α and β. In other embodiments, a model with three parameters, α, β, and γ may be used. 
     If core parameters are not available, a reference set may be used with the standard methodology described above. That is, the methodology may be used to determine u and v, substituting the odds of conversion (at the result vector level) with estimated odds of conversion. The raw scores may be scaled, such as according to equation (4) (above). 
     Unless the context clearly requires otherwise, throughout the description and the claims, the words “comprise,” “comprising,” and the like are to be construed in an inclusive sense, as opposed to an exclusive or exhaustive sense; that is to say, in the sense of “including, but not limited to.” As used herein, the terms “connected,” “coupled,” or any variant thereof, means any connection or coupling, either direct or indirect, between two or more elements; the coupling of connection between the elements can be physical, logical, or a combination thereof. Additionally, the words “herein,” “above,” “below,” and words of similar import, when used in this application, shall refer to this application as a whole and not to any particular portions of this application. Where the context permits, words in the above Detailed Description using the singular or plural number may also include the plural or singular number respectively. The word “or,” in reference to a list of two or more items, covers all of the following interpretations of the word: any of the items in the list, all of the items in the list, and any combination of the items in the list. 
     The above detailed description of embodiments of the system is not intended to be exhaustive or to limit the system to the precise form disclosed above. While specific embodiments of, and examples for, the system are described above for illustrative purposes, various equivalent modifications are possible within the scope of the system, as those skilled in the relevant art will recognize. For example, while processes or blocks are presented in a given order, alternative embodiments may perform routines having steps, or employ systems having blocks, in a different order, and some processes or blocks may be deleted, moved, added, subdivided, combined, and/or modified to provide alternative or subcombinations. Each of these processes or blocks may be implemented in a variety of different ways. Also, while processes or blocks are at times shown as being performed in series, these processes or blocks may instead be performed in parallel, or may be performed at different times. 
     The teachings of the methods and system provided herein can be applied to other systems, not necessarily the system described above. The elements and operation of the various embodiments described above can be combined to provide further embodiments. 
     While certain aspects of the technology are presented below in certain claim forms, the inventors contemplate the various aspects of the technology in any number of claim forms. For example, while only one aspect of the invention is recited as embodied in a computer-readable medium, other aspects may likewise be embodied in a computer-readable medium. Accordingly, the inventors reserve the right to add additional claims after filing the application to pursue such additional claim forms for other aspects of the technology. 
     From the foregoing, it will be appreciated that specific embodiments of the invention have been described herein for purposes of illustration, but that various modifications may be made without deviating from the spirit and scope of the invention. Accordingly, the invention is not limited except as by the appended claims.