Patent Document ID: 7725340
Application ID: 12050371
Patent Status: 1

Claim One:
1. A computer-implemented ranking-based method for evaluating regression models by obtaining predictions y from one or more models to be evaluated on a test set of customers for which the true value y of the quantity of interest has been observed comprising the steps of: storing the test set in an electronic database sorted in increasing order such that “large” corresponds to one of a plurality of customers and/or potential customers with largest perceived spending budget, and “small” corresponds to one of a plurality of customers and/or potential customers with smallest perceived spending budget; applying all models to customer data from the test set to obtain predictions for the spending for each model and customer; converting the predictions into ranks and storing for each model in one or more electronic databases as a model ranking table, a number of ranking order switches relative to ranking of the observed customer spending being calculated for each model and ranking order switches being defined as those changes in ranking position of the prediction relative to the order of the true observations y; calculating a measure of a magnitude of erroneous ranking from a weighted sum of ranking order switches; transforming the number of ranking switches and weighted sum of ranking order switches into a range of [−1, 1] wherein −1 corresponds to making all possible errors (inverse ranking) and 1 corresponds to a perfect model wherein said number of ranking switches has been transformed to represent a difference between a probability that the ranking of two customers and/or potential customers are in the same order versus the probability that two of the customers and/or potential customers are in different orders from the originally obtained rank; normalizing transformed measures of order switches into a range of [0,1] wherein 1 corresponds to perfect ranking and 0 corresponds inverse ranking; calculating a variance of measures of order switches and determining confidence intervals for each ranking measure, further comprising: calculating using computing resources a variance that said measures of said ranking order switches, wherein said calculating using computing resources a variance implements a relationship: V ⁢ a ^ ⁢ r ⁡ ( τ ^ ) = ( 2 n ⁡ ( n - 1 ) ) 2 · 2 · ( 2 ⁢ ∑ i ⁢ ⁢ C i 2 - ∑ i ⁢ ⁢ C i - ( 2 ⁢ n - 3 ) n ⁡ ( n - 1 ) · ( ∑ i ⁢ ⁢ C i ) 2 ) wherein, a set of variables of said set of relationships includes: C i = ∑ j < i ⁢ ⁢ 1 ⁢ { y i > y j } + ∑ j > i ⁢ ⁢ 1 ⁢ { y i < y j } is a number of observations that are concordant with observation i, i is an integer from 1 to (n−1), j is an integer from (i+1) to n, n is a size of a set of customers and/or potential customers, and y is a predicted response of the model y=m(x); calculating using computing resources a confidence interval for each measure of said ranking order switches; and constructing using computing resources graphical representations of said ranking order switches; updating the model performance table with the ranking measures and their confidence intervals; and outputting the ranking measures and confidence levels as well as graphical representations thereof to a domain expert via electronic display on a computer monitor, who will choose based on this information the best model m* of the one or more models evaluated, and storing the predictions of y=m*(x) in the optimal prediction table.