Source: http://www.google.com/patents/US8170841?dq=6499693
Timestamp: 2015-07-06 04:06:41
Document Index: 419107273

Matched Legal Cases: ['art.\n18', 'art.\n20', 'art.\n21', 'art.\n25', 'art.\n27', 'art.\n28', 'art.15']

Patent US8170841 - Predictive model validation - Google PatentsSearch Images Maps Play YouTube News Gmail Drive More »Sign inAdvanced Patent SearchPatentsModels are generated using a variety of tools and features of a model generation platform. For example, in connection with a project in which a user generates a predictive model based on historical data about a system being modeled, the user is provided through a graphical user interface a structured...http://www.google.com/patents/US8170841?utm_source=gb-gplus-sharePatent US8170841 - Predictive model validationAdvanced Patent SearchPublication numberUS8170841 B2Publication typeGrantApplication numberUS 10/826,947Publication dateMay 1, 2012Filing dateApr 16, 2004Priority dateApr 16, 2004Also published asUS20050234753, US20120197608Publication number10826947, 826947, US 8170841 B2, US 8170841B2, US-B2-8170841, US8170841 B2, US8170841B2InventorsStephen K. Pinto, Richard Mansfield, Marc Jacobs, Donald RubinOriginal AssigneeKnowledgebase Marketing, Inc.Export CitationBiBTeX, EndNote, RefManPatent Citations (88), Non-Patent Citations (43), Referenced by (9), Classifications (15), Legal Events (7) External Links: USPTO, USPTO Assignment, EspacenetPredictive model validation
US 8170841 B2Abstract
In connection with a project in which a user generates a predictive model based on historical data about a system being modeled, the user is enabled to validate the model development process with cross-validation between at least two subsets of the historical data; the validated model development process is enabled to be reapplied.
in connection with a project, selecting a first subset of historical data about a system being modeled for use in generating a tentative predictive model for the system, the first subset of historical data comprising less than all of the historical data;
selecting a process for developing a predictive model for the system from among multiple possible model development processes;
applying the selected model development process to the first subset of historical data to generate the tentative predictive model;
selecting a second subset of the historical data, the second subset being less than all of the historical data and being at least a portion of a complementary dataset of the first subset or being randomly selected from the historical data and independent of the first subset;
applying the tentative predictive model to the selected second subset, determining whether results of applying the tentative predictive model to the selected second subset validate that the selected model development process will produce a final predictive model that is accurate for data that is not part of the historical data, if the selected model development process is so validated, then applying the validated model development process to a full set of historical data that includes the first and second subsets to generate a final predictive model, and using the final predictive model.
2. The method of claim 1 also including displaying on a user interface project goals to enable the user to assess performance of the project, wherein the project goals comprise at least one of: cumulative lift over an interval of interest, degree of monotonicity, or concordance scores.
3. The method of claim 2 also including
identifying that the tentative predictive model does not produce at least a predefined degree of lift for at least one of the subsets.
4. The method of claim 3 also including
enabling a user to choose interactively at least one model development criterion change or transformation or interaction of variables to improve a fit of the predictive model.
5. The method of claim 4 also including
graphically displaying and comparing measures of performance for a validation dataset and a training dataset.
6. A machine-based method comprising:
in connection with a project, selecting a model development process from multiple model development processes to apply on a first subset of less than all of a set of historical data to generate a first tentative predictive model, applying the selected model development process including (a) automatically transforming variables of the first subset of the historical data, (b) automatically generating the first tentative predictive model, and (c) automatically generating performance measures of the first tentative predictive model, determining a validity of the selected development process based on the performance measures of the first tentative predictive model;
applying the validated model development process to the full set of historical data to generate a second, final predictive model, and using the final predictive model.
7. The method of claim 6 also including
generating measures of the performance of the predicative model for the at least two datasets, the performance measures being generated separately percentile by percentile.
8. The method of claim 6 also including
graphically displaying and comparing measures of the performance for at least two datasets.
9. The method of claim 6 also including
persistently storing the validated model development process and a validated model for computing propensities for at least one target outcome variable, the propensities serving as indices of a score for non-historical data.
10. The method of claim 6 also including providing a user interface for assessing project goals against performance.
11. The method of claim 6 also including providing a user interface for selecting at least one subset of the historical data in addition to a training subset.
12. The method of claim 6 providing a user interface for displaying the performance of the first predictive model for at least two subsets of the historical data for an interval of interest.
13. The method of claim 6 enabling a user to choose interactively at least one transformation or interaction of variables to improve the model development process.
14. The method of claim 6 also includes cross-validating the final model using random portions of the historical data.
15. The method of claim 6 providing a user interface that enables the user to select at least one validation dataset and invoke a process for validating the model development process.
16. The method of claim 6 providing a user interface that enables the user to point and click to cause display of information about the validation of the model development process.
17. The method of claim 16 in which the information about the validation of the model development process includes at least one of: a statistical report card with a link to the statistical report chart, a cumulative lift chart with a link to the cumulative lift chart, and a non-cumulative lift chart with a link to the non-cumulative lift chart.
18. The method of claim 17 in which invocation of the link to the statistical report card causes display of the statistics of the validation of the model development process.
19. The method of claim 17 in which invocation of the link to the cumulative lift chart causes display of a cumulative lift chart.
20. The method of claim 17 in which invocation of the link to the cumulative lift chart causes display of a non-cumulative lift chart.
21. The method of claim 17 in which a user is enabled to choose interactively at least one performance criterion change or transformation or interaction of variables to improve the model development process.
22. The method of claim 6 also including providing a user interface that enables the user to select at least one machine automated model development process applied to the entire set of the historical data for the validated model development process.
23. The method of claim 6 also including providing a user interface that enables the user to point and click to cause display of information about the performance of the validated model development process applied to the entire set of historical data.
24. The method of claim 23 in which the information about the performance of the first model for two independent datasets includes at least one of the following:
a statistical report card with a link to the statistical report chart, a cumulative lift chart with a link to the cumulative lift chart, a non-cumulative lift chart with a link to the non-cumulative lift chart.
25. The method of claim 24 in which the invocation of the link to the statistical report card causes display of the statistics of the validation of the model development process.
26. The method of claim 24 in which the invocation of the link to the cumulative lift chart causes display of a cumulative lift chart.
27. The method of claim 24 in which the invocation of the link to the cumulative lift chart causes display of a non-cumulative lift chart.
28. The method of claim 6 also including storing the final model and validation results of the model development process persistently.
In general, in one aspect, the invention features a machine-base method comprising, in connection with a project in which a user generates a predictive model based on historical data about a system being modeled, enabling the user to validate a model process with a predictive model between at least two subsets of the historical data. Implementations may include one or more of the following features. User interface for assessing project goals against performance. User interface for selecting at least one subset of the historical data in addition to the training subset. User interface for displaying the performance of the model for at least two subsets of the historical data for the interval of interest. A user is enabled to choose interactively at least one transformation or interaction of variables to improve the model validation process. Means of determining whether the model generalizes to the data other than the subsample, if so, applying the possible model to all of the data to generate a final model, and cross-validating the final model using random portions of the data. The user interface enables the user to select at least one validation dataset and invoke a model process validation method. The user interface enables the user to point and click to cause display of information about the model process validation. The information about the model process validation includes at least one of: a statistical report card with a link to the statistical report chart, a cumulative lift chart with a link to the cumulative lift chart, a non-cumulative lift chart with a link to the non-cumulative lift chart. The invocation of the link to the statistical report card causes display of the statistics of model process validation. The invocation of the link to the cumulative lift chart causes display of a cumulative lift chart. The invocation of the link to the cumulative lift chart causes display of a non-cumulative lift chart. A user is enabled to choose interactively at least one performance criterion change or transformation or interaction of variables to improve model validation process. The user interface enables the user to select at least one machine automated model development process applied to the entire dataset for a validated model process. The user interface enables the user to point and click to cause display of information about the performance of the validated model process applied to the entire set of historical data. The information about the model performance for two independent data subsets includes at least one of the following: a statistical report card with a link to the statistical report chart, a cumulative lift chart with a link to the cumulative lift chart, a non-cumulative lift chart with a link to the non-cumulative lift chart. The invocation of the link to the statistical report card causes display of the statistics of model process validation. The invocation of the link to the cumulative lift chart causes display of a cumulative lift chart. The invocation of the link to the cumulative lift chart causes display of a non-cumulative lift chart. The final model and the model process validation results are stored persistently.
Square: :X ^2
Nonlinear interaction: : X*:Y
Missing value: If(Is Missing(:X), 0, If(:X=1, 1, 0))
G 2(A i ; T/z)=2�(ln2)�E*(z)�MI (A i ; T/z) [7]
F 0(G(x 1 1 , . . . , x 1 n))−>H(x 2 1 , . . . , x 2 m) [1]
F 1(H(x 2 1 , . . . , x 2 m)−>J(x 3 1 , . . . , x 3 p) [2]
F 2(H(x 2 1 , . . . , x 2 p))−>J(x 3 1 , . . . , x 3 q) [3]
where H(x2 1, . . . , x2 p)=1 and J(x3 i, . . . , x3 q)=1 are the hyperplanes defining the respective successive models. In effect, the second filter reduces the dimensionality of the modeling space from p to q.
In the third stage 174, the subspace, XQ, is expanded by including all significant cross-products, xj k*xp q, where xj k and xp q are in XQ, then applying a filter 176 to retain only significant variables, e.g., x1 k, whose probability of non-contribution, [1−Pr(y|x4 k)], is less than a fixed constant, C3,
F 3(J(x 4 1 , . . . , x 4 q))−>K(x 5 1 , . . . , x 5 r) [4]
In the fourth stage 178, the augmented subspace, X# Q is further iteratively expanded with all the subspaces, xj k*xp q+zr s, which are now significant where zr s are from XM-XP, then applying a filter 180 to retain only significant variables.
F 5(K(x 5 1, . . . , x5 r))−>L(x 6 1 , . . . , x 6 s) [5]
logit[Pr(Y 1=1|X 1 , . . . , X k)]=C(0)+Sum(j=l,k)C(j)X j [6]
log it[Pr(Y 1=1|X 1 , . . . , X k)]=C(0)+Sum(j=l,k)C(j)X j [8]
Pr(Y i=1|X ik =X k)=exp(γk0+γk1 �x k)/[1+exp(γk0+γ k1 �x k)] [10]
Pr(Y i=1|X ik =X k)=exp(γk0+γk1 �x k)/[1+exp(γk0+γ k1 �x k)]s/F i [11]
The model generation platform provides a series of actionable reports profiling customers' past and future behaviors. In the context of product marketing, these reports, derived from predictive analytics, yield a feature spectrum, for example, on customer segments for generating creative marketing development, mass marketing promotions, marketing planning, and product development.
The model generation platform is designed for'straight through processing (STP) of customer data records (CDRs) from input, to modeling, to scoring and the deployment of actionable lists. For sensitive information, the model generation platform employs an encryption library so that, for example, the commercial standard TripleDES encryption is built into list scoring post-processing.
Dimension Reduction Stare
Model Rains for Sample
Patent CitationsCited PatentFiling datePublication dateApplicantTitleUS5479523Mar 16, 1994Dec 26, 1995Eastman Kodak CompanyConstructing classification weights matrices for pattern recognition systems using reduced element feature subsetsUS5491629Mar 4, 1994Feb 13, 1996Strategic Weather ServicesSystem and method for determining the impact of weather and other factors on managerial planning applicationsUS5544281Mar 23, 1994Aug 6, 1996Hitachi, Ltd.Method of supporting decision-making for predicting future time-series data using measured values of time-series data stored in a storage and knowledge stored in a knowledge baseUS5583763Sep 9, 1993Dec 10, 1996Mni InteractiveMethod and apparatus for recommending selections based on preferences in a multi-user systemUS5592560Sep 8, 1994Jan 7, 1997Credit Verification CorporationMethod and system for building a database and performing marketing based upon prior shopping historyUS5608620Feb 17, 1995Mar 4, 1997Lundgren; Carl A.Method of eliciting unbiased forecasts by relating a forecaster's pay to the forecaster's contribution to a collective forecastUS5621812May 17, 1993Apr 15, 1997Credit Verification CorporationMethod and system for building a database for use with selective incentive marketing in response to customer shopping historiesUS5659593Nov 30, 1994Aug 19, 1997Lucent Technologies Inc.Detection of deviations in monitored patternsUS5692107Sep 20, 1996Nov 25, 1997Lockheed Missiles & Space Company, Inc.Method for generating predictive models in a computer systemUS5701400Mar 8, 1995Dec 23, 1997Amado; Carlos ArmandoMethod and apparatus for applying if-then-else rules to data sets in a relational data base and generating from the results of application of said rules a database of diagnostics linked to said data sets to aid executive analysis of financial dataUS5826250Jun 19, 1996Oct 20, 1998Pegasystems Inc.Rules bases and methods of access thereofUS6026397May 22, 1996Feb 15, 2000Electronic Data Systems CorporationData analysis system and methodUS6185543May 15, 1998Feb 6, 2001Marketswitch Corp.Method and apparatus for determining loan prepayment scoresUS6243696Mar 24, 1998Jun 5, 2001Pavilion Technologies, Inc.Automated method for building a modelUS6269325Oct 21, 1998Jul 31, 2001Unica Technologies, Inc.Visual presentation technique for data mining softwareUS6301471Nov 2, 1998Oct 9, 2001Openwave System Inc.Online churn reduction and loyalty systemUS6317752Dec 9, 1998Nov 13, 2001Unica Technologies, Inc.Version testing in database miningUS6321205Aug 17, 1998Nov 20, 2001Value Miner, Inc.Method of and system for modeling and analyzing business improvement programsUS6338066Sep 25, 1998Jan 8, 2002International Business Machines CorporationSurfaid predictor: web-based system for predicting surfer behaviorUS6381504Dec 31, 1998Apr 30, 2002Pavilion Technologies, Inc.Method for optimizing a plant with multiple inputsUS6430539 *May 6, 1999Aug 6, 2002Hnc SoftwarePredictive modeling of consumer financial behaviorUS6523015Oct 14, 1999Feb 18, 2003KxenRobust modelingUS6542894Dec 9, 1998Apr 1, 2003Unica Technologies, Inc.Execution of multiple models using data segmentationUS6556977Aug 14, 1998Apr 29, 2003Adeza Biomedical CorporationMethods for selecting, developing and improving diagnostic tests for pregnancy-related conditionsUS6567812Sep 27, 2000May 20, 2003Siemens AktiengesellschaftManagement of query result complexity using weighted criteria for hierarchical data structuringUS6611726Sep 15, 2000Aug 26, 2003Carl E. CrosswhiteMethod for determining optimal time series forecasting parametersUS6631360Nov 6, 2000Oct 7, 2003Sightward, Inc.Computer-implementable Internet prediction methodUS6636862Jul 5, 2001Oct 21, 2003Camo, Inc.Method and system for the dynamic analysis of dataUS6640215Mar 15, 2000Oct 28, 2003Marketswitch CorporationIntegral criterion for model training and method of application to targeted marketing optimizationUS6684208May 15, 2001Jan 27, 2004Rockwell Technologies, LlcSystem and method for quantifying an extent to which a data mining algorithm captures useful information in input dataUS6687696Jul 26, 2001Feb 3, 2004Recommind Inc.System and method for personalized search, information filtering, and for generating recommendations utilizing statistical latent class modelsUS6782390Mar 7, 2002Aug 24, 2004Unica Technologies, Inc.Execution of multiple models using data segmentationUS6807535Mar 8, 2001Oct 19, 2004Lnk CorporationIntelligent tutoring systemUS6820070Dec 8, 2000Nov 16, 2004Insyst Ltd.Method and tool for data mining in automatic decision making systemsUS6834266Oct 11, 2001Dec 21, 2004Profitlogic, Inc.Methods for estimating the seasonality of groups of similar items of commerce data sets based on historical sales data values and associated error informationUS6836773Sep 27, 2001Dec 28, 2004Oracle International CorporationEnterprise web mining system and methodUS6839682 *Oct 3, 2000Jan 4, 2005Fair Isaac CorporationPredictive modeling of consumer financial behavior using supervised segmentation and nearest-neighbor matchingUS6859529Feb 25, 2002Feb 22, 2005Austin Logistics IncorporatedMethod and system for self-service scheduling of inbound inquiriesUS6879971Jun 5, 2001Apr 12, 2005Pavilion Technologies, Inc.Automated method for building a modelUS6882977Jul 31, 2000Apr 19, 2005Hewlett-Packard Development Company, L.P.Method and facility for displaying customer activity and valueUS6952688Jun 7, 2000Oct 4, 2005Insyst Ltd.Knowledge-engineering protocol-suiteUS6954758Jun 30, 2000Oct 11, 2005Ncr CorporationBuilding predictive models within interactive business analysis processesUS6956941Apr 12, 2000Oct 18, 2005Austin Logistics IncorporatedMethod and system for scheduling inbound inquiriesUS7035786 *Oct 26, 2001Apr 25, 2006Abu El Ata Nabil ASystem and method for multi-phase system development with predictive modelingUS7035855Jul 6, 2000Apr 25, 2006Experian Marketing Solutions, Inc.Process and system for integrating information from disparate databases for purposes of predicting consumer behaviorUS7043461May 23, 2001May 9, 2006Genalytics, Inc.Process and system for developing a predictive modelUS7047251Nov 22, 2002May 16, 2006Accenture Global Services, GmbhStandardized customer application and record for inputting customer data into analytic modelsUS7349827Sep 18, 2002Mar 25, 2008Doubleclick Inc.System and method for reporting website activity based on inferred attribution methodologyUS7499897Apr 16, 2004Mar 3, 2009Fortelligent, Inc.Predictive model variable managementUS7562058Apr 16, 2004Jul 14, 2009Fortelligent, Inc.Predictive model management using a re-entrant processUS7725300Apr 16, 2004May 25, 2010Fortelligent, Inc.Target profiling in predictive modelingUS7730003Apr 16, 2004Jun 1, 2010Fortelligent, Inc.Predictive model augmentation by variable transformationUS7933762Apr 16, 2004Apr 26, 2011Fortelligent, Inc.Predictive model generationUS20020052836Aug 30, 2001May 2, 2002Yuri GalperinMethod and apparatus for determining a prepayment score for an individual applicantUS20020052858Dec 4, 2001May 2, 2002Insyst Ltd.Method and tool for data mining in automatic decision making systemsUS20020123923Mar 1, 2001Sep 5, 2002Stefanos ManganarisMethod and system for assessing intrinsic customer valueUS20020127529Dec 6, 2000Sep 12, 2002Cassuto Nadav YehudahPrediction model creation, evaluation, and trainingUS20030004777Mar 22, 2001Jan 2, 2003Phillips Alan Paul RollestonController for controlling a systemUS20030004781Jun 18, 2001Jan 2, 2003Mallon Kenneth P.Method and system for predicting aggregate behavior using on-line interest dataUS20030018601Mar 7, 2002Jan 23, 2003Unica Technologies, Inc., A Massachusetts CorporationExecution of multiple models using data segmentationUS20030088565Oct 15, 2002May 8, 2003Insightful CorporationMethod and system for mining large data setsUS20030154442Feb 13, 2002Aug 14, 2003Karen PapierniakVisualization tool for web analyticsUS20030171829Mar 7, 2002Sep 11, 2003Insyst Ltd.System and method for complex process optimization and controlUS20030176931Mar 11, 2002Sep 18, 2003International Business Machines CorporationMethod for constructing segmentation-based predictive modelsUS20030212678Mar 10, 2003Nov 13, 2003Bloom Burton H.Automated model building and evaluation for data mining systemUS20030220860Apr 24, 2003Nov 27, 2003Hewlett-Packard Development Company,L.P.Knowledge discovery through an analytic learning cycleUS20040002833 *Jun 28, 2002Jan 1, 2004Microsoft CorporationSystems and methods for mining model accuracy display for multiple state predictionUS20040030667Aug 2, 2002Feb 12, 2004Capital One Financial CorporationAutomated systems and methods for generating statistical modelsUS20040111314 *Oct 16, 2002Jun 10, 2004Ford Motor CompanySatisfaction prediction model for consumersUS20040215599Dec 23, 2003Oct 28, 2004Eric AppsMethod and system for the visual presentation of data mining modelsUS20050004786Nov 13, 2003Jan 6, 2005Koninklijke Philips Electronics N.V.State machine modellingUS20050089768Aug 27, 2004Apr 28, 2005Satoshi TanakaMethod of creating predictive model, method of managing process steps, method of manufacturing semiconductor device, method of manufacturing photo mask, and computer program productUS20050096950Oct 29, 2003May 5, 2005Caplan Scott M.Method and apparatus for creating and evaluating strategiesUS20050234688Apr 16, 2004Oct 20, 2005Pinto Stephen KPredictive model generationUS20050234760Apr 16, 2004Oct 20, 2005Pinto Stephen KTarget profiling in predictive modelingUS20050234761Apr 16, 2004Oct 20, 2005Pinto Stephen KPredictive model developmentUS20050234762Apr 16, 2004Oct 20, 2005Pinto Stephen KDimension reduction in predictive model developmentUS20050234763Apr 16, 2004Oct 20, 2005Pinto Stephen KPredictive model augmentation by variable transformationUS20060161403Dec 10, 2003Jul 20, 2006Jiang Eric PMethod and system for analyzing data and creating predictive modelsUS20060241923May 2, 2006Oct 26, 2006Capital One Financial CorporationAutomated systems and methods for generating statistical modelsUS20100010878Sep 23, 2009Jan 14, 2010Fortelligent, Inc.Predictive model developmentUS20110071956May 26, 2010Mar 24, 2011Fortelligent, Inc., a Delaware corporationPredictive model developmentCA2241119A1Jun 19, 1998Dec 20, 1998Felix KollerProcess for removing nitrogen from flue gasesEP1085429A1Aug 24, 2000Mar 21, 2001NCR International, Inc.Classifying data in a databaseWO1999022328A1Oct 26, 1998May 6, 1999Charles L Jones IiiSystem and method of targeted marketingWO2002019061A2Aug 30, 2001Mar 7, 2002Marketswitch CorpMethod and apparatus for determining a prepayment score for an individual applicantWO2003005232A2Jul 8, 2002Jan 16, 2003Angoss Software CorpA method and system for the visual presentation of data mining modelsWO2005011749A2Jul 28, 2004Feb 10, 2005Steris IncMethod and device for deactivating items and for maintaining such items in a deactivated state* Cited by examinerNon-Patent CitationsReference1"On-line Seminar: How to keep customers and increase profits using predictive analytics powered by Clementine", www.spss.com/spssbi/directresponse/clementinewebinar/index.cfm?dcode=d4183.2"SAS Enterprise Miner", www/sas.com/technologies/analytics/datamining/miner.3Angus, Enterprise Miner 5.1 Digs Into Data, InforWorld, Aug. 9, 2004, vol. 26, No. 32, pp. 22-23.4Application for U.S. Appl. No. 12/564,998, filed Sep. 23, 2009.5Autobox 5.0 for Windows-User's Guide, Automatic Forecasting Systems, Inc. (Dec. 10, 1999).6Autobox 5.0 for Windows—User's Guide, Automatic Forecasting Systems, Inc. (Dec. 10, 1999).7Bhattacharyya , Evolutionary Algorithms in Data Mining: Multi-Objective Performance Modeling for Direct Marketing, Proceedings of the Sixth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 465-473, Aug. 2000.8 *C Bounsaythip, E Rinta-Runsala "Overview of Data Mining for Customer Behavior Modeling"-Finland: VTT Information Technology, Research Report TTE1-18, 2001.9 *C Bounsaythip, E Rinta-Runsala "Overview of Data Mining for Customer Behavior Modeling"—Finland: VTT Information Technology, Research Report TTE1-18, 2001.10Cabena et al., Intelligent Miner for Data Applications Guide, IBM Redbook, SG24-5252-00, Mar. 1999, pp. 9-17, 27-35, 69-77, 87-108, 111-128.11Chapman et al., CRISP-DM 1.0 Step-by-Step Data Mining Guide, Aug. 2000, pp. 27-31, 53-59, 71-79.12Clifford C. Clogg, et al., "Multiple Imputation of Industry and Occupation Codes in Census Public-Use Samples Using Bayesian Logistic Regression", Journal of the American Statistical Association, vol. 86, No. 413, pp. 68-78 (Mar. 1991).13Commercial software: SPSS Clementine: www/spss.com/Clementine/index.htm.14Declaration filed in related U.S. Appl. No. 10/826,949, filed Apr. 16, 2004 along with claim chart.15Donald B. Rubin, "Using Empirical Bayes Techniques in the Law School Validity Studies", Journal of the American Statistical Association, vol. 75, No. 372, pp. 801-816 (Dec. 1980).16Eizion et al., e-CLV: A Modeling Approach for Customer Lifetime Evaluation in e-Commerce Domains, With an Application and Case Study for Online Auctions, IEEE International Conference on e-Technology, e-Commerce and e-Service, pp. 149-156, Mar. 2004.17ForecastPro Product Brochure (www.forecastpro.com), Business Forecasting Systems (2000), (Retrieved from Archive.org Dec. 20, 2005).18Harrison, H.C., "An Intelligent Business Forecasting System," ACM May 1993, pp. 229-236.19Haughton et al., "A Review of Software Packages for Data Mining," The American Statistician, Nov. 2003, vol. 57, No. 5, pp. 290-309.20International Preliminary Report on Patentability for App. Ser. No. PCT/US05/11749, dated Nov. 29, 2006, 8 pages.21New technology Card News, 10(6) (Apr. 3, 1995).22Pair Transaction History for U.S. Appl. No. 10/826,949, filed Apr. 16, 2004.23Pair Transaction History for U.S. Appl. No. 12/564,998, filed Sep. 23, 2009.24Pair Transaction History for U.S. Appl. No. 12/787,956, filed May 26, 2010.25Pearce et al., Experimentation and Self Learning in Continuous Database Marketing, IEEE International Conference on Data Mining, pp. 775-778, Dec. 2002.26Pinto, "New Marketing Technologies for a World of Product Proliferation," Fortelligent Inc., Whitepaper, 2 pages (2002).27Pinto, "Smart Data Use Can Improve Bank Insurance Marketing National Underwriter," 106(20) (Jul. 2002).28Pinto, "Using Customer Data to Dramatically Improve Marketing ROI," Fortelligent Whitepaper, 3 pages (2002).29Roderick J.A. Little. et al., Statistical Analysis with Missing Data (2nd Edition), John J. Wiley & Sons, Inc., (2002), pp. 19-22; 166-188; and 292-309.30SAS JMP, www.jmp.com.31Schikora et al., Efficacy of End-User Neural Network and Data Mining Software for Predicting Complex System Performance, International Journal of Production Economics, vol. 84, pp. 231-253, Jun. 2003.32SmartSoftware Web Pages (www.smartcorp.com), Smart Software Inc., (Jul. 2001) (Retrieved from Archive.org Dec. 20, 2005).33Tashman, "Automatic Forecasting Software: A Survey and Evaluation," International Journal of Forecasting, 7(2) (1991).34U.S. Appl. No. 10/826,452, filed Apr. 16, 2004.35U.S. Appl. No. 10/826,453, filed Apr. 16, 2004.36U.S. Appl. No. 10/826,624, filed Apr. 16, 2004.37U.S. Appl. No. 10/826,630, filed Apr. 16, 2004.38U.S. Appl. No. 10/826,711, filed Apr. 16, 2004.39U.S. Appl. No. 10/826,949, filed Apr. 16, 2004.40U.S. Appl. No. 10/826,950, filed Apr. 16, 2004.41U.S. Appl. No. 12/564,998, filed Sep. 23, 2009.42U.S. Appl. No. 12/787,956, filed May 26, 2010.43Wang et al., "An Expert System for Forecasting Model Selection," IEEE (1992).* Cited by examinerReferenced byCiting PatentFiling datePublication dateApplicantTitleUS8494995 *Nov 17, 2010Jul 23, 2013International Business Machines CorporationSystems and methods for dynamically reconfiguring predictive analytics within workflows using selective user feedbackUS8620802 *Sep 12, 2012Dec 31, 2013United Services Automobile Association (Usaa)Consumer-level financial performance analysisUS8738549 *Aug 19, 2011May 27, 2014International Business Machines CorporationPredictive modelingUS8751273May 26, 2010Jun 10, 2014Brindle Data L.L.C.Predictor variable selection and dimensionality reduction for a predictive modelUS8892452 *Nov 9, 2012Nov 18, 2014Hartford Fire Insurance CompanySystems and methods for adjusting insurance workflowUS20120123987 *Nov 17, 2010May 17, 2012International Business Machines CorporationSystems and methods for dynamically reconfiguring predictive analytics within workflows using selective user feedbackUS20120158624 *Aug 19, 2011Jun 21, 2012International Business Machines CorporationPredictive modelingUS20120323835 *Aug 28, 2012Dec 20, 2012International Business Machines CorporationDynamically reconfiguring predictive analytics within workflows using selective user feedbackUS20130073322 *Nov 9, 2012Mar 21, 2013Hartford Fire Insurance CompanySystems and methods for adjusting insurance workflow* Cited by examinerClassifications U.S. Classification703/2, 703/6, 700/44, 705/38, 700/97International ClassificationG06G7/60, G06Q10/00, G06F19/00, G06Q30/00Cooperative ClassificationG06Q30/02, G06Q10/04, G06Q40/025European ClassificationG06Q10/04, G06Q30/02, G06Q40/025Legal EventsDateCodeEventDescriptionAug 27, 2004ASAssignmentOwner name: FORTELLIGENT, MASSACHUSETTSFree format text: CORRECTIVE ASSIGNMENT TO CORRECT FIRST NAMED INVENTOR, PREVIOUSLY RECORDED ON REEL/FRAME 0150;ASSIGNORS:PINTO, STEPHEN K.;MANSFIELD, RICHARD;JACOBS, MARC;AND OTHERS;REEL/FRAME:015842/0923;SIGNING DATES FROM 20040823 TO 20040825Owner name: FORTELLIGENT, MASSACHUSETTSFree format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:PINT, STEPHEN K.;MANSFIELD, RICHARD;JACOBS, MARC;AND OTHERS;REEL/FRAME:015089/0786;SIGNING DATES FROM 20040823 TO 20040825Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:PINT, STEPHEN K.;MANSFIELD, RICHARD;JACOBS, MARC;AND OTHERS;SIGNING DATES FROM 20040823 TO 20040825;REEL/FRAME:015089/0786Free format text: CORRECTIVE ASSIGNMENT TO CORRECT FIRST NAMED INVENTOR, PREVIOUSLY RECORDED ON REEL/FRAME 015089/0786;ASSIGNORS:PINTO, STEPHEN K.;MANSFIELD, RICHARD;JACOBS, MARC;AND OTHERS;SIGNING DATES FROM 20040823 TO 20040825;REEL/FRAME:015842/0923Owner name: FORTELLIGENT, MASSACHUSETTSFree format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:PINT, STEPHEN K.;MANSFIELD, RICHARD;JACOBS, MARC;AND OTHERS;SIGNING DATES FROM 20040823 TO 20040825;REEL/FRAME:015089/0786Owner name: FORTELLIGENT, MASSACHUSETTSFree format text: CORRECTIVE ASSIGNMENT TO CORRECT FIRST NAMED INVENTOR, PREVIOUSLY RECORDED ON REEL/FRAME 015089/0786;ASSIGNORS:PINTO, STEPHEN K.;MANSFIELD, RICHARD;JACOBS, MARC;AND OTHERS;SIGNING DATES FROM 20040823 TO 20040825;REEL/FRAME:015842/0923Apr 8, 2005ASAssignmentFree format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:PINTO, STEPHEN K.;MANSFIELD, RICHARD;JACOBS, MARC;AND OTHERS;SIGNING DATES FROM 20040823 TO 20040825;REEL/FRAME:016039/0731Owner name: FORTELLIGENT, INC., MASSACHUSETTSFree format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:PINTO, STEPHEN K.;MANSFIELD, RICHARD;JACOBS, MARC;AND OTHERS;SIGNING DATES FROM 20040823 TO 20040825;REEL/FRAME:016039/0731Owner name: FORTELLIGENT, INC., MASSACHUSETTSFree format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:PINTO, STEPHEN K.;MANSFIELD, RICHARD;JACOBS, MARC;AND OTHERS;REEL/FRAME:016039/0731;SIGNING DATES FROM 20040823 TO 20040825Owner name: FORTELLIGENT, INC., MASSACHUSETTSFree format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:PINTO, STEPHEN K.;MANSFIELD, RICHARD;JACOBS, MARC;AND OTHERS;REEL/FRAME:016039/0731;SIGNING DATES FROM 20040823 TO 20040825Jul 13, 2011ASAssignmentOwner name: FORTELLIGENT LLC, MASSACHUSETTSFree format text: CHANGE OF NAME;ASSIGNOR:FORTELLIGENT, INC.;REEL/FRAME:026583/0709Effective date: 20061229Jan 23, 2012ASAssignmentOwner name: KNOWLEDGEBASE MARKETING, INC., TEXASFree format text: CHANGE OF NAME;ASSIGNOR:FORTELLIGENT LLC;REEL/FRAME:027574/0488Effective date: 20111228Jun 26, 2012CCCertificate of correctionOct 22, 2012ASAssignmentOwner name: BRINDLE DATA L.L.C., DELAWAREFree format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:KNOWLEDGEBASE MARKETING, INC.;REEL/FRAME:029169/0618Effective date: 20120913Owner name: KNOWLEDGEBASE MARKETING, INC., TEXASFree format text: CORRECTION TO THE COVER SHEET PREVIOUSLY RECORDED AT REEL 027574, FRAME 0488, CHANGING THE NATURE OF CONVEYANCE TO MERGER;ASSIGNOR:FORTELLIGENT LLC;REEL/FRAME:029170/0124Effective date: 20111228Nov 20, 2012CCCertificate of correctionRotateOriginal ImageGoogle Home - Sitemap - USPTO Bulk Downloads - Privacy Policy - Terms of Service - About Google Patents - Send FeedbackData provided by IFI CLAIMS Patent Services