Package: predictionscorer
Python Prediction ScorerPython Prediction ScorerMotivationStatusGoalsBackgroundInstallationSystem requirementsUsageChoice predictionsBrierLogarithmicPracticalQuadraticChangelogGoals and judgment callsFeature completeEasy to read (for as many people as possible)Easy and predictable to use with modern toolsContributingTable of contents generated with markdown-tocMotivationThe purpose of this library is to score predictions.BackgroundSome of the code in this library comes from my work atEmpiricast, a forecasting startup I co-founded.For a thorough introduction to scoring rules, seeCalibration Scoring Rules for Practical Prediction TrainingbySpencer Greenberg.Features4 scoring rules for choice predictions:BrierLogarithmicPracticalQuadraticFully type hinted100 percent test coverageREST APIPython Prediction Scorer is also available as a REST API. This is useful if you are not using Python or if you are using a Python version that we don’t support. The documentation is available onhttps://predictionscorer.herokuapp.com/docs.Installationpip install predictionscorerSystem requirementsPython Prediction Scorer requires Python 3.8.UsageChoice predictionsForchoice predictions, the forecaster assigns probabilities to different answers. As an example, let’s say that George and Kramer made the following forecasts for the result of a game where the home team ended up winning:ResultGeorgeKramerCorrectHome team wins40 %65 %YesTie30 %10 %NoAway team wins30 %25 %NoKramer assigned a higher probability to the correct answer than George did, so his forecast was better. But how much better? In order to find out, we mustquantify the qualityof their predictions. That’s what this library does.We have fourscoring rulesto determine this:BrierLogarithmicPracticalQuadraticLet us look at each of them.BrierBrier scores range from 0 to 2. Lower is better.frompredictionscorer.rulesimportbrier_scoregeorge_probability=0.4kramer_probability=0.65george_score=brier_score(george_probability)# 0.72kramer_score=brier_score(kramer_probability)# 0.2450LogarithmicLogarithmic scores range from approaching infinity (worst) to 0 (best):frompredictionscorer.rulesimportlogarithmic_scoregeorge_probability=0.4kramer_probability=0.65george_score=logarithmic_score(george_probability)# 1.32kramer_probability=logarithmic_score(kramer_probability)# 0.62PracticalPractical scores range from approaching negative infinity (worst) to a configurable maximum — we use 2:frompredictionscorer.rulesimportpractical_scoregeorge_probability=0.4kramer_probability=0.65george_score=practical_score(0.4)# -0.64kramer_score=practical_score(0.65)# 0.76practical_scoreaccepts two optional parameters:NameDefaultmax_practical_scoreDecimal(2)max_probabilityDecimal("0.9999")QuadraticQuadratic scores range from -1 (worst) to 1 (best):frompredictionscorer.rulesimportquadratic_scoregeorge_probability=0.4kramer_probability=0.65george_score=quadratic_score(george_probability)# 0.28kramer_score=quadratic_score(kramer_probability)# 0.76ChangelogSeeCHANGELOG.md.ContributingPleaseopen an issue on GitHubif you discover any problems or potential for improvement. They are very welcome. Comments on the API design are especially useful at this point.Also, seeCONTRIBUTING.md.