Package: comparecast
comparecastCode and Python package accompanying our paper,Comparing Sequential Forecasters,
published toOperations Research(2023).
A preprint version is also available onarXiv.In the paper, we derive anytime-valid and distribution-free inference procedures for comparing sequential forecasters that each make a prediction over a sequence of events.
This accompanying package, namedcomparecast, includes implementations of:confidence sequence (CS): a sequence of confidence intervals that estimate the average score difference between forecasters A and B (which may vary over time);e-process: a sequence of evidences against the null hypothesis that forecaster A scores no better than forecaster Bon average.Confidence sequences can be continuously monitored without having to worry about "sampling to a foregone conclusion," which happens when using a standard, fixed-time confidence interval and repeatedly peeking at the data.
E-processes also allow peeking at the data before deciding when to stop the experiment ("anytime-valid").
Seeherefor an overview of these methods, orherefor a standalone implementation of confidence sequences.In the plot below, we plot the 95% (empirical-Bernstein) CS and its corresponding e-processes (one for each forecaster) for
comparing the probability forecasts ofFiveThirtyEightandVegas oddson all Major League Baseball games from 2010 to 2019.
The averageBrier scoreslightly favors Vegas odds over time,
and the CS closely tracks this average score and represents the uncertainty in estimating the score while accounting for the randomness of data.Seenb_comparecast_baseball.ipynbfor the code that generated this plot.InstallationRequires Python 3.7+.Frompip:pipinstall--upgradepip
pipinstall--upgradepandasseaborntqdmconfseq
pipinstall--upgradecomparecastFrom source:gitclonehttps://github.com/yjchoe/ComparingForecasterscdComparingForecasters

pipinstall--upgradepip
pipinstall-rrequirements.txt
pipinstall-e.Data SourcesSeedata/README.md.Sample UsageAlso see experiment notebooks below.PythonThe following sample code generates some simulated data and forecasters.
Then, it computes the CS and the e-processes for comparing two of these forecasters,
and saves the relevant plots inside theplots/testdirectory.importcomparecastascc# Generate/retrieve synthetic datadata=cc.data_utils.synthetic.get_data("default",size=1000)# Calculate, save, and plot the forecastsforecasters=["k29_poly3","laplace","constant_0.5"]data=cc.forecast(data,forecasters,out_file="data/test.csv")cc.plot_forecasts(data,forecasters,plots_dir="plots/test")# Compare forecasts using confidence sequences & e-valuesresults=cc.compare_forecasts(data,"k29_poly3","laplace",scoring_rule="brier",alpha=0.05,compute_cs=True,compute_e=True,)# returns a pandas DataFrameresults.tail(5)#      time       lcb       ucb         e_pq      e_qp# 995   996  0.012868  0.072742  2025.725774  0.021684# 996   997  0.013050  0.072879  2157.262456  0.021672# 997   998  0.012635  0.072492  1886.687861  0.021596# 998   999  0.012824  0.072637  2013.209084  0.021583# 999  1000  0.012447  0.072275  1783.204679  0.021519# Draw a comparison plot and save in plots/test/*.pdfresults,axes=cc.plot_comparison(data,"k29_poly3","laplace",scoring_rule="brier",alpha=0.05,baselines=("h","acs"),plot_e=True,plot_width=True,plots_dir="plots/test",)Command Line Interface(Clone this repo first.)# Generate synthetic data and forecastspython3forecast.py-ddefault-n1000-fall\-oforecasts/test.csv-pplots/test# Compare forecasts and plot resultspython3plot_comparisons.py-dforecasts/test.csv\-pk29_poly3-qlaplace--baselineshacs-oplots/test# Compare 538 and vegas forecasterspython3plot_comparisons.py-dforecasts/mlb_2010_2019.csv\-pfivethirtyeight-qvegas--baselinesacs-oplots/test/mlb_2010_2019\--ylim-scale0.01ExperimentsMain experiments (appearing in the main paper):nb_comparecast_synthetic.ipynb:
Experiments on synthetic data and forecasts.
Includes comparison with a fixed-time CI.
Section 5.1 in our paper.nb_comparecast_scoringrules.ipynb:
Experiments on synthetic data and forecasts using different scoring rules.
Section 5.1 (Figure 4) in our paper.nb_comparecast_baseball.ipynb:
Experiments on Major League Baseball forecasts,
leading up to the 2019 World Series.
Section 5.2 in our paper.nb_comparecast_weather.ipynb:
Experiments on postprocessing methods for ensemble weather forecasts.
Includes e-value comparison withHenzi & Ziegel (2021).
Section 5.3 in our paper.Additional experiments (some appearing in the E-Companion/Appendix of our paper):nb_comparecast_dmgw_miscoverage.ipynb:
Validity comparison with classical comparison methods (DM & GW).nb_comparecast_comparison_with_dm_power.ipynb:
"Power" comparison with classical comparison methods (DM & GW).nb_comparecast_weather_eda.ipynb:
Exploratory plots on the ensemble weather forecast dataset.nb_iid_mean.ipynb:
Comparison of uniform boundaries on the mean of IID data.
Partly reproduces Figure 1 fromHoward et al. (2021).nb_cgf_convexity.ipynb:
Illustration of the Exponential CGF-like function.nb_eprocess_ville.ipynb:
Illustrating some properties of (sub-exponential) e-/p-processes
in the context of game-theoretic statistical inference.
Not used in our paper.Code LicenseMITAuthorsYJ ChoeandAaditya RamdasReferencesIf you use parts of our work, please cite our paper as follows:Text:Choe, Y. J., & Ramdas, A. (2023). Comparing sequential forecasters.Operations Research.https://doi.org/10.1287/opre.2021.0792BibTeX:@article{choe2023comparing,title={Comparing sequential forecasters},author={Choe, Yo Joong and Ramdas, Aaditya},journal={Operations Research},year={2023},doi={https://doi.org/10.1287/opre.2021.0792},publisher={INFORMS}}