Package: online-conformal
Improved Online Conformal Prediction via Strongly Adaptive Online LearningThis library implements numerous algorithms which perform conformal prediction on data with arbitrary distribution
shifts over time. This is the official implementation for thepaperBhatnagar et al.,
"Improved Online Conformal Prediction via Strongly Adaptive Online Learning," 2023. We include reference implementations
for the proposed methods Strongly Adaptive Online Conformal Prediction (SAOCP) and Scale-Free Online Gradient Descent
(SF-OGD), as well as Split Conformal Prediction (Vovk et al., 1999),
Non-Exchangeable Conformal Prediction (Barber et al., 2022), and
Fully Adaptive Conformal Inference (FACI,Gibbs & Candes, 2022).Replicating Our ExperimentsFirst install theonline_conformalpackage by cloning this repo and callingpip install ..
To run our time series forecasting experiments, first clone theMerlionrepo
and install theirts_datasetspackage. Then, you can callpythontime_series.py--model<model>--dataset<dataset>--njobs<njobs>where<model>can be one ofLGBM,ARIMA, orProphet;<dataset>can be one ofM4_Hourly,M4_Daily,M4_Weekly, orNN5_Daily; and<njobs>indicates the number of
parallel cores you wish to parallelize the file with. The results will be written to a sub-directoryresults.To run our experiments on image classification under distribution shift, first installPyTorch.
Then, you can callpythonvision.py--dataset<dataset>where dataset is one ofImageNetorTinyImageNet. Various intermediate results will be written to
sub-folders, and checkpointing (e.g. for model training) is automatic.Using Our CodeTo use our code, first install theonline_conformalpackage by callingpip install online_conformal.
You can alternatively install the package from source by cloning this repo and callingpip install ..Each online conformal prediction method is implemented as its own class in the package. All methods share a common API.
For time series forecasting, we leverage models implemented inMerlion.
Below, we demonstrate how to useSAOCPto create prediction intervals for multi-horizon time series forecasting.
The update loop is a simplified version of callingsaocp.forecast(time_series=test_data.iloc[:horizon], time_series_prev=train_data), whose implementation you can findhere.importpandasaspdfrommerlion.models.factoryimportModelFactoryfrommerlion.utilsimportTimeSeriesfromonline_conformal.datasetimportM4fromonline_conformal.saocpimportSAOCP# Get some time series data as pandas.DataFramesdata=M4("Hourly")[0]train_data,test_data=data["train_data"],data["test_data"]# Initialize a Merlion model for time series forecastingmodel=ModelFactory.create(name="LGBMForecaster")# Initialize the SAOCP wrapper on top of the model. This splits the data# into train/calibration splits, trains the model on the train split,# and initializes SAOCP's internal state on the calibration split.# The target coverage is 90% here, but you can adjust this freely.# We also do 24-step-ahead forecasting by setting horizon=24.horizon=24saocp=SAOCP(model=model,train_data=train_data,coverage=0.9,calib_frac=0.2,horizon=horizon)# Get the model's 24-step-ahead prediction, and convert it to prediction intervalsyhat,_=saocp.model.forecast(horizon,time_series_prev=TimeSeries.from_pd(train_data))delta_lb,delta_ub=zip(*[saocp.predict(horizon=h+1)forhinrange(horizon)])yhat=yhat.to_pd().iloc[:,0]lb,ub=yhat+delta_lb,yhat+delta_ub# Update SAOCP's internal state based on the next 24 observationsprev=train_data.iloc[:-horizon+1]time_series=pd.concat((train_data.iloc[-horizon+1:],test_data.iloc[:horizon]))foriinrange(len(time_series)):# Predict yhat_{t-H+i+1}, ..., yhat_{t-H+i+H} = f(y_1, ..., y_{t-H+i})y=time_series.iloc[i:i+horizon,0]yhat,_=saocp.model.forecast(y.index,time_series_prev=TimeSeries.from_pd(prev))yhat=yhat.to_pd().iloc[:,0]# Use h-step prediction of yhat_{t-k+h} to update SAOCP's h-step prediction intervalforhinrange(len(y)):ifi>=h:saocp.update(ground_truth=y[h:h+1],forecast=yhat[h:h+1],horizon=h+1)prev=pd.concat((prev,time_series.iloc[i:i+1]))For other use cases, you can initializesaocp = SAOCP(model=None, train_data=None, max_scale=max_scale, coverage=0.9).
Here,max_scaleindicates the largest value you expect the conformal score to take. Then, you can obtain the conformal
score corresponding to 90% (or your desired level of coverage) by callingscore = saocp.predict(horizon=1)[1], and
you can use this value to compute the prediction set{y: S(X_t, y) < score}using your own custom code. Finally, after
you observe the true conformal scorenew_score = S(X_t, Y_t), you can update the conformal predictor by callingsaocp.update(ground_truth=pd.Series([new_score]), forecast=pd.Series([0]), horizon=1).