Package: conditionalconformal
Conditional Conformalconditionalconformalis a Python package for conformal prediction with
conditional guarantees.For example, given a collection of groups $\mathcal{G}$,conditionalconformalissues
a prediction set $\hat{C}(\cdot)$ satisfying$$\mathbb{P}(Y_{n + 1} \in \hat{C}(X_{n + 1}) \mid X \in G) \geq 1 - \alpha \quad \text{for all $G \in \mathcal{G}$}.$$Alternatively, given a collection of covariate shifts $\mathcal{F}$, the package issues
a prediction set $\hat{C}(\cdot)$ satisfying$$\mathbb{P}_ f(Y_{n + 1} \in \hat{C}(X_{n + 1})) \geq 1 - \alpha \quad \text{for all $f \in \mathcal{F}$}.$$If the collection of shifts is unknown, we also provide a methodology
for providing finite-sample guarantees over arbitrary shifts. To query for the guarantee (which can
depend on the shift of interest), we provide theestimate_coveragefunction.Installationconditionalconformal can be installed with pip.$pipinstallconditionalconformalExamplesThe easiest way to start using conditionalconformal may be to go through the following notebook:Synthetic DataUsageTheCondConfClassTheCondConfclass has the following API:classCondConf:def__init__(self,score_fn:Callable,Phi_fn:Callable,infinite_params:dict={}):"""Constructs the CondConf object that caches relevant information forgenerating conditionally valid prediction sets.We define the score function and set of conditional guaranteesthat we care about in this function.Parameters---------score_fn : Callable[np.ndarray, np.ndarray] -> np.ndarrayFixed (vectorized) conformity score function that takes inX and Y as inputs and returns S as outputPhi_fn : Callable[np.ndarray] -> np.ndarrayFunction that defines finite basis set that we provideexact conditional guarantees overinfinite_params : dict = {}Dictionary containing parameters for the RKHS component of the fitValid keys are ('kernel', 'gamma', 'lambda')'kernel' should be a valid kernel name for sklearn.metrics.pairwise_kernels'gamma' is a hyperparameter for certain kernels'lambda' is the regularization penalty applied to the RKHS component"""defsetup_problem(self,x_calib:np.ndarray,y_calib:np.ndarray):"""setup_problem sets up the final fitting problem for aparticular calibration setThe resulting cvxpy Problem object is stored inside the CondConf parent.Arguments---------x_calib : np.ndarrayCovariate data for the calibration sety_calib : np.ndarrayLabels for the calibration set"""defpredict(self,quantile:float,x_test:np.ndarray,score_inv_fn:Callable,S_min:float=None,S_max:float=None):"""Returns the (conditionally valid) prediction set for a giventest pointArguments---------quantile : floatNominal quantile levelx_test : np.ndarraySingle test pointscore_inv_fn : Callable[float, np.ndarray] -> .Function that takes in a score threshold S^* and test point x andoutputs all values of y such that S(x, y) <= S^*S_min : float = NoneLower bound (if available) on the conformity scoresS_max : float = NoneUpper bound (if available) on the conformity scoresReturns-------prediction_set"""defestimate_coverage(self,quantile:float,weights:np.ndarray,x:np.ndarray=None):"""estimate_coverage estimates the true percentile of the issued estimate of theconditional quantile under the covariate shift induced by 'weights'If we are ostensibly estimating the 0.95-quantile using an RKHS fit, we maydetermine using our theory that the true percentile of this estimate is only 0.93Arguments---------quantile : floatNominal quantile levelweights : np.ndarrayRKHS weights for tilt under which the coverage is estimatedx : np.ndarray = NonePoints for which the RKHS weights are defined. If None, we assumethat weights corresponds to x_calibReturns-------estimated_alpha : floatOur estimate for the realized quantile level"""defpredict_naive(self,quantile:float,x:np.ndarray,score_inv_fn:Callable):"""If we do not wish to include the imputed data point, we can sanity check thatthe regression is appropriately adaptive to the conditional variability in the databy running a quantile regression on the calibration set without any imputation.When n_calib is large and the fit is stable, we expect these two sets to nearly coincide.Arguments---------quantile : floatNominal quantile levelx : np.ndarraySet of points for which we are issuing prediction setsscore_inv_fn : Callable[np.ndarray, np.ndarray] -> np.ndarrayVectorized function that takes in a score threshold S^* and test point x andoutputs all values of y such that S(x, y) <= S^*Returns-------prediction_sets"""defverify_coverage(self,x:np.ndarray,y:np.ndarray,quantile:float):"""In some experiments, we may simply be interested in verifying the coverage of our method.In this case, we do not need to binary search for the threshold S^*, but only need to verify thatS <= f_S(x) for the true value of S. This function implements this check for test pointsdenoted by x and yArguments---------x : np.ndarrayA vector of test covariatesy : np.ndarrayA vector of test labelsquantile : floatNominal quantile levelReturns-------coverage_booleans : np.ndarray"""CitingThis code is available for use under the MIT license.
If you use this code in a research project, please cite the forthcoming paper.@article{gibbs2023conformal,
    title={Conformal Prediction with Conditional Guarantees},
    author={Isaac Gibbs, John J. Cherian, Emmanuel J. Cand\`es},
    publisher = {arXiv},
    year = {2023},
    note = {arXiv:2305.12616 [stat.ME]},
    url = {https://arxiv.org/abs/2305.12616},
}