Package: torchcp
TorchCP is a Python toolbox for conformal prediction research on deep learning models, using PyTorch. Specifically, this
toolbox has implemented some representative methods (including posthoc and training methods) for
classification and regression tasks. We build the framework of TorchCP based
onAdverTorch. This codebase is still under construction and
maintained byHongxin Wei's research group at SUSTech.
Comments, issues, contributions, and collaborations are all welcomed!OverviewTorchCP has implemented the following methods:ClassificationYearTitleVenueCode LinkImplementation2023Class-Conditional Conformal Prediction with Many ClassesNeurIPSLinkclassification.predictors.cluster2023Conformal Prediction for Deep Classifier via Label RankingArxivLinkclassification.scores.saps2021Learning Optimal Conformal ClassifiersICLRLinkclassification.loss.conftr2020Uncertainty Sets for Image Classifiers using Conformal PredictionICLRLinkclassification.scores.raps2020Classification with Valid and Adaptive CoverageNeurIPSLinkclassification.scores.aps2019Conformal Prediction Under Covariate ShiftNeurIPSLinkclassification.predictors.weight2016Least Ambiguous Set-Valued Classifiers with Bounded Error LevelsJASAclassification.scores.thr2015Bias reduction through conditional conformal predictionIntell. Data Anal.classification.scores.margin2013Applications of Class-Conditional Conformal Predictor in Multi-Class ClassificationICMLAclassification.predictors.classwiseRegressionYearTitleVenueCode LinkImplementation2023Conformal Prediction via Regression-as-ClassificationRegML @ NeurIPS 2023linkregression.predictors.r2ccp2021Adaptive Conformal Inference Under Distribution ShiftNeurIPSLinkregression.predictors.aci2019Conformalized Quantile RegressionNeurIPSLinkregression.predictors.cqr2016Distribution-Free Predictive Inference For RegressionJASALinkregression.predictors.splitTODOTorchCP is still under active development. We will add the following features/items down the road:YearTitleVenueCode Link2022Training Uncertainty-Aware Classifiers with Conformalized Deep LearningNeurIPSLink2022Adaptive Conformal Predictions for Time SeriesICMLLink2022Predictive Inference with Feature Conformal PredictionICLRLink2022Conformal Prediction Sets with Limited False PositivesICMLLink2021Optimized conformal classification using gradient descent approximationArxivInstallationTorchCP is developed with Python 3.9 and PyTorch 2.0.1. To install TorchCP, simply runpip install torchcpTo install from TestPyPI server, runpip install --index-url https://test.pypi.org/simple/ --no-deps torchcpExamplesHere, we provide a simple example for a classification task, with THR score and SplitPredictor.fromtorchcp.classification.scoresimportTHRfromtorchcp.classification.predictorsimportSplitPredictor# Preparing a calibration data and a test data.cal_dataloader=...test_dataloader=...# Preparing a pytorch modelmodel=...model.eval()# Options of score function: THR, APS, SAPS, RAPS# Define a conformal prediction algorithm. Optional: SplitPredictor, ClusterPredictor, ClassWisePredictorpredictor=SplitPredictor(score_function=THR(),model=model)# Calibrating the predictor with significance level as 0.1predictor.calibrate(cal_dataloader,alpha=0.1)########################################## Predicting for test instances########################################test_instances=...predict_sets=predictor.predict(test_instances)print(predict_sets)########################################## Evaluating the coverage rate and average set size on a given dataset.########################################result_dict=predictor.evaluate(test_dataloader)print(result_dict["Coverage_rate"],result_dict["Average_size"])You may find more tutorials inexamplesfolder.DocumentationThe documentation webpage is on readthedocshttps://torchcp.readthedocs.io/en/latest/index.html.LicenseThis project is licensed under the LGPL. The terms and conditions can be found in the LICENSE and LICENSE.GPL files.CitationIf you find our repository useful for your research, please consider citing the followingtechnical report:@misc{wei2024torchcp,
      title={TorchCP: A Library for Conformal Prediction based on PyTorch}, 
      author={Hongxin Wei and Jianguo Huang},
      year={2024},
      eprint={2402.12683},
      archivePrefix={arXiv},
      primaryClass={cs.LG}
}We welcome you to cite the following works:@article{huang2023conformal,
  title={Conformal Prediction for Deep Classifier via Label Ranking},
  author={Huang, Jianguo and Xi, Huajun and Zhang, Linjun and Yao, Huaxiu and Qiu, Yue and Wei, Hongxin},
  journal={arXiv preprint arXiv:2310.06430},
  year={2023}
}

@article{xi2024does,
  title={Does Confidence Calibration Help Conformal Prediction?},
  author={Xi, Huajun and Huang, Jianguo and Feng, Lei and Wei, Hongxin},
  journal={arXiv preprint arXiv:2402.04344},
  year={2024}
}ContributorsHongxin WeiJianguo HuangXuanning Zhou