Package: ppi-python
Prediction-powered inference (PPI) is a framework for statistically rigorous scientific discovery using machine learning.
Given a small amount of data with gold-standard labels and a large amount of unlabeled data, prediction-powered inference allows for the estimation of population parameters, such as the mean outcome, median outcome, linear and logistic regression coefficients.
Prediction-powered inference can be used both to produce better point estimates of these quantities as well as tighter confidence intervals and more powerful p-values.
The methods work both in the i.i.d. setting and for certain classes of distribution shifts.This package is actively maintained, and contributions from the community are welcome.Getting StartedIn order to install the package, runpipinstallppi-pythonThis will build and install the most recent version of the package.Warmup: estimating the meanTo test your installation, you can try running the prediction-powered mean estimation algorithm on thegalaxiesdataset.
The gold-standard labels and model predictions from the dataset will be downloaded into a folder called./data/.
The labels, $Y$, are binary indicators of whether or not the galaxy is a spiral galaxy.
The model predictions, $\hat{Y}$, are the model's estimated probability of whether the galaxy image has spiral arms.
The inference target is $\theta^* = \mathbb{E}[Y]$, the fraction of spiral galaxies.
You will produce a confidence interval, $\mathcal{C}^{\mathrm{PP}}_\alpha$, which contains $\theta^*$ with probability $1-\alpha=0.9$, i.e.,$$\mathbb{P}\left( \theta^* \in \mathcal{C}^{\mathrm{PP}}_\alpha\right) \geq 0.9.$$The code for this is below. It can be copy-pasted directly into the Python REPL.# Importsimportnumpyasnpfromppi_pyimportppi_mean_cifromppi_py.datasetsimportload_datasetnp.random.seed(0)# For reproducibility's sake# Download and load datasetdata=load_dataset('./data/',"galaxies")Y_total=data["Y"];Yhat_total=data["Yhat"]# Set up the inference problemalpha=0.1# Error raten=1000# Number of labeled data pointsrand_idx=np.random.permutation(Y_total.shape[0])Yhat=Yhat_total[rand_idx[:n]]Y=Y_total[rand_idx[:n]]Yhat_unlabeled=Yhat_total[n:]# Produce the prediction-powered confidence intervalppi_ci=ppi_mean_ci(Y,Yhat,Yhat_unlabeled,alpha=alpha)# Print the resultsprint(f"theta={Y_total.mean():.3f}, CPP={ppi_ci}")The expected results look as below $^*$:theta=0.259, CPP=(0.235677274705698, 0.26595223970754855)($^*$ these results were produced withnumpy=1.26.0, and may differ slightly due to randomness in other environments.)If you have reached this stage, congratulations! You have constructed a prediction-powered confidence interval.
Seethe documentationfor more usages of prediction-powered inference.ExamplesThe package somes with a suite of examples on real data:Proteomic Analysis with AlphaFold (alphafold.ipynb)Galaxy Classification with Computer Vision (galaxies.ipynb)Gene Expression Analysis with Transformers (gene_expression.ipynb)Deforestation Analysis with Computer Vision (forest.ipynb)Health Insurance Analysis with Boosting Trees (census_healthcare.ipynb)Income Analysis with Boosting Trees (covariate shift) (census_income_covshift.ipynb)Plankton Counting with Computer Vision (label shift) (plankton.ipynb)Ballot Counting with Computer Vision (ballots.ipynb)Income Analysis with Boosting Trees (census_income.ipynb)Usage and DocumentationThere is a common template that all PPI confidence intervals follow.ppi_[ESTIMAND]_ci(X,Y,Yhat,X_unlabeled,Yhat_unlabeled,alpha=0.1)You can replace[ESTIMAND]with the estimand of your choice. For certain estimands, not all the arguments are required, and in this case, they are omitted. For example, in the case of mean estimation, the function signature is:ppi_mean_ci(Y,Yhat,Yhat_unlabeled,alpha=0.1)All the prediction-powered point estimates and confidence intervals implemented so far can be imported by runningfrom ppi_py import ppi_[ESTIMAND]_pointestimate, ppi_[ESTIMAND]_ci. For the case of the mean, one can also import the p-value asfrom ppi import ppi_mean_pval.Full documentation is availablehere.Repository structureThe repository is organized into three main folders:./ppi_py/./examples/./tests/The first foler,./ppi_py, contains all the code that eventually gets compiled into theppi_pypackage. Most importantly, there is a file,./ppi_py/ppi.py, which implements all the prediction-powered point estimates, confidence intervals, and p-values for different estimators.
There is also a file,./ppi_py/baselines.py, which implements several baselines.
Finally, the file./ppi_py/datasets/datasets.pyhandles the loading of the sample datasets.The folder./examplescontains notebooks for implementing prediction-powered inference on several datasets and estimands. These are listedabove. There is also an additional subfolder,./examples/baselines, which contains comparisons to certain baseline algorithms, as in the appendix of the original PPI paper.The folder./testscontains unit tests for each function implemented in theppi_pypackage. The tests are organized by estimand, and can be run by executingpytestin the root directory. Some of the tests are stochastic, and therefore, have some failure probability, even if the functions are all implemented correctly. If a test fails, it may be worth running it again. Debugging the tests can be done by adding the-sflag and using print statements orpdb. Note that in order to be recognized bypytest, all tests must be preceded bytest_.The remainder of the files/folders are boilerplate and not relevant to most users.ContributingThank you so much for considering making a contribution toppi_py; we deeply value and appreciate it.The contents of this repository will be pushed to PyPI whenever there are substantial revisions. If there are methods or examples within the PPI framework you'd like to see implemented, feel free to suggest them on theissues page. Community contributions are welcome and encouraged as pull requests directly onto the main branch. The main criteria for accepting such pull requests is:The contribution should align with the repository's scope.All new functionality should be tested for correctness within our existingpytestframework.If the pull request involves a new PPI method, it should have a formal mathematical proof of validity which can be referenced.If the pull request solves a bug, there should be a reproducible bug (within a specific environment) that is solved. Bug reports can be made on the issues page.The contribution should bewell documented.The pull request should be of generally high quality, up to the review of the repository maintainers.
The repository maintainers will approve pull requests at their discretion. Before working on one, it may be helpful to post a question on the issues page to verify if the contribution would be a good candidate for merging into the main branch.Accepted pull requests will be run through an automatedBlackformatter, so contributors may want to run Black locally first.