[17] Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, Tony Lee, Etienne David, Ian Stavness, Wei Guo, Berton A. Earnshaw, Imran S. Haque, Sara Beery, Jure Leskovec, Anshul Kundaje, Emma Pierson, Sergey Levine, Chelsea Finn, and Percy Liang. WILDS: A Benchmark of in-the-Wild Distribution Shifts, 2021.
[18] Ananya Kumar, Percy S Liang, and Tengyu Ma. Verified Uncertainty Calibration. Advances in Neural Information Processing Systems, 32, 2019.
[19] Tomo Lazovich, Luca Belli, Aaron Gonzales, Amanda Bower, Uthaipon Tantipongpipat, Kristian Lum, Ferenc Huszar, and Rumman Chowdhury. Measuring Disparate Outcomes of Content Recommendation Algorithms with Distributional Inequality Metrics. *Patterns*, 3(8):100568, 2022.
[20] Jean-Yves Le Boudec. Rate Adaptation, Congestion Control and Fairness: A Tutorial. Web Page, November, 4, 2005.
[21] Amit Moscovich, Boaz Nadler, and Clifford Spiegelman. On the Exact Berk-Jones Statistics and Their  $p$ -Value Calculation. Electronic Journal of Statistics, 10(2):2329-2354, 2016.
[22] John Platt et al. Probabilistic Outputs for Support Vector Machines and Comparisons to Regularized Likelihood Methods. Advances in Large Margin Classifiers, 10(3):61-74, 1999.
[23] R Tyrrell Rockafellar, Stanislav Uryasev, et al. Optimization of Conditional Value-at-Risk. Journal of Risk, 2:21-42, 2000.
[24] Yaniv Romano, Rina Foygel Barber, Chiara Sabatti, and Emmanuel Candès. With Malice Toward None: Assessing Uncertainty via Equalized Coverag. Harvard Data Science Review, 2(2):4, 2020.
[25] Yaniv Romano, Stephen Bates, and Emmanuel J. Candès. Achieving Equalized Odds by Resampling Sensitive Attributes. arXiv:2006.04292, June 2020.
[26] Halsey Lawrence Royden and Patrick Fitzpatrick. Real Analysis, volume 32. Macmillan New York, 1988.
[27] Glenn Shafer and Vladimir Vovk. A Tutorial on Conformal Prediction. Journal of Machine Learning Research, 9(3), 2008.
[28] Galen R. Shorack and Jon A. Wellner. Empirical Processes with Applications to Statistics. Society for Industrial and Applied Mathematics, January 2009.
[29] Anthony F Shorrocks. The Class of Additively Decomposable Inequality Measures. Econometrica: Journal of the Econometric Society, pages 613-625, 1980.
[30] Jake C Snell, Thomas P Zollo, Zhun Deng, Toniann Pitassi, and Richard Zemel. Quantile Risk Control: A Flexible Framework for Bounding the Probability of High-Loss Predictions. International Conference on Learning Representations (ICLR), 2023.
[31] James Taylor, Berton Earnshaw, Ben Mabey, Mason Vectors, and Jason Yosinski. Rrx1: An Image Set for Cellular Morphological Variation Across Many Experimental Batches. In International Conference on Learning Representations (ICLR), volume 22, page 23, 2019.
[32] Robert Williamson and Aditya Menon. Fairness Risk Measures. In International conference on machine learning, pages 6786-6797. PMLR, 2019.
[33] Shlomo Yitzhaki. Relative Deprivation and the Gini Coefficient. The Quarterly Journal of Economics, 93(2):321-324, 1979.
[34] Shlomo Yitzhaki and Edna Schechtman. The Gini Methodology: a Primer on a Statistical Methodology. Springer, 2013.