C ONSERVATIVE P REDICTION VIA T RANSDUCTIVE C ONFIDENCE M INIMIZATION

Errors of machine learning models can be prohibitively costly, especially in safetycritical settings such as healthcare. However, machine learning may be applicable to such scenarios if the learned model can abstain and defer to a human on difficult examples instead of making errors. In safety-critical settings, we prefer conservative models that defer to humans at the cost of some overall accuracy. Unfortunately, selective classification and out-of-distribution detection are notably difficult as it is hard to anticipate all possible examples. To mitigate this challenge, we focus on the transductive setting, where unlabeled examples from the test distribution are available during training. We propose Transductive Confidence Minimization (TCM), which minimizes prediction confidence on unlabeled test examples while simultaneously optimizing the training objective. We theoretically show that TCM learns a lower bound on the true confidence, and that this property can be leveraged to provably detect examples that are sufficiently different from training examples, regardless of what distribution they came from. In our experiments, TCM consistently shows high performance, achieving the highest OOD detection performance compared to 6 other methods on 9 out of 10 ID→OOD pairs and consistently outperforming methods for selective classification in settings where we test on data from a previously unseen distribution. A PSEUDOCODE FOR TCM Algorithm 1 TCM for OOD Detection Input: Training data Dtr, Unlabeled data Du, Hyperparamter λ Initialize weights θ ← θ0 while Not converged do Sample mini-batch Btr ∼ Dtr Update θ using∇θLxent(f,Btr) while Not converged do Sample mini-batches Btr ∼ Dtr, Bu ∼ Du Update: ∇θLxent(f,Btr) + λLconf(f,Bu) Algorithm 2 TCM for Selective Classification Input: Training data Dtr, Validation data Dval, Hyperparameter λ Initialize weights θ ← θ0 while Not converged do Sample mini-batch Btr ∼ Dtr Update: ∇θLxent(Btr, f) Get correct set D◦ val ← {(x, y) ∈ Dval | fθ(x) = y} Get error set D× val ← {(x, y) ∈ Dval | fθ(x) ̸= y} while Not converged do Sample mini-batches Btr ∼ Dtr ∪D◦ val, B× val ∼ D × val Update: ∇θLxent(Btr, f) + λLconf(B val, f) B RELATED WORK Out-of-distribution detection and unlabeled data. Many existing methods for OOD detection use a criterion based on the activations or predictions of a model trained on ID data (Hendrycks & Gimpel, 2016; Liang et al., 2017b; Lee et al., 2018; Liu et al., 2020). However, as noted in Tajwar et al. (2021), the performance of these methods are not consistent across different ID-OOD dataset pairs, which suggests that the OOD detection problem may be too challenging in the absence of additional information. Our method leverages an unlabeled dataset which contains a mix of ID and OOD data, similarly to Ţifrea et al. (2020). However, this method requires an ensemble of models to measure disagreement, while TCM only uses a single model. Similarly to our method, Hendrycks et al. (2018) minimizes confidence on an unlabeled set, but they do so on one big dataset regardless of the OOD data, and the support of this dataset is disjoint with the support of most OOD distributions. We additionally present theoretical results showing the benefit of minimizing confidence on an unlabeled set that includes inputs from the OOD distribution. Our experiments provide further support for this claim, showing that this transductive setting results in substantial performance gains, even if the unlabeled set is a mixture of ID and OOD data. More generally, unlabeled data has been shown to be beneficial for performance, especially in conditions of distribution shift (Sagawa et al., 2021; Lee et al., 2022). Conservative prediction and selective classification. Prior works have studied selective classification, also known as reject option, for many model classes including SVM, boosting, and nearest neighbors (Hellman, 1970; Fumera & Roli, 2002; Cortes et al., 2016). Because deep neural networks generalize well but are often overconfident (Guo et al., 2017; Nixon et al., 2019), mitigating such overconfidence using selective classification while preserving its generalization properties is