Aligning Model Properties via Conformal Risk Control
Authors: William Overman
Jacqueline Jil Vallon
Mohsen Bayati
Venue: NeurIPS 2024 poster (2024)
Keywords: Alignment
Conformal Prediction
Conformal Risk Control
Property Testing
Primary area: safety_in_machine_learning
TL;DR: Proposes a novel approach for post-training model alignment using conformal risk control, defining alignment using ideas from property testing

Abstract: AI model alignment is crucial due to inadvertent biases in training data and the underspecified machine learning pipeline, where models with excellent test metrics may not meet end-user requirements. While post-training alignment via human feedback shows promise, these methods are often limited to generative AI settings where humans can interpret and provide feedback on model outputs. In traditional non-generative settings with numerical or categorical outputs, detecting misalignment through single-sample outputs remains challenging, and enforcing alignment during training requires repeating costly training processes.
In this paper we consider an alternative strategy. We propose interpreting model alignment through property testing, defining an aligned model $f$ as one belonging to a subset $\mathcal{P}$ of functions that exhibit specific desired behaviors. We focus on post-processing a pre-trained model $f$ to better align with $\mathcal{P}$ using conformal risk control. Specifically, we develop a general procedure for converting queries for testing a given property $\mathcal{P}$ to a collection of loss functions suitable for use in a conformal risk control algorithm. We prove a probabilistic guarantee that the resulting conformal interval around $f$ contains a function approximately satisfying $\mathcal{P}$. We exhibit applications of our methodology on a collection of supervised learning datasets for (shape-constrained) properties such as monotonicity and concavity. The general procedure is flexible and can be applied to a wide range of desired properties. Finally, we prove that pre-trained models will always require alignment techniques even as model sizes or training data increase, as long as the training data contains even small biases.

Aligning Model Properties via Conformal Risk Control
William Overman'
Jacqueline Jil Vallon
Mohsen Bayati'
Stanford Graduate School of Business (rpp.,bayati)@staafoor edu.
Management Science and Engineering jjvallon @ealummisstanfrr..co
Abstract
AI model alignment is crucial due inadvertent biases in training data and the underspecified machine learning pipeline. where models with excellent test metrics may not meet end-user requirements. While post-training alignment via human feedback shows promise, these methods are often limited to generative Al settings where humans can interpret and provide feedback on model outputs. traditional non-generative settings with numerical or categorical outputs, detecting misalign- ment through single-sample outputs remains challenging, and enforcing alignment during training requires repeating costly training processes. In this paper we con- sider an alternative strategy. We propose interpreting model alignment through property testing, defining aligned model / as one belonging to a subset P of functions that exhibit specific desired behaviors. We focus on post-processing a pre-trained model to better align with P using conformal risk control. Specif- ically, we develop a general procedure for converting queries for testing a given property P to collection of loss functions suitable for in conformal risk control algorithm. We prove a probabilistic guarantee that the resulting conformal interval around contains function approximately satisfying P. We exhibit applications our methodology on a collection of supervised learning datasets for (shape-constrained) properties such monotonicity and concavity. The general procedure is flexible and can be applied to a wide range of desired properties. Finally, prove that pre-trained models will always require alignment techniques even as model sizes or training data increase, as long as the training data contains even small biases.
Introduction
The emergence of large foundation models has increased the attention to the problem of alignment.
Aligned models are artificial intelligences designed to pursue goals that align with human values.
principles, and intentions (Leike et al., 2018 Ouyang et al.] 2022; Hendrycks et al., 2023) Ngo al. 2024). Although the alignment problem is predominantly examined in the context of potential artificial general intelligence (AGI), large language models (LLMss, and reinforcement learning (RL) agents, it also has roots in the modern machine learning pipeline (D' Amour al.) (022). Motivated by this, we introduce a broader notion of alignment in this paper, extending beyond the aforementioned generative models to include even tabular regression models.
As an example, consider a regression task where property P represents models that are monotonically decreasing given feature (covariate). For example, predicting cancer patient survival should be monotonically decreasing in cancer stage (Vallon et al. 2022, 2024). Constraining prediction model during training maintain monotonicity in this feature can be viewed as form of alignment. For pre-trained model ff that was trained without such constraints, ensuring monotonically decreasing predictions this feature can be more complex. This complexity arises particularly non-genere
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

settings where the user cannot update f or obtain any outputs other than point predictions f(X) for given input X.
In this work, we propose an approach to aligning a pre-trained model f that motivated by property testing (Ron 2008 Goldreich| 2017) and conformal risk control (Angelopoulos et al. (024). Property testing aims to design efficient algorithms for determining membership to the set 7 of functions with given property, that require fewer resources than learning algorithms for d (Ron 2008). This is
particularly relevant for modern deep learning, where a user may need to determine if a pre-trained model f belongs to without the resources to train model of comparable size.
Property testing algorithms use local queries to determine, with high probability, whether a function has given global property or is far from having it. We map such queries for property P to set of loss functions, which we then use in conformal risk control procedure (Angelopoulos et al.) 2024) to establish a notion of alignment for P. We prove that this procedure yields conformal interval around/ f containing a function close to P.
We demonstrate our methodology real-world datasets for the properties of monotonicity and concavity, Motivated by the potential for systematic under- or over-estimation bias in f. we provide a straightforward extension of Angelopoulos et al. (2024) to obtain asymmetric conformal intervals with multi multi-dimennional parameters. While we examine both monotonicity and concavity constraints, the majority of our focus is on monotonicity, as these constraints have been shown promote crucial aspects of alignment to human values, such as fairness and adherence to social norms (Wang and Gupta 2020).
While our methodology provides a way to align pre-trained models, one may question whether such techniques will remain necessary as Al capabilities advance. Given the outstanding capabilities of modern AI models with substantially large numbers of parameters and training data, one may argue that the alignment problem may naturally disappear such advances continue (Kaplan et al. 2020). However, another contribution of this paper is to refute this argument a stylized setting, building on recent advances in the theory of linearized neural networks (Mei and Montanari; Misiakiewiez and Montanari, 2023). Specifically, we show that increasing the size of the training data or the number of parameters in random feature model (a theoretically tractable neural network proxy where hidden layer weights are randomly initialized and fixed (Rahimi and Recht (007)) cannot help it satisfy a property P, if the pre-training data has biased labels. Our simulations show that the result holds even only a small fraction of the training labels are impacted by the bias.
Summarizing our main contributions, we: (1) introduce an alignment perspective based on property testing, (2) use conformal risk control to post-process predictions pre-trained models for better alignment, and (3) demonstrate that increasing training data and parameters in random feature model does not eliminate the need for alignment. We discuss related work in Section 6) particularly connections to Yadkori et al. (2024). who use conformal risk control to address large language model hallucinations (Ji et al. 2023).
Preliminaries
In this section we provide key definitions drawn from property testing as well as a condensed overview to conformal prediction and conformal risk control. We provide short introduction propery testing in the extended version of the paper Overman et al. (2024) and an extensive introduction to the field can be found in Goldreich (2017).
2.1 Properties and Property Testing for Set- Valued Functions
Our perspective on alignment in this work is motivated by the field of property testing (Goldreich 2017 Ron 2008). Property testing studies algorithms that, by making small number of local queries to a large object (such as a function or a graph), can determine whether the object has a certain property is significantly far from having it.
Classic examples include linearity testing of Boolean functions (Blum et al. 1993). testing whether a function is a low-degree polynomial (Kaufman and Ron 2006 (Bhattacharyya et al. 2009). and testing k-juntas (Blais) 2009). These algorithms generally operate by randomly sampling and querying the object, leveraging local information infer global properties.

---

In this work, we focus on set-valued functions, which are functions that map elements of a domain X' to subsets of a codomain ) i.e.,, F: :x= + 2 While the standard definitions of property testing are technically sufficient for our purposes- since we can consider set-valued functions as functions with range 2'"-we we introduce specialized definitions to maintain clarity and to facilitate the transition between discussing )' and 2
Definition (Satisfying and Accommodating a Property). Let property P denote a specific subset of all functions that map X' to y. A function f X + y satisfies the property P if f E P.
functioo A set-valued function F: X + 2d accommodates property P if there exists a function g P such
that g(x) E F(x) for all a E X.
Intuitively, accommodates P if it contains at least one function satisfying P within its possible outputs.
We extend the notion of e-farness from a property (as defined in the extended version of the paper Overman al. (2024) to set-valued functions. For set- val-vel functions, we measure the distance based on how often the outputs of any function E P fall within the sets provided by F.
Definition 2 (e-Faraway). For a set-valued function F F - 2 distribution 'D over X, E > 0, and a property P, we say F is e-Faraway from P with respect to 'D if (F) > e, where
(F) def inf (D( and 8p((.g) gEP
def
Pr (x() & F(X)). X
Note. Throughout this work, we assume that D is the empirical distribution of a fixed and finite calibration dataset, and thus has finite support. While this assumption is not strictly necessary, most property testing results are over finite domains. Property testing over functions with Euclidean domains is in general a difficult problem. though there have been notable recent successes (Fleming Arora et al., 2023)..
and Yoshida 2020:
With these definitions in place, we can define testers for set-valued functions. We focus one-sided error testers, which are algorithms that take in a set-valued function F, a distribution DD and a distance parameter and output either Accept or Reject. These algorithms never reject function that accommodates the property. The standard definition of one-sided error testers (provided in the extended version of the paper Overman et al. (0224) extends naturally to set-valued functions by replacing the notion of satisfying a property with accommodating it.
Definition 3 (One-Sided Error Tester for Set- Valued Functions). A one-sided error tester for property P in the context of set-valued functions is a probabilistic oracle machine M that, given a distance parameter 0, oracle access set-valued function F X + 2", and oracle access to samples from a fixed but unknown distribution D over X, satisfies:
1. If F accommodates P, then Pr[ME: (E) = Accept] = 1.
2. If F is E-Faraway from P with respect to D, then FPP)) = Accept| < 1--
Here, M..D(()) (E) denotes the execution of the tester M when given oracle access to the function F. the distribution D. and the parameter e.
Note that M itself is an abstract algorithm; F.D is the instantiation of this algorithm with specific oracle access to F and D.
In many property testing algorithms, the parameter e is used only to determine the number of iterations or samples required, not the core logic of the tester. This leads the concept of proximityoollivious testers (POTs), where the basic testing procedure is independent of s. The general definition of POTs (given in the extended version of the paper Overman al. (2224) also extends naturally set-valued functions.
Definition& 4 ProxxinttyOOllivoous Oblivious Tester for Set- Valued Functions). A moximitty-bllvion tester for a property P in the context of set-valued functions is probabilistic oracle machine 'T that satisfies:
1. If F accommodates P, then Pr[TEED- = Accept] =
2. There exists a nec-decreasing function p: (0, 1] + (0, 1] (called the detection probability) such that if F is e-Faraway from P.
Pr(T Reject] 2 p(E).
3

---

Here, TFD denotes the execution of the tester T when given oracle access to the function F and the distribution D.
To obtain one-sided error tester with parameter e, we can make O
independent calls to p(c)
the POT T and accept if and only if all the calls accept (Goldreich and Ron 2008 We denote by T (X) the output when applied
specific sample X ~ D, and note that with abuse of notation we will late consider D to be the empirical distribution of calibration dataset {XX, Yi) in which case we write F.P (X,, Yi) for the output on this specific sample from D.
Example. Consider functions / : Rd - R. and P denote the property that f constant in the k-th dimension. This property has has connections to fairness among other applications Caton and Haas 2024). Assume is the empirical distribution of the inputs X E R. for some fixed dataset.
Restrict to set-valued functions F' that output compact and connected intervals of the form [a, b] C IR for a, b R. The candidate POT 7 F.D for whether such set-valued function J' acccommodates P is
then as follows: sample X, X' ~ D, If ((X) n F(X') 4 Ø, then Accept; otherwise, Reject. We
prove that this satisfies Definition 4 in the extended version of the paperOverman Overman al. (2024).
2.2 Conformal prediction and conformal risk control
Our main tool for achieving alignment from this property perspective built on conformal prediction and conformal risk control (Vovk et al. 2005 Bates al. 202 Angelopoulos et al. 2024). Con- formal prediction post-processes the outputs of any model to create prediction intervals C()) that
ensure certain statistical coverage guarantees. Using a calibration dataset ((x,, Y)) consisting of ground truth input-output pairs, conformal prediction constructs intervals around the predictions of f such that Pr[Y.+1 & i(),+1)] S a for a user-specified error rate a on test point (Xa+1, Yn+1).
This guarantee notably distribution-Fii randomness in all n + points; both the calibration set and the test point. The construction of C(-) depends on both the model f and the draw the calibration data.
and holds for any function f. The probability is over
The conformal risk control framework extends conformal prediction to notions of error beyond miscoverage (Angelopoulos al. 2024). Consider a paramater set 2g that is bounded subset of the nonnegative reals. Given exchangeable collection of non- non-increasing, random loss functions L; : A + (-00, B), 1, I,...,n 1, conformal risk control uses the first loss functions and calibration data {(,,, 1) to determine Ā such that
E[ (\) S
Consider loss functions of the form (A(A) (Xx), Y,), where Cx(X,) is set of outputs constructed by f and the calibration data. Larger values of À generate more conservative prediction sets (x(.). Let the risk on the calibration data for a given À be (A) 1 Ei (A)). For user-specified risk rate a, we let
i inf
R (A)
# S a
n n+
This choice of À guarantees the desired risk control E[ (?)] S a AAngelopoulos et al. 2024
3 Conformal property alignment
Our main methodology is to use conformal risk control to create prediction intervals that align with specific properties P. Our approach allows post-process the outputs of pre-trained model f to ensure that within the resulting conformal band, with a given probability, there exists predictions that adhere to desired properties such as monotonicity.
3.1 Multi-lambda conformal risk control
We make particular use of the conformal risk control algorithm to allow for k-dimensional vector of tuning parameters X (A1, Aa), where larger values of À A c 1² yield more

---

conservative outputs, where A C R is a bounded subset of B This works by mapping À to scalar and then applying standard conformal risk control. We emphasize that this result is not new and follows essentially directly from Angelopoulos et al. (2024). The construction of the output set (X) S y' depends on the specific application and provides flexibility how the function f(X) and the parameters À are utilized.
Definition 5 (Construction of (x)))) Let f= y be a given function. For each E RR, define the set-valued function Fx X - 2" such that, for each X X, Fx(X) is of predictions for constructed from f and X. The specific construction of Fx(X) should satisfy the following properties:
1. When X O. we have Fo(X) (5(X))
2. For any X, A. E R i (i.e. S X Vi 1,0=1,2... 1,2...., then Fx(X) S FX))..
This definition ensures that increasing the parameters À leads to larger (more conservative) prediction sets, and that when all parameters are zero, the prediction set reduces to the point prediction given by f(X).
Following the original scalar À setting, we assess Fx using non con-increasing random loss functions Y,) E (-o0, B] for B < oc. In particular, we consider an exchangeable collection of non-increasing random functions L3 A + (- B), i = 1,... + 1, where A C R is bounded subset of R with bound Xmnax in each dimension j [R]].
L, =
As in Angelopoulos et al. 22024), we use the first n functions to determine Å so that the risk on the (n + 1)-th function is controlled, specifically so that E1 ( S a.
We apply a similar algorithm. Given a E (oo, B) and letting (A) + define
B
Amin min A n
Sa
n
to be the set of minimal elements (Boyd and Vandenberghe 2004) of A that satisfy the condition B n (A) + n+1 I a. Let A IR be a strictly increasing function such that (A) is non- n+1
increasing with respect to the level sets defined by g()). Then select Å Amin to be minimizer of I over Amin-
We then deploy the resulting set- valued function Fx on the test point X For this choice of Å, we have a risk control guarantee that mimies the result of Angelopoulos al. (2024), specifically:
Proposition 1. Assume that L,(X) is non-incte non-increasing with respect to the partial ordering of A inherited from from strictly increasing function g A + R. Also assume L; is right-contimuous in each dimension. L max) S a, and supx (A) I B < oc almost surely. Then
R Additionally, assume that L;(A) is non-increasing with respect to g(A) for some
E (Å) & a.
The proof is similar to the proof of the guarantee for the conformal risk control algorithm in Angelopoulos et al. (2024) and deferred to the extended version of the paper Overman et al. (2024)
To provide intuition on (A), we note that for our primary use case we will take g(A) = Σk_ A Clearly this function is strictly increasing in À and intuitively it is reasonable to consider loss functions L, that are non-increasing as the sum the components of increases.
3.2 Conformal property alignment from proximity oblivious testers
We now demonstrate how to construct conformal risk control problem using proximity-oblivinn testers (POT') for given property P. Suppose we are given pre-trained model / X + y. We aim to extend the point predictions of to prediction sets, where the size conservativeness of the set is parameterized by a parameter Let Fx X' - 2 denote the set-valued function that outputs, for each X X, the set (X(X y determined by X, and X.
5

---

Let T.D be proximity-oblivion tester for whether set-valued function F accommodates the property as given y Definition We denote the random output of T evaluated (X,Y) D by (X.).
We define loss function, generated from TF.D, which will be crucial in formulating conformal risk control problem.
Definition (Loss Function Generated from a POT). Let T.D be a preximity-ollivion tester for a property P. We define the loss function L3
0, ifT Accept,
L 1, otherwise,
where (X,, Yi) are samples from the distribution
Example. Consider the POT for the property P of function f : R - R being constant, as mentioned Section 2.1 Assume we have access to a calibration set {(X,, Y:) of size n. We use two-dimensional parameter X (A=A and define the set-valued function:
FX(X) [f(x) _ f(x) + Xt],
for each E R. This creates prediction intervals around the point prediction f(X), with widths controlled by X- and X+.
We then apply the loss function generated by T Fx.DD as given Definition 6 and use conformal risk control tune À such that the expected loss on the (n + 1)th point falls below given target level
Note that in this case the tester and loss function does not depend on the Y, This is because the property of f being constant does not depend on the Y; from the calibration set and here D is only used obtain samples of the X;. This is not the case in general, however, and properties can be defined with respect to the whole sample (X,, Yj) ~ D. For example, we could consider the property have o (X) S Y. Now we state our main
that does not over-predict that is, for (X,Y) theorem.
Theorem Let T be proximity-oblivious lester for a property P with detection probability function p(-). Assume access calibration dataset {(X,, Y.)) sampled independenty from distribution D. Suppose we run conformal risk control on this calibration datasel using risk parameter a and loss functions T4 for property generated from T (as in Definition ④ Then, for any such that
p(E) > a. the probability that FsS is s- E-Faraaay from P satisfies:
Pr (... (Fx, is e-Faraway from P) S
a p(e)*
Proof. Let & denote the event that Fx is e-Faraway from the property P. Our goal is to bound the probability (E)...
The conformal risk control procedure ensures that the expected loss on a new sample (Xn+1, Ya+1) satisfies:
(X). (mnt)) a.
Now, we can write
E B[ Pr(E) [T[T 8| Pr(E")
B[ E7)
When & occurs, Fj is e-Faraway from P. By the properties of the proximity-oblivious tester T, we have:
Pr (X.Y)~
(x,))(x,)) = Reject E] 2 p(e).
Thus, the conditional expected loss satisfies:
B[ E] = Pr "T D (Xn+1, Yn+1) = Reject [ p(s).
And when since Pr(E9)
(18))-ELLnaa E[Ln+1 [8] non-negative because Lu+1 is non-negative, we obtain
E L Pr(E) p(e).

---

Combining this result with our guarantee from the conformal risk control procedure,
E [[ 2 Pr(E) p(c).
This implies:
Pr(E) S αt
p(e)
Therefore, the probability that Fx is e-Faraway from P satisfies:
Pr (X,, (X..
(F& is E-Faraway from P) S p(E)
Amplifying Detection Probability via Independent Calls. When the detection probability p(=) of the proximity-oblivion tester 7 is less than or close to the risk parameter o, the bound provided by Theorem 1 may not be tight meaningful (since a/ a(e) could be greater than equal to 1). To address this issue, we can amplify the detection probability by performing multiple independent exccutions of T and combining their results appropriately.
To increase the detection probability beyond a, we execute the proximity-oblivion tester T inde- pendently k times on independent samples and define new tester T' that rejects if any of the k executions reject (i.e., by applying a logical OR to the outcomes). This amplification technique yields an adjusted detection probability
(e) = 1 - (1 -1- (())),
representing the probability that least one of the k independent executions rejects when the function e-Faraway from P.
In this approach, the calibration dataset needs to be partitioned into n' disjoint batches, each containing samples. Each batch provides the independent samples required for the it executions of T per calibration point. As result, the effective sample size available for calibrartion becomes n' due to this batching of samples.
4 Examples
4.1
Monotonicity
Monotonic behavior is important in various applications. We focus monotonicity in single feature, where we expect that f(X) should have monotonically increasing or decreasing behavior with respect to a certain feature x² when other features x - are held fixed. While there long- standing literature on using monotonic constraints for regularization (Brunk et al., 1973 Sill and Abu-MMsttfaa 1996, You et al. 2017 Bonakdarpour et al. 2018) and on integrating such monotonic shape constraints into prediction models (Groeneboom and Jongbloed, 2014 Cano al. 2018 Runje and Shankaranarayana 2023), our aim not to view monotonicity as a possible means to improve test accuracy, but rather as user-desired property for safe fair deployment of a given model. For example, Wang and Gupta (2020) highlight the importance of monotonicity in models for criminal sentencing, wages, and medical triage.
Consider a user given pre-trained model f thhtt was not trained with monotonic constraints. The user, however, wishes for the sake of safe fair deployment to make predictions in way that is as monotonic as possible. In particular, let be the property that / monotonically decreasing in dimension k. To apply our methodology we consider the proximity oblivious tester T for P as given in Algorithm
We prove in the extended version of the paper Overman et al. (2024) that Algorithm I is indeed POT for the property P of being monotonically decreasing given dimension. Then let M be the one-sided error tester for P resulting from ((1/p(E)) calls to T. Now assume we have access to a calibration dataset {(X,, Y sampled from of size n E ((1/p(E)). We will use this calibration dataset to determine the setting of X = (X+ X-) via conformal risk control where the loss function is generated as in Definition 6) Here the set-valued function will be constructed as Fx(X) = (x() probability ((e) > a we expect to obtain a set-valued function F3 at most e from P. We now investigate this empirically.
>= ((X + A+). Then by Theorem if the tester has sufficient detection
7

---

Algorithm POT T for property P of monotonically decreasing in dimension I:
1: Sample X1 ~ D. Let X1 = (#1,
2: Sample æ2 from the marginal distribution 'D in dimension Set X2 = (#2, x x=h)
3: if' Ø1 < 22 and max FXX1) min F(22) then
4: return Reject
5: else if 22 < 21 and max F(X2) min F(X1) then
6: return Reject
7: end if
8: return Accept
Setup. We align for monotonicity various UCI ML repository datasets (Dua and Graff 2023
with a 70-15-15 train-calibrate-tes split, averaged over 30 random splits. We use XGBoost regression models (Chen and Guestrin] 2016). For each dataset, select feature for which we desire the model to be monotonic, not with the intention of improving test-set accuracy, but from the perspective of user who desires this property.
We train two models per dataset: one unconstrained, trained on the training set, and another con- strained to be monotonic, trained both the training and calibration sets. The conformal risk control procedure is applied to the unconstrained model using the calibration data. The constrained model can be considered best possible from the user's perspective, using all available pre-test data and satisfying the monotonicity property P during training.
To compare performance with respect to the training metric of accuracy, we convert conformal intervals into point predictions by taking k-quantiles of the constrained feature, linearly interpolating between adding A at the lowest quantile to subtracting À at the highest quantile for monotonically decreasing, or vice versa for monotonically increasing.
Results. Table 4.1 presents results on the test set for the Combined Cycle Power Plant dataset (Teekci and Kayal 2014). In practice, Exhaust-vacuum is known to negatively influence turbine efficiency (Tiekci and Kaya 2014). The conformal procedure outperforms the constrained model in terms of MSE for all a, which a fortuitous but unexpected outcome. The constrained model should be seen oracle benchmark in the sense that the model was given to the user already trained to satisfy the desired property. The risk metric closely matches the thcoretical guarantee from conformal risk control and achieves optimal performance of ( for the constrained model. Additional datasets and results are detailed in the extended version of the paper Overman et al. 2024).
Table I: Power Plant, n 9568. Monotonically decreasing on Exhaust Vacuum. max (10, 10).
a
A
Metric
Unconstrained
Adjusted
Constrained
A+ (10.
MSE
10.19 (0.19
10.47 (10.46)
16.21(10.45)
0.1
A 0.76(1
Risk
0.75(10.09)
0.10(10.001)
0.00
A+ 1.09
MSE
10.19(20.46)
11.42
16.21 16.21(+0.45)
0.05
A 1.61 (+0.60))
Risk
0.75(20.09)
0.05
(.000)0.00)
A+ 2.39 (20.82)
MSE
10.19
14.46
16. 16.21(40.48)
0.01
x = 3..3
Risk
0.75 0.75
0.01
(.00
4.2
Concavity
Concavity and convexity are crucial behaviors many applications. In this context, we focus on concavity in a single feature. A common example where users might expect concave behavior is in recommendation or preference prediction models. According to economic theory, the utility function with respect to the quantity of an item is often quasi-concave, reflecting the principle of diminishing marginal utility (Mas-Colell et al. 1995). JJenkins et al.) (202T) propose a novel loss function to account for this expected concavity, which aligns the model with the concavity property P during training. Here we again consider aligning a pre-trained model, not trained satisfy P. using a proximity oblivious tester T for P as described in Algorithm

---

Original Model
Adjusted Prediction
Risk Control Band Violation(s)
465
Target Variable (Power Output)
460
455
50
70
Feature of Interest (Exhaust Vacuumn)
Figure 1: Univariate partial dependence plot of unconstrained model. Risk control band for a = 0.05.
Dashed line exemplifying Theorem demonstrating existence of monotonically decreasing function falling within the conformal band on 0.975 > 1 fraction of the domain.
Algorithm 2 POT T for property of concavity dimension
1: Sample Xmid 'D
2: Sample Seett Öright from empirical differences in feature k:
3: Set Xjent by decreasing feature & of Xmid by ōjon
4: Set Xright by increasing feature
Xmid by dright
5: Query FXXmaa), F(X F(Xee), and F ((Krigha)
Xright [k]
Xmia [k]
6: Compute
X [k] X1ot [k]
7: if min F(Xmid)
> a max F (Xien) + (1 a.) max F F(Xrigh)) then
8: return Reject
9: end if
return Accept
can use a calibration dataset to determine the setting of À (*+,^-) via conformal risk control where the loss function is generated as in Definition 6 The set-valued function will be constructed as (X) = [X) x f(X) + X*]. We demonstrate running conformal risk control with this loss function real-world dataset in the extended version of the paper Overman et al. (2024)
5 A stylized examination of alignment persistence in Al models
Consider data generated as:
y == 9 h(X) +E,
where E is mean-zero noise with variance 72 independent of X. Here, h(X) is biased noise we want ignore, aiming leam only ( Consider the case in which experts expect data to follow g(X) + unbiased noise, but biased noise h (X) can obscure this.
One potential reason for the presence of biased noise in data could be due to measurement error of outcome that is correlated with select features, leading to an incorrectly calculated outcome. biased mcasurement error could occur if there is incomplete data and the presence of the incomplete

---

data is correlated with select features in an unexpected, systematic way. Our goal is to understand how this bias affects model behavior when trying to learn g(X) alone.
Given n. i.i.d. samples {(x,, Y;)) from the above model, we denote this dataset by Dn. We use a random feature model:
fRF (X;a, (w))ie(N)) = 1
E (oo((X,,)))
VN
JE[N]
where a E RN are learned weights, and {w, [iele are fixed random weights. The squared loss is minimized by ridge regression:
â사 = arg min
Σ VE(n)
(Y - JRF (X,)) + Al|al12
BERN
Users expect a model to exhibit a property P. satisfied by g(X) but not necessarily by g(X) + h(X). We can constrain training to ensure P. Let Cp {al I and fRF (X; a) satisfies P), yielding
a constrained model: åx,P arg minagCp Σie(>) (Y (RF,) M|aalE.
Assuming 9 and h are polynomials with deg, < degn, and given specific conditions data size and model parameters, we consider two settings: (i) Classic: d'es, +5 < < < ddegn -6, and (ii)
Underspecified: N > ddegs +6 for a small 6 > 0.
In the extended version of the paper Overman et al. 2024 we utilize results from Misiakiewicz and Montanari (2023) to derive insights into the impact of model complexity and data size on adherence P. In particular, we show that under certain assumptions, including small noise bias and robustness of property P. the constrained and unconstrained models have zero distance in the classic setting: As = åx- However, in the underspecified setting, the constrained and unconstrained models will differ, resulting in non-zero distance: âx,P o åx. This result implies that in the presence of noise bias, the overparameterized models (i.e., underspecified setting) fail to satisfy the property P. and this cannot be remedied as the data size increases.
6 Related work
Our paper draws from broad range of areas, hence we refer the reader textbooks and surveys in
alignment (Everitt et al. [2118] Hendrycks et al. 2022 Ji et al. 2024 Hendrycks 2024). conformal
prediction (Angelopoulos and Bates 20022) property lesting (Ron. 2008 Goldreich 2017), and
linearized neural networks (Misiakiewicz and Montanari] 2023).
RLHF (Chistianoo et al.) 2017) has been notably effective in aligning LLMs with human values and intentions, as demonstrated by (Ouyang et al.) (022). Our work considers attempts at alignment that generalzies to models without human-interpretabbe
outputs, which has connections to the scalable oversight problem (Irving et al.| 20188; Christiano et al.) 2018 Wu et al. 2021). Goal misgeneralization (Langosco al. 2022 Shah et al.] 2022) has potential connections to the underspecified pipeline (D' Amour et al.) 2022) considered in this paper in the sense that models with equivalent performance according the training metric may differ in some other user-desired property during deployment. One of the main methods of assurance (Batarseh et al. 202I). which is concerned with assessing the alignment of pre-trained AI systems, is safety evaluations (Perez et al., 2022 (Sevlane al. 2023) meant to assess risk during deployment, which also has connections to our approach.
The work of Yadkori et al. (2024) closely aligns with ours in both methodology and theme, utilizing conformal risk control to reduce LLM hallucinations (Ji et al. (023). We discuss connections to this work in the extended version of the paper
paperOverrman Overman al. (2024).
7 Discussion
We introduce a method to align pre-trained models with desired user properties using conformal risk control. By post-procsssing outputs using property dependent loss functions, we provide probabilistic guarantees that conformal intervals contain functions close to the desired set. This allows for alignment without retraining, effective in both generative and non-generative contexts. Future work should extend these techniques to more properties, explore sample complexity and adaptive querying, and potentially apply them to policy functions in MDP settings for RL agent safety guarantees.
10

---

References
Anastasios N. Angelopoulos and Stephen Bates. A gentle introduction to conformal prediction and distribution-free uncertainty quantification, 2022.
Anastasios Nikolas Angelopoulos, Stephen Bates, Adam Fisch, Lihua Lei, and Tal Schuster. Confor- mal risk control. In The Tweifth International Conference on Learning Representations, 2024.
Vipul Arora, Arnab Bhattacharyya, Noah Fleming, Esty Kelman, and Yuichi Yoshida. Low Degree Testing over the Reals, pages 738-792 2023. doi: 10. 17117//1977111119775444488 hattps://epon siam. otg/ 1177/11.98111117775565 ch31
URL
Feras A. Batarseh, Laura Freeman, and Chih--Ho Huang. survey on artificial intelligence assurance. Journal of Big Data, 8(1), April 2021. ISSN 2196-1115. doi: 10.1186//40537-221-004444 URL http://dx. dow.:xdoo..rggg..... 10. 1186/ 0051///40000022-000000
Stephen Bates, Anastasios Angelopoulos, Lihua Lei, Jitendra Malik, and Michael I. Jordan. Distribution-free, risk-controlling prediction sets, 2021.
Arnab Bhattacharyya, Swastik Kopparty, Grant Robert Schoenebeck, Madhu Sudan, and David Zuckerman. Optimal testing of reed-muller codes. 2010 IEEE 51st Anmual Symposium on Foundations of Computer Science, pages 488 497, 2009. https:/:://pp http://app..smmnnicscchhh.o.aaea
org/ Btg/CorpusIDD.. 25924
Eric Blais. Testing juntas nearly optimally. pages 151-158, 05 2009. doi: 10.1 10.11451156611415556637
Manuel Blum, Michael Luby, and Ronitt Rubinfeld. Self:testing/correcttnng with applications to numer-
ical problems. Journal of Computer and System Sciences, 47(3):549-59) 1993. ISSN 0022-0000. doi: com cotp:/ccceeee//ccccceee///////00
W. URL. https: http:://wwwwwww.cediiie...
Matt Bonakdarpour, Sabyasachi Chatterjee, Rina Foygel Barber, and John Lafferty. Prediction rule reshaping, 2018.
Stephen Boyd and Lieven Vandenberghe. Convex optimization. Cambridge university press, 2004.
Hugh D. Brunk, Richard E. Barlow, David J. Bartholomew, and Joan M. Bremner. Statistical inference under order restrictions the theory and application of isotonic regression. International Statistical Review, 41:395, 1973. URL https://:///pp www.sticsmcsonootoeeeo. arr org/ Ttrg/CorpusID:: 1203:9544499
José-Ramón Cano, Pedro Antonio Gutiérrez, Bartosz Krawczyk, Michal Wožniak, and Salvador García. Monotonic classification: overview algorithms, performance measures and data sets, 2018.
Simon Caton and Christian Haas. Fairness in machine learning: A survey. ACM Comput. Surv., 56 (7), April 2024. ISSN 0360-0300. doi: 10.1 (0.1145/3616863 URL. htttps://doi. org/ /10 1145/ 3616865
Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '16. ACM, August 2016. doi: 10. 10.114572939672.2999733. URLhttpp: //dp: doi org. 1145/ 2939672. 2939785
Paul Christiano, Buck Shlegeris, and Dario Amodei. Supervising strong learners by amplifying weak experts, 2018.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep rein- forcement learning from human preferences. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL hec.:/007.URLLLtttt///reeeeeded neurips cc/ pdf
paper_ httpp//.....poooooooooocooooaoan

---

Alexander D' D'Amour, Katherine Heller, Dan Moldovan, Ben Adlam, Babak Alipanahi, Alex Beutel, Christina Chen, Jonathan Deaton, Jacob Eisenstein, Matthew D. Hoffman, Farhad Hormozdiari, Neil Houlsby, Shaobo Hou, Ghassen Jerfel, Alan Karthikesalingam, Mario Lucic, Yian Ma, Cory McLean, Diana Mincu, Akinori Mitani, Andrea Montanari, Zachary Nado, Vivek Natarajan, Christopher Nielson, Thomas F. Osborne, Rajiv Raman, Kim Ramasamy, Rory Sayres, Jessica Schrouff, Martin Seneviratne, Shannon Sequeira, Harini Suresh, Victor Veitch, Max Vladymyrov, Xuezhi Wang, Kellie Webster, Steve Yadlowsky. Taedong Yun, Xiaohua Zhai, and D. Sculley, Underspecification presents challenges for credibility in modern machine learning. Journal of Machine Learning Research, 23(1), jan 2022. ISSN 1532-4435.
Dheeru Dua and Casey Graff. Uci machine learning repository, 2023. uttp:////arcrr uci. edu/ml
Tom Everitt, Gary Lea, and Marcus Hutter. Agi safety literature review, 2018.
Noah Fleming and Yuichi Yoshida. Distribution-free testing of linear functions on IR", In Thomas Vidick. editor, 11th Innovations in Theoretical Computer Science Conference (ITCS 2020), volume 151 of Leibniz International Proceedings in Informatics (LIPlcs), pages 22:1-22:19, Dagstuhl, Germany, 2020. Schloss Saarttuul--eiiniiZzeetteee für Informatik. ISBN 97883-99999997 doi: 1tt.00/.PPc....CS..200000 2020/11707
httpss//::///rr wwwpsuhl..tttdccooom http://ppx/oollettee
Oded Goldreich. Introduction to Property Testing. Cambridge University Press, 2017.
Oded Goldreich and Dana Ron. On proximity-oblivious testing. Electronic Colloquium on Computa- tional Complexity (ECCC), 15, 01 2008. doi: 10.1 10.1137/1007896466
Piet Groeneboom and Geurt Jongbloed. Nonparametric Estimation under Shape Constraints: Estimators, Algorithms and Asymptoties. Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, Cambridge, 2014. ISBN 978-0-521-86400155 doi:
10.1017/CBB998811990200
Dan Hendrycks. AI Safety, Ethics, and Society. Center for Al Safety, 2024. URL httph:t/p/// www
aisafetybook.com com/
Dan Hendrycks, Nicholas Carlini, John Schulman, and Jacob Steinhardt. Unsolved problems in ml safety, 2022.
Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn Song, and Jacob Steinhardt. Aligning ai with shared human values, 2023,
Geoffrey Irving, Paul Christiano, and Dario Amodei. Ai safety via debate, 2018.
Porter Jenkins, Ahmad Farag, J. Stockton Jenkins, Huaxiu Yao, Suhang Wang, and Zhenhui Li. Neural utility functions. Proceedings of the AAAI Conference on Artificial Intelligence, 35(9): 7917-7925, May 2021. doi: 10. 10.1660/aaii.v3555 16966. URL https:///o/sss aaai. aeaai.rg//a brg/index. php /AA//AAA//aaa http://rrrreee/ee/e66//6..
Jaming Ji, Tianyi Qiu, Boyuan Chen, Borong Zhang, Hantao Lou, Kaile Wang, Yawen Duan, Zhonghao He, Jiayi Zhou, Zhaowei Zhang. Fanzhi Zeng, Kwan Yee Ng. Juntao Dai, Xuehai Pan, Aidan O' Gara, Yingshan Lei, Hua Xu, Brian Tse, Jie Fu, Stephen McAleer, Yaodong Yang, Yizhou Wang, Song-Chun Chun Zhu, Yike Guo, and Wen Gao. Ai alignment: A comprehensive survey, 2024.
Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. Survey of hallucination in natural language generation. ACM March 2023. ISSN 1557-7341, doi: 10. 10.1145/3571730. URL htttp://dx.do// doi. /tag////... 1145/3571700
Computing Surveys, 55(12))1-38)
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020.
Tali Kaufman and Dana Ron. Testing polynomials over general fields. SIAM J. Comput., 36:779-802, 10 2006. doi: 10. 1011177750097
12

---

Lauro Langosco Di Langosco, Jack Koch, Lee D Sharkey, Jacob Pfau, and David Krueger. Goal mis- generalization in deep reinforcement learning. In Kamalika Chaudhuri. Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 12009-12019. PMLR, 17-23 Jul 2022. httthtttpp//coceeeeeee langoaco22a.html
mlr www.:mmes././//.mmt..t.. press/v162) vv62/
Jan Leike, David Krueger, Tom Everitt, Miljan Martic, Vishal Maini, and Shane Legg. Scalable agent alignment via reward modeling: a research direction, 2018.
Andreu Mas-Colell, Michael D. Whinston, and Jerry R. Green. Microeconomic Theory. Oxford University Press, New York, 1995.
Song Mei and Andrea Montanari. The generalization error of random features regression: Precise asymptotics and the double descent curve. 75(4))667-766. ISSN 1097-0312. doi: 10. 1002/ppa 22008.
Theodor Misiakiewicz and Andrea Montanari. Six Lectures on Linearized Neural Networks, arXiv e-prints, art. arXiv:2308. 13431, August 2023. doi: 10.48550/arxiv.2308.. 13431.
Richard Ngo, Lawrence Chan, and Sören Mindermann. The alignment problem from deep learning perspective, 2024.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder. Paul F Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 27730-27744 Curran Associates, Inc., 2022. URL https: https::/pproceeddnggs cttp://eedeeeeeeeeeeeeeeeee....e cct.rrips.ccc/appaaaa ff files/ fwwes/papelleeeee /aperr/0022//11ee 1actee5beeaa//44440000007772...0
pdf
William Overman, Jacqueline Jil Vallon, and Mohsen Bayati. Aligning model properties via conformal risk control, 2024. URL wtW.ttttt::: http://aaxxvvoorgaaa// 18777
Ethan Perez, Sam Ringer, Kamilé Lukošiūté, Karina Nguyen, Edwin Chen, Scott Heiner, Craig Pettit, Catherine Olsson, Sandipan Kundu, Saurav Kadavath, Andy Jones, Anna Chen, Ben Mann, Brian Israel, Bryan Seethor, Cameron McKinnon, Christopher Olah, Da Yan, Daniela Amodei, Dario Amodei, Dawn Drain, Dustin Li, Eli Tran-Johnson, Guro Khundadze, Jackson Kernion, James Landis, Jamie Kerr, Jared Mueller, Jeeyoon Hyun, Joshua Landau, Kamal Ndousse, Landon Goldberg. Liane Lovitt, Martin Lucas. Michael Sellitto, Miranda Zhang, Neerav Kingsland, Nelson Elhage, Nicholas Joseph, Noemí Mercado. Nova DasSarma, Oliver Rausch, Robin Larson, Sam McCandlish, Scott Johnston, Shauna Kravec, Sheer El Showk, Tamera Lanham, Timothy Tellcen- Lawton, Tom Brown, Tom Henighan, Tristan Hume, Yuntao Bai, Zac Srefiell-Doddde Jack Clark, Samuel R. Bowman, Amanda Askell, Roger Grosse, Danny Hernandez, Deep Ganguli, Evan Hubinger, Nicholas Schiefer, and Jared Kaplan. Discovering language model behaviors with model-written evaluations, 2022.
Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. In J. Platt, D. Koller, Y. Singer, and S. Roweis, editors, Advances in Neural Information Processing Systems. volume 20. Curran Associates, Inc., 2007. URL https::/pprcceeddnggg www.:ggs..........e... cc/paper_
Paper .df
Dana Ron. Property testing: A learning theory perspective. Found. Trends Mach. Learn., 1(3): 307-402, 2008. doi: 10. 10.1561/2200000000
htttttttt:///doo http::p::/ooo..rgg/0......./. 1ttp:/11..5111/20000000000
Davor Runje and Sharath M. Shankaranarayana. Constrained monotonic neural networks, 2023.
Rohin Shah, Vikrant Varma, Ramana Kumar, Mary Phuong, Victoria Krakovna, Jonathan Uesato, and Zac Kenton. Goal misgeneralization: Why correct specifications aren't enough for correct. goals, 2022.
13

---

Toby Shevlane, Sebastian Farquhar, Ben Garfinkel, Mary Phuong, Jess Whittlestone, Jade Leung, Daniel Kokotajlo, Nahema Marchal, Markus Anderljung, Noam Kolt, Lewis Ho, Divya Siddarth, Shahar Avin, Will Hawkins, Been Kim, Iason Gabriel, Vijay Bolina, Jack Clark, Yoshua Bengio, Paul Christiano, and Allan Dafoe. Model evaluation for extreme risks, 2023.
Joseph Sill and Yaser Abu-Mostafa. Monotonicity hints. In M.C. Mozer, M. Jordan, and T. Petsche, editors, Advances in Neural Information Processing Systems, volume 9. MIT Press, 1996. URL
neurissssnnnnnnnu.o..e.........s cc/ wwwpp/p.ccppppppppeeeeeee http///leeee/pppr/99/////eee 66601770a66771111666477777777--3 .pf
Pnar Tfekci and Heysem Kaya. Combined Cycle Power Plant. UCI Machine Learning Repository,
2014. DOL: http:://bi.orr11..24522/650
Jacqueline J. Vallon, William Overman, Wanqiao Xu, Neil Panjwani, Xi Ling, Sushmita Vij, Hilary P. Bagshaw, John Leppert, Sumit Shah, Geoffrey Sonn, Sandy Srinivas, Erqi Pollom, Mark K. Buyyounouski, and Mohsen Bayati. On aligning prediction models with clinical experiential learning: A prostate cancer case study. Unpublished manuscript, 2024.
JJJ. Vallon, N. Panjwani, Ling, S. Vij, E. Pollom, H.P. Bagshaw, S. Srinivas, Leppert, M. Bay- ati, and M.K. Buyyounouski. Clinically consistent prostate cancer outcome prediction mod- els with machine learning. International Journal of Radiation Oicology" 114(3, Supplement): 126-e117, 2022. ISSN 0360-3016. doi: https///ddoi.or/100110000 1jjrobp.2022.07.955. 50360301622016723
URL https: wwwww.sciieet swwencedirect.commm com/scciianc//articcepp.... ASTRO Annual 2022 Meeting.
Vladimir Vovk, Alexander Gammerman, and Glenn Shafer. Algorithmic Learning Random World, Second Edition. January 2005. doi: 10. 1110079978-3-031-66649-99 Springer- New York, Inc. 2005.
Serena Wang and Maya Gupta. Deontological ethics by monotonicity shape constraints, 2020.
Jeff Wu, Long Ouyang, Daniel M. Ziegler, Nisan Stiennon, Ryan Lowe, Jan Leike, and Paul Christiano. Recursively summarizing books with human feedback, 2021.
Yasin Abbasi Yadkori. Ilja Kuzborskij. David Stutz, András György, Adam Fisch, Arnaud Doucet, luliya Beloshapka, Wei-Hung Weng, Yao Yua- Yang. Csaba Szepesvári, Ali Taylan Cemgil, and Nenad Tomasev. Mitigating Ilm hallucinations via conformal abstention, 2024.
Seungil You, David Ding, Kevin Canini, Jan Pfeifer, and Maya Gupta. Deep lattice networks and partial monotonic functions, 2017.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction clearly state the contributions of proposing model alignment approach via conformal risk control and property testing, with applications demonstrated real-world datasets. The claims supported by theoretical results and experimental validation. See Section 3 and 4.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The paper mentions the assumptions made for the theoretical results, such as the nature of the data distribution and the properties of the models. The discussion section also reflects on the limitations regarding the types of properties and models which the methodology can be applied. See Section 3.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model vell-specifiiattin, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
15

---

3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: The theoretical results are supported by complete proof's provided in both the main text and the extended version of the paper. Assumptions are clearly stated along with theorems. See Section and the extended version of the paper.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: The experimental setup, including data splits, model parameters, and the procedure for applying conformal risk control, is described in detail. This information is sufficient to reproduce the main experimental results. See Section and the extended version of the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
16

---

(d)
reproducible, they We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: The paper provides links to the UCI ML repository for the datasets used, and the code used for experiments will be made available upon publication. Sufficient instructions are included for reproducing the experiments. See Section 4 and the extended version of the paper.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/pips..cc/pulic//guu/ http://udiieesseeeeee httpc//ppo..o/oooioooooeeeeeebaa
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
The paper includes detailed information about data splits, model training, hyperparameters, and the selection process, allowing readers to understand and reproduce the results. See Section and the extended version the paper.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
17

---

The paper reports mean squared errors and risks, and the experiments are averaged over multiple random splits, providing sufficient information on variability and significance suitable for our context. See Section and the extended version of the paper.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (c.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The paper specifies the use of XGBoost models and mentions typical execution times and resources used for the experiments. See the extended version of the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
The research adheres to ethical guidelines by ensuring transparency, repro- ducibility, and careful consideration societal impacts, especially in the context of model alignment and fairness.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
18

---

The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
The discussion includes potential positive impacts of improved model align- ment on fairness and transparency, as well as the challenges and limitations that need to be addressed. See Section 1.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairesss considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper does not involve high-risk data models.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
19

---

Answer: [Yes]
Justification: The paper uses publicly available datasets from the UCI ML repository, which are properly credited and cited. See Section and the extended version of the paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwwppsppppeeaeeeeeee thcode com datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not introduce new datasets or models.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [NA]
Justification:
The paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
20

---

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
21


Reviews:
[{"review_id": "XBTaCrlMZx", "reviewer": "Reviewer_9EgD", "summary": "This paper proposes a method to post-process a pre-trained model to align with a subset of hypotheses on which the specific desired behaviors can be attained. The proposed method relies on proximity oblivious testers to give detection for the misalignment, based on which a conformal risk control process is used to calibrate the prediction interval to guarantee the farness to the desired subset of hypotheses.", "questions": "N.A.", "limitations": "N.A.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The considered problem is a general and important. The properties to be aligned can include many possible instances.\n2. The paper is overall well written and easy to follow.", "weaknesses": "1. The proposed aligning method is largely built on conformal risk control to calibrate the prediction interval, so the original technical contribution needs to be highlighted.\n2. The generalization to multi-dimension conformal risk control is a bit straightforward and an immediate result from the original conformal risk control. The technical challenges need to be highlighted also.\n3. The proposed method depends on the reliability of the POT, as mentioned in Line 181. However, to guarantee a rigorous prediction, it is important to consider the failure and errors from POT and how to guarantee the valid prediction in this case in details.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for their time and effort in reviewing our paper and providing valuable feedback. We appreciate that the reviewer acknowledges the generality and importance of the problem we considered, as well as the flexibility of our approach concerning the properties that can be addressed. Below, we provide responses to each of the three listed weaknesses.\n\nThe first listed weakness:\n“1. The proposed aligning method is largely built on conformal risk control to calibrate the prediction interval, so the original technical contribution needs to be highlighted.”\n\nWe appreciate the call to better highlight our technical contribution. An important contribution of this work is combining two unrelated topics of conformal risk control and property testing to tackle the alignment problem. To the best of our knowledge, this is the first. We feel this methodological approach, along with our main theorem, is a strong contribution. We also want to bring to the review’s attention our technical contributions in Section 5. We bring attention to these results in our global response, but repeat here for convenience. \n\nOur theoretical results in this section build on random feature model theory (Mei and Montanari 2023, Misiakiewicz and Montanari 2023) to obtain insights regarding the impact of model size and sample on adherence to properties $\\mathcal{P}$. We show that even if the true data generating process adheres to $\\mathcal{P}$ if there is small noise bias then overparameterized models will fail to satisfy a desired property $\\mathcal{P}$ regardless of how much data is collected. This result has significant implications with respect to the persistent need for alignment techniques regardless of how much training data large models are trained on. \n\nWe hope this clarifies the strength of the technical contributions of our paper. \n\nThe second listed weakness:\n2. The generalization to multi-dimension conformal risk control is a bit straightforward and an immediate result from the original conformal risk control. The technical challenges need to be highlighted also.\n\nWe agree with this point; we attempted to clearly state in both the proposition statement in the main text and the proof in the appendix that the proof closely mimics the original result. \n\nHowever, we do feel that the literature has overlooked the potential of something like multi-lambda conformal risk control so far, even in instances in which it would have been helpful. In particular, as mentioned in the global response, Yadkori et al. (2024) utilize two separate conformal procedures to calibrate two separate functions, which could have been simultaneously calibrated in tandem using multi-lambda conformal risk control. Other works, such as “Conformal Risk Control for Ordinal Classification” (Xu, 2023), assume that lower and upper bounds on a prediction set are given by functions l(lambda) and u(lambda) but requiring both ends of the interval to grow simultaneously may not be generally necessary; multi-lambda conformal risk control provides a flexible implementation of the conformal risk control algorithm in such settings.\n\nThe third weakness:\n3. The proposed method depends on the reliability of the POT, as mentioned in Line 181. However, to guarantee a rigorous prediction, it is important to consider the failure and errors from POT and how to guarantee the valid prediction in this case in details.\n\nWe believe there is room for clarification on this point. While for general property testing algorithms, there may be two-sided errors, for POTs, we can guarantee that if $f$ satisfies property $\\mathcal{P}$, then with probability one, the POT will correctly accept this function. In other words, there can be no false negatives or type II errors. The POTs can indeed have a false positive or type I error. This does not impact our main result, however, since for any valid testing algorithm (which we assume $T$ is), any function that $T$ accepts on at least $1-\\alpha$ fraction of points is by definition as most $\\alpha$ far from having the property that $T$ tests for. \n\nIt is possible, however, for future work, that results concerning the distribution of the loss functions on test points (since here we only have a guarantee a bound on the expectation of the loss) may need to take into account the error probability of the tester. We thank the reviewer for bringing up this point, as it is closely aligned with ideas on our minds for potential future directions. \n\nWe hope that our responses to each of these listed weaknesses sufficiently address the reviewer’s concerns and can convince them of the work’s contribution. As the reviewer mentioned, the address problem of this work is both general and interesting, and our approach is powerful and flexible. We hope the reviewer is more convinced that the approach is also theoretically and technically strong regarding both the new approach to alignment and the results regarding the persistent need for alignment based on the random feature model theory. Again, we would like to thank the reviewer for engaging with our work and hope their judgment of the work has become increasingly positive.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iQPvCnALto", "reviewer": "Reviewer_7tC7", "summary": "The authors propose a way to solve the problem of alignment in Machine Learning using Conformal Risk Control. They first expand the previous work of conformal risk control to multidimensional parameters $\\boldsymbol{\\lambda}$, then used this extension to propose a way to test if a function belongs to a certain class of functions $\\mathcal{P}$ that we assumed are aligned with user interests. Finally, they demonstrate how this method can be used to test monotonicity of functions.", "questions": "see weakness in concavity.", "limitations": "* The authors advertise this paper as being useful in GenAI. What are examples of potential class of functions that would satisfy alignment with a generated text / image for example. It is indeed a good first step to test the belonging to a class, but defining that class can be very hard in the most important application", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The theoretical foundations of the paper seems pretty solid. The methodology is clear and the paper is quite easy to follow\n* The monotonicity and concavity examples are pretty convincing\n* Detailing the linearity case help understand the theoretical foundation and reasoning of the method", "weaknesses": "* There is only a single example with results. Presenting a method for concavity is interesting but it would be nice to see it applied to a real world examples\n* There is still a significant gap between monotonicy/concavity and alignment as we understand it in AI.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for their positive assessment of our paper. We appreciate the recognition of the solid theoretical foundations, the clarity of our methodology, and the overall readability of the paper. We also value the opportunity to address your concerns and clarify our contributions.\n\nRegarding the reviewer’s first stated weakness: \"There is only a single example with results. Presenting a method for concavity is interesting, but it would be nice to see it applied to real-world examples,\" we believe there may be a misunderstanding. We included two additional examples on real datasets for monotonicity in the appendix and a real-world example for the concavity method. We briefly mentioned these at the end of the Results section in 4.1, but we will make this more explicit in the final version. We hope the reviewer finds these additional examples convincing of the strength of our paper.\n\nThe reviewer’s other stated weakness concerns the gap between monotonicity/concavity and alignment in AI. We appreciate the desire for a more detailed discussion on this matter. As discussed in our abstract and introduction, our goal is to propose a framework for interpreting and performing alignment in settings with outputs less interpretable and amenable to human feedback. Thus, our goal was to show that this framework could extend from properties like monotonicity to more complex properties of generative models.\nOur methodology allows us to define and align desirable properties of generative models, provided we can test for these properties at a per-input level. We included this discussion in our general response but will restate it here for the reviewer’s convenience.\n\nWe want to highlight this claim via a section that may have been too buried originally. The connection to Yadkori et al. (2024) in Section 6 Related Work was not only to acknowledge their work but also to highlight the applicability of our methodology to a generative AI setting. We discuss that their use of conformal risk control to mitigate LLM hallucination can be captured as a specific case of our more general property alignment perspective.\n\nWe discuss that “not hallucinating” can be considered a property of the function $(a, f)$, where $a$ determines whether the model should abstain, and if not, the function $f$ gives the output. This example shows how our approach of forming the conformal intervals is not limited to subsets of the reals since, in this application, the conformal interval is either just $f(X)$ or $\\{f(X), Abstain\\}$ for sufficiently conservative lambda.\n\nWhile including an application of our approach for LLM alignment is beyond the scope of this paper and not feasible for this rebuttal, we believe this discussion highlights the potential applicability of our approach to generative AI. Our methodology potentially enables one to define and align desirable properties of generative models as long as there is a way to test for these properties at a per-input level.\nWe also wish to draw attention to the importance of monotonicity through an additional example provided in Figure 1 of the attached PDF. Although we lack the time or scope to provide a comprehensive comparison of our method to the extensive fairness literature, we have included a figure for results of applying our method to the UCI ML repo “Student Performance” dataset. This dataset is often used as a benchmark in the fairness literature (see “A survey on datasets for fairness-aware machine learning” Quy et al. 2022). We take “sex” to be the protected attribute and the final grade “G3” to be the real-valued outcome of interest. The attached figure demonstrates the lambdas needed to achieve a constant predicted grade when varying “sex” with a tolerance of 0.1 for $\\alpha$.\n\nGiven the reference to applications of monotonicity in medical triage at the beginning of Section 4.1, we have included Figure 2 in the attached PDF showing the results of our method on the UCI ML repo “Liver Disorders” dataset. Here, we take the target of the average number of drinks per day (specified as the target in the repo) to be monotonically increasing in the feature “mcv,” which is known to be predictive of liver disorders. This example from the high-stakes medical domain emphasizes that monotonicity significantly improves trust, interpretability, and the adoption of clinical models.\n\nWe also point the reviewer to another newly included experiment in Figure 3a and 3b at the end of the general response, highlighting the applicability of the framework to a more complex safety alignment problem of avoiding side-effects. The details of this experiment are included in the global response.\n\nWe hope these examples demonstrate that while monotonicity and concavity may not match the usual notion of alignment in generative AI, they are important properties of models. Our methodology allows for application to more complex settings, such as generative AI and RL.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PYDpf82qHx", "reviewer": "Reviewer_8oYs", "summary": "This article offers an alternative approach to alignment by testing whether trained models follow specific properties. This is done, for instance, on monotone or concave functions. A small modification for the use case is made to the Conformal Risk Control setting to allow vectorial parameters. Using CRC and property testers (POTs), a guarantee is obtained that the predictor approximately satisfies the property.", "questions": "Although early in the article, it's specified that \" we can imagine a user who wishes to determine whether a pre-trained model f belongs to P but is unable to train a model of f ’s size\" (42), line 178 proposes to \"modify\" the model. How is that done?\nMoreover, in the appendix (table 4), unconstrained models have varying performance when alpha varies, how is that explained? (moreover, no notion of uncertainty is provided on the metrics)", "limitations": "I believe it's worth mentioning that this multidimensional  CRC is essentially one-dimensional, as it replaces the non-increasingness in lambda by one in a mapping of lambda to R. \nMoreover, the monotonicity in the l1 norm of lambda is often not verified (think of different slopes for the different dimensions), and several experiments do not use this result.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The topic of the paper is interesting, and the approach is novel, to my knowledge. \nThis approach's applications are quite broad and should interest communities interested in alignment, robustness, or, for instance, linking to certain physical applications. A few applications are presented in experiments.\nFinally, I find the paper, overall, well written.", "weaknesses": "However, several parts could be improved. Section 2.1 is unclear, and the introduction of lambda without details is bound to confuse readers not familiar with CRC. \nMoreover, I found section 3.2 to be overly verbose and lacking clear, explicit mathematical formulations of the loss, particularly but not exclusively.\nIn case it is a misunderstanding, I believe a rewriting and clear formalism would help in understanding.\nMoreover, I want to emphasize the lack of links between different sections of the papers. Notations, like the function \"g\" are reused with different meanings, and the threshold \"epsilon\" disappears from the method.\n\nAdditional typo: line 147 \"setof\".", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for what we believe is an overall positive assessment of our paper. We especially appreciate that they find the topic interesting and our approach novel. We also agree this is a standout strength of this work. Additionally we thank the reviewer for observing that the generality and flexibility of our approach can make the work attractive to a broad audience, including alignment researchers, AI safety in robotics and other physical applications, etc. \n\nWe also feel optimistic about our ability to alleviate the reviewer’s questions, perceived weaknesses, and remarks concerning limitations of this work. We find almost all of the comments to be simultaneously fair and readily addressable. \n\nWe begin with the questions. Firstly, we address the question “Although early in the article, it's specified that, ‘we can imagine a user who wishes to determine whether a pre-trained model $f$ belongs to $\\mathcal{P}$ but is unable to train a model of $f$ ’s size’ (42), line 178 proposes to \"modify\" the model. How is that done?” \n\nThe “modification” to f that we are referring to in Definition 3 is abstract in the sense that it is considering the case in which $f$ had output $\\tilde{Y}$ instead of its actual output $f(X)$. No actual modification to the model is made, the underlying function $f$ remains a black box. We will edit this definition to make this point more clear. We thank the reviewer for bringing this confusion to our attention. \n\nThe second question, “Moreover, in the appendix (table 4), unconstrained models have varying performance when alpha varies, how is that explained? (moreover, no notion of uncertainty is provided on the metrics)” is even easier for us to address. This was simply a typo; the 114 for $\\alpha=0.05$ should have been 115 as for $\\alpha=0.1$ and $\\alpha=0.01$, after which you can observe the performance of the unconstrained model is consistent across varying $\\alpha$. We thank the reviewer for pointing out this typo.\n\nThe reviewer expressed concerns regarding multi-lambda risk control. We agree that the generalization is rather straightforward from the original conformal risk control and essentially mimics the one-dimensional case due to the mapping to $R$. We attempt to clearly state in both the proposition statement in the main text and the proof in the appendix that the proof closely mimics the original result. It is not our intention to frame this result as a significant technical contribution but rather a novel perspective on conformal risk control with application to alignment. \n\nWe also believe that even if the generalization to multi-lambda conformal risk control is straightforward, the literature has overlooked it so far, even in instances in which it would have been helpful. In particular, as mentioned in the global response, Yadkori et al. (2024) utilize two separate conformal procedures to calibrate two separate functions, which could have been simultaneously calibrated in tandem using multi-lambda conformal risk control. Other works, such as “Conformal Risk Control for Ordinal Classification” (Xu, 2023), assume that lower and upper bounds on a prediction set are given by functions l(lambda) and u(lambda) but requiring both ends of the interval to grow simultaneously may not be generally necessary; multi-lambda conformal risk control provides a flexible implementation of the conformal risk control algorithm in such settings.\n\nWe thank the reviewer for informing us that Section 2.1, introducing the preliminaries for property testing, is insufficiently clear. We hope to better explain the main ideas needed from property testing in this section and provide a more comprehensive introduction to property testing in the appendix in the final version, as suggested by Reviewer Hsyg. \n\nWe appreciate the feedback on insufficient clarity regarding the definition of the loss function in Section 3.2. We can rewrite this definition similarly to as follows:\n\n$$\nL_i = \\ell \\left( C_\\lambda (X_i), Y_i \\right) = \n0 \\text{ if } \\exists \\hat{Y} \\in C_\\lambda (X_i) \\text{ s.t. } T_i (\\hat{f}, X_i, Y_i) = \\text{Accept} \\text{, otherwise } 1\n$$\n\n$$\n\\text{where} \\quad \\hat{f} (X) = \nf(X) \\text{ if } X \\not= X_i \\text{, or } \\hat{Y} \\text{ if } X = X_i\n$$,\n\nWhere $T_i$ for $i=1,...$ is an infinite sequence of testers with fixed randomness for each $i$. That is, each $T_i$ is a deterministic function of its inputs $(\\hat{f}, X_i, Y_i)$.\n\nWe hope this definition provides sufficient clarity on the loss function. \n\nWe also look forward to the chance to clean and align the notation around $g$ and $\\varepsilon$, as mentioned by the reviewer. Thanks to the reviewer for pointing out the typo “setof” on line 147. We will fix these errors in the final version. \n\nWe hope all of these responses are satisfactory to the reviewer. As mentioned at the beginning, we appreciate that the reviewer gave us an overall positive review with respect to the significance, novelty, and applicability of our work. We also appreciate the 3s across the board from the reviewer on Soundness, Presentation, and Contribution. We hope our comments above have sufficient gravity to positively shift the reviewer’s overall score and assessment of our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Yv3EJeyFrb", "reviewer": "Reviewer_Hsyg", "summary": "The paper connects conformal risk control to define prediction sets containing results that satisfy a property. In short, assume that function $f$ (a trained model) does not satisfy some property $\\mathcal{P}$, we can define a prediction interval around results of $f$ s.t. those intervals satisfy $\\mathcal{P}$ with $1 - \\alpha$ probability. The authors first generalize conformal risk control to multidimensional conservative parameter, through a proxy from $\\mathbb{R} ^d \\mapsto \\mathbb{R}$, and use that in addressing the property testing. They define the risk as if there is no output in the prediction interval that satisfies the property.", "questions": "1. Why did the authors choose monotonicity and concavity as the properties to test. Is there any real-life use-case of those properties or does more justifiable properties remain as a possible future work? Ofcourse there is a dicussion on applications of monotonicity to causal inference, and fairness but is there any chance that the authors could provide experiments leading to those areas?\n\n2. Definition 2 has some confusing points:\n- Should there be any nested property over the prediction sets for various lambdas? Like if $\\lambda_a \\le \\lambda_b$ then $\\mathcal{C}(x_i; \\lambda_a) \\subseteq \\mathcal{C}(x_i; \\lambda_b)$? If yes then how the order is defined in this vector space? Does the nested property hold for a linear combination of $\\lambda_i$ values or the nested property should be present for any $\\lambda_i$. \n- How the interval is defined over function? Is it just the function value $\\pm$ some radius around it or is it a something apply over function parameters?\n\n3. In line 156, shouldn't we find the minimal $\\boldsymbol{\\lambda}_\\mathrm{min}$ instead of $\\boldsymbol{\\Lambda}_\\mathrm{min}$?", "limitations": "To the best of my understanding, the paper defines the guarantee over a property, and a joint guarantee of property + accuracy is not defined in its framework. I see that their extension of risk control supports multi-dimensional set constant not a multi-dimensional risk. However, I believe this is not a shortcoming of the method but a possible open problem to solve.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The idea is novel. Defining conformal property satisfaction based on property testing and conformal risk control opens a new area of work based on black-box post-hoc modifications of the model toward a desired property. I believe it can be further used in model-explainability, fairness, or tweaking the model toward constrained predictions.\n\nThe idea is nicely developed and nicely presented. It can easily engage an audience how is not familiar with conformal risk control or property testing. I would recommend an additional introduction to the aforementioned topics in the appendix, however the current version is also really nice.\n\nI also find the way the authors address fairness in section 5 (through bias in measurement error) really interesting.", "weaknesses": "1. Notations in the paper can be introduced in a better way. For instance a mathematic notation of $\\mathcal{P}$, which can be a subset of the function space.\n\n2. There are minor typos in the mathematical notation e.g. in line 88, $x \\sim \\mathcal{D}$ but $f(X) \\neq g(X)$. Also I do not understand why the authors used $\\mathrm{Min}$ instead of $\\min$; I believe that might have to relate to the fact that they are minimizing over vectors but still it could be indicated by $\\min_{\\boldsymbol{\\lambda}\\in\\boldsymbol{\\Lambda}}$. Also shouldn't we find the minimal $\\boldsymbol{\\lambda} _\\mathrm{min}$ instead of $\\boldsymbol{\\Lambda} _\\mathrm{min}$?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for a positive and confident assessment of our paper and appreciate the clear attention to detail shown by the reviewer. Both the strengths and weaknesses/questions/limitations make it evident to us that the reviewer invested significant time understanding the core of our paper. \n\nWe thank the reviewer for the direct assessment of our work as novel and capable of opening a new area in post-hoc modification of models to align with desired properties. We also thank the reviewer for finding our paper engaging to read and potentially applicable to domains such as explainability and fairness, and appreciate the interest given to Section 5 on the random feature model theory and its connections to alignment and fairness. \n\nWe find all of the reviewer's listed weaknesses helpful and easily addressable. We completely agree with the reviewer on the listed notation fixes and typos, we are happy to address those in a final version. In particular, we agree that we can use a lower case $\\min$ notation in place of Min. With regards to the question of finding the minimal $\\lambda_{\\text{min}}$ instead of $\\Lambda_{\\text{min}}$, this distinction was drawn to clarify that the set of minimal elements (which can have cardinality more than 1) is first formed by finding the set of $\\lambda$ that give a bounded risk below $\\alpha$, and then $\\lambda_{\\text{min}}$ is found by finding an argmin of the function $g$ from this initial set. Ultimately we agree that the notation in this section should be more clearly clarified to avoid confusion. Thanks to the reviewer for highlighting this and delving into detail on the multi-lambda risk control algorithm. \n\nWe thank the reviewer for their questions that clearly indicate engagement with our work. \nFor question 1, we feel that monotonicity is a natural first property to study in this framework. Monotonicity is well-established as a desirable property in many settings as described by the Wang and Gupta (2020) paper we cite at the beginning of Section 4.1. Moreover, it is also well-studied in the property testing literature. Following the reviewer's request for additional experiments, we next demonstrate new experiments in fairness and medicine to highlight the usefulness of monotonicity. \n\nAlthough we lack the time or scope to provide a comprehensive comparison of our method to the extensive fairness literature, given the reviewer’s request we have included in the pdf document attached to the global rebuttal a figure for results of applying our method to the UCI ML repo “Student Performance” dataset. This dataset is often used as a benchmark in the fairness literature (see “A survey on datasets for fairness-aware machine learning” Quy et al. 2022). Following the literature, we take “sex” to be the protected attribute and the final grade “G3” to be the real-valued outcome of interest. The attached figure demonstrates the lambdas needed to achieve a constant predicted grade when varying “sex” with a tolerance of 0.1 for $\\alpha$.\n\nAlso given the reference made to applications of monotonicity in medical triage at the beginning of Section 4.1, we have included another figure in the attached pdf showing the results of our method on the UCI ML repo “Liver Disorders” dataset. Here we take the target of average number of drinks per day (specified as target in the repo) to be monotonically increasing in the feature “mcv,” of which increasing levels are known to be predictive of liver disorders. This example from the high-stakles medical domain emphasizes that we believe monotonicity has significant importance in improving trust, interpretability, and hence adoption of clinical models.\n\nFor question 2, we wish to thank the reviewer for sharing their confusion with Definition 2. This definition was intentionally abstract, but we realize now it may have lacked sufficient clarity. \n\nIn the original CRC paper, $C_\\lambda$ is just taken to be any arbitrary function of the model and calibration data that outputs a prediction set, and outputs an increasingly large set as $\\lambda$ increases. It is not stated explicitly but appears to be assumed implicitly that if $\\lambda_a \\leq \\lambda_b$, then $C_{\\lambda_a} (X_i) \\subseteq C_{\\lambda_b} (X_i)$. \n\nWe operate under the same implicit assumption that for $\\lambda_a, \\lambda_b \\in \\mathbb{R}^k$, if $\\lambda_a \\leq \\lambda_b$, then $C_{\\lambda_a} \\subseteq C_{\\lambda_b}$. However, this does not have to be strictly increasing; we can have $\\lambda_a < \\lambda_b$, but $C_{\\lambda_a} = C_{\\lambda_b}$. This nested property holds for every dimension due to the fact that we follow the partial ordering of $\\mathbb{R}^k$ as defined in the main text.\n\n\n\nFor the question of how the interval is defined over function, we can define this as $C_{\\boldsymbol{\\lambda}}(f):\\mathcal{X} \\to \\mathcal{P}(\\mathcal{Y})$ is a set valued function where at each point $X_i$ we have $C_{\\boldsymbol{\\lambda}}(f)(X_i) = C_{\\boldsymbol{\\lambda}}(X_i)$. Thus it does not necessarily have to be a radius around the function output $f(X_i)$, but this is natural when operating in the reals. We point the reviewer to our discussion of the connection to Yadkori et al. (2024) within Section 6 Related work, which exemplifies a use case of our methodology that does not involve real-valued outputs. We hope the above discussion helps make Definition 2 more mathematically precise, and we look forward to the chance to revise this definition in a final version. \n\n\nFor question 3, similar to the discussion above on the lambda notation we agree that this can be improved and will do so for the final version. \n\nFinally, we agree with the reviewer with respect to both of the two stated limitations and also believe that these are exciting avenues for future research. \n\nAgain we would like to thank the reviewer for their compellingly positive and extensive assessment of our work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers for their positive feedback and critical assessment of our paper. We aim to address remaining concerns and reinforce the contributions of our work. This response focuses on key points that may have been underemphasized, aiming to further convince the reviewers who gave us Accept (7) scores of our work’s quality and persuade those who gave us Borderline reject (4) scores to reconsider positively.

Reviewers appreciated the flexibility and applicability of our approach to various problems, noting its potential for domains such as generative AI, physical applications, and fairness. There were questions, however, around our focus on monotonicity and how our approach extends to generative AI. We want to emphasize that our methodology allows us to define and align desirable properties of generative models as long as we have a way to test for these properties at a per-input level.

We wish to highlight this claim via a section of the paper that may have been too buried originally. The connection to Yadkori et al. (2024) within Section 6 Related work was not intended only for paying correct dues to their work, but also to highlight the applicability of our methodology to a generative AI setting. We discuss that their use of conformal risk control to mitigate LLM hallucination can be captured as a specific case of our more general property alignment perspective.

We discuss there in Section 6 that “not hallucinating” can be considered the property of the function $(a, f)$, where $a$ determines whether the model should abstain, and if not, the function $f$ gives the output. This example also shows how our approach of forming the conformal intervals is not limited to subsets of the reals since, in this application, the conformal set is either just $f(X)$ or $\\{f(X), \text{Abstain}\\}$ for sufficiently conservative $\lambda$. We also highlight the use of multi-lambda conformal risk control at the end of Section 6, since Yadkori et al. (2024) use a separate conformal procedure for calibrating their match function and abstention function, but using multi-lambda conformal risk control and our methodology would allow these functions to be calibrated in tandem.

Including a generative AI experiment is beyond this paper’s scope, but our discussion highlights our approach's potential. Our methodology enables defining and aligning generative models' desirable properties if testable at a per-input level.

A significant technical contribution, perhaps underemphasized, is in Section 5. Our theoretical results in this section build on random feature model theory (Mei and Montanari 2023, Misiakiewicz and Montanari 2023) to obtain insights regarding the impact of model size and data quality and volume on adherence to properties $\mathcal{P}$. We show that even if the true data generating process adheres to $\mathcal{P}$, small noise bias will cause overparameterized models to fail to satisfy the desired property $\mathcal{P}$, regardless of how much data is collected for training. This result has significant implications: even if we keep making models bigger and use unlimited training data, we still need alignment techniques because training data always has some small imperfections. 

Finally we want to share in this global response an additional experiment that we hope hammers home the general applicability of our method. We consider the “box” or sokoban environment [1] as presented in [2,3]. In this sokoban environment, of which we attached an image in the pdf, the Agent is rewarded for reaching the goal state in as few moves possible (-1 reward per move, 50 reward and termination of episode upon reaching goal state). There is also a box, and if the agent moves in the direction of the box the box is pushed in that direction. We train a tabular Q-learning agent for this goal, which expectedly converges to the optimal performance of 45. 

However, the authors of [2] consider the case in which we wish to penalize the agent from committing an “irreversible” action, which in this case is an action that pushes the box into a corner. They add a penalty term to the reward that punishes deviations from the initial environment’s trajectory to account for this. 

We use our alignment via conformal risk control to approach this safety problem. Our pretrained tabular Q-learning agent was not trained for side effects. Here, $f$ is $Q(s,a)$ mapping state, action pairs to q-values. The property of interest is whether it induces irreversible actions—whether the $\text{argmax}_a q$ value action for a state is irreversible.
Our calibration set is $X_i = s_i$ and $Y_i = 1$ if $a_i = \text{argmax}_a Q(s_i,a)$ is irreversible, 0 otherwise. The loss function for CRC is 0 if the highest q-value action is reversible or if the interval around the highest q value includes a reversible action’s q-value, otherwise 1.
Using CRC, we find optimal lambda pairs, with $\lambda^+=15.3$, $\lambda^-=0$. We provide a q-values distribution in the attached pdf. Our agent chooses the action with the max q-value if its interval is disjoint from others; otherwise, we defer to a safe action checker, choosing the highest q-value among safe actions. This approach is akin to the abstention problem in Yadkori et al. (2024), allowing the pretrained model to output its optimal action unless deemed too risky, in which case it checks for safe actions. Through our alignment via conformal risk control procedure we obtain an agent that achieves the optimal score of 43 while avoiding potential side-effects. 

[1] AI safety gridworlds. J Leike et al.

[2] Penalizing side effects using stepwise relative reachability. V Krakovna, L Orseau, R Kumar, M Martic, S Legg

[3] Avoiding side effects by considering future tasks. V Krakovna, L Orseau, R Ngo, M Martic, S Legg - Advances in Neural Information Processing Systems, 2020

Meta-review: This work discusses model alignment, and suggests a new frameworks towards doing so by combining ideas from property-testing and conformal risk prediction. And this idea seems novel and interesting as mentioned by some of the reviewers.
Nevertheless, the writing and clarity of the manuscript need to be improved as per the comments of the reviewers; and the technical novelty is quite mild. Nevertheless, conceptually the paper does bring novelty, and may very well spark new research directions.
From these reasons I view the paper as borderline, with tendency to accept.
Decision: Accept (poster)