Adaptive Learn-then-Test: Statistically Valid and Efficient Hyperparameter Selection
Authors: Matteo Zecchin
Sangwoo Park
Osvaldo Simeone
Venue: ICML 2025 spotlightposter (2025)
Keywords: Calibration
Learn-then-Test
Hyperparameter Selection
Primary area: probabilistic_methods->everything_else
TL;DR: An adaptive learn-then-test calibration procedure for efficient and statistically valid hyperparameter selection.

Abstract: We introduce adaptive learn-then-test (aLTT), an efficient hyperparameter selection procedure that provides finite-sample statistical guarantees on the population risk of AI models. Unlike the existing learn-then-test (LTT) technique, which relies on conventional p-value-based multiple hypothesis testing (MHT), aLTT implements sequential data-dependent MHT with early termination by leveraging e-processes. As a result, aLTT can reduce the number of testing rounds, making it particularly well-suited for scenarios in which testing is costly or presents safety risks. Apart from maintaining statistical validity, in applications such as online policy selection for offline reinforcement learning and prompt engineering, aLTT is shown to achieve the same performance as LTT while requiring only a fraction of the testing rounds.

Adaptive Learn-then-Test Statistically Valid and Efficient Hyperparameter Selection
Matteo Zecchin
Sangwoo Park
Osvaldo Simeone
Abstract
In the current scaling-centric era of large, Internet-scale, data sets (Xiao, 2024), hyperparameter optimization is typi- cally modeled as bandit problem in which the goal is the minimization of the training loss over a set of discrete con- figurations. The configurations may include prompt designs for the fine- seeettunig of large language models (LLMs) (Zhou et al., 2023; Quach al., 2023; Schneider al., 2024), ar- chitectural choices related model scale (Nasir et al., 2024) to the timing of carly decisions (Schuster et al., 2022), or policy settings for offline reinforcement learning (Paine et al., 2020; Fujimoto & Gu, 2021).
We introduce adaptive learn-then-test (aLTT), an efficient hyperparameter selection procedure that provides finite-sample statistical guarantees on the population risk of Al models. Unlike the exist- ing Searn-then-tes (LTT) technique, which relies conventional p-value-bol based multiple hypothe- sis testing (MHT), aLTT implements sequential data-dependent MHT with early termination by leveraging e-processes. As a result, al.TT can re- duce the number of testing rounds, making par- ticularly well-suited for scenarios in which testing is costly or presents safety risks. Apart from main- taining statistical validity, in applications such as online policy selection for offline reinforcement learning and prompt engineering, aLTT is shown to achieve the same performance as LTT while requiring only a fraction of the testing rounds.
The learn-then-test (LTT) method introduced by (An- gelopoulos et al., 2021) has recently emerged as a formal framework to address hyperparameter selection over dis- crete space. LTT treats the calibration task of hyperpa- rameter selection as multiple hypothesis testing (MHT) problem. Accordingly, it associates each hyperparameter in candidate set to the null hypothesis that the hyperparame- ter does not meet a reliability requirement on the population risk. Hypotheses are tested using p p-values, and the probabil- ity of mistakenly detecting hyperparameter as reliable guaranteed via familyy error rate (FWER)-controlling statistical procedures (Keselman & Rogan, 1977). This way, I.TT ensures finite-sample, high-probability guarantees on the population risk of the selected hyperparameters.
1. Introduction
1.1. Context and Motivation
The safe and reliable deployment of AI applications, or apps for short, hinges the possibility of certifying their perfor- mance (Seshia et al., 2022: Tegmark& & Omohundro, 2023). Depending on the problem, this may require controlling the missed detection probability in medical imaging (Lu et al., 2022; Mchrtash et al., 2020), ensuring safety measures for control policies (Lindemann et al., 2023; Zecchin et al., 2024), or verifying the correctness of the answers given by large language model (LLM) (Quach al., 2023).
To prevent p- p-hacking (Head et al., 2015), LTT's guarantees apply only to non-adaptive MHT procedures. However, re- lated work on hyperparameter optimization has shown the significant benefits that can be accrued by adaptive explo- ration strategies that test hyperparameters sequentially in data-driven manner (Swersky et al., 2014; Rakotoarison et al., 2024). This work aims at improving the data efficiency of LTT by leveraging recent advances in sequential MHT based on e-processes (Vovk & Wang. 2021; Waudby-Smin & Ramdas, 2024; Xu et al., 2021).
In practice, before deployment, AI apps can be often cali- brated by selecting hyperparameters based on data set aside for this purpose or based on rounds of real-world testing.
'Centre for Intelligent Information Processing Systems, De- partment of Engineering, King's College London, London, United Kingdom. Correspondence Sangwoo Park, Matteo Zecchin <sangwoo.parks www.aaaa.a......c. kcl.ac.uk uk, mattco. zecchin @ kel..cckk>>
1.2. Related Work
Finite-sample statistical guarantees on inferential outputs can be obtained via conformal prediction methods (Shafer Vovk, 2008; Angelopoulos al., 2023) and, more generally, via conformal risk control and risk-controlling set-valued predictions (Angelopoulos et al., 2022; Bates et al., 2021).
Proceedings of the nnnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 20255 by the author(s).

---

Adaptive Learn-then-Tess Test: Statistically Valid and Efficient Hyperparameter Selection
Hyperparameter Pre-Selection
Hyperparanntter Selection via aLTT
Post-calibration Seloction
Darssead takonaa
Candidate Hyperperameters
I' CCA
Predicted Reliable Hyperparandder
Selection Cttterim
Selecsed Hpperarraeeee
Brcetto an My ware aeater ows.
(1] cao mooeuend makk
(p...... Av)
1 I
Ard SA
I"
A ApMl
Figure 1. An example application of aLTT to reliable prompt optimization (Zhou et al., 2023; Quach et al., 2023; Schneider et al., 2024). A set A of candidate prompts for movie recommender is generated using L.M and/or prior experience. Prompts serve an example of discrete set of hyperparameters to be optimized using aL.T. The goal is to select subset Azel C A of prompts that guarantee a sufficiently high recommendation accuracy. To this end, aLTT applies a sequence of data-dependent testing rounds with adaptive termination. Specifically, at cach testing round f, aLTT estimates the performance of a subset of hyperparameters I' C A through held-out data or real-world testing. The subset be tested selected based on prior testing outcomes, and the process stops as soon as sufficiently large reliable subset A is identified. An additional post-calibration selection step can be applied to choose a single hyperparameter À from the selected subset Á based on users preferences.
These methods calibrate classification or regression models by setting& threshold hyperparameter based on held-out data to control the size of prediction set. Calibrated pre- dictors can be leveraged in geedictthen-optimiiz control tasks to offer reliability guarantees (Vovk& & Bendtsen, 2018; Lindemann et al., 2023; Zecchin et al., 2024).
bandit testing (Xu et al., 2021), two-sample testing (Shekhar & Ramdas, 2023), and mean estimation of bounded random variables (Waudby-Smith & Ramdas, 2024).
Hyperparameter optimization is vast field focused on the optimization of the hyperparameters of training algorithms, such as learning rate, weight decay, and dropout rate (Swer- sky et al., 2014; Pedregosa, 2016; Maclaurin et al., 2015; Lindauer et al., 2022). Hyperparameter optimization typi- cally operates in continuous domains, and it can serve as a preliminary step for the identification of candidate hyper- parameters. While hyperparameter optimization does not provide statistical guarantees on the population risk, the goal of hyperparameter selection methods such as LTT is to formally test a subset of candidate hyperparameters for statistical validity.
Beyond prediction sets, the problem of calibrating an Al app via the selection of hyperparameters from a candidate pool has been addressed through the LTT framework (An- gelopoulos et al., 2021). For example, as illustrated in Fig- ure 1, for the problem of prompt engineering (Zhou et al., 2023), the initial set of candidate hyperparameters encom- passes instruction prompt templates generated by an L.M and/or via prior experience. Leveraging p-value-basee MHT via FWER-controlling procedures, LTT selects a subset of candidates that come with probbabbbbii population risk guarantees.
1.3. Main Contributions
FWER guarantees are often too conservative, potentially resulting in empty calibration sets. The false discovery rate (FDR) is an alternative and less strict criterion that is often preferred for MHT in fields such as genetics (van den Oord & Sullivan, 2003), neuroimaging (Genovese et al., 2002), online advertising (Berman & Van den Bulte, 2022), and finance (Harvey & Liu, 2020).
The main contributions of this paper are as follows.
We introduce adaptive LTT (aLTT), a data-efficient hy- perparameter selection method that provides finite-sample guarantees on the population risk of Al apps. The main tech- nical underpinning of aLTT is e-process-bused MIIT, which supports statistical validity while enabling data detandepend sequential testing (Xu et al., 2021). Unlike I.T,, as illus- trated in Figure I, aLTT adaptively tests subsets of hyperpa- rameters that are chosen based on the evidence accumulated in the previous rounds, allowing also for the early termina- tion of the calibration process. aLTT guarantees rigorous control over FWER and FDR, while significantly reducing the number of testing rounds.
E-values have gained popularity in MHT due to their advan- tages over p-values ( (Vovk& & Wang, 2021; Shafer & Vovk, 2019; Ramdas et al., 2020). Similarly to p-values, e-values measure the statistical plausibility of the null hypothesis. Specifically, an e-value can be thought of as a special type of p-value that has additional robustness properties. The key property of interest in this paper is that, unlike p-values, e-values can be readily combined to obtain e-processes, making it possible to devise sequential testing strategies with anytime safety (Wang& & Ramdas, 2022; Ramdas et al., 2023). E- E-processes have been applied to problems such as sequential change detection (Shin al., 2022), multiple
We study two practical scenarios requiring hyperparam- eter selection, namely online policy selection for offline reinforcement learning (Fujimoto Gu, 2021) and auto- mated prompt engineering (Zhou et al., 2023). In both cases, aLTT is shown to deliver reliable and effective hyperparame- ters using only small fraction of the testing rounds required
2

---

Adaptive teen-thhh---stt Statistically Valid and Efficient Hyperparameter Selection
by LTT.
with the average evaluated with respect the distribution of the subset Arel
2. Problem Definition
Accordingly, testing procedure that outputs (a, 6)FDDR controlling sets guarantees that the expected fraction of un- reliable hyperparameters in the predicted set Å År is bounded by 8. Thus, ensuring the (a, 6)-FWER condition automati- cally also guarantees the (a, 5)-FDR requirement.
2.1. Setting
Let M2 be an Al app whose operation is determined by a vector of hyperparameters X. The performance of a hyper- parameter vector when tested at imput data Z is measured by a risk function R(,, Z) E [0. 1]. Accordingly, the popu- lation risk with respect to unknown data distribution Pz is defined as
Since any FWER FDR level can be trivially satisfied by procedure that returns the empty set Å = 0, it is important to gauge the informativeness of the testing procedure via the Irue positive rale (TPR), which corresponds to the expected fraction of reliable models in the predicted set Arel, i.c.,
R(A) = Ep, [R(A, Z)
(1)
As illustrated in Figure 1, for a given discrete subset = (AI....,Aw, of hyperparameters and a user-specified reliability level [0, 1], we aim determining the subset of hyperparameters in set A that conforms with the required reliability level a, i.e.,
TPR(Á - A n A Arel|
(6)
2.3. Sequential and Adaptive Hyperparameter Selection
To produce the estimated subset of reliable hyperparameters, A rrl we adopt a general sequential testing procedure that, at each round 2 1, operates follows.
A = {A E A R(A) S a}.
(2)
The complementary set, comprising unreliable hyperparam- eters, is accordingly defined as
L Hyperparameter subset selection: A subset of hyperparameters I' S is selected for testing.
A unrell A A () E A: R(A) > a}.
(3)
Since identifying the entire set Arel in (2) is impossible ow- ing to the lack of knowledge about the data distribution Pzz the goal is producing subset of hyperparameters rel S A that contains as many reliable hyperparameters from subset Are as possible while controlling the number of unreliable hyperparameters from subset A unrel mistakenly included in subset Arel_
2 Testing: Empirical risk estimates R [R(),, Zi) are obtained, one for each candi- date hyperparameter A in the selected subset It, us- ing held-out data or real-world testing. The random variable Z ~ Pz describes the data used to test hy- perparameter X; round t. The random variables {2, hizz can be arbitrarily dependent, and thus one may reuse the same data to test all hyperparameters T
2.2. Performance Criteria
Definition 2.1 ((o,, D--FWER-controllin
set). For a given 3 Evidence update: Evidence accumulated up to time t, including both the observed risks and the sub- set of queried models, is updated as D' = D' U {(I'RR')
reliability level a E [0. 1] and an error level E [0, 1]. a hy- perparameter subset Arel Ç A is (a, - set if satisfies the requirement
FWER(A*>)) == Pr [i | n Azely 2 i]
< 8. (4)
The testing procedure outlined above is fully specified by the tuple 1l = (( A,,,,, encompassing family of acquisition policies (Q' h22,, a decision rule A, and a calibration horizon T' which are defined as follows.
where the probability is evaluated with respect to the distri- bution of the subset re)
The FWER guarantee (4) imposes that the probability that the calibration set Å rel contains an unreliable hyperparame- ter is bounded by 5.
Acquisition policy: At each round f, the acquisition policy Oc determines the hyperparameters I' to be tested at step If the policy Q' uses the evidence 1 to select the hyperparameter set I', it is said to be adaptive; otherwise, it is non-udaptive. Both adaptive and non-adaptive acquisition policies can incorporate prior knowledge, which we denote as D
Definition 2.2 ((a, -D-oDtccnnto set). For a given reliability level E [0, 1] and an error level 8 e [0, 1], hyperparameter subset Arel C A is (a, -D-tooooonn set if it satisfies the inequality
unrell Arel
(A :
Aell
>
< 5,
(5)
Decision rule: The decision rule A uses the evidence DT available at the end of the last calibration round T to
|Asel|
3

---

Adaptive Learn-then-Test Statistically Valid and Efficient Hyperparameter Selection
produce the estimated set Arel of reliable hyperparameters.
each null hypothesis H, in (7), an e-value Eg is a non- negative random variable with expectation larger than when Hi is true, i.e.
Number of calibration rounds: The number of calibration rounds T, is said to be adaptive, if the stopping condition T = t is determined by the evidence D Otherwise, when it is predetermined based solely on prior knowledge DD, the calibration horizon T is said to be non-adaptive.
H4 < 1.
(9)
By Markov's inequality, an e-value E, can be turned into a p p-value Pa as P. = 1/ E,, since the inequality (8) is satisfied as
3. (Non-Adaptive) I Learn-then-Tess
1 E
Pr
x H4 S LEE S x, Ve E 10, 1]. (10)
In this section, we review LTT, non-adaptive hyperparam- eter selection procedure devised to meet the (a, 8) 8 guarantee (Angelopoulos et al., 2021). LTT associates to each hyperparameter X E A the null hypothesis
For each null hypothesis H4, given observation Z and fixed μ E (0, 1/(1 a)). a valid e-value is given by (Waudby-Smith & Ramdas, 2024)
H₂s : R(()) > a
(7)
FS = (1 + j(a- - R(,, Z)))-
(11)
that the population risk R(A,) in (1) violates the target relia- bility level a. For each null hypothesis Hi a p-value P; is non-negative random variable that satisfies the inequality
The e-value (11) has the interpretation of wealth growth in betting setting. Accordingly, one can think of parameter j in (11) as the amount of the current wealth that the gambler bets on the hypothesis H4 being false, i.e., on the validity of the assumption R(Ax) S a that the hyperparameter A; is reliable. In fact, if μ > O, when R(A, 2) S a, the gambler's wealth in (11) increases; while, when R(A,, Z) > a, the quantity (9) the gambler's ' wealth (11) decreases.
Pr[P: S [H.. S
(8)
for every x E [0. By the definition (8) a p-value P. provides evidence for the validity of hypothesis Hi, This is in the sense that a small value of P; is unlikely to occur if Hg is true.
Given pre-specified calibration horizon T and non- adaptive acquisition policy {Q')21, LTT queries at each round& t 2 the subset of hyperparameters Q' Q'(D") I' obtaining the corresponding risk estimates R¹ = {R(A,, 7)))x
An e-process for hypothesis H4 is a sequence of random variables {E | such that, for any stopping time T, which may depend on all previously collected evidence, the ran- dom variable F." is valid e-value. Using the e-value (11), considering the general iterative testing framework in Sec- tion 2.3, an e-process for the null hypothesis Hi in (7) can be obtained as the product
LTT uses the accumulated evidence at the end of the test- ing process, DT,, to compute a valid p-value for the null hypothesis (7) using, for instance, the Hoeffding-Bentkrr concentration inequality introduced in (Bates et al., 2021). Based on the collection of p-values P = { LTT selects a subset of hyperparameters ÄLTT using a FWER- controlling algorithm .A (F). A variant of LTT that is FDR FDR-controlling can be readily obtained by using an FDR- controlling selection rule (FDR(P), Examples of FWER and FDR controlling procedures are provided in the Supple- mentary Material.
E
II (1+17 (R(A 2))),, (12)
where the betting strategy 14 E 10, - a)) can be optimized as function of the past risk estimates ( (R(), Z7 Zf)) and E. = 1. Based on the discussion above, the e-process (12) represents the wealth accumulated up to time by gambler making sequential bets (14 Jree (Shafer & Vovk, 2019; Waudby-Smitt & Ramdas, 2024). As such, the gambler's wealth up to time t, E,, provides evidence against the null hypothesis that the hyperparameter X is unreliable.
4. Adaptive Learn-Then-Tes
In this section, we introduce adaptive LTT (aI.TT), a hyper- parameter selection scheme that supports adaptive acquisi- tion policies and an adaptive number of calibration rounds. The algorithmic description of aLTT is given in Algorithm
In similar way, anytime-valit p-value for hypothesis H, is sequence of random variables {P! such that, for any stopping time T, the random variable Pf is valid p-value. Given an e-process (E) for hypothesis H,, the sequence
P
(13)
4.1. Hypothesis Testing via E-Processes
max,51 E
The proposed aLTT scheme applics MHT based on e-values and e-processes (Shafer, 2021; Ramdas et al., 2023). For
is an anytime-vali valid p-value for the hypothesis H4 (Ramdas et al., 2023).
4

---

Adaptive Learn-then-Tess Statistically Valid and Efficient Hyperparameter Selection
Algorithm 1 Adaptive Leam-Then-Test (aLTT)
Require: Candidate hyperparameters A, prior knowledge reliability level a, error tolerance level 6, acquisition policy
(Q') h22, FWER/FDR-controllin selection rule (-WER())/APDR) (.), betting strategy (11)22 maximum
number of iterations max and minimal hyperparameter cardinality d hyperparameter set A
Ensure: (a.6) F
t-1
while S fmax A |A |
11 < d do
Select hyperparameters I' (E and receive risk estimates R = {R(Ai, MeT Update evidence D' = D ( (²)} and e-processes Ex as in (14)
if coERoc then
Compute p-values Pt as in (13)
Ä + AFWER(Pt)
else if FDR-control then
A A < (E")
end if
4- t+1
end while Return: AuITT,
4.2. Adaptive Acquisition Policy
4.3. Hyperparameter Subset Selection
aLTT applies an adaptive acquisition policy {2' h22 and an adaptive calibration horizon T. Specifically, each calibration round /. 1, aLTT's acquisition policy Q' uses the e-processes -t- = [E] /i=1 in (12) to choose which subset of hyperparameters, I', to test next. Examples of acquisition functions Z' = Qf (g will be provided in the next section.
At the end of the calibration process, aLTT uses the current e-processes Ed to generate the final prediction set Á A By the anytime validity properties explained in Section 4.1, if an FWER- FWER-controlling method .A AFWER(P") is used, the resulting set Ä is (a,, 8)-FW))-FFOF Contooootoolinng: while if an FDR FDR-controlling method is used, the resulting set ALLTT, = =AFDR(E" is (o, -)-FoDR-oont
For the selected hyperparameters set I', aLTT obtains the risk estimates R' (R(),, b,,ze and updates the associated e-processes using the recursive formula (12), i.e.,
Theorem 4.1. Given a hyperparameter set A, a reliability level a [0, 1). and an error level å E [0, 1], aLTT with FWER FDR-controlling selection rule returns a final prediction set Å ALLTT, that satisfies (a, 8--FEER control or (a, 8)-FDR control, respectively.
(1 (1+ RACGG-R(AA,, 2()) E F111
A E otherwise. (14)
5. Applications
5.1. Online Policy Selection for Offline Reinforcement Learning
Offline reinforcement learning enables the training of con- trol policies based on fixed data set collected by using possibly unknown behavior policy, without any online interaction with the environment (Levine et al., 2020). How- ever, the estimate of the performance of the trained policies obtained from offline data can differ substantially from the actual performance the real world. This makes prac- tically essential to validate the policies' performance via online interaction with the environment (Paine et al., 2020; Liu et al., 2023).
With this information, prediction set ÀaLTT. is evalu- ated by employing either FWER-controlling method A FWER (P') based on the p-values p = {P in (13); or an FDR-controlling procedure AFDR such as the e- --Bnjaammni--- -orhhirg (eBH) method, reviewed in the Sup- plementary Material (Wang& & Ramdas, 2022), using directly the e-values (14).
aLTT terminates the calibration procedure whenever there are at least d hyperparameters in set A&LTT, i.e., A 2 d, or a maximum number of iterations faax have been reached. This allows aLTT to stop the data ac- quisition phase early when a sufficiently large number of reliable hyperparameters have been identified..
To reduce the cost and potential harm of online validation procedures, the number of online interactions of the pre- trained candidate policies with the real world must be kept to a minimum (Gareıa & Fernández, 2015). To this end, in
5

---

Adaptive Learn-then-Teee Test: Statistically Valid and Efficient Hyperparameter Selection
FWER Control
FDR Control
1.0
1.0
aLTT, e-gredd aeq-
1.0
FDRt-couzoo
aLTT, non-adap.
0.5
0.8
0.8
LTT
0.8
Tarset, Beguineeen.
0.5
0.6
FWR
Position
0.6
EDR
0.4
0.4
0.4
0.4
0.2
FWEI--comzel control
0.2
0.0
0.0
0.0
0.0
0.2
0.6
0
2000
4000
2000
4000
Round (8)
Round (t)
Figure 3. Comparison of the FWER and FDR levels obtained by aLTT under FDR-contron (solid lines) and TWER-control (dashed lines) for different maximum tolerated error (FWER or FDR) levels 6.
Figure 2. True positive rate of LTT and aLTT with e-greody ac- quisition policy for e E (0.25, 0.5, 0.75, 0.95} and non- -dnaadaptvve acquisition. On the left panel. the prediction sets satisfy FWER control while on the right FDR control. both cases, the tolerance level is d = 0.1.
a), while the complementary set A unrel \ Areel includes the policies that do not satisfy the given reliability require- ment.
this subsection, we investigate the potential benefits of the proposed aLTT framework as a means to select a subset of candidate policies that enjoy performance guarantees with respect to the real-world environment.
Control policies are tested sequentially by following the procedure described in Section 2.3, such that at each cali- bration round a policy X E A is tested using online inter- actions with the MDP
5.1.1. PROBLEM DEFINITION
obtain the episodic reward value Ri = R( Z in (15). For an error threshold ō [O, 1). the goal of reliable online policy selection is to return pre- diction set Arel C A that is either (a,, 6)-FDR controlling or (a, 8)-FRER controlling with a TPR that is as large as possible.
We assume a standard Markov decision process (MDP) (S, A, Pa Pr]s.e} specified by state space S; an action space A; transition kernel as defining the conditional distribution of the next state s' E S given the current state s E S and action E A; and a conditional re- ward distribution Pr|s.a given state s E S and action a E .A. We assume that the reward is bounded and normalized in the [0, 1] interval.
5.1.2. RESULTS
In our experiments, we consider the Half Cheetah control problem from the OpenAl Gym MuJoCo tasks (Todorov et al., 2012) and use control policies obtained via the offline reinforcement learning algorithm TD3+BC (Fujimoto & Gu, 2021). The TD3+BC algorithm leverages an offline data set D to optimize policies by maximizing the standard deterministic policy gradient objective (Silver et al., 2014), combined with a behavioral cloning regularization term, whose strength is controlled by a hyperparameter in [Eq. 5](Sile et al., 2014). We produce N 20 different control policies by setting the hyperparameter À in the TD3+BC training objective on an evenly spaced grid in the interval [0.25, 5]. Unless stated otherwise, we consider a target reliability a = 0.57 and a target FDR requirement ó = 0.1.
We are given set of pre-trained control policies II ( NN mapping an observed state E S to the random action ~ Ts))) E A. Each policy m, is identi- fied by a hyperparameter X; The goal is to select a subset rel C A of byperparameters that yield reliable policies by using a limited amount of interactions with the real world.
Reliability is measured via the cumulative reward obtained by policy À on the MDP ε, which is defined
K
1 K
R(A, 2)
(15)
where Ki is the length of the episode, and the per-episode random variable Z encompasses the initial state s Pai along with the sequence a d',r',s² s2, a², y2 r?....,, K a ,. with actions k <s((s) rewards n Pr+t a a and MDP transitions *+1 ~ Pat+i For a user-specified reliabil- ity level a E (0. 1), the subset of reliable policies Tel ç A includes all policies in A with average cumulative reward larger than a, rel = (A E R(A) = Er, [R(,, Z)) >
We evaluate aLTT with an e-greedy acquisition policy Q' that, at every calibration round t, with probability selects the hyperparameter X; not included in A that is associated with the largest e- e-process value; otherwise, it picks uniformly at random a hyperparameter not in ÄaLTT, For reference, we also consider aLTT with a non- non-adaptive acquisition policy that, at each round t. picks uniformly at random the hyperparameter to be tested regardless of the
6

---

Adaptive Learn-then-Tess Statistically Valid and Efficient Hyperparameter Selection
prediction outcome ÀaLTT, and the e-process values. As benchmark, we implement LTT with random uniform quisition policy and p-values obtained from the e-processes as in (13). Recall that LTT produces a decision at the end of the calibration process, here at round 7 = 5000.
FWER Control
FDR Control
0.8
0.8
LLTT, c-groody aoq.
L.TT non-adap. zoq.
Trae Positive Hate
0.6
LTT
0.6
esii 3G
Finally, the value of the parameter μ in aLTT is set by following the approximate growth rate adaptive to the par- ticular alternative (aGRAPA) betting strategy in (Waudby- Smith & Ramdas, 2024) with other adaptive and non- adaptive betting strategies evaluated in the Supplementary Material.
0.1
0.1
0.2
0.2
75
0.0
0.0
10a
10a
10a
10a
Round (4)
Round (8)
In Figure 2, we compare the TPR of LTT and aLTT as a function of the calibration round I. We target FWER control on the left and FDR control on the right. By construction, LTT returns uninformative hyperparameter sets up until the termination of the testing procedure. The performance of LTT is the same as aL. with a non-adaptive acquisition policy, i.c. with e = 1, at f = T. aLTT with an e-greedy acquisition function can benefit from the accumulated evi- dence to adaptively determine the models to test next. As decreases, and thus the acquisition policy becomes increas- ingly driven by evidence, the TPR increases from 0.32 to 0.85 in the case of FWER control and from 0.4 to 0.85 in the case of FDR control. Finally, we note that the TPR of the schemes under FDR control is larger than that ob- tained under FWER control, reflecting the stricter reliability requirement of FWER control.
Figure 4. True positive rate of LTT and aLTT with e-greedy ac- quisition policy for e E [0.25, 0.5, 0.75, 0.95] and non-adaptive acquisition. On the left panel, the prediction sets satisfy FWER control, while on the right panel they meet FDR requirements. In both cases, the tolerance level 6 0.1. Results are averaged over the tasks in (Honovich ct al., 2022) that yield non-empty reliable prompts set.
tions and identify a sufficiently large number of prompts that meet user-defined reliability requirements.
5.2.1. PROBLEM DEFINITION
In Figure 3, we report the FWER and FDR of aLTT with 0.25 function of the tolerated error level 8. As the error level ò increases, the empirical FWER and FDR increase accordingly, remaining below the maximum target level 6. However, since FWER control is a stricter requirement, the aLTT prediction set obtained under FWER control delivers lower FDR and FWER levels as compared to aLTT with FDR control.
Consider a target LLM f() that, when fed the concatenation [A, X] of an instruction À and data X, produces an output text f([,, For example, in the task of movie recommen- dation illustrated in Figure 1, each prompt À corresponds to an instruction to generate movie titles similar to those provided in the input list X; while the output f(X,, rep- resents a list of recommended movies. Following (Zhou et al., 2023; Zhang et al., 2023), the candidate set A of instructions, corresponding to the hyperparameters to be tested, is generated using a separate LLM. In particular, we use the Llama3 8B Instruct LLM (Dubey et al., 2024) as the target model f()), while the initial instruction set A is generated through the Llama3.3 70B Instruct model (meta, 2025) using the forward generation mode detailed in (Zhou et al., 2023).
5.2. Reliable Automated Prompt Engincering
Prompt engineering focuses on designing and refining in- put instructions for LLMs (Reynolds McDonell, 2021: Shin et al., 2020). Recent studies have demonstrated the effectiveness of methods that automate this search using LLMs prompt generators (Zhou et al., 2023; Zhang et al., 2023). However, while LLMs are capable of producing high-quality instruction templates, at a level comparable to human annotators, supervision and testing remain essential to filter out poorly performing prompts.
For each instruction X, the ability of the prompted model S(A, XI) to generate high-quality outputs for a test datum Z (X,Y) is measured by task-dependent loss function R(A, 2) ((f((,, XJ), Y) E [0, 1]. Focusing on tasks from the instruction induction data set (Honovich et al., 2022), we specifically adopt the 0-1 loss tailored to the given task (Zhou et al., 2023). The goal is to determine a subset Ael C A that complies with the condition (2) with a target execution error a = 0.2, and FWER and FDR requirements given by the tolerance parameter 8 = 0.1.
In this section, propose applying aLTT to automated prompt engineering (Zhou al., 2023) to generate instruc- tions with statistical performance guarantees. As illustrated in Figure 1, given a set A of candidate instructions generated by an LLM, we use aLTT to sequentially test the instruc-
7

---

Adaptive Learn-then-Teee Test: Statistically Valid and Efficient Hyperparameter Selection
sults are averaged over tasks with final non-empty set Åre while plots for individual tasks are included in Appendix B.3. Stricter accuracy requirements are seen reduce the number of discovered reliable instructions, leading to an increase in the minimal instruction length within Arel, However, across all reliability levels a, aLTT consistently delivers the short- est instructions, outperforming alternative schemes. This advantage be attributed to the data-adaptive acquisition strategy implemented by aLTT, which enables the discovery of a larger number of reliable instructions.
FWER Control
FDR Control
Lengh of the Shortest Instruction in Awi
50
50
a.T.., e-greddy scq.
L.TT,, non-adap. seq.
45
LTT
45
1od,00
1od.0
40
35
35
30
30
0.4
0.6
0.8
0.4
0.6
0.8
Target Arcurayy
Target Accuracy
6. Conclusion
Figure 5. Length of the shortest instruction in the predicted set of reliable hyperparameter A returned by LTT and TLT.. Instruc- tions tested under different accuracy requirements and with fixed testing budget T = 2000.
We introduced aLTT, a novel framework for hyperparameter selection that implements data-dependent sequential test- ing via early termination. Unlike the existing LTT, which builds on p-value multiple hypothesis testing (MHT), aLTT is based on sequential MHT via e-processes (Xu et al., 2021). In practical scenarios, including the problem of prompt engi- neering, this results in more efficient and flexible calibration procedures that maintain statistical validity and the same discovery power as LTT by using only a fraction of testing rounds.
5.2.2. RESULTS
For the outlined prompt engineering problem, we eval- uate aLTT using an e-greedy acquisition strategy with e E f0.25, 0.5, 0.75, 0.95), as well as a non-adaptive ac- quisition strategy whereby the instruction to be tested is selected randomly and independently from prior testing rounds. All schemes employ an online Newton step betting strategy (Waudby-Smitl & Ramdas, 2024) for the e-value (12).
Potential extensions of the aLTT framework include the study of scenarios characterized by distribution shift, data reuse (Wang et al., 2025) and simulation -ided calibration.
A Acknowledgments
In Figure 4, we report the TPR of LTT and aLTT as a func- tion of the number of rounds t. Results are averaged over tasks that yield a final non-empty set Are Plots for in- dividual tasks are included in Appendix B.3. Under both FDR and FWER control, aLTT reduces the number of test- ing rounds, or equivalently the number of LLM executions, required to achieve a given TPR. In particular, LTT pro- duces a non-empty set only at the end of the testing process, attaining a final TPR less than half of the TPR achieved by aLTT with e-greedy acquisition strategy and e = 0.25. Moreover, aLTT with e 0.25 identifies 50% of the reliable instructions within the first 1000 testing rounds, while the non-adaptive testing procedure identifies less than 10% of the reliable instructions with the same number of rounds.
This work was supported by the European Union's 's Hori- zon Europe project CENTRIC (101096379). The work of Osvaldo Simeone was also supported by the Open Fellow- ships of the EPSRC (EP/W024101/I) and by the EPSRC project (7PX01185211) Thanks also to the Advanced Re- search and Invention Agency (ARIA) for supporting broader foundational work in this space.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
The improved efficiency of aLTT translates into superior performance in downstream tasks. To illustrate this point, we follow the described hyperparameter selection process with post-selection phase that identifies a single hyperpa- rameter Ā from the estimated set Are of reliable instructions (see Figure 1). Specifically, we adopt the shortest instruction A E Arel This post-selection criterion is motivated by the computational benefits of processing shorter prompts.
References
Angelopoulos, A. N., Bates, S., Candès, E. J., Jordan, M. I., and Lei, L. Learn then test: Calibrating predic- tive algorithms achieve risk control. arXiv preprint 1 2021.
Angelopoulos, A. N., Bates, S., Fisch, A., Lei, L., and Schuster, T. Conformal risk control. arXiv preprint arXiv:2208.0283 2022.
In Figure 5, we report the length of the selected instruction Ā as function of the target execution accuracy a. Re-
8

---

Adaptive Learn-then- -hen--stt: Statistically Valid and Efficient Hyperparameter Selection
Angelopoulos, A. N., Bates, S., et al. Conformal predic- tion: A gentle introduction. Foundations and Trends in Machine Learning, 1644))494-59, 2023.
Ferdosian, N., Othman, M., Ali, B. M., and Lun, K. Y. Greedy-knapsack algorithm for optimal downlink re- source allocation in LTE networks. Wireless Networks, 22:1427-1440, 2016.
Baruah, S. K., Cohen, N. K., Plaxton, C. G., and Varvel, D. A. Proportionate progress: A notion of fairness in resource allocation. In Proceedings of the twenty-fifth annual ACM symposium on Theory of computing, pp. 1993.
Fujimoto, S. and Gu, S. S. A minimalist approach offline reinforcement learning. Advances in Neural Information Processing Systems, 34:20132-201444 2021.
Garcia, J. and Fernández. F. A comprehensive survey safe reinforcement learning. Journal of Machine Learning Research, 16(1):1437-1400, 2015.
Bates, S., Angelopoulos, A., Lei, L.,, Malik, J., and Jor- dan, M. Distribution- free, rist-controlling prediction sets. Journal of the ACM (JACM), 68(6): 2021.
Genovese, C. R.,, Lazar, N. A., and Nichols, T. Thresholding of statistical maps in functional neuroimaging using the false discovery rate. Neuroimage, 15(4))870-878, 2002.
Bauer, P. Multiple testing in clinical trials. Statistics in medicine, 10(6):871-890 1991.
Harvey, C. R. and Liu, Y. False (and missed) discoveries in financial economics. The Journal of Finance, 75(5): 2503-2553, 2020.
Benjamini, Y. and Hochberg, Y. Controlling the false dis- covery rate: a practical and powerful approach multiple testing. Journal of the Royal statistical society: series B (Methodological), 57(1 3:29-3300, 1995.
Head, M. L., Holman, L.,, Lanfear, R., Kahn, A. T., and Jen- nions, M. D. The extent and consequences of p-hacking in science. PLoS biology, (3(3)e1002100) 2015.
Benjamini, Y. and Yekuticli, D. The control of the false dis- covery rate in multiple testing under dependency. Annals of statistics, pp. 1165-1188 1888, 2001.
Honovich, O., Shaham, U., Bowman, S. R., and Levy, O. Instruction induction: From few examples to natural language task descriptions. arXiv preprint arXiv:2205. 10782, 2022.
Berman, R. and Van den Bulte, C. False discovery in A/B testing. Management Science, ((9))6762-6782, 2022.
Bonferroni, C. Teoria statistica delle classi e calcolo delle probabilita. Pubblicazioni del R Istituto Superiore di Science Economiche Commericiali di Firenze, 5:3-62, 1936.
Keselman, H. and Rogan, J. C. The Tukey multiple com- parison test: 1953-1976. Psychological Bulletin, 84(5): 1050, 1977.
Laros III, J. H., Pedretti, K., Kelly, S. M., Shu, W., Ferreira, K., Van Dyke, J., Vaughan, C., Laros III, J. H., Pedretti, K., Kelly, S. M., al. Energy delay product. Energy- Efficient High Performance Computing: Measurement and Tuning, pp. 51-55, 2013.
Dabney, W., Rowland, M., Bellemare, M., and Munos, R. Distributional reinforcement learning with quantile re- gression. Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018
Levine, S., Kumar, A., Tucker, G., and Fu, Offline rein- forcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.0166 2020.
de Sant Ana, P. M. and Marchenko, N. Radio access schedul- ing using CMA-ES for optimized QoS in wireless net- works. In 2020 JEEE Globecom Workshops (GC Wkshps, pp. 1-6. IEEE, 2020.
Li, C., Song, S., Zhang, J., and Letaief, K. B. Maximizing energy efficiency in wireless networks with minimum average throughput requirement. In 2012 IEEE wireless communications and networking conference ( (WCNC), pp- 1130-1134. IEEE, 2012.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama herd of models. arXiv preprint arXiv:2407.21783, 2024.
Lindauer, M., Eggensperger, K. Feurer, M., Biedenkapp. A., Deng, D.,, Benjamins, C., Ruhkopf, T., Sass, R., and Hutter, F. SMAC3: versatile Bayesian optimization package for byperparameter optimization. Journal of Machine Learning Research, 23(54):1 2022.
Elbamby, M. S., Perfecto, C., Liu, C.-F., Park, Sama- rakoon, S., Chen, X., and Bennis, M. Wireless edge computing with latency and reliability guarantees. Pro- ceedings of the IEEE, 107(8):1177-1377 2019.
Farzaneh, A., Park, S., and Simeone, O. Quantile learn- then-test: Quantile-base risk control for hyperparameter optimization. IEEE Signal Processing Letters, 2024.
Lindemann, L.,, Cleaveland, M., Shim, G., and Pappas, G. J. Safe planning in dynamic environments using conformal prediction. IEEE Robotics and Automation Letters, 2023.

---

Adaptive Learn-then- Test: Statistically Valid and Efficient Hyperparameter Selection
Liu, V., Nagarajan, P., Patterson, A., and White, M. When is offline policy selection sample efficient for reinforcement learning? arXiv preprint arXiv:2312.02355 2023.
Ramdas, A., Ruf, J., Larsson, M., and Koolen, W. Admissi- ble anytime-valid sequential inference must rely non- negative martingales. arXiv preprint 2rXiv:2009.03167, 2020.
Lu, C., Lemay, A., Chang, K., Höbel, K., and Kalpathy- Cramer, J. Fair conformal predictors for applications in medical imaging. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 12008-12016, 2022.
Ramdas, A., Grünwald, P., Vovk, V., and Shafer, G. Game- theoretic statistics and safe anytime-valid inference. Stu- tistical Science, 38(4):576-601, 2023.
Reynolds, L. and McDonell, K. Prompt programming for large language models: Beyond the few-shot paradigm. In Extended abstracts of the 2021 CHII conference on human factors in computing systems, pp. 1021
Maclaurin, D., Duvenaud, D., and Adams, R. Gradient- based hyperparameter optimization through reversible learning. In International Conference on Machine Learn- ing, pp. 2112-2122. PMLR, 2015.
Schneider, L., Wistuba, M., Klein, A., Golebiowski, J., Zap- pella, G. and Merra, F. A. Hyperband-based bayesian op- timization for black-box prompt selection. arXiv preprint arXiv:2412.07820, 2024.
Mahapatra, R., Nijsure, Y., Kaddoum, G., Hassan, N. U., and Yuen, C. Energy efficiency tradeoff mechanism to- wards wireless green communication: A survey. IEEE Communications Surveys & Tutorials, 18(1):686-705, 2015.
Schuster, T., Fisch, A., Gupta, J., Dehghani, M., Bahri, D., Tran, V., Tay, Y., and Metzler, D. Confident adaptive language modeling. Advances in Neural Information Processing Systems, 35:17456-17472, 2022.
Mehrtash, A., Wells, W. M., Tempany, C. M., Abolmaesumi, P., and Kapur, T. Confidence calibration and predictive uncertainty estimation for deep medical image segmen- tation. IEEE Transactions on Medical Imaging, 39(1)): 3868-3878, 2020.
Seshia, S. A., Sadigh, D., and Sastry, S. S. Toward verified artificial intelligence. Communications of the ACM, 65 (7):46-55, 2022.
meta. Llama. https: /wL./LLLLMAM................. com LLaMA3, 2025. [Online; accessed 20055
Shafer, G. Testing by betting:
strategy for statistical and scientific communication. Journal of the Royal Statistical Society Series A: Statistics in Society, 184(2):407-431, 2021.
Nasir, M. U., Earle, S., Togelius, J., James, S., and Cleghorn, C. Llmatic: neural architecture search via large language models and quality diversity optimization. In proceedings the the Genetic and Evolutionary Computation Conference,
Shafer, G. and Vovk, V. A tutorial on conformal prediction.
Journal of Machine Learning Research, 9(3), 2008
pp. 1110-1118, 2024.
Shafer, G. and Vovk, V. Game-theoretic foundations for probability and finance, volume 455. John Wiley & Sons, 2019.
Nokia. Wireless suite, 2020. URL https:///ithh/ com/nokian wookkiawwiree rew.ss-sussssseeeeee
Paine, L., Paduraru, C., Michi, A., Gulcehre, C., Zolna, K., Novikov, A., Wang, Z., and de Freitas, N. Hyper- parameter selection for offline reinforcement learning. arXiv preprint arXiv:2007.09055 2020.
Shekhar, S. and Ramdas. A. Nonparametric two-sample testing by betting. IEEE Transactions on Information Theory, 2023.
Shin, J., Ramdas, A., and Rinaldo, A. E-detectors: non- parametric framework for sequential change detection. arXiv preprint WtXiv:2203.035322 2022.
Pedregosa, F. Hyperparameter optimization with approxi- mate gradient. In International Conference on Machine Learning, pp. 737-746. PMLR, 2016.
Shin, T., Razeghi, Y., Logan IV, R. L., Wallace, E., and Singh, S. Autoprompt: Eliciting knowledge from lan- guage models with automatically generated prompts. In Proceedings of the 2020 Conference on Empirical Meth- ods in Natural Language Processing (EMNLP). Associa- tion for Computational Linguistics, 2020.
Quach, V., Fisch, A., Schuster, T., Yala, A., Sohn, J. H., Jaakkola, T. S., and Barzilay, R. Conformal language modeling, arXiv preprint arXiv:2306. 201.93330.00193, 2023.
Rakotoarison, H., Adriaensen, S., Mallik, N., Garibov, S., Bergman, E.,, and Hutter, F. In-context freeze-thaw Bayesian optimization for hyperparameter optimization. arXiv preprint a arXhv:2404.16795 2024.
Silver, D.,, Lever, G., Heess, N., Degris, T., Wierstra, D., and Riedmiller, M. Deterministic policy gradient algorithms. International Conference on Machine Learning, pp. 387-395. Pmlr, 2014.
Ramdas, A. and Wang, R. Hypothesis testing with e-values. arXiv preprint arXiv:2410.2 236/44 2024.
10

---

Adaptive Learn-then- Statistically Valid and Efficient Hyperparameter Selection
Stanczak, S., Wiczanowski, M., and Boche, H. Fundamen- fals of resource allocation in wireless networks: theory and algorithms, volume 3. Springer Science & Business Media, 2009.
Representations, 2023. URL net/ (et.rrrmiisnnttBBCC.. jUHkr.
Zhou, Y., Muresanu, A. I., Han, Z., Paster, K., Pitis, S., Chan, H., and Ba, J. Large language models are human-level prompt engineers. In The Eleventh In- lernational Conference Learning Representations, 2023. URL https://apenrew:: /openreview net ffrrm??r 1d=92gvkB2DE-
Swersky, K., Snoek, J., and Adams, R. P. Freeze-thaw Bayesian optimization. arXiv preprint arXiv:1406.3800 2014.
Tegmark, M. and Omohundro, S. Provably safe sys- tems: the only path to controllable AGI. arXiv preprint arXiv:2309.01933 009999,93,,,...............,,333 2023.
Todorov, E., Erez, T., and Tassa, Y. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ Inter- national Conference Intelligent Robots and Systems, pp. 5026-5033. IEEE, 2012.
van den Oord, and Sullivan, F. False discoveries and models for gene discovery. Trends in Genetics, 19(10): 537-542, 2003.
Vovk, V. and Bendtsen, Conformal predictive decision making. In Conformal and Probabilistic Prediction and Applications, pp. 52-62. PMLR, 2018.
Vovk, V. and Wang, R. E-values: Calibration, combination and applications. The Annals of Statistics, 49(3):1736 1754, 2021.
Wang, H., Dandapanthula, S., and Ramdas, A. Anytime- valid fdr control with the stopped e-bh procedure. arXiv preprint arXiv:2502.085.3 08539, 2025.
Wang, R. and Ramdas, A. False discovery rate control with e-values. Journal of the Royal Statistical Society Series B: Statistical Methodology, 84(3):822-852, 2022.
Waudby-Smith, I. and Ramdas, A. Estimating means of bounded random variables by betting. Journal of the Royal Statistical Society Series B: Statistical Methodol- ogy, 86(1):1-27, 2024.
Xiao, Rethinking conventional wisdom in machine learning: From generalization scaling. arXiv preprint 27Xiv22409.15150 2024.
Xu, Z., Wang, R., and Ramdas, A. A unified framework for bandit multiple testing. Advances in Neural Information Processing Systems, 34:16833-16845 2021.
Zecchin, M., Park, S., and Simeone, O. Forking uncertain- ties: Reliable prediction and model predictive control with sequence models via conformal risk control. IEEE Journal on Selected Areas in Information Theory, 2024.
Zhang, Z., Zhang, A., Li, M., and Smola, A. Automatic chain of thought prompting in large language models. In The Eleventh International Conference on Learning

---

Adaptive Learn-then-Tee Statistically Valid and Efficient Hyperparameter Selection
A. FWER and FDR-Controlling Procedures
In this section we review popular FWER and FDR-controlling strategies applied to the problem of hyperparameter selection.
A.1. FWER Control
A.1.1. BONFERRONI CORRECTION
For set of p-values { a simple (a, -oERooollnnnn procedure given by the Bonferroni correction
A AFWER (P), which returns the hyperparameter set
A
rel Bon
A, P S N
(16)
The hyperparameter set Arel is guaranteed be d-FWER-controlling (Bonferroni, 1936).
A.1.2. TIXED-SEQUENT TESTING
If the designer has access to prior knowledge D about which hyperparametens are likely to be more reliable, it is possible to increase the power of the statistical test, and thus the cardinality of the hyperparameter Arel, by using fxfiedd-eeee eexuenee testing procedure FWEER (P, ) In fixed sequence hypothesis testing, the hypotheses (H4 are ordered hased on the A prior knowledge D from the most likely to reliable to the least likely to be reliable. Denote the k-th hypothesis the corresponding ordered sequence as H(K), and the associated p-value as P(R). With fixed-sequence testing, the hypotheses are sequentially tested at a reliability level until one is accepted. Accordingly, the resulting hyperparameter set contains all hypotheses up until the first acceptance, i.e.,
Aps (>G) P()) S0 Vi 5
(17)
For any ordering of the p-values, the hyperparameter set Apg is guaranteed to be 6-FWER-controlling (Bauer, 1991).
A.2. FDR Control
1. BENAMIN--HOCIBBER PROCEDURE
Given a set of independent p-values P, denote the k-th smallest value in the set as P(k) and the associated hyperparameter as A(a)- For an error level the Benjamini-Hochberg Hochberg procedure AFDR (P) returns the prediction set
re] BH
A0 P(o I
id N
(18)
By (18), BH applies a larger threshold to hyperparameters À E A that have larger p-values and are thus less likely to be reliable. the p-values in set P are independent, the Honjaminn-HHocheee (BH) procedure FDR (FD(P) (P) guarantees FDR control level (Benjamini & Hochberg, 1995).
A.2.2. BENJAMINI- YEKUTIELI PROCEDURE
In case of arbitrarily dependent p-values in set P the enror level 5 has to be adjusted by multiplicative factor (Σ 1 1/n)-1 The resulting FDR-controlingg procedure, also known as the Benjamini-Yckuttelii Yekutieli (BY) procedure, yields the set
{ Aa1= P() < ΣN N/n
id
A Any
(19)
The BY procedure 4PDR (P) is 6-FDRR-contooling (Benjamini Yekutieli, 2001).
12

---

Adaptive Learn-then-Teee Test: Statistically Valid and Efficient Hyperparameter Selection
FWER Control
FDR Control
1.0
aGBAPA
1.0
&GRAPA
LBOW
LBOW
0.8
ONS
0.8
True Posiive Bate
Max Bet
True Podiive Rate
Max
0.6
Unit
0.6
Unis
0.2-
0.2
0.0
0.0
1000)
2000
3000
4000
1000
2000
3000
4000
Round (0)
Round (!)
Figure 6. True positive rate of aLTT with e-greedy acquisition policy with = 0.25 under different betting strategies. On the left panel. the prediction sets satisfy FWER control while on the right FDR control. In both cases, the tolerance level is 0.1.
A.2.3. E-BENIAINN--OCHEEEE PROCEDURE
Alternatively, given the set of e-values g = [ denote by the k-th largest e-value in as E and the associated hyperparameter as A(j). For error level 8, the @-Benjamini-Hochheer procedure ABHH (E) outputs the hyperparameter subset
AFDR(E)
{N0) A1)
N 10
B
(20)
Following the same basic principles underlying BH in (18), eBH applies smaller threshold to hyperparameters À E A that have smaller e-values and are thus less likely to be reliable. The eBH procedure returns an 6-FDR-controlling set even for arbitrarily dependent e-values (()) without the need to adjust the error level & as in the case of the p-value-based BH procedure (Xu et al., 2021).
B. Additional Experiments
B.1. Online Policy Selection for Offline Reinforcement Learning
B.I.1. EFFECT OF THE BETTING STRATEGY
In Section 5.1, we studied the TPR of aLTT under the approximate growth rate adaptive the particular alternative (aGRAPA) betting strategy. In the following, we compare its performance against non-adaptive betting strategies, specifically the maximum bet (MaxBet) strategy, which sets the bet μu to the maximum allowed value of 1/a, and the umitary bet (UnitBet) strategy, where 1 = 1. Additionally, we evaluate two alternative adaptive online betting strategies based on approximate wealth maximization (Waudby-Smin & Ramdas, 2024): lower bower-bound on the wealth (LBOW) and online Newton step (ONS).
Under the online policy selection for offline reinforcement learning set up in Section 5.1, in Figure 6 we compare the TPR of aLTT with an e-greedy acquisition policy. The left panel targets FWER control, while the right focuses FDR control. All adaptive betting strategies exhibit similar performance, with aGRAPA showing slight advantage. Interestingly, in this scenario, the non-adaptive MaxBet strategy performs surprisingly well, while UnitBet achieves the lowest TPR.
B. 1.2. QUANTILE RISK CONTROL
In this experiment, we consider the same set-up of Section with the caveat that, instead of identifying policies with a large enough average reward, we focus on policies with a small enough quantile risk risk definition that explicitly captures safety (Dabney et al., 2018; Farzaneh et al., 2024). The quantile risk of hyperparameter À at level is defined as
R2(A) (r: Pr(R(A, Z) <r\> 1-9) 9)
(21)
13

---

Adaptive Learn-then-Tess Statistically Valid and Efficient Hyperparameter Selection
PWER Control
FDR Control
1.0
aLTT, e-greddy acq:
1.0
non-adap. uoq.
Troe. Posétive Rate.
0.8
LIT
0.8
0.6
Goll
Coff
0.4
0.4
0.2
0.0
2000
40000
6000
0
2000
4000
0000
Round (4)
(9)
Figure 7. True positive rate of LTT and aLTT with e-greedy acquisition policy for e E (0.25, 0.5, 0.75, 0.95] and non-adaptive acquisition. On the left panel, the prediction sets satisfy FWER control while the right FDR control. In both cases, policy considered reliable if the quantile risk at level 0.1 is lower than 0.37, and the tolerance level is 5 = 0.1.
FWER Coutrol
FDR Control
1.0
1.0
0.8
0.8
Thoo Poellive Rtde
0.0
Ket
0.5
0.4
0.4
0.2
0.2
top- 0.25-greedy neq
0.0
0.0
2000
4000
2000
4000
(2)
(8)
Figure True positive rate of aLTT as function of number testing rounds, using a top-K c-greedy acquisition policy with 0.25 and K E 3142,3,4).
Controlling the quantile risk ensures guarantees on the average fraction times the policy risk exceeds the level (A).). For the RL setting, we deal with rewards, so the inequalities the definition are reversed.
We aim to identify policies with quantile risk at the level q 0.1 that is smaller than a 0.57. In Figure report the true positive rate of aLTT using the e-greedy acquisition policy and LTT. aLTT significantly outperforms LTT, identifying up to 75% of the policies that satisfy the given reliability requirement within the maximum testing round, T. In contrast, LTT is unable to identify any policy due to its non-adaptive testing strategy.
1.3. TOP-K K e-GREEDY ACQUISITION STRATEGY
In Figure 2, we instantiate aLTT using an c-greedy acquisition strategy. With probability 1 c, the strategy selects the hypothesis associated with the largest e-process among those that have not yet been rejected. With probability e, it selects a hypothesis random from the remaining unrejected set. A natural generalization of this strategy is the top-K e-greedy acquisition strategy. In this variant, with probability - c, aLTT tests the set of the K hypotheses associated with the largest e-processes that have not yet been rejected; with probability it tests K hypotheses selected uniformly at random from the entire set of unrejected hypotheses.
In Figure 8, we report the FDR of aLTT using this top-K c-greedy strategy with c = 0.25 and varying values of (1,2,3, 4]. As K increases, the TPR performance of aLTT improves. This is expected, as the effective number evaluations per testing round increases proportionally with K allowing aLTT discover reliable models in less testing rounds.
14

---

Adaptive Carm-thhe-Tesst Statistically Valid and Efficient Hyperparameter Selection
FWER Control
FDR Control
155
Predicted Set Sage
6-
4
4-
aLTT, e-greedy acy.
0-
LIT
0.36
0.57
0.58
0.56
0.57
0.58
Reliability Level (a)
Reliability Level
Figure 9. Size of the prediction set returned by LTT and aLTT with a top- e-greedy acquisition policy for varying reliability levels a. As increases, the reliability requirement becomes stricter, requiring larger expected reward R(A) Ery [R(A, Z)] > a. For all values of a, aLTT is able to discover larger number of reliable policies.
B. 1.4. RELIABILITY LEVEL
In the following, we compare the efficiency of aLTT and LTT under different values of the reliability level a. In Figure 9, we vary a E [0.565, 0.58) and report the expected size of the prediction set returned by LTT and aLTT after T = 5000 testing rounds.
In the context of the online policy selection problem, a higher reliability level a imposes a stricter requirement on the expected reward of the policies, namely R(A)- Ep [R,,, Z)] > a. As a result, the size of the prediction sets decreases as increases. However, for all values of a, aLTT is capable of identifying a larger number of reliable policies compared to LTT. For example, under FWER control, when 0.575, LTT fails to find any reliable policies, whereas aLTT discovers up to four reliable policies.
B.2. Reliable Hyperparameter Selection for Wireless Resource Allocation
In wireless communication systems, resource allocation is essential functionality that regulates access to the spectrum for users and services (Stanezak et al., 2009). The performance of resource allocation policies is evaluated by using key performance indicators (KPIs) such as throughput, delay, and energy efficiency. Despite the randomness inherent in the network conditions, some services require strict reliability guarantees in terms of KPIs. For instance, gaming applications must meet latency constraints (Elbamby et al., 2019), streaming connections are subject to throughput requirements (Li et al., 2012), and battery-powerer transmitters have strict energy-efficiency constraints (Mahapatra al., 2015).
B.2.1. PROBLEM DEFINITION
As illustrated in Figure 10, we consider downlink resource allocation problem in which, at every transmission frame k > 1, base station serves users by communicating bits from the corresponding queues. This is done by assigning physical resource blocks (PRBs) to users based on descriptor s of the network conditions that includes the state of users' transmission buffers, users' priorities, and channel conditions. Following the simulation software Nokia Wireless Suite (Nokia, 2020), we consider three different types of PRBs assignment policies: a proportional fair scheme (Baruah al., 1993), a knapsack allocation policy (Ferdosian al., 2016), and a learning- based scheme based on a Covariance Matrix Adaptation Evolution Strategy (CMA-ES) (de Sant Ana & Marchenko, 2020). Overall, a resource allocation scheme is specified by the hyperparameters À {>1, with X (proportional fair, knapsack, CMA-ES) identifying the scheduling policy, and d2 determining the transmit power level. We let the hyperparameter x2 vary from 0 dB to 24 dB in I dB increments. This way, the set A contains 75 candidate byperparameters.
An instance of the resource allocation problem is described by the random quantity Z, which encompasses channel conditions and packet generation. Furthermore, the performance of the resource allocation hyperparameter À is measured by the following KPIs: (i) averuge transmission delay of packet Tex (A, Z); (ii) overall energy efficiency, which is defined as the ratio between the number of transmitted bits and the overall transmitted energy within the episode, p(), Z) EK_ B*/ / E where B and E are the number of transmitted bits and transmit energy at slot k; (iii) average queue length occupancy (2,, Z), which is normalized by the maximum queue size Qmax: (iv) average mnrgy-delay product,
15

---

Adaptive Searn-then-Tesh Statistically Valid and Efficient Hyperparameter Selection
Packet Arrivals Uscrs' Butfers
High peiority
Resource Allocation
Packet Transmission
Low peiority
Network's state Rosouree ussignment scheme (A₂)
Transmit Power Level (A²)
Figure 10. System diagram of the considered resource allocation problem. At each transmission frame k; new packets randomly generated and stored in the corresponding users' buffers. A resource allocation scheme, determined by hyperparameter A1. assigns resources to the users, deciding whose users schedule round k:. Packets from the scheduled users' queues are then transmitted over wireless channel using a transmit power dictated by the hyperparameter A2.
8 FWER Control
FDR Control
Energe- Delay Product jubx msl
1.2
sLTT, e-greedy soq.
LTT
1.0
0.8
0.6
0.4
0.2
0.0
0.15
0.20
0.25
0.15
0.20
0.25
Avg. Queue Oerupancy
Avg. Quear Ooepanccy
Figure 11. Average energy-delay product of the resource allocation policy returned by LTT and aLTT with e-greedy acquisition policy with e E (0.25, 0.5, 0.95] under different queue occupancy efficiency requirements. The final hyperparameter is obtained by selecting the hyperparameter in the estimated set of reliable policies rel that is associated with the lowest empirical energy-delay product. On the left panel, the prediction sets satisfy FWER control, while on the right FDR control, with the tolerance level 5 0.1.
k== E* /K, quantifying the overall performance in terms of energy and delay (Laros III et al., EDP(A, Z) (A, Z) Ek_ 2013).
B.2.2. RESULTS
We study a setting with differentiated service requirements. particular, we enforce a constraint on the average queue occupancy and energy efficiency for high-priority users, while making a best-effort attempt at minimizing the energy- delay product for low-priority users. Writing as (HI( Px (A, Z)] the average queue occupancy of high- priority users, and ا Er, (A, Z)] the average energy efficiency of high-priority users, and as (01, the corresponding requirements. A hyperparameter is reliable if it meets both the queue occupancy and energy efficiency requirements for high-priority users, i.e., A = {A E A (") < n pP""(A) 2 a2). An e-process for the hypothesis Hi QHI (Ax) > an n (Ar) < a2 can be obtained as the minimum between an e-process for hypothesis HC (A+) > a1, and an e-process for the hypothesis HZ (X,) < 022 Note that any e-merging function can be applied to ensure validity over the joint hypothesis (see, e.g., (Vovk & Wang, 2021; Ramdas Wang, 2024)), we choose minimum for simplicity.
(p
compare the performance of LTT and aLTT for calibration horizon T 4000 by choosing in the estimated set of reliable hyperparameters Arel the hyperparameter Å that minimizes the empirical energy-delay product of low-priority users based on all the data DT collected across the testing steps. We vary the queue occupancy reliability level a while fixing the emeryy-efficience requirement a₂ = 0.01 Mbit/Joule. As the queue occupancy target aj increases, the requirement becomes less stringent, and the energy-delay product the low-priority users under the returned policy decreases. In both cases FWER control on the left and FDR control on the right-the the performance of aLTT is significantly
In Figure 11,
16

---

Adaptive Caan-tthen-Test. Statistically Valid and Efficient Hyperparameter Selection
FWER Control
FDR Control
37.5
aLT' w/ cugredy ucq.
LIT
35.0
35.0
0m0.0
cm0.0
5
32.5
32.5
Average Dolay
em0.75
30.0
30.0
25.0
25.0
22.5
22.5
0.225
0.50
0.75
0.22
0.50
0.75
Energ Eff. Rug. [Mbit/Joule]]
Energy
Rog. [Mbit/Jonke]
Figure 12. Average delay of the resource allocation policy returned by LTT and aLTT with e-greedy acquisition policy with (0.25, 0.5, 0.95) under different energy efficiency requirements. The final policy is obtained by selecting the policy the estimated set of reliable policies Are that is associated to the lowest empirical delay. On the left panel, the prediction sets satisfy FDR control, while on the right FWER control, with the tolerance level 6 0.1.
superior to that of LTT. What is more, in this example, the looser guarantees provided by FDR do not entail any reliability loss, since the FWER of the hyperparameter Ā returned by FDR-controlling procedures was found be significantly below 6 = 0.1.
B.2.3. MINIMIZING AGER DELAY UNDER AN ENERGY EFFICIENCY REQUIREMENT
Now we consider the wireless resource allocation problem under a reliability requirement on the average energy efficiency, Specifically, we consider the population risk R(A) = Ep [p(A, 2)], so that the set of reliable hyperparameters is given by A (X A R(A) Z a for the given target energy efficiency a. After T 1000 testing steps, with each step consisting of a resource allocation episode of K 2500 frames, the hyperparameter selection scheme selects from the estimated subset of reliable hyperparameters A the hyperparameter À that minimizes the empirical communication delay (x,(,, D 1/ | (A, Z) based all the data D" collected across the testing steps.
In Figure 12 we report the average delay (Ā) = Ep [Tix (², Z)], estimated on hold-out data, of the selected policy Ā as a function of the reliability level a for LTT and aLTT with an e-greedy acquisition function, when setting the target error level 8 = 0.1. On the left panel, consider FWER control, while the right presents the performance with FDR control.
As the reliability requirement increases, becoming more stringent, the average delay of the selected hyperparameter increases. However, thanks to adaptive testing, aLTT returns larger prediction set rel providing hyperparameters with lower delays as compared to LTT. Furthermore, due to the less stringent FDR requirements, FDR-controlling testing procedures generally return policies with average delays smaller than FWER-controlling.
B.3. Reliable Prompt Selection for Natural Language Processing
In Figures and 5, we report the performance averaged over ten tasks from the instruction induction dataset (Honovich et al., 2022). Complementing the results in the main text, here we provide a breakdown of the methods' performance across the ten tasks. In Figure 13, we show the TPR of the schemes considered in Section 5.2.2 under FDR and FWER control with tolerance 6 0.1. Across all tasks, aLTT consistently outperforms LTT, which returns empty sets for many tasks.
In Figure 14, we plot the length of the shortest instruction in the predicted set r returned by the different hyperparameter selection schemes a function of the execution accuracy 1 a. If prediction scheme returns empty predicted hyperparameter set, we set the instruction length to the length of the longest instruction in the candidate set A. For all tasks, aLTT can identify shorter instructions compared to LTT under the same testing budget.
B. EXAMPLE PROMPTS
list of the tested prompts together with their accuracy levels can be found at:
17

---

Adaptive Learn-then-Tese Statistically Valid and Efficient Hyperparameter Selection
FWER Cantrol
Iarger animal
diff
Bow, word lettes
letters lust
taconieny animal
0.4
eigendy any
The Footrr Bate
6.4
Toa Fostum Tato
0.3
aLTTU notvadaapp..o oay
02.00
0.23
The
6.9
0.0
16f
107
DP
J0
107
10
Reand (1)
Roard
Round 85
Roodd (0)
Beand
to seriid
rlymoss
transiation mhii
singrllar fear pluna
The Poatte Boan.
Pastire Buee
0.63
0.00
0.23
3.00
0.0
0.0
107
107
107
10
10
10"
1P
10"
Reand (1)
Toand (6)
Dound (1)
Domnd (1)
Tound
lurer Junncal
Sast wood, Jetter
lettess,lat
Tasononny animal
Tue Poltowe 2tAt
0.4
alll, adyy anc
ETT
0.50
Poitor
0.2
0.0
0u3
040
0.0
0.0
(1)
Bound
Round
(1)
cum to verkel
rhymes
troradatiin ente
singuler plorsl
The Pualee Bute
Thee Pualie Bite
0.4
0.4
1..0
Tou
02
0.0
107
Board 61
Dound (5)
Beand
Round
(9)
Figure 13. Breakdown of the average performance of methods presented in Figure 4 across the 10 tasks in the instruction induction dataset (Honovich et al., 2022) which a non-empty hyperparameter sets A returned. Across all tasks, aLTT delivers the highest true positive rate.
git
18

---

Adaptive Learn-then-Tess Statistically Valid and Efficient Hyperparameter Selection
FWER Cantrel
Iager Jnime
Eist Jattor
letton
Tacnnoniy aniimal
300
esppedo adu
LLTT, amaeoday sou
63.3
40
650-
0.4
0.5
0.5
0.4
0.6
0.4
0.6
0.8
0.5
0.5
0.0
Tram to webbal
trotelation en
singtiar to plunal
00.0
10-
01.5
4.-
50
20
03-
42.5
0.66
0.4
1..
6.4
0.01
0.4
0.8
0..6
0.8
0.5
000
6.1
01.
65
Control
poor Jetter
lettons
enamy.
50.0
50
50
50
47.5
1271
10-
20-
30
InNI tu verbvll
Flymes
teamelacion,
eingnlar fo. piuml
50
02
50
506
30
50
Figure 14. Length of the shortest instruction in Á as a function target execution accuracy - a. for each task in the instruction induction dataset (Honovich et al., 2022) for which the methods presented in Figure 5 return non-empty hyperparameter A
19


Reviews:
[{"review_id": "bSceguEjxa", "reviewer": "Reviewer_JkP4", "summary": "The submission proposes an adaptive algorithm for bandit multiple hypothesis testing with e-processes and anytime-valid false discovery rate control. The algorithm is dubbed \"adaptive learn then test\" (aLTT) for its connections to the \"learn then test\" (LTT) framework, which finds a subset of hyper-parameters that provide risk control, but in a non-adaptive fashion. \n\nExperiments on a reinforcement learning and a prompt selection task showcase the behavior of the proposed method in comparison to LTT. aLTT displays larger true positive rates across experiments, while controlling false discovery rate (or family-wise error rate).", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful comments. Below we address your question point by point:\n\n**Theorem and Stopped e-process**: In the revised version, we will formally state the aLTT guarantee in a theorem and provide a proof in the Appendix, which follows from the validity of the e-processes. “*Given a hyperparameter set $\\\\Lambda$, a reliability level $\\\\alpha \\\\in [0,1]$, and an error level $\\\\delta \\\\in [0,1]$, aLTT with an FWER or FDR-controlling selection rule returns a final prediction set $\\\\hat{\\\\Lambda}^{{\\\\rm aLTT},T}$ that satisfies $(\\\\alpha,\\\\delta)$-FWER control or $(\\\\alpha,\\\\delta)$-FDR control, respectively.*” \nRegarding the validity of the e-BH procedure, we will add the following statement clarifying that e-BH maintains statistical validity under the scenario discussed in the paper, while for alternative scenario in which data is re-used at later testing round it is necessary to use the procedure of [1] “*Finally, note that while the eBH procedure directly applies to the sequential testing of Section 2.3, where we allow risk estimates $\\\\mathcal{R}^t=\\\\{R(\\lambda_i, Z_i^t)\\\\}_{\\\\lambda_i\\\\in \\\\mathcal{I}^t}t$ be arbitrarily dependent --- for example, by using the same data to evaluate all models in $\\\\mathcal{I}^t$ --- reusing the same data for evaluation of different hyperparameters at later stages can invalidate the e-process validity unless this is properly corrected [1]*\"\n\n[1] Wang et al. Anytime-valid FDR control with the stopped e-BH procedure. arXiv preprint arXiv:2502.08539. 2025 Feb 12\n\n**Hyperparamater Definition**: We consider a broad definition of hyperparameters, including train-time, architectural, and post-processing parameters. For example, in the RL experiment (Sec. 5.1), “*We produce $N = 20$ control policies by setting the hyperparameter $\\\\lambda$ in the TD3+BC training objective.*” The revised text will clarify that aLTT applies to both train-time and inference-stage hyperparameters.\n\n**Xu et al. :** We agree that aLTT relies on the sequential multiple hypothesis testing framework based on e-processes introduced by Xu et al. (2021). In this sense, aLTT is to Xu et al. (2001) what the original LTT is to multiple hypotheses testing with FWER control. In the original text we wrote “*This work aims at improving the data efficiency of LTT by leveraging recent advances in sequential MH based on e-processes (Vovk & Wang, 2021; Waudby-Smith & Ramdas, 2024; Xu et al., 2021)*”. In the revised text, this relation will be further clarified as “*This work aims to improve the data efficiency of LTT by leveraging recent advances in sequential multiple hypothesis testing (MHT) based on e-processes (Vovk & Wang, 2021; Waudby-Smith & Ramdas, 2024; Xu et al., 2021). Unlike LTT, which employs standard p-value hypothesis testing, we propose an adaptive learn-then-test strategy based on the hypothesis testing framework proposed in Xu et al. 2021 to perform statistically valid hyperparameter selection for machine learning models and generate finite-sample performance certificates.*”  \n\n**Alternative Data Acquisition:** The $\\\\epsilon$-greedy strategy generalizes to a top-$K$ approach, where the $K$ largest e-processes are selected with probability $1-\\\\epsilon$, and $K$ random hyperparameters are chosen otherwise. We will revise the text accordingly and add an experiment ( https://ibb.co/d46Q2yPK ).\n\n**Why p-values from e-processes**: While for FDR control there exists procedures that directly operate with e-values, for FWER e-values are first converted to p-values using e-to-p calibration and then FWER procedure are based on p-values are used. [1] \n\n[1] Vladimir Vovk and Ruodu Wang. E-values: Calibration, combination and applications.\n\n**Highlighted box page 3:** We clarify that while data for each hyperparameter is i.i.d., risk estimates across hyperparameters may be dependent due to shared data reuse (see also first point).\n\n\n**Experiments**:\n- Unless the experiments are conducted using synthetic data, the ground truth is generally unknown. In real-data experiments, the ground truth is estimated from large hold-out samples, which will be clarified in the revision. \n- The average number of tested prompts is 66 across tasks.\n- We consider tasks from the Instruction induction dataset; in Appendix B.3 we report the performance per tasks, specifying which task we consider. In the revised version we will subsample some interesting good/bad prompt example from the exhaustive list ( https://ibb.co/1GV2MHyj ).\n- In the revised text, we will clarify that this statement is made in an empirical and average sense. In fact, due to the randomness of testing, it is possible that the discovered sets are not exactly subsets of one another. However, we find that a stricter requirement leads to a smaller predicted set of reliable prompts (on average) and an increase in the minimal instruction length (on average).\n- Here we consider plot failure probability instead of minimal length (https://ibb.co/QSCN4sy).", "claims_and_evidence": "The connections between this submission and Xu et al. (2021) should be clarified. The proposed aLTT algorithm boils down to Algorithm 1 in the reference. It should be made clear whether the contribution of the submission is to apply the algorithm of Xu et al. to machine learning settings, or to propose a novel algorithm.", "methods_and_evaluation_criteria": "The evaluation criteria are appropriate", "theoretical_claims": "Theoretical claims are mentioned in the main text without formal proofs. Anytime FDR validity does not immediately follow from validity of the e-BH procedure in the general case (see Wang et al., \"Anytime-valid FDR control with the stopped e-BH procedure\"). Within the context of the submission, the arms are independent so FDR validity should be satisfied. It should be discussed when the proposed algorithm fails to avoid misuse.", "experimental_designs_or_analyses": "Could the authors expand on how ground-truth is defined in each experiment to evaluate true positive rate?", "relation_to_broader_scientific_literature": "The submission builds upon bandit multiple hypothesis testing, conformal risk control, and e-processes, which are well-established ideas in their research areas.", "essential_references_not_discussed": "Connections with Xu et al. (2021) should be discussed", "other_strengths_and_weaknesses": "**Strengths**\n* The problem of statistically-valid hyper-parameter selection is compelling\n* The idea of using anytime valid inference for this problem is neat\n\n**Weaknesses**\n* Presentation is hard to follow at times\n* Certain claims are hand-wavy and should be made precise\n\nI will expand on my comments and I am looking forward to discussing with the authors!", "other_comments_or_suggestions": "**Definition of hyper-parameter**\n\nHyper-parameters are introduced in the submission and generally understood by the community as train-time quantities. The LTT framework and the proposed method, however, focus on calibration of fixed, pre-trained models, or am I missing something?\n\nIn the offline reinforcement learning setting, it was somewhat confusing to see the policies themselves being treated as hyper-parameters. I thought the experiment was going to tackle the problem of choosing, for example, the strength of a regularization parameter to guarantee risk control after training.\n\n**Clarification questions on proposed method**\n\nThe considered problem boils down to the bandit multiple hypothesis testing problem described in Xu et al. (2021). That is, at each round of the algorithm, the adaptive mechanism chooses a subset of arms to query, and then updates their respective wealth processes according to the outcomes of the queries. It should be clarified and discussed how aLTT differs from the previous work of Xu et al.\n\nThe aLTT procedure is introduced as querying a subset of the arms at each round. In practice, the $\\epsilon$-greedy procedure only selects one arm at a time. This comes short of showcasing the proposed method in full. Could the authors expand on reasonable choices of adaptive mechanisms different from the greedy one?\n\nIt should be discussed under which assumptions aLTT provides anytime valid FDR control, as this property does not immediately follow from validity of the e-BH procedure in a general setting (see Wang et al., \"Anytime-valid FDR control with the stopped e-BH procedure\").\n\nCould the authors expand on the choice of providing FWER control by converting the e-processes to p-processes?\n\nCould the authors expand on the message they are trying to convey in point 2 of the highlighted box on page 3? I am not sure I understand with respects to what it is claimed that the $Z_i$ \"can be arbitrarily dependent\". $Z_i^t \\sim P_Z$, so the $Z_i^t$'s are i.i.d.\n\n**Experiments**\n\nHow is ground-truth determined to compute TPR in both experiments?\n\nI have a few questions on the prompt engineering experiment:\n* How many prompts are being tested?\n* What kind of tasks are being considered? It would be interesting to include some examples of good and bad prompts to verify whether there are distinctive features in the two groups.\n* I am not sure how to verify the claim that \"stricter accuracy requirements are seen to reduce the number of discovered reliable prompts, leading to an increase in the minimal instruction length\". I could have a set {$p_1, p_2$} with minimal prompt length $\\ell_1$, and a larger set {$p_3, p_4, p_5$} with minimal instruction length $\\ell_2 > \\ell_1$.\n* Instead of minimal instruction length, it might be more informative to report the bottom $\\delta$ quantile in order to account for the failure probability.\n\n---\n\n**Minor comments**\n* Lines 16-17, right column: repetition of \"or\"\n* Lines 99-100, left column: e-values are not a p-values.\n* Definition 2 is not the usually considered notion of FDR: $FDR = \\mathbb{E}[FDP] = \\mathbb{E}[\\frac{\\lvert S \\cap S_0 \\rvert}{\\| S \\| \\vee 1}] = \\mathbb{E}[\\frac{\\lvert S \\cap S_0 \\rvert}{\\| S \\|} \\mid \\|S\\| > 0] \\cdot \\mathbb{P}[\\lvert S \\rvert > 0]$.\n* Algorithm 1: the mechanism $\\mathcal{Q}$ should also be a function of $\\mathcal{D}^{t-1}$ to reflect adaptability?\n* Line 296, left column: $\\mathcal{E}$ is overloaded with the e-processes\n* Figures: it might be more readable to report LTT performance as a horizontal line since it does not depend on $t$", "questions_for_authors": "I have no further questions for the authors", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "P6bCDw49iM", "reviewer": "Reviewer_82X1", "summary": "The authors propose Adaptive Learn-Then-Test (aLTT), a method designed for calibration of AI applications over discrete sets of hyperparameters. The problem is formulated as a multiple hypothesis testing (MHT) task. Hypotheses are tested using e-processes, enabling adaptive and sequential selection of hyperparameters while updating the e-processes based on accumulating evidence over time. The method is applied to two settings: policy selection for offline Reinforcement Learning and prompt engineering, showing improvements in true positive rate compared to non-adaptive baselines.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review our paper! Below is our response to your question and your comment regarding the absence of formal proofs: \n\n- The intuition behind aLTT’s inability to discover all hyperparameters is that, in our experiments, we use finite datasets, which may sometimes be insufficient for aLTT to accumulate enough evidence to identify the entire set of valid hyperparameters.\n\n- In the revised text we have decided to state formally the validity of aLTT with theorem as follows “Given a hyperparameter set $\\\\Lambda$, a reliability level $\\\\alpha \\\\in [0,1]$, and an error level $\\\\delta \\\\in [0,1]$, aLTT with an FWER or FDR-controlling selection rule returns a final prediction set $\\\\hat{\\Lambda}^{{\\\\rm aLTT},T}$ that satisfies $(\\\\alpha,\\\\delta)$-FWER control or $(\\\\alpha,\\\\delta)$-FDR control, respectively.”", "claims_and_evidence": "The claims made in the submission are supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "The proposed method and evaluation criteria are appropriate and well-justified for the calibration of AI applications.", "theoretical_claims": "There are no formal theoretical proofs presented in the paper. However, I reviewed the formulas provided and verified that they are consistent with their descriptions in the text. The claims regarding the algorithm’s behavior, as derived from these formulas, appear sound and align with the expected outcomes.", "experimental_designs_or_analyses": "The experimental design and analyses are sound, clearly described, and well-aligned with the goals of the paper.", "relation_to_broader_scientific_literature": "Key contributions:\n- Introduction of e-processes for calibration in AI applications.\n- Adaptive, sequential selection of hyperparameters in a statistically principled manner.", "essential_references_not_discussed": "There are no essential references missing from the current discussion.", "other_strengths_and_weaknesses": "Strengths:\n- Clear and accessible writing.\n- Well-explained and principled algorithm design.\n- Strong experimental evidence.\n\nWeaknesses:\n- No noticeable weaknesses.", "other_comments_or_suggestions": "No additional comments or suggestions.", "questions_for_authors": "1. What is the intuition behind the inability of aLTT to uncover the entire set of valid hyperparameters in the experiments?\nUnderstanding this limitation more deeply could help assess the practical utility and potential extensions of the method.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YcXcl8h2dl", "reviewer": "Reviewer_G4NT", "summary": "This paper essentially proposes the following method, as summarized in Section 2.3: Sequential and Adaptive Hyperparameter Selection. Given candidate hyperparameters $\\Lambda = \\lambda_1, \\dots, \\lambda_N$, for rounds $t = 1, \\dots, T$, where $T$ is some stopping time:\n\n- Choose $\\mathcal{I}^t \\subset \\Lambda$ based on an adaptive acquisition policy (that is, based only on information seen so far.) For each $\\lambda_i^t \\in \\mathcal{I}^t$, sample new data $Z_i^t$ independent of the past. \n- Return a set $\\hat \\Lambda^{\\textup{aLTT}, t}$ according to some decision rule on the data, which is an estimate of $\\{\\lambda \\in \\Lambda : E_{P_Z}[R(\\lambda_i^t, Z_i^t)] \\leq \\alpha\\}$. The rule is in the form of a multiple testing algorithm for the $N$ null hypotheses that $E_{P_Z}[R(\\lambda_i^t, Z_i^t)] > \\alpha$, which corrects for multiple testing over all of $\\Lambda$. \n\nIn this work, the authors are free to use any adaptive acquisition policy and stopping time $T$, due to the special form of the decision rule, which is based on multiple testing algorithms applied to e-processes. The theoretical observation is that the final set $\\hat \\Lambda^{\\textup{aLTT}, T}$ has the risk controlling properties of the original LTT proposal, and the main, empirical observation is that it has much better sample efficiency, which is useful for scenarios where testing is costly.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful comments. Below we address your question:\n\n**Questions For Authors:** \nYes, adaptive LTT (aLTT) substantially outperforms LTT, even when LTT uses fixed-time valid p-values. The table below reports final TPR for aLTT and LTT using p-values from Waudby-Smith & Ramdas (2024) and Hoeffding-Bentkus p-values (Angelopoulos et al., 2021). The latter is tight for 0-1 losses. In all cases, aLTT with adaptive acquisition ($\\epsilon<1$) achieves significantly higher true positive rates (TPR), demonstrating its advantage stems from data-adaptive acquisition rather than LTT’s use of weaker p-values. We will include these results in the revised paper.\n\n|           | LTT WSR | LLT HB p-value | aLTT $\\\\epsilon=1$ (non adaptive) | aLTT $ \\\\epsilon=0.25$ (adaptive) |\n|-----------|---------|----------------|--------------------|-------------------------|\n| Online RL | 0.14$\\pm0.07$    | 0    $\\pm0.0$          | 0.15 $\\pm0.06$             | **0.86** $\\pm0.05$                   |\n| Automated Prompt Engineering      | 0.27$\\pm0.06$     | 0.27$\\pm0.06$            |      0.27$\\pm0.06$               | **0.65**$\\pm0.05$                    |\n\nP-values based on the central limit theorem are asymptotically valid but unsuitable for rigorous finite-sample guarantees. Such p-values can inflate FDR and lack statistical soundness—for instance, prediction sets from these p-values yield an empirical FWER of 0.154, exceeding the intended $\\\\delta=0.10$.\n\n\n**Claims and Evidence**\n- We acknowledge the ambiguity in the term “testing” and will replace it with “evaluation” when referring to the data-adaptive acquisition policy. Specifically, at each time step $t$, a subset of hyperparameters $\\\\mathcal{I}^t \\\\subseteq \\\\Lambda$ is selected for evaluation, updating only the e-processes in $\\\\mathcal{I}^t$ (see Eq. 14). This clarifies that all hypotheses in $\\\\Lambda$ are tested, but only a subset is evaluated at each step.\n\n- We believe Figure 3 aligns with theory, as both empirical FWER and FDR stay below $\\delta$. It shows FWER (blue) and FDR (red) for an FWER-controlling procedure (dashed triangle) and an FDR-controlling procedure (solid square). Since FWER is stricter than FDR, for either scheme (triangle or square), the FWER curve (blue) is always higher than the FDR curve (red). Likewise, as the FWER-controlling procedure (dashed triangle) is stricter, its FDR and FWER curves are lower than those of the FDR-controlling procedure (solid square).\n\n**Methods and Eval**\n- RL and LLMs are widely used in high-stakes and high-risk applications.  For example, in the field of robotics, RL algorithms are used for robotic surgery, autonomous navigation, robotic arm control, and more. In these cases, policies are often trained offline using simulators to mitigate the risks associated with training policies in the real world. However, due to the mismatch between simulators and the real world, the actual performance of these policies is often unknown. In Section 5.1 we exemplify how aLTT can be used to identify policies that satisfy real-world performance guarantees, while limiting the number of real-world interactions.\nSimilarly, LLMs are used in critical fields such as medical diagnosis and legal applications. For example, Med-PaLM is a large language model (LLM) designed to provide high-quality answers to medical questions. The performance of these model can vary depending on how are prompted, rendering their use potentially harmful. In the automated prompt engineering example proposed in Section 5.2 we showcase how aLTT can be used to identify prompt templates whose accuracy falls above the minimum reliability level. \n\n- The FWER criterion ensures high-confidence reliability, whereas FDR prioritizes statistical power in large-scale testing. FDR control is common in exploratory research, including genomics, neuroimaging, and social sciences, enabling broader discovery while accepting a higher false positive rate.\n\n- We assume the standard episodic RL setting [2], where agent interactions occur in episodes with resets to the initial state. Risk (15) measures average episodic performance, and reliability requirements apply over the initial state distribution $P_{s^1}$ and policy roll-out.\n\n[2] Sutton RS, Barto AG. Reinforcement learning: An introduction. \n\n**Related Work**\n- In the revised text, this relation will be further clarified as “This work aims to improve the data efficiency of LTT by leveraging recent advances in sequential multiple hypothesis testing (MHT) based on e-processes (Vovk & Wang, 2021; Waudby-Smith & Ramdas, 2024; Xu et al., 2021). Unlike LTT, which employs standard p-value hypothesis testing, we propose an adaptive learn-then-test strategy based on the hypothesis testing framework proposed in Xu et al. (2021) to perform statistically valid hyperparameter selection for machine learning models and generate finite-sample performance certificates.”\n- Thanks for bringing to our attention Jamieson and Jain!", "claims_and_evidence": "In Section 4.3 it is claimed that $\\hat \\Lambda^{aLTT, T}$ is an $(\\alpha, \\delta)$-FWER or FDR controlling as defined in Definition 2.1 and 2.2. No proof is provided, as it follows immediately from the anytime-valid properties of e-processes and their resulting p-values. \n\nRegarding this claim, however, it is important that the multiple testing procedure involves a correction over *all* of $\\Lambda$ at each step, rather than just $\\mathcal{I}^t$. But Section 2.3 states, as item 1, that $\\mathcal{I}^t \\subset \\Lambda$ is selected for \"testing\". This is misleading; they are for sampling the $Z_i$'s, not for testing, as evidenced by the notation in Section 5, which refers to the p-values $1,\\dots, N$. On the contrary, every parameter in $\\Lambda$ is tested at every time step. \n\nOn the empirical side, in Section 5 they show that their method indeed performs as expected w.r.t FWER and FDR, and has good performance in terms of some metric (true positive rate, length of the prompt) with few rounds of sampling required. These claims are well supported, though it should be mentioned in Figure 3 that the FDR of the FDR controlling method is nearly zero, while its FWER is less than $\\delta$---a curious behavior which is *not* implied by the theory. \n\nHowever, the claim that aLTT is much better than LTT is at least partly unfair to LTT. In the experiment of Section 5.1, LTT is the same algorithm as aLTT at the final time step, where the stopping time $T = 5000$ is a fixed constant. However, even though LTT cannot use adaptive acquisition policies and stopping times (decreasing its power), it can take p-values as input that were not derived from e-processes (increasing its power). I suggest specific comparisons to LTT later.", "methods_and_evaluation_criteria": "The original LTT motivation was to give a guarantee of absolute safety for high risk applications like medical imaging, which was why the FWER guarantee which appears in that paper has to hold in finite samples. In the RL and LLM examples, it's unclear why a finite-sample guarantee would be absolutely required. \n\nThat caveat aside, the method itself and the FWER criteria makes sense for these problems, and the TPR and prompt length criteria of Section 5 seem like good measurements (though I would defer to the opinion of an applied MLer here.) Less clear to me is why the FDR criterion would make sense here---if $\\alpha = 0.2$, then on average, 20% of the $\\lambda$'s might be unreliable, which doesn't seem attractive if the goal is to pick a final reliable one. \n\nAs for the evaluation criteria, perhaps this is my unfamiliarity with RL, but it would seem that Section 5.1 requires the MDP to be stationary every time it is queried, since aLTT needs the transcripts $Z$ of state/action/rewards to be iid. I would think that in the real world, interacting with the environment repeatedly changes the underlying MDP. Also, perhaps the loss function in Secton 5.2.1 should be stated explicitly?", "theoretical_claims": "No proofs to check, though as previously mentioned, ideally it's stated clearly that $\\mathcal{I}^t$ is only for sampling, not for testing.", "experimental_designs_or_analyses": "The experiments showcase well the utility of aLTT on the example problems. Though, as previously mentioned, the RL problem may be too stylized.", "relation_to_broader_scientific_literature": "Xu et al (2021) stated the e-process bandit framework from which this article inherits. Waudby-Smith and Ramdas (2024) proposed the particular e-process that is used here. And Angelopoulos et al (2021) cast the hyperparameter optimization problem as a multiple testing problem in the form of LTT. \n\nThis paper might be distinguished by the recognition that these tools can be fruitfully applied to settings that require expensive queries from an environment, such as policy selection in offline RL and prompt engineering for LLMs. Even if these methods are not directly applied (due to the conservatism displayed in Figure 3), perhaps they could be adjusted in practice and used as a heuristic when the finite sample guarantee is not precisely required.", "essential_references_not_discussed": "The Xu et al (2021) is cited, but the bandit multiple testing setup is precisely the one for which the current one is a special case, and this framing is not discussed. An earlier paper using the same framework, but without e-processes, is Jamieson and Jain (2018), and is not cited.", "other_strengths_and_weaknesses": "The paper is relatively clear, but there was a point of frustration regarding whether $\\mathcal{I}^t$ is tested over or simply used for sampling, as previously mentioned. \n\nI think this is an interesting paper with an application to a domain not previously considered by Xu et al (2021) and Angelopoulos et al (2021), but methodologically it is not new. Its strength would then lie in its impact to, e.g. prompt engineering or RL policy selection, though I don't think I can evaluate this area.", "other_comments_or_suggestions": "- To remind the reader, you could refer to $N$ as the cardinality of $\\Lambda$ wherever it appears. \n- Equations (4) and (5) should be $\\alpha$, not 1.\n- Box in Section 2.3: change \"selected for testing\" to something like \"selected for estimating the risk more carefully\"?\n- Figure 3: Dashed lines are hard to see, why not say \"triangles\"? Also, it might be nice to walk the reader through this figure more.", "questions_for_authors": "To more convincingly show this method's utility compared to LTT, we should equip LTT with fixed-time valid $p$-values, which are more powerful than anytime-valid. So: \n(1) Does your method still dominate LTT with non-adaptive acquisition, when LTT uses $p$-values based on the hedged capital process of Waudby-Smith and Ramdas (2024), which Angelopoulos et al (2021) call the WSR bound?\n(2) What about $p$-values based on the central limit theorem, since $\\frac{1}{\\sqrt T} \\sum_{t = 1}^T R(\\lambda_i^t, Z_i^t)$ is asymptotically normal?\n\nThis would serve as a more fully convincing demonstration that aLTT can outdo LTT. When the stopping time $T$ is fixed, my hunch is that it cannot always be the case, but may be the case in your applications of interest. I would change my score from Reject to Weak Reject if this is confirmed to be indeed the case (and possibly more if other reviewers concur that the application examples are compelling).", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ThCpBVzaQ9", "reviewer": "Reviewer_wnwq", "summary": "This paper extends the LTT strategy to accommodate adaptive MHT procedures. The resulting procedure is more data-efficient and can be used in situations where the loss function as a function of lambda is not a-priori known, but must be evaluated (with some non-negligible associated cost) on a per-lambda basis. Experiments are performed on offline RL via MuJoCo, prompt engineering, and wireless resource allocation (Appendix).", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review our paper and for your positive comments! Below, we address each comment point by point:\n\n- *\"Why no main theorem/proof?...\"*. -  Thank you for the suggestion! In the revised version, we will add the following theorem on the statistical validity of aLTT  “Given a hyperparameter set $\\\\Lambda$, a reliability level $\\\\alpha \\\\in [0,1]$, and an error level $\\\\delta \\\\in [0,1]$, aLTT with an FWER or FDR-controlling selection rule returns a final prediction set $\\\\hat{\\\\Lambda}^{{\\\\rm aLTT},T}$ that satisfies $(\\\\alpha,\\\\delta)$-FWER control or $(\\\\alpha,\\\\delta)$-FDR control, respectively.” and provide a proof in the Appendix.\n\n- *\"It may be a good idea to reference selectivenet...\"*. - Thanks for bringing this related work to our attention, we will include it in the related work.\n\n**Other Comments Or Suggestions**\n- We consider the stopping criterion $ |\\\\hat\\\\Lambda^{\\\\rm aLLT,t}|>d$ or $t=t_{\\\\rm max}$ because we don’t make any assumption on the relation between the value of $\\\\lambda$ and its risk (e.g. monotonicity as in RCPS). For example, in case of the prompt selection, there is no clear notion of larger or smaller prompt. However, if some preference or information about the parameters is available, it is possible to include this information in the design of the stopping criterion. We will clarify this in the revised text.\n- Yes, assuming that the loss function is monotone, one can use aLTT in a similar manner to RCPS. The e-processes in aLTT can be used to obtain upper confidence bounds on the risk, which can then be used to implement the selection procedure as in RCPS. The validity would follow as in the RCPS paper. However, note that our paper operates in the most general setting, where we remain agnostic to any relationship between risk and hyperparameters.\n- Thanks for spotting the typo!", "claims_and_evidence": "For the most part, yes. It is clear to me why this procedure works. \n\nHowever, why no main theorem/proof? To a reader who is an expert in both LTT and e-processes, it is clear that combining these tools leads to a stronger result than in the original LTT paper (accounting for temporal dependencies, allowing early stopping, etc.) But a theorem would substantially help to clarify this point and speak to the power of aLTT in comparison. And a proof, as well. Right now it feels like the paper is incomplete if there’s no theorem or proof of the main theoretical claim (that this is a valid risk-controlling procedure) even though I believe this to be true based on the arguments made in the text of the paper.", "methods_and_evaluation_criteria": "I believe the method is very well evaluated.", "theoretical_claims": "Copy-pasting above: \n\nWhy no main theorem/proof? To a reader who is an expert in both LTT and e-processes, it is clear that combining these tools leads to a stronger result than in the original LTT paper (accounting for temporal dependencies, allowing early stopping, etc.) But a theorem would substantially help to clarify this point and speak to the power of aLTT in comparison. And a proof, as well. Right now it feels like the paper is incomplete if there’s no theorem or proof of the main theoretical claim (that this is a valid risk-controlling procedure) even though I believe this to be true based on the arguments made in the text of the paper.", "experimental_designs_or_analyses": "Yes. Seems sound.", "relation_to_broader_scientific_literature": "It may be a good idea to reference selectivenet: https://proceedings.mlr.press/v97/geifman19a\n\nAt this point, it is known that the selectivenet procedure is not statistically correct. In other words, the guarantee does not hold.\n\nHowever, it does have a similar adaptive search through $\\Lambda$-space idea presented here, so I believe it is worth a cite.", "essential_references_not_discussed": "See above: https://proceedings.mlr.press/v97/geifman19a", "other_strengths_and_weaknesses": "I think the paper is very strong. The idea is truly great, and can really improve the statistical and computational efficiency of LTT. The experiments are fantastic, very strong, relevant, and modern. \n\nMy main gripe is the lack of formal theoretical statement being made. It makes it unclear what new algorithm with accompanying theory the paper is proposing.", "other_comments_or_suggestions": "As stopping criteria, the authors suggest (1) stopping once $|\\hat{\\Lambda}^{\\rm aLTT, t}| \\geq d$ or (2) stopping once $t = t_{\\rm max}$. Are these the most natural stopping criteria?I would have thought there would be a different criterion. For example, in some cases you may want the smallest or largest $\\lambda$ satisfying the constraint of having a population risk less than alpha.\n\nIs there any way to show/argue that this strategy is tight? For example, what does it do in the monotone case? (Can it somehow reduce to RCPS, or something close to RCPS?)\n\n“potentially resulting in empty calibration sets” —> “potentially resulting in empty set of selected prompts $\\hat\\Lambda^{\\rm rel}$.”", "questions_for_authors": "No further questions. Great job, this is a wonderful paper!", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HNRH31yjEm", "reviewer": "Reviewer_uVNw", "summary": "This paper makes progress on the problem of hyperparameter selection with statistical guarantees. The work provides a new, adaptive algorithm for hyperparameter selection that presents an extension of the Learn-Then-Test (LTT) framework that was previously introduced. The LTT framework uses p-values of hypotheses corresponding to hyperparameter selections to obtain high probability bounds on the selection of reliable hyperparamters. The present manuscript extends this framework by utilizing e-value rather than p-values. E-values are using to accumulate evidence measures multiplicatively which allows for combining evidence across studies, or in this case, across sets of collected data. The paper provides some theoretical intuition first and demonstrates the utility of the proposed approach in offline-to-online RL and prompt engineering.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful comments. Below we address your question point by point:\n\n**Claims And Evidence**\n- **Missing Theorem**: We will add the following theorem on the statistical validity of aLTT  “*Given a hyperparameter set $\\\\Lambda$, a reliability level $\\\\alpha \\\\in [0,1]$, and an error level $\\\\delta \\\\in [0,1]$, aLTT with an FWER or FDR-controlling selection rule returns a final prediction set $\\\\hat{\\\\Lambda}^{{\\\\rm aLTT},T}$ that satisfies $(\\\\alpha,\\\\delta)$-FWER control or $(\\\\alpha,\\\\delta)$-FDR control, respectively.*” and provide a proof in the Appendix.\n\n- **Reliability of selected parameters** Models in the predicted set have reliability above $\\\\alpha$ with a probability at least $1-\\\\delta$, ensuring valid hyperparameter selection. However, in our experiments, realized performance often exceeds the target reliability ( https://ibb.co/CpGgwVtP ). We agree that reporting the best model’s reliability is insightful and will include it in the revised version.\n\n- **Methods And Evaluation Criteria** Training an agent online is not always feasible due to safety concerns (e.g., robotic surgery). Alternatively, training with off-policy data requires accounting for covariate shift using propensity scores [1]. Even with known propensity scores, valid performance guarantees must be established via policy rollouts, as done in aLTT. \n\n[1] Uehara M, Shi C, Kallus N. A review of off-policy evaluation in reinforcement learning. \n\n**Experimental Designs Or Analyses**: \n\n- A single grid search has poor TPR and lacks FWER/FDR control, making it unsuitable for statistically valid hyperparameter selection. As an alternative, we benchmark validation-based selection, which tests hyperparameters on a grid and selects those with empirical risk below $\\alpha$. Our experiments show that this approach results in high empirical FWER/FDR, making it unreliable.\n\n- The reliability $\\\\alpha$ parameter must be chosen by the user based on the desired level of reliability for the application. Accordingly, the choice of this parameter must be informed by some knowledge about realistic values for the average risk. In the RL example, one could use off-policy data log to estimate a reasonable target reliability level $\\\\alpha$. To evaluate the impact of $\\\\alpha$ here we perform an ablation study by varying its value ( https://ibb.co/xq5fxXTn ).\n\n**Relation To Broader Scientific Literature:** In the paper we mention various approaches based on Bayesian optimization [1-4]; however, we do not mention evolutionary strategies or PAC-Bayes methods. Thanks for pointing out this, in the revised paper we will add references to [5-8]. ( References https://ibb.co/twNJDDCd ) \n\n**Weaknesses**: Our approach extends to multiple correlated hyperparameters by associating a multivariate parameter with a single hypothesis. Appendix B.2 presents a wireless resource allocation experiment where hyperparameters include transmit power level and scheduling policy, which are inherently correlated. Even in such cases, aLTT significantly improves testing efficiency and system performance. If space allows, we will move this discussion to the main text.\n\n**Clarifying e-values**: We acknowledge that readers unfamiliar with these concepts may need more guidance. In the revised version, we will explicitly direct them—already in the introduction—to Section 4.1, which explains e-values, e-processes, and their role in testing.\n\n**Questions For Authors:** \n\n- A simple grid search selecting the hyperparameter with the lowest empirical risk yields a maximum TPR of 1/|reliable hyperparameters|, which can be much lower than 1 when reliable hyperparameters are numerous. For instance, in our RL experiment, with nine reliable hyperparameters, grid search would achieve at most 0.11 TPR, far below aLTT’s 0.85. Moreover, this approach lacks FWER/FDR guarantees, as shown in our experiment.\n\n- The reliability requirement is an expectation over multiple roll-out. The definition in (15) is the reward for a single roll-out, while the expectation of (15) over the roll-out distribution defines the reliable hyperparameters, i.e. $\\\\Lambda^{\\\\rm rel}=\\\\{\\\\lambda\\\\in\\\\Lambda: R(\\\\lambda)=\\\\mathbb{E}_{P_Z}[R(\\\\lambda,Z)]> \\\\alpha\\\\}$. We will clarify this in the main text as “*The reliability for a single roll-out is measured via the cumulative reward obtained by a policy $\\\\lambda$ on the MDP $\\\\mathcal{E}$, which is defined as…*”\n\n-\tWe choose to compare aLTT only against methods that are statistically valid, as the goal of this paper is to perform ‘statistically valid hyperparameter selection.’ To illustrate that methods without formal guarantees can violate the target FWER and FDR levels we consider a validation based baseline that picks an hyperparameter if its empirical risk is below $\\alpha$. Figures: RL experiment ( https://ibb.co/xq2KnrGH ) and the APE experiment (https://ibb.co/QFxgkc0G ) .", "claims_and_evidence": "The main contribution of this paper is a novel algorithm for hyperparameter testing called adaptive LTT (aLTT). The main claim about this algorithm is that it guarantees rigorous control over the FWER and FDR metrics while reducing the number of testing rounds compared to the non-adaptive version of the algorithm.\n\nTo provide evidence for this claim, the work outlines the mathematical properties of e-values and then provides an application study which includes experiments in offline RL as well as prompt engineering.\n\nEven though the paper claims that the algorithm provides guarantees, there is no formal proposition statement that would quantify that. In line 246R, the text states that “if an FWER-controlling method [...] is used, the resulting set [hyperparameter set] is (α, δ)-FWER-controlling”. I believe it would be beneficial to state this as an explicit theorem of the algorithm and provide a proof, even if this proof simply refers to properties or propositions analyzed by other works.\n\nFor the experimental section, the claim that aLTT requires fewer rounds and is supported in both applications in Figures 1-4. \nIn both cases, the algorithm also achieves a higher true positive rate indicating it selected fewer hyperparameters that have poor performance. What would be nice to see and is missing is a measure of the achieved reliability.", "methods_and_evaluation_criteria": "The proposed benchmarks are both problems where hyperparameter tuning is relevant. However, in both cases, the hyperparameter problem becomes significantly more difficult when one does not have access to an online version of the problem. This is not being addressed and it seems that unlimited access is assumed. As a result, e.g. in Figure 2, there are 5000 rounds. If each round collects only a single trajectory of say 500 steps in Half-Cheetah, one gets access to 2.5 million environment steps. This is sufficient to train online agents rather than hyperparameter tune on offline data.", "theoretical_claims": "As mentioned above, the theoretical claims are solely in prose and I think a formal proposition would round out section 4 nicely. Since things are only in prose, there are no proofs to evaluate. I am not very familiar with this type of work but I am willing to believe that the algorithm is correct to the extent that it follows from direct application of e-value properties and corresponding optimization algorithms.", "experimental_designs_or_analyses": "The experimental design is mostly reasonable I believe. The paper evaluates the TPR which seems to be the key indicator relevant to the approach. However, I think there is a lack of simple baselines to put things into perspective. For instance, how well could I do with simple gridsearch? What happens if I use a very small $\\varepsilon$ such as 0.01. While the main claims are supported, I think more ablations to understand the relative behavior of the approach and to put it into perspective with other approaches in the domain would be quite useful.\n\nIt is also not clear to me how the choices for $\\alpha$ were made. $0.57$ in the RL settings seems quite arbitrary. An ablation of varying alpha might be useful to remove any concerns about the choice of this value.", "relation_to_broader_scientific_literature": "It seems to me that the closer scientific literature has been discussed. However, the introduction in line 10R states \"hyperparameter optimization is typically modeled as a bandit problem\". This is a very broad statement. There are many other approaches for hyperparameter optimization beside bandits. There is lots of work on Bayesian optimization for adaptive hyperparameter selection and evolutionary strategies. There is also work on PAC-Bayes hyperparamter selection. All these are not mentioned.", "essential_references_not_discussed": "I’m not sufficiently familiar with the literature to make a statement here.", "other_strengths_and_weaknesses": "Weaknesses: \n* It is unclear to me how this approach could handle multiple correlated hyperparameters. The experimental setting only considers one fixed hyperparameter over which needs to be searched. In practice, the problem is much more high-dimensional.", "other_comments_or_suggestions": "I believe the text would benefits from a better description of e-values and e-processes. I had to conduct my own search to gain clarity on what these are. Specifically, in line 100L, the text states the key property of e-values is that they can be combined to obtain e-processes which are then not explained. This section would benefit from conciseness about what the property is that makes this possible and an explanation of e-processes.\n\nI think line 162L is supposed to be \"with the expectation\".\n\nOverall, I think this is a decently written paper that requires some fine-tuning here and there thus I am recommending weak accept.", "questions_for_authors": "Q1. What would happen if I, for instance, applied a simple gridsearch and took an argmax? I might be able to achieve TPR=1, correct?\n\nQ2: Why is reliability in line 313L approximated over a single rollout and not an expectation over multiple? That seems like an odd choice especially for highly stochastic domains such as those considered in the paper but maybe I am missing something.\n\nQ3. The work states that there are other methods without guarantees (line 010). Could you elaborate why there are no comparisons against such methods?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper extends the Learn Then Test (LTT) paradigm of hyperparameter selection towards an adaptive and on-the-fly approach called adaptive LTT (aLTT). The main benefits of moving away from the static p-value approach and towards a dynamic e-values/processes approach is improving the sample efficiency of such algorithms, making them more amenable to practical applications. To this end, the authors illustrate the improvements of aLTT in a number of diverse tasks, from prompt engineering to offline RL.

Reviewers were more or less unanimous in their positive feedback of this work. The core idea is simple, elegant and well motivated, and  the paper is well-written such that all the reviewers were able to achieve a strong understanding. There were no real major concerns, and the authors seemed to address the minor concerns in the rebuttal phase.

Therefore taking all these points together, the paper merits inclusion into the program.
Decision: Accept (spotlight poster)