Multi-Objective Hyperparameter Selection via Hypothesis Testing on Reliability Graphs
Authors: Amirmohammad Farzaneh
Osvaldo Simeone
Venue: NeurIPS 2025 poster ()
Keywords: Hyperparameter Selection
Conformal Prediction
Hypothesis Testing
Primary area: probabilistic_methods

Abstract: The selection of hyperparameters, such as prompt templates in large language models (LLMs), must often strike a balance between reliability and cost. In many cases, structural relationships between the expected reliability levels of the hyperparameters can be inferred from prior information and held-out data -- e.g., longer prompt templates may be more detailed and thus more reliable. However, existing hyperparameter selection methods either do not provide formal reliability guarantees or are unable to incorporate structured knowledge in the hyperparameter space. This paper introduces reliability graph-based Pareto testing (RG-PT), a novel multi-objective hyperparameter selection framework that maintains formal reliability guarantees in terms of false discovery rate (FDR), while accounting for known relationships among hyperparameters via a directed acyclic graph. Edges in the graph reflect expected reliability and cost trade-offs among hyperparameters, which are inferred via the Bradley-Terry (BT) ranking model from prior information and held-out data. Experimental evaluations demonstrate that RG-PT significantly outperforms existing methods such as learn-then-test (LTT) and Pareto testing (PT) through a more efficient exploration of the hyperparameter space.

Multi-Objective Hyperparameter Selection via Hypothesis Testing on Reliability Graphs
Amirmohammad Farzaneh Osvaldo Simeone
Centre for Intelligent Information Processing Systems
Department of Engineering
King's ' College London
London, United Kingdom
{ami fammrrmaamma..faa farzaneh, osvaldo (0l.meeeee@kkkll.. ac ac.uk
Abstract
The selection of hyperparameters, such as prompt templates in large language models (LLMss), must often strike a balance between reliability and cost. In many cases, structural relationships between the expected reliability levels of the hyperparameters can inferred from prior information and held-out data - e.g., longer prompt templates may be more detailed and thus more reliable. However, existing hyperparameter selection methods either do not provide formal reliability guarantees unable incorporate structured knowledge in the hyperparameter space. This paper introduces reliability graph-based Pareto testing (RG-PT), a novel multi-objeccivv hyperparameter selection framework that maintains formal reliability guarantees in terms false discovery rate (FDR), while accounting for known relationships among hyperparameters via directed acyclic graph. Edges in the graph reflect expected reliability and cost trade-offs among hyperparameters, which are inferred via the Bradley-Terry (BT) ranking model from prior information and held-out data. Experimental evaluations demonstrate that RG-PT significantly outperforms existing methods such learn-then-test (LTTT) and Pareto testing (PT) through more efficient exploration of the hyperparameter space.
Introduction
1.1 Context and Motivation
Consider the problem of selecting prompt templates sentiment analysis task []] In this setting, the L.M receives natural language prompt template along with a movie review as input, and the goal is to determine whether the sentiment expressed in the review is positive negative. As illustrated in Fig. [ the prompt templates À are chosen from a set of pre-determined choices A, and the objective is to identify prompt templates that elicit consistently accurate responses across inputs [
large language model (LL.M)-based
Longer prompts often yield more reliable outputs (2). However, they are also more costly to the end user when pay- per-ttenn billing schemes are applied. This is commonly the case for enterprise software incorporating Al-driven analytics or hosted L.MM endpoints via the LLM application programming interface (API) 3 As exemplified in Fig. Ta this suggests that the reliability of different prompt templates follows a directed acyelic graph (DAG) structure with nodes closer the roots corresponding to more costly prompt templates with higher expected reliability.
An ideal prompt engincering scheme would apply hyperparameter selection methods capable of selecting prompt templates that are as short as possible while enduring formal reliability guarantees. Existing hyperparameter selection methods, however, either do meet formal reliability require-
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

on poilie appitel (evien,
Expsctel relubiilty
unee pos ce regane jerieg yes
A
le postie onglte jerin, ye!
PT
undieen undie pir bonnee oeann jeriey yei)
Aç
porkee arg ongane" jering
RG-PT
yoatre segettce (ovee
10
Length of shartest prompt
50
60
(a)
Figure 1: Illustrative example for prompt engineering in LLM-based sentiment analysis: (a) Prompt template candidates in set have expected reliability levels that can be arranged on a reliability graph (RG), so that each parent prompt template is expected to be more reliable than its child prompts: (b) Distribution of the length of the shortest prompt templates identified by LTT [4, PT (5]. and the proposed RG-PT for the Stanford Sentiment Treebank dataset [6] (see Sec. 4.1 for details).
ments
8 or cannot incorporate the structured knowledge encoded graph like the DAG in Fig.
In particular, Learn-Then- Test (LTT) 4 pioneered the use of multiple hypothesis resting (MHT) for hyperparameter selection, providing formal guarantees on the reliability the returned subset of hyperparameters. However, LTT cannot incorporate structured information about relative pected reliability levels of the hyperparameters. Pareto Testing (PT) [S] builds on LTT to address multi-objective optimization problems. Specifically, PT infers global, linear ordering over hyperpa- rameters from held-out data based on their expected relative reliability. Thus, PT cannot account for more complex structured relationships between candidate hyperparameters in the example of Fig.
Ta
This paper proposes novel framework, reliability graph-based PT (RG-PT), that systematically captures and exploits interdependencies between hyperparameter configurations for hyperparameter selection. RG-PT models the hyperparameter space DAG, termed the reliability graph (RG). In an RG, nodes correspond to candidate hyperparameter configurations, such as prompt templates, and edges encode reliability relationships. If there is edge from hyperparameter A; another Xy in the RG, then A; is expected to be more reliable than Xj-
For the running example of prompt design, as shown in Fig. 1b (detailed in Sec. 4. RG-PT is seen experimentally to select shorter prompt templates than LTT and PT, while still satisfying formal guarantees in terms of false discovery rate (FDR). The FDR measures the fraction of unreliable prompt templates returned by the hyperparameter selection scheme. This advantage of RG-PT stems from its ability to encode rich structural relationships among prompt templates, to explore the hyperparameter space more efficiently during the MHT procedure.
1.2 Further Related Work
Hyperparameter Selection: State-of-the- techniques for hyperparameter optimization, such as Bayesian optimization (BO) (9. multi-armed bandits (MAB) [ and gradient- based optimization
10,, provide satisfactory empirical performance, but they lack post-selection error control such as FDR guarantees. Under idealistic assumptions such as the correct specification kernel models, BO and MAB can provide bounds on cumulative or simple average regret [1] 12] [3]. However, these bounds do not offer the type of assumption-fire free guarantees on the risk the selected parameters that can be instead provided by FDR-controlling methods. LTT addresses this gap by incorporating MHT the hyperparameter selection process [4. Extensions of LTT are surveyed in [4]. Note that, while reference 4 mentions the possible use of graph- based approaches, these are limited to fixed dser-eefined graphs or linear directed graphs (chains).
Multi- Oojjii-Obeeetive Optimization: Modern AI applications often require optimizing multiple objectives such as accuracy, efficiency, and cost. This can be formally done through Pareto optimization to identify all the feasible trade-off points among different objectives [15. PT [5] extends LTT to settings with multiple objectives, inferring a linear testing order in the hyperparameter space based on held-out data.

---

1.3 Main Contributions
The main contributions of this paper are as follows.
Methodology: We propose RG-PT, novel multi-objective hyperparameter selection framework that systematically infers and utilizes interdependencies among the expected reliability levels of candidate hyperparameter configurations. RG-PT first constructs a DAG, known as RG, based on prior information and held-out data via the Bradley-Terry (BT) ranking model [16] and the non-negative Lasso 17 Then, it applies MHTT-based hyperparameter selection on the RG by following DAGGER, graphical testing method introduced in [18]
Applications: We demonstrate the effectiveness of RG-PT through experiments in LLM prompt engineering, sequence toaeeettt-soooee to-sequence translation, object detection, image classification, and telecommu- nications, highlighting its advantages over existing methods.
The rest of this paper is organized as follows. In Sec. 2 we define the multi objetivo hyperparameter selection problem. Sec. details the proposed RG-PT framework. Experimental results are presented
in Sec. 4 We conclude the paper in Sec. 5)
2 Multi-Objective Hyperparameter Selection
In this section, we define the problem of multi-ohjective hyperparameter selection, and we show how this problem can be formulated via MHT by following references 51
2.1 Problem Definition
Consider predefined discrete and finite set A of hyperparameters X, which govern the performance
of machine learning model such as an LLM (see, e.g., (I9]). The discrete set A is populated in
preliminary pre-selection step [20 2T 22 using methods including LLM judges (33 and continuous
optimizers like Bayesian optimization
7 and Hyperband
In multi-objective setting with I. risk functions, when tested data point Z, hyperparameter attains risk values ri (2, (2,A) X) for ( = 1...., L. The risk functions r1(2, A) are negatively oriented, meaning that lower risk values correspond better-performing hyperparameters. The risks normalized within the range 0 < rr(2, < 1. Furthermore, for each performance criterion
1= 1......, L, the average risk function is defined
Ra(A) = Ez [Z)
where the expectation taken over the distribution Pz of the data Z.
We partition the set of L risk functions into the following two groups:
Reliability risk functions: The first set of risk functions {A) must be controlled via the choice of the hyperparameter A. In particular, a hyperparameter configuration À is said to be reliable it guarantees the constraints
R((A) aq for all 1= Le.
(2)
Auxiliary risk fanctions: The second set performance measures (Ri())) are uncon- strained, and optimized in best-effort fashion via the selection of the hyperparameter X. Accordingly, the goal of hyperparameter selection is defined as the multi-objective problem
min {RL, R.....))......
xyλ
subject to (A(A) og for all 5 Los
which targets the minimization of the auxiliary risk functions (R(( under constraints the reliability risk functions {R, For example, in the setting of Fig. we wish to minimize the prompt length, while ensuring constraint on the accuracy of the LLM'S outputs.
Solving a multi-objective optimization problem such as 3 ideally entails identifying the entire Pareto front of dominant solutions À e A. at least obtaining specific solutions corresponding to scalar
3

---

However, the problem (3) cannot be directly addressed since the data distribution Pz assumed be unknown. Instead, we assume have access i.i.d. data و drawn from the unknown data distribution Pz. For any data subset 2 C 2, the empirical estimate of risk function (A) can be obtained as
criteria [24 25].
(A
1
Σ (()
2
ZEŽ
2.2 Hyperparameter Selection as Multiple Hypothesis Testing
As proposed in [.. hyperparameter selection can be formally addressed as an MHT problem. Accordingly, for each hyperparameter À A, define the null hypothesis H. that hyperparameter violates the reliability constraints (2) i.e..
Hx there exists / E {L,....Lc} such that R((A) > aq.
(5)
Thus, rejecting the null hypothesis HA implies that hyperparameter À meets all the constraints in 2 A rejection is also referred to discovery. A discovery is false if the selected hyperparameter X, i.e. the hyperparameter for which the corresponding null hypothesis H, is rejected, is actually unreliable, satisfying the null hypothesis H2,
Given dataset 2 S Z, evidence against the reliability of each candidate hyperparameter À E A can be measured by a p-value derived by applying Hoeffding's 's inequality (26) to the empirical mean of bounded losses. Specifically, for each reliability risk function l,.... Les we define the statistic
(2) exp(-2)ž[(ar R ())
(6)
Combining the statistic 6 across all T.g risk functions via the maximum
px (2)
max (2).
(7)
yields a valid p-value for null hypothesis HA in the p-value px(2 (2) yields a reliability test that controls type-I error with finite-sample guarantees.
5
15
Appendix A.2]. Therefore, thresholding
Furthermore, by formulating hyperparameter selection an MHT problem, we can leverage statistical tools that guarantee false discovery rate (FDR) requirements [27] To elaborate, define as Åz the subset of hyperparameters selected by an MHT mechanism. The FDR defined as the expected proportion of unreliable hyperparameters in set Âz. Therefore, controlling the FDR amounts to finding a subset Åg S A that satisfies the inequality
Exer, 1( > on for any / I,..,,, Le}
Ez
<6,
(8)
max((Azl,1)
where 1(-) is the indicator function, and the expectation is taken over the unknown data distribution Pz. The FDR constraint (8) ensures that the average fraction of unreliable hyperparameters in set
Åz is upper bounded by 8.
3 Reliability Graph-Based Pareto Testing
In this section, we introduce RG-PT, novel hyperparameter selection strategy based on MHT that adopts testing schedule based on the novel concept of RG.
3.1 Overview
The design of RG-PT starts from the observation that the reliability of some hyperparameters can be highly predictive of the reliability of other hyperparameters, and that this structure can be encoded by DAG as in Fig. Ta By incorporating the DAG structure in the MHT process of hyperparameter selection, RG-PT supports a more efficient hyperparameter selection procedure, while meeting formal reliability constraints in terms of the FDR

---

As illustrated in Fig. 2 using partition Z = (Zorr, _MHT} of the data set 2, RG-PT applies the following steps:
D Estimating the Puretofront for all risk measures: Following PT S RG-PT uses the dataset Zorr to identify the subset AOFT S A of hyperparameters that the Pareto front of the space of estimated
risk measures (Ai(A) 1 This is done by addressing the multi-objective optimization problem
3 with the estimates { (o) in lieu the true risks {R((A) Zopr) using any suitable
multi-objective optimization algorithm [5].
2 Learning the reliability graph: Rather than ordering the hyperparameters in subset AOPT in a linear sequence as done by PT [5]. RG- PT creates an RG, with nodes given by the hyperparameters
AOPT
DAGGER Åg
subset ΛOPT- This is done by following the principle that hyperparameters E АOPT whose reliability levels are predictive of the reliability levels of other hyperparameters A' C ΛOPT should be tested before the hyper- parameters A'. As detailed in Sec. 3.2 the RG construction leverages the BT model to incor- porate prior information and the non-negative Lasso to determine the links in the graph.
Identify
2 Learn the RG
S- Apply controlling MHT via DAGGER
hyperparameters the Pareto front
Figure 2: Illustration of the main steps of RG-PT: ! Estimate the hyperparameters AOPT lying on the Pareto front pf problem (2); 2 Build the RG over the selected hyperparameters ΛOPT: 3 Apply an FDR-controlling MHT procedure, DAGGER, to the RG to obtain the selected set Åg S АOPT.
3 FDR-controlling MHT: Using the data set 2MHT. contcoonoo MHT carried out by incorporating the structure encoded by the RG. As explained Sec. 3.2 this is done by using
DAGGER 18], returning the subset Âg S Aorr-
4 Addressing the multi-ohjective optimization problem: Given the subset Åz, RG-PT addresses the problem
min
Rr. (AI ZoPT))
where the auxiliary risk functions (A) in are replaced with the corresponding empirical estimates obtained with data ZOPT-
3.2 Learning the Reliability Graph
After obtaining the estimated Pareto front ΛOPT, RG-PT constructs an RG to encode the expected relationships between the reliability levels attained by the candidate hyperparameters in the set ΛOPT- The RG is a DAG in which each node represents a hyperparameter À E АOPT, and edges are directed to describe reliability hierarchy. Specifically, edges encode the expectation that parent nodes are predictive of the reliability of their child nodes.
Accordingly, starting from the nodes with no parents and following the direction of the edges in the RG, one encounters hyperparameters that are estimated to be increasingly unreliable. Generalizing the linear ordering assumed by PT, the DAG structure adopted by RG-PT can thus assign the same expected reliability ranking to multiple hyperparameters. Specifically, all the hyperparameters the same depth in the DAG are deemed to have the same relative reliability level. This partial ordering enables more efficient exploration of the hyperparameter space.
In order to learn the RG, RG-PT leverages the data set ZOPT, as well as, possibly, prior information about the relative reliability of pairs of hyperparameters. This done via the following two steps:
Depth assignment: The hyperparameters set Aorr are ranked in terms of their expected reliability, allowing for multiple hyperparameters be ranked equally. This step thus assigns depth level d in the DAG to each hyperparameter E AOPT- As explained in Sec. 3.2.1 this done by leveraging the BT ranking model [16).
Learning the directed edges: Given any hyperparameter À at some depth d, RG-PT selects a subset of hyperparameters at the previous depth level d to serve parents of the hyperparameter X. As
5

---

specified in Sec. 3.2.2 this is done by choosing the hyperparameters at depth d - that are most predictive the rehability level of hyperparameter À via the non-negative Lasso [77
3.2.1 Depth Assignment
Fix number D Σ |Aorr] of levels for the DAG. With D = |APPT] one can assign each hyperpa- rameter À E ΛOPT a distinct level, yielding a global ordering and recovering PT. Conversely, with D -=1 1, all hyperparameters À E ÅOPT are assigned the same level. The setting of interest is thus I << D < |APPI, which is assumed from now on. A procedure for choosing a suitable value for depth D is presented in Appendix
Depth assignment is carried out by first obtaining a score s(A) for all hyperparameters E AOPT using the data set ZOrr. and then partitioning the set AOrr into D clusters according the obtained scores.
To compute the scores (A) for hyperparameters À E ΛOPT, we use the BT model [66] The BT model converts pairwise counts Wij for all pairs of hyperparameters X and X2 in subset ΛOPT into perrayyprparrmeteer scores sA) for all À AOPT. The pairwise count wi measures the number of times that hyperparameter X; was found to be more reliable than hyperparameter Ay. In RG-PT, we propose to evaluate the pairwise counts Wij by leveraging two sources of information:
Prior information is encoded by pairwise probabilities S JJ S for each pair of hyperparameters (Ar, X,) in Aorr. This probability reflects the expected rate at which hyperparameter X is observed to be more reliable than hyperparameter Ay. Note that we have "/i "/j- The strength of the prior information is determined via pseudocount variable np as in the standard categorical-D Diricchlt model [28] A larger pseudocount np indicates a stronger trust in the prior information. Importantly, the statistical guarantees of RG-PT do not depend on the choice of the prior probabilities {Na, and pseudocount np, which can, however, improve the capacity of RG-PT to optimize the auxiliary risk functions. Note that in the absence of prior information, one can set np = 0, and a principled approach to choosing np is discussed Appendix F
Prior information:
Data: Using the p-values Px, (Zorr) and Px, (Zort) in 77, we construct the pairwise comparison
score
Px. (ZOPT)
Pis (ZOP)) PA, (ZOPT) + Px, (ZoPT)
(10)
which satisfies Piy (ZOPT) = 1 Pja( ZOPT). This construction of a comparison score is instance of
the broader class of models used for paired comparison data [29]. Other possible choices include the
Thurstone model (30 Pty (Zorr) = (p ( (ZOPT) PA, (ZPP))). where ob-) is the normal CDF, and
the BT model 31 Pty (OPT) 1/(1 + e (Px, (Zorr) (2nrr)). While we adopt (10). alternative
comparison models could be used without affecting the theoretical guarantees of RG-PT.
Overall, the pairwise count Wij is obtained by combining prior information and data as
Wij = Zopr (P.y (Zort) + 7p
(11)
so that the relative strength of the prior information in depends on the ratio np ZoPr between the pseudocount n, and the number of data points | The pairwise count waj in (1 can be viewed as smoothed estimate of the reliability of hyperparameter A; relative to hyperparameter Ay. where prior and empirical counts combine additively to form the effective posterior sufficient statistie. The weighting mimics a Bayesian update, ensuring that both prior knowledge and observed evidence contribute the final comparison score
32 33 34
Using the BT model, the scores s sAx:) for all hyperparameters X E Aorr are obtained by maximizing the likelihoddo [16]
Aoeer
|Aper
sW,) s(A:) + s(A,)
Wiy In
(12)
==1
with w. = 0 for all S i | With this design, in the absence of prior information (n, = 0), the BT model reduces to assigning scores directly proportional to the p-values px px (ZorT).
After obtaining the scores s(A.) for all S i |APPT], depth assignment done via clustering,
producing disjoint subsets A Ap. The cluster A₁ S AOPT contains the hyperparameters with

---

the highest expected reliability, and the remaining clusters ......, Ap are sorted in descending order of expected reliability. All hyperparameters cluster Ag are assigned depth level d.
Clustering can be implemented by using methods such as K-means or hierarchical clustering. We recommend using agglomerative hierarchical clustering, which begins with each hyperparameter in its own cluster and iteratively merges clusters [35].
3.2.2 Learning the Directed Edges
Having obtained the clusters A1,..,y Ap. the RG is constructed by: (i) including one node for each hyperparameter À E Aopr; and (ii) selecting for each hyperparameter À Ad at depth level d subset of hyperparameters in cluster Ad-1 to serve as parents of À for all depth levels 2 S d < K. The resulting directed edges are intended to represent inferred reliability dependencies.
To this end, we implement feature selection via the non-negative Lasso [77 Further discussions the choice this standard algorithm can be found in Appendix [ Specifically, given hyperparameter
Ad. consider the problem of predicting the risks {rr Z, ) from the risks (r, x) attained by the hyperparameters E Ad-1 at the previous depth level. The use of non-negative Lasso regression ensures that only positive correlations are represented in the DAG, preserving hierarchical reliability relationships between parent and child nodes.
Formally, using the data set ZOPT, for each hyperparameter À E Ad address the problem
min Σ ZoF
((2,)) E Bxr(2,X) N'AA-1
+ Σ
Bx
(13)
X'E
where (3x is the vector of non-negative regression coefficients corresponding to each potential parent node X' Ad-1; r(Z, A) is the vector containing the values {rr(Z, A) H= || |l2 represents the Eg norm; and 7 > is a regularization parameter that controls the degree of sparsity in the solution. The procedure for choosing a suitable value for the regularization parameter discussed in Appendix F After solving the convex problem (13). only the hyperparameters X E Ay-1 for which the corresponding coefficient B8 are positive are selected as parent nodes of hyperparameter A. As for the variables (np. {ny }) in the BT likelihood [12), the choice of the parameter T does not affect the validity properties of RG-PT.
3.3 FDR-Controlling Multiple Hypothesis Testing
Given the obtained RG. RG-PT performs MHT via DAGGER 18] an FDR-controlling algorithm that operates on DAGs. DAGGER begins testing at the root nodes of the RG, i.e., at the hyperparameters in cluster A1, and proceeds with the clusters A2, Ag..... AD. guided by the outcomes of prior tests. If a hyperparameter is deemed unreliable, none of its descendants are tested.
The test for each hyperparameter À AOPT detects X; reliable if the p-value px, (EMITT) in is no larger than threshold 6,, i.e., if Px. (2MHT) I 6,- The testing level 8, is determined by DAGGER based several factors, including the overall target FDR level in constraint (8). the number of reliable hyperparameters identified among those tested prior to A; and the structure of the graph rooted λę We refer the reader to Appendix and [18] for details.
An algorithmic overview RG-PT is provided in Appendix D The following proposition states the theoretical guarantees provided by RG-PT.
Proposition 3.1. The set Â of hyperparameters returned by RG-PT controls the FDR below the pre-specified threshold as in
Proof. RG-PT applies the DAGGER algorithm 18 to DAG over the candidate hyperparameters, using valid p-values defined in 7 Since DAGGER controls the FDR at level å for any DAG structure when supplied with valid p-values under arbitrary dependence (see Appendix B). the result follows directly.

---

4 Experiments
In this section, we evaluate the proposed RG-PT hyperparameter selection strategy on a prompt engineering problem [1] and a sesguencce-t-seqqeeee translation task [36). Additional experiments
including on object detection [4] and telecommunications [37 can be found in Appendix H
Throughout the experiments, we adopt as benchmarks LTT and PT [5] To the best of our knowledge, LTT and PT are the only existing hyperparameter selection methods that guarantee (8). justifying this choice. L.TT is implemented by applying Benjamini- Hochbbrrg (BH) (38] as the "DR-controlling algorithm, while PT follows [5], with the caveat that FDR conttoongg FST [39] is used in lieu of an FWER-controlling scheme. LTT uses the entire calibration data set Z to evaluate the p-values used in BH, while PT and RG-PT partition Z into data sets ZOFT and ZMHrr-
statistical validity in the sense of the FDR constraint
Across the experiments, following common practice in the MHT literature 38 including the LTT and PT papers, we set 8 = 0.1 in (8), corresponding to a confidence level of 90% for the reliability of the selected hyperparameters. Additionally, the choice for the threshold values on, < S Le, in (8) depends on the downstream application. In our experiments, we selected these thresholds via preliminary validation sweeps to identify values that effectively separate low-risk configurations, while maintaining adequate statistical power.
4.1 Reliable Prompt Engineering
Problem Setup. In this experiment, we focus on prompt engineering for the following three tasks from the instruction induction data set [1: 1. Sentiment analysis: In this task, based the Stanford Sentiment Treebank dataset [6], each data point Z = (X, Y) encompasses a movie review X, and the corresponding sentiment Y E {positive, negative), ). 2. Sentence similarity: In this task, based on the Semantic Textual Similarity Benchmark dataset [40), each data point = (X, Y) comprises two sentences input X, along with semantic similarity label Y [0. .0......3 5). 3. Word in context: In this task, based on the Word-in-Context dataset [1], each data point 7 = (X, Y) consists of a
target word and two context sentences as input X, paired with binary label Y {same, not same] indicating whether the target word shares the same meaning across both contexts.
For each task, we use 1000 examples each for the data sets Zorr and ZMHr, well as for the test data set. Furthermore, following the forward generation mode detailed in [42], we use the LLaMA3-70B-Img Ins-rrrutctt model [33 to generate set A (>1,...,AA10) of distinct instruction-style prompt templates for each task.
Given prompt À and an input X, the smaller LL concatenated input [), X], producing the output f((A, X]). For each input- pair 7 (X, Y), a task-specific 0-1 prompt loss is calculated as "prompt Z, A) (() Y) E {0. 1}. indicating whether the task was performed correctly. The objective is to find prompts in set A that control the average prompt loss Rgrompt (A) Ez prpompp (Z, A)] below a target level of
model 44
is queried with the
0.2, while
minimizing the average prompt length. For this selection, we wish to control the FDR 5 = 0.1.
at level
Prior Information via LLM-as-a-Judge- To incorporate prior structure into the reliability graph, we adopt the a-duee framework (23]. For each pair of prompts X ) E AOPT. we query the GPT-4 Turbo 100-4-0225pprvieee temperature = O, max tokens = 10) model 45 with a task-specific prompt template to assess which instruction is more likely to elicit correct or helpful response. We perform this comparison once per hyperparameter pair A Ay E АOPT. and define the binary pairwise preference Tij = 1 if GPT-4 Turbo selects тą₃ more reliable, and T/i 0 otherwise. For each reliable set of prompts returned by each method, we choose the shortest prompt as the final hyperparameter choice. We set the pseudocount n,p to 1,000.
Results. For each task, we plot the distribution of the length of the shortest reliable prompt for 100 independent runs over random splits of the data set, for LTT. PT, and RG-PT in Fig. [bb] Fig. Ba and Fig. 3b for the sentiment analysis task, the sentence similarity task, and the word in context task, respectively. The figures demonstrate that RG-PT identifies more concise instructions compared to
'The code for the experiments can be found at the anonymous Github repository http://wwoynooooooopooo.cacc/ces
8

---

LTT and PT by leveraging the prior information provided by the LLM judge. Note that all schemes satisfy the FDR constraint (not shown). For instance, RG-PT achieved an average FDR of 0.089, 0.095, and 0.092 for the the sentiment analysis, the sentence similarity, and the word in context tasks, respectively.
LTT
LIT
RG-PT
RG-PT
50
37.5
45.0
50.0
Length of shortest prompt
Length sbortest prompt
(b)
Figure 3: Distribution of the length of the shortest prompt templates identified by LTT [4] PT (5). and the proposed RG-PT for (a) the Semantic Textual Similarity Benchmark dataset 40 and (b) the Word-in-Contex dataset 41
An ablation study the effect of the RG depth D, well the effect of misspecified prior information for this experiment can be found in Appendix H.3
4.2 Sequence-to-Sequence Language Translation
We consider a sequence-to-sequenee language translation task on the WMT16 Romanian-Engliss dataset [46], using BLEU [47] and ROUGE-L 48 as the objectives, Following (36), the dataset is
preprocessed with SentencePiece tokenization Two key hyperparameters are considered:
49 and an LSTM-baser encoder-decoder trained.
1. The hyperparameter p controls the sparsity of the output distribution using Entmax [5 transition- ing between dense output with softmax (p- = 1) and sparsemax (p = 2) [1].
2. The Fenchel- Fenchel-Young label smoothing strength is training regularization hyperparameter that determines the extent to which one-hot targets are mixed with uniform noise based on Fenchel- Young losses [36]. Accordingly, the original one-hot targets are assigned weight e, while the uniform distribution over all possible classes assigned the weight e.
To create the initial candidate set A, we selected hyperparameters over a grid of 32 combinations, using 8 logarithmically spaced values the interval [1. 2] for p. and values in the set (0.0, 0.01, 0.05, 0.13 for e. This selection is in line with reference (36]
To up RG-PT, leveraged the prior knowl- edge that less sparse settings may be more reli- able than their sparser counterparts. Specifically, for any two hyperparameters A; (pi, <;) and A -pj:e,) where Pi Pj. we assigned prior probability Thj = 1, reflecting this prior reliability assumption. Furthermore, the pseu-
0.38
LET
0.57
ROUGE-1
0.96
docount parameter np, which determines the weight of prior information, was be equal to -.PT|
BLEU
Figure 4: Test ROUGE-L scores achieved by LTT, PT, and RG-PT methods as function the target reliability value for the BLEU score.
Denote RELEU(A) (X) and PROUGE()) the age BLEU and ROUGE-L scores, respectively, obtained for given hyperparameter configura- tion À = (p, e). The goal is to guarantee the BLEU score to be above threshold a, while maximizing the ROUGE-L score. This amounts to
an instance of problem 33) with L = 2, L. = 1, R1 (A) = (A), R2(A) - (A),GI(()).
and 8 0.1. After MHT, all the schemes choose the hyperparameter À E Åg with the maximum

---

estimated value RROUGE (A), i.e., minimum R₂ (2 for LTT, and minimum R2(A) ZoFT) for PT and RG-PT. The data set sizes are | 400, |ZOrr| 200, and ZMHTI 200.
Fig. 4 illlustrates the ROUGE-L score achieved on the test data by each calibration method, plotted against the target value for the BLEU score. The results demonstrate that RG-PT consistently maintains higher ROUGE-L scores, even under stricter requirements for the BLEU score. This highlights RG-PT's advantage in effectively exploring the hyperparameter space, enabling a more efficient testing procedure and identifying superior hy peprpaaraeeter configurations that still statistically satisfy the desired conditions on the risk functions.
To assess the practical implications of the different theoretical guarantees offered by BO 9 and MAB 8 methods, implemented both approaches on this task, and compared their achieved FDR with LTT, PT, and RG-PT. For BO, we employed the scikit-opt ommizzz implementation with the Upper Confidence Bound (UCB) acquisition function [9 11. For the MAB baseline, we used Thompson Sampling, a canonical stochastic bandit algorithm with Bayesian regret guarantees [52, (3). We observed that only RG-PT, LTT, and PT satisfied the target FDR level a 0.1, while the achieved FDR on the test dataset for BO and MAB was 0.14 and 0.21, respectively.
5 Conclusion, Limitations, and Future Work
In this paper, we have introduced RG-PT, a novel framework for multi-objective hyperparameter selection that integrates MHT with the concept of RGs capture interdependencies among candidate hyperparameters. By leveraging a DAG structure informed by prior knowledge and data, RG-PT enables more powerful parallel testing of hyperparameters compared the state-of-the-art methods LTT and PT. RG-PT provides statistical guarantees through FDR control, while expanding the space of reliable hyperparameter configurations, leading to a superior optimization of auxiliary objectives.
Limitations this work include the exclusive applicability to settings with discrete hyperparameter spaces and the lack of theoretical results on the power and sample efficiency of the method. Future work may focus on optimizing the RG structure to maximize power, on the use of synthetic data for the derivation of an RG, as well as the integration with sequential testing methods based on e-processes [54].
Acknowledgments
This work was supported by the European Union's Horizon Europe project CENTRIC (101096379), by the Open Fellowships of the EPSRC (EP/W024101/1), and by the EPSRC project (EP/X01188) 1852/1).
References
[1] Or Honovich, Uri Shaham, Samuel R Bowman, and Omer Levy. Instruction induction: From few examples to natural language task descriptions. arXiv preprint arXiv:2205.1077 2022.
[2] Jinlan Fu, See-Kiong Ng. Zhengbao Jiang, and Pengfei Liu. Gptscore: Evaluate you desire. arXiv preprint arXiv:2302.0044 2023.
[3] Lingjiao Chen, Matei Zaharia, and James Zou. FrugalGPT: How use large language models while reducing cost and improving performance. Transactions on Machine Learning Research, 2024.
[4] Anastasios N Angelopoulos, Stephen Bates, Emmanuel Candès, Michael I Jordan, and Lihua Lei. Learn then test: Calibrating predictive algorithms to achieve risk control. arXiv preprint arXiv:2110.01002,2000 2021.
[5] Bracha Golnsseenl, Adam Fisch, Regina Barzilay, and Tommi S. Jaakkola. Efficiently controlling multiple risks with Pareto testing. In Proc. International Conference on Learning Representations, 2023.
[6] Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over
10

---

sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1631-1642, 2013.
[7] James Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl. Algorithms for hyper- parameter optimization. Advances in neural information processing systems, 24, 2011.
[8] Lisha Li, Kevin Jamieson, Giulia DeSalvo, Afshin Rostamizadeh, and Ameet Talwalkar. Hy- perband: A novel bandit-based approach to hyperparameter optimization. Journal of Machine Learning Research, 18(185)1-52, 2018.
[9] Jasper Snock, Hugo Larochelle, and Ryan P Adams. Practical bayesian optimization of machine learning algorithms. Advances in neural information processing systems, 25, 2012.
[10] Dougal Maclaurin, David Duvenaud, and Ryan Adams. Gradient-based hyperparameter opti- mization through reversible learning. In International conference on machine learning, pages 2113-2122. PMLR, 2015.
[11] Niranjan Srinivas, Andreas Krause, Sham Kakade, and Matthias Seeger. Gaussian process optimization in the bandit setting: no regret and experimental design. Proceedings of the 27th International Conference International Conference Machine Learning, ICML' 10, page 1015-1022, Madison, WI, USA, 2010. Omnipress.
[12] Sébastien Bubeck, Nicolo Cesa-Bianchi, et al. Regret analysis of stochastic and nonstochastic multi-armed bandit problems. Foundations and Trends@ in Machine Learning, 5(1):1-122. 2012.
[13] Tor Lattimore and Csaba Szepesvári. Bandit algorithms. Cambridge University Press, 2020.
[14] Amirmohammad Farzaneh and Osvaldo Simeone. Ensuring reliability via hyperparameter selection: Review and advances. arXiv preprint arXiv:2502.04200 2025.
[15] Kalyanmoy Deb, Amrit Pratap, Sameer Agarwal, and TAMT Meyarivan. fast and elitist multiobjective genetic algorithm: NSGA-II. IEEE transactions on evolutionary computation, 6(2):182-197, 2002.
[16] David R Hunter. Mm algorithms for generalized bradley-terry models. The annals of statistics, 3211):384 406, 2004.
[17] Robert Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society Series B: Statistical Methodology, 58(1):267-288, 1996.
[18] Aaditya Ramdas, Jianbo Chen, Martin J Wainwright, and Michael Jordan. A sequential algorithm for false discovery rate control on directed acyclic graphs. Biometrika, 106(1):69-86) 2019.
[19] James Bergstra and Yoshua Bengio. Random search for hyper-parameter optimization. Journal of machine learning research, 13(2), 2012.
[20) Katharina Eggensperger, Philipp Müller, Neeratyoy Mallik, Matthias Feurer, René Sass, Aaron Klein, Noor Awad, Marius Lindauer, and Frank Hutter. Hpobench: A collection of reproducible multi-fidelity benchmark problems for hpo. arXiv preprint arXiv:2109.06716, 2021.
[21] Stefan Falkner, Aaron Klein, and Frank Hutter. Bohb: Robust and efficient hyperparameter optimization at scale. In International conference on machine learning, pages 1437-1446. PMLR, 2018.
[22] Noor Awad, Neeratyoy Mallik, and Frank Hutter. DEHB: evolutionary hyberband for scalable, robust and efficient hyperparameter optimization. In Zhi-Hua Zhou, editor, Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21, pages 2147-2153. International Joint Conferences on Artificial Intelligence Organization, 8 2021. Main Track.
[23] Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, et al. A survey on ilm-as-a-judge. arXiv preprint a55552241.15554, 2024.
11

---

[24] Kalyanmoy Deb, Karthik Sindhya, and Jussi Hakanen. Multi-objective optimization. In Decision sciences, pages 161-200. CRC Press, 2016.
[25] Kevin Jamieson and Ameet Talwalkar. Non-stochastic best arm identification and hyperparame- ter optimization. In Artificial intelligence and statistics, pages 240-248. PMLR, 2016.
[26] Wassily Hoeffding. Probability inequalities for sums of bounded random variables. Journal of the American statistical association, 58(301):13-30, 1963.
[27] George Casella and Roger Berger. Statistical inference. CRC press, 2024.
[28] Christopher Bishop and Nasser M Nasrabadi. Pattern recognition and machine learning. volume 4. Springer, 2006.
[29] Manuela Cattelan. Models for paired comparison data: A review with emphasis dependent data. 2012.
[30] Louis L Thurstone. A law of comparative judgment. In Scaling, pages 81-92. Routledge, 2017.
[31] Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: the method of paired comparisons. Biometrika, 39(3/4):324-345, 1952.
[32] José M Bernardo, Adrian FM Smith, and Mark Berliner. Bayesian theory, volume 586. Wiley Online Library, 1994.
[33] Persi Diaconis and Donald YIvisaker. Conjugate priors for exponential families. The Anmals of statistics, pages 269-281, 1979.
[34] Andrew Gelman, John B Carlin, Hal Stern, and Donald Rubin. Bayesian data analysis. Chapman and Hal//CRC, 1995.
[35] Anil K Jain and Richard C Dubes. Algorithms for clustering data. Prentice-Hai Hall,, Inc., 1988.
[36] Ben Peters and André FT Martins. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2642-2654, 2021.
Smoothing and shrinking the sparse seq2seq search space.
[37] Alvaro Valcarce. Wireless Suite: A collection of problems in wireless telecommunications.
2020.
[38] Yoav Benjamini and Yosef Hochberg. Controlling the false discovery rate: a practical and powerful approach to multiple testing. Journal of the Royal statistical society: series B (Methodological), 57(1):289-300, 1995.
[39] Gavin Lynch, Wenge Guo, Sanat K. Sarkar, and Helmut Finner. The control of the false discovery rate fixed sequence multiple testing. Electronic Journal of Statistics, 11(2):4649- 4673, 2017.
[40] Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lapez-Gazpio, and Lucia Specia. Semeval-2017 task 1: Semantic textual similarity preprint 17Xvv:1708.0055 2017.
similarit-multilinguaa multilingual and cross-lingual focused evaluation. arXiv
[41] Mohammad Taher Pilehvar and Jose Camacho-Collados. Wic: the word-in-context dataset for evaluating context-sensitive meaning representations. arXiv preprint arXiv: 09100, 2018.
[42] Yongehao Zhou, Andrei loan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. Large language models are human-level prompt engineers. In The Eleventh International Conference on Learning Representations, 2022.
[43] meta. Llama. https://LLaM. meta. Com/LLaMA3 2025. [Online; accessed 28-Jan-2025).
[44] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, al. The llama 3 herd of models. arXiv preprint ar arXi::2407.21788, 2024.
12

---

[45] OpenAl. GPT-4 technical report, 2023, https://opents cww. com/gpt-4 gpt-4
[46]
Ondicj Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Kochn, Varvara Logacheva, Christof Monz, Matteo Negri, Aurélie Névéol, Mariana Neves, Martin Popel, Matt Post, Raphael Rubino, Carolina Scarton, Lucia Specia, Marco Turchi, Karin Verspoor, and Marcos Zampieri. Findings of the 2016 conference on machine translation. In Ondřej Bojar, Christian Buck, Rajen Chatterjee, Christian Federmann, Liane Guillou, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Aurélie Névéol, Mariana Neves, Pavel Pecina, Martin Popel, Philipp Koehn, Christof Monz, Matteo Negri, Matt Post, Lucia Specia, Karin Verspoor, Jörg Tiedemann, and Marco Turchi, editors, Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers, pages 131-198, Berlin, Germany, August 2016. Association for Computational Linguistics.
[47] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. BLEU: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311-318, 2002.
[48] Chin- Chin-Yewll Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 14-81, 2004.
[49] Taku Kudo. Subword regularization: Improving neural network translation models with multiple subword candidates. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 66-75, 2018.
[50] Ben Peters, Vlad Niculae, and André FT Martins. Sparse sequence-to-seqqeene models. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1504-1519, 2019.
[51] Andre Martins and Ramon Astudillo. From softmax to sparsemax: A sparse model of attention and multi-label classification. In International conference on machine learning, pages 1614- 1623. PMLR, 2016.
[52] William R Thompson. On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika, 25(3/4):285 2944, 1933.
[53] Daniel J Russo, Benjamin Van Roy, Abbas Kazerouni, lan Osband, Zheng Wen, et al. A tutorial on thompson sampling. Foundations and Trends@ in Machine Learning, 11(1):1-96, 2018.
[54] Matteo Zecchin, Sangwoo Park, and Osvaldo Simeone. Adaptive learn-then-test: Statistically valid and efficient hyperparameter selection. arXiv preprint arXiv:2409.15844 2024.
[55] John A. Rice. Mathematical Statistics and Data Analysis. Belmont, CA: Duxbury Press., third edition, 2006.
[56] Yoav Benjamini and Daniel Yekuticli. The control of the false discovery rate in multiple testing under dependency. Annals of statistics, pages 1165-1188, 2001.
[57] Daniel Müllner. Modern hierarchical, agglomerative clustering algorithms. arXiv preprint 10X 2011.
[58] Jerome H Friedman, Trevor Hastie, and Rob Tibshirani. Regularization paths for generalized linear models via coordinate descent. Journal of statistical software, 33:1-22, 2010.
[59] Aaditya K Ramdas, Rina F Barber, Martin J Wainwright, and Michael I Jordan. A unified treatment of multiple testing with prior knowledge using the p-filter. The Annals of Statistics, 2755):2790-2821, 2019.
[60] Peter Rousseeuw. Silhouettes: a graphical aid to the interpretation and validation of cluster analysis. Journal of computational and applied mathematics, 20:53-65, 1987.
[61] Leonard Kaufman and Peter Rousseeuw. Finding groups in data: an introduction to cluster analysis. John Wiley & Sons, 2009.
[62] Bradley Efron and Carl Morris. Stein's estimation rule and its competitors-an empirical bayes approach. Journal of the American Statistical Association, 68(341):117-130, 1973.
13

---

[63] Nicolai Meinshausen and Peter Bühlmann. Stabiliit selection. Journal of the Royal Statistical Society Series Statistical Methodology, 72(4):417-473, 2010.
[64] Rajen D Shah and Richard Samworth. Variable selection with error control: another look at stability selection. Journal of the Royal Statistical Society Series B: Statistical Methodology, 75(1):55 80, 2013.
[65] Martin Wainwright and Max Chickering. Estimating the" wrong" graphical model: Benefits in the computation-limin setting. Journal of Machine Learning Research, 7(9), 2006.
[66] Luca Franceschi, Michele Donini, Valerio Perrone, Aaron Klein, Cédric Archambeau, Matthias Seeger, Massimiliano Pontil, and Paolo Frasconi. Hyperparameter optimization in machine learning. arXiv preprint arXiv:2410.22854 2024.
[67] Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft COCO: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740-755. Springer, 2014.
[68] Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan- Wen Lo, and Ross Girshick. Detectron2. http:ooowooo.ercccceeeeettteeeee
https://gg t/ht:///gbbu.........gg. thub.
2019. Accessed: 2024-11-19,
[69] DM Powers. Evaluation: from precision, recall and f-measure to roc, informedness, markedness & correlation. Journal of Machine Learning Technologiex, 2:37, 2011.
[70] Dani Yogatama, Lingpeng Kong, and Noah A Smith. Bayesian optimization of text represen- tations. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. 2015.
[71] Pedro M de Sant Ana and Nikolaj Marchenko. Radio Access Scheduling using CMA-ES for optimized QoS in wireless networks. In 2020 IEEE Globecom Workshops (GC Wkshps), pages 1-6. IEEE, 2020.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The abstract and introduction clearly state the contributions of the paper, including the proposal of the RG-PT method, its theoretical guarantees (e.g., FDR con- trol), and empirical improvements over the saae-offtheean PT and LTT. These claims are substantiated through theoretical analysis and experiments, as discussed throughout the paper.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The paper discusses the main limitations of the proposed method in Sec. and Sec. 5
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory assumptions and proofs
15

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: The main theoretical result of the paper, the FDR control guarantee of RG-PT, is formally stated and proven Proposition 3.1 along with all necessary assumptions.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
The algorithm is described in detail in Appendix D and the experimental setup. including datasets, objectives. model architectures. and hyperparameter grids, fully documented in Sec. 4 and Appendix H This information is sufficient to reproduce the main results without access to the code.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often good way to accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
16

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Anonymized code and data are presented as an anonymous Github repository. All datasets used are publicly accessible, and instructions for reproduction are included in Sec. 4 and Appendix H
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//iips.cc/ppll wttpcccppppppppeeeeeeeeeeeeeeeee
httpa///guuieee/eeeeebbiiesssss Coodeeooidsssiiiieccca sonooooocsss for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: All training/test details (c.g., data splits, priors, objectives, and hyperparame- ters) are described in Sec. and Appendix H with key elements summarized in Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
17

---

Where appropriate, such as in Fig. random data splits visualized using boxplots. Guidelines:
Justification:
lb Fig. and Fig. 9 the variability due to
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Computer resources used to run the experiments are clearly stated in Appendix H.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: Yes, the authors have read the NeurlPS Code of Ethics, and confirm that the research conducted in this paper conforms with the guidelines.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- cration due to laws or regulations their jurisdiction).
10. Broader impacts
18

---

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
The potential positive societal impact is discussed in Sec. 1 where the authors
highlight that reliable hyperparameter selection with statistical guarantees can improve the safety and trustworthiness of machine learning in high-stakes domains.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deploymennt of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All external code and datasets used in the paper are properly cited with references to their original sources, and their licenses and terms of use are respected and followed specified by the original creators.
19

---

Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The code for the proposed algorithm, RG-PT, is linked in the paper as an anonymous Github repository.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
20

---

Answer:
[NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not non non- stanstandard components.
involve LLMs as any important, original,
Please refer to our LLM policy for what should or should not be described.
21

---

A Fixed Sequence Testing
In this section, provide brief overview of FST for controlling the FDR, which used in our simulations for PT. While PT. as outlined in [5). is designed to support control of the FWER, our focus in this paper is FDR control.
MHT methods such as the Bonferroni correction for FWER control [55] and the Benjamini- Benjamini-Yekkutill (BY) procedure for FDR control [56] do not leverage any side information about the relative reliability of the hyperparameters. When such information available during calibration, FST can be used to test hyperparameters in order of expected reliability. When the ordering information is accurate, FST can be beneficial to reduce the FDR [39]. In this section, we briefly describe the FST procedure for FDR control.
With FST, the candidate hyperparameters are ordered as A(1) .... AQAAI) using side information. The ordering ideally lists the hyperparameters from the most to the least likely to meet the reliability criterion (
Starting with = 1, each hyperparameter Age) is tested sequentially based on its p- p-value Pxgę against an adjusted critical value 6; that decreases with the index i =
A/. At each step i, the hyperparameter ( is deemed to be reliable if Pxra S 6z Testing continues until hyperparameters deemed to be unreliable, at which point testing stops. The choice of the integer & is typically set as a small proportion, often around 5- 5-10% of the total number of hypotheses, |.].
The critical values 6; are adapted to account for the position of each hypothesis in the testing sequence. These values are specifically designed to control the FDR under various dependency structures among the y-values. For the case of interest here, which is arbitrary dependence of the p-values, the critical levels can be set as (39
isk k if i>k:
8, CIA 8+16
(14)
where 8 is the target FDR level and l is the number of unreliable hyperparameters allowed before testing stops.
The final of reliable hyperparameters Âz (Aa)....,An)) where j corresponds to the index of the last hyperparameter tested before stopping. This ensures that the FDR is rigorously maintained below level 8.
B Summary of DAGGER
This section outlines the step-up procedure used in DAGGER 18 to dynamically adjust the testing thresholds. DAGGER determines the testing thresholds adaptively based on the structure of the DAG and on the outcomes of previously tested hypotheses. At each depth level of the DAG, thresholds are updated dynamically to control the FDR, while respecting the hierarchical dependencies encoded by the DAG.
At each depth d, only the hyperparameters with no unreliable parents are considered for testing. If any parent of a hyperparameter is deemed unreliable by DAGGER, all of its descendants are also automatically deemed unreliable. The threshold for testing the i-th hypothesis at depth d is given by
5
(,(r) V
(15)
8(m; + Ri
where r E [1, [Aal] is parameter set as detailed below; v; is the effective number of leaves in the subgraph rooted at the current node; V is the total number of leaves in the DAG; m, is the effective number of nodes in the subgraph rooted at the current node; R1.d-1 is the total number of rejections at depths 1 through d 1; and ((-) is a reshaping function, such as the Benjamini-Yckuttelii (56) function BBY (x) x Ek=1 1/k, which is designed to ensure FDR control under arbitrary dependence.
The effective number of leaves v; and the effective number nodes m, for node are calculated as follows. If i is a leaf, then we have Vi m, = 1. Otherwise, the values and m. are calculated recursively from leaves roots as
0/
(16)
parents(
(Echildran()
22

---

Unioredd typeruneeeee
pypapprander (rlected typothoii)
hyperperamcerr (acoepted mall typetiena)
Input
V
Pi
Vi 0.5
3.25
0.05
6,(r) 2.25
Pi ( 0.0117
0.01
0.5
1.75
P:
9 0.5
1.5
6,(1 0.75
0.05
R R=
P: 6-2)) 0.0181
0.02
Y4 0.5
0.05
8
0.05
6,(r)
P5 6,1) 0.05
2.
All descondants of u'rrliabbl htyperparamsters deemed unr unreliable
0.08
0,0%
24 > 5(1) 0.0133
2.
Figure 5: Illustration of the DAGGER algorithm's operation to control the FDR at = 0.1. At each step, a hyperparameter tested, starting from the root nodes and progressing level by level through the DAG. The testing thresholds 5; are computed for each hyperparameter X; using the step-up procedure in (15) and 18). using the identity function as B(-). The p-value of each hyperparameter compared against its respective threshold ó; to assess the reliability of A4.
and
m
m; = 1
(17)
pprrents
(ECchildren(s)
where children(1) and parents(i) denote the sets of the children and the parents of node respectively,
The parameter r at depth needs to be determined before testing can begin. To maximize the number of rejections while ensuring FDR control, DAGGER calculates the value
R = arg
max r=,..... Aal
Σ (ZMHT) S 6,(r)) 2r AA
(18)
The threshold 6,(R) is then used to perform the testing for hyperparameter A
The step-up procedure (15) ensures that thresholds are increasingly relaxed, i.e., increased, move further down the DAG. The overall algorithm is described Algorithm [ and an example illustrating DAGGER's operation is shown in Fig- [ The figure highlights how thresholds
updated
and decisions propagated through the DAG. For simplicity, the figure assumes the reshaping function First, the values for the effective number of leaves and the effective number of nodes m, are calculated for each hyperparameter X; Next, going level by level, the thresholds (,) calculated using 1.5 and [8 and each hyperparameter X is tested by comparing Px, (ZмHT) with (R).
8(.) to be the identity function ((x)
C Computational Complexity of Constructing the Reliability Graph
Constructing the RG involves three main steps: training the BT model, clustering, and Lasso regression. The training of the BT model has a time complexity Onn²), because it involves cyclic optimization over the model parameters associated the pairs of configurations [16] Sec. 3]. Hierarchical clustering has complexity (n²) 57 Sec. 3], while I Lasso regression has per-iteration complexity ((n) (58 Sec. 2. Therefore, the overall complexity of RG construction is of the order On?²).
23

---

Algorithm DAGGER 18
Input: DAG structure, p-values (px (ZwHr)) }. target FDR level Output: Set of reliable hyperparameters Az for depth = 1...... do
for each hyperparameter X; in cluster Ag do if all parent hyperparameters of X; are deemed as reliable then Evaluate threshold 6;( using (15) and (18)
if Px, (2MHT) x (:( then Detect A; reliable
else
Detect X, as unreliable
end if
end if
end for
Update Åz with all the hyperparameters detected reliable depth d end for
return Åz
D RG-PT Algorithm
Algorithm
provides summary of RG-PT.
Algorithm 2 Reliability SraphhBaseed Pareto Testing (RG-PT)
1: Input: Hyperparameter set A, calibration dataset 2, FDR level 6, reliability thresholds {ai L. number of DAG levels
1==1
2: Output: Reliable hyperparameter subset Âz
3: Split calibration data: 2 = Zopt J ZMHT
4: Estimate Pareto front ΛOPT S A using žOPT
5: Construct Reliability Graph (RG):
6: Compute pairwise comparisons using ZOPT and optional priors
7: Estimate scores s(A) via Bradley-Teryy model
8: Cluster AOrT into D levels using scores
9: for each level d = 2 to D do
for each Aa do
11: Select parents in Aa-1 via non-negative Lasso
end for
13: end for
Run DAGGER on RG with data 2MHT:
15: for each node in topological order do
16: if' all parents
are reliable then
17: Compute p-value px
18: pA < 5x then
19: Add to Åz
20: end if
21: end if
22:
end for
23:
return Åς
E Sample Efficiency and Finite-Sample Behavior
DAG-structured multiple testing yields higher statistical power compared to linear testing whenever the graph encodes true logical dependencies among hypotheses. We illustrate this through a simple analytical comparison based on FST, which can be seen as special case of DAGGER with chain graph.
As demonstrated
24

---

Setup.
Suppose we wish to test K hypotheses Hı...., Hk. Let m < K of these hypotheses correspond to non-null hypotheses (reliable hyperparameters), and the remaining K m be true nulls (unreliable hyperparameters). Assume all K p-values (p: K i= are independent and identically distributed. Under the null, p; ~ Onif 1], so the CDF Fo( = p. Under the alternative, pt is stochastically smaller, with CDF (P) satisfying Fa(a) = 3
At testing threshold a, the type-I error probability a, while the power (probability of correctly rejecting a non-null) is 3.
Consider an ideal ordering in which all m non-nulls appear first. FST proceeds sequentially, rejecting H4 if p; S a. The probability of correctly rejecting all m non-nulls is
II Pr(p: aa) = 8mh
(19)
Now suppose r 2 null hypotheses appear before the m non-nulls, i.e., among the first m + positions. Assume pessimistically that all nulls come before any non-nulls. FST terminates if it encounters a null with p. > a (probability - a). To reach and reject all m non non-nulls, the procedure must: 1. reject each of the preceding nulls (probability a') and 2. reject all m non-nulls (probability (3").. Thus, the probability of correctly rejecting all m non-nulls is
a" 8Th
(20)
This calculation shows that even a single misplaced null reduces power by a factor of a < 1. In the worst case, with nulls preceding all non-nulls, the power degradation becomes exponential in
In DAG-structured testing framework, when the DAG accurately reflects the logical dependencies among hypotheses, non-nulls tend to be concentrated in subtrees rather than arbitrarily interleaved with nulls. In this case, DAGGER avoids early termination caused by misplaced nulls, yielding higher power and improved sample efficiency compared to fixed linear orders. In the RG setting, this insight underscores the importance of incorporating structural information: cell-specffied edges increase the probability of detecting all reliable hyperparameters at given sample size, while FDR control remains valid regardless of structure.
F Selection of RG-PT Hyperparameters
The theoretical FDR guarantee of RG-PT holds regardless the choices of the hyperparameters D, Up, and r. These parameters affect the construction of the RG and hence influence the statistical power of the procedure, but not its validity. For practical use, however, guidance on their selection is essential. Below we describe the automated, data-driven procedures we employed in our experiments, which allow RG-PT to be applied in reproducible manner without user intervention.
Graph Depth D. We determine the number of levels in the RG by computing the Silhouette score [60 for each candidate value D E 3..3..... D m ) and selecting the value that maximizes the average Silhouette score. The maximum depth can be set |Zorr| or capped at a smaller constant (e.g.. Dmax = 20) for computational efficiency. The Silhouette score quantifies intra-cluster cohesion and inter-cluster separation and is standard criterion for cluster number selection (6]. This approach is fully data-driven, with each cluster naturally corresponding to a DAG level in RG-PT.
Prior Weight up- We set the prior weight ny using predictive empirical Bayes procedure 62 which calibrates the reliance on prior pairwise information Mi according its predictive value on held-out data. Specifically:
1. Define
grid of candidate [0.25] Zorrl, Zorrl, 0.75 ZorrI}.
values
for
np-
c.g.,
2. For each np. compute BT scores (A) for À AOPT.
3. Evaluate the predictive log-likelihood on a validation dataset Zval as
(1p)
Σ
Yig Yiy log (o + (1 Biy) log (a
((A)))
("(Ay)
(21)
A.ASTAOPT
where Vig
1 if X; outperforms Ay in Zval.
25

---

4. Select the n maximizing (pp)
This ensures the prior contributes proportionally its predictive usefulness.
Lasso Regularization т. We tune the non-negative Lasso regularization parameter via stability selection (63) principled approach obtain reproducible sparse structures. The procedure is:
1. Generate multiple random subsamples of the calibration dataset ZoPT, each containing 50% of the data.
2. For each T, fit the non non-negative Lasso each subsample and construct the RG.
3. For each edge (A, + Ay), record its selection frequency across subsamples.
4. Compute the edge stability score for as the average selection frequency.
5. Choose the that maximizes stability, subject to a sparsity constraint on the graph (e.g.. average node degree < 3) to avoid overfitting and preserve power [64].
G Semantics of Edges Inferred via Non-Negative Lasso
The purpose of the RG is to encode reliability dominance relationships among candidate hyperparam- eters, rather than mere predictive correlations. We provide here the structural motivation for our use of non-negative Lasso to infer edges in the DAG.
Let G = -V, denote the RG, where V = - A/AOPT is the set of candidate hyperparame- ters, and _ V V the set of directed edges. An edge (A; - X2) E E indicates that the reliability of X is dependent that of X;, i.e., X is "parent" of Xj in the DAG.
We assume that each configuration X; E has an unobserved binary reliability label R2 E {0, 1}. where R2 1 indicates that Xj is reliable. The relationship between reliability variables can be expressed as the stochastic Boolean structural equation
R3 f( (A+ 21) E}) +
(22)
where (5(-) is an unknown binary-valued function, and ej is noise term, possibly correlated with the output of [ (-). For example, (-( may encode that X is likely to be reliable whenever a sufficient number of its parents are. The noise term allows for the possibility that child configuration is unreliable (Ry- = 0) due to random effects not captured by f-).
While the true reliability labels }Ry} are unobserved, we do observe scalar-valued performance vectors Y(A)) E R² (e.g., per-task or per-example risks), which act as continuous surrogates for reliability. We approximate the Boolean process with the linear regression model
Y(2) ~ Σ Big ((A)))
(23)
i
subject to non-negativity constraints 3.j 2 O to preserve monotonic influence. This yields the non-negative Lasso objective, which provides both sparsity (retaining only the most predictive and interpretable relationships) and monotonicity (reflecting the domain prior that improving parent cannot reduce the reliability the child).
This formulation does not assert that reliability is intrinsically linear. Instead, it provides a tractable linearization of an unknown monotonic Boolean process, in line with influence modeling approaches in graphical models [65]. Crucially, the formal FDR control of RG-PT remains valid regardless of the DAG learning strategy; the graph structure influences statistical power but not correctness of the guarantees.
H Additional Experiments
In this section, we present four new experiments across language model calibration [36], object detection [4) image classification [66], and telecommunications engineering [37], to demonstrate
the advantages of our method over LTTT and PT. We begin by detailing the hardware used for the simulations and the specific parameter settings chosen for each experiment.
26

---

Average FDR achieved by LTT, PT, RG-PT across tasks for a target FDR threshold of H.1 Experimental Setups
All experiments were conducted using dedicated computational resources. Specifically, RG-PT, LTT. and PT runs, along with data generation for the object detection, image classification, and telccommunications engincering tasks, were executed on machine equipped with Apple MI Pro chip (10-core CPU, 16-core GPU, 16 GB RAM). Data generation for the prompt engineering experiment (Section 4.1 and the sequence-to-sequence translation task was performed on an NVIDIA A100 GPU (40 GB VRAM), using CUDA 11.3 and 40 GB system memory.
The RG-PT parameter settings for each experiment are summarized in Table
Table RG-PT parameter settings for each experiment.
Experiment
n2
Prompt Engineering
17
1000
0.1
Sequence-to-sqquee translation
10
200
0.1
Object Detection
20
0
0.1
Image Classification Low Dimension
10
0
0.1
Image Classification High Dimension
0
0.1
Telecommunications Engineering
10
0
0.1
H.2 FDR Analysis Across All Experiments
To begin with, we present high-level summary our experimental validation of Proposition 3.1 The specific details of each experiment, including their objectives, datasets, and task configurations, are provided their respective sections. Briefly, for each task, we ran RG-PT 100 times using different random splits of the available dataset into Zoer, ZMHT, and an independent test set. The target was to control the average FDR on the test set below a threshold 5 = 0.1. We then measured the average FDR the test set across runs, and the results, summarized in Table 2 confirm that RG-PT satisfies the FDR condition as theoretically established in Proposition 3.1 For completeness, also evaluated the average FDR of LTT and PT under the procedure, thereby validating that both methods also maintain FDR control in practice.
Table 2:
5 = 0.1.
Task
Prompt Engineering
Object Detection
Language Translation
Image Classification
Radio Access Scheduling
RG-PT
0.089
0.093
0.095
0.084
0.093
LTT
0.073
0.091
0.097
0.071
0.090
PT
0.079
0.087
0.093
0.071
0.087
H.3
Ablation Study
In this section, we use the prompt engineering task in Sec. 4.1 to perform an ablation study over the RG depth D, well as the effect of misspecified prior information.
H.3.1 Effect of DAG Depth
To assess the impact of the DAG depth D in RG-PT, we vary it from 1 (flat graph, equivalent to LTT) to 100 (fully linear, equivalent to PT), and measure both the length of the shortest prompt in jÂz and the test FDR. F illlustrate the average shortest reliable prompt length and average FDR over 100 runs for each depth, showing that FDR remains valid across depths. Additionally, it can be seen that there exists an intermediate depth that minimizes the average prompt length, indicating that the depth < D < |AoPT| can be chosen to optimize power.
27

---

50
15-
M
30
01003
01103)
0,085
Figure Average shortest prompt length in the returned prompt set Åg and the average FDR achieved by RG-PT as a function of RG depth D.
50
0.100
0.075
0.025
FOR threabold
0.000
0.4
Figure 7: Average shortest prompt length in the returned prompt set Åg and the average FDR achieved by RG-PT as a function of fraction f of flipped pairwise prior probabilities.
H.3.2 Effect Misspecified Priors
To simulate prior misspecification, inject noise into the pairwise priors Vl-- For every pair As r E Aopr, we independently swap the priors 7/ÿ and 7ýj with probability f E [0, 1]. When - the prior remains intact, and when every pairwise preference is completely reversed.
Fig. illustrates the average shortest prompt length in the returned set Å of reliable prompts and the average FDR, across 100 random data splits, as function of the flipped fraction f of pairwise preferences Tī.j. As increases, the shortest prompt length grows, indicating the effect of reduced prior information quality. Despite this, the FDR remains controlled below the target level = 0.1, demonstrating robustness misspecified priors. Notably, for f > 0.6, RG-PT returns empty set Å across all runs, correctly avoiding any potentially unreliable hyperparameters in the face of highly corrupted priors.
H.4 Image Segmentation for Object Detection
We now evaluate the proposed RG-PT framework multi-objective image segmentation task for object detection, leveraging the MS-COCO dataset [67] and a pretrained detector from Detectron2 688 as done in [4]. The task involves three distinct objectives: (i) detecting objects within an image (object detection); (ii) delincating object boundaries (image segmentation); and (iii) assigning correct labels to detected objects (object classification). These tasks measured using recall, intersection- over-union (IoU), and classification accuracy, respectively. The goal is to control classification errors while optimizing recall and segmentation quality, addressing the trade-offs among these objectives.
The performance the detection is determined by three hyperparameters:
1. The object recull threshold (A1) controls the threshold for selecting objects based on confidence scores. Reducing the value of X lowers the confidence threshold, which allows more objects to be selected at the cost of, potentially, increasing false positives.
28

---

Figure 8: Illustration of the benefits of the proposed RG-PT hyperparameter selection scheme the state-of-the-ar LTT and PT for an object detection application [4]. The red arrows mark the objects not detected by object recognition model calibrated using LTT or PT that are instead detected by the same model calibrated via RG-PT (see Appendix H.4 for details).
PT
RG-PT
Moxohbeea Leletted
Loss obgere Stected
Object recall threshold (4))
Figure 9: Example of of hyperparameter distributions for LTT, PT, and RG-PT methods. The box plots show the range (denoted by the horizontal black whiskers). median (represented by the thick black vertical lines), and interquartile range (depicted by the boxes) of the object recall threshold hyperparameter Åy.
2. The mask size threshold (X2) tunes the size of the bounding masks used to segment objects, impacting the IoU score.
The classification certainty level (Ag) controls the certainty level required for object classification, adjusting the tolerance for inclusion in the set of labels assigned to each detected object.
The candidate hyperparameter set A was constructed as per 4 by taking all combinations of 50 linearly spaced values in [0.2, 0.5] for Å1. 5 linearly spaced values in [0.3, 0.7] for X2, and 25 logarithmically spaced values in [ 0.00046, 0) for Ag. These discretization choices were optimized
Denote Rí (A). R₂ (A), and R3 (A) the risks associated with recall, IOU, and coverage, respectively, for hyperparameter À = (A1, A2, Aa). Controlling these risks in the context of problem (3) is equivalent to having = and I.. = Additionally, as 4 we set the targets as an 0.5.
02 = 0.5, and 03 = 0.75 with 0.1. Within the set Åz of reliable hyperparameters returned by the algorithm choice, we choose the hyperparameter in subset Åg with the lowest value of Y in order to increase the number of detected objects much as possible. No prior knowledge was leveraged in creating the RG in this experiment by setting 72p =
We compare the distribution of the hyperparameters returned by LTT, PT, and RG-PT. The distribution obtained by running 200 trials for each algorithm over different splits of calibration data Z into subsets Zorr and ZNHT with Zoorr| = 1500 and ZxHrl = 1500. As shown in Fig. 9 the results demonstrate that RG-PT tends to return lower values for X than both LTT and PT. In particular, both the mean and dispersion for RG-PT are lower than those for LTT and PT. A lower threshold λ allows the detector to select more objects, which directly enhances object recall, while still maintaining controlled levels of segmentation and classification accuracy (see also Fig. g)

---

Notably, this experiment shows that RG-PT still exhibits larger power compared to LTT and PT even when ordering among hypotheses are not hard-coded into the testing procedure. In particular, in this task, we did not assume access to any prior reliability information and set Mp = 0. Despite this, RG-PT still outperformed both L.TT and PT, demonstrating its ability to discover and exploit latent structure purely from empirical data.
H.S
Image Classification
we consider the problem of hyperparameter selection for support vector machine (SVM) model used to classify images from the Fashion MNIST dataset. The Fashion MNIST is a widely used benchmark for image classification, consisting of 70,000 grayscale images of 10 different clothing categories.
In this experiment, following 66
We consider two risk functions (L = 2) in problem namely the classification error Rem (A) and the recall, Ri (A). The classification error (A) measures the proportion of incorrectly classified images out of the total number of samples. The recall measures the ability of a model to correctly identify all the relevant instances of each class. Accordingly, for each class, the recall is computed as the ratio of correctly identified instances of that class to the total number of actual instances of the same class in the dataset. The recall Ree (A) represents the average of the recall values across all classes.
With reference to problem 3 we aim at minimizing recall, i.c. R2(A) = RAce while keeping the classification error rate below 0.3, i.c. R1 Recc()) La 1, and an = 0.3. The goal is therefore defined as
min (A) subject Racee (A) < 0.3.
(24)
This is non-trivial problem since the accuracy maximizing model may not also optimize the recall 669.
The SVM model requires the selection of two hyperparameters [66]. The regularization parameter, C, controls the desired trade-off between maximizing the margin and minimizing the classification error. Lower values of C' allow for a softer margin that can overlook some misclassification errors, while higher values enforce stricter classification error requirements. The kernel coefficient, 7, determines the impact of single training example on the decision boundary, with higher values capturing finer details in the data set but risking overfitting. To create the initial candidate set A, we selected hyperparameters over a grid of 25 combinations, using five logarithmically spaced values in the intervals [-3, 3] and [-4, 1] for C' and y, respectively. This selection is in line with approaches such as [70 for SVM hyperparameter selection.
We used 5,000 data points for training the SVM, and used an additional 5,000 data points as calibration data Z. The calibration data set Z was in turn divided into two groups of size 2,500, for the data sets ZOPT and ZMHHT. respectively.
Fig. TO illustrates the testing procedures of LTT 4 PT [5), and RG-PT. The x- and y-axes represent the logarithmic scales of the two hyperparameters C and y, while the contours indicate levels of recall (A), evaluated on the test data set, function of the hyperparameters À = (c, 7). The numbers illustrate the testing order for each testing method. Note that than LTT, which uses BY, does not follow any inferred order the hyperparameters, and thus does not have the order labels in the figures. Furthermore, while LTT and PT test hyperparameters one by one, following a linear trajectory, RG-PT proceeds along DAG, testing at the same time all hyperparameters at the same depth in the DAG.
LTT and PT are seen to stop at the sixth tested hyperparameter, yielding the set of reliable hyper- parameters Å marked as green dots. In contrast, RG-PT returns much larger set Åg of reliable hyperparameters, also marked as green dots. Choosing within these sets the hyperparameter that minimizes the estimated recall as per problem (24) yields the solutions indicated as green stars, corresponding to a test recall of 0.727 for LTT and PT, and 0.332 for RG-PT.
It is important to note that all three methods yield test accuracies below the 0.3 threshold in 24 Specifically, the hyperparameters selected by LTT and PT result in a test accuracy error of 0.267, while those chosen by RG-PT achieve slightly higher accuracy error of 0.286. Although the accuracy of RG-TPT is closer to the threshold, remains consistent with the statistical guarantee outlined in (24). In fact, RG-PT achieves lower recall while maintaining the desired accuracy constraint, whereas
30

---

0773
0.773
0.223
0.22
0.63
EX22
0.403
0.473
0.425
0.425
00373
00
40.73
10
0.225
0.225
kgg
(a) LTT
PT
6.778
0.775
Real E312
0.835
MAPT
0,355
0.375
(c) RG-PT
Figure 10: Illustration of the hyperparameter selection procedure followed by LTT (a), PT (b), and
RG-PT (c) for the setting studied in Sec. H..5 Each node represents a hyperparameter = (C, 7).
with the numbers representing the testing order. Green nodes show the hyperparameters included in the reliable set ε and the star node shows the hyperparameter in set Â with the lowest recall rate.
LTT and PT follow a more conservative approach, leading to reliable hyperparameter with higher recall.
To demonstrate the scalability of RG-PT high-dimensional hyperparameter spaces, we repeated the previous experiment over grid of 10,000 hyperparameter configurations instead of 25. This grid was constructed using 100 logarithmically spaced values for and y over the same intervals as before. The average FDR across 100 runs is reported in Table highlighting RG-PT's robustness and effectiveness even in high-dimensional settings.
H.6 Radio Access Scheduling
In this section, we study telecommunications engineering problem, namely the optimization of radio access scheduler (37]. In this setup, each user equipment (UE) belongs one of four quality- of-service (QoS) classes, assigned at random, each with its own delay and bit rate requirements [71). The goal is to control the delay of UEs in given QoS class, while simultaneously minimizing the delays for UEs in the other three QoS classes.
Accordingly, in the context of problem (3), choose L = 4 and L. 1, and set the risk R, (A(A) to be equal to the average delay of QoS class for < i S 4. We aim to keep R2 (A) below 15 ms, while minimizing Rí (A), (A),,, and Ra (A). Formally, the problem is stated
min R2 (A), R3 R4 subject to Rg (A) < 15 ms. XEA
(25)
The scheduling algorithm at the base station allocates spectral resources to the UEs. As in [7]] the UEs are randomly distributed within km² area containing a centrally located base station. Each UE has initial buffer of 100 packets, and moves random speeds and directions. Resource
31

---

20
6.0
18-
Average delay (ms)
6.2
16
Target
5.5
6.0
14
12.
5.0
5.6-
10
5.4
LTT
PT
RG-PT
LTT
PT
RG-PT
LTT
PT
RG-PT
LTT
PT
RG-PT
(a) Class
(b) Class
(c) Class 3
(d) Class 4
Figure 11: Distribution of the average delay for the four QoS classes using hyperparameters optimized by PT (left column) and RG-PT (right column). The dashed red line indicates the target threshold for the average delay in QoS class 2.
allocation is carried out in intervals of ms, called transmission time intervals (TTIs), over 10,000 TTIs per episode.
The resource allocation algorithm is controlled by set of hyperparameters where each A; adjusts a specific criterion in the reward model detailed in 337 Hyperparameters
(A1, A2, A3 A4) E A,
λ1. λ2. λ3, and A determine respectively the channel quality for each UE, the total queue sizes at the UEs, the age of the oldest packet in each UE's buffer, and the fairness resource block allocation among UEs.
Calibration and test data generated using the Nokia wireless suite 37 For each run, used 100 episodes for calibration and 100 episodes for testing.
The candidate hyperparameter set A generated by keeping À₁ and X2 at the values \j and X recommended by [71,, and linearly sweeping hyperparameters λ and X4 in [0.02, 0.2] and [-0.1, 0.1]. respectively, with 10 steps each, resulting in total of 100 combinations.
Fig. I1 presents the results of using PT and RG-PT optimize the hyperparameter (A1, A3, M4). Both methods successfully meet the statistical guarantee of (2(A) < 15 ms for class However, RG-PT demonstrates greater ability explore the hyperparameter space A. identifying configurations that more effectively minimize the average delay across the other three classes.
32


Reviews:
[{"review_id": "KTU4Pz4vfR", "reviewer": "Reviewer_ar1u", "summary": "This paper introduces a new framework for multi-objective hyperparameter selection named Reliability Graph-based Pareto Testing (RG-PT). The method addresses the common challenge of balancing performance with cost, such as selecting prompt templates in LLMs. RG-PT models the relationships between hyperparameters as a Reliability Graph (RG), which is inferred from prior information and data using the Bradley-Terry model and non-negative Lasso. This structure allows for a more efficient exploration of the hyperparameter space compared to existing methods such as Learn-Then-Test (LTT) and Pareto Testing (PT). The framework uses the DAGGER algorithm to perform multiple hypothesis testing, which formally controls the false discovery rate (FDR). Experiments on tasks including prompt engineering, object detection, and sequence-to-sequence translation demonstrate that RG-PT outperforms LTT and PT by identifying better-performing hyperparameters.", "questions": "- The paper claims that methods like Bayesian optimization and bandit-based approaches 'lack statistical guarantees' (Line 59). This statement requires clarification. While these methods may not offer FDR control, they are supported by theoretical guarantees, such as bounds on cumulative regret. The paper should more precisely define the type of statistical guarantee it focuses on (i.e., post-selection reliability via FDR control) and provide a clearer justification for why this framework is more suitable for the target applications than regret-minimization frameworks.\n- In Line 117 ('A discovery is false if the selected hyperparameter A is actually unreliable'), for improved clarity and consistency with standard hypothesis testing terminology, consider replacing 'selected' with 'rejected'.\n- The derivation of the p-value in Equation (6) is not explicitly provided. Citing the original source or providing a brief derivation would strengthen the paper's self-containedness.\n- The term 'data-driven probability' used to describe Equation (10) is not standard. Please clarify its meaning and provide a justification for why this ratio of p-values can be interpreted as a probability of one hyperparameter being more reliable than another.\n- The formulation for the pairwise count $w_{ij}$ in Equation (11) appears to be a heuristic combination of data-driven and prior information. Please clarify if this is a standard approach and provide references if available, or justify the specific weighting scheme used.\n- The paper's motivating example (Fig. 1a) has a clear, intuitive structure. In cases where no obvious reliability hierarchy exists among hyperparameters, how does the RG construction perform? For instance, would a lack of structure lead to a flat or fully-connected graph, and what would be the performance implications?\n- The paper uses non-negative Lasso to select parents for a node (Line 245). Is it possible for all regression coefficients ($\\beta_{\\lambda'}$) to be positive, and if so, would this mean a node has all potential parents from the previous level? Please clarify this edge case.\n- The choices for the reliability threshold ($\\alpha=0.2$) and the FDR level ($\\delta=0.1$) are presented without justification. Are these standard values for these tasks, or were they chosen specifically for these experiments? Providing some context or guidance on how to set these crucial parameters in practice would be beneficial.\n- The paper reports the achieved FDR for RG-PT (Lines 311-313), confirming it meets the target. For a more complete comparison, please also report the achieved FDR for the LTT and PT baselines in the experiments.", "limitations": "NaN", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths\n\n* The core idea of leveraging prior knowledge to build a Reliability Graph that reflects the structure of the hyperparameter space is compelling. This innovative approach represents a promising research direction for more efficient hyperparameter selection.\n* The paper provides comprehensive experimental validation across a diverse set of applications, including prompt engineering, language translation, and object detection. These results convincingly demonstrate the superior performance and broad applicability of RG-PT compared to the LTT and PT baselines.\n\n### Weaknesses\n\n* Several key technical concepts lack sufficient explanation or citation. For example, Equations (6) and (7) are presented without justification. To improve clarity and rigor, the paper should provide brief derivations or cite the specific sources from which these formulas are drawn.\n* Proposition 3.1 establishes that RG-PT controls the FDR by inheriting the guarantees of the DAGGER algorithm, but it does not offer theoretical insight into *why* RG-PT outperforms LTT and PT. The paper would be significantly strengthened by a formal discussion of the statistical power or sample efficiency gains that the graph-based structure provides.\n* The paper's central motivation is providing a formal statistical guarantee (FDR control), which is used to justify excluding methods like Bayesian optimization from the experimental comparison. However, the practical utility of this specific guarantee over other frameworks (e.g., regret minimization) is not clearly demonstrated within the context of the chosen experiments.\n* The main text lacks a discussion on the computational complexity and scalability of building the Reliability Graph. While an analysis is present in the appendix, the main paper should address how the method scales with the number of hyperparameters on the Pareto front to clarify its feasibility for larger-scale problems.", "was_revised": true, "final_justification": "I increased my score from 3 to 4 since the authors have responded to my questions and concerns in details. I am convinced that the proposed method is a sufficiently solid algorithm with good experiments supported.", "rebuttal": "We thank the reviewer for the thoughtful and detailed comments. We address each concern below and outline the specific clarifications and additions we will make in the revised manuscript.\n\n1. Lack of theoretical insight into why RG-PT outperforms LTT and PT\n\nPlease see point 4 of our response to reviewer LYAN.\n\n2. Clarification on claim that BO and MAB “lack statistical guarantees”\n\nWe will revise Line 59 to clarify that BO and MAB methods lack post-selection error rate control such as FDR or FWER guarantees. Under assumptions such as the correct specification of kernel models, BO and MAB can  provide bounds on cumulative or simple average regret. These do not offer the type of assumption-free guarantees on the risk of the selected parameters that can be instead provided by FWER- or FDR-controlling methods. \n\n\n\nTo validate the practical impact of the different  theoretical guarantees by BO and MAB, we have now implemented BO and MAB on the seq2seq task, comparing the achieved FDR with LTT, PT, and RG-PT. For BO, we used the scikit-optimize implementation with the UCB acquisition function. For MAB, we used a standard stochastic bandit algorithm, namely Thompson Sampling. \n\nIt was observed that only RG-PT, LTT, and PT satisfy the target FDR level $\\alpha = 0.1$, while the achieved FDR on the test dataset for BO and MAB was 0.14 and 0.21, respectively. \n\n3. Terminology: “selected” vs “rejected” in Line 117\n\nThank you for this suggestion. While “rejected” aligns with classical hypothesis testing, we intentionally used “selected” to match the hyperparameter selection framing. We will clarify this distinction in the revised text for consistency with standard terminology.\n\n4. Derivation of the $p$-value in Equation (6) and (7)\n\nEquation (6) is derived using Hoeffding's inequality, applied to the empirical mean of bounded loss values in $[0,1]$, similar to the approach in LTT [Angelopoulos et al., 2021] and PT [Laufer et al., 2022]. Additionally, the proof that eq. (7) forms a valid p-value can be found in Appendix A.2 of the PT paper [Laufer et al, 2022]. We will include a short derivation and cite these works explicitly to make the paper more self-contained.\n\n\n5. Interpretation of Equation (10) as a “data-driven probability”\n\nWe agree that this terminology is unclear and will revise it. This construction belongs to a broader class of metrics used in paired comparison and choice models [Cattelan, 2011]. A function $f: \\mathbb{R} \\to [0,1]$ is called a link function if it satisfies the symmetry condition\n$$\nf(x) + f(-x) = 1,\n$$\nand is increasing in $x$. In pairwise comparison models, such a link function is used to compare two items with respective scores $p_i$ and $p_j$ (with higher values indicating better performance) by evaluating $f(p_i - p_j)$. This quantity can then be interpreted as the relative likelihood that item $i$ is preferable to item $j$. Examples of commonly used link functions include [Cattelan, 2011]:\n\n- Thurstone model [Thurstone, 1927]: $\\displaystyle f(p_i- p_j) = \\Phi(p_i - p_j)$, where $\\Phi(\\cdot)$ is the normal CDF.\n- BT model [Bradley \\& Terry, 1952]: $\\displaystyle f(p_i- p_j) = \\frac{1}{1 + e^{-(p_i - p_j)}}$,\n- Transformed BT model (our choice) [Bradley \\& Terry, 1952]: $\\displaystyle f(\\log p_i- \\log p_j) = \\frac{1}{1 + e^{-(\\log p_i - \\log p_j)}} = \\frac{p_i}{p_i + p_j}$.\n\nWe chose the linear ratio (transformed BT model) form for its simplicity, numerical stability, and effective integration with the Lasso-based DAG construction. In the revised manuscript, we will clearly state this design choice and emphasize that alternative transformations from this family could be substituted without affecting the theoretical guarantees of RG-PT.\n\n\n6. Justification of Equation (11)'s weighting scheme\n\nEquation (11) defines a convex combination of prior and data-driven comparisons to estimate the pairwise reliability strength between hyperparameters $\\lambda_i$ and $\\lambda_j$. The ratio $n_p / |\\mathcal{Z}_\\text{opt}|$ encodes our confidence in the prior information relative to the observed data, where $n_p$ serves as a pseudocount analogue.\n\nThis formulation is not ad hoc, but is directly inspired by the theory of conjugate priors in exponential family models [Diaconis and Ylvisaker, 1979]. In such models, the natural parameters of the posterior distribution are formed by additive combinations of prior and empirical sufficient statistics. For example, in the Beta-Binomial model, the posterior mean of a Bernoulli parameter $\\theta$ is\n$$\n\\mathbb{E}[\\theta \\mid \\text{data}] = \\frac{\\alpha + s}{\\alpha + \\beta + n},\n$$\nwhere $(\\alpha, \\beta)$ are prior pseudocounts and $(s, n-s)$ are observed successes and failures. This additive structure generalizes to Dirichlet-Multinomial models (of which the BT model is a special case), where prior and empirical pairwise counts are combined to produce regularized estimates of comparison probabilities.\n\nIn our setting, the constructed count $w_{ij}$ can be viewed as a smoothed estimate of the relative reliability of $\\lambda_i$ over $\\lambda_j$, constructed analogously to the posterior update in exponential family models with conjugate priors, where the prior and observed sufficient statistics combine additively to determine the posterior natural parameter. The weighting in Equation (11) thus mimics a Bayesian update, where the prior and data each contribute proportionally to the final score. This principle is well-established in the Bayesian literature [Bernardo and Smith, 1994; Diaconis and Ylvisaker, 1979; Gelman et al., 1995], and we will make this connection explicit in the revised manuscript.\n\n\n7. Behavior when there is no structure among hyperparameters\n\nIn scenarios where no prior structural knowledge is available, we explicitly set $n_p = 0$, resulting in an RG that is learned entirely from held-out data via the BT model and non-negative Lasso. This setting is described in Lines 211 and 221 of the manuscript. Even in this purely data-driven regime, RG-PT retains the formal FDR control guarantees of DAGGER.\n\nClearly, if the hypothesis space is completely unstructured, one cannot hope to learn useful information through this approach. That said, our experiments confirm that RG-PT still exhibits a larger  power compared to LTT and PT even when ordering among hypotheses are not hard-coded into the testing procedure. In particular, in the object detection task, we did not assume access to any prior reliability information and set $n_p = 0$. Despite this, RG-PT still outperformed both LTT and PT, demonstrating its ability to discover and exploit latent structure purely from empirical data. We will clarify this point more explicitly in the revised version.\n\n\n8. Edge case: all non-negative Lasso coefficients are positive\n\nYes, it is possible that a node has all previous-layer nodes as parents if all Lasso coefficients are positive. In such cases, the node will be tested only if all these parents are declared reliable. For example, the bottom node in Fig. 1 has all the nodes from the previous level as a parent, illustrating this case.\n\n9. Justification for the reliability threshold $\\alpha$ and FDR level $\\delta$\n\nThe threshold $\\alpha$ defines what it means for a hyperparameter to be considered “reliable,” i.e., having risk below $\\alpha$. Its choice depends on the downstream application and the desired level of caution. In our experiments, we selected $\\alpha$ via preliminary validation sweeps to identify values that effectively separate low-risk configurations while maintaining adequate statistical power. For example, values of $\\alpha$ in the range $[0.1, 0.3]$ are commonly used in the applications considered in the LTT paper  [Angelopoulos et al., 2021].\n\n\nThe parameter $\\delta$ sets the desired control level for the FDR, specifying the maximum acceptable expected proportion of selected hyperparameters whose risk exceeds the target level $\\alpha$. Following common practice in the MHT literature, including the LTT and PT papers, we set $\\delta = 0.1$, corresponding to a confidence level of 90\\% for the reliability of the selected hyperparameters. We will clarify this rationale and include appropriate citations (e.g., [Benjamini and Hochberg, 1995]) in the revised manuscript.\n\n\n\n10. Report achieved FDR for LTT and PT\n\nThe empirical FDR achieved by RG-PT for all the tasks is already reported in Table 2 of the Appendix. In the revised manuscript, we will extend this table to also include the achieved FDR values for LTT and PT as well. The results for LTT and PT were originally excluded, as the FDR control of these two methods has already been validated experimentally in their respective papers. Importantly, the experiments all demonstrate that LTT, PT, and RG-PT all satisfy the FDR constraint, as backed up by theory.\n\n\nWe thank the reviewer again for the constructive feedback. We believe the above clarifications and additions will significantly improve the clarity and rigor of the paper.\n\n\nReferences:\n\n- Cattelan, M. (2011). Models for paired comparison data: A review with emphasis on dependent data. Statistical Science, 26(3), 412–433.\n\n- Thurstone, L. L. (1927). A law of comparative judgment. Psychological Review, 34(4), 273–286.\n\n- Bradley, R. A., \\& Terry, M. E. (1952). Rank analysis of incomplete block designs: I. The method of paired comparisons. Biometrika, 39(3/4), 324–345.\n\n- Diaconis, P., \\& Ylvisaker, D. (1979). Conjugate priors for exponential families. The Annals of Statistics, 7(2), 269–281.\n\n- Bernardo, J. M., \\& Smith, A. F. M. (1994). Bayesian Theory. Wiley.\n\n- Gelman, A., Carlin, J. B., Stern, H. S., \\& Rubin, D. B. (1995). Bayesian Data Analysis. Chapman and Hall/CRC.\n\n- Benjamini, Y., \\& Hochberg, Y. (1995). Controlling the false discovery rate: a practical and powerful approach to multiple testing. Journal of the Royal Statistical Society: Series B (Methodological), 57(1), 289–300."}, {"review_id": "OYeDU1hGHv", "reviewer": "Reviewer_T6Dk", "summary": "The paper introduces RG-PT, a principled framework for multi-objective hyperparameter selection that elevates classical Learn-Then-Test and Pareto-Testing paradigms by embedding them in a reliability-graph structure. Each candidate configuration is a node in a DAG whose hierarchy and edges are learned from data-driven Bradley-Terry scores and non-negative Lasso, allowing the method to exploit transitive reliability relationships and to incorporate external priors. A DAGGER-based multiple-hypothesis procedure then traverses the graph, guaranteeing global FDR control while aggressively pruning implausible regions of the search space.\nExperiments on LLM prompt engineering and NMT decoding show that RG-PT achieves the same reliability guarantees as baselines.", "questions": "My main concern has already been listed in the weaknesses section, and I look forward to the authors’ response.", "limitations": "yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. Reliability-Graph Formulation: The paper introduces a graph-based abstraction that generalises both Learn-Then-Test and Pareto Testing, letting the method exploit transitive reliability relations instead of a fragile total order;\n\n2. Strong Statistical Guarantees: By embedding the DAGGER procedure in the graph traversal, the approach proves global FDR control under arbitrary p-value dependence, offering firm, theory-backed reliability;\n\n3. Practical Impact Demonstrated: Experiments on LLM prompt engineering and machine translation show the method finds cheaper or higher-quality configurations than LTT/PT while respecting the same FDR budget, validating both efficiency and effectiveness;\n\n4. The paper’s arguments and experiments are quite comprehensive.\n\nWeaknesses:\n\n1. Users need to manually specify parameters such as the graph depth D, the regularization strength of the non-negative Lasso, yet the paper provides no guidance for automatically selecting or tuning these values;", "was_revised": true, "final_justification": "I have thoroughly considered the authors’ response and find that it adequately addresses my earlier concerns. I recommend incorporating the clarifications into the final manuscript. Since my current evaluation is already positive, I will not revise my score.", "rebuttal": "We thank the reviewer for their helpful review. We would like to first clarify that the theoretical FDR guarantee of RG-PT holds regardless of the choices of the parameters $D$, $n_p$, and $\\tau$. These parameters affect the construction of the RG and hence influence statistical power, but not the validity of the testing procedure.\n\nThat said, we agree that practical guidance is important for users, and we plan to include in the revised manuscript a description of a specific  procedure for selecting these hyperparameters. This is the  method we employed in our experiments, and we will include it in the appendix of the revised version to aid future applications.\n\nGraph Depth $D$: We select $D$ by computing the Silhouette score [Rousseeuw, 1987] for each candidate value $D \\in \\\\{2,3,\\dots,D\\_{\\max}\\\\}$ and select the value that maximizes the average Silhouette score. The maximum depth can be set  to $|\\mathcal{Z}\\_\\text{OPT}|$, or a fixed, smaller,  value such as setting $D\\_{\\max} = 20$ for computational efficiency. The Silhouette score quantifies both intra-cluster cohesion and inter-cluster separation, and is a well-established criterion for model-free cluster number selection [Kaufman \\& Rousseeuw, 2009]. This approach is fully data-driven,  and fits naturally with the structure of RG-PT, where each cluster corresponds to a DAG level.\n\nPrior Weight $n_p$: We set $n_p$ using a predictive empirical Bayes procedure, which calibrates the trust in the prior information $\\eta\\_{ij}$ based on its ability to generalize to held-out data. Specifically, we follow the steps below:\n\n- Create a set of candidate values for $n_p$, e.g., $\\\\{0.25|\\mathcal{Z}\\_\\text{opt}|, 0.5|\\mathcal{Z}\\_\\text{opt}|, 0.75|\\mathcal{Z}\\_\\text{opt}|, |\\mathcal{Z}\\_\\text{opt}|\\\\}$\n\n- For each candidate $n_p$, compute the BT scores $s(\\lambda)$ for the hyperparameters in $\\lambda \\in\\Lambda\\_\\text{OPT}$.\n- Evaluate the predictive log-likelihood of the pairwise outcomes on a held-out validation dataset $\\mathcal{Z}\\_\\text{val}$ as\n            $$\n            \\mathcal{L}\\_{\\text{val}}(n_p) = \\sum\\_{\\lambda_i \\in \\Lambda_{\\text{OPT}}} \\sum\\_{\\lambda_j \\in \\Lambda\\_{\\text{OPT}}} \\left[ y_{ij} \\log \\frac{e^{s(\\lambda_i)}}{e^{s(\\lambda_i)} + e^{s(\\lambda_j)}} + (1 - y\\_{ij}) \\log \\frac{e^{s(\\lambda_j)}}{e^{s(\\lambda_i)} + e^{s(\\lambda_j)}} \\right]\n            $$\nwhere $y\\_{ij} = 1$ if $\\lambda_i$ outperformed $\\lambda_j$ in $\\mathcal{Z}\\_\\text{val}$, and 0 otherwise.\n\n- Choose the $n_p$ that maximizes $\\mathcal{L}\\_{\\text{val}}(n_p)$.\n\nThis procedure reflects standard practice in empirical Bayes model selection [Efron and Morris, 1973] and ensures that the prior contributes proportionally to its predictive usefulness.\n\nLasso Regularization $\\tau$: We select the Lasso regularization parameter $\\tau$ using stability selection [Meinshausen \\& Bühlmann, 2010], a principled technique for tuning regularization parameters. Our goal is to obtain a DAG structure that is both sparse (to maximize testing power and interpretability) and stable (i.e., robust to data perturbations). Stability here refers to the reproducibility of selected edges under subsampling. The procedure is as follows.\n\n- We generate multiple random subsamples of the calibration dataset $\\mathcal{Z}\\_{\\text{OPT}}$, each containing a fixed proportion of the data (e.g., 50\\%) as in standard stability selection [Meinshausen \\& Bühlmann, 2010].\n\n- For each candidate value of $\\tau$, we fit the non-negative Lasso independently on each subsample, and follow the steps to create the RG.\n\n- For each edge $(\\lambda_i \\rightarrow \\lambda_j)$, we record its selection frequency, the fraction of subsamples in which it appears in the inferred DAG.\n\n- We then compute the edge stability score for a given $\\tau$ as the average selection frequency over all edges. This metric reflects the reproducibility of the graph structure.\n\n- Among all candidate $\\tau$ values, we select the one that maximizes stability, subject to a sparsity constraint on graph complexity (e.g., average node degree $\\leq 3$), to prevent overfitting and preserve statistical power [Shah \\& Samworth, 2013].\n\n\nWe will include these procedures in the appendix of the final version. This ensures that RG-PT can be applied in a fully automated and reproducible manner, without compromising its theoretical guarantees.\n\n\nReferences:\n\n- Rousseeuw, P. J. (1987). Silhouettes: a graphical aid to the interpretation and validation of cluster analysis. Journal of Computational and Applied Mathematics, 20, 53–65.\n\n- Kaufman, L., \\& Rousseeuw, P. J. (2009). Finding groups in data: an introduction to cluster analysis. John Wiley \\& Sons.\n\n- Efron, B., \\& Morris, C. (1973). Stein’s estimation rule and its competitors—an empirical Bayes approach. Journal of the American Statistical Association, 68(341), 117–130.\n\n- Meinshausen, N., \\& Bühlmann, P. (2010). Stability selection. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 72(4), 417–473.\n\n- Shah, R. D., \\& Samworth, R. J. (2013). Variable selection with error control: another look at stability selection. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 75(1), 55–80."}, {"review_id": "f1cgC0BjHa", "reviewer": "Reviewer_5Xxa", "summary": "This paper proposes RG-PT, a new framework for multi-objective hyperparameter selection with formal reliability guarantees. It builds upon LTT and PT by replacing the global linear ordering of candidate hyperparameters with a DAG, termed the reliability graph (RG). The RG is learned using BT ranking and non-negative Lasso regression based on prior information and held-out data. RG-PT applies DAGGER to perform FDR-controlling MHT. The method is validated across a wide range of tasks like prompt engineering, sequence-to-sequence translation, object detection, and telecommunications.", "questions": "See the weaknesses above, in particular regarding the interpretation of Lasso-based edges, selection of hyperparameter D, and missing baseline.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strength:\n1. The incorporation of a learned DAG structure to guide the hypothesis testing process represents a meaningful extension to existing LTT and PT frameworks. And the proposed framework maintains formal FDR guarantees under arbitrary DAG structures using DAGGER, building on established statistical results.\n2. The experiments cover diverse tasks in NLP and vision, including prompt engineering, sequence-to-sequence translation, object detection, and telecommunications. \n3. The paper is generally well-written and easy to follow.\n\nWeakness:\n1. In the depth assignment step of the RG learning, the BT model first generates scalar scores, and then these scores are clustered into D clusters to assign depth. Although an ablation study over D is done and shows that intermediate depths can improve power, this parameter is manually set and optimized via a grid sweep. This introduces an extra hyperparameter into the RG-PT itself. Given that the method is intended for hyperparameter selection and automation, it is somewhat contradictory that a crucial internal parameter still relies on user intervention and manual tuning. I think the paper could be improved if the authors could add more discussions of whether a data-driven strategy could determine D adaptively, how sensitive the results are to the choice of D across datasets, or whether the chosen D generalizes across data splits. \n\n2. From my understanding, the central assumption for constructing the DAG is that a hyperparameter $\\lambda$ can be predicted by $\\lambda'$'s risk vector implies $\\lambda'$ is more reliable. My major concern is that non-negative Lasso only captures linear correlation, not reliability dominance relationship. Is it possible that the linear predictability between two hyperparameters arises from shared noise,  task structure, or other facts, rather than indicating a true reliability dominance relationship? This assumption may break the intended semantics of the DAG: edges are meant to indicate reliability dominance, not statistical dependency. Is there any theoretical justification or empirical validation for this assumption?\n\n3. Missing baseline: RG-PT is only compared against LTT and PT. The paper omits any comparison against non-statistical methods which may offer better empirical trade-offs. Moreover, the paper does not compare against simpler but informative baselines that would help clarify where RG-PT's actual advantage comes from. Specifically, I would suggest the authors consider:\n- Methods that use weak or uncertain prior signals to sort the candidates, followed by standard PT, without learning a strict DAG structure, and can often be implemented at much lower computational cost. Comparing RG-PT to such baselines would help reveal whether the use of a learned DAG actually contributes to improved reliability discovery or is simply a surrogate for rough sorting.\n- PT variants with randomly shuffled orders, or uniform sampling from the candidates. Although such baselines do not provide statistical guarantees, they serve as important references. Without including them, it's difficult to assess whether RG-PT significantly improves naïve strategies.\n\n4. Given the reliance on the DAG structure for statistical validity and selection efficiency, the paper lacks evaluation or discussion of the structural sensitivity of the DAG. For example, can the authors compare different DAG construction pipelines or explore how perturbations to the DAG (e.g., pruning edges, changing cluster assignments) affect the final selected hyperparameters or FDR?\n\n5. (Minor) The font size in Figure 1(a) is small and difficult to distinguish.\n\n6. (Minor)It would be helpful to include a table of notations and definitions in the Appendix for easy reference.", "was_revised": true, "final_justification": "I maintain my original positive assessment of this paper.", "rebuttal": "We thank the reviewer for the thoughtful and detailed comments. We address each concern below and outline the specific clarifications and additions we will make in the revised manuscript.\n\n\n1. On the choice of graph depth $D$:\n\nAs noted by Reviewer T6Dk as well, we agree that this parameter plays a meaningful role in shaping the power of RG-PT. In our rebuttal to Reviewer T6Dk, we have now proposed and adopted a principled data-driven procedure for choosing $D$.\n\n\n\n2. On the semantics of edges inferred via non-negative Lasso:\n\nWe appreciate the reviewer’s insightful question. Indeed, our goal is to encode reliability dominance in the DAG edges, not merely predictive correlation. To clarify this, we now provide a structural motivation and explain why a linear model is a reasonable and practical choice.\n\nLet us denote the RG as $ G = (V, E) $, where $ V = \\\\{\\lambda_1, \\dots, \\lambda_{|\\Lambda_{\\text{OPT}}|}\\\\} $ is the set of candidate hyperparameters and $ E \\subseteq V \\times V $ is the set of directed edges. An edge $ (\\lambda_i \\rightarrow \\lambda_j) \\in E $ indicates that the reliability of $ \\lambda_j $ is dependent on that of $ \\lambda_i $, i.e., $ \\lambda_i $ is a ``parent\" of $ \\lambda_j $ in the DAG.\n\nWe assume that each configuration $ \\lambda_j \\in V $ has an unobserved binary reliability label $ R_j \\in \\\\{0, 1\\\\} $, where $ R_j = 1 $ indicates that $ \\lambda_j $ is reliable. We can model the relationship between these reliability variables using a stochastic Boolean structural equation\n$$\nR_j = f_j\\big(\\\\{R_i : (\\lambda_i \\rightarrow \\lambda_j) \\in E\\\\}\\big) + \\varepsilon_j,\n$$\nwhere $ f_j $ is an unknown binary-valued function, and $ \\varepsilon_j $ is a noise term, which is generally correlated with the function $f_j(\\cdot)$'s output. The function $f_j(\\cdot)$ may, for instance, encode that hyperparameter $\\lambda_j$ is likely to be reliable as long as a sufficiently large number of parent nodes are.  Moreover, the noise term captures  the possibility that a child configuration is unreliable ($R_j=0$) due to random effects not captured by function   $f_j(\\cdot)$.\n\nAlthough we do not observe the true reliability labels $ \\\\{R_j\\\\} $, we do observe scalar-valued performance vectors $ Y(\\lambda_j) \\in \\mathbb{R}^T $, such as per-task or per-example risks, which serve as continuous surrogates for reliability. We then approximate the above relationship using a linear regression model:\n$$\nY(\\lambda_j) = \\sum_{i \\neq j} \\beta_{ij} Y(\\lambda_i) + \\varepsilon_j,\n$$\nwith non-negativity constraints $ \\beta_{ij} \\geq 0 $, which preserve the monotonicity of influence. This leads to the non-negative Lasso objective we solve to infer the edge set $ E $.\n\nThis formulation does not claim that reliability is fundamentally linear in nature. Rather, it is a linearization of an unknown monotonic Boolean process, which may be justified as done in the literature on  influence modeling [Wainright \\& Chickering, 2006]. The sparsity induced by Lasso ensures that only the most predictive and interpretable relationships are retained. The non-negativity constraint encodes the domain prior that improving the parent should not reduce the reliability of the child. \n\nUltimately,  the FDR control of RG-PT remains valid regardless of any modeling choice made for the construction of the DAG. The DAG only affects the power of the procedure, not its validity.\n\n\n\n\n\n3. On missing baselines:\n\nWe thank the reviewer for highlighting the importance of benchmarking against simpler, heuristic, or non-statistical baselines. We note that this comparison has already been conducted in prior work: both LTT and PT evaluate their performance against non-risk-controlling baselines (e.g., \"$\\alpha$-constrained'' and \"$(\\alpha, \\delta)$-constrained'') and demonstrate improved power under statistical control. Since our method consistently outperforms LTT and PT across all benchmarks, we implicitly improve over these naive baselines as well. Nevertheless, we appreciate the reviewer’s suggestion and have added additional heuristic baselines in the object detection experiment.\n\n- Chain graph PT: We use our proposed graph learning method to construct a chain graph by setting $D = |\\Lambda_{\\text{OPT}}|$, and then apply PT using this learned chain as the ordering.\n\n- Randomly shuffled PT: We run PT on a chain graph where the hyperparameters have been randomly shuffled, effectively removing prior structural information.\n\n- Uniform Sampling: We uniformly sample $k$ hyperparameters (without replacement) and return them as the selected subset, mimicking random selection without statistical guarantees.\n\nThe median object recall threshold ($\\lambda_1$) achieved by chain graph PT, randomly shuffled PT, and uniform sampling was 0.61, 0.52, and 0.49, respectively, while for PT and RG-PT it was 0.56 and 0.68, respectively. The results show that randomly shuffled PT underperforms compared to standard PT, while chain graph PT offers a modest improvement over PT but remains weaker than RG-PT, highlighting the benefits of using a learned DAG structure. Finally, uniform sampling performs erratically, the selected hyperparameters span the full performance range (reflecting the lack of selection precision), and the method exhibits a high average FDR of 0.54, confirming its lack of statistical reliability.\n\n\n4. On structural sensitivity of the DAG:\n\nWe agree with the reviewer that the quality of the learned DAG influences the power of the procedure. That said, the FDR guarantee remains intact regardless of DAG structure, as DAGGER is valid under arbitrary acyclic graphs.\n\nOur approach assumes that there exists an unknown latent DAG that reflects true reliability dependencies. The goal of the learning procedure is to approximate this latent structure from available signals (BT scores, prior edges, and observed risk vectors). There are indeed many ways to learn such a DAG. We proposed a computationally tractable pipeline based on convex surrogates and empirical priors. Our ablations (e.g., on graph depth $D$) already explore aspects of structural variation. We will consider expanding the ablation scope (e.g., edge pruning and alternative clustering methods) in future work, though we note that defining ground truth reliability DAGs remains challenging in real-world datasets.\n\n5. Minor points:\nWe will enlarge the font size in Figure 1(a) and add a table of notations to the Appendix for reader convenience.\n\nWe thank the reviewer once again for the constructive feedback, and we hope that these additions and clarifications improve their view of the work and alleviate their concerns.\n\n\nReferences:\n\n- Wainwright, M. J., \\& Chickering, M. (2006). Estimating the \"wrong\" graphical model: Benefits in the computation-limited setting. Journal of Machine Learning Research, 7(9), 1829–1859."}, {"review_id": "APInJzzeoB", "reviewer": "Reviewer_LYAN", "summary": "This paper introduces Reliability Graph-based Pareto Testing (RG-PT), a novel framework for multi-objective hyperparameter selection that provides formal reliability guarantees. The authors propose modeling the hyperparameter space as a directed acyclic graph (DAG) called a reliability graph (RG), where nodes represent candidate hyperparameters and edges encode expected reliability relationships. This approach extends existing methods like Learn-Then-Test (LTT) and Pareto Testing (PT) by incorporating structured knowledge about the hyperparameter space, which enables more efficient exploration during multiple hypothesis testing.", "questions": "1.How might the approach be extended to continuous hyperparameter spaces? Could techniques like discretization or Gaussian process modeling be incorporated?\n2.The paper mentions that the choice of prior parameters doesn't affect the validity properties of RG-PT, but how do they impact its power? Is there a principled way to select these parameters?\n3.How does the computational cost of RG-PT compare to LTT and PT in practice, especially for large hyperparameter spaces?\n4.What is the sample complexity of RG-PT as a function of the DAG structure? Can you provide finite-sample guarantees rather than asymptotic ones, and prove that RG-PT achieves better sample efficiency than PT under specific conditions? This would require analyzing how the graph learning process affects the statistical efficiency of the subsequent hypothesis testing procedure, particularly when the number of hyperparameters is large relative to the sample size.\n5.The learned reliability graphs potentially contain valuable domain knowledge about hyperparameter relationships. Can you provide a deeper analysis of these structures across different domains to extract actionable insights? For instance, in the LLM prompt engineering application, what linguistic patterns or semantic relationships are captured in the graph structure that could inform better prompt design principles beyond the specific experimental settings?", "limitations": "The authors adequately address the limitations of their work, particularly noting the exclusive applicability to discrete hyperparameter spaces and the lack of theoretical results on power and sample efficiency. They also acknowledge potential future work directions, including optimizing the RG structure to maximize power and integration with sequential testing methods based on e-processes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths ：\n\n1.The paper provides formal guarantees for FDR control, extending the statistical rigor of existing hyperparameter selection methods to settings with structured knowledge.\n2.The authors evaluate their method across diverse application domains, demonstrating consistent improvements over state-of-the-art methods.\n3.The framework naturally applies to important real-world scenarios, particularly in prompt engineering for LLMs where trading off reliability and cost is crucial.\n4.The method elegantly incorporates prior information through the Bradley-Terry model, allowing domain experts to guide the hyperparameter selection process.\n\n\nWeaknesses：\n\n1.While the paper demonstrates that RG-PT works well, it's unclear how sensitive the performance is to the specific choice of DAG structure and depth parameter D. The ablation study in Appendix E.3 only partially addresses this concern.\n2.The paper mentions computational complexity in Appendix C, but doesn't provide empirical runtime comparisons with LTT and PT, which would be valuable for practical applications.\n3.Ironically, RG-PT itself introduces hyperparameters (D, np, τ) that need to be selected. The paper provides values in Table 1 but doesn't fully explain how these were chosen.\n4.The paper acknowledges the limitation to discrete hyperparameter spaces, but doesn't explore potential extensions to continuous spaces.\n5.While FDR control is proven, there is no theoretical analysis of the statistical power of RG-PT compared to PT and LTT, which would strengthen the theoretical contribution.", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their constructive and thoughtful feedback. Our responses to each of the concerns are provided below.\n\n1. Extension to continuous hyperparameter spaces:\n\n\nAs mentioned in lines 88–89 of the draft, a pre-selection step using methods such as LLM judges, Bayesian optimization, and Hyperband can be used to construct a discrete candidate set from a continuous hyperparameter space. For example, Bayesian optimization can efficiently explore the continuous domain using acquisition functions, and the most promising configurations can be selected to form the input to RG-PT [Snoek et al., 2012]. Similarly, Hyperband adaptively allocates resources to a large pool of randomly sampled configurations from a continuous domain, ultimately yielding a high-performing discrete subset [Li et al., 2017].\n\n\n2. On the impact and selection of prior parameters (e.g., $D$, $n_p$):\n\nWe agree that these parameters influence the power (though not the statistical validity) of RG-PT. We have provided a detailed and principled procedure for selecting both the DAG depth $ D $ and the prior weight $ n_p $ in our response to Reviewer T6Dk.\n\n3. Computational complexity:\n\nA detailed complexity analysis and comparison with LTT and PT is already provided in Appendix C of the paper. In practice, we found RG-PT to run efficiently even on large candidate sets (e.g., 100+ configurations), with runtimes under 5 minutes on a standard Apple MacBook Pro (M1 chip). We are happy to include empirical runtime comparisons in the revised appendix to complement the theoretical analysis in Appendix C.\n\n\n4. Sample efficiency and finite-sample behavior:\n\nAs demonstrated through various experiments in [Ramdas et al., 2017], DAG-structured testing yields higher power compared to linear testing when the graph encodes true logical dependencies among hypotheses. To illustrate this more rigorously, we provide next a straightforward  analytical comparison in the context of fixed sequence testing (FST), which can be seen as a special case of graph-based (DAGGER) testing with a chain graph. \n\nSuppose we wish to test $K$ hypotheses $\\mathcal{H}_1, \\ldots, \\mathcal{H}_K$. Let $m < K$ of these hypotheses correspond to non-null hypotheses, i.e.,  reliable  hyperparameters, and the remaining $K - m$ be true nulls, i.e.,  unreliable hyperparameters. Assume all $K$ $p$-values $\\\\{p_i\\\\}_{i=1}^{K}$, one per hypothesis $\\mathcal{H}_i$, are mutually independent and identically distributed. In particular, let $F_0(p) = p$ be the CDF of all p-values under the null, and $F_A(p)$ be the CDF under the alternative, where $F_A(\\alpha) = \\beta > \\alpha$ reflects the higher concentration of small $p$-values under the alternative hypothesis. Accordingly, for a testing threshold given by $\\alpha$ (i.e., $p_i \\leq \\alpha$ rejects the null),  the type-I probability (of incorrectly rejecting the null)  is $\\alpha$, while the probability of correctly detecting a non-null hypothesis is $\\beta$. \n\nAssume now a well-informed testing order whereby FST is applied in the order from $\\mathcal{H}\\_1$ to $\\mathcal{H}\\_K$. FST proceeds through these hypotheses,  deciding for the alternative (reliable hyperparameter) if $p_i \\leq \\alpha$. The  probability of correctly rejecting all $m$ non-null hypotheses is\ngiven by $\\prod_{i=1}^m \\mathbb{P}(p_i \\leq \\alpha) = \\beta^m$ .\n\nNow consider a misspecified testing order in  which $r \\geq 1$ null hypotheses  appear among the first $m + r$ positions. Let us pessimistically assume that these $r$ nulls occur before all $m$ non-nulls. The test may stop at the first null encountered if its $p$-value exceeds $\\alpha$, which happens with probability $1 - \\alpha$. Therefore, to reach and reject all $m$ non-nulls, we must reject all $r$ preceding nulls (each with probability $\\alpha$) and then reject all $m$ non-nulls (each with probability $\\beta$). Thus, the probability of correctly rejecting all $m$ non-nulls is \n$\\alpha^r \\cdot \\beta^m.$\n\nThis derivation shows that even one null placed early in the sequence reduces power by a factor of $\\alpha$. In the worst case, with multiple early nulls, the power degradation becomes exponential in $r$.\n\n\n\nSimilar calculations can be done by considering a more complex structure in the hypothesis space via a DAG. In this case, when the DAG is well specified,  the set of true null hypotheses (reliable hyperparameters) corresponds to a rooted subtree of the DAG. \n\n\n5. On extracting domain knowledge from the learned DAGs:\n\nWe agree with the reviewer that the learned reliability graphs may encode meaningful domain-specific structure, especially in rich settings such as prompt engineering. For instance, the DAG could implicitly capture relationships between prompt length, linguistic specificity, or formatting cues and their impact on performance. Extracting such patterns could indeed yield actionable design principles for future prompt development.\n\nThat said, our focus in this paper is on statistical reliability and multi-objective selection, rather than semantic analysis or pattern mining. A thorough analysis of graph interpretability across domains would require additional infrastructure (e.g., prompt annotation, linguistic taxonomy alignment), which is beyond the current scope. Nonetheless, we fully agree that this is a compelling direction for future work and will explicitly mention it in the conclusion section.\n\n\nWe thank the reviewer once again for the constructive feedback, and we hope that these additions and clarifications improve their view of the work and alleviate their concerns.\n\n\nReferences:\n\n- Snoek, J., Larochelle, H., \\& Adams, R. P. (2012). Practical Bayesian optimization of machine learning algorithms. Advances in Neural Information Processing Systems, 25, 2951–2959.\n\n- Li, L., Jamieson, K., DeSalvo, G., Rostamizadeh, A., \\& Talwalkar, A. (2017). Hyperband: A novel bandit-based approach to hyperparameter optimization. Journal of Machine Learning Research, 18(1), 6765–6816.\n\n- Ramdas, A., Barber, R. F., Wainwright, M. J., \\& Jordan, M. I. (2017). A unified treatment of multiple testing with prior knowledge using the p-filter. Annals of Statistics, 47(5), 2790–2821."}]

Author rebuttal:

Meta-review: This work introduces a new framework for multi-objective hyperparameter selection named Reliability Graph-based Pareto Testing (RG-PT). The selection method balances performance with cost. Relationships between hyperparameters are modeled as a Reliability Graph (RG), which is inferred from prior information and data using the Bradley-Terry model and non-negative Lasso. The method is. compared to competing baselines.

All reviewers voted for acceptance post rebuttal. The main reason being the following:
1/ The proposed method is sound and novel. The approach is evaluated on a diverse set of applications, including prompt engineering, language translation, and object detection. 
2/ The core idea of leveraging prior knowledge to build a Reliability Graph that reflects the structure of the hyperparameter space is compelling;
3/ The method exhibits strong statistical guarantees. By embedding the hypothesis testing procedure in the graph traversal, the approach proves global FDR control under arbitrary p-value dependence, offering firm, theory-backed reliability;
4/ The paper is generally well-written and easy to follow.
Decision: Accept (poster)