FedHPO-Bench: A Benchmark Suite for Federated Hyperparameter Optimization
Authors: Zhen WANG
Weirui Kuang
Ce Zhang
Bolin Ding
Yaliang Li
Venue: Submitted to ICLR 2023 (2023)
Keywords: federated learning
hyperparameter optimization

Abstract: Hyperparameter optimization (HPO) is crucial for machine learning algorithms to achieve satisfactory performance. Its research progress has been boosted by existing HPO benchmarks. Nonetheless, existing efforts in benchmarking all focus on HPO for traditional centralized learning while ignoring federated learning (FL), a promising paradigm for collaboratively learning models from dispersed data. In this paper, we first identify some uniqueness of HPO for FL algorithms from various aspects. Due to this uniqueness, existing HPO benchmarks no longer satisfy the need to compare HPO methods in the FL setting. To facilitate the research of HPO in the FL setting, we propose and implement a benchmark suite FedHPO-Bench that incorporates comprehensive FedHPO problems, enables flexible customization of the function evaluations, and eases continuing extensions. We also conduct extensive experiments based on FedHPO-Bench to provide the community with more insights into FedHPO. We open-sourced FedHPO-Bench at https://github.com/FedHPO-Bench/FedHPO-Bench-ICLR23.

Under review as a conference paper at ICLR 2023
FEDHPO-BENCH: A BENCHMARK SUITE FOR FEDER- ATED HYPERPARAMETER OPTIMIZATION
Anonymous moonymous authors Paper under double-blind review
ABSTRACT
Hyperparameter optimization (HPO) is erucial for machine learning algorithms to achieve satisfactory performance. Its research progress has been boosted by existing HPO benchmarks. Nonetheless, existing efforts in benchmarking all focus on HPO for traditional centralized learning while ignoring federated learning (FL), a promising paradigm for collaboratively learning models from dispersed data. In this paper, we first identify some uniqueness of HPO for FL algorithms from various aspects. Due to this uniqueness, existing HPO benchmarks no longer satisfy the need to compare HPO methods in the FL setting. To facilitate the research of HPO in the FL. setting, we propose and implement a benchmark suite FEDHPO-BEE BENCH that incorporates comprehensive FedHPO problems, enables flexible customization of the function evaluations, and eases continuing extensions. We also conduct extensive experiments based FEDHPO FEDHPO-BENCH provide the community with more insights into FedHPO. We open-sourced FEDHPO-BI BENCH at http://ghhhhcccooo//////////eeed
INTRODUCTION
Most machine learning (ML) algorithms expose many design choices, which can drastically impact the ultimate performance. Hyperparameter optimization (HPO) (Feurer & Hutter 2019) aims at making the right choices without human intervention. To this end, HPO methods usually attempt to solve min x fA), where each A₁ corresponds to the candidate choices of specific heprrparameter, e.g., taking the learning rate from A₁ [0.01, 1.0) and the batch size from A2 (16,32, 32, 64). For each specified X, f(A) the output result ge. validation loss) of executing the considered algorithm configured by X. A solution X found for such problem is expected to make the considered algorithm lead to superior generalization performance. Research in this line has been facilitated by HPO benchmarks (Gijsbers
al. 2019 Eggensperger et al.) 2021 Fineda-Arange et al. 2021), ) which prepare many HPO problems so that different HPO methods can be effortlessly compared, encouraging fair, reliable, and reproducible empirical studies.
However, existing HPO benchmarks all focus on traditional learning paradigms, where the functions to be optimized correspond to centralized learning tasks. Federated learning (FL.) (McMahan et al. paradigm for collaboratively learning model from
2017 Li et al., 2020a), as a distributed data, has not been considered. Actually, along with the increasing privacy concerns from the whole society, FL has been gaining more attention from academia and industry. Meanwhile, HPO for FL algorithms (denoted by FedHPO from now on) is identified as critical and promising open problem in FL (Kairouz et al. 2019).
In this paper, we first elaborate on several differences between FedHPO and traditional HPO (see Section 2.2). which essentially come from FL.ss distributed nature and the heterogeneity among FL's participants. These differences make existing HPO benchmarks inappropriate for studying FedHPO and, in particular, unusable for comparing FedHPO methods. Consequently, several recently proposed FedHPO methods (Zhou et al.) 2202 Dai et al., 22000] Khodak et al./ [021] Zhang et al. [202]] Guo et al. 2022) evaluated on respective problems and have not been uniformly implemented in one FL framework and well benchmarked.
Motivated by FedHPO's uniqueness and the successes of existing HPO benchmarks, propose and implement FEDHPO- -BENCH, a dedicated benchmark suite, to facilitate the research and application of FedHPO. FEDHPO- is featured by satisfying the desiderata as follows:

---

Under review as a conference paper at ICLR 2023
Comprehensiveness. FL tasks diverse in terms of domain, model architecture, heterogeneity among participants, etc. The objective functions of their corresponding FedHPO problems are thus likely to be diverse. Hence, FEDHPO-BENCH provides comprehensive collection of FedHPO problems for drawing an unbiased conclusion from comparisons of HPO methods.
Flexibility. Users may have different levels of privacy and fairness concern,, which may correspond to different multi-objective optimization problems. Meanwhile, the execution time for function evaluation depends the system condition. Thus, FEDHPO-BENCI allows users to flexibly tailor the FedHPO problems to their privacy protection needs, fairness demands, and system conditions.
Extensibilityy As developing field, new FedHPO problems and novel FedHPO methods constantly emerge, and FL's best practice continuously evolves. Thus, we build HEDHPO-BENNHE on a popular FL framework, FederatedScope (FS) (Xie et al., 2022). and make more of benchmarking tool that can effortlessly incorporate novel ingredients.
To our knowledge, FEDHPO-BENCE the first FedHIPO benchmark. We conduct extensive empirical studies with it to validate its usability and attain more insights into FedHPO.
2 BACKGROUND AND MOTIVATIONS
We first give a brief introduction to the settings of HPO and its related benchmarks. Then we present and explain the uniqueness of FedHPO to show the demand for dedicated FedHPO benchmarks. Due to the space limitation, more discussions about related works are deferred to Appendix B
2.1 PROBLEM SETTINGS AND EXISTING BENCHMARKS
In the literature FFuurer& & Hutter 2019), HPO is often formulated as solving min xxx f(A). where each A& corresponds to candidate choices of a specific hyperparameter, and their Cartesian product (denoted by x) constitute the search space. In practice, such A₃ is often bounded and can be continuous (e.g., an interval real numbers) or discrete (e.g., set of categories/intgges)) Each function evaluation specified hyperparameter configuration À means to execute the corresponding algorithm accordingly and return the value of considered metric (e.g., validation loss) as the result f()). HPO methods generally solve such problem with series of function evaluations. As full- fidelity function evaluation is extremely costly, fullti-iieelty methods exploit low-fidelity function evaluation, e.g., training for fewer epochs (Swersky et al. 2014 Domhan et al. (015) or subset of data (Klein et al., [017] [Petrak] [2000] (Swersky et al.] 2013)) to approximate the exact result. Thus, Ak, 6 E B1
it would be convenient treat f as f),,b), A₁ X corresponds to the possible choices of a specific fidelity dimension.
Bt, where each B
HPO benchmarks (Gijsbers al., 2019 Eggensperger al., 202 Fineda-Arange al. 202 have
prepared many HPO problems, i.c., various kinds of objective functions, for comparing HPO methods. To evaluate these functions, HPO benchmarks, e.g., HPOBench (Eggensperger al. 2021. D. often
provide three modes: (1) "Raw" means truly executing the corresponding algorithm: (2) "Tabular" means querying a lookup table, where each entry corresponds to a specific [A b); (3) "Surrogate" means querying surrogate model that might be trained on the tabular data.
2.2 UNIQUENESS OF FEDERATED HYPERPARAMETER OPTIMIZATION
Generally, traditional HPO methods are applicable to FedHPO problems where, in each trial, the value f(X, 6) is evaluated, that is to say, an accordingly configured FL training course is conducted, iilustrates. Conceptually, there are N clients, each of which has as the dashed black box in Figure T
its specific data, and server coordinates them leam model collaboratively by an FL algorithm such as FedAvg (McMahan et al., 2017) and FedOpt (Asad et al. 2020).. Such FL algorithms are iterative. In the t-th round, the server broadcasts the global model (()) to sampled clients; then, these clients make local updates and send the updates back; finally, the server aggregates the updates to produce ((++1), After executing the FL algorithm configured by À for several such rounds, e.g.. #round= T according to the specified fidelity b, the performance, e.g., best validation loss ever achieved during this FL course, is returned as f(),b) b).
Desppte the various scenarios in literature, restrict our discussion about FedHPO to one of the most general FL. scenarios that have been adopted in existing FedHPO works (Khodak al. 2021 Zhang al. 202T
2

---

Under review as a conference paper at ICLR 2023
Standard FL training
FedHPO method concurrently explores client side search during the
g
Server
aeer
gleey
Server
aggr
o
2857
800
h
6m0
Clienty
Client,
Clienty
Traditional HPO method
Client,
Client,
Clients
(r. (199,8)
Local update
Evaluste
(49)
a (4bboor...b
2= (0))
Figure 1: Solving a FedHPO problem by a traditional HPO method solely or as the wrapper of FcdHPO method: the (-th round of a trial is shown in the dashed black box and dashed blue box, respectively. The faded clients not sampled in that round; "ager" denotes certain aggregation operation; here FedEE is considered the wrapped FedHPO method, which learns policy ı to determine X each (-) is regarded as client- approximation of f(x(),
As a distributed ML scenario, the procedure of each FL training round consists of two subroutines local updates and aggregation. Thus, À can be divided into server-side and client-side hyperparameters according to which subroutine each hyperparameter influences. Denoting it À (X(), X()), the original optimization problem can be straightforwardy restated as its bi-level counterpart mingo [()(), X(e)*), ), s.t., ( = minxo f(((), X(9).. With such a point of view, let's see how some FedHPO methods leverage the distributed nature of FL to solve the lower-level sub-problen efficiently and approximately.
Concurrent exploration. In each FL training course, all sampled clients have executed local updates configured by client-side hyperparameters for one or more rounds. Given specific A if we regard clients as replicas of the black-box box function fA((), -) or, at least, similar such functions whose evalu- ation results help fitting f(x(),.) -), it is natural to try out client-side hyperparameter configurations client-wisely so that we evaluate (()((), (()) for more than one ( in each FL training course. We summarize this idea as concurrent exploration, which recently proposed FedHPO methods, such as FedEx (Khodak et al.) 202T) and FTS (Dai et al. (020), have instantiated. Specifically, FedEx incorporates concurrent exploration with the weight-sharing trick (Liu et al. 2019) to achieve one- shot learning (i.e., by one FL training course) of policy for determining the optimal lower-level response (+)) = min se) f(A(s), ( Hence, we could solve the bi-level optimization problem by letting traditional HPO method, as a wrapper, choose ( ( and replace each standard FL training course with a FedHPO method, as the dashed blue box in Figure T shows.
As FedHPO methods are fused with the training course, existing HPO benchmarks become unusable for comparing them. Moreover, since this fusion makes the implementations of such FedHPO methods tightly coupled with that of FL training courses, and no existing FL framework has incorporated such FedHPO methods, researchers cannot compare them in a unified way. How we incorporate several recent FedHPO methods into FEDHPO- -BENIIII and make it extensible is discussed in Section 3.3 and whether concurrent exploration is useful is empirically answered in Section 4.2
Personalization. The non- IIDness among clients' data is likely to make them have different optimal configurations (Koskela & Honkela 2020). where making decisions by the same policy, such as FedEx, would become unsatisfactory. This phenomenon tends to become severer when federated
hetero-task learning (Yao et al. 2022) is considered. Trivially solving the personalized FedHPO
,fft....... problem min (00, ......$ [[(1s), X ...... (e) where X denotes the client-side hyperparameter configuration of the i-th client, is intractable, as the search space exponentially increases with N.
To promote studying personalized FedHPO, we provide personalized FedHPO problem featured
by heterogeneous tasks among the clients and the corresponding tabular benchmark (see H.1 for detailed description).
In practice, the traditional HPO method also determines the FedHPO algorithm's 's hyperparameters and subset of the original client-side search space be explored. We omit these to keep brevity.
3

---

Under review as a conference paper at ICLR 2023
Multi-objective optimization. Despite the model's performance. researchers often concerned about other issues, such as privacy protection and faimess. Regarding privacy, the FL algorithm is often incorporated with privacy protection techniques such as differential privacy (DP) (Kairouz et al. 2019), where the DP algorithm also exposes its hyperparameters. Intuitively, a low privacy budget specified for such algorithms indicates lower risk of privacy leakage yet more significant degrada- tion of the model's performance. As for fairness, namely, the uniformity of the model's performances across the clients, more and more FL algorithms have taken it into account (Li al. 2021a Wang et al. 2021b), which contains some hyperparameter(s) concerning fairesss. Therefore, researchers may be interested in searching for a hyperparameter configuration that guarantees an acceptable privacy leakage risk (e.g., measured by Rényi-DP (Mironov 2017) and fairness measurement (e.g..
the standard deviation of client-wise performances) while optimizing the model's performance.
Thus, FedHPO benchmark is expected to expose a vector-valued objective function instead of a scalar-valued one, where the entries of a returned vector could be the quantitative measures corresponding to performance, privacy leakage risk, fairness, etc. Then, researchers are allowed to
study multi-objective HPO (Hernández et al. 2021 Deb et al., 2002 Abdolshah et al. 2019).
Runtime estimation and system-dependent trade-offs. For the research purpose, an FL train- ing course is usually simulated in a single computer rather than executed in a distributed sys- tem. As result, simply recording the consumed time is meaningless for measuring the cost of a function evaluation in studying FedHPO. Meanwhile, FL's distributed nature introduces a new fidelity ciimension-clieee celientssampl__aeeeeeee____ sample_ratt which determines the fraction of clients sampled in each round. When considering a client_sample pamtleerateeaaeeeeeee less than that of a full-fidelity function evaluation, each round of the FL course would take less time because there is less likely be a straggler. However, client_ slsppssmmppaaaaee tratee correlates with another fidelity limension-#round client_
where a lower client_sample sample rate often leads to federated aggregation with larger variance, which is believed need more rounds for convergence. How we should balance these two fidelity dimensions to achieve more economical eecuracy-efficienne trade-offs strongly depends on the system condition, e.g., choosing large #round but small tiientssmpleeratte r_rtee when the straggler issue is severe.
As existing HPO benchmarks focus on centralized learning tasks, they overlook a runtime estimation functionality for studying FedHPO. In Section 3.2 we present FEDHPO-B -ENCI''s system model, with which we conduct an empirical study to show the effect of balancing client_ pecet__saple_raee rate and #round in Appendix
Due to the uniqueness mentioned above, existing HPO benchmarks are inappropriate for studying FedHPO. FedHPO calls for dedicated benchmarks that incorporate objective functions corresponding to FL algorithms and respecting realistic FL settings.
OUR PROPOSED BENCHMARK SUITE: FEDHPO-BENCH
We present overview of FEDHPO- BEDHPO-BENCH in Figure Conceptually. FEDHPO SEDHPO-BENCH en- capsulates function evaluation and provides a unified interface for HPO methods to interplay with it. Following the design of HPOBench (Eggensperger et al.) 2021), function evaluations can be conducted in either of the three modes: raw, tabular, or surrogate. For the raw mode, we chose to build FEDHPO-BENCE upon the well-known FL platform FederatedScope (FS) (Xie al. 2022). which has provided its docker images so that we can containerize FEDHPO-BENCE effortlessly by executing each FL algorithm in FS docker container. To generate the lookup table for tabular mode, we truly execute the corresponding FL algorithms with the grids of search space as their configurations. These lookup tables are adopted as training data for the surrogate models, which are expected to approximate the objective functions (more details about this approximation are discussed in Appendix [..4.2.. It's important to note that the distributed nature of FL makes very expensive to run FL course, so, in FedHPO, the tabular and surrogate modes are much in demand to meet the efficiency requirement. For the convenience of users, we keep FEDHPO HEDHPO-BENCHT 's interface basically the same HPOBench's s. Meanwhile, instantiation of a benchmark. We defer the discussion of the relationship between FEDHPO-BENCE and HPOBench to Appendix B.I
expose extra arguments for users to customize the
In this section, we elaborate on three highlights of FEDHPO FEDHPO-BENCH. In addition to the off-the-shelf Fl.-related components that FS already provides, we contribute new datasets, models, and algorithms
4

---

Under review as a conference paper at ICLR 2023
Interface design and code example
Function evaluation
Off-line prepracess
S(A
FedH
Container
System stadiatics
8,
System model
Tabular
Surrogate
Raw
X, as config
Exeo.ts FodAigo or FocHPO
A, ( 1}
HPO
Beselmteriace
f(AA, b)
FederatesScopp
HPO
Collect
Surrogate moce
8)
Datasel
Model
FedAlgo
A, input
(A,
Froe Fedrpabh benchsorks isport Tebolarfeeccharkk
Training
bencheurk: Tohal arBenchears " "Fesnist", "
f(, i)
b).8 111
get hyperporaneters space
configgspuce
gwtggeett_oooooeeeeee fourrttttoonn (paceC)
Performance e.g. loss
Fideliiy space benchsork.gge. Fidelity space()
get Fidelity space
Aa
Ax:
8
((A,A)
Container
get
BenchnorkkConff
sorple configuration()
Ass as confii
Execute FedMlge.
Fidel Fidelity.. tyy.-tyceee.....o sumplee le.onniguraatiiooo
folimess_ Fancing foimess_ reg. fxd seede1)
Figure 2: Overview of FEDHPO-BENCH
to FS to prepare comprehensive collection of FedHPO problems (see Section 3.1) Moreover, users are allowed to flexibly tailor these problems their specific cases (see Section 3.2) Also, FS's
event-driven framework allows us to easily extend COOHPO--BNCHEE by incorporating more FedHPO problems and methods, which is valuable for this nascent research direction (see Section 3.3
3.1 COMPREHENSIVENNSS
There is no universally best HPO method (Gijsbers et al. 2019). Therefore, for the purpose of fairly comparing HPO methods, is necessary to compare them on a variety of HPO problems that correspond to diverse objective functions and thus can comprehensively assess their performances.
To satisfy this need, leverage FS to prepare various FL tasks, where their considered datasets and model architectures are quite different. Specifically, the data can be images, sentences, graphs, or tabular data. Some datasets are provided by existing FL benchmarks, which are readily distributed and thus conform to the FL setting. Some are centralized initially, which we partition by FS's splitters to construct their FI. version with various kinds of Non-IIDness among clients. All these datasets are publicly available and can be downloaded and preprocessed by our prepared scripts. More details of these datasets can be found in Appendix D Then the corresponding suitable neural network model is applied to handle each dataset, involving fully-connected networks, convolutional networks, the latest attention based model, etc. It is worth noticing that our prepared FL tasks cover both cross-silo and cross-device scenarios. In drosce scenario, there are lot more clients and much lower dliet_samplle__ rate than in cross-silo scenario.
For each such FL. task, we basically employ FL algorithms, FedAvg and FedOpt, to learn the model, respectively. Then the FedHPO problem is defined as optimizing the design choices of the FL algorithm on each specific FL task. So, without further explanation, we use the triple <dataset, model, algorithm> to index particular benchmark in the remainder of this paper. We summarize our currently provided FedHPO problems in Table ! and more details can be found in Appendix H For each problem, #round and client saatttssamp____e_ pate are adopted as the fidelity dimensions.
We study the empirical cumulative distribution function (ECDF) for each model type the cross-silo benchmarks. Specifically, in creating the lookup table for tabular mode, we have conducted function evaluations for the hyperparameter configurations located on a very dense grid over the search space, resulting in finite set {(A, f())}} for each benchmark. Then we normalize the performances (i.e.. f()) and show their ECDF Figure 3 where these curves exhibit different shapes. For example, the amounts of top-tier configurations for GNN on PubMed are remarkably less than on other graph datasets, which might imply a less smoothed landscape and difficulty in seeking the optimal configuration. As the varying shapes of ECDF curves have been regarded of the diversity of benchmarks (Eggensperger et al., (00T)) we can conclude from Figure that FRDHPO- -E-HH enables evaluating HPO methods comprehensively.
indicator
5

---

Under review as a conference paper at ICLR 2023
Table 1: Summary of benchmarks in current FEDHPO-BENCE BENCH: MF refers to matrix factorization. Rec. and Algo. are short recommendation and algorithm, respectively. #Cont. and #Disc. denote the number of hyperparameter dimensions corresponding to continuous and discrete candidate choices, respectively. The unit of the budget is either day (d) or second (s).
Scenario
Model
#Dataset
Domain
#Client
#Algo.
#Cont.
#Disc.
Budget
CNN
2
CV
200
2
4
2
20d
BERT
NLP
2
2
20d
Cross-Silo
GNN
Graph
2
4
1
1d
GNN
Hetero
1
1
ld
LR
Tabular
2
1
21,600s
MLP
7
Tabular
5
2
4
3
43,200s
MF
Rec.
480,189
2
Cross-Device
LR
NLP
~3300
2
1
1d
1.9
1.3
10
9.2
IRAEIT
3.
03
86
60
60
(a) CNN
(b) BERT
(c) GNN
(d) LR
(c) MLP
Figure 3: Empirical Cumulative Distribution Functions: The normalized regret is calculated for all evaluated
configurations of the respective model on the respective FL task with FedAvg.
3.2 FLEXIBILITY
We allow users to instantiate each benchmark with arguments other than the <dataset, model, algo- rithm> triple. So, both the underlying objective function and the amount of time it takes to calculate its value can be customized according the cases specified by the users.
Objective function. FEDHPO -BECH provides the basic interface to support multi-objective optimization concerning arbitrary specified performance metrics and privacy and fairness-relatee metrics. For privacy protection, we employ representative FL+DP algorithm NbAFL (Wei et al.) 2020) provided by FS, where users can specify any valid value for the privacy budget. As for fairness, FS has provided many personalized FL algorithms and fairness-awand aggregations, and FEDHPO- BENCH can record client-wise performances. In designing the interface of FEDHPO-BENC allow users to specify their preferred measurements of privacy leakage risk and fairness. Then the execution of an FL algorithm can be regarded as evaluating a vector-valued function rather than scalar-valued one. By default, FEDHPO- -ENCC transforms the vector result into scalar one by treating privacy and fairness-related values as soft constraints to penalize.
we
System model. In addition to customizing the objective function, it is also very helpful to customize the exccution time of function evaluation because the exccution time of the same FL course can vary lot when deployed in environments with different system conditions. Many existing HPO benchmarks record the execution time of the training courses, which cannot be adapted regarding users' system conditions. Despite recorded execution time, provide system model to estimate the time consumed by evaluating f(x, b) in realistic scenarios, which is configurable so that users with different system conditions can calibrate the model their cases (Mohr al.) 202T)) D. Based on the analysis of such a system model and a basic instance (Wang et 202) la), we propose and implement our system model, where the execution time T(f. (ff, X, b) for each round in evaluating f(), 6) is the summation of time consumed by computation and communication. Roughly, the time for communication is the summation of the time for downloading and uploading transferred information and the latency for establishing connections. The time for computation is the summation of the time for the server's aggregation step and that for the straggler client's local updates. Our system model exposes several adjustable parameters, for which we provide default choices based on the records collected from creating the tabular benchmarks. Meanwhile, users are allowed to specify these parameters according their scenarios other system statistic providers, e.g., estimating the latency of stragglers by sampling from FedScale (Lai et al. 2022). We defer the details about our system model to Appendix

---

Under review as a conference paper at ICLR 2023
3.3 EXTENSIBILITY
As FedHPO is springing up, we must reduce the effort of introducing more FedHPO problems and novel FedHPO methods -BNCHH.
Recall that a FedHPO problem is characterized by the <dataset, model, algorithm> triple. With FS, we can apply the off-the-shelff data splitters transform an arbitrary centralized dataset into an FL dataset, reuse any open-sourced model implementation by registering it in FS, and develop novel FL algorithm via plugging the hook function that expresses its unique step(s).
Traditional HPO methods are decoupled from the procedure of function evaluation, with a concep- tually standard interface for interaction (see Figure and Figure (2. Thus, any new method this line can readily interplay with FEDHPO BIPHHOO-BENCH.. However, FedHPO methods, such as FTS and FedEx, are fused with the FL training course to make concurrent exploration, as the dashed blue box in Figure I and the red color "FedHPO" module in Figure shows,. Thus, we need to implement such methods in FS if we want to benchmark them on FEDHPO-BENT BENCH.
shows.
At a high level, such FedHPO methods essentially aim to learn policy collaboratively, along with the original FL course. As FS is featured by its event-driven programming paradigm, a standard FL course is modularized into wentthhndder pairs that express all the subroutines. Benefiting from this event-driven paradigm, all we need to develop are augmenting the messages exchanged by FL participants (i.e., re-defining events) and plugging those policy learning-related operations into the event handlers. As a result, we have implemented FTS, FedEx, and a personalized FedEx in FS, where their differences mainly lie in just the definition and implementation of those plug-in operations. We defer more implementation details to Appendix
4 EXPERIMENTS
We conduct extensive empirical studies with our proposed FEDHPO-BENCH Basically, we ex- emplify the use of FEDHPO-BENNC in comparing HPO methods, which, in the meantime, can somewhat validate the correctness of SEDHPO--BENCHE Moreover, we aim to gain more insights into FedHPO, answering two research questions: (RQ1) How do traditional HPO methods behave in solving FedHPO problems? (RQ2) Do recently proposed FedHPO methods that exploit "concurrent exploration" (see Section 2.2) significantly improve traditional methods? We conduct empirical studies in Section and Section (4.2ion442 answer RQ1 and RQ2, respectively. All scripts concerning the studies here have been committed to FEDHPO-BENCE so that the community can quickly reproduce our established benchmarks.
4.1 STUDIES ABOUT APPLYING TRADITIONAL HPO METHODS IN THE FL SETTING
To answer RQI, we largely follow the experiment conducted in HPOBench (Eggensperger et al. 2021 but focus on the FedHPO problems HEDHPO-BENCEE provided.
Protocol. We employ up to ten optimizers (i.c., HPO methods) from widely adopted libraries (sce Table σ for more details). For black-box optimizers (BBO), consider random search (RS), the evolutionary search approach of differential evolution (DE (Storn & Price 1997, Awad et al. 2020 ). and bayesian optimization with: GP model (BOGr), random forest model (BO (Hutter et al.) 201 and a kernel density estimator (BOxDE (Falkner et al.) 2018b)), ) respectively. For multi- fidelity optimizers (MF), we consider Hyperband (HB (Li et al. 2017)) its model-based extensions with KDE-based model (BOHB (Falkner et al.) 2018a)), and differential evolution (DEHB (Awad) al. 2021)). and Optuna's 's implementations
TPE with median stopping (TPEMD) and TPE with Hyperband (TPE (Akiba et al. (211)). We apply these optimizers to solve the cross-silo FedHPO problems summarized in Table where the time budget is relaxed for these traditional HPO methods to satisfy multiple full-fidelity function evaluations rather than a one-shot setting. For the sake of efficiency, we conduct this experiment with the tabular mode of FEDHPO-BENCIL and consider #round as the fidelity dimension for HPO methods to control (while keeping ciienissample rrtplee__atee1.)). To compare the optimizers uniformly and fairly, we repeat each setting five times in the same runtime environment but with different random seeds. The best-seen validation loss is monitored for each optimizer (for multi fidei-fii optimizers, higher fidelity results are preferred over lower ones). We sort the optimizers by their best-seen results and compare their mean ranks all the considered
7

---

Under review as a conference paper at ICLR 2023
FedHPOO problems. Following HPOBench, sign tests judge (1) whether advanced methods outperform their baselines and (2) whether mulli-fideliin methods outperform their angle-fidelitg counterparts. We refer our readers Appendix for more details.
Mean rank
Mean rank
1c.4
1c.3 1c.2 le.1
1eo4
1c.3
1e.2 le1
1664
1c.3
16:1
Fraction of budget
Fraction budget
Fraction of budget
(a) BBO
(b) MF
(c) All
RS
BOGP
BORF DEHB
BOXDE TPEND
DE
BOHB
TРEμB
Figure 4: Mean rank over time on all FedHIOO problems (with FedAvg).
Results and Analysis. We show the overall results in Figure 4 and we defer detailed results to Appendix Overall, their eventual mean ranks do deviate remarkably, For BBO, the performances of optimizers are close at the beginning but become more distinguishable along with their exploration. Ultimately, BPGp has successfully sought better configurations than other optimizers. In contrast to BBO, MF optimizers perform pretty differently in the early stage, which might be rooted in the vast variance of low-fidelity function evaluations. Eventually, HB and BOHB become superior to others while achieving very close mean rank. We consider optimizers' final performances on all the considered FedHPO problems, where, for each pair of optimizers, one may win, tie, or lose against the other. Then in Table and 3 (1) Comparing these advanced optimizers with their baselines, only B0Gr, BORF, and DE win more than half of the problems but have significant improvement, which is inconsistent with the non-FL setting. It is worth noting that a similar phenomenon can also be observed for HPO problems in general (Pushak & Hoos 2022). show any ad advantage in exploiting experience, which differs from non-FL cases. We presume the reason lies in the distribution configurations' performances (see Figure 3 From Table 3 we draw part of the answer to RQI as that MF optimizers always outperform their corresponding single-fidelity version, which is consistent with the non-FL. settings.
conduct sign tests compare pairs of optimizers, where results presented
(2) Meanwhile, MF optimizers
Table 2: P-value of a sign test for the hypothesis these advanced methods surpass the baselines.
B0GP 0.0637
B0нr
DE B0xDE
p-value agains RS win-tie-loss
0.2161
0.1649
0.7561
13 13/0/7
12/0/8
7/0/13
11/0/9
ВOHB
DEHB
TPEMD
TPEm
p-value against HB
0.4523
0.9854
0.2942
0.2454
win-tic-loss
7/0/13
9/0/11
9/0/11
9/0/11
Table 3: P-value of sign test for the hypothesis MFF methods surpass corresponding BBO methods.
HB vs.
DEHB vs. DE
BOHB vs. BOKDE
p-value 0.1139 win-tie-loss
0.2942 13/0/7
0.0106 16/0/4
13/0/7
4.2 STUDIES ABOUT CONCURRENT EXPLORATION
To answer RQ2, we select the superior optimizers from Section 4.1 to compare with FedEx (Khodak
al. 2021), As mentioned in Section 2.2 FI. allows HPO methods take advantage of concurren
8

---

Under review as a conference paper at ICLR 2023
exploration, which somewhat compensates for the number of function evaluations. We are interested in methods designed regarding these characteristics of FedHPO and design this experiment to see how much concurrent exploration contributes.
Protocol. We consider the FedHPO problem <FEMNIST, CNN, FedAvg), i.c. FedAvg is applied to learn a 2-layer CNN on FEMNIST. As full-fidelity function evaluation consumes 500 rounds on this dataset, we specify RS, BP.p, R0RF. BOXDE: HB, and BOHB to limit their total budget to 2,500 (i.e., times budget of full-fidelity evaluation) in terms of #round. Precisely, each BBO method consists of 50 trials, each of which runs for 50 rounds. For MF optimizers, we set the 1) of Successive Halving Algorithm (SHA) (Jamieson & Talwalkar 2016) to 3, the minimal budget rounds, and the max budget to 81 rounds. Then we adopt these optimizers and FedEx wrapped by them (X+ FedFxx to optimize the design choices FedAvg, respectively. The wrapper is responsible for determining the arms for each execution of FedEx. We consider validation loss the metric of interest, and function evaluations are conducted in the raw mode. We repeat each method three times and report the averaged best-seen value at the end of each trial. Meanwhile, for each considered method, we entirely run the FL course with the optimal configuration it seeks. Their averaged test accuracies compared.
4.0
RS
3.5
80, RF
Methods
W/O FedEx
W/ FedEx
3.0
80,
HB
RS
79.93 + 2.45
82.03 # 2.08
2.5
B0Gr
82.18 0.94
83.20 1.24
RS+FedEx
2.0
80, GP+redEx
BORF
81.86 1.10
82.20 # 0.54
1.5
8D_ FedEx
BOKDE
81.34
82.11 0.46
80, RodEx
1.0
HB+FedEX
HB
80.26 2.02
82.47 + 0.04
0.5
ВОHВ FedEx
BOHB
79.59 2.09
84.02 + 0.50
1e-2
le-1 Fraction of budget
Compare the searched configura- tions: Mcan test accuracy (%) standard de- viation. Underline indicates improvements.
Table 5:
Figure 5: Mean validation cross- cross-entropy loss over time,
and and Table For FedAvg, the best-seen mean validation losses of all wrapped FedEx decrease slower than their corresponding wrapper. significantly better than
Results and Analysis. We present the results in Figure
However, their searched configurations' generalization performances their wrappers, which strongly confirms the effectiveness of concurrent exploration. Thus, we have clear answer RQ2: concurrent exploration methods significantly improve traditional methods.
5 CONCLUSION AND FUTURE WORK
In this paper, we first identify the uniqueness of FedHPO, which ascribe the distributed nature of FL and its heterogeneous clients. This uniqueness prevents FedHPO research from leveraging existing HPO benchmarks, which has led inconsistent comparisons between some recently proposed methods. Hence, we suggest and implement a comprehensive, flexible, and extensible benchmark suite, FEDHPO-BENCE
We further conduct extensive HPO experiments on FEDHPO-BEET -ENCHH, validating its correctness and applicability to comparing traditional and FedHPO methods. We have open-sourced FFDHPO- BENCH with an Apache-2.. 2.0 license and will actively maintain in the future (Maintenance of SEEDPOO-BENCE is discussed in Appendix A). We believe (OOHPO--BNNCE can serve as the stepping stone to developing reproducible FedHPO works.
In our next step, tasks other than federated supervised learning will be incorporated. At the same time, we aim to extend FEDHPO- -BENCH to include different FL settings, e.g., HPO for vertical FL (Zhou et al. 2021). Another issue the current version has not touched is the risk of privacy leakage caused by HPO methods (Koskela & Honkela, 2020), which we should provide related metrics and testheds in the future.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Majid Abdolshah, Alistair Shilton, Santu Rana, S. Gupta, and Svetha Venkatesh. Multi-objecttve bayesian optimisation with preferences over objectives. ArXiv, abs/1902.04228, 2019.
Steven Adriaensen, André Biedenkapp, Gresa Shala, Noor H. Awad, Theresa Eimer, Marius Thomas Lindauer, and Frank Hutter. Automated dynamic algorithm configuration. ArXiv, abs/2205.13881, 2022.
Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. Optuna: next-generation hyperparameter optimization framework. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, pp. 2623-2631, 2019.
Muhammad Asad, Ahmed Moustafa, and Takayuki Ito. Fedopt: Towards communication efficiency and privacy preservation in federated learning. Applied Sciences, 10, 2020.
N. Awad, N. Mallik, and F. Hutter. Differential evolution for neural architecture search. In Proceedings of the 1st workshop on neural architecture search( @ ICLR (0). 2020.
Noor Awad, Neeratyoy Mallik, and Frank Hutter. Dehb: Evolutionary hyberband for scalable, robust and efficient hyperparameter optimization. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21, pp. 2147-2153, 2021.
James Bennett and Stan Lanning. The netflix prize. 2007.
James Bergstra and Yoshua Bengio. Random search for hyper-parametes parameter optimization. J. Mach. Learn. Res., pp. 281-305, 2012.
André Biedenkapp, Furkan Bozkurt, Theresa Eimer, Frank Hutter, and Marius Thomas Lindauer. Dynamic algorithm configuration: Foundation of a new meta-algorithmic framework. In European Conference on Artificial Intelligence, 2020.
Bernd Bischl, Giuseppe Casalicchio, Matthias Feurer, Frank Hutter, Michel Lang, Rafael G Man- tovani, Jan N van Rijn, and Joaquin Vanschoren. Openml benchmarking suites. arXiv preprint 1.XXv:1708.03331 2017.
Sebastian Caldas, Sai Meher Karthik Duddu, Peter Wu, Tian Li, Jakub Konečný, H Brendan McMa- han, Virginia Smith, and Ameet Talwalkar. Leaf: benchmark for federated settings. arXiv preprint arXiv:
2018.
Zhongxiang Dai, Bryan Kian Hsiang Low, and Patrick Jaillet. Federated bayesian optimization via thompson sampling. In NeurlPS, 2020.
Kalyanmoy Deb, Samir Agrawal, Amrit Pratap, and T. Meyarivan. A fast and elitist multiobjective genetic algorithm: Nsga-ii. IEEE Trans. Evol. Comput., 6: 6:182-197, 2002.
Tobias Domhan, Jost Tobias Springenberg, and Frank Hutter. Speeding up automatic hyperparameter optimization of deep neural networks by extrapolation of learning curves. In Twenty-fourth international joint conference on artificial intelligence, 2015.
Xuanyi Dong and Yi Yang. NAS-Bench-201: Extending the scope of reproducible neural architecture search. In International Conference on Learning Representations (ICLR), 2020.
Benchmarking nas Xuanyi Dong. Lu Liu, Katarzyna Musial, and Bogdan Gabrys. Patern Analysis and Machine algorithms for architecture topology and size. IEEE Transactions Intelligence (TPAMI), 2021.
Katharina Eggensperger, Philipp Müller, Neeratyoy Mallik, Matthias Feurer, Rene Sass, Aaron Klein, Noor Awad, Marius Lindauer, and Frank Hutter. HPOBench: A collection of reproducible multi-fidelity benchmark problems for HPO. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021.
Stefan Falkner, Aaron Klein, and Frank Hutter. Bohb: Robust and efficient hyperparameter optimiza- tion at scale. In International Conference Machine Learning, pp. 1437-1446 201 8a.
10

---

Under review as a conference paper at ICLR 2023
Stefan Falkner, Aaron Klein, and Frank Hutter. Bohb: Robust and efficient hyperparameter optimiza- tion at scale. In International Conference on Machine Learning. 2018b.
Matthias Feurer and Frank Hutter. Hyperparameter optimization. In Automated machine learning. pp. 3-33. 2019.
Gijsbers, E. LeDell, S. Poirier, J. Thomas, B. Bischl, and J. Vanschoren. An open source automl benchmark. arXiv preprint htt1::/997.000000 [cs.LG], 2019.
Ronald L.. Graham, Donald E. Knuth, and Oren Patashnik. Concrete Mathematics: Foundation for Computer Science. Addison-Westee 1989.
Pengfei Guo, Dong Yang, Ali Hatamizadeh, An Xu, Ziyue Xu, Wenqi Li, Can Zhao, Daguang Xu, Stephanie A. Harmon, Evrim B Turkbey, Baris I Turkbey, Bradford J. Wood, F. Patella, Elvira Stellato, Gianpaolo Carrafiello, Vishal M. Patel, and Holger R. Roth. Auto-fedrl: Federated hyper- parameter optimization for multi-institutional medical image segmentation. ArXiv, abs/2203.00333. 2022.
Nikolaus Hansen, Anne Auger, Olaf Mersmann, Tea Tusar, and Dimo Brockhoff. Coco: a platform for comparing continuous optimizers in black-box setting. Optimization Methods and Software pp. 114 144, 2021.
Florian Hase, Matteo Aldeghi, Riley J. Hickman, Loic M. Roch, Melodie Christensen, Elena Liles, Ja- son E. Hein, and Alán Aspuru-Guzik. Olympus: a benchmarking framework for noisy optimization and experiment planning. Machine Learning: Science and Technology, 2021.
Alejandro Morales Hernández, Inneke Van Nieuwenhuyse, and Sebastian Kojas-Gonzalee A sur- vey on multi-objective hyperparameter optimization algorithms for machine learning. ArXiv, abs/21 0as/2111.113555 2021.
Stephan Holly, Thomas Hiessl, Safoura Rezapour Lakani, Daniel Schall, Clemens Heitzinger, and Jana Kemnitz. Evaluation of wysperparamtteroopimizzee approaches in an industrial federated learning system. ArXiv, abs/21 10.08202, 2021.
Dzmitry Huba, John Nguyen, Kshitiz Malik, Ruiyu Zhu, Mike Rabbat, Ashkan Yousefpour, Carole- Jean Wu, Hongyuan Zhan, Pavel Ustinov, Harish Srinivas, et al. Papaya: Practical, private, and scalable federated learning. Proceedings of Machine Learning and Systems, 2022.
Frank Hutter, Holger H Hoos, and Kevin Leyton-Brown. Sequential model- optimization for general algorithm configuration. In International conference on learning and intelligent optimization, pp. 507-523. Springer, 201 la.
Frank Hutter, Holger H Hoos, and Kevin Leyton-Brown Sequential model-based optimization for general algorithm configuration. In International conference learning and intelligent optimization, 201
Frank Hutter, Holger Hoos, and Kevin Leyton-Brown. Parallel algorithm configuration. pp. 55-70, 2012.
Frank Hutter, Manuel lópez-Ibáñez, Chris Fawcett, Marius Thomas Lindauer, Holger H. Hoos, Kevin Leyton-Brownn and Thomas Stützle. Aclib: A benchmark library for algorithm configuration. In LION, 2014.
Kevin Jamieson and Ameet Talwalkar. Non-stochastic best arm identification and hyperparameter optimization. Proceedings of the 19th International Conference on Artificial Intelligence and Statistics, pp. 240-248, 2016.
D. R. Jones, A taxonomy global optimization methods based on response surfaces. Journal of Global Optimization, 21:345-383, 2001.
Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. Ad- vances and open problems in federated learning. arXiv preprint arXiv: 1912::0112.04777 2019.
11

---

Under review as a conference paper at ICLR 2023
Mikhail Khodak, Renbo Tu, Tian Li, Liam Li, Nina Balcan, Virginia Smith, and Ameet Talwalkar. Federated hyperparameter tuning: Challenges, baselines, and connections to weight-sharing, In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021.
Aaron Klein, Stefan Falkner, Simon Bartels, Philipp Hennig, and Frank Hutter. Fast bayesian optimization of machine learning hyperparameters large datasets. In Artificial intelligence and statistics, pp. 528-536, 2017.
Antti Koskela and Antti Honkela. Learning rate adaptation for differentially private learning. In Proceedings the Twenty Third International Conference Artificial Intelligence and Statistics, pp. 2465 2475, 2020.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Fan Lai, Yinwei Dai, Sanjay S. Singapuram, Jiachen Liu, Xiangfeng Zhu, Harsha V. Madhyastha, and Mosharaf Chowdhury. FedScale: Benchmarking model and system performance of federated learning at scale. In International Conference on Machine Learning (ICML), 2022.
Lisha Li, Kevin Jamieson, Giulia DeSalvo, Afshin Rostamizadeh, and Ameet Talwalkar. Hyperband: A novel bandit-based approach to hyperparameter optimization. The Journal of Machine Learning Research, pp. 6765-6816, 2017.
Tian Li, Anit Kumar Sahu, Ameet Talwalkar, and Virginia Smith. Federated learning: Challenges, methods, and future directions. IEEE Signal Processing Magazine, pp. 50-60, 2020a.
Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization heterogeneous networks. Proceedings of Machine Learning and Systems, pp. 429-450, 2020b.
Tian Li, Shengyuan Hu, Ahmad Beirami, and Virginia Smith. Ditto: Fair and robust federated learning through personalization. In ICML, pp. 6357-6368, 2021a.
Xiaoxiao Li, Meirui JIAGG, Xiaofei Zhang. Michael Kamp, and Dou. FedBN: Federated learning on non-lID features via local batch normalization. In International Conference on Learning Representations, 2021b.
Zitao Li, Bolin Ding, Ce Zhang, Ninghui Li, and Jingren Zhou. Federated matrix factorization with privacy guarantee. Proc. VLDB Endow., 15:900-913, 2021c
Marius Lindauer, Katharina Eggensperger, Matthias Feurer, André Biedenkapp, Difan Deng, Carolin Benjamins, Tim Ruhkopf, René Sass, and Frank Hutter. Smac3: A versatile bayesian optimization package for hyperparameter optimization. J. Mach. Learn. Res., pp. 54-1, 2022.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. DARTS: Differentiable architecture search. In International Conference Learning Representations, 2019.
Brendan McMahan, Fider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera Arcas.
Communication--fficiieee Learning of Deep Networks from Decentralized Data. In Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, pp. 1273-1282, 2017.
Ilya Mironov. Rényi differential privacy. In 2017 IEEE 30th Computer Security Foundations Symposium (CSF), pp. 263-275, 2017.
Felix Mohr, Marcel Wever, Alexander Toreeee, and Eyke Hüllermeier. Predicting machine learning pipeline runtimes in the context of automated machine learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43:3055-3066, 2021.
Hesham Mostafa. Robust federated learning through representation matching and adaptive hyper- parameters, 2020.
12

---

Under review as a conference paper at ICLR 2023
Fabian Pedregosa, Gaël Varoquaux. Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg. Jake Vanderplas, Alexandre Passos, David Couraaeaau, Matthicu Brucher, Matthieu Perrot, and Edouard Duchesnay, Scikit-learm Machine learning in python. Journal of Machine Learning Research, pp. 2825-2830) 2011.
Johann Petrak. Fast subsampling performance estimates for classification algorithm selection. In Proceedings of the ECML--00 Workshop on Meta-Learning: Building Automatic Advice Strategies for Model Selection and Method Combination, pp. 3-14, 2000.
Florian Pfisterer, Lennart Schneider, Julia Moosbauer, Martin Binder, and Bernd Bischl. Yahpo gym-an efficient multi-objective multi-fidelity benchmark for hyperparameter optimization. In First Conference on Automated Machine Learning (Main Track), 2022.
Sebastian Sineda-Aranges Hadi S. Jomaa, Martin Wistuba, and Josif Grabocka. HPO-B: A large-scale reproducible benchmark for black-box HPO based on openml. Neural Information Processing Systems (NeurIPS) Track on Datasets and Benchmarks, 2021.
Yasha Pushak and Holger H. Hoos. Automl loss landscapes. ACM Transactions on Evolutionary Learning and Optimization, 2022.
Prithviraj Sen, Galileo Namata, Mustafa Bilgic, Lise Getoor, Brian Galligher, and Tina Eliassi-Rad. Collective classification in network data. Al magazine, pp. 93-93, 2008
Bobak Shahriari, Kevin Swersky, Ziyu Wang, Ryan P. Adams, and Nando de Freitas. Taking the human out of the loop: review of bayesian optimization. Proceedings of the IEFE, pp. 148-175, 2016.
Rainer Storn and Kenneth Price. Differential evolution-a simple and efficient heuristic for global optimization over continuous spaces. Journal of global optimization, pp. 341-359, 1997.
Kevin Swersky, Jasper Snock, and Ryan Adams. Multi-task bayesian optimization. Advances in neural information processing systems, 2013.
Kevin Swersky, Jasper Snoek, and Ryan Prescott Adams. Freeze-thaw bayesian optimization. arXiv preprint Xiv: vr..ivv406...5 1406.3866, 2014.
R. Turner and D. Eriksson. Bayesmark: Benchmark framework to easily compare bayesian optimiza- tion methods on real machine learning tasks, 2019.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprini 1804.0880.00745 07461, 2018.
Jianyu Wang, Zachary Charles, Zheng Xu, Gauri Joshi, H Brendan McMahan, Maruan Al-Shedivat, Galen Andrew, Salman Avestimehr, Katharine Daly, Deepesh Data, et al. A field guide to federated optimization. arXiv preprint arXiv:2107.00 177.7, 2021a.
Zhen Wang, Weirui Kuang, Yuexiang Xie, Liuyi Yao, Yaliang Li, Bolin Ding, and Jingren Zhou. Federatedscope-gmm gnn: Towards a unified, comprehensive and efficient package for federated graph learning. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and 2022.
Data Mining, pp.
Zheng Wang, Xiaoliang Fan, Jianzhong Qi, Chenglu Wen, Cheng Wang, and Rongshan Yu. Federated learning with fair averaging. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21, pp. 1615-1623, 2021b.
Kang Wei, Jun Li, Ming Ding, Chuan Ma, Howard H. Yang, Farhad Farokhi, Shi Jin, Tony Q.S. S. Quek, and H. Vincent Poor. Federated learning with differential privacy: Algorithms and performance analysis. Trans. Info. For. Sec., pp. 3454-3469, 2020.
Yuexiang Xie, Zhen Wang, Daoyuan Chen, Dawei Gao, Liuyi Yao, Weirui Kuang, Yaliang Li, Bolin Ding, and Jingren Zhou. Federatedscope: A flexible federated learning platform for heterogeneity. arXiv preprint arXiv:2204.0501 2022.
13

---

Under review as a conference paper at ICLR 2023
Zhilin Yang, William Cohen, and Ruslan Salakhudinov. Revisiting sumi-supee learning with graph embeddings. International conference on machine learning. pp. 40-48, 2016.
Liuyi Yao, Dawei Gao, Zhen Wang, Yuexiang Xie, Weirui Kuang, Daoyuan Chen, Hao- hui Wang, Chenhe Dong, Bolin Ding, and Yaliang Li. Federated hetero-task learning. https:/urxiv.or//ab//2.26.. 033///bb/2200003336., 2022.
Huanle Zhang, Mi Zhang, Xin Liu, Prasant Mohapatra, and Michael DeLucia. Automatic tuning of federated learning syper-parameters from system perspective. arXiv preprint 10.00011,, 2021.
Yi Zhou, Parikshit Ram, Theodoros Salonidis, Nathalie Baracaldo, Horst Samulowitz, and Heiko Ludwig. Flora: Single-shot hyper-parameter optimization for federated learning. arXiv preprint 12.00221,..0824, 2021.
14

---

Under review as a conference paper at ICLR 2023
A MAINTENANCE OF FEDHPO-BENCH
In this section, we present our plan for maintaining FEDHPO-BENCH following Eggensperger et al. (2021).
Who is maintaining the benchmarking library? FEDHPO-BENCE is developed and maintained by FedHPO-Bence team (to de-anonymized later).
How can the maintainer of the dataset be contacted (e-g., email address)? Users can reach out to the maintainer via creating issues on the Github repository with FEDHPO- BENCH label.
Is there erratum? No.
Will the benchmarking library be updated? Yes, as we discussed in Section 5 we will add more FedHPO problems and introduce more FL tasks to the existing benchmark. We will track updates and Github release on the README. In addition, we will fix potential issues regularly.
Will older versions of the benchmarking library continue to be sup- ported//oste/mmaintaneddd??
All older versions are available and maintained by the Github release, but limited support will be provided for older versions. Containers will be versioned and available via AliyunOSS.
If others want to extend/augmmen//uild on/contribute to the dataset, is there mechanism for them to do so? Any contribution welcome, and all commits to FEDHPO-BENCH must follow the guidance and regulations at FedHPO- BetiioO-Bencc//edPPO-BBeeee Bttneech-TCRR////bo///aaaaaaaaaa READMEEEREEEE........... mdl
com/
B RELATED WORK
Hyperparameter Optimization (HPO). Generally, HPO is an optimization problem where the objective function is non-analytic, non-convex, and even non-differentiable. Therefore, most HPO methods solve such an optimization problem
trial-and-erre manner, with different strategies for balancing exploitation and exploration. Model-free methods such as random search (RS) (Bergstra Bengio 2012) and grid search query a set of initially determined hyperparameter configurations without any exploitation. Model-based methods such as Bayesian Optimization (BO) (Shahriari et al. 2016) employ a surrogate model to approximate the objective function. Methods in this line (Hutter al. [01 20 Lindauer et al.) [022 Hutter et al. [201 Falkner al., 2018b) mainly differ from cach other in their surrogate model and how they determine the next query. There are also Evolutionary Algorithms (EAs) that iteratively maintain population. We consider differential EAs (Storn & Price
1997: Awad et al. 2020) in our experiments.
As training deep neural network a large- scale dataset is very costly, the full-fidelity function evaluations made by BO methods are often unaffordable in practice. Naturally, researchers consider trading off the precision of function evaluation for its efficiency by, e-g., training fewer epochs and training on a subset of the data. Hyperband (Li et al. 2017) is representative multi-fidelity method that calls the Successive Halving Algorithm (SHA) Jamieson & Talwalkar 2016) again and again with different number of initial candidates. However, in each execution of SHA, the initial candidates are randomly sampled without any exploitation. To exploit the experience of previous SHA executions, researchers combine BO methods with Hyperband (Falkner et al. 20 Awad ct al. 202T).
Benchmarking HPO. AutoML-rrelted optimization benchmarks have been proved helpful for promoting fair comparisons of related methods and reproducible research works. There have been
many successful examples (Hutter et al. 2014; Hansen et al. [021 Hase al. 2021 Turner & Eriksson 2019 Dong & Yang 2020: Dong et al.) 2021; Gijsbers et al. 2019). Noticeably, HPO- B Srineda-Arango al. 20211) is highlighted by its support for benchmarking transfer-Hie HPO methods, and HPOBench (Eggensperger et al. (202)) fills the gap of missing multi- fillii-fiiiityy HPO benchmarks.
However, existing HPO benchmarks mainly focus centralized ML, yet FL, a promising learning paradigm, has been ignored. In this paper, we identify the uniqueness of FedHPO (see Section 2.2 and implement FEDHPO- SEHH--BBENCH satisfy the demand for FedHIPO benchmark suite.
15

---

Under review as a conference paper at ICLR 2023
Federated Learning (FL). In this paper, restrict our discussion of FL to the "standard" scenario introduced in Section 2.2] where FedAvg (McMahan et al.] (017)) is widely adopted. Fancy FL optimization algorithms, including FedProx (Li et al., 2020b) and FedOpt (Asad et al. 2020). are mainly designed to improve the convergence rate and/or better handle the non-IlDness among clients (Wang et al.] (021a). Despite these synchronous optimization algorithms, asynchronous ones (Huba et al. 2022] Xie et al. 2022) are proposed to keep a high concurrency utility.
Sometimes, learning one global model is insufficient to handle the non-IIDness, which calls for personalized FL (Kairouz et al., 22119] Wang et al. 202 Many popular pFL algorithms, such as FedBN (Li et al. (2021b) and Ditto (Li et al. 2021a), have been incorporated into FS (Xie al. 2022). with their unique hyperparameters exposed. Thus, we can further extend FEDHPÓ-BENCH by considering FedHPO tasks of optimizing the hyperparameters of such algorithms.
FedHPO. When we consider HPO in the FL setting, as mentioned in Section 2.2 there is some uniqueness that brings in challenges while, at the same time, can be leveraged by deliberately designed FedHPO methods. For example. in contrast to traditional HPO methods that query one configuration in each trial, FedEx (Khodak et al. 2021) maintains one policy for determining the client- side hyperparameters and independently samples each client's configuration in each communication round. Different configurations may be evaluated with the same model parameters, which is in analogy to the weight-sharing idea in neural architecture search (NAS) (Liu et al., 2019). as summarized by the authors of FedEx. However, due to the non-IIDness among clients, clients' HPO objective functions tend to be different, where determining their configurations by only one policy might be unsatisfactory. Regarding this issue, FTS (Dai et al.) 2020) can be treated as a personalized FedHPO method, where each client maintains its own policy. During the learning procedure, the clients benefit each other by sharing the policies in a privacy-preserving manner and conducting Thompson sampling.
It is worth mentioning that parallel algorithms have been utilized in HPO (Jones 2001 Hutter et al. p.12). However, in FedHPO, the clients actually do not correspond the same black- box function due to the heterogeneity among them. Essentially, FedHPO methods instantiate the concurrent exploration idea with extra assumptions. Besides, vanilla parallel HPO methods may leak privacy in the aggregation step, which has been carefully taken into account by FTS.
Dynamic algorithm configuration methods (Biedenkapp et al., 2020 Adriaensen et al. 2022 employ reinforcement learning to learn policies for online adjustments of algorithm parameters, since different parameter values can be optimal at different stages. In contrast to DAC methods, the policy 'T learned in FedEx is responsible for determining the optimal lower-level response of the bi-level optimization problem discussed in Section 2.2 which can be regarded as a multi-armed bandit problem rather than Markov decision process. In other words, combined with the concurrent exploration strategy, FedEx tries out one arm at a client in each round, where the underlying reward function assumed to be unchanged across the clients and the whole training course.
As an emerging research topic, existing works relating to FedHPO include Ted-Tuning (Zhang et al. (02T) concerning about system-related performance, learning rate adaptation (Koskela & Honkela 2020). FLoRA for Gradient Boosted Decision Trees (GBDT), online adaptation scheme- based method (Mostafa, 2020). Auto-FedRL (Guo et al. 2022) for RL-based hyperparameter adaptation, and an insightful comparison between local and global HPO (Holly et al. (002)). These methods can also be easily incorporated into FS, enabling BEDHPO-BENCEE to benchmark them.
B.1 RELATION TO HPOBENCH
HPOBench (Eggensperger et al. 2021) is a collection of multi-fidelity HPO benchmarks, highlighted by their efficiency, reprodueibility. and flexibility. These benchmarks can be accessed either tabular, surrogate, or raw mode. On the one hand, the tabular and surrogate modes enable function evaluation without truly executing the corresponding ML algorithm and thus are efficient. On the other hand, the raw mode means execution in docker container, which ensures reproducibility. HPOBench provides twelve families of benchmarks that correspond to different data domains, model types, fidelity spaces, etc., and thus flexible usages to validate HPO methods. This collection of HPO benchmarks can promote fair comparisons of related works and reproducible research work, so HPOBench has gained more and more attention from the community.
16

---

Under review as a conference paper at ICLR 2023
As pointed out in Section 2.2 that evaluating the objective function that corresponds to FL algorithm COOHHPO-BENCHE also prepares tabular and surrogate modes for users to
is extremely expensive, avoid truly executing FL courses. Mcanwhile, we provide the raw mode to truly execute FL course in the docker container of FederatedScope (FS) (Xie et al. 2022).
Sharing the same modes, question naturally arises is it possible to reuse HPOBench's interface for FEDHPO-BENCH? We answer this question by discussing their commonality and the unique ingredients of FEDHPO-BENCI:
Commonality, As the code snippet in Figure 2 shows, the instantiation of a benchmark class, the "ConfigSpace" package based specification of search space, and the protocol for the interaction between an HPO method and benchmark object are roughly consistent with HPOBench.
Uniqueness. In addition enabling users to tailor one benchmark to their scenarios (see Section 3.2) To this end, users are
collection of benchmarks, TEDHPO-BENCE is flexible in terms of allowed instantiate specific benchmark object with extra optional arguments:
Privacy budger with which function evaluation corresponds to the execution of NbAFL (Wei et al. 2020) instead of vanilla FedAvg. Taking the tabular mode, for example, this means looking up privacy budget-specific table.
The type of fairness metric and its strength with which PEDHPO-BENCE will consider a vector-valued objective function (i.e., client-wise results) rather than scalar-valued objective function. Besides, the return value of calling the function evaluation will be the mean performance regularized by the specified fairness regularizer.
The parameter(s) for our system model with which the execution time is estimated regarding the user's system condition. Without using system model, REDHPO-BENNCE can provide the recorded execution time in the creation of this benchmark.
Currently, we implement the interfaces of FEDHPO-BENCH by ourselves, where the style of our interfaces is kept similar to HPOBench for the convenience of users who are familiar with HPOBench. We also provide several examples
Coom/FedHPO -cecch-Bee.ch// FedHPO- Bench ICLR23/ERR CCCLR23/teee/main// demo) to access our tabular, surrogate, and raw bench- marks by implementing HPOBench's abstract base class. As first step, we are going to contribute more such subclasses to the repository of HPOBench so that users can access our benchmarks via HPOBench's interfaces, where flexible customization cannot provided temporarily. our next step, we plan to extend the interfaces of HPOBench such that the benchmarks of FEDHPO-BENCH can be accessed with our proposed flexible customization.
C HPO METHODS
As shown in Table 6 we provide an overview of the optimizers (i.e., HPO methods) we use this paper.
Table 6: Overview of the optimizers from widely adopted libraries.
Name
Model
Packages
version
RS (Bergstra & Bengio
2012
HPBandster
0.7.4
BOGP (Hutter et
201
Lindauer et al.
2022
GP
SMAC3
1.3.3
BO (Hutter et al.
201
RF
SMAC3
1.3.3
BOKpg
(Flkknrr et
80)
KDE
HPBandster
0.7.4
DE (Stor & Price
1997
Awad et al.
2020
DEHB
git commit
HB Li et al. (017))
HPBadtster
0.7.4
BOHB
Falkner et al.
2018a)
KDE
HPBantster
0.7.4
DEHB
Awad et al.
2021)
DEHB
git commit
TPEMD
Akiba et al.
20T9)
TPE
0 Optuna
2.10.0
TPEμ8
Akiba et al.
2019
TPE
0 Optuna
2.10.0
17

---

Under review as a conference paper at ICLR 2023
C.I BLACK- BOX OPTIMIZERS
RS (Random search) is priori-free free HPO method, i.., each step of the search does not exploit the already explored configuration. The random search outperforms the grid search within small fraction of the computation time.
GP is Bayesian optimization with Gaussian process model. B0GP uses Matérn kernel for continuous hyperparameters, and a hamming kernel for categorical hyperparameters. addition, the acquisition function is expected improvement (EI).
RORF is a Bayesian optimization with a random forest model. We set the hyperparameters of the random forest follows: the number of trees is 10, the max depth of each tree is 20, and we use the default setting of the minimal samples split, which is
BOKDE is Bayesian optimization with kernel density estimators (KDE), which is used in BOHB (Falkner et al. 2018a). It models objective function as Pr(x ygood) and Pr(x yood)- We set the hyperparameters for BOKDE follows: the number of samples to optimize EI is 64, and 1/3 of purely random configurations are sampled from the prior without the model; the bandwidth factor is encourage diversity, and the minimum bandwidth is 1e-3 keep diversity.
DE uses the evolutionary search approach Differential Evolution. We set the mutation strategy to randl and the binomial crossover strategy to bin In addition, we use the default settings for the other hyperparameters of DE, where the mutation factor is 0.5, crossover probability is 0.5, and the population size is 20.
C.2 MULTI-FIDELITY OPTIMIZERS
HB (Hyperband) is an extension on top of successive halving algorithms for the pure-exploration nonstochastic infinite-armed bandit problem. Hyperband makes trade-off between the number of hyperparameter configurations and the budget allocated each hyperparameter configuration. We set 77 to 3, which means only fraction of 1/n of hyperparameter configurations goes the next round.
BOHB combines HB with the guidance and guarantees of convergence of Bayesian optimization with kemel density estimators. We set the hyperparameter of the BO components and the HIB components of BOHB to be the same as BOKDE and HB described above, respectively.
DEHB combines the advantages of the bandit- based method HB and the evolutionary search approach of DE. The hyperparameter of DE components and BO components are set to exactly the same as DE and HB described above, respectively.
TPENp is implemented in Optuna and uses Tree-structured Parzen Estimator (TPE) as a sampling algorithm, where each trial, TPE fits two Gaussian Mixture models for each hyperparameter. One is to the set of hyperparameters with the best performance, and the other is to the remaining hyperparameters. In addition, it uses the median stopping rule as a pruner, which means that it will prune if the trial's best intermediate result is worse than the median (MD) of intermediate results of previous trials the same step. We use the default settings for both TPE and MD.
TPEнB is similar to TPEND described above, which uses TPE as a sampling algorithm and HB as pruner. We set the reduction factor to 3 for HIB pruner, and all other settings use the default ones.
D DATASETS
As shown in Table 7 we provide detailed description of the datasets we use in current FEDHPO- BENCIL. For comprehensiveness, we use 16 FL datasets from domains, including CV, NLP, graph, tabular, and recommendation (Xie al. 2022 Wang al. 2022 Eggensperger et al. 2021). Some of them are inherently real-world FL datasets, while others are simulated FL datasets split by the splitter modules of FS. Notably, the name of datasets from OpenML is the ID of the corresponding task.
Please refer to httpt:tt///// http://gtttttuuttt thub. wtw.bb..oomauuoooo /DDHH//l wwtp:/owwaaaaeeeeeeeeeeeeeeeeeet www.atteeRRRAMM... for details.
18

---

Under review as a conference paper at ICLR 2023
Table 7: Statistics of the datasets used in current FEDHPO--ERS
Name
#Client
Subsample
#Instance
#Class
Split by
FMNIST
3,550
5%
805,263
62
Writer
CIFAR-10
5
100%
60,000
10
LDA
CoLA
5
100%
10,657
2
LDA
SST-2
5
100%
70.042
2
LDA
Cora
5
100%
2,708
7
Community
CiteSeer
5
100%
4,230
6
Community
PubMed
5
100%
19,717
3
Community
Hetero-task
5
100%
6,760
2-6
Task
credit-ga1
5
100%
1,000
2
LDA
vehicles3
5
100%
846
LDA
kcl 3917
5
100%
2,109
2
LDA
blood-transf..10.0r
100% 5
748
2
LDA
Australian 46818
5
100%
690
2
LDA
car146821
5
100%
1,728
4
LDA
segment, L16822
116822
5
100%
2,310
7
LDA
FedNetflix Twitter
480,189
100%
100,000,000
5
User
660,120
0.5%
1,600,498
2
User
FEMNIST is an FL image dataset from LEAF Caldas et al. 2018), whose task is image classification. Following (Caldas et al. 2018), we use a subsample of FEMNIST with 200 clients, which is round 5%. And use the default train/valid/test splits for each client, where the ratio is 60% 20% 20%.
CIFAR-10 (Krizhevsky et al. 2009) is from Tiny Images dataset and consists of 60,000 32 x 32 color images, whose task image classification. We split images into 5 clients by latent dirichlet allocation (LDA) to produce statistical heterogeneity among these clients. We split the raw training training and validation sets with ratio 4
that ratio of final train/valid/test splits is
66.7%: 6..6%%:16.67%% 66.67.:66...7..16666667777
SST-2 is dataset from GLUE (Wang al. 2018) benchmark, whose task is binary sentiment classification for sentences. We also split these sentences into 5 clients by LDA. In addition, we use the official train/valid/test splits for SST-2.
CoLA is also dataset from GLUE benchmark, whose task is binary classification for sentences- whether is grammatical English sentence. We exactly follow the experimental setup in SST-2,
Cora & CiteSeer & PubMed (Sen et al. 2008 Yang et al. 2016) are three widely adopted graph datasets, whose tasks node classification. Following FS-G (Wang al.) (2022), community splitter is applied to each graph to generate five subgraphs for each client. We also split the nodes into train/validdtest sets, where the ratio is 60%:20%:20%%
Hetero-task is a graph classification dataset adopted from Graph-DC (Yao et al. 2022), which contains 5 clients. Each client has different but similar graph classification task, such as molec- ular attribute prediction. In addition, we set the ratio of train/valid/test splits in each client to 80%:10%: 10%:10%
Tabular datasets are consist of 7 tabular datasets from OpenML (Bischl et al. 2017), whose task ids (name source data) are 31 (credit-g). 53 (vehicle), 3917 (kel), 10101 (blood-trant wewndddrransusionsseeeeee center), 146818 (Australian). 146821 (car) and 146822 (segment). We split each dataset into clients by LDA, respectively. In addition, we set the ratio of train/valid/test splits to 80%: 10%:: 1 10%.
FedNettlix is recommendation dataset from The Netflix Prize (Bennett & Lanning 2007). whose task is to predict the ratings between users and movies. Netflix consists of around 100 million ratings between 480,189 users and 171,770 movies. We split the Netflix dataset into 480.189 clients by users. In addition, we set the ratio of train/valid/test splits to 80%: 10%::0%
Twitter is sentiment analysis dataset from LEAF (Caldas al. 2018), whose task is determine sentiment of sentences. We use a subsample Twitter with around 3300 clients. Moreover, we use the train/valid/test splits for each client, where the ratio is 80% 10% 10%. It is worth noting that
19

---

Under review as a conference paper at ICLR 2023
the average number of samples is only 1.94, which means some clients do not have valid split or test split, and we evaluate the performance on a shared test split merged by all clients.
E SYSTEM MODEL
In this section, we will discuss the system model in detail we have proposed and implemented. The total execution time of FL consists of the time consumed by communication and the time consumed by calculation, thus, the system model is as follows:
T( b) T comm! (f, T Toomp (S,,, X, b),
Teomm (J, b) Sdown (f. () a(N),
Brown
Bup
(I)
T Toopp (f, b) peccc Exp((
lappl, 1.1= [max (T, (cliem) })] + (server) (f, b),
where N denotes the number of clients sampled this round, ((N) denotes the latency, which an increasing function of N but is independent of the message size (contains the time needed to establish the transmission between the server and the clients), S(f. denotes the download/upload size, B denotes the download/upload bandwidth of client, T'server) is the time consumed by server-side computation, and 7 (cllet))) denotes the computation time consumed by i-th client, which is sampled from an exponential distribution with c(f, X, b) as its mean. This design intends to simulate the heterogeneity among clients' computational capacity, where the assumed exponential distribution has been widely adopted in system designs (Wang et al., 2021a) and is consistent with real-world applications (Huba et al. 2022).
We provide default parameters of our system model, including c(f, A, Bup Brown and (server) based on observations collected from FL trials we have conducted and real-world network bandwidth. Users are allowed to specify these parameters according to their scenarios or other system statistic providers, e.g., estimating the computation time of stragglers by sampling from FedScale (Lai et al. 2022). As for the network bandwidth, we set Baown ~ 0.75MB/secs Bup 0.25MB/ /scss following (Lai et al. 2022 Wang et al., 2021a). The default value of c(f, b) is obtained by averaging the recorded client wise time costs trials of tabular mode benchmarks. Due the limit on the number of ports of the server, we set the default value of the maximum number of connections in calculating a (NN) to 65535.
To implement our system model, we use the following proposition to calculate Eq. analytically, where we use as a shorthand for c(J, b) keep clarity.
Proposition 1. When the computation time of clients is identically independently distributed, fol- lowing an exponential distribution Exp(-1?), then the expected time for the straggler of N uniformly sampled clients is EN1 1
What we need to calculate is the expected maximum of i.i.d. exponential random variable. Proposi- tion states that, for N exponential variables independently drawn from Exp(-I)), the expectation is There many ways to prove this useful proposition, and we provide proof starting from studying the minimum of the exponential random variables.
Proof. At first, the minimum of N such random variables obeys Expp+1 Graham et al. 1989). Denoting the i-th minimum of them by T, T₁ ~ Exp(-I N) and Ty is what we are interested in. Meanwhile, it is well known that exponential distribution is memoryless, namely, Pr(X > 8 + +|XX > 8) = Pr(X > (). Thus, T2 Ti obeys the same distribution as the minimum of N 1 such random variables, that is to say, T2 - Ti a random variable drawn from Exp(-) Similarly, (T++1 T:) ~ Exp(-1 N-1 ) i = 1, 1,..., N - 1. Thus, we have:
N-1
N-1
N
E E = E + Σ
(T - Ta) = N
+ Σ
= N -
c
(2)
1=1
20

---

Under review as a conference paper at ICLR 2023
x/t10
amar(s)
") a (A 1. (41))
(49.5) (44)))
Global policy Global model 6/
sync(
oft syncl sy A/5)
Broadcast plo, y or Al
Seed 1/110) 4 or R. (49)
28 a/" Larral updatte on
ovail-)
1.th (49)
Clienta
Clienty
[xf
update (2 1.
Figure 6: A general algorithmic view for FedHIPO methods: They are allowed to concurremly explore different client-side configurations the same round of FL., but the clients heterogeneous, iee, corresponding different functions f (-). Operators in brackets are optional.
which concludes this proof.
is worth noting that we provide several optional system models. For example, for point-to-pont to-point transport protocols, Teomm should contain the time the server sends the model to each client.
F DETAILS OF THE IMPLEMENTATIONS OF FEDEX AND FTS
We first present general algorithmic view in Figure which unifies several such methods well as their personalized counterparts, At a high level, a policy æ for determining the optimal lower-level response ( = minxer f(x(), X is to be federally Iearned, along with the FL course itself. In the f-th communication round: (1) In addition to the model g(t), either the policy =(1) or its decisions ( is also broadcasted. (2) For the i-th client, if (+() is received, it needs to synchronize its local policy ni with this global one and then sample hyperparameter configuration X from its local policy. (3) Either received or locally sampled, x!0) is used to specify the local update procedure of FL. which results in updated local model o((+1), (4) Then (((-1) is evaluated to provide the result of
(client-specific) function evaluation f!() (2(0)) (5) For personalized FedHPO methods that maintain
a local policy mis updated w.r.t. (A;, Ji(A;)) to produce m
(t+1)
(6) In addition the local
model o((++)) either the local policy x;
(t-1) or the feedback (2/9) ft (2(0) sent the server.
(7) Finally, the server aggregates g(t+1)s s into 1(++1) and m respectively.
(++1) s/ (2(0) A (2(0) s into ((++1)
In FedEx (Khodak et al. 2021), A, are independently sampled from म, and the aggregation operator "aggr, is exponential gradient descent. FTS (Dai et al. 2020). the broadcasted policy x(!) the samples drawn from all clients' posterior beliefs. The synchromous operator "sync," can be regarded as mixing Gaussian process (GP) models. The update operator "update," p corresponds to updating
local GP model. Then a sample drawn from local GP posterior belief is regarded as π; uploaded. Finally, the aggregation operator "aggr," " is packing received samples together.
((+1) and
G STUDIES ABOUT THE NEW FIDELITY
In FL, a larger client client_sammpaea sample_raa rate leads to minor variance of the aggregated model in each round, which is believed to need less #round for convergence and to perform better. Therefore, we tend to set the client sample_ rate as close in Section 3.2 a large sitent_sample_____ pattee leads to an increase in latency (a(N), which makes the communication cost higher. To answer RQ3, we use tabular mode and study the trade-off between these two fidelity dimensions: client Lecenttssmpl_____eeee rate and #round. We simulate two distinct system conditions by specifying different parameters for our system model.
as possible. However, according our system model
Protocol. We compare the performance of IIB with different client_sane caatrtt_sapp__eeee rrats to learn 2-layer CNN with 2,048 hidden units on FEMNIST. To simulate a system condition with bad network status, set the upload bandwidth Bup to 0.25MB/second and the download bandwidth Boowa to 0. 0.75MB (Wang et al. 2021a). As for good network status, we set the upload bandwidth Bup to 0. 0.25GB/ GB/secod and the download bandwidth B (doww) to 0.75GBBsecond. In both cases, we consider
21

---

Under review as a conference paper at ICLR 2023
20%
20%
40%
60%
60%
Loss [tog)
0.5
0.5
80%
Less
100%
100%
0.0
-0.5
0.0
0.2
0.4
0.6
0.8
1.0
0,0
0.2
0.4
0.6
0.8
1.0
Fraction of budget
Fraction of budget
(a) With bad network status
(b) With good network status
Figure 7: Performances of different client_sanen cectet_sampl_raeee nate under different system conditions.
different computation overhead so that it is negligible and significant, respectively. As for the rest settings, we largely follow that in Section 4.1
Results and Analysis. We have an answer to RQ3: with the same time budget, the FL. procedure with lower client_sample rate achieves better result than higher client sspett_spppe____ l_reeeeeeeee--e with the bad network status. In comparison, that with higher client_sample sample_rate achieves a better result than lower client_saan siteettsample_dtte rate in the good network status. In conelusion, this study suggests best practice of pursuing more economic securacy-ffficinne trade-off by balancing client client_sample rate with #round, w.r.t. the system condition. Better choices tend achieve more economical accuracy- efficiency trade-off's for FedHPO.
H DETAILS ON FEDHPO-BENCE BENCHMARKS
FEDHPO-BENE BENCH consists of serveral categories of benchmarks on the different datasets (see Appendix D) with three modes. If not specified, we use the model as the name of the benchmark in cross-silo scenario. In this part, we provide more details about how we construct the FedHPO problems provided by current FEDHPO-BENG and the three modes to interact with them.
H.1 CATEGORY
We categorize our benchmarks by model types. Each benchmark is designed to solve specific FL HPO problems on its data domain, wherein CNN benchmark on CV, BERT benchmark on NLP, GNN benchmark on the graphs, and LR & MIP benchmark tabular data. All benchmarks have several hyperparameters on configuration space and two on fidelity space, namely sample rate of FL and FL round. And the benchmarks support several FL algorithms, such FedAvg and FedOpt.
CNN benchmark learns a two-layer CNN with 2048 hidden units on FEMNIST and 128 hidden units on CIFAR-10 with five hyperparameters on configuration space that tune the batch size of the dataloader. the weight decay, the learning rate, the dropout of the CNN models, and the step size of local training round in client each FL communication round. The tabular and surrogate mode of the CNN benchmark only supports FedAvg due to our limitations in computing resources for now, but with more results as soon as possible.
we will update FEDHPO-
BERT benchmark fine-tunes pre-trained language model, BERT-Tiny. which has two layers and 128 hidden units, on CoLA and SST-2. The BERT benchmark also has five hyperparameters on configuration space which is the same CNN benchmark. In addition, the BERT benchmark support FedAvg and FedOpt with all three mode.
GNN benchmark learns two-layer GCN with 64 hidden units Cora, CiteSeer and PubMed. The GNN benchmark has four on hyperparameters configuration space that tune the weight decay, the learning rate, the dropout of the GNN models, and the step size of local training round in client each FL communication round. The GNN benchmark support FedAvg and FedOpt with all three mode.
Hetero benchmark learns two-layer GCN with 64 hidden units backbone to be aggregated. The Hetero benchmark has two on hyperparameters configuration space that tune the learning rate and the step size of local training round in client each FL. communication round. For each client, there
22

---

Under review as a conference paper at ICLR 2023
Table 8: The scarch space of our benchmarks, where continuous search spaces discretized into several bins under tabular mode.
Benchmark
Name
Type
Log
#Bins
Range
batch_sit
int
{16. 32, 64)
weight _deaa
float
4
[0, 0.001]
Client
dropout
float
[0, 0.5]
step_size
int
4
[1,4]
CNN
learning _aate rale
float
10
[0.01, 1.0]
momentum
float
2
[.90 0.91
Server
learning rate
float
[0.1. 1.0]
client client_sample, r_aee round batch_sizz weight decay
float
[0.2. 1.0]
Fidelity
int
X
250
[00]
int
{8,
16, 32, 64, 128)
float
4
[0, 0.001]
Client
dropout
float
[0. 0.5]
step_size
int
4
[1,4]
BERT
learning_re _aate
float
10
[0.01, 1.0]
momentum
float
2
0.99
Server
learning rate
float
X
[0.1, 1.0]
client_sample sasisssamlleeeeaeee _rate
float
[0.2, 1.0]
Fidelity
round
int
X
40
[1, 40]
weight weight_decay
float
[0, 0.001]
dropout
float
2
[0, 0.5]
Client
step step_size size
int
8
[1. 8]
learning_ rate
float
V
10
[0.01, 1.0]
GNN
momentum
float
[0.0, 0.9]
Server
learning rrae
float
3
10.1,
client_sane cactttssappe_rre rrate
float
10.2,
Fidelity
round
int
500
[00]
learning _aal
float
2
[0.001, 0.01]
Client
step_size
int
2
[1.4]
Hetero
client_sample sample_maaaee _rate
client
float
5
[0.2. 1.0]
Fidelity
round
int
500
500]
batch_sizz
int
7
[4, 256]
weight_decae
float X
[0, 0.001]
Client
step_size
int
4
[1,4]
learning_ne rate
float
6
[0.00001, 1.0]
LR
momenium
float
2
[0.0, 0.9]
Server
learning_rat rale
float X
3
[0.1, 1.0]
client cliettssmple sampll_mmaeeaee rate round
_aate
float
[.2. 1.0]
Fidelity
int
500
[00]
batch_size
int
[4, 256]
weight_ _decu_deccyya_
float
4
[0, 0.001]
Client
step_size
int
X
[1,4]
learning rate float depth
6
[0.00001, 1.0]
int
3
[1,3]
MLP
width momenium float
int
7
[16. 1024)
X
[0.0, 0.9
Server
learning rate
float
3
[0.1, 1.0]
Fidelity
client client_sample sample_maaa
pate
float
X
5
[.02
round int
500
[00
23

---

Under review as a conference paper at ICLR 2023
personalized encoder and classifier to handle different tasks. Thus, the Hadamard product of each client's configuration makes the search space.
LR benchmark learns an Ir seven tasks from OpenML, see Appendix D for details. The LR benchmark has four on hyperparameters configuration space that tune the batch size of the dataloader, the weight decay, the learning rate, and the step size of local training round in client each FL communication round. The LR benchmark support FedAvg and FedOpt with all three mode.
MLP benchmark's 's the vast majority of settings are the same LR benchmark. But in particular, add depth and width of the MLP to search space in terms of model architecture. The MLP benchmark also support FedAvg and FedOpt with all three mode.
Cross-device. In cross-device scenarios, there can be large number of clients in total, but only a few participate in each communication round. This benchmark contains two datasets, Twitter and FedNetflix. We use a bag of words model with LR and tune the learning_ grate, weight_decay, and step_stt size_izeeeoe of local training round in Twitter. As for FedNetflix, we tune an HMFNet (Li et al.) 2021c) in the learning rate, batch_ s_ize, and step_size of local training round. Due to the time limit, the results FedNetflix is incomplete, and we present the ECDF of Twitter in Figure
H.2 MODE
Following HPOBench (Eggensperger et al. 2021), HEDHPO-BENCEE provides three different modes for function evaluation: the tabular mode, the surrogate mode, and the raw mode. The valid input hyperparameter configurations and the speed of acquiring feedback vary from mode to mode. Users can choose the desired mode according to the purposes of their experiments.
Tabular mode. The idea is to evaluate the performance of many different hyperparameter configura- tions in advance so that users can acquire their results immediately. For efficient function evaluation, we implement the tabular mode of FEDHPO- BEEHH by running the FL algorithms configured by the grid search space in advance from our original search space (see Table B). For hyperparameters whose original search space is discrete, we just preserve its original one. As for continuous ones, we discretize them into several bins (also see Table [8] for details). To ensure that the results are reproducible, we execute the FL procedure in the Docker container environment. Each specific configuration À is repeated three times with different random seeds, and the resulted performances, including loss, accuracy and fl scose under train/validationneest splits, are averaged and adopted as the results of (()). Users can choose the desired metric as the output of the black-box function via PEDHHPOBENCH'S APIs. Besides, we provide not only the results of f(A) (i.e., that with full-fidelity) but also results of b), where b is enumerated across different #round and different client sample rate. pate. Since exccuting function evaluation is much more costly in FL than traditional centralized learning, such lookup tables are precious. In creating them, spent about months of computation time on six machines, each with four Nvidia V100 GPUs. Now we make them publicly accessible via the tabular mode of FEDHPO-BENCH.
Surrogate mode.
As tabular mode has discretized the original search space and thus cannot respond queries other than the grids, we train random forest models on these lookup tables, i.e., {(X, [(, b))}. These models serve surrogate of the functions to be optimized and can answer any query À by simply making an inference. Specifically, we conduct 10-fold rrossvvaldatiion to train and evaluate random forest models (implemented in scikit-learn (Pedregosa et al. ( on the tabular data. Meanwhile, we search for suitable hyperparameters for the random forest models with the number of trees in (10, 20) and the max depth in [10, 15, 20). The mean absolute error (MAE) of the surrogate model w.r.t. the true value is within an acceptable threshold. For example, in predicting the true average loss on the CNN benchmark, the surrogate model has a training error of 0.00609 and testing error of 0.00777. In addition to the off-the- surrogate models we provide, FEDHPO REDHPO-BENCH offers tools for users to build brand-new surrogate models. Meanwhile, we notice the recent successes of neural network-based surrogate, e.g., YAHPO Gym (Pisterer et al. 2022), and we will also try it in the next version of FEDHPO- BEECH.
Raw mode. Both of the above modes, although they can respond quickly, are limited to pre-designed search space. Thus, we introduce raw mode to FEDHPO- SEDHPO-BENCH, where user-defined search spaces allowed. Once TEDHPO-BENCH'S APIs are called with specific hyperparameters, a containerized
24

---

Under review as a conference paper at ICLR 2023
and standalone FL procedure (supported by FS) will be launched. It is worth noting that although use standalone simulation to eliminate the communication cost, raw mode still consumes much more computation cost than tabular and surrogate modes.
H.3 NEW HTYPERPARAMETIN
The FL setting introduces hyperparameters such server-side learning pearnning_att,,,, rate, momentum for FedOpt (Asad et al. 2020) and client-side chocal_update, _tep.. Different FI. algorithms have different parameters, which correlate with hyperparameters related to general MI. procedures. In this section, we first adopt FedProx (Li et al. (2020b) to study the impact of server-side hyperparameters mu, the coefficient of the regular term, the results, And then we compare the landscape of the federated learning method and non-federated method.
H.3.1 TRENDS WITH DIFFERENT REGULARITY IN FEDPROX
To extend the tabular benchmarks with more FL algorithms, we adopt FedProx (Li et al. 2020b to GNN benchmark. Based on Table 8 we tune the server-side hyperparameters mu, the coefficient of the regular term, in {0.1, 1, 1.0, 5.0} to study the trends with different regularity in FedProx. We show the landscape in Figure I0 with learning rate in [0.01, 1.0] and mu [0.1, 5.0], and we observe that when the learning rate is low, the effect of mu has little impact on the accuracy; however, when the learning rate large, the increase of mu can seriously damage the accuracy.
H.3.2 LANDSCAPES ON MI.REELATED HYPERPARAMETERS
In this section, to study the validation loss landscape of the federated learning method (FedAvg) and non-federated method (Isolated), we consider learning rate and batch_ ,izz, the hyperparameters of the ML algorithm, as the coordinate axis to build the loss landscapes. We fix other ML-related hyperparameters weight_decte decay 0.0, dropout to 0.5 for both FedAvg and Isolated, which is the best configuration chosen from the tabular benchmark <CNN, FEMNIST, FedAvg> under 1.0 client poont_sspple___a rate. As the loss landscapes shown in Figure X with learning rate in [0.01, 1.0] and batch size in 16, 32, 64, we observe that the FedAvg with higher learning rate achieves better results, while the non-federtted method (Isolated) prefer a lower learning rate. Their differences suggest the uniqueness of FedHPO's objective functions.
8.6
Bach que
0.6
So
(a) FedAvg
(b) Isolated
Figure 8: Landscape with the hyperparameters of the ML algorithm FEMNIST.
H.4 DATA ANALYTICS
H.4.1 TRENDS IN DIFFERENT PRIVACY BUDGETS
We extend the tabular benchmarks with different levels of privacy budgets FEMNIST and Cora. To explore the trends of optimal configurations under different privacy budgets, adopt NbAFL. (Weil
25

---

Under review as a conference paper at ICLR 2023
et al. 2020 with {1, 10, 20). We observe that the best configuration varies under different levels of privacy budgets in and Cora, as shown in Table Under different privacy budgets, large step_ size all leads to good performance. However, when the noise is intense, higher learning rate is preferred, while lower learning_ran paae will perform better when the noise is weak.
1.0
5.6
learning rate
weight decay
dropout
step size
step_size
Test Acc. (%)
1.0
0.001
0.5
7
63.87 6.38
0.59948
0.0
0.5
87.02 1.16
20
0.59948
0.001
0.5
6
87.30 0.54
3.0
Tatter
Noma lond regnet
Figure 9: ECDF on Twitter.
Table 10: Best configuration with different levels of privacy budgets in Cora.
H.4.2 ERRORS SURROGATE BENCHMARKS
we report the regression error of training surrogate model in Table 12 Meanwhile, we present the mean rank over time of optimizers with surrogate modes in Figure 22 and Figure 23] Compared to the results of tabular modes in and [4] B0Gp shows good performance in both modes, while Random Search does not. This show the consistent performance of the same optimizer when it interplays with surrogate and tabular benchmarks.
As we mentioned in Section H.2
Model
Dataset
Algo.
Train MAE
Test MAE
CNN
FEMNIST
FedAvg
0.00609
0.00777
FedAvg
0.04724
0,05454
CoLA
FedOpt
0.02426
0.02959
BERT
FedAvg
0.02597
0.03227
SST2
FedOpt
0,02802
0.03166
FedAvg
0.04702
0.04839
0.
Cora
FedOpt
0.05703
0.05893
22
9
GNN
CiteSeer
FedAvg
0.01334
0,01381
FedOpt
0.01652
0.01717
FedAvg
0.04042
0.04148
PubMed
FedOpt
0.04816
0.05699
Figure Landscape with the different
regularity of FedProx on Cora.
Table 12: The regression error of surrogate models.
H.4.3 VARIANCE OF DIFFERENT SAMPLE RATE
As we build our tabular benchmark from FL courses executed in docker images provided by FS, we can fully reproduce all the results given the same random seed in raw mode. Other than that, study the noise of different federated optimization, we analyze the variance of validation loss with 500 rounds under different sample rates FEMNIST. And the mean standard deviation validation loss is [1.94 1..94 1.7e-2, 1.728e-2, 1.715e-2, 1.43e-2) with sample rate (0.2, 0.4, 0.6, 0.8, 1.0), which shows that the higher sample rate tends to have lower variance. The reason is apparent: the lower sampling rate, the more inconsistent the set of clients sampled during the training process leads to this error.
H.4.4 ECDF WITH DIFFERENT HETEROGENEITY
We extend our LR benchmarks with different heterogeneity settings. As discussed in Appendix D we split the tabular dataset with LDA. whose a is in (0.1, 0.5, 0.7) (the smaller the alpha, the more the heterogencous). We show the ECDF of the normalized regret of evaluated configurations with different a in Figure [T] which shows that as the a decreases, it harder to find a good configuration. This phenomenon shows the necessity of tuning hyperparameters in FL with heterogeneous data.
26

---

Under review as a conference paper at ICLR 2023
0.8
0.8
0.8
0.6
credi-g vehicle
0.6
credit
0.6
medltg
vehicle
00.4
Hond-trandd.
0.2
Australian
0.2
Australion
0.2
Australlan la
segment
0.0
0.2 0.4
0.6
0.8
1.0
0.0
0.2
0.6
0.8
1.0
0.0
0.2
0.4
0.8
1.0
Normalized regret
Normalized regret
Normalized regret
(a) a 0.1
(b) a 0.5
(c) a 0.7
Figure 11: Empirical Cumulative Distribution Functions with different heterogeneity in LR benchmark.
I
MORE RESULTS
In this section, show the detailed experimental results of the optimizers FEDHPO-BE -EEOHH benchmarks under different modes. We first report the averaged best-seen validation loss, from which the mean rank time for all optimizers be deduced. Due time and computing resource constraints, we do not have complete experimental result of the raw mode, which we will supplement soon as possible.
1.1 TABULAR MODE
Following Section we show the overall mean rank overtime all FedHPO problems with FedOpt, whose pattern is similar to that of FedAvg in Figure Then, we report the final results with FedAvg and FedOpt in Table and [4] respectively. Finally, we report the mean rank over time in Due to time and computing resource constraints, the results on CNN benchmark are incomp ete (lacking that with FedOpt), which we will supplement soon possible.
Figure 13 21
10
Mean rank
Mean rank P
2
16-4
16:
164
161
10-3
16:1
budget
budget
budget
BBO
(b) MF
(c) All
RS
BOGP BOHB
BORF
BOKDB
DE TPEμB
HB
DEHB
TPE
Figure 12: Mean rank over time on all FedHPO problems (with FedOpt).
Table 13: Final results of the optimizers on tabular mode with FedAvg (lower is better).
Nandmmak
INI
17M
CNNquoos
0.88790000011
0.0855200065
50504420006
0,0028330004
0,0026600062
0,9945210009
04996500061
0.516221
0.514Ren
ONNNaa
83355200034
83335257000
0333550
0,304480016
664695300333
0644220.0006
61699930009
0L64422310000
0164536300001
663874003077
0154
06402201000
0635
05145100116
0S1190006199
0.9001
01809411/012
0.0944L00010
0.906100177
0.4044L 01.9
TMG.
2017
5992240 3547
77561e0 2536
2.627161
1.7519636000
386862.
L6384ch
3.113622 9836
2.13461 1068
265fiel 1446
LR
11 145800
8 0350820000000
U 545552200000
0.50876010008
UUS48 0000
11. 0.58066000047
0.35
0.5133411 1909
0.5157411001
LR
0552944000006
0520010000000
0052955000006
125892200004
0.52918200000
O 0.52922300005
0.52985.0004
0.52002000001
0 0055
0.53872n 0185
044738200335
846644010088
0.4221200112
0.8443200210
0.667RL0010
0.4147300122
0477072112003
043766000077
"ITF:
11 10ST
IE 1010--1000
MI P.
I 7709 40
I 2L079-
ET 2209400176
03IT3A 1407
04191-11099 (M1
17667-0-11 DI
0.2861.0000000
8 0.303720U1000
1835542200108
155554200088
0.34320U0074
O 0.3895630000
0.2011200009
0408421104477
0,3979
0.3988211 DOOS
MLP DI1
40144620113
0551255500076
0.5139436011
0.5199EIL011S
0,5059gf
0.519228
0.54990199
MIL..
114410 01:9
D 0.18S11007
0.1912L1..005
0.1866LE
0.176L 0LD4VS
ML..
B 0138900012299
015171000147
0120281
012855100045
0.244000117
0,274511.0054
27

---

Under review as a conference paper at ICLR 2023
Table 14: Final results of the optimizers on tabular mode with FedOpt (lower is better).
beodunak
R5
80gz
BOar
DE
HB
SOHB
DENB
37E
TPE
-0049
0-8325-001155
046001-010000
0004
303022000111
0.3256210005
0.68TTs 0M33
0.549751
M350
B 86633f000072
0.645een 0025
0.64225 0034
06463e IX29
D.Ge8tel 0006
e 6.6455el..00 2007
84722400019 18
04771940 1006
04118:11000
0.6899 01
64318 IMC
149664990006
0 0.44001 0067
000-000 1015
US15500111 19
0.N3550 1000
0.30
2.252a
2.22tsnn553
2,347241
0.3144ah 8655
0.55238500078
0.558550 0033
0 0.50025000022
0.5498020012
0.504400.00 9039
0.5476500177
6.5222n 0036
A.5612600022 02201
0.355700.001
LR
0.51 (9050
$058230100
0.508420 M037
0.5133200005
0,5007200000
0.502200 200004
CA978at 0000
0.4081800049
0.500420.005 01.5
3999-010144
0.1721-0110
D.
0.300--U -00000
014887405487
MILF.
025012200012
0.291445000044
03901220
02902200002
0.9011800007
C3911a
MEPS
DESSS 1800
0.846-11
10167 1776
0.9499
1216
1:1224000164
0.1862 IRTS
B 1877 10001001100
D. 0.386711100055
D (960
(44001 00000
D 8.000).B.00 L0031
MLP II:
0.2219000016
041977
84011-00005
D.30Seel 1309
04115-4100000
C4215
0.40020003340
0.500ce1 1980
MLP
0.116968/01226
0.0915e0
166440000000
0.0891 01.50
0.536a 0270
011051=000155
MLP
0.296348.0224
1220142002215
0240
030252000442
0.2779L0
0.279930.02.1
0.257,0 0220
0.25 (RL0.0015
Moin cakk
Moon
Fraction burge!
Fracton of busget
Practon of budget
(a) ALLFENNIST
(b) BBOTEMNIST
MF MFPEENIST
RS
BOer BOHB
BORF DEHB
BOXDE TPEND
DE TPEнB
HB
Figure 13: Mean rank over time CNN benehmark (FedAvg).
Moan 2IR
Not ank
10.3
1m
Tec1
1--1
1
3
budget
(a) BLL
(b) BBOBERT
(c) MFBERT
Mian 0a1
Moapronk
Hea makk
10:3
1m.2
10.2 20-3
Fraction
Fraction
Fraction
(d) ALL
(e) BBOCaLA
( MFCOLA
Noat 177
Tracttion Sudger
Fraczan of bucget
Traction of budget
(g) ALLsSTa
(h) BBOsST
(i) MFSST=2
RS
BOGp BOHB
BORF DEHB
BOKDE
DE TPEnB
HB
TPE
Figure 14: Mean rank over time BERT benchmark (FedAvg).
28

---

Under review as a conference paper at ICLR 2023
Moon rane
104
Pratiion Lunges
Frccion of budget
Fraction budget
(a) ALLRERT
(b) BBOBERT
(c) MFBERT
Mean SIR
Moan cukk
1044
Frodtion budge
Fractin of budget
Fractio of budget
(d) ALL
(e) BBOCaLA
(f) MFCOLA
10
Mosn cak
263
101
104
123
IP3
Dudge!
budget
(g) ALLSsE2
(h) BBOssU2
(i) MFSSE2
RS
B0er BOHB
BORF DEHB
BOxDE
DE TPERB
HB
TPEND
Figure 15: Mean rank over time on BERT benchmark (FedOpt).
1.2 SURROGATE MODE
We report the the final results with FedAvg on FEMNIST and BERT benchmarks in Table 15 Then we present the mean rank over time of the optimizers in Figure 22 and Figure 23
Table 15: Final results of the optimizers in surrogate made (lower better).
benchmark
RS
B0er
B0KDE
DE
HB
ВОНB
DEHB
TPEnp
TPEμи
CNNEMNNIST
0.0508
0.0478
0.0514
0.0492
0.0503
0.0478
0.048
0.0469
0.0471
0.0458
BERTsTT2
0.4909
0.4908
0.4908
0.4908
0.4908
0.4908
0.4908
0.4917
0.4908
0.4908
CaRT
0.5013
0.4371
0.4113
0.487
0.444
0.4621
0.4232
0.4204
0.3687
0.3955
29

---

Under review as a conference paper at ICLR 2023
Moan ont
Frantion of burgee
Fraction of busget
Fraction of bunget
(a) ALLoNN
(b) BBOCxN
MFGNN
00
Moan Noaraan
30.4
Proction budge:
Fraczin of budgut
Rractin of budges
(d) ALLcam
(e) BBO
(1) MFcon
5
Woan Waanrank cak
2012
1822 12.
12:4
161 Fration budgec
Dodge:
budget
(g) ALLensssr
(h) BBO
(i) MFCnnsee
5
Moan cark
Wath
EC 2012 101l Fradtion
123 101
1P- Praction
(i) ALLpusMed
(k) BBO
(I) MFPaMMedd
RS
BOer BOHB
BORF DEHB
BOKDE TPEND
DE
HB
TPE
Figure 16: Mean rank time on GNN benchmark (FedAvg).
30

---

Under review as a conference paper at ICLR 2023
Moan ont
Frantion of hurgge
Fraction of busget
Fraction of burgget
(a) ALLoNN
(b) BBOCxN
MFGNN
Vean 1812
Noarane
30.4
Prodtion budge:
Fracton of budgut
Rracton of Dudges
(d) ALLcam
(e) BBO
(1 MFcon
Wuan Maanrank cak
Men
103 20-2
123 18
12:4
Dodge:
budget
Frottin budget
(g) ALLensser
(h) BBOciusesr
(i) MFcnnsser
Moan cark
Wath
E-d 2012 1011 Fradtion
123 1012
124
LES Praction
(j) ALLpusMed
(k) BBO
(I) MFPaaMedd
RS
BOer BOHB
BORF DEHB
BOKDE TPEND
DE
HB
TPE
Figure 17: Mcan rank over time GNN benchmark (FedOpt).
31

---

Tell Under review as a conference paper at ICLR 2023
Moon Pan
Fration Lunges
Frccion of budget
Foztion wugge
ALLZR
(b) BBOLR
(c) MFLR
Moan SAR
Moan rak
104
Frodtion budge:
Fracion of budget
Fraction budget
(d) ALLa
(e) BB0
(f) MF 510geeML
Mea TaRK
Monn cak
Menn
102
budge!
budget
(g) ALLoxopenML
(h) BBO
(i) M MFssppesML.
Mea T9RT
Monn ank
20-4
20.5
Fractiin
20-2
20.4
I012 Fraction
Praction
(j) ALL101010mmM
(k) BBO101010pemML
(1) MfF11010peeML
Mean 1ar2
Manyank
Hon
Proction budge:
Fracton of bugget
Fractin of burget
(m) ALL106s100peMM
(n) BBO 1800165550peaMM
Wean Manrank
Vean
104
Prodtion bodge:
Frccion of budgut
Roction of budget
(p) 1LL14G2200eaM
(q) BBO 1668210pMMM..
(r) MF MF 1082100eeMM.
Moan cak
IE'3 J012
1/44
183
budge:
budget
budget
(s) ALL14asz2Opennt
(t) BBO, 468120penMLL
(u) MF 14682200peMM
RS
BOGP
BO
BOXDE
DE
HB
BOHB
DEHB
TPEND
TPERB
Figure 18: Mean rank time LR benchmark (FedAvg). 32

---

Under review as a conference paper at ICLR 2023
Voan
2012 Faation budged
Fraciin budget
Froddii of budget
(a) ALLLR
(b) BBOLR
MFLR
Mean SAR
Moan ruk
104
Frastinn bugge
Frctoon of budget
Fractio of budget
(d) ALLa
(e) 8B0
(f) MF 510genML
Mea Taan
Monn aak
2632
101
104
IP2
Duege!
budget
(g) ALLazopenML
(h) BBO
(i) M MFsappesML
10
Meah T9RK
Monn ank
2
20-2
2014
I012
Fractiin
Fraction
Fraction
(j) 1LL101010mmM
(k) 1BO101010peMML
(1) MM 10100000M
50
Mean nar2
Manyank
Honnre
Proction budge
Fracton of budget
Fractio of budget
(m) ALLSS0S10
(n) BBO 1400168550peaMML
MF1 1081
Veam nnn
Mean Man
104
Proction budge:
Frccion of budgut
Prction of budget
(p) AGL1462200MMM
(q) BBO 1468210pmmM.
(r) MF MF 10821000eMM.
Moan cakk
10'3 J03
104
183
10
124
Iel
budge:
budget
budget
(s) 1LL.465220p
(t) BBO 4681200MMML
(u) MF 14682200peML
RS
BOGP
BO DEHB
BOXDE
DE TPERB
HB
BOHB
TPEND
Figure 19: Mean rank time LR benchmark (FedOpt). 33

---

Under review as a conference paper at ICLR 2023
Vocm cant.
Voxi
Frationn Ludges
Facionn budget
Foztion budget
ALL
(b) MBO
(c) MFMPP
Mean SaR
Mouan rukk
1044
Frodtion bugger
Fration of budget
Fraction budget
(d) ALLa
(e) BB0
(f) MF M100enMMM
Mea TaRe
Moin cak
104
IPZ
Dudge!
budget
(g) ALLoadpenML
(h) BBO
(i) M MFsappeeMM
08 5.
Monn ank
Moep mny.
20.5
20.2 10.4
Fractiin
Fraction
(j) ALL101010mmM
(k) 10010100)
(1) 10000010pemML
Oean 1912
Mon ann
Proction budge:
Fractio of budget
Fraction of budget
(m) ALL106s1S0peM
(n) BBO 14B0L
MF11 1051
Veam nnt
Moan DA
104
Prodtion budge:
FOctOn of budgut
Poction of budget
(p) AGL1462200MMM
(q) BBO 16622100MMM
(r) MF 1011222200.
8 5
Wenn min
Moan rakk
J03
104
124
lell
budge:
budget
budget
(s) ALL 1LL1465220enML
(t) BBO 4681200mMML
(u) MF 14682200peMML
RS
BOGr
BO DEHB
BOXDE
DE
HB
BOHB
TPEND
TPERB
Figure 20: Mean rank over time MLP benchmark (FedAvg).
34

---

Under review as a conference paper at ICLR 2023
Mocn Pant.
Fration Ludge
Facionn of budget
Foztion bugge
ALLALP
(b) BBOMLP
(c) MFMPP
Mean Matn-ane
Mouan rukk
Moni
104
Frodtion budger
Facton of budget
Fraction of budget
(d) ALLanO
(e) BB0
(f) MF 510genML
10
Mea mane
Mosn cakk
Moen
20.2 101
burge!
budget
(g) ALLoxopenML
(h) BBO
(i) M MFssopesMLL
Monn ank
20-4
2g.5
20-2
10.4
1012
Fraction
Fraction
Praction
(j) ALL101010pmM
(k) BBO101010penML
() 10101010pemMM
Mean nart
Mon
Proction budge
Fractio of budget
Fractinn of burgee
(m) ALL1o0s1S0peML
(n) BBO 10881
MM11 10sspeMM
R.R.
Veam nnt
Moan Mnn
How Montrnn
10.4
Prodtion budge:
Factinn of budgut
Praction of budget
(p) ALL1 Aos
(q) BBO 1668210pMMM..
(r) MF M01
Oeann 1012
Moan ant
How
Trantion of burgge
Fractan of bucget
Traction of burgee
ALL ALL1466220peMLL.
(t) B800200220pmmM
MF 1468220PeaML
RS
BOGp BOHB
BO DEHB
BOXDE
DE TPEнB
HB
TPEND
Figure 21: Mean rank over time on MLP benchmark (FedOpt).
35

---

Under review as a conference paper at ICLR 2023
Moan SAC
Moon culk
104
153
11:
Frodtion Luige:
Faction of budget
Fractio budget
(a) ALLEEMNIST
(b) BBOVEMNIST
(c) MFFEMNIST
RS
BOGr BOHB
BORF
BOxDE
DE
HB
DEHB
TPE
TPERB
Figure 22: Mcan rank over time on CNN benchmark under surrogate mode (FedAvg).
Moan on
Mopp
Fraction budge:
Fraction of budget
Praction of budget
(a) ALL
(b) BBOsERT
(c) MF MFBERT
BERT
10
Honn 1952
Mos aa
Now
3m-1
1
1
Frantion burger
Fraction of budget
Fraction budgee
(d) ALL
(e) BBOcaLA
(f) MFcOLA
Muan ont
204
1:3
2C32
10-
1-3
1m2
16c
1
161
(g) ALL
(h) BBOSST2
(i) MFsST2
RS
BOGr BOHB
BORF DEHB
BOxDE TPEND
DE TPERB
HB
Figure rank over time on BERT benchmark under surrogate mode (FedAvg).
36


Reviews:
[{"review_id": "iBoHeQZQXZ", "reviewer": "Reviewer_BCqK", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a benchmark suite for hyperparameter optimization (HPO) specifically tailored to the problem of federated learning. They argue that the problem features several distinct characteristics, which are not accounted for in standard benchmarks, but should be. FedHPO-Bench was designed to alleviate this situation with the corresponding differences in mind. In order to increase the size of available federated HPO problems, the authors also present a way to generate a federated HPO problem from a standard one. In an experimental study, the authors show how standard and federated HPO approaches perform on federated HPO problems from the proposed benchmark.", "summary_of_the_review": "Although I very much like the idea of investigating the HPO problem for federated learning, I have to suggest a rejection of the paper for the following reasons: \n\n1. The overall story of the paper is not well written making it unclear a) what the federated HPO problem actually is and in how far it differs from the standard HPO problem, b) why existing HPO methods are not well suited, c) why this benchmark is needed and what it offers in detail.\n1. The evaluation presented in the paper has several flaws, which should be fixed.\n", "strength_and_weaknesses": "### Strengths\n* As federated learning becomes more and more important in practice, considering HPO for federated learning is an important research direction. The idea of having a corresponding benchmark is in general an important contribution. \n* Language-wise the paper is well-written.\n\n### Weaknesses\n* The overall structure of the paper is not very compelling. The authors spend quite some space on explaining the differences between the standard HPO and the federated HPO setting without properly defining the latter. There is a vague description of the overall process on Page 7 (directly before the experiments), which should be a) much clearer and b) much earlier in the paper to convince the reader that the two problems are truly distinct. Overall, I am missing a clear path to follow through the paper, which makes it not very convincing.\n* The points made by the authors regarding the differences between the standard HPO and federated HPO problem are not convincing to me. Each difference detailed on Page 3 can very well be modeled within the realm of standard HPO: \n    * Function evaluation in FL: The argument for FL algorithms being iterative is not well supported. Many standard ML models such as neural networks are also iteratively trained making FL not very different from it. However, we still tune neural networks using standard HPO methods. Similarly, the heterogeneity between FL clients is not well elaborated on. Why does this change the HPO problem in particular?\n    * Hyperparameter dimensions: New hyperparameters are not a reason for a new HPO formulation and benchmark. If this was the case, we would need a new benchmark with each new ML algorithm or variant thereof. Showing the figure in Appendix H.3.1 is not very convincing as different HPO problem instantiations (e.g. with different algorithms) also can have different shapes. Moreover, the figure is rather destructive to the overall motivation for the federated HPO problem as it suggests that the optimal region is in a very large plateau, which can easily be found by random search. In fact this observation is even corroborated in the experiments later on where the authors argue that none of the standard HPO approaches shows a significant improvement. In principle, this is also not such a new insight as something similar can also be observed for HPO problems in general [7].\n    * Fidelity dimensions: We can have different fidelity dimensions in standard HPO as well. Moreover, the argument conducted simply shows that some of the hyperparameters are correlated - an issure that we also have on standard HPO. \n    * Privacy: While this is certainly an important property to be analyzed, it seems that this is, once again, simply controlled via a (set of) hyperparameters, which could also be modeled in the standard HPO setting. Privacy simply seems to be a second objective suggesting that federated HPO is actually a multi-objective HPO problem. Unfortunately, this is never mentioned by the authors. \n    * Fairness: First of all, fairness often has a very different notion in the area of machine learning nowadays, which might be very confusing for readers of this paper. The authors should make much clearer, what they mean by privacy as was done in the corresponding references given by the authors. Second, the description here suggests that federated HPO should be multi-objective. \n    * Concurrent exploration: Once again, this is also something that exists in standard HPO and is in fact quite beneficial to perform in the standard setting. For example, [2] suggests a UCB based acquisition function, which can be used in concurrent exploration as done in D-SMAC [3]. \nOverall this makes the description of differences not very convincing. To be clear: I am very sure that the problem features several important characteristics, which make it inefficient to treat it using standard HPO methods. But this is simply not well conveyed by the authors. Most importantly, the overall setup, i.e. how the process works (description directly before evaluation) already suggests that tuning should NOT be performed as standard in the sense that a single function evaluation consists of several rounds between the clients and the master until convergence, but that the iterative process itself should be abused. In fact, this suggests the usage of multi-fidelity approaches or, if the tuning should be performed duringthe training of the federated approaches as it is suggested in this work, using methods from dynamic algorithm configuration [4,5]. The authors should really consider completely rethinking the way they present (a) the federated learning problem and (b) its differences to the standard setting.\n* It is unclear why the authors do not build upon existing benchmarks for federated learning such as [1] and why existing federated HPO methods have not been well benchmarked as claimed by the authors. Once again, this does not make a compelling point for the need for this work.\n* The evaluation is not very compelling. \n    * Firstly, RQ3 is raised in the main paper, but never again mentioned there. It is answered in the appendix, but this makes the main paper not self-contained. The authors should either answer the question within the main paper or not raise it there in this form in the first place, but completely defer it to the appendix. \n    * Secondly, I do not see on the basis of which federated HPO problems RQ1 is answered. While Figure 1 looks interesting, the results are not well put into context. As mentioned earlier, Appendix H.3.1 suggests that the problem can be solved well by random search. Correspondingly, the result \"Comparing these advanced optimizers with their baselines, only BOGP, BORF, and DE win on more than half of the problems but have no significant improvement, which is inconsistent with the non-FL setting.\" is not very surprising, which is not mentioned. Moreover, the second result \"Meanwhile, no MF optimizers show any advantage in exploiting experience, which differs from non-FL cases.\" cannot be verified from my point of view. It seems that some optimizers do work better than others at earlier stages of the optimization from Fig. 4. Other than that, I am not sure how you come to that conclusion from the results shown in the paper. \n    * Thirdly, the experiment to answer RQ2 is limited to only a single federated HPO task, which seems strange considering that the authors suggest a complete benchmark here, which should feature many more problems. Moreover, it is unclear which loss function is displayed in Figure 5 (validation of what loss?) making the figure meaningless as it cannot be properly interpreted. \n    * Fourthly, the evaluation completely lacks error bands or standard deviation values to put the result into context.\n* Important details regarding the benchmark are missing in the description. For example, what are the benchmark problems contained? How many problems does it contain? How exactly does it support the user in evaluating federated HPO algorithms? Which algorithms are already implemented? Will the benchmark be further extended by the authors? \n* There exist several notational and descriptional problems in the work. For example, the HPO problem description in the introduction is not great as the in the end one is not interested in optimizing the validation performance, but rather the test performance. In fact, one is actually interested in optimizing the out of sample error, which is approximated by the test error. Similarly, several symbols of Eq. 1 are not defined: What is S_{down}, S_{up}, B_{down}, B_{up}?\n\n## Minor remarks\n* Figure 1: text \"$\\lambda, b$ as query\" is too long and is displayed above another part of the figure\n* p.6: \"As a result, the learning dynamics of FL change, and thus different privacy budgets often correspond to different HPO objective functions\" -> I disagree. The objective function does not change with a different privacy hyperparameter, as the objective is always to maximize performance, as far as I can tell. What might change is your additional objective, if you explicitly make privacy an objective, but I do not read anything regarding multi-dimensional HPO in the paper.\n* p.6: model execution time calibration -> The authors should consider citing [6], where something similar is done to adhere to varying hardware. \n* Eq. 1: Several symbols undefined -> not possible to understand the idea from the main paper here\n    * What is S_{down}, S_{up}, B_{down}, B_{up}\n* p. 7 last sentence: unclear - what is \"#round\"? \n\n### Typos: \n* p.4: \"much helpful\" -> very helpful\n* p.7: \"it needs to synchronous\" -> \"it needs to synchronize\"\n\n[1] He, C., Li, S., So, J., Zhang, M., Wang, H., Wang, X., Vepakomma, P., Singh, A., Qiu, H., Shen, L. and Zhao, P., 2020. Fedml: A research library and benchmark for federated machine learning. arXiv preprint arXiv:2007.13518\n[2] Donald R. Jones. A taxonomy of global optimization methods based on response surfaces. Journal of Global Optimization, 21(4):345–383, 2001.\n[3] Frank Hutter, Holger H. Hoos, & Kevin Leyton-Brown. Parallel algorithm configuration. In International Conference on Learning and Intelligent Optimization, pages 55–70. Springer, 2012.\n[4] Biedenkapp, André, et al. \"Dynamic algorithm configuration: foundation of a new meta-algorithmic framework.\" ECAI 2020. IOS Press, 2020. 427-434.\n[5] Adriaensen, Steven, et al. \"Automated Dynamic Algorithm Configuration.\" arXiv preprint arXiv:2205.13881 (2022).\n[6] Mohr, Felix, et al. \"Predicting machine learning pipeline runtimes in the context of automated machine learning.\" IEEE Transactions on Pattern Analysis and Machine Intelligence 43.9 (2021): 3055-3066.\n[7] PUSHAK, YASHA, and HOLGER H. HOOS. \"AutoML Loss Landscapes.\" ACM Transactions on Evolutionary Learning and Optimization (TELO) (2022).\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "VumUnWodUh", "reviewer": "Reviewer_QKdH", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduced a new benchmark suite for Federated HPO called FedHPO-Bench. The FedHPO-Bench is considered distinct from other HPO benchmark suites such as HPO-Bench because of the inherent requirements for FL algorithms.\n\nThe paper first identified the need for a FedHPO benchmark suite because of its distinct separation between client and server hyperparameters, and then provided some results to evaluate the performance of their benchmark suite. The analysis of the performance differences between separated client and server hyperparameters and standard HPO algorithms defined the majority of the reasoning behind the separation of HPO and FedHPO, and the corresponding need to create their benchmark suite. The authors clearly described this distinction, but did not provide convincing reasoning behind the need for their benchmark suite. \nAs stated in this paper, the benchmark suite is very closely related to HPO-Bench, and does not appear to provide a truly novel solution. Instead, it appears as a small addition to an already existing technology.", "summary_of_the_review": "I would not recommend this paper for acceptance due to the fact that it does not provide a novel enough contribution to the field. Although it may definitely improve benchmarking for FedHPO, it is specific to only a single field, and it provides few novel aspects of HPO benchmark suites in general.\n\nWhile I can see how there may be some need for a solution to easily benchmark FedHPO algorithms, I remain unconvinced that this is novel enough to warrant publication.\n", "strength_and_weaknesses": "trengths: This paper clearly described the the current FL landscape. \nThe explanation of its extensibility, and the experimental results give a clear idea of the given justification of FedHPO-Bench as a novel research path.\n\nWeaknesses: There were some instances throughout the paper where information was not put forth in an easily understandable format. For instance, Equation 1 provides could be summarized by saying that the total time is bounded by the summation of the slowest client’s computation, the server’s computation, and data transfer speeds. Figure 1 also introduces another issue like this, wherein the interface design is difficult to understand. Also, on page 19 (Appendix E), citing Wikipedia isn't a usable source for an academic conference due to the nature of the source of its information.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "UsfzBxKVP5b", "reviewer": "Reviewer_eSy6", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new benchmark suite for federated hyperparameter optimization (FedHPO), called FedHPO-bench. FedHPO-bench focuses on three main desiderata, which are: (1) comprehensiveness -- which is the diversity of tasks; (2) flexibility -- which is the customizability of the federated environment; and (3) extensibility -- which is the ease of evolving the benchmark suite. The paper demonstrates the usefulness of the proposed benchmark suite on a substantial collections of experiments and through which reveals interesting insights about FedHPO.", "summary_of_the_review": "I recommend a marginally below acceptance for now, because I'm not convinced by the empirical demonstration. I hope that the authors can provide me with good reasons to upgrade my score, because the proposed bench seems like a fundamental contribution if executed well.", "strength_and_weaknesses": "Strength: The paper is well motivated. The proposed software has great practical values and is potentially impactful in the growing field of Fed Learning. While I believe there is much more to be done, the current implementation might be sufficient as a prototype.\n\nWeakness: My main concern is that the empirical studies are not very well designed. The authors mentioned that there are three desiderata for a good benchmark suite. The experiments should focus on demonstrating these three desiderata. The insights described in Sec 4 are good to know, but not entirely relevant. My specific comments regarding this are as follows:\n\n-- Comprehensiveness: In my opinion, this is the most well demonstrated desiderata. The authors show in Fig. 2 a wide range of hyperparameters-performance distributions, obtained over different categories of model types. But these are confined to the vanilla FL setting. It would be much better if the authors can demonstrate on other interesting settings, such as increasing/decreasing the heterogeneity of the federated tasks. For example, I would expect that increasing the heterogeneity will make the ecdf curves of most datasets be more like that of the PubMed dataset (i.e., harder to find a good configuration that works well on all clients). Many existing papers on non-IID FL have done this study before, so I believe it would be useful if this feature is implemented as an environment parameter that users have control over.\n\n-- Flexibility: None of the experiments really demonstrate how easy it is to use the proposed software to generate interesting experiments. Varying the heterogeneity (as pointed out above) would be a great feature to have (and to conduct demonstration for). In addition, I also find it unintuitive to customize the objective function (to reflect different privacy budget/fairness level -- Sec 3.2). Personally, I think it is better to separately implement DP and fairness metrics and let the users decide which objective function to use. As the system is designed to be extensible, the dev team can incorporate implementations of privacy and fairness preserving FL objectives later. \n\n-- Extensibility: This is arguably harder to demonstrate, and I think the description in Sec 3.3 did a fairly good job to convince me that the bench can be extended in principle. My concern is that while three concurrent exploration methods were implemented (FedEx, FTS, personalized FedEx), it seems that only FedEx was demonstrated in the experiments. In addition, does the unification result in other meaningful concurrent exploration methods by varying the sync, update & aggr operations? I would recommend the authors trying several sensible configurations and show that they yield decent results (not necessarily better) to confirm that the unification is indeed useful. That will also demonstrate the extensibility of the bench. Otherwise, it seems pointless to unify these methods.\n\nOther concerns:\n\n-- The authors should make clear the distinction between HPO and FedHPO in the problem setting (Sec 2.1 or beginning of 2.2). Is the goal learning one HP setting for all clients or a personalized setting for each client?\n-- I do not get the discussion about the fidelity dimensions (Sec 2.2). I understand what they are and they seem like parameters of the federated environment, which FedHPO-bench should give users control over (i.e., flexibility). I also do get that it is interesting to demonstrate the effect of varying these parameters (which is done in Appendix G), but it is unclear from the description whether FedHPO-bench does have this customization feature? If so, the authors should explicitly state it.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject