Unlocking the Potential of Global Human Expertise
Authors: Elliot Meyerson
Olivier Francon
Darren Sargent
Babak Hodjat
Risto Miikkulainen
Venue: NeurIPS 2024 poster (2024)
Keywords: Human-AI collaboration
Evolution
Distillation
Neural Networks
Primary area: machine_learning_for_other_sciences_and_fields
TL;DR: Framework to address global challenges: (1) Distill diverse human solutions, (2) Recombine and elaborate upon them with evolution.

Abstract: Solving societal problems on a global scale requires the collection and processing of ideas and methods from diverse sets of international experts. As the number and diversity of human experts increase, so does the likelihood that elements in this collective knowledge can be combined and refined to discover novel and better solutions. However, it is difficult to identify, combine, and refine complementary information in an increasingly large and diverse knowledge base. This paper argues that artificial intelligence (AI) can play a crucial role in this process. An evolutionary AI framework, termed RHEA, fills this role by distilling knowledge from diverse models created by human experts into equivalent neural networks, which are then recombined and refined in a population-based search. The framework was implemented in a formal synthetic domain, demonstrating that it is transparent and systematic. It was then applied to the results of the XPRIZE Pandemic Response Challenge, in which over 100 teams of experts across 23 countries submitted models based on diverse methodologies to predict COVID-19 cases and suggest non-pharmaceutical intervention policies for 235 nations, states, and regions across the globe. Building upon this expert knowledge, by recombining and refining the 169 resulting policy suggestion models, RHEA discovered a broader and more effective set of policies than either AI or human experts alone, as evaluated based on real-world data. The results thus suggest that AI can play a crucial role in realizing the potential of human expertise in global problem-solving.

Unlocking the Potential of Global Human Expertise
Elliot Meyerson' Olivier Francon'
Darren Sargent' Babak Hodjat' Risto Miikkulainen' 1.2
'Cognizant Al Labs 'The University of Texas at Austin (elliot. wew.oo.mmyyrrsonnoooiiiiii olivier francon, darren. sargent, babak, Soo.mtiiccnnnnnnnt..cmmmm
Abstract
Solving societal problems global scale requires the collection and processing ideas and methods from diverse sets of international experts. As the number and diversity of human experts increase, does the likelihood that elements in this collective knowledge can be combined and refined to discover novel and better solutions. However, it is difficult identify, combine, and refine complementary information in an increasingly large and diverse knowledge base. This paper argues that artificial intelligence (AI) can play crucial role in this process. An evolu- tionary AI framework, termed RHEA, fills this role by distilling knowledge from diverse models created by human experts into equivalent neural networks, which are then recombined and refined in population paaset search. The framework was implemented formal synthetic domain, demonstrating that transparent and systematic. It was then applied to the results of the XPRIZE Pandemic Response Challenge, in which over 100 teams of experts across 23 countries submitted models based on diverse methodologies to predict COVID-19 cases and suggest non non-pharmaceutial intervention policies for 235 nations, states, and regions across the globe. Building upon this expert knowledge, by recombining and refining the 169 resulting policy suggestion models, RHEA discovered broader and more effective set of policies than either AI or human experts alone, as evaluated based on real-world data. The results thus suggest that can play a crucial role realizing the potential of human expertise in global problem-solving.
Introduction
Integrating knowledge and perspectives from a diverse set of experts is essential for developing better solutions societal challenges, such as policies eurb ongoing pandemic, slow down and reverse climate change, and improve sustainability 33
64). Increased diversity in human teams lead improved decision-making (25 62 but the scale of the problem and size of the team increases, it becomes difficult to discover the best combinations and refinements of available ideas (37) This paper argues that artificial intelligence (AI) can play a crucial role in this process, making it possible to realize the full potential of diverse human expertise. Though there are many AI systems that take advantage of human expertise improve automated decision-making 4 31 66), an approach to the general problem must meet a set of unique requirements: It must be able to incorporate expertise from diverse sources with disparate forms; must be multi-objective since conflicting policy goals will need to be balanced; and the origins of final solutions must be traceable so that credit can distributed back to humans based on their contributions. An evolutionary Al framework termed RHEA (for Realizing Human Expertise through AT) is developed in this paper to satisfy these requirements. Evolutionary AI, or population based search, is biologically-inspiee
41
57
63
method that often leads to surprising discoveries and insights 55] 15 39
48
67 it is also natural fit
here since the development of ideas in human teams mirrors an evolutionary process [1 17 38
Implementing RHEA for particular application requires the following steps (Fig.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

context
Natrrcal data
outcomes
actions
progesphic region
predictor
future cases
prescriptor
fiture scheuiee
grogr region
polccy
shtiggetry
Define
2. Gather
Evolve
Dishtl
Figure 1: The RHEA (Realizing Human Expertise through AI) framework. The framework consists of four components: Defining the prediction and prescription tasks, gathering the human solutions, distilling them into canonical form, and evolving the population of solutions further. a, The predictor maps context and actions to outcomes and thus constitutes a surrogate, or a "digital twin", of the real world. For example, in the Pandemic Response Challenge experiment, the context consisted of data about the geographic region for which the predictions were made, c.g., historical data of future schedules of intervention policies
COVID-19 cases and intervention policies; actions for the region; and outcomes were predicted future cases of COVID-19 along with the stringency of the policy. b, Given predictor, the prescriptor generates actions that yield optimized outcomes across contexts. c, Humans are solicited to contribute expertise by submitting prescriptors using whatever methodology they prefer, such decision rules, epidemiological models, classical statistical techniques, and gradient- neural network that replicates its behavior. e, This population of neural networks evolved further, i.e., the distilled models are recombined and refined in a parallelized, iterative search process. They build synergies and extend the ideas in the original solutions, resulting in policies that perform better than the original ones. For example, in the Pandemic Response Challenge, the policies recommend interventions that lead to minimal cases with minimal stringency.
methods. d, Each submitted prescriptor is distilled into canonical
1. Define. Define the problem in formal manner so that solutions from diverse experts can be compared and combined.
2. Gather. Solicit and gather solutions from a diverse of experts. Solicitation can take the form of an open call or a direct appeal to known experts.
3. Distill. Use machine learning to convert (distill) the internal structure of each gathered solution into canonical form such neural network.
4. Evolve. Recombine and refine the distilled solutions using population-based search to realize the complementary potential the ideas in the expert-developed solutions.
RHEA is first illustrated through formal synthetic example below, demonstrating how this process can result improved decision- massion-making, RHEA is then put to work in a large-scale international experiment on developing non- pharmaceutical interventions for the COVID-19 pandemic. The results show that broader and better policy strategies can discovered in this manner, beyond those that would be available through AI or human experts alone. The results also highlight the value of soliciting diverse expertise, even some of it does not have immediately obvious practical utility: Al may find ways to recombine it with other expertise to develop superior solutions.

---

To summarize, the main contributions of this paper are as follows: (1) Recognizing that bringing together diverse human expertise is a key challenge in solving many complex problems; (2) Identifying desiderata for an Al process that accomplishes this task; (3) Demonstrating that existing approaches do not satisfy these desiderata; (4) Formalizing a new framework, RHEA, to satisfy them; (5) Instantiating first concrete implementation of RHEA using standard components; and (6) Evaluating this implementation in a global application: The XPRIZE Pandemic Response Challenge.
2 Illustrative Example
In this section, RHEA is applied to formal synthetic setting where its principles and mechanics are transparent. It is thus possible to demonstrate how they can lead to improved results, providing a roadmap for when and how to apply it to real-world domains (see App. B for additional details).
Consider policy-making scenario in which many new cassonable-sounding policy interventions are constantly being proposed, but there are high levels of nonlinear interaction between interventions and across contexts. Such interactions are major reason why it difficult to design effective policies and the main challenge that RHEA is designed to solve. They are unavoidable in complex real- world domains such as public health (e.g.. between closing schools, requiring masks, or limiting international travel), traffic management (e.g., adding buses, free bus tokens, or bike lanes), and climate policy (e.g., competing legal definitions of "net-zero" or "green hydrogen", and environmental feedback policymakers, economists, scientists, local community leaders, and other stakcholders- whose imput is worth soliciting before implementing interventions. In RHEA, this policy-making challenge can be formalized as follows:
loops) 19 52, 60]. In such domains there exist diverse
Define. Suppose we are considering policy interventions 01,..., On. A policy action A consists of some subset of these. Suppose we must be prepared to address contexts e E {cI..... Cr h. and we have a black-box predictor őbc, A) to evaluate utility (Fig. Db.. In practice, will be a complex dynamical model such an agent-based to highlight the core behavior of RHEA, & is a simple-to-define function containing the kinds of challenging nonlinearities we would like to address, such as context dependence, synergies, anti- synergies, threshold effects, and redundancy (the full utility function is detailed in Eq. [T.. Similarly, e is a simple cost function, defined as the total number of prescribed policy interventions. A prescriptor is a function ((c)= = A (Fig. [b). The goal is to find a Pareto front of prescriptors across the outcomes of utility & and cost 1b. Note that the search space is vast: There are n possible prescriptors.
wewllllettooo-bbbeeeeee predictor. In this example,
Gather. Suppose prescriptors of unknown functional form have been gathered (Fig. (e) from three experts: one "generalist", providing general knowledge that applies across contexts (see Fig. 2 for an example); and two "specialists", providing knowledge that of higher quality (i.e. lower cost-per-utility) but applies only to a few specific contexts (Fig. 22-b).
Distill. Datasets for distillation can be generated by running each expert prescriptor over all contexts. The complete behavior of a prescriptor can then be visualized as a binary grid, where a black cell indicates the inclusion of an intervention in the prescription for a given context (Fig. 22-c). This data can be used to convert the expert prescriptors into rule sets or neural networks (Fig. Td. App. B.2).
Evolve. These distilled models can then be injected into initial population and evolved using multi-objective optimization [16] (Fig. [D). The full optimal Pareto front is obtained as a result.
With this formalization, it is possible to construct synthetic example of RHEA in action, as shown in Fig. 2 It illustrates the optimal Pareto front. Importantly, this front is discoverable by RHEA, but not by previous machine learning techniques such as Mixture-of-Eixe -xperts (MoF) 42 or Weighted Ensembles [18], or by the experts alone. RHEA is able to recombine the internal structure of experts across contexts (c.g., by adding 03, 04, a5 to ai, ag in c1). It can innovate beyond the experts by adding newly-applicable interventions (ag). It can also refine the results by removing interventions that are now redundant or detrimental (aş in 02), and by mixing in generalist knowledge. In contrast, the discoveries of MoE are restricted to mixing expert behavior independently at each context, and Weighted Ensemble solutions only choose single combination of experts to apply everywhere.
The domain also illustrates why is important to utilize expert knowledge in the first place. The high-dimensional solution space makes it very difficult for evolution alone (i.e. not starting from distilled expert prescriptors) to find high-quality solutions, akin to finding needles in a haystack. Experimental results confirm that RHEA discovers the entire optimal Pareto front reliably, even
3

---

Mal Fandlo froe
Dissmos Persto Prant
Exprt ution
10 3 5 2 100
Gonteot
Tooi Da0
Contost
20
Thtel Cost Pakyy Action
Corfec
Figure Illustration of RHEA a Synthelic Domain. The plots show the Pareto front of prescrip- tors discovered by RHEA vs. those of alternative prescriptor combination methods, highlighting the kinds of opportunities RHEA is able to exploit. The specialist expert prescriptors a and b and the generalist expert prescriptor
useful but suboptimal on their own (purple s). RHEA recombines and innovates upon their internal structure and is able to discover the full optimal Pareto front (blue (MoE; green x x's), which can only mix expert
*'s). This front dominates that of Mixture-of- behavior independently in each context. It also dominates that of Weighted Ensembling (yellow + s), which can only choose single combination of experts to apply everywhere. Evolution alone (without expert knowledge) also struggles in this domain due to the vast search space (App. Fig. (6) do MORL methods (App. Fig. 78. Thus, RHEA unlocks the latent potential in expert solutions.
as the number of available interventions increases, while evolution alone does not (App. Fig. 6 Multi-objective reinforcement learning (MORL) methods also struggle in this domain (App. Fig. Thus, RHEA harnesses the latent potential of expert solutions. uses pieces of them as building blocks and combines them with novel elements take full advantage of them. This ability can be instrumental in designing effective policies for complex real-world tasks. Next, RHEA is put to work on one particularly vexing task: optimizing pandemic intervention policies.
3 The XPRIZE Pandemic Response Challenge
The XPRIZE Pandemic Response Challenge [10] 11 presented an ideal opportunity for demonstrating the RHEA framework. XPRIZE is an organization that conducts global competitions, fueled by large cash prizes, to motivate the development of underfunded technologies. Current competitions target wildfires, desalination, carbon removal, meat alternatives, and healthy aging [81] 2020 and 2021, the XPRIZE Pandemic Response Challenge was designed and conducted 778 challenging participants to develop models to suggest optimal policy solutions spanning the tradcoff between minimizing new COVID-19 cases and minimizing the cost of implemented policy interventions.
Define. The formal problem definition was derived from the Oxford COVID-19 government response tracker dataset 27 [4 [4], which was updated daily from March 2020 through December 2022. This dataset reports government intervention policies ("IPs"" on a daily basis, following standardized classification of policies and corresponding ordinal stringency levels in Z, (used to define IP "cost") to enable comparison across geographical regions ("geos"), which include nations and subnational regions such as states and provinces. The XPRIZE Challenge focused on 235 geos (App. Fig 9 and those IPs over which governments have immediate daily control [54]: school closings, workplace closings, cancellation of public events, restrictions on gathering size, closing of public transport,

---

stay at home requirements, restrictions on internal movement, restrictions on international travel, public information campaigns, testing policy, contact tracing, and facial covering policy. Submissions for Phase were required to include runnable program ("predictor") that outputs predicted cases given a geo, time frame, and IPs through that time frame (Fig. [T)). Submissions for Phase were required to include set of runnable programs ("prescriptors" ) which, given a geo, time frame, and relative IP costs, output a suggested schedule of IPs ("prescription") for that geo and time frame (Fig. [D). By providing a set of prescriptors, submissions could cover the tradeoff space between minimizing the cost of implementing IPs and the expected number of new cases. Since decision makers for particular geo could not simultaneously implement multiple prescriptions from multiple teams, prescriptions were evaluated not in the real world but with a predictor & (from Phase 1), which forecasts how case numbers change result of prescription. The formal problem definition, requirements, API, and code utilities are publicly available [10]. Teams were encouraged to incorporate specialized knowledge in geos with which they were most familiar. The current study focuses on the prescriptors created in Phase 2. There are x 10020 possible schedules for a single geo for 90 days, so brute-force search is not an option. To perform well, prescriptors must implement principled ideas to capture domain-specific knowledge about the structure of the pandemic.
Gather. Altogether, 102 teams of experts from 23 countries participated in the challenge. Some teams were actively working with local governments to inform policy [49 53]; other organizations served as challenge partners, including the United Nations ITU and the City of Los Angeles [80). The set of participants was diverse, including epidemiologists, public health experts, policy experts, machine learning experts, and data scientists. Consequently, submissions took advantage of diverse methodolo- gies, including epidemiological models, decision rules, classical statistical methods, gradient- based optimization, various machine learning methods, and evolutionary algorithms, and exploited various auxiliary data sources to get enhanced views into the dynamics of particular geos (79) (Fig. [b).
The Phase 2 evaluations showed substantial specialization different geos for different teams, strong indication that there was diversity that could be harnessed. Many submissions also showed remarkable improvement over strong heuristic baselines, indicating that high-quality expertise had been gathered successfully. Detailed results of the competition are publicly available [[]; this study focuses on the ideas in them in the aggregate.
Distill. A total of 169 prescriptors were submitted to the XPRIZE Challenge. After the competition, for each of these gathered prescriptors n; an autoregressive neural network (NN) म, with learnable parameters 0 was trained with gradient descent to mimic its behavior, i.e. to distill it (30 31 (Fig. Tdl: App.(C.. C.I Each NN was trained dataset of 212,400 input-output pairs, constructed by querying the corresponding prescriptor Th times, i.e., through behavioral cloning:
(:) p(a) (g) - * (n((n(,, (1)))))): 0):00) 11,
0 = argmin 8,
R argmin ng
1
12g Σ
(.(g)) - #o (n(g. n. (15)
(2)
0:
where q E Q is a query and K is a function that maps queries (specified via the API in Define) to input data, i.e., contexts, with a canonical form. Each (date range, geo) pair defines query q, with
(T(9) E 200022 the policy generated by T; for this geo and date range, and Φ(g. (q) E R the predicted (normalized) daily new cases. Distilled models were implemented in Keras and trained with Adam 335 using L1 loss (since policy actions were
ordinal scale) (see App.
Evolve. These 169 distilled models were then placed in the initial population of evolutionary AI process ( This process was based on the same Evolutionary assioga Prescription (ESP) method 24 previously used to evolve COVID-19 19 IP prescriptors from scratch (50]. In standard ESP, the initial population (i.c.. before any evolution takes place) consists only of NNs with randomly generated weights. By replacing random neural networks with the distilled neural networks, ESP starts from diverse high-quality expert- expert-based solutions, instead of low- low-quality random ones. ESP can then be run as usual from this starting point, recombining and refining solutions over a series of generations to find better tradeoff's between stringency and cases, using Pareto- based multi- optimization [66] (App. C.2) Providing a Pareto front of policy strategy options is critical, because most saacision-makers will not simply choose the most extreme strategies (i.e. IPs with maximum stringency, or no IPs at all), but are likely to choose a tradeoff point appropriate for their particular political, social and economic scenario (Fig. Bl shows the real-world distribution of IP stringencies).
5

---

Streggrccee
Eecood
Eveleot
Excled
Rasdon
3.12-
Gaa <De
Row Geo Data
Collict
Disiiee
Doolie
1.10
IMEA
20
2:95
10
10
3:82-
3.00
6-110
12'
6.100
2x110
6.110
2x-12"
Crrl
REM
RUN
DR
h
MOR
1541
1-3
43000
66-
0.-
2000
19900
60
Dalodd
Pandey
Briaed
R6G
Aansdm
Cpsteed
Figure 3: Quantitative comparison of solutions. a, Objective values for all solutions in the final population of single representative run of each method. b, Pareto curves for these runs. Distilled provides improved tradeoff's over Random and Evolved (from random), and RHEA pushes the front out beyond Distilled. c, Overall Pareto front of the union of the solutions from these runs. The vast majority of these solutions are from RHEA. d, The distribution of actual stringencies implemented in the real world across all geos at the prescription start date, indicating which Pareto solutions real-world decision makers would likely select, i.e., which tradeoffs they prefer. e, Given this distribution, the proportion of the time the solution selected by a user would be from a particular method (the REM metric); almost all of them would be from RHEA. f, The same metric, but based on uniform distribution of tradeoff preference (RUN) g, Domination rate (DR) w.r.t. Distilled, i.c. how much of the Distilled Pareto front is strictly dominated by another method's front. While Evolved (from scratch) sometimes discovers better solutions than those distilled from expert designs, RHEA improves 75% of them. h, Max reduction of cases (MCR) compared to Distilled across all stringency levels. i, Dominated hypervolume improvement (HVI) compared to Distilled. For each metric, RHEA substantially outperforms the alternatives, demonstrating that it creates improved solutions over human and AI design, and that those solutions would likely be preferred by human decision-makers (Bars show mean and st.dev. See App. C.3 for technical details of each metric.)
Evolution from the distilled models was run for 100 generations in 10 independent trials to produce the final RHEA models. As baseline, evolution was run similarly from scratch. As a second baseline, RHEA was compared the full set of distilled models. A third baseline was models with randomly initialized weights, which is often meaningful starting point in NN-based policy search [68] All prescriptor evaluations, including those during evolution, were performed using the same reference predictor as in the XPRIZE Challenge itself; this predictor was evaluated in depth in prior work (50)
Results. The performance results are shown in Fig. 3 As is clear from the Pareto plots (Fig. h-c) and across range of metrics (Fig. -iii, the distilled models outperform the random initial models, thus confirming the value of human insight and the efficacy of the distillation process. Evolution then improves performance substantially from both initializations, with distilled models leading to the best solutions. Thus, the conclusions of the illustrative example are substantiated this real-world domain: RHEA is able to leverage knowledge contained in human-developed models to discover solutions beyond those from the AI alone or humans alone. The most critical performance metric is the empirical Rl-etri (REM; (28).. ). which estimates the percentage of time decision-maker with a fixed stringency budget would choose a prescriptor from a given approach among those from all approaches. For RHEA, REM is nearly 100%. In other words, not only does RHEA discover policies that perform better, but they are also policies that eecision-makers would be likely to adopt.
4 Characterizing the Innovations
Two further sets of analyses characterize the RHEA solutions and the process of discovering them. First, IP schedules generated for each geo by different sets of policies were projected to 2D via

---

Stagg
(80
Figure 4: Dynamics of IP schedules discovered by RHEA. a, UMAP projection of geo IP schedules generated by the policies (App. C.4). The schedules from high-performing submitted expert mod- els are concentrated around a 1-dimensional manifold organized by overall cost (seen as a yellow arc). This manifold provides a scaffolding upon which RHEA elaborates, interpolates, and expands. Evolved policies, on the other hand, are scattered more discordantly (seen as blue clusters), un- grounded by the experts. b, To characterize how RHEA expands upon this scaffolding, five high-level properties of IP schedules were identified and their distributions were plotted across the schedules. For each, RHEA finds balance between the grounding of expert submissions (i.e., regularization) and their recombination and elaboration (i.c., innovation), though this balance manifests distinct ways. For swing and separability. RHEA is similar to real schedules, but finds that the high separability proposed by some expert models can sometimes be useful. RHEA finds the high focus of the expert models even more attractive; in practice, they could provide policy-makers with simpler and clearer messages about how to control the pandemic. For focus, agility, and periodicity, RHEA pushes beyond areas explored by the submissions, finding solutions that humans may miss. The example schedules shown in a(i-v) illustrate these principles
practice (rows are IPs sorted from top to bottom as listed in Sec. 3 column are days in the 90-day period; darker color means more stringent). (i) Real-world examples demonstrate that although agility and periodicity require some effort to implement, they have occasionally been utilized (e.g. in Portugal and France); (ii) simple example of how RHEA generates useful interpolations of submitted non- non-Pareto schedules, demonstrating how it realizes latent potential even in some low- wwwffofooming solutions, far from schedules evolved from scratch; (fii) another useful interpolation, but achieved via higher agility than Pareto submissions; (iv) high-stringency RHEA schedule that trades swing and separability for agility and periodicity compared to its submitted neighbor; and (v) a medium mreiiun-ttinngecy RHEA schedule with lower swing and separability and higher focus than its submitted neighbor. Overall, these analyses show how RHEA realizes the latent potential of the raw material provided by the human-created submissions.
UMAP 45 to visualize the distribution of their behavior (Fig. 4a). Note that the schedules from the highest-pefforming (Pareto) submitted policies form a continuous 1D manifold across this space, indicating continuity of tradeoffs. This manifold serves as scaffolding upon which RHEA recombines, refines, and innovates; these processes are the same as in the illustrative example, only more complex. Evolution alone, on the other hand, produces a discordant scattering of schedules, reflecting its unconstrained exploratory nature, which is disadvantageous in this domain. What kind of structure does RHEA harness to move beyond the existing policies? Five high-level properties were identified that characterize how RHEA draws on submitted models in this domain: swing measures the stringency difference between the strictest and least strict day of the schedule; separability measures to what extent the schedule can be separated into two contiguous phases of different stringency levels; focus is inversely proportional to the number of IPs used; agility measures how often IPs change; and periodicity measures how much of the agility can be explained by weekly periodicity
7

---

Pantoo solutón uncestral comp saly
Paner Creee LocE
133
325
Loc
333
9.32
73
150 2.00
6.2
0.00
Cart
Figure 5: Dynamics of evolutionary discovery process. a, Sample ancestries of prescriptors on the RHEA Pareto front. Leaf nodes are initial distilled models: the final solutions are the root. The history of recombinations leading to different solutions varies widely in terms of complexity. with apparent motifs and symmetries. The ancestries show that the search is behaving as expected, that the cost of the child usually lands between the costs of its parents (indicated by color). This property also visualized in b (and c). where child costs (and cases) are plotted over all recombinations from all trials (k-NN regression, k 100). d, From ancestries, one can compute the relative contribution of each expert model to the final RHEA Pareto front (App C.5). This contribution remarkably consistent across the independent runs, indicating that the approach is reliable (mean and st.dev, shown). e, Although there a correlation between the performance teams expert models and their contribution to the final front, there are some teams with unimpressive quantitative performance in their submissions who end up making outsized contributions through the evolutionary process. This result highlights the value of soliciting broad diversity of expertise, even if some of it does not have immediately obvious practical utility. Al can play a role in realizing this latent potential.
(Fig. 4p: App. C.4). Some ideas from submitted policies, e.g.. increased separability and focus, are readily incorporated into RHEA policies. Others, e.g. increased focus, agility, and periodicity, RHEA is able to utilize beyond the range of policies explored by the human designs. The examples in Fig. 4h illustrate these properties in practice. Example (i) shows number of real policies, suggesting that geos are capable of implementing diverse and innovative schedules similar to those discovered by RHEA; e.g., weekly periodicity was actually implemented for time in Portugal and France. Examples (ii-v) show RHEA schedules and their nearest submitted neighbors, demonstrating how innovations can manifest interpolations or extrapolations of submitted policies. For instance, one opportunity is to focus on a smaller set of IPs; another is to utilize greater agility and periodicity. This analysis shows how RHEA can lead insights on where improvements are possible.
Second, to understand how RHEA discovered these innovations, an evolutionary history can be reconstructed for each solution, tracing back its initial distilled ancestors (Fig- 5). Some final solutions stem from a single beneficial crossover of distilled parents, while others rely on more complex combinations of knowledge from many ancestors (Fig. 5p). While the solutions are more complex, the evolutionary process is similar to that of the illustrative example: It proceeds a principled manner, with child models often falling between their parents along the case-stringency tradcoff (Fig. 5b-c). Based on these evolutionary histories, one can compute the relative contribution of each expert model to the final RHEA Pareto front (App C.3). These contributions are highly consistent across independent runs, indicating that the approach is reliable (Fig. (d).. Indeed in the XPRIZE competition, this contribution amount was used as one of the quantitative metrics of solution quality [1]. Remarkably, although there is correlation between the performance of expert models and their contribution end up making outsized contributions through the evolutionary process (Fig. 5e; see also Fig. (ii). This result highlights the value of soliciting a broad diversity of expertise, even some of it does not have immediately obvious practical utility. Al can play role in realizing this latent potential.
final front, there are also models that do not perform particularly well, but

---

5 Discussion
Alternative Policy Discovery Methods. Our implementation of RHEA uses established methods in both the Distill and Evolve steps; the technical novelty comes from their unique combination in RHEA to unlock diverse human expertise. Popular prior methods for combining diverse models include ensembling [18] and Mixture-of -ixture-of-Experts [42], but, as highlighted in Fig. 2 although multi- variants have been explored in prior work [36), neither of these methods can innovate beyond the scaffolding provided by the initial experts. Evolution is naturally suited for this task: Crossover is powerful way to recombine expert models, mutation allows innovating beyond them, and population- based search naturally supports multiobjective optimization. Other approaches for policy optimization include contextual bandits (73), planning-base based methods [66], and reinforcement learning [29] 69], and an interesting question is how they might play role in such system. One approach could be to use evolutionary search for recombination and use another method for local improvement, akin to hybrid approaches used in other settings 6 (See App. A for longer discussion).
Theory. is intuitive why expert knowledge improves RHEA's search capability. However, any theoretical convergence analysis will depend on the particular implementation of RHEA. The present implementation uses NSGA-II, the convergence of which has recently been shown to depend critically on the size of jumps in the optimization landscape, i.e. roughly the maximum size of non-convex benchmark, the tightest known upper-bound for convergence to the full ground truth Pareto front is O /((2)"/O(k)) where ls is measure of the jump size, n is the problem dimensionality. and N is the (sufficiently large) population size. In other words, smaller jump size leads to drastic convergence speed up. Distilling useful, diverse experts is conceptually analogous to decreasing the jump size. This effect is apparent in the illustrative domain, where the experts provide building blocks that can be immediately recombined discover better solutions, but that are difficult to discover from scratch (Fig. (2). This interpretation is borne out in the experiments: RHEA continues to converge quickly as the action space (i.e. problem dimensionality) increases, whereas evolution regresses to only being able to discover the most convex (easily-discoverable) portions of the Pareto front (App. Fig. 6)
regions 20 21] On the ONEJUMPZZROJJU
domains where (1) the problem can formalized with contexts, actions, and outcomes, (2) there exist diverse experts from which solutions can be gathered, and (3) the problem is sufficiently challenging. In contrast, RHEA would not be effective, (1) if the problem too easy, so that the input from human experts would not be necessary, (2) if the problem is hard, but no useful and diverse experts exist, and (3) if there is no clear way to define context and/or action variables upon which the experts agree.
Generalizability. RHEA can be applied effectively to
The modularity of RHEA allows different implementations of components to be designed for different domains, such as those related to sustainability, engineering design, and public health. One particularly exciting opportunity for RHEA is climate policy, which often includes complex interactions between multiple factors 46). For example, given the context of the current state of the US energy grid and energy markets, the green hydrogen production subsidies introduced by the Inflation Reduction Act will in fact lead to increases carbon emissions, unless the Treasury Department enacts three distinct regulations in the definition of "green hydrogen" (60] It is precisely this kind of policy combination that RHEA could help discover, and such a discovery process could be an essential part of climate policy application. For example, the En-Roads climate simulator supports diverse actions across energy policy, technology, and investment, contexts based on social, economic, and environmental trajectories, and multiple competing outcomes, including global temperature, cost of energy, and sea-level rise [8. Users craft policies based on their unique priorities and expertise. RHEA could be used with predictor like En Roads to discover optimized combinations of expert climate policies that trade-off across temperature change and other the outcomes that users care about most.
Ethics and Broader Impact. As part of the UN AI for Good Initiative, we currently building a platform for formalizing and soliciting expert solutions to SDG goals more broadly [35]. Ethical considerations when deploying such systems are outlined below. See App. D for further discussion.
Fairness. such problems with diverse stakeholders, breaking down costs and benefits by affected populations and allowing users to input explicit constraints prescriptors can be crucial for generating feasible and equitable models. In this platform, RHEA could take advantage of knowledge that local experts provide and learn to generalize it; by treating each contributed model as black box.

---

it is agnostic to the type of models used, thus helping to make the platform future- proure. Fairness constraints can also be directly included in RHEA's multiple objectives.
Governance and Democratic Accountability. An important barrier in the adoption of Al by real-world For example, such systems could be used to justify the decisions of bad actors. RHEA provides an advantage here: If the initial buman-developee
decision-makers is trust 44 655
models it uses are explainable (e.g. are based on rules or decision trees), then a user can trust that suggestions generated by RHEA models are based on sensible principles, and can trace and interrogate their origins. Even when the original models are opaque, trust can be built by extracting interpretable rules that describe prescriptor behavior, which is feasible when the prescriptors are relatively compact and shallow 72). as in the experiments in this paper. That is, RHEA models can be effectively audited critical property for AI systems maintained by governments and other badlic-g organizations.
Data Privacy and Security. Since experts submit complete prescriptors, no sensitive data they may have used to build their prescriptors needs to be shared. In the Gather step in Sec. each expert team had an independent node to submit their prescriptors. The data for the team was generated by running their prescriptors on their node. The format of the data was then automatically verified, to ensure that it complied with the Defined API. Verified data from all teams was then aggregated for the Distill & Evolve steps. Since the aggregated data must fit an API that does not allow for extra data to be disclosed, the chance of disclosing sensitive data in the Gather phase is minimized.
External Oversight. Although the above mechanisms could all yield meaningful steps in addressing a broad range of ethical concerns, they cannot completely solve all issues of ethical deployment. So, it is critical that the system is not deployed in an isolated way, but integrated into existing democratic decision- processes, with appropriate external oversight. Any plan for deployment should include disclosure of these risks to weigh against the potential societal benefits.
Sustainability and Accessibility. Due to the relatively compact model size, RHEA uses orders of magnitude less compute and energy than many other current Al systems, which is critical for creating uptake by cacision-makers around the world who do not have access to extensive computational resources or for whom energy usage is becoming an increasingly central operational consideration.
Limitations. Understanding the limitations of the presented RHEA implementation is critical for establishing directions for future work. The cost measure used in this paper was uniform over IPs, an unbiased way to demonstrate the technology, but, for prescriptor to be used in particular geo, costs of different IPs should be calibrated based on geo-specific cost- analysss.. The geo may also have some temporal discounting in its cases and cost objectives. For consistency with the XPRIZE, they were not included in the experiments in this paper but can be naturally incorporated into RHEA in the future. When applying surrogate-developed policies the real world, approximation errors can compound over time. Thus, usei-facing applications of RHEA could benefit from the inclusion of uncertainty measures (26) (58), inverse reinforcement learning 12 70), as well as humans-in ieetans-i--the-lopp to prevent glaring errors. Distillation could also be limited in cases where expert models use external data sources with resulting effects not readily approximated by the inputs specified in the defined API. If this were an issue in future applications, it could be addressed by training models that generalize across domain spaces as prescriptors in the XPRIZE. but not yet in hands-on user studies. Hands-on user evaluation is a critical step but requires completely different kind of research effort, i.e. one that is political and civil, rather than computational. Our hope is that the publication of the results of RHEA makes the real-world incorporation of these kinds of AI seecision-assistamn more likely.
47 59
RHEA prescriptors were evaluated in the same surrogate setting
Conclusion. This paper motivated, designed, and evaluated framework called RHEA for bringing together diverse human expertise systematically to solve complex problems. The promise of RHEA was illustrated with an initial implementation and an example application; it can be extended to other domains in future work. The hope is that, as a general and accessible system that incorporates input from diverse human sources, RHEA will help bridge the gap between human-only decision-making and Al-from-data-onle approaches. As a result, docisiin-maker can start adopting powerful AI decision-suppor systems, taking advantage of the latent real-world possibilities such technologies illuminate. More broadly, the untapped value of human expertise spread across the world is immense. Human experts should be actively encouraged to continually generate diverse creative ideas and contribute them to collective pools of knowledge. This study shows that AI has a role to play in realizing the full value of this knowledge, thus serving catalyst for global poblem-solviigg
10

---

Acknowledgements
We would like to thank XPRIZE for their work in instigating, developing, publicizing, and adminis- tering the Pandemic Response Challenge, as well as the rest of the Cognizant AI Labs research group for their feedback on experiments and analysis. We would also like to thank Conor Hayes for advice on running the MORL comparisons, and Benjamin Doerr for advice NSGA-II theory.
References
[]] L. N. Alegre, A. L. Bazzan, D. M. Roijers, A. Nowé, and B. C. da Silva. Sample-efficient multi-objective learning via generalized policy improvement prioritization. Proceedings of the 2023 International Conference Aulonomous Agents and Multiagent Systems, pages 2023.
[2] S. Arora and P. Doshi. A survey of inverse reinforcement learning; Challenges, methods and progress. Artificial Intellignnce, 297:103500, 2021.
[3] Ba and Caruana. Do deep nets really need to deep? Adv. Neural Inf. Process. Syst., 2014.
[4] Buchanan and Smith. Fundamentals of expert systems. Annu. Rev. Comput. Sci., 1988.
[5] F. Chicano, D. Whitley, G. Ochoa, and R. Tinós. Optimizing one million variable NK landscapes by hybridizing deterministic recombination and local search. Proceedings of the Genetic and Evolutionary Computation Conference, GECCO 17, pages 753-760, July 2017.
[6] Chicano, D. Whidly, G. Ochoa, and R. Tinós. Optimizing one million variable nk landscapes by hybridizing deterministic recombination and local search. In Proceedings of the genetic and evolutionary computation conference, pages 753-760, 2017.
[7] Chollet and others. Keras: The python deep learning library, June 2018.
[8] Climate Interactive. En-roads climate solutions simulator, 2024. [Computer software]. 7/en-roads climateinteeaactive..o
https
[9] Cognizant AI Labs. XPRIZE pandemic response challenge (github repository). https://
2021.
[10]
Cognizant Labs and XPRIZE, XPRIZE pandemic response challenge guidelines. https
pdf 2020.
[11] Cognizant Al Labs and XPRIZE Pandemic response challenge phase 2 results. https
//phase2. eeoouiiiiivolltion...
ml/ 2021. Accessed: 2022-1-22.
[12] Cognizant Al Labs and XPRIZE. Phase quantitative evaluation 2. https:
ml/ htppp//ppiie//aaaee//00000 Anon. Anon. 2021. Accessed: 2022-1-22.
[13] H. Dan. How much did AlphaGo zero cost? 2020.
www. www. www.:uuuhh.ccmm/tttaaaaaacccesee
June
[14] R. Dawkins. The Selfish Gene. Oxford University Press, 1976.
[15] K. Deb and C. Myburgh. Breaking the Billion-Variable barrier in Real-World optimization using customized evolutionary algorithm. In Proceedings of the Genetic and Evolutionary Computation Conference 2016, GECCO 16, pages 653-660. ACM, July 2016.
[16] K. Deb, A. Pratap. S. Agarwal, and T. Meyarivan. A fast and elitist multiobjective genetic algorithm: NSGA-IL. IEEE Trans. Evol. Comput., 182):182-197, Apr. 2002.
[17] D. C. Dennett. From Bacteria to Bach and Back: The Evolution of Minds. W. W. Norton & Company, Feb. 2017.
[18] G. Dietterich al. Ensemble learning. The handbook of brain theory and neural networks, 2(1):110-125, 2002.

---

[19] C. Ding and S. Song. Traffic paradoxes and economic solutions. Journal of Urban Management, 1(1):63-76, 2012.
[20] B. Doerr and Z. Qu. From understanding the population dynamics of the nsga-ii to the first proven lower bounds. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages
2023.
[21] B. Doerr and Qu. Runtime analysis for the nsga-ii: Provable speed-ups from crossover. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 12399-12407, 2023.
[22] M. M. Drugan and A. Nowe. Designing multi-objective multi-armed bandits algorithms: A study. In The 2013 international joint conference on neural networks (IJCNN), pages 1-8. IEEE, 2013.
[23] F. Felten, L. N. Alegre, A. Nowe, A. Bazzan, E. G. Talbi, G. Danoy, and B. C da Silva. A toolkit for reliable benchmarking and research in multi-objective reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
[24] O. Francon, S. Gonzalez, B. Hodjat, E. Meyerson, R. Miikkulainen, X. Qiu, and H. Shahrzad. Effective reinforcement learning through evolutionary aurroaate-assiserd prescription. In Proc. of the Genetic and Evolutionary Computation Conference, June 2020.
[25] R. B. Freeman and W. Huang. Collaboration: Strength in diversity. Nature, 513((5188)305. Sept. 2014.
[26] Gawlikowski, C. R. N. Tassi, M. Ali, J. Lee, M. Humt, J. Feng, A. Kruspe, R. Triebel, P. Jung, R. Roscher, et al. A survey of uncertainty in deep neural networks. arXiv preprint arXiv:210..03342.20000 2021.
[27] T. Hale, N. Angrist, R. Goldszmidt, B. Kira, A. Petherick, T. Phillips, S. Webster, E. Cameron- Blake, L. Hallas, S. Majumdar, and H. Tatlow. A global panel database of pandemic policies (oxford COVID-19 government response tracker). Nature Human Behaviour, 5(4):529-538 Mar. 2021.
[28] M. P. Hansen and A. Jaszkiewicz. Evaluating the quality of approximations to the non-dominated set. Technical Report IMM-REP-1998-1 Institute of Mathematical Modelling, Technical University of Denmark, 1994.
[29] C. F. Hayes, R. Rādulescu, E. Bargiacchi, J. Källström, M. Macfarlane, M. Reymond, T. Ver- straeten, L. M. Zintgraf, Dazeley, F. Heintz, et al. A practical guide to multi-objective reinforcement learning and planning. Autonomous Agents and Multi-Agent Systems, 36(1):26, 2022.
[30] Hinton, Vinyals. and Dean. Distilling the knowledge in neural network. arXiv preprint arXiv: /503. 02531.
[31] A. Hussein, M. M. Gaber, E. Elyan, and C. Jayne. Imitation learning: A survey of learning methods. ACM Comput. Surv., 50(2):1 Apr. 2017.
[32] Renzullo, M. Moses, W. Weimer, and S. Forrest. Neutral networks enable distributed search in evolution. Genetic Improvement Workshop at the International Conf. on Software Engineering (ICSE), 2018
[33] M. Jit, A. Ananthakrishnan, M. McKee, O. J. Wouters, P. Beutels, and Y. Teerawattananon. Multi-country collaboration in responding to global infectious disease threats: lessons for europe from the COVID-19 pandemic. Lancet Reg Health Eur, 9:100221, Oct. 2021.
[34] M. Jordan and T. M. Mitchell. Machine learning: Trends, perspectives, and prospects. Science,
24956245):555-26,, 2015.
[35] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. Dec. 2014.
[36] D. Kocev, C. Vens, Struyf, and S. Džeroski. Ensembles of multi-ohjective decision trees. In
18th European Conference Machine Learning, pages 624-631. Springer, 2007.
12

---

[37] S. W. J. Kozlowski and B. S. Bell. Work groups and teams in organizations. In N. W. Schmitt, editor, Handbook of psychology: Industrial and organizational psychology. Vol, volume 12. pages 412-469. John Wiley & Sons, Inc., xvii, Hoboken, NJ, US, 2013.
[38] C. Le Goues, S. Forrest, and W. Weimer. The case for software evolution. In Proceedings of the FSE/SDP workshop Future of software engineering research, pages 205-210, Nov. 2010.
[39] J. Lehhann, J. Clune, D. Misevic, C. Adami, L. Altenberg. J. Beaulieu, P. J. Bentley, S. Bernard,
G. Beslon, D. M. Bryson, N. Cheney, P. Chrabaszez, A. Cully, S. Doncieux, F. C. Dyer, K. O.
Ellefsen, R. Feldt, S. Fischer, S. Forrest, A. Frenoy, C. Gagne, L. Le Goff, L. M. Grabowski,
B. Hodjat, F. Hutter, L. Keller, C. Knibbe, P. Kreah, R. E. Lenski, H. Lipson, R. MacCurdy, C. Maestre, R. Mikkulainen, S. Mitri, D. E. Moriarty, J.-B. Mouret, A. Nguyen, C. Ofria, M. Parizeau, D. Parsons, R. T. Pennock, W. F. Punch, T. S. Ray, M. Schoenauer, E. Schulte, K. Sims, K. O. Stanley, F. Taddei, D. Tarapore, S. Thibault, R. Watson, W. Weimer, and Yosinski. The surprising creativity of digital evolution: collection anecdotes from the evolutionary computation and artificial life research communities. Artif. Life, 26(2):274-366 Apr. 2020.
[40] J. Lehman and K. O. Stanley. Abandoning objectives: Evolution through the search for novelty alone. Evolutionary computation, 19(2):189-223, 2011.
[41] Li, K. Guo, E. H. Viedma, H. Lee, J. Liu, N. Zhong, F. Autran Monteiro Gomes, G. Filip, S.-C. Fang. M. S. Özdemir, X. Liu, G. Lu, and Y. Shi. Culture versus policy: More global collaboration effectively combat COVID-19. Innovation (Camb), 1(2):100023, Aug. 2020.
[42] S. Masoudnia and R. Ebrahimpour. Mixture of experts: literature survey. The Artificial Intelligence Review, 42(2):275, 2014.
[43] M. Mauldin. Maintaining diversity in genetic search. In Proceedings of the Fourth AAAI Conference Artificial Intelligence, pages 247-250, 1984.
[44] A. McGovem, Ebert-Uphoff. D. Gagne, and A. Bostrom. Why we need to focus on devel- oping ethical, responsible, and trustworthy artificial intelligence approaches for environmental science. Environmental Data Science, 1:66, 2022.
[45] L. McInnes, Healy, and J. Melville. UMAP: Uniform manifold approximation and projection for dimension reduction. Feb. 2018 8.
[46] J. Meckling, N. Kelsey, E. Biber, and J. Zysman. Winning coalitions for climate policy. Science, 349(6253):1170-111 2015.
[47] E. Meyerson and R. Mükkkllaneen. The traveling observer model: Multi-task learning through spatial variable embeddings. International Conference on Learning Representations, 2021.
[48] E. Meyerson, X. Qiu, and R. Mükkuulainen. Simple genetic operators universal approximators of probability distributions (and other advantages of expressive encodings). In Proceedings of the Genetic and Evolutionary Computation Conference, pp. 739-748. ACM, 2022.
[49] Miguel Angel Lozano, et al. Open data science to fight COVID-19; Winning the 500k XPRIZE pandemic response challenge. International Joint Conference on Artificial Intelligence, pages 5304-5308, 2022.
[50] Miikkulainen, Francon, Meyerson, Qiu, Sargent, Canzani, and Hodjat. From prediction to prescription: Evolutionary optimization of Non-Pharmaccuttial interventions in the COVID-19 pandemic. IEEE Trans. Evol. Comput., 2021.
[51] R. Mükkullanen. Creative ai through evolutionary computation: Principles and examples. SN Computer Science, 2:163, 2021.
[52] A. Muscillo, Pin, and T. Razzolini. Covid19: Unless one gets everyone to act, policies may be ineffective or even backfire. PloS one, 150))e0237005, 2020.
13

---

[53] N. Oliver. Data science for social good: The valencian example during the COVID-
19 pandemic. https://www esade. wwwd//ccpo/wppooo. cttp:///pppcttttttttuolallaaaads httppp/ladd//////////
22//2/22222202222 07/ AAFF
EcPol-OICI PaperSerries_ Satptieee______________e__ ENG_ DEF compressed pdf 2022. Accessed: 2022-9-7.
[54] A. Petherick, B. Kira, N. Angrist, T. Hale, T. Phillips, and S. Webster. Variation in govermment responses to COVID-19. Technical report, Oxford University, 2020.
[55] Project Resilience. Platform. Global Initiative on Al and Data Commons, 2022. //github.com/Poo/fii http:/oojeect-eeelleeneeeeeee/ee
https
[56] J. K. Pugh, L.. B. Soros, and K. O. Stanley. Quality diversity: A new frontier for evolutionary computation. Frontiers in Robotics and AI, 3:40, 2016.
[57] K. Pulkkinen, S. Undorf, F. Bender, P. Hikman-Svahn F. Doblas-Reyes, C. Flynn, G. C. Hegerl, A. Jönsson, G.-K. Leung, Roussos, T. G. Shepherd, and E. Thompson. The value of values in climate science. Nat. Clim. Chang,, 12(1))4-6, Jan. 2022.
[58] X. Qiu, E. Meyerson, and R. Mikkulainen. Quantifying point-prediction uncertainty in neural networks via residual estimation with i/o kernel. In International Conference on Learning Representations, 2019.
[59] S. Reed, K. Zolna, E. Parisotto, S. G. Colmenarejo. A. Novikov, G. Barth-Maron, M. Gimenez, Y. Sulskuk, J. Kay, J. T. Springenberg, al. A generalist agent. arXiv preprint ar a7Xiv:2205.06//5. 2022.
[60] W. Ricks, Q. Xu, and J. D. Jenkins. Minimizing emissions from grid-based hydrogen production in the united states. Environmental Research Letters, 18(1):014025, 2023.
[61] N. Riquelme, C. Von Lücken, and B. Baran. Performance metrics in multi-objective optimization.
In 2015 Latin American Computing Conference (CLEI), pages 1-11, Oct. 2015.
[62] Rock and Grant. Why diverse teams are smarter. Harv. Bus. Rev., 2016.
[63] M. Romanello, A. McGushin, C. Di Napoli, Drummond, N. Hughes, L. Jamart, H. Ken- nard, P. Lampard, B. Solano Rodriguez. N. Arnell, S. Ayeb- Karlsson, K. Belesova, W. Cai, D. Campbell-Lendrum, S. Capstick, J. Chambers, L. Chu, L. Ciampi, C. Dalin, N. Dasandi,
S. Dasgupta, M. Davies, P. Dominguez-Salar R. Dubrow, K. I., Ebi, M. Eckelman, P. Ekins, Hartinger, K. He, C. Heaviside, J. Hess, S.-C. Hsu, S. Jankin, M. P. Jimenez, I. Kelman, G. Kiesewetter, P. L.. Kinney, T. Kjellstrom, D. Kniveton, J. K. W. Lee, B. Lemke, Y. Liu, Z. Liu, M. Lott, R. Lowe, J. Martinez-Urtaza, M. Maslin, L. McAllister, C. McMichael, Z. Mi, J. Milner, K. Minor, N. Mohajeri, M. Moradi-Lakeh, K. Morrissey, S. Munzert K. A. Murray, T. Neville, M. Nilsson, N. Obradovich, M. O. Sewe, T. Oreszczyn, M. Otto, F. Owfi, O. Pearman, D. Pencheon, M. Rab- baniha, E. Robinson, J. Rocklöv, R. N. Salas, J. C. Semenza. J. Sherman, L. Shi, M. Springmann, M. Tabatabaei, J. Taylor, J. Trinanes, J. Sauuake-Guillemen B. Vu, F. Wagner, P. Wilkinson, M. Winning, M. Yglesias, S. Zhang, P. Gong. H. Montgomery, A. Costello, and I. Hamilton. The 2021 report of the lancet countdown health and climate change: code red for healthy future. Lancet, 398 (100011):166 1):1619-1662. Oct. 2021.
L. F. Escobar, I.. Georgeson, D. Grace, H. Graham, S. H. Gunther,
[64] M. Schoon and M. E. Cox. Collaboration, adaptation, and scaling: Perspectives on environmen- tal governance for sustainability. Sustain. Sci. Pract. Policy, 10(3):679, Mar. 2018.
[65] Siau and Wang, Building trust in artificial intelligence, machine learning, and robotics. Cutter business technology journal, 2018
[66] D. Silver, A. Huang, C. J. Maddison, A. Guez, I.. Sifre, G. van den Driessche, J. Schrittwieser,
I. Antonoglou, V. Panneershelvam, M. Lanctot, S. Dieleman, D. Grewe, J. Nham, N. Kalch- brenner, I. Sutskever, T. Lillicrap, M. Leach, K. Kavukeuoglu, T. Graepel, and D. Hassabis. Mastering the game of go with deep neural networks and tree search. Nature, 529(7587):844- 489, Jan. 2016
[67] K. O. Stanley, J. Clune, Lehman, and R. Miikkulainen. Designing neural networks through neuroevolution. Nature Machine Intelligence, 1(1):24-35, Jan. 2019.
14

---

[68] F. P. Such, V. Madhavan, E. Conti, J. Lehman, K. O. Stanley, and J. Clune. Deep neuroevo- lution: Genetic algorithms are competitive alternative for training deep neural networks for reinforcement learning. Dec. 2017.
[69] R. S. Sutton and A. G. Barto. Reinforcement learning: An introduction. MIT press, 2018.
[70] G. Swamy, D. Wu, S. Choudhury, D. Bagnell, and S. Wu. Inverse reinforcement learning without reinforcement learning. In International Conference on Machine Learning, pages 33299-33315 PMLR, 2023.
[71] Thrun. Extracting rules from artificial neural networks with distributed representations. Adv Neural Inf. Process. Syst., 7, 1994.
[72] G. G. Towell and J. W. Shavlik. Extracting refined rules from knowledge-based neural networks. Mach. Learn., 13(1):71-101, Oct. 1993.
[73] E. Turgay, D. Oner, and C. Tekin. Multi-objective contextual bandit problem with similarity information. In International Conference on Artificial Intelligence and Statistics, pages 1673- 1681. PMLR, 2018.
[74] University of Oxford. Codebook for the Oxford COVID-19 government response tracker. github. codebook. md 2020.
[75] P. Virtanen, R. Gommers, T. E. Oliphant, M. Haberland, T. Reddy, D. Cournapeau, E. Burovski, Peterson, W. Weckesser, J. Bright, S. van der Walt, M. Brett, Wilson, K. Millman, N. Mayorov, A. R. J. Nelson, E. Jones, R. Kern, E. Larson, C. J. Carey, I. Polat, Y. Feng, E. W. Moore, J. VanderPlas, D. Laxalde, Perktold, R. Cimrman, I. Henriksen, A. Quintero, C. R. Harris, A. M. Archibald, A. H. Ribeiro, F. Pedregosa, P. van Mulbregt, and SciPy 1.0 Contributors. SciPy 1.0: fundamental algorithms for scientific computing in python. Nut. Methods, 17(3):2112222, Mar. 2020.
[76] M. Waskom. seaborn: statistical data visualization. J. Open Source Softw., 6(60):3021, Apr. 2021.
[77] L. F. Wolff Anthony, B. Kanding, and R. Selvan. Carbontracker: Tracking and predicting the carbon footprint of training deep learning models. July 2020.
[78] XPRIZE. sendemicresponee
Pandemic response challenge.
orapiizee.rg/chaleengg...
[79] XPRIZE. Technical team descriptions. cessed: 2021-5-11.
html Ac-
[80] XPRIZE. Pandemic response challenge: Prize partners.
ftops://www.xpriz/. org/
2020.
[81] XPRIZE. xprizee org. 2022. Accessed: 2022-1-22.
[82] R. Yang, Sun, and K. Narasimhan. generalized algorithm for multi-objective reinforcement learning and policy adaptation. Advances in neural information processing systems, 32, 2019.
[83] K. L. Yeager and F. M. Nafukho. Developing diverse teams to improve performance in the organizational setting. European Journal of Training and Development, 36(4):388 408, Jan. 2012.
[84] Y. You, Z. Zhang, C.-J. Hsich, J. Demmel, and K. Keutzer. ImageNet training in minutes. In Proceedings of the 47th International Conference on Parallel Processing, number Article in ICPP 2018, pages 1-10. ACM, Aug. 2018.
[85] Y. Zhang and Q. Yang. A survey multi-task learning. JEEE Transactions on Knowledge and Data Engineering, 34(12):5566-560) 2021.
15

---

Appendix
A
Related Work
The RHEA method builds on long tradition of leveraging diversity in machine learning, as well as methods for policy discovery in general.
A.1 Harnessing diversity in AI
Machine learning (ML) models generally benefit from diversity in the data on which they are trained 34]. At higher level. it has long been known that diverse models for a single task may be usefully combined to improve performance on the task. Methods for such combination usually fall under the label of ensembling [18]. By far, the most popular ensembling method is to use a linear combination of models. Mixture-of-Exxee -f-xxerrtss (MoE) approaches use a more sophisticated approach of conditionally selecting which models to use based on the input [42]. However, as highlighted in Fig.B although some multi-objective variants have been explored in prior work [36). neither of these methods are inherently sufficient for the kind of policy discovery required by RHEA. In particular, such methods are not multi-objective, and provide no method of innovating beyond the scaffolding provided by the individual experts.
An orthogonal approach to harnessing diversity within single task is to exploit regularities across multiple tasks, by learning more than one task in single model [85]. In the extreme case, a single model may be trained across many superficially unrelated tasks with the goal of learning shared structure underlying the groblem-solving universe 47 [9]] In this paper, it was possible to specialize the expert models to different regions and pandemic states, but the input-output spaces of the models were uniform to enable consistent API. Future work could generalize RHEA to cases where the expert models are trained on different, but related, problems that could potentially benefit from one another.
Finally, there is rich history of managing and exploiting diverse solutions in evolutionary algorithms: from early work on preserving diversity to prevent premature convergence 43 and well-established work on multi-objective optimization [16), to more recent research on novelty search and diversity for diversity's 's sake [40], and to the burgeoning field of Quality Diversity, where the goal is to discover geethpefforming solutions across an array of behavioral dimensions [56]. RHEA is different from these existing methods because it is not about discovering diverse solutions de novo, but rather about harnessing the potential of diverse human-created solutions. Nonetheless, the scope and success of such prior research illustrate why evolutionary optimization is well-suited for recombining and innovating upon diverse solutions.
A.2 Alternative approaches policy discovery
In this paper, evolutionary optimization was used discovery method because it is most naturally suited for this task; crossover is a powerful way to recombine expert models, mutation allows innovating beyond them, and population-base Other approaches for policy optimization include contextual bandits (73), planning-based methods [66], and reinforcement learning (69] and an interesting question whether they could be used in this role as well.
search naturally supports multiobjective optimization.
Although less common than in evolutionary optimization, multi-objective approaches have been developed for such methods (22 82). However, because they aim at improving a single solution rather than population of solutions, they tend to result in less exploration and novelty than evolutionary approaches [51]. One approach could be to use evolutionary search for recombination and use one of these non-evolutionary methods for local improvement. Such hybrid approaches have been used in other settings [6], and would be an interesting avenue of future work with RHEA.
B Illustrative Example
This section details the methods used in the formal synthetic example.
16

---

B.1
Definition of Utility Function
The utility predictor ø is defined to be compact and interpretable, while containing the kinds of nonlinearities leading to optimization challenges that RHEA is designed to address:
1, if 01 A A = {a1,a2]
2,
if C₁ A A {ai, 02 03 d4, as}
3,
if c A
{a1, 02 0g d1 ug (6)
4,
if C2 A A = {al, 02 032 d4, as as
((c, A) =
(I)
5,
if 02 A A = {ai, 02 0z as, ao)
1,
if A A = {az a4 a5}
1,
i (07, 0gg ag. a10)
0, otherwise.
In this definition, the non-zero-utility cases represent context-dependent synergies between policy interventions; they also represent threshold effects where utility is only unlocked once enough of the useful interventions are implemented. The interventions that are not present in these cases yield anti-s synerisess,,,s i.e. they negate any positive policy effects. The contexts e₁ and C2 represent similar but distinct contexts in which similar but distinct combinations of interventions are useful and can inform one another. In 62, a5 becomes redundant once ag included.
B.2 Analytic Distillation
Since the context and action spaces are discrete in this domain, preseriptors can analytically distilled based on the dataset describing their full behavior (i.c., the binary grids in Fig. 2). For example, these prescriptors can be distilled into rule-based or neural
prescriptors.
Consider rule- based prescriptors of the form: (ci...., Cm and A, S {ai...., } are subsets of the possible contexts and policy interven- tions, respectively. These prescriptors have variable number of rules r 2 O. Given a context e, (e) prescribes the first action A, such that e E C., and prescribes the empty action A. = if no C2 contains c. Then, the gathered expert prescriptors with behavior depicted in Fig. D e can be compactly distilled as का = [c] + {a1, (2)), T2 = [[2)) + {az, 04, {s)), and T3 = [[,, C2, C3. C1, C3: Co, (7) + {17, ag; 0g, 11 respectively.
[C1 + AI....,C, C, + Ar), where C, S
Similarly, consider neural-network-bee a1...., an. and hidden nodes with ReLU activation and no bias. For every unique action A; prescribed by a prescriptor æ, let C, be the set of contexts e for which (c)) A, Add a hidden node h; connected to each input e E C; and each output When using this model, include policy intervention a, the prescribed action if its activation is positive. Then, distilled versions of the expert prescriptors can be compactly described by their sets of directed edges: क = (((1, (hi, a1), (/1,a2)), ). 22 = {(22, l1), (h1 0z), ; (4), (h1, as)}, and {(c1, 1n), h1) (c2,11)....., (07, h1) (hi, 07), (hi, as), (h1, (9), (h1,a10) (11))
prescriptors with input nodes ci...., Co.. output nodes
A; Let all edges have weight one.
Both rules and neural networks provide distilled prescriptor representation amenable to evolutionary optimization.
B.3 Evolution of Analytically Distilled Models
For experimental verification, the distilled rule-set set models were used to initialize minimal multi- objective evolutionary Al process. This process was built from standard components including method for recombination and variation of rule sets, non-dominated sorting [16], duplicate removal, and truncation selection. In the RHEA setup, the distilled versions the gathered expert prescriptors were used to initialize the population and were reintroduced every generation. In the evolution alone setup, all instances of distilled models were replaced with random ones. The Python code for running these experiments can be found at https: : /ttp//ggttttttttt wtt.tttw..ooocmeeeeoeommeeeettte wtt. coom/connzaatt.... ai labs/ waaas/hhaaaaaeaaaa hte//rheaaaaa demo
17

---

front by Evolution generations
mean
% ofont 500 ger pemeratorr
moce
602
01.0
71.7
300-
63.6
generatione
46.5
222
26.4
21.3
600
20
35
policy Interventiionss
polioy interventionns ons
Figure 6: Experimental results comparing RHEA vs. Evolution alone (i.e., without knowledge of guthered expert solutions) in the illustrative domain. Whiskers show 1.5x the middle bar is the median. a, RHEA exploits latent expert knowledge to reliably and efficiently discover the full optimal Pareto front, even the number of available policy interventions increases (there are 2" possible actions for each context; 100 trials each). b, Evolution alone does not reliably discover the front even with 10 available interventions, and its performance drops sharply the number increases (100 trials each). Thus, diverse expert knowledge is key to discovering optimal policies.
B.4 Comparison to multi-objective reinforcement learning
Multi-objective reinforcement learning (MORL) is growing area of research that aims deploy the recent successes of reinforcement learning (RL) to multi-objective domains [29]. A natural question is: Is RHEA needed, or MORL methods be directly applied from scratch (without expert knowledge) and reach similar or better performance?
To answer this question, comparisons were performed with suite of state-of-the-an MORL tech- niques 23 in the Illustrative domain. Preliminary tests were run with several the recent algorithms, namely, GPI-LS (1) GPI-PD [I, and Envelope Q-Learning [2]. The hyperparameters were those found to work well in the most similar discrete domains the benchmark suite") Due to computa- tional constraints, the comparisons then focused GPI-LS for scaling up larger action spaces because (1) it has the best recorded results in this kind of domain (23], and (2) none of the other MORL methods in the suite were able outperform GPI-LS in the experiments. Note that the more sophisticated GPI-PD yields essentially the same results as GPI-LS in this discrete context and action domain.
In short, even the baseline multi-objective evolution method strongly outperforms MORL (Fig. The reason is that evolution inherently recombines blocks of knowledge, whereas MORL techniques struggle when there is no clear gradient of improvement.
C Pandemic Response Challenge
This section details the methods used in the application of RHEA to the XPRIZE Pandemic Response Challenge.
C.1 Distillation
the goal is to fit model with fixed functional form to capture the behavior of each initial solution, by solving the following minimization problem:
In distillation 3 30 31
gr = argmin
p(g)
TT(C) - *,
n: m:(g). (n)((mmm(()))))) 0): |
(2)
0.
1 12g
F argmin
(T(())) - #; ( ((4)))))
(3)
0:
ng
j=1
18

---

Convergence: 10 Actions
Convargence: 30 Actions
Converge toc: 50 Actions
4-b
GPIFD
%o furk huad
2.8
2.6
1B16
MISA
0.2
20-
00-
03-
3
b
Figure 7: Convergence curve comparisons. a-c, Convergence curves for 10, 30, and 50 actions, respectively, in the Illustrative domain. RHEA converges the full Pareto front in all cases, whereas the other methods converge to lower values as the action space grows. Evolution substantially outperformed the MORL baselines in all cases. With 10 actions, all MORL baselines converged relatively quickly to the same performance. Due to computational limitations, only the most relevant comparison, GPI-LS (which state-of-the the-art in discrete domains), was run the experiments with more actions (lines are means; shading is standard deviation).
% of Parato front found 500 generations
Evelution mean
Evolitoon: median
81.8
GP-LS:
24.5
27.3
91
0.
Figure 8: MORL scaling comparison. GPI-LS discovered less of the true Pareto front than the Evolution baseline (100 trials each). The performance of both methods decreases the problem complexity, i.e., the number of actions, increases. This plot complements Fig. 6b. Recall from Fig. 6 that RHEA discovers the entire Pareto front in all trials.
where q E Q is query, स. the initial solution, मं the distilled model with learnable parameters 0, and is function that maps queries (which may be specified via high-level API) to input data, i.c., contexts, with a canonical form that can be used to train ft In practice, क. is trained by optimizing 0, with stochastic gradient descent using data derived from the n, queries for which data available.
In the Pandemic Response Challenge experiment, prescriptors were distilled into an evolvable neural network architecture based on one previously used evolve preseriptors from scratch this domain [50] with the following changes: (1) In addition to the IPs used in that previous work, new IPs were used that were released in the Oxford data set since that work [27] 74), and that were used in the XPRIZE Pandemic Response Challenge; (2) Instead of case growth rate, the case data input to the models were presented as cases per 100K residents. This input was found to allow distilled models to fit the training data more closely than the modified growth rate used in previous work. The reason for this improvement that cases per 100K gives a more complete picture the state of the pandemic; the epidemiological maaogiicc--mmdll-liiiiieeeeeeeee ratio used in prior work captures the rate of change in cases explicitly but makes it difficult to deduce how bad an outbreak is at any particular moment. Since many diverse submitted prescriptors took absolute case numbers into account, including these values in the distillation process allows the distilled prescriptors to align with their source model more closely.
Data for training distilled model ân was gathered by collecting the prescriptions made by n; in the XPRIZE Pandemic Response Challenge. Data was gathered for all prescriptions made with uniform
19

---

Afghanistan, Albania, Algoria, Andorra. Angola, Argentina, Aruba, Australia, Austria, Azerbaijan, Bahamas, Bahrain. Bangladesh, Barbados, Belarus, Belgium, Belize, Benin, Bermuda, Bhutan, Bolivia, Bosnia and Herzegovina, Botswana, Brazil, Brunci, Bulgaria, Burkina Fiso, Burundi, Canbodia, Cameroon, Canada, Cape Verde, Central Africun Republic, Chad. Chile. China, Colombia, Comaros, Congo, Costa Rica, Cote d' I'oviree Croatia, Caha, Cyprus, Czech Republic, Democratic Republic of Congo. Denmark, Djibouti, Dominica, Dominican Republic, Ecuador, Eggpp, Bl Salvador, Eriirea, Estonia, Eswatimi, Ethiopia, Fueroe Islands, Fiji. Finland, France, Gabon, Gambia, Geurgia, Germany, Ghana, Greese, Greenland, Guam, Guatemala, Guinea, Guyana, Haiti, Honduras, Hong Kong. Hungary, Ioeland, India, Indonesia, Tran, Iraq, Ireland, Israel, Italy, Jamaica, Japan, Jordan, Kazakhstan, Kenya, Kosovo, Kuwair, Kyrgyz Republic, Laos, Latvia, Lebanon, Lesotho. Liberia, Libya, Lithoania, Luxembourg Macso, Madagasear, Malawi, Malaysia. Mali, Mauritania, Mauritius, Mexico, Moldova, Monaco, Mongulia, Moroevo, Mocambique, Myanmar, Namibia, Nepal, Netherlands, New Zealand, Nicuragua. Niger. Nigeria, Norway, Omnn, Pakistan, Palestine, Panama, Papua New Guinea, Paraguay, Penu, Philippines. Poland, Portugal, Puerto Rico, Qatar, Romania, Russia, Rwanda, San Marino, Saudi Arabia, Senegal, Serbin, Seychelles, Siera I Leone, Singapore. Slovak Republic, Slovenia, Solomon Islands, Somalia, South Africa, South Korea, South Sudan, Spuin, Sri Lanka, Sudan, Suriname, Sweden, Swizzerland, Syria, Taiwan, Tajikistan, Tanzania, Thailand, Timor-I Lste, Togo, Trinidad and Tobago, Tunisia. Turkey, Uganda, Ukraine, United Arab Emirates, United Kingdom Beglland United Kingdom Northern Ireland, United Kingdom / Scotland. United Kingdom / Wales, United Kingdom, United States/ / Alabema, United States/ / Alaska, United States Arizona, United States / Arkansas, United States / California, United States Colorado, United States/ Connecticut, United States/ Delaware, United States/ / Florida, United States / Georgia, United States / Hawaii, United States Idaho, United States / Illnnois, United States Indiana. United States / Towa, United States / Kansas, United States / Kentucky, United States Louisiana, United States / Maine, United States / Maryland, United States / Massachusetts, United States Michigan, United States / Minnesota, United States/ Mississippi. United States/ Missouri, United States / Montana, United States / NNehraska, United States / Nevada, United States / New Hampshire, United States / New Jersey, United States / New Mexico, United States / New York, United States North Carolina, United States North Dakota, United States / Ohio. United States / Oklahoma, United States / Oregon, United States / Ponnsylvania, United States Rhode Island, United States / South Carolina. United States South Dakota, United States/ / 'Tennessee. United States/ / Texas, United States/ Utah, United States / Vermont, United States/ / Virginia. United States / Washington, United States / Washington DC, United States/ / West Virginia, United States/ / Wisconsin, United States / Wyoming, United States, Uruguay, Uzhekistan, Vanuatu, Venezuela, Vietnam, Yemen, Zambia, Zimbabwe
Figure 9:
List of the 235 geos (i.c., countries and subregions) whose data (from the Oxford dataset was used in XPRIZE competition and in experiments in this paper.
27 54 74
IP weights. This data consisted of ten date ranges, each of length 90 days, and 235 geos (Fig 2) resulting in 212,400 training samples for each prescriptor, a random 20% of which was used for validation for early stopping. More formally, each (date range, geo) pair defines a query g. with (i) E 255 220x12 the policy generated by я; for this geo and date range. The predicted daily new cases for this geo and date range given this policy is ((g. n, (q) E R Let be the vector of daily historical new cases for this geo up until the start of the date range. This query leads to 90 training samples for मेद: For each day i, the target is the prescribed actions of the original prescriptor n. (g),. and the input is the prior 21 days of cases (normalized by 100K residents) taken from h for prior days before the start of the date range and from Φ(g. T, (4)) for days in the date range.
Distilled models were implemented and trained in Keras 77 using the Adam optimizer absolute error (MAF) was used as the training loss (since policy actions were on an ordinal scale), with targets normalized to the range [0, 1]. The efficacy of distillation was confirmed by computing the rank correlations between the submitted expert models the XPRIZE challenge and their distilled counterparts with respect to the two objectives: For both cases and cost, the Spearman correlation was & 0.7, with p < 10 10-20, 20 demonstrating that distillation was successful. In such a real-world scenario, a correlation much closer to 1.0 is unlikely, since many solutions are close together in objective space, and may have different positions on the Pareto front depending on the evaluation context.
35 Mcan
C.2 Evolution
In the Pandemic Response Challenge experiment, the evolution component was implemented inside using the Evolutionary Surrogate-assisted Prescription (ESP) framework (24), which was previously used to evolve prescriptors for IP optimization from scratch, i.e., without taking advantage of distilled models (50). The distillation above results in evolvable neural networks म ... Rn. which approximate Ti ... RR,, respectively. These distilled models were then placed into the initial population of run of ESP, whose goal is to optimize actions given contexts. In ESP, the initial population (i.e., before any evolution takes place) usually consists of neural networks with randomly generated weights. By replacing random neural networks with the distilled neural networks, ESP starts from diverse high-quality solutions, instead of low-quality random solutions. ESP can then be run as usual from this starting point.
In order to give all distilled models chance to reproduce, the "population removal percentage" parameter was set to 0%. Also, since the experiments were run as a quantitative evaluation of teams in the XPRIZE competition [10 [ 12) distilled models were selected for reproduction inversely
20

---

proportional to the number of submitted preseriptors for that team. This inverse proportional sampling creates fair sampling at the team level.
Baseline experiments were run using the exact same algorithm but with initial populations consisting entirely of randomly initialized models (i.e., instead of distilled models). The population size was 200; in RHEA, 169 of the 200 random NNs in the initial population were replaced with distilled models. Ten independent evolutionary runs of 100 generations each were run for both the RHEA and baseline settings.
The task for evolution was to prescribe IPs for 90 days starting February 12, 2021, for the 20 regions with the most total deaths at that time. Internally. ESP uses the Pareto-based selection mechanism from NSGA-II to handle multiple objectives [16].
The current experiments were implemented with ESP because it an already established method in this domain. Note, however, that such distillation followed by injecting in the initial population could be used in principle to initialize the population of any multi-ohjective evolution- -aaed method that evolves functions.
C.3 Pareto-based Performance Metrics
This section details the multi-objective performance evaluation approach used in this paper. is based on comparing Pareto fronts, which are the standard way of quantifying progress in multi-objective optimization. While there are many ways to evaluate multi-objective optimization methods, the goal in this paper is to do it in manner that would be most useful to a real-world decision-maker. That is, ideally, the metrics should be interpretable and have immediate implications for which method would be preferred in practice.
In the Pandemic Response Challenge experiment, each solution generated by each method m in the set of considered methods M yields a policy with a particular average daily cost e E [0, 34] and corresponding number of predicted new cases a 2 0 [50]. Each method returns set of N,, solutions which yield set of objective pairs S,, = {(,, a.) J=z Following the standard definition, one solution = (c1, a1) is said to dominate another 82 = (c2, .2) if and only if
(cl < C2 ai < <a²) V (cI S c2 ai < a2),
i.e., it is at least as good on each metric and better on at least one. If 81 dominates 82, we write 81 > 82- The Pareto front Fm of method is the subset of all s; (c,, az) Sm that are not dominated by any 8j = (cj: ag) E Sm. The following metrics are considered:
Hypervolume Improvement (HVI) Dominated hypervolume is the most common general- -urrpss metric used for evaluating multi-objective optimization methods [6]. Given a reference point in the objective space, it is the amount of dominated area between the Pareto front and the reference point. The reference point is generally chosen to be a "worst-possible" solution, so the natural choice in this paper the point with maximum IP cost and number of cases reached when all IPs are set 0. Call this reference point = (Co, a.). Formally, the hypervolume is given by
f 1 13s E Fm : 2 sAs So ds,
Hmmm)
(5)
where 1 is the indicator function. Note that HV can be computed in time linear in the cardinality of Fm. HVI, then, is the improvement in hypervolume compared to the Pareto front Fm, of a reference method m.:
HVI(m) HV HV
(6)
The motivation behind HVI is to normalize for the fact that the raw hypervolume metric is often inflated by empty unreachable solution space.
Domination Rate (DR) This metric is head-to-head variant of the "Domination Count" metric used in Phase 2 evaluation the XPRIZE, and goes by other names such as "Two-set Coverage" 61 It is the proportion of solutions in the Pareto front Fm. of reference method m. that are dominated by solutions in the Pareto front of method m:
DR(m)= = [Pmol
1(00 E Fmo (3s Fm : x s.)H
(7)
21

---

The above generic multi multi-objecctve metrics can be difficult to interpret from a policy- inileyimmpleennatioon perspective, since, e.g., hypervolume is in units of cost times cases, and the domination rate can be heavily biased by where solutions on the reference Pareto front tend to cluster. The following three metries are more interpretable and thus more directly usable by users of such a system.
Maximum Case Reduction (MCR) This metric is the maximum reduction in number of cases that solution on Pareto front gives over the reference front:
MCR(m) = max a. a, (8o- = (Co, (o) E Fm,:: 8. = (c,,a.) E Fm) : 8n ł 80
(8)
In other words, there is solution in F., such that one can reduce the number of cases by (CR(m). with no increase in cost. If MCR is high, then there are solutions on the reference front that can be dramatically improved.
The final two metrics, RUN and REM, are instances of the RI metric for multi-objectivv evaluation 28 [1]. which is abstractly defined the probability of selecting solutions from one set versus another given distribution over decision-maker utility functions.
R1 Metric: Uniform (RUN) This metric captures how often a decision-mate maker would prefer solutions from one particular Pareto front among many. Say a they are willing to pay when selecting a policy. The RUN for method m is the proportion of costs whose nearest solution on the combined Pareto front F. (the Pareto front computed from the union of all Fm A m E M) belong to m!
maker has particular cost
RUN(m) = c/mm
1 argmin ||c cx E Fm de, s.EF.
Gyin
Caiia
where s. (Ca, a.).. Here, Gmin O, and Gmax 34, since that is the of the maximum settings across all IPs. Note that RUN can be computed in time linear in the cardinality of F..
RUN gives complete picture of the preferability of each method's Pareto front, but is agnostic to the real preferences of prcision-makers. In other words, it assumes a uniform distribution over cost preferences. The final metric adjusts for the empirical estimations of such preferences, that the result more indicative real-world value.
R1 Metric: Empirical (REM) This metric adjusts the RUN by the real-world distribution of cost preferences, estimated by their empirical probabilities p(c) at the same date across all geographies of interest:
REM(m)
Crase
p(c) 1 argmin ||c ||c-c.1l c.ll E Fm de.
(10)
S.EF.
In this paper, p(c) is estimated with Gaussian Kernel Density Estimation (KDE; Fig. 3d), using the scipy implementation with default parameters (75). For the metrics that require reference Pareto front against which performance is measured (HVI, DR, and MCR), Distilled is used as this reference; it represents the numan-developee solutions, and the goal is compare the performance of Au (i.e. RHEA) to human alone.
All of the above metrics are used to compare solutions in Fig. of the main paper. They all consistently demonstrate that RHEA creates the best solutions and that they also would be likely to be preferred by human decision makers.
C.4 Analysis of Schedule Dynamics
The data for the analysis illustrated Fig. is from all submitted prescriptors and single runs of RHEA, evolution alone, and real schedules. Each point in Fig. 4 corresponds to schedule S 92 produced by policy for one of the 20 geos used in evolution. For visualization, each S was reduced to Š [o, 52 by taking the mean of IPs across time, and these 12-D Š vectors dist=1 and all other parameters
were processed via UMAP 45 with n__negghbors25 default.
Below are the formal definitions of the high-level behavioral measures computed from S and used in Fig. 4b. Let S [0, 34]90 be the total cost over IPs in S for each day.
22

---

Swing measures the range in overall stringency of a schedule:
Swing(S) max 2
S -
(11)
Separability measures to what extent the schedule can be separated into two contiguous phases of differing overall stringency:
} ト S , -, 5
Separability()) max
(12)
( E S. - 5})=
Focus increases as the schedule uses a smaller number of IPs:
Focus(S)
Σ 1(S 0).
(13)
Agility measures how often IPs change:
89 (S # 1(1-1))).
Agility(S)- = max Σ
(14)
Periodicity measures how much of the agility can be explained by weekly periodicity in the schedule:
Ee (Sax + 1) - 1Stk * S(L-TT))
Periodicity(S) = max 0, max
(15)
Σx SS ((()))a)
These five measures serve to distinguish the behavior of schedules generated by different sets of policies an aggregate level.
The violin plots in Fig. 4p were created with Seaborn 76 using default parameters aside from cut=0, scale= wiith', and linewidth=1 (https://seabon) www.aaab..pddaaaaaa.a.aaaaaaaaaa
gergg//nerttteeeeeeeeeeee seaborr
violinplot hhtminploo.tmm....... The violin plots have small embedded boxplots for which the dot is the median, the box shows the interquartile range, and the whiskers show extrema.
C.5 Pareto Contributions
To measure the contribution of individual models, the ancestry individuals on the final Pareto front of RHEA is analyzed. For each distilled model RG the number of final Pareto front individuals who have ft as an ancestor is counted, and the percentage of genetic material on the final Pareto front that originally comes from is caleulated. Formally, these metrics computed recursively. Let Par(m) (T) be the parent set of π in the evolutionary tree. Individuals in the initial population have an empty parent set; individuals in further generations have parents. Let F be the set of all individuals on the final Pareto front. Then, the ancestors of æ are
Ø (UrEPa(A)) Ane(m') U Par(T)) otherwise,
Par( Ø,
Anc(T)
(16)
and the Pareto contribution count is
PCCount(T) = HT' : E Anc(T') and x E F),
(17)
while the percentage of ancestry of ग due to x'
Par( = Ø, π 7 n², Par( = 6,, π n²
APercent,
(18)
[Par(x) Σ "EPa(())) APercent, (x")
otherwise,
with the Pareto contribution percentage
PCPercent(T) (T)
Σ
APercent, (T')
(19)
F
x'EF
In the experiments, these two metrics are highly correlated, so only results for PCPercent are reported (Fig.) 5
23

---

C.6 Energy Estimates
The relatively compact model size in RHEA makes accessible and results in a low environmental impact. Each run of evolution in the Pandemic Response experiments an estimated 3.9 x 10 This computation is orders-of-min -magnnitd less energy intensive than many other current AI systems: For instance, training AlphaGo took many limited-availability and expensive TPUs, consuming & 8.8 x J [3]: training standard image and language models on GPUs can consume = 6.7 x 1 84 and 2 6.8 x 10"l] (77,, respectively. Specifically:
16 CPU cores, consuming
Each training run of RHEA for the Pandemic Response Challenge experiments takes 29 hours on 16-core m5a.4xlarge EC2 instance. At 100% load, this instance runs at 1220W littps://engineerinng teads teads wwt./com/sssaaeeee sustainabbiity wtt.:pprrrnneeeetiimmtto-o-o-ooe wwwtasssiitsssssss.nss yielding total of x 3.9 x
10
The energy estimate of training AlphaGoo was based on httpps:/wwww /uupyyuhhww.... com data/ agz cost. html] with 6380 TPUs running 40W for 40 days, yielding a total of R 8.8 x 10
The energy estimate for image models is based on training ResNet-50 on ImageNet for 200 epochs on a Tesla M40 GPU. The training time is based on https://axiivorrg/att 709.05011 [84: the energy was computed from httph:ttt/mmmtte
github. io/ hms/ac//p//
The energy estimate for language models was based on an estimate of training GPT-3 (See Appendix D in http://aaxxivoorg//d///000/ http:///d/...........o... 033051.pdff (77).
Each training run of RHEA in the Illustrative Domain takes only few minutes on a single
CPU.
D Ethics
This section considers ethical topics related to deploying RHEA and similar systems in the real world.
Fairness. Fairness constraints could be directly incorporated into the system. RHEA's multi-objective optimization can use any objective that can be computed based on the system's 'ss behavior, so a fairness objective could be used if impacts on the subgroups can be measured. A human user might also integrate this objective into their calculation of unified Cost objective, since any deviation from ideal fairness is a societal cost. In deployment, an oversight committee could interrogate any developed metrics before they are used in the optimization process to ensure that they align with declared societal goals.
Governance and Democratic Accountability This is key topic of Project Resilience (55), whose goal is to generalize the framework of the Pandemic Response Challenge to SDG goals more broadly. We are currently involved in developing the structure of this platform. For any decision-making project there are four main roles: Decision-maker, Experts, Moderators, and the Public. The goal is to bring these roles together under a unified governance structure.
At a high-level, the process for any project would be: the Docision-maker defines the problem for which they need help: Experts build models for the problem and make them (or data to produce Dis- tilled versions) public; Moderators supervise (transparently) what Experts contribute (data, predictors or prescriptors); The Public comments the process, including making suggestions in particular contexts, and on ways to improve the models (e.g., adding new features, or modifying objectives); Experts incorporate this feedback to update their models; after sufficient discussion, the Decision-makes uses the platform to make decisions, looking at what the Public has suggested and what the models suggest, using the Pareto front make sense of key trade-off'; The Decision-male communicates about their final decision, i.e., what was considered, why they settled on this set of actions, etc. In this way, key elements of the docision-making process are transparent, and decision- makers can be held accountable for how they integrate this kind of AI system into their decisions. By enabling a public discussion alongside the modeling/optimization process, the system attempts to move Al-assisted decision-making toward participative democracy grounded in science. The closest example of an existing platform with similar interface https//www.mettauluu mew.culusu com/home/, but is for predictions, not prescriptions, and problems are not linked to particular decision-makers
what to do
24

---

It is important that the public has access to the models via an "app", giving them a way to directly investigate how the models are behaving, as in existing Project Resilience proof-of-conceple along with a way of flagging any issuescconcems/nsighn they come across. A unified governance platform like the one outlined above also would enable mechanisms of expert vetting by the public, decision-makers, or other experts.
The technical framework introduced in this paper provides mechanism for incorporating demo- cratically sourced knowledge into a decision-making process. However, guaranteeing that sourced knowledge is democratic is a much larger (and more challenging) civil problem. The concepts of power imbalances and information asymmetry are fundamental to this challenge. Our hope is that, by starting to formalize and decompose decision-makinn making processes more clearly, it will become easier to identify which components of the process should be prioritized for interrogation and modification, toward the goal of system with true democratic accountability.
For example, the formal decomposition of RHEA into Define, Gather, Distill, Evolve, enables each step be interrogated independently for further development. The implementation in the paper starts with the most natural implementation of each step as proof-of-concepl, which should serve as a foundation for future developments. For example, there is major opportunity to investigate the dynamics of refinements of the Distill step. In the experiments in this paper, classical aggregated machine learning metrics were used to evaluate the quality of distillation, but in a more democratic platform, experts could specify exactly the kinds of behavior they require the distillation of their models to capture. By opening up the evaluation of distillation beyond standard metrics, we could gain new view into the kinds of model behavior users really care about. That said, methods could also be taken directly from machine learning, such as those discussed App.[A] However, we do not believe any of these existing methods are at point where the humans can be removed from the loop in the kinds of real-world domains the approach aims to address.
Data Privacy and Security. Since experts submit complete prescriptors, no sensitive data they may have used build their prescriptors needs be shared. In the Gather step, each expert team had an independent node to submit their prescriptors. The data for the team was generated by running their prescriptors on their node. The format of the data was then automatically verified, to ensure that it complied with the Defined API. Verified data from all teams was then aggregated for the Distill & Evolve steps. Since the aggregated data must fit an API that does not allow for extra data to be disclosed, the chance of disclosing sensitive data in the Gather phase is minimized. One mechanism for improving security is to allow the user of each role to rate sources, data, and models for a quality, reliability, and security standpoint, similar to established approaches in cybersecurity[]
External Oversight, Although the above mechanisms all could yield meaningful steps in addressing a broad range of ethical concerns, they cannot completely solve all issues of ethical deployment. So, it critical that the system not deployed in an isolated way, but integrated into existing democratic decision decision-making processes, with appropriate external oversight. Any plan for deployment should include disclosure of these risks to weigh against the potential societal benefits
Sustainability and Accessibility. See App. C.6 for details on how energy usage estimates were computed.
E Data Availability
The data collected from the XPRIZE Pandemic Response Challenge (in the Define and Gather phases) and used to distill models that were then Evolved can be found on AWS S3 at hitps //s3. 1ss3.us-oe west- 2. amazonaws com cowid- xprize- anon (i.c., in the public S3 bucket named coovid-pprze--nonn
so it is also accessible via the AWS command line). This is the raw data from the Challenge, but with the names of the teams anonymized. The format of the data is based on the format developed for the Oxford COVID-19 Government Response Tracker 27
25

---

Code Availability
The formal problem definition, requirements, API, and code utilities are for the XPRIZE, including the standardized predictor, are publicly available IOO 9.. The prediction and prescription API, as well as the standardized predictor used in the XPRIZE and the evolution experiments can be found at /ttps::/gi gitbb....b.... com hamm//oggizaatteee ai 1abs/ coooss/oviixpppp experiments was implemented a proprietary implementation of ESP framework, but the algorithms used therein have been described in detail in prior work (24. Code for the illustrative domain was implemented outside of the proprietary framework and can be found at https:// github. www.coonizanttaaaeee.coo ai labs thea-dent
The Evolve step in the
26

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction clearly describe the paper's contributions. The main contributions are listed the end of the introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Limitations are explicitly discussed in the Discussion.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "I "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
27

---

Answer: [Yes]
Justification: The only analytical result the distillation of experts in the illustrative example, which is clearly described under Analytic Distillation in the Appendix. The Theoretical Motivation discussed in the Discussion includes the critical assumptions and references the foundational work.
Guidelines:
The answer NA means that the paper does not include theoretical results. the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof' can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Code for the illustrative domain accompanies the paper. The implementation of the Pandemic Response Challenge experiments is well-detailed in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
28

---

5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: The paper provides open access to the code for reproducing the illustrative domain results. The Pandemic Response Challenge results used proprietary implementation of ESP (24), the implementation is well-detailed in the Appendix, and the framework introduced not inherently tied to this implementation. As mentioned in Code Availability, the data used to distill prescriptors can be found on AWS S3 at httttps://s3 us us-west-2 amazonaws.. , com/ covid- covid-xprize xprize-aae
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
public/guides// htwas//oooooubbssiooooooooooooo
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
The specifications detailed in Sections C.1 and
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: The main results include error bars, and raw data are plotted when there are 10 fewer data points.
Guidelines:
The answer NA means that the paper does not include experiments.
29

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results are out range (e.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The resources used are detailed under Carbon Footprint in the Discussion and under Energy Estimates (C.6).
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics http://ap/aaaaa...c////////////a http:////c../.ccccccccieeeeeeeee
Answer: [Yes]
Justification: Like many decision-making tools, the results of RHEA could be used justify decisions of actors with bad intentions. As detailed in the Broad Impacts section of the discussion, we hope the auditability of RHEA will help ameliorate these concerns.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
30

---

Justification:
These are discussed in subsections in the Discussion.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (c.g., pretrained language models, image generators, scraped datasets)?
Answer: [NA]
Justification: No data or models released in the paper have a high risk for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
The paper cites the sources of the data and existing framework components.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
31

---

The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperssi thoode com comddatasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The format of the released data is well-documented in the cited XPRIZE repo and the released code well documented in the rhea-demo repo.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
No crowd-sourcing experiments or research with human subjects was done.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper contains research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
32

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
33


Reviews:
[{"review_id": "7MSPUquiis", "reviewer": "Reviewer_R6Y1", "summary": "This paper introduces RHEA (Realizing Human Expertise through AI), a framework for combining diverse human expertise to solve complex problems using artificial intelligence. The key contributions are:\n* Recognizing the challenge of integrating diverse human expertise to solve global problems like those in public health.\n* Identifying requirements for an AI process to effectively combine human expertise.\n* Proposing the RHEA framework to meet these requirements, consisting of four steps: 1) Define the problem formally 2) Gather diverse expert solutions 3) Distill solutions into a canonical form (ie neural networks) 4) Evolve the distilled solutions to discover improved solutions\n* Demonstrating RHEA on a synthetic example.\n* Applying RHEA to a real-world problem: optimizing COVID-19 intervention policies.\n* Analyzing, for that real world problem: how RHEA recombines and innovates upon human expertise to discover improved solutions.\n\nThe paper shows (at least for their examples) that RHEA can discover broader and more effective policy strategies than either AI or human experts alone. It highlights RHEA's ability to realize latent potential in diverse human expertise, even from solutions that may not seem immediately useful. The authors argue this approach could help bridge the gap between human-only and AI-only decision-making for complex global problems.", "questions": "* How confident are you that RHEA can be applied effectively to domains beyond COVID-19 policy optimization? Could you provide examples of other complex problems where you believe RHEA would be particularly effective, and explain why? And also where it would not be effective?\n\n* Have you conducted any comparisons between RHEA and other state-of-the-art methods in policy optimization or ensemble learning? If so, what were the results? If not, which methods do you think would be most relevant for comparison?\n\n* How does the computational complexity of RHEA scale with the number of experts and the complexity of the problem space? Are there any foreseeable bottlenecks in applying RHEA to much larger or more complex problems?\n\n* Have you conducted any theoretical analysis of RHEA's properties, such as convergence guarantees or performance bounds? If so, could you elaborate on these? If not, what kind of theoretical results do you think would be most valuable to pursue?\n\n* How interpretable are the solutions produced by RHEA compared to the original expert inputs? Can you provide a detailed case study of a specific evolved solution, explaining how it combines and improves upon the original expert knowledge?\n\n* Could you provide more details on the distillation process? How do you ensure that the neural network accurately captures the essence of the expert's solution, especially for complex or nuanced strategies? For example, you find a Spearman correlation of 0.7 which does not seem particularly high to me. Also, I assume this 0.7 on the test set? What is the spearman correlation on the training set?\n\n* Does RHEA have mechanisms to incorporate new expert knowledge or feedback over time? How adaptable is the framework to changing conditions or new information?", "limitations": "The authors discuss the potential for approximation errors when applying the policies to the real world.\nThe paper also mentions the need for user studies to fully evaluate the real-world effectiveness of RHEA prescriptors. The authors recognize that their cost measure was uniform over interventions, which may not reflect real-world scenarios.\n\nI think this is generally a good set of limitations. Potential areas where the authors could improve:\n* The authors could explore more deeply the potential long-term societal impacts of widespread adoption of AI-assisted policy-making tools like RHEA.\n* While briefly mentioned, the paper could elaborate on how to ensure the RHEA process remains transparent and interpretable to non-expert stakeholders.\n* The limitations section could address how RHEA might handle rapidly changing conditions or unexpected scenarios not covered in the initial expert knowledge.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Originality: The paper presents a novel framework, RHEA (Realizing Human Expertise through AI), which creatively combines human expertise with an evolutionary framework. This approach is original in several ways: The framework introduces a process of distilling human-created solutions into a canonical form (neural networks) that can then be evolved using population-based search.\n\nQuality: The authors provide a step-by-step explanation of the RHEA framework, though it’s unclear to me to how immediately useful those steps are as they are quite high level.\n* The framework is tested first on a synthetic example making it easier to understand.\n* The analysis of results on the COVID example is thorough, using multiple performance metrics and visualizations to support their claims.\n* The authors acknowledge some of the limitations and potential issues\n\nClarity: The paper is generally well-structured and clearly presented:\n* The paper is well written.\n* The introduction effectively sets up the problem and the paper's contributions.\n* The illustration is helpful for explaining the concept.\n* The use of figures, particularly in explaining the RHEA framework and visualizing results, aids in understanding.\n\nSignificance: The paper's significance lies in its potential impact on AI research and applications:\n* It addresses the important challenge of leveraging human expertise in AI systems, which is crucial for tackling complex global problems.\n* The RHEA framework could be applied to a wide range of domains beyond COVID-19 policy optimization.\n* The work demonstrates a practical approach to combining diverse expert knowledge, which could be valuable for collaborative problem-solving in various fields.", "weaknesses": "Here's a substantive assessment of the weaknesses of the paper:\n\n* Limited generalizability: The paper's primary application focuses on COVID policy optimization, which, while relevant, may not sufficiently demonstrate the framework's broad applicability. The authors could strengthen their claims by: a) Providing a theoretical analysis for why RHEA should work in other domains. b) Discussing potential applications in other fields, with specific examples of how RHEA might be adapted. \n\n* Novelty: The evolutionary algorithm comes from other work so the idea is basically to initialize the parameters with distilled neural networks. This is a powerful way to solve this specific problem but given the limited analysis of the algorithm both empirically (when does it work) and theoretically, I am not sure if this work should be published at an ML conference or if it would be better at a more general purpose venue. \n\n* Insufficient comparison to existing methods: While the paper compares RHEA to evolution alone and distilled models, it lacks comparison to other state-of-the-art methods in policy optimization or ensemble learning. The authors should: a) Include comparisons to relevant baseline methods (e.g., reinforcement learning approaches, other multi-objective optimization techniques). b) Discuss how RHEA compares to other methods of combining expert knowledge, such as boosting or stacking.\n\n* Generalizability concerns: The paper does not sufficiently address what the operating regime is of RHEA. What problems does it work? What's the scale at which it works? Does it work if you have thousands of experts? \n\n* Lack of Theoretical guarantees: While the empirical results are promising, the paper lacks theoretical analysis of RHEA's properties. The authors could strengthen the paper by: a) Providing theoretical bounds on the performance of RHEA under certain conditions. b) Analyzing the convergence properties of the evolutionary process. c) Discussing how the distillation process affects the solution space and optimization landscape.\n\n* The structure of the paper: Some important details are in the appendix while  there is a lot of analysis of the specific solutions and details (like the carbon footprint)  which should be in the appendix. For example, as the specific ways this work is technically novel are unclear, the authors should move some of the related work to the main text.\n\nAddressing these weaknesses would significantly strengthen the paper, providing a more comprehensive and robust presentation of the RHEA framework and its potential impact.\n\nPost rebuttal update:", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Our responses to remaining comments are grouped by topic:\n\n_NeurIPS vs. general science venue:_\n\nWe agree that this approach could be well-appreciated by broad audiences. Indeed, NeurIPS encourages application papers that have broad appeal, and we believe the introduction of this framework to the NeurIPS community could catalyze ML researchers to consider practical concerns that lead to interesting ML problems to solve. Consequently, in the revision we will add additional comparisons to multi-objective reinforcement learning (MORL) methods, expand the theoretical analysis, and showcase technical details and context in the main paper, thus strengthening this impact.\n\n_Comparisons:_\n\nW.r.t. comparisons to ensembling methods, the MoE comparison in the Illustrative Domain (Figure 2) serves as a multi-objective Oracle for stacking, since it shows the performance of a stacker that optimally selects experts based on context. Boosting does not apply to our problem: it relies on experts having access to each other’s models, whereas we assume that the expert models are submitted in a single traunch. We will clarify these points in the paper. This also relates to your question on adaptively adding new experts during optimization, which is discussed below. \n\n_Paper structure:_\n\nThank you for this feedback. We will move the most relevant portions of the related work to the main text, specifically the discussion of multi-objective reinforcement learning and ensembling methods. We will also move technical methodological details to the main paper, such as those on distillation, to make the main paper technically self-contained.\n\n_Scaling to more complex problems:_\n\nThe population-based search used by RHEA extends readily to thousands of experts; the theoretically optimal population size is often quite large [4, 5]. We will note these details in the final version of the paper.\n\nBottlenecks could come from adversarial attacks in submissions based on overwhelming the system with non-useful experts. In this paper we have assumed that the humans submitting solutions are well-intentioned. We will mention this potential concern in the discussion.\n\n_Interpretable case study:_\n\nFirst, the illustrative domain provides a clear case study exactly along the lines suggested. Second, in the COVID-19 domain we get at this quantitatively through the topological analysis of the final solutions (Figure 4) and evolutionary dynamics (Figure 5). Figure 4 illustrates several complete schedules generated from RHEA solutions, and Figure 5 shows how evolved solutions in RHEA solutions tend to lie between their parents in objective space. We will add a detailed case study of a single solution from Figure 4 and its expert parents to make the interpretation more detailed and concrete.\n\n_Distillation process:_\n\nThe process is detailed in the Appendix; details will be moved to the main paper. While other implementations are possible, this one is simple, intuitive, and was shown to work well based on the quantitative results in the overall framework. In the future, we can investigate different distillation processes and their effects on the overall framework. In this kind of real-world scenario, correlation much closer to 1.0 is unlikely, since many solutions are close together in objective space, and may have different positions on the Pareto front depending on the evaluation context.\n\nFull autoregressive schedule roll-outs were not generated in the Distill step, as the models were trained on next-step prediction. The mean MAE over distilled models on both the training and validation splits of the distillation dataset was ~0.1, i.e., the mean difference between the actual and predicted actions (which could range from 0-5) was small. \n\n_Online adaptation:_\n\nThis is a great avenue for future work! The present implementation of RHEA naturally supports the incorporation of new expert knowledge over time, since it maintains a population of solutions over time, so new expert solutions could at any time be distilled and inserted into the populations. Since it is likely that the objective quality of the new knowledge falls somewhat behind the current Pareto front, a mechanism might be required to ensure the new knowledge survives for long enough in the population to be effectively exploited.\n\nIt would also be possible to update the predictor based on new data, as has been demonstrated in earlier work on evolutionary surrogate-assisted prescription [Francon et al 2020], and even increase the number of context or action variables online (e.g., in the case of pandemic interventions, new kinds of interventions could be implemented).\n\n_Long term societal impacts:_\n\nThis is a very interesting social question. One risk with AI-assisted policy-making tools is that as human users come to rely on them more there may be less incentive to gather useful and diverse knowledge from human experts. The hope with RHEA is that, by centering the value of soliciting and gathering human expert knowledge, humans will be rewarded for the perspectives they share, and it will encourage expert diversity to thrive. We will clarify this discussion of risk/reward trade-off in the discussion of the paper.\n\n_Transparency:_\n\nRHEA can already assign contribution percentages of submitted experts to final solutions, as depicted in Figure 5. One can further imagine an interface, where upon selecting a solution for inspection, a decision-maker is alerted to who contributed to this solution, at which point they have the opportunity to investigate the contribution, e.g., raise a flag if many selected solutions have the majority of their contribution from a single actor or small group of actors. Analysis of such a scenario could reveal outstanding objective expertise by this group, or some unintended bias in the system. We will clarify this point in the discussion. \n\n\nThank you again for articulating these issues. We agree that our response strengthens the work and its impact.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SSGVsScTqS", "reviewer": "Reviewer_fdTu", "summary": "The authors introduce a new evolutionary framework for combining expert policy predictions called RHEA and evaluate performance in both a synthetic domain that is highly interpretable and in a predictor from the XPRIZE Pandemic Response Challenge. The solutions are analyzed by qualitatively and quantitatively and compared against less sophisticated baselines.", "questions": "I did not get the sense that the “policy intervention problem” is something that is well studied. How is the problem statement here different from a contextual bandit or similar formalisms (knapsack problems?)\nAre there other baselines for this problem besides lesioned versions of the current framework? \nWhat are the computational differences between RHEA and the baselines was each algorithm run until convergence? Is it possible to show learning curves?", "limitations": "Yes", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is clearly written and focuses on an important problem. I liked that the authors explain the framework in the context of a highly interpretable example and then do significant qualitative and quantitative analysis on the solutions discovered by in the XPRIZE domain.", "weaknesses": "See questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> I did not get the sense that the “policy intervention problem” is something that is well studied. How is the problem statement here different from a contextual bandit or similar formalisms (knapsack problems?) Are there other baselines for this problem besides lesioned versions of the current framework? What are the computational differences between RHEA and the baselines was each algorithm run until convergence? Is it possible to show learning curves?\n\n*Response to Questions above:*\n\nThese clarifications are indeed useful; thank you for pointing them out. The problem that the Evolve portion of the framework solves is formulated as a general decision-making problem. It can indeed be framed for multi-objective reinforcement learning (MORL) by casting the predictor as an environment that prescriptors interact with as they learn. We ran such an experiment for this response, and describe the results in our Main Response. In short, the state-of-the-art MORL methods do not do well on these kinds of problems because they have a hard time recombining blocks of knowledge in useful ways, whereas this process is natural in evolutionary methods.\n\nIn the pdf attachment to our Main Response, we provide learning curves for RHEA, Evolution, and the MORL methods. Each was given the same number of evaluations, i.e., calls to the predictor. In terms of wall-clock time, RHEA and Evolution were also much faster than the MORL methods, especially model-based methods that require expensive gradient updates. The theoretical analysis in the Main Response also addresses questions of method convergence.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bfyGqQoZPO", "reviewer": "Reviewer_CmcD", "summary": "This paper designs a framework called Realizing Human Expertise through AI (RHEA) to combine and distill solutions from a diverse set of models or solutions provided by experts. The steps are: Define Problem -> Gather Solutions -> Distill Model -> Evolve Solution. Examples were presented to demonstrate the framework's effectiveness in different domains.", "questions": "1) The concern is the efficiency of this framework, as the Gather step requires waiting for experts to respond. The framework involves a human-in-the-loop process. How can bias be reduced when gathering solutions from the experts? (if these experts' solutions are biased or limited)\n2) During a crisis, what happens if there are insufficient solutions? If there are two opposing solutions, what will be the outcome?\n3) The description of the Distill process lacks sufficient detail for the reader to understand how it works.", "limitations": "This work has limited real-world evaluation. Fairness also needs to be considered in developing this framework.", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The author proposes a potential solution to collective knowledge that can be combined to discover solutions. It could be a way to find a better solution from the information stream when there is more than one solution.", "weaknesses": "1. Some explanations of the framework are unclear and require additional details to better explain the \"Distill\" process. \n2. The contribution paragraph needs more information to clearly describe the real contribution. \n3. The conclusion is too broad and needs to be more specific.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "_Response to question on details of the Distill process:_\n\nWe will move the core details of Distill from the Appendix to the main text and ensure that there is enough in the main text to have a clear idea of the process.\n\n_Response to suggestions to add more specific details to the Contributions and Conclusion paragraphs:_\n\nThank you for the suggestion. We will revise both these paragraphs to focus more on the technical contributions of the paper, specifically, details of the framework, implementation, and experiments.\n\n_Response to questions around the effects of having humans playing a role in the process:_\n\nThese are very interesting practical considerations. This work focuses on the technical aspect of how to set up the framework, but the problem of how to improve efficiency in gathering expert solutions is a social/civil problem. It could be improved e.g. through improved communication/solicitation methods. Such considerations are outside the scope of this paper, but constitute interesting avenues of future work, as will be noted in the discussion.\n\nOn the other hand, RHEA has a natural method to overcome bias in expert solutions, since, through the process of evolution, the objectively highest quality components can recombine and persist. If expert knowledge is limited, RHEA will simply revert to the evolution baseline.\n\n_Response to: “During a crisis, what happens if there are insufficient solutions? If there are two opposing solutions, what will be the outcome?”_\n\nThis is exactly the problem that RHEA is designed to solve! RHEA yields a broad Pareto front of solutions, to maximize the chance that one will be useful in any particular scenario.\n\n_Response to “This work has limited real-world evaluation”:_\n\nWhile further real-world evaluation would always be useful, this paper actually includes an unusually substantial such evaluation already. The COVID-19 intervention application required soliciting, incentivizing, and gathering the input of over 100 teams of experts in over 20 countries. Such a real-world demonstration of the Define and Gather steps at scale is a major and unique contribution of this work. Further, the optimization results were evaluated with an expansive dataset of real-world pandemic interventions and outcomes in over 200 countries and two years—the first such dataset to be created. We will clarify these contributions in the paper.\n\n_Response to “Fairness also needs to be considered in developing this framework.”:_\n\nRHEA has mechanisms for fairness, by assigning probabilities to expert knowledge for recombination, and providing traces of where contributed knowledge comes from. However, we agree that it is possible for fairness issues to enter the system, especially if actions with bad intentions submit solutions (as also outlined in the response to Reviewer 1). Detecting and preventing such adversarial behavior is an interesting avenue for future work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Main Response:

Thank you to the reviewers for the feedback. We were glad the reviewers agreed that the work was valuable, and suggested how it could be strengthened further by addressing some outstanding questions. This main response focuses on three main points brought up by multiple reviewers:

1. Comparisons: We have performed further experimental comparisons to multi-objective reinforcement learning (MORL) methods, highlighting how they fall short, creating a research opportunity for RHEA.
2. Theory: We have provided theoretical justification for the main methodology of evolutionary optimization.
3. Generalizability: We have clarified the scope of the method and detailed how it can be applied to other domains.

**Comparisons**

In response to questions about additional baselines, we performed comparisons to a suite of MORL techniques [1] in the Illustrative domain. We ran preliminary tests with several of the recent algorithms, namely, GPI-LS [2], GPI-PD [2], and Envelope Q-Learning [3]. Due to computational constraints, we then focused on GPI-LS for scaling up to larger action spaces because (1) it has the best recorded results in this kind of domain [1], and (2) none of the other MORL methods in the suite were able to outperform GPI-LS in our experiments.

In short, the baseline multi-objective evolution method strongly outperforms MORL (see plots in rebuttal pdf). The reason is that evolution inherently recombines blocks of knowledge, whereas MORL techniques struggle when there is no clear gradient of improvement. We will add this discussion and the code for the MORL comparisons to the paper and its online supplement. Further, we will move the most critical portions of Related Work from the Appendix to the main paper to clarify the motivation for these comparisons.

**Theory**

The theory will depend on the particular implementation of the RHEA framework. For this paper, we can provide a theoretical analysis based on recent convergence analysis of NSGA-II [4, 5], which is the multi-objective evolutionary algorithm used in the paper. 
In theoretical settings, the performance of this algorithm has been shown to depend critically on the size of “jumps” in the optimization landscape, roughly, the maximum size of non-convex regions in the landscape. When these regions are minimal, the method converges to the full ground truth Pareto front in $O(N n \lg n)$ evaluations. When the jump size increases, the best known bound is $O(N^2 n^k / \Theta(k)^k)$, where $k$ is a measure of the jump size, $n$ is the problem dimensionality, and $N$ is the population size. In other words, an increasing jump size causes a roughly exponential slowdown of convergence.

Distilling useful, diverse experts can be viewed as a way of decreasing the jump size. This process is clearly shown in the illustrative domain, where the experts provide building blocks that can be immediately recombined to discover better solutions, but that are difficult to discover from scratch. This interpretation is borne out in the experiments, as RHEA continues to converge quickly as the action space (i.e. problem dimensionality) increases, whereas evolution regresses to only being able to discover the most convex (easily-discoverable) portions of the Pareto front.

We will articulate this theoretical framework and the requisite assumptions fully in the revision.

**Generalizability**

RHEA can be applied effectively to policy-discovery domains where (1) the problem can be formalized with contexts, actions, and outcomes; (2) there exist diverse experts from which solutions can be gathered; and (3) the problem is sufficiently challenging. In contrast, RHEA would not be effective, (1) if the problem is too easy, so that the input from human experts would not be necessary; (2) if the problem is hard, but no useful and diverse experts exist; (3) if there is no clear way to define context and/or action variables upon which the experts agree. These aspects of generality will be clarified in the revision.

The modularity of the overall framework means that different implementations of components can be designed for different domains, such as those related to sustainability, engineering design, and public health. One particularly exciting opportunity for RHEA mentioned several times in the paper is climate policy. The paper gives the example of defining “green hydrogen” as a concrete example of the kind of challenge RHEA could solve, but this could be just a small part of a climate policy application. For example, the En-Roads climate simulator supports diverse _actions_ across energy policy, technology, and investment; _contexts_ based on social, economic, and environmental trajectories; and multiple competing _outcomes_, including global temperature, cost of energy, and sea-level rise (https://en-roads.climateinteractive.org/). Users craft policies based on their unique priorities and expertise. RHEA could be used with a predictor like En-Roads to discover optimized combinations of expert climate policies that trade-off across temperature change and other the outcomes that users care about most. We will add this discussion to the paper.



A point-by-point response to remaining comments in each review is provided below. The same reference numbers are used throughout the responses.

[1] Felten, et al. “A Toolkit for Reliable Benchmarking and Research in Multi-Objective Reinforcement Learning”. NeurIPS 2023. https://github.com/LucasAlegre/morl-baselines.

[2] Alegre, et al. “Sample-Efficient Multi-Objective Learning via Generalized Policy Improvement Prioritization”. AAMAS 2023.

[3] Yang, Sun, and Narasimhan. “A Generalized Algorithm for Multi-Objective
Reinforcement Learning and Policy Adaptation”. NeurIPS 2019.

[4] Doerr and Qu. “From Understanding the Population Dynamics of the NSGA-II to the First Proven Lower Bounds”. AAAI 2023.

[5] Doerr and Qu. “Runtime Analysis for the NSGA-II: Provable Speed-Ups from Crossover”. AAAI 2023.

Meta-review: This paper proposes a framework for aggregating expertise to help inform policymaking with steps: Define Problem -> Gather Solutions -> Distill Model -> Evolve Solution.

All reviewers agreed that the approach was interesting, novel, and potentially impactful, and appreciated the exposition around the real-world importance of the problem. There were significant questions about whether the approach would generalize beyond the specific question where it was tested, but reviewers were willing to overlook this since the area as a whole is quite new and has such clear  connections to so many other areas of emerging importance for NeurIPS like democratic accountability, social welfare measurement, etc.
Decision: Accept (poster)