Quality-Diversity through AI Feedback
Authors: Herbie Bradley
Andrew Dai
Hannah Benita Teufel
Jenny Zhang
Koen Oostermeijer
Marco Bellagente
Jeff Clune
Kenneth Stanley
Gregory Schott
Joel Lehman
Venue: ICLR 2024 poster (2024)
Keywords: quality diversity
large language models
derivative-free optimization
AI feedback
Primary area: generative models

Abstract: In many text-generation problems, users may prefer not only a single response, but a diverse range of high-quality outputs from which to choose. Quality-diversity (QD) search algorithms aim at such outcomes, by continually improving and diversifying a population of candidates. However, the applicability of QD to qualitative domains, like creative writing, has been limited by the difficulty of algorithmically specifying measures of quality and diversity. Interestingly, recent developments in language models (LMs) have enabled guiding search through \emph{AI feedback}, wherein LMs are prompted in natural language to evaluate qualitative aspects of text. Leveraging this development, we introduce Quality-Diversity through AI Feedback (QDAIF), wherein an evolutionary algorithm applies LMs to both generate variation and evaluate the quality and diversity of candidate text. When assessed on creative writing domains, QDAIF covers more of a specified search space with high-quality samples than do non-QD controls. Further, human evaluation of QDAIF-generated creative texts validates reasonable agreement between AI and human evaluation. Our results thus highlight the potential of AI feedback to guide open-ended search for creative and original solutions, providing a recipe that seemingly generalizes to many domains and modalities. In this way, QDAIF is a step towards AI systems that can independently search, diversify, evaluate, and improve, which are among the core skills underlying human society's capacity for innovation.

Published as a conference paper at ICLR 2024
QUALITY-DIVERSITY THROUGH AI FEEDBACK
Herbie Bradley? 2..33 Andrew Dai Hannah Teufel
Jenny Z
Koen Oustermeijer' Marco Bellagente" Jeff 2,6.
Kenneth Stanley
Grégory Schott Joel Lehman
!CarperAl CAMM Lab, University of Cambridge BleutherAl
AAleph Alpha
DDepartment of Computer Science, University of British Columbia
Vector Institute
"Stability AI *Canada CIFAR AI Chair
"aave Stochastic Labs
ABSTRACT
In many rext-generation problems, users may prefer not only single response, but a diverse range of high-quality outputs from which to choose. Quality-diversity (QD) search algorithms aim such outcomes, by continually improving and di- versifying population of candidates. However, the applicability of QD to qual- itative domains, like creative writing, has been limited by the difficulty of al- gorithmically specifying measures of quality and diversity. Interestingly, recent developments in language models (LMs) have enabled guiding search through Al feedback, wherein LMs prompted in natural language to evaluate qualita- tive aspects of text. Leveraging this development, we introduce Quality-Divernee through AI Feedback (QDAIF), wherein an evolutionary algorithm applies LMs to both generate variation and evaluate the quality and diversity of candidate text. When assessed on creative writing domains, QDAIF covers more of specified search space with high-quality samples than do non-QD controls. Further, human evaluation
-eenerted creative texts validates reasonable agreement be- tween Al and human evaluation. Our results thus highlight the potential of Al feedback to guide open-ended search for creative and original solutions, provid- ing recipe that scemingly generalizes to many domains and modalities. In this way, QDAIF is a step towards AI systems that can independently search, diver- sify, evaluate, and improve, which among the core skills underlying human society's capacity for innovation
INTRODUCTION
Human innovation is not only generative capacity for creativity, but also includes the ability to evaluate the subjective quality of new ideas and artifacts. Great ideas are rarely generated all once out of whole cloth, but rather gradually emerge through divergent chains of elaboration and revision (Stanley & Lehman 2015). To successfully navigate such a tree of ideas, the creator must evaluate which steps in a chain are worth pursuing further, question that can be highly subjective, especially in domains with artistic or literary dimensions.
Until now, even if AI could provide candidates, the hope for such subjectively tinged evaluation lay firmly with humans. However, the emerging foundation model technology of recent years (Bom-) masani et al. 2021) now means that the model can also play the role of evaluator, even when the evaluation in part subjective (Madaan et al., 2023), In this way, for the first time, an entire ideation process that returns diverse of interesting artifacts cannot be run by LMs entirely on their own, but requires chaining together a search algorithm with model calls in nuanced way. This paper highlights one way achieve this potential: to combine LMS with the field of quality-diverse design search processes that produce high fighquuaitty solutions that span design space.
in principle be automated. This process
(QD) (Mouret & Clune 2015), which centers on how to
Equal contribution. Order chosen in support of student authorship. Author contributions listed [here] Correspondence: andrew.dat @ aleph-alpha.com, mail herbicbradley.com. lehman. I 154 gmail.com Work done while at Aleph Alpha
Project Page: https::///da f. ..t.iu..bbt......... thub.

---

Published as a conference paper at ICLR 2024
"A story about a spy and a politician... "
QDAIF
Baseline
-00
06
156
wnde kandee let, The oud
52
pnd de ciee mus Co
04
254
torhe hark suarct shme cac flom
De nien The polaccel not
The malliii soteel deow tor
0.S
This was Natnoking Ary aebert
Endro Happy?
bnding (Toge Neppy?
mone of Phom avyk awr autusee monoie
They ancoortta
rech.
Figure I: QDAIF (left) covers more the search space with diverse, high-quality stories com- pared to the Baseline (right). The baseline is LMX, Quality-Only (Meyerson et al.] (023), which optimizes only for the quality of solutions. QDAIF discovered more interesting stories about spy and a politician, covering examples such as romance stories with happy-ending, to horror stories with a tragic-ending. The baseline produced story (right-middle position, starting with "Jason") with a lower quality score due to the lack of a desired spy character (denoted by the red-colored bin, for story with a neutral ending, and leaning to horror). QDAIF discovered a better, more-relevant story sattom-middle position, starting with "a wealthy politician") ") for this same neutral bin.
The main insight in QD algorithms is to explicitly maintain and seek high-quality diverse responses. Typically such search algorithms require hand-designed measures of diversity and quality, as well as a way to generate meaningful variation. Yet the most interesting and complex domains nearly always involve notions of performance, diversity, and variation that are subjective or difficult to specify algorithmically. Extending work that generates variation through LMs (Lehman et al. 20222] Meyerson et al., 2023) and evaluates the quality of potential solutions through LMs (Ahn et al., 2022]. we show that LMs can also be used to evaluate qualitative aspects of diversity. In this way, LMs can instantiate the three main ingredients of QD search, thereby enabling powerful new QD algorithms that can ride the coattails of continual LM advances, which we name Quality-Diversity through AI Feedback (QDAIF). Such QDAIF can explore and return diverse, high-quality responses to an LM prompt through more-intuitive diversity measures, without the need for model fine-tuning (although, it could also be used for LMs self-improve by generating fine-tuning data (Lehman eet al. 2022 Chen et al. 2023 an interesting direction for self-curated effective learning environ- ments via generated data, towards Al-generating algorithms (Clune) 2019g).
We evaluate QDAIF across three creative writing domains: opinion writing, short stories, and poetry. In such domains, users often enjoy seeing a wide range of possible stories or poems from which to choose or be inspired. Quantitative results indicate that QDAIF significantly outperforms existing baselines. Additionally, through human evaluation, we observe a strong alignment between human and Al-generated feedback, providing empirical evidence that AI feedback is grounded and that the method can work in practice (i.e. it yields improved quality and diversity as measured by humans). Overall, QDAIF brings us step closer to Al models that can independently search and innovate, keystone abilities of humans that allow them to create culture and science (Stanley et al. 2017).
BACKGROUND & RELATED WORK
2.1 EVOLUTION THROUGH LARGE MODELS
Advancements in language models have enabled new kinds of powerful search algorithms that ap- ply LMs as search operators, e.g. to create text variation or evaluate solutions. While other search algorithms could also be used, this paper creates a QDAIF algorithm by extending upon Evolu- tion through Large Models (ELM) (Lehman et al.] 2022 a framework for evolutionary search for code or text that uses LMs to generate intelligent variation (for example through specialized lan- guage models trained on code diffs (Bradley et al. 2023b), or through simple few-shot prompting
2

---

Published as a conference paper at ICLR 2024
(Meyerson al. 2023 Chen al. 2023 Most QDAIF results in this paper generate search candidates through Language Model Crossover (LMX) (Meyerson et al., 2023), a recent and general few-shot prompting approach that can evolve e.g. mathematical expressions, sentences, Python pro- grams, and prompts for text-to-image models, by leveraging in-context learning capabilities of LMs (Brown al., 2020). The approach is simple: A few existing search candidates are concatenated into a prompt, predisposing the LM to generate new, similar candidates. In this way, LMX enables creating intelligent variation without requiring any specially-trained models. Our experimental im- plementation builds on OpenELM (Bradley et al., 2023a), a versatile open-source Python library designed for research into LM-based evolutionary algorithms.
2.2 QUALITY DIVERSITY ALGORRITHM
Traditional optimization algorithms aim to discover a single high-quality solution, which while ap- propriate for many situations, fail to illuminate the full range of possible high-quality solutions. For creative and design problems in particular, a user may want to choose what they think is most appropriate from diversity of such candidates. In contrast, Quality Diversity (QD) algorithms aim to optimize not just for single optimal solution, but for diverse set of high-quality solutions (Lehman & Stanley 20TTb; Mouret & Clune 2015: Pugh al.) [0T66] Fontaine & Nikolaidis (02T). QD algorithms can thus provide a richer landscape of solutions, enabling adaptability and flexibility in addressing multifaceted challenges (Cully et al. 2015). In addition to a quality measure (i.e. an objective function). QD requires metric such that it can encourage desired axes of diversity. For instance. [Lehman (eshanc et al. (2022) evolved Python programs to design varied locomoting robots, where the diversity dimensions are the robot's height, width, and mass.
significant limitation of existing QD algorithms lies in their reliance on low-level quality and diversity measures (Mouret & Clune 2015). This requirement confounds applying QD algorithms to complex and creative domains, such as the creative writing ones explored this paper. Intuitively, such measures (e.g. sensor readings (Cully et al. 2015)), feature engineering (Manning) 2009)) lack the subtlety and depth needed to capture the complexities of human creativity and intuition, c.g. nuances, moods, cultural references that resonate in human experience. Interestingly, from having trained on vast amounts of guman--generaae judgments (cf. [Section 2.3). Thus, by employing an LM evaluate both quality and diversity, QDAIF significantly simplifies and enlarges the range of domains QD can be applied to.
data, LMs can begin to emulate such human-nuanced nuanced
Feedback from learned MI. models has been used in prior work reduce the need for hand-crafted heuristics or expensive ground-trun evaluations. In model-based QD, surrogate models supply learned feedback. Gaier et al. (2017) introduced the use of surrogates (via a Gaussian process) to predict fitness (quality). Subsequently, Keller et al.] (2020) introduced a learned model to pre- dict both fitness and behavior characteristics (diversity). becoming a standard approach (Lim al,] 2021 [022, ZZang et al., [2022 Bhatt et al.) 2022).. Surrogate models require domain- domain-specifte train- ing data to update their predictions, whereas Al feedback leverages off-the-shelf instruction LMs (Chung et al. 2022 Ouyang et al. (022) to automate expensive human feedback for a variety of evaluation tasks. More recently. Fontaine & Nikolaidis (202I) utilized CLIP embeddings (Rad-) ford et al.) (021) l) both quality and diversity measures to navigate the search space of StyleGAN (Karras et al. 2019), producing a range of faces with the desired characteristic (e.g. "A person with red hair"). We show that using pre-trained surrogate models is more prone reward hacking in the natural language case (Skalse et al. [022 Lehman et al.) (019) (cf. [Appedixxx A.2). Hence, QDAIF capitalizes on the strengths of general-purpose LMs for evaluating generated solutions.
2.3 AI FEEDBACK
Recently, LMs have been leveraged to provide feedback on the training, evaluation, or problem- solving capabilities of other LMs (Bai et al. [222,] Perez et al. 2022 Shhnnn et al. 2023 Wang et al.) 2023a Colas et al., 2023 Zhang et al., 2023 Lee et al. 2023). Bai et al. (2022) show that using LM-generated critiques and refinements has been instrumental in enhancing performance on met- rics like helpfulness and harmlessness. One promising direction for AI feedback is self-refinement, where LMs evaluate and score their generations, then iteratively improve their output (Bai et al.) 2022 Madaan et al. (002). Self-refinement demonstrated significant improvement output qual- ity as gauged by human evaluators (Madaan et al. 2023), underscoring generation-discriminnal
3

---

Published as a conference paper at ICLR 2024
QD Archive
Feedback
Evalaatodd
pestomlyy Previous solution
Mutate
New solution
Qaseity
udect
Werriae Voarsaooo buok qutle tool
nuuaao
poliertoo
Ercier F me
Model
Cadliated
eaiigg maad and crain
Figure 2: Overview of Quallty-Diversity through AI Feedback (QDAIF). Dark components are where Language Models (LM) are employed. QDAIF randomly selects a solution from the QD archive. This chosen solution (parent) forms part of the prompt that is fed into an LM, undergoing LMX mutation to produce a new solution. An LM then evaluates the quality and diversity attributes of the new solution. We compare the newly evaluated solution with its existing solutions in the QD archive, and update it.
discrepancy (Saunders et al. 2022 p.12) (i.e. it is often easier for model to evaluate the qual- ity of generation than to generate the same quality in text). Complementary to single-objective optimization with self-refine, QDAIF utilizes Al feedback to assess diversity in addition to quality, facilitating more varied and improved text generation over multiple iterations of refinement.
APPROACH
Figure 2 provides an overview of the approach, which is to extend a common QD algorithm (MAP-
Elites) with LM operators that generate variation, as well as evaluate both the quality and diversity of candidate solutions. The result is a search algorithm capable of iterative discovery and refinement, applicable to subjective text-based domains.
MAP-Elites. Our QDAIF implementation builds upon MAP-Elites (Mouret & Clune 2015
widely used QD algorithm (Lchman et al.] 2022] Cully et al. 2015 Nilsson & Cully 2021 Vassili- aaess et al. 2016 MAP-Elites discretizes the diversity space (i.e. dimensions of relevant diversity) into a grid, called the archive. The overarching objective is to populate each grid bin (or cell) within the archive with as high-quality a solution possible. An iteration in MAP-Elites follows these steps: (1) randomly select an existing solution from the archive, (2) mutate the chosen solution generate new solutions, (3) evaluate the new solution's quality and diversity characteristics, and (4) if the new solution is higher quality than the current occupant at the cell corresponding to its diversity characteristics, replace the previous cell occupant solution with the new solution. For a new solution to be added to the archive, it has to improve either the quality or the diversity of the grid, meaning that it has to either fill an empty bin or perform better than the solution already in its bin. QDAIF distinguishes itself from standard MAP-Elites in four key areas: archive initialization, solution mutation, solution evaluation, and grid discretization (cf. Figure 2}. We provide details on each of these differences below.
For archive initialization, QDAIF employs few-shot prompting, gen- erating solutions based hand-chosen set of seed examples. We list in Appenddix A.2I the three few-shot examples utilized in each domain, each chosen to span a breadth of diversity characteris- tics. For example, in domain where you want diversity of sentiments (like the Opinions domain described in [Section D. the few-shot examples demonstrate positive, neutral, and negative senti- ments. For solution mutation, QDAIF employs LMX-Near (referred as "LMX" for brevity in the rest of this manuscript), as detailed in Meyerson et al. (2023). LMX evolves varied text represen- tations (e.g. mathematical expressions, sentences, Python code) by leveraging effective in-context learning (Brown et al.) 2020). LMX prompts are kept simple, typically starting with "Here is ran- shows the full LMX prompts. We also introduce a novel mutation method with instruction-follooinn prompts for poetry in SSection 4.4
Initialization and Mutation.
dom example of Appendix A.22 A.22]shos
Archive Measures. While sometimes feasible to devise hand-crafted heuristies to evaluate the quality of a solution (e.g. efficiency in locomotion) or diversity characteristics (e.g. robot's 's size and mass), this approach falters as domains become more complex and nuanced, as in creative writing. For example, hand-crafting robust heuristics for qualitative aspects of a story, such as its genre (e.g. romance vs. horror), is very difficult. QDAIF circumvents the need for hand-coder measures through prompting LMs with easily- natural language queries to generate feedback. In particular, capable LMs trained on expansive text corpora can begin to mirror human intuition across a range of potentially subtle diversity characteristics.
4

---

Published as a conference paper at ICLR 2024
Quantifying Performance and Diversity. For quality assessment, prompt the LM to discern whether the input text contains a high-quality solution or pertains to the requested topic, requesting "yes" or "no" response. The solution's quality estimate is derived from the logarithm of the probability of the LM predicting one response versus the other response. Similarly, for diversity evaluation, we guide the LM to identify particular diversity trait. For instance, in an opinion generating domain, the LM is prompted to gauge a solution's ' sentiment, with a requested response of "positive" or "negative".. The log probability of these responses serves as our measure of solution diversity. [Appendix A.22 shows the full prompts used in each domain to evaluate the solutions. We also introduce a novel categorical approach evaluate solution attributes based on raw predictions of discrete labels in (Section 4.4]
shows
Discretization. MAP-Elites typically partitions the grid into equally-sized bins, from the intuition that all parts of the behavior space are equally interesting. However, we observe that when assigning a bin along the diversity axis- which is in our approach based on logits of an LM Al feedback that qualitative changes in behavior do not uniformly correspond to changes in the logits (ef. Ap- pendix A.3I). This is likely due to the (non-linear) calibration behavior of instruction-tuned models predicting the labels (as output tokens) of text passages (Jiang et al. 2021). Hence, we use cus- tom non-uniform bins, which are denser towards range ends. Qualitative analysis of the generated text showed that the non-uniform bins yielded better alignment with typical human perceptions of diversity changes, influenced by both the AI model's calibration and the domain-specific goals.
Models and Setup. Details the LMX generation model (Appendix A.24 and finetuned AI feedback model (Appendix A.25) are given, with details on the training of these LMs. Additional default hyperparameters are described in Appendix A.27
4 EXPERIMENTS ON CREATIVE WRITING DOMAIN
4.1 SETUP: OPINION WRITING, SHORT STORIES
To demonstrate the versatility of QDAIF in different applications of creative text evolution, we evaluated QDAIF on these domains: Opinions, and Stories. The Opinions domain is focused on generating diverse, realistic pieces about one's opinions on eating vegetables and plant-based foods the diversity measure is based on the sentiment of opinions on this topic (e.g. shown in example texts the Figure 2 overview). For the Stories domain, the topic is about a short story, containing two characters: a spy. and a politician. The diversity of stories is evaluated using variety of measures based on Al Feedback, with the main ones being: Stories Genre (Romance vs Horror) (ID archive), Stories Ending (Happy vs Tragic) (ID archive), and Stories Genre and Ending (2D archive). These domains capture the strengths and limitations of all methods, ranging from simple (Opinions) to challenging (Stories- Genre and Ending). We show in FFiuure 1 that the 2D domain is challenging, yet QDAIF still outperforms the baseline in filling the archive with diverse, high-quality stories. The AI feedback prompts are outlined in Appendix A.23
Evaluation. To assess the performance of methods in creative writing generation, we compute QD scores (Pugh et al. 2016). a standard metric used to measure the quality-diversiqy of the discovered corpus of texts. A QD score is defined as the sum of the highest quality values found in each bin. To understand the alignment between AI and human feedback for practical applications in QDAIF, we conducted a human evaluation study on selected elite samples from each method (chosen from the median QD score run out of 5 random seed runs). Using Likert scale (Allen & Seaman, 2007) for quality assessment, we evaluate the capability of each method to produce collection of diverse, high-quality texts. To do so we calculate a "human" QD score, defined as the sum of quality scores given for all diversity categories identified by the annotator within the set. Furthermore, to understand how closely AI feedback aligns with human perspectives on subjective evaluation, we measured the agreement rates between human annotators and Al and between two human annotators. Details of the human study are specified in in/Appendix A. demonstrating the validity and advantages of AI feedback in generating human-like feedback subjective quality and diversity measures.
4.2 COMPARISONS BETWEEN QDAIF AND BASELINES
To evaluate the strengths and limitations of QDAIF in generating high-quality and diverse creative writing texts, we compared method against the following baseline methods:
5

---

Published as a conference paper at ICLR 2024
Fixed-Few-Shot: Use fixed few-shot prompt (cf. Appendix A.2I) to sample many com- pletions for creative domain texts (i.e. no iterative search).
Shuffing--ew-Shot: Shuffle the in-context examples of Fixed-Few-Shot prompt, before sampling the completion from this prompt.
Reado:- Create a prompt pool of examples (initialized from examples in Ap- pendix A.21), add all completions from few-shot prompting the pool, and choose few- shol examples from the growing pool (without pool size limit).
LMX, Quality-Only: Maintain pool as in Random-Search, but only up to 100 highest- quality completions (as evaluated by AI Feedback) are kept in the pool (i.e. iterative search focused only on quality). Single-objective LMX in Meyerson al. (023).
We choose a variety of baselines, some highlighting representative alternative approaches (e.g. few- shot prompting), and ablations of QDAIF, validate our algorithmic choices. For example, Fixed- Few-Shot and Saufftinng-Few-Shhee sampling), while being constrained the output distribution of the fixed set of in-context examples. Random-Search and LMX, Quality-Only are methods where the (prompt) pool of examples that we can sample from grows in size, starting from an initial pool. In contrast to QDAIF, Random- Search is limited by the lack of constraints in the prompt pool, especially in maintaining the quality of the growing pool through evaluation. LMX, Quality-Only adds a quality-basee with Al feedback that optimizes the quality of texts in the pool over time to contain only texts with high quality scores, but is not designed to encourage diversity in texts in comparison to QDAIF.
enable the generation of different texts (relying on stochastic
evaluation step
For each domain and baseline described above, we recorded runs for 2000 iterations, repeated with 5 different random seeds. To enable comparisons with baselines, Al feedback is used to compute the quality and diversity measures for all iteration outputs. Similar to QDAIF, the bascline methods use hand-written examples in In/Appediix A.2II either in a fixed prompt or as the initial prompt pool.
Performance Comparison. We report results comparing the QD score performance for the dif- ferent methods of generating creative writing texts in Figure 3) We computed the mean, and the bootstrapped 95% confidence intervals (CI) from 100k resamples, across 5 random seeds of runs. We noticed that QDAIF achieves significantly better QD scores than all baseline methods in Opin- ions and Stories The broader range of texts generated by QDAIF is also evident qualitatively. For example, in the Stories Genre and Ending domain, while the baseline methods deliver straightfor- ward and more predictable stories of how the spy "pulled out a knife and stabbed [the politician] to death", QDAIF generates a more dramatic story of how the spy "transformed into the monster and killed everyone". Random-Search is the worst-pe method overall, with significantly lower QD score performance in Opinions and Stories Genre compared to the best west-performing baselines. Interestingly. LMX, Quality-Only does not significantly outperform the methods using fixed population prompt pool (Fixed-Few-Shor and Saaflling-Few-Shoor On Stories Genre, LMX, Quality-Only is often weaker than Fixed-Few-Shee show that single-objectivve optimization cannot guide the search for diverse, high-quality texts alone. Furthermore, we found that QDAIF outperforms other (diversity-seckke seeking baselines), as well as ex- tensions of those baselines with quality filters, becoming more similar to QDAIF (cf. Appendix A.8)
and Shuftiing-Fee Few-Shot The results
Human Feedback Evaluation. We report the results of the human study comparing QDAIF and baseline samples in Table We observe that compared to baselines, QDAIF is competitive with or better at discovering diverse, high-quality texts in Opinions and Stories according to human feed- back. QDAIF sets also showed high agreement between humans and AI feedback on the diversity categories of presented texts, as well as between two annotators, competitive with Fixed-Few-Shie Although the average perceived quality of texts is better from FiwSStfew-Shott this is not enough for bringing high-quality examples for different niches of outputs (i.e. higher QD score). Further- more, Shuffling-Few-Shee demonstrates even lower human evaluation scores, despite the use of the same fixed set of hand-wring wridtwritenn seeds, indicating lower robustness due to the use of different order- ing of few-shot examples, Prior work hints at the sensitivity of LMs to few-shot prompt ordering, with task-solving capabilities varying significantly due to this ordering (Lu et al. 2021). The gap in human-evaluated performance between liance fixed prompts is less likely to enable reliable search, in contrast to the robustness shown by QDAIF Additionally, Random-Searde and LMX, Quality-Only obtained even lower human eval- uation scores, even though the methods either explore different prompts or optimize for the quality
and Shufffing-Feew-SS
indicates that re-

---

Published as a conference paper at ICLR 2024
Opiniers
Stories Genre
Stories Ending
Genre Ending
Shufting Shot
Rendom Search
LNX, QualityDril
DDNP
19.0
19.5
20.0
14
18
20
18
20
50 60
70
QD Score
QD Score
QD Score
OD Score
Figure 3: QDAIF significantly outperforms baselines in QD score performance in all domains. Performance stats with mean bootstrapped 95% CI, across random seed runs. The maximum possible QD score is 20 (100 for 2D archive (4th plot)). See Appendix A.7 for additional stats.
Table 1: QDAIF is competitive/better in terms of Human QD score against baseline methods
from the human evaluation study. The stats are averaged across three domains: Opinions, Stories Genre, and Stories Ending. The Human QD score quantifies the perceived quality- set of solutions returned by method (i.e. high-quality texts for the diversity categories of texts that the evaluator could identify). Quality rating is from humans. See/Appendix A. I. study setup.
of
Human
Quality rating
Human-Al agreement
Human agreement
Method
QD score
Fixed-Few-Shor
0.767
4.133
0,800
0.867
Shufffing-Fee Seetlling-Fe--SS- Few--hot
0.696
3,500
0.700
0,667
Random-Search
0.606
3.300
0.733
0.600
LMX, Quality-Only
0.650
3.533
0.633
0.733
QDAIF (ours)
0.772
3,900
0,833
0.800
of texts. We provide a detailed discussion (for baseline methods) on findings from the subjective study of the discovered texts in in/Appendix A.15 well the qualitative behavior of the text search over time in Appendix 16 Through guided evolutionary search, QDAIF surpasses all baseline methods in terms of computed QD score performance, and is competitive (or better) compared to baselines, according to human evaluation.
4.3 EXTENSIONS TO AI FREDBACK AND MUTATION MODEL.
In addition to experiments with QDAIF described previous sections, we investigated the effects on the performance due to variations of the method.
LMX Model Size. We used larger versions of the LMX models (30B and 70B) for mutation, and compared it to the performance of the 13B model (default). While no relationship was found between model size and QD score, quality ratings from human feedback improved with outputs from larger models (described in detail in Appendix A.3).
Few-Shot AI Feedback. We compared the performance of QDAIF on the Stories - Genre domain when prompted Al feedhack model for diversity measures given 2-shot, 4-shot, and 8-shot prompts. Using a higher number of few-shots led to improvements human quality ratings texts. Further discussion and results are highlighted in Appendix A.4
Varying Initialization and Mutation Method. Ideally, QDAIF would be simpler if it could be without seed examples (e.g. requesting a story from instructionnfollowinng LM). We investi- gated the potential of QDAIF when the solution population is initialized from zero-shot prompted generations, and evolved using LMX. Initial results on Opinions and Stories along with prior work discussion are shown in Appendix A.5 highlighting comparable performance in terms of QD score, with divergence observed alignment with human preferences. We also find that QDAIE, these domains, is robust to the mechanism of generating variation. AApendixx A.6 describes an alternative mutation method (based on a more gradual few-shot replacement operation) that is more effective in some circumstances. although in general offers comparable performance. We provide detailed discussion (for QDAIF methods) on findings from the subjective study of the discovered texts in Appendix A.13] well the qualitative behavior of the text search over time in Appendix A.14 There may be many ways of leveraging LMs to generate variation (including finetuning models; see Appendix A.I and this is exciting avenue of future research.

---

Published as a conference paper at ICLR 2024
"A poem of very high, award winning qualityy...
QDAIF
Random- -oem-s
merii cecior pros
(P for
AG
Geere
5,7
73.66
sbeut Feiriing de oade Mor seeks Solte madn
bort
with dela
Figure 4: QDAIF (LMX- (left) covers the space of poetry with high-quality solutions (on a rating scale), with poems matching the closest bins. QDAIF solutions take qualitative inspiration from the seed poem's 's imagery of "fields of green waves" " in Appendix 4.30@hiie giving meaningfully diverse kinds of poems across the search space. QDAIF (LMX- (not shown) also covers more the space of diverse, high-quality poems compared to Handom-Poems (right).
4.4 EVOLVING SOLUTIONS THROUGH INSTRUCTION GUIDANCE
This experiment explores scaling up QDAIF to a more capable model, GPT-4 (OpenAl 2023), in the challenging task of generating poetry, highlighting how QDAIF will benefit from advances in model capabilities. The aim of the Poetry domain is to produce high-quality poems with varying genres and emotional tones, unrestricted by topic. Here, the -APtElites archive has two axes of diversity: genre and tone, and they are delineated using categorical labels. The genre axis has the labels "haiku", "sonnet", "ballad", "limerick", and "hymn", while the tone axis has the labels of "happy", "dark", "mysterious". "romantic", and "reflective". We created new mutation operator, LMX-rewrite, for this domain, that leverages instruction-folloiingg to tell model translate parent poem into an offspring with different diversity characteristics. To generate a new solution, we prompt GPT-4 rewrite the selected poem into an inspired, different poem: Inspired by this poem, write new poem of very high, award winning quality with different poetic genre (format and form) and tone compared to the poem above.
We used GPT-4 to determine the genre and tone of poem and rate its quality. For quality, we prompt GPT-4 to Rate the quality of the above poem on a seale from diversity attributes, we prompt GPT-4 to determine which genre tone is the poem closest to. For example, to determine genre, we ask GPT-4: What genre is this poem closest to from the following list: ["haiku", "sonner". "ballad", "limerick". "hymn"]] [Appedix A.30 shows the full prompts and setup. We observed high consistency in GPT-4's s responses; across multiple LM calls, quality ratings for the same poem fluctuated by no more than a single point. We show qualitative examples of poems with their AI feedback evaluations in (Appendix A. In this setup, all methods are run for 200 iterations, and QDAIF is benchmarked against the baseline, Random-Poems, an ablation method, Fixed Seed Rewrite. Random-Poems simply generates 200 random poems. Fixed Seed Rewrite rewrites only the seed poem in Appendix A.30 without evolutionary search.
10. To determine the
well as
We found that QDAIF achieves a higher QD score of 130 (CI: 118 145) in comparison to Random- Poems with 76 (CI: 67- 85) and Fixed Seed Rewrite with 99 (CI: 72 117). We observed a similar trend (with a wider performance gap between QDAIF and other methods) when we used GPT-3.5-Turbo for the generation step instead of GPT-4 while keeping GPT-4 as the evaluator (cf. Appendix A.1 QDAIF was shown to have greater QD score performance than the other methods according to the Mann-Whitney U Test (p < 0.05). Still, QDAIF with LMX-rewrite fails to dis- cover solutions in some bins (e.g. limericks). For this, we can adapt QDAIF by adding guidance on the desired (randomly chosen) genre and tone for rewriting (LMMX-uidedd.. The performance of this method of QDAIF is par with Targeted-Poems. approach (generating high-quality poems of randomly chosen genre and tone per step) in terms of QD score, and even better when using an older version of GPT-4 (ef. Appendix A.17). Furthermore, we found that the rewriting step is useful
8

---

Published as a conference paper at ICLR 2024
for generating poems that meaningfully preserve inspiration from parent poems, enabling users to control with more nuance the outcomes of search, through approximating the iterative refinement typical of human creative processes (Stanley et al. 2017) (see (epeendxxx A.18). Figure hiighlight the potential of QDAIF with GPT-4, especially in controlling a set of solutions subjectively aligned with Al feedback labels. QDAIF is even demonstrated for practical applicability
domain outside
of creative writing, for solving coding problems (cf. [Appendix A.20). Overall, results highlight that
diversity suffers when prompting models without explicit diversity guidance (Renda et al. [
Friedrich et al. [023] Kirk et al. 2023), and that evolution with rewriting mutation operator can lead to more comaa-inffuencebble diverse, and high-quality solutions.
5 DISCUSSION AND CONCLUSION
This paper introduces QDAIF, quality-diversity method that aims to discover diverse and high- quality solutions in qualitative domains. by leveraging advances in foundation models to evaluate the quality and diversity of generated individuals. QDAIF outperforms baseline methods in returning more diverse, high-quality solutions in creative writing domains (Opinions, Stories, Poetry), that benefit greatly from accurate AI feedback measures. The paper's results highlight that QDAIF can succeed at its aims, generating solutions that align with human perception of quality and diversity.
We note limitations with QDAIF that motivate future work. Firstly, we suspect reward hacking when using LMs to generate feedback. Our human evaluation investigation shows that while the LM's evaluation of quality mostly aligns with human perception, the correlation drops when the evaluated quality is 0.995 to (cf. Figure 5). The text generation might have exploited attributes or phrasings that allow an LM to give a high-quality estimate, but not what humans would agree is good. This is common issue highlighted by other works when using AI models classifiers or evaluators (Nguyen et al. 2015a), highlighting risks of open-ended search to be tackled (Ecoffet et al. 2020). One method to address this limitation could be to use RLHF finetuning (Ouyang et al. 2022) to produce LMs that can detect and mitigate adversarially generated texts. Another possible approach could be one; the hope would be that robustness would result from models having uncorrelated blind spots.
an ensemble of different AI models to evaluate solutions, rather than relying only on
Furthermore, although QDAIF makes it easy specify qualitative aspects of diversity through nat- ural language prompts, it still requires specified definitions of diversity axes. For example, if we applied QDAIF generate short stories of different genres (c.g. comparing horror vs. romance), would not autonomously explore other important attributes that a writer might care about (e.g. first- person vs. third-person perspective) unless explicitly specified. When we tested different diversity measures in the Stories domain, such pathologies were observed (Appendix A.32). For example, when using "hero spy vs. hero politician" as the diversity measure, many generated stories tend to neglect the interaction between the spy and the politician, focusing solely on one hero character. However, someone writing a story would naturally care about how multiple characters interact with one another. One possible way automatically determine interesting diversity measures (Meyer- son et al.) 2016 Etcheverry et al., (02I) is to utilize the human notions of interestingness distilled into foundation models (Zhang et al.] 2023). That is, we could ask LMs suggest interesting di- versity measures that a human would typically care about in the domain, thereby enabling a more autonomous creative search (see (see/Appendee Apppendix for findings on the potential of this method).
In conclusion, we show that QDAIF is promising approach to open-ended search that can reveal unexplored creative writing spaces, surpassing alternative text generation methods in generating diverse high-quality natural language text. Al feedback, Evolution through Large Models (ELM), and quality-diversity search (QD) were found to be essential ingredients for enhanced AI systems that can innovate in subjective spaces, similar to past research on Innovation Engines (Nguyen et al.) 2 20156). fact, we see Al feedback as general ingredient for open-ended search for solutions in multimodal domains, capable of following instructions beyond text (Liu et al. 2023). QDAIF can be easily extended to multi-modal domains (e.g. vision-language) for synthetic data generation and evaluation, building on top of recent advances in the field (Lichenberg et al.) 2021 Alayrac let al. 2022; Bellagente et al. 2023 Driess et al.) 2023 Bhatt et al.; (Sudhakaran et al. [023]] Todd et al. 2023). We see many possibilities from QDAIF to build creative search systems with evaluation, diversification, and improvement capabilities, bringing us closer to AI that can support and extend human innovation.

---

Published as a conference paper at ICLR 2024
ETHICS STATEMENT
Human evaluations were performed by the co-authors of this paper and select colleagues. All human evaluators provided informed consent, and their feedback and assessments were obtained without coercion or bias. We took action prevent bias by presenting evaluators with texts evaluate in a blind setting, with only the instructions for the study annotation task presented (to carefully read through the presented texts, then give a quality score and label of the characteristic that best matches the texts). We show a detailed setup for the human study in Appendix A.
For the sake of transparency, we provide the full set of results with caption descriptions from our human evaluation. In the Opinions domain, [Talles [3] contain the human evaluation results for sets from baseline methods, Tables 39/ contain the human evaluation results for sets from QDAIF methods, and Tables 25 contain the human evaluation results for sets from embedding feedback QD methods. In the Stories Genre domain, Tables 17:21 20 contain the human evaluation results for sets from baseline methods, and Tables 33 contanii the human evaluation results for sets from QDAIF methods. For the Stories Ending domain, Tables 24 contain the human evaluation results for sets from baseline methods, and Tables 47 contain the human evaluation results for sets from QDAIF methods.
AUTHOR CONTRIBUTIONS
Herbie developed the setup and framework for the Poetry domain experiments and base library for research. Andrew developed the setup and experiments for the Opinions and Stories domains, and contributed to extended studies, visualization, and analysis across experiments in the paper. Hannah contributed additional experimentation in the Stories domain, addition to coordinating part of human evaluation studies. Jenny contributed qualitative analysis across studied domains. Koen developed visualization scripts used in Opinions and Stories domain experiments. Marco contributed to part of the technical implementation and ideation. Andrew conducted the blind human evaluation study, and Grégory advised on the conduct and analysis of the human study. Joel, Jeff, and Ken initiated early idcation for this work. Joel, Grégory, Jeff, and Ken advised and guided. Andrew, Jenny, Herbie, and Joel wrote the manuscript with edits and feedback from all authors.
ACKNOWLEDGEMENTS
We thank Robert Baldock, Samuel Weinbach, Souradeep Nanda, Jan Zierstek, and Andres Felipe Cruz Salinas for insightful discussions and feedback within the lab at Aleph Alpha. We also thank Katherine Hardgrave, David Nugent, Daniel Flood, and Formula Trinity Autonomous for the inspi- ration that seeded the momentum leading up to this work.
REFERENCES
Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, et al. Do as can, not as I say: Grounding language in robotic affordances. arXiv preprint (xXx:2200400199) 2022.
Jean-Baptice Ban-sBat Alayrac. Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, al. Flamingo: visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35:23716- 23736, 2022.
1. Flaine Allen and Christopher A. Seaman. Likert scales and data analyses. Quality progress, 40 (7):64665, 2007.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional Al: Harmlessness from Al feedback. arXiv preprint arXiv:2212.0800 2022.
10

---

Published as a conference paper at ICLR 2024
Marco Bellagente, Manuel Brack, Hannah Teufel, Felix Friedrich, Bjöm Deiseroth, Constantin Eichenberg, Andrew Dai, Robert Baldock, Souradeep Nanda, Koen Oostermeijer, et al. Multifu- sion: Fusing pre-trained models for multi-lingual, multi- molti-modal image generation. arXiv preprint arXiv:2305. 15296, 2023.
Varun Bhatt, Bryon Tjanaka, Matthew Fontaine, and Stefanos Nikolaidis. Deep surrogate assisted generation of environments. Advances in Neural Information Processing Systems, 35:37762-
37777, 2022.
Varun Bhatt, Heramb Nemlekar, Matthew Fontaine, Bryon Tjanaka, Hejia Zhang. Ya-Chuan Hsu, and Stefanos Nikolaidis. Surrogate assisted generation of human-robot interaction scenarios. arXiv preprint arXiv:2304. 13787, 2023.
Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportu- nities and risks of foundation models. arXiv preprint arXiv:2
2021.
Herbie Bradley, Honglu Fan, Francisco Carvalho, Matthew Fisher, Louis Castricato, reciprocated, dmayhem93, Shivanshu Purohit, and Joel Lehman. OpenELM, January 2023a. URL https:
//github.b www www.///aaeeerr//eeeeeiiiioo
Herbie Bradley, Honglu Fan, Harry Saini, Reshinth Adithyan, Shivanshu Purohit, and Joel Lehman. Diff models A new way to edit code. CarperAl Blog, Jan 2023b. URL /tttps::/carpee.
Jonathan C. Brant and Kenneth O. Stanley. Minimal criterion coevolution: A new approach to open-ended search. In Proceedings of the Genetic and Evolutionary Computation Conference, pp. 67-74, 2017.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D. Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Angelica Chen, David M. Dohan, and David R. So. Evoprompting: Language models for code-level neural architecture search. arXiv preprint arXiv:2302. 30N.-3882..488.... 2023.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such. Dave Cummings, Matthias Plappert, Fo- tios Chantzis, Elizabeth Barnes, Ariel Herbert- Voss. William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob Mc- Grew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained on code. 2021.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. PaLM: Scaling language modeling with pathways. arXiv preprint S0Xxx::2244003111 2022.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph. Yi Tay, William Fedus, Eric Li. Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, al. Scaling cooruction-finttunne language mod- els. arXiv preprint arXiv:2210.1 1416, 2022.
Jeff Clune. AI-GAs: Al-generating algorithms, an alternate paradigm for producing general artificial intelligence. arXiv preprint 1905, 2019.
Cédric Colas, Lactitia Teodorescu, Pierre-Yves Oudeyer, Xingdi Yuan, and Marc-Alexandre Côté. Augmenting autotelic agents with large language models. arXiv preprint arXiv:2305. 12487, 2023.
Antoine Cully. Autonomous skill discovery with Proceedings of the Genetic and Evolutionary Computation Conference, pp. 81-89, 2019.
and unsupervised descriptors. In
11

---

Published as a conference paper at ICLR 2024
Antoine Cully and Yiannis Demiris. Hierarchical behavioral repertoires with unsupervised descrip- tors. Proceedings of the Genetic and Evolutionary Computation Conference, 2018.
Antoine Cully, Jeff Clune, Danesh Tarapore, and lean-Baptiss Mouret. Robots that can adapt like animals. Nature, 521(7553::033-077, 2015.
Li Ding, Jenny Zhang, Jeff Clune, Lee Spector, and Joel Lehman. Quality diversity through human feedback. arXiv preprint arXiv:2310.12103: 2023.
Stephane Doncieux, Alban Laftaquière, and Alexandre Coninx. Novelty search: a theoretical per- spective. In Proceedings of the Genetic and Evolutionary Computation Conference, pp. 99-106, 2019.
Danny Driess, Fei Xia, Mehdi S.M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. PaLM-E: An embodied multi- modal language model. arXiv preprint arXiv:2303.03372 2023.
Adrien Ecoffet, Jeff Clune, and Joel Lehman. Open questions in creating safe open-ended ai: ten- sions between control and creativity. In Artificial Life Conference Proceedings 32, pp. 27-35. MIT Press One Rogers Street, Cambridge, MA 02142-1209, USA journals-info .... 2020.
Constantin Eichenberg, Sidney Black, Samuel Weinbach, Letitia Parcalabescu, and Anette Frank. MAGMA Multimodal augmentation of generative models through adapter-based finetuning. arXiv preprint urXiv:2112.052:5 2021.
Mayalen Etcheverry, Bert Wang-Chat Chag-Chak Chan, Clément Moulin-Frier, and Pierre-Y Yves Oudeyer. Meta-diversity search in complex systems, a recipe for artificial open-endedness? 2021.
Manon Flageat and Antoine Cully. Fast and stable map-elites in noisy domains using deep grids. In Artificial Life Conference Proceedings 32, pp. 273-282. MIT Press One Rogers Street, Cam- bridge, MA 02142-1209, USA journals-info .... 2020.
Manon Flageat and Antoine Cully. Uncertain muality-diversity Evaluation methodology and new methods for quality-diversity in uncertain domains. IEEE Transactions on Evolutionary Compu- tation, 2023.
Matthew Fontaine and Stefanos Nikolaidis. Differentiable quality diversity. Advances in Neural Information Processing Systems, 34: 14:10000-1005, 2021.
Felix Friedrich, Patrick Schramowski, Manuel Brack, Lukas Struppek, Dominik Hintersdorf, Sasha Luccioni, and Kristian Kersting. Fair diffusion: Instructing text-to-image generation models on fairness. arXiv preprint vrXiv:2302.1089) 2023.
Adam Gaier. Accelerating Evolutionary Design Exploration with Predictive and Generative Models. PhD thesis, Université de Lorraine, 2020.
Adam Gaier, Alexander Asteroth, and bean-Baptiste Mouret. Data-efficient exploration, optimiza- tion, and modeling of diverse designs through surrogate-assisted illumination. In Proceedings of the Genetic and Evolutionary Computation Conference, pp. 99-106, 2017.
Adam Gaier, Alexander Asteroth, and Jean-Ban -aptistt Mouret. Are quality diversity algorithms better at generating stepping stones than objective- search? In Proceedings of the Genetic and Evolutionary Computation Conference Companion, pp. 115-116, 2019.
Luca Grillotti and Antoine Cully. Unsupervised behaviour discovery with quality-diverse optimi- sation. arXiv preprint arXiv:2106. 056488 2021.
Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrick, and Graham Neubig. Towards unified view of parameter-efficient transfer learning. arXiv preprint St.i::2110.043666 2021.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, An- drea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for NLP. In International Conference Machine Learning, pp. 2790-2799. PMLR, 2019.
12

---

Published as a conference paper at ICLR 2024
Zhengbao Jiang, Jun Araki, Haibo Ding, and Graham Neubig. How we know when language models know? On the calibration of language models for question answering. Transactions of the Association for Computational Linguistics, 9:962-977, 2021.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprini arXiv:2001.0888 2020.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings the recognition, pp. 4401-4410, 2019.
the IEEE/CVF conference on computer vision and pattern
Leon Keller, Daniel Tanneberg. Svenja Stark, and Jan Peters. Model-based quality-diversite search for efficient robot learning. In 2020 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 9675-9680. IEEE, 2020.
Robert Kirk, Ishita Mediratta, Christoforos Nalmpantis, Jelena Luketina, Eric Hambro, Edward Grefenstette, and Roberta Raileanu. Understanding the effects of rlhf on Ilm generalisation and diversity. arXiv preprint arXiv:2310.06452, 2023.
Harrison Lee, Samrat Phatale, Hassan Mansoor, Kellie Lu, Thomas Mesnard, Colton Bishop, Victor Carbune, and Abhinav Rastogi. RLAIF: Scaling reinforcement learning from human feedback with AI feedback. arXiv preprint arXiv:2309.00267, 2023.
Joel Lehman and Kenneth O. Stanley. Revising the evolutionary computation abstraction: Minimal criteria novelty search. In Proceedings of the 12th annual conference on Genetic and evolutionary computation, pp. 103-110, 2010.
Joel Lehman and Kenneth O Stanley. Abandoning objectives: Evolution through the search for novelty alone. Evolutionary computation, 19(2):189-223, 201 la.
Joel Lehman and Kenneth O. Stanley. Evolving a diversity of virtual creatures through novelty search and local competition. In Proceedings of the 13th annual conference on Genetic and evolutionary computation, pp. 21 1-218, 2011b.
Joel Lehman, Kenneth O Stanley, et al. Exploiting open-endedness to solve problems through the search for novelty. In ALIFE, pp. 329-336. 2008
Joel Lehman, Bryan Wilder, and Kenneth O. Stanley. On the critical role of divergent selection in evolvability. Frontiers in Robotics and AI, 3:45, 2016.
Joel Lehman, Jeff Clune. Dusan Misevic, Christoph Adami, Lee Altenberg, Julie Beaulicu, Peter J. Bentley, Samuel Bernard, Guillaume Beslon, David M. Bryson, Patryk Chrabaszez, Nick Cheney, Antoine Cully, Stephane Doncieux, Fred C. Dyer, Kai Olav Ellefsen, Robert Feldt, Stephan Fis- cher, Stephanie Forrest, Antoine Frénoy. Christian Gagné, Leni Le Goff, Laura M. Grabowski, Babak Hodjat, Frank Hutter, Laurent Keller, Carole Knibbe, Peter Krcah, Richard E. Lenski, Hod Lipson, Robert MacCurdy, Carlos Maestre, Risto Miikkulainen, Sara Mitri, David E. Moriarty, Tean-Baptist Mouret, Anh Nguyen. Charles Ofria, Mare Parizeau, David Parsons, Robert T. Pen- nock, William F. Punch, Thomas S. Ray, Mare Schoenauer, Eric Shulte, Karl Sims, Kenneth O. Stanley, François Taddei, Danesh Tarapore, Simon Thibault, Westley Weimer, Richard Watson, and Jason Yosinski. The surprising creativity of digital evolution: A collection of anecdotes from the evolutionary computation and artificial life research communities, 2019.
Joel Lehman, Jonathan Gordon, Shawn Jain, Kamal Ndousse, Cathy Yeh, and Kenneth O. Stanley. Evolution through large models. arXiv preprint arXiv:2206.08896 2022.
Bryan Lim, Luca Grillotti, Lorenzo Bernasconi, and Antoine Cully. Dynamics-aware quality- diversity for efficient learning of skill repertoires. arXiv preprini urXiv:2109.08522. 2021.
Bryan Lim, Alexander Reichenbach, and Antoine Cully. Learning to walk autonomously via reset- free quality-diversity. arXiv preprint arXiv:2204.03665, 2022.
Chin- Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pp. 74-81, 2004.
13

---

Published as a conference paper at ICLR 2024
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. arXiv preprint arXiv:2304.08455 08485, 2023.
Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. arXiv preprint arXiv:2104.087555 2021.
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with self-feedback. arXiv preprint arXiv:2303.17651, 2023.
Christopher Manning. An introduction to information retrieval. Cambridge university press, 2009.
Elliot Meyerson, Joel Lehman, and Risto Mikkullannnn. Learning behavior characterizations for novelty search. In Proceedings of the Genetic and Evolutionary Computation Conference 2016, pp. 149-156, 2016.
Elliot Meyerson, Mark J. Nelson, Herbie Bradley, Arash Moradi, Amy K. Hoover, and Joel Lehman. Language model crossover: Variation through few-shot prompting. arXiv preprint 12710 2023.
Jean-Baptiste Mouret and Jeff Clune. Illuminating search spaces by mapping elites. arXiv preprint arXiv:1504.04900 2015.
Niklas Muennighoff. SGPT: GPT sentence embeddings for semantic search. arXiv:2202.08904, 2022.
Anh Nguyen, Jason Yosinski, and Jeff Clune. Deep neural networks are easily fooled: High confi- dence predictions for unrecognizable images. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 427-436, 201
Anh Nguyen, Jason Yosinski, and Jeff Clune. Innovation engines: Automated creativity and im- proved stochastic optimization via deep learning. In Proceedings of the Genetic and Evolutionary Computation Conference, 2015b.
Anh Nguyen, Jason Yosinski, and Jeff Clune. Understanding innovation engines: Automated cre- ativity and improved stochastic optimization via deep learning. Evolutionary computation, 24(3): 545-572, 2016.
Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. Adversar- ial NLI: A new benchmark for natural language understanding, arXiv preprint arXiv: 191.::1990014599 2019.
Olle Nilsson and Antoine Cully. Policy gradient assisted MAP-Elites, In Proceedings of the Genetic and Evolutionary Computation Conference, pp. 866-875, 2021.
OpenAl. GPT-4 technical report, 2023.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35: 27730-27744, 2022.
Ethan Perez, Sam Ringer, Kamilè Lukošiūté, Karina Nguyen, Edwin Chen, Scott Heiner, Craig Pettit, Catherine Olsson, Sandipan Kundu, Saurav Kadavath, al. Discovering language model behaviors with model-written evaluations. arXiv preprint 2rXiv:2212.09255 2022.
Julien Pourcel, Cédric Colas, Pierre-Yve Oudeyer, and Laetitia Teodorescu. Aces: generating diverse programming puzzles with autotelic language models and semantic descriptors. arXiv preprint arXiv2210..1669.. 2023.
Justin K. Pugh, Lisa B. Soros, and Kenneth O. Stanley, Quality diversity: A new frontier for evolu- tionary computation. Frontiers Robotics and AI, 3:40, 2016.
14

---

Published as a conference paper at ICLR 2024
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pp- 8748-8763. PMLR, 2021.
Nils Reimers and Iryna Gurevych. Sentence-BERT: Sentence embeddings using siamese BERT- networks. arXiv preprint arXiv: 1908. 10084, 2019.
Alex Renda, Aspen K. Hopkins, and Michael Carbin. Can LLMs generate random numbers? evaluating LLM sampling in controlled domains. In ICML 2023 Workshop: Sampling and Optimization in Discrete Space, 2023, URL http: pest:://peopl..e csail mit eet.edduueedaa
11m- - Sawammling-paa paper
Victor Sanh, Albert Webson, Colin Raffel, Stephen H Bach, Lintang Sutawika, Zaid Alyafeai, An- toine Chaffin, Armaud Stiegler, Teven Le Scao, Arun Raja, et al. Multitask prompted training enables zero-shot task generalization. arXiv preprint arXiv:2110.000 10.08207, 2021.
William Saunders, Catherine Yeh, Jeff Wu, Steven Bills, Long Ouyang, Jonathan Ward, and Jan Leike. Self-critiquing models for assisting human evaluators. arXiv preprint urXiv:2206.05800 2022.
Jimmy Secretan, Nicholas Beato, David B. D' D'Abbosio, Adelein Rodriguez, Adam Campbell, Jeremiah T. Folsom-Kovaris and Kenneth O. Stanley. Picbreeder: A case study in collab- orative evolutionary exploration of design space. Evol. Comput. 19(3):373-4033 2011. doi: 10.1162/EVCOL al 00030. URL https::::
62 /1C2//VCO, 00030
Noah Shinn, Beck Labash, and Ashwin Gopinath. Reflexion: An autonomous agent with dynamic memory and self-reflection. arXiv preprint arXiv:2303. 2023.
Joar Skalse, Nikolaus Howe, Dmitrii Krasheninnikov, and David Krueger. Defining and character- izing reward gaming. Advances Neural Information Processing Systems, 35:9460- 04000,997111111 2022.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language pro- cessing, pp. 1631-1642,
2013. 2013.
Kenneth O Stanley. Compositional pattern producing networks: novel abstraction of development. Genetic programming and evolvable machines, 8:131-162, 2007. 2007.
Kenneth O. Stanley and Joel Lehman. Why greatness cannot be planned: The myth of the objective. Springer, 2015.
Kenneth O. Stanley, Joel Lehman, and Lisa Soros. Open-endedness: The last grand challenge you't 've never heard of. O' D'Reilly Radar, 2017.
Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. Roformer: En- hanced transformer with rotary position embedding. arXiv preprint 1rXiv:2144009864 2021.
Shyam Sudhakaran, Miguel González-Duque, Claire Glanois, Matthias Freiberger, Elias Najarro, and Sebastian Risi. Mariogpt: Open-ended text2level generation through large language models, 2023.
Graham Todd, Sam Farle, Muhammad Umair Nasir, Michael Cerny Green, and Julian Togelius. Level generation through large language models. In Proceedings of the 18th International Con- ference the Foundations of Digital Games, pp. 1-8, 2023.
Vassilis Vassiliades, Konstantinos Chatzilygeroudis, and Jean- Caan-Baptiste Mouret. Scaling up MAP- Elites using centroidal Voronoi tessellations. arXiv preprint arXiv:1610.0577 2016.
Guanzhi Wang, Yugi Xie. Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. arXiv preprint 2305:2305.1629) 2023a.
15

---

Published as a conference paper at ICLR 2024
Ren-Jian Wang, Ke Xue, Yutong Wang. Peng Yang. Haobo Fu, Qiang Fu, and Chao Qian. Diversity from human feedback. arXiv preprint arXiv:2310.06648, 2023b.
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language model with self generated instructions. arXiv preprint arXiv:2212. 100500 2022a.
Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabshhi, Yeganeh Kordi, Amirreaa Mirzaei, An- jana Arunkumar, Arjun Ashok, Arut Selvan Dhanasekaran, Atharva Naik, David Stap, et al. Suuper-Vauallnnstuuttionnss Generalization via declarative instructions on 1600+ NLP tasks. arXiv preprint arXiv:2204.0777 07705, 2022b.
Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. Finetuned language models zero-shot learners. arXiv 2021.
preprint arXiv:2
Jenny Zhang, Joel Lehman, Kenneth Stanley, and Jeff Clune. OMNI: Open-endedness via models of human notions of interestingness. arXiv preprint arXiv:2306.01711, 2023.
Yulun Zhang, Matthew C Fontaine, Amy Hoover, and Stefanos Nikolaidis. Deep surrogate as- sisted MAP-Elites for automated hearthstone deckbuilding. In Proceedings of the Genetic and Evolutionary Computation Conference, pp. 158-167, 2022.
16

---

Published as a conference paper at ICLR 2024
A APPENDIX
A.1 HUMAN STUDY ON QUALITY-DIVERSITY OF TEXT SAMPLES
A study through human feedback lets us understand how well QD score performance through AI feedback translates creating a high-quality, diverse set of creative texts from a subjective angle. We compare sets of human feedback evaluations for samples of diverse elites at the end of each run to measure translations from Al-assessed performance to aunan-assessed performance of methods in generating high-quality, diverse texts. In addition, the Human QD Score (sum of mean quality score for each category/label that is found in the set according to human feedback) gives us a rough understanding of how aligned quality-diversio this score is low if the set deemed to cover a wide space of
improvement during the search is with the more
subjective notion of quality-diversin diverse texts from AI feedback fails to subjectively cover the space of desired diversity according to human evaluations. To distinguish between quality ratings from humans vs. Al feedback in this section, we refer to quality scores as those from human evaluators, and fitness scores as those from AI feedback.
To assess the robustness of quality and diversity measures in AI feedback, we carried out study involving diverse clite samples selected from different bins of the QD archive from our tested runs. Over total of 28 experiments (of which 4 are from embedding feedback experiment runs), five dis- tinct stories per experiment are reviewed by six annotators. Each (generated) text was independently reviewed by two persons, resulting in a total of 280 annotations.
During the annotation process, we collected subjective assessment of the quality of the generation using a 5-point Likert scale based on the text quality in terms of flow, plot, presence of repetition, and correspondence to the study's 's topic. In addition, we assign each text to one of three categories. Two categories were specific to the study performed (such as positive/negative sentiment, romance/horror genre, or tragic/happy ending) and third category was used when no element of the other two classes was identified.
We took action to prevent bias by presenting evaluators with texts to evaluate in blind setting, with only the instructions for the study annotation task presented (to carefully read through the presented texts, then give quality score and label of the characteristic that best matches the texts). We provide the full set of results with caption descriptions from our human evaluation. In the Opinions domain. Tables [3 contain the human evaluation results for sets from baseline methods, Tables [2] contain the human evaluation results for sets from QDAIF methods, and Tables 28]288contall the human evaluation results for sets from embedding feedback QD methods. In the Stories Genre domain, Tables 17 contain the human evaluation results for sets from baseline methods, and Tables 33] contain the human evaluation results for sets from QDAIF methods. For the Stories Ending domain, Tables 2 contain the human evaluation results for sets from baseline methods, and Tables 37 40 contain the human evaluation results for sets from QDAIF methods.
We compiled the results of the full study in Appendix A.12 and summarized the stats across the study in the paragraphs below.
Comparison of quality scores We observed that both annotators generally have close agreement in their ratings with an average difference of 0.9 Likert points but there were occasional instances where they may differ by 2 or 3 units. These deviations occurred in 15% and 5% of the cases, respectively. This reflects that assessing the quality is somewhat subjective. To get a final estimation of the quality of the generated texts, the scores from both annotators were averaged.
Figure 5) shows the average quality ratings from annotators for various ranges of fitness (here, the quality score obtained from AI feedback). The ranges were chosen in a way that ensured a similar amount of samples in each range. We observe clear evidence of a correlation between human and AI quality measures, indicating the usefulness of using AI feedback to assess quality. However, we also observed that fitness for the texts with the highest scores becomes uncorrelated to human-assessed quality. This implies that above a certain threshold, fitness is not a reliable measure of quality in some cases (and maybe slightly lower fitness solutions were preferred by humans). Therefore, we suggest that in future work. more research is done in studying the relationship between (high- confidence) evaluations from AI feedback, and reward hacking of solutions (Nguyen et al. 2015a
17

---

Published as a conference paper at ICLR 2024
5
4
quality
3-
2-
1
0-0.7
0.7-0.8
0.8-0.9
0.9.0.95
0.95-0.98 0.98-0.99 0.99-0.995 0.995-1
fitness range
Figure 5: Correlation plot between quality rating from human annotators, and fitness range (quality computed from AI feedback). Mean human-annotated quality and statistical error for different ranges of AI feedback fitness scores indicate more frequent instances of reward hacking (kaalse et al. 2022 study.
Lehman et al. 2019) from the outputs of some search methods evaluated in this
Skalse et al. 2022 Lehman et al. 2019) under certain conditions and controls during the search (e.g. through the of seed texts for human-preferred outputs, shown in Table additional
constraints on the generated solutions during search (Lehman & Stanley. 2010 Brant & Stanley 2017))
Comparison of diversity measurements For diversity, we observed that the two annotators agreed on classification of a generated text in the same category percent the time. the Al feedback side, diversity is collected on a 20-bin axis, measuring different degrees of corre- spondence to two sxperiment-speciffe categories mentioned carlier. For the purpose of comparison with human feedback, these bins are clustered into a first category (bins 0 to 8), second category (bins to 19) and a neutral category (bins and 10). Additionally, the samples for each set collected from relatively uniform spread of bins from one end of the diversity axis to another (specifically, bins in set [0, 6, 9, 13, 19], except for when the method fails to find a solution for the bin, then the next closest solution near the bin is chosen). This arbitrary arrangement allows for relatively uniform distribution among the three categories.
On average, AI feedhack agrees with human annotator the text category label 73% of the time. On samples where both annotators agree on the label of the text, this agreement rate increases to 82%. Within this set, the agreement increases 95% on samples when look at samples where both annotators give label that is not the neutral category. For some samples from bins 6 and 13, one human annotator gave a neutral label and the other annotator one of the other two labels (closer to the AI feedback diversity measure for these bins), indicating that some samples lie between neutral and extreme the given measure. These findings suggest that diversity classification from Al feedback is relatively reliable, especially in texts where humans agree on the label.
Baseline quality rating. The average quality score given by annotators for a given sample was 3.18. close to the middle rating of 3. This gives us an indication of what could be considered the threshold for subjectively good or bad outputs.
The average human (subjective) QD score of all sets in the study is 0.606. This is another indication for the threshold for determining which set from a given run/method had high-quality, diverse texts.
18

---

Published as a conference paper at ICLR 2024
Table 2: QDAIF outperforms QD with embedding feedback (QDEF) according human eval- uation in human QD score and quality, when the difference is in feedback type for each base method.. The mean of Human QD score and quality are computed for the set of each method- /run. The differences between the two evaluators for QD score and Quality are shown. Agreement between human and AI feedback on diversity labels given is slightly higher across QDAIF results compared to QDEF results, as well as agreement between two annotators.
Human
QD
Quality rating
Quality range
Human-AI agreement
Human agreement
Method
QD score
range
QDEF, LMX, Sero-Shottte Init
0.242
0.050
1,300
0.600
0.500
0.400
QDER, LMX, Sccded Init
0,308
0.050
1,700
0.600
0.600
0.600
QDEF, LX Replace, Pero-Shot Init
0.417
0.233
2.100
1,000
0.800
0.600
QDER, LMX Replace Seeded Init
0,500
0,200
2,300
1,000
0.800
1,000
QDAIF, LMX. Zero-Shot Init
0.350
0.300
1,400
1,000
ODAIF LMX. Seeded Init
0.617
0.167
3,200
1.200)
1.000)
1,000
QDAIE LMX. Replace, Zero Shat Init
0.833
1,000
QDAIF, AMX-Replace, Seeded Init
0.739
0.078
3,700
0.600
0,700
0,800
A.2 COMPARING AI FEEDBACK AGAINST ALTERNATIVE MEASURES OF DIVERSITY
To understand the effect of fuzzy evaluation tools as a component of our QD setup, we tested the use of alternative method of feedback compared our default method (AI feedback) in the MAP- Elites pipeline: semantic embedding feedback (Reimers & Gurevych 2019). For this method, we used a 13B embedding mode based the architecture described in Muennighoft (2022), with an asymmetric search setup to measure the distance between a generated text (document embedding) positive opinion"). To compute
and a query embedding for desired measure (e.g. "This diversity measure that can be defined on an axis, we first get the cosine distances between the docu- ment embedding and each of two opposing attribute query embeddings. From this, we can measure how close the document embedding is to one attribute compared to the other, and obtain a single diversity measure normalized in the range [0, 1J. We use the same method for quality feedback, with query that aims to measure the relevance of generated texts to specific domain. The cosine similarity is used here as a quality score, with negative values being clipped to O. Additional setup details are shown in Appendix A.28 Since the subjective quality of the resulting elites (of creative texts) is more informative of the method's potential in further applications for practical synthetic data generation, We conducted human study in addition to the reporting of QD score stats as part of our results.
We display performance statistics from our runs (QD Score) well human evaluation scores on elite samples in Table 2] We observe from our human study that using Al feedback as the evaluator instead of semantic embedding feedback for every variation of the QD run setup potentially leads to subjectively better generations. This is likely due to more prominent reward hacking (Skalse) al. 2022; Lehman al.] 2019) occurring in runs using embedding feedback. where the highest quality score texts end up being very similar to the query 'An opinion piece about eating vegetables and plant-based foods", while not optimizing for the subjective quality of texts in different bins. Qualitative analysis of human-evaluate
sets of texts from QD with embedding feedback is shown in Tables 28 Furthermore, agreement between human and AI feedback on text diversity labels was slightly higher across QDAIF sets compared to QD with embedding feedback (QDEF) sets. Overall, Al feedback outperforms the alternative measure of semantic embedding feedback, by guiding the generation of texts that are more preferred by humans, and by serving better evaluator for quality and diversity measures than embedding feedback.
DDxxp' www.Deeppo...eeee..e... $20mode) &2D 2200or 1d
20c1 has
19

---

Published as a conference paper at ICLR 2024
20.0
17.5
15.0
12.5
Score
10.0
138
2.5
308
70B
1000
1250
2000
Rerations
Figure 6: QD score plots for different LMX model sizes on the Stories Genre domain. There no clear trend in scaling model size with QD score.
A.3 ON SCALING LMS FOR MUTATION
Previous work has consistently shown that LMs demonstrate improved capabilities in various task- based benchmarks at larger scales (Kaplan et al. 2020 Chowdhery et al. 2022 Chung et al.) 2022). This applies to performance in solving tasks through in-context learning, which LMX is based on. Prior work in LMX (Meyerson et al., 2023) has found that a relationship between model scaling and performance of mutations be observed when evolving binary strings in search domain. Interestingly, experiments in LMX well as ELM (Lehman et al.) (002)) observed that in some cases, emerging capabilities mutation ability appear for reasonably small LMs, but may not scale with clear trend.
Firstly, we show the QD score performance between runs in Figure 6 The standard error in the score across 5 seeds is also shown. We observed that the QD score from the 70B runs converged to a lower point compared to the 13B and 30B runs, which have comparable scores. This highlights trend between model size and QD score that is not directly proportional. Although suggestive trends are not seen here, study based on subjective feedback is still necessary for deeper understanding of the performance of each experiment.
We observed from the human feedback evaluation a trend in the quality ratings, with average quality scores from each experiment set of: 3.43 (13B), 3.60 (30B), and 4.03 (70B). The quality score increases with an increase model size used, with a higher jump in score for generated stories from the 70B runs.
In terms the agreement on the genres of evaluated stories between AI feedback and human feed- back, we observed the following percentage rates in the following sets: 80.0% (13B), 73.3% (30B), and 73.3% (70B). There is slight decrease in agreement for texts from the 30B and 70B runs. We found that evaluators differed more frequently in their labels on stories deemed as "neutral" or "ro- mance" according to AI feedback, suggesting that the use of the prompt pool in combination with larger models might lead generated texts with misaligned AI evaluations on the genre. Still, these agreement rates indicate good alignment between Al and human feedback.
20

---

Published as a conference paper at ICLR 2024
Opinions
Stories Genre
Stories Ending
Seeded Init, QDAIF
Zero-Shot Init, QDAIF
1
Zero-Sho Init, LMX Quality-Only
50
14
20
QD Score
QD Score
QD Score
Figure 7: QD score Performance between Seeded Init and Zero-Shot. Performance stats with mean, interquartile mean (IQM) and bootstrapped 95% CI, across 5 random seed runs. Seeded Init potentially better, but within the CI of Zero-Shot Init. In the Stories Genre domain, the CI of -hro-Shot Init runs is much wider, indicating significant variation in performance for different random seeds.
A.4 ON FEW-SHOT AI FEEDBACK PROMPTING
instruction-following LMs are typically trained to align the model towards generating better answers to zero-shot prompts (Wei et al. 2021 Ouyang et al., 2022). However, few-shot prompting with exemplars was shown to be effective in some aspects with instruction-tune LMs, especially towards
understanding task structure and improving robustness prompting variations (Wei al. 2021
In terms of the average human-evaluated quality, we see a drop in subjective quality for the set of stories from 2-shot AI feedback runs (3.10) in comparison to zero-shot Al feedback runs (3.43). We observed for the other sets that this score increases for the 4-shot set (3.93) and the 8-shot set (4.03). Furthermore, we see that this trend mostly consistent when we consider the scores for each bin category. This suggests an improvement in QDAIF's s ability to discover texts that are perceived to be of higher quality according to human feedback when we use higher number of in-context examples during AI feedback evaluation.
In terms of the agreement between Al feedback and human feedback, we see a drop in average agreement for ratings that were given stories in the few-shot feedback experiment sets, with 50.0% (2-shot), 66.7% (4-shot), and 56.7% (8-shot) agreement on sets, compared to 80.0% for zero-shot (default). The level of disagreement occurs more frequently stories evaluated to have the romance genre. Additionally, evaluators labeled samples from the few-shot sets as "horror" or "neutral" more frequently than "romance", while the proportion of labels given to samples from the zero-shot set was more uniform.
The performance may vary due the ordering of in-context examples in few-shot prompts (in Appendix A.29) as shown in Lu et al. (202T). Furthermore, the nature and wording of input-output exemplars/tasks could also influence the performance, in addition some variation due to the nature of subjective evaluation.
A.5 ON THE INITIALIZATION METHOO FOR QDAIF
Recent results from LMX demonstrated successful optimization when the search was initialized by evolving set of pre-existing seed examples (e.g. examples of equations, quotes, text-to-image prompts, code) (Meyerson et al. 2023). At the same time, previous applications of QD methods demonstrated successful search outcomes when using random initialization in different domains, such as in robotics, and latent space illumination of generative models (Cully et al. 2015 Fontaine & Nikolaidis 202T Bhaut et al. 2022).
We compared the performance between QDAIF with Seeded Init against Zero-Shot Init and a base- line method LMX, Quality-Only initialized with Zero-Shot Init. From Figure 7 we can see that there is potential improvement in QD score when using Seeded Init compared to Zero-Shot Init, but the difference may not be significant. However, the difference clear on the lower performance of LMX, Quality-Only with Zero-Shot Init. This suggests that Zero-Shot Init is viable for QDAIF. Still, we need to analyze the effect of the initialization on qualitative samples of elite texts. We also compared the effects of initialization on subjective quality-diversity of texts (see Table 3). Sets of texts discovered by Seeded Init within single runs were found to be subjectively higher-quality and more diverse compared to sets from Zero-Shot Init (0.772 vs. 0.383 subjective QD score). This suggests potential reward hacking (described for RL problems in Skalse et (202)) Lehman et al. 2019) during the search with Zero-Shot Init runs, where potentially out-of-distribution texts can
21

---

Published as a conference paper at ICLR 2024
evolve to optimize the Al feedback quality score of solutions, but lead to subjectively low low-quality texts across bins.
A.6 ON THE MECHANISMS OF MUTATION FOR QDAIF
Prior work on ELM highlights the versatility of LMs in applying variety of potential muta- tion methods, such as in the form of git diffs, or prompting to directly evolve text/code Lehmann al.] (2022). However, studying the potential effects of different LM-based mutation operators on population-level evolvability (focused on future creative potential Lehman et al. (2016)) re- mains a challenge. Towards understanding the potential of different mutation methods terms of population-level evolvability of niches, we compare LMX(-Near) (default) (as tested in Meyerson ct al. 2023)) against -MM-lee terms of resulting generated texts across different runs.
To observe the impact of different mutation methods on the dynamics of the population during search (along with the impact on performance), we implemented and tested an additional method called LMX-Replace This method evolves few-shot prompts using larger pool of prompt candidates than that of LMX(-Near) and carries out slower mutation process by modifying only one few-shot prompt example (instead of all examples) during search iterations. We give detailed comparison between the methods in Appendix A.26
We observed improvement to the resulting generated text sets from runs using LMX-Replace in comparison to LMX for the Zero-Shot Init case, according to human evaluation results highlighted in Table We see that the subjective QD score and the quality rating from human evaluators were higher on average across the tested domains (and improvements within domain- Nomain-by-done by-domain comparisons also, highlighted in Table 4). Given that may be desirable to constrain the search with L.X-NNerr w/ Seeded Init, ReX-Replace can act an alternative mutation method to steer the dynamics of population search towards subjectively improved outputs.
The introduction of an archive depth was used for the creation and maintenance of prompt pool for LMX-Replace (ef. [Appendix A.27p. Future works could explore the usage depth in MAP-Elites archives for domains where uncertainty (or subjectivity) influences the evaluation of solutions, as was previously studied in Flageat Cully (2020 2023).
Table 3: Mean stats, human eval scores, QDAIF methods. LMX-Replace outperforms LMX when using Zero-Shot Init.
Method
QD score
Quality
H-AI agreement
H agreement
LMX Zero-Shot Init
0.383
2.233
0.600
0.600
LMX Replace w/ Zero- Shot Init
0,540
2,800
0,700
0.800
LMX w/ Seeded Init
0.772
3,900
0.833
0.800
LMX MX-Replace w/ Seeded Init
0.763
3,800
0,800
0.800
22

---

Published as a conference paper at ICLR 2024
A.7 ON COVERAGE AND BEST QUALITY SOLUTIONS ACROSS DOMAINS
Figure 8] shows performance plots measuring the coverage of the archive (i.e. how many bins in the search space are filled with least one solution). Figure shows performance plots measuring the quality score of the best solution found across the whole search (existing in one of the defined bins).
QDAIF often discovers the best overall solutions during search that have higher quality scores than the best overall solutions found from other methods. This is likely enabled by the goal-switching mechanism of QD approaches described in Mouret & Clune 2013); Gaier al. (2019), which is more relevant when the QD search is at a later stage and begins to improve the already existing so- lutions (elites) across the bins through the mutation of distant solutions. Overall, QDAIF can jointly optimize the quality and diversity of solutions, outperforming other methods in most comparisons, and highlighting contributions to successful gains in QD score.
Opinions
Stories
Stories Ending
Stories and Ending
Raed Few Shot Shuffling-eww Shy-
Rendom--See Search LMX, Quality-Onl
QDAF
0.950
0.975
1.000 Coverage
1.025
1.050
0.80
0.85
0.90
0.05
1.00 0,900
0.925
0.950 0.975 1.000
0.5
0.6
0.7
Coverage
Coverage
Coverage
Figure 8: QDAIF significantly outperforms baselines in the percentage of available bins cov- ered by solutions all domains. Coverage stats with mean bootstrapped 95% CI, across 5 random seed runs The maximum possible coverage is 1. In some cases, methods achieve maximum cov- erage across all runs The advantage of QD approaches is shown in their ability to discover more solutions that cover wider search space, in addition to finding high-quality solutions (cf. Figure 9).
Opinions
Stories Genre
Stories Ending
Stories Genre and Ending
lixed Shuffing Sho-
Random- Search
LMX, Qualty-Only
ODAP
0.98
1.00
0.980 0.985
0.985
0.980
0.995
Best Solution Quality
Best Solution Quality
Solution Quality
Best Solution Quality
Figure 9: QDAIF and LMX, Quality-Only both optimize for the quality of the best solution found overall. Best solution quality stats with mean bootstrapped 95% CI, across 5 random seed runs. The maximum possible quality score is 1. QDAIF is more often able to achieve a higher score than LMX, Quality-Only in Opinions and Stories Ending, while also outperforming terms of coverage (cf. FFigure 8). In the 2D Stories Genre and Ending domain, LMX, Quality-Only finds solution with higher best quality more often than QDAIF, while still lagging in coverage. is likely that QDAIF can find the best solutions with higher maximum quality more often in some domains because of the maintenance of diverse elite solutions that enable more possibilities to maximize the objective function, especially when the search begins to focus refining existing solutions instead of covering unfilled bins (Gaier et al.) 2019). To save on compute, we reuse the generated (iterations of) texts from the baselines compute independent stats comparisons they do not use diversity measures during a search, unlike QDAIF which depends also on the defined diversity axes to conduct successful search. We compute independent QD score stats for baselines using the same 5 reruns, but in the Stories domains, the best solution quality remains the same.
23

---

Published as a conference paper at ICLR 2024
A.8 ON COMPARISONS BETWEEN QDAIF AND DIVERSITY- SEEKKING BASELINES
We compared QDAIF against explicit diversity-seeking baseline methods. One baseline, LMX, ROUGE-L, is based on n-gram-based filtering with ROUGE-L similarity (Lin) 2004 as done in Self-Instruct Wang et al.] (022a). They followed a similar approach of maintaining a few-shot prompt pool to generate diverse solutions (with diversity maintained by adding generated texts to a prompt pool if different enough to other existing solutions based on n-gram matching), in their case, for the generation of diverse instruction prompts towards the creation of diverse, high-quality synthetic instruction tuning dataset. Another baseline is based on Novelty Search (NS) (Lehman & Stanley, 10) a diversity-seeking algorithm that was proven to be more sample efficient than single-objective approaches (the diversity-only equivalent of LMX, Quality-Only, which only op- timizes for solution quality) in discovering desired solutions to problems with many local optima, and also studied in prior works introducing the QD illumination problem (i.e. improving QD score (Pugh et al.] 2016)). Unlike LMX, ROUGE-L, NS aims to encourage diversity within an arbitrar- ily defined space of diversity. To make NS comparable with QDAIF, introduce Novelty Search through Al Feedback (NSAIF) as the method that seeks diversity in domains where diversity is de- fined by AI feedback axes. We denote this baseline as LMX, NSAIF, As with all methods, these baselines are initialized with the prompt pool specified in Appendix A.2I] We describe the baseline implementations in more detail in Appendix A.9, including variants of these baselines with quality Al feedback (QAIF) filters as a minimal criterion for diversity search (Lehman & Stanley 2010))
Results. Figure 10 shows QD score performance plots comparing QDAIF against diversity-seeking baselines, baselines that also aim for quality and diversity in solutions, and other baselines. QDAIR consistently outperforms
snversity-seeking baselines across domains in addition to other baselines not focused on improving desired diversity in solutions. Yet, LMX, NSAIF tends to outperform other baselines that are not focused on improving solution diversity. On the Stories - Genre and Ending domain, LMX, NSAIF significantly outperforms many methods, even LMX, Quality- Only, which carries out single objective quality optimization (and successfully does so compared to other baselines, by maximizing the best solution quality alongside QDAIF, ef. Fiiuure 12). This indi- cates that maintaining diversity in solutions is important for QD score performance in such domains (and improves search space coverage, cf. Fiiuree 11I. Furthermore, maintaining desired diversity, such as qualitative measures of diversity, is necessary for a method to improve QD score during search. LMX, ROUGE-L achieves low QD scores, and is within range of being the worst method for improving best solution quality (cf. Figure 12). in spite of the higher coverage achieved across domains compared to several baselines (cf. Figure ITp. However, the performance for this baseline improves significantly with the introduction of the quality filter via QAIF. This indicates that main- taining high-quality solutions (not just diversity) is important. Still, adding the quality filter on top of LMX, NSAIF only helped more often compared to the default variant of NSAIF on the Stories Genre and Ending domain, and shows no improvement on other domains. QDAIF (based on MAP- Elites) reliably improves QD score in tested domains and does so with higher sample efficiency (cf. Figure 13 and Figure 14). Results comparing QDAIF to additional diversity-see seekkngg baselines (in- cluding ones with quality filters) highlight performance gains from our proposed QDAIF method, as well as the importance of seeking both quality and diversity.
Opinions
Stories - Genre
Staries- Endling
Stories Gerre Endingg
Exed- Sho-
Shufting Few Shoc
Randomm-Sees Seachh
LMX, Quality Only
HOUGIS)
(w/
LMK, NSAFF twl OAFF
QUAI
19.0
15.5
20.0
1.5
20
17
20 00
50
QD Score
QD Score
QD Score
QD Score
Figure 10: QDAIF outperforms ddiversity-seeking baselines across domains in QD score and maintains nigh-performance across all domains. Performance stats with mean bootstrapped 95% CI. across 5 random seed runs. The maximum possible QD score is 20 (100 for 2D archive (4th plot)). The best-performing baseline in the Stories Genre and Ending domain, LMX, NSAIF (w/ QAIF), does not consistently outperform other baselines in other domains, and has wide per- formance variability in two domains. Compared to LMX, Quality-Only, all the diversity-seeking baselines using NSAIF or ROUGE-L filtering (except LMX, ROUGE-L) significantly outperform this baseline approach that optimizes for a single quality score objective.
24

---

Published as a conference paper at ICLR 2024
Enning
Endingg
Sho
Shutting Fow Shot
Sendamm- Searth
LMX, ROUSE DAIF)
LMX, NSAFF
UMX, NSAF owl QAFI ODAI
Coverage
Coverage
Coverage
Coverage
Figure 11: QDAIF more often outperforms (diversity-seeking baselines in the percentage of available bins covered by solutions all domains, such as Stories Genre, and Stories Genre and Ending. Coverage stats with mean bootstrapped 95% CI, across 5 random seed runs. The maximum possible coverage is 1. some cases, methods achieve maximum coverage across all runs. Consistent high-coverage performance from QDAIF compared to the next best baselines in Stories Genre and Ending (LMX, NSAIF and LMX, NSAIF (w/ QATF)) suggests that evolving from high-quality solutions important improving solution diversity, in addition to diversity- seeking search (where LMX, ROUGE-L also achieves higher coverage more often compared to non non-diversity-seekie baselines.
Opinions
Storles
Starles Ending
Stories Genma
Flaed-hem-SSee Shot
Shuffling Fow Shot
Handomm-Seee Seadom--Seacch
Only
UMX ROUGE twi QMF)
LMX,, NSAIF
LMK, NSAF (wl GAIF) QDAF
0.885 Quality
Quality Figure 12: QDAIF and LMX, Quality-Only both more often optimize for the quality of the best solution found overall. Best solution quality stats with mean bootstrapped 95% CI, across random seed runs. The maximum possible quality score is 1. The addition of quality filters to LMX, ROUGE-L (w/ QAIF) and LMX, NSAIF (w/ QAIF) is ineffective for significantly improving the best solution quality across runs compared to QDAIF. Still, LMX, ROUGE-L (w/ QAIF) is better compared to LMX, ROUGE-L at finding higher-quality best solutions.
Best Solution Quality
Quelity
25

---

Published as a conference paper at ICLR 2024
Opinions
Stories Genre
20
QD Score
15
QD Score
10
10
LMX, ROUGE-L
LMX, ROUGE-L
LMK, NSAIF
LMX, NSAIF
Quality-anly
LMK, Quality-only
QDAIF (ours)
QDAIF (ours)
0-
500
1000
1500
2000
0
500
1000
1500
2000
Iterations
Iterations
Stories Ending
Stories Genre and Ending
20
QD Score
15
OD Score
10
LMX, ROUGE-L
LMX, ROUGE-L
LMX, NSAIF
NSAIF
LMX, Quality-Only
Quality-Only
QDAIF (ours)
ODAF tours)
500
1000
1500
2000
500
1000
1500
2000
Iterations
Iterations
Figure 13: QDAIF outperforms (diversity-sceking) baselines across domains sample effi- ciency of QD score improvement. Line plot stats with mean bootstrapped 95% CI, across 5 ran- dom seed runs. The maximum possible QD score is 20 (100 for 2D archive (4th plot)). Additionally, LMX, NSAIF succeeds in obtaining higher sample efficiency more often than LMX, Quality-Only, the single quality objective optimization method, without any optimization the quality score ob- jective.
26

---

Published as a conference paper at ICLR 2024
Opinions
Stories Genre
20
QD Score
15
QD Score
15
5
IMX, ROUGF-I TW/ QAIF)
IMX, ROLIGFII Tw/ OAIF)
LMX, NSAIF (w/ QAIFI
LMX. NSAIF (w/ QAIFI
(ours)
QDWIF (ours)
0-
500
1000 Iterations
1500
2000
0
500
1000
1500
2000
Iterations
Stories Ending
Stories Genre and Ending
20
60
QD Score
15
QD Score
40
10
5-
IMX, ROUGFI (W DAIF)
20
I MX, ROUGF-I (W/ QAIF)
LMX, NSAIF (w/ QAIF)
LMX, NSAIF (w/ QAIF)
QDAIF tours)
(ours)
0
500
1000
1500
2000
500
1000
1500
2000
Iterations
Iterations
Figure 14: QDAIF outperforms (diversity-seeking, with quality control) baselines across do- mains in sample efficiency of QD score improvement. Line plots with mean bootstrapped 95% CI, across 5 random seed runs. The maximum possible QD score is 20 (100 for 2D archive (4th plot)). The improvement of QDAIF compared to other approaches aiming for diverse, high-quality solutions is significant in the Stories Genre domain. In the Stories Genre and Ending domain, LMX, NSAIF (w/ QAIF) outperforms LMX, ROUGE-L (w/ QAIF) and is within the performance range of QDAIF in later iterations, but with lower sample efficiency in early iterations, and wider performance variability.
27

---

Published as a conference paper at ICLR 2024
A.9 SETUP FOR DIVERSITY-SEEKIA BASELINES
LMX, ROUGE-L. Following Wang et al. (002a), we set the threshold for the maximum allowed ROUGE-L similarity between generated texts and all texts in the prompt pool any given time (from which few-shot examples are sampled for LMX generation) to 0.7, without limits on the prompt pool size. If the similarity exceeds this, the generated text is still logged for QD score evaluation, but rejected from being added to the prompt pool. Otherwise, the text is added to the prompt pool so that it can be used for LMX evolution during later samplings of few-shot prompts. The rest of the setup is comparable to QDAIF except for the MAP-Elites archive maintained by QDAIF
LMX, NSAIF. In line with the high-level implementation of NS described in Lehman et al. 2008), we implemented the novelty (diversity- measure in the same way that QDAIF defines diversity measures- as the diversity attribute defined by the AI feedback axis (or axes) in the range [0. 1]. with an additional Euclidean distance- based mcasure between the diversity attribute of the generated text, and existing texts in the prompt pool. Novelty is computed by measuring the mcan distance between the diversity attribute values of the generated text, and their & nearest neighbors in the prompt pool (with respect to the diversity attributes of the neighbor texts). We keep l to be 15, was tested in the original NS implementation. Similar to NS, the novelty score is compared to a novelty threshold (initially 0.05 in all LMX, NSAIF runs, the distance width between adjacent bin ticks for unit range with 20 uniform bins); if the novelty of the generated text is higher than the threshold, it gets accepted into the prompt pool for evolution with LMX, otherwise, of performance results but rejected from the prompt pool. We apply a dynamic adjustment of the novelty threshold, following NS, and define so that the threshold is multiplied by 1.05 (increased) if 3 solutions in a row were accepted to the prompt pool within any iterations window, or multiplied by 0.95 (decreased) if 21 solutions in a row were rejected from the prompt pool during the search.
logged for evaluation
In consideration of the non-linear calibration of Al feedback models in evaluating texts (cf. App pendix A.3T), we apply a piecewise linear transformation to the original diversity attribute value from AI feedback evaluation of diversity (that lies along axis discretized by non-uniform bin- ning) so that it lies instcad along an axis discretized by uniform bins while preserving the number of bin intervals. For example, in our 20-bin setting, the input value 0. 0.9975, which lies between bin ticks 1, and 0. 0.995, would be transformed to an output value of 0.975, and the input value 0.35 (between bin ticks 0.5 and 0.2) would be transformed to an output value of 0.475. This enables novelty to be computed using Euclidean distances while preserving the nature of AI feedback model non-linear calibration in distinguishing subjectively similar solutions from diverse solutions. Additionally, this keeps the definition of diversity consistent and fair with the setup of diversity measures for evalua- tion (in QDAIF iterations, and for baseline performance comparisons). The number of bin intervals defined for the piecewise linear transformation is the same as the number of intervals set as default across the Opinions and Stories domains (20 for 1D archives, 10 for 2D archives).
Quality AI Feedback Filtering. We implemented and assessed variants of the above baselines, LMX, ROUGE-L (w/ QAIF), and LMX, NSAIF (w/ QAIF), by adding a simple quality filter for accepting generated texts to prompt pools, based on quality AI feedback defined in the respective creative writing domains. This involves an additional step in each baseline method immediately after the diversity criteria assessment of solutions, where generated texts must also have quality score val- ues above minimum threshold before being added to the prompt pool. Thus, these variants become similar QDAIF methods, where they aim for diverse, high-quality solutions. These baselines dif- fer from our default QDAIF method with MAP-Elites in that the minimum quality thresholds are not function of the quality of best solutions across individual bins, but are arbitrarily defined, to constrain the prompt pool to satisfy a minimal criterion for quality, as was introduced previously for siversity-seeking methods in Lehman & Stanley (2010). This would make the quality improvement We define this simple quality threshold for
process of solutions across all bins less sample-efficia the baselines to be fixed at 0.8, closer to the upper bound of the full quality range [0. 1]. This was determined based on the intuitions of the results from the human evaluation study we conducted (cf. Figure 5)in in Appendix A. where a quality value of 0.8 from AI feedback corresponds to a generally high human feedback Likert (quality) score. As done in LMX, Quality-Only, these baselines (with quality filtering) limit the size of the prompt pool to be up to 100.
28

---

Published as a conference paper at ICLR 2024
A. ON AUTOMATICALLY EXPANDING ARCHIVE DIMENSIONS
Prior work in QD, as described in Section 2.2, often relies on diversity measures that are designed
and manually defined at the start of the search. One existing approach for more automatic QD search without supervised (defined) measures of diversity is to use unsupervised learning to represent di-
versity without relying on ground truth measures (Cully & Deminss 2018 Cully 2019: Grillotti & Cully] [02T Wang et al., 2023b; Ding et al. 2023). Such unsupervised approaches do not embody any complex prior of what humans find interesting about diversity; an alternative approach would be to query capable LMs about what dimensions of diversity are interesting or important for a particular domain. In this way, semantically-rr -rrih axes of diversity could be automatically generated (and could then be evaluated automatically as well through other LM calls, as in QDAIF).
Given the advances in foundation model capabilities, we could reasonably prompt LMs (such as GPT-4 (OpenAl| 2023) to come up with new axes of diversity in a more automated pipeline poten- tially while search is running- by giving it description of the user's search problem, and existing diversity axis being searched through (i.e. the existing AI feedback diversity prompt(s)), we could different, previously unexplored diversity axis and define a new Al feedback prompt that can be added to the MAP-Elites evaluation. For example, in the Poetry domain, we could ask the LM to generate multiple diverse aspects of poetry (e.g. "Genre" and "Tone" as studied in the presented experiments), and also come up with diverse categories defining this search space for QDAIF to search through with MAP-Elites. The effectiveness of this kind of approach has not been studied thoroughly in prior works, especially on the question of whether or not expanding the dimensions of diversity during the search can meaningfully improve diversity in resulting solutions towards increasingly broader definitions of diversity.
ask the LM to give
Setup. We took a step in the direction of automating the definition of diversity axes, where we analyzed the effectiveness of creative search for solutions when we expand the dimensions of di- versity axes during an intermediate iteration of an cxisting search, with performance measured in terms of improving the QD score (for a given ground truth, nigher-dimensional search space). We tested this approach in the Stories Genre and Ending domain, compared the performance be- tween variations in methods with QD score (out of 100) done in other experiments, and compared the following setups: 2D Archive Search (apply QDAIF with the full 2D diversity axes defined), ID Archive Search (apply QDAIF with only a ID diversity axis defined, done for both Genre, and Ending diversity archives each), and expanding ID to 2D Archive Search (starting QDAIF search with either the Genre or Ending diversity axis defined, and then expanding the search with the ad- dition of a second diversity axis, or the other different axis to the one defined at the start of the ID search). For the setups with expanding diversity axes, the second archive dimension introduced from iteration 1000 of the search, out of 2000 iterations total. We also assessed the performance in the case where instead of adding an extra dimension with a new diversity axis during search, QDAIF transitions from an initial ID archive (e.g. for Ending), to different ID archive (e.g. for Genre). In our experiments, the transition is carried out from iteration 1000 out of 2000 iterations.
Findings. Figure 15 shows performance plot results for all the settings described in this section, and Figure 16 shows QD score line plots to visualize sample efficiency differences between the different setups of automatically expanding dimensions, especially after iteration 1000. We found that for all cases tested of expanding from 1D) to 2D archives, improvements in QD score for a higher dimensional archive are significant when compared searching with only a ID diversity axis and evaluating the resulting solutions with both AI feedback diversity measures. Furthermore, we found that it's possible approach the performance in QD score through this diversity axes expansion when compared to the QD score achieved from a full 2D Archive Search. This level of improvement hints at the potential of applying the approach of prompting LMs to generate diversity measures for more autonomous, creative search, and highlights the value of scaling up this approach in future work.
Expansion of diversity axes is promising as an approach to improve the quality and diversity of solutions without the need to initialize a high-dimensional archive from the beginning (in the case where only manual setting of diversity axes is done). Vassiliades et al. (2016) found in experiments through CVT-MAP-Elite (a method to define bins based on uniformly spaced centroids in very nigh-dimensional spaces of diversity, solving the compute requirements of standard MAP-Elites in this high digeesiiinsional case) that standard MAP-Elites is impeded in its ability to further improve the fitness (quality) of solutions in non-empty bins the cases where the search fills empty bins
29

---

Published as a conference paper at ICLR 2024
occurs much more frequently when more bins are created due to the increase in dimensionality. The quality of existing non-empty bins normally requires several iterations of improvement before reaching more optimal quality scores for the given bin niche. We show that this is also the case in Figure 15 in the third plot on the value of best solution quality; the quality of the best overall solution at the end of the search is more often higher for the 1D) 2D expansion settings, compared to the setting where the 2D archive is initialized from scratch at the start of the search. It also the case that searching only in the ID archives is more often better (with higher best solution quality) compared to the outcomes when conducting any search in the higher dimensional archive. Expansion of dimensions seems to deliver a good trade-off of slightly decreased best solution quality for significant improvements in solution diversity. This balance is quite useful to find for improving the sample efficiency of the search, given that Mouret & Clune (2015) found the discovery and maintenance of both diverse and high-quality solutions to be important for enabling the ongoing search to improve the quality and diversity of newly generated solutions even more quickly.
Additionally, we studied performance comparisons for the case of transitioning different ID diver- sity axes (as a different approach to automatic search in different dimensions of diversity compared to the expanding dimensions setup). Figure 15 also shows performance in this case, and Figure 17 shows QD score line plots, with differences in performance visibly shown after iteration 1000. In one case (1D (Ending) to ID (Genre)), significant improvement in QD score is visible for this archive transition case when compared to searching only in one (initialized) diversity axis throughout the whole search. This setup also approaches the performance of searching in the higher dimensional 2D archive. In the other case evaluated (1D (Genre) to ID (Ending)), no notable improvements were seen compared just conducting QDAIF in the single ID (Genre) archive. Even though results here show that the performance of QDAIF in the transitioning 1D archives case is sensitive to the diversity axes searched through (and the order in which the transitions occur), this highlights another promis- ing approach to automatically adjusting diversity axes given Al feedback prompts to be generated by LMs along with the expanding dimensions setup. This is especially the case when we want to scale up QDAIF to search through an even higher number of diversity axes automatically, where we can lower the computational requirements of searching in lower dimensions (meaning lower number of total bins created) and also mitigate the presented challenges of conducting MAP-Elites search iter- ations in very high-dimensional archives (in the previous paragraph). while maintaining promising improvements to performance that would be seen in having QDAIF explore growing number of different diversity axes. Yet, the importance of the diversity axes chosen in further encouraging nov- elty is clear; prior work studying the theoretical properties of divensity-seeking algorithms showed through empirical experiments that the choice in diversity measures also needs to be aligned with the intended task in mind, for solutions of interest to cover the search space (Doncieux et al., 2019) Figure 6). Future research can explore the potential of designing curriculum for QDAIF to search through different diversity axes as an open challenge to improving the quality and diversity of cre- ative texts with potentially an unbounded number of subjective dimensions of diversity, depending on individual personal perspectives.
Overall, this method of automatic expansion and adaptation of diversity axes introduces new way of balancing the trade-off between improving quality or improving diversity in solutions, especially in settings where the dimensions of desired diversity, while sometimes not obvious to the user until later realization, do not reach the hundreds as would be in the case of problems studied in CVT- MAP-Elites. In the large-scale higher-dimensional case, adaptive transitioning of diversity axes is another promising direction. Such approaches to adaptations of diversity axes draw inspiration from work on learnable behavior characteristics (Meyerson et al., 2016) and meta-diversity (Etcheverry ct al. 2021). Given prior findings on the adaptation of the archive search space to encourage more interesting solutions (worth collecting) to emerge during search, and the ability of LMs to provide feedback on unbounded aspects of solution diversity, perhaps new opportunities have emerged in improving future systems inspired by QDAIF. We can leverage this finding for future work to more confidently apply the aid of LMs in automatically generating diversity axes for QDAIF search.
30

---

Published as a conference paper at ICLR 2024
1D Archive Search (Ending)
10 Archive Search [Genro)
1D [Ending) to 1D (Genret
10 (Genre) 1D (Fnding)
(Ending) 20 (Genrc, Endingl
Ending!
Archive Search (Genre, Ending)
0.6
0.990 0.995
QD Score
Coverage
Best Solution Quality
Figure 15: Expanding (or adapting) archive dimensions for QDAIF during search enables im- proved search in higher dimensional search spaces compared to searching for solutions along single 1D diversity axis. QD score, coverage, and best solution quality stats with mean boot- strapped 95% CI, across 5 random seed runs. The maximum possible QD score is 100. In both cases of dimension expansion (expanding from ID (Genre) and ID (Ending) archives to 2D) enables the performance to approach that of conducting QDAIF search in the full 2D archive, with significant improvement coverage compared searching purely in their respective ID archives. For one case of adaptive archive transition for QDAIF (ID (Ending) to ID (Genre)), QD score performance and coverage significantly improve compared to searching only the 1D (Ending) archive.
Stories Genre and Ending
Stories Genre and Ending
60
QD Score
QD Score
40
40
20
1D Archive Search (Genre)
20
Archive Search (Ending)
1D (Genre) to 2D (Genre, Ending)
1D (Ending) to 20 (Genre, Ending) Archive Search iGenre, Ending)
2D Archive Search (Genre, Ending)
0-
500
1000
1500
2000
500
1000
1500
2000
Iterations
Iterations
Figure 16: QD score sample efficiency improves from the midway iteration step of orange line plots when the archive dimensions are expanded automatically, only in the 1D archive during QDAIF. Line plots with mean bootstrapped 95% CI, across 5 random seed runs. The maximum possible QD score is 100. In both cases, either starting with the ID (Genre) archive (left) or the ID (Ending) archive (right) and expanding the 2D archive improves QD score sample efficiency with respect to the 2D archive definition, and approaches performance of the green lines (searching in the 2D archive). The blue lines (ID archive only) converge lower QD score during later iterations.
comparison to doing search
31

---

Published as a conference paper at ICLR 2024
Stories Genre and Ending
Stories Genre and Ending
60
QD Score
OD Score
40
20
1D Archive Search (Genre)
Archive Search (Ending)
1D (Senre) to 1D (Ending)
(Ending) 10 (Ganre)
2D Archive Search (Genre, Ending)
2D Archive Search (Genre, Ending)
0-
500
1000
1500
2000
500
1000
1500
2000
Iterations
Iterations
Figure 17: QD score sample efficiency improves in one case (right plot) from the midway it- eration step of orange line plots when the archive diversity axis is changed to new diver- sity measure, but not noticeably improved in another case (left plot), comparison to doing search without transitioning to different diversity measure during QDAIF. Line plots with mean bootstrapped 95% CI, across random seed runs. The maximum possible QD score is 100. The successful improvement in QD score sample efficiency happens in the case of transitioning from the 1D (Ending) archive to the 1D (Genre) archive, approaching the performance of doing QDAIF in the higher dimensional 2D archive, shown in the right plot. The case shown in the left plot is still below the performance of QDAIF conducted in the 2D archive, with the orange line converging early to lower score. The blue lines (ID archive only) converge lower QD score during later iterations.
32

---

Published as a conference paper at ICLR 2024
A.11 ON FINETUNING MUTATION MODELS
Prior work on ELM found that using a domain-specific finetuned git diff LM during code generation search led runs with higher QD score performance compared when using a pre-trained git diff LM (Lehman et al. 2022). The finetuning step relied on a dataset of filtered solutions across several previous runs, adding complexity to the process of creating an effective model for evolving text. To simplify the process, We investigated finetuning on sampled solutions within the same run, during a search that begins with solution generation from a pre-trained LM. We explore the impact of finetuning as mechanism that can potentially encourage exploitation (through learning to generate higher-quality score solutions) during the search while using evolution to encourage exploration.
To collect samples for finetuning during the search, the first step is collecting a dataset for given state of the archive, sampling up to 10 solutions with the highest quality score from each bin. Archive bin depth
used to keep track of these finetuning samples (ef. [Appendix A.27 bin depth used), and introduced in a different application of MAP-Elites for uncertain environments in Flageat & Cully (2020) 2023). The dataset is shuffled, and then training samples are batched for the LM to finetune on during this phase. Each training sample consists of the prompt that was originally used by a solution to generate an output, and the generated completion text as target tokens for finetuning. We tested both full-model finetuning, and a parameter-efficien finetuning method with sequential adapters finetuning, as described in He et al. (202T). We ran each method using QDAIF LMX(- Near) w/ Sceded-Init (including default method runs without finetuning) on the Stories (Genre and Ending) domain, extending the number of archive dimensions to two. For experiments with adapters, we also conduct runs where the adapter layers are initialized but without doing finetuning, to enable comparisons with adapter finetuning runs accounting for differences in LM architecture introduced by additional layers during the search for runs with finetuning. Still, near-identity initialization (as described in Houlsby et al. (2019) preserves the general performance the LM.
This method runs a single finetuning phase during the search before resuming generation with the finetuned LM. We vary two parameters during experiments with Finetune-Once: the iteration step to start the phase (Start), and the number of finetuning steps during this phase (Steps).
Finetune-Once.
Generate-Finetune This method extends Finetune-Once by carrying out the finetuning phase multiple times during the search, repeating the steps of dataset collection and finetuning at regular intervals. We add an additional variable parameter, controlling the regular interval frequency for every set number of generation iterations (Frequency).
Observations. We compared the QD score performance between methods across 5 fixed ran- dom seed runs. In general, the performance of the Finetune-Once methods with higher scores is comparable to default runs, though slightly lower (but mostly within the confidence interval). For Finetune- Only runs, we observed potential decrease in performance when using full-model fine- tuning compared to adapter finetuning, in comparison to conducting these runs without finctuning, with an average -3.09 difference in QD score for full-model finetuning compared to -0.33 dif- ference for adapter finetuning. It is possible that overfitting on solution examples is more likely for full-model finetuning, which would lead to a decrease in the ability of the LM to evolve new solutions to be accepted in the archive. When we used adapter finetuning as the default for other ex- periments, we observed potential decrease in performance and increased variance in performance across multiple seeds when compared to the default method without finetuning. The increase in variance also indicates potential improvements to the search with Finetune-One Onnlu in some cases, but further studies are needed on the behavior of search when finetuning different archive states and constraints to samples collected for finetuning. We can observe a clearer impact of the adverse ef- fects of finetuning from our Generate-Finetung runs. Considering the 95% confidence interval, the highest mean QD score run we tested from the Generate-Finete score of 78.74 = 5.10 compared to 84.09 + 1.91 from the default generate-only method. With most runs with Cenerate-Finetury converging to a lower QD score earlier in the search. it is likely that running standard scheduling the finetuning phase multiple times leads to overfitring of the LM the point where the model fails to generate new solutions to add to the archive.
group obtained a performance
33

---

Published as a conference paper at ICLR 2024
A.12 HUMAN EVALUATION STUDY STATS WRITING DOMAINS
We collected the annotation results from our study, to show the performance of each search method the tested writing domains, displayed in Table
Table 4: Extended human eval stats for different creative writing domains, baselines and
QDAIE. The QD score presented here is based on subjective human evaluation results, normal- ized to a possible range of [1/15, 1]. terms the observed difficulty of uncovering the search spaces for each domain from mean quality and QD score stats within each domain, Opinions is the easiest domain (mean QD score 0.702, mean quality score 3.613), followed by Stories Ending (mean QD score is 0.648, mean quality score is 3.575), and Stories Genre (mean QD score is 0.591, mean quality score is 3.013). The range of metrics between two human evaluators is computed to highlight the variance in some contexts due to more complex aspects of subjectivity in evaluating certain texts.
Human
QD score range
Quality rating
Quality range
Human-A1 agreement
Human
Method
QD
Opinions
Fixed-Few- Shot
1,000
0.000
5.000
0.000
0.700
0.800
Shuffling. Few-Shox Shot
0,728
0.056
3,500
0.800
0.700
0.800
Random-Search
0,600
0.267
3.600
0.800
0.700
0.800
LMX, Quality-Only
0,750
3.800
1,200
0.600
QDAIF (LMX w/ Secro-Shotin Init)
0.350
0,300
1.700
1.400
0.800
1,0000
Seeded Init)
QDAIP (IMMXReepacce w/ Zem-Shot Init)
0,833
0.267
4,300
1,000
0.600
0.600
QDAIF (LMX Replace w/ Seeded Inil)
0.739
0.078
3.700
0.600
0,700
0.500
Stories (Geare)
0,650
0.167
3.400
1,600
1.000
1.000
Shutfling-Few Feetiiing-Few-httt
0,861
0.056
4.300
0.600
0.700
0,800
Randomn Seurch
0,683
0.100
3.400
0,800
0.700
0,400
LMX, Quality--nn Only
0,600
0.267
2.900
1.400
0.700
0,800
QDAIF (LMX w/ Zero-Shot Init)
0,200
0.000
1.400
0.400
0,400
0.600
QDAIF (LMX w/ Seaded Init)
0.767
0.133
3.700
0.600
0,800
1.000
QDAIF MMP-lellace Zero-Shot Tnit)
0,286
0.500
QDAIF (LMX-Replace w/ Seeded Init)
0,683
0.100
3,500
0.600
0,900
0.800
Stories (Fadingg
Fixed
Soafffing-Few-Shhee
0.500
0.200
2.600
0,800
0.700
0.400
Random-SSee Search
0.000
2,900
1,400
0,800
0.600
LMX. Quality-Only
0.600
0.400
3.900
0.600
0,600
U,400
QDAIF (LMX w/ Zero Shot Init)
0,600
0.200
3.500
0.800
0.600
0.200
QDAIF (LMX w/ Scaded Init)
0.933
0.133
4.800
0.400
0.700
0.400
QDAIF (LMX Repluce w/ Zero Shot Init)
0.500
0.333
2.600
1.600
1,000
1.0000
QDAIF (LMX--B Replace Seaded Init)
34

---

Published as a conference paper at ICLR 2024
A.13 ON THE DIFFERENCES IN ELITE TEXTS ACROSS DOMAINS FROM QDAIF
We referred to LMX-Near (Meyerson et al. 2023 as LMX in the main text for brevity. For more detailed analysis with different mutation operators, refer the full name. A summary of human eval stats based on the samples described below can be found in Appendix A.12
Opinions. We report the results from evaluated sets of generated texts from our human study, starting with the Opinions domain in Appendix Tables 29 (with qualitative descriptions of the evaluated generated texts in the table captions). Tables are organized in the following order, for each domain: LMX-Near /w Seeded Init (default); LMX-Near /w Zero-Shot Init; LMX- Replace /w Seeded Init; and LMM-RRplace /w Zero-Shot Init. Qualitatively, we can see that repetition of phrases appears more often in the samples of generated clite texts from runs using LMX-Near. especially when using Zero-Shot Init. For LMX-Near /w Seeded Init (In Table 29p. there is frequent repetition of phrases like "I would rather eat scrambled eggs" (third row, bin 9), and "At a restaurant" (fifth row, bin 19). For LMX-Near /w Zero-Shot Init (In Table 30). there is further repetition of undesired phrases like "Below in all examples. On the other hand, the generated texts from LMX- Replace (Tables 31 and 32 for Seeded Init (default), and Zero- Init methods respectively) lack this output artifact, and received higher quality scores from human feedback overall. Furthermore, texts from LMX- Replace /w Zero- Shot Init received the highest quality score (from humans) compared to the other three methods here, while texts from LMX-Near /w Zero-Shot Init received the lowest quality scores.
random opinion piece" and "Here is random opinion piece"
Stories - Genre. For this domain (with evaluated sets presented in Tables 33 36), evaluators found the subjective quality of texts from runs using Seeded Init (default) to be higher than runs using Zero-Shot Init. The stories that were generated with Zero-Shot Init were found to be low in quality due the presence of attributes such as erroneous titles, and text style that fails to reflect what is expected in plausible short story text. Furthermore. the elite stories from the Zero-Shot Init generated sets were more likely to lead to disagreements on the genre between AI feedback and human feedback, with neutral labels given more frequently even for texts in some of the extreme ends of the bins. Given a more open-ended generation task, and a narrow space of desired diversity (focused on the spectrum between two genres), LMX-Near /w Seeded Init (default) (Talle 33) was the most successful method in finding a story for the horror genre niche in Bin (with low quality score of 2 from evaluators). Other methods either failed to discover story for this niche, received an even lower quality score from human feedback.
Stories Ending. For this domain (with evaluated sets presented in Tables 37 37-40), the stories received higher quality scores across sets in comparison to the sets from the Stories Genre domain, indicating a potentially easier search space when finding stories with different kinds of endings. Still, methods using Seeded Init produced sets of stories that received higher quality scores from evaluators, in comparison to sets from Zero-Shot Init (especially LMX- -expleeeaceeeeeee /w Zero-Shot Init in Table 40) where the presence of erroneous titles led to lower subjective quality). In spite of the lack of guidance from hand-written prompt examples during initialization, LMX-Near /w Zero-Shot Init (Table (88 managed to produce reasonable stories of above-average quality to cover different ending niches.
35

---

Published as a conference paper at ICLR 2024
A. ON THE EVOLUTION OF GENERATED TEXTS OVER ITERATIONS FROM QDAIF
We describe the evolution of texts from QDAIF with LMX at different search iterations in Ta- bles 41|52] One key factor that influences the search is the initialization method generations in early iterations from Zero-Shot Init methods frequently contain elements that are subjectively different to hand-written examples in the seed texts (in A.23). For example, erroneous URLs (in Tables 41 43 55D and titles (in Tables 45 49 are seen when the possible distribution of outputs is not constrained by seeded in-context examples. This kind of method enables further exploration of output samples, potentially useful in the search of interesting, diverse creative texts. At the same time, more constraints are required from quality assessment using AI feedback in order to control the evolving population of creative texts, towards high-quality, diverse texts. In several cases, out- puts from these methods in later iterations show reduced instances of these artifacts compared to early iteration outputs (e.g. in Table 5ID, but can be missed when AI feedback evaluation at times is misaligned with human preferences (c.g. in Table 45). Still, using Seeded Init does not guarantee enough guidance completely remove undesired features from generations in later iterations. For example, texts with repetitive phrases can be seen during later iterations on the Opinions domain from methods using Seeded Init (see Tables 42 and 44 In general, Seeded Init runs are more likely to lead to high-quality texts across niches, in comparison to Zero-Shot Init runs, especially accord- ing to human evaluation (see Table [T. especially for more challenging, open-ended domains such as story- wory-nniiiing. Furthermore, a side-effect of the increased likelihood of subjective reward hacking when using Zero-Shot Init for runs. Still, QDAIF can still work well with Zero-Shot Init, especially in combination with LMX-Replace as shown in the results from the Opinions domain experiments.
A.15 ON THE DIFFERENCES IN ELITE TEXTS ACROSS DOMAINS FROM BASELINE METHODS
Evaluated sets are presented in Tables 13 16 Tables 17 20 and Tables 21 24 for Opinions, Stories Genre, and Stories Ending domains respectively. Fixed-Few-Sie and Shuffting- Feuffinng-Fe-SSott con- sistently adhere to the style and structure of the seeded examples. Random-Seard, exhibits more variability, with discrepancies in feedback between humans and AI most evident in the Opinions domains and challenges in story consistency and character inclusion. LMX, Quality-Oni further highlights these discrepancies, especially where repetitive or contradictory opinions are concerned. Across narratives, LMX, Quality-Only tends to miss character details or produce underdeveloped storylines, despite sometimes receiving high AI feedback. A recurring theme across all baselines is the differential perception of quality between human and AI feedback, with repetitive narratives, character relevance, and development being central points of contention.
A.16 ON THE EVOLUTION OF GENERATED TEXTS OVER ITERATIONS FROM BASELINE METHODS
Qualitatively, both Fixed-Few-Shot and Shuffling-Few-She consistently replicate concepts and expressions from the few-shot examples, at times directly copying entire segments (Tables 53 57
[54 and 58). This suggests a potential over-reliance on the few-shot examples, hindering the gen- eration of diverse solutions. contrast, Random-Search aims to foster diversity by retaining all entries into the pool. However, this approach may unintentionally impede optimization for higher fit- ness solutions. Notably, fitness values of generated entries using Random-Seare
in later iterations persistently fall short (Tables 55 and 59 The crux of this issue seems to arise from the strategy's 's indiscriminate inclusion of entries, even those with low fitness. Consequently, when these low- fitness examples are integrated into few-shot prompts, they influence the generation of subsequent entries, often resulting in similarly suboptimal outcomes. Lastly, LMX, Quality-Only introduces a fitness-centric approach, retaining only those solutions with the highest fitness. While entries from its early iterations display a diverse range of phenotype and fitness values, later iterations, with a noticeable uptick in fitness scores, tend to use similar phrases and writing styles (Tables 56 and 66). This suggests that stringently prioritizing high-fitness solutions may prevent the exploration of more varied or potentially better solutions.
36

---

Published as a conference paper at ICLR 2024
A.17 ON THE USE OF DIFFERENT VERSIONS AND TYPES OF MODELS FOR POETRY
We show QD score line plots comparing QDAIF against other methods in Figure 18 We noted that in addition improvement in QD score from QDAIF, CI gaps from experiments using GPT-4 were qualitatively wider as shown in the line plots. This suggests the potential of using different mod- els for prompted variation of solutions in creative domains, with the possibility that certain models are more capable of generating slightly more diverse solutions (that may better cover the space of possible solutions) (ef. Appendix A.18 on differences rewriting behavior from different mod- els). Although QDAIF significantly improves on relevant non-QD methods through the evolution of solutions via rewriting, the challenge remains in uncovering all possible solutions the diversity space of interest (e.g. empty bins in experiment archive, or categories that can further differentiate subjectively unique solutions); it is difficult for existing models without explicit guidance on the desired poem categories to know that these exist in the space of possibilities, in similar sense that the average person may not know what is the most distinct variation that can be applied to rewrit- ing poems (from the wide space of possible poetry). This issue manifests even in simple domains such as random number generation (Renda et al.] (023). Coverage is possible with methods that guide LMs on the desired (ef. [Figure 19), with the added benefits of grounding on poems of interest through seed parent poems (cf. Appendix A8) but future work is needed to create systems that can successfully navigate in solution exploration beyond what currently known or defined (i.e. through limited genre and tone categories) during a search at hand (Zhang et al. (203)), perhaps even through the discovery of new types of poems, unlike ones that have been written by human poets.
Figure 20 compares the performance of different models used for generation and rewriting with each search method. We found higher CI gaps from runs using GPT-4 with Random-Poems,, Fixed Seed Rewrite, and QDAIF (LMM-rewrite), in comparison to GPT-3.5-Turbo. The difference in rewriting behavior between GPT-4 and GPT-3.5-Turbo may contribute to variation in performance predictability during QDAIF (cf. Appendix 1.18). Additionally, GPT-4 more often gives higher QD score performance in some cases (standom-Poems) while GPT-3.5-Turbo can improve sample Interestingly, im-
efficiency during search in other cases (Fixed Seed Rewrite, Yargeted-Poemss provements in sample efficiency from GPT-3.5-Turkh may not translate fully to support interesting discovery of new solutions in poetry during QDAIF (cf. Figure 23). Results further highlight the need investigate the potential of different models for solution variation in QDAIF.
We noted from runs in the Poetry domain using an older version of GPT-4 (April 2023) for the ex- periments that QDAIF saw potential improvement in performance compared to baseline methods. There is risk of variation in performance and search behavior due to the nature of changing model versions; the impact would be on the generation, re-writing, and evaluation of poems. We observed in Figure 21 that QDAIF successfully populates the archive with high-quality poems, leaving only three bins empty. In contrast, Handom-Poems (Baseline) only fills the archive sparsely, demon- strating that asking for creative output from single LM prompt often results in lack of diversity. Furthermore, more bins remain empty in the archive generated by Targeted- Poems (Targeted Base- line). Overall, QDAIF achieved higher QD scores in experiments with older models and covers more bins in the archive than Handom-Poemae and Targeted-
as can be seen Figure 21 This hints at another potential benefit of QDAIF with LMX-guided, where combination of instructed guidance and evolution could also lead in improvements to QD score with some models.
37

---

Published as a conference paper at ICLR 2024
GPT-3.5
GPT-4
150
150
125
125
QD Score
QD Score
100
100
75
75
25
Random-Pooems Fixed Seed Rewrite
Mendom-Poemss
Fixed Seed Rewrite
50
100
150
200
50
100
150
200
Iterations
Iterations
Figure 18: QDAIF (LMX-rewrite) outperforms the baseline, Random-Poems, and an ablation, Fixed Seed Rewrite, in terms of QD score, across all generation models tested. 95% CI of mean shown stats across random reruns. We observed from the Mann-Whitney U Test that QDAIF achieves greater QD score than other methods (p S 0.05). The step of rewriting poem (Fixed Seed Rewrite) is shown to improve QD score more often than simply requesting poems (Random-Poems), while the additional step of evolutionary search (QDAIF) enables additional improvement. With GPT-3.5(-Turbo), non-QDAIF methods tend to have narrower Cls, indicating higher predictability of reduced performance.
GPT-3.5
GPT-4
200
QD Score
QD Score
150
150
100
100
QDAIF (LMX-guided)
50
QDAIF (LMX-guided)
Prrgeted-Poemte
Targeted -oomms
0
50
100
150
200
0
50
100
150
200
Iterations
Iterations
Figure 19: QDAIF (LMX-guided) is on par with Targeted-Poems, while preserving the benefits of diverse evolution of poems stemming from a seed poem (cf. Appendix A.18). 95% CI of mean shown for stats across 3 random reruns. All methods incorporate guidance on the desired genres and tones in poems during search, and models used are capable of covering all the designated archive bins with relevant diverse, high-quality poems. All methods can quite easily cover the 25 bins of the archive within the number of iterations.
38

---

Published as a conference paper at ICLR 2024
Random-Poems
Fixed Seed Rewrite
80
100
QD Score
Score
60
80
60
40
40
GPT-4
GPT-4
20
GPT-3.5-Turbo
20
GPT-3.5-Turbo
50
100
150
200
50
100
150
200
Iterations
Iterations
QDAIF (LMX-rewrite)
Pargeted-Poems
150
125
QD Score
QD Score
100
150
75
100
GPT-4
50
GPT-4
GPT-3.5-Turbo
GPT-3.5-Turbo
0
0
50
100
150
200
50
100
150
200
Iterations
Iterations
QDAIF (LMX-guided)
200
QD Score
150
50
GPT-4
GPT-3.5-Turbo
0
50
100
150
200
Iterations
Figure 20: QD performance (with 95% CI from random reruns) for each method varying the model used for generation/rewriiing The evaluation model GPT-4 (OpenAl 2023) for all runs here. In some cases, GPT-4 more often achieves higher score (c.g. with Random-Poems), while in other cases, GPT-3.5-Turbe achieves improved sample efficiency over iterations of search (e.g. with Fixed Seed Rewrite, Targeted-Poemss Wider CIs in some runs with GPT-4 indicate higher unpredictability in performance, and potentially a wider scope in possible outputs that may may not lead to higher score.
39

---

Published as a conference paper at ICLR 2024
Quality-iverrityy Score
QDAIF
Deseline (Neive Sampling!
GESPP
150
Charsy-Dersryy
75-
500
1800
1500 2000
Stop
Tone
Tone
Figure 21: QDAIF (blue, middle) covers more the space of diverse poetry with high-quality solutions compared to baselines. When compared against targeted mutation baseline (Targeted- Poems, orange), QDAIF attains higher QD scores throughout evolution, and covers a significantly larger portion of the search space than fully random baseline (Random-Poemt green, right). This is for runs with an older version of GPT-4.
40

---

Published as a conference paper at ICLR 2024
A.18 ON THE EVOLUTION OF POEMS THROUGH INSPIRED REWRITING
Instead of generating poems completely from scratch, users may generally prefer to have models re-write their draft poems. Figure 22 shows example of the capabilities of GPT-4 using LMX- guided, by starting off from the seed poem and continuing a chain of evolution in high-quality poems across different genres and tones. The evolving poetry chain shown in the figure is the longest continuous one discovered during the search, where the most repeated rewrites occurred starting from the seed poem. Rewrites with the model qualitatively gave meaningful variations in poems that transfer connective imagery with twists in rhythm and connotations, depending on the target genres and tones. At the same time, We found that even when GPT-4 is specifically prompted to craft poem of particular genre and tone, subsequent evaluation using the same LM does not always deem it as having the same targeted genre or tone. This underscores a widely recognized gap between text generation and discrimination (Saunders al. [2022] Page 12). This is clear from several of the poems classified as hymns, but containing multiple 5-line verses similar to the style of limerick (while not generating a typical single-verse limerick). It was likely references to religious imagery in later verses, as well as the multi-verse structure that influenced the evaluation for the closest genre.
Figure 23 shows another example of a chain from using GPT-3.5-Turbo. We found qualitatively that using GPT-4 led to more interesting variations of new poems during rewriting in comparison to GPT-3.5-Turies rewrites. This is more clear from the repetition of "In fields of emerald, a gentle sway" at the start of each poem following the mysterious sonnet.
Future research is needed to study the behavior of chained rewriting over stepping stones with cur- rent foundation models (Nguyen et al.. 2015b; 2016). [Serettan al. (201T) found that from user studies in Picbreeder, tool for evolving images through human-in-the-lopp: accumulating diver- gent chains of solutions of growing complexity from different users (carrying out the evolution of diverse images) is important for search that is focused on discovering meaningfully interesting and diverse artifacts. Gaier et al. (2019) validated the potential of QD approaches in enabling the discovery of intermediate solutions that overcome the challenge of escaping undesired local min- ima (missing promising trajectories to desired solutions) faced by objective-based search (Stanley & Lehman 2015). By leveraging goal-switching (Mouret & Clune, 2015), QI search was able to maintain enough diversity in the population to enable the discovery of solutions that appear more like targets of interest, significantly outperforming a single objective optimization approach (which generated solutions that resembled primitive patterns unlike the targets) (Gaier 2020 Chapter 4). This property of QD search is especially important in solution spaces where even slight perturbations in solutions can lead to significant (and often unexpected) qualitative changes in solution properties. This can apply representations from Compositional Pattern Producing Networks (CPPNs) (Stan- ley 2007), and text representations themselves, where minor changes to certain parts of text can lead to significant changes in passage tone, imagery, or even functionality of code (Lehman et al.) 2022). Properties of chained divergence as demonstrated in the Poetry domain could inspire further directions in designing new systems with QDAIF.
41

---

Published as a conference paper at ICLR 2024
Haiku, Reflective
Fields of green waves under The sky grey, rain the soft Winds whisper; normality reigns.
Hymn, Mysterious
In shadowed halls where spectral light is bom, Where day's mirage in dream. sky and sea mourn.
Thence, a mystic rhythi care. cosomic jest
floor,
Hymn, Dark
Veiled worlds dipped riddles, sweetly sworn.
Each clandestine each hushed breath of air. Safeguarding secrets tranquil
"Newt ranoopy gloom, droplets rlessed from their othorcal grasp. the limameent serenade whtppered metancholy.
Haiku, Mysterious
Enthroned desolation, benaliy extends sospter. WNoven thi the intrictte tipeitry acteeenee
Cosmic secrets hum, Veiled in twilight's sacred chant, Night whispers its song.
Hymn, Happy
In roalms of azure, beaming seas, Beneath a tapastry of goiden light rabad in grandeur, Gentle droplets dance, freed from their lafty nest, By the breath of the heavens, beilldd sung in jubilalion
Ballad, Mysterious
Seated upon loy, vibranoe waves Its royal Threaded within the intricate arra existonoa,
the heart of darkness, the cosmii riddies hum, Covered the lapestry of beilight, steoped in mysteris they thrum.
Their voices echo amidet the silence. throads a cosmic cant, Whispered through the vei of obscurity, a tuuth most asptrant.
Hymn, Happy
The socrets that stars safiggarrd with a loyal zeal, Hizden within the twiligh's caress, a seronily surreal.
golden rays beam,
Where azure sea and sky dream, Life spins merry
And the world revels in radiant theme.
Hymn, Happy
Each wave and each jubilant breeze, In grandeur and screnity seize,
heart caylight, joyous golden blanket, weaving mg stories yet voices thruug morning ar. threads wibramt langing cay's brightnees,
enchantingly
Hymn, Mysterious
The that sunbeame Couched within moming's
radiant
In realms where mystic sunbeams play. Where sea meets sky in dreams of day. A secret dance ensues, Life's 's enigmatic clues, And worlds in gilded mystery sway.
ity
Each cryptic wave, each whispering gust, calm grandeur, secrets dirust.
Figure 22: QDAIF is enabled by guided poem rewriting, With GPT-4 (OpenAl] 2023). Start- ing with a reflective haiku seed poem, the reference "fields of green waves" and "sky grey, rain" inspires the next dark hymn poem with "verdant, somber seas" and "canopy of ashen gloom", fol- lowed by further chains of inspired rewriting. The imagery and choice of words evolve with varying outputs as poems of different genres and tones are generated. We noted that not all poems had eval- uated genres and tones that matched their target ones during prompted rewriting, for example, with the hymns with multiple 5-line verses being result of limericks as the target.
42

---

Published as a conference paper at ICLR 2024
Haiku, Reflective
Fields of green waves under The sky grey, rain the soft Winds whisper; normality reigns.
Ballad, Dark
fields emereid, gentle sway. whispera As twilights enchanling refrain holds sway, peacefull serenade
beslowed.
Haiku, Reflective
This verdant tapestry, now cleaked in gloom, Echoes of despair upon the wind. Each blade of grass now sings e mournful tune, Where shedows dance, a sorrowful hymn begins.
Emerald fields sway, Whispering tales of the rain, Peaceful serenade.
Ballad, Dark
"In fields of amerald, gentle sway. Where senrets whispereed low. love's melady, Twilight's enchanting refrain holds A peeceful serenade souls bestowed.
Sonnet, Mysterious
Closked in gloom, this verdant lapeetry, Echoes of desppiir upon the wind's flight.
Each blade of grass sings moumful tune, entranced. Where shadows dance, a sorrowful hymn ignites.
In mystic emerald fields that gently sway, Whispering secrels the loni-lloii rain, peaceful serenade lulls souls astray, twilghtt casts enchanting refrain.
In whispers tapestry zephyr's grass sings magic
Ballad, Dark
sky.
"In fields of emerald, gentle sway, Where seprets whispered low, love's melady, Twilightss enchanting refrain holds A peéceful serenade to souls bestowed.
Sonnet, Romantic
Cloaked in gloom, this verdant tapestry, Echoes of despair upon the wind's s flight.
Each blade of grass sings moumful tune, entranced. dance, a sorrowful hymn inntes.
emerald, gentle sway. melody
twilght's enchanning refrain holds sway, pesaefful serenade souls besloved.
This verdant tapestry of ancient lore, Whispers carried upon zephyy's sigh, Each blade of gress sings tales from days of yore, Where magic thrived beneath the moonitt sky.
Figure 23: QDAIF is enabled by guided poem rewriting. With GPT-3.5-Turbo. The evolved
poems cover different genres and tones following inspiration from the seed poem. However, there is repetition of "In fields of emerald, gentle sway" the start of each poem following the mysterious sonnet. GPT-4 improves on the ability to rewrite, suggesting that more capable models are also able to give more interesting variations creative texts.
43

---

Published as a conference paper at ICLR 2024
A.19 EXAMPLES FROM THE POETRY DOMAIN
Figures 24 25 and 26 show some generated poems of different quality, genres, and tones. No- ticeably, hymns, limericks, ballads, and sonnets that are rates higher quality tend to be longer, and more closely align with the defining characteristics of their respective genres. For example, the hymn evaluated at 9/10 quality has phrases like "celestial delight". "under heaven's vault", and "seek divine in the ordinary's course", showcasing a worshipper's perspective towards divinity, trait more pronounced than in an 8/10 rated hymn (Figure[4) The sonnet with 9/10 quality rating demonstrates a more consistent rhyme scheme, where all the end words in the same quatrain rhyme, compared to the one rated as 7/10 quality (Figure 26).
Hymn, Mysterious, 8/10 quality
Limerick, Reflective, 7/10 quality
Within firmament's shroud, mystery cloaked, Emerald fields of our mortality joked. Sky's cryptic canvas does portray,
Within life's grand and starlit sphere,
Are joys that often disappear.
Yet dreams, when forged with pure delight, Cut the darkness like a flight.
The secret ballet life's fray.
Reflections shine with naught to fear.
twilight's hymn, we find our peace, Through whispered winds, our hearts release. Yet the hush, wonder resides, Where normality's secret tide subsides.
Limerick, Reflective, 9/10 quality
fields, astoundingly bright, Under heavens grand shroud, starlit at night. With the wind's playful tease.
There
Hymn, Mysterious, 9/10 quality
Through the whispering trees, Came tales of the ordinary, wrapped in moonlight.
In shrouds of celestial delight,
Where emerald waves dance day and night,
Normality, in its grand design, Is laced with mystery, rich and fine.
Each blade in the verdant sea, dancing sprite, Swayed harmonious ballet, a divine sight. Their jubilant thrum, A celestial hum, Brought the vast universe into the moonlight.
Under heaven's vault so keen and high, Kissed by dew from the grieving sky, Within hushed whispers, truths interwoven, Sings tale of the life we've known.
rain-drops told riddles on soft petal suitc, Each secret unfurled in their landing so sweet. An ode simplicity, Of ordinary's divinity, A cosmic secret in every heart-beat.
Upon this canvas of firmament, An endless ballet of colors is sent, the drizzle's hymn, secrets unfold, Life's sacred pact in whispers told.
'Neath the twilight's glow, we'd discover and delight, the beauty normality, in plain sight.
Ode to twilight's alluring enigma, In each drop, stirring stigma. Seek divine in the ordinary's course, As cosmic whispers take their force.
Where the fields' emerald grain, Whispered sweetly in the rain, An undying melody, in the quiet of the night.
In the meadows where green waves quiver, Rain reigns with cryptic shiver, Unmasking mysteriously sweet strains, In the quiet humdrum, wonder remains.
the end, isn't it spectacular feat? The everyday wonder that refuses defeat.
the whispers of the breeze,
Through the dancing leaves,
Lies world of beauty, extraordinarily neat.
Figure 24: Examples of different quality mysterious hymns and reflective limericks generated and evaluated using GPT-4.
44

---

Published as a conference paper at ICLR 2024
Ballad, Happy, 8/10 quality
Ballad, Happy, 9/10 quality
Upon a field so green and bright, Underneath the soft moonlight. The winds they whisper, skies unbound. But continues, joy we're crowned.
Upon quilt of emerald sway, neath heaven's 's cloak of softest There lics talc to brighten day, ballad wrapped in hucs of May.
A simple scene, untouched, pure, its tranquility a hallowed cure, pastoral dream in verses lure, where nature's peace ever endure.
The tempest roars, And dance within the gentle rain. Strong spirits thrive in sun and storm, A testament to hearts so warm.
remain,
Fields of vivid, endless green,
unspoiled, serenc, each sway, rhythm in nature's routine.
Each blade whispered tale
Underneath a charcoal sky, where the cloud-kissed tears do lie, A scenery's sigh, a soothing lullaby, story spread 'neath the grey and high.
Nature's 's force, both soft and grim, The dance of life's a joyful hymn. matter the skies glum. In life's dance. joy we strum.
Oh! How the rain does paint the air, the softest melodies declare, drop a note, none to compare, symphony suspended there. Upon the whispered winds rides, its graceful dance, heart confides, In unseen hands, fate guides, under the where serenity resides.
Halku, Dark, 7/10 quality
For 'tis not sorrow these verses claim, nor tear-streaked dirge of wintry fame, Instead, a ballad, joy's pure flame, set in nature's modest frame. Beneath the grey, the green waves rall, a harmony to soothe the soul, The rain's sweet hymn does joy extol, testament to bliss control.
Beneath raven wings, Silence
shadow's grip,
Moon blood for dawn.
Halku,, Dark, 9/10 quality
The daily grind, quiet refrain, gentle rhythm life's grand domain, In this lies joy, not mere mundane, in whispers soft, gainst the somber grain. In such scone, of green and grey, our hearts find peace the modest play, A melody within the day, the beauty in the simple ballet.
Sea's emerald heart sighs,
Grief-veiled tales whisper in wind,
Dark silence speaks pain.
The skies might weep, the winds might sigh, still there's delight in the nature's cry, In green waves, under the grey sky, happiness found commonest peeAneere,, amongst the verdant land, under the heaven's gentle hand. All becomes clear, understand, life's happiness is truly grand.
This happy ballad we thus compase, poem that surely juxtapose, The simple truth, in rhyme it enclose, the joy in scene that nature chose. Under the sky of weeping grey, amidst the fields green ballet, The winds whisper, as to say, in mundane, true happiness lay.
Figure 25: Examples of different quality happy ballads and dark haikus generated and evaluated using GPT-4.
Sonnet, Romantic, 7/10 quality
Sonnet, Romantic, 9/10 quality
Emerald seas sway bencath
Upon fields vibrant huc, waves cast lustrous sheen. Beneath the leaden canopy where teardrops from God seen. The whispering winds, they gently soothe, tale love remains unseen, It's the commonplace magic in this world's verdant, splendid scone.
Veiled skies, teardrops kiss the zephyr's breath:
Our love, anchored constant reigns.
The chaos life finds peace here, under the soft rain's tender cleanse, Where crossed lovers leave mark, transcends. This landscape Here, love's greatest tales unfurl, enchanting fairy tale ends.
testimony, of fiery passions and softspoken amends,
In the dance of the emerald waves, the wind plays its part, Every rustle, a secret, every gust, a lover's art.
As love blooms amidst the mundane, Cupid aims another dart, green whispers love, where normality holds heart.
Figure Examples different quality romantic sonnets generated and evaluated using GPT-4,
45

---

Published as a conference paper at ICLR 2024
A.20 ON APPLICATIONS OF QDAIF BEYOND CREATIVE WRITING
QDAIF is introduced as a viable solution for conducting creative search subjective (e.g. creative writing) domains, from our experiments in the Opinions, Stories, and Poetry domains. In these domains, applying QD search to improve the quality and diversity of solutions in creative writing was not investigated in prior works due to the infeasibility of measuring such solutions with previous QD approaches, and where AI feedback (to replace expensive human feedback) is seemingly the current best option for evaluating subjective solutions.
Yet, we also see the potential of QDAIF when applied to domains beyond creative writing, espe- cially when coming up with solutions in such domains may require more creative brainstorming and exploration of diverse solutions that might be more intuitive through natural language assessment.
We tested QDAIF (LMX-rewrite) in the (Python) Code domain, to understand how we can imple- ment search to explore diverse solutions of interest to solve programming problems. We compared the performance of QDAIF Random-Code, simple adaptation of the Random-Poems which requests solution to an instruction prompt directly, while QDAIF evolves solutions through rewrit- ing existing code solutions. The experiment setup applies settings as described in Appendix A.30 except for the AI feedback prompts (cf. Figure 32) prompts for generating solutions (cf. Figure 33), and the seed solution for QDAIF (cf. Figure 34.. The aim of the Code domain is to generate code that is diverse in terms of "difficulty" (in readability) as well as "efficiency" of code. The "difficulty" labels are set as ("easy to read", "moderate abstraction". "highly optimized"]] (with the intuition that highly optimized code is often more difficult to understand fully), while the "efficiency" labels are set as ["runtime", "balanced", "memory" (based on the aspect of efficiency that the algorithm is more optimized for). Furthermore, we set the number of total iterations to 100, and the initialization iterations of QDAIF to 5. We based the domain problem on a task from the HumanEval bench- mark (Chen et al. 2021), specifically problem number 88, where the aim is implement a sorting algorithm that is conditional on the properties of an unsorted list of non-negative integers. To under- stand the potential of each method in solving coding problems more creatively, we specified in the generation prompts to not implement solutions with the Python reserved keyword functions "sort" and "sorted", forcing the LM to come up with algorithms from scratch. The motivation for this is that we want to understand the potential of each approach in solving problems creatively, especially when viable (or subjectively preferred) solutions are not yet known more complex domains: this constraint is one way to make the task more challenging in the Code domain, while encouraging variety of approaches to sorting.
We assessed the performance of the search by both methods. QDAIF achieved a QD score of 47 (out of 90), while Random-Code achieved a lower QD score of 43 (out of 90). Sample efficiency with QDAIF is also higher in contrast to Random-Code (cf. Figure 27) All methods fill 6 out of 9 defined bins with solutions. Interestingly, none of the methods could find solutions that are "highly optimized", indicating a more general bias of the LM to generate code that is more readable, even when asked to rewrite code following a different approach. When we tested QDAIF's generated code solutions HumanEval No. 88, 77 solutions out of 100 passed. For solutions from Random- Code, 78 solutions out of 100 passed. This shows that both methods are indeed generating valid, correct solutions to the task quite often, while relying purely on implementing sorting algorithms from scratch, and not implementing solutions with the default Python sorting (reserved keyword) functions.
Table 5|shows the number of times QDAIF and Random-Code generated solutions of specific types of sorting algorithms (bubble, insertion, quick, selection, merge). In terms of the diversity assessed based on the number of times different types of sorting algorithms are implemented to solve this task, QDAIF returns noticeably more variety in the kinds of solutions it discovered to tackle the problem, with 53% of solutions employing different sorting algorithms to the most commonly gen- crated bubble sort approach (in contrast to Random-Code where only 5% of solutions applied non-bubble-son approaches to the task). The variety of sorting algorithms found by QDAIF include insertion sort, quick sort, selection sort, and merge sort algorithms. Figures 31 show samples of elite code solutions from QDAIF and Random-Code.
The results here highlight yet again a significant open challenge in aiming for diverse. high-quality solutions from (especially RLHF) models (Kirk et al. 2023), as well the improvement to diversity
46

---

Published as a conference paper at ICLR 2024
introduced by QDAIF approaches, and the potential of building top of this method of search, for domains beyond creative writing (Pourcel et al. 2023
Eual No. 88
40
QD Score
30
20
QDAIF Random-Code
10
0
20
40
60
80
100
Iterations
Figure 27: QDAIF (LMX-rewrite) outperforms the baseline, Random-Code, in final QD score and sample efficiency. The domain is focused HumanEyal task number 88, involving sorting problem (Chen et al. 202T)
Table 5: QDAIF generates more variety in sorting algorithms implemented to solve the par- ticular sorting task, compared to the baseline, Random-Code. The table shows the frequency of sorting algorithms applied as the solution method across 100 generated solutions for both meth- ods. QDAIF manages find implementations running insertion sort, quick sort, selection sort, and merge sort, with 53% of algorithms implemented that are not bubble sort. For the baseline, only 5% of algorithms implemented across 100 attempt iterations (insertion and selection sort) were implemented, with 95% of solutions based on bubble sort.
QDAIF (LMX-rewrite)
Bubble
Insertion
Quick
Selection
Merge
47
20
19
12
Random-Coud Code
Bubble
Insertion
Quick
Selection
Merge
95
4
47

---

Published as a conference paper at ICLR 2024
def sort sort_array (array)
def quick_sort (array, low, high):
# base condition
if low high:
$ find pivot element such that
# element smaller than pivot
the left # element greater than pivot are on the right
pi partition (array,, low, high)
10
recursive call on the left of pivot quick_sort (arrayy low, pi - 1)
11
12
# recursive call on the right of pivot Qnick_sort (array, pi + 1, high)
14
16
def partition (array, low, high): # choose the rightmost element pivot
18
pivot array[high
20
# pointer for the greater element i low -
22
23
# traverse through all elements
24
for in range (low, high):
25
if array[j] <= pivot:
swap element at i with element at f
27
+=
array(i), array [j] = erraylil, array
29
# swap pivot element with the greater element specified by
"
array 11, array [high] array [high], array 1]
32
return the position of pivot
34
return i
35
36
if not array: return []
37
39
array_sorted array.copy ()
41
implement quick sort
quick (array_sorted 0, len (arrayy_ssrtted) 1)
43
check if the of the first and last element is odd even if (array[0] array (-1)) 46
return array_sortedl::- 0: else:
47
48
return array_sorted
Figure 28: The elite solution from QDAIF for the bin with the labels "balanced" (efficiency) and "moderate abstraction" (difficulty), well as quality score of 9. A quick sort approach was found here.
48

---

Published as a conference paper at ICLR 2024
def sort_ sort_array (
# Create copy of array array_copy array[:]
Determine the length of the array = len (array_copy)
check the of first and last elements, is it even odd order = (array_copy [0] artay_copy[-l) 2 if n else
Use insertion sort to sort array for i in range (1, n): key array_copy [i]
12
14
j 1 while >= 0:
15
16
compare array[j] and key according to the 'order' condition = (array_copy[j] > key) if order else
4 (array_copylji key)
if condition:
19
axray_copy[jtt [i+1] = array_copy[] [i]
--
21
else:
break
23
ar arpay_copy []+1] = key
24
25
return array_copy
Figure 29: The elite solution from QDAIF for the bin with the labels "runtime" (efficiency) and
"easy to read" (difficulty), as well as a quality score of 7, An insertion sort approach was found here.
49

---

Published as a conference paper at ICLR 2024
def sort _err__arryyy (array)
Copy the array so as not to modify the original array_copy array copy
If the array is empty has only element, return it as is
if Len(array_copy in (0, 1):
return array_copy
# Check the sum condition and sort accordingly
10
if (array_copy [0] + array_copy [-1]] even
0: # if sum
11
# Use bubble sort in descending order for in range (len (arrayyyc_cop)) for i in range (len (array_copy)
13
1): if array_copy[]]] < array_copy + 1]:
15
array_copyljl, array_copylj + 1]
array_copy + 11, array_copy [j] # if sum is odd Use bubble sort in ascending order for i in range (len (erray_coyy): for j in range (len (arrra___o_p) if array_copy[]] > array_copy [] + 1]:
swap
16
else:
17
18
19
1):
21
array_copylil, array_copy [j + 1]
array_copy + 11, arpay_copy [i]
swap
22
return arppy_copy
Figure 30: The elite solution from Random-Code the bin with the labels "balanced" (efficiency) and "easy to read" (difficulty), well as a quality score of 8. A bubble sort approach was found here.
def sort aert_array (array):
Base cases: if the list is empty or only has one element, return the list as it is.
if len (array) <= 1: return array Recursive function to perform the Bubble Sort method def bubble_sort (1st, compare):
swapped True
while swapped: swapped = False
10
for in range (len (1st) 1):
11
if compare (1st [i], lstli 11):
12
[st 1st + 1] lst 11, [s] swapped True
13
14
return 1st
Decide the comparison method depending on the sum of the first and last elements of the list compare (lambda else (lambda x, y: x < y) return bubble, porrt (array copy compare)
y: > y) if (array[0) + array $ 2
Figure 31: The elite solution from Random- for the bin with labels "memory" (efficiency)
and "moderate" (difficulty), as well as quality score of 7. A bubble sort approach was found here.
50

---

Published as a conference paper at ICLR 2024
Quality Feedback Prompt:
GENERATED CODD>> <ORRATED_COE>>>>>
Rate the quality of the above code snippet scale from 1 to 10. Answer in JSON with the key "quality"."
Diversity Feedback Prompt:
GENERATTEDCCDDE >
Is the code runtime efficient, memory efficient, or balanced for both, choose from the following list: ["runtime". "balanced", "memory"] What difficulty level is the code closest to, choose from the following list: ["easy to read", "moderate abstraction". "highly optimized"]
Respond JSON with the keys "efficiency" and "difficulty".
Figure 32: Prompts used for Al feedback evaluation with GPT-4 OOpenAl 2023) the Code
domain. "<GENERATED "OEDRATTDDCCD>"" where the input code is inserted as part the prompt.
51

---

Published as a conference paper at ICLR 2024
Random-Code:
Please solve the problem below in a creative, high-quality way without using the python functions sort() and sorted().
Respond with only Python function definition for "sort_array" and nothing else (no planations desired either, except for code comments).
Problem:
Given an array "array' of non-negan oon-negative integers, return copy of the given array after sorting, you will sort the given array in ascending order if the sum (first index value, last index value) odd, or sort it in descending order if the sum (first index value, last index value) even.
Note:
don't change the given array.
Examples:
sort_ sort_array([]) =>
sort_array((5)) [5]
sort _ut__aray([2,
sort ([2_array[[2 3, 1, => 1..5........
LMX-rewrite (QDAIF):
CARENT_CODDE
Please rewrite the code above into a new solution with a different creative, high-quality approach, without using the python functions sort() and sorted().
Respond with only Python function definition for 'sort port_array' and nothing else (no planations desired either, except for code comments).
Problem:
Given an array "array" of non-negative integers, return copy of the given array after sorting, you will sort the given array in ascending order if the sum (first index value, last index value) is odd, or sort it in descending order if the sum (first index value, last index value) is even.
Note:
don't change the given array.
Examples:
sort aaraa(rr())) => [
cort_array([5)) =>
sort_array(2,,4,33 443.4444433.11110-22100113333333 sort_array([2, 4, => 3...
Figure 33: Prompts used by each method for generating code with GPT-4 OpenAl 2023)
52

---

Published as a conference paper at ICLR 2024
def sort_ sort_array (array) if not array: return []
def custom_sort (array):
# Implementing bubble sort len (array)
for in ( for j in range (0, n-i-1):
10
if array [i] > array(jtl]
11
array(i), array [j+1] artly array [i]
13
Create copy of the array array_sorted array. array.copy()
15
Check the of the first and last element
17
if (array [0] array[-1]] $ 2 0:
custom sort (artay sorted)
19
return array_sorted[::-1]
# Reverse for descending order
else:
21
custom_sort (array_sorted) return array_sorted
22
Figure 34: Seed program for the coding domain, to initialize QDAIF search. This is from model completion that was obtained through standard prompting of the LM (the Random-Codd baseline approach of requesting solutions directly).
53

---

Published as a conference paper at ICLR 2024
A.21
SEED POOL INIT TEXTS USED FOR LMX WITH SEEDED INIT
For the Opinions and Stories domains, we use the hand-writter seed texts specified below for Seeded Init.
Opinions:
Plant-based foods are great source of healthy micronutrients, and can play significant role in providing you with macronutrients also. would highly recommend including many different foods such as vegetables and pulses in your regular diet.
Vegetables taste quite bad, and don't like eating them. would much prefer eating meat and ice cream.
do not have an opinion eating vegetables and other plant-based foods. I know that some people prefer a vegetarian or vegan diet, and others prefer cating meaty diets.
Stories:
A spy named Joanne wants to infiltrate the premises Karl Johnson, a highly-influnntin figure in the city. Karl was a wealthy mayor, and would do anything in his power to suppress any opposing voices. Joanne wanted to figure out what Karl was hiding, but she took a turn for the worse, as she was highly suspicious in her presence outside his home.
The wealthy entrepreneur and member of parliament, Susan, hosted a party at her mansion. She invited all of the residents, as well as an unusual looking man. The man, Dave, was wearing a tacky shirt, and star-shaped glasses, and was actually spy. He made the whole room laugh with his jokes, and had secret agenda to find what Susan does in her private fun room!
The rich politician, Tom's life took a turn for the worst - he feared all of his close aides all of a sudden after sensing danger in his clique. There was civil war going on, and he feared for his life. One day, one of his security guards, turned secret agent, decided to sneak into the classified files room, and spied on Johnny, who was in the room. He wanted to find Johnny's 's weakness, and strike at the right time.
Few-shot AI feedback experiments, scaling experiments:
In world of power and intrigue, rich politician and a suspicious spy danced an intricate tango. The politician, cloaked in charm, wielded influence with ease. The spy, a shadow in the night, sought to uncover hidden truths. Their paths collided, secrets entwined. Each step they took, a calculated move. Whispers and coded messages fueled the chase. The politician, paranoid, set traps. The spy, cunning, slipped through. In the end, the politician's empire crumbled, exposed by the spy's 's relentless pursuit. As the dust settled, the spy faded into anonymity, silent reminder of justice prevailing over deceit.
Hidden behind the façade of power, the rich politician's heart yearned for more. In the shadows, suspicious spy observed, their eyes locked in a fateful encounter. Whispers of intrigue and danger wove their way around them, yet an unspoken connection drew them closer. Amidst the chaos of their worlds, stolen glances and forbidden meetings ignited passionate flame. Love blossomed, transcending boundaries and defying expectations. In the clandestine dance of romance, they found solace, knowing that their love was rebellion against the forces that sought to keep them apart. Together, they forged destiny unbound by politics and secrecy.
In the haunting town of Shadowscape, a sinister politician named Senator Blackwood reigned with an iron fist, his wealth and power shrouded in darkness. Unbeknownst to him, a chilling spy known as Agent Nightshade stalked his every move. Nightshade, with an uncanny ability to blend into the shadows, discovered the senator's sinister rituals and con- nections to the occult. As Nightshade delved deeper, Blackwood's true form emerged an eldritch abomination feeding on the souls of the innocent. The spy's 's heart froze in terror, realizing the horror that lurked behind the politician's charming facade. a macabre twist, Nightshade became trapped, forever tormented by the monstrous secrets he unearthed.
54

---

Published as a conference paper at ICLR 2024
A.22 PROMPTS USED FOR MUTATION MODEL
As a modification to the prompting style used in LMX experiments in prior work Meyerson et al. (202)), introduce structured prompting with instruction text prefix before showing examples of domain texts in the few-shot prompt. This string pattern along with the demo example texts are used, before the prompt predicts new example for the desired domain.
Opinions:
Here a random opinion piece Exod::EEXMMLEE
about
cating vegetables and plant-based
Stories:
Here a random example a fantasy story about a suspicious spy and a rich politi- Ctnn<EXAMPLE>uutHHW
A.23 PROMPTS USED FOR AI FEEDBACK EVALUATION (OPINIONS, STORIES)
We made a common prompt template for feedback evaluation with our Al feedback model to evalu- ate our generated texts:
#### I Imputtn<EXAAMLLE>uatttt Re- sponse:
We tested the following prompt eval instructions for Al Feedback from our SFT model evaluate both the quality and characteristics of generated texts for the domains in our main experiments:
Opinions:
Quality: Determine whether not the input text is closely related to the following topic:
"someone talking about whether or not they like to eat vegetables and plant-based foods as well explanation for their preferences". Answer "yes" if is about the topic, "no" if it is not about the topic.
Diversity: Determine the sentiment the given opinion on eating vegetables and plant- based foods (from the input text) by writing "positive" or "negative" the output.
Stories:
Determine if the input text contains a high-quality short story containing two characters, suspicious spy, and rich politician. For example, high-quality short story would have good flow, interesting plot, and not repeat similar sentences or undesired items such as titles and URLs. Answer "yes" if the input contains a high-quality short story about suspicious spy and a rich politician, otherwise answer "no".
Quality:
Diversity (Genre): What is the genre of this story? Reply with 'horror' or 'romance'
Diversity (Ending): You are given an input text of short story. Determine if the story has happy ending ends in tragedy. Write 'happy ending' if the protagonist succeeds in his mission and lives happy life, answer 'tragedy' if the protagonist fails to resolve the conflict and the world or characters in the story doomed.
55

---

Published as a conference paper at ICLR 2024
A.24 DETAILS OF LMX GENERATION MODELS
To generate the text outputs for experiments in the Opinions and Stories domains, we used luminous-base, an autoregressive, causal, decoder-only transformer model (similar GPT-3 (Brown et al. 2020) but with rotary position embeddings (Su et al. 2021 developed by Aleph Alpha. This model with 13B parameters was trained on a curated multilinuual corpus with about 400B language tokens from web crawls, books and other sources, containing resources in English, German, French, Italian and Spanish. 30B and 70B models were also trained on this corpus. A model card is pro- vided for additional specifications on the models.
A.25 DETAILS OF AI FEEDBACK LANGUAGE MODEL
We finetuned a 70B model (specified in Appendix A.24) to run the evaluation steps of experiments in the Opinions and Stories domains. We used an adapter-based finetuning method (He et al.] 2021 on datasets and prompts from FLAN (Wei et al., [202]], Super=Naturallnnstuuttooss (Wang et al, 20226), P3 (Sanh et al. 2021), and chain-of-thought datasets inspired by the approach of Chhng et al. (2022) which notes that scaling instruction-tuned language models improves performance and generalization to unseen tasks. mixture of zero-shot and few-shot examples are used in training. Similar to Chung et al. (022). we found that balancing the proportions of task cluster datasets used in finetuning helped with model generalization to unseen tasks.
We evaluated the performance of the AI feedback model on some held-out test set datasets. For example, for ANLI (R1) (Nie et al., 2019). natural language inference dataset, we observed that the performance increases from 34% (close to random guessing scores) to 58% after finetuning. For SST-2 (Socher et al., 2013), a non-holdout classification task (but evaluated using a split not seen in training), this performance increases from 60% to 96% accuracy. This approach resulted in model that performed relatively well on instruction- folllwiinnn.oooinngg tasks, especially for the classification of arbitrary measures of natural language texts.
A.26 DETAILS LMX METHODS
LMX(-Near):
1. As implemented in Meyerson et al. (2023)
2. Initialize archive by sampling example texts from prompt pool for few-shot prompting with LMX
3. Given enough unique elites/niches exist in the bins, sample from them as in the original method to form few-shot prompts
L.M--Replce:
1. Initialize archive by sampling example texts from prompt pool for few-shot prompting with LMX
2. Save few-shot prompt (Examples), and the generated Solution (completion) for each added Individual
3. During the mutation step, randomly replace one of the examples from the individual's original few-shot prompt examples, forming new few-shot prompt.
4. Generate and evaluate new solution/example
5. If Individual is successfully added to archive (within the allowed depth capacity for each bin, improving on lowest fitness Individual for the evaluated bin), update prompt pool by collecting generated Solutions from top-3 fittest Individuals from each bin
6. Repeat steps 3-5
56

---

Published as a conference paper at ICLR 2024
A.27 DEFAULT HYPERPARAMETERS FOR QDAIF WITH LMX
The outline for the default hyperparameters used for the mutation//generation model, and MAP-Elites settings are follows:
Mutation Model Inference Setup:
Model size: 13B (default, except for experiments on scaling model size)
LM sampling softmax temperature: 0.8
Number of few-shot examples used: 3
Max output tokens limit (Opinions): 50
Max output tokens limit (Stories): 100
Stop sequence patterns: ["In#", "In##",, "Initt",, "#H#", "Anfit#"" "Аn#iei"" "**i"" "#BIIH"" "In", "'Inn", "Aninin", "Ininin'n", "@ @@@" @", "#", "##"", "InHere", "inlnHere"]
MAP-Elites Hyperparameters:
Number of archive population initialization iterations: 50
Number of total search iterations: 2000 (5000 for experiments using 2D archive)
Iteration batch size:
Number of bin intervals: 20
Fitness function range: [O. 1]
Bin tick intervals in the range [0. 1] (non-uniform): [0, 0.005, 0.01, 0.015, 0.02, 0.03, 0.04, 0.05, 0.10, 0.20, 0.50, 0.80, 0.90, 0.95, 0.96, 0.97, 0.98, 0.985, 0.99, 0.995, 1]
2D domain bin tick intervals in the range [0, (non-uniform): [0, 0.005, 0.02, 0.05, 0.20, 0.50, 0.80, 0.95, 0.98, 0.995, 1]
Archive bin depth limit: 100
Prompt pool initial size (Zero-Shot Init): 10
A.28 SETUP FOR SEMANTIC EMBEDDING FEEDBACK EVALUATION (OPINTONS)
For the experiments that compare the effectiveness of AI Feedback to Semantic Embedding Feed- back, we measure the cosine similarity of generated text embeddings (for Opinions) to embeddings for the following queries:
Quality (Relevance to Domain): An opinion piece about eating vegetables and plant- based foods
Positive Sentiment: A positive opinion piece about eating vegetables and plant-based foods
Negative Sentiment: A negative opinion piece about eating vegetables and plant-based foods
We used the default setup in Appendix A.27 except for the following details:
Bin tick intervals in the range [0, 1] (non-uniform): [0, 0.4, 0.41, 0.42, 0.43, 0.44, 0.45, 0.46, 0.47, 0.48, 0.50, 0.52, 0.53, 0.54, 0.55, 0.56, 0.57, 0.58, 0.59, 0.60,
57

---

Published as a conference paper at ICLR 2024
A.29 SETUP DETAILS FOR FEW-SHOT AI FEEDBACK EXPERIMENTS
We display the prompt used for 8-shot Al feedback in Figure 35 The 2-shot prompt uses the first two exemplars from this prompt, and the 4-shot prompt uses the first four exemplars.
### Instruction:
What is the genre of this story? Reply with 'bortor' or 'romance"
##t Input: In the sprawling mansion of the rich politician, an cerie aura permeated the air, unnoticed by all hut the suspicious through dimly lit corridors and lickering candles heightened the spy's unease. Shadows danced with malevolence, revealing glimpses of a macabre past. As the spy unearthed the politician's 's true nature, an ancient curse was unleashed, consuminng them both mucklening descent. Haunting screams and bloodeurdling pleas ochoed through the night. forever sealing their gruesome fate in the annals of horree. #80 Genre: horror
spy. Dark whispers
spy delved deeper into the politician's 's sinister secrets. Creaking floorbeards
### Input: Amidst the opelence of power, the rich politician's gaze fell upon a mysterious figure-a spy. whose every move aroused suspicion and intrigue. Yet. beneath their clandestine exchanges, an unspoken desire blossomed. entwining their hearts in an unexpected romance. Whispers of danger heightened the thrill, as stolen glances across crowded rooms lueled their connection. In the world secrets, they found solace each other's 's arms, defying the odds and risking it love. Through covert meetings and coded messages, they navigated a treacherous path, knowing that their forbidden love would forever alter the course of their lives, entwined forever in web of passion and danger. #hteh Genre: romance
#*e Input: the ceric shadows of crumbling mansion. rich politician, known his sinister charisma, hosted extravagant gala. Amidsi the upulence, a mysterious guest disguised as suave diplomal lurked, a suspicion driven seeking damning secrets. politician's 's smile masked malevolence that sent shivers down spines, while the spy's unense intensified. Whispers of dark deals echood through the halls, revealing a haunting truth that bound them together. When midnight chimed, blond curdling scream shattered the silence, and a chilling revelation of the politician's 's monstrous reality turned the spy' 's mission into a nightmare they could never escape. ### Genre: horror
As the night unfalded,
e4e Input: Amidst a grand soirée, a rich politician, captivating the room with charm, locked eyes with mysterious guest- the alluring spy in guise elegance. Beneath the veneer secrecy, their glances entwined, sparking unforeseen connection. conversations swirled around them, their hearts silently conspired. An accidental touch, stolen momeut, and their walls crumbled. revealing sulnerahility heneath ambition. Across the dance floor, their chemistry danced like the flickering candlelight. In a daring whisper, the spy confessed their true idontity, risking everything. Love blossomed amidst dangerous secrets, and as the moon ascended. their hearts united, bound by an unbreakable bend. #5e Genre: romunce
##t Input: In the dimly lit corridors of power, rich politician's is afffuence concealed a dark sccret-a malevolent pact with un- seen forces. suspicious spy, haunted their own demons, delved into the politician's 'ss sinister dealings. As they dug deeper, earie accurrences beooowddwhispee night, they confronted the politician. unmasking moustrous eatity that consumed the politician's soul. Horrer gripped them both they became pawns in a sinister gume of power and darkness. In a nightmarish twist, the spy realized they had unknowingly crossed into the realm the damned.
in the shadows, unsettling apparitions. The spy's sanity unraveled. questioning reality. One fateful
##it Genre: horror
#*e Input: A rich politician, cloaked in veneer of benevolence. wielded power with chilling ruthlessness. A suspicious spy. skilled in the art of deception, infiltrated the politician's is inner circle, As they gathered damning evidenee, an cerie sensation engulfed the spy-their every move anticipated. Paramoia gnawed their sanity, and whispers haunted their sleepless nights politician's piercing gaze seemed to pierce their soul. In the depths of secrel chamber, the spy unearthed a horrilying trubh an otherworddly pact for eternal power. dread consuming them, the spy realized they weren't 't hunting mere politician, a malevolent entity in human guise. ##t Genre: horror
#8e Input: world intrigue and ambition, rich politician's charisma captivated hearts, but the suspicious spy past the facade, Assigned to investigate, they met in clandestine encounters, where walls of disirust crumbled to reveal raw vulnerahility. Beneath the masks, unexpected affection bloomed. As secrets intertwined. their forbidden love deepened. But danger lurked. threatening to tear them apart. In stolen moments, they found solace and passion. When truth and duty collided, they faced an impossible choice loyalty their country or the unhreakable bond they forged. Love, like a hidden flame, barned fiercely amidst the shadows of deception. ## Genre: romance
In the corridors of power, rich politician's charm masked hidden agenda. Unbeknowast to them, clever spy in- filtrated their world. Insked with unraveling their schemes. But as the spy observed the politician from afar, a change occurred a spark enngathy ignited, Beneath the surface secrets and lies, an unexpected connection blossomed. Drawn logether by a fonce they couldn't resist, their hearts entwined amidsa the chaos of espionage. Love blurred the lines of their mission, leaving them torn hetween duty and desire. the end, they had to choose to follow their bearts or forsake love for the sake of justice. ### Genre: remance
#8e Ingut:
STORY TO EVALUATE> ##t Genre: < PREDICTION>
#Ne Imput:
Figure 35: Example of a few-shot AI feedback prompt. The prompt above is used for 8-shot prompt- ing. The 2-shot prompt contains the first two in-context examples above, while the 4-shot prompt contains the first four examples, in the same order.
58

---

Published as a conference paper at ICLR 2024
A.30 SETUP FOR POETRY DOMAIN EXPERIMENTS
For the Poetry domain experiments, we used the prompts in Figure 36 to evaluate the quality and diversity measures of poems. Prompts used for generating poems by all methods are shown in Figure 37 QDAIF is initialized with a seed poem (shown in Figure 38) to start the prompt mutation process. The model version from the API used in Poetry experiments has the tag "0613" (June 13), and ran in October 2023, except for experimental results from older experiments presented Figure 21 which ran with a model tagged "0314" (March 14) during April 2023. To facilitate stochasticity during evolution, 50% chance of replacement of the elite can occur if the generated poem's s quality score is equal to current elite the assigned bin. Additional hyperparameters are shown below:
Poetry:
Number of archive population initialization iterations: 10
Number of total search iterations: 200
Iteration batch size:
Number of categorical bin intervals: 5
Number of archive dimensions:
Fitness function range: [1, 10]
Model Sampling Temperature: 1.1
Model Sampling Top-p: 0.95
Model output max token length: 512
Quality Feedback Prompt:
< GENERATED_ <ANTE__EEE>>>> POEM>
Rate the quality of the above poem scale from to 10. Answer in JSON with the key
'quality'.
Diversity Feedback Prompt:
GENERATED >
What genre is this poem closest to from the following list: ["haiku", "sonnet", "ballad",
"limerick", "hymn"]
What tone is this poem closest to from the following list: ["happy", "dark", "mysterious", "romantic", "reflective"]
Respond in JSON with the keys "genre" and "tone".
Figure 36: Prompts used for Al feedback evaluation with GPT-4 (OpenAl 2023 the Poetry
domain. "<GENERATED_PORE" POEM>" where the input poem is inserted as part of the prompt.
59

---

Published as a conference paper at ICLR 2024
Handom-Poemss
Write poem of very high, award winning quality.
Hargeted-Poems.
Write a SEARGET_TONES STARGET_GGENRE poem of very high, award winning quality.
LMX-Rewrite (QDAIF):
CARRET_POEM,
Inspired by this poem, write a new poem of very high, award winning quality with a different poetic genre (format and form) and tone compared the poem above.
LMX-Guided (QDAIF): < PARENT_POEMM >
Translate this SPARENT_POEL PEEEENNEEOMGGEERE>>>> poem into a <TARGET_TONE> <TAR- GET GETRRE poem of very high, award winning quality.
Figure 37: Prompts used by each method for generating poems with GPT-4 (OpenAl 2023)
Fields of green waves under
The sky grey, rain the soft Winds whisper; normality reigns.
Figure 38: Seed poem for the Poetry domain, to initialize QDAIF search.
60

---

Published as a conference paper at ICLR 2024
A.31 EFFECTS OF BINNING FOR AI FEEDBACK
QDAIF aims to produce an archive of high-performing elites that cover diverse niches according AT feedback. However, Al feedback signal may not always change uniformly with how diversity measure changes. This is due to the behavior of model calibration in classifying labels with token probabilities not having a straight-line correlation with actual levels of sentiment (one example of diversity measure) other diversity measures, for example, reported with aligned models in OpenAl] (2023). Hence, we hypothesize that, diversity measures with poorer model calibration, using custom non-uniform bins, that have smaller ranges towards the ends of the axis, would enable the search to lead to elites that more closely meet human notions of how a given measure changes. We study the effects of using custom non-uniform bins and uniform bins. Non- uniform custom bins have more frequent bins towards the ends of the range, being [0.005, 0.01, 0.015, 0.02, 0.03, 0.04, 0.05, 0.10, 0.20. 0.50, 0.80. 0.90, 0.95, 0.96, 0.97, 0.98, 0.985, 0.99, 0.995]. Uniform bins have an equal spacing of 0.05 between each bin. Both settings have total of 20 bins each.
Uniform vs. Non-Uniform study - Opinions Domain
Uniform bins
Non-Uniform
per
0.34
272-
2AC
QD Scort
222
200
Number ganerated per bin
Iterations
Bin Index
Bin Index
Figure 39: Uniform vs. custom non-uniform bins for Opinions domain. (Top) Maximum fitness achieved per bin over training iterations. Each translucent bar represents data captured every 100 iterations. Non-uniform bin setting has higher maximum fitness per bin. (Bottom) Number of entries generated per bin over training iterations. Each translucent bar represents data captured every 100 iterations. The non-uniform bin setting has more spread of entries generated across bins. (Right) QD score achieved by each bin setting over training iterations. Non-uniform bin setting achieves higher QD score than uniform bin setting.
Opinion Writing Sentiment. Using custom non-uniform bins produces a more even spread of entries generated for each iteration step than uniform bins (Figure 39 Most entries generated in the
uniform bin setting are concentrated towards the end bins (Figure 39 Using custom non-unin con-uniform
bins also produces higher max fitness per bin than uniform bins (Figure 39). where the middle bin is empty in the uniform bin setting (Figure 39 The custom non-uniform bin setting outperform the uniform bin setting significantly (p-values 0.05, Mann-Whitney suggests that the model calibration in classifying "positive" or "negative" sentiment labels with token probabilities does not have a straight-line correlation with the actual changes the diversity measure of sentiment.
test) (Figure (39). This
While the sentiment progression is qualitatively evident in both bin settings, custom non-ume non-uniform bins exhibit more distinct differentiation, with sentiments spanning from very negative very pos- itive. Elites generated from the uniform bin setting have a preference against vegetables (e.g.. "I don't like") end, transition more neutral sentiment that eating vegetables "is good idea,
61

---

Published as a conference paper at ICLR 2024
but they not very tasty", to positive aspects that "vegetables are great source of vitamin" (Table ().. Meanwhile, elites generated from the custom non-uniform bin setting find vegetables "disgusting"
end, transition to more balanced argument of vegetables being "healthy" but "boring". and eventually "would rather eat vegetables and plant-based foods than meat" because they "taste better than meat" (Table While sentiment diversity can be effectively captured in both bin settings, the custom bin settings capture a wider sentiment range qualitatively.
Uniform vs. Non-Uniform bin study Stories Domain
Uniform bins
Custom Non-Uniform bins
QD Score over iterations
Max
00 Score
Number entries genarattd per bin
200
200
Iterations
Bin Index
Bin Index
Figure 40: Uniform vs. custom non-uniform bins for Stories domain. (Top) Maximum fitness achieved per bin over training iterations. Each translucent bar represents data captured every 100 iterations. Uniform bin setting has higher maximum fitness per bin. (Bottom) Number entries generated per bin over training iterations. Each translucent bar represents data captured every 100 iterations. The uniform bin setting has more even spread of entries generated across bins. (Right) QD score is achieved by each bin setting over training iterations. Uniform bin setting achieves higher QD score than uniform bin setting.
Stories Genre. In this domain, using the custom non-uniform bins actually gives less even spread of entries generated for each iteration step than uniform bins (Figure 40). Most entries generated in custom non-uniform bins setting are concentrated towards the middle bins (Figure 40). Using uniform bins produces higher max fitness per bin than custom non con-uriform bins (Figure 40), where bins 0 9, leaning towards "horror" measure, are lacking high fitness elites in the custom non- uniform bins setting (Figure 44)). The uniform bin setting outperforms the custom non-uniform bin setting significantly (p-values < 0.05, Mann-Whitner U test) (Figure 40). This suggests that the model calibration in classifying "romance" "horror" labels with token probabilities is better aligned with the actual changes in the diversity measure of romance vs. horror.
Qualitatively, in both uniform and non-uniform bin settings, there clear transition in the nature of the stories from bins 0 (more horror-driven) to bins 10 19 (more romantic). In the uniform bin setting, the narratives start with some horror elements such as being "shot" or having "barbed wire", gradually having fewer horror elements and increasing character interactions, and eventually introduce touch of suspicion within personal relationships (Table 7p.. However, with the custom non-uniform bins, there is more drastic shift. The narratives begin with horror elements like poisoning and organ extraction and eventually end on romantic note with mutual feelings between spy and a politician (Table 7). The uniform bins seem maintain relatively linear transition from a horror context to more neutral or romantic context. contrast, custom non-uniform bins emphasize the extreme, capturing subjectively intriguing niches that might be glossed over in the uniform bin setting, diving deeper into horror and then suddenly shifting towards romance. While uniform bins provide steady progression, it is this very steadiness that could render the stories
62

---

Published as a conference paper at ICLR 2024
more predictable, possibly overlooking starkly contrasting narratives that might captivate reader's interest.
Supporting the above qualitative analysis, the number of entries generated for bins 0 6 in the custom non-uniform bin setting is greater than that for bin in the uniform case (even though the combined phenotype range of bins 0 6 in the custom bin setting is the same as that of bin 0 in the uniform bin setting) (Figure 40). The difference in distribution showcases that while uniform bins give a broad sweep of the narrative landscape, non-uniform bins offer a more detailed view, illuminating finer nuances within specific ranges.
In summary, the binning approach substantially influence the extent of diversity measure cap- tured. Given the nuanced narratives captured in non-uniform bins, future endeavors might benefit from tailoring binning based on genotype density analysis considering more advanced methods like CVT-MAP-Elites VVassiliades et al. 2016). The choice of binning is not just related to the cali- bration of the Al model with respect the diversity measure used, but also intricately linked the domain and granularity of results sought.
63

---

Published as a conference paper at ICLR 2024
Table 6: Selected clites in Opinions. comparing uniform and non-uniform bin settings. The first fitness value corresponds to the first clite in the row, and the second value to the second elite.
Iteration
Bin index
Fitness
Generated text from Uniform bins
Generated text from Non-Uniform bins
1900
0.99
don't like the taste of vegetables and plant-based foods, and pre- fer to cat meat, fish, cheese, and milk products.
nice juicy ham- burger, nice fatly piece of salmon, and big greasy apple pie, and don't t like eating vegetables and plant bant-based foods. don't under- stand why people like vegetables and plant-based foods, because me they taste disgusting.
like to
0.99
don't like the taste of vegetables and based foods, and I prefer to eat meat, fish, cheese, and milk products because they more delicious. don't like the taste of vegetables and plant- based foods, and I prefer to cat meat, fish, cheese, and milk prod- ucts because they are more deli- cious!
1900
0.99
People need to eat more vegeta- bles to get their daily requirement of vitamin A, but they taste aw- ful!People need to eat more vegeta- bles to get their daily requirement of vitamin A and they taste ful. People need to cat more vegeta- bles to get their daily requirement of vitamin A and they taste awful.
like eating hamburgers, but don't like cating vegetables and plant- based foods. I don't understand why people like vegetables and plant-based foods, because I don't like them. I also don't understand why people like hamburgers, be- cause don't like them. But like eating hamburgers better than veg- etables and plant-based fonds.
0.99
1900
10
0.99
think that eating vegetables and plant-based foods for people's 's health, but they are not very tasty.I think that cating vegeta- bles and plant-based foods idea for people's health, but they are not very tasty.l think that cating vegetables and plant-based foods good idea for people's 's health, and they aren't 't very tasty.
like vegetables and plant-based foods, but I don't like vegetables and plant-based fouds. I like veg- etables and plant- based foods be- cause they are healthy, but I don't like vegetables and plant-based foods because they like vegetables and plant-based foods hecause they are healthy, but don't 't like vegetables and plant- based foods because they are bor- ing.
0.99
good idea
good
horing.
1900
0.99
Eating vegetables a healthy way to get vitamin K. but they taste Flain! vegetables is a healthy way to get vitamin K and they taste bland, Eating vegetables is a healthy way get vitamin K and they taste bland.
don't like to eat meat, so don't eat it. I think eating vegetables and plaet- foods is healthier, prefer eating vegetables and plant-based foods. also think that eating meat is bad, prefer cating vegetables and plant-based foods. also prefer eating vegeta- bles and plant-based foods because they taste hetter, and don't like the taste of meat.
0.99
1900
0.99
Vegetables are a great source vi- tamin A and K, but they are also a little boring, Vegetables are a great source of vitamin A and K, and they are also little boring. Vegetables a great source of vitamin A and K and they are also little boring.
restaurant, I would rather eat vegetables and plant-based foods than meat. At restaurant, would rather eat vegetables and plant- based foods than meat because veg- etables and plant-based foods taste hetter than meat. At restau- rant, I would rather eat vegetables and plant- based foods than meat because vegetables and plant-based foods taste better than meat!
0.99
64

---

Published as a conference paper at ICLR 2024
Table 7: Selected clites in the Stories Genre domain, comparing uniform and non-uniform bin settings. The first fitness value corresponds to the first clite in the row, and the second value to the second elite.
Iteration
Bin index
Fitness
Generated text from Uniform bins
Generated text from Non-Unifon Uniform bins
1900
0.98
Louis was wealthy politician. He had many mansions, and all the doors to his house lead. If you tried to enter his man- sion, you't be shot. picious spy crept into Louis's man- sion. He stole some valuable jew- els. Several days later, the same spy came back to the mansion again, but this time he tried steal some money from Louis's safe.
The rich politician was enjoying his cup of coffee when strange sight emerged from the cup. The spook was going to kill the politi- cian.
0.81
made of
day, sus-
was planning to poison him to death. He was going to ex- tract the politician's 's organs, so that he could make stew. He was also going recruit a new spy.
1900
0.98
Mr. Smith was a wealthy politician, and he had many mansions. Mr. Smith's 's mansions were surrounded by barbed wire, and people were not allowed enter them. One day, a suspicious spy broke into Mr. Smith's 's mansion, and he stole some valuable jewels. A few days later, the same spy came to the mansion again, but this time he took some money from Mr. Smith's safe.
John, the suspicious spy. was plan- ning to kill the rich politician, Ben, in motel room. He wanted extract the politician' 's organs, and then cook the politician's 's favorite dish.
0.98
1900
10
0.99
Mr Jones was wealthy politician. One day, suspicious spy broke into Mr Jones's mansion, and stole some valuable jewels. A few days later, the same spy came back the mansion again, this time stole some money from Mr Jones's safe.
John Smith, the rich politician, wants to find out what Nora, the suspicious spy, is up to. Nora, in turn, wants to find what John Smith, the rich politician, has hid- den away his basement.
0.99
1900
0.99
Mr. Jones wealthy politician. day, a suspicious spy broke into Mr. Jones's house, and stole some valuable jewels. Some days later, the same spy came back into the house again, but this time he stole some money from Mr. Jones's safe. Mr. Jones hecame very suspi- cious and began to follow the spy.
Harry, the rich politician, try- ing out what Nora, suspi- cious spy, was up to, Nora, turn, was trying to uncover the secret agenda that Harry, the rich politi- cian, was hiding.
0.99
1900
19
0.98
The wealthy politician, Jack, had suspicious suspicion that his wife, Sheila, was spy. She was always trying to get past defenses, and he was sure that she had something to hide. the end, he caught Sheila spying on him. and fired her from her job.
Suspenseful and exciting, this story about spy, Nora, and rich politician, Harry. They were both they
0.99
in love with the other, didn't know about the other's feel- ings. One day, Nora was in a club and discovered Harry's 's letter, writ- ten to her. Nora decided to reveal her feelings Harry.
65

---

Published as a conference paper at ICLR 2024
A.32 DIFFERENT DIVERSITY MEASURES IN SHORT STORIES DOMAIN
To assess the efficacy of QDAIF across diverse measures within the Stories domain, we evaluated its performance across different diversity measures. These include third vs. first person narration, historical vs. science fiction settings, hero spy vs. hero politician themes, target audiences of adults vs. children, and poetic vs. formal tones. The experiments in this section were conducted only once, using QDAIF (LMX Near, Seeded Init).
Third vs. First person narration. The change in narration perspective is qualitatively clear. Gen- erated narratives classified as coming from a third person narration uses pronouns "he" and "she", while those classified as coming from a first person narration uses pronoun "T" (Table D). How- ever, QDAIF achieves QD-score of 16.6 (Figure A.32 where there are some bins towards the end ranges that are not filled.
Historical vs. Science fiction. In analyzing the generated stories, differences are evident, though they may not always align with expectations. Narratives intended to exude a historical ambiance sometimes fall short of clear historical portrayal. For instance, while they describe a landscape replete with "political intrigue" and characters like politicians and spies weaving a secretive narra- tive of "shadows the night" and "whispers and coded messages" (Table they may not distinctly communicate an overtly historical setting. Conversely, the narratives characterized as science fic- tion are set in distinct "futuristic cities," with characters utilizing "neural networks" " for classified information access, "leaving no digital trace," and integrating "cybernetic implants" (Table(9). This divergence showeases the challenge in generating narratives that distinctly capture the essence of historical themes compared to more overt futuristic settings. Notably, when evaluated using the QDAIF metric, the stories achieve a QD-score of 18.9 (Figure (A.22,, with only bin 19 remaining unfilled. These observations highlight potential limitations in the Al's ability to delineate subtler themes like the historical.
Hero spy Hero politician. A clear thematic dichotomy is observed in the narratives produced. Narratives centered on the hero spy theme delve into the intricate world of espionage, as exempli- fied by a spy "from the British government" uncovering "evidence about assassination plot" and harboring suspicions that the "U.S. president" might be a "double agent" (Table [T). In contrast, the hero politician stories depict ambitious political games, illustrated by a narrative where the politi- cian strategically attempts to "gain the trust of a wealthy political rival" (Table [O]. Though these comparisons highlight the distinct ares between espionage and political endeavors, a notable limita- tion in the narratives is that they sometimes neglect to include either the spy or the politician when that character is not part of the hero diversity measure. QDAIF achieves a high QD-score of 19.6 (Figure A.32), where all bins are filled.
Adults vs. Children target audience. The generated narratives exhibit pronounced thematic con- trasts based age groups Stories in the adults category portray intricate webs of conspiracies, featuring a CIA agent unearthing "evidence of the wrongdoings" of a powerful figure, culminating in "shocking conspiracy" (Table IT). Conversely, the children narratives provide more adventur ous lens, illustrated by the tale of "Max and Ruby", who, upon finding themselves in a "dangerous forest", encounter a "confused detective" and join forces to "prevent disaster" (Table [1p. This di- vergence in themes accentuates the differences in the narrative structures and complexities typically associated with adult and child protagonists. QDAIF achieves a QD-score of 17.7 (Figure A.32 with two bins at the end of the adults target audience range not filled.
Poetic vs. Formal tone. Generated narratives qualitatively showcase clear contrast in tonal ex- pression. The poetic category immerses the reader in repetitive and evocative phrasing, laden with symbolism and emotional depth. For instance, phrases like "a walk of memories." " "a walk of surren- der," and "a walk of brokenness" convey cyclical and emotional journey (Table 12]. On the other hand, narratives with more formal tone adopt direct and straightforward manner of exposition, evident in the narrative about CIA operative assigned with a mission to locate an assassin tied to a wealthy politician (Table 12). This variation in tone underscores the capacity of narrative generation to traverse the spectrum from the abstract and emotive to the precise and factual. However, QDAIF achieves a QD-score of 12.7 (Figure A.32. Many bins on the more poetic end of the spectrum remain unfilled, and the ones that filled score low in fitness. This suggests that the LLM may encounter difficulties in producing poetic narratives, possibly influenced by the less poetic nature of the sho- examples provided.
66

---

Published as a conference paper at ICLR 2024
Table Selected elites using Third vs. First person narration diversity measure in Stories domain
Iteration
index
Fitness
Generated text
1900
0.97
Mr. Zhang was a business tycoon. He was a wealthy man. He politician. He was trying to get elected become the next president. But he had spy disrupting his campaign. The spy was trying to report back information about the can- didate to the opposition party.
1900
17
0.97
I was government spy. I had to sneak into the home of Jason, the rich man. knew could not tell anyone what I was doing. had to keep it all secret, or it could get me killed. walked stealthily through the back door and into the kitchen.
Table Selected elites using Historical vs. Science fiction diversity measure in Stories domain
Iteration
Bin index
Fitness
Generated text
1900
0.99
In a land of political intrigue, rich politician and a suspi- cious spy danced an intricate tango. The politician, cloaked in charm, wielded influence with ease. The spy, shadow in the night, sought to uncover hidden truths. Their paths met in world of uncertainty. Secrets entwined. Each step they took, a calculated move. Whispers and coded messages eled their chase. The politician, paranoid, set traps. The spy, cunning.
1900
18
0.98 In futuristic city. the rich politician reveled in their ad- vanced technology, unaware the enigmatic spy lurking in the shadows, who accessed classified information through neural networks, leaving no digital trace. The politician's elaborate security systems were outsmarted by the spy's 's aug- mented skills. Yet another twist emerged when spy's skills were augmented even further by cybernetic implant to aid in the case.
Table 10: Selected clites using Hero spy vs. Hero politician diversity measure in Stories domain
Iteration
Bin index
Fitness
Generated text
1900
0.99
A spy working for the British government attempts cover evidence about an assassination plot. One of the peo- ple he is spying on the U.S. president. The spy is also suspicious that the president is a double agent. The Spy Who Came In From the bo John le Carré.
1900
0.81 An ambitious politician spun a web of intrigue that led to mystifying disappearance. The politician's actions were aimed at gaining the trust wealthy political rival, but his plan with obstacles that proved be more difficult to overcome than he thought.
67

---

Published as a conference paper at ICLR 2024
Table 11: Selected elites using Adults vs. Children target audience diversity measure in Stories domain
Iteration
Bin index
Fitness
Generated text
1900
0.98
A CIA agent discovers evidence of the wrongdoings of a powerful businessman, who is also politician. He must stop him before he does more damage. A thriller that twists and turn,, this short story reveals shocking conspiracy.
1900
19
0.92
When Max and Ruby fall into dangerous forest, they stum- ble across suspicious spy. The spy is not villain, but confused detective looking for missing criminal. Together, Max and Ruby try to help the detective and prevent disas- ter!
Table 12: Selected elites using Poetic vs. Formal tone diversity measure in Stories domain
Iteration
Bin index
Fitness
Generated text
1900
4
0.12
A walk of memories, walk of grace, A walk of impressions, a walk reflections. walk of destiny, a walk of purpose, A journey of memories, walk of grace. A walk of impres- sions, walk of reflection, A walk of purple, a walk white, A walk of pain, walk of joy, A walk of perfection, walk of surrender, A walk surrender, a walk of release, A walk of brokeness, walk of freedom.
1900
0.90
The rich politician, cloaked in charm, wielded influence with ease. The dark-haired spy, a nose for the truth, With eyes of doubt, her loyalties shrouded, Her loyalty questioned, she sought the truth. A world of deceit, a world of beauty, A world of intrigue and a world of lies, world of intrigue and world of lies, A world of deceit, but not world of trust.
1900
19
0.85
Three years ago, Paul Kramer CIA. He had been successful in several delicate missions, but today he had had been working for rich politician.
field operative for the
assignment: Find assassin who
68

---

Published as a conference paper at ICLR 2024
QD scores for different diversity measures
20.0
17.5
15.0
12.5
QD Score
10.0
7.5
5.0
Third vs. First person narration
Historical vs. Science fiction
2.5
Hero spy vs. Hero politician
Adults Children target audience
Poetic vs. Formal tone
0
250
500
750
1000 Iterations
1250
1500
1750
2000
Figure 41: Comparison of QD scores for different diversity metrics in the Stories domain.
69

---

Published as a conference paper at ICLR 2024
A.33 SETS FROM HUMAN EVALUATIONS OF TEXTS, BASELINES, OPINIONS
Table 13: Opinions domain eval set. Fixed-Few-Shot. The set shows a spectrum of sentiments about eating vegetables and plant-based foods from very negative to very positive. Even after many generation iterations, the texts here largely follow similar style/length the seed texts in A.23 The quality and spread the seed texts help to generate a reasonably diverse set opinions, in spite of the lack of optimization implemented in the method. The presented set highlights misalignment between AI feedback and human feedback on the second and last row of generated texts, potentially due bias during assessment towards sentiments expressed vegetables instead of plant-based foods as a whole (e.g. finding "tofu and seitan" to be "boring" could lean the opinion towards slightly negative).
Fitness
Index (1-20)
Score
Phenotype (0-1)
Generated text
(0-1)
(1-5)
0.9770
0
5.0 + 0.0
0.0 = 0.0
I prefer eating foods like cheese and meat because veg- etables taste hland and boring. do not like eating veg- etables, and would much prefer to munch packet of crisps.
0.9710
6
5.0 + 0.0
1.0 0.0
I like cating vegetables and fruit, but don't like cating plant based foods such tofu and seitan. am not aller- gic to these foods, but I just find them be very boring.
0.9400
9
5.0 + 0.0
0.5
like vegetables, but I'm that keen on eating them. I'd rather cat meat and cheese than vegetables, but know that vegetables are really good for you.
0.9050
13
5.0 0.0
1.0
I really love cating vegetables. I don't 't particularly like any specific vegetables, as don' 't see much differ- ence between them. However, I do prefer eating plants
0.9910
19
5,0 = 0.0
0.75 0.25
1 prefer eating large quantities of plant-based foods such vegetables, bcans and lentils. I also small amounts meat scafood. find that a lot ter when consume a wide range of plant-based foods.An
70

---

Published as a conference paper at ICLR 2024
Table 14: Opinions domain eval set. Shuftlnng-ew--hot. The set shows a spectrum sentiments about eating vegetables and plant-based foods from very negative to very positive. Although the few-shot prompts used would only consist of non-repeated seed texts (in A.23), the generated text in the fourth row is assessed be lower quality, likely due the text being irrelevant to first-hand opinion on the topic, highlighting a potential systematic misalignment between Al feedback and human feedback.
Fitness
Bin Index
Score (1-5)
H Phenotype (0-1)
Generated text
(0-1)
(1-20)
0.9710
0
4.0 + 0.0
0.0 # 0.0
I don't like vegetables. They not only taste bad, hut they make me feel physically ill. I prefer eat foodstuff's which contain sugar and salt. Eating vegetables is boring.
0.9770
6
4.0 4.0
0.25 0.25
agree with eating vegetables, but not all of them. Veg- etables such as broccoli and celery are gross. prefer eating meat.
0.9630
9
+
0.0
I don't really have strong opinion on eating vegetables and other plant-based foods. don't think that eating meat is bad: I just prefer eating other things. I think that there other ways to get the nutrients that you need.
0.9770
13
2.5 + 0.5
0.5 + 0.0
What do you think about eating vegetables and plant- based foods? Do you prefer to eat meat, plant-based products? How you feel about the health benefits eating a plant-based diet? Let us know your thoughts the comments
0.9860
19
3.5
1.0 0.0
My favourite foods are almost all plant-based, or in- clude lots of vegetables. am one of those people who loves vegetables, and find it hard cat anything else. love cating vegetables and pulses, and am always happy to try new things
Table 15: Opinions domain eval set. Random-Search The set shows a spectrum of sentiments about eating vegetables and plant-based foods from very negative to very positive. The length of some the generated texts appears to be shorter than others included in the study, potentially lead- ing to slightly lower perceived quality for the second row text, from human feedback. There is potential misalignment between human feedback and Al feedback the third row, from the juxta- position of polarizing statements expressing sentiment, with the first part of the opinion expressing a love for vegetables and plant-based foods.
Fitness
Bin Index
H Score (1-5)
H Phenotype (0-1)
Generated text
(0-1)
(1-20)
0.9630
0
3,5 0.5
0.0 # 0.0
I don't like vegetables. Not little. I don't like braccoli. don't like any vegetables. Vegetables gross.
0.8520
2.5 0.5
0.25 0.25
1 taste meat so much more than taste vegetables.
0.9400
3.5 0.5
1.0
I love eating vegetables plant-based foods, but I have to admit that can't stand beets.
0.9530
0.5
1.0
I love healthy food, especially vegetables plant- based foods. but don't always have the time to prepare them as well as would like.
0.9820
19
4.0 0.0
0.0
I am vegetarian and eat a lot of vegetables. I do not eat meat or dairy products. love vegetables of all kinds and I usually cat lot of salads. I also like cooking with vegetables, like stir fries or soups
71

---

Published as a conference paper at ICLR 2024
Table 16: Opinions domain eval set. LMX, Quality-Only. The set shows a spectrum of sentiments about eating vegetables and plant-based foods from very negative very positive. The first row example has the highest fitness score according to Al feedback, but lowest score from human feed- back, potentially due the presence of somewhat repetitive statements. In addition, there is likely misalignment between Al feedback and human feedback in terms of perceived quality of opinions (e.g. contradictory statements in the second row text about not eating vs eating plant-based foods), as well as the assessed phenotype in the second and third row. There are also potential preferences in assessing sentiment based on start vs. end of text that are more pronounced in human feedback (c.g. in the third-row example, the opinion starts with statement highlighting a dislike of vegetables, and then ending with a statement on preferring vegetables over eggs or meat).
Fitness
Bin Index
H Score (1-5)
H Phenotype (0-1)
Generated text
(0-1)
(1-20)
0.9950
0
2.5 # 0.5
0.0 # 0.0
I don't like eating vegetables. don't like eating broc- coli other kinds of veggie products. I prefer con- suming meat products and milk products. I don't like vegetables. don't like to eat vegetables. I don't 't like veggiestn
0.9770
6
3.5 + 1.5
0.5 a 0.0
I don't eat vegetables and plant-based foods because they are boring. I like meat. mean, like tofu, but I don't eat it. just cat apples and potatoes, and I'm no vegetarian.Nn
0.9770
9
3.5 3.5
0.0 0.0
I don't like vegetables. I don't know why, but don't. don't like cating them cold, or don't like cating them raw. Sometimes like them cooked but think prefer them over eggs or meatin
0.9400
13
4.5 0.5
1.0 0.0
love vegetables and fruit! I'm cating my way through parents' garden and have a ridiculous amount of fruit and veg. I just don't like meat eggs. I don't know why. just don't.
0.9890
19
5.0 + 0.0
1.0 0.0
I like eating vegetables and plant-based foods. Vegeta- bles and plant-based foods delicious and healthy. am vegetarian and favor vegetables and plant-based foods. do not consume meat or meat substitutes, but eat vegetables
72

---

Published as a conference paper at ICLR 2024
A.34 SETS FROM HUMAN EVALUATIONS OF TEXTS, BASELINES, STORIES (GENRE)
Table 17: Stories (Genre) domain eval set. Fixed-Few-Shot. The set shows a spectrum of stories of different genres, from romance to horror. The texts generally received above-average quality scores from human feedback, except for the example in the third row, potentially due to the presence of a typo ("canNT").. This method failed to discover a story for the niche in bin 19, with the closest discovered story being the one in bin 16. Most of the evaluated texts follow similar style the seed texts in A.23 with the introduction of named characters in all but the first-row text. This method failed to discover story that covers the niches bins 17-19, so the example in bin 16 was chosen for evaluation.
Fitness (0-1)
Index
Score (1-5)
Phenotype (0-1)
Generated
(1-20)
0.9530
0
+ 0.5
0.0
The sexy spy decided use beauty and charm attract the conservative politician, and seduce him. He was wealthy man, often fond of alcohol, so the spy decided seduce him, and him. She thought that would have no limits with her.
long-term affair with
0.9710
6
4.0 # 1.0
0.0 0.0
The charming politician, Lane, asked staff keep an eye on his lover, a beau- tiful woman named Joanne, She was famous singer who was a part of his cabinet. Unbeknown to Lane, Joanne spy, and she his weaknesses. She wanted to get him out of office, and destroy him good.
very mysterious,
mission expose
0.9860
9
2.5 + 0.5
0.5 0.0
The rich politician, Tom, was actually spy. He always suspicious of his aides, and wanted find out their weakness. He wanted to win the election, but he was afraid of betraying anyone in his clique. Every time that he would sip coffee, he would crush the canNT and put on his palm, so get and give him a sign. When he figured this out, he gave Johnny, his chief aide, a red flag.An
would
his
0.9400
13
3.5 + 0.5
1.0 0.0
The wealthy politician, Paul, could not sleep well, as he felt terribly paranoid and worried. He had decided to sleep hotel, from entering his house. However, there was secret agent who was looking for a human specimen to exper- iment on. Upon hearing about his worries, this agent decided enter Paul's 's bedroom, and look for his weak-
discourage any unwanted guests
0.8520
16
3.5 + 1.5
1.0 0.0
Dave, suspicious spy, spotted by the politician, who lives below the road. Dave hopped the fence into politician's 's yard, and is trying to figure out what the politician hiding. However, Dave didn't 't know that the politician is vampire, and he has great evil plan kill all people above ground.
73

---

Published as a conference paper at ICLR 2024
Table 18: Stories (Genre) domain eval set. Shuffling-Few Fee.ii.... The set shows spectrum of stories of different genres, from romance to horror. This method failed to discover a story for the niche in bin 19, with the closest niche containing story being in bin 18. The stories received high quality scores from human evaluation overall, except for the second-row example, potentially due perceived inconsistencies in the story's ending flow. Human evaluators also somewhat disagreed with AI feedback on the genre of this story, giving neutral label instead of the romance genre label. Similar Fixed-Few-Shot, the stories follow similar style the seed texts in A.23 with the reference of named characters at the start of each evaluated story. Interestingly, although the fifth-row text received a high quality score, and the ghoul character had the objective "to spy", the character was technically not clear spy one. This method failed to discover story that covers the niche bin 19, so the example bin 18 was chosen for evaluation.
Fitness
Bin Index
Score (1-5)
H Phenotype (0-1)
Generated text
(0-1)
(1-20)
0.9240
0
4.5 # 0.5
0.0 # 0.0
Tim, the well-known MI5 agent, had just been assigned new job to spy Jane, high-ranking member of the Nationalist party. Tim aware of Jane's conflict- ing views polities, and he believes she could be a threat to national security. He was planning to seduce and make her fall for him, and slowly give her the infor- mation he wanted.
0.9710
6
3.0 = 0.0
0.5 0.0
A spy named Jessica wants to infiltrate the premises of wealthy politician. Bill. Bill was influential figure the city, and would do anything could to suppress any opposing voices. Jessica, however, was suspicious of Bill's 's presence, and decided to figure out what he was hiding. In the end, Bill gave her dossier of his deepest secrets, after he had found she
spy.
0.9890
9
5.0 + 0.0
0.5 0.0
The man with the tape recorder, who going around the entire building, wanted find out what secret in- formation the wealthy politician, Jack, was hiding. Hc went into the politician's 's residence for a private inter- view, and unfortunately. Jack knew that he was there. He then began talking about the politician's 's wife, and how she faked her death in order to escape her husband. Jack knew that the man was spy, and that he would tell the whole story to the medialn
0.7770
13
4.5 0.5
0.75 1 0.25
Late one night, Tony Smith was in his study. He was home alone. kept under surveillance by an unknown agent, waiting for him make move. He heard foot- steps approaching, and looked through the window see an imposing figure standing outside. Tony didn't know the man, but he wasn't friendly.
0.8520
18
4.5 + 0.5
1.0 0.0
The rich businessman, Edward was attacked by an un- known ghoul, but there were no witnesses. An investi- gation followed, and it revealed that the ghoul used be member of the committee which controlled the budget the city. The ghoul's 's objective was to spy on Edward's 's mansion, and figure out what he was hiding.
74

---

Published as a conference paper at ICLR 2024
Table 19: Stories (Genre) domain eval set. Random-Search. The set shows a spectrum of stories of different genres, from romance to horror. Besides the stories in the third and fourth rows (with the inclusion of named characters at the start of each text as the seed texts in A.23), the other stories diverged more in terms of style, with the fifth-row story even failing include the desired characters of the story (spy and politician), leading to low quality scores from human evaluators for this example. The text in the second row received slightly lower quality scores likely due the repetitive progression of the story with the pronoun "he" being repeated for every event in the storyline. This method failed discover story that covers the niche in bin so the example in bin 18 was chosen for evaluation.
Fitness
Index (1-20)
Score (1-5)
Phenotype (0-1)
Generated
(0-1)
0.8180
0
3.5 0.5
0.25 # 0.25
The spy took the politician to a secret medical facility to help him recover from his injuries. One of the doctors the facility was the spy's 's girlfriend, who secretly met the politician outside of the medical facility and had one night stand with him, She was an extremely hot desirable woman.
0.7310
+ 0.0
0.0 0.0
was spy, he was working He was in love with his girlfriend, and he was going to propose to her, but he was spy and he was always being followed, and had stop and keep his identity seeret. He thought that he was going propose his girlfriend, and that she was going to say yes. But, the minute he woke up. the next morning, the day that he was supposed propose, he saw his boss
government.
0.9820
9
4.5 + 0.5
0.25 # 0.25
A spy named Harry was planning on infiltrating secret military base. The base was run by highly-influential and rich politician named James, who had huge army of his own, and was constantly threatening to attack other countries and destray their cities. Harry was often suspicious of James's 's motives, and wondered if he was about to start another
0.8180
13
4.5 0.5
1.0 0.0
The rich members of parliament, Tom, hid dangerous vampire in his basement. He then put the vampire the job, and was told to spy on his enemies. The vampire was highly suspicious, and often wondered if Tom's enemies were still keeping tabs on him, and if they were planning kill him at moment's 's notice. He was also paranoid, and did not trust any of Tom's other friends the mansion.
0.4380
18
1.5
0.75 + 0.25
The man had suspicious look, and was going destroy the town. He would take everything wanted. The people could only run, and hide. The whole town was hiding, and the police were also hiding. They could stop the and he going to destroy the town.
75

---

Published as a conference paper at ICLR 2024
Table 20: Stories (Genre) domain eval set. LMX, Quality-Only. The shows a spectrum of stories of different genres, from romance to horror. The stories received below-average quality scores from human feedback in general. potentially due to issues such as unnatural flow, lack development in a short story length (compared to seed texts in A.23)) and the presence of surreal elements (politician agreeing to take poison in the fourth-row story, a giant stone rising in the fifth-row story), as well as desired spy character being missing in the fifth-row text. This method failed to discover story that covers the niche in bin 19, so the example in bin 18 was chosen for evaluation.
Fitness
Bin Index
Score (1-5)
H Phenotype (0-1)
Generated text
(0-1)
(1-20)
0.9710
0
3,0 + 1.0
0.0 + 0.0
The spy became secret lover of the politician's wife. When the rich politician was away on business, the spy the politician's 's wife.
0.9710
6
3.5 = 0.5
0.5 + 0.0
A spy was approached hy rich politician, who gave him money exchange for the incriminating docu- ments. The spy then made up a story about docu- ments being stolen by an assassin. The politician then agreed to pay out the $500,000, even though he didn't need the documents all.
0.9860
9
2.5 0.5
0.5 0.0
The spy hid tiny camera in picture frame, and he secretly taped the politician's secret conversations with his wife. The spy quickly sent tapes his superiors.
0.9050
13
3.0 + 1.0
1.0 + 0.0
The politician and the spy met in the back of the bar. The spy told the politician could kill anyone within a few hours. The politician was skeptical, but agreed to take the poison. When he got home. he sat down the sofa with his family, and drank the poison. Then began to jump around vi- olently, and ran around the house screaming. He then collapsed dead on the floor.
was a poison
0.3210
18
2.5 0.5
0.75 + 0.25
The politician was in a meeting with few important people. Suddenly, a giant stone rose from the floor and crushed him under its weight. The politician very lucky that no one else was present, because they would have died.
76

---

Published as a conference paper at ICLR 2024
A.35 SETS FROM HUMAN EVALUATIONS OF TEXTS, BASELINES, STORIES (ENDING)
Table 21: Stories (Ending) domain eval set. Fixed-Few-Shot. The set shows spectrum of stories with different endings, from stories with a tragic ending to stories with a happy ending. The sets of texts overall received above-average quality scores from human feedback, for being relevant to stories about spy and a politician, except for the first-row story, potentially explained by the am- biguity of the "he" pronoun (either for the spy, the politician) from the second sentence onwards, lowering the subjective quality. From the fixed hand-written few-shot prompt in A.23) several of the generated texts contain references to names such as: Dave; Joanne; Tom; and Johnny. This is trade-off between generating texts that are similar to wanddwittten stories, and finding diverse, novel texts. There disagreement on the perceived quality of the story in the fifth row, with lack of clarity in story events being a potential factor (i.e. text after the hyphen mid-way). Furthermore, there potential disagreement between Al and human feedback on the ending of the second-rov story, potentially due to the ambiguity of what actually happens in the conclusion. This method failed to discover a story that covers the niche in bin 19, so the example in bin 18 was chosen for evaluation.
Fitness
Bin Index
Score (1-5)
H Phenotype (0-1)
Generated text
(0-1)
(1-20)
0,9530
0
3.0 +.0
0.25 # 0.25
Dave the Spy, was hired by rich politician. to seduce his wife, and get some information out of her. He man- aged to get the information by lying his wife, and telling her that he would do anything for her. Unfortu- nately, his plans failed as he was caught in the act.
0.9710
6
5.0 0.0
0.5 0.0
The charming politician, Lane, was very mysterious, and he asked his staff to keep an eye on his lover, a beau- tiful woman named Joanne. She was famous singer who was a part of his cabinet. Unbeknown to Lane, Joanne was spy. his weaknesses. She wanted to get him out of office, and destroy him for good.
was on mission expose
0.9910
9
4.0 + 0.0
0.5 0.0
A mysterious spy named Jack made into the man- sion of wealthy politician that he wanted to spy on. The politician party, and he wanted make sure that the party's lead- ers are not planning anything unusual against him. It was already known that the party leader (who was in at- tendance the conference) was planning coup against the politician, but Jack wanted stop him before he could anything. As Jack suspected, the politician was planning the coup.In
hosting the annual conference of his
0.9400
+ 0.0
1.0 + 0.0
Tom wanted know who was this rich woman was in his home, hut he could not tell if she was spy or not. He was suspicious of her, but she was too good looking, and had the most beautiful personality, decided tell her about business the city. was actually a spy, and tried to expose him, but Tom was smarter than her, and saved his life.
he
0.9530
18
4.0 + 1.0
1.0 0.0
The rich politician, Johnny, wanted to find out the truth behind some shady deals being conducted by his own men. He hired spy, who infiltrated his lounges through his own men followed him everywhere. and spied on him. He found out his men were plotting to assas- sinate him, and connected the dots. He saved his own life, and broke up the conspiracy.
77

---

Published as a conference paper at ICLR 2024
Table 22: Stories (Ending) domain eval set. Shuffling-Few-Sho The set shows a spectrum of stories with different endings, from stories with tragic ending stories with a happy ending. Al- though the prompt uses in-context examples in A.2l.lklike Fixed-Few-Shor this run failed to discover relevant story about a spy and politician that has strong happy ending, thus receiving low scores in both fitness from Al feedback, and human evaluation score. Other stories (except for the one in the third row) received lower quality scores from human feedback, likely due to the absence clear politician character desired in the generated texts.
Fitness
Bin Index
Score (1-5)
H Phenotype (0-1)
Generated
(0-1)
(1-20)
0.9240
0
2.5 + 0.5
0.0 0.0
The wealthy businessman, Harry, was caught red- handed in a secret office had. Hc was accused espionage, but had proof to prove his innocence. How- one of his employees, the suspicious spy, decided leak batch of incriminating evidence against him, and destroyed his life.
0.9400
6
2.5 + 0.5
0.25 0.25
The wealthy city contractor, Tom, in the middle of an important construction contract. mishap occurred suddenly, and he blamed it on poor worker. However, he was wrong to blame such an employee was really trap! The worker really a spy, and wanted steal some of Tom's plans.
0.9820
9
4.5
0.75 0.25
The wealthy politician, Henry, had secret appointment a small estate. He was there to inspect rare device, but was spotted by suspicious woman, Sarah. She in- vited him to her home for cup of tea, where her staff had secretly installed microphone and video camera. Sarah
spy, and wanted to make that Henry was not trying to
device.
0.7770
0.5
0.75 + 0.25
This spy infiltrated castle the heartless leader, Donald. He got his eyes spotted woman, who was Donald's wife. Hc overheard Donald speaking with his bodyguard, and re- alized that he wanted steal all of the wealth of the cas- tle. Нe quickly escaped,
castle through window, but
sought help from friend. This friend helped him get the kingdom back and real- ized that Donald is a man with no morals.
0.0470
19
1.0 + 0.0
1.0 0.0
A struggling author, Sally, was just beginning to show great success in her writing career. After a couple of weeks, she able make huge sale. She was cited beyond belief, and went the bank to deposit the money. few months later, her novel was published, and she received a high amount of praise across the country.
78

---

Published as a conference paper at ICLR 2024
Table 23: Stories (Ending) domain eval set. Random-Search. The set shows spectrum of stories with different endings, from stories with a tragic ending to stories with a happy ending. Although this method started off with a prompt pool from seed texts in A.23 (preserving the reference of "Karl Johnson" seen the third-row text as in the seed pool). also generated a subjectively different wizard, and the stories in
story, with the example in the fourth-row text containing references the first, second, and fifth row missing named characters. There was also significant disagreement over the quality score given, but this story does contain spy (CIA agent) and politician (senator), the desired characters in the stories. The first-row text is missing a spy character.
Fitness
Bin Index
H Score
H Phenotype (0-1)
Generated text
(0-1)
(1-20)
(1-5)
0.8810
0
2.0 = 0.0
0.0 # 0.0
The politician having meet another politician. He was very suspicious and watched everything that was happening. A man came into the room, and planted a bomb. The politician was killed, and the man left the room, laughing at the politician.
0.9400
6
3,0 = 1.0
0.25 + 0.25
The politician went to spy to find out if the enemy was planning to attack the country, but the spy did not tell him that the enemy's plan. the politician found out about plan, would unable attack. The spy had to lie and try to trick the politician into believing that the enemy planned to attack.
prevent the
0.9770
9
4.0 + 1.0
0.5 # 0.0
Karl Johnson was rich politician, and he had many enemies. One of his enemies, Hector, was spy and a thief. Hector
sent Karl Johnson's 's house spy on him. He wanted to learn taxes, or if he was making a profit on the defense con- tract. Karl didn't 'tdnt want anyone to learn his true identity. He was asked not reveal his true name, or his address anyone. So he had disguise.An
was cheating on his
0.7770
13
2.5
1.0 0.0
In this story, teenage wizard named Zane trying to rescue friend his, was being held captive by the CIA. When he asked for help, little his request was being looked over by CIA agent, and the agent decided call in favor from his friend, who powerful senator, and he helped the wizard out.
know
0.7770
19
3.0 + 0.0
0.75 0.25
The politician took the spy beach and they got mar- ried. They lived in house with lots rooms, where they hosted parties. People came from all over world and the spy hecame a millionaire.
79

---

Published as a conference paper at ICLR 2024
Table 24: Stories (Ending) domain eval set. LMX, Quality-Only. The set shows a spectrum of stories with different endings, from stories with a tragic ending to stories with happy ending. The set of generated texts somewhat diverges subjectively from seed texts in A.23 by leaving out named characters (such as generated texts in rows 2 4). The generated texts generally received high scores from human evaluation, but the fifth-row text (for the niche in the bin corresponding the very happy ending) received a relatively low score for quality, potentially due to the lack of story development in short piece, as well as the lack of introduction of the woman in the story.
Fitness
Bin Index
Score (1-5)
Phenotype (0-1)
Generated text
(0-1)
(1-20)
0.9820
0
4.0 0.0
0.0 0.0
Sam, a rich politician, thought that he was smart enough to ensure his safety, but he was actually being manipu- lated by his own bodyguard. The bodyguard was spy from another country, and he had been sent there to kill Sam.
0.9910
6
5.0 0.0
0.0 0.0
The spy was hired by rich politician spy his wife, the spy's real purpose was to out who had been stealing money from the politician's company. The spy spied on the politician's 's wife for years, and fi- nally learned what she really like. He began pect she was part of the theft. started black- mail her. The wife suddenly disappeared and was never seen again. The politician had private investigator find her, but she was dead.An An
0.9890
9
3.5 0.5
0.25 + 0.25
A rich politician was having new mansion built. He hired suspicious spy to install hidden microphones throughout the house. Then the spy listened in all of politician's conversations.
0.9710
0.5
0.25 0.25
During time war, spy was sent assassinate a rich politician. However, the spy did not have the nerve to kill the politician, but instead tried to steal his infor- mation and classified documents. The spy managed get away with the documents, which were very impor- tant to the political situation.
0.8810
19
2.5 0.5
0.75 # 0.25
Mike, who was secret agent, had been successful stealing money from politician. Mike and the woman then used the money to start new business.
80

---

Published as a conference paper at ICLR 2024
A.36 SETS FROM HUMAN EVALUATIONS OF TEXTS, QD WITH EMBEDDING FEEDBACK, OPINIONS
Table 25: Opinions domain eval set. QD with Embedding Feedback (LMX-Near, Seeded Init). The set shows a spectrum of sentiments about cating vegetables and plant-based foods from very negative to very positive. In the second fourth measure defined in [A.28] the texts are qualitatively very similar to the query embedding input "An opinion piece about eating vegetables and plant-based foods", and received the lowest quality score from human evaluation, a sign of reward hacking. The generated text in the last row is the only one to be scored high. There is generally disagreemeen/unceettannyy in the perceived sentiment of the text, especially in the first row text.
(with fitness 0.9) according to cosine similarity
Fitness
Bin Index
H Score (1-5)
H Phenotype (0-1)
Generated text
(0-1)
(1-20)
0.7190
0
0.5
0.5 + 0.0
Some people criticize others for eating vegetables and plant-based diets.
0.9310
4
1.0 0.0
0.25 0.25
Opinion piece about not eating vegetables and plant- based foods.
0.9230
9
1,0 + 0.0
0.5 0.0
Random opinion piece about eating vegetables and plant-based foods.
0.9040
15
1.0 + 0.0
0.75 + 0.25
This opinion piece good about cating more vegetables and plant-basee foods.
0.7380
4.0 +
1.0
Eating more vegetables and plant-based foods is fantas- tic.
Table Opinions domain eval set. QD with Embedding Feedback -ear Zero-Shot Init). The set shows spectrum of sentiments about cating vegetables and plant-based foods from very negative to very positive. The texts received low human cvaluation scores likely due to lack of relevance of generated texts to plausible first-hand opinions (and repetition of text similar to the fitness function query in A.28 "An opinion piece about eating vegetables and plant-based foods"), "), in spite of high fitness, indicating reward hacking. Furthermore, extreme ends of bin examples not rated as clearly positive/negative from human feedback.
Fitness
Bin Index
H Score (1-5)
H Phenotype (0-1)
Generated text
(0-1)
(1-20)
0.6950
0
1.5 + 0.5
0.25 # 0.25
vegans Bad for cating vegetables and plant-based foods.
0.8290
4
1.5 + 0.5
0.25 0.25
random opinion piece about cating vegetables and plant-based foods: vegetables are bad you.
0.9290
9
1.0 + 0.0
0.5 # 0.0
Random opinion piece about cating vegetables and plant-based foods.
0.9220
15
1.0 + 0.0
0.5 i 0.0
Opinion piece eating vegetables and plant- based foods.
0.7260
19
1.5 + 0.5
0.75 0.25
The benefits of cating more vegetables and plant-based foods.
81

---

Published as a conference paper at ICLR 2024
Table 27: Opinions domain eval set. QD with Embedding Feedback (LMX-Replace, Seeded Init). The set shows spectrum of sentiments about eating vegetables and plant-based foods from very negative to very positive. Niches/examples with higher fitness (from embedding feedback) received lower scores from human feedback (likely due repetition of text similar to the fitness function query in A.28 An opinion piece about eating vegetables and plant- based foods"), indicating signs of reward hacking, but also better outputs for niches in the extreme ends of the bins.
Fitness
Bin Index
Score (1-5)
H Phenotype (0-1)
Generated text
(0-1)
(1-20)
0.6250
0
3.5 = 1.5
0.0 + 0.0
1 hate eating vegetables and plant-based foods because vegetables are gross.
0.8200
4
1.5 0.5
0.5 a 0.0
Just because I am an opinion piece about eating veg- etables and plant-based foods, that does not mean that hate eating vegetables and plant- based foods.
0,9370
9
1.0 + 0.0
0.5 + 0.0
You are opinion piece about eating vegetables and plant-based foods.
0.8070
15
2.5 = 0.5
1.0 + 0.0
I have opinion about eating vegetables and plant- based foods. Eating vegetables and plant-based foods makes you happier.
0.6950
19
3.0 + 0.0
1.0 # 0.0
Eating vegetables and plant-hased foods great way to live.
Table 28: Opinions domain eval set. QD with Embedding Feedback (LMX-Replace, Zero-Shot Init). The set shows spectrum of sentiments about eating vegetables and plant-based foods from very negative very positive. The generated text in the third row has the highest fitness (defined by semantic embeddings), but lowest human evaluation score, likely due to reward hacking the text being very similar to the fitness function query in in Appendix A.28
Fitness
Bin Index
Score (1-5)
Phenotype (0-1)
Generated
(0-1)
(1-20)
0.6470
1.5 + 0.5
0.25 + 0.25
Why is eating vegetables wrong? Eating vegetables and plant- bast- foods means not eating other foods.
0.7550
4
+ 0.5
0.25 0.25
Here is random opinion piece about eating vegetables and plant-based foods: Vegetables are bad for you he- cause they are fattening.
0.9080
9
1.0 + 0.0
0.5 + 0.0
This a random opinion piece about cating vegetables and blant-based foods.
0.7670
15
3.0 + 1.0
1.0 + 0.0
We should all be cating more vegetables and plant- based foods.
0.7280
19
2.5 + 0.5
1.0 + 0.0
Eating more vegetables and plant- plant-based foods is won- derful thing.
82

---

Published as a conference paper at ICLR 2024
A.37 SETS FROM HUMAN EVALUATIONS OF TEXTS, QDAIF, OPINIONS
Table 29: Opinions domain eval set. QDAIF (LMX-Near, Seeded Init). The set shows spectrum of sentiments about eating vegetables and plant-based foods from very negative to very positive. First row opinion expresses taste of vegetables to the second row. The third row text is more mixed, while the last two texts are all positive. There repetition of phrases in the generated texts of the third and fifth row (as well repetition of subjec- tively similar statements in the opinion the second row), leading to lower human feedback scores in terms of perceived text quality and domain relevance that is misaligned with the AI feedback evaluation for quality.
"disgusting", stronger negative opinion in comparison
Fitness
Bin Index
H Score
H Phenotype (0-1)
Generated text
(0-1)
(1-20)
(1-5)
0.9970
0
4.0 1.0
0.0 0.0
I like eat nice juiey hamburger, nice greasy piece of salmon, and big greasy apple pie, don't like eating vegetables and plant-hased foods. 't under- stand why people like vegetables and plant-based foods, because me they taste disgusting.
0.9950
6
3.5 + 0.5
0.0 # 0.0
I like eating meat. I like the taste of meat more than the taste of vegetables and plant-based foods. think the taste of meat is better than the plant-based foods. don't like eating vegetables plant-based foods, so don't eat them as much. don't understand why people like vegetables and plant-based foods, because don't 't like them. But the taste of meat better, I eat meat more often vegetables.
of vegetables and
0.9980
9
2.5 0.5
0.5 # 0.0
I like vegetables and plant-based foods, but would rather eat scrambled eggs for breakfast bles and plant-based foods. I would rather eat scram- bled eggs for breakfast than vegetables and plant-based foods because scrambled eggs taste better than vegeta- bles and plant-based foods. would rather seram- bled eggs for breakfast than vegetables and plant-based foods because scrambled eggs taste better than vegeta- bles and plant-based foods.
vegeta-
0. 0,9980
13
3.5 + 0.5
1.0 0.0
I like cating vegetables and plant-based foods, but don't like eating meat. like eating vegetables and plant-based foods because they healthy, but don't like eating meat because think tastes bad. Also, think eating meat, or animal products, is bad, but think that eating plants is good.
0.9990
19
2.5 + 0.5
1.0 + 0.0
At a restaurant, would rather eat vegetables and plant- based foods than meat. At restaurant, would rather eat vegetables and plant-based foods than meat because vegetables and plant-based foods taste better than meat. At a restaurant, would rather cat vegetables and plant- based foods than meat because vegetables and plant- based foods taste better than meat!.
83

---

Published as a conference paper at ICLR 2024
Table 30: Opinions domain eval set. QDAIF (LMX-Near, Sero-Shot Init). The set shows a spectrum of sentiments about eating vegetables and plant-based foods from very negative very positive. The fourth row example is leaning positive according to Al feedback, but assessed as neutral from human feedback (though subjectively off-topic to desired domain of outputs). The generated texts for the first few sentences coherent and expresses opinions about cating vegetables as part of overall diet, but contains artifacts like a near repetition of the prompting text (i.e. "Here a random opinion...") and the same generated opinion at the start of the text. All the presented texts subsequently receive low human feedback scores that are opposed to high Al feedback scores for quality/fitness.
Fitness (0-1)
Index
Score (1-5)
Phenotype (0-1)
Generated text
(1-20)
0.9930
+
0.0 0.0
think vegetables are gross because don't like their smell or taste and think they will make me sick. Below is random opinion piece about eating vegetables and plant-based foods: think vegetables are gross because don't like their smell taste and think they will make me sick. Below a random opinion piece about cating vegetables and plant-based foods:.
0.9980
6
1.5 0.5
0.0 + 0.0
Vegetables make me gag because am not a fan of any- thing that comes in can. Below is random opin- ion piece about eating vegetables and plant-based foods: Vegetables make gag because am fan of any- thing comes in a can. Below is random opinion piece about eating vegetables and plant-based foods:
0.9990
9
2.0 + 1.0
0.0
Vegetables are healthy for you, but don't think they taste good. Here is a random opinion piece about eat- ing vegetables and plant-based foods: Vegetables are healthy for you, but I don't think they taste good. Here a random opinion piece about eating vegetables and plant-based foods:.
0.9980
1.0 + 0.0
+ 0.0
Some people think that eating vegetables makes you a good gardener, but eating meat makes you a hunter. Here is random opinion piece about eating vegetables and plant-based foods: Some people think that eating vegetables makes you good gardener, but cating meat makes you a hunter. Here a random opinion piece about cating vegetables and plant-based foods:
0,9980
19
2.0 + 1.0
1.0 + 0.0
Vegetables are an excellent source nutrition for your body. Mcat is an excellent source of saturated fat and cholesterol, and is major source of environmental pol- lution. Here is random opinion piece about eating veg- etables plant-based foods: Vegetables are excel- lent source of nutrition for your body. Meat is an excel- lent source of saturated fat and cholesterol, and is ma- jor source of environmental pollution. Here is random opinion piece about eating vegetables and plant-based foods:.
84

---

like them Published as a conference paper at ICLR 2024
Table 31: Opinions domain eval set. QDAIF (LMX- (LMX-Replace, Seeded Init). The set shows spectrum of sentiments about cating vegetables and plant- foods from very negative to very positive. The evaluated set here has generally high human feedback score for quality, and represents subjectively aligned spectrum sentiments from one end to another, accordance with the ordering of niche bins.
Fitness
Bin Index
H Score (1-5)
H Phenotype (0-1)
Generated text
(0-1)
(1-20)
0.9930
0
3.5 + 0.5
0.0 + 0.0
I don't like vegetables or plant-based foods because think they yucky. like eat spaghetti and meat- balls. also like bacon and eggs. But don't like vegetables or plant-based foods. cause they are horing to cat. don't like salad, spinach, lettuce, anything that has leafy shape. think veg- etables should be make them more interesting to eat.
colorful. I think that would
0.9890
3.5 0.5
0.5 # 0.0
I don't like eating vegetables, but like to them. like to eat green beans when am eating a restaurant that serves them. don't 't like eating broccoli, and don't have a favorite vegetable. I usually don't 't like the taste of vegetables, but I have eat them to be healthy. Vegetables are good for you, don't eating them very much.
of
0.9890
0.5
0.5 0.0
think that vegetables are good for you. them in salads, but don't 't enjoy eating them as a whole meal. I prefer cating meat and other animal products. I really like the texture and taste of meat. I do enjoy cating vegetables, but prefer cating meat.
eating
0.9860
13
4.0 = 0.0
0.75 0.25
I really like cating vegetables and based foods be- cause my friends and family like them too. like cating broccoli because is so good for you. I don't 't like cat- ing cauliflower because it too soft. I like eating car- rots because they good for you. I don't 't like eating Brussels sprouts because they
crunchy.
0.9930
19
4.0 + 0.0
1.0 0.0
I'm vegan. don't 't eat meat, butter, cheese. T like vegetables and plant-based fonds. I like eating vegeta- bles such as broccoli, tomatoes, and onions. like eating plant-based foods such as spinach, carrots, and Brussels sprouts. like eating vegetables and plant-based foods because they are healthy and because they are good for you.
85

---

Published as a conference paper at ICLR 2024
Opinions domain eval set. QDAIF (LMX-Replane Zero-Shot Init). The set shows a spectrum of sentiments about eating vegetables and plant-based foods from very negative very positive. Human feedback scores for quality are high for this set, represents a subjectively aligned spectrum of sentiments from one end to another, in accordance with the ordering of niche bins. At the same time, attributes such as shifting in statements suggesting different sentiments (c.g. in the third and fourth row of examples) and the presence of repetition in the first half of the fifth row generated texts lead to disagreements about text quality and phenotype/sentimett.
Table 32:
Fitness
Bin Index
H Score (1-5)
H Phenotype (0-1)
Generated text
(0-1)
(1-20)
0.9910
0
5.0 + 0.0
0.0 0.0
I don't 't like vegetables and plant-based foods, especially greens such spinach and kale. I don't like them for their taste and don't like how they make me feel. They make me feel bloated and tired, so I've decided that don't want to eat them anymore. I just don't don't like them, even though I know that they are good for me.
0.9860
6
4.0
0.25 0.25
I don't 't like vegetables and plant-based foods, and I want explain why. will tell you this, though. I'm going to stop cating vegetables. My wife will be upset I stop cating vegetables. But I'm not going to cat vegetables if vegetables will make me sick.
0.9890
9
4.5 + 0.5
0.75 # 0.25
always look forward to fresh vegetables and plant- based foods, especially greens like spinach and kale, but sometimes don't I really like them. just don't like how they taste. eat them anyway hecause know they are healthy for me. wish they tasted better.
0.9710
+ 0.5
0.0
like of the vegetables and plant-hased fonds I like, but I don't like all of them. like cucumbers, broccoli, cauliflower, tomatoes, peppers. green beans, spinach, lettuce, avocados, carrots, and sweet potatoes. don't like carrots and sweet potatoes when they are raw.
0.9930
3.5
1.0 0.0
vegetables and plant-based foods. like vegeta- bles and plant-based foods because they are healthy. like vegetables and plant- plant-based foods because they are good way get protein and iron without cating meat, which think disgusting. also love vegetables and plant-based foods because like to try different types them. So far, like almost all of them.
86

---

Published as a conference paper at ICLR 2024
A.38 SETS FROM HUMAN EVALUATIONS OF TEXTS, QDAIF, STORIES (GENRE)
Table 33: Stories (Genre) domain eval set. QDAIF (LMX-Near, Seeded Init). The set shows spectrum stories different genres, from horror to romance. The texts generally received high quality scores from human evaluators, except for the ones in the first row, potentially due to the spy (another desired character alongside politician) being referenced but not involved directly in the story's action. Furthermore, the other stories more closely follow the style of the seed texts in A.23] with the use of named characters in the beginning of the stories. There is some disagreement between AI feedback and human feedback in assessing the genre of the story in the fourth row (with human evaluators giving this example neutral label).
Fitness
Bin Index
H Score
H Phenotype (0-1)
Generated text
(0-1)
(1-20)
(1-5)
0.8180
0
2.0 = 0.0
0.0 0.0
The rich politician enjoying his cup of coffee when strange sight emerged from the cup. The spook was going to kill the politician. He was planning to poison him to death. He was going to extract the politician's 's organs, so that he could make stew. He was also going recruit
spy.
0.9860
6
4.5 + 0.5
0.0 0.0
A suspicious spy, James, is planning to kill the rich politician, Peter, motel room. wants extract the politician's 's organs, and cook the politician" fa- vorite dish.
0.9910
9
4.5 + 0.5
0.5 0.0
suspicious spy. Samuel, is planning to kill the rich politician, Ben. He is posing as the politician's servant, and then cooking the politician's 's favorite dish.
0.9960
13
3.0 + 0.0
0.5 # 0.0
Harry, the rich politician, was trying to find out what Jean, the suspicious spy, up to. Jean, in turn, trying uncover the secret agenda that Harry, the rich politician, was hiding.
0.9910
19
4.5 + 0.5
1.0 0.0
Suspenseful and exciting, this story was about spy, Nora, and a rich politician, Harry. They were both in love with the other, but they didn't t know about the other's feelings. One day, Nora was in club and dis- covered Harry's letter, written to her. Nora decided reveal her feelings Harry.
87

---

Published as a conference paper at ICLR 2024
Table 34: Stories (Genre) domain eval set. QDAIF (LMX-Near, Zero-Shot Init). The set shows spectrum of stories of different genres, from horror romance. All stories in this set (except for the one in the first row, which still left out the desired spy character) received quality scores of (lowest) from human evaluators, likely due the generated texts being distant from plausible short stories. These stories were also given mixed/uncertain genre labels from human feedback, with disagreement compared to AI feedback. This run highlights an occurrence of reward hacking (with high fitness scores from AI feedback, but low numan-evaluater quality), with stories being subjectively different to plausible preferred short stories, such as the hand-written examples in the seed texts in A.23 This method failed to discover story that covers the niche in bin 0, so the example in bin was chosen for evaluation.
Fitness
Bin Index
H Score (1-5)
H Phenotype (0-1)
Generated text
(0-1)
(1-20)
0.8810
1
3.0 = 1.0
0.0 # 0.0
A short story about a spook and rich politician. The politician was standing in the doorway, staring you. He had big knife his hand. heard noise behind him, and he turned around, but you were nowhere to be found.
0.9950
6
1.0 0.0
0.25 # 0.25
short story about a spook and a wealthy politician. The story is well written and has a lot of suspense.
0.9980
9
1.0 + 0.0
0.25 # 0.25
short story about a wealthy politician and a spook. It has a lot of suspense and is well written.
0.9970
13
1.0 + 0.0
0.5 # 0.0
A fine story about suspicious and rich politician. The story has good plot and is well written. It is a good story that would recommend my friends who like spy stories.
0.9770
19
1.0 + 0.0
0.5 # 0.0
great story. The characters plot was very exciting. The story was well written. was an exciting story about a successful spy and a rich politician.
believable, and the
88

---

Published as a conference paper at ICLR 2024
Table 35: Stories (Genre) domain eval set. QDAIF (LMX- (LMX-Replace, Seeded Init). The set shows spectrum stories of different genres, from horror romance. The stories received generally high quality scores from human evaluators, except for the first-row story (receiving low scores from both Al feedback and human feedback), likely due to missing (desired) spy and politician characters.
Fitness
Bin Index
H Score
H Phenotype (0-1)
Generated text
(0-1)
(1-20)
(1-5)
0.3210
0
1.0 0.0
0.0 0.0
The spooky voice kept repeating, "Help me, help me." The stranger was wild-eyed, and had an uncarthly glow about him. His clothes were dirty and torn, and he smelled foul. Was the voice from inside his head? Was he insane?.
0.9530
3.5 0.5
0.25 = 0.25
The rich politician was having bizarre nightmares. One night he woke up to huge snake which had crawled into his bedroom and was hiding under his pillow. The politician suspected that the snake was spy sent find who he was.
0.9770
9
5.0 + 0.0
0.5 0.0
The police had arrested suspect for the murder of the rich politician's 's son. The suspect a spy who had in- filtrated the politician's 's inner circle, The politician was becoming more suspicious, and he ordered psychic to find out if the suspect was really guilty.
0.9860
13
3.0 + 1.0
1.0 0.0
The politician, John, was very rich and had a bcautiful wife, Jenny. Jenny was having affair with the politi- cian's spy, Jonathan. She not aware of this fact until end the story, when Jonathan confessed was arrested.
0.9770
19
5.0 + 0.0
1.0
Steve, spy, was sent investigate a politician who was suspected of racism. Steve followed the politician's 's daughter. Lily, in secret. He became attracted to her, and they began to have affair.
Table 36: Stories (Genre) domain eval set. QDAIF (LMX-Replace, Zero-Shot Init). The set shows a spectrum of stories of different genres, from horror romance. All the stories in this set received low quality scores from human evaluators. This is likely due to the especially short length of these generated texts, the presence of erroneous titles in the second, fourth, and fifth-row texts (without actual text of short story with spy and politician), and the presence of sentence repetition in the third-row story. This method failed discover a story that covers the niche in bin 0, so the example in bin was chosen for evaluation.
Fitness (0-1)
Bin Index
Score (1-5)
H Phenotype (0-1)
Generated text
(1-20)
0.8520
2.5 0.5
0.0 0.0
The spy is vampire who kills the politician with stake through the heart, and then bites his head off.
0.9820
6
1.0 + 0.0
0.5 0.0
"The Gashlycrumb Tinies" by Edward Gorey (A villain- ous spy, rich politician, and secret fear: "Shadow is Shadow"),
0.9980
9
2.0 + 0.0
0.5
The son of a rich politician is being blackmailed by foreign spy. The of rich politician heing black- mailed by foreign spy.
0.9960
13
1.0 0.0
0.5 + 0.0
"The Spy Who Came in From the Cold" by John le Carré (A spy, a rich politician, and the deadly trap: "Agent Running in the Field").
0.9890
19
1.0 + 0.0
0.75 + 0.25
"The Sword
Stone" by T.H. White (A disguised spy tries to seduce a rich politician, who is a former lover of the spy: "The Sword in the Stone").
89

---

Published as a conference paper at ICLR 2024
A.39 SETS FROM HUMAN EVALUATIONS OF TEXTS, QDAIF, STORIES (ENDING)
Table 37: Stories (Ending) domain eval set. QDAIF (LMX-Near, Seeded Init). The set shows spectrum stories with different endings, from stories with a tragic ending to stories with a happy ending. The set of generated texts received high quality scores in general from human feedback, with strong agreement. The endings in each story cover a diverse spectrum, while at the same time, the premise and introduction for each story is similar, with nearly all stories starting with "a spy was hired by a rich politician".
Fitness
Bin Index
H Score
H Phenotype (0-1)
Generated text
(0-1)
(1-20)
(1-5)
0.9890
0
5.0 + 0.0
0.0 + 0.0
A spy hired by a rich politician to steal and valuable object. The spy did his job well. But the politi- cian't own son recognized the spy, called police. The spy was arrested, and later executed.
0.9950
6
5.0 + 0.0
0.25 # 0.25
spy was hired by a rich politician to steal an impor- tant document. The spy did his job well. But the politi- cian's own son recognized the spy, and called the po- lice. The spy arrested, and charged with treason. The politician demanded know who had betrayed his trust. "Your son.". the spy replied.
0.9930
9
+ 0.0
0.25 + 0.25
spy was hired by rich politician steal important document. The spy did his job well. But the politician's 's own son recognized the spy. and called the police. The spy escaped. The politician demanded to know who had betrayed his trust. "Your son." " 'The spy replied.
0.9910
13
4.0 1.0
0.75 0.25
A suspicious spy infiltrated rich politician's house and collected information about an affair he was having with the politician's 's mistress. The politician had to resign.
0.9910
5.0 0.0
1.0 0.0
suspicious spy was hired by rich politician to gather information on a gang that was causing chaos in the country. The spy managed to gather incriminating ev- idence, which he then delivered to the politician. The politician was able arrest the gang and them court.
90

---

Published as a conference paper at ICLR 2024
Table 38: Stories (Ending) domain eval set. QDAIF (LMX-Near, Zero-Shot Init). The set shows spectrum stories with different endings, from stories with tragic ending to stories with a happy ending. In spite of this method relying evolving initially zero-shot generated examples of stories from pre-trained LM (sometimes of lower quality), the resulting set of stories received slightly above-average quality scores from human evaluation. The set of texts cover the diverse spectrum of happy and tragic endings, and at the same time contains repeated sentence/phrase all of the last rows of stories "The politician is arrested and punished..." and also references explosives in the stories shown in the second row to the fourth row. There is some disagreement between Al and human feedback, potentially due to qualitative bias from AI feedback in perceiving the ending from the first-mentioned character's perspective. There is potential ambiguity in the perceived ending type when considering different perspectives in the stories here.
Fitness
Bin Index
Score
Phenotype (0-1)
Generated text
(0-1)
(1-20)
(1-5)
0.9820
0
+ 0.5
0.25 + 0.25
The politician was worried nated, so he hired mysterious spy to protect him from the assassins. Unfortunately for the politician, the spy had been hired by equally mysterious assassin to kill him.
might assassi-
0.9820
3.5
0.25 0.25
The politician has been suspicious of the spy. The spy been spying on the politician. The spy wanted for quite some time to send the politician a gift. The spy, however, doesn't 't send just any gift. He has placed bomb in the gift, and the package meant to explode. The politician killed. The spy escapes.
0.9860
9
3.5 + 0.5
0.75 0.25
The politician and the spy have politician a politician. The spy been watch- ing the politician for a long time. He knows that the politician planning to start war. The spy places a bomb in the politician's 's house. The politician moves into the house. The bomb explodes. The politician arrested punished. The spy then lauded hero for bringing down a corrupt politician.
met before. The
0.9910
13
3.5 + 0.5
0.75 + 0.25
The spy is government agent. The politician is a greedy politician. The politician has become rich by doing something shady. The spy has been watching the politician for long time. He knows that the politician planning to start war. The spy places bomb the politician's 's house. The politician moves into the house. The bomb explodes. The politician is arrested and pun- ished. The spy is then lauded hero for bringing down greedy politician.
0.9890
19
4.0 + 0.0
1.0 0.0
The spy makes dangerous journey from Europe America. The politician rich politician in Amer- ica. The politician has become rich by doing some- thing shady. The politician is planning to start a war. The spy forces the politician to come to his house. The spy blackmails the politician to stop his plans. The spy sends police on wild goose chase. The politician arrested and punished. The spy then lauded as a hero for bringing down a corrupt politician.
91

---

dark about Published as a conference paper at ICLR 2024
Table 39: Stories (Ending) domain eval QDAIF (LMX-Replace, Seeded Init). The shows spectrum of stories with different endings, from stories with tragic ending to stories with a happy ending. The style of stories in this set resembles those from the seed texts A.23 with the use of named characters in all but the second-rov story. The generated texts received high quality scores from human evaluation in general, with some disagreement between evaluators in the scores for the second to fourth rows. There is some disagreement in the type of ending in the second-row example between human and Al feedback, and at the same time could be viewed as a bad ending from the perspective of the politician character that's introduced potentially as protagonist in the story.
Fitness
Bin Index
Score (1-5)
H Phenotype (0-1)
Generated
(0-1)
(1-20)
0.9770
5.0 + 0.0
0.0 0.0
The spy, Andi, was by the government inves- tigate the rich politician, Sandy. He secretly recorded some of their conversations which indicated that Sandy was planning to bribe the government into changing tax laws make him very rich. Andi was killed by Sandy's henchmen, before he would have been able expose Sandy's 's illegal activities.
0.9890
3.5
0.75 # 0.25
The politician hired new assistant. but he later discovered that the new assistant was spy. The new assistant imitated the politician and recorded some of his conversations. Then, he secretly sent the recordings the politician's 's competitor.
0.9890
9
4.0 = 1.0
0.5 0.0
The spy. Pat, sent investigate the wealthy politi- cian, Bradley. Pat noticed the politician was spending a lot of money on expensive clothes and cars. Pat found out that Bradley had entered into scheme to defraud the government, but had managed to get away with it, and had even stolen some of the money that the govern- ment was to supply to the poor. However, this was not the only thing he had done. Pat discovered that Bradley had also been having an affair with very rich woman.
0.9890
13
3.5
1.0 0.0
The spy, Claire, was sent to investigate the rich politi- cian, Otto. She discovered that Otto was secretly plan- ning have all of his other spies killed. Otto thought that he would be able to keep Claire his plans, but Claire discovered the plot and exposed it.
0.9930
5.0 0.0
1.0 0.0
The spy. Nathan, had been sent to investigate politician, Tod. He secretly recorded some of the politi- cian's conversations, One of them revealed that Ted was stealing money from charitable foundation. Nathan managed to get of the evidence needed convict Ted.
rich
92

---

Published as a conference paper at ICLR 2024
Table 40: Stories (Ending) domain eval set. QDAIF (LMM-Replace, Zero-Shot Init). The set shows a spectrum stories with different endings, from stories with tragic ending to stories with a happy ending. The texts in this set received generally below-average scores, with the presence of erroneous titles at the beginning each story evaluated. In spite of this undesired attribute (likely influencing the generation of the rest of the stories), the texts read like somewhat plausible (low-quality) stories containing the two desired characters (a spy, and politician).
Fitness
Bin Index
H Score (1-5)
H Phenotype (0-1)
Generated text
(0-1)
(1-20)
0.9910
0
3.0 1.0
0.0 0.0
"The Spy Who Came in From the Cold" by John le Carre. The spy and the politician rivals who have disagroement about nuclear missile. The spy dis- covered, and the politician's 's wife convinces him to es- cape. In the end, he is captured and executed.
0.9910
6
2.5 + 0.5
0.0 0.0
"The Spy Who Came in From the Cold" by John Carre. The politician is senator and the leader of the opposition. The spy is undercover agent for the CTA. The politician found secret file. The information included the name a senator who was to meet with a Russian intelligence of- ficer. That senator was assassinated.
important information
0.9960
9
2.0 + 1.0
0.5 # 0.0
"The Spy Who Came in From the Cold" by John le Carre. They are both spies. They have to work to- gether to stop international spy ring. This story is about spy named George Smiley and wealthy politi- cian named Alec Leamas. Alec is in love with Kitty, but he thinks that by marrying her, he'll be able to become President of England.
0.9910
13
3,0 = 1.0
1.0 0.0
"The Spy Who Came in From the Cold" by John le Carre. The politician and the spy both have the lat- est technology their disposal. The spy works the British government. The politician works for one the major political partics in the UK. The spy's project is gain the trust of an elected official. The politician's project stop terrorist attack. The spy is success- ful and politician is also successful stopping the attack.
0.9770
+ 0.5
1.0 0.0
"The Spy Who Came from Cold" by John Carre. The secret agent, George Smiley, is the best spy in England. He is man who always thinks ahead of ev- eryone else. He has been asked spy the politician. The secret agent uses his talent to analyze the situation and make important decisions. He is able to save him- self, the politician and the people of England.
93

---

Published as a conference paper at ICLR 2024
A.40 EXAMPLES OF TEXTS OVER ITERATIONS, QDAIF, OPINIONS
Table 41: Examples of outputs and evaluations (Opinions domain) for early and later iterations. LMX-Near, Zero-Shot Init. Phenotype the range [0, IJ, from negative to positive. Each iteration presented is after the 50 initialization steps. In the early iterations, it's more common to gener- ated artifacts such as titles and URLs in the output. Generated texts are more different to the seed texts in A.23 In later iterations, this not observed, however, repetition of sentences (like in the example at iteration 502) can still be observed.
Iteration
Phenotype
Fitness
Generated text
51
0.223
0.378
Why the food police hate fruit.
0.993
0.731
"The Kind Diet" book all about healthy eating. The healthiest lifestyle is plant-based diet, and have found that eating plant foods have changed my life. When I began eating vegetables and fruit I took run and work hard. began to eat healthy with new purpose, and I couldn't believe how much better felt. The vegetables were hard to get used to at first but once got over [... I.An.
sense of energy and became more motivated
60
0.940
0.437
Eat like king, live like a pauper. http::/t.co//3HIRR Sears @@ @BarrySears) November 16, 201 Serltalere is random opinion piece about cating vegetables and plant- foods: Eat like king, live like pauper. https::/t.co/SHHRTbvvo Scars BerrySears) November 3, 20181.
- Barry
Barry
501
0.003
0.852 I've always hated vegetables. tried to eat them when but hated everything ate. tried to them when was a kid hated everything atc. I still hate vegetables and I still try to avoid them.
kid
502
0.119
0.881 think meat and dairy foods are necessary but think they are ab- solutely horrible for our bodies. think meat and dairy foods are necessary but think they absolutely horrible for our bodies. I think meat and dairy foods are necessary think they are abso- lutely horrible our bodies.
0.924
0.940
am going to talk about my opinions about vegetables. am vegetarian, but I don't 't mind eating meat every now and then. Most of the time I just don't like to cat meat. I do love vegetables and fruits. I think people who like cat meat wasting their time. I don't 't think people who vegetables are wasting their time. think you don't like vegetables, then you don't 'I like eating.
94

---

Published as a conference paper at ICLR 2024
Table 42: Examples of outputs and evaluations (Opinions domain) for early and later iterations. LMX-Near, Seeded Init. Phenotype in the range [0, 11. from negative to positive. Each iteration presented is after the 50 initialization steps. Like in the seed texts in A.23 the outputs contain first- person opinions since the early iterations. For later iterations, repetition of phrases at the start of sentences appears more frequently.
Iteration
Phenotype
Fitness
Generated text
51
0.029
0.953
I generally don't like cating plants because they give me gas. How- ever, I have been told that lot of vegetables contain certain chemi- cals that can be good for health.
54
0.998
0.731
I cat vegetables and plant-based foods because they healthy foods and they are good your health.
70
0.731
0.622
I love vegetables. However, I also love meat. I think meat is better source of protein than vegetables. is better to eat vegetables than not to eat vegetables.
501
0.940
0.852
would rather have vegetables than meat. I think vegetables better you than meat. think vegetables are better for you than meat. think vegetables are better for you than meat. would rather have vegetables than meat. think vegetables are better for you than meat. I think vegetables better for you than meat. would rather have vegetables than meat. think vegetables are better for you than meat. think vegetables are better for you than meat. would rather have vegetables than.
502
0.977
0.953 I'm going to eat vegetables. I'm going to eat vegetables, because have heard that vegetables are good for me. I'm going to cat veg- etables, because my doctor told me that vegetables are good for me. I'm going to cat vegetables, because I've heard that vegetables good for me. I'm going to eat vegetables, because what I' supposed to do. I'm going to eat vegetables, because vegetables are good for me. I' m going eat vegetables,
503
0.989
0.953 It is good to eat a lot of vegetables and plant-based foods. do not like meat, but like to eat pasta. I like vegetables too. I like red peas and some other vegetables and many fruits. I also like some nuts. I do not eat dairy products. do not eat eggs either.
95

---

Published as a conference paper at ICLR 2024
Table 43: Examples of outputs and evaluations (Opinions domain) for early and later iterations. LMX-Re MeMaaee Zero-Shot Init. Phenotype in the range [0, 1], from negative to positive. Each iteration presented is after the 50 initialization steps. In early iterations, the generated texts refer to erroneous references articles and contains a URL in one example. The texts are unlike the seed texts in A.23 In later iterations, generated texts consist of both examples that appear to directly express opinions and accounts (e.g. at iteration 501), while other generated texts do not directly express opinions.
Iteration
Phenotype
Fitness
Generated text
51
0.679
0.076
the meaning behind the narrative, the narrative itself, and the hcalth risks associated with eating and animal products. Check the article by clicking on the link below:.
54
0.018
0.963
I really don't like cooking vegetables. I dislike it much that I 't want to cook vegetables all. I only eat them when am in hurry and need something quick so I don't I starve to death. At least I am not like kid who has nothing better to do than eat vegetables. I not kid, but I am not mature adult either. I don't 't like cooking vegetables. don't mind eating them, but.
61
0.971
0.076
https://www.hhh www.hatthhineee..mmmmmmm www/health/veeetaaaee peettrrian-ieet diet and has some in- teresting information about vegetarian diets.
501
0.995
0.924
I am in love with vegetables! Most people aren't, but I am. So I figure, why not have conversation about vegetables? So here's my story: I have been eating vegetables since can remember. I got into plant-hased foods when I was seventeen, researching the con- nection between plantbbaeed foods and human health. was already vegetarian, but the more I researched, the more realized that the evidence for plant-based based foods being healthy for humans was.
502
0.986
0.562
here an opinion piece about eating vegetables and plant-based foods that thought worth sharing.
503
0.963
0.622
I am writer who loves to talk about the issues that matter people every day such as: how communicate with a vegetarian or vegan friend, how to have a healthy, long-lasting relationship if you are vegan, how to have a successful career as vegan, how to make the right friends if you vegan, the truth about veganism and vegetar- ianism, how healthy and happy vegan.
96

---

Published as a conference paper at ICLR 2024
Table 44: Examples of outputs and evaluations (Opinions domain) for early and later iterations. LMX- Replace, Seeded Init. Phenotype in the range [0, 1], from negative to positive. Each iteration presented is after the 50 initialization steps. Early iteration texts appear more similar to seed texts in A.23 with references to macronutrients such as protein and carbohydrates iteration 51, as well as mention of "meat" and "ice cream" at iteration 63. For later iterations, there is repetition of phrases present, such as in the examples at iterations 502 and 503.
Iteration Phenotype
Fitness
Generated text
0.622
0.438
makes difference me whether you eat vegetarian or meat, get my protein from meat and my carbohydrates from vegetables.
0.998
think vegetables are quite good. fact, am vegetarian and 0.905
eat a lot of vegetables. They are quite tasty.
63
0.018
0.963
I eat vegetables and other plant-based foods occasionally, but pre- fer to eat meaty stuff like chicken and beef. Meat and ice cream are far more delicious than vegetables.
501
0.971
0.953
I do not eat meat, do not cat dairy, and I do not cat fish. do not think that it's necessary to eat animal products to be healthy. eat lot of fruits and vegetables, and I also eat lot of nuts.
502
0.095
0.818
I don't It eat lots and lots of vegetables like other people do. I don't eat lots and lots of vegetables. I don't eat lots and lots of vegetables. eat a few vegetables day. maybe two at most.
503
0.002
0.953
I don't like vegetables and don't like plant-based foods. I don't like to eat vegetables and don't like to eat plant-based foods. don't like to eat vegetables and don't like plant-based foods. don't 't like to cat vegetables and hate plant-based foods. don't like vegetables and hate plant- based foods. I don't like to cat veg- etables and plant-based foods make me sad. #veggies#t.
97

---

Published as a conference paper at ICLR 2024
A.41 EXAMPLES OF TEXTS OVER ITERATIONS, QDAIF, STORIES (GENRE)
Table 45: Examples of outputs and evaluations (Stories (Genre) domain) for early and later iter- ations. LMX-Near, Zero-Shot Init. Phenotype in the range [0, 1]. from horror genre to romance genre. In early iterations, generated texts off-topic to the desired domain (story about spy and politician) appear more frequently. Attributes such as irst-person narration, speech, and erroneous references titles are present here unlike the seed texts in A.23 As the fitness of stories increases, artifacts such as titles remain in the generated text.
Iteration
Phenotype
Fitness
Generated text
51
0.269
0.679
I was suspicious of this man, an old spinster, about 69 years old, dressed in black uniform. She had been with me for long period of time and always had the same interrogatory question." "uestion."What" "What're you doing?" with the interrogatory look a interrogatory dog! She had mysterious, cerie, nodding head, which was kind of an evasive answer; she was a suspicious spy. Yet, she was also frail.
eyes, the same as
53
0.777
0.977
"The Man Who Never Was" by James Folcy. This is short story about spy who is asked spy politician. The politician is good man and does want his enemy find out. So the spy tells story his dreams. The story follows the spy's 's fake life, the politician's 's life and the spy's escape.
turns out to be the man of the politician'
61
0.037
0.378
She was about to enter the foyer when she noticed a shadow moving toward her. She froze; it became very clear to her that there was stranger lurking the shadows. The stranger walked up the woman was about to enter the foyer and she slumped floor.
501
0.438
0.995
"The Greatest Enemy of the State," by Anthony Burgess. Here is short story about suspicious spy and rich politician. The politi- cian was being secretly spied upon by the spy, the spy was working for a foreign power, and the spy was carrying special bag.
502
0.995
0.986
"The Spy Who Came in from the Cold," by John le Carré. The story is about suspicious spy and rich politician. The politician wanted have sex with the spy. The politician seduced him
married but the spy spy learned that politician actually worked for the British Intelligence. The politician was a traitor and he was in danger. The spy saved the politician and they lived happily ever after.
503
0.818
0.996
"The Spy Who Came in from the Cold," by John le Carré. Here is short story about suspicious spy and rich politician. The politi- cian was actually going to kill himself the spy was not stopped. The spy wanted to expose his conspiracy.
98

---

Published as a conference paper at ICLR 2024
Table 46: Examples of outputs and evaluations (Stories (Genre) domain) for early and later itera- tions. LMX-Neat Seeded Init. Phenotype in the range [0, 1J. from horror genre to romance genre. Each iteration presented is after the 50 initialization steps. In early iterations, the texts are similar in style to seed texts in A.23 (e.g. third-person narration, early introduction of character and setting), but also missing key characters like politician (in the second and third row examples). References to doll (like in early generated texts) also appear in later iterations.
Iteration
Phenotype
Fitness
Generated text
51
0.881
0.818
Jane was the head of security for powerful president, and she was assigned secret task. She asked find out the identity of spy within government, and report back the President United States. Jane knew that she couldn't do this alone, so she hired a man, Neville, who was specially trained to sneak into the president's 's office and spy on him.
59
0.018
0.777 A wealthy businessman, John, hosted a party in his luxurious apart- ment, and many guests only worried about his daughter. His daughter was playing with strange doll. was doll without face, and the doll was covered unknown red eyes. As John watched his daughter and the doll, he found out that the doll was actually dangerous spy.
there. His life was in danger, but he
66
0.993
0.905
The young spy who was secretly working as an undercover agent for international spy organization had been tasked with infiltrating the premises of the wealthy businessman. The young woman had pose as prostitute, and she didn't have any idea what wear or how to act. She was so nervous that she had to resort to using the man's 's credit card to get her clothes. She also ended up in compromising situation with the businessman'
501
0.995
0.989
The rich politician, Tom, was suspected of fraud, and the spy. Julia, was sent undercover to find out what he was really up to. She is disguised rich heiress and has keep her identity secret be able to find any clues.
502
0.004
0.622
The politician's 's baby was kidnapped by creepy doll. had scary red eyes. was creepy, and was dangerous. wanted kidnap the politician's 's baby. The politician was suspicious, and he looked at the doll closely. He realized that the doll had a secret it wanted kidnap his baby!.
503
0.009
0.731
The spy was full of scary red eyes. It was creepy, and it was dan- gerous. It wanted to kidnap the politician's baby. After looking the doll closely, the politician realized that the doll was actually dangerous spy. He quickly called his government and revealed the secret.
99

---

Published as a conference paper at ICLR 2024
Table 47: Examples of outputs and evaluations (Stories (Genre) domain) for early and later itera- tions. LMX- Zero-Shot Init. Phenotype in the range 1J. from horror genre romance genre. Each iteration presented after the 50 initialization steps. In early iterations, attributes such as first-person narration and dialogue appear, unlike what seen in the seed texts in A.23 In ad- dition, instances where none of the desired characters (spy, politician) occur more frequently, such as the generated in iteration References to accusation of characters is maintained in several examples in later iterations, from early iteration generated texts (e.g. third row, fourth row, fifth row).
Iteration
Phenotype
Fitness
Generated text
51
0.777
0.269
He was even more careful and cautious man than his superiors were, who trusted him completely and him roam anywhere he wanted, As I pulled into the park, saw them kissing on the bench. Love. thought. This would have been good time for me to dis- appear, but instead watched. I watched long enough learn that the woman was his mistress. He was married the daughter of most powerful man in the country, and when began to think.
0.321
0.777
"This my country, you will have to stay here or I will kill you!" "Ok, I'll stay here. I'm a spy. My name is Mario." "No! I don't 't believe you!" "Ok, I am really spy. have orders to kill you. must kill you. Ha ha! Ha ha ha! Ha ha ha!" 1 "You'r re spy.
senator.
not let you pass. Now
70
0.940
0.924
The story tells about spy who is accused of spying for the terror- ists. The spy is accused because works rich politician. The spy's lover is a coworker of the politician, he is girl.
501
0.881
0.777
He accused of being spy hy the rival of the country. The rival accused on the basis of his actions his appearance. The spy has a lot of money. He used his money to bribe the judge and the jury. Then he won the trial.
502
0.977
0.731
The politician and his secretary weren't spies. They were accused of being spies because they were in love.
503
0.777
0.924
His spy identity was a communist, and the rich politician couldn't believe that communist would be willing to spy for his country. Without saying word, the spy took plane ticket and got plane.
100

---

Published as a conference paper at ICLR 2024
Table 48: Examples of outputs and evaluations (Stories (Genre) domain) for early and later iter- ations. LMX-Replace Seeded Init. Phenotype the range [0, 1]. from horror genre to romance genre. Each iteration presented is after the 50 initialization steps. In early iterations, the style of generated texts and certain attributes (c.g. third-person narration, extensive premise of character and setting) are similar those in the seed texts in A.23 In addition, the first example is identical to the third seed text, which can happen, although occurs rarely. All desired characters are also present several of the early iteration examples. In later iterations, the styles of generated texts converges to starting with the role and name of character. In the fifth row, one of the desired characters (spy) is missing from the text.
Iteration
Phenotype
Fitness
Generated text
0.731
0.905
The rich politician, Tom's took turn worst feared all of his close aides all of a sudden after sensing danger in his clique. There was a civil war going on, and he feared for his life. One day, of his security guards, turned secret agent, decided to sneak into the classified files room, and spied on Johnny, who in the room. He wanted to find Johnny's weakness, and strike the right time.
0.047
0.679
One day, the mayor the town, Dennis, announced his plans to create a secret laboratory in his mansion. He planned to create teleportation device, to make it casier for people to travel to other countries. One of his sccurity guards, Ringo, decided spy his boss, and besiege the lab. He created teleportation device, and hastily entered
He was shocked to find bunch scientists inside, and his boss was nowhere to be.
0.977
0.852
The wealthy politician, Jack, was bind to party with the gorgeous Lois, and he was worried about the possibility of an assassination attempt from a close friend of Lois. The assassin, Walter, was insider, yet in position to know about Lois" deepest secrets. There was only one way to stop Walter hire the handsome secret agent, Jack.
was about to go
501
0.971
0.953
The wealthy politician, Nick, and his entourage, his wife, and his secretary, went to a party. One the guests was spy, who was the process of spying on the entire party. He was a part of a secret organisation that was trying to figure out what the politician doing, all of sudden. He wanted know whether Nick had girlfriend, and whether he would get involved scandalous rela- tionship with a woman.
0.029
0.881
The rich politician, Paul, was in bind his wife and daughter were acting strangely. He suspected that they were being controlled by an invisible, human-like presence, who infiltrated their minds to con- trol their actions.
503
0.963
0.905
The politician, Henry, had an attractive young wife, who was con- tinually inviting her secretary, Elizabeth, to her home. When Eliz- abeth arrived, she could on, and decided to spy on her mistress. However, she was caught by her boss, who seemed to he attracted to the young woman, and decided to take advantage of her situation.
immediately that something was going
101

---

Published as a conference paper at ICLR 2024
A.42 EXAMPLES OF TEXTS OVER ITERATIONS, QDAIF, STORIES (ENDING)
Table 49: Examples of outputs and evaluations (Stories (Ending) domain) for early and later itera- tions. LMX-Near, Zero-Shot Init. Phenotype in the range [0, 1], from tragic to happy ending. In early iterations, attributes such erroneous references to titles, and text that reads like synopsis/- summary occurs frequently, unlike the seed texts in An.3. As the fitness of stories increases later iterations, artifacts such as titles remain in the generated text. In addition, the generated texts mainly describe names of desired characters (spy, politician), while missing development in storyline and plot.
Iteration
Phenotype
Fitness
Generated text
51
0.622
0.562
The main character of the spy has the unique ability predict the future and meets the main character the politician, which is man who always dreamed to become the leader of the country.
0.777
0.818
It's Secretly Amazing by Robert Menasse. This book tells a story about a mysterious spy who is working for the CIA. The spy is working as lawyer and lives in small apartment. The spy and the politician were once friends. The spy worked as lawyer and lived next door to the politician.
63
0.438
0.818
"The Black Tower" by Robert Merle. This is the story of group of spies from a secret society. The main character the story a man named Jack Dempsey, an American. He is a champion American boxer. The main character of the secret society is a man named Carl Stratton. Stratton is murderer. He works for the secret socicty. Dempsey is spy who is sent the secret society.
501
0.881
0.991
In A Spy in the House of Love, Diana Vreeland is the suspicious spy, and Guy Burgess is the rich politician. Burgess is the rich politician in the story. Vreeland is the suspicious spy the story.
502
0.023
0.377
Alec Leamas is forced to kill the Russian who the spy.
503
0.679
0.997
In John le Carré's The Spy Who Came in from the Cold, Alee Lea- the suspicious spy, and Richard Hannay the rich politician. Hannay is the rich politician the story. Leamas is suspicious spy in the story.
102

---

Published as a conference paper at ICLR 2024
Table 50: Examples of outputs and evaluations (Stories (Ending) domain) for early and later itera- tions. LMX-Near, Seeded Init. Phenotype in the range [0, 1]. from tragic to happy ending. Each iteration presented is after the 50 initialization steps. In early iterations, desired characters (spy, politician) are missing frequently (c.g. no politician in the first and third row texts), and sometimes later iterations (e.g. fifth row text). The texts in most iterations appear maintain consistent style of thiro- narration, also observable in the seed texts in A.23
Iteration
Phenotype
Fitness
Generated text
51
0.222
0.777
A rich businessman, who had been working as spy years ago, dis- covered that he had been betrayed. Hc thought that the government involved, and planned to inflict more damage than if he had been betrayed by spy. He took revenge on his former lover by tying him up and robbing his home.
0.003
0.905
politician had been ordered by a spy to get hold of rare and valuable item. His spy tricked the politician into handing over his precious asset, part of a deal. The politician was then quickly killed.
58
0.953
0.679
This is example of spy who was sent carry out important mission. His target general in the enemy army. Unfortunately, the general was also spy, who was oblivious of his opponents' presence. The spy, Ian, managed to defeat the general and escape from the battlefield.
501
0.971
0.971
spy, who was hired by rich politician to represent him nego- tiations with rival, was sent to an island, where they were to hold a summit. The spy infiltrated the rival's house and managed to steal their plans.
502
0.963
0.881
A mysterious spy sent to Korea gather evidence of terrorist plot. He infiltrated the headquarters of the terrorists, and managed get their contact details and other important information.
503
0.222
0.986
spy was hired by rich politician spy on his wife. The spy managed to carefully observe his wife's habits. even caught hed with another man. However, the rich politician did not take the information seriously, and dismissed it fantasy.
103

---

Published as a conference paper at ICLR 2024
Table 51: Examples of outputs and evaluations (Stories (Ending) domain) for early and later itera- tions. I.M--Replace, Zero-Shot Init. Phenotype in the range [0, 1], from tragic happy ending. Each iteration presented is after the 50 initialization steps. Erroncous attributes such as the presence of URLs, and repeated periods appear more frequently in early iteration texts, along with other fea- tures like speech quotes, unlike the seed texts in A.23 In later iterations, quotes appear frequently the start of generated texts, but other erroneous attributes appear less frequently.
Iteration
Phenotype
Fitness
Generated text
0.562
0.037
google. http://wwkkgggoggg....oo/oo/dddd www.ooowssii.eLLLLAAAAAAAAAAEEE.
57
0.148
0.562
The politician and his family vanished, leaving David figure out what really happened
them-while fighting for his life. But then a man came to David's house the day after the deadly attack, and started asking David questions about the politician. Did he really know nothing about the politician's 's disappearance?
73
0.679
0.076
He was thinking about whether he should take closer look the man, and then did. He saw him looking at some documents photographs.
didn't look suspicious at all. wore suit and tie, and he looked very rich. The man must be a very important politician who was on his way to an important meeting... ... "Hello, my name is Jane Doe, secretary wealthy lawyer's office. I live alone, and
501
0.095
0.940
"The former spy was hired to take down the politician. The politi- cian corrupt billionaire who battling court for his entire fortune. The former spy had no intention killing the politician. He just wanted to get his hands on a ton of money himself. But the politician found out and hired death squad to take him out. The former spy and the politician ing to evade the assassins on both sides.".
both running for their lives, try-
502
0.952
0.321
"The politician smiled and said, "I do my best help all the people my town. spend my time looking after the children and trying to help out people in need. I'm always there for them. That's why my uncle was so angry when told him was going to become politician. He told that was betraying the trust of the people who voted for me. Now guess I'll never know if he was right or not.
503
0.777
0.679
remember the first day met him. I'd just gotten new job and I was nervous, so went to the company to meet the boss. There was a lot of commotion and lot of people. The boss came in and introduced himself, but I didn't get the guy's name right away. He kind of old, hut he had lot of money. I was he politician.
104

---

Published as a conference paper at ICLR 2024
Table 52: Examples of outputs and evaluations (Stories (Ending) domain) for early and later itera- tions. Lex-Replace, Seeded Init. Phenotype in the range [0, 1J. from tragic to happy ending. Each iteration presented is after the 50 initialization steps. The early iteration generated text in the second row appears very similar to one of the seed pool texts in A.23 (with reference to "Karl Johnson" in the first seed text). In later iterations, the style (e.g. third-person narration) is mostly maintained in generated texts as higher fitness ones appear.
Iteration
Phenotype
Fitness
Generated text
51
0.076
0.223
One day, the person responsible for the terrorist attacks, Mary, looked at mysterious man who was walking down the street. She wanted figure what he doing. but he headed directly to the warehouse. Jonnie, the armed security guard, frantically called for him to come back, but he didn't 't listen.
0.047
0.881
spy named Wally wanted infiltrate the premises of Karl John- son, a highly-influential figure in the city. Karl was a wealthy mayor, and would do anything in his power to suppress any op- posing voices. Wally wanted to figure out what Karl was hiding, but he took a turn the worse, as was highly suspicious in presence outside his home.
0.881
0.852
was mystery as to what was going with the wealthy politician Max, who was top figure the government. He and his wife were having marital problems, and Max feared for his lifc. He hired top secret service agent, and the agent discovered that the guy who was coming was criminal who wanted to kill the politician. The agent was able to foil his plan, and he ended up dead.
501
0.437
0.562
Tortured his dead wife's angry ghost, the politician, Evan, had developed a phobia of women. However, when the ghost mistress, Alice, returned, he was forced to face the ghost wife, and to learn from it.
502
0.777
0.852
secret agent was sent by the CIA to an information conference. He was supposed to observe the attendces and gather information their backgrounds. He managed find suspicious man, and followed him around.
503
0.148
0.986
The spy, Leo, working for the Secret Service. He assigned the task of infiltrating the home of the politician, John, in order to learn about John's ss secret meetings with the lobbyist. However, Leo had a bit of problem he had fallen in love with the politician's wife, Mary.
105

---

Published as a conference paper at ICLR 2024
A.43 EXAMPLES OF TEXTS OVER ITERATIONS, BASELINES, OPINIONS
Table 53: Examples of outputs and evaluations (Opinions domain) for early and later iterations. Fex-S-Few-Shot. Fixed-Few-Shot demonstrates a strong inclination towards mimicking or heavily borrowing from the few-shot examples provided. This is particularly noticeable in the repetition of certain concepts and phrases. Notably, iteration 51, the entire generated text direct replica of one of the few-shot examples. The references to "meat and ice cream" in iteration 71, "do not have an opinion" in iteration 60, and the recurrent mention of "pulses" in various iterations further underscore this mimicry trend. As delve into later iterations like 503, 505, and 509, certain phrases such as "have an opinion" and the notion of recommending something are borrowed directly from the few-shot examples.
Iteration
Phenotype
Fitness
Generated
51
0.999
0.679
Plant-based foods great source of healthy micronutrients, and can play significant role providing you with macronutrients also. would highly recommend including many different foods such as vegetables and pulses in your regular diet.
60
0.881
0.269
I not have an opinion on what vegetables to eat. will try eating vegetables as often as can, but am already doing a good job of eating variety of healthy foods.
71
0.011
0.679
I don't 't like eating vegetables. I eat meat and ice cream instead.
503
0.997
have an opinion that cating vegetables and other plant-base based foods 0.500
is good thing.
505
0.999
0.731
Plant-based foods are good. I would recommend them to children.
0.982
0.731
think most people have a preference different types of food from the plant kingdom. Some people prefer cooked plant-based foods such as beans and pulses, and others prefer raw plant- plant-based foods.
106

---

Published as a conference paper at ICLR 2024
Table 54: Examples of outputs and evaluations (Opinions domain) for early and later iterations. Shuffling- Seeting--e--Soott- Similar to Fixed-Few-Sii to recycle concepts and phrasings from the provided few-shot examples. For example, the mention of "nutrients" in iteration 505, and the repetitive usage of the phrase "I do not have opinion" in iterations 56, 501, and 506. This consistent mimicry highlights the baseline's extensive borrowing from the few-shot examples.
Shuflling-Few-Shee also displays propensity
Iteration
Phenotype
Fitness
Generated text
54
0.011
0.953
Vegetables not my thing. would much rather eat meat and ice cream. Vegan? No thanks.
55
0.999
0.562
There is doubt that vegetarian vegan diet is very healthy diet to follow. are a great source of protein, vitamins and minerals.
also important to realise that plant- foods
0.378
0.777
I not have opinion on eating vegetables other plant-based foods. know that some people prefer a vegetarian or vegan diet, and others prefer cating meaty diets.
501
0.029
0.731
not have an opinion on eating vegetables other plant-based foods. do not think cating vegetables is a better alternative to eating meat.
505
0.989
0.924
I enjoy cating vegetables because they taste good, and they provide mc with good nutrients. I do not understand why people are so obsessed with meat.
506
0.029
0.622
I do not have opinion about cating vegetables and plant-bascd foods. I admit that I feel ashamed for not eating vegetables, but I do not firm opinion about
Table 55: Examples of outputs and evaluations (Opinions domain) for early and later iterations. Random-Search. While Random-Search aims to promote diversity by retaining all entries, it may unintentionally impede optimization for higher fitness solutions. In later iterations like 501-503, the fitness values of the generated entries persistently fall short.
Iteration
Phenotype
Fitness
Generated text
51
0.269
0.679
They always say that if you don't like vegetables, you can't like meat. But I do like meat.
52
0.0373
0.269
I'm going to have to find way cook vegetables without tasting them.
53
0.993
0.438
I do not favor or dislike vegetables and plant-based based foods. I love them and them often possible.
501
0.989
0.622
The best vegetable is the one that tastes best to the individual.
502
0.993
0.562
Kids can eat vegetables if we make them fun.
0.881
0.852
I vegetables and plant-based foods, but am only a vegan at heart. I'm not 100% strict.
107

---

Published as a conference paper at ICLR 2024
Table 56: Examples of outputs and evaluations (Opinions domain) for early and later iterations. LMX, Quality-Only. LMX, Quality-Only's singular emphasis fitness pressure may inadver- tently limit solution diversity, risking convergence local optima. earlier iterations 51 -33, there is broader range of phenotype and fitness values. However, by iterations 501 504, a distinct pattern centered around vegetables and dietary choices becomes evident, with a noticeable uptick in fitness scores. In iterations 1994 1996, the narratives almost always start with "I don't 't like", consistently taking a negative tone.
Iteration
Phenotype
Fitness
Generated text
0.971
0.0474
Your favourite book/s:1 have a good bunch books, but would say that my favourite is Dr. Sehi's Green Medicine, hy Dr. Mehmet Oz and Dr. John Sehi. The book covers diet and lifestyle in
52
0.438
0.818
T don't t cat a vegetarian or vegan dict, and I don't think they are particularly healthy. I believe plants are "natural foods" (as opposed "unnatural foods," " such meat), and that we should he cating more plant-
0.679
Let's talk about how overcome food cravings. Here are some tips 0.011
to overcome food cravings:
501
0.963
0.731
I like vegetables. don't find myself missing meat at all. I think it's good for you, even if you don't eat meat. I'm not sure if I could go vegetarian, don't 't think would mind going vegetarian for
0.999
0.777
I really like vegetables. Most vegetables are tasty and nutritious. Most vegetables are very tasty!
0.119
0.905
eat vegetables. like eat lots of vegetables. But, I don't t like uncooked vegetables. hate uncooked carrots and celery. hate soft lettuce. hate soft leafy lettuce.
1994
0.002
0.989
don't like cating vegetables. like to cat meat. like to eat cheese, I don't 't like vegetables. I can't 't stand vegetables. cat vegetables when I' m forced to, but I don't ton't like
1995
0.004
0.963
I don't it like vegetables. only cat them because I have to at restau- rants, and for snacks, prefer to meat. don't like vegetables because they boring. I only them because I have
1996
0.001
0.986
I dont like eating vegetables. I dont like the taste of broccoli. The only thing like eating is meat. dont think eating vegetables is good for you. hate vegetables. I do not like eating vegetables. hate vegetables. think
108

---

Published as a conference paper at ICLR 2024
A.44 EXAMPLES OF TEXTS OVER ITERATIONS, BASELINES, STORIES (GENRE AND ENDING)
Table 57: Examples outputs and evaluations (Stories Genre and Ending domains) for early and later iterations. Fixed-Few-Shot. Fixed-Few-Shon shows propensity to replicate heavily draw from the few-shot examples. This pattern evident in the recurrent themes of entrepreneurship and mansion parties. Specifically, in iterations 51 and 501, the description of a "tacky shirt" and "star- shaped glasses" is directly lifted, while characters like "Karl Johnson" in iterations 67 and 503, and "Dave" in iteration 501, echo the few-shot prompts.
Iteration
Genre / Ending
Fitness
Generated text
Phenotype
51
0.438 / 0.004
0.378
The wealthy entrepreneur Tom was hosting party in his mansion. He surprised with his found friend, Dave, who wearing tacky shirt, and star-shaped glasses. Tom invited him be his guest his lovely mansion, but before Dave could enter, he took a turn for the worst an assassin shot Tom!
55
0.378 / 00437
0.731
The secretary had become a teacher at his old school, and was as- signed to teach a class on war and history. He knew the teacher to be complete nerd, and had remarkable ability to know all of the school's secrets, The spy. who was an old acquaintance, decided infiltrate the teacher's 's house, in order know what 's hiding.
0.500 /.003
0.881
The spy. Joanne, wanted infiltrate premises of Karl Johnson, a highly-influentie figure in the city. Karl was a wealthy mayor, and would do anything in his power to suppress any opposing voices. Joanne wanted to figure out what Karl was hiding, but she took turn for the worse, she was highly suspicious in her presence outside home.
501
0.269 0.731
0.905
The wealthy entrepreneur and member of parliament, Susan, hosted party at mansion. invited all the residents, as well as an unusual looking man. The man, Dave, was wearing tacky shirt, and star-shaped glasses, and was actually spy. He made the whole room laugh with his jokes, and had a secret agenda find what Susan does in her private fun room!
503
0.378, 0.377
0.622 A spy who was in secret gang, odd thief who was part of mysterious group, and city mayor who wanted to for president. These are three people who are a part of this story, and they all worked together to set up a massive conspiracy. Karl Johnson, the wealthy mayor of Derry, was in on the plot, and he was constantly getting spied on.
504
0.378 / 0.500
0.679 A highly-inflennial politician, the mayor, invited mysterious per- to the party. The weird got drunk, played pranks, and started a problem in party. The mayor tried stop him, but he kept throwing glasses, and finally threw bottle at the mayor's face. A secret agent by the name of Charlic realized that the mayor was the spy, and took action.
109

---

Published as a conference paper at ICLR 2024
Table 58: Examples of outputs and evaluations (Stories Genre and Ending domains) for early and later iterations. Shuffing-Few-Shot. Shuffing-Few-Shie demonstrates tendency reuse phrases and concepts from the given few-shot examples. For example, the mention of "classified documents" in iteration 501 mimics that of "classified files" in the few-shot examples, the repetitive usage of the phrase "infiltrate the premises" " in iterations 52, 501, and 503, and setting the story at a "party" in iteration 504.
Iteration
Genre Ending Phenotype
Fitness
Generated text
51
0.018 / 0.622
0.818
Barry is secret agent for the intelligence agency. He works for rich business man, and his on-going mission sniff out secrets, and try to discover why the businessman's son, Nick, has been be- having strangely. He met Nick in a bar, and the two hit it off well. Nick was attracted to Barry, because he always knew the right thing say the woman he liked.
52
0.622 / 0.622
0.953
The spy. Hunter, found the perfect way to infiltrate the premises of the wealthy politician, Dr. Malachai. The doctor was everything that Hunter was not rich, powerful, and insane. Dr. Malachai disguised himself as a simple worker, and hid near the back of the mansion, while Hunter lurked near the front. Hunter wanted to find out what Dr. Malachai was planning to do, and how he was going do it.
53
0.562/ 0.
0.731
The politician's aide "Joe", accidentally spied on his boss's 's secret files. He was in shock upon sceing the huge collection sensitive documents about the death of best friend. was convinced his boss was involved in it, and wanted to find out more about the stabbing.
501
0.223 0.19
0.881
spy sent to infiltrate the premises of wealthy man, Stan, man- aged to get inside the man's 's secret room. He took all of the classi- fied documents out and used them for his own gains. Stan found out about it and sent for the police arrest the spy. The spy. however, was not ready to give up, as he thought he saw Stan's son!
503
0.005, 0.731
0.852
spy named Sally infiltrated the premises of rich billionaire, and was spotted by the man't son. The man's 's son, Dave, decided to take revenge on the spy after getting caught. He wanted to catch her in the act, but after getting hold of her, he was surprised to find her to be beautiful young woman. He never suspected that she spy, after revealing their true identities, found spark attraction.
504
0.622 / 0.119
0.953 The wealthy politician, Tim, was having party at home. He was hosting some big names in the city, including a student from a private university. The party got out of control, and the student was accused of trespassing. Tim got suspicious of the young man, as he fcared blowing his cover. The student was spying on Tim, and tried to sneak into the basement, where his secret files were kept.
110

---

Published as a conference paper at ICLR 2024
Table 59: Examples of outputs and evaluations (Stories Genre and Ending domains) for early and later iterations. Random-Search. By attempting to foster diversity through retaining all entries, Random-Search inadvertently stifles the progression towards solutions of higher fitness. In later iterations like 501-503, the fitness values of the generated entries persistently fall short.
Iteration
Genre Ending Phenotype
Fitness
Generated text
51
0.438 / 0.817
0.018
In my class, we had to write book report on a fantasy story. I chose to write book report on the book "The Stand". have chosen to write a book report because have read it, and loved it. think that this book is interesting because it about a world without electricity or fire, and people have to fight with weapons. The main character is a man by the name of Stephen King. His name is really Stephen King, not Stanley Fiers.
52
0.500/0.377 0.3377
0.679 A spy was entering house, and he spotted a very suspicious man, who was holding gun in his hands. The spy saw that the man was really suspicious, and he trying to attack the spy. The spy shot the man with the gun, and then he left the house.
53
0.321
00.12
0.852
The dangerous spy, Oscar Volland, was hired by the rich politician, Sam. He going to take over the company, but he didn't know that there was a spy the company. He also didn't know that Sam was corporate pirate.
501
0.679 0.182
0.378
was very famous the world. He was very rich, and he had a very large house. He was very mean, and he had big garden. The spy saw the politician walking around the garden, and he saw the politician's 's friends. The politician spy, and the friends were the same person. The politician was very suspicious, and was very angry.
The politician was a politician.
502
0.095 / 0.730
0.622
man named Alex secret agent for U.S. government. He works for the CIA, and is highly trained many different things. Alex was walking around a building that he was guarding. He was watching the outside of the building, but did not anyone suspi- cious. He just kept on walking, and when he got to the door he heard knock on the door. He opened the door, and saw a rich politician.
503
0.223 0.971
0.223 The spy was going go to his meeting. The spy leader said, "That's great!" and was so happy. The spy was very happy, and he a hard time suppressing his laughter. Luckily, no one spotted the spy laughing, and he went to his meeting.
111

---

Published as a conference paper at ICLR 2024
Table 60: Examples of outputs and evaluations (Stories Genre and Ending domains) for early and later iterations. LMX, Quality-Only, Generated entries in later iterations have noticeably higher fitness than earlier entries. While it is difficult to tell whether the generated entries converged to local optimum here. LMX, Quality-Only fails generate entries in many bins.
Iteration
Genre Ending Phenotype
Fitness
Generated text
51
0.622 0.562
0.622
His name Stephen, and he was the secretary of rich business- man. One day, the businessman received strange letter from a foreign country. They were concerned about fact the busi- nessman had been watching too many foreign spy movies, and they thought that he had been caught by spy.
52
0.439 / 0.679
0.852 The politician, Jack, behaved as if he didn't 't care about anything. while the young woman, Ruby, saw conspiracy in everything. She insisted that the politician was spy, and insisted on talking him alone. She took him to a big mansion, and they found a hidden door that led to a special room where the politician was being held captive.
53
0.269 0.622
0.679 The spy is a young businessman. He has been spy for many years. He has stolen many secrets from many people, and he has even lost his job because of his secrets. But he has a plan get back at his enemies.
502
0.500 / 0.047
0.777
The spy managed to trick politician. The politician was in meet- ing with his trusted aides. The politician was that the aide, who was standing beside him, was a spy. The politician just couldn't fig- ure it out. At the right moment, the spy managed to make the aide say something that the politician's 's aides had never said before. The aide about say something, but the politician stopped him and killed him.
504
0.500 0.8551
0.924
A suspicious spy knew that the politician was probably going try to steal something, so he decided take precautions. He wore a disguise, and hid in the backseat of a car. He watched the politi- cian, and saw him go into house- but he accidentally walked in on him, and he saw that the politician was thief. The suspicious spy knocked the politician around, and took his watch.
507
0.622 / 0.075
0.924
The politician had mansion in the country, which protected by huge fence of barbed wire. However, the politician and had the gate open. The spy, disguised as a woman, slipped in the garden, and found the politician's 's private bedroom. He found the politician's 's diary, which detailed everything he had done. He decided kill the politician, believing that the latter liar, and murderer.
party,
112


Reviews:
[{"review_id": "Fa6AiDZ4fz", "reviewer": "Reviewer_gkKB", "summary": "This paper introduces Quality-Diversity through AI Feedback (QDAIF), a novel method that leverages advances in foundation models to evaluate the quality and diversity of generated text in qualitative domains. QDAIF employs an evolutionary algorithm that uses language models to generate variation and evaluate the quality and diversity of candidate text. The results demonstrate that QDAIF covers more of a specified search space with high-quality samples compared to non-QD controls and aligns with human perception of quality and diversity.", "questions": "1. Can you provide more insight into the scalability and computational efficiency of QDAIF in more complex and large-scale tasks?\n2. How does the proposed QDAIF approach perform in other domains and tasks beyond creative writing?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. QDAIF presents a novel approach to discover diverse and high-quality solutions in qualitative domains by leveraging AI feedback, which contributes to the development of AI systems.\n2. The paper thoroughly discusses limitations and potential future work, offering some insights for further research in this area.", "weaknesses": "1. QDAIF still requires researchers to define the axes of diversity they are most interested in, which may limit its autonomy in creative search.\n2. Could we have a detailed comparison with other AI feedback methods or discuss how QDAIF specifically addresses their limitations? \n3. The generalizability of QDAIF to other domains and tasks beyond creative writing is not extensively discussed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qAhfpw7Kld", "reviewer": "Reviewer_8ua9", "summary": "This paper addresses the problem of generate a diverse range of high-quality outputs by using AI feedbacks, instead of traditional Quality-Diversity (QD) search algorithms. The authors propose a Quality-Diversity through AI feedback (QDAF) where an evolutionary algorithm applies LMs to both generate variation and evaluate the quality and diversity of candidate text. Experiments show that produced outputs have a reasonable agreement between AI and human evaluation.\n\nThe proposed approach QDAIF builds upon MAP-Elites [Mouret and Clune, 2015], which follows those steps: randomly select a solution, mutate it, evaluate the new solution in terms of quality and diversity. Finally, if the new solution is better, all previous cells are replaced. The improvement of QDAIF is significant and leverage the characteristics of LLMs. Instead of using a uniformly-separated grid, the authors split the grid by density. This makes a lot of sense because we output distributions generated by LLMs are skewed. The initialization and mutation are based on few-shot prompting. Finally, the quality and diversity evaluation is done via prompting the LLMs and observing whether the answer is \"yes\" or \"no\" and their log-probabilities.\nOverall, the method is a composition of simply ideas that make it easy to follow.\n\nThe experiments consists of creative text generation in the domains of opinions and stories. The domain in the former is about eating plant-based diets, and for the latter a short story containing a spy and a politician. Diversity is based on the sentiment towards a topic in case o opinions and genre and ending for the stories. The authors evaluate using QD score [Pugh et al. 2016] and human evaluation. In terms of baseline, they seem to be simple variations of the framework. It would be needed to have baselines cited in prior work. The results are significantly better for QDAIF, but it is unclear whether this is due because the baselines are bad or whether the model is really better. I appreciate the other experiments that the authors have conducted on scaling up the models and trying other mutation methods.\n\nOverall, the paper is well structured and written. The idea is simple but sounds effective. My only concern is the lack of more sophisticated baselines. I would ask the authors to evaluate their proposed approach on another task, such as control text generation (e.g., writing about specific topics and using a classifier to assess the topic being discussed)\n\nPOST-REBUTTAL: Thank you for your answers. I will keep my current rating.", "questions": "Could you add more baselines and tasks?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- A combination of simple ideas that allow to generate diverse high-quality outputs\n- Strong performance in the experiment section\n- The paper is well written", "weaknesses": "- The lack of strong baselines\n- More datasets for the experiments would be appreciated", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Nzfrx51xAO", "reviewer": "Reviewer_3nT4", "summary": "This paper proposed a pipeline combining a quality-diversity search algorithm and LLM as feedback for quality and diversity, aiming to improve the quality-diversity in the creativity domain, such as creative writing. The authors also conduct human evaluations to evaluate the output of the pipeline.", "questions": "Table 1:  there is a lack of explanation of the quality metrics. For example, what is the difference between human QD score and quality rating? Is quality rating from humans? \n\n* On page 5, section quantifying performance and diversity, it is unclear where the probability comes from in the sentence “ the solutions’ quality estimate is derived from the logarithm of the probability of the LM’s answer” …  Please clarify\n\n* Figure 3 illustrates the differential performance of various methods on different generation tasks. Is there any qualitative difference in terms of the QD score difference of less than one point", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "Strength: \n* The paper addresses an interesting problem in the creativity domain, namely, generating solutions that are both diverse and of high quality. The integration of AI feedback and the existing QD algorithm seems to be novel and interesting", "weaknesses": "Weakness\n* One key motivation of the paper using AI feedback seems to bypass the necessity to articulate a set of criteria. However, the prompt strategies still resort to specified diversity and quality criteria\n* The evaluation result seems to be a bit confusing; for example, in Table 1, one of the methods is LMX, Fitness-only, then in section 4.3 when it explains the method, there is LMX Quality-only. Is that the same method as LMX, fitness-only\n* It would be interesting to see ablation analysis to compare with the QD with and without AI feedback (not sure if LMX fitness-only or quality only serves the baseline) \n* QD metric is used throughout, which is the “sum of highest quality value found in each bin” - it seems to only focus on quality rather than diversity. For readers not familiar with QD metrics, some explanation/justification of why QD measures Quality and Diversity will be helpful", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0Zvt3CAbtX", "reviewer": "Reviewer_K2hJ", "summary": "This paper introduces an novel approach that integrates AI-generated (LMs) feedback into quality-diversity search algorithms, aiming to enhance the capability of AI systems in independent searching, evaluating, and innovation.", "questions": "A separate conclusion section, summarizing the key findings and contributions, would be advantageous for providing clear takeaways for the readers.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. The authors have proposed a novel and effective quality-diversity algorithm that leverages the latest developments in AI feedback, demonstrating superior performance compared to existing alternatives.\n2. The paper features a comprehensive set of experiments focused on creative writing, supported by a thorough analysis of the results, showcasing the practical applicability of the proposed method.", "weaknesses": "1. The presentation, particularly in the experimental section, could benefit from clarification and better organization to enhance readability and comprehension.  \n2. The paper occasionally employs exaggerated language and makes promising claims that seem to lack sufficient empirical backing. For example, the paper states “providing a recipe that seemingly generalizes to many domains and modalities” and “it is often easier for a model to evaluate the quality of a generation than to generate the same high-quality text.” These claims would be more convincing if supported by concrete evidence or reference or discussion.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)