ShinkaEvolve: Towards Open-Ended and Sample-Efficient Program Evolution

## Abstract

Abstract We introduce ShinkaEvolve 1 1 1 The Japanese term ‘shinka’ translates to ‘evolution’ or ‘innovation’. ShinkaEvolve refers to the vision of an open-ended self-refining innovation engine. While ShinkaEvolve might be a bit funny for some Japanese readers, as it sounds like Evolve-Evolve or 進化ー進化, similar bilingual repetitive terms appear quite often in Japan. : a new open-source framework leveraging large language models (LLMs) to advance scientific discovery with state-of-the-art performance and unprecedented efficiency.
Recent advances in scaling inference time compute of LLMs have enabled significant progress in generalized scientific discovery. These approaches rely on evolutionary agentic harnesses that leverage LLMs as mutation operators to generate candidate solutions.
However, current code evolution methods suffer from critical limitations: they are sample inefficient, requiring thousands of samples to identify effective solutions, and remain closed-source, hindering broad adoption and extension. ShinkaEvolve addresses these limitations, introducing three key innovations: a parent sampling technique balancing exploration and exploitation, code novelty rejection-sampling for efficient search space exploration, and a bandit-based LLM ensemble selection strategy.
We evaluate ShinkaEvolve across diverse tasks, demonstrating consistent improvements in sample efficiency and solution quality. ShinkaEvolve discovers a new state-of-the-art circle packing solution using only 150 samples, designs high-performing agentic harnesses for AIME mathematical reasoning tasks, identifies improvements to ALE-Bench competitive programming solutions, and discovers novel mixture-of-expert load balancing loss functions that illuminate the space of optimization strategies.
Our results demonstrate that ShinkaEvolve achieves broad applicability with exceptional sample efficiency. By providing open-source accessibility and cost-efficiency, this work democratizes open-ended discovery across diverse computational problems. Code https://github.com/SakanaAI/ShinkaEvolve

## 1 Introduction

The rapid advancement of large language models (LLMs) has transformed scientific discovery through agentic systems that autonomously conduct experiments and test hypotheses (Lu et al., 2024b; Yamada et al., 2025; Novikov et al., 2025; Zhang et al., 2025). These frameworks leverage LLMs as sophisticated mutation operators, iteratively refining candidate solutions with successful variants propagating through successive generations. This methodology has proven effective across domains such as competitive programming (Li et al., 2022), mathematical optimization (Romera-Paredes et al., 2024), and automated agentic design (Hu et al., 2024).
However, current implementations face significant practical limitations. The primary challenge is substantial sample inefficiency as existing approaches typically require thousands of evaluations, making them computationally expensive and time-consuming. This inefficiency stems from naive exploration strategies that fail to effectively leverage accumulated knowledge from previous generations. Additionally, most leading systems remain closed-source, creating barriers to reproducibility and limiting community-driven improvements.
ShinkaEvolve addresses these challenges through three key algorithmic innovations that work synergistically to enhance sample efficiency. Our adaptive parent and LLM sampling intelligently balances exploration of novel regions with exploitation of known high-quality areas. Next, our code proposal novelty rejection sampling ensures efficient program mutations. Finally, our bandit-based LLM ensemble selection strategy dynamically adapts to the evolving state of the sampled archive parents and inspiration programs.
Experimental validation across diverse domains demonstrates substantial improvements in both efficiency and solution quality, with ShinkaEvolve achieving state-of-the-art results using orders of magnitude fewer evaluations than existing approaches. By releasing our complete implementation as open-source software, we aim to democratize access to advanced evolutionary discovery tools and enable broad community contributions. In summary:

- 1.
We introduce ShinkaEvolve, an evolutionary framework that substantially improves sample efficiency through three key algorithmic innovations: a novel parent program sampling strategy, code novelty rejection-sampling, and adaptive performance-based LLM ensemble selection.
- 2.
We demonstrate ShinkaEvolve’s ability to innovate beyond human and LLM-generated solutions with comprehensive experimental validation across four distinct problem domains: mathematical optimization (circle packing), agentic design (AIME tasks), competitive programming (ALE-Bench), and LLM training design (mixture-of-expert load balancing loss).
- 3.
We release ShinkaEvolve as open-source software under the Apache 2.0 license, including implementation details, and an interactive visualization tool for monitoring the search process.

## 2 Related Work

Evolutionary Code Optimization with LLMs. One particular flavor of test-time compute is evolutionary code optimization: the usage, mutation, and recombination of previously generated code to produce new samples. This approach has previously been used to optimize reward and preference objectives (Lu et al., 2024a; Ma et al., 2023), mathematical science code (Romera-Paredes et al., 2024), and other applications (Lehman et al., 2022; Lange et al., 2024; Meyerson et al., 2023; Berman, 2025; Lange et al., 2025). Through prompting, LLMs are used as recombination engines (Lange et al., 2023; Meyerson et al., 2023), and are capable of simulating crossover between diverse code snippets and the rationales that produced them. These types of program archive-building systems resemble a population-based LLM-guided tree search (Jiang et al., 2025; Inoue et al., 2025). Most closely related to our work are AlphaEvolve (Novikov et al., 2025), OpenEvolve (Sharma, 2025), and LLM4AD (Liu et al., 2024a). We advance this line of work, demonstrating unprecedented sample efficiency with our combination of rejection-sampling, LLM prioritization, and online meta-scratchpad drafting.

Open-Ended Agentic Discovery. The integration of LLMs with open-ended evolutionary principles enables agentic systems capable of continuous innovation (Stanley et al., 2017; Zhang et al., 2025). Unlike traditional novelty search that relies on explicit diversity metrics (Lehman et al., 2008; Lehman and Stanley, 2011), LLM agents leverage learned representations to generate creative solutions while maintaining semantic coherence (Faldor et al., 2024; Hu et al., 2024; Novikov et al., 2025). These agents construct evolutionary trees of programs where LLM-guided mutations connect related solutions across generations (Lehman et al., 2020). ShinkaEvolve systematically combines stepping stones, suboptimal intermediate solutions that serve as building blocks for breakthrough innovations, by employing LLM agents to both generate mutations and evaluate program relationships, enabling successful patterns to rapidly propagate across search branches through recombination.

## 3 Method

Algorithm Overview. ShinkaEvolve’s control-flow entails three main phases:

- 1.
Parent and inspiration sampling from an archive of island program subpopulations. Importantly, we emphasize the trade-off between exploration and exploitation in parent program selection.
- 2.
Program mutation via LLM-guided code edit proposals. We utilize novelty rejection-sampling based on code embedding similarity and an LLM-as-a-novelty-judge assessment.
- 3.
Program execution and world feedback guiding the LLM ensemble selection probabilities and online meta-scratchpad drafting for documentation and knowledge diffusion.

### 3.1 Parent and inspiration sampling

Archive Maintenance, Island Populations & Mutation Context Construction. ShinkaEvolve maintains a fixed-size archive of previously evaluated programs with fitness scores and meta information, implementing an elite size constraint. The mutation context incorporates a primary parent program alongside inspiration programs drawn from top-performing solutions and random archive samples, providing the LLM with diverse exemplars for creative recombination. We follow Romera-Paredes et al. (2024); Novikov et al. (2025) and employ an island model approach with independent subpopulations seeded from the same initial program. The islands evolve in parallel to enhance diversity and prevent premature convergence. Island members can occasionally migrate between islands to diffuse knowledge across “discovery substreams”. To protect the uniqueness of each island, we prevent the island-specific best-performing program from migrating (Tanese, 1989; Romera-Paredes et al., 2024). Sampling occurs hierarchically: with the island ID first sampled uniformly from the archive, later used as the origin for both parent and inspirations. Afterwards, we sample random archive programs and the top-K performing programs to use them as context programs.

Balancing Exploration & Exploitation: Parent Program Selection. Given an island subpopulation, ShinkaEvolve implements multiple different parent sampling strategies that balance exploration and exploitation:
First, we employ power law sampling where programs are ranked by fitness with ranks $r_{i}$ ($r_{i}=1$ for the best program). The selection probability follows $p_{i}=\frac{r_{i}^{-\alpha}}{\sum_{j=1}^{n}r_{j}^{-\alpha}}$, where $\alpha$ controls exploitation intensity. Setting $\alpha=0$ yields uniform sampling, while $\alpha\to\infty$ implements hill-climbing.
Inspired by Zhang et al. (2025), we contrast this with weighted sampling, incorporating performance and novelty. Given programs with offspring count $N(P_{i})$, we first compute the median fitness $\alpha_{0}=\text{median}(\{F(P_{1}),F(P_{2}),...,F(P_{n})\})$. The performance component uses sigmoid scaling: $s_{i}=\sigma(\lambda\cdot(F(P_{i})-\alpha_{0}))$ where $\sigma(x)=\frac{1}{1+e^{-x}}$ and $\lambda$ controls selection pressure. The novelty component $h_{i}=\frac{1}{1+N(P_{i})}$ favors programs with fewer offspring. The final probability combines these: $p_{i}=\frac{w_{i}}{\sum_{j=1}^{n}w_{j}}$ where $w_{i}=s_{i}\cdot h_{i}$ balances performance and novelty. The strategies are illustrated in [Figure˜2](https://arxiv.org/html/2509.19349v1#S3.F2).

Figure: Figure 2: ShinkaEvolve Parent Sampling. The strategies range from pure exploration (uniform sampling) to pure exploitation (hill-climbing) to a combination of performance and novelty.
Refer to caption: x2.png

### 3.2 Program mutation and novelty assessment

LLM-Guided Program Mutations. To generate new programs, ShinkaEvolve starts by sampling a specific LLM and a set of sampling parameters (e.g., temperature or reasoning budget) from a pre-specified pool. Our framework provides support for models from leading API providers, including GPT, Gemini, Claude, and DeepSeek (OpenAI, 2023; Team, 2025; Anthropic, 2024; Guo et al., 2025). After sampling a model, ShinkaEvolve employs three distinct mutation approaches to foster diversity and creativity in the LLM-generated program variants:

- 1.
Diff-Based Edits. We implement diff edits using LLMs following the approach outlined in Novikov et al. (2025), utilizing SEARCH/REPLACE blocks for targeted modifications.
- 2.
Full Rewrites. We enable full program rewrites to allow greater flexibility, programmatically ensuring that non-mutable blocks remain unchanged during the LLM rewrite process.
- 3.
Crossover Mutation. We leverage crossover mutations (Lehman et al., 2022; Lange et al., 2025) where an additional archive program is sampled and an LLM is prompted to combine programs.

Following Novikov et al. (2025), we use text markers (EVOLVE-BLOCK-START & EVOLVE-BLOCK-END) to ensure that immutable code is left unchanged during the LLM rewrite process. After obtaining a code change proposal, we enforce that the immutable code is not touched and resample a new proposal if a patch is invalid, providing parsing feedback using Reflexion (Shinn et al., 2024).

Program Diversity via Novelty Rejection Sampling. To enhance the creativity of executed code proposals, we leverage a foundation model ensemble combined with temperature sampling. Additionally, we introduce code novelty rejection sampling using an embedding model to embed mutable parts of the program code. Afterwards, we compute cosine similarity scores across the island subpopulation programs. If the maximal score exceeds a threshold (e.g., $\eta=0.95$), we query an LLM to further assess whether the program is meaningfully different. The approach is illustrated in [Figure˜3](https://arxiv.org/html/2509.19349v1#S3.F3).

Figure: Figure 3: ShinkaEvolve Program Novelty Rejection Sampling. ShinkaEvolve embeds mutable code snippets, computes similarities across the archive; if the maximal score exceeds a threshold, another LLM is queried to assess whether the program is meaningfully novel.
Refer to caption: figures/f2_b_rejection.png

### 3.3 Execution and world feedback

Multi-Objective Optimization & Textual Feedback. After a program obtained with the above steps is executed, ShinkaEvolve performs multi-objective assessment yielding both its scalar fitness value $r_{i}$ together with a set of exposed “public metrics” and textual feedback. ShinkaEvolve then stores this full multi-objective assessment in the population archive to provide an informative context for future generations of language model mutations using a simple prompting format:

Adaptive LLM sampling evolution. The performance of different LLMs to propose mutations can vary across problem domains and based on the current state of the sampled archive parents and inspiration programs. ShinkaEvolve dynamically adapts to this non-stationarity by evolving the LLM sampling probability throughout at the end of each generation. Our approach is based on the UCB1 algorithm (Auer et al., 2002), associating each LLM with a visitation counter and an estimate of the expected score updated with the performance of its sampled mutations. We introduce changes tailored to the domain of LLM-driven discovery. In particular, rather than the absolute fitness of each mutation $r_{i}$, we update the LLM distribution using: $r_{i}^{u}=\exp\!\left(\max(r_{i}-r_{i}^{b},0)\right)-1$, where $r_{i}^{b}$ is the baseline reward for program $i$ computed as the maximum between its parent program and the initial program in the database, ensuring each LLM is evaluated based on its relative improvement to account for the non-stationarity of the program archive. At the same time, the $\exp(\cdot)$ and $\max(\cdot,0)$ operations help precisely promote LLMs able to come up with bold, high-risk, high-reward mutations, over “safer” minor improvements. We use the tracked statistics over the observed rewards to normalize $r_{i}^{u}$ and ensure invariance to the fitness scale of each domain.

Meta-Scratchpad & Online Refinement. ShinkaEvolve implements a meta-scratchpad system that periodically analyzes successful solutions to accelerate learning. Every $T$ generations, we summarize the recent program evaluations and identify common optimization strategies and design principles. The meta-agent synthesizes insights into actionable recommendations appended to the mutation prompt, providing high-level guidance from accumulated evolutionary experience. The approach is illustrated in [Figure˜4](https://arxiv.org/html/2509.19349v1#S3.F4).

Figure: Figure 4: A ShinkaEvolve Meta-Scratchpad. It consists of individual program summaries, global insights, and implementation recommendations, which are appended to the mutation prompt.
Refer to caption: figures/f2_c_scratchpad.png

## 4 Results

### 4.1 Circle Packing: Reproducing & Improving AlphaEvolve Results

##### Task Description.

The circle packing optimization problem requires placing 26 circles within a unit square such that the sum of their radii is maximized while ensuring no circles overlap and all circles remain fully contained within the square boundary. This constrained optimization challenge combines discrete placement decisions with continuous radius optimization, making it a complex benchmark for evolutionary algorithms. The problem exhibits multiple local optima and requires sophisticated search strategies to discover high-quality solutions, as naive approaches often converge to suboptimal configurations with poor space utilization.

##### ShinkaEvolve ’s Discovery Dynamics.

ShinkaEvolve was evaluated over 150 evolutionary generations, demonstrating remarkable sample efficiency compared to existing approaches that typically require thousands of evaluations ([Figure˜1](https://arxiv.org/html/2509.19349v1#S0.F1)).
[Figure˜5](https://arxiv.org/html/2509.19349v1#S4.F5) (left) illustrates the improvement trajectory, exhibiting three distinct phases: an initial rapid improvement phase where the algorithm quickly discovers fundamental radii optimization strategies, a sustained exploration phase with incremental gains as more sophisticated techniques emerge (constraint-based optimization), and a final convergence phase where the best solutions are refined through restarts.
The tree structure in [Figure˜5](https://arxiv.org/html/2509.19349v1#S4.F5) (right) reveals how successful innovations propagate through the population, with high-performing solutions (shown in green and yellow) serving as parents for subsequent generations.
Notably, the algorithm demonstrates sophisticated exploration patterns, with multiple evolutionary branches exploring different algorithmic approaches before converging toward the optimal solution path highlighted in black. We provide various ablation studies in [Section˜5](https://arxiv.org/html/2509.19349v1#S5).

Figure: Figure 5: ShinkaEvolve on Circle Packing Task. Left: ShinkaEvolve outperforms AlphaEvolve’s solution within less than 150 program evaluations. Right: ShinkaEvolve’s program evolution tree demonstrates the iterative composition of stepping stones into high-performing solutions.
Refer to caption: x3.png

##### ShinkaEvolve ’s Discovered Solution.

The evolved algorithm ([Section˜C.1](https://arxiv.org/html/2509.19349v1#A3.SS1)) combines three key innovations: (1) a sophisticated initialization strategy that places circles in a structured golden-angle spiral pattern with strategic corner and edge positioning, (2) a hybrid optimization approach integrating SLSQP gradient-based refinement with simulated annealing for global exploration, and (3) intelligent perturbation mechanisms that alternate between local circle movements and global ring rotations to escape local optima. The discovered solution employs adaptive temperature scheduling with reheating strategies to prevent premature convergence, while maintaining feasibility through constraint-aware radius computation. This multi-level approach, from structured initialization through meta-heuristic exploration to gradient-based polishing, exemplifies how ShinkaEvolve can discover sophisticated algorithmic compositions that outperform hand-designed baselines.

### 4.2 AIME: Evolving Agent Scaffolds for Math Reasoning

##### Task Description.

We evaluate ShinkaEvolve on AIME 2024 (AIM, 2024) mathematical reasoning problems, consisting of 30 challenging competition-level questions requiring sophisticated problem-solving strategies (Hu et al., 2024).
The task involves evolving agent scaffold designs constrained to a maximum of 10 LLM queries per problem for computational efficiency.
Using gpt-4.1-nano as the base model, we discover scaffold designs for 75 generations, with each candidate evaluated across three independent runs on the complete question set.

##### ShinkaEvolve ’s Discovery Dynamics.

The evolutionary process systematically explores prompting strategies, ensemble methods, and verification techniques to identify optimal agent architectures.
ShinkaEvolve discovers scaffold designs that significantly outperform hand-designed baselines, including simple single-query agents and sophisticated majority-voting approaches.
The search reveals a Pareto frontier between efficiency and performance ([Figure˜6](https://arxiv.org/html/2509.19349v1#S4.F6), left), with 7 LLM queries yielding maximum performance while an alternative scaffold achieves comparable results using the full 10-query budget.
Generalization experiments reveal important insights into the scaffold’s robustness.
Evaluating on 2023 and 2025 AIME problems shows different transfer patterns ([Figure˜6](https://arxiv.org/html/2509.19349v1#S4.F6), middle): smaller improvements on 2023 problems suggest potential saturation due to training data contamination, while larger gains on 2025 problems indicate successful generalization to recent, unseen challenges.
Cross-LLM model transfer experiments validate robustness, with successful adaptation to gpt-4.1-mini, gpt-4.1, and o4-mini demonstrating that discovered architectures capture generalizable strategies rather than model-specific optimizations ([Figure˜6](https://arxiv.org/html/2509.19349v1#S4.F6), right).

Figure: Figure 6: ShinkaEvolve for Agent Scaffold Design. Left: ShinkaEvolve discovers a Pareto frontier between performance and LLM query budget. Middle: The discovered scaffold generalizes to unseen AIME problems. Right: The scaffold improves performance regardless of the underlying LLM.
Refer to caption: x4.png

##### ShinkaEvolve ’s Discovered Solution.

The evolved agent implements a three-stage architecture leveraging diverse expert personas, critical peer review, and synthesis mechanisms.
Three specialized experts generate independent solutions using distinct approaches: a meticulous step-by-step reasoner, an intuitive pattern-recognition specialist, and an algorithmic computer science-oriented mathematician, each operating at 0.7 temperature to balance creativity with reliability.
The second stage introduces critical peer review, where each solution undergoes rigorous scrutiny from a skeptical reviewer at low temperature (0.1).
The reviewer validates pattern-based reasoning by testing patterns on multiple examples, identifies logical flaws, and provides corrections when necessary, significantly improving solution quality.
The final synthesis stage employs an editor-in-chief persona operating at zero temperature to analyze all solutions and critiques, identify the most reliable approach, and construct a canonical solution.
Robust fallback mechanisms resort to majority voting among reviewed solutions, then original solutions, ensuring reliable output when components fail.
This architecture effectively utilizes 7 LLM calls (3 generation + 3 review + 1 synthesis) within the 10-call constraint.
The complete discovered agent scaffold can be found in [Section˜C.2](https://arxiv.org/html/2509.19349v1#A3.SS2).

### 4.3 ALE-Bench: Evolving Programs for Combinatorial Optimization

##### Task Description.

We apply ShinkaEvolve to the ALE-Bench LITE (Imajuku et al., 2025) benchmark, a collection of 10 competitive programming contests hosted by AtCoder and designed to test the performance of LLMs on heuristic problems. Here, we explore whether ShinkaEvolve can successfully improve high-performing solutions discovered by LLMs. We leverage the best programming solution discovered by ALE-Agent (Imajuku et al., 2025) as an initial program for each problem and apply ShinkaEvolve to improve on top of it. We run ShinkaEvolve for 50 generations, leveraging the score calculated on the public test set as the fitness function. Afterwards, we submit the best solution to the private test set and report the score.

##### ShinkaEvolve ’s Discovery Dynamics.

ShinkaEvolve is able to improve the solutions discovered by ALE-Agent by approximately 2.3% across the 10 tasks on average ([Figure˜7](https://arxiv.org/html/2509.19349v1#S4.F7)). Furthermore, on one task, ahc039, the combination of ShinkaEvolve with ALE-Agent resulted in the second place submission on the [AtCoder leaderboard](https://atcoder.jp/contests/ahc039/submissions/68865341) if they had participated. While these improvements resulted from detailed implementation improvements, we observe that the proposed changes by ShinkaEvolve remained algorithmically close to the original ALE-Agent’s initialization solution.

Figure: Figure 7: ShinkaEvolve for Improving ALE-Bench solutions. Left: ShinkaEvolve improves the solutions discovered by ALE-Agent by $\sim 2.3\%$. Right: On one task, ahc039, the solution improved from 5th to 2nd place submission on the AtCoder leaderboard if it had participated in the contest.
Refer to caption: x5.png

##### ShinkaEvolve ’s Discovered Solution.

We focus on two tasks to illustrate the discovered improvements of ShinkaEvolve, ahc039 and ahc025.
The objective of ahc039 is to find an optimal, axis-aligned polygon to maximize the number of mackerels it contains minus the number of sardines, subject to given constraints. The base solution by ALE-Agent applies simulated annealing with kd-tree data structure (5th, 2880 performance). ShinkaEvolve further improved the solution (2nd, 3140 performance) by introducing modifications such as caching the validation process and enhancing neighborhood operators. For the caching, the kd-tree was augmented to cache subtree statistics, including bounding boxes and fish counts, at each node. For the neighborhood operators, a novel “targeted edge move” was introduced, which heuristically identifies a misclassified fish (e.g., a mackerel outside the polygon) and greedily moves the nearest edge to correct its state. These changes strengthened the directionality of the search.
For ahc025, the task is to use a balance scale to compare the total weights of any two subsets of items, aiming, after a fixed number of weighings, to partition the items into groups with as equal total weights as possible. ShinkaEvolve improved the ALE-Agent’s simulated annealing baseline by introducing faster caching, refining fallback weight estimation, and ultimately replacing simulated annealing with a more focused optimization combining greedy moves and targeted local search. Comparison with top human solutions suggests that for many tasks, there is ample room for improvement. Furthermore, often times ShinkaEvolve tended to explore modifications staying close to the ALE-Agent’s solution. This indicates the potential of overfitting to the initialization solution.

### 4.4 LLM Training: Evolving Losses for Balanced and Effective Experts

##### Task Description.

The Mixture-of-Expert (MoE) architecture (Szymanski and Lemmon, 1993; Shazeer et al., 2017; Lepikhin et al., 2020; Fedus et al., 2022) has been a critical advancement, ubiquitous amongst modern open and closed-source flagship models (Google AI Blog, 2024; Guo et al., 2025; Meta-AI, 2025; Yang et al., 2025; Team, 2025). The basic idea is simple: replace traditional large feed-forward residual blocks with ensembles of efficient smaller modules (the “experts”) that can each specialize in distinct problem domains (Fedus et al., 2022). For each MoE layer and token, only the outputs of the top-K experts selected by a router classifier are computed, effectively splitting the computation and making both training and inference cheaper and faster. However, due to the non-differentiability of the top-K expert selection operation, it is critical to provide the router with an auxiliary load balancing loss (LBL), which serves to avoid early collapse toward uneven expert distribution of the token load. We deploy ShinkaEvolve precisely to tackle this open architectural design challenge, which has been one core focus driving recent MoE advancements (Shazeer et al., 2017; Fedus et al., 2022; Du et al., 2022; Zoph et al., 2022; Xue et al., 2024; Dai et al., 2024; Qiu et al., 2025; Muennighoff et al., 2024): Devising an effective load balancing loss to incentivize efficiency and specialization, without hindering the model’s expressivity.

##### ShinkaEvolve ’s Discovery Dynamics.

We ground the problem of LBL design by pretraining a MoE model with 556M parameters, $N_{E}=64$ total experts of which only $K=8$ active for any given token. This results in only 82M parameters sparsely activated in each forward pass, excluding the token embeddings. We train this small model on over 2B tokens from fineweb (Penedo et al., 2024) by minimizing the MoE loss function adding the LBL, weighted by $\lambda=0.01$, to the model’s cross-entropy loss (CE). The fitness function of each program then measures a simple objective: minimize the sum of the final CE together with the model’s “load imbalance” as measured by the L1 deviation from a uniform distribution of tokens between the MoE experts. Given the cost of pretraining, we run ShinkaEvolve for only 30 iterations.
We evaluate the generality of ShinkaEvolve’s best-performing solutions by training a much larger MoE with 2.7B parameters on slightly under 30B fineweb tokens across three LBL coefficients $\lambda\in{0.001,0.01,0.1}$, yielding different levels of regularization. We then compare with the “global-batch LBL” used to train some of the most popular open LLMs (Yang et al., 2025), in terms of final perplexity (Figure [8](https://arxiv.org/html/2509.19349v1#S4.F8), left) and downstream task performance (Figure [8](https://arxiv.org/html/2509.19349v1#S4.F8), center) as evaluated across seven different benchmarks (Talmor et al., 2018; Zellers et al., 2019; Mihaylov et al., 2018; Bisk et al., 2020; Sap et al., 2019; Sakaguchi et al., 2021; Clark et al., 2018). We provide our results below as a function of load imbalance, showing that ShinkaEvolve’s new loss achieves a consistent edge across our training configurations, growing larger with the value of the $\lambda$ coefficient.

Figure: Figure 8: ShinkaEvolve for discovering Mixture-of-Experts Load Balancing Loss Functions. Left: Downstream task performance across seven benchmarks. Middle: Final perplexity across different missroute fractions. Right: Load imbalance gradient as a function of the token allocation.
Refer to caption: x6.png

##### ShinkaEvolve ’s Discovered Solution.

The discovered LBL is a new twist on the established global-batch LBL, which was used for seeding the evolutionary search. ShinkaEvolve complements this popular LBL with a new term, specifically targeted toward regularizing the MoE layers with individual under-specialized experts. Concretely, let $f_{\ell,i}$ and $P_{\ell,i}$ correspond to the selection frequency and the average router probabilities for each expert $i$ located in layer $\ell$. ShinkaEvolve’s LBL uses a normalized complement to the entropy in each layer $s(P_{\ell})=0.5+\Bigl(1-\frac{H(P_{\ell})}{\log N_{E}}\Bigr)$ and a minimum usage threshold target $\tau=0.064/{N_{E}}$ to compute:

$$ $\displaystyle L_{\mathrm{LBL}}$ $\displaystyle=\underbrace{\color[rgb]{0.12890625,0.44140625,0.7109375}\definecolor[named]{pgfstrokecolor}{rgb}{0.12890625,0.44140625,0.7109375}N_{E}\cdot\frac{1}{L}\sum_{\ell=1}^{L}\sum_{i=1}^{N_{E}}f_{\ell,i}\,P_{\ell,i}}_{\text{Global-batch LBL}}+\underbrace{\color[rgb]{0.90234375,0.33203125,0.05078125}\definecolor[named]{pgfstrokecolor}{rgb}{0.90234375,0.33203125,0.05078125}\frac{0.1}{L}\sum_{\ell=1}^{L}s(P_{\ell})\,\sum_{i=1}^{N_{E}}\max\!\bigl(0,\tau-f_{\ell,i}\bigr)}_{\text{{ShinkaEvolve} new regularization}}\,.$ (1) $$

The effects of ShinkaEvolve’s new regularization term can be visualized through its induced gradients acting on the router’s token allocation in a simplified two-expert scenario (Figure [8](https://arxiv.org/html/2509.19349v1#S4.F8), right). Intuitively, this term softly affects the MoE router of any layer, with experts getting allocated a fraction of tokens less than $\tau$. The multiplier $s(P_{\ell})$ makes this push stronger when the layer’s routing entropy $H(P_{\ell})$ is low and the router is concentrating on fewer dominating experts. This closes a potential blind spot of the global-batch LBL: the dot product $f\!\cdot\!P$ can look “balanced” even if few experts are barely touched. Thus, ShinkaEvolve’s new term can be seen as a safety net that adaptively activates and vanishes once an expert crosses the floor, providing dead experts and avoiding over-regularizing well-balanced layers.

## 5 Ablations & Analysis

Figure: Figure 9: ShinkaEvolve Method Ablation Studies on Circle Packing. Left: Weighted parent sampling outperforms random search and hill climbing. Middle: Bandit-based LLM ensembling slightly improves the performance over a fixed uniform ensemble distribution. Right: Embedding-based rejection sampling with LLM as a novelty judge strongly outperforms no rejection sampling.
Refer to caption: x7.png

##### Impact of Parent Selection Strategies.

To understand the importance of parent selection, we compare different strategies for choosing which programs to evolve. The Best-of-N baseline ignores the evolutionary history, always using the initial program as parent without feedback. In contrast, Hill Climbing represents a greedy approach that consistently selects the highest-performing program as the parent for subsequent mutations. Our proposed Weighted Sampling strategy balances exploration and exploitation by probabilistically selecting parents based on their fitness and number of offspring.

Takeaways. Weighted sampling consistently outperforms both random search and hill climbing across all tasks. Hill climbing shows strong initial performance but plateaus quickly, while weighted sampling maintains steady improvement throughout evolution. Random search demonstrates the poorest convergence, highlighting the importance of leveraging fitness-based parent selection.

##### Impact of LLM Ensembling and Prioritization.

Evolutionary agents can benefit from diverse coding capabilities by leveraging multiple LLMs. We investigate this hypothesis by comparing a Single LLM baseline (GPT-5-nano) against ensemble approaches. The Fixed LLM Ensemble provides diversity by sampling uniformly from a predetermined set of models, while our Bandit-Based LLM Ensemble adaptively learns which models contribute most effectively to fitness improvements, balancing exploration of underutilized models with exploitation of high-performing ones.

Takeaways. The bandit-based LLM ensemble significantly outperforms both single LLM and fixed ensemble approaches. While the fixed ensemble shows moderate improvements over single LLM usage, the adaptive bandit strategy achieves the highest performance by dynamically prioritizing more effective models based on their contribution to fitness improvements.

##### Impact of Code Embedding-Based Rejection Sampling.

Similar code variants can waste computational resources without advancing the search frontier. To address this challenge, we examine different novelty filtering mechanisms. The No Rejection Sampling baseline accepts any LLM proposal, potentially allowing near-duplicate programs to proliferate. Our Embedding-Based Rejection Sampling approach leverages text embeddings to identify and reject proposals with similarity scores exceeding 0.95. We also explore an Additional LLM-as-a-novelty-judge variant that supplements embedding-based filtering with explicit LLM assessment of program novelty.

Takeaways. Code embedding-based rejection sampling provides substantial performance gains over no rejection sampling by preventing redundant mutations. The additional LLM-as-a-novelty-judge offers marginal improvements, suggesting that embedding similarity is already an effective proxy for novelty assessment without requiring additional computational overhead.

## 6 Discussion

##### Summary.

This work introduces ShinkaEvolve, an evolutionary framework addressing critical limitations in LLM-driven scientific discovery through improved sample efficiency and open-source accessibility.
ShinkaEvolve achieves state-of-the-art results across four domains: circle packing with 150 evaluations (orders of magnitude improvement), sophisticated AIME reasoning scaffolds, ALE-Bench algorithmic improvements, and novel mixture-of-expert load balancing.

##### Limitations.

Our implementation uses fixed configurations with limited automatic control over exploration-exploitation balance, which may vary across domains.
Task specification requires manual human expertise for objective functions and evaluation.
The framework is constrained to problems with well-defined numerical objectives, limiting its applicability to diverse evaluation domains.

##### Future Directions.

Automated task specification through LLM task generation could enable greater autonomy and unlock applications in unexplored domains.
Transitioning to true open-endedness, where systems generate their own objectives, represents a compelling frontier.
Self-referential refinement and online meta-learning offer opportunities for continuously improving discovery.

##### Broader Impact & Ethical Considerations.

ShinkaEvolve’s open-source release further democratizes advanced evolutionary optimization, making it accessible to researchers and practitioners previously lacking access to proprietary systems.
The framework’s exceptional sample efficiency reduces computational barriers for resource-constrained environments.
However, API costs from large-scale LLM usage could create economic barriers, potentially constraining democratization goals.

## Acknowledgments

We thank David Ha, Takuya Akiba, Taishi Nakamura, Luca Grilloti, Yutaro Yamada, and the rest of the Sakana AI team for helpful discussions throughout the project.

## Author Contribution List

##### Robert Tjarko Lange

Initiated and led the project, designed the core ShinkaEvolve codebase, and implemented as well as collected results for Circle Packing, AIME, and ALE-Bench. Wrote the manuscript.

##### Yuki Imajuku

Helped setting up the ALE-Bench infrastructure, advised on the ALE-Bench results and supported writing the manuscript section on ALE-Bench.

##### Edoardo Cetin

Was involved in design discussions for ShinkaEvolve and came up as well as implemented the adaptive LLM sampling method and the Hydra configuration. He implemented andcollected results for LLM training and Mixture-of-Experts evolution. Co-wrote the manuscript.

## References

- AIM (2023)
American invitational mathematics examination, 2023.
Problems and solutions, 2023.
Published by the Mathematical Association of America / AMC contests.
- AIM (2024)
American invitational mathematics examination, 2024.
Problems and solutions, 2024.
Published by the Mathematical Association of America / AMC contests.
- AIM (2025)
American invitational mathematics examination, 2025.
Problems and solutions, 2025.
Published by the Mathematical Association of America / AMC contests.
- Anthropic (2024)
Anthropic.
The claude 3 model family: Opus, sonnet, haiku, 2024.
URL [https://www-cdn.anthropic.com/de8ba9b01c9ab7cbabf5c33b80b7bbc618857627/Model_Card_Claude_3.pdf](https://www-cdn.anthropic.com/de8ba9b01c9ab7cbabf5c33b80b7bbc618857627/Model_Card_Claude_3.pdf).
- Auer et al. (2002)
Peter Auer, Nicolo Cesa-Bianchi, and Paul Fischer.
Finite-time analysis of the multiarmed bandit problem.
*Machine learning*, 47(2):235–256, 2002.
- Berman (2025)
Jeremy Berman.
How i got a record 53.6% on arc-agi.
[https://jeremyberman.substack.com/p/how-i-got-a-record-536-on-arc-agi](https://jeremyberman.substack.com/p/how-i-got-a-record-536-on-arc-agi), 2025.
Accessed: 2025-02-08.
- Bisk et al. (2020)
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al.
Piqa: Reasoning about physical commonsense in natural language.
In *Proceedings of the AAAI conference on artificial intelligence*, volume 34, pages 7432–7439, 2020.
- Clark et al. (2018)
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord.
Think you have solved question answering? try arc, the ai2 reasoning challenge.
*arXiv preprint arXiv:1803.05457*, 2018.
- Dai et al. (2024)
Damai Dai, Chengqi Deng, Chenggang Zhao, RX Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Yu Wu, et al.
Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts language models.
*arXiv preprint arXiv:2401.06066*, 2024.
- Du et al. (2022)
Nan Du, Yanping Huang, Andrew M Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, et al.
Glam: Efficient scaling of language models with mixture-of-experts.
In *International conference on machine learning*, pages 5547–5569. PMLR, 2022.
- Faldor et al. (2024)
Maxence Faldor, Jenny Zhang, Antoine Cully, and Jeff Clune.
Omni-epic: Open-endedness via models of human notions of interestingness with environments programmed in code, 2024.
URL [https://arxiv.org/abs/2405.15568](https://arxiv.org/abs/2405.15568).
- Fedus et al. (2022)
William Fedus, Barret Zoph, and Noam Shazeer.
Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity.
*Journal of Machine Learning Research*, 23(120):1–39, 2022.
- Google AI Blog (2024)
Google AI Blog.
Our next-generation model: Gemini 1.5.
[https://blog.google/technology/ai/google-gemini-next-generation-model-february-2024/](https://blog.google/technology/ai/google-gemini-next-generation-model-february-2024/), February 2024.
Accessed: 2025-07-13.
- Guo et al. (2025)
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al.
Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.
*arXiv preprint arXiv:2501.12948*, 2025.
- Hu et al. (2024)
Shengran Hu, Cong Lu, and Jeff Clune.
Automated design of agentic systems.
*arXiv preprint arXiv:2408.08435*, 2024.
- Imajuku et al. (2025)
Yuki Imajuku, Kohki Horie, Yoichi Iwata, Kensho Aoki, Naohiro Takahashi, and Takuya Akiba.
Ale-bench: A benchmark for long-horizon objective-driven algorithm engineering.
*arXiv preprint arXiv:2506.09050*, 2025.
- Inoue et al. (2025)
Yuichi Inoue, Kou Misaki, Yuki Imajuku, So Kuroki, Taishi Nakamura, and Takuya Akiba.
Wider or deeper? scaling llm inference-time compute with adaptive branching tree search.
*arXiv preprint arXiv:2503.04412*, 2025.
- Jiang et al. (2024)
Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al.
Mixtral of experts.
*arXiv preprint arXiv:2401.04088*, 2024.
- Jiang et al. (2025)
Zhengyao Jiang, Dominik Schmidt, Dhruv Srikanth, Dixing Xu, Ian Kaplan, Deniss Jacenko, and Yuxiang Wu.
Aide: Ai-driven exploration in the space of code.
*arXiv preprint arXiv:2502.13138*, 2025.
- Lange et al. (2023)
Robert Lange, Tom Schaul, Yutian Chen, Tom Zahavy, Valentin Dalibard, Chris Lu, Satinder Singh, and Sebastian Flennerhag.
Discovering evolution strategies via meta-black-box optimization.
In *Proceedings of the Companion Conference on Genetic and Evolutionary Computation*, pages 29–30, 2023.
- Lange et al. (2024)
Robert Tjarko Lange, Yingtao Tian, and Yujin Tang.
Large language models as evolution strategies.
*arXiv preprint arXiv:2402.18381*, 2024.
- Lange et al. (2025)
Robert Tjarko Lange, Aaditya Prasad, Qi Sun, Maxence Faldor, Yujin Tang, and David Ha.
The ai cuda engineer: Agentic cuda kernel discovery, optimization and composition.
Technical report, Technical report, Sakana AI, 02 2025, 2025.
- Lehman and Stanley (2011)
Joel Lehman and Kenneth O Stanley.
Abandoning objectives: Evolution through the search for novelty alone.
*Evolutionary computation*, 19(2):189–223, 2011.
- Lehman et al. (2008)
Joel Lehman, Kenneth O Stanley, et al.
Exploiting open-endedness to solve problems through the search for novelty.
In *ALIFE*, pages 329–336, 2008.
- Lehman et al. (2020)
Joel Lehman, Jeff Clune, Dusan Misevic, Christoph Adami, Lee Altenberg, Julie Beaulieu, Peter J Bentley, Samuel Bernard, Guillaume Beslon, David M Bryson, et al.
The surprising creativity of digital evolution: A collection of anecdotes from the evolutionary computation and artificial life research communities.
*Artificial life*, 26(2):274–306, 2020.
- Lehman et al. (2022)
Joel Lehman, Jonathan Gordon, Shawn Jain, Kamal Ndousse, Cathy Yeh, and Kenneth O. Stanley.
Evolution through large models, 2022.
URL [https://arxiv.org/abs/2206.08896](https://arxiv.org/abs/2206.08896).
- Lepikhin et al. (2020)
Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen.
Gshard: Scaling giant models with conditional computation and automatic sharding.
*arXiv preprint arXiv:2006.16668*, 2020.
- Li et al. (2022)
Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al.
Competition-level code generation with alphacode.
*Science*, 378(6624):1092–1097, 2022.
- Liu et al. (2024a)
Fei Liu, Rui Zhang, Zhuoliang Xie, Rui Sun, Kai Li, Xi Lin, Zhenkun Wang, Zhichao Lu, and Qingfu Zhang.
Llm4ad: A platform for algorithm design with large language model.
*arXiv preprint arXiv:2412.17287*, 2024a.
- Liu et al. (2024b)
Liyuan Liu, Young Jin Kim, Shuohang Wang, Chen Liang, Yelong Shen, Hao Cheng, Xiaodong Liu, Masahiro Tanaka, Xiaoxia Wu, Wenxiang Hu, et al.
Grin: Gradient-informed moe.
*arXiv preprint arXiv:2409.12136*, 2024b.
- Loshchilov and Hutter (2017)
Ilya Loshchilov and Frank Hutter.
Decoupled weight decay regularization.
*arXiv preprint arXiv:1711.05101*, 2017.
- Lu et al. (2024a)
Chris Lu, Samuel Holt, Claudio Fanconi, Alex James Chan, Jakob Nicolaus Foerster, Mihaela van der Schaar, and Robert Tjarko Lange.
Discovering preference optimization algorithms with and for large language models.
In *The Thirty-eighth Annual Conference on Neural Information Processing Systems*, 2024a.
URL [https://openreview.net/forum?id=erjQDJ0z9L](https://openreview.net/forum?id=erjQDJ0z9L).
- Lu et al. (2024b)
Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob Foerster, Jeff Clune, and David Ha.
The ai scientist: Towards fully automated open-ended scientific discovery.
*arXiv preprint arXiv:2408.06292*, 2024b.
- Ma et al. (2023)
Yecheng Jason Ma, William Liang, Guanzhi Wang, De-An Huang, Osbert Bastani, Dinesh Jayaraman, Yuke Zhu, Linxi Fan, and Anima Anandkumar.
Eureka: Human-level reward design via coding large language models.
*arXiv preprint arXiv:2310.12931*, 2023.
- Meta-AI (2025)
Meta-AI.
The Llama 4 herd: The beginning of a new era of natively multimodal AI innovation, 2025.
URL [https://ai.meta.com/blog/llama-4-multimodal-intelligence/](https://ai.meta.com/blog/llama-4-multimodal-intelligence/).
- Meyerson et al. (2023)
Elliot Meyerson, Mark J Nelson, Herbie Bradley, Adam Gaier, Arash Moradi, Amy K Hoover, and Joel Lehman.
Language model crossover: Variation through few-shot prompting.
*arXiv preprint arXiv:2302.12170*, 2023.
- Mihaylov et al. (2018)
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal.
Can a suit of armor conduct electricity? a new dataset for open book question answering.
*arXiv preprint arXiv:1809.02789*, 2018.
- Muennighoff et al. (2024)
Niklas Muennighoff, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Jacob Morrison, Sewon Min, Weijia Shi, Pete Walsh, Oyvind Tafjord, Nathan Lambert, et al.
Olmoe: Open mixture-of-experts language models.
*arXiv preprint arXiv:2409.02060*, 2024.
- Novikov et al. (2025)
Alexander Novikov, Ngân Vũ, Marvin Eisenberger, Emilien Dupont, Po-Sen Huang, Adam Zsolt Wagner, Sergey Shirobokov, Borislav Kozlovskii, Francisco JR Ruiz, Abbas Mehrabian, et al.
Alphaevolve: A coding agent for scientific and algorithmic discovery.
*arXiv preprint arXiv:2506.13131*, 2025.
- OpenAI (2023)
OpenAI.
Gpt-4 technical report, 2023.
- Penedo et al. (2024)
Guilherme Penedo, Hynek Kydlíček, Anton Lozhkov, Margaret Mitchell, Colin A Raffel, Leandro Von Werra, Thomas Wolf, et al.
The fineweb datasets: Decanting the web for the finest text data at scale.
*Advances in Neural Information Processing Systems*, 37:30811–30849, 2024.
- Qiu et al. (2025)
Zihan Qiu, Zeyu Huang, Bo Zheng, Kaiyue Wen, Zekun Wang, Rui Men, Ivan Titov, Dayiheng Liu, Jingren Zhou, and Junyang Lin.
Demons in the detail: On implementing load balancing loss for training specialized mixture-of-expert models.
*arXiv preprint arXiv:2501.11873*, 2025.
- Romera-Paredes et al. (2024)
Bernardino Romera-Paredes, Mohammadamin Barekatain, Alexander Novikov, Matej Balog, M Pawan Kumar, Emilien Dupont, Francisco JR Ruiz, Jordan S Ellenberg, Pengming Wang, Omar Fawzi, et al.
Mathematical discoveries from program search with large language models.
*Nature*, 625(7995):468–475, 2024.
- Sakaguchi et al. (2021)
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi.
Winogrande: An adversarial winograd schema challenge at scale.
*Communications of the ACM*, 64(9):99–106, 2021.
- Sap et al. (2019)
Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi.
Socialiqa: Commonsense reasoning about social interactions.
*arXiv preprint arXiv:1904.09728*, 2019.
- Sharma (2025)
Asankhaya Sharma.
Openevolve: an open-source evolutionary coding agent, 2025.
URL [https://github.com/codelion/openevolve](https://github.com/codelion/openevolve).
- Shazeer (2020)
Noam Shazeer.
Glu variants improve transformer.
*arXiv preprint arXiv:2002.05202*, 2020.
- Shazeer et al. (2017)
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean.
Outrageously large neural networks: The sparsely-gated mixture-of-experts layer.
*arXiv preprint arXiv:1701.06538*, 2017.
- Shinn et al. (2024)
Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao.
Reflexion: Language agents with verbal reinforcement learning.
*Advances in Neural Information Processing Systems*, 36, 2024.
- Stanley et al. (2017)
Kenneth O Stanley, Joel Lehman, and Lisa Soros.
Open-endedness: The last grand challenge you’ve never heard of.
*While open-endedness could be a force for discovering intelligence, it could also be a component of AI itself*, 2017.
- Su et al. (2024)
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu.
Roformer: Enhanced transformer with rotary position embedding.
*Neurocomput.*, 568(C), February 2024.
ISSN 0925-2312.
[10.1016/j.neucom.2023.127063](https:/doi.org/10.1016/j.neucom.2023.127063).
URL [https://doi.org/10.1016/j.neucom.2023.127063](https://doi.org/10.1016/j.neucom.2023.127063).
- Szymanski and Lemmon (1993)
Peter T Szymanski and Michael D Lemmon.
Adaptive mixtures of local experts are source coding solutions.
In *IEEE International Conference on Neural Networks*, pages 1391–1396. IEEE, 1993.
- Talmor et al. (2018)
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant.
Commonsenseqa: A question answering challenge targeting commonsense knowledge.
*arXiv preprint arXiv:1811.00937*, 2018.
- Tanese (1989)
Reiko Tanese.
*Distributed genetic algorithms for function optimization*.
University of Michigan, 1989.
- Team (2024)
Chameleon Team.
Chameleon: Mixed-modal early-fusion foundation models.
*arXiv preprint arXiv:2405.09818*, 2024.
- Team (2025)
Gemini Team.
Google deepmind. gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities.
Technical report, Technical Report v2. 5, Google DeepMind, 2025.
- Xue et al. (2024)
Fuzhao Xue, Zian Zheng, Yao Fu, Jinjie Ni, Zangwei Zheng, Wangchunshu Zhou, and Yang You.
Openmoe: An early effort on open mixture-of-experts language models.
*arXiv preprint arXiv:2402.01739*, 2024.
- Yamada et al. (2025)
Yutaro Yamada, Robert Tjarko Lange, Cong Lu, Shengran Hu, Chris Lu, Jakob Foerster, Jeff Clune, and David Ha.
The ai scientist-v2: Workshop-level automated scientific discovery via agentic tree search.
*arXiv preprint arXiv:2504.08066*, 2025.
- Yang et al. (2025)
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al.
Qwen3 technical report.
*arXiv preprint arXiv:2505.09388*, 2025.
- Zellers et al. (2019)
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi.
Hellaswag: Can a machine really finish your sentence?
*arXiv preprint arXiv:1905.07830*, 2019.
- Zhang et al. (2025)
Jenny Zhang, Shengran Hu, Cong Lu, Robert Lange, and Jeff Clune.
Darwin godel machine: Open-ended evolution of self-improving agents.
*arXiv preprint arXiv:2505.22954*, 2025.
- Zoph et al. (2022)
Barret Zoph, Irwan Bello, Sameer Kumar, Nan Du, Yanping Huang, Jeff Dean, Noam Shazeer, and William Fedus.
St-moe: Designing stable and transferable sparse expert models.
*arXiv preprint arXiv:2202.08906*, 2022.

## Appendix

## Appendix A Shinka Implementation Details

- •
ShinkaEvolve uses a queue based implementation where LLMs generate program proposals sequentially. Afterwards, they are added to a job evaluation queue. Each proposal is based on all jobs that have completed so far and are stored in the database.
- •
Throughout development, we experimented with a fully asynchronous implementation that leverages both a job and a proposal queue. This allows for higher throughput but introduces a degree of "off-archiveness" in the sense that new code proposals are generated in advance and not based on all the previously submitted jobs. Furthermore, jobs from faster to query models will be executed earlier since their proposal jobs will be processed earlier. Many open research questions remain regarding the optimal trade-off between throughput, sample efficiency, and off-archiveness.
- •
Below we provide an overview of the Python API. It roughly adopts the high-level interface of OpenEvolve (Sharma, 2025):

Figure: Listing 1: Minimal ShinkaEvolve configuration and usage example.

evaluate.py - Evaluation Script

initial.py - Starting Solution

## Appendix B Task Implementation Details

### B.1 Circle Packing Problem

Detailed Task Description. The circle packing task requires placing 26 circles within a unit square such that the sum of their radii is maximized while ensuring no circles overlap and all circles remain fully contained within the square boundary.

Verification Methodology with Slack. For the main
ShinkaEvolve run presented in the paper, we employed the
verification script provided by OpenEvolve (Sharma, 2025), which allows for $1\times 10^{-6}$ numerical
slack. To ensure the robustness of our results, we
additionally validated our solutions using AlphaEvolve’s (Novikov et al., 2025) exact
verification code. We found that our discovered solution can
be made trivially exact by reducing each circle’s radius by
$1\times 10^{-8}$, demonstrating the high precision of our
approach. The adjustment from the relaxed to exact
formulation reduces the sum of radii for our discovered
solution by a negligible amount, from 2.635983099011548 to 2.
6359828390115476, representing a relative change of less than
$10^{-6}$.

Figure: Figure 10: Discovered Circle Packing solution by ShinkaEvolve.

Verification Methodology with Exact Constraint. Additionally, we replicated the solution using the exact verification code from AlphaEvolve [Figure˜11](https://arxiv.org/html/2509.19349v1#A2.F11) with a score of 2.63597770931127. The discovery of the solution requires more samples to be evaluated. This illustrates an important principle: surrogate relaxed tasks can be effectively used during evolution and subsequently post-processed to discover exact state-of-the-art solutions.

Figure: Figure 11: Circle packing asynchronous evolution results for exact circle packing verification showing convergence behavior and solution quality over time.
Refer to caption: x9.png

Baseline Comparisons. Our performance benchmarks are established against solutions from three primary sources. The AlphaEvolve sum of radii is taken from their paper (Novikov et al., 2025). The OpenEvolve baseline scores are derived from their [official implementation and examples](https://github.com/codelion/openevolve/tree/main/examples/circle_packing) available in their repository. Additionally, we compare against LLM4AD results, specifically their [circle packing implementations](https://github.com/Optima-CityU/LLM4AD/blob/main/example/circle_packing/circle_packing_result.ipynb) and [Evolution of Heuristics (EoH) experimental configurations](https://github.com/Optima-CityU/LLM4AD/blob/main/example/circle_packing/EoH_settings%26logs/run_eoh.py). These baselines provide comprehensive coverage of existing automated algorithm design approaches, enabling fair and thorough performance evaluation of our method.

ShinkaEvolve’s Hyperparameter Configuration.

**Table 1: ShinkaEvolve hyperparameter configuration for the Circle Packing task.**
| Parameter | Value | Parameter | Value |
| --- | --- | --- | --- |
| Database configuration |  |  |  |
| Archive size | 40 | Elite selection ratio | 0.3 |
| Archive inspirations | 4 | Top-$k$ inspirations | 2 |
| Migration interval | 10 | Migration rate | 0.0 |
| Island elitism | true | Parent selection strategy | weighted |
| Parent selection $\lambda$ | 10.0 | Number of islands | 2 |
| Evolution configuration |  |  |  |
| Patch types | [diff, full, cross] | Patch type probs | [0.45, 0.45, 0.1] |
| Generations | 150 | Max parallel jobs | 5 |
| Max patch resamples | 3 | Max patch attempts | 3 |
| Meta recommendation interval | 10 | Max meta recommendations | 5 |
| Embedding model | text-embedding-3-small | Max novelty attempts | None |
| Code embed sim threshold | 0.95 | Problem implementation | Python |
| LLM dynamic selection | ucb1 | Exploration coefficient | 1.0 |
| LLM models |  |  |  |
| gemini-2.5-pro | $\times$ | gemini-2.5-flash | $\times$ |
| claude-sonnet-4 | $\checkmark$ | o4-mini | $\checkmark$ |
| gpt-5 | $\times$ | gpt-4.1-nano | $\checkmark$ |
| gpt-4.1 | $\checkmark$ | gpt-4.1-mini | $\checkmark$ |
| LLM settings |  |  |  |
| Temperatures | [0.0, 0.5, 1.0] | Max tokens | 16,384 |
| Meta models | [gpt-5-nano] | Meta temperatures | [0.0] |
| Novelty models | [gpt-5-nano] | Novelty temperatures | [0.0] |

### B.2 AIME Math Reasoning Agentic Harness

Figure: Figure 12: ShinkaEvolve’s Discovery Trajectory for Math Agent Scaffold Design.

ShinkaEvolve’s Hyperparameter Configuration.

**Table 2: ShinkaEvolve Hyperparameter Configuration for the Math Reasoning Agentic Harness.**
| Parameter | Value | Parameter | Value |
| --- | --- | --- | --- |
| Database configuration |  |  |  |
| Archive size | 40 | Elite selection ratio | 0.3 |
| Archive inspirations | 4 | Top-$k$ inspirations | 2 |
| Migration interval | 10 | Migration rate | 0.1 |
| Island elitism | true | Parent selection strategy | weighted |
| Parent selection $\lambda$ | 10.0 | Number of islands | 4 |
| Evolution configuration |  |  |  |
| Patch types | [diff, full, cross] | Patch type probs | [0.6, 0.3, 0.1] |
| Generations | 75 | Max parallel jobs | 1 |
| Max patch resamples | 3 | Max patch attempts | 3 |
| Meta recommendation interval | 10 | Max meta recommendations | 5 |
| Embedding model | text-embedding-3-small | Max novelty attempts | 3 |
| Code embed sim threshold | 0.95 | Problem implementation | Python |
| LLM dynamic selection | null | Exploration coefficient | 0.0 |
| LLM models |  |  |  |
| gemini-2.5-pro | $\checkmark$ | gemini-2.5-flash | $\times$ |
| claude-sonnet-4 | $\checkmark$ | o4-mini | $\checkmark$ |
| gpt-5 | $\times$ | gpt-5-nano | $\times$ |
| gpt-4.1 | $\times$ | gpt-4.1-mini | $\times$ |
| LLM settings |  |  |  |
| Temperatures | [0.0, 0.5, 1.0] | Max tokens | 16,384 |
| Meta models | [gpt-4.1] | Meta temperatures | [0.0] |
| Novelty models | [gpt-4.1] | Novelty temperatures | [0.0] |

##### Detailed Task Description.

For the agent scaffold design task, we evaluate ShinkaEvolve on AIME 2024 mathematical reasoning problems, consisting of 30 challenging competition-level questions requiring sophisticated problem-solving strategies (AIM, 2024). We limit the maximum number of LLM queries per problem to 10 for computational and cost efficiency. Using gpt-4.1-nano as the base model, we evolve scaffold designs over 75 generations. Additionally and to combat stochasticity in LLM queries, we evaluated each candidate evaluated across three independent runs on the complete question set. After evolution, we evaluate the discovered scaffold designs on 2023 and 2025 AIME problems (AIM, 2023, 2025) to assess generalization as well as robustness to different base agent language models.

### B.3 ALE-Bench Problems

##### Detailed Task Description.

The ALE-Bench benchmark (Imajuku et al., 2025) is a collection of heuristic programming problems previously used in competitive programming contests (AtCoder). We evaluate ShinkaEvolve on the LITE subset of problems, which consists of 10 problems. We follow the evaluation protocol of ALE-Agent (Imajuku et al., 2025) and use the score calculated on the public test cases as the fitness function. Afterwards, we submit the best solution to the private test set and report the score. Additionally, in [Figure˜7](https://arxiv.org/html/2509.19349v1#S4.F7), we provide scores for evaluating the top-5 publicly scored solutions and taking their maximum score on the private test set. While this does not resemble the traditional competitive programming setting, it allows us to assess the generalization ability of the discovered solutions. The average solution score improves by a neglible amount from 1923.5 to 1927.0. Hence, we do not observe significant evidence for overfitting to the public test cases.

ShinkaEvolve’s Hyperparameter Configuration.

**Table 3: ShinkaEvolve Hyperparameter Configuration for the ALE-Bench Problems.**
| Parameter | Value | Parameter | Value |
| --- | --- | --- | --- |
| Database configuration |  |  |  |
| Archive size | 50 | Elite selection ratio | 0.3 |
| Archive inspirations | 2 | Top-$k$ inspirations | 2 |
| Migration interval | 10 | Migration rate | 0.1 |
| Island elitism | true | Parent selection strategy | weighted |
| Parent selection $\lambda$ | 10.0 | Number of islands | 2 |
| Evolution configuration |  |  |  |
| Patch types | [diff, full, cross] | Patch type probs | [0.6, 0.3, 0.1] |
| Generations | 50 | Max parallel jobs | 1 |
| Max patch resamples | 3 | Max patch attempts | 3 |
| Meta recommendation interval | 5 | Max meta recommendations | 5 |
| Embedding model | None | Max novelty attempts | None |
| Code embed sim threshold | None | Problem implementation | C++ |
| LLM dynamic selection | ucb1 | Exploration coefficient | 1.0 |
| LLM models |  |  |  |
| gemini-2.5-pro | $\checkmark$ | gemini-2.5-flash | $\checkmark$ |
| claude-sonnet-4 | $\checkmark$ | o4-mini | $\checkmark$ |
| gpt-5 | $\checkmark$ | gpt-5-mini | $\checkmark$ |
| gpt-4.1 | $\times$ | gpt-4.1-mini | $\times$ |
| LLM settings |  |  |  |
| Temperatures | [0.0, 0.5, 1.0] | Max tokens | 16,384 |
| Meta models | [gpt-5-mini] | Meta temperatures | [0.0] |
| Novelty models | None | Novelty temperatures | None |

### B.4 Mixture-of-Experts Load Balancing Loss

##### Detailed Task Description.

**Table 4: MoE architectures and training setup.**
| Hyperparameter | Small MoE (evolution) | Large MoE (evaluation) |
| --- | --- | --- |
| Model architecture |  |  |
| Model parameters | 556M | 2.7B |
| Model parameters | 82M | 404M |
| Number of experts ($N_{E}$) / active per token ($K$) | 64 / 8 | 64 / 8 |
| Hidden size | 512 | 1024 |
| Hidden size in each MoE expert | 384 | 768 |
| Number of hidden layers | 12 | 16 |
| Number of attention heads | 8 | 16 |
| Number of key–value heads | 8 | 8 |
| Head dimension | 128 | 128 |
| Attention bias | false | false |
| Attention dropout | 0.0 | 0.0 |
| Initializer range | 0.02 | 0.02 |
| RoPE $\theta$ | 1,000,000 | 1,000,000 |
| Tied word embeddings | true | true |
| Output router logits | true | true |
| Decoder sparse step | 1 | 1 |
| Router auxiliary loss coefficient ($\lambda$) | 0.01 | 0.001, 0.01, 0.1 |
| Computation dtype | bfloat16 | bfloat16 |
| Training setup |  |  |
| Optimizer | AdamW | AdamW |
| Learning rate | $1.0\times 10^{-3}$ | $3.0\times 10^{-4}$ |
| Weight decay | 0.1 | 0.1 |
| Adam parameters $(\beta_{1},\beta_{2},\epsilon)$ | (0.9, 0.95, $1\!\times\!10^{-8}$) | (0.9, 0.95, $1\!\times\!10^{-8}$) |
| Learning rate scheduler | Cosine decay | Cosine decay |
| Warmup steps | 70 | 490 |
| Maximum sequence length | 1024 | 1024 |
| Global train batch size (sequences) | 1024 | 2048 |
| Tokens per training step | 1,048,576 | 2,097,152 |
| Maximum steps | 2000 | 14,000 |
| Total tokens | 2.10B | 29.36B |
| Dataset | fineweb | fineweb |

The Mixture-of-Expert (MoE) architecture (Szymanski and Lemmon, 1993; Shazeer et al., 2017; Lepikhin et al., 2020; Fedus et al., 2022) has been a critical advancement, enabling scaling breakthroughs in large language model training. MoEs are currently ubiquitous amongst modern open and closed-source flagship models (Google AI Blog, 2024; Guo et al., 2025; Meta-AI, 2025; Yang et al., 2025; Team, 2025). The core principle behind the MoE design is to replace traditional large feed-forward residual blocks with ensembles of smaller modules (the “experts”), which can be efficiently sharded during training and only partially activated during inference (Fedus et al., 2022). Each expert is itself a small feed-forward network $E_{\ell,i}$ located within a larger ensemble of size $N_{E}$ at layer $\ell$. The router, a layer-specific linear classifier $h_{\ell}$, selects the top-$K$ most relevant experts for each token, computing only their outputs:

$$ $y_{\ell}(x)=\sum^{N_{E}}_{i=1}g_{\ell,i}(x)\,E_{\ell,i}(x),\quad g_{\ell,i}(x)=\begin{cases}\frac{e^{h_{\ell,i}(x)}}{\sum_{j\in\mathcal{T}_{K}(x)}e^{h_{\ell,j}(x)}},&\text{if }i\in\mathcal{T}_{K}(x)\\ 0,&\text{otherwise}\end{cases}$ (2) $$

where $\mathcal{T}_{K}(x)$ denotes the set of indices corresponding to the top-$K$ router logits $h_{\ell,i}(x)$. This sparsely activated design allows different experts to specialize in distinct problem domains, enabling greater efficiency, scalability, and adaptability in handling diverse prompts.

However, due to the non-differentiability of the top-$K$ expert selection operation, it is critical to provide the router with an auxiliary load balancing loss (LBL). The LBL prevents collapse toward uneven token distributions and under-specialized experts. Devising an effective load balancing loss that simultaneously encourages efficiency and expert specialization, without hindering expressivity, remains an open design challenge that has driven much of the recent progress in MoEs (Shazeer et al., 2017; Fedus et al., 2022; Du et al., 2022; Zoph et al., 2022; Xue et al., 2024; Dai et al., 2024; Qiu et al., 2025; Muennighoff et al., 2024). Minor design variations have been shown to significantly affect both efficiency and specialization ability (Dai et al., 2024; Jiang et al., 2024; Team, 2024; Liu et al., 2024b; Qiu et al., 2025).

One of the most widely adopted designs is the “global-batch” LBL introduced by Shazeer et al. (2017), which underpins several state-of-the-art open models such as Qwen 3 (Yang et al., 2025). For a layer $\ell$ with $N_{E}$ experts, it is defined as:

$$ $L_{LB}=N_{E}\cdot\frac{1}{L}\sum_{\ell=1}^{L}\sum_{i=1}^{N_{E}}f_{\ell,i}\cdot P_{\ell,i},$ (3) $$

where

$$ $f_{\ell,i}=\frac{\text{Tokens routed to expert }i}{\text{Total tokens in layer }\ell},\qquad P_{\ell,i}=\frac{\sum_{x}h_{\ell,i}(x)}{\sum_{x,j}h_{\ell,j}(x)}.$ $$

This formulation encourages token usage across experts to align with the router’s average soft assignment probabilities.

We evaluate ShinkaEvolve by pretraining a MoE model with 556M parameters, $N_{E}=64$ experts of which only $K=8$ are active for each token, corresponding to 82M sparsely activated parameters per forward pass (excluding embeddings). Training is performed on 2B tokens from fineweb (Penedo et al., 2024). For each program, we define a fitness function consisting of the cross-entropy (CE) loss together with an LBL term weighted by $\lambda=0.01$. To additionally measure load imbalance, we track the L1 deviation from a uniform distribution of token allocations:

$$ $L_{\mathrm{imb}}=\frac{1}{2}\sum_{i=1}^{N_{E}}\bigl|f_{\ell,i}-\tfrac{1}{N_{E}}\bigr|\,,$ (4) $$

with lower values indicating more even load distribution. This grounding provides ShinkaEvolve a two-fold search objective: minimize CE while improving load balance. To avoid local noise affecting the cross-entropy calculations, we average it over the last 10M tokens. The final fitness score used during evolution is then the negated sum of the two:

$$ $r=-(L_{\mathrm{CE}}+L_{\mathrm{imb}}).$ (5) $$

Given the expense of pretraining, we run ShinkaEvolve for only 30 iterations, focusing on gpt-4.1, gemini-2.5-pro, and claude-sonnet-4. To evaluate generality, we scale to a larger 2.7B-parameter MoE of which 404M active (excluding embeddings), trained on slightly under 30B fineweb tokens, and compare across three LBL coefficients $\lambda\in\{0.001,0.01,0.1\}$. We used AdamW (Loshchilov and Hutter, 2017) as the optimizer with cosine decay, and linear warmup. As common practice in modern training regimes, we used rotary positional embeddings (Su et al., 2024), SwiGLU MLPs (Shazeer, 2020), and half-precision bfloat16 to efficiently keep our model’s weights on device. For the small model used during ShinkaEvolve’s evolution, we use a batch size of slightly over $1M$ tokens, for 2K steps. For the larger MoE used double the batch size and seven times the total number of steps. After training, we benchmark against the global-batch LBL baseline in terms of perplexity (Figure [8](https://arxiv.org/html/2509.19349v1#S4.F8), left) and downstream performance across seven standard evaluations: CommonSense QA (Talmor et al., 2018), HellaSwag (Zellers et al., 2019), OpenBook QA (Mihaylov et al., 2018), PIQA (Bisk et al., 2020), SIQA (Sap et al., 2019),
WinoGrande (Sakaguchi et al., 2021), and ARC (Clark et al., 2018), truncating the number of questions to 1000 for large benchmarks as done by (Penedo et al., 2024).

Figure: Figure 13: LBL loss comparison.
Refer to caption: figures/load_balancing_visualization_app.png

As described in Section [4](https://arxiv.org/html/2509.19349v1#S4) and detailed in Appendix [C](https://arxiv.org/html/2509.19349v1#A3), ShinkaEvolve discovers a new twist on the global-batch LBL from Equation [3](https://arxiv.org/html/2509.19349v1#A2.E3), which was used for seeding evolutionary search. ShinkaEvolve discovers an augmentation of this loss with an additional regularization term to target under-specialized experts. As defined in Equation [3](https://arxiv.org/html/2509.19349v1#A2.E3), let $f_{\ell,i}$ and $P_{\ell,i}$ denote the selection frequency and average router probabilities for expert $i$ in layer $\ell$. Furthermore, define $s(P_{\ell})=0.5+\Bigl(1-\frac{H(P_{\ell})}{\log N_{E}}\Bigr)$ as a normalized complement of the routing entropy, and $\tau=0.064/N_{E}$ as a minimum usage threshold. The final discovered LBL is:

$$ $\displaystyle L_{\mathrm{LBL}}$ $\displaystyle=\underbrace{\color[rgb]{0.12890625,0.44140625,0.7109375}\definecolor[named]{pgfstrokecolor}{rgb}{0.12890625,0.44140625,0.7109375}N_{E}\cdot\frac{1}{L}\sum_{\ell=1}^{L}\sum_{i=1}^{N_{E}}f_{\ell,i}\,P_{\ell,i}}_{\text{Global-batch LBL}}+\underbrace{\color[rgb]{0.90234375,0.33203125,0.05078125}\definecolor[named]{pgfstrokecolor}{rgb}{0.90234375,0.33203125,0.05078125}\frac{0.1}{L}\sum_{\ell=1}^{L}s(P_{\ell})\,\sum_{i=1}^{N_{E}}\max\!\bigl(0,\tau-f_{\ell,i}\bigr)}_{\text{{ShinkaEvolve} new regularization}}\,.$ (6) $$

Figure: Figure 14: Mixture-of-Experts LBL design additional results.
Refer to caption: figures/load_balancing_comparison_app.png

In addition to the results from Section [4](https://arxiv.org/html/2509.19349v1#S4), in Figure [14](https://arxiv.org/html/2509.19349v1#A2.F14), we provide additional results comparing the global-batch LBL and ShinkaEvolve’s discovered LBL. In particular, we report the average task performance, final perplexity, and the fraction of missrouted tokens, as a function of the LBL coefficient $\lambda$ used for training the MoEs. Consistent with our previous analysis, ShinkaEvolve’s LBL appears to improve from the original LBL across both axes. However, we also note that the architecture used for evolving and testing the employed LBL was quite similar, and the training budget was still limited. However, the consistent generalization results across training budgets and coefficients $\lambda$ provide an optimistic outlook for future extensions to much longer training regimes, where even small efficiency gains could scale to significant cost savings.

ShinkaEvolve’s Hyperparameter Configuration.

**Table 5: ShinkaEvolve Hyperparameter Configuration for the MoE LBL Discovery.**
| Parameter | Value | Parameter | Value |
| --- | --- | --- | --- |
| Database configuration |  |  |  |
| Archive size | 20 | Elite selection ratio | 0.3 |
| Archive inspirations | 4 | Top-$k$ inspirations | 2 |
| Migration interval | 10 | Migration rate | 0.1 |
| Island elitism | true | Parent selection strategy | weighted |
| Parent selection $\lambda$ | 10.0 | Number of islands | 2 |
| Evolution configuration |  |  |  |
| Patch types | [diff, full] | Patch type probs | [0.5, 0.5] |
| Generations | 20 | Max parallel jobs | 1 |
| Max patch resamples | 10 | Max patch attempts | 10 |
| Meta recommendation interval | 10 | Max meta recommendations | 5 |
| Embedding model | text-embedding-3-small | Max novelty attempts | 3 |
| Code embed sim threshold | 0.95 | Problem implementation | Python |
| LLM dynamic selection | ucb1 | Exploration coefficient | 1.0 |
| LLM models |  |  |  |
| gemini-2.5-pro | $\checkmark$ | gemini-2.5-flash | $\times$ |
| claude-sonnet-4 | $\checkmark$ | o4-mini | $\times$ |
| gpt-5 | $\times$ | gpt-5-nano | $\times$ |
| gpt-4.1 | $\checkmark$ | gpt-4.1-mini | $\times$ |
| LLM settings |  |  |  |
| Temperatures | [0.0, 0.5, 1.0] | Max tokens | 16,384 |
| Meta models | [gpt-4.1] | Meta temperatures | [0.0] |
| Novelty models | [gpt-4.1] | Novelty temperatures | [0.0] |

## Appendix C ShinkaEvolve Discovered Solutions

### C.1 Circle Packing Problem

Figure: Listing 2: ShinkaEvolve Discovered Circle Packing Solution.

### C.2 AIME Math Reasoning Agentic Harness

Figure: Listing 3: ShinkaEvolve Discovered AIME Agent Scaffold Design.

### C.3 ALE-Bench Problems

#### C.3.1 ALE-Bench LITE task: ahc039

Figure: Listing 4: ShinkaEvolve Discovered ahc039 Solution.

#### C.3.2 ALE-Bench LITE task: ahc025

Figure: Listing 5: ShinkaEvolve Discovered ahc025 Solution.

### C.4 Mixture-of-Experts Load Balancing Loss

Figure: Listing 6: ShinkaEvolve Discovered Mixture of Experts Load Balancing Loss.

## Peer Reviews

confidence: 2
contribution: 2
initial_rating: 4
presentation: 2
questions: 1. Could you explain why it claims the bandit ensemble “significantly outperforms” both baselines (line 455), yet also notes it only “slightly improves” (Figure 8) over a fixed uniform ensemble? Quantitative deltas are not explicit.

2. What’s the exact parent-selection formula? The ablation plot labels “Novelty Weighted,” but the description elsewhere emphasizes fitness and offspring-count weighting—does novelty enter the score, and how?

3. Why choose a 0.95 embedding-similarity threshold—and how sensitive are results to it? The threshold is fixed for several tasks without a sweep or rationale.

4. Why is novelty filtering disabled on ALE-Bench but enabled on AIME/circle-packing? What trade-offs dictated this per-domain choice?
soundness: 2
strengths: 1. **Clear Motivation**: The paper targets sample inefficiency in LLM-driven program evolution and proposes three synergistic components:  novelty-weighted parent selection, code-novelty rejection sampling, and bandit-based LLM ensemble selection—all described with algorithmic detail.

2. **Concrete Reproducibility Details**: The paper includes hyperparameters, task harness details, and states that anonymized code is provided with plans to open-source.

3. **Multi-domain Evidence**: Extends beyond a single toy task: evolves AIME reasoning scaffolds, improves ALE-Bench solutions, and proposes an MoE load-balancing regularizer for training—broadening the paper’s relevance.
summary: This paper introduces ShinkaEvolve, an evolutionary framework to overcome the sample efficiency problems in prior works with three synergistic pieces: Parent sampling, code-novelty rejection sampling, and Bandit-based LLM ensemble selection. Experiments on several benchmarks and ablations prove its sample efficiency and broad utility.
weaknesses: **Heavy System Complexity and Tuning without Sufficient Sensitivity Analysis**: The framework has many moving parts and hyperparameters (islands, migration, thresholds, patch types, UCB1, etc.), increasing the risk that performance depends on careful engineering. Specifically, a fixed embedding-similarity cutoff (0.95) is used; aside from showing it helps, there’s little analysis of the sensitivity or cross-task robustness of that threshold.

**Modest Gains without Statistical Significance**: Average improvement for ALE-Bench is about 2.3% over a strong baseline across 10 tasks—useful but relatively small; variance or statistical significance isn’t discussed. The SOTA claim for Circle-packing comparisons is presented mainly relative to AlphaEvolve; the breadth of baselines in this domain isn’t deeply examined. 

**Effect-size Ambiguity for Ensembling**: Text/figure messaging is inconsistent, see question 1.

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: the bandit-based LLM ensemble approach uses a relatively simple UCB1 variant. More sophisticated multi-armed bandit algorithms or contextual bandits that consider the current state of the search could potentially yield better results. The paper doesn't justify why this approach was chosen.
soundness: 2
strengths: 1.	The study successfully demonstrates improvements across four diverse domains: geometric optimization, mathematical reasoning, competitive programming, and neural architecture design. It shows consistent performance gains with proper ablation studies validating each algorithmic component's contribution, and includes generalization experiments such as the AIME scaffold working across different LLMs that strengthen the practical applicability claims.
2.	The work introduces a well-integrated three-component system combining adaptive parent sampling, embedding-based novelty rejection, and bandit-based LLM selection that addresses real limitations in existing approaches. The framework produces immediately applicable results like the discovered MoE load balancing loss for training large language models and competitive programming solutions that outperform expert baselines, effectively bridging evolutionary computation and modern AI research with both principled algorithmic design and strong empirical validation.
summary: This paper proposes ShinkaEvolve, a framework that addresses sample inefficiency in LLM-driven program evolution through three key innovations: adaptive parent sampling balancing exploration and exploitation, code novelty rejection-sampling to avoid redundant mutations, and bandit-based LLM ensemble selection. The approach is validated across four domains including circle packing, mathematical reasoning, competitive programming, and neural architecture design, achieving state-of-the-art results with orders of magnitude fewer evaluations than existing methods.

Strengths:
1. The paper successfully demonstrates improvements across four diverse domains: geometric optimization, mathematical reasoning, competitive programming, and neural architecture design. It shows consistent performance gains with proper ablation studies validating each algorithmic component's contribution, and includes generalization experiments such as the AIME scaffold working across different LLMs that strengthen the practical applicability claims.
2. The work introduces a well-integrated three-component system combining adaptive parent sampling, embedding-based novelty rejection, and bandit-based LLM selection that addresses real limitations in existing approaches. The framework produces immediately applicable results like the discovered MoE load balancing loss for training large language models and competitive programming solutions that outperform expert baselines, effectively bridging evolutionary computation and modern AI research with both principled algorithmic design and strong empirical validation.

However, the bandit-based LLM ensemble approach uses a relatively simple UCB1 variant. More sophisticated multi-armed bandit algorithms or contextual bandits that consider the current state of the search could potentially yield better results. The paper doesn't justify why this approach was chosen.
In addition, this study is an applied engineering method of LLMs, the innovation of technical theory is relatively general.
weaknesses: the bandit-based LLM ensemble approach uses a relatively simple UCB1 variant. More sophisticated multi-armed bandit algorithms or contextual bandits that consider the current state of the search could potentially yield better results. The paper doesn't justify why this approach was chosen.
In addition, this study is an applied engineering method of LLMs, the innovation of technical theory is relatively general.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 4
questions: 1. Could you provide an estimate of the system's sensitivity across domains? Does the performance observed in one domain generalize to a new one without requiring extensive retuning of the system’s parameters?
2. Could you please elaborate on the limitations of using an embedding model for code novelty rejection-sampling? Also, in Table 3 (ALE-Bench config) it seems like it’s not being used.
3. Could you further clarify the core technical novelty of ShinkaEvolve and how it distinguishes itself form existing state-of-the-art evolutionary-based methods for LLM scientific discovery (e.g. AlphaEvolve [1])?

[1] Novikov et al. "AlphaEvolve: A coding agent for scientific and algorithmic discovery." (2025).
soundness: 2
strengths: - The paper aims to establish the building blocks for efficient agentic scientific discovery, which is a potentially high impact problem across multiple fields.
- The paper is well-structured, demonstrating clear explanations and principled ablations supporting its key contributions.
- The experiments cover four diverse and relevant domains, including an MoE architectural design challenge which is more open-ended. The results seem convincing and the discovered LBL regularization term found through this method is interesting (at the very least).
- The authors have shared their code and scripts for all experiments.
summary: The paper introduces ShinkaEvolve, a method that uses LLMs in a sample-efficient way to generate code and algorithms for scientific discovery. The system is built to improve sample inefficiency by guiding the LLMs to only pursue promising ideas through three key steps: adaptive parent sampling, code novelty rejection-sampling and a bandit-based LLM selection strategy. The authors demonstrate their results on four scientific and engineering tasks.
weaknesses: - The system is relatively complex, raising concerns that it may require quite different configurations across tasks or domains, which could limit its generalizability.
- A potential concern with using embedding similarity for code novelty rejection-sampling is that it may favor semantic or structural similarity, overlooking small but critical code modifications. Also, the LLM-as-a-judge component seems redundant with only marginal improvements at a high cost.
- The paper claims "order of magnitudes" fewer iterations compared to baselines for the circle packing task. However, it seems like the observed efficiency is at least partially an artifact of optimizing a surrogate objective of a relaxed task (Section B.1, Figure 10).

---

confidence: 4
contribution: 1
initial_rating: 0
presentation: 1
questions: Unfortunately, even if the authors can prove there are no ethical issues, I cannot understand the novelty of the contributions from the writing.
soundness: 1
strengths: The setting is promising and timely. Generally, the text was typo-free.
summary: This paper claims to introduce a new method that leverages LLMs to perform evolutionary algorithm search for the benefit of scientific discovery. Empirical results across a handful of domains are presented.
weaknesses: The paper is not coherently written, and I suspect it is entirely generated by an LLM. A (very non-exhaustive) list of suspicious features include the following.
- Introduction is mostly the abstract repeated verbatim.
- Paragraph starting line 110 is suspicious. Many terms are imprecisely used or not defined, e.g. “elite size,” “mutation context,” “primary parent,” “archive samples.” I can’t imagine a human writing “diverse exemplars for creative recombination.” The rest of the paragraph is similarly flowery language while not being quite precise enough for an ML audience.
-The overall problem is not defined anywhere
- Line 129: F, P_i are not defined before use. p_i is defined twice, differently. 
- r_i is both used for the rank (line 124) and for fitness value (line 210)
- The name ShinkaEvolve is never explained. 

Even if the paper was not LLM generated, it does a poor job of explaining the concepts with enough details that I would be able to reconstruct the algorithm or convince me that the authors understand the novelty and components.

---

confidence: 2
contribution: 3
initial_rating: 6
presentation: 3
questions: N/A
soundness: 3
strengths: SHINKAEVOLVE’s key strength is its high sample efficiency, achieving state-of-the-art results in as few as 150 iterations, which is orders of magnitude faster than prior evolutionary LLM systems. It also demonstrates generalization across domains, improving solutions in optimization, mathematical reasoning, competitive programming, and even neural network training. Additionally, it introduces reasonable mechanisms—like novelty-aware sampling and adaptive LLM selection—that make the search process both effective and scalable, setting a foundation for more open-ended AI discovery systems.
summary: SHINKAEVOLVE is a framework that uses LLMs to evolve programs efficiently for scientific and engineering discovery. It addresses a major limitation of prior systems—sample inefficiency—by introducing three innovations: novelty-aware parent selection, code novelty rejection sampling, and a bandit-based strategy to choose the most effective LLMs during evolution. These components work together to dramatically reduce the number of evaluations needed to find high-quality solutions. On the classic circle packing optimization task, SHINKAEVOLVE achieves state-of-the-art results in only 150 program evaluations, orders of magnitude fewer than previous methods. It also generalizes across domains, improving math reasoning agents (AIME), competitive programming solutions (ALE-Bench), and even discovering a new load-balancing loss for training mixture-of-experts models. The system demonstrates that LLM-driven evolutionary search can be both powerful and sample-efficient, advancing progress toward open-ended automated discovery. However, it still relies on human-defined fitness functions and structured problem objectives, leaving full autonomy as future work.
weaknesses: 1. #1 claim (sample efficiency) is shown on one canonical task with incomplete fairness controls. They highlight “SOTA in <150 evaluations” on circle packing and show an evolution tree and cost curve (Fig. 4), but I don’t see: multiple seeds, wall-clock comparisons, or matched compute/budget against the strongest recent baselines. The paper states 150 generations and that it “outperforms AlphaEvolve in less than 150 evaluations,” but it seems not to report variance or rigorous ablation under equalized budgets. That makes the “orders-of-magnitude fewer iterations” headline hard to audit.

2. Every domain here is driven by an explicit numeric fitness (packing objective; CE+load-balance proxy for MoE; AIME accuracy under a fixed query budget). That’s fine for optimization, but the paper’s “towards open-ended discovery” framing isn’t backed by tasks where the objective is underspecified or emergent. 

3. For the MoE study, SHINKAEVOLVE runs only 30 iterations during pretraining and optimizes a fitness that is the sum of CE and a load-imbalance term (L1 distance to uniform), with downstream generalization only implied. They later “scale up” to 2.7B parameters, but again emphasize CE/LBL trade-offs rather than end-task metrics; this makes it unclear whether the new regularizer robustly improves model quality beyond balancing tokens.


Decision: Accept (Poster)