Venue: NAACL 2025

# Exploiting Edited Large Language Models as General Scientific Optimizers

Qitan Lv\({}^{1}\), Tianyu Liu\({}^{1*}\), Hong Wang\({}^{1+}\),

\({}^{1}\) University of Science and Technology of China

{qitanlv, tianyu_liu, wanghong!700}@mail.ustc.edu.cn

Equal Contributions.The Corresponding author.

###### Abstract

Large language models (LLMs) have been widely adopted in mathematical optimization in scientific scenarios for their extensive knowledge and advanced reasoning capabilities. Existing methods mainly focus on utilizing LLMs to solve optimization problems in a prompt-based manner, which takes observational feedback as additional textual descriptions. However, due to LLM's **high sensitivity to the prompts** and **tendency to get lost in lengthy prompts**, these methods struggle to effectively utilize the observational feedback from each optimization step, which severely hinders the applications for real-world scenarios. To address these challenges, we propose a conceptually simple and general bi-level optimization method, namely **G**eneral **S**cientific **O**ptimizers (GSO). Specifically, GSO first utilizes inner-level simulators as experimental platforms to evaluate the current solution and provide observational feedback. Then, LLMs serve as knowledgeable and versatile scientists, generating new solutions by refining potential errors from the feedback as the outer-level optimization. Finally, simulations together with the expert knowledge in LLMs are jointly updated with bi-level interactions via model editing. Extensive experiments show that GSO consistently outperforms existing state-of-the-art methods using _six_ different LLM backbones on _seven_ different tasks, demonstrating the effectiveness and a wide range of applications.

## 1 Introduction

Optimization is ubiquitous across a wide range of scientific domains, spanning mathematics, physics, chemistry, pharmacology, etc Amari (1993); Intriligator (2002); Cova and Pais (2019). Various research endeavors innovate within its field, creating methods tailored to its specific challenges and nuances to automate and accelerate the process of mathematical optimization in scientific scenarios Wang et al. (2023b). However, the need for customization of optimization algorithms to address specific challenges highlights the absence of a universally applicable philosophy Popper (2005); Fortunato et al. (2018), which is crucial for establishing a standardized optimization framework and enhancing the efficiency of scientific research. We aim to transcend specific domains and provide a generalized approach to boost mathematical optimization in scientific scenarios.

Standing out as versatile tools with vast knowledge repositories, large language models (LLMs) have recently risen to prominence in optimization across scientific domains for their expansive knowledge bases, advanced reasoning capabilities, and human-friendly natural language interface AI4Science and Quantum (2023). Extensive research efforts have been devoted to boosting general mathematical optimization in scientific scenarios. Canonical methods mainly focus on fine-tuning LLMs using domain-specific data

Figure 1: GSO achieves state-of-the-art performance on a broad range of scientific optimization tasks compared with existing methods, using LLama 3 SB Team (2024b) as the backbone. Results of other five LLMs are in Figures 10 and 11.



to align natural language with scientific information, such as chemical structures (Li et al., 2024; Chithrananda et al., 2020) or drug structures (Liu et al., 2021). However, these approaches are constrained to specific domains and require substantial amounts of data and extensive computation resources for broader applicability. Recently, prompt-based iterative optimization methods--which enhance the inherent capabilities of pre-trained LLMs by incorporating the optimization feedback to LLMs--have emerged as a promising approach for advancing scientific optimization (Yang et al., 2023). Extensive researches have explored leveraging LLMs as optimizers or agents (Zhang et al.) for tasks such as mathematical problem-solving (Romera-Paredes et al., 2024; Yang et al., 2023), conducting chemical experiments (Boiko et al., 2023), advancing physical scientific discovery (Ma et al.), molecular discovery (Li et al., 2024), and drug discovery Sharma and Thakur, 2023).

Albeit with multiple benefits of the prompt-based methods, they confront one significant challenge that severely hinders their general applications--struggling to effectively utilize observational feedback. This challenge primarily stems from two limitations inherent in existing prompt-based methods: **(i) LLMs are shown to be sensitive to the prompt format**(Lu et al., 2022; Wei et al., 2023; Madaan and Yazdanbakhsh, 2022). In particular, semantically similar prompts can yield drastically different performance (Kojima et al., 2022; Zhou et al., 2023; Yang et al., 2023), and the optimal prompt formats may be model-specific and task-specific, which severely limits the generalizability across different scientific tasks. **(ii) LLMs may get lost in lengthy prompt**(Lv et al., 2024). In multi-round iterative optimization, the input prompt can become increasingly lengthy to trace the optimization trajectory, which may distract the LLMs' reasoning and result in the _lost in the middle issue_(Shi et al., 2023). Despite LLMs' ability to process long contexts, performances significantly decrease as the input grows longer, even for models explicitly designed for long contexts Lv et al. (2024).

To address this challenge, we propose a conceptually simple, flexible, and general method, namely **G**eneral **S**cientific **O**ptimizers (GSO). Specifically, GSO is a bi-level optimization method involving **inner-level** optimization and **outer-level** optimization, together with **bi-level** interaction between them. For a given optimization task, GSO will iteratively conduct the following process: (i) the inner-level optimization first employs simulators serving as an experimental platform to provide observational feedback for reasoning and refining; (ii) the **outer-level** optimization then utilizes knowledgeable LLMs as reasoning agents to generate hypotheses, reason with observational feedback, and refine the previous hypothetical solutions. It also devises a dynamic exploit-and-explore strategy to adaptively adjust the LLM reasoning trajectory based on observational feedback; (iii) finally, the **bi-level** optimization jointly updates the simulations together with the knowledge embedded within LLMs via model editing.

GSO is a novel framework to boost general mathematical optimization in scientific scenarios. As shown in Figure 1, extensive experiments demonstrate the effectiveness and generalization of our GSO, leading significant and consistent superiority using _six_ different LLM backbone on _seven_ different tasks than existing state-of-the-art methods.

## 2 Preliminaries

Model EditingModel editing mainly focuses on altering the internal knowledge within LLMs. It aims to update a range of intricate learned concepts, such as logical reasoning, spatial awareness, and numerical understanding, to make tailored adjustments to the model's behavior. In this paper, we follow Zhong et al. (2023); Zhang et al. (2024) to update factual knowledge represented as (subject \(s\), relation \(r\), object \(o\)). An LLM is expected to retrieve a memory corresponding to \(o\) when given a natural language prompt. Editing a fact involves replacing the existing knowledge triple \((s,r,o)\) with a new one \((s,r,o^{*})\). For simplicity, an edit is denoted as \(e=(s,r,o,o^{*})\). Given a set of edits \(\mathcal{E}=\{e_{1},e_{2},\ldots\}\) and an original model \(f_{\theta_{0}}\), sequential model editing applies each edit consecutively, i.e., \(\mathcal{F}(f_{\theta_{n-1}},e_{n})=f_{\theta_{n}}\), where \(f_{\theta_{n}}\) refers to the model after \(n\) edits.

## 3 Related Work

Model EditingModel editing involves modifying the memorized knowledge contained in LLMs. Various kinds of complex learned beliefs such as logical, spatial, or numerical knowledge are expected to be edited. Many studies explore the role of MLP layers in Transformers, revealing that these layers store knowledge, which can be localized to specific neurons and edited Da et al. ( 

2021; Geva et al., 2020, 2022). KE De Cao et al., 2021) and MEND Mitchell et al.) train a hypernetwork to compute gradient adjustments for updating model parameters. ROME Meng et al. (2022) and MEMIT Meng et al. (2020) apply the Locate-Then-Edit strategy, which first locates MLP storing factual knowledge, and then edits such knowledge by injecting a new key-value pair in the MLP module.

LLMs for Scientific OptimizationA recent line of research explores methods to enhance LLM performance by incorporating natural language feedback as prompts to revise model outputs in improving reasoning Shinn et al. (2023); Madan et al. (2024), code generation Chen et al. (2023); Olausson et al. (2023), dialogue applications Nair et al. (2023); Yuan et al. (2024); Liu et al. (2024b), and so on Wang et al. (2023a); Kim et al. (2024). LLMs have also been demonstrated to optimize complex problems by utilizing tools Sumers et al. (AlphaGeometry's Trinh et al. (2024) success in solving complex geometry problems without human demonstrations underscores the potential of LLMs in automating complex tasks. OPRO Yang et al. (2023) employs LLMs as black-box optimizers for complex reasoning tasks. Eureka Ma et al. (2023b) generates multiple solutions in each step to increase the success rate of produced codes. Funsearch Romera-Paredes et al. (2024b) utilizes an evolutionary strategy to avoid local optima by using LLMs along with a systematic evaluator. SGA Ma et al. (2023) introduces a bilevel optimization framework to enhance the knowledge-driven capabilities of LLMs by integrating simulations.

## 4 Method

### Bi-level Optimization Pipeline

We first briefly outline the bi-level pipeline of our GSO, describing how the inner-level and the outer-level optimization are jointly conducted. Specifically, for a scientific optimization task \(y\) (e.g., the traveling salesman problem), GSO aims to predict the optimal solution \(\hat{s}\), by which GSO iteratively optimizes the solution \(s_{k}\) from an initial solution \(s_{0}\), where \(k\) denotes the iteration step. Each itera 

tion consists of three parts: the inner-level simulation platform, outer-level LLM optimization, and bi-level interactions. An overview of GSO is in Figure 2.

The inner-level simulator **first** serves as an experimental platform, where a simulator \(\Phi\) is employed to take a intermediate solution \(s_{k}\) and its scientific expression \(\mathcal{E}_{k}\) (e.g., coordinates of the nodes in the TSP task) as inputs, and outputs the corresponding observational feedback \(f_{k}\) and the optimization objective \(\mathcal{L}_{k}\) (e.g., the total distance of the current route in the TSP task):

\[f_{k},\mathcal{L}_{k}=\Phi\left(s_{k};\mathcal{E}_{k}\right).\]

The outer-level optimization **then** utilizes an LLM with its parameter \(\theta_{k}\) (denote by \(\mathcal{M}_{\theta_{k}}\)) as a reasoning agent, propose a new hypothesis to optimize the current solution \(s_{k}\):

Here \(\mathcal{P}\) denotes the input prompt to LLM, and \(\{\mathcal{L}_{i},f_{i},\mathcal{E}_{i},s_{i}\}_{i=0}^{k}\) represents the historical optimization trajectory.

The bi-level interaction finally utilizes the feedback \(f_{k}\) from inner-level simulation and the solutions \(s_{k}\) provided by the outer-level LLM as a new key-value pair to update the LLM parameters \(\theta\) through model editing:

where \(\theta_{k+1}\) represents the new parameter matrix obtained by applying a model editing step to the previous parameters \(\theta_{k}\), \(\mathcal{M}_{\theta_{k+1}}\) represents the according updated LLM, and \(\mathcal{F}\) denotes the model editing process. We can then define the overall optimization task:

\[\min_{s}\mathcal{L}\left(y\left(\mathcal{E},s;\Phi\right)\right)\] s.t. \[G\left(\mathcal{E},s;\Phi\right)\leq 0,\]

where \(G\left(\cdot\right)\leq 0\) indicates that the current solution satisfies the constraints of the given optimization problem (for example, in the TSP task, the solution must visit all points and return to the starting point).

### Inner-level Simulation Platforms

The inner-level optimization conducts numerical simulations for a hypothetical solution. Simulations can provide domain-specific knowledge, transferring information from the simulation to optimization outputs (e.g., feedback in the form of total distance for the TSP task). These outputs, paired with the predicted solution \(s\), are then fed back into the LLMs to iteratively refine the hypothesis. The feedback can include the simulation loss relative to the target metric \(\mathcal{L}\), as well as auxiliary information during the optimization process, which can provide more insights for improving solutions. For instance, if \(\mathcal{L}\) represents the total distance of a solution for a TSP task, the auxiliary information may include the specific distance between two given points or the relative magnitude of distances between pairs of points (e.g. simulator can provide auxiliary information that \((v_{4},v_{1})\) has a larger distance than \((v_{2},v_{4})\) in Figure 2).

### Outer-level LLM Optimization

Many studies have demonstrated that LLMs are capable of discovering high-quality solutions and can match or even outperform hand-designed heuristic algorithms Yang et al. (2023); Romera-Paredes et al. (2024c). Therefore, we design outer-level optimization to: **(i)** analyze the current optimization task and make reasonable hypotheses; **(ii)** utilize the feedback from the simulations (including experimental phenomena and loss values, etc.), and update the LLM's knowledge through model editing; **(iii)** design and propose new potential solutions and input into the simulation platforms.

Inspired by the scientific optimization process by human scientists, we observe that when confronted with a new problem, scientists tend to be _daring adventurers_, utilizing expert knowledge and problem parameters to thoroughly explore the search space, eliminating improbable parameter regions. This may significantly reduce the subsequent solution space, yielding solutions that satisfy optimization constraints, though they may not be optimal Wuestman et al. (2020). Then, after gaining a deeper understanding of the problem through initial attempts, they change to be _cautious followers_ to keep the trajectory and trail previous solutions, adhering more closely to previous optimizations. This allows them to continually improve the metrics while satisfying optimization constraints, ultimately approaching the optimal solution. We thus design a **dynamic exploitation and exploration strategy** to mimic this process.

Specifically, we calculate the loss change as \(\Delta L=\frac{L_{\text{prev}}-L_{\text{rev}}}{L_{\text{prev}}}\). If \(\Delta L>0\), this indicates that the current hypothetical solutions are better than 

the previous ones. We then apply a lower decoding temperature to allow the LLM to follow the trace, i.e., \(T_{\text{cur}}=T_{\text{pre}}\times\left(\frac{1}{1+\Delta L}\right)\); if \(\Delta L<0\), this indicates that current solutions are worse than the previous ones. We thus apply a higher decoding temperature to be more exploratory and adventurrous, i.e., \(T_{\text{cur}}=T_{\text{pre}}\times(1+|\Delta L|)\). We also clip the temperature to \([0,1]\) in case that \(T_{\text{cur}}<0\) or \(T_{\text{cur}}>1\). Empirically, we divide solutions into \(\mathcal{S}_{\text{exploit}}\) and \(\mathcal{S}_{\text{explore}}\). We observe that (i) \(\mathcal{S}_{\text{exploit}}\) often contains repetitive solutions from previous iterations, and (ii) \(\mathcal{S}_{\text{explore}}\) tends to yield solutions to be informative for guiding optimization, or otherwise infeasible.

### Bi-level Interactions

The key challenge in integrating the two levels of optimization is developing a protocol that facilitates efficient, structured, and flexible communication between them. Therefore, we propose an interaction strategy considering the two aspects.

**On the one hand, from simulation platforms to LLMs**, drawing upon Meng et al. (2022), we incorporate the feedback from simulation platforms as new knowledge edited into the model. We **first** locate how facts are stored within the parameters of LLMs. We begin by analyzing and identifying which specific layers and their parameters \(W\) have the strongest causal effect on predictions of individual facts through **causal tracing**Meng et al. (2022)Detailed steps and results of the causal tracing for different LLMs are in Appendix B. **Next**, we integrate the experimental results from the simulation platforms as new knowledge, formatted as triples, into the model. For instance, in the context of the TSP problem, suppose the simulation platform's feedback template includes: (i) the path length for the solution \(v_{1}\to v_{2}\to v_{3}\to v_{4}\) is \(108\), (ii) the distance between \(v_{1}\) and \(v_{4}\) is greater than that between \(v_{2}\) and \(v_{4}\), and etc. These statements are then transformed into triples, such as \((v_{1}\to v_{2}\to v_{3}\to v_{4},~{}\text{has distance of},108)\) and \((v_{1},v_{4}),~{}\text{greater than},~{}(v_{2},v_{4}))\), which are subsequently edited into the model. **Finally**, by collecting such triples \((s,r,o)\), we compile a series of key-value pairs for a set of vector keys \(S=[s_{1},s_{2},\ldots,s_{n}]\) and corresponding vector values \(O=[o_{1},o_{2},\ldots,o_{n}]\) in each iteration step. Therefore, for the \(u\) new \((s,r,o)\) pairs obtained from the next iteration step, the objective of editing can be given by the following formula:

Here, \(W\) denotes the original matrix and \(W_{1}\) denotes the edited weight matrix. Due to this straightforward algebraic structure, any fact can be inserted directly once \((s,o)\) is determined. We can update the parameters of the LLM by editing the newly obtained \(u\) parameter-feedback pairs all at once, which adaptively utilizes the feedback from each simulation platform interaction into the LLM. It directly updates the LLM's weight parameters and prevents the input prompt length from increasing with iterations, alleviating the loss-in-the-middle issue during long-text and multi-round optimization processes. **On the other hand, from LLMs to simulation platforms**, we leverage the LLM as a domain expert to guide the setup of each subsequent simulation based on the results of the previous results. This process is akin to an experienced scientist providing tailored guidance for the experimental configuration of the next step simulation based on the current results. The simulation platforms can then continue running experiments based on the guidance of the LLM to provide observational feedback. Specifically, each time the outer-level LLM receives feedback from the previous iteration of simulation platforms, it can simultaneously request intermediate results from the inner-level platforms to aid in its reasoning. Taking the TSP problem as an example, when the LLM provides the solution \(v_{1}\to v_{2}\to v_{3}\to v_{4}\to v_{1}\), it can also request the platforms to provide the specific distance between two points. This additional feedback can then assist in refining the reasoning for the next iteration.

## 5 Experiments

### Problem Descriptions

We provide a brief definition of each task. More detailed definitions are in Appendix F. For the **linear system regression** task, the objective is to estimate the linear coefficients to model relationships between input and corresponding output Fisher (1922). We use **"#steps"** (optimization steps for successfully finding the optimal solution) **as the metric**. We follow the same dataset setting as OPRO Yang et al. (2023). For the **TSP** task, the objective is that given a set of \(n\) nodes with known coordinates, it seeks to find the shortest possible route 

that visits each node once and returns to the starting point. We follow the same dataset setting as OPRO (Yang et al., 2023) and use the Gurobi solver (Gurobi Optimization, LLC, 2024) to construct the oracle solutions and compute the **optimality gap as the metric** (the difference between the distance in the solution by the evaluated approach and by the oracle solution, divided by the distance of the oracle solution). For the **constitutive law prediction** task, we consider both fitting linear and non-linear materials and follow the same dataset setting as SGA (Ma et al.). We use **mean square error (MSE) as the metric**. For the **molecular property prediction** task, we consider three tasks: predict a molecule's highest occupied molecular orbital (HOMO), lowest unoccupied molecular orbital (LUMO), and the HOMO-LUMO gap based on their conformations and quantum mechanical properties. We follow SGA to use the QM9 dataset (Ramakrishnan et al., 2014) for experiments. We also use **MSE as the metric**.

### Experiment Setups

Implementation DetailsWe apply LLMs including GPT-J 6B (Radford et al., 2019), Llama38B (Team, 2024b), Mistral 7B (Jiang et al., 2023), Llama213B (Team, 2023c), Yi9b (Young et al., 2024), and Internlm 7B (Team, 2023b). We follow (Ma et al., 2023b) to conduct all experiments five times using different random seeds to guarantee stable and reproducible results. More details of implementation details are in Appendix F.

We consider **six** strong baselines for evaluation: (i) vanilla LLMs without additional modules. Vanilla LLMs represent the original capabilities of LLMs. (ii) **CoT** prompting (Wei et al., 2022) solves the problem by looking at step-by-step solutions from examples. We provide \(5\) examples with explanations as the initial solutions. (iii) **Funsearch**(Romera-Paredes et al., 2024b) utilizes evolutionary strategy to avoid local optimum. (iv) **Eureka**(Ma et al., 2023b) generates multiple solutions in each iteration to improve the success rate.

\begin{table}
\begin{tabular}{l l c c c c c c c} \hline \hline  & & \multicolumn{2}{c}{**Linear System**} & \multicolumn{2}{c}{**Travel Salesman**} & \multicolumn{2}{c}{**Constitutive Law**} & \multicolumn{2}{c}{**Molecule Property**} \\ \cline{3-10}
**Backbone** & **Method** & **(a)**\(\downarrow\) & **(b)**\(\downarrow\) & **(c)**\(\downarrow\) & **(d)**\(\downarrow\) & **(e)**\(\downarrow\) & **(f)**\(\downarrow\) & **(g)**\(\downarrow\) \\ \hline \multirow{8}{*}{**GPT-J 6B**} & **Vanilla** & N/A & 6.0 \(\pm\) 2.0 & N/A & N/A & N/A & N/A & N/A \\  & **CoT** & N/A & 4.4 \(\pm\) 1.9 & N/A & N/A & N/A & N/A & N/A \\  & **Funsearch** & 29.7 \(\pm\) 10.2 & 0.9 \(\pm\) 0.1 & 77.4 \(\pm\) 19.5 & 58.7 \(\pm\) 17.2 & 230.8 \(\pm\) 30.0 & 301.6 \(\pm\) 51.9 & 34.9 \(\pm\) 6.8 \\  & **Eurcka** & 45.1 \(\pm\) 19.8 & 1.9 \(\pm\) 0.8 & 155.3 \(\pm\) 31.2 & 91.7 \(\pm\) 18.5 & 390.0 \(\pm\) 33.8 & 388.0 \(\pm\) 70.3 & 55.3 \(\pm\) 8.4 \\  & **OPRo** & 41.0 \(\pm\) 18.0 & 0.4 \(\pm\) 0.2 & 60.9 \(\pm\) 22.6 & 81.3 \(\pm\) 10.2 & 459.5 \(\pm\) 83.3 & 155.8 \(\pm\) 18.9 & 23.7 \(\pm\) 4.0 \\  & **SGA** & 23.9 \(\pm\) 5.0 & 0.2 \(\pm\) 0.1 & 84.7 \(\pm\) 17.9 & 73.2 \(\pm\) 21.0 & 238.1 \(\pm\) 17.5 & 89.5 \(\pm\) 13.0 & 15.5 \(\pm\) 3.1 \\  & **GSO (ours)** & **12.3 \(\pm\) 4.8** & **0.0 \(\pm\) 0.0** & **15.7 \(\pm\) 5.0** & **55.4 \(\pm\) 8.0** & **70.1 \(\pm\) 17.7** & **83.1 \(\pm\) 13.9** & **8.5 \(\pm\) 2.0** \\ \hline \hline \end{tabular} 
\begin{tabular}{l l c c c c c c c} \hline \hline  & & \multicolumn{2}{c}{**Linear System**} & \multicolumn{2}{c}{**Travel Salesman**} & \multicolumn{2}{c}{**Constitutive Law**} & \multicolumn{2}{c}{**Molecule Property**} \\ \cline{3-10}
**Backbone** & **Method** & **(a)**\(\downarrow\) & **(b)**\(\downarrow\) & **(c)**\(\downarrow\) & **(d)**\(\downarrow\) & **(e)**\(\downarrow\) & **(f)**\(\downarrow\) & **(g)**\(\downarrow\) \\ \hline \multirow{8}{*}{**Llama38 B**} & **Vanilla** & N/A & 6.0 \(\pm\) 2.0 & N/A & N/A & N/A & N/A & N/A \\  & **CoT** & N/A & 4.4 \(\pm\) 1.9 & 33970.0 \(\pm\) 298.9 & N/A & N/A & N/A & N/A & N/A \\  & **Funesearch** & 15.7 \(\pm\) 8.3 & 1.3 \(\pm\) 0.5 & 198.9 \(\pm\) 30.8 & 335.9 \(\pm\) 112.1 & 433.7 \(\pm\) 25.5 & 175.5 \(\pm\) 28.8 & 91.4 \(\pm\) 19.8 \\  & **Eureka** & 18.3 \(\pm\) 5.1 & 2.1 \(\pm\) 0.2 & 255.6 \(\pm\) 80.9 & 401.3 \(\pm\) 98.0 & 260.7 \(\pm\) 38.9 & 130.1 \(\pm\) 15.9 & 155.1 \(\pm\) 28.3 \\  & **OPRO** & 17.0 \(\pm\) 4.7 & 0.3 \(\pm\) 0.1 & 74.1 \(\pm\) 10.3 & 227.8 \(\pm\) 33.2 & 537.1 \(\pm\) 69.7 & 115.5 \(\pm\) 35.3 & 51.9 \(\pm\) 10.4 \\  & **SGA** & 10.2 \(\pm\) 3.7 & 0.5 \(\pm\) 0.1 & 89.1 \(\pm\) 5.9 & 150.0 \(\pm\) 40.3 & 235.1 \(\pm\) 55.7 & 94.1 \(\pm\) 22.0 & 30.5 \(\pm\) 15.3 \\  & **GSO (ours)** & **5.1 \(\pm\) 1.0** & **0.0 \(\pm\) 0.0** & **8.1 \(\pm\) 2.1** & **20.1 \(\pm\) 3.9** & **30.1 \(\pm\) 14.9** & **20.9 \(\pm\) 9.3** & **9.7 \(\pm\) 3.6** \\ \hline \hline \end{tabular} 
\begin{tabular}{l l c c c c c c c} \hline \hline  & & \multicolumn{2}{c}{**Linear System**} & \multicolumn{2}{c}{**Travel Salesman**} & \multicolumn{2}{c}{**Constitutive Law**} & \multicolumn{2}{c}{**Molecule Property**} \\ \cline{3-10}
**Backbone** & **Method** & **(a)**\(\downarrow\) & **(b)**\(\downarrow\) & **(c)**\(\downarrow\) & **(d)**\(\downarrow\) & **(e)**\(\downarrow\) & **(f)**\(\downarrow\) & **(g)**\(\downarrow\) \\ \hline \multirow{8}{*}{**Mistral 7B**} & **Vanilla** & N/A & 6.0 \(\pm\) 2.0 & N/A & N/A & N/A & N/A & N/A \\  & **CoT** & N/A & 4.4 \(\pm\) 1.9 & N/A & N/A & N/A & N/A & N/A \\  & **Funesearch** & 67.3 \(\pm\) 14.7 & 1.3 \(\pm\) 0.3 & 317.9 \(\pm\) 43.0 & 199.4 \(\pm\) 31.0 & 335.9 \(\pm\) 61.3 & 310.4 \(\pm\) 33.1 & 86.8 \(\pm\) 17.4 \\  & **Eureka** & 37.9 \(\pm\) 13.1 & 2.5 \(\pm\) 0.8 & 337.3 \(\pm\) 51.1 & 379.0 \(\pm\) 139.1 & 201.3 \(\pm\) 33.7 & 331.7 \(\pm\) 25.4 & 110.4 \(\pm\) 19.5 \\  & **OPRo** & N/A 

(v) **OPRO**Yang et al. (2023) highlights the advantages of involving a sorted optimization trajectory. (vi) **SGA**Ma et al. utilizes a top-k heap to generate more diverse solutions.

### Main Results

We conduct our experiments on the \(7\) designed tasks using GPT-J 6B, Llama3 8B, and Mistral 7B as three representative backbone models in Table 1. We also provide more results for other _three_ different backbone models in Table 5 in Appendix E to demonstrate the versatility of our GSO across different backbones. GSO enables tasks, which are challenging to effectively optimize with traditional Vanilla and CoT methods, to become feasible. We also observe that GSO significantly and consistently outperforms existing methods on the scientific optimization tasks, which demonstrates the effectiveness of our GSO. Notably, for the molecule property prediction task on predicting the HOMOLUMO gap, GSO achieves a maximum precision improvement of over 32.6\(\times\) when utilizing Mistral 7B as the backbone model. These results underscore the importance of effectively utilizing the observational feedback to adaptively adjust its optimization directions. The universality of _six_ popular open-source backbone models in Appendix E also suggests the generalizability of our GSO.

### Ablation Study

To further investigate the contribution of each component within GSO, we conduct a series of ablation experiments on the entire framework. Specifically, we denote GSO without _edit_ as GSO\({}_{w/o~{}edit}\), GSO without the dynamic strategy as GSO\({}_{w/o~{}dynamic}\), respectively. We use Llama3 8B as the backbone model, the results of other backbone models are in Appendix G. We present the ablation results of GSO using Llama 3 8B as the backbone model in Table 2. More Results using the other five different backbone models are in Appendix G. As shown in Table 2, the absence of any component within GSO results in a performance degradation of the entire framework. Notably, GSO\({}_{w/o~{}edit}\) exhibits more significant impacts on the performance of GSO, which demonstrates the importance of effectively utilizing the observational feedback to adaptively adjust the optimization direction.

### Case Study

Robustness of the PromptAs mentioned in Section 1, one appealing feature of our GSO compared to other methods is its robustness to prompts. Prompts with similar semantics do not require meticulous crafting to yield consistently promising results. To provide more insights into our GSO, we manually generated two prompts and used OpenAI o1 to rewrite an additional three based on the original task prompts for each task (details of the generated prompts are in Appendix D). As shown in Table 3, we observe that prompt-based methods tend to be sensitive to the prompts, with semantically similar prompts often leading to fluctuating results across many tasks. This necessitates significant effort from users to perform prompt 'tuning' during application. In contrast, **our GSO consistently produces robust results across different prompts, with minimal variation between semantically similar prompts.** This reduces the need for extensive prompt "tuning," highlighting its potential for broader real-world applications.

Figure 3: We visualize the average MSE loss values of each method for the non-linear Constitutive Law task (d) across five random seeds at the same optimization steps using Mistral 7B as the backbone model, with shading representing the standard deviation.

\begin{table}
\begin{tabular}{l c c c c c c c} \hline \hline \multirow{2}{*}{**Method**} & **Linear System** & **Travel Salesman** & \multicolumn{4}{c}{**Constitutive Law**} & \multicolumn{2}{c}{**Molecule Property**} \\ \cline{2-9}  & **(a)**\(\downarrow\) & **(b)**\(\downarrow\) & **(e)**\(\downarrow\) & **(d)**\(\downarrow\) & **(e)**\(\downarrow\) & **(f)**\(\downarrow\) & **(g)**\(\downarrow\) \\ \hline
**GSO\({}_{w/o~{}edit}\)** & 33.7 ± 10.1 & 0.7 ± 0.3 & 116.1 ± 30.1 & 141.5 ± 36.9 & 307.7 ± 31.4 & 331.2 ± 89.2 & 164.3 ± 38.9 \\
**GSO\({}_{w/o~{}dynamic}\)** & 6.5 ± 3.5 & 0.0 ± 0.0 & 23.7 ± 9.9 & 76.9 ± 11.4 & 35.0 ± 17.1 & 45.0 ± 19.3 & 21.9 ± 7.3 \\ \hline
**GSO** & **5.1 ± 1.0** & **0.0 ± 0.0** & **8.1 ± 2.1** & **20.1 ± 3.9** & **30.1 ± 14.9** & **20.9 ± 9.3** & **9.7 ± 3.6** \\ \hline \hline \end{tabular}
\end{table}
Table 2: The results of the ablation study of our GSO on the seven scientific optimization tasks, using Llama3 8B as the backbone model (We provide more results for the other _five_ backbone models, in Appendix G).

 

Loss Curve and Decoding TemperatureWe also investigate the impact of the number of lengthy input prompts as optimization iterations increase on each method. We present the feedback loss trend for the non-linear constitutive law task (d) in Figure 3 as an example. We observe that GSO achieves significantly lower loss and exhibits a clear convergence trend compared to existing baselines. This demonstrates that GSO can effectively leverage feedback from each iteration to achieve stable and consistent improvements in scenarios involving lengthy prompts. Notably, some methods show improvement at the beginning (i.e., loss reduction) when the initial prompt is short and the optimization space is large, but as the number of optimization steps increases, the feedback loss fluctuates, making further optimization difficult. In contrast, GSO effectively leverages each step feedback to adaptively adjust its optimization direction, leading to a stable and consistent loss decrease. These results demonstrate that **GSO can effectively observational feedback from lengthy prompts to facilitate further optimization, thereby alleviating the loss in the middle issue.**

Comparison with Advanced Closed-source LLMsWe also compared our GSO framework on open-source models with the current state-of-the-art closed-source model, GPT-4o1 and Claude-3-Sonnet2. As shown in Table 4, our GSO utilizing Mistral 7B can consistently outperform GPT-4o and Claude-3-Sonnet, demonstrating the effectiveness of our approach. Note that our method is orthogonal to the choice of backbone model, making it a versatile plug-and-play module that can be directly applied to more advanced LLMs to further achieve enhanced results.

Footnote 1: [https://openai.com/01/](https://openai.com/01/)

Footnote 2: [https://www.anthropic.com/news/claude-3-5-sonnet](https://www.anthropic.com/news/claude-3-5-sonnet)

## 6 Conclusion

In this paper, we propose a novel **G**eneral **S**cientific **O**ptimizers method, effectively enabling LLMs to utilize observational feedback from each optimization step. Specifically, GSO consists of a bi-level optimization framework: outer-level LLMs function as knowledgeable and versatile scientists, generating new hypotheses to optimize experimental hyperparameters; inner-level simulations function as experimental platforms to perform numerical simulations to these hypotheses and provide observational feedback; a bi-level interaction then update the simulators together with the expert knowledge within LLMs via model editing. GSO effectively guides LLMs to derive a more precise and stable optimization direction, yielding superior optimization results. Extensive experiments on _six_ different open-source and _seven_ different scientific tasks demonstrate the superiority of our GSO, delivering consistent, robust, generalizable, and nearly monotonic improvement. We view our GSO as a trailblazer, establishing a new paradigm for utilizing LLMs and simulations as bi-level optimization to further advancements in scientific optimizations.3

\begin{table}
\begin{tabular}{l c c c c c c c} \hline \hline \multirow{2}{*}{**Method**} & **Linear System** & **Travel Salesman** & \multicolumn{2}{c}{**Constitutive Law**} & \multicolumn{2}{c}{**Molecule Property**} \\ \cline{2-9}  & **(a)**\(\downarrow\) & **(b)**\(\downarrow\) & **(e)**\(\downarrow\) & **(d)**\(\downarrow\) & **(e)**\(\downarrow\) & **(f)**\(\downarrow\) & **(g)**\(\downarrow\) \\ \hline
**GPT-4o** & 6.4 ± 1.5 & 0.2 ± 0.0 & 59.7 ± 7.4 & 45.5 ± 9.1 & 181.4 ± 33.7 & 313.5 ± 45.4 & 67.8 ± 10.9 \\
**Claude-3** & 12.1 ± 2.0 & 0.2 ± 0.1 & 128.1 ± 13.5 & 103.0 ± 5.2 & 162.3 ± 21.2 & 519.3 ± 37.4 & 37.8 ± 8.3 \\ \hline
**GSO** & **5.6 ± 3.0** & **0.1 ± 0.0** & **3.2 ± 1.7** & **13.0 ± 2.1** & **23.9 ± 5.1** & **2.8 ± 1.5** & **1.7 ± 0.4** \\ \hline \hline \end{tabular}
\end{table}
Table 4: Results of our GSO against popular closed-source LLMs, including GPT-4o and Claude-3.5. We present the results of our GSO Mistral 7B as the backbone model for comparison.

\begin{table}
\begin{tabular}{l l r c c c c c c c} \hline \hline \multirow{2}{*}{**Backbone**} & \multirow{2}{*}{**Method**} & \multicolumn{2}{c}{**Linear System**} & \multicolumn{2}{c}{**Travel Salesman**} & \multicolumn{2}{c}{**Constitutive Law**} & \multicolumn{2}{c}{**Molecule Property**} \\ \cline{3-10}  & & **(a)**\(\downarrow\) & **(b)**\(\downarrow\) & **(e)**\(\downarrow\) & **(d)**\(\downarrow\) & **(e)**\(\downarrow\) & **(f)**\(\downarrow\) & **(g)**\(\downarrow\) \\ \hline \multirow{4}{*}{**Mistral 7B**} & **Funsearch** & 52.2 ± 29.4 & 2.4 ± 1.5 & 411.0 ± 193.4 & 254.9 ± 109.3 & 473.3 ± 130.2 & 403.1 ± 135.0 & 173.0 ± 66.3 \\  & **Eureka** & 60.25 ± 29.2 & 3.5 ± 2.6 & 303.7 ± 180.3 & 179.0 ± 65.8 & 219.0 ± 40.6 & 287.9 ± 99.0 & 148.9 ± 55.0 \\  & **OPRO** & N/A & 0.4 ± 0.3 & 107.6 ± 61.4 & 224.8 ± 34.2 & 94.3 ± 50.6 & 593.7 ± 266.8 & 40.5 ± 28.0 \\  & **SGA** & 38.7 ± 20.3 & 0.3 ± 0.2 & 55.4 ± 43.9 & 203.2 ± 22.3 & 211.7 ± 98.0 & 73.0 ± 18.8 & 62.9 ± 33.2 \\  & **GSO (ours)** & **8.3 ± 3.3** & **0.1 ± 0.0** & **5.5 ± 3.1** & **20.9 ± 8.3** & **18.8 ± 7.0** & **3.9 ± 2.1** & **2.2 ± 0.4** \\ \hline \hline \end{tabular}
\end{table}
Table 3: We evaluate our GSO against other baseline methods on five prompts that vary in format but are semantically similar, and report their average results of each method, using Mistral 7B as the representative backbone model. The symbol N/A indicates that the model cannot provide a feasible solution for the current task.

 



## Acknowledgements

The authors would like to thank all the anonymous reviewers for their insightful comments.

## 7 Limitations

We consider a few limitations and future directions. (i) The content generated by LLMs exhibits a certain degree of randomness, and the optimization process cannot guarantee interpretability or transparency. (ii) LLM inference requires large computational resources and thus increases expense. It paves the way for research on LLM inference acceleration to expedite our GSO Leviathan et al. (2023); Liu et al. (2024a). (iii) GSO requires access to model weights, which limits its applicability to closed-source models like GPT-4 and Claude 3.5. We believe that as the community progresses and the performance gap between open-source and closed-source models narrows, our GSO will be able to demonstrate its capabilities more effectively.

## References

* M. Al4Science and M. Azure Quantum (2023)The impact of large language models on scientific discovery: a preliminary study using gpt-4. arXiv preprint arXiv:2311.07361. Cited by: SS1.
* S. Amari (1993)Backpropagation and stochastic gradient descent method. Neurocomputing5 (4-5), pp. 185-196. External Links: ISSN 0021-909X Cited by: SS1.
* D. A. Boiko, R. MacKnight, B. Kline, and G. Gomes (2023)Autonomous chemical research with large language models. Nature624 (7992), pp. 570-578. External Links: Document, Link Cited by: SS1.
* T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. (2020)Language models are few-shot learners. Advances in neural information processing systems33, pp. 1877-1901. External Links: ISSN 1050-3215, Link, Document Cited by: SS1.
* A. Chen, D. Dolhan, and D. So (2024)EvoPrompting: language models for code-level neural architecture search. Advances in Neural Information Processing Systems36. Cited by: SS1.
* A. Chen, J. Scheurer, T. Korbak, J. C. Campos, J. S. Chan, S. R. Bowman, K. Cho, and E. Perez (2023)Improving code generation by training with natural language feedback. arXiv preprint arXiv:2303.16749. Cited by: SS1.
* H. Chen, X. Shen, Q. Lv, J. Wang, X. Ni, and J. Ye (2024)Sac-kg: exploiting large language models as skilled automatic constructors for domain knowledge graph. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 4345-4360. External Links: ISBN 978-3-319-5035-5 Cited by: SS1.
* X. Chen, M. Lin, N. Scharli, and D. Zhou (2019)Teaching large language models to self-debug. In The Twelfth International Conference on Learning Representations, Cited by: SS1.
* X. Chen and Y. Tian (2019)Learning to perform local rewriting for combinatorial optimization. Advances in neural information processing systems32. Cited by: SS1.
* Y. Chen, X. Song, C. Lee, Z. Wang, R. Zhang, D. Dohan, K. Kawakami, G. Kochanski, A. Doucet, M. Laurelio Ranzato, et al. (2022)Towards learning universal hyperparameter optimizers with transformers. Advances in Neural Information Processing Systems35, pp. 32053-32068. External Links: ISSN 1050-3215, Link, Document Cited by: SS1.
* S. Chithrananda, G. Grand, and B. Ramtsundar (2020)Chemberta: large-scale self-supervised pretraining for molecular property prediction. arXiv preprint arXiv:2010.09885. Cited by: SS1.
* T. F. Cova and A. A. Pais (2019)Deep learning for deep chemistry: optimizing the prediction of chemical patterns. Frontiers in chemistry7, pp. 809. Cited by: SS1.
* J. Da, R. Le Bras, X. Lu, Y. Choi, and A. Bosseltu (2021)Analyzing commonsense emergence in few-shot knowledge models. arXiv preprint arXiv:2101.00297. Cited by: SS1.
* N. De Cao, W. Aziz, and I. Titov (2021)Editing factual knowledge in language models. arXiv preprint arXiv:2104.08164. Cited by: SS1.
* M. Devdon, P. Cournut, A. Lacoste, Y. Adulyasak, and L. Rousseau (2018)Learning heuristics for the tsp by policy gradient. In Integration of Constraint Programming, Artificial Intelligence, and Operations Research: 15th International Conference, CPAIOR 2018, Delft, The Netherlands, June 26-29, 2018, Proceedings 15, pp. 170-181. Cited by: SS1.
* B. Dhingra, M. Zaheer, V. Balachandran, G. Neubig, R. Salakhutdinov, and W. W. Cohen (2018)Differentiable reasoning over a virtual knowledge base. In International Conference on Learning Representations, Cited by: SS1.
* Vanessa Didelez and I. Pigeot (2001)Causality: models, reasoning, and inference. Cited by: SS1.
* R. A. Fisher (1922)On the mathematical foundations of theoretical statistics. Philosophical Transactions of the Royal Society of London. Series A222, pp. 309-368.



Santo Fortunato, Carl T Bergstrom, Katy Borner, James A Evans, Dirk Helbing, Stasa Milojevic, Alexander M Petersen, Filippo Radicchi, Roberta Sinatra, Brian Uzzi, et al. 2018. Science of science. 359(6379):eaao0185.
* Frantar and Alistarh [2023] Elias Frantar and Dan Alistarh. 2023. Qmoe: Practical sub-1-bit compression of trillion-parameter models. _arXiv preprint arXiv:2310.16795_.
* Geva et al. [2022] Mor Geva, Avi Caciularu, Kevin Ro Wang, and Yoav Goldberg. 2022. Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space. _arXiv preprint arXiv:2203.14680_.
* Geva et al. [2020] Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. 2020. Transformer feed-forward layers are key-value memories. _arXiv preprint arXiv:2012.14913_.
* Golden et al. [1980] Bruce Golden, Lawrence Bodin, T Doyle, and W Stewart Jr. 1980. Approximate traveling salesman algorithms. _Operations research_, 28(3-part-ii):694-711.
* Gurnee et al. [2020] Wes Gurnee, Neel Nanda, Matthew Pauly, Katherine Harvey, Dmitrii Troitskii, and Dimitris Berrsimas. Finding neurons in a haystack: Case studies with sparse probing. _Transactions on Machine Learning Research_.
* Optimization [2024] Gurobi Optimization, LLC. 2024. Gurobi Optimizer Reference Manual.
* Gutin and Punnen [2006] Gregory Gutin and Abraham P Punnen. 2006. _The traveling salesman problem and its variations_, volume 12. Springer Science & Business Media.
* Guu et al. [2020] Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. 2020. Retrieval augmented language model pre-training. In _International conference on machine learning_, pages 3929-3938. PMLR.
* Hanna et al. [2024] Michael Hanna, Ollie Liu, and Alexandre Variengien. 2024. How does gpt-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model. _Advances in Neural Information Processing Systems_, 36.
* Hao et al. [2023] Shibo Hao, Bowen Tan, Kaiwen Tang, Bin Ni, Xiyan Shao, Henzhe Zhang, Eric Xing, and Zhiting Hu. 2023. Bertnet: Harvesting knowledge graphs with arbitrary relations from pretrained language models. In _Findings of the Association for Computational Linguistics: ACL 2023_, pages 5000-5015.
* Helsgaun [2017] Keld Helsgaun. 2017. An extension of the linkernighan-helsgaun tsp solver for constrained traveling salesman and vehicle routing problems. _Roskilde: Roskilde University_, 12:966-980.
* Hsieh et al. [2023] Cheng-Yu Hsieh, Chun-Liang Li, Chih-Kuan Yeh, Hootan Nakhost, Yasuhisa Fujji, Alexander Ratner, Ranjay Krishna, Chen-Yu Lee, and Tomas Pfister. 2023. Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes. _arXiv preprint arXiv:2305.02301_.
* Hu et al. [2021] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. _arXiv preprint arXiv:2106.09685_.
* Intriligator [2002] Michael D Intriligator. 2002. _Mathematical optimization and economic theory_. SIAM.
* Jiang et al. [2023] Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral 7b. _arXiv preprint arXiv:2310.06825_.
* Jiang et al. [2016] Chenfanfu Jiang, Craig Schroeder, Joseph Teran, Alexey Stomakhin, and Andrew Selle. 2016. The material point method for simulating continuum materials. In _Acm siggraph 2016 courses_, pages 1-52.
* Junger et al. [1995] Michael Junger, Gerhard Reinelt, and Giovanni Rinaldi. 1995. The traveling salesman problem. _Handbooks in operations research and management science_, 7:225-330.
* Meng-Wei Chang Kenton and Kutina Toutanova [2019] Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In _Proceedings of naacL-HLT_, volume 1, page 2. Minneapolisolis, Minnesota.
* Kim et al. [2024] Geunwoo Kim, Pierre Baldi, and Stephen McAleer. 2024. Language models can solve computer tasks. _Advances in Neural Information Processing Systems_, 36.
* Kojima et al. [2022] Takeshi Kojima, Shixiang Shane Gu, Michel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. _Advances in neural information processing systems_, 35:22199-22213.
* Kool et al. [2018] Wouter Kool, Herke Van Hoof, and Max Welling. 2018. Attention, learn to solve routing problems! _arXiv preprint arXiv:1803.08475_.
* Lehman et al. [2022] Joel Lehman, Jonathan Gordon, Shawn Jain, Kamal Ndousse, Cathy Yeh, and Kenneth O. Stanley. 2022. Evolution through large models. _Preprint_, arXiv:2206.08896.
* Leviathan et al. [2023] Yaniv Leviathan, Matan Kalman, and Yossi Matias. 2023. Fast inference from transformers via speculative decoding. In _International Conference on Machine Learning_, pages 19274-19286. PMLR.
* Li et al. [2024a] Jiatong Li, Yunqing Liu, Wenqi Fan, Xiao-Yong Wei, Hui Liu, Jiliang Tang, and Qing Li. 2024a. Empowering molecule discovery for molecule-caption translation with large language models: A chatgpt perspective. _IEEE Transactions on Knowledge and Data Engineering_.
* Li et al. [2020] Junxian Li, Di Zhang, Kunzhi Wang, Zeying Hao, Jingdi Lei, Qian Tan, Cai Zhou, Wei Liu, Yaotian Yang, Xinrui Xiong, Weiyu Wang, Zhe Chen, 

Wenhai Wang, Wei Li, Shufei Zhang, Mao Su, Wanli Ouyang, Yuqiang Li, and Dongzhan Zhou. 2024b. Chemvlm: Exploring the power of multimodal large language models in chemistry area. _Preprint_, arXiv:2408.07246.
* Li et al. (2024c) Junxian Li, Di Zhang, Xunzhi Wang, Zeying Hao, Jingdi Lei, Qian Tan, Cai Zhou, Wei Liu, Yaqutian Yang, Xinrui Xiong, Weiyuan Wang, Zhe Chen, Wenhai Wang, Wei Li, Shufei Zhang, Mao Su, Wanli Ouyang, Yuqiang Li, and Dongzhan Zhou. 2024c. Chemvlm: Exploring the power of multimodal large language models in chemistry area. _Preprint_, arXiv:2408.07246.
* Liu et al. (2024a) Tianyu Liu, Yun Li, Qitan Lv, Kai Liu, Jianchen Zhu, and Winston Hu. 2024a. Parallel speculative decoding with adaptive draft length. _Preprint_, arXiv:2408.11850.
* Liu et al. (2024b) Tianyu Liu, Qitan Lv, Jie Wang, Shuling Yang, and Hanzhu Chen. 2024b. Learning rule-induced subgraph representations for inductive relation prediction. _Advances in Neural Information Processing Systems_, 36.
* Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. _Preprint_, arXiv:1907.11692.
* Liu et al. (2021) Zhichao Liu, Ruth A Roberts, Madhu Lal-Nag, Xi Chen, Ruili Huang, and Weida Tong. 2021. Ai-based language models powering drug discovery and development. _Drug Discovery Today_, 26(11):2593-2607.
* Lu et al. (2022) Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2022. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 8086-8098.
* Luo et al. (2023) Yun Luo, Zhen Yang, Fandong Meng, Yafu Li, Jie Zhou, and Yue Zhang. 2023. An empirical study of catastrophic forgetting in large language models during continual fine-tuning. _arXiv preprint arXiv:2308.08747_.
* Lv et al. (2024) Qitan Lv, Jie Wang, Hanzhu Chen, Bin Li, Yongdong Zhang, and Feng Wu. 2024. Coarse-to-fine highlighting: Reducing knowledge hallucination in large language models. In _Forty-first International Conference on Machine Learning_.
* Ma et al. (2023a) Pingchuan Ma, Peter Yichen Chen, Bolei Deng, Joshua B Tenenbaum, Tao Du, Chuang Gan, and Wojciech Matusik. 2023a. Learning neural constitutive laws from motion observations for generalizable pde dynamics. In _International Conference on Machine Learning_, pages 23279-23300. PMLR.
* Ma et al. (2024) Pingchuan Ma, Tsun-Hsuan Wang, Minghao Guo, Zhiqing Sun, Joshua B Tenenbaum, Daniela Rus, Chuang Gan, and Wojciech Matusik. Llm and simulation as bilevel optimizers: A new paradigm to advance physical scientific discovery. In _Forty-first International Conference on Machine Learning_.
* Ma et al. (2023b) Yecheng Jason Ma, William Liang, Guanzhi Wang, DeAn Huang, Osbert Bastani, Dinesh Jayaraman, Yuke Zhu, Linxi Fan, and Anima Anandkumar. 2023b. Eureka: Human-level reward design via coding large language models. In _The Twelfth International Conference on Learning Representations_.
* Madaan et al. (2024) Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shimai Prabhumoye, Yiming Yang, et al. 2024. Self-refine: Iterative refinement with self-feedback. _Advances in Neural Information Processing Systems_, 36.
* Madaan and Yazdanabakhsh (2022) Aman Madaan and Amir Yazdanabakhsh. 2022. Text and patterns: For effective chain of thought, it takes two to tango. _arXiv preprint arXiv:2209.07686_.
* Meng et al. (2022) Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022. Locating and editing factual associations in gpt. _Advances in Neural Information Processing Systems_, 35:17359-17372.
* Meng et al. (2023a) Kevin Meng, Arnab Sen Sharma, Alex J Andonian, Yonatan Belinkov, and David Bau. Mass-editing memory in a transformer. In _The Eleventh International Conference on Learning Representations_.
* Meyerson et al. (2023a) Elliot Meyerson, Mark J Nelson, Herbic Bradley, Adam Gaier, Arash Moradi, Amy K Hoover, and Joel Lehman. 2023a. Language model crossover: Variation through few-shot prompting. _arXiv preprint arXiv:2302.12170_.
* Meyerson et al. (2023b) Elliot Meyerson, Mark J Nelson, Herbie Bradley, Adam Gaier, Arash Moradi, Amy K Hoover, and Joel Lehman. 2023b. Language model crossover: Variation through few-shot prompting. _arXiv preprint arXiv:2302.12170_.
* Mitchell et al. (2023) Eric Mitchell, Charles Lin, Antoine Bosseltut, Chelsea Finn, and Christopher D Manning. Fast model editing at scale. In _International Conference on Learning Representations_.
* Montgomery et al. (2021) Douglas C. Montgomery, Elizabeth A. Peck, and Geoffrey G. Vining. 2021. _Introduction to Linear Regression Analysis_, 6th edition edition. Wiley.
* Nair et al. (2023) Varun Nair, Elliot Schumacher, Geoffrey Tso, and Anitha Kannan. 2023. Dera: enhancing large language model completions with dialog-enabled resolving agents. _arXiv preprint arXiv:2303.17071_.
* Nazari et al. (2018) Mohammadrezza Nazari, Afshin Oroojlooy, Lawrence Snyder, and Martin Takac. 2018. Reinforcement learning for solving the vehicle routing problem. _Advances in neural information processing systems_, 31 

* Neuberg (2003) Leland Gerson Neuberg. 2003. Causality: models, reasoning, and inference, by judea pearl, cambridge university press, 2000. _Econometric Theory_, 19(4):675-685.
* Olaussson et al. (2023) Theo X Olaussson, Jeevana Priya Inala, Chenglong Wang, Jianfeng Gao, and Armando Solar-Lezama. 2023. Demystifying gpt self-repair for code generation. _arXiv preprint arXiv:2306.09896_.
* OpenAI (2020) OpenAI. 2020. Chatgpt: A large-scale generative model for conversation.
* OpenAI (2023) OpenAI. 2023. GPT-4 technical report. _Preprint_, arXiv:2303.08774.
* Popper (2005) Karl Popper. 2005. _The logic of scientific discovery_. Routledge.
* Radford et al. (2018) Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. 2018. Improving language understanding by generative pre-training.
* Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. _OpenAI blog_, 1(8):9.
* Ramakrishnan et al. (2014) Raghunathan Ramakrishnan, Pavlo O Draal, Matthias Rupp, and O Anatole Von Lilienfeld. 2014. Quantum chemistry structures and properties of 13 kilo molecules. _Scientific data_, 1(1):1-7.
* Romero-Paredes et al. (2024a) Bernardino Romero-Paredes, Mohammad Barekatain, Alexander Novikov, Matej Balog, M Pawan Kumar, Emilien Dupont, Francisco JR Ruiz, Jordan S Ellenberg, Pengming Wang, Omar Fawzi, et al. 2024a. Mathematical discoveries from program search with large language models. _Nature_, 625(7995):468-475.
* Romera-Paredes et al. (2024b) Bernardino Romera-Paredes, Mohammad Barekatain, Alexander Novikov, Matej Balog, M Pawan Kumar, Emilien Dupont, Francisco JR Ruiz, Jordan S Ellenberg, Pengming Wang, Omar Fawzi, et al. 2024b. Mathematical discoveries from program search with large language models. _Nature_, 625(7995):468-475.
* Romera-Paredes et al. (2024c) Bernardino Romera-Paredes, MohammadBarekatain, Alexander Novikov, Matej Balog, M Pawan Kumar, Emilien Dupont, Francisco JR Ruiz, Jordan S Ellenberg, Pengming Wang, Omar Fawzi, et al. 2024c. Mathematical discoveries from program search with large language models. _Nature_, 625(7995):468-475.
* Rosenckrantz et al. (1977) Daniel J Rosenkrantz, Richard E Stearns, and Philip M Lewis, II. 1977. An analysis of several heuristics for the traveling salesman problem. _SIAM journal on computing_, 6(3):563-581.
* Seber and Lee (2012) George A. F. Seber and Alan J. Lee. 2012. _Linear Regression Analysis_, 2nd edition edition. Wiley.
* Sharma and Thakur (2023) Gaurav Sharma and Abhishee Thakur. 2023. Chatgpt in drug discovery.
* Shi et al. (2023) Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dolan, Ed H Chi, Nathanael Scharli, and Deny Zhou. 2023. Large language models can be easily distracted by irrelevant context. In _International Conference on Machine Learning_, pages 31210-31227. PMLR.
* Shinn et al. (2023) Noah Shinn, Beck Labash, and Ashwin Gopinath. 2023. Reflexion: an autonomous agent with dynamic memory and self-reflection. _arXiv preprint arXiv:2303.11366_, 2(5):9.
* arXiv_,_Comell University
* Sulsky et al. (1995) Deborah Sulsky, Shi-Jian Zhou, and Howard L Schreyer. 1995. Application of a particle-in-cell method to solid mechanics. _Computer physics communications_, 87(1-2):236-252.
* Sumers et al. (2024a) Theodore Sumers, Shunyu Yao, Karthik Narasimhan, and Thomas Griffiths. Cognitive architectures for language agents. _Transactions on Machine Learning Research_.
* Team (2024a) Anthropic Team. 2024a. The claude 3 model family: Opus, sonnet, haiku.
* Team (2023a) Gemini Team. 2023a. Gemini: A family of highly capable multimodal models. _Preprint_, arXiv:2312.11805.
* Team (2023b) InternLM Team. 2023b. Intermlm: A multilingual language model with progressively enhanced capabilities.
* Team (2023c) Llama2 Team. 2023c. Llama 2: Open foundation and fine-tuned chat models. _Preprint_, arXiv:2307.09288.
* Team (2024b) Llama3 Team. 2024b. The llama 3 herd of models. _arXiv preprint arXiv:2407.21783_.
* Team (2022) PaLM Team. 2022. Palm: Scaling language modeling with pathways. _Preprint_, arXiv:2204.02311.
* Trinh et al. (2024) Trieu H Trinh, Yuhuai Wu, Quoc V Le, He He, and Thang Luong. 2024. Solving olympiad geometry without human demonstrations. _Nature_, 625(7995):476-482.
* Wang et al. (2023a) Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. 2023a. Voygier: An open-ended embodied agent with large language models. _arXiv preprint arXiv:2305.16291_.
* Wang et al. (2023) Hanchen Wang, Tianfan Fu, Yuanqi Du, Wenhao Gao, Kexin Huang, Ziming Liu, Payal Chandak, Shengchao Liu, Peter Van Katwyk, Andreeea Deac, et al. 2023b. Scientific discovery in the age of artificial intelligence. _Nature_, 620(7972):47-60 

Hongyu Wang, Shuming Ma, Li Dong, Shaohan Huang, Huaijie Wang, Lingxiao Ma, Fan Yang, Ruiping Wang, Yi Wu, and Furu Wei. 2023c. Bitnet: Scaling 1-bit transformers for large language models. _arXiv preprint arXiv:2310.11453_.
* R. Wang et al. (2022) Kevin Ro Wang, Alexandre Varienigien, Arthur Conny, Buck Shlegeris, and Jacob Steinhardt. Interpretability in the wild: a circuit for indirect object identification in gpt-2 small. In _The Eleventh International Conference on Learning Representations_.
* Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. _Advances in neural information processing systems_, 35:24824-24837.
* Wei et al. (2023) Jerry Wei, Jason Wei, Yi Tay, Dustin Tran, Albert Webson, Yifeng Lu, Xinyun Chen, Hanxiao Liu, Da Huang, Denny Zhou, et al. 2023. Larger language models do in-context learning differently. _arXiv preprint arXiv:2303.03846_.
* Wuesman et al. (2020) Mignon Wuestman, Jarno Hoekman, and Koen Frenken. 2020. A typology of scientific breakthroughs. _Quantitative Science Studies_, 1(3):1203-1222.
* Yang et al. (2023) Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V. Le, Denny Zhou, and Xinyun Chen. 2023. Large language models as optimizers.
* Young et al. (2024) Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, et al. 2024. Yi: Open foundation models by 01. ai. _arXiv preprint arXiv:2403.04652_.
* Yuan et al. (2024) Weizhe Yuan, Kyunghyun Cho, and Jason Weston. 2024. System-level natural language feedback. In _Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 2773-2789.
* Zhang et al. (2024a) Di Zhang, Wei Liu, Qian Tan, Jingdan Chen, Hang Yan, Yuliang Yan, Jiatong Li, Weiran Huang, Xiangyu Yue, Dongzhan Zhou, et al. 2024a. Chemllm: A chemical large language model. _arXiv preprint arXiv:2402.06852_.
* Zhang et al. (2024b) Ningyu Zhang, Yunzhi Yao, Bozhong Tian, Peng Wang, Shumin Deng, Mengru Wang, Zekun Xi, Shengyu Mao, Jintian Zhang, Yuansheng Ni, et al. 2024b. A comprehensive study of knowledge editing for large language models. _arXiv preprint arXiv:2401.01286_.
* Zhang et al. (2021) Starkson Zhang, Alfredo Pearson, and Zhenting Wang. Autonomous generalist scientist: Towards and beyond human-level automatic research using foundation model-based ai agents and robots (a position).
* Zhao et al. (2024) Yilong Zhao, Chien-Yu Lin, Kan Zhu, Zihao Ye, Lequn Chen, Size Zheng, Luis Ceze, Arvind Krishnamurthy, Tianqi Chen, and Baris Kasikci. 2024. Atom: Low-bit quantization for efficient and accurate llm serving. _Proceedings of Machine Learning and Systems_, 6:196-209.
* Zhong et al. (2023) Zexuan Zhong, Zhengxuan Wu, Christopher D Manning, Christopher Potts, and Danqi Chen. 2023. MQulex: Assessing knowledge editing in language models via multi-hop questions. _arXiv preprint arXiv:2305.14795_.
* Zhou et al. (2020) Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. Large language models are human-level prompt engineers. In _The Eleventh International Conference on Learning Representations_.
* Zhu et al. (2020) Chen Zhu, Ankit Singh Rawat, Manzil Zaheer, Srinadh Bhojanapalli, Daliang Li, Felix Yu, and Sanjiv Kumar. 2020. Modifying memories in transformer models. _arXiv preprint arXiv:2012.00363_.
* Zhu et al. (2023) Xunyu Zhu, Jian Li, Yong Liu, Can Ma, and Weiping Wang. 2023. A survey on model compression for large language models. _arXiv preprint arXiv:2308.07633_.

 

More Related Works

### Classical Language Model Methods for Scientific Optimization

Besides the methods using LLMs for scientific optimization in Section 3, these are several classical studies that have fine-tuned language models (LMs) to function as mutation and crossover operators within evolutionary algorithms Meyerson et al. (2023); Zhang et al. (2024); Li et al. (2024). LMX Meyerson et al. (2023) employs language models guided by few-shot exemplars to generate evolutionary crossovers in tasks like image and code generation, enhancing the model's ability to adapt and innovate across diverse domains. ELM Lehman et al. (2022) trains an LLM to generate code diffs, which serve as the mutation operator, and introduces a fine-tuning method to enhance performance in the Sodarace domain for robotic simulation. EvoPrompting Chen et al. (2024) leverages large language models to evolve neural network architectures by integrating evolutionary search with soft prompt tuning. OptFormer Chen et al. (2022) incorporates trajectories as input for optimization by training a transformer model on extensive hyperparameter optimization datasets. These methods primarily focus on fine-tuning LLMs with domain-specific data to align natural language with scientific information. However, these approaches are domain-bound and demand substantial data which also limits their broader applicability.

### Large Language Models

Language models such as GPT Radford et al. (2018), BERT Kenton and Toutanova (2019), RoBERTa Liu et al. (2019), and Megatron-LM Shoeybi et al. (2019) have led to a learning paradigm shift in natural language processing (NLP). Models are first pre-trained on extensive volumes of unlabeled text corpora with language modeling objectives and then fine-tuned on downstream tasks. Recently, large language models (LLMs) including LLama Team (2024, 2023) ChatGPT OpenAI (OpenAI, 2020) GPT4 OpenAI (2023), PaLM Team (2022), GeminiTeam (2023), and Claude3 Team (2024) have shown great performance in both few-shot and even zero-shot scenarios Brown et al. (2020). To further enhance the interpretability of these LLMs, some research endeavors explain LLMs through attribution analysis Wang et al. (2024); Gurnee et al. (2024). Another line of work aims to retrieve the knowledge explicitly from LLMs as the basis for interpreting them, including the reasoning task Shi et al. (2023) and the QA task Hao et al. (2023); Dhingra et al. (2020).

## Appendix B Results of the Causal Tracing for Different LLMs

Causal tracing has emerged as a pivotal methodology for dissecting and understanding the internal mechanisms of model Didelez and Pigeot (2001). This technique facilitates the identification and modification of specific factual associations within a model without necessitating comprehensive retraining Neuberg (2003). In the context of model editing, causal tracing enables precise interventions by isolating the neural correlates responsible for particular behaviors or outputs. We follow Meng et al. (2022) to build a **causal graph**Neuberg (2003) to describe dependencies between the hidden variables. This graph illustrates numerous pathways from the input on the left to the output (next-word prediction) at the lower right. Our aim is to **determine whether specific hidden state variables are more important than others in the process of recalling a fact**.

To quantify each state's contribution to a correct factual prediction, we analyze all of LLM's internal activations across three runs: **a clean run** that accurately predicts the fact, **a corrupted run** where the prediction is impaired, and **a corrupted-with-restoration run** that evaluates the ability of a single state to restore the correct prediction.

Let \({\mathbb{P}}[o]\), \({\mathbb{P}}_{*}[o]\), and \({\mathbb{P}}_{*,\text{clean $h^{i}_{t}$}}[o]\) denote the probability of emitting the given entity \(o\) under the clean, corrupted, and corrupted-with-restoration runs, respectively; dependence on the input \(x\) is omitted for notational simplicity. The **total effect** (TE) is defined as the difference between two probabilities:

\[\text{TE}={\mathbb{P}}[o]-{\mathbb{P}}_{*}[o].\]

The **indirect effect** (IE) of a specific mediating state \(\hat{h_{i}^{l}}\) is defined as the difference between the probability of \(o\) under the corrupted condition and the probability of \(o\) when that state is restored to its clean version, while the subject remains in a corrupted state:

\[\text{IE}={\mathbb{P}}_{*,\text{clean $h_{i}$}!}[o]-{\mathbb{P}}_{*}[o].\]

By averaging over a sample of statements, we can derive the **average indirect effect (AIE)** for 

each hidden state variable. Specifically, when restoring hidden states from the original run, we substitute the values computed originally for the corresponding layer and token, allowing subsequent computations to proceed without further modification. Taking Llama3 8B as an example, as shown in Figure 4, we observe that the \(18\)-\(22\)th layers for the last subject token demonstrate the most causality in the AIE metric. Therefore, we use the \(18\)-\(22\)th MLP layers as the layers where model edits are applied to update knowledge by modifying the parameters. We also visualize other mentioned LLMs in Figures 5, 6, 7, 8, and 9 to provide a more comprehensive results.

## Appendix C Prompt Templates for Each Task

We list the prompt templates for different tasks to offer more visually intuitive results for each task in Table 7, respectively. More detailed prompt information for the best performance of each task and dataset can be seen within the code.

## Appendix D Details of Augmented Prompts

As mentioned in Section 5.5, we discussed the robustness of the GSO method compared to traditional prompt-based approaches when handling semantically similar prompts. Specifically, we manually generated five prompts and used OpenAI o1 to rewrite an additional five prompts based on the original task prompts. We now list these augmented prompts in Tables 8 and 9 to provide more insights

\begin{table}
\begin{tabular}{l l c c c c c c c} \hline \hline  & \multirow{2}{*}{**Funsearch**} & \multicolumn{2}{c}{**Eureka**} & \multicolumn{2}{c}{**OPRO**} & \multicolumn{2}{c}{**SGA**} & \multicolumn{2}{c}{**GSO**} \\ \cline{1-1} \cline{6-10}
201.1 \(\pm\) 39.0 & 291.0 \(\pm\) 20.8 & 190.0 \(\pm\) 10.3 & 59.7 \(\pm\) 5.9 & **17.1 \(\pm\) 4.0** \\ \hline \hline \end{tabular}
\end{table}
Table 6: We consider an imaginary constitutive law setting to prevent the LLM from cheating by memorization and report the results using Mistral 7B as the representative backbone model.

\begin{table}
\begin{tabular}{p{11.4pt} p{11.4pt} p{11.4pt} p{11.4pt} p{11.4pt} p{11.4pt} p{11.4pt} p{11.4pt} p{11.4pt} p{11.4pt}} \hline \hline \multirow{2}{*}{**Backbone**} & \multirow{2}{*}{**Method**} & \multicolumn{2}{c}{**Linear System**} & \multicolumn{2}{c}{**Travel Salesman**} & \multicolumn{2}{c}{**Constitutive Law**} & \multicolumn{2}{c}{**Molecule Property**} \\ \cline{3-13}  & & **(a)**\(\downarrow\) & **(b)**\(\downarrow\) & **(c)**\(\downarrow\) & **(d)**\(\downarrow\) & **(e)**\(\downarrow\) & **(f)**\(\downarrow\) & **(g)**\(\downarrow\) \\ \hline \multirow{6}{*}{**Llama2 13B**} & **Vanilla CoT** & N/A & 6.0 \(\pm\) 2.0 & N/A & N/A & N/A & N/A & N/A & N/A \\  & **ConT** & N/A & 4.4 \(\pm\) 1.9 & N/A & N/A & N/A & N/A & N/A \\  & **Funsearch** & N/A & 2.5 \(\pm\) 1.0 & 170.9 \(\pm\) 20.1 & 255.1 \(\pm\) 31.1 & 108.5 \(\pm\) 13.0 & 115.5 \(\pm\) 20.7 & 59.7 \(\pm\) 8.3 \\  & **Enreka** & N/A & 2.7 \(\pm\) 1.5 & 211.7 \(\pm\) 50.4 & 131.7 \(\pm\) 21.9 & 98.2 \(\pm\) 11.3 & 220.1 \(\pm\) 30.1 & 39.3 \(\pm\) 13.9 \\  & **OPRO** & 28.7 \(\pm\) 10.8 & 0.9 \(\pm\) 0.3 & 55.8 \(\pm\) 10.4 & 165.8 \(\pm\) 52.9 & 168.5 \(\pm\) 31.7 & 355.4 \(\pm\) 43.7 & 35.9 \(\pm\) 10.7 \\  & **SGA** & 30.0 \(\pm\) 10.2 & 0.2 \(\pm\) 0.1 & 31.7 \(\pm\) 10.1 & 55.4 \(\pm\) 15.8 & 87.1 \(\pm\) 17.0 & 89.5 \(\pm\) 17.7 & 27.6 \(\pm\) 5.4 \\  & **GSO (ours)** & **5.0 \(\pm\) 1.6** & **0.1 \(\pm\) 0.1** & **2.9 \(\pm\) 1.3** & **7.4 \(\pm\) 3.1** & **48.1 \(\pm\) 12.0** & **79.1 \(\pm\) 13.3** & **8.3 \(\pm\) 3.2** \\ \hline \hline \end{tabular}
\end{table}
Table 5: Results of our GSO against **6** baselines using Llama2 13B, Yi 9B, and Interlim 7B as the backbone models. Our experiments encompass **7** different tasks, which are divided into linear system regression (LSR) (**a**), travel salesman problem (TSP) (**b**), constitutive law prediction (**c-d**), and molecule property prediction (**e-g**). For the LSR task, we use the number of steps of successfully finding the optimal solution as the metric. For the TSP task, we use the optimality gap as the metric. For the rest five tasks, we use MSE loss as the metric. We calculate the mean \(\pm\) standard error of each optimization result. The symbol N/A indicates that the model is unable to provide a feasible solution for the current task. A lower value is preferable across all tasks. The best results are highlighted in **bold** text.

 

into our GSO.

## Appendix E More Results of Different Edited Models

As mentioned in Section 5.3, we select GPT-J 6B, Llama3 8B, and Mistral 7B as representative models in Table 1. In this section, to further demonstrate the generalization and versatility of GSO, we also conducted experiments on several popular open-source LLMs at different scales, including Llama2 13B, Yi-9B, and Internlm 7B. As shown in Table 5, we can still observe that our GSO method significantly outperforms existing baselines. This further demonstrates the effectiveness of our GSO approach. We also present radar charts for each model to provide a more intuitive performance comparison in Figures 10 and 11. We apply a linear mapping to assign a score of \(100\) to values with a zero loss. Higher loss values correspond to progressively lower scores, with an inability to answer the question resulting in a score of zero. For formatting reasons, we also provide the radar chart of Llama3 8B. The effectiveness of our GSO across various popular open-source models further demonstrates its strong versatility and generalization capabilities.

## Appendix F More Details of Experiment Setups and Task Definitions

We present more details of experiment setups and task definitions in this section.

Experiment SetupsFor the experimental setup, we apply several representative open-source LLMs: Llama3 8B4, GPT-J-6B5, Llama2 13B6, Yi 9B7, IntemLM 7B8, and Mistral 7B9. Other open-source models can also be replaced based on specific requirements. All experiments were conducted on a single Nvidia A100 GPU (80GB). Our approach is implemented using PyTorch 2.4.11 and Huggingface's Transformers 4.44.21. For each task, we set a maximum of \(100\) iterations, with a limit of \(2048\) tokens generated per iteration. The optimization process terminates if no performance improvement or accurate result is observed after \(10\) consecutive iterations, or if an accurate result is reached earlier. More detailed configurations for the best perfor

Figure 4: Causal tracing visualization results for Llama3 8B. The causal impact on output probability is mapped for (a) the effect of each hidden state on the prediction, (b) the effect of MLP activations alone, and (c) the effect of attention activations alone. We also give according to mean causal traces of over a sample of 1000 factual statements, shown as a line plot with 95% confidence intervals, which is below the first three figures. The confidence intervals confirm that the distinctions between peak and non-peak causal effects at both early and late sites are significant.

 

mance of each task and dataset can be seen within our code.

For task definitions, we provide a more detailed of each task definition and setting.

Linear System RegressionIn linear system regression, the objective is to estimate the linear coefficients that most effectively model the relationship between input variables and their corresponding responses, within a probabilistic framework (Fisher, 1922). We follow the dataset setting as OPRO (Yang et al., 2023) which we include \(437\) samples. Linear system regression is a fundamental technique in statistics and machine learning, widely applied in both theoretical and practical settings (Montgomery et al., 2021; Seber and Lee, 2012).

Traveling Salesman Problem (TSP)The Traveling Salesman Problem (TSP) (Junger et al., 1995; Gutin and Punnen, 2006) is a classical combinatorial optimization problem that has garnered significant attention in the literature, with numerous algorithms proposed, ranging from heuristic methods to exact solvers (Rosenkrantz et al., 1977; Golden et al., 1980; Gurobi Optimization, LLC, 2024; Helsgaun, 2017). Recently, the problem has also been approached through the use of deep neural networks (Kool et al., 2018; Deudon et al., 2018; Chen and Tian, 2019; Nazari et al., 2018), underscorting its adaptability to modern machine learning techniques. Formally, given a set of \(n\) nodes with known coordinates, the TSP seeks to determine the shortest possible route that visits each node exactly once and returns to the starting point.We follow the dataset setting as OPRO (Yang et al., 2023), which we include \(177\) TSP problems. At each optimization step, a maximum of \(8\) new solutions are generated. To evaluate the performance of various methods, we utilize the Gurobi solver (Gurobi Optimization, LLC, 2024) to generate oracle solutions and compute the optimality gap. The optimality gap is defined as the relative difference between the distance of the solution obtained by the tested approach and that of the oracle solution, normalized by the oracle solution's distance.

Constitutive Law PredictionIdentifying constitutive laws from elastic material remains one of the most challenging tasks in fields such as physics, materials science, and mechanical engineering. In this paper, we follow the dataset settings as (Ma et al., 2023a; Ma et al.), including \(357\) different linear and non-linear materials and utilizing differentiable Material Point Method (MPM) simulators (Sulsky et al., 1995; Jiang et al., 2016). The primary objective of this task is to uncover both the discrete structure and continuous parameters of a constitutive law, specifically identifying the material models \(\varphi(\cdot)\) along with their associated mate

Figure 5: Causal tracing visualization results for GPT-J 6B. The causal impact on output probability is mapped for (a) the effect of each hidden state on the prediction, (b) the effect of MLP activations alone, and (c) the effect of attention activations alone. We also give according to mean causal traces of over a sample of 1000 factual statements, shown as a line plot with 95% confidence intervals, which is below the first three figures. The confidence intervals confirm that the distinctions between peak and non-peak causal effects at both early and late sites are significant.

 

rial parameters \(\theta\), from a ground-truth trajectory of particle positions \(\hat{X}_{t\in[1,...,T]}\), where \(T\) represents the number of time steps. In this context, we consider two classes of constitutive laws: \(\varphi_{E}(\cdot;\theta_{E})\) for modeling elastic materials and \(\varphi_{P}(\cdot;\theta_{P})\) for modeling plastic materials, both of which contain both linear and non-linear materials and are formally defined as the following:

\[\begin{split}&\varphi_{E}\left(\mathbf{F};\theta_{E}\right)\mapsto \boldsymbol{\tau}\\ &\varphi_{P}\left(\mathbf{F};\theta_{P}\right)\mapsto\mathbf{F}^{ \text{corrected}},\end{split}\]

where \(\mathbf{F}\in\mathbb{R}^{3\times 3}\) is the deformation gradient, \(\tau\in\mathbb{R}^{3\times 3}\) is the Kirchhoff stress tensor, \(\mathbf{F}^{\text{corrected}}\in\mathbb{R}^{3\times 3}\) is the deformation gradient after elastic return-mapping correction, and \(\theta_{E}\) and \(\theta_{P}\) are the continuous material parameters for elastic and elastic constitutive laws respectively. Given a specific constitutive law, we input it to the differentiable simulation and yields a particle position trajectory:

\[X_{t\in[1,...,T]}=\text{sim}\left(\varphi\left(\cdot;\theta\right)\right),\]

and we optimize the constitutive law by fitting the output trajectory to the ground truth \(\hat{X}_{t\in[1,...,T]}\).

#### Molecule Property Prediction

Predicting the Highest Occupied Molecular Orbital (HOMO), Lowest Unoccupied Molecular Orbital (LUMO), and HOMO-LUMO gap value of molecules is a key challenge in computational chemistry, particularly in the fields of quantum chemistry, material science, and drug design. In this task, we aim to predict the HOMO value based on molecular descriptors, such as molecular weight, Topological Polar Surface Area (TPSA), Octanol-water partition coefficient (logP), etc (maximum to 2k different properties). Although traditional quantum mechanical methods such as Density Functional Theory (DFT) are widely used for HOMO calculations, their computational cost can be prohibitive for large datasets. In this work, we adopt a machine learning approach to predict the HOMO value using these physical-chemical properties as input features. We follow the dataset settings as Ma et al. (2023a); Ma et al. (2023b), including \(238\) different molecules.

The primary goal of this task is to construct a model that accurately predicts the HOMO energy level \(\epsilon_{\text{HOMO}}\) by learning a mapping \(\psi(\cdot)\) between the input molecular properties \(x\in\mathbb{R}^{n}\), such as molecular weight, TPSA, and logP, and the HOMO energy \(\epsilon_{\text{HOMO}}\in\mathbb{R}\). The machine learning model is trained on a dataset containing known molecular properties and their corresponding HOMO values, and optimized to minimize the prediction error relative to the true HOMO values \(\hat{\epsilon}_{\text{HOMO}}\).

Given a set of molecular descriptors, the model outputs the predicted HOMO value:

\[\epsilon_{\text{HOMO}}=\psi(x),\]

Figure 6: Causal tracing visualization results for Llama2 13B. The causal impact on output probability is mapped for (a) the effect of each hidden state on the prediction, (b) the effect of MLP activations alone, and (c) the effect of attention activations alone. We also give according to mean causal traces of over a sample of 1000 factual statements, shown as a line plot with 95% confidence intervals, which is below the first three figures. The confidence intervals confirm that the distinctions between peak and non-peak causal effects at both early and late sites are significant.

 

and the objective is to minimize the loss function, defined as the difference between the predicted and true HOMO values:

\[L=\left\|\psi(x)-\hat{\epsilon}_{\text{HOMO}}\right\|^{2}.\]

This approach provides a computationally efficient alternative to traditional quantum mechanical methods, offering the potential for high-throughput screening of molecular libraries for their electronic properties, including HOMO values.

## Appendix G More Results of Ablation Study

In Section 5.4, we report the results of the ablation study using Llama3 8B as the backbone model. In this section, we will further present the results using GPT-J-6B, Llama2 13B, Yi 9B, InternLM 7B, and Mistral 7B as backbone models to obtain more insights into the individual components constituting GSO across various backbone models. As illustrated in Tables 11, 12, 13, 14, and 15, we still observe that the absence of each component within GSO leads to a decline in performance across diverse domains for almost all applied backbone models in the seven tested scientific optimization tasks, which further demonstrates that GSO organically integrates the two-level optimization into a unified framework as well. We also observe that the absence of model editing leads to a more significant decline in accuracy, which further demonstrates the importance of effectively utilizing observational feedback to adaptively adjust the optimization direction.

## Appendix H More Case Study on Generalization or Memorization

To investigate whether the improvements brought by our method are solely due to the LLM having seen solutions during the training phase, we designed an experiment aimed at eliminating this factor by having the model invent an imaginary constitutive law that does not exist on Earth. We combined the von Mises plasticity constitutive law, granular material, and weakly compressible fluid in proportions of \(50\%\), \(30\%\), and \(20\%\), respectively, creating a new constitutive law that represents an exceedingly complex hypothetical material. As shown in Table 6, our method was still able to discover the constitutive law with minimal quantitative loss compared to other existing baselines. These results also indicate that GSO does not simply rely on memorization to achieve results but instead indeed effectively performs optimization.

## Appendix I Inference Time Comparisons

We note that GSO requires an additional process to utilize the optimization feedback by applying model editing techniques to update LLM's parameters. Compared to the vanilla model, this process

Figure 7: Causal tracing visualization results for Yi 9B. The causal impact on output probability is mapped for (a) the effect of each hidden state on the prediction, (b) the effect of MLP activations alone, and (c) the effect of attention activations alone. We also give according to mean causal traces of over a sample of 1000 factual statements, shown as a line plot with 95% confidence intervals, which is below the first three figures. The confidence intervals confirm that the distinctions between peak and non-peak causal effects at both early and late sites are significant.

 

could potentially introduce extra inference time. However, we also observe that GSO effectively reduces the input prompt length through this approach, thereby decreasing the LLM's inference time. Hence, to explore the interplay between these two effects, we record and compare the average inference time per sample of different methods using a single Nvidia A100 GPU (80GB), including vanilla, CoT, OPRO, Eureka, Funsearch, SGA, and GSO on each mentioned scientific optimization task to explore the influence of additional inference time and provide more insight of our GSO.

We report the average inference time per sample as a metric, as shown in Table 10. We observe from the table that although the incorporation of model editing within the GSO framework introduces just marginal additional inference time costs. On average, the increase in inference time cost per sample due to the introduction of GSO, compared to the vanilla LLM, is \(7.75\text{s}\). Notably, when utilizing Mistral 7B as the backbone model, this additional inference time is only \(5.8\text{s}\), yet it offers a maximum precision improvement of 32.6\(\times\) in the HOMO-LUMO gap prediction task compared to all other methods. Furthermore, GSO exhibits higher inference efficiency compared to existing methods. We may focus on exploring ways to further reduce the time cost of the GSO, including lightweight-LLMs to get a faster calculation Zhu et al. (2023); Hsieh et al. (2023) or adopting more efficient Wang et al. (2023c); Zhao et al. (2024); Frantar and Alistarh (2023) and rational large model inference strategies such as speculative decoding Leviathan et al. (2023); Liu et al. (2024a) as for future works.

## Appendix J Scientific Artifacts

The data we collect in specialized domains is publicly available and viewable online. The data owners have indicated that the data can be used for scientific research or have not indicated that the data cannot be used for scientific research, and our collection process is also in compliance with regulations. Moreover, there is no unique identification of individuals or offensive content in these data.

## Appendix K More Discussions On GSO

What is the key advantage of using LLMs to optimize over some traditional optimization algorithms, especially on classical optimization problems?

The key advantage is that one can use natural language to describe the optimization problem. Instead of formally defining the optimization problem and deriving the update step with a programmed solver. This makes optimization more accessible to general users who may not have extensive domain knowledge of the specific types of optimization tasks in question, and it may also enhance

Figure 8: Causal tracing visualization results for Internlm 7B. The causal impact on output probability is mapped for (a) the effect of each hidden state on the prediction, (b) the effect of MLP activations alone, and (c) the effect of attention activations alone. We also give according to mean causal traces of over a sample of 1000 factual statements, shown as a line plot with 95% confidence intervals, which is below the first three figures. The confidence intervals confirm that the distinctions between peak and non-peak causal effects at both early and late sites are significant.

 

the productivity of optimization experts who work on these tasks daily. Moreover, employing LLMs for optimization is both domain-agnostic and task-agnostic, offering excellent universality and generalization. It enables rapid understanding and optimization of relatively unknown problems and can assist in finding more refined solutions. With the rapid advancement of LLMs, their abilities are improving swiftly, indicating the great potential of LLM-based optimization methods to address more complex and even unknown optimization tasks.

What specific impact does the adoption of the exploitation/exploration phase actually have, given that sometimes satisfactory results can be achieved without introducing it?

To investigate how the dynamic exploitation/exploration strategy works, we designed an ablation experiment in Section 5.4 and Appendix G. Statistically, the strategy has a relatively positive effect, especially for tasks like predicting the HOMO-LUMO gap of molecules. We find that: **Without exploitation**, GSO sometimes finds it difficult to effectively provide consistent high-quality parameter hypotheses. **Without exploration**, GSO sometimes becomes trapped in local optima, unable to further optimize the task to gain additional insights. Consequently, using a balanced exploitation/explo

Figure 10: GSO achieves state-of-the-art performance on a broad range of scientific optimization tasks compared with existing methods, using LLama 3 7B, GPT-J 6B, and Llama2 13B as backbone models, respectively. We linearly map the evaluation metrics to [0, 100] for presentation.

Figure 9: Causal tracing visualization results for mistral 7B. The causal impact on output probability is mapped for (a) the effect of each hidden state on the prediction, (b) the effect of MLP activations alone, and (c) the effect of attention activations alone. We also give according to mean causal traces of over a sample of 1000 factual statements, shown as a line plot with 95% confidence intervals, which is below the first three figures. The confidence intervals confirm that the distinctions between peak and non-peak causal effects at both early and late sites are significant.

 

ration strategy enables one to escape local optima and obtain consistent loss decay. As shown in Figure 3, GSO with this strategy is the only one that continues to progress toward better results, whereas others exhibit plateaued stagnation curves.

Is GSO still effective when dealing with high-dimensional data or extremely intricate optimization tasks?

Our experimental design includes both simple, low-dimensional ideal, and high-dimensional complex scientific optimization problems aimed at minimizing the sim-to-real gap to the greatest extent possible. This ensures that GSO generates meaningful and practical optimizations rather than questionable boys. We believe this is a strength rather than a weakness for GSO. Moreover, the success of GSO stems from the effective utilization of optimization feedback at each step and a dynamic exploration/exploitation strategy, which is domain-agnostic and can be applied to other domains. We also acknowledge that when the underlying physics of the opti

\begin{table}
\begin{tabular}{p{142.3pt}|p{170.7pt}} \hline \hline
**Optimization Tasks** & **Prompt Templates** \\ \hline Linear System Regression & You will help me minimize a function with two input variables w, b. I have some (w, b) pairs and the function values at those points. The pairs are arranged in descending order based on their function values, where lower values are better. \\  & Below are some examples: str(Examples) \\  & Give me a new (w, b) pair that is different from all pairs above, and has a function value lower than any of the examples. \\ \hline Travel Salesman Problem & You are given a list of points with coordinates below: str(Nodes) \\  & Below are some previous traces and their lengths. The traces are arranged in descending order based on their lengths, where lower values are better: str(Traces) \\  & Give me a new trace that is different from all traces above, and has a length lower than any of the above. The trace should traverse all points exactly once. The trace should start with ¡trace¿ and end with ¡/trace¿. \\ \hline Constitutive Law Prediction (Linear and Non-Linear) & You are given a list of materials with corresponding properties below: str(Properties) \\  & Below are some previous stress-strain responses for each material, sorted in descending order based on compliance, where lower compliance values indicate better structural integrity: str(Examples) \\  & Provide a new constitutive law that differs from all the laws above, ensuring that it produces a compliance value lower than any of the existing responses. The constitutive law should describe the relationship between stress and strain accurately for the material and should start with ¡law¿ and end with ¡/law¿. \\ \hline Molecule Property Prediction (HOMO value, LUMO value, and HOMO-LUMO gap) & You are given a list of molecules with their chemical properties below: str(Properties). Your goal is to predict the HOMO (can be adjusted to LUMO or HOMO-LUMO gap depending on different tasks) values for the listed molecules. Below are some examples: str(Examples) \\  & The HOMO value for each molecule should be distinct from any previously reported values, and the predictions should ensure an accurate representation of their electronic properties. Each predicted value should be formatted to start with ¡value¿ and end with ¡/value¿ \\ \hline \hline \end{tabular}
\end{table}
Table 7: Prompt templates for each task. In the prompts, “str(Examples)” represents initial solutions for each task. “str(Nodes)” and ”str(Properties)” represent some basic properties of the materials/molecules.

 



\begin{table}
\begin{tabular}{l|p{142.3pt}} \hline
**Optimization Tasks** & **Augmented Prompts** \\ \hline Linear System Regression & (i) You will help me minimize a function with two input variables w, b. I have some (w, b) pairs and the function values at those points. The pairs are arranged in descending order based on their function values, where lower values are better. \\ Please provide a new (w, b) pair distinct from those above, with a lower function value than any previous pair. \\ (ii) You are assisting in minimizing a function with two variables w and b. Provided are some (w, b) pairs along with their function values, sorted in descending order where lower values are better. \\ Generate a (w, b) pair not seen in the above list, ensuring it yields a function value lower than any listed. \\ (iii) Help me find the minimum of a function dependent on variables w and b. Below are (w, b) pairs and their corresponding function values, arranged from highest to lowest (lower is better). \\ Give a (w, b) pair not seen in the above list, ensuring it yields a function value lower than any listed. \\ (iv) I need assistance in minimizing a function with inputs w and b. Here are some (w, b) pairs and their function values, listed in descending order of their function values (lower values indicate better results). \\ Give me a better (w, b) pair that is not included above. \\ (v) Your task is to help minimize a function of two variables w and b. The following are (w, b) pairs and their function values, sorted from highest to lowest (lower values are preferable). \\ Provide a new and better (w, b) pair from those above. \\ \hline Travel Salesman Problem & (i) Given the coordinates of points: strr(Nodes). Here are some previous routes and their lengths, sorted from longest to shortest (shorter is better). Give me a new trace that is different from all traces above, and has a length lower than any of the above. The trace should traverse all points exactly once. The trace should start with ¡trace¿ and end with ¡/trace¿. \\ (ii) Consider the following points with coordinates: strr(Nodes). Generate a new trace that differs from all previous traces and is shorter than any of them. Ensure that this trace covers each point exactly once, begins with ¡trace¿, and ends with ¡/trace¿. \\ (iii) You have a set of points at these coordinates below: strr(Nodes). Please provide a unique trace that is distinct from all preceding traces and has a length shorter than any of the prior traces. This trace should start with ¡trace¿ and conclude with ¡/trace¿, while visiting each point one time. \\ (iv) Here are points with their coordinates: strr(Nodes). Please craft a new trace that is unique from all previous traces and shorter in length than any listed above. This trace should traverse all points a single time, beginning with ¡trace¿ and ending with ¡/trace¿. \\ (v) You are provided with points with their coordinates: strr(Nodes). Produce a trace that does not resemble any of the existing traces and has a length less than the shortest one listed. It should begin with ¡trace¿, finish with ¡/trace¿, and cover each point exactly once. \\ \hline \end{tabular}
\end{table}
Table 8: Augmented prompt templates for linear system regression and travel salesman problem. In the prompts, ”str(Examples)” represents initial solutions for each task. “str(Nodes)” represents some basic properties of the materials.





\begin{table}
\begin{tabular}{p{142.3pt}|p{85.4pt}} \hline
**Optimization Tasks** & **Augmented Prompts** \\ \hline Constitutive Law Prediction (Linear and Non-Linear) & (i) You have been provided with a list of materials and their associated properties: strr(Properties). You will find previous stress-strain responses for each material, where lower compliance indicates stronger structural integrity: strr(Examples) Create a new constitutive law that is distinct from all those listed. The law should begin with <law> and end with </law>. (ii) Provided below are some materials with their properties: str(Properties). It also includes prior stress-strain responses for each material, where lower values reflect better structural integrity: strr(Examples). Develop a new constitutive law that is different from all previous laws. The law should start with <law> and end with </law>. (iii) You have a list of materials and their properties outlined below: str(Properties). Alongside, there are previous stress-strain responses for each material with lower compliance suggesting stronger structural integrity: strr(Examples). Formulate a new constitutive law. This law should start with <law> and end with </law>. (iv) Below is a list of materials and their corresponding properties: str(Properties) The data is in descending order, where a lower compliance value indicates greater structural integrity: strr(Examples). Construct a new constitutive law that is distinct from all the above, ensuring that it yields a compliance value below all existing responses. The law should start with <law> and end with </law>. (v) A list of materials and their properties is given below: str(Properties). Below are previous responses, where lower values suggest better structural integrity: str(Examples). Design a new constitutive law. The law should begin with <law> and end with </law>. (i) Provided below is a list of molecules and their properties: str(Properties). Predict the HOMO values for these molecules (or adjust to LUMO/HOMO-LUMO gap as needed). Ensure each HOMO value accurately represents electronic properties. Format predictions with <value> at the start and </value> at the end. (ii) Below are molecules with their chemical properties: str(Properties). Your task is to predict HOMO values (or LUMO/HOMO-LUMO gap) for each. Make sure predictions reflect electronic properties accurately, formatted as <value>...</value>. (iii) You have a list of molecules with properties: str(Properties). Predict the HOMO (or LUMO/HOMO-LUMO gap) values. Use <value>...</value> to format predictions. (iv) Here are molecules and properties: str(Properties). Predict HOMO values (or LUMO/HOMO-LUMO gap) that capture properties accurately. Prediction should begin with <value> and end with </value>. (v) Provided is a list of molecules: str(Properties). Predict HOMO values (or LUMO/HOMO-LUMO gap), ensuring accurate electronic property representation. Format each as <value>..</value>. \\ \hline \end{tabular}
\end{table}
Table 9: Augmented prompt templates for constitutive law prediction and molecule property prediction. In the prompts, “str(Examples)” represents initial solutions for each task. ”str(Properties)” represent some basic properties of the molecules.



mization problem is too complex, this represents a limitation of our approach. A potential extension could focus on data compression by pre-training an auto-encoder to project high-dimensional data into a latent space.

What is the difference between updating LLM parameters using model editing and employing methods like fine-tuning?

The differences are two-folds:(**i**) the key difference lies in their **scope and flexibility**. Fine-tuning typically involves adjusting a large portion of the model's parameters over a dataset, often requiring extensive computational resources and time Hu et al. (2021); Zhu et al. (2020), and it can result in overfitting or catastrophic forgetting of prior knowledge Luo et al. (2023). Model editing is a more targeted approach, allowing for precise modifications to specific parts of the model in response to new information or tasks without altering the broader knowledge encoded within the model. (ii) Another difference is its **efficiency**; it enables rapid updates to the model without the need for extensive retraining. This makes it particularly suitable for dynamic optimization tasks where quick adjustments are necessary. Moreover, model editing preserves the generalization abilities of the LLM while fine-tuning may risk degrading its performance on unrelated tasks. In this sense, model editing offers a more controlled and adaptive method for refining LLM behavior, especially in domain-agnostic and task-agnostic scenarios, making it an ideal tool for iterative optimization processes.

\begin{table}
\begin{tabular}{l c c c c c c c} \hline \hline Backbone Models & Vanilla & CoT & OPRO & Eureka & Funsearch & SGA & GSO (ours) \\ \hline Llama3 BB & 21.9 & 21.9 & 34.0 & 32.1 & 40.9 & 37.3 & 28.9 \\ GPT-J 6B & 20.1 & 21.3 & 30.8 & 27.5 & 44.5 & 35.8 & 29.0 \\ Llama2 13B & 40.2 & 41.4 & 53.3 & 47.6 & 61.0 & 55.3 & 51.0 \\ Yi 9B & 28.8 & 28.9 & 38.8 & 41.0 & 53.3 & 50.5 & 35.9 \\ Internlm 7B & 16.9 & 17.4 & 25.4 & 21.0 & 36.9 & 37.0 & 23.8 \\ Mistral 7B & 18.2 & 18.5 & 27.2 & 20.4 & 33.9 & 37.3 & 24.0 \\ \hline \hline \end{tabular}
\end{table}
Table 10: The inference time (per sample on average) on the TSP task for the baseline methods, including Vanilla, CoT, OPRO, Eureka, Funsearch, SGA, GSO, respectively. We report the results using Llama3 8B, GPT-J 6B, Llama2 13B, YiqB, Internlm 7B, and Mistral 7B as backbone models, respectively. (Unit: seconds)

Figure 11: GSO achieves state-of-the-art performance on a broad range of scientific optimization tasks compared with existing methods, using Yi 9B, Internlm 7B, and Mistral 7B as backbone models, respectively. We linearly map the evaluation metrics to [0, 100] for presentation.

 



\begin{table}
\begin{tabular}{l|c c c c c c c} \hline \multirow{2}{*}{**Method**} & **Linear System** & **Travel Salesman** & \multicolumn{3}{c}{**Constitutive Law**} & \multicolumn{3}{c}{**Molecule Property**} \\ \cline{2-9}  & **(a)**\(\downarrow\) & **(b)**\(\downarrow\) & **(e)**\(\downarrow\) & **(d)**\(\downarrow\) & **(e)**\(\downarrow\) & **(f)**\(\downarrow\) & **(g)**\(\downarrow\) \\ \hline
**GSO\({}_{w/o~{}obst}\)** & 9.0 \(\pm\) 1.5 & 0.4 \(\pm\) 0.1 & 55.4 \(\pm\) 10.1 & 408.1 \(\pm\) 39.0 & 821.0 \(\pm\) 112.0 & 982.1 \(\pm\) 101.5 & 155.0 \(\pm\) 30.1 \\
**GSO\({}_{w/o~{}dynamic}\)** & 5.5 \(\pm\) 1.7 & **0.0 \(\pm\) 0.0** & 19.1 \(\pm\) 3.0 & **80.9 \(\pm\) 5.0** & 73.0 \(\pm\) 5.3 & 194.0 \(\pm\) 20.1 & 9.0 \(\pm\) 2.3 \\ \hline
**GSO (ours)** & **3.0 \(\pm\) 0.8** & **0.0 \(\pm\) 0.0** & **5.9 \(\pm\) 2.1** & 89.1 \(\pm\) 33.9 & **67.9 \(\pm\) 23.1** & **172.9 \(\pm\) 43.1** & **5.5 \(\pm\) 2.1** \\ \hline \end{tabular}
\end{table}
Table 13: The results of the ablation study of our GSO on the seven scientific optimization tasks, using Yi 9B as the backbone model.

\begin{table}
\begin{tabular}{l|c c c c c c c} \hline \multirow{2}{*}{**Method**} & **Linear System** & **Travel Salesman** & \multicolumn{3}{c}{**Constitutive Law**} & \multicolumn{3}{c}{**Molecule Property**} \\ \cline{2-9}  & **(a)**\(\downarrow\) & **(b)**\(\downarrow\) & **(e)**\(\downarrow\) & **(d)**\(\downarrow\) & **(e)**\(\downarrow\) & **(f)**\(\downarrow\) & **(g)**\(\downarrow\) \\ \hline
**GSO\({}_{w/o~{}obst}\)** & 13.0 \(\pm\) 2.5 & 0.3 \(\pm\) 0.1 & 55.4 \(\pm\) 13.5 & 31.7 \(\pm\) 10.4 & 151.1 \(\pm\) 20.0 & 45.4 \(\pm\) 10.1 & 115.3 \(\pm\) 20.1 \\
**GSO\({}_{w/o~{}dynamic}\)** & 9.4 \(\pm\) 2.0 & **0.0 \(\pm\) 0.0** & 14.1 \(\pm\) 3.0 & 16.0 \(\pm\) 3.0 & 47.3 \(\pm\) 6.8 & 10.5 \(\pm\) 1.9 & 16.8 \(\pm\) 4.0 \\ \hline
**GSO (ours)** & **5.6 \(\pm\) 3.0** & 0.1 \(\pm\) 0.0 & **3.2 \(\pm\) 1.7** & **13.0 \(\pm\) 2.1** & **23.9 \(\pm\) 5.1** & **2.8 \(\pm\) 1.5** & **1.7 \(\pm\) 0.4** \\ \hline \end{tabular}
\end{table}
Table 15: The results of the ablation study of our GSO on the seven scientific optimization tasks, using mistral 7B as the backbone model.

\begin{table}
\begin{tabular}{l|c c c c c c c} \hline \multirow{2}{*}{**Method**} & **Linear System** & **Travel Salesman** & \multicolumn{3}{c}{**Constitutive Law**} & \multicolumn{3}{c}{**Molecule Property**} \\ \cline{2-9}  & **(a)**\(\downarrow\) & **(b)**\(\downarrow\) & **(e)**\(\downarrow\) & **(d)**\(\downarrow\) & **(e)**\(\downarrow\) & **(f)**\(\downarrow\) & **(g)**\(\downarrow\) \\ \hline
**GSO\({}_{w/o~{}obst}\)** & 13.0 \(\pm\) 2.5 & 0.3 \(\pm\) 0.1 & 55.4 \(\pm\) 13.5 & 31.7 \(\pm\) 10.4 & 151.1 \(\pm\) 20.0 & 45.4 \(\pm\) 10.1 & 115.3 \(\pm\) 20.1 \\
**GSO\({}_{w/o~{}dynamic}\)** & 9.4 \(\pm\) 2.0 & **0.0 \(\pm\) 0.0** & 14.1 \(\pm\) 3.0 & 16.0 \(\pm\) 3.0 & 47.3 \(\pm\) 6.8 & 10.5 \(\pm\) 1.9 & 16.8 \(\pm\) 4.0 \\ \hline
**GSO (ours)** & **5.6 \(\pm\) 3.0** & 0.1 \(\pm\) 0.0 & **3.2 \(\pm\) 1.7** & **13.0 \(\pm\) 2.1** & **23.9 \(\pm\) 5.1** & **2.8 \(\pm\) 1.5** & **1.7 \(\pm\) 0.4** \\ \hline \end{tabular}
\end{table}
Table 11: The results of the ablation study of our GSO on the seven scientific optimization tasks, using GPT-J 6B as the backbone model. The symbol N/A indicates that the model is unable to provide a feasible solution for the current task.