PINNsAgent: Automated PDE Surrogation with Large Language Models
Authors: Qingpo Wuwu
Chonghan Gao
Tianyu Chen
Yihang Huang
Yuekai Zhang
Jianing Wang
Jianxin Li
Haoyi Zhou
Shanghang Zhang
Venue: ICML 2025 poster (2025)
Keywords: pinns
llm-agent
Primary area: applications->chemistry_physics_and_earth_sciences

Abstract: Solving partial differential equations (PDEs) using neural methods has been a long-standing scientific and engineering research pursuit. Physics-Informed Neural Networks (PINNs) have emerged as a promising alternative to traditional numerical methods for solving PDEs. However, the gap between domain-specific knowledge and deep learning expertise often limits the practical application of PINNs. Previous works typically involve manually conducting extensive PINNs experiments and summarizing heuristic rules for hyperparameter tuning. In this work, we introduce PINNsAgent, a novel surrogation framework that leverages large language models (LLMs) to bridge the gap between domain-specific knowledge and deep learning. PINNsAgent integrates Physics-Guided Knowledge Replay (PGKR) for efficient knowledge transfer from solved PDEs to similar problems, and Memory Tree Reasoning for exploring the search space of optimal PINNs architectures. We evaluate PINNsAgent on 14 benchmark PDEs, demonstrating its effectiveness in automating the surrogation process and significantly improving the accuracy of PINNs-based solutions.

PINNsAgent: Automated PDE Surrogation with Large Language Models
Qingpo Wuwu 1* Chonghan Gao Tianyu Chen Yihang Huang² Yuekai Zhang' Jianing Wang'
Jianxin Li Haoyi Zhou? Shanghang Zhang'
Abstract
ods, such as finite difference (Strikwerda, 2004), finite cle- ment (Hughes, 2000), and finite volume methods (Le (LeVeque, 2002), often incur significant computational costs, struggle to handle nonlinearities and complex geometries (Canuto et al., 2007: Berger & Oliger, 1984), motivating the devel- opment of data-driven alternatives. Physics-Infee Informed Neu- ral Networks (PINNs) have recently emerged as a promis- ing deep learning- based approach for solving PDEs (Raissi et al., 2017; 2019). However, designing effective neural architectures for PINNs heavily relies on expert knowledge and often requires extensive trial-and-error to mitigate train- ing pathologies (Wang et al., 2023b), prompting extensive research to explore optimal PINNS architectures and hyper- parameters through numerous manual experiments. (Wang et al., 2022) conducted a comprehensive study to understand the relationship between PINNs architectures and their per- formance, while (Kaplarevié-Mališić et al., 2023) manually explored various evolutionary strategies for PINNs archi- tecture optimization. Similarly, (Wang & Zhong, 2024) investigated the impact of different architectural choices on PINNs performance through extensive experiments. These works often summarize their findings as thumb rules or guidelines for several PDE families to assist non-experts in manually tuning PINNs architectures and hyperparameters. However, this manual approach is time-consuming, labor- intensive, and may not generalize well to a wide range of PDEs, highlighting the need for an automated framework capable of hierarchically formulating and optimizing PINNs architectures for given PDEs.
Solving partial differential equations (PDEs) us- ing neural methods has been long-standing scientific and engineering research pursuit. Physics-Informed Neural Networks (PINNs) have emerged as a promising alternative to traditional numerical methods for solving PDEs. However, the gap between domain-specifie knowledge and deep learning expertise often limits the practical application of PINNs. Previous works typically involve manually conducting extensive PINNs ex- periments and summarizing heuristic rules for hyperparameter tuning. In this work, we intro- duce PINNsAgent, a novel surrogation framework that leverages large language models (LLMs) bridge the gap between domain-specifie knowl- edge and deep learning. PiNNsAgent integrates Prysscs--uided Knowledge Replay (PGKR) for efficient knowledge transfer from solved PDEs to similar problems, and Memory Tree Reason- ing for exploring the search space of optimal PINNs architectures. We evaluate PINNsAgent on 14 benchmark PDEs, demonstrating its effective- ness in automating the surrogation process and significantly improving the accuracy of PINNs- based solutions. Project website: https:// Langgoowuw..grthuie 1o/P INNsAgent/.
1. Introduction
Recent advancements in large language model (LLM) intelligent agents have showcased their capacity in the realm of scientific computing, automating tasks such as code gen- eration (Nijkamp et al., 2023; Huang et al., 2023; Madaan et al., 2024; Wang et al., 2023c), hyper-parameter tuning (Zhang et al., 2023), and physical modeling (Alexiadis Ghiassi, 2024; Ali-Dib & Menou, 2023). These agents leverage the vast knowledge encoded within LLMs pro- vide intelligent assistance and recommendations, opening up new possibilities for accelerating scientific research and development. However, their application in developing deep learning- based solvers for PDEs has yet to fully exploit the potential of leveraging existing database as foundational knowledge and experimental logs as iterative feedback. We
Solving partial differential equations (PDEs) is fundamen- tal challenge with wide-ranging applications across various scientific and engineering domains, including fluid dynam- ics (Kutz, 2013), quantum mechanics (Teschl, 2014), and cli- mate modeling (Stocker, 2011). Traditional numerical meth-
"Equal contribution 'State Key Laboratory of Multimedia Infor- mation Processing, School of Computer Science, Peking Univer- sity "School of Computer Science, Beihang University "School of Artificial Intelligence, Beijing Normal University. Correspondence to: Shanghang Zhang <shanghang @ www.cdu.en......
Proceedings of the 4nnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 20255 by the author(s).

---

PINNsAgent: Automated PDE Surrogation with Large Language Models
aim to bridge this gap by developing an LLM-based intelli-
2. Related Work
gent agent (Brown et al., 2020) framework to autonomously optimize Physics-Informee Neural Networks (PINNs) archi- tectures for solving Partial Differential Equations (PDEs) without relying on manual tuning or expert heuristics.
2.1. Learned PDE Solvers
Data-driven PDE solvers have garnered significant atten- tion since (Raissi et al., 2017; 2019) first propose physics- informed neural networks (PINNs) to solve nonlinear PDEs by using automatic differentiation to embed PDE residuals into the loss function. However, vanilla PINNs exhibit vari- ous limitations in accuracy, efficiency, and generalizability, prompting extensive research towards developing improved differentiable neural network PDE solvers (Cuomo et al., 2022). For instance, (Yu et al., 2022; Wang et al., 2021) proposed loss functions that incorporate gradient enhance- ment of the PDE residual to improve model stability and accuracy. Another approach, described (Jagtap al., 2020b;a; 2022), introduced adaptive activation functions to reduce the inefficiency of trial and error in network train- ing. Furthermore, to enhance computational efficiency and adapt to complex geometries, (Nabian et al., 2021; Shukla et al., 2021; Jagtap & Karniadakis, 2020) introduced impor- tance sampling and domain decomposition. Despite these advancements, the selection and design of PINNS still pose barriers for non-experts. Our proposed PINNsAgent frame- work aims to provide an automated surrogate framework for proposing PINNs architectures to solve user-provided PDEs.
To this end, we introduce PINNsAgent, an innovative LLM- based surrogate framework that leverages LLMs as intelli- gent agents to develop and optimize PINNS autonomously. PINNsAgent comprises multi-agent system, including database that accumulates past experimental logs, a plan- ner that generates candidate architectures and guides the exploration process, a programmer that translates designed architectures into executable code, and a code bank for stor- ing and retrieving successful implementations. To efficiently utilize the knowledge stored in the database, we propose new retrieval framework called Physics-Guide Knowledge Replay (PGKR) that encodes the essential characteristics of PDEs. Inspired by insights from existing literature (Wang et al., 2023b; 2022; Rathore et al., 2024; Saratchandran et al., 2024), we assign appropriate weights to different PDE features for better determining similarity. This weighted en- coding enables efficient knowledge transfer from solved PDEs to similar problems by ranking their similarity scores. Additionally, to further explore and optimize the hyperpa- rameter configurations provided by PGKR, we introduce the Memory Tree Reasoning Strategy (MTRS), which guides the planner in exploring the PINNs architecture space. This approach continuously improves PINNs architecture com- ponents via online learning with experiment feedback.
2.2. LLM- Autonomous SciML Agents
Large language models have demonstrated powerful general knowledge and linguistic capabilities since the release of (O0-a.5 et al., 2022), leading to various studies that extend their application towards specific tasks, known as large language model (LLM) agents. These agents have been applied to both general tasks and scientific disciplines (AI4Science & Quantum, 2023; Bran et al., 2024; Boiko et al., 2023). In the field of Scientific Machine Learning (SciML), various studies have highlighted the capability of LLMs to simulate physical phenomena, for instance, (Ali- Dib & Menou, 2023; Alexiadis & Ghiassi, 2024) demon- strate the use of LLMs in developing numerical PDE solvers, thereby accelerating scientific inquiries and discoveries. Ad- ditionally, (Kumar et al., 2023) combined LLMs with PDE solvers (PINNs and DeepONet), creating agents that assist in data preprocessing, model selection, and result interpre- tation. Similarly, (Lin et al., 2024) introduced a physics- informed LLM agent specifically designed for power con- verter modulation. However, these previous works have only preliminarily explored the potential of LLMs in solv- ing PDE problems and conducted some case studies. Our work not only further validates the effectiveness of LLMS in these applications but also proposes a novel framework that seamlessly integrates LLMs with PINNs, contributing to the SciML community.
Evaluated on 14 diverse PDEs, PINNsAgent demonstrates superior performance compared to state-of-the-art methods, showcasing its ability to autonomously develop and opti- mize PINNS architectures. The critical contributions of our work can be summarized as follows:
We propose PINNsAgent, a novel LLM-based surro- gate framework that autonomously develops and ex- plores optimal PINNs for given PDEs without relying expert heuristics deep learning. The framework consists of a multi-agent system, including a database, planner, a programmer, and a code bank, which work together to generate and optimize PINNs architectures.
We introduce Physics-Guided Knowledge Replay (PGKR), which encodes essential characteristics of PDEs and enables efficiint knowledge transfer from solved PDEs to similar problems, significantly improv- ing learning efficiency.
We develop the Memory Tree Reasoning Strategy (MTRS) that guides the planner in navigating the PINNs architecture space, facilitating continuous im- provement through iterative feedback.
2

---

PINNsAgent: Automated PDE Surrogation with Large Language Models
PINNsAgent
Database
Code Generation
Query
0.0
Plan
Programmer
Write
SCRIPT </>
Execute
PDE Descriptions
Literatures
Exp Record
Jeb Code
Feedback
Topk
Update
Retrieve
Debug
Config Generation
Code Bank
Arch
Arch
Input
SCRIPT </>
Execute
YAML
Planner
Config
Base Code
Feedback
Memory Tree
Revise Loop x N
Figure 1. The workflow of the PINNsAgen''s Framework. The PINNsAgent operates in two modes: Code Generation and Config Generation. It leverages LLM agents to generate and refine executable code and YAML configuration files for optimizing hyperparameters PINNs. The planner and programmer collaborate to devise experimental plans and generate training code, utilizing a central Code Bank and top-K cases from the Database.
2.3. LLMs enabled AutoMI.
traditional methods.
Automated Machine Learning (AutoML) has revolutionized the field of machine learning by automating the selection of optimal models and their hyperparameters (He al., 2021; Karmaker et al., 2021). Early approaches in AutoML fo- cused on efficiently exploring hyperparameter spaces, pro- cess known as Hyperparameter Optimization (HPO) (Feurer & Hutter, 2019). Representative methods include Random Search (Bergstra Bengio, 2012), Grid Search (Liashchyn- skyi & Liashchynskyi, 2019), Bayesian Optimization (Wu al., 2019), and Evolutionary Computation (Liu et al., 2023). With the advent of large language models (LLMs), HPO methodologies have significantly evolved. LLMs can automate and enhance HPO by generating predictive and insightful hyperparameter suggestions based on their exten- sive training data (Wang et al., 2023a; Tornede et al., 2024). Their reasoning capabilities allow them to propose initial hy- perparameters by analyzing training tasks and datasets (Guo et al., 2024). Additionally, LLMs leverage cross-domain knowledge to improve model configurations and can gen- erate parts or entire neural network architectures (Yu et al., 2023; Zheng et al., 2023). Our work differs from previ- ous studies by introducing novel multi-agent framework that automates the design of PINNs at two levels: datahase retrieval (PGKR), hyperparameter optimization (MTRS). This approach provides balanced trade-off between effi- ciency and accuracy, offering significant improvements over
3. Methods
In this session, we introduce PINNsAgent, a novel multi- agent framework for optimizing PINNs architectures.
Section 3.1 introduces the background of the proposed framework. Section 3.2 provides an overview of the critical components of PINNsAgent. In Section 3.3, we propose novel retrieval framework called Physics-Guide Guided Knowl- edge Replay (PGKR), which leverages the mathematical and physical properties of PDEs to identify promising hy- perparameter configurations. Finally, Section 3.4 introduces the Memory Tree Reasoning Strategy within PINNsAgent for efficient exploration the search space.
3.1. Preliminary
3.1.1. PROBLEM FORMULATION: SOLVING PDES WITH PINNS
Partial Differential Equations (PDEs) are foundational to modeling various physical phenomena across science and engineering disciplines, including fluid dynamics (Kutz, 2013), quantum mechanics (Teschl, 2014) 4), and climate mod- eling (Stocker, 2011). l). A general form of a PDE is expressed as:
3

---

PINNsAgent: Automated PDE Surrogation with Large Language Models
where H denotes the hyperparameters iteration is the PDE formulation, and f is feedback from the prior iteration. We introduce two agents: planner Ppl and programmer Ppr. The planner generates new hyperparame- ter settings based on the problem formulation т, previous settings H and feedback ft-1, The programmer ex- ecutes training scripts for PINNs using the settings H' and produces ft-1. The initial setting H. is defined as p(H() PR(T, B), where PR is retriever querying pre- established database B with the PDE formulation r to es- tablish a starting point 7t"
F(r, u, Vu, Vu,,...)
where u u(x) denotes the unknown function, the spatial coordinates, and Vu, V the first and higher-order spatial derivatives of u.
Physics-Informed Neural Networks (PINNs) provide a mesh-free method to solve PDEs by utilizing the univer- sal approximation capabilities of deep neural networks. PINNS enforce the compliance of the neural network solu- tion u(0, H) with the underlying physical laws represented by the PDEs. The overall formulation of PINNs is given by:
3.2. PINNsAgent
PINNsAgent is LLM-based multi-agent framework that integrates three key components: the Database, the Planner, and the Code Bank. These components work collaboratively to develop optimal PINNs architectures for target PDEs.
[(u H)) = LPDE(u(0, 10) + (u(00 (2)
where v(0,H) H) is the neural network approximation of u, parameterized by weights 0 and hyperparameters H. The PDE-residual loss LEDE is computed as:
As illustrated in Figure I, the PINNsAgent operates two distinct modes: Config Generation and Code Generation. The Config Generation mode, which is the primary focus of this study, is designed to handle scenarios where the target PDE already exists in the Code Bank. In this mode, the LLM-based agent, termed the planner, is tasked with gen- erating YAML configuration files. These files delineate the settings within a predefined search space, optimizing the hy- perparameters in accordance with the specific requirements and constraints of the target PDEs. The Code Generation mode is designed to address scenarios where the user spec- ifies a PDE that is not present in the Code Bank, enabling PINNsAgent to handle user-specified PDEs that are new to the Code Bank.
N F( u. Vu, V2 V......)
(PDE(u(0,H))
(3)
where N represents the number of collocation points used to evaluate the PDE residuals.
The boundary condition loss LBC enforces agreement with the prescribed boundary (and/or initial) conditions, and is typically defined as:
NBc Σ
(Bc(u(0,11) NBC
us (x (x) (xc(xx())))))))))))))
(x)
3.2.1. STEP 1: DATABASE RETRIEVAL
(4)
The Database acts as a central repository for archiving both the literature related to PINNs and the successful hyperpa- rameter configurations derived from prior experiments. To capitalize on this accumulated experience efficiently, we introduce novel retrieval strategy named Physics-Guidde Knowledge Replay (PGKR). Upon querying the Database with a detailed description of the PDE, select the top hy- perparameter settings that best align with the requirements of the target PDE. These selected settings are then forwarded to the planner, which synthesizes this information to devise comprehensive experiment plan. The details of PGKR are elaborated in Section 3.3.
where NBC is the number of boundary (or initial) sample points, x_3 denotes their coordinates, and UBC is the pre- scribed value at BCC
3.1.2. HYPERPARAMETER OPTIMIZATION VIA LLM
Selecting optimal hyperparameters H. such as learning rates, layer depths, neuron counts per layer, and activation func- tions, is crucial for the training efficacy and solution accu- racy of PINNs. Traditional methods like grid or random search are often inefficient and computationally demanding. Large Language Models (LLMs) offer a novel iterative ap- proach to generate hyperparameter settings. Formally, the LLM-based HPO method encompass an iterative loop:
3.2.2. STEP 2: EXPERIMENT PLAN GENERATION
In this step, the planner plays a pivotal role in generating candidate architectures and guiding the exploration of the hyperparameter search space for PINNs. Utilizing the top K initial configurations sourced from the database, the planner functions as policy model tasked with the strategic explo-
(L)
(f - 1.- ft-1), 11-1
Σ
(5)
4

---

PINNsAgent: Automated PDE Surrogation with Large Language Models
ration of the PINNs architecture search space. To navigate this search space efficiently, we have developed the Memory Tree Reasoning Strategy (MTRS), novel method designed to optimize the selection process of hyperparameters by evaluating their exploration scores. The details of MTRS are thoroughly claborated in Section 3.4. After selecting H using MTRS, the planner proceeds to develop a comprehen- sive experimental plan. This plan serves as a blueprint for the programmer implement the PINNS according to the specified configurations. In the case of Config Generation mode where the programmer is not involved, the planner is required to generate the configuration files in YAML format, based on the feedback ! of the former iteration.
signing PINNS.
To address this issue, we conducted 3000 parameter fine- tuning experiments on the datasets provided by PINNa- cle (Hao et al.. 2024). These experimental results enable us to leverage past experiences to solve new PDEs. Sub- sequently, we developed a new retrieval method named Physics-Guided Knowledge Replay (PGKR), which ex- tends the general Knowledge Replay concept by incorpo- rating domain-specifee knowledge. PGKR first encodes PDE's mathematical and physical properties into a struc- tured format. This allows the method to find retrieval PDEs with similar structures to the target PDE. By comparing the encoded representations, PGKR identifies the most relevant PDEs from the knowledge base, providing valuable insights into the appropriate PINNs architectures and hyperparame- ter settings.
3.2.3. STEP 3: CODE EXECUTION
To facilitate the deployment of PINNs models, we construct Code Bank to store the reusable code snippets, providing the programmer with successful examples and API instances. In the Code Generation mode, The programmer retrieves pre-defined templates, libraries, and best practices from the Code Bank and translates the candidate architectures generated by the planner into executable code following the experiment plan. If errors are reported, the terminal feedback is then replayed to the programmer identify and resolve bugs. In the Config Generation mode, we ex- tract the base code that can directly run on the generated configuration file to get the final results.
To encode the mathematical and physical properties of PDEs into structured format, we define a comprehensive set of labels L = (4, ./,,,.....,,..,... that capture the key features of each PDE, including equation type (e.g.. parabolic, elliptic, hyperbolic), spatial dimensions, linearity, time dependence, boundary and initial conditions, coefficient type, time scale, and geometric complexity. These labels are then encoded into feature vectors F = (f1, f2.... fn) using a prede- fined encoding scheme. Based on findings from previous studies, we assign higher weights to certain critical features, particularly the PDE type, to enhance similarity determina- tion. The encoding process can be formally defined as a function & : C + F, which maps each label to its corre- sponding weighted feature vector:
3.2.4. STEP 4: REVISION
Upon completing the execution step, the evaluation results f", including detailed training logs, performance metrics, and visualization results, utilized update the Database. This process ensures that each iteration enriches the repos- itory with new insights and empirical evidence, contribut- ing to a more comprehensive knowledge base. Using the feedback f', the Planner revises the experimental plan to enhance the model's performance in subsequent iterations. This feedback-driven revision process involves systemati- cally adjusting the hyperparameter settings and potentially exploring new architectural modifications. The revision pro- cess is meticulously executed through loop that iterates N times. In each iteration, the Planner assesses the current performance, identifies areas for improvement, and makes informed adjustments to the hyperparameters.
E (4. wifi, i= =, 1,2.....
(6)
where we is the weight assigned to the i-th feature.
The encoded feature vectors for all PDEs are concatenated to form feature matrix X E Raxm, where is the num- ber of PDEs and m is the dimensionality of the feature space. Further details on the encoding scheme and weight assignment can be found in Appendix A.
To measure the similarity between PDEs, we employ a weighted cosine similarity, which quantifies the cosine of the angle between two weighted feature vectors. Given two PDEs represented by their feature vectors fo and fy, the weighted cosine similarity Siy is computed as:
3.3. Database Exploitation: Physics-Guided Knowledge Replay (PGKR)
(W (Wf)) Sij | Sill [[
i n
Experience Replay is a paradigm for reusing past experimen- tal data and expert knowledge to address new challenges (Rolnick al., 2019). However, there a lack of sufficient databases and literature supporting the retrieval of optimal architectures for PDEs in database components when de-
where W is a diagonal matrix of weights. This weighting scheme allows us to emphasize the importance of certain features in determining similarity. The resulting similarity matrix S E R captures the pairwise similarities between all PDEs in the knowledge base. The top-k most similar
5

---

PINNsAgent: Automated PDE Surrogation with Large Language Models
PDE
Memory Tree
Burgers
PDE
Updated Memory Tree
Burgers
Activation
Rella
Tanh
Activation
Relu
Tanh
Depth
Depth
Width
128
128
128
64
Width
64
128
64
128
64
128
64
128
Case
MSS
MSP-)
MSB-6R
MSS-601
MSF-602
M85-33
MSS-13
MSS-6/001
dasbal
Optincl
Optinal
Optincl)
Step2: Select Unexplored
Step3: Generate Config
Code
Step1: Analyze Memory Tree
Burgers
Step4: Execute
Tanh
Planner
SCRIPT </>
CSV
StepS: Back- Propagation
L serve as policy model
Result
YAML
Base Code
Revise Loop x
Figure 2. Memory Tree Reasoning Strategy. The root node represents the corresponding PDE, with subsequent levels corresponding different hyperparameters. The planner selects unexplored branches and generates configurations, which are executed to obtain MSE scores. This process iterates refine the and find the global optimal architecture (Arch 8 with the lowest MSE).
PDEs are retrieved by ranking the similarity scores, along with their associated best-performing PINNs configurations. These configurations serve as the starting points for the surrogate model search process.
root node 8q. The action a, at step i involves selecting a specific hyperparameter from the subsequent layer. Con- sequently, each leaf node at the final layer represents complete hyperaarameter setting. The reward is simply de- signed as the negative Mean Squared Error (MSE) score of the selected setting. To leverage the LLM agent planner for guiding the expansion and exploration the most promising nodes tree, we maintain a state-action value function Q S A + R, where (ss, a) estimates the expected future reward of taking action state
3.4. Guided Exploration: Memory Tree Reasoning Strategy
Physics-Guided Knowledge Replay (PGKR) provides effective sub-optimal hyperparameter configuration initial point. To further refine and optimize this configura- tion, inspired by Monte Carlo Tree Search (MCTS) (Browne et al., 2012), we introduce the Memory Tree Reasoning Strategy (MTRS) within PINNsAgent. The Memory Tree abstracts the hyperparameter optimization process, il- lustrated in Figure 2, enabling the agent to utilize prior knowledge and feedback to guide the exploration of the Physics-Infonne Informed Neural Networks (PINNs) architecture space.
3.4.1. SELECTION
The first step is to select the most promising actions within the search space. To achieve this, we employ the well- known Upper Confidence bounds applied to the Trees (UCT) algorithm (Kocsis & Szepesvari, 2006):
In N(s) N(s
In the Memory Tree abstraction, the root node represents the PDE be solved, and each subsequent level corresponds specific hyperparameter, such as optimizer activation function. The child nodes within each level represent the possible values for the corresponding hyperparameter.
a- arg max aEA(ss)
((xpal))
Ampe(al))
(8)
where N(s) is the number of times state has been visited, N (8,, a) is the number of times action is taken at node and is a constant that balances exploration and exploita- tion. The planner, serving as the policy model (a)s)) uses the distribution of the LLM's output to determine the following action to take.
The Hyperparameter Optimization of PINNs can thus be formulated as Monte Carlo Tree Search (MCTS) process. In this formulation.en. each node in the tree represents a state, denoted as s,, which encapsulates the unique path the
6

---

PINNsAgent: Automated PDE Surrogation with Large Language Models
3.4.2. EXPANSION
analyze the effectiveness of PINNsAgent in solving PDEs. Finally, Section 4.3 presents ablation study to investigate the contributions of PGKR and the Memory Tree.
This step expands the memory tree by adding new child nodes to the previous state. If the selected node is a terminal node, this step is skipped, and the process proceeds directly to the paop-propngttion step. We limit the range of selection to avoid generating unreasonable architecture.
4.1. Experimental Settings
Dataset. We leverage the PINNacle benchmark dataset (Hao et al., 2024), a comprehensive collection of 20 repre- sentative PDEs spanning ID, 2D, and 3D domains. These PDEs encompass diverse characteristies, including varying goometries, multi-scale phenomena, nonlinearity, and high dimensionality, providing a challenging testbed for evaluat- ing PINNs architectures. Detailed descriptions are provided in Appendix B.
3.4.3. SIMULATION
The planner iteratively selects new actions and expands the existing memory tree until terminal nodes are reached. During this process, the top-K K and temperature values of the planner can balance the exploration and exploitation of the memory tree. For instance, the LLM's decisions are more diverse with higher temperatures.
Hyperparameter Search Space. We extend the hyperpa- rameter search space defined by (Wang et al., 2022; Wang & Zhong, 2024), with additional hyperparameters carefully curated from previous hyperparameter optimization (HPO) works (Klein Hutter, 2019) to provide a more compre- hensive exploration of the architectural landscape of PINNs. The configuration space, shown in Table 1, encompasses architectural choices: network type, activation functions, width, and depth, along with 5 hyperparameters: optimizer, initializer, learning rate, loss weight coefficients, and do- main/boundarr/iiitaal points.
3.4.4. BACK- PROPAGATION
After selecting an unexplored path, the planner generates configuration, integrates it into the base code extracted from the code base, and obtains the execution results. At this stage, only the MSE score is needed to calculate the reward of H' defined as R(H') - (u((,, H²), ugg). The back-propagation algorithm of MCTS then executed to update the Q(s, a) by aggregating the rewards from all future steps of the nodes along the path.
Task Description and Experimental Details. We eval- uate the performance of PINNsAgent on the task of Hy- perparameter Optimization. In this task, PINNsAgent is required to optimize the hyperparameter configuration for given PDE within 5 iterations. We implement PINNsAgent with GPT-4 model. To evaluate the ability of PINNsA gen! to solve unseen PDEs, we did not provide the relevant database for the target PDE. During the implementation of PGKR, we selected topk = 1. For each PDE, we conducted ten repeated experiments and took the average of the lowest MSE to mitigate randomness, with temperature of 0.7. We compare PINNsAgent with two baseline methods: (1) Random Search, a basic hyperparameter tuning method that selects configurations randomly, and (2) Bayesian Search, which uses Bayesian optimization to select configurations. We also provide PINNacle benchmark's 's best reported results as oracle/upper bound for reference.
Table Hyperparameter Search Spaces for PINNs Optimization Tasks
Hyperparameter
Details
Net
FNNs, LAAFS, GAAFs
Elu, Selu, Sigmoid, SiLu, ReLU, Tanh, Swish, Sin, Gaussian
Activation
Width
8 to 256 (Increment: 4) 3 to 10 (Increment: 1) SGD, Adam, MultiAdam, L-BFGS
Depth
Optimizer
Glorot Normal/Uniform. He Normal/Uniform, Zeros 10-6 10
Initializer
Learning Rate
Points
100 to 9600 (Increment: 500)
(Dom/Bnd/Init)
4.2. Main Results
The comparative end-to-end performance of PINNsAgent and the baseline approaches on 14 different PDEs pre- sented in Table 2. The results demonstrate that PINNsAgent consistently outperforms the baselines, achieving the best performance on 12 out of 14 PDEs. Notably, PINNsAgem shows significant improvements over Random Search and Bayesian Search in complex PDEs such as NS-C, Heat- MS, and Heat-ND. For instance, on the NS-C equation,
4. Experiments
In this section, we discuss the experimental methodology used to evaluate the performance of our PINNsAgent.
Section 4.1 describes the experimental settings, including the dataset, hyperparameter search space, and baselines for comparison. In Section 4.2, we present the main results and

---

PINNsAgent: Automated PDE Surrogation with Large Language Models
Table Comparative performance (MSE) of PINNsAgent and haseline approaches 14 different PDFs for Task 1. Results are averaged over 10 runs to mitigate randomness. Values in parentheses represent standard deviations. The best performances are highlighted in bold. PINNacle benchmark's s best reported results are shown in gray for reference.
Random
Bayesian Search
PINNacle
PDEs
PINNsAgent
Search
Benchmark
Burgers
6.63E-02 (+1 10B-01)
8.70E-02 (+665118-3))
6,51E-05 (=1.665-05)
7. 7.90E-05
ID
Wave-C
1.50E-01 (65-00))
1.78E-01
(#3.84B-02)
3,33E-02
(=:3,00E-02)
3.01E-03
KS
1.09E+00 (1..58E)02)
1.10E+00
(12.55E-03)
1.09E+00
(133.20E 02)
1.04E+00
Burgers-C
2.48E-01
(#:4044B-33)
2.42E-01
(#8.96E-03)
2.04E-01
(=:17118-02)
109E-01
Wave-CG
2.87E-02 (+4.98E-04)
2.11E-02
(+1.12B-02)
5.40E-02
(#7.89E-03)
2.99E-02
Heat-CG
3,96E-01 (1.32226.1))
1.17E-01
(1:3.245.02)
1.80E-03
(1:1.044.0) 03)
8.53E-04
NS-C
4.02F-03 (1:5.938.03)
5,12E-03
(1:1.33E-03
8,50E-06
(1:6.80F.06)
2.33E-05
2D
GS
4.28F-03 (1:2.238-05)
4.03F-003
(1:4.7F.004)
4.32F-03
(1:3.07F.05)
4.32F-00
Heat-MS
1.84E-02 (#1.18B-02)
7.48E-03
(4:38185-3)
3,57E-05
(#:233B-55
5.27E-05
Heat-VC 3,57E-02 (#8.728-03)
3,93E-02
(#221176-03)
5.52E-03
(d:3.895-03)
1.76E-03
Poisson-MA
5,87E+00 (+1.17E:00)
5,82E+00
(+2.30E+000
3.16E+00
(#9.92E-00)
183E+00
3D
Poisson-CG
3,82E-02 (#22118-022)
2.55E-02 (4:56655-03)
1.59E-02
(1:1.115-02)
9,51E-04
Poisson-ND
1.30E-04 (#2.785-04)
4,72E-05 (#2.768-06)
2.09E-06
(+:1.0EE-05)
2.09E-06
ND
Heat-ND
2.58E-02 (19.876-02)
1.188-04:28992-000
3.51E-07 (+7.926-07)
8.52E+00
Table Ablation study: Comparative performance (MSE) of PINNsAgent variants using GPT-4 on 12 PDEs. Results are averaged over runs (mean + std). Best performances are highlighted in bold.
Burgers
WareC
Burgers-C
Ware-CG
Heat-GG
NS-C
Heat-M
Heat-VC
Paisson-CGG
Polsson-ND
Heat-ND
Method
68
6.51E-05
3.33E-02
2.04E-01
5,40E-02
1.80-03 8.50E-06
4.32E-03
3,57E-05
5,52E-03
1.59E-02
2.09E-06
3.51E-07
PINNsAgent
11.63E.05
1.360E-02
11.71E
17.89E-03
11.04E-03
16.800-66
13.077.05
1..33.05
1.3891.03
11.11.02
11.066
1.7.920.07
7.41E-05 2.87E-02
2.17E-01
3.19E-02
1.38E-02
1.53E-05
4.31E-03
3.85E-05
69
2.27E-02
2.02E-05
4.92E-06
w/o PGKR
1 05 13.911.02
11.551.002
12888.03
17.94000
11.588.05
13.066.05
11.020.04
14.381.03
11.599..22
11.02.011
11I1E
w/o PGKR
8.44E-05
2.88E-02 2.25E-01
3.53E-02
6.97E-02
1.08E-05
2.59E+08
8.13E-05
1.10E-02
2.58E-02
2.43E-05
6.59E-07
& .MTSS
15.977.05
43.32F.02
11.877.02
11.299.02
12.07-01
18.55-06 17.777-08 11430.04 11.36F-02
1: 688-020 42.77.05 11.01..06
PINNsAgent achieves an MSE of 8.50E-06, which is several orders of magnitude better than Random Search (4.02E-03) and Bayesian Search (5. (5.12E-03). These results highlight the effectiveness of PINNsAgens in optimizing PINNs archi- tectures across a diverse range of PDEs.
Effectiveness of PGKR and MTRS. Table 3 presents the performance of three variants of PINNsAgent: (1) the complete PINNsAgent framework, (2) PINNsAgent with- out PGKR (w/o PGKR), and (3) PINNsAgent without both PGKR and MTRS (w/o PGKR & MTRS).
These results confirm that the knowledge encoded and reused by PINNsAgent can be effectively transferred to dif- ferent PDE classes and datasets, further highlighting the practical value of our approach for automated scientific ma- chine learning.
The results demonstrate that both PGKR and MTRS con- tribute significantly to the performance of PINNsAgent. The complete PINNsAgen framework achieves the best per- formance on 9 out of 12 PDEs. Removing PGKR leads to performance degradation on most PDEs, with notable excep- tions on Wave-C and Wave-GG. Further removing MTRS results in additional performance drops, most dramatically the GS equation where the MSE increases from 4.3 IE 03 to 2.59E+08. These results validate the effecctvveness of leveraging prior knowledge through PGKR and MTRS, demonstrating their crucial role in enhancing the perfor- mance and robustness of PINNsAgent across a diverse range of PDEs.
4.3. Ablation Study
To gain a deeper understanding of the contributions of Physics-Guided Knowledge Replay (PGKR) and the Mem- ory Tree Retrieval Strategy (MTRS) in PINNsAgent, we conducted an ablation study by removing these components individually and comparing the performance with the com- plete framework.
8

---

PINNsAgent: Automated PDE Surrogation with Large Language Models
4.4. Computational Cost Analysis
complex physical systems. Users should maintain healthy skepticism about automatically generated configurations and always validate results against physical intuition and established numerical benchmarks before applying them to real-world problems with safety implications.
We also conduct a computational cost analysis by reporting the average computation time per PDE for all methods. As shown in Table 4, the additional overhead introduced by LLM inference in PINNsAgent is only about 8.2% higher than Random Search and 4. % higher than Bayesian Opti- mization.
References
A14Science, M. R. and Quantum, M. A. The impact of large language models on scientific discovery: a pre- liminary study using gpt-4, ArXiv, abs/2311.07361,
Table 4, Average computation time (scconds) per PDE across all 14 benchmarks.
Method
Average Computation Time (s)
2023. URL https: //pi: semanticschola. /ts//CCopUSII::555500000
Random Search
3462.24 + 2631.55
Bayesian Search
3598.47 1: 2792.83
PINNsAgent
3747.78 1 2965.62
Alexiadis, A. and Ghiassi, B. From text to tech: Shaping the future of physics- based simulations with ai-driven generative models. Results in Engineering, 21:101721, 2024.
Our approach therefore remains efficient and practical for real-world scientific machine learning tasks.
Ali-Dib, M. and Menou, K. Physics simulation capabilities of llms. CoRR, abs/2312.02091, 2023. doi: 10.48550/ 1220V22312.02091 URL https://doo org/10. 10150//axXi..o www.:iiii.ii.....mm 2312 02091.
5. Conclusion
In this work, we introduced giNNsAgent, a novel LLM- based surrogate framework that automates the development and optimization of PINNs for solving PDEs. By leveraging the knowledge and reasoning capabilities of large language models, PiNNsAgent effectively bridges the gap between domain-specific knowledge and deep learning expertise, enabling non-experts to harness the power of PINNs without extensive manual tuning.
Berger, M. J. and Oliger, J. Adaptive mesh refinement for hyperbolic partial differential equations. Journal of computational Physics, 53(3):484-512, 1984.
Bergstra, J. and Bengio, Y. Random search for
hyper-parameter optimization. J. Mach. Learn. Res., 13:281-305, 2012. URL http:::/api.. semanticscholi wttp//oooosI::1100000....
Acknowledgments
This work was supported by the National Science and Tech- nology Major Project (No. 20222D0117800)
Boiko, D. A., MacKnight, R., Kline, B., and Gomes, G. Au- tonomous chemical research with large language models. Nature, 2023. URL ht https: : s $41586-024 06792-0.
Impact Statement
We acknowledge several potential societal implications of PINNsAgent: (1) Democratization vs. expertise dilution. While PINNsAgent aims make PINNs technology more accessible non-experts, there is concern that widespread automation might lead to insufficient understanding of the underlying physics and mathematical principles. We empha- size that PINNsAgent should serve as an educational tool that guides users toward better understanding rather than replacing the need for scientific rigor and domain exper- tise. (2) API dependency and cost barriers. The reliance on commereial LLM APIs creates potential barriers for researchers with limited funding or those in regions with restricted access to such services. We recommend explor- ing integration with open-source language models to ensure broader accessibility. (3) Overconfidence in automated solutions. There is risk that automated optimization may create false confidence in PINNs solutions, particularly for
Bran, A. M., Cox, S., Schilter, O., Baldassari, C., White, A. D., and Schwaller, P. Augmenting large language mod- els with chemistry tools. Nat. Mac. Intel., 6(5):525-535, 2024. doi: 10. 101100/542266-004000822222 8. URL https: //doi. wwroo///////// 10. 1038 /D/.:///4225555 0200022256660000022---202
Brown, T. Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877-1901, 2020.
Browne, C. B., Powley, E., Whitehouse, D., Lucas, S. M., Cowling, P. I., Rohlfshagen, P., Tavener, S., Perez, D., Samothrakis, S., and Colton, S. A survey of monte carlo tree search methods. IEEE Transactions on Computa- tional Intelligence and in games, 4(1): 111):1-43, 2012.

---

PINNsAgent: Automated PDE Surrogation with Large Language Models
Canuto, C., Hussaini, M. Y., Quarteroni, A., and Zang. T. A. Spectral methods: fundamentals in single domains. Springer Science & Business Media, 2007.
Jagtap, A. D., Kawaguchi, K., and Karniadakis, G. E. Adap- tive activation functions accelerate convergence in deep and physics-informed neural networks. Journal of Com- putational Physics, 404:109136 2020b.
Cuomo, S., Di Cola, V. S., Giampaolo, F., Rozza, G., Raissi, M., and Piccialli, F. Scientific machine learning through thysies-informed neural networks: Where we are and what's next. Journal of Scientific Computing, 92(3):88, 2022.
Jagtap, A. D., Shin, Y., Kawaguchi, K., and Karniadakis, G. E. Deep kronecker neural networks: A general frame- work for neural networks with adaptive activation func- tions. Neurocomputing, 468:165-188 180, 2022.
Feurer, M. and Hutter, F. Hyperparameter optimization. Au- tomated machine learning: Methods, systems, challenges, pp. 3-33, 2019.
Kaplarevié-Maliiie, A., Andrijević, B., Bojović, F., Nikolić, S., Krstić, L.,, Stojanović, B., and Ivanović, M. Identi- fying optimal architectures of physics- inffrsed neural networks by evolutionary strategy. Applied Soft Comput- ing, 146: 10646, 2023.
Guo, S., Deng, C.,, Wen, Y., Chen, H., Chang, Y., and Wang, J. Ds-agent: Automated data science by em- powering large language models with case-based reason- ing. In Forty-first International Conference on Machine Learning. ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024, URL htwpstt/t//// BotLhtttt///pereeee //tps////peee...eee//m//mee/eeee net/ formm i d=I. G.g.NFF...
Karmaker, S. K., Hassan, M. M., Smith, M. J., Xu, L., Zhai, C., and Veeramachaneni, K. Automl to date and beyond: Challenges and opportunities. ACM Computing Surveys (CSUR), 34(8):1-36, 2021.
Klein, A. and Hutter, F. Tabular benchmarks for joint architecture and hyperparameter optimization. CoRR, 0s//1905004970, 2019. URL http:://ttiiiv.ee//et abs / 0490000497.
Hao, Z., Yao, J., Su, C., Su, H., Wang, Z., Lu, F., Xia, Z., Zhang, Y., Liu, S., Lu, L., and Zhu, J. Pinnacle: A comprehensive benchmark physics-informed neural networks for solving pdes. In Globersons, A., Mackey, L.,, Belgrave, D., Fan, A., Paquet, U., Tomezak, J. M., and Zhang, C. (eds.), Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024, 2024.
Koesis, L. and Szepesvari, C. Bandit based monte-carlo planning. In European Conference Machine Learning, 2006. URL https:/ http://///ppsssmaniiissssseeeeee org/ Co hesp/CorpusTD:. 11188477
wwwanticscho
Kumar, V. V., Gleyzer, L., Kahana, A.. Shukla, K., and Karniadakis, G. E. Mycrunchgpt: A chatgpt assisted framework for scientific machine learning. Journal of Machine Learning for Modeling and Computing, 2023. URL https: /appi. org /ttp/CoopSII::5522222223
He, X., Zhao, K., and Chu, X. Automl: survey of the state-of tft Knowledge based systems, 212:106622, 2021.
Huang. D., Bu, Q., Zhang, J. M., Luck, M., and Cui, H.
Agentcoder: Multi aulti-ageet-baseee code generation with it- erative testing and optimisation. CoRR, abs/2312.13010 2023. doi: 10.4550/ARXX1.2312.10001 URL https:
Kutz, J. N. Data-driven modeling & scientific computation:
methods for complex systems & big data. OUP Oxford, 2013.
//doi. org / 10. /0t55550///iiv../////
LeVeque, R. J. Finite volume methods for hyperbolic prob- lems, volume 31, Cambridge university press, 2002.
Hughes, T. J. The Finite Element Method: Linear Static and Dynamic Finite Element Analysis. Dover Publications, 2000.
Liashchynskyi. P. and Liashchynskyi. P. Grid search, ran- dom search, genetic algorithm: A big comparison for nas, 2019.
Jagtap, A. D. and Karniadakis, G. E. Extended physics- informed neural networks (xpinns): A generalized space- time domain decomposition based deep learning frame- work for nonlinear partial differential equations. Commu- nications in Computational Physics, 2 2020.
Lin, F., Liu, J., Li, X., Zhao, S., Zhao, B., Ma, H., and Zhang, X. PE-GPT: A physics- informed interactive large language model for power converter modulation design. CoRR, abs/2403. 14059, 2024. doi: 10.48550/ARXII, 2403.14059, URL https: /tt///do... 110g//0...00000.. 48550/ arXiv #=Xiv.2403.11. 2405.
Jagtap, A. D., Kawaguchi, K., and Em Karniadakis, G. Lo- cally adaptive activation functions with slope recovery for deep and physics-infee informed neural networks. Proceedings of the Royal Society A, 47662239) 2020a.
Liu, Y., Sun, Y., Xue, B., Zhang, M., Yen, G. G., and Tan, K. C. A survey on evolutionary neural architecture search.
10

---

PINNsAgent: Automated PDE Surrogation with Large Language Models
IEEE Transactions on Neural Networks and Learning Systems, 34(2):550-570, February 2023. ISSN 2162- 2388. doi: 10. 10.1009/mlls2222.31005555 URL http:// dx. doi. org/ 10. 110./TNN1SSSSS.. 2021. 3100554.
Rolnick, D., Ahuja, A., Schwarz, J., Lillicrap, T. P., and
Wayne, G. Experience replay for continual learning. In Wallach, H. M., Larochelle, H., Beygelzimer, A., d' Alché- Buc, F., Fox, E. B., and Garnett, R. (eds.), Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada. pp. 348-358, 2019.
Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, L., Wiegreffe, S., Alon, U., Dziri, N., Prabhumoye. S., Yang. Y., et Self-refine: Iterative refinement with self- feedback. Advances in Neural Information Processing Systems, 36, 2024.
Saratchandran, H., Ch Ch'ng. S., and Lucey, S. Architectural strategies for the optimization of physics-informed neural networks. CoRR, tttss/24220021111 2024. doi: 10.48550/ ARXIV.2402.0271 URL https: ttttps://di... //t::///ooioooo org//0. 4 48550/arXiy 2402.02711.
Nabian, M. A., Gladstone, R. J., and Meidani, H. Efficient training of physics-informed neural networks via impor- tance sampling. Comput. Aided Civ. Infrastructure Eng., 36(8):962-977, 2021. doi: 10.1111/MICE. 12685, URL https: /dw.i//do////oo org/ 10. 11111111111 /micee..m..... met.m................
Shukla, K., Jagtap, A. D., and Karniadakis, G. E. Parallel physics-informed neural networks via domain decompo- sition. Journal of Computational Physics, 447:11068: 2021.
Nijkamp, E., Pang, B., Hayashi, H., Tu, L., Wang, H., Zhou, Y., Savarese, S., and Xiong, C. Codegen: An open large language model for code with multi-turn program synthesis. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL https: //openrevies ew, net for:://ormmmmmmm wwt.:iidaaaaaaa.aa.....
Stocker, T. Introduction to climate modelling. Springer
Science & Business Media, 2011.
Strikwerda, J. C. Finite difference schemes and partial differential equations. SIAM, 2004.
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, I.., Simens, M., Askell, A., Welinder, P., Christiano, P. F., Leike, J., and Lowe, R. Training language models to follow instructions with human feedback. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurlPS 2022, New Orleans, LA, USA, November 28- December 9, 2022, 2022.
Teschl, G. Mathematical methods in quantum mechanics, volume 157. American Mathematical Soc., 2014.
Tornede, A., Deng, D., Eimer, T., Giovanelli, J., Mohan, A., Ruhkopf, T., Segel, S., Theodorakopoulos, D., Tornede, T., Wachsmuth, H., and Lindauer, M. Automl in the age of large language models: Current challenges, future opportunities and risks. Trans. Mach. Learn. Res., 2024, 2024. URL https::/ttp/o.ttttt /ttp////ppeeee.e.e//e//e/. her/forum? id=cAthubStyG.
Wang. H., Gao, Y., Zheng, X., Zhang, P., Chen, H., and Bu, J. Graph neural architecture search with GPT-4, CoRR, abs/2310.01433 2023a. doi: 10.48550/ARXIV, 2310.01436. URL https: /ttp:///ooo..//0/0//00000/ ArXiv.2310 01436.
Raissi, M., Perdikaris, P., and Karniadakis, G. E. Physics informed deep learning (part I): data-driven solutions of nonlinear partial differential equations. CORR, abs/1711.10561, 2017. URL http://arxiv.or abs/1711.1055
Wang, S., Teng, Y., and Perdikaris, P. Understanding and mitigating gradient flow pathologies in physics-informed neural networks. SIAM Journal on Scientific Computing, 43(5):A3055-A30811 2021.
Raissi, M., Perdikaris, P., and Karniadakis, G. Physics- informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics, 378:686-707, 02 2019. doi: 10. 10166/00jcjpp.. 2018 10.045.
Wang. S., Sankaran, S., Wang, H., and Perdikaris, P. An ex- pert's 's guide to training CoRR, abs/2308 08468, 2023b. doi: 10.48550/ARXIV, URL https: http//i..oo/..0005555131 arXiv. sxxiv.2308... 2308. 68.
informed neural networks.
Rathore, P., Lei, W., Frangella, Z., Lu, L., and Udell, M. Challenges in training pinns: A loss landscape perspective. In Forty-first International Conference on Machine Learning. ICML 2024, Vienna, Austria, July 21-27, 2024. OeenReview.net, 2024. URL https: / //openreview.o net/ forum? d=mJGi Fr8jjiiFFrrra..
Wang. Y. and Zhong, L. Nas-pinn: neural architecture
search-guided physics-informet neural network for solv- ing pdes. Journal of Computational Physics, 496: 126603, 2024.

---

PINNsAgent: Automated PDE Surrogation with Large Language Models
Wang, Y., Han, X., Chang, C., Zha, D., Braga- Neto, U., and Hu, X. Auto-pinn: Understanding and optimizing physics-informed neural architecture. CoRR, abs/2205.13744 2205.13748, URL https::/t/////.... httpt::/ttt......... http://oo.oorrgoo o0o.oorgr/0.. 48550/ Dexiv.2200. 13748.
2022. doi: 10.48550/ARXIV.
Wang, Y. Le. H., Gotmare, A., Bui, N. D. Li, J., and Hoi, S. C. H. Codet5+: Open code large lan- guage models for code understanding and generation. In Bouamor, H., Pino, J., and Bali, K. (eds.), Proceed- ings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023. Singapore, December 6-10, 2023, pp. 1069-1088. Association for Computational Linguistics, 2023c. doi: 10.18653/VI/ 2023.EMN.P-MAIN.68. URL https:: /woi/oioooooo org/ 10.18653/v1/2022 emnlp-main....... 68.
Wu, J., Chen, X.-Y., Zhang, H., Xiong, L.-D., Lei, H., and Deng, S.-H. Hyperparameter optimization for machine learning models based on bayesian optimization. Journal of Electronic Science and Technology, 17(1):26-44) 40, 2019.
Yu, C., Liu, X., Feng, W., Tang, C., and Lv, GPT- NAS: evolutionary neural architecture search with the generative pre-trained model. CoRR, abs/2305 .03351, 2023. doi: 1044550/ARRX112230005533 URL https: //doi org.i.orrg/1... wwt.o/////....////////// w0w..4440000//........ 2305. 05351.
Yu, J., Lu, L., Meng, X., and Karniadakis, G. E. Gradient- enhanced physics-informed neural networks for forward and inverse pde problems. Computer Methods in Applied Mechanics and Engineering, 393:1 14823, 2022.
Zhang, M. R., Desai, N., Bae, J., Lorraine, J., and Ba, J. Us- ing large language models for hyperparameter optimiza- tion. In NeurlPS 2023 Foundation Models for Decision Making Workshop, 2023.
Zheng, M., Su, X., You, S., Wang, F., Qian, C., Xu, C., and Albanie, S. Can GPT-4 perform neural architecture search? CoRR, abs/2304. 10970, 2023. doi: 10.48550/
ARXIV.2304.10970, URL 20550//rxiv.234..000
12

---

PINNsAgent: Automated PDE Surrogation with Large Language Models
Appendix
A. PDE Feature Encoding and Similarity Analysis
A.1. PDE Feature Encoding
To enable effective knowledge reuse in the PINNsAAgett framework, we define a comprehensive set of labels L 4, 12............
..,, In that capture the essential mathematical and physical characteristics of each PDE problem. These labels include:
PDE Type (Burgers, Poisson, Heat, NS, Wave, Chaotic)
Equation Type (Parabolic, Elliptic, Hyperbolic, Mixed)
Spatial Dimensions (ID, 2D, 3D, ND)
Linearity (Linear, Nonlinear)
Time Dependence (Time-Independdnt, Time-Dependent)
Boundary Conditions (Dirichlet, Neumann, Mixed Robin, Periodic)
Initial Conditions (Presence or Absence)
Coefficient Type (Constant, Variable)
Time Scale (Short-Time, Long-Time)
Geometric Complexity (Simple, Complex)
We then encode these labels into binary feature vectors F = fi, f2,..... ,,, Sax where n is the number of PDEs, using predefined encoding scheme, as shown in Table 5.
A.2. PDE Labels
With the defined labels and encoding scheme, we assign a comprehensive set of labels to each PDE problem in our study. Table 6 and 7 present these labels for each PDE.
This encoding process can be formally defined as a function E: C C which maps each label to its corresponding binary feature vector. The encoded feature vectors for all PDEs are concatenated to form feature matrix E -=** where the number of PDEs and the dimensionality of the feature space. In our study, n 14 and m 30.
A.3. PDE Similarity Analysis
Having obtained the PDE labels L and their corresponding feature vectors F, we employ cosine similarity to quantify the similarity between PDEs. which measures the cosine of the angle between two feature vectors. The resulting similarity matrix S E RRxx captures the pairwise similarities between all PDEs in the knowledge base.
B. Datasets
We conduct our experiments using the datasets provided in PINNacle (Hao et al., 2024), which include 20 representative partial differential equations (PDEs) one, two, and three dimensions. These equations exhibit a variety complexities, including different geometries, multi-scale phenomena, nonlinearity, and high dimensionality.
For our study, we selected 14 PDEs from this collection. We excluded Pinv and Hlnv because they used for conducting inverse problems and Heat2d-LT and NS24-LT because they present long-time PDEs that are poorly fitted by PINNs regardless of the hyperparameter configuration. Thus, their optimal hyperparameter configurations are not meaningful and can negatively impact the Physics-Guided Knowledge Replay (PGKR) approach during retrieval.
Below, we provide key properties of the PDEs used in this paper. For full details on the data generation process and the hyperparameters used generate the PDE dataset, we refer the reader to (Hao al., 2024).
13

---

PINNsAgent: Automated PDE Surrogation with Large Language Models
Table Feature encoding scheme for PDE labels
Label
Description
Binary Encoding
Burgers
[1. 0, 0, 0, 0, 0]
Poisson
[0. I, 0, 0, 0]
Heat
[0. 0,
0, 0]
PDE Type
Navier-Stokes
[0, 0, 0, 1, 0]
Wave
[0, 0, 1,
Chaotic
[0. 0, 0, 0, 1]
Parabolic
0, 0]
Elliptic
[0, 0, 0]
Equation Type
Hyperbolic
00,
Mixed
0,
1D
[1, 0, 0, 0]
2D
[0, 0]
Spatial Dimensions
3D
[0, 0, 0]
ND
[0. 0, 0, 1]
Linear
[1. 0] [0, 1]
Linearity
Nonlinear
Lime-Independen
[1, 0] [0, 1]
Time Dependence
DemenDepndd
Dirichlet
[1, 0, 0, 0]
Neumann
[0, 0, 0]
Boundary Conditions
Mixed Robin
[0, 0, 1, 0]
Periodic
[0, 0, 0, 1]
Initial Condition
[1, 0]
Initial Conditions
No Initial Condition
[0, 1]
Constant Coefficient
[1, 0]
Coefficient Type
Variable Coefficient
[0, 1]
Short- Time
[1,0]
Time Scale
Long- Time
[0, 1]
Simple Geometry
[1,
Geometric Complexity
Complex Geometry
[0, 1]
B.1. PDE Descriptions
1.1.1. BURGERS EQUATION (ID)
The Burgers equation is a fundamental nonlinear partial differential equation (PDE) extensively used to model various fluid dynamics systems, including shock flows, wave propagation combustion chambers, and vehieular traffic movement. The equation is given by:
0.01 82u u
du
O,
01
dr
π Ox.2
where z [-1,1] and t E [0, 1]. Here, 0.01 represents the diffusion coefficient of the fluid. The initial and boundary conditions are
u(x 0) sin n,.
(10)
u(-1,t) = u(1, = 0.
14

---

PINNsAgent: Automated PDE Surrogation with Large Language Models
Table PDE labels and their corresponding feature
PDE
Labels
Burgers, parabolic, 1d, nonlinear, time-dependent,
BurgersId
Dirichlet, initial-condition, constant-coefficien
short-time, simple-geometry
Burgers, paraholic, 2d, nonlinear, time-dependent, periodic, initial-condition, constant-coefficien,,
Burgers2d
short-time, simple-geometry Poisson, elliptic, 2d, linear, time-independent, dirichlet, no-initial-condition, constant-coefficien,, short-time,
Poisson2d-C
Poisson, elliptic, 2d, linear, iime-independent, dirichlet, no-initial-condition, variable-coeffccient, short-time, complex-geomett
Poisson2d-CO
Poisson, elliptic, 3d, linear, time- meme-iddependen,, neumann, no-initial-condition, variable-coefficienn, short-time,
Poisson3d-CCG
Poisson, elliptic, 2d, linear, iine-independent, robin, no-initia-coonittoo,, cnffable-coeffccienn short-time, complex-geomest
Poisson2d-MS
Heat, parabolic, 2d, linear, time- dependenn,, dirichlet, initial-condition, variable-coefficiees short-time, simple-geometry
Heat2d-VC VC
Heat, parabolic, 2d, linear, time-dependent, dirichlet, initial-condition, constant-coefficion time, simple-geone geomeyy
Hcat2d-MS
B.1.2. 2D COUPLED BURGERS EQUATION (BURGERS 2D)
The 2D Coupled Burgers equation, which extends the ID Burgers equation to two dimensions, is expressed as:
+ u Vu ~vuu 0,
(1I)
where (x, 40, and t E [0, 1]. Here, represents the diffusion coefficient of the fluid. The periodic boundary
conditions are:
= u(x,0,t) u(x, L,t)
(12)
The initial conditions are given by:
L
(1x y) Σ Σ
au (in(2 + +z)) + bi cos(22iix + +y))
(13)
u (ar, y. 0) 2w(xx v)
where aijy big, and normally distributed random variables with mean 0 and variance 1, i.e., aij bij, e N(0, 1).
1.1.3. POISSON 2D CLASSIC (PoissoN2D-C)
The Poisson 2D equation is widely used to describe various physical phenomena, such as electrostatics and fluid dynamics. models the distribution of a scalar field u(x, y) governed by Laplace's 's operator:
Au
(14)
15

---

PINNsAgent: Automated PDE Surrogation with Large Language Models
Table PDE labels and their corresponding feature
PDE
Labels
Heat, parabolic, 2d, linear, time-dependent, robin, initial-condition, constant-coeffcieen short-time, complex-geometry
Heat2d-CG
NS, mixed, 2d, nonlinear, time-independent, dirichlet, no-initial-condition, constant-cocfficien,, short-time, Simple-geometty
NS2d-C
NS, mixed, 2d, nonlinear, time-independent, dirichlet, no-initial-condition, constant-cocfficien. short-time, complex- -geomeeyg
NS2d-CG
Wave, hyperbolic, 1d, linear, imeedependent, dirichlet, initial-condiitin,, constant-coefffccent, short-time, simple-geometry
Waveld-C
Wave, hyperbolic, 2d, linear, unee-dependent, neumann, initial-condition, variable-coefficientt short-time,
Wave2d-CG
complex-geone geometry
Wave, hyperbolic, 2d, linear, time-dependent, dirichlet, initial-condition, constant-coeffiiient long-time. simple-geometry
Wave2d-MS
Chaotic, parabolic, 2d, nonlinear, time- deneedeeendent, Do-boundary-condiiion initial-condition,
GS
constant-coefficient, long-time, simple-geomedt Chaotic, parabolic, 1d, nonlinear, time- dependennn,nt,, periodic, initial-condition, constant constant-coefficient, short-time, simple-geometry Poisson, elliptie, nd, linear, time-independent, dirichlet, no-initialccondiioon, constant-cocffiiient short-time, simple-geomety
KS
PNd
Heat, parabolic, nd, linear, time- denendeenn,nt,
HNd
mixed-boundary. initial-condition, constant-coefffcieet short-time, simple-geometry
We define the regions R1, R2, R3, and R4 as:
R1 {1,,)) : (z- 0.3) + ( (y- 0.3) < 0.12),
R₂ {(x, (z).)) (x+
(y3033) < 0.12), R3 = {(x, y) (x (x- 0.3) (y+) < 0.12),
(15)
R4 = {(x, y) (x (x - (y + 0.3)² < 0.12].
This equation operates within the spatial domain Ω Pree \ {Ri), where Pree [-0.5, 0.5)a denotes a rectangular region excluding the four circular areas R, The boundary condition is defined as:
u 0, x E OR;
(16)
u 1, E dmgee
B. 1.4. ROISSON-BOLTZMANN (HELMHOLTZ) 2D IRREGULAR GEOMETRY (POISSON2D-CG)
The Poisson-BEi Bortzmmnn (Helmholtz) 2D equation extends the classic Poisson equation by including a term proportional to the scalar field u. The equation given by:
16

---

PINNsAgent: Automated PDE Surrogation with Large Language Models
Au kZ f(r,
(17)
The computational domain is defined as [-1, 1]2 excluding several circular regions Qeirele = Rs which are defined by:
R1 (z,y): (z 0.5) + (v- 50)))
R2 (z. : (r- 0.4)² (y 0.4")
(18)
R3 {z, (x+ ++++0)))+++++0777) < 0.1²)
(y
R4 (x, (x+ ( 0 0.3
The boundary conditions are:
0.2, 00
(19)
1, x E Wignele
B.1.5. POISSON 3D COMPLEX GEOMETRY WITH Two DOMAINS (POISSON3D-CG)
The Poisson 3D equation with two distinct regions is described by:
-uAu kiu (1,,,, i= 1,2.
(20)
The computational regions are defined as follows:
Ω1 = j0, 1] x [0,1] x [0,0.5 U R
i=1
(21)
Ω2 = 10. 1] 10, 1] x 0.5. 1] U R
==1
The spherical regions R; are given by:
R1 {xx (x, (r +1 + (2-00))3+))) <02"
R₂ {x,,,,2) (x- 0.6) + +1 0.7) + +(2 (2-) 0.22).
(22)
R3 = (2,,,,2) (x - 0.2)² 1o+++y-00 (v--0))² + (=-007)²
R1 (z, (x 0.6) + ( 0.2) + (2
0.3) s0.1).
The boundary condition is:
Ou On = 0, x E 02
(23)
B. B.1.6. 2D POISSON EQUATION WITH MANY SUBDOMAINS (POISSON2D-MS)
The PDE and boundary condition given by:
V (..xa() = f(x, v), x
du
(24)
On + = 0, x E 82.
Here, the domain is (z, y) E Ω = [010, 10)2. The entire domain is divided into many small squares, with axx) being a piecewise linear function within each square.
17

---

PINNsAgent: Automated PDE Surrogation with Large Language Models
B.1.7. 2D HEAT WITH VARYING COBFFICIENTS (HEAT2D-VC)
The heat equation models the distribution of temperature in a given region over time, describing how heat diffuses through medium. The 2D heat equation with varying source is given by:
du Qt
V(a())Vu) = f(x,t),
(25)
where the computational domain is 12 T = [0, 1]2 x 5). The initial and boundary conditions are defined as:
u(x, y, 0) 0, u(x,, y, f) 0, z E 00.
(26)
B.1.8. 2D HEAT EQUATION WITH MULTI-SCALE FEATURES (HEAT2D-MS)
The 2D heat equation with multi-scale features is described by:
tu aL
Uxx 7 vv O,
(27)
(500T)²
over the domain Ω x T = =], x [0, 5). The initial and boundary conditions are given by:
u(x, y. 0) = sin (20mx)) sin(Ty), æ E Q
(28)
u(x,y, = O, x E 82.
B. 2D HEAT COMPLEX GEOMETRY (HEAT EXCHANGER, HEAT2D-CG)
The 2D heat equation for a complex geometry given by:
Ou
Au
(29)
Øt
The domain defined as ([-8,8) 12, 12) R,),,) [0,3].
1.1.10. 2D HEAT LONG TIME (HEAT2D-LT)
The long-time timettmmene 2D heat equation is defined by:
Du
0.001Au + 5 (ku²) (1+ 2si
π -
sin (miTT) sin (m2ny),
(30)
Øt
where the computational domain is Ω x T = =00, 1|2 [0, 100) The initial and boundary conditions given by:
u(x, y, 0) sin (mx)) sin(3 E Ω u(z, t) O, e 02
(31)
B. B.1.11. 2D STVIERSSTOES LID-DRIVEN FLOW (NS2D-C)
The 2D Navier-Stokes equations describe the motion of fluid substances and are fundamental in fluid dynamics. The governing equations for describing lid-driven cavity flow are defined as:
Vu + Vp
Au 0, 20 E Re
(32)
V =0 0, x E ft
where Ω 132 the domain. The boundary conditions are specified as follows:
18

---

PINNsAgent: Automated PDE Surrogation with Large Language Models
u(x) (4x - 0),
1,
u(x) (0, 0), p 0,
T2+
(33)
(0, 0),
where the top boundary denoted as 11 and the left, right, and bottom boundaries are denoted as 2-
1.1.12. 2D BACK STEP FLOW (NS-CG)
For the 21) back step flow, the Navier-Stokes equations and boundary conditions described as follows:
1 Au 0, V u O,
u Vu + Vp
Re
(34)
The domain defined as 5 [0,4] x [0.2]] (10,2) [1,2] U R,), which excludes the top-left quarter. The boundary conditions are:
Uin 4y y),
0, at the outlet,
(35)
a O, no-slip condition,
B.1.13. ID BASIC WAVE EQUATION (WAVEID-C)
The ID wave equation is second-order partial differential equation that describes the propagation of waves, such sound or light waves, through medium. The standard form of the wave equation in dimension is given by:
4ugz
0
(36)
The domain for this problem [0, [0,1] [0, 1). The boundary conditions are:
u(0, t) u(1, =
(37)
The initial conditions are:
u(x,0) sin(xx) 1 sin(4 (x(x, 0
(38)
1.1.14. 2D WAVE EQUATION IN HETEROGENEOUS MEDIUM (WAVE2D-CG)
The governing PDE for the 2D wave equation a heterogeneous medium given by:
02 u(x, t) =
2
(39)
c(x) 0t?
The domain is Ω [-1, x [],, and the initial conditions are:
u(2, 0) exp (- 1x 111 ) Ω
20
(40)
Iu
[z, 0) = 0, E Ω
0t
The boundary condition is:
19

---

PINNsAgent: Automated PDE Surrogation with Large Language Models
Ou
0,
02
(41)
On
B.1.15. 2D DIFFUSION- REAITION GRAY-SCOTT MODEL (GS)
The Gray-Scott model describes the pattern formation in reaction-diffusion systems. The governing PDEs are:
U, = E Au - b(1 u) wy²
(42)
Av de + ur²
The domain 12 = [|1, 200 The initial conditions are:
u(x, y, 0) = exp (-80 ((x + 0.05)² + (y 0.02))) exz 0) = exp (-80 (a--0000 0.05) + (v - -.00))))
(43)
B.1.16. 2D KURAMOTO-SIVAASIISKY EQUATION (KS)
The Auramoto-Sivashnnse equation models the chaotic behavior in systems such as flame fronts and fluid surfaces. The governing PDE is:
Ue + artlbyz + Bugr + VMarrz
(44)
The domain f2 x T = [0, 2T] x 1]. The initial condition is:
u(x, 0) =
(45)
B. 1.1. N-DIMENSIONAL POISSON EQUATION (PND)
The Poisson equation is fundamental PDE in potential theory and electrostatics. The N-Dimensional Poisson Equation is given by:
π
π
Ar
sin
Tq
(46)
4
1=1
The domain is defined by Ω2 [0, 1]".
B. 1.1.88. N-DIMENSIONAL HEAT EQUATION (HND)
The heat equation describes the distribution heat (or variation in temperature) given region over time. The N- Dimensional Heat Equation is:
Ou kAu + [(x,), (),
a TEΩx [0,
ōt
(47)
n Vu g(x, a x 0000x x [0,1] u(z,00 0) g(x,0)
xEΩ
The geometric domain (x |al2 & 1) a unit sphere in d-dimensional space.
20

---

PINNsAgent:
Automated PDE Surrogation with Large Language Models
C. Pseudocodes
Algorithm Physics-Guided Knowledge Replay (PGKR) for New PDE
1: Input: Knowledge base K {(P:, 4,, C;) where:
2:
-P; is the i-th PDE in the knowledge base,
3: - 14 is the label set of the i-th PDE,
4: -C, is the best- performmng PINNs configuration for the i-th PDE,
5: is the total number of PDEs the knowledge base.
6: Input: New PDE Pew with labels Leew
7: Output: Top-k most similar PDEs, their best-performing PINNs configurations, and the updated knowledge base
8: Initialize empty set S {} to store the similarities
9: Encode the labels of the new PDE: fnew E (fnew)
10: for = 1, 2,..., 71 do
11: Encode the labels of the i-th PDE the knowledge base: f; {44
12: Compute the cosine similarity between the new PDE and the i-th PDE:
fe [f:ll
Spew,s [
14: Add the similarity to the set S: S = S U {(P,, nnow,a) }
15: end for
Sort the set descending order based on the similarity scores
Select the top-k most similar PDEs: Sne. })
18: Retrieve the best-performing PINNs configurations for the top-k PDEs: C C (Cy1(P3. Snew E
19: Use the retrieved PINNS configurations C starting points for solving the new PDE Pnew
20: Fine-tune the PINNs configurations in C for the new PDE Prew
21: Select the best-performin PINNs configuration Cnew for the new PDE Peww
22: Add the new PDE, its feature vector, and its best pest-performing PINNs configuration to the knowledge base:
23: K IC U (Pnow: fnew, Caew)]
Return: C, and the updated knowledge base
D. Prompt
D.1. Task Description
Task Description
[Task]
You are an expert in the field of neural architecture search (NAS) and physics- inforre neural networks (PINNs). Your task to suggest a hyperparameter configuration for solving the Burgers- ID equation, Partial Differential Equation (PDE), using a PINNs model. The objective is to maximize the model's performance, measured by minimizing the Mean Squared Error (MSE) and running time.
The Burgers- 1D equation is given by:
u_t + u u_X = nu u.xx
The domain is defined as: (x, 1) Omega [-1, 1]x [0,
The initial and boundary conditions are:
u(x, 0) = -sin(pi x)
u(-1, 1) = u(1, 1) 0
The parameter is:
nu 0.01/pi
The search space includes various options for the network architecture, optimization algorithm, and other settings:
""net" options: FNN (Fully-connected neural network), LAAF (Locally Adaptive Activation Functions), GAAF (Globally Adaptive Activation Functions)
21

---

PINNsAgent: Automated PDE Surrogation with Large Language Models
"optimizer" options: adam (Adam optimizer), multiadam (Multiscale Adam optimizer), Ibfgs (Limited-memory BFGS algorithm)
The full search space is provided in the following JSON file:
yaml]
"name": "Transfer_Learning
"seed": 44, "Log_evezy": 100, "plot_every": 2000,
"repeat": 1,
"iter": 20000,
"pde_list": [' ['BurgerslD',,
"activation": ("elu", "selu", "sigmoid", "silu", "relu", "tanh", "swish" "gaussian"),
"net": ["fnn", "laaf", "geaf"), "optimizer": ["adam", "multiadam", "Ibfgs") "loss "loss_weight". ["none"],
"width": [8, 16, 20, 24, 28, 32, 36, 40, 44, 48, 52, 56, 60, 64, 68, 72,
76, 80, 84, 88, 92, 96, 100, 104, 108, 112, 116, 120, 124, 128, 132, 136, 140, 144, 148, 152, 156, 160, 164, 168, 172, 176, 180, 184, 188, 192, 196, 200, 204, 208, 212, 216, 220, 224, 228, 232, 236, 240, 244, 248, 252, 256],
"depth": [3, 4, 5, 6, 8, 9, 10), "Ir": [1e-6, 1e-5, 1e-4, 1e-3, 1e-2, 1e-1], daam_dddmiinnponttet
[100, 600, 1100, 1600, 2100, 2600, 3100, 3600, 4100, 4600, 5100, 5600, 6100, 6600, 7100, 7600, 8100, 8600, 9100, 9600],
[100, 600, 1100, 1600, 2100, 2600, 3100, 3600, 4100, 4600, 5100, 5600, 6100, 6600, 7100, 7600, 8100, 8600, 9100, 9600],
"num www__iintaaal_inntsett [100, 600, 1100, 1600, 2100, 2600, 3100, 3600, 4100, 4600, 5100, 5600, 6100, 6600, 7100, 7600, 8100, 8600, 9100, 9600],
"initializer": ["Glorot normal", "Glorot uniform" "He normal", "He uniform" "zeros"]
D.2. PGKR
PGKR
Here are the most relevant hyperparameter configurations from previous experiments on PDEs that share similar characteristics with the PDE in the current [Task]. These configurations have shown good performance the past, so they can serve as a valuable starting point for your hyperparameter search. However, keep in mind that the optimal configuration may still differ due the unique properties of the current PDE. Use these configurations guide, but don't hesitate to explore other values within the provided search space.
(experiment_logs] task run time mse 12rel mxe crmse activation net optimizer width depth ir domain_points boundary_points initial_points initializer iter
Heat2D_VaryingCon 1940.0 0.00165 0.205 0.255
sin fnn multiadam
22

---

PINNsAgent: Automated PDE Surrogation with Large Language Models
100 5
001
8192
2048
2024
Glorot normal
NaN
0 00131
0.00174
20000
Please provide single hyperparameter configuration that aims improve performance by selecting one value for each key from the search space. Return the configuration properly formatted JSON object enclosed in triple backticks (*)).
D.3. Optimization with the Guidance of Memory Tree
Memory Tree
Iteration 30:
Config: n'name': 'exp_1_Tee (exp___Transfer_iaernni
'seed' 44, 'log_every's 100, 'plot_every's 2000, repeat': 1, 'iter': 200, 'pde_list': ['BurgersiD'] ), 'activation' 'swish' 'net': ffn'', 'optimizer' 'adam', 'loss_weight's 9600,
'none', :width': 120, 'depth': 8, 'Ir': 0.001, 'uum 9600, 'initializer's
hnun_boundary_ponnt 'He normal')
9600,
MSE: 4.7700e-02 Run Time: 59.50s
Exploration Scores: activation: 1.1592 net: 0.9660 optimizer: 1.1592 width: 0.7728 depth: 0.7728 Ir: 0.7728 num wwn_ddmmannppoint.. 0.2898 num_ hum_boundary_ppp points: 0.2898 num initiial int.tiiillloinsss:: 0.2898 initializer: 0.3864 Please provide an updated JSON configuration enclosed in triple backticks ("") for the next iteration. Adjust hyperparameters to improve both the Mean Squared Error (MSE) and Run Time.
Ensure select values from the provided search space in [train.yaml].
Foeus tuning the following parameters: activation, optimizer, net. These parameters have the highest exploration scores, indicating they are the most promising for improving performance based on previous iterations. Note that we have only a GPU with 24GB of memory. Return the configuration as properly formatted JSON object enclosed in triple backticks.
23


Reviews:
[{"review_id": "MWub3Zor0Q", "reviewer": "Reviewer_Yfa3", "summary": "In this work, the authors introduce PINNsAgent, a surrogation framework that leverages large language models (LLMs) enabling efficient knowledge transfer from solved PDEs to similar problems. By leveraging LLMs and exploration strategies, PINNsAgent enhances the automation and efficiency of PINNs-based solutions. PINNsAgent is evaluated on 14 benchmark PDEs, demonstrating its effectiveness in automating the surrogation process.", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Reply to Reviewer Yfa3\n\nWe sincerely thank the reviewer for their detailed assessment of our paper. We address each concern below:\n\n## Novelty and Technical Depth\n\nOur multi-agent LLM framework is the first comprehensive system to fully automate PINNs development without expert intervention. In our pipeline, the Physics-Guided Knowledge Replay mechanism introduces a novel physics-informed approach to knowledge transfer across different PDEs, while our Memory Tree Reasoning Strategy offers a structured exploration method that captures the hierarchical dependencies unique to PINNs design.\n\n## LLM Output Reliability and Ablation Studies\n\nRegarding the reliability of LLM-generated hyperparameters, our extensive experiments in Tables 2 and 3 demonstrate that our approach significantly outperforms traditional methods like Random Search and Bayesian Optimization across diverse PDE types. This empirical evidence confirms that our LLM-based reasoning framework adds substantial value to the hyperparameter optimization process for PINNs.\n\nTo address the reviewer's concern about different LLM performance, we conducted additional experiments comparing GPT-3.5 and GPT-4. The results show that while both models improve over traditional methods, GPT-4 achieves not only better average performance but also significantly lower variance (±8.47E-02) compared to GPT-3.5 (±2.03E-01), demonstrating more consistent and reliable reasoning capabilities.\n\n**Performance comparison with different LLMs (Average MSE across 12 PDEs)**\n\n\n| **Method**           | **Average MSE**      |\n| -------------------- | -------------------- |\n| Random Search        | 6.13E-01 ± 1.49E-01 |\n| Bayesian Search      | 5.88E-01 ± 1.86E-01 |\n| PINNsAgent (GPT-3.5) | 3.89E-01 ± 2.03E-01 |\n| PINNsAgent (GPT-4)   | 3.52E-01 ± 8.47E-02 |\n\n## Handling Rare PDE Configurations\n\nFor rare PDE configurations, PINNsAgent leverages both the PGKR mechanism and the LLM's reasoning abilities. The PINNacle benchmark used in our evaluation includes a diverse range of PDEs with varying characteristics, including some with complex boundary conditions and multi-scale phenomena. Our consistent performance already across this diverse set demonstrates the framework's robustness to different PDE types.\n\nWhen encountering a completely new PDE type, the PGKR component retrieves the most similar (though not identical) PDEs from the database, and the LLM uses these as starting points for reasoning about appropriate hyperparameters. The iterative refinement process through MTRS then allows the system to adapt these initial configurations to the specific requirements of the new PDE.\n\n## MTRS Implementation Details\n\nWe appreciate the request for clarification regarding Equation 7 and the MTRS implementation. In our framework:\n\n1. The initial tree is constructed based on the LLM's knowledge and the most similar PDEs retrieved by PGKR. For each state $s$ (representing a partial hyperparameter configuration), the LLM generates a probability distribution over possible actions $a$ (hyperparameter choices).\n2. The policy model $\\pi_{pl}(a|s)$ is implemented as the LLM itself. It uses both its pre-trained knowledge and the retrieved similar PDE configurations to generate probabilities for different hyperparameter choices. These probabilities are then used in the UCT formula (Equation 7) to balance exploration and exploitation.\n3. The tree is expanded using the standard MCTS process: selection, expansion, simulation, and backpropagation. The key difference is that our expansion and simulation steps are guided by the LLM's output probabilities rather than random sampling or a separately trained policy network.\n4. The policy model does not require separate updating since the LLM adapts its recommendations based on the feedback from previous iterations, effectively implementing an adaptive policy.\n\n## Other Issues\n\nThank you for pointing out the typo in line 310. We will correct \"Figure ??\" to \"Figure 2\" in the revised manuscript.\n\nWe believe these clarifications address the reviewer's concerns and highlight the technical novelty and depth of our approach. The empirical results in Tables 2 and 3, along with the additional LLM comparison in Table 4, provide strong evidence of the effectiveness of our framework.", "claims_and_evidence": "1. PINNsAgent is an agent to enhance the process of automatically searching for the best hyperparameters settings for a PDE, may be useful for non-expert users of PINNs. \n2. This work is mainly engineering-oriented, lacking the depth suitable for ICML. The PGKR module is simple and the MTRS is a simple application of conventional Monte Carlo Tree Search technique. \n3. This work utilizes LLM's output hyperparameters to determine the action. Are the hyperparameters output by LLM reliable? Only prompt is given in the appendix. It is recomended to give examples or experiments to show the effectiveness of LLM's output. What is going to happen if we input some rare PDE configurations to LLM? Also, only GPT-4 is used in experiments, ablation study should be conducted to show the effect of using different LLMs. \n4. In the MTRS module, there is eq.7 to select the best action for a state, and \"The planner, serving as the policy model πpl(a|s), uses the distribution of the LLM’s output to determine the following action to take\". These seem confusing. Please descibe them in detail, including how the best action is selected and how the distribution of the LLM’s output is used. Is the LLM used for exploration?\n5. For the training of MCTS, what is the initial tree? Does the policy model need update?", "methods_and_evaluation_criteria": "Standard benchmark PDEs are used in experiments.", "theoretical_claims": "no theoretical part.", "experimental_designs_or_analyses": "Only GPT-4 is used in experiments, ablation study should be conducted to show the effect of using different LLMs.", "relation_to_broader_scientific_literature": "The proposed LLM-based agent to automatically search for suitable hyperparameters for PINN training seems to be new.", "essential_references_not_discussed": "no", "other_strengths_and_weaknesses": "no", "other_comments_or_suggestions": "typo: line 310, Figure ??", "questions_for_authors": "no", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9PfqyoyHS6", "reviewer": "Reviewer_znat", "summary": "The paper introduces PINNsAgent, an automated framework using LLM to design and optimize PINNs for solving PDEs. It addresses the limitations of manual hyperparameter tuning by incorporating two novel methods: Physics-Guided Knowledge Replay for efficient knowledge transfer from past experiments, and the Memory Tree Reasoning Strategy for systematic hyperparameter optimization. Experiments on various PDEs demonstrate that PINNsAgent outperforms traditional approaches.\n\n## update after rebuttal\nAfter reviewing the other reviewers' comments and the corresponding responses, the reviewer keeps the current rating.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Reply to Reviewer znat\n\nWe appreciate the reviewer's thorough assessment of our paper. Below, we address the key concerns raised:\n\n## Methods and Evaluation Criteria\n\n1. **Knowledge Transfer Evidence**: The reviewer questions whether improvements come from learning genuine transferable knowledge or merely from exhaustive hyperparameter search. As shown in our experimental setup (Section 4.1), while baseline methods (Bayesian optimization and random search) required 10 iterations to reach their best performance, PINNsAgent achieved superior results in just 5 iterations across most PDEs. This 50% reduction in required iterations demonstrates efficient knowledge transfer rather than exhaustive search.\n\n2. **PGKR's Effectiveness**: Our ablation study in Table 2 provides clear empirical evidence of PGKR's effectiveness. When PGKR is removed (\"w/o PGKR\"), performance degrades significantly across most PDEs. This direct comparison isolates PGKR's contribution to PINNsAgent's performance, demonstrating that the knowledge encoded and retrieved by PGKR substantially improves PDE solving capabilities. \n   \n\nWhile our approach builds on existing techniques, its novelty lies in the unique integration and adaptation of these methods specifically for PINNs optimization. Our multi-agent LLM framework represents the first comprehensive system to fully automate PINNs development without expert intervention.\n\n## Experimental Designs and Analyses\n\nWe appreciate the reviewer's feedback on our experimental design and have conducted additional analyses to address these concerns:\n\n### Computational Cost Analysis\n\n| **Method** | **Average Computation Time (s)** |\n|------------|--------------------------------|\n| Random Search | 3462.24 ± 2631.55 |\n| Bayesian Search | 3598.47 ± 2792.83 |\n| PINNsAgent | 3747.78 ± 2965.62 |\n\nRegarding computational cost concerns, we conducted additional analysis across all 14 benchmark PDEs, with results shown in the table. The total computation time for PINNsAgent is only about 8.2% higher than random search and 4.1% higher than Bayesian optimization. This additional cost primarily comes from LLM inference and PGKR retrieval processes.\n\n### Knowledge Transfer Verification\n\nTo demonstrate broader applicability, we also evaluated PINNsAgent on the PDEBench dataset. As shown in the table below, PINNsAgent consistently outperforms both baseline methods on these additional PDEs.\n\n| **PDE** | **PINNsAgent** | **Random Search** |\n|---------|---------------|-------------------|\n| Reaction-Diffusion 1D | 3.75E-08 ± 1.36E-08 | 4.45E-05 ± 2.84E-05 |\n| Darcy Flow 2D | 5.31E-06 ± 6.60E-08 | 9.22E-06 ± 2.31E-07 |\n\n**Relation To Broader Scientific Literature**\nWe appreciate the reviewer's insights. Our focus on PINNs is deliberate as these models are particularly sensitive to hyperparameter choices - far more than standard neural networks or even neural operators. Small configuration changes in PINNs can lead to order-of-magnitude differences in accuracy, making automated tuning especially valuable for this domain, an observation also confirmed by Wang et al. 2024 in their NAS-PINN work. \n\n* [1] Wang, Yifan, and Linlin Zhong. \"NAS-PINN: Neural architecture search-guided physics-informed neural network for solving PDEs.\" *Journal of Computational Physics* 496 (2024): 112603.\n\n## Essential References Not Discussed\nWe agree with the reviewer's assessment that our paper's primary focus is on LLM-enabled AutoML for hyperparameter tuning rather than PINNs methodology itself. As noted, we have already included the essential references related to this focus in Section 2.3. We will enhance the literature review by incorporating additional relevant references on hyperparameter tuning approaches to provide a more comprehensive context for our work.\n\n## Other issues\n\n1. **Definition of L_BC:** We agree that L_BC should be formally defined in the Preliminaries section. We will add the boundary condition loss definition alongside the other loss components to ensure completeness.\n   \n$$ L_{BC} = \\frac{1}{N_{BC}} \\sum_{i=1}^{N_{BC}} \\left| u_{\\theta}(\\mathbf{x}_i^{BC}) - u_{BC}(\\mathbf{x}_i^{BC}) \\right|^2 $$\n\n2. **Figure reference:** Thank you for catching this error. We will correct \"Figure ??\" to \"Figure 2\" in line 310.\n\n3. **Scope and PINNs architecture:** Our framework is designed to be flexible regarding model architecture and training strategies. As mentioned in Section 4.2, our configuration files allow users to specify various PINN variants and training techniques. This includes addressing known issues like spectral bias (Krishnapriyan et al., NeurIPS 2021) through techniques such as curriculum learning, adaptive weighting, and alternative network architectures. The LLM agents can select and configure these options based on the specific PDE characteristics. We will clarify this flexibility more explicitly in the revised manuscript to address this concern.", "claims_and_evidence": "The authors' experiments partially provide empirical evidence supporting their claims. However, there are several limitations:\n\n1.\tThe experiments can provide empirical evidence for the claims to some extent, but the results do not conclusively show that the proposed approach successfully learns or transfers domain-specific knowledge. It remains unclear whether the observed improvements come from learning genuine transferable knowledge or merely from exhaustive hyperparameter search.\n\n2.\tThere is insufficient theoretical analysis and empirical evidence to substantiate that the information encoded by PGKR is both effective and transferable across different PDEs.\n\n3.\tAdditionally, the authors have not provided adequate theoretical justification or empirical validation for the effectiveness of representing the hyperparameter tuning problem explicitly as a tree-structured search.", "methods_and_evaluation_criteria": "The proposed method and evaluation criteria make sense to the problem. But it is worth pointing out here that a large portion of the proposed method seems to be a combination of existing methods while acknowledging there are some new aspects added to the existing work.", "theoretical_claims": "There is no theoretical claim in this paper.", "experimental_designs_or_analyses": "Although the authors have compared their proposed methods with two baselines and conducted the ablation studies, the experiments are insufficient. \n\nFirst, the paper lacks detailed analysis or discussion regarding the computational cost associated with implementing and running the proposed framework, especially given the iterative nature of the Memory Tree method. It could be extremely time consuming when the search space for hyperparameters is large.\n\nSecond, there is no experiments to verify that the PINNs actually can learns or transfers domain-specific knowledge from PINNsAgent.\n\nThird, there is insufficient theoretical analysis and empirical evidence to substantiate that the information encoded by PGKR is both effective and transferable across different PDEs.\n\nFourth, the authors have not provided adequate theoretical justification or empirical validation for the effectiveness of representing the hyperparameter tuning problem explicitly as a tree-structured search.", "relation_to_broader_scientific_literature": "As it primarily addresses hyperparameter tuning specifically for PINNs, there seem to be some contributions to the scientific community interested in using PINNs. However, it is less clear if the proposed method has been thoroughly studied (e.g., computational costs, accessibility -- the current one uses GPT 4, variants of PINNs architectures)  could be generally applicable to other relevant methods (such as neural operators).", "essential_references_not_discussed": "No. Although there are some PINNs-related papers that could be discussed here in this paper, the main focus of the paper is not the PINN itself, but hyper-parameter tuning. Regarding the hyper-parameter tuning, not extensive, but essential papers seem to be included.", "other_strengths_and_weaknesses": "S1: The authors propose their own framework to reduce manual effort and reliance on expert knowledge for solving PDE.\n\nS2: The authors conduct some experiments with the proposed methods.\n\nW1: This paper's technical contributions do not seem to reach the bar. \n\nW2: The scope of this paper is rather limited as it is only designed for hyperparameter tuning for PINNs (only the vanilla PINNs architecture, which is known to suffer from many technical issues e.g., spectral bias in PINNs failure mode by Krishnapriyan et al, NeurIPS 2021).\n\nW3: This paper lacks theoretical evidence, and the experiments are insufficient to verify the effectiveness of their proposed method. \nW4: Different components lack motivations. For instance, the authors should explain why they formulate the searching for hyperparameters as a tree structure, and why MCTS process is a potential optimal choice.", "other_comments_or_suggestions": "Some minor issues: \n- The author should provide the provide the definition of L_BC in the Preliminary. \n\t\n- In line 310 “Figure ??” should be “Figure 2”.", "questions_for_authors": "The major questions are relevant to the points raised in \"Experimental Designs Or Analyses\". Including those, other points raised in the above sections (weaknesses, etc) would be the questions.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Htzx4uTYhI", "reviewer": "Reviewer_gzrR", "summary": "This paper introduces PINNsAgent, a framework that uses large language models (LLMs) to automate the development and optimization of Physics-Informed Neural Networks (PINNs) for solving partial differential equations (PDEs). The key components are:\n1. Physics-Guided Knowledge Replay (PGKR) – encodes PDE characteristics and associated PINN configurations into a structured format to enable knowledge transfer between similar PDEs.\n2. Memory Tree Reasoning Strategy (MTRS) – abstracts the hyperparameter optimization process as MCTS.\n\nThe framework is evaluated on 14 benchmark PDEs and demonstrates strong performance compared to random search and Bayesian optimization.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Reply to Reviewer gzrR\n\nWe sincerely thank the reviewer for their thoughtful assessment and insightful questions. We address each point below.\n\n## Baseline Details and Computational Costs\n\n We ran all experiments on 8 NVIDIA V100 (32GB) GPUs, providing sufficient computational resources to complete all benchmark PDEs in the PINNacle dataset. \n\nCompared to traditional methods, the additional computational overhead in PINNsAgent comes primarily from LLM inference and PGKR retrieval processes. We also observed that the LLM tends to recommend slightly larger (though still reasonable) network architectures, which marginally increases training time. To address the reviewer's concern about computational costs, we conducted additional experiments comparing all methods (Random Search, Bayesian Search, and PINNsAgent) with 5 iterations each. As shown in the table below, PINNsAgent introduces only modest computational overhead (approximately 8.2% compared to Random Search) while delivering substantially better performance.\n\n| **Method** | **Average Computation Time (s)** |\n|------------|-----------------------------------|\n| Random Search | 3462.24 ± 2631.55 |\n| Bayesian Search | 3598.47 ± 2792.83 |\n| PINNsAgent | 3747.78 ± 2965.62 |\n\n## PDE-Dependent Hyperparameter Sensitivity\n\nRegarding the reviewer's question about hyperparameter variation across PDEs, we found that PINNs exhibit significant hyperparameter sensitivity compared to neural operator methods, even with identical architectures. We found clear patterns in optimal hyperparameters across different PDE types:\n\nDifferent PDE types consistently favor certain optimizers. For time-dependent diffusion equations (Heat series: HeatND, Heat2D_Multiscale, Heat2D_VaryingCoef), the LBFGS optimizer significantly outperforms other choices. For example, on HeatND, LBFGS (5.64E-08) outperforms MultiAdam (7.93E-08) by approximately 29%. This advantage persists even with increasing problem dimensionality, suggesting that the diffusion mechanism, rather than dimensionality, drives optimizer selection. For static problems (Poisson class), MultiAdam performs better on complex geometries.\n\nThe second most important hyperparameter is the activation function. Problems with smooth solutions (Heat, Poisson) benefit most from tanh and sin functions, while problems with sharp gradient changes (Burgers, NS) perform better with gaussian and swish activations. On NS2D_LidDriven, for instance, gaussian (9.89E-06) outperforms sin (1.27E-05).\n\nNetwork size only needs to be reasonable; excessively large networks do not significantly improve PINN performance but substantially increase computational burden. We also found that advanced architectures like LAAF and GAAF (Jagtap et al., 2020) do not consistently deliver the best performance--the original PINN architecture often proves more robust across different PDEs. These findings validate our approach's ability to identify nuanced relationships through the PGKR framework.\n\n## Additional References\n\nWe thank the reviewer for suggesting the additional references. We will incorporate these papers in our discussion of related work.", "claims_and_evidence": "Convincing evidence is provided for the benefits of PINNsAGents for PINN hyperparameter search:\n* Performance improvements are demonstrated through comprehensive experiments on 14 diverse PDEs. PINNsAgent consistently outperforms random search and Bayesian optimization on most benchmark problems. Results are also averaged over 10 runs to account for randomness. It also outperforms the best PINNacle MSEs on a number of tasks, with especially promising results on Heat-ND.\n* Ablation studies validate the contributions of both PGKR and MTRS components.", "methods_and_evaluation_criteria": "The methods and evaluation approach appear mostly sound, though more details about the evaluation setup and baselines would be helpful (see questions below):\n* The benchmark set (PINNacle) includes a diverse range of PDEs with varying characteristics. Furthermore, this is a standard benchmark used in the PINN literature.\n* Performance is measured using a reasonable metric (MSE), although it would also be helpful to report relative error metrics.\n* Baselines include both simple (random search) and sophisticated (Bayesian optimization) approaches. However, details about the baseline methods, including compute budget vs. performance, seem to be missing.", "theoretical_claims": "N/A, no theoretical claims made.", "experimental_designs_or_analyses": "The experimental analyses seem sound, though details about the baselines are missing. See questions below.", "relation_to_broader_scientific_literature": "The paper builds on several research directions:\n* Physics-informed neural networks (PINNs) for solving PDEs\n* LLM-based automation of machine learning pipelines\n* Neural architecture search / AutoML for scientific computing\n\nPrior work finds that the performance of PINNs depends crucially on the choice of architecture (e.g. activation function) and optimizer. This positions hyperparameter optimization for PINNs as an important problem for improving the performance of ML methods on PDEs.", "essential_references_not_discussed": "The discussion of related work is generally thorough. A couple missing references regarding the intersection of LLMs and Bayesian optimization or Neural Architecture Search are:\n* Large Language Models to Enhance Bayesian Optimization. Liu et al, ICLR 2024.\n* EvoPrompting: Language Models for Code-Level Neural Architecture Search. Chen et al, NeurIPS 2023.\n* LLMatic: Neural Architecture Search via Large Language Models and Quality Diversity Optimization. Nasir et al, GECCO 2024.", "other_strengths_and_weaknesses": "n/a", "other_comments_or_suggestions": "n/a", "questions_for_authors": "1. Standard hyperparameter optimization papers show the tradeoffs between performance and time/compute/number of HPO iterations. However, these evaluations seem to be missing, and the “random search” and “Bayesian search” baselines seem to only report one point along this performance-cost tradeoff curve. Could the authors describe the baselines in more detail, including number of iterations or compute cost? This is crucial for understanding the main results (Table 2).\n  * Furthermore, could the authors clarify the total computational cost of running PINNsAgent compared to baselines? This is important for understanding practical trade-offs.\n\n2. How much do the optimal hyperparameters vary across the different PDEs, and how much do they depend on all of the PDE features encoded within the PGKR scheme? This information would clarify whether an HPO method like PINNsAgent is necessary for different PDEs, vs. if existing PINN architectures/hyperparameters are simply undertuned.\n  * Intuitively, I might expect the optimal hyperparameters should depend mostly on a few features, e.g. equation type and time-dependence. Did the authors conduct any investigation about which features are the most important?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7n0eaZJb1K", "reviewer": "Reviewer_mBxJ", "summary": "The paper introduces a framework that utilizes LLMs to design and optimize PINNs to solve PDEs. It facilitates solving PDEs with PINNs more efficiently without tuning parameters and choosing architectures manually. The paper demonstrated its effectiveness on dataset PINNacle.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Reply to Reviewer mBxJ\n\nWe sincerely thank the reviewer for their thoughtful comments and constructive feedback. We address each concern below and explain how we will improve the manuscript accordingly.\n\n## The Unique Role of LLM in Our Framework\n\nThe LLM serves several critical and unique functions in PINNsAgent that would be difficult to achieve with smaller language models. To illustrate this, we conducted additional experiments comparing GPT-3.5 and GPT-4 (see table below). The results show that while both models improve over traditional methods, GPT-4 achieves not only better average performance but also significantly lower variance (±8.47E-02) compared to GPT-3.5 (±2.03E-01), demonstrating more consistent and reliable reasoning capabilities. These unique functions include:\n\n| **Method** | **Average MSE** |\n|------------|-----------------|\n| Random Search | 6.13E-01 ± 1.49E-01 |\n| Bayesian Search | 5.88E-01 ± 1.86E-01 |\n| PINNsAgent (GPT-3.5) | 3.89E-01 ± 2.03E-01 |\n| PINNsAgent (GPT-4) | 3.52E-01 ± 8.47E-02 |\n\n1. **Reasoning across physics and deep learning domains**: The LLM planner must simultaneously understand PDE characteristics (equation type, dimensionality, boundary conditions) and relate them to appropriate neural architectures. This cross-domain reasoning requires the sophisticated knowledge integration capabilities of large language models.\n\n2. **Memory Tree Reasoning Strategy (MTRS) implementation**: As detailed in Section 3.4, our MTRS approach requires the LLM to function as a policy model π_pl(a|s) that guides the exploration of the hyperparameter search space. The LLM's probabilistic outputs are used directly in the UCT formula to balance exploration and exploitation.\n\n3. **Adaptive hyperparameter generation**: The LLM analyzes feedback from previous iterations f^(t-1) and adjusts hyperparameters accordingly. This requires understanding complex relationships between hyperparameters (e.g., how learning rate interacts with optimizer choice) and PDE characteristics.\n\n## Pipeline for PDEs Not Already in the Code Bank\n\nFor PDEs not in the Code Bank, PINNsAgent operates in the Config Generation mode, as the Code Bank contains base code that can be applied to any unseen PDE with the appropriate configuration. The complete pipeline is:\n\n1. **PDE encoding**: The system encodes the mathematical and physical properties of the new PDE using the comprehensive set of labels described in Section 3.3.\n\n2. **Physics-Guided Knowledge Replay (PGKR)**: PGKR computes weighted cosine similarity between the encoded target PDE and all PDEs in the database. The top-K most similar PDEs are retrieved along with their best-performing configurations.\n\n3. **Configuration generation**: The planner uses these retrieved configurations as starting points and generates YAML configuration files for the new PDE.\n\n4. **Base code application**: The system applies the generated configuration to the appropriate base code from the Code Bank. This base code is designed to be flexible and can handle any PDE when provided with the correct configuration.\n\n5. **Memory Tree-guided exploration**: Following the MTRS approach described in Section 3.4, the system iteratively refines the hyperparameter configuration.\n\n6. **Database update**: Successful configurations are added to the database to benefit future queries.\n\n## Additional Benchmark Datasets\n\nWe appreciate the suggestion to evaluate on additional benchmark datasets. As shown in the table below, we conducted experiments on two representative PDEs from the PDEBench dataset: Reaction-Diffusion 1D and Darcy Flow 2D. PINNsAgent consistently outperforms both Random Search and NAS-PINNs on these additional PDEs, demonstrating its broader applicability beyond the PINNacle benchmark.\n\n| **PDE** | **PINNsAgent** | **Random Search** |\n|---------|---------------|-------------------|\n| Reaction-Diffusion 1D | 3.75E-08 ± 1.36E-08 | 4.45E-05 ± 2.84E-05 |\n| Darcy Flow 2D | 5.31E-06 ± 6.60E-08 | 9.22E-06 ± 2.31E-07 |\n\nThese additional experiments provide more comprehensive evidence of PINNsAgent's effectiveness and generalizability across different PDE types and benchmarks.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria are well-suited to the problem at hand. However, I believe the paper could be improved by clarifying the following two aspects of the proposed method:\n\n1. What is the unique role of the LLM in your framework? Could a smaller language model be used instead? If the focus is solely on configuration generation, is an LLM even necessary?\n\n2. Could you provide more details on the pipeline for PDEs that are not already in the code bank? Perhaps an additional section would help clarify this process.", "theoretical_claims": "There's no theoretical claims in the paper.", "experimental_designs_or_analyses": "The experimental design in the paper is solid. However, the study was conducted on only one dataset, which is not a commonly used benchmark. I recommend that the authors conduct more extensive experiments to evaluate the framework’s performance on widely used benchmark datasets, such as PDEBench or PDEArena. This would better demonstrate the framework’s real-world applicability, particularly by comparing its performance with other PDE solvers and showing whether it produces comparable or superior results.", "relation_to_broader_scientific_literature": "The scientific computing community can benefit from the proposed framework, as it automates the workflow for solving PDEs using PINNs. While PINNs are effective, they have a significant drawback—the process of tuning and designing the model structure requires substantial human effort. This framework helps bridge that gap.", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "I believe the paper presents an innovative idea, and the problem it aims to solve is well-motivated with broad applications.\n\nHowever, the primary weakness, in my view, is the lack of comprehensive experimental results.", "other_comments_or_suggestions": "NA", "questions_for_authors": "Please see above comments.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The reviewed work designs an LLM-based agent that uses PINNs to solve PDEs. The reviewers mostly appreciated the novelty of this approach and found the paper interesting. However, some reviewers found that the paper lacked conceptual depth. A widespread concern was that the provided experiments were not quite convincing enough as far as the generalizability of the proposed method is concerned. In light of these concerns, I can only weakly recommend this paper for acceptance at the current point. I personally believe that the avenue persued by the authors is a natural yet innovative one that could spur interesting follow-up work, not just on automating PINNs but numerical methods more generally. I therefore believe it would be a valuable addition to the conference program.
Decision: Accept (poster)