<figure><img src="image_1.png" /></figure>

HAL
open science

Augmented-GDA: LLM In-Loop Gradient Descent
Algorithm for Smooth Unconstrained Optimization

Belgacem Ben Ziada, Thomas Mongaillard, Hang Zou, Samson Lasaulce,
Anthony Girardin, Vineeth S. Varma, Mérouane Debbah

► To cite this version:

Belgacem Ben Ziada, Thomas Mongaillard, Hang Zou, Samson Lasaulce, Anthony Girardin, et al. Augmented-GDA: LLM In-Loop Gradient Descent Algorithm for Smooth Unconstrained Optimization. The 3rd International Conference on Foundation and Large Language Models (FLLM2025), IEEE, Nov 2025, Vienne, Austria. hal-05388565

HAL Id: hal-05388565

https://hal.science/hal-05388565v1

Submitted on 2 Dec 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

Public Domain
# Augmented-GDA: LLM In-Loop Gradient Descent Algorithm for Smooth Unconstrained Optimization

Belgacem Ben Ziada*, Thomas Mongaillard* Hang Zou†, Samson Lasaulce‡

Anthony Girardin§, Vineeth S. Varma‡, Mérouane Debbah†

*École Polytechnique, 91120 Palaiseau, France

†6G Research Center, Khalifa University, 127788 Abu Dhabi, UAE

‡Université de Lorraine, CNRS, CRAN, F-54000 Nancy, France

§DAES, Geneva, Switzerland

**Abstract**—The problem of exploiting large language models (LLMs) for solving optimization problems is new and existing papers rarely address the enhancement of standard solvers through LLM capabilities. We introduce an approach to augment gradient descent algorithms (GDAs) by adopting an agentic perspective. Indeed, GDAs remain central to optimization and modern machine learning, yet its efficiency often depends on brittle and time-consuming hyperparameter tuning. We investigate whether an LLM can contribute inside a running gradient method for smooth, unconstrained, single-objective standard-form optimization problems (SFOP) with known or estimable gradients. We present Augmented GDA (AGDA), an LLM in-loop controller that monitors progress and adapts learning-rate schedules in real time while keeping the optimizer family fixed within each run. This places the LLM where feedback is most informative, reducing manual trial-and-error without altering the numerical core. Experiments on benchmark functions and two famous deep learning tasks show that in-loop scheduling is competitive with strong static baselines and can yield faster convergence or lower loss.

**Index Terms**—Optimization, gradient descent, large language models, hyperparameter scheduling, agentic systems

## I. INTRODUCTION

Optimization is a backbone of scientific computing and machine learning: advances in control, design, inverse problems, and large-scale training pipelines are routinely unlocked by better optimizers. Yet the objectives encountered in practice are heterogeneous and often nonconvex; their geometry may be ill-conditioned, anisotropic, or dominated by wide plateaus; evaluations may be noisy or costly; and the same fixed recipe can behave very differently across tasks and scales. In this regime, practical performance hinges on a web of interdependent choices that must be adapted to the problem at hand, including the optimizer family, learning-rate (LR) schedule, initialization strategy, and ancillary hyper-parameters (for example momentum, batch size, and stopping rules). This tuning is painstaking and brittle, difficult to reproduce, and it rarely transfers cleanly across datasets or compute budgets.

The rapid development of large language models (LLMs) such as GPT-5 [1], Gemini-2.5 [2] and DeepSeek-R1 [3] has transformed them from assistants for routine tasks to general-purpose reasoning systems capable of solving hard math problems and complicated code generations tasks. Especially, LLMs are increasingly used along the optimization pipeline,

from translating informal problem statements into formal optimization problems (OPs) to assisting or even replacing numerical solvers. Recent works such as OptLLM [4] and OptimAI [5] translate natural language requests into formal optimization artifacts by chaining retrieval, schema filling, and code generation. These “front-end” systems address specification, constraint extraction, and solver selection, showing that LLMs can operationalize optimization tasks end-to-end when input is textual and underspecified. OPRO [6] frames optimization as prompt evolution, where the LLM proposes candidate solutions using a meta-prompt with scores, effectively acting as a black-box optimizer. TextGrad [7] generalizes this idea by treating critiques as “textual gradients”, while LEO [8] explores evolutionary search entirely within the model’s proposal space. These approaches are compelling for unknown or non-differentiable objectives but operate outside numerical updates. AlphaEvolve [9] orchestrates an evolutionary pipeline of LLMs that iteratively edit code modules with automated evaluation, reporting discoveries in algorithms and system kernels. In applied operation research (OR), stacks such as OptimAI [5] similarly plan, synthesize, and test solver code from textual goals, with planner/critic loops grounded by execution.

Other work uses LLMs to improve existing heuristics: for example, augmenting a construct, merge, solve and adapt (CMSA) heuristic with LLM-generated suggestions yields consistent gains over a human-designed baseline [10]. Hybrid strategies also appear, where an LLM partitions the search space or coordinates multiple solvers [11]. Beyond code synthesis, systems have been built to solve optimization tasks from natural language directly. VRQ2Vec [12] converts voice requests into power-scheduling vectors via three LLM components for intent recognition, parameter extraction, and solver interfacing, illustrating both feasibility and the trade-off between catalogue breadth and recognition accuracy. In hyper-parameter optimization, frameworks such as AgentHPO [13] and OptFormer [14] use LLMs to propose configurations, justify them, and adaptively refine choices based on evaluation feedback. These examples highlight the practical value of hybrid designs, though most still keep the LLM outside the numerical loop.

External hyperparameter optimization (HPO) and black-
box LLM optimizers operate between runs, consuming many full evaluations and leaving intra-run feedback unused; they help pick configurations but do not react to transient regimes (plateaus, oscillations) as they emerge. Such transient regimes correspond to phases where the optimizer temporarily slows down or oscillates before reaching a stable descent direction; during these phases, adaptive step-size or momentum adjustments can significantly improve convergence speed and stability. Pure-LLM approaches in discrete spaces (e.g., prompt/algorithm editing) demonstrate high-level reasoning but typically lack a well-specified control surface over a numeric core for continuous OPs. Conversely, classical adaptive methods (momentum schedules, step-size heuristics) are hand-engineered for specific regimes and do not leverage problem priors or cross-run experience. To our best knowledge, there is no framework that (i) positions an LLM in-loop over OPs with known or approximable gradients, (ii) specifies a principled static vs dynamic contract with guardrails, and (iii) evaluates the resulting controller as a solver-level capability rather than a single heuristic. This is the gap we address.

We consider the so-called *meta-solver* methods: an in-loop LLM controls several task-specific decision agents that coordinate multiple levers of a conventional gradient method. A compact history module summarizes each run and informs the next, turning in-loop LLM control into a solver-level capability that composes strategies over time, reconciles global heuristics with local signals, and adapts without restarts. Instead of introducing a new update rule, we define a bounded control surface and contracts that make in-loop decisions practical, safe, and reproducible for smooth, unconstrained, single-objective problems with known or estimable gradients.

Besides, our goal is not to build a super-solver but to test whether in-loop LLM controls can meet or exceed human choices within a fixed family of gradient descent algorithm (GDA) variants. We use a human-optimality lens (HOL): compare against the best grid-searched configuration in-family, report slowdown, success probability under a fixed budget, and dispersion, and ask whether the controller matches experts on easy regimes, surpasses engineer-level heuristics, and stabilizes outcomes. This places our study in the hybrid, in-loop, known-objective standard-form optimization problem (SFOP) setting, distinct from black-box prompt evolution or math word problems. Our main contributions are summarized here:

*   **Meta-solver formulation with agentic structure.** We formalize an LLM in-loop controller that coordinates multiple levers of a gradient method under a principled *static vs dynamic* contract. The design treats the controller as a solver-level capability rather than a single heuristic.

*   **Interfaces, guardrails, and reproducibility.** We specify clean inputs/outputs for decisions, define when actions can be taken, and describe guardrails (schema checks, bounds, conservative fallbacks) that make free-form proposals safe to execute.

*   **Evidence across standard benchmarks and multiple applications (HOL view).** Within the smooth, unconstrained setting (with exact or approximated gradients),

the meta-solver consistently competes with strong static recipes under identical conditions and often stabilizes outcomes; it surpasses engineer-level hyperparameter choices while setting a realistic baseline relative to expert tuning in the same class.

The remainder of the paper is organized as follows. We formulate the problem formally in Sec. II. Then we explain the design principle of our framework and then propose our AGDA by presenting the functionality of LLM-powered agents in Sec. III. After that, in Sec. IV, we validate the effectiveness of the proposed method through several analytical test functions and two practical deep learning tasks. Finally, we conclude the paper in Sec. V.

## II. PROBLEM FORMULATION

We focus on **smooth, unconstrained, single-objective optimization** in the standard form, allowing for nonconvex objectives. Values of the objective function are available and gradients are either computed exactly or approximated by standard estimators e.g., finite differences or stochastic perturbations such as simultaneous perturbation stochastic approximation (SPSA), when exact derivatives are not exposed. Unless otherwise stated, we do not require global differentiability or Lipschitz continuity; “smooth” here simply denotes functions for which a reliable descent direction can be obtained. Without loss of generality, we consider unconstrained, single-objective optimization in $\mathbb{R}^d$:

$$ \min_{x \in \mathbb{R}^d} f(x). \qquad (1) $$

The numerical core of GDA can be written as:

$$ x_{t+1} = x_t - \eta_t g_t(x_t; \theta_t), \qquad (2) $$

where $x_t \in \mathbb{R}^d$, $\eta_t > 0$ is the stepsize at iteration $t$, and $g_t(\cdot; \theta_t)$ is a descent direction with internal state $\theta_t$ (e.g., momentum or moments). A run is declared *converged* as soon as either a function-value or a gradient-norm condition is met,

$$ f(x_T) \le \varepsilon_f \quad \text{or} \quad \|\nabla f(x_T)\|_2 \le \varepsilon_g, \qquad (3) $$

for some iteration $T < T_{\max}$. The exact tolerances $(\varepsilon_f, \varepsilon_g)$ and the cap $T_{\max}$ are specified per benchmark in the experimental protocol. If neither condition is met by $T_{\max}$, the run terminates unsuccessfully. Within a run, a controller monitors stagnation with a patience window of length $K$. We define best value so far of the objective function as $f_{\text{best}}(t) = \min_{0 \le i \le t} f(x_i)$ and the improvement at iteration $t$ as:

$$ \Delta_t \equiv f_{\text{best}}(t-1) - f(x_t). \qquad (4) $$

An iteration is said to be of *significant improvement* if $\Delta_t \ge \varepsilon_{\text{imp}}$ with $\varepsilon_{\text{imp}}$ a given threshold; otherwise the patience counter is incremented. We declare stagnation when there has been no significant improvement for $K$ consecutive iterations and the gradient is small:

$$ \text{STAGNANT}(t) := (P_t \ge K) \wedge \|\nabla f(x_t)\|_2 \le \varepsilon_{g, \text{stag}}, \qquad (5) $$
with $\epsilon_{g,stag}$ a predefined threshold. A run terminates either when the convergence criteria are satisfied or when the iteration cap $T_{max}$ is reached. Each optimization session operates under a budget of at most $B_{max}$ complete runs. Run $r \in \{1, \dots, B_{max}\}$ proceeds until convergence as in Eq. (3) or until reaching $T_{max}$. Without loss of generality, We consider with the following classical gradient descent (GD) optimizer families in the canonical forms.

* **Gradient Descent.** The standard form is

$$x_{t+1} = x_t - \eta_t \nabla f(x_t). \quad (6)$$

* **Momentum [15] (exponential moving average or EMA of gradients).** An exponential moving average of gradients is used as direction:

$$v_t = \beta v_{t-1} + (1-\beta) \nabla f(x_t), \quad x_{t+1} = x_t - \eta_t v_t. \quad (7)$$

This “EMA-momentum” differs algebraically from Polyak’s heavy-ball below.

* **Heavy-Ball (Polyak) [16].** A velocity is accumulated on the parameter updates:

$$v_t = \mu v_{t-1} - \eta_t \nabla f(x_t), \quad x_{t+1} = x_t + v_t. \quad (8)$$

Unlike the EMA-momentum, the momentum acts on step increments.

* **Nesterov’s accelerated gradient (NAG) [17].** In the standard look-ahead form,

$$y_t = x_t + \mu (x_t - x_{t-1}), \quad x_{t+1} = y_t - \eta_t \nabla f(y_t), \quad (9)$$

i.e., the gradient is evaluated at the look-ahead point $y_t$.

* **Adam [18].** Adaptive momentum estimation:

$$\begin{aligned} m_t &= \beta_1 m_{t-1} + (1 - \beta_1) \nabla f(x_t), \\ v_t &= \beta_2 v_{t-1} + (1 - \beta_2) (\nabla f(x_t))^{\odot 2}, \\ \hat{m}_t &= \frac{m_t}{1 - \beta_1^t}, \quad \hat{v}_t = \frac{v_t}{1 - \beta_2^t}, \quad x_{t+1} = x_t - \eta_t \frac{\hat{m}_t}{\sqrt{\hat{v}_t + \epsilon}}. \end{aligned} \quad (10)$$

* **Nadam [19] (Nesterov-accelerated Adam).** A Nesterov look-ahead is applied to the first moment:

$$m_t^{\text{nest}} = \beta_1 \hat{m}_t + \frac{1 - \beta_1}{1 - \beta_1^t} \nabla f(x_t), \quad x_{t+1} = x_t - \eta_t \frac{m_t^{\text{nest}}}{\sqrt{\hat{v}_t + \epsilon}}. \quad (11)$$

With these concepts clearly defined, we can discuss the role of LLM in GDA. We model the decision process of an LLM in GDA as

$$P_{\text{LLM}}((\eta_t, \theta_t) | H_t, p_t), \quad (12)$$

where $H_t = \{(x_k, f(x_k), \nabla f(x_k))\}_{k \le t}$ is the trajectory history and $p_t$ is the current prompt. The controller samples or takes the argmax to obtain $(\eta_t, \theta_t)$, applies the update in Eq. (2), then forms next prompt $p_{t+1}$ from the new state and an optional feedback signal $c_t$ from the critic. The core problem for an in-loop LLM controller is to specify which control decisions an LLM may influence, when it may act, and how these actions are validated before affecting the state of the optimizer, which we will present in the next section.

### III. PROPOSED METHOD

We expose two interfaces to the LLM that reflect distinct reliability profiles and an explicit trade-off between correctness and exploration. Static decisions are drawn from a finite, validated menu (classifier-like). Because each option is pre-validated against interface invariants, these choices guarantee admissibility (syntactic and semantic validity) and reproducibility at the cost of expressivity. Dynamic decisions are free-form proposals (generator-like): they offer greater flexibility and exploratory power, yet come with uncertainty (the model may err or overreach). We intentionally place safety-critical structure in the static channel (e.g., selecting a stable optimizer family per run, or fixing the mode by which directions are computed), while delegating high-payoff, context-sensitive adaptations to the dynamic channel (e.g., proposing a piecewise-constant schedule, refreshing it on observed stagnation; proposing distributional initializations with tunable weights). Dynamic outputs flow through lightweight guardrails such as schema/consistency checks, bounds, and conservative fallbacks so that free-form suggestions cannot invalidate the run. This contract makes the in-loop controller both auditable and effective, explicitly balancing guaranteed correctness (static) with guided exploration (dynamic).

Our LLM-in-loop AGDA consists of six LLM-driven agents, namely, optimizer variant agent, init/distribution agent, LR agent, gradient-estimation agent, hyper-parameter agent and critic agent. Their roles are as follows:

**Optimizer Variant Agent:** At the start of each run, the controller asks the LLM to select an optimizer family and keep this choice static and fixed for the whole run. The motivation is twofold: (i) every choice is a well-specified, numerically sound update with a known state $\theta_t$, and (ii) reproducibility is preserved because runs are comparable when the inner dynamics are fixed. This conservative anchor keeps the controller within a human-designed family of descent methods, leaving variant invention for future work so that the present study isolates the value of in-loop control.

**Init/Distribution Agent:** We let the LLM either provide a single $x_0 \in \mathbb{R}^d$ or specify a parametric distribution from which the controller samples the starting point(s). The family of distributions (e.g., Gaussian with diagonal covariance, uniform box, or a small mixture) is drawn from a short validated list, but the parameters (means, scales, bounds, mixture weights) are proposed dynamically. This mixed design gives the model geometric freedom to encode priors about promising regions (wide boxes for exploration, narrow Gaussians for focused entry into a valley) while keeping parsing and sampling robust. In practice, this is where the LLM can express its “shape intuition” about the objective without risking malformed inputs.

**LR Agent:** At run start, the LLM proposes a piecewise-constant schedule $\eta_t$ with at most four segments. When the stagnation trigger fires, it receives a compact snapshot and returns either continue or an update that replaces the remaining schedule from the current iteration. The interface is dynamic and lightweight, which keeps changes auditable
<table><caption>TABLE I: Full-controller results by function and dimension, computed from aggregated run logs. Dashes indicate that no run met the success criterion under the budget.</caption><thead><tr><th>Function</th><th>Dimension</th><th>Success (%)</th><th>T<sub>conv</sub> median</th><th>T<sub>conv</sub> mean</th><th>f<sub>final</sub> mean</th><th>f<sub>final</sub> variance</th></tr></thead><tbody><tr><td rowspan="2">Rosenbrock</td><td>2</td><td>54.69</td><td>2807</td><td>2721.4</td><td>9.16 × 10<sup>-4</sup></td><td>1.00 × 10<sup>-6</sup></td></tr><tr><td>10</td><td>80.66</td><td>2615</td><td>2670.9</td><td>6.15 × 10<sup>-1</sup></td><td>1.91 × 10<sup>0</sup></td></tr><tr><td rowspan="2">Ackley</td><td>2</td><td>—</td><td>—</td><td>—</td><td>1.47 × 10<sup>0</sup></td><td>2.54 × 10<sup>0</sup></td></tr><tr><td>10</td><td>—</td><td>—</td><td>—</td><td>3.60 × 10<sup>0</sup></td><td>4.24 × 10<sup>0</sup></td></tr><tr><td rowspan="2">Beale</td><td>2</td><td>100.00</td><td>197</td><td>347.4</td><td>—</td><td>—</td></tr><tr><td>10</td><td>34.38</td><td>1064.5</td><td>1130</td><td>9.29 × 10<sup>-1</sup></td><td>3.83 × 10<sup>-1</sup></td></tr><tr><td rowspan="2">Himmelblau</td><td>2</td><td>100.00</td><td>130</td><td>133.8</td><td>—</td><td>—</td></tr><tr><td>10</td><td>100.00</td><td>170</td><td>169.1</td><td>—</td><td>—</td></tr><tr><td rowspan="2">Rastrigin</td><td>2</td><td>30.86</td><td>53</td><td>51.0</td><td>1.89 × 10<sup>-1</sup></td><td>3.97 × 10<sup>0</sup></td></tr><tr><td>10</td><td>—</td><td>—</td><td>—</td><td>1.04 × 10<sup>1</sup></td><td>1.12 × 10<sup>2</sup></td></tr></tbody></table>

configuration within each family as our expert-in-hindsight reference. Initializations are not randomized: each function is evaluated from two fixed, canonical starting points that probe different basins or conditioning, ensuring comparability across methods.

**Results with Full Controller:** For each benchmark function and dimension we report: (i) **success rate** within budget, i.e., probability of convergence before max iterations; (ii) **convergence iterations** $T_{conv}$ among successful runs (median and mean); (iii) **final objective** $f_{final}$ (mean and variance across all runs). The results are summarized in Tab. I. With the full controller active (history + stagnation-triggered LR suffix refresh), we observe the intended behavior: on smoother objectives (Beale, Himmelblau) success is essentially guaranteed and iteration counts are small; on Rosenbrock, the controller maintains solid success with competitive $T_{conv}$; on highly multi-modal landscapes (Rastrigin, Ackley) the strict success target under the current budget yields lower rates (and none for Ackley), consistent with their ruggedness.

**Results without Stagnation Module:** We turn off the stagnation module to test its effect. Across Ackley, Rastrigin, Himmelblau, and Beale, the single-run AGDA performs on par with the best static grid configuration: runs converge in a similar number of iterations to final objectives of the same order of magnitude. In other words, on these smooth, well-studied objectives, once a reasonable learning-rate schedule is in place, different optimizers behave comparably from canonical starts. This suggests the suite is not sufficiently challenging to stress-test the controller's adaptivity when the stagnation agent is disabled for fairness with the grid baseline. The only exception is Rosenbrock: in both $d=2$ and $d=10$ we consistently observe tangible gains from LLM-driven configuration. We therefore report full tables for Rosenbrock in Tab. II.

Originally, the distribution initialization agent had a single point initializer (one start chosen by an agent). This raises two issues: (i) “what is a good starting point?”, in the limit it is the minimizer itself; and (ii) in practice we observed that, even when the function name was withheld and only the formula was given, the LLM often “guessed” the global minimizer (either by legitimate reasoning or, more likely, contamination), effectively starting at the solution and short-circuiting the

<table><caption>TABLE II: Rosenbrock results for $d=2$ and $d=10$. For $d=10$ and Start 1 both methods reach the iteration cap $T=4000$, while the augmented GDA attains a much lower objective.</caption><thead><tr><th>Dim</th><th>Start</th><th>Method</th><th>f<sub>final</sub></th><th>T<sub>conv</sub></th></tr></thead><tbody><tr><td>2</td><td>Start 1</td><td>Grid (best)</td><td>1.18956 × 10<sup>-10</sup></td><td>2473</td></tr><tr><td>2</td><td>Start 1</td><td>AGDA</td><td>7.35571 × 10<sup>-11</sup></td><td>2002</td></tr><tr><td>2</td><td>Start 2</td><td>Grid (best)</td><td>6.62056 × 10<sup>-11</sup></td><td>3554</td></tr><tr><td>2</td><td>Start 2</td><td>AGDA</td><td>2.17487 × 10<sup>-11</sup></td><td>718</td></tr><tr><td>10</td><td>Start 1</td><td>Grid (best)</td><td>1.4778 × 10<sup>-3</sup></td><td>4000</td></tr><tr><td>10</td><td>Start 1</td><td>AGDA</td><td>3.90255 × 10<sup>-8</sup></td><td>4000</td></tr><tr><td>10</td><td>Start 2</td><td>Grid (best)</td><td>4.40339 × 10<sup>-11</sup></td><td>2748</td></tr><tr><td>10</td><td>Start 2</td><td>AGDA</td><td>2.56568 × 10<sup>-11</sup></td><td>949</td></tr></tbody></table>

<table><caption>TABLE III: DIP and SIREN results with the full controller. Lower is better for loss; higher is better for PSNR. Protocols are identical between methods within each task. In particular, it is seen, that the PSNR of images can be increased by 2dBs just by leaving the choice of the GD learning rate to an LLM instead of the best human choice.</caption><thead><tr><th>Task</th><th>Method</th><th>Loss</th><th>PSNR (dB)</th></tr></thead><tbody><tr><td rowspan="2">DIP</td><td>Baseline</td><td>1.04 × 10<sup>-4</sup> (var 1.798 × 10<sup>-9</sup>)</td><td>N/A</td></tr><tr><td>AGDA</td><td><b>9.10</b> × 10<sup>-5</sup> (var 1.027 × 10<sup>-10</sup>)</td><td>N/A</td></tr><tr><td rowspan="2">SIREN</td><td>Baseline</td><td>0.00024 ± 0.00003</td><td>37.72 ± 0.64</td></tr><tr><td>AGDA</td><td><b>0.00014</b> ± 0.00004</td><td><b>39.89</b> ± 1.71</td></tr></tbody></table>

descent. We therefore switched to a **distributional** initialization agent: the LLM reasons about the problem's geometry to propose families (box/Gaussian/mixture) with bounded parameters, and the run samples a concrete start, restoring a meaningful optimization trajectory while keeping guardrails.

## B. Applied DL Tasks

To reduce reliance on textbook knowledge and highlight optimizer decisions rather than memorization, we also evaluate two applied objectives, Deep Image Prior (DIP) [20] and SIREN [21], comparing LLM-selected hyperparameters with human-engineered choices under identical budgets. Task-specific metric, i.e., peak signal-to-noise ratio (PSNR) is reported for SIREN as well. The aim is to show that the controller's selections (variant, schedule, inits) can outperform strong human heuristics in realistic settings while remaining
auditable. To keep the comparison fair, the *baseline* and *AGDA* (our controller) use the same architectures, seeds, iteration budgets, and loss definitions; only the controller's decisions differ. Wall-clock time is comparable under equal iteration counts and therefore omitted from the tables below. DIP optimizes the weights of a randomly initialized convolutional generator so that its output matches a target/observation (super resolution, denoising, inpainting); the architecture acts as an image prior without any training data. SIREN fits a continuous signal with sinusoidal activations; for images, we regress RGB values over normalized pixel coordinates. For both tasks, proposed AGDA outperforms the baseline solution in different metrics, confirming the effectiveness of the LLM-in-loop controller.

## V. CONCLUSION

We studied whether an LLM can add value inside a running gradient method for smooth, unconstrained, single-objective optimization by making scoped interventions within a conventional gradient descent algorithm. The proposed Augmented-GDA controller separates decisions into a static channel with validated menus and a dynamic channel with free-form proposals checked by guardrails. A history-aware critic links runs under fixed budgets. Empirically, the controller matches strong static schedules on smooth benchmarks and yields clear gains when trajectories face regime changes, such as on Rosenbrock. It also improves performance on two deep learning tasks. Beyond average outcomes, the controller reduces manual tuning and narrows dispersion, consistent with a human-optimality view of choices within a fixed optimizer family. Future work includes mid-run variant switching, richer schedules, stronger stability guarantees, cost-aware policies, and extensions to stochastic and multi-objective settings.

## REFERENCES

[1] OpenAI, "Introducing GPT-5," *OpenAI Blog*, 2025.

[2] G. Comanici et al., "Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities," *arXiv preprint arXiv:2507.06261*, 2025.

[3] D. Guo, D. Yang, H. Zhang et al., "DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning," *Nature*, vol. 645, pp. 633-638, 2025.

[4] J. Zhang, W. Wang, S. Guo, L. Wang, F. Lin, C. Yang, and W. Yin, "Solving general natural-language-description optimization problems with large language models," *arXiv preprint arXiv:2407.07924*, 2024.

[5] R. Thind, Y. Sun, L. Liang, and H. Yang, "OptimAI: Optimization from Natural Language Using LLM-Powered AI Agents," *arXiv preprint arXiv:2504.16918*, 2025.

[6] C. Yang, X. Wang, Y. Lu, H. Liu, Q. V. Le, D. Zhou, and X. Chen, "Large language models as optimizers," in *The Twelfth International Conference on Learning Representations*, 2023.

[7] M. Yuksekgonul, F. Bianchi, J. Boen, S. Liu, Z. Huang, C. Guestrin, and J. Zou, "Textgrad: Automatic "differentiation" via text," *arXiv preprint arXiv:2406.07496*, 2024.

[8] S. Brahmachary, S. M. Joshi, A. Panda, K. Koneripalli, A. K. Sagotra, H. Patel, A. Sharma, A. D. Jagtap, and K. Kalyanaraman, "Large language model-based evolutionary optimizer: Reasoning with elitism," *Neurocomputing*, vol. 622, p. 129272, 2025.

[9] A. Novikov, N. Vü, M. Eisenberger, E. Dupont, P.-S. Huang, A. Z. Wagner, S. Shirobokov, B. Kozlovskii, F. J. Ruiz, A. Mehrabian et al., "AlphaEvolve: A coding agent for scientific and algorithmic discovery," *arXiv preprint arXiv:2506.13131*, 2025.

[10] C. C. Sartori and C. Blum, "Improving existing optimization algorithms with LLMs," *arXiv preprint arXiv:2502.08298*, 2025.

[11] Z. Guo, M. Liu, Z. Ji, J. Bai, Y. Guo, and W. Zuo, "LLM as a Complementary Optimizer to Gradient Descent: A Case Study in Prompt Tuning," *arXiv preprint arXiv:2405.19732*, 2024.

[12] T. Mongaillard, S. Lasaulce, O. Hicheur, C. Zhang, L. Bariah, V. S. Varma, H. Zou, Q. Zhao, and M. Debbah, "Large language models for power scheduling: A user-centric approach," in *2024 22nd International Symposium on Modeling and Optimization in Mobile, Ad Hoc, and Wireless Networks (WiOpt)*. IEEE, 2024, pp. 321-328.

[13] S. Liu, C. Gao, and Y. Li, "Large language model agent for hyper-parameter optimization," *arXiv preprint arXiv:2402.01881*, 2024.

[14] Y. Chen, X. Song, C. Lee, Z. Wang, R. Zhang, D. Dohan, K. Kawakami, G. Kochanski, A. Doucet, M. Ranzato et al., "Towards learning universal hyperparameter optimizers with transformers," *Advances in Neural Information Processing Systems*, vol. 35, pp. 32053-32068, 2022.

[15] N. Qian, "On the momentum term in gradient descent learning algorithms," *Neural Networks*, vol. 12, no. 1, pp. 145-151, 1999.

[16] B. T. Polyak, "Some methods of speeding up the convergence of iteration methods," *USSR Computational Mathematics and Mathematical Physics*, vol. 4, no. 5, pp. 1-17, 1964.

[17] Y. Nesterov, "A method for solving a convex programming problem with convergence rate $o(1/k^2)$," *Soviet Mathematics Doklady*, vol. 27, no. 2, pp. 372-376, 1983.

[18] D. P. Kingma and J. Ba, "Adam: A method for stochastic optimization," in *Proceedings of the 3rd International Conference on Learning Representations (ICLR)*, 2015.

[19] T. Dozat, "Incorporating nesterov momentum into adam," in *International Conference on Learning Representations (ICLR)*, Workshop Track, 2016.

[20] D. Ulyanov, A. Vedaldi, and V. Lempitsky, "Deep Image Prior," in *Proceedings of the IEEE conference on computer vision and pattern recognition*, 2018, pp. 9446-9454.

[21] V. Sitzmann, J. Martel, A. Bergman, D. Lindell, and G. Wetzstein, "Implicit neural representations with periodic activation functions," *Advances in neural information processing systems*, vol. 33, pp. 7462-7473, 2020.