# AMSTRAMGRAM: Adaptive Multi-cutoff Strategy
## Modification for ANaGRAM

Nilo Schwencke, Cyriaque Rousselot, Alena Shilova, Cyril Furtlehner

► To cite this version:

Nilo Schwencke, Cyriaque Rousselot, Alena Shilova, Cyril Furtlehner. AMSTRAMGRAM: Adaptive Multi-cutoff Strategy Modification for ANaGRAM. ENUMATH 2025 - MS31 - Geometric Optimization Methods for Scientific Machine Learning, Marius Zeinhofer; Johannes Müller, Sep 2025, Heidelberg, Germany. hal-05245453

HAL Id: hal-05245453

https://hal.science/hal-05245453v1

Submitted on 8 Sep 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
Joint work:

Nilo Schwencke, Cyriaque Rousselot, Alena Shilova, Cyril Furtlehner

2025-09-04

<figure><img src="image_2.png" /></figure>

université
PARIS-SACLAY

<figure><img src="image_3.png" /></figure>

Laboratoire des Sciences du Numérique (LISN) | Université Paris-Saclay | CNRS | Inria

Team: Apprentissage & Optimisation (A&O) | Inria TAU

Gif-Sur-Yvette, France

Presented by Cyriaque Rousselot

ENUMATH 2025 - MS31 Geometric Optimization Methods for Scientific Machine Learning
* **Follow-up work** on ANaGRAM algorithm (Schwencke and Furtlehner 2025)

* **Empirical analysis** with geometric insights into PINN training dynamics

* **This presentation:**

1. Refresh on PINNS for PDES

2. Natural Gradient and the ANaGRAM algorithm

3. Quantities of interest for ANaGRAM

4. Empirical training dynamics

5. Adaptive algorithm

6. Experimental results

**Goal:** Understand and improve natural gradient methods for PINNs through empirical analysis
**Setting:** Given a domain $\Omega \subset \mathbb{R}^n$ with boundary $\partial\Omega$

• Domain measure: $\mu$ (probability measure on $\Omega$)

• Boundary measure: $\sigma$ (probability measure on $\partial\Omega$)

• Function space: $\mathcal{H}$ (Hilbert space of functions $u : \Omega \to \mathbb{R}^m$)

**Differential operators:**

$$D : \mathcal{H} \to L^2(\Omega \to \mathbb{R}, \mu), \quad B : \mathcal{H} \to L^2(\partial\Omega \to \mathbb{R}, \sigma)$$

**PDE Problem:**

$$\begin{cases} D[u] = f & \text{in } \Omega \\ B[u] = g & \text{on } \partial\Omega \end{cases}$$

**Goal:** Find $u \in \mathcal{H}$ satisfying the PDE with given source $f$ and boundary data $g$.
**General PINN Loss** (Raissi et al. 2019):

$$ \ell_{D,B}(\theta) = \frac{1}{2S_D} \sum_{i=1}^{S_D} |D[u_{\theta}](x_i^D) - f(x_i^D)|^2 + \frac{1}{2S_B} \sum_{i=1}^{S_B} |B[u_{\theta}](x_i^B) - g(x_i^B)|^2 $$

where:

* $\theta \in \mathbb{R}^p$: neural network parameters

* $u_{\theta}$: parametric model

* $S_D, S_B$: number of domain/boundary samples

* $x_i^D, x_i^B$: sampled collocation points

**Gradient Descent:**

$$ \theta_{t+1} = \theta_t - \eta \nabla \ell(\theta_t) $$

Has a lot of failing modes for PINNS (Krishnapriyan et al. 2021)

**Mirror Descent:** Uses a Bregman divergence $D_{\varphi}$ with convex function $\varphi$:

$$ \theta_{t+1} = \arg \min_{\theta} \left\{ \langle \nabla \ell(\theta_t), \theta \rangle + \frac{1}{\eta} D_{\varphi}(\theta, \theta_t) \right\} $$

**Natural Gradient** (Amari 1998) as Mirror Descent:

With $\varphi(\theta) = \frac{1}{2}\langle\theta, G_{\theta}\theta\rangle$:

$$ \theta_{t+1} = \theta_t - \eta G_{\theta_t}^{-1} \nabla \ell(\theta_t) $$

$$ G_{(\theta_t)_{p,q}} = \langle \partial_p u_{\theta_t}, \partial_q u_{\theta_t} \rangle_{L^2(\Omega, \mu)} $$
**Simplified Setting:** For $f \in L^2(\Omega, \mu)$, we focus on:

$$ \ell(\boldsymbol{\theta}) = \frac{1}{2S} \sum_{i=1}^{S} |v_{\boldsymbol{\theta}}(x_i) - f(x_i)|^2 $$

with $v_{\boldsymbol{\theta}}: x \in \Omega \rightarrow D[u_{\boldsymbol{\theta}}](x)$

This discretizes the functional loss:

$$ \mathcal{L}(v) = \frac{1}{2} \|v - f\|_{L^2(\Omega, \mu)}^2 $$

where $v \in L^2(\Omega, \mu)$, $v: x \in \Omega \rightarrow D[u](x)$

**Fréchet Derivative** of $\mathcal{L}(v) = \frac{1}{2}\|v - f\|_{L^2(\Omega, \mu)}^2$:

$$ d\mathcal{L}|_v(h) = \langle v - f, h \rangle_{L^2(\Omega, \mu)} $$

**Functional Gradient:** $\nabla \mathcal{L}|_v = v - f$

**Idealized Steepest Descent:** In unconstrained $L^2(\Omega, \mu)$:

$$ v_{\text{new}} = v - \eta \nabla \mathcal{L}|_v = v - \eta(v - f) $$

**Reality:** Updates are constrained to the tangent space:

$$ \begin{aligned} T_{\boldsymbol{\theta}}\mathcal{M} &= \operatorname{Im}(dv_{\boldsymbol{\theta}}) \\ &= \operatorname{Span}(\partial_p v_{\boldsymbol{\theta}} : 1 \le p \le P) \subset \mathcal{H} \end{aligned} $$

of the parametric manifold $\mathcal{M} = \{v_{\boldsymbol{\theta}} : \boldsymbol{\theta} \in \mathbb{R}^P\}$

**Idealized Update rule:** Performing updates with

$$ \theta_{t+1} = \theta_t - \eta dv_{\theta_t}^{\dagger} \left( \Pi_{T_{\theta_t} \mathcal{M}}^{\perp} \left( \nabla \mathcal{L}_{\theta_t} \right) \right) $$
<figure><img src="image_4.png" /></figure>

**Ideal Update Rule (Verbockhaven et al. 2024):**

$$ \theta_{t+1} = \theta_t - \eta dv_{|\theta_t}^{\dagger} \left( \Pi_{T_{\theta_t M}}^{\perp} (\nabla \mathcal{L}_{\theta_t}) \right) $$

**Natural Gradient Update (Ollivier et al. 2017):**

$$ \theta_{t+1} = \theta_t - \eta G_{\theta_t}^{\dagger} \nabla \ell(\theta_t) $$

$$ G_{(\theta_t)_{p,q}} = \langle \partial_p u_{\theta_t}, \partial_q u_{\theta_t} \rangle_{L^2(\Omega, \mu)} $$

**Problem: $O(P^3)$ complexity**
**Solution:** Use empirical approximation via SVD

**Empirical Feature Matrix** $F \in \mathbb{R}^{S \times P}$:

$$F_{i,p} = \partial_p v_{\theta}(x_i)$$

**Empirical Functional Gradient** $\nabla \widehat{\mathcal{L}}_{\theta} \in \mathbb{R}^S$:

$$\nabla \widehat{\mathcal{L}}_{\theta,i} = v_{\theta}(x_i) - f(x_i)$$

**Efficient Update:**

$$\theta_{t+1} = \theta_t - \eta (F^{\dagger})^T \nabla \widehat{\mathcal{L}}_{\theta}$$

**SVD:**

$$F = U D V^T$$

* Computational cost: $O(\min(PS^2, P^2S))$

* Pseudo-inverse:

$$F^{\dagger} = V D^{\dagger} U^T$$
## Hard Cutoff Regularization:

Uses a binary threshold with cutoff parameter $\epsilon$:

$$D_i^\dagger = \begin{cases} D_i^{-1} & \text{if } D_i \ge \epsilon \\ 0 & \text{otherwise} \end{cases}$$

**Key Parameter:** The cutoff $\epsilon$ determines the effective rank of the approximation and **significantly impacts** training dynamics.

1: **function** ANAGRAM()

2: ▷ **Input:** $v_\theta$: neural network, $\theta_0$: initialization, $f$: target, $(x_i)$: batch, $\epsilon$: cutoff

3:

4: **while** not convergence **do**

5: $F \leftarrow (\partial_p v_{\theta(x_i)})_{i,p}$ via autodiff

6: $U, D, V^T \leftarrow \text{SVD}(F)$

7: $D \leftarrow \text{truncate at cutoff } \epsilon$

8: $\hat{\nabla} \leftarrow (v_\theta(x_i) - f(x_i))_i$

9: $d \leftarrow V D^\dagger U^T \hat{\nabla}$

10: $\eta^* \leftarrow \text{line search } \text{argmin}_\eta \{L(\theta - \eta d)\}$

11: $\theta \leftarrow \theta - \eta^* d$
**Goal:** Solve $D[u] = f$ by minimizing
$$ \|D[u] - f\|_{L^2(\Omega, \mu)}^2 $$

**Generalized Sobolev Norm:** For linear operator $D$:

$$ \|u\|_{D} = \sqrt{\|u\|_{L^2(\Omega, \mu)}^2 + \|D[u]\|_{L^2(\Omega, \mu)}^2} $$

**Fundamental Inequalities:**

$$ \|u\|_{L^2(\Omega, \mu)} \le \|u\|_{D} \quad \text{and} \quad \|D[u]\|_{L^2(\Omega, \mu)} \le \|u\|_{D} $$

**Regularization control:**

$$ \exists \alpha > 0, \forall u \in \mathcal{H}_D : \|u\|_D \le \alpha \|D[u]\|_{L^2(\Omega, \mu)} $$

If you can invert $D$, it is equivalent to:

$$ (\exists \alpha > 0, \forall f \in L^2(\Omega \to \mathbb{R}, \mu), \|D^{-1}[f]\|_D \le \alpha \|f\|_{L^2(\Omega \to \mathbb{R}, \mu)}) $$

**Issue:** This fails to be inverted when:

1. $D$ has a non-trivial null space: $\ker D \ne \{0\}$
**Idea:** Add $\alpha^2 I_d$ to Gram matrix $G_\theta$

**Regularized Model:** Replace $u : \mathbb{R}^P \to L^2(\Omega, \mu)$

with:

$$u^{\alpha} : \begin{cases} \mathbb{R}^P \to L^2(\Omega, \mu) \times \mathbb{R}^P \\ \boldsymbol{\theta} \mapsto (u_{\theta}, \alpha\theta) \end{cases}$$

**Regularized Target:** $(f, \alpha\theta)$ instead of $f$

## Key Properties:

1. Convergence: $u^{\alpha} \to u$ as $\alpha \to 0$ in operator norm

2. Injectivity: $\ker du^{\alpha} = \{0\}$ (removes null space issues)

3. Continuous Invertibility: $\alpha\|\boldsymbol{\theta}\| \le \|du_{\boldsymbol{\theta}}^{\alpha}\|$

## Price to Pay:

$$(f, 0) \notin \mathrm{Im}(du_{\boldsymbol{\theta}}^{\alpha}) \text{ even if } f \in \mathrm{Im}(du_{\boldsymbol{\theta}})$$

Ridge solutions never solve the original equation!
**SVD Approach:** $du_{\theta} = UDV^T$ with singular values $D_i$

**Cutoff Pseudoinverse:**

$$D_i^{\dagger} = \begin{cases} D_i^{-1} & \text{if } D_i \ge \alpha \\ 0 & \text{otherwise} \end{cases}$$

**Restricted Domain:** $\Theta_{\alpha} = \text{Span}(V_i : D_i \ge \alpha)$

**Cutoff Operator:** $du_{\theta}^{\alpha} = du_{\theta}|_{\Theta_{\alpha}}$

**Key Properties:**

1. Domain Restriction: $\Theta_{\alpha} \subset \Theta_{\beta}$ for $\alpha > \beta$

2. Exact on Subspace: $du_{\theta}^{\alpha} = du_{\theta}$ on $\Theta_{\alpha}$

3. Continuous Invertibility: $\alpha \|h\| \le \|du_{\theta(h)}\|$ for $h \in \Theta_{\alpha}$

**Advantage:** Cutoff solutions solve the original equation on the restricted subspace.
# Ridge vs Cutoff: Geometric Comparison

## Ridge Regression:

* Modifies the operator: $D \to D^\alpha$

* Adds regularization dimension

* Solutions never exact (even as $\alpha \to 0$)

* Always invertible but changes the problem

**Target:** $(f, \alpha u)$ instead of $f$

## Cutoff Regularization:

* Restricts the domain: $\mathcal{H}_D \to \mathcal{H}_D^\alpha$

* Keeps original operator structure

* Solutions are exact on subspace

* Invertible by domain restriction

**Target:** Still $f$, but on restricted space

**Key Insight:** Cutoff regularization preserves the original problem structure while ensuring well-posedness
**Given:** SVD decomposition $F = UDV^T$ with rank $r$

**1. Singular Values:** Diagonal entries $D_i$ for $i = 1, ..., r$

**2. Reconstruction Error:** For empirical tangent spaces $\mathcal{T}_{M,N} = \text{Span}(V_i : M \le i \le N)$:

$$ \text{RCE}_{N}^{S} = \frac{1}{\sqrt{S}} \left\| V \Pi_{N}^{0} V^{T} \widehat{\nabla \mathcal{L}}_{\theta} - \widehat{\nabla \mathcal{L}}_{\theta} \right\|_{\mathbb{R}^{S}} $$

where $\Pi_N^M$ projects onto components $N + 1, ..., M$:

$$ \Pi_N^M = \sum_{p=N+1}^{M} e^{(p)} e^{(p)T} $$

**Description of RCE:**

$$ \begin{aligned} \lim_{S \to \infty} \text{RCE}_N^S &= \left\| \nabla \mathcal{L}_{|u_\theta} - \Pi_{\mathcal{T}_N^0}^\perp \nabla \mathcal{L}_{|u_\theta} \right\|_{L^2(\Omega \to \mathbb{R}, \mu)} \\ &= \left\| \Pi_{(\mathcal{T}_N^0)^\perp}^\perp \nabla \mathcal{L}_{|u_\theta} \right\|_{L^2(\Omega \to \mathbb{R}, \mu)} \end{aligned} $$

where $\mathcal{T}_N^0 = \text{Span}(f_V^{(i)} : 1 \le i \le N)$

**Variation of RCE:**

$$ (\text{RCE}_M^S)^2 - (\text{RCE}_N^S)^2 = \frac{1}{S} \left\| \Pi_N^M V^T \widehat{\nabla \mathcal{L}}_\theta \right\|^2 $$
<figure><img src="image_5.png" /><figcaption>Figure 2: Median absolute $L^2$ errors and Test losses for the Heat equation.</figcaption></figure>
Iteration 0: Intersection before cutoff

* fixed $\epsilon = 1e-3$

* two layers model [2,32,32,1]

* no adaptive sampling

$$ (\mathrm{RCE}_{0}^{S})^{2} = \frac{1}{S} \|\nabla \mathcal{L}_{\theta}\|^{2} = \ell(\theta) $$

<figure><img src="image_6.png" /></figure>

<figure><img src="image_7.png" /></figure>
<figure><img src="image_8.png" /></figure>

<figure><img src="image_9.png" /></figure>
Iteration 90: Intersection passes cutoff

<figure><img src="image_10.png" /></figure>

<figure><img src="image_11.png" /></figure>
$$ (\mathrm{RCE}_M^S)^2 - (\mathrm{RCE}_N^S)^2 = \frac{1}{S} \| \Pi_N^M \mathbf{V}^T \widehat{\nabla \mathcal{L}_\theta} \|^2 $$

<figure><img src="image_12.png" /></figure>

<figure><img src="image_13.png" /></figure>
## Phase 1: Feature Development

* Reconstruction error and singular values both evolve

* Intersection point shifts rightward toward cutoff

## Phase 2: Flattening

* Once intersection passes cutoff: RCE flattens over $[N_{flat}, r_k]$

* Eventually $N_{flat} \to 0$ and RCE becomes constant

## Connection to Training Loss:

$$ (\mathrm{RCE}_{0}^{S})^{2} = \frac{1}{S} \| \widehat{\nabla \mathcal{L}_{\theta}} \|^{2} = \ell(\theta) $$

The final flattening directly causes the sharp training loss drop!

$$ (\mathrm{RCE}_{M}^{S})^{2} - (\mathrm{RCE}_{N}^{S})^{2} = \frac{1}{S} \| \Pi_{N}^{M} V^{T} \widehat{\nabla \mathcal{L}_{\theta}} \|^{2} $$
**Key Question:** What if we adjust cutoff to retain exactly $N_{flat}$ components?

<figure><img src="image_14.png" /><figcaption>Figure 12: Standard cutoff
$10^{-3}$</figcaption></figure>

<figure><img src="image_15.png" /><figcaption>Figure 13: Adaptive cutoff
(30 components)</figcaption></figure>

<figure><img src="image_16.png" /><figcaption>Figure 14: After one adaptive
step</figcaption></figure>

**Instant flattening:** Single step with adjusted cutoff completes flattening

**Insight:** Dynamic cutoff adjustment based on intersection dynamics accelerates convergence
**Intersection Rank:**

$$r_{\text{int}(t)} = \max\{j : \text{RCE}_{j}^{S(t)} \le \hat{\Delta}_{\theta_{t}, j}\}$$

**Precision Rank:**

$$r_{\varepsilon(t)} = \max\{j : \text{RCE}_{j}^{S(t)} \ge \varepsilon\}$$

**Adaptive Cutoff Strategy:**

$$\text{Cutoff}(t) = \begin{cases} r_{\text{int}(t)} & \text{if } \text{RCE}_{r_{\text{int}(t)}}^{S(t)} > \varepsilon \\ r_{\varepsilon(t)} & \text{if } \text{RCE}_{r_{\text{int}(t)}}^{S(t)} \le \varepsilon \end{cases}$$

1: **function** AMSTRAMGRAM()

2: ▷ **Input:** $u_{\theta}$: neural network, $\theta_{0}$: initialization, $f$: target, $(x_{i})$: batch, $\varepsilon$: target precision

3: **while** $r_{\varepsilon} > 0$ **do**

4: $F \leftarrow (\partial_{p} u_{\theta(x_{i})})_{i,p}$ via autodiff

5: $U, D, V^{T} \leftarrow \text{SVD}(F)$

6: $\hat{\nabla} \leftarrow (u_{\theta(x_{i})} - f(x_{i}))_{i}$

7: $r_{\text{int}} \leftarrow \text{intersection rank from RCE vs } \hat{\Delta}$

8: $r_{\varepsilon} \leftarrow \text{precision rank from RCE vs } \varepsilon$

9: **if** $\text{RCE}_{r_{\text{int}}}^{S} > \varepsilon$ **then**

10: cutoff $\leftarrow r_{\text{int}}$

11: **else**

12: cutoff $\leftarrow r_{\varepsilon}$

13: $D \leftarrow \text{truncate at adaptive cutoff}$

14: $d \leftarrow V D^{\dagger} U^{T} \hat{\nabla}$

15: $\eta \leftarrow \text{line search}$

16: $\theta \leftarrow \theta - \eta d$
Iteration 0 min; cutoff 6.881671565793e-05

<figure><img src="image_17.png" /></figure>

<figure><img src="image_18.png" /></figure>
Iteration 1 max; cutoff 3.000308302963762e-08

<figure><img src="image_19.png" /></figure>

<figure><img src="image_20.png" /></figure>
Iteration 1 min; cutoff 3.000308302963762e-08

<figure><img src="image_21.png" /></figure>

<figure><img src="image_22.png" /></figure>
Iteration 2 max; cutoff 8.10280342796075e-09

<figure><img src="image_23.png" /></figure>

<figure><img src="image_24.png" /></figure>
Iteration 7 min; cutoff 4.250319199042198e-09

<figure><img src="image_25.png" /></figure>

<figure><img src="image_26.png" /></figure>
Iteration 8 max; cutoff 4.250319199042198e-09

<figure><img src="image_27.png" /></figure>

<figure><img src="image_28.png" /></figure>
Iteration 11 min; cutoff 4.250319199042198e-09

<figure><img src="image_29.png" /></figure>

<figure><img src="image_30.png" /></figure>
Iteration 12 max; cutoff 4.250319199042198e-09

<figure><img src="image_31.png" /></figure>

<figure><img src="image_32.png" /></figure>
Iteration 14 min; cutoff 4.250319199042198e-09

<figure><img src="image_33.png" /></figure>

<figure><img src="image_34.png" /></figure>
Iteration 17 min; cutoff 4.250319199042198e-09

<figure><img src="image_35.png" /></figure>

<figure><img src="image_36.png" /></figure>
Iteration 20 min; cutoff 4.250319199042198e-09

<figure><img src="image_37.png" /></figure>

<figure><img src="image_38.png" /></figure>
Iteration 29 min; cutoff 4.250319199042198e-09

<figure><img src="image_39.png" /></figure>

<figure><img src="image_40.png" /></figure>
Iteration 32 min; cutoff 4.250319199042198e-09

<figure><img src="image_41.png" /></figure>

<figure><img src="image_42.png" /></figure>
Iteration 33 min; cutoff 4.250319199042198e-09

<figure><img src="image_43.png" /></figure>

<figure><img src="image_44.png" /></figure>
Iteration 34 min; cutoff 4.250319199042198e-09

<figure><img src="image_45.png" /></figure>

<figure><img src="image_46.png" /></figure>
Iteration 35 min; cutoff 4.250319199042198e-09

<figure><img src="image_47.png" /></figure>

<figure><img src="image_48.png" /></figure>
**Experimental Setup:** Standard PINN benchmarks with identical settings

<table><thead><tr><th>Problem</th><th>MSE Test (Ours)</th><th>MSE Test (ANaGRAM)</th><th>L<sub>2</sub> Error (Ours)</th><th>L<sub>2</sub> Error (ANaGRAM)</th></tr></thead><tbody><tr><td>Heat Equation</td><td>8.51e-22 ± 8.06e-23</td><td>8.56e-11 ± 7.05e-11</td><td>6.50e-14 ± 2.06e-14</td><td>1.28e-06 ± 1.75e-06</td></tr><tr><td>Laplace 2D</td><td>5.11e-18 ± 5.50e-18</td><td>4.27e-13 ± 4.66e-13</td><td>1.91e-11 ± 9.53e-12</td><td>3.49e-09 ± 3.58e-09</td></tr><tr><td>Laplace 5D</td><td>2.04e-08 ± 1.16e-08</td><td>6.37e-08 ± 7.01e-08</td><td>6.85e-05 ± 2.16e-05</td><td>4.00e-05 ± 2.93e-05</td></tr><tr><td>Allen-Cahn</td><td>3.76e-09 ± 3.03e-09</td><td>2.19e-04 ± 4.16e-04</td><td>1.56e-04 ± 2.01e-05</td><td>4.32e-03 ± 5.93e-03</td></tr></tbody></table>

**Baseline:** (Urbán et al. 2025) methods

<table><thead><tr><th>Problem</th><th>MSE (Ours)</th><th>L<sub>2</sub> Error (Ours)</th><th>L<sub>2</sub> Error (Baseline)</th></tr></thead><tbody><tr><td>1D Burgers</td><td>1.50e-06 ± 9.43e-07</td><td>1.50e-06 ± 9.43e-07</td><td>2.9e-06 ± 4e-07</td></tr><tr><td>Non-Linear Poisson</td><td>8.51e-24 ± 2.24e-24</td><td>6.81e-10 ± 1.41e-09</td><td>3e-09 ± 1e-09</td></tr></tbody></table>

**Advantages:**

* No need for specialized boundary handling
Example of training configuration with Overfitting on 1D Burgers

<figure><img src="image_49.png" /><figcaption>Burgers Equation Results - Seed 70 (Final Iteration: 929)</figcaption></figure>

<figure><img src="image_50.png" /></figure>
<figure><img src="image_51.png" /></figure>
<figure><img src="image_52.png" /></figure>
<figure><img src="image_53.png" /></figure>
<figure><img src="image_54.png" /></figure>
Key Insights

* RCE analysis reveals hidden Information left to be exploited

* Selecting proper adaptive regularization leads to performance improvements

* Overfitting is possible with PINNs

* Strategic sampling remains crucial for complex PDE problems
Amari S-I (1998) *Natural Gradient Works Efficiently in Learning. Neural computation* 10:251–276. https://doi.org/10.1162/089976698300017746

Krishnapriyan A, Gholami A, Zhe S, et al (2021) Characterizing Possible Failure Modes in Physics-Informed Neural Networks. In: Advances in Neural Information Processing Systems. Curran Associates, Inc., pp 26548–26560

Ollivier Y, Arnold L, Auger A, Hansen N (2017) *Information-Geometric Optimization Algorithms: A Unifying Picture via Invariance Principles*. Journal of Machine Learning Research 18:1–65

Raissi M, Perdikaris P, Karniadakis G (2019) *Physics-Informed Neural Networks: A Deep Learning Framework for Solving Forward and Inverse Problems Involving Nonlinear Partial Differential Equations*. Journal of Computational Physics 378:686–707. https://doi.org/10.1016/j.jcp.2018.10.045
Schwencke N, Furtlehner C (2025) ANaGRAM: A Natural Gradient Relative to Adapted Model for Efficient PINNs Learning. In: The Thirteenth International Conference on Learning Representations

Urbán JF, Stefanou P, Pons JA (2025) Unveiling the Optimization Process of Physics Informed Neural Networks: How Accurate and Competitive Can PINNs Be?. Journal of Computational Physics 523:113656. https://doi.org/10.1016/j.jcp.2024.113656

Verbockhaven M, Chevallier S, Charpiat G (2024) Growing Tiny Networks: Spotting Expressivity Bottlenecks and Fixing Them Optimally. http://arxiv.org/abs/2405.19816. Accessed 25 Jul 2024