Title: PINN Balls: Scaling Second-Order Methods for PINNs with Domain Decomposition and Adaptive Sampling
Venue: NeurIPS 2025
Accepted: yes
Keywords: Physics-Informed Neural Networks

Second-order optimization

Domain Decomposition

Adversarial Adaptive Sampling
Abstract: Recent advances in Scientific Machine Learning have shown that second-order methods can enhance the training of Physics-Informed Neural Networks (PINNs), making them a suitable alternative to traditional numerical methods for Partial Differential Equations (PDEs). However, second-order methods induce large memory requirements, making them scale poorly with the model size. In this paper, we define a local Mixture of Experts (MoE) combining the parameter-efficiency of ensemble models and sparse coding to enable the use of second-order training.  Our model -- PINN Balls -- also features a fully learnable domain decomposition structure, achieved through the use of Adversarial Adaptive Sampling (AAS), which adapts the DD to the PDE and its domain. PINN Balls achieves better accuracy than the state-of-the-art in scientific machine learning, while maintaining invaluable scalability properties and drawing from a sound theoretical background.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe submission suggests algorithmic improvements to physics-informed neural networks (PINNs): it proposes to combine domain decomposition, adaptive sampling, and second-order optimisation.\nSpecifically, the approach is to replace a single neural network with an ensemble, combining the outputs of each ensemble member using a locally-supported weighting function (akin to a partition of unity).\nThe collocation points for the PINN are sampled from the same weighing function, and the resulting model is trained using the Levenberg--Marquardt optimiser. \nThe submission also includes an analysis of the existence of solutions to the minimax problem underlying the adaptive sampling algorithm, which extends a previous result. Numerical results show that the resulting algorithm is competitive with prior approaches.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\nThe submission is a nice paper, and I enjoyed reading it. It does not just contain a well-explained combination of ideas; merging domain decomposition and adaptive sampling appears to be effective, and the resulting algorithm is sufficiently simple to motivate reimplementation in future PINN studies. Furthermore, the theory is novel, convincing, and a valuable addition to the literature on adaptive sampling for PINNs.\n\n## Weaknesses\nWhile I recommend accepting the paper, I am currently assigning a borderline score due to the weaknesses explained below. I am open to revising my evaluation if my concerns turn out to be easy to resolve.\n### Second-order optimisation\n\nThe second-order optimisation parts of the paper are not as convincing as the other components. The main optimisation contribution of the paper is replacing [33]'s stochastic gradient descent with Levenberg--Marquardt. However, the paper does not demonstrate that for the PINN-Balls, second-order methods outperform first-order methods. Instead, the motivation for choosing second-order methods is that they are promising for PINNs, as indicated by recent studies [4, 24]. \n- If using second-order optimisers for PINN-Balls is considered a contribution of the paper, there needs to be a corresponding benchmark or analysis. \n- If referencing [4, 24] is considered a sufficient argument for using second-order methods, then I think that using second-order methods for PINN-balls is not a contribution of this paper.\n\nSection 4.1 is dedicated to discussing the sparsity of the Gauss--Newton matrix $D L^\\top DL$ with the corresponding savings in memory and runtime. This perspective overlooks matrix-free linear solvers, such as conjugate gradients, which do not need sparsity as long as matrix-vector products are efficient. For $G=  DL^\\top DL$, these matrix-vector products can be implemented with automatic differentiation. I believe this omission is a weakness in Section 4.1; a discussion or benchmark of alternative linear algebra approaches would be appropriate.\n\n### Experimental clarity\n\nThe numerical experiments leave several open questions, and I believe this hinders the appreciation of the results to the extent they should be appreciated. For example, I would like the following information added to the context of Figures 1 and 2 and Table 2:\n- Figure 1:\n    - The sparsity of the $DL^\\top DL$ matrix should depend on the distribution of points, so it should vary quite a lot during optimisation. Why is this experiment, which benchmarks only a single Levenberg--Marquardt step, representative of the sparsity of the system matrix?\n    - All but the left-centre plot would be much easier to read on a logarithmic scale. As is, I find trends difficult to identify.\n    - The far-left figure suggests that the sparsity does not yield any runtime improvements. Is this because the problem is too small? What happens to larger matrices?\n    - All those numbers are on CPU, do I understand correctly? Do the authors have any intuition about what they look like on GPU? In my experience, CPU and GPU runtimes differ considerably when running linear algebra routines.\n    - Do I read the figure correctly that the more PINN balls, the more efficient the linear algebra? How many balls are \"too many\" in terms of approximation quality?\n- Figure 2:\n    - What does $L^2$ error mean here? Is this a comparison to a reference solution or the $L^2$ norm of the residual? If it is the former, where does the reference come from?\n    - What precision is used? L2 errors of $10^{-10}$ sound like double precision, but in my experience, this is difficult on GPUs.\n    - It appears that the more PINN balls are used, the lower the loss. What happens to more than 20 Pinn balls?\n- Table 2: The results seem to disagree with Figure 2 in the sense that in Figure 2, 20 balls achieve lower error than five balls, but in Table 2, the roles are reversed. Where does this discrepancy come from?\n\nI list some additional, more minor questions under \"Questions\" below. (For reference, I don't expect all to be answered in the rebuttal. It would be nice, but I understand that time and space for the rebuttal are limited.)\n\nDespite these issues, the paper's strengths outweigh the weaknesses, and I recommend acceptance. However, resolving the issues I discuss above (or explaining how I misunderstood something) would lead me to improve my score.\n\nQUESTIONS:\nHere is a collection of some more minor questions. They don't affect my score, but they might be helpful feedback for revising the paper. (I'd still be interested in their answers, though I understand that rebuttal time and space may be limited.)\n\n1. Line 187: \"Ensuring \\phi > 0 can be done by choosing a sufficiently large $s$ at initialisation\" -> is this always true?\n2. Equation 10: Does this loss assume the negative entropy regularisation? \n3. Equation 10 (again): How does the domain decomposition interact with the boundary?\n4. Equation 11: I thought Equation 10 (\"J\") was the preferred loss, not Equation 4 (\"L\"). What have I missed? Same in Line 228.\n5. Line 241: The exponential decay in learning rate feels quite aggressive. How important is that? And what kind of exponential decay is used?\n6. Line 245: What's the domain of the problem? The target function $f(x) = sin(4x)sin(y)$ has 4 \"bumps\" if x and y are between 0 and $\\pi$. Depending on the domain, should there be a \"correct ensemble size\" of four, or is the number of ensembles less straightforward to imagine?\n7. Line 247: What are $K$-means applied to?\n8. Line 266: Are those results in the paper? I can't find them, but I would like to see them.\n9. Line 268: To what extent is the condition number of $DL$ relevant for Figure 1? Doesn't the experiment measure a single step of Levenberg-Marquardt implemented with a direct solver? In my understanding, this should be independent of any condition numbers. Have I missed something?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces an ensemble model for PINNs, where each submodel learns a compactly supported Radial Basis Function (RBF). By ensuring the supports of RBFs cover the entire domain, the ensemble model autonomously learn domain decomposition. Employing the Levenberg-Marquardt update rule and Adversarial Adaptive Sampling (AAS) with a specified probability density function, it achieves notable performance. Furthermore, the paper includes theoretical extension of [33] to a more general class of regularizers.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1. The proposed method autonomously learns domain decomposition without requiring any a priori knowledge and eliminates the need to explicitly manage interfaces between subdomains.\n2. The model outperforms existing baselines across several benchmarks.\n3. The theoretical results on Adversarial Adaptive Sampling (AAS) are extended to more general settings, increasing their practical relevance.\n\nWeaknesses\n1. While accuracy improves with an increasing number of submodels, this comes at the cost of learning stability. Such instability may hinder deployment in real-world applications.\n2. Although the concept of learnable domain decomposition is appealing, the paper lacks experimental validation on complex domains where such decomposition would be most advantageous.\n3. Several important implementation details are provided only in the appendix and are not referenced in the main text. For example, the network architecture used in experiments is described in the appendix, but not mentioned or cited in the main body of the paper.\n\nQUESTIONS:\n1. If there is an empirical result on complex domain such as dolphin-shaped interface, it would strengthen the generalizability of the proposed method.\n2. The theorem said that probability density function $p_n$ converges to a uniform distribution. If this is observed in real experiments, could we expect the similar results by using uniform distribution rather than using AAS?\n3. Considering the ensemble nature of the proposed framework, could the submodels be added in an adaptive manner? For example, after training PINNBalls-10, would it be possible to introduce an additional set of 10 submodels whose RBF centers are initialized in regions exhibiting high residual error? If the support of these new submodels is restricted to those high-error regions, this strategy might enable further improvement in accuracy without degrading performance in areas that are already well-learned.\n\nLIMITATIONS:\nyes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nPINN Balls is a model that combine second-order training with domain decomposition on PINNs. An overlapping RBF-based domain decomposition (\"balls”) is used to create a sparse Jacobian. Sparse Levenberg–Marquardt updates are adopted for second-order optimization. Adversarial Adaptive Sampling (AAS) is used to learn the decomposition along with the PINN training. Experimental results are presented on 2D Helmholtz, Burgers, and Navier–Stokes equations.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n-- Scaling higher-order methods is an open problem. This paper considers this problem along with learning the adaptive domain decomposition.\n\n-- Results on Helmholtz especially is impressive. \n\n-- Extension of AAS theory, proving convergence to sampling under Dirichlet or entropic regularization.\n\nWeaknesses:\n\n-- Although sparse Jacobians allow for larger models, all benchmarks are on small 2D PDEs solvable by dense Levenberg–Marquardt. There is no demonstration of a problem that dense second-order methods OOM but PINN BALLS completes. \nMoreover, assembling the CSR matrix has non-negligible overheads at moderate to large scale and could outweigh the gains from sparsity. GPU sparse kernels are also not as efficient as dense kernels.\n\n-- Fluctuating losses with increasing balls point to unstable AAS maximization.\n\n-- Benchmarks are three textbook PDEs on 2D domains. No demonstration on multiphysics and heterogeneous boundary conditions where PINNs often struggle.\n\n-- Comparisons to XPINN/APINN use reported numbers rather than reproduced results under identical settings.\n\nThis paper presents good small-scale results, but falls short on demonstrating practical scalability which is the motivation for sparsifying second-order PINNs.\n\nQUESTIONS:\n1. Consider adding experiments on larger-scale PDEs, e.g., 3D or fine-resolution 2D, where dense LM cannot run due to GPU memory limits, but sparse LM succeeds and outperforms other models. \n\n2. How sensitive is the method to the initialization of RBF centers and radii?\n\n3. PINN Balls relies on AAS to train the DD. What are the benefits of AAS to residual-based methods? An ablation study of PINN Balls with residual-based sampling and AAS (with Dirichlet/entropic regularization) reporting final errors and convergence speed would be a good addition in the appendix.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces PINN BALLS, a sparse Mixture-of-Experts architecture for Physics-Informed Neural Networks (PINNs).\nKey ideas are:\n- trainable radial-basis “balls” that decompose the domain and create a partition of unity.\n- second-order Levenberg-Marquardt updates applied locally so Jacobians stay sparse and fit GPU memory.\n- Adversarial Adaptive Sampling (AAS) used as the gating distribution, with new convergence theorems that replace the original Lipschitz constraint by Dirichlet, and entropy-based regularizers.\n- extensive empirical study on Helmholtz, Burgers, and 2-D Navier–Stokes benchmarks plus ablations on memory/runtime.\nA comparison table highlights how the approach differs from XPINN, FBPINN and APINN (sparsity, no interface loss, adaptive sampling, second-order training)\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- While mixture-of-experts in PINNs is not a new idea, and prior works already explored sparse gating and soft domain decomposition, the idea proposed here is to combine the benefits of DD, second-order training methods and Adversarial Adaptive Sampling (AAS)\n- A radial basis “ball” formulation is derived for soft domain decomposition that leads to both partition-of-unity gating and Jacobian sparsity. \n- The paper provides theoretical convergence results for Adversarial Adaptive Sampling (AAS), relaxing prior assumptions and generalizing earlier proofs using Dirichlet and entropy regularizers.\n- In addition to empirical evaluation, Authors provide ablation studies across ball count, sampling, and sparsity.\n- Transparent discussion of limitations such as sparse Jacobian overhead and current AD support issues, which raises my respect and trust to their science work quality.\n\nWeaknesses:\n- No ablation study on the effect of different components the authors proposed in their study. For example, comparing the following variations can shed light on which components are truly necessary and and where most of the gains come from: Full PINN BALLS, vs without AAS, vs Without LM update step, etc.\n- Lack of comparison with non-PINN state-of-the-art models: Since the ultimate goal is to train an accurate surrogate for PDE solutions, it is important to evaluate how PINN BALLS performs against other state-of-the-art models in terms of predictive accuracy, regardless of the underlying optimization algorithm or architecture type. I suggest that the authors include a comparison with data-driven models (such as FNO or DeepONet) as part of their experimental evaluation.\n- No public code at the time of submission exists and their answer to NeurIPS checklist question must be \"No\". This can limit the reproducibility. Even an anonymized repo for the results reported in this paper would help.\n- The notation can be dense and intimidating for readers unfamiliar with second-order methods and sparse linear algebra. A high-level algorithm box or diagram of the training pipeline would improve clarity.\n\nQUESTIONS:\n- Have you attempted to scale PINN BALLS to 3D problems or complex multi-physics PDEs? Would sparse Jacobian construction and solver performance remain tractable in such high-dimensional or coupled systems?\n- Could you elaborate on why you chose quartic radial basis functions for the gating balls? Did you explore other compactly supported kernels or make the RBF shapes learnable? How sensitive is the model to this design choice?\n- How sensitive is model performance to the number or initial placement of balls? \n- Would your method require significant architectural or training changes to handle time-dependent PDEs.\n- Can you provide more quantitative analysis of the time and memory trade-offs of sparse Jacobian construction and sparse LM solving compared to dense LM or first-order optimizers?\n\nLIMITATIONS:\n- Reproducibility is limited by the lack of a code release at submission time, though the authors state their intent to share it. Open access would allow others to test and verify scalability claims.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}