Message-Passing GNNs Fail to Approximate Sparse Triangular Factorizations
Authors: Vladislav Trifonov
Ekaterina Muravleva
Ivan Oseledets
Venue: Accepted by TMLR ()

Abstract: Graph Neural Networks (GNNs) have been proposed as a tool for learning sparse matrix preconditioners, which are key components in accelerating linear solvers. We present theoretical and empirical evidence that message-passing GNNs are fundamentally incapable of approximating sparse triangular factorizations for classes of matrices for which high-quality preconditioners exist but require non-local dependencies. To illustrate this, we construct a set of baselines using both synthetic matrices and real-world examples from the SuiteSparse collection. Across a range of GNN architectures, including Graph Attention Networks and Graph Transformers, we observe low cosine similarity ($\leq0.7$ in key cases) between predicted and reference factors. Our theoretical and empirical results suggest that architectural innovations beyond message-passing are necessary for applying GNNs to scientific computing tasks such as matrix factorization. Moreover, experiments demonstrate that overcoming non-locality alone is insufficient. Tailored architectures are necessary to capture the required dependencies since even a completely non-local Global Graph Transformer fails to match the proposed baselines.

Published in Transactions on Machine Learning Research (February/202)
Message-Passing GNNs Fail to Approximate Sparse Triangular Factorizations
Vladislav Trifonov
wlndrifono@gmaailccom com
ATC, Skoltech
A148 Center, Sberbank of Russio
Ekaterina Muravleva
wwwrroleea@skoolectth
AI48 Center, Sberbank of Russia
AIC, Skoltech
Ivan Oseledets
oseledets@airi..n
ATRI
AIC, Skoltech
Reviewed on OpenReview:
https: operreveee.....e....m net/ forum? id= VIr9S=D3C9
Abstract
Graph Neural Networks (GNNs) have been proposed as tool for learning sparse matrix preconditioners, which are key components in accelerating linear solvers. We present theo- retical and empirical evidence that messago-passing GNNs are fundamentally incapable of approximating sparse triangular factorizations for classes of matrices for which high-quality preconditioners exist but require non-local dependencies. To illustrate this, we construct set of baselines using both synthctic matrices and real-world examples from the SuiteSparsc collection. Across a range of GNN architectures, including Graph Allention Networks and Graph Transformers, we observe low cosine similarity (S 0.7 in key cases) between predicted and reference factors. Our theoretical and empirical results suggest that architectural in- necessary for applying GNNs to scientific computing novations beyond message-passing Moreover, experiments demonstrate that overcoming tasks such as matrix factorization. non-locality alone is insufficient. Tailored architectures are necessary to capture the re- quired dependencies since even a completely non-local Global Graph Transformer fails to match the proposed baselines.
1 Introduction
Preconditioning sparse symmetric positive definite matrices is fundamental problem in numerical linear algebra (Benzi 2002). The goal is to find a precondtioner matrix such that X-1 A is well-conditioned. This results in faster convergence of iterative methods when solving linear systems (Saad 2003). A well- established choice for symmetric positive definite matrices to use an incomplete Cholesky factorization the direct preconditioner X LLT A. Recently, there has been significant interest in using graph neural networks (GNNs) to predict such preconditioners (Li et al. 2024 Trifonov et al. 2024 Häusner et al. 2023) 2024). The key idea is to represent the sparse matrix A as a graph where edges correspond to the non-zero entries and to use GNN architectures to predict the entries of the preconditioner X, minimizing a certain functional.
Although GNNs show promisc, we demonstrate their fundamental limitation in tasks related solving lincar systems: their locality prevents them from learning non-local preconditioners. Specifically, we demonstrate that there classes of matrices for which optimal sparse preconditioners exist, yet they exhibit. non-local

---

Published in Transactions on Machine Learning Research (obbuaryy/220
dependencies. Starting with simple matrices, such tridiagonal matrices arising from discretization of partial differential equations (PDEs), we demonstrate that updating a single entry in A can significantly affect all entries in L.
To promote further development in the field, we introduce a new benchmark dataset of matrices for which optimal sparse preconditioners are known to exist but require non-local computations. We construct this dataset using both syuthetic examples and real-world matrices from the SuiteSparse collection (Davis 2024). For the synthetic benchmarks, carefully design tridiagonal matrices for which the Cholesky factors depend non-locally the matrix elements by leveraging properties of rank-1 semiseparable matrices. For real- world problems, we explicitly compute so-called K-optimal preconditioners based on the inverse matrix with sparsity patterns matching the lower triangular part of the original matrices.
Although message- pariannggagsingggggg GNNs can naturally be applied to sparse linear algebra problems, their inherent locality poses fundamental limitations learning non-local lincar algebra operations such as proconditioner construction. Our contribution is as follows:
We demonstrate the fundamental limitations of prssage-passing GNNs in approximating generally non-local preconditioner matrices for sparse linear systems.
We construct benchmark dataset to validate novel architectures for GNN-based preconditioner construction routines and to validate novel non-local GNN architoctures. The datasct consists of both synthetic and real-world matrices. The real-world matrix dalaset is based on the solution to an optimization problem, resulting in optimal sparse preconditioners.
We demonstrate that specific architectures are required for the task of constructing GNN-based preconditioners, since even completely non-local architectures, such as graph transformers, fail to approximate the proposed benchmark, indicating the issue is architectural mismatch, not solely insufficient receptive field.
2 Problem Formulation
Let A be a sparse symmetric positive definite matrix. The goal is to find sparse lower triangular matrix L such that LL approximates A well. In other words, the condition number of L-T AL- is small. Moreover, a good preconditioner should produce a tightly clustered spectrum of I. AL-1. The key idea in approx- imating L with GNNs is representing the sparse matrix as a graph where the edges correspond to the non-zero entries. Nodes can be represented as variables {Li el al. (022),, diagonal entries of A (Li et al., 2024) or omitted (Trifonov et al.) (2044). A GNN then processes this graph predict the non-zero entries of L, preserving the sparsity pattern of A.
Multiple rounds of pressage-asssing aggregate information from neighboring nodes and edges. This architec- ture is local, meaning that if we modify a single entry of A, the change will propagate only to neighboring nodes and edges. The size of this neighborhood is limited by the GNN's receptive field, which is proportional to the network's depth, i.e. the number of message pasogggg layers. It is worth noting that this task is a regression on edges, which appear less frequently than typical node-level or graph-level tasks, for which most classical GNNs are formulated.
In principle, Cholesky factorizations admit formulations entirely in terms of message passing. Therefore they can be expressed as message-pase passing algorithms within modern GNN frameworks, analogous to the examples described in Moore et al. (2025). Such a message-passing realization would require network depth scaling with the problem size making the architecture impractically deep for realistic systems. Since our work focuses on learning approximate factors with GNNs without explicitly encoding the elimination procedure, we do not exploit this direct algorithmic rcalization.
Related work The first attempts at learning sparse factorized preconditioners used convolutional neural networks (CNNs) (Sappl et al.) 2019: Ackmann et al.] (2020) Cali et al., (023).. However, CNNs are not feasible for large linear systems and can only be applied efficiently with sparse convolutions. heless
2

---

Published in Transactions on Machine Learning Research (abruary/2020
sparse convolutions do not utilize the underlying graph structure of sparse matrices. Therefore, GNNs were naturally applied to better address the structure of sparse matrricss as has already been considered in several works {Li et al. 2023 Trifonov et al.) 2024 Häusner et al. 2023 Li et al. 2024 Booth et al, 2024). GNNs can also be introduced into iterative solvers from other perspectives, not just for learning sparse incomplete factorizations. For example, the authors Chen (2024) suggest using GNN as a nonlinear
preconditioner, and in works Taghibakhshi al. (2022] 2023) authors use GNNs for a domain decomposition method. Although; our work focuses on the sparse factorized preconditioners, the illustrated limitation is also relevant to the GNN-based approaches discussed above. Another line of work covers iterative solvers rather than preconditioners; see, for example, Luo et al. (2024).
The main limitations of GNNs, such as over-smoothing (Rusch et al. 2023), over-squashing (Topping et al. 2021) and expressive power bound (Xu et al. 2018) have already been highlighted in modern research. Mitigating these problems is indeed an important task, but in long-range dependent graphs, the receptive field must still be really large. Furthermore, deeper stacking of graph convolution layers may indeod work well for well-known GNNs benchmarks since they are lypically sparse and connected, and one can expect small graph diameter. However, for the problems considered in this paper, the diameter of the graphs be up to ONN), making deeper stacking of layers infeasible.
2.1 Limitations of GNN-based Preconditioners
Consider the mapping f: A + L, where A is given symmetric positive definite matrix, and L is sparse lower triangular matrix with given sparsity pattern. In this section we will provide a example of sparse matrices A, when:
A is a sparse matrix and there exists an ideal factorization A = L.I.T, where L is a sparse matrix.
The mapping of A not local: a change in one entry of A can significantly affect all entries of I., beyond the GNN's receptive field.
The simplest class of such matrices are positive definite tridiagonal matrices. These matrices appear from the standard discretization of one-dimensional PDEs. Such matrices known to have bidiagonal Cholesky factorization
4=LT
(1)
where L bidiagonal matrix, and that is what we are looking for: the ideal sparse factorization sparse matrix. Our goal is show that the mapping equation [] not local. Lets consider first the case of discretization of the Poisson equation on a unit interval with Dirichlet boundary conditions. The matrix A is given by the second order finite difference approximation,
2 -1
0
0
1
-1 ... 0
A =
0
-1
2
:
(2)
-1
2
The Cholesky factor L is bidiagonal in this case.
The question is, how do the elements of L change if we change single entry of A in position (1, 1)? The change in the diagonal is shown in Figure | on the left, where one can observe the decay. This decay is algebraic and aligned with the properties of the Green's functions of the PDEs. For this matrix, the dependence is local. However, we can construct pathological examples where the dependence is not local single change in A will change almost all elements of L.
Theorem 2.1. Let be tridiagonal symmetric positive definite n matriz. Then it can be factorized as
A L.I.
3

---

Published in Transactions on Machine Learning Research (February/22) 2026)
1.0
Offerrce 3.000r aterree
0:38
Offfrece m-:aay 2o4
&
Figure Difference of the diagonal elements between the Cholesky factor L and perturbed factor L' in a single entry An of the tridiagonal matrix. (Left) ID Laplacian matrix. (Right) Counterexample.
where L is bidiagonal lower triangular matria, and then mapping A + L is not local, which meons that there erist matriz A and A' such that A' has only one non-zero element, where the difference L L- L' has dense support: many entries change significantly, even though only a single entry was perturbed.
Proof. Consider the matrix A given by A LLT where L is a bidiagonal matrix with Li = -. 1,..., n and Li = 1,1 2. Then is a symmetric positive definite tridiagonal matrix with elements An = Ar, -1 ,, = A1, = t, = I,...,
Now, consider the matrix A' = A + eje,, where E1 is the first column of the identity matrix. Let A' L'L'T be its Cholesky factorization. The matrix L' is bidiagonal. The element 1/1 is equal to v2, and for each i -,..., have the well-known formulas
Li,i-1
1/,511
14
1-1,1-1
and 14, V Acs (I), Let d; (14,)² then di = 2, Φ.. 11- From this recurrence relation it is easy see that de converges to
0o.
Consequently, for general matrix A, finite-depth message passing architecture robustly recover L under such perturbations, empirical failures expected. The difference between diagonal clements of L and L' is shown on Figure on the right.
3 Constructive Approach
We will use the class of tridiagonal matrices as the basis for our synthetic benchmarks for learning triangular preconditioners. What approaches can we take for other, more general sparse positive definite matrices? In this section, we present a constructive approach to building high-quality preconditioners that cannot represented by GNNs, as demonstrated our numerical experiments Section D
To accomplish this task, draw attention to the concept of K-condition number, introduced in Kaporin 1994h. By minimizing this condition number, A ~ LLT for many matrices, where the sparsity paltern of L, matches that of the lower triangular part of A. The K-condition number of a symmetric positive definite matrix A is defined as
constructively build sparse preconditioners of the form
1 (T((A)
K(A) (det(A))*
The interpretation of equation 3 is that it represents the arithmetic mean of the eigenvalues divided by their geometric mean. For matrices with positive eigenvalues, this ratio is always greater than 1 and equals only

---

Published in Transactions on Machine Learning Research (obbraary/2000
when the matrix is a multiple of the identity matrix. Given preconditioned matrix M, evaluate its quality using K (M). This metric can be used to construct incomplete factorized inverse precondilioners (K--
porin 1994 Chen et al. 2024 A-1 R LL where L is sparse, However, we focus on constructing incomplete
factorized preconditioners A LL with a sparse Therefore, we propose minimizing the functional:
K(L A(L"A-1)) A- min,
(4)
where L is sparse lower triangular matrix with predetermined sparsity pattern. Utilizing the inverse matrix in preconditioner optimization is promising strategy that has been explored in other works [Li Trifonov et al. 2024) through the functional:
et al. 2023
ILL A - T1] - min.
(5)
The distinctive advantage of the functional equation is that the minimization problem can be solved explicitly using lincar algebra techniques. This allows construct pairs of matrices (A₃, L2) for small and medium-sized problems, where L acts effective preconditioner. These pairs are valuable benchmarks for evaluating and comparing the properties of preconditioner learning algorithms against matrices that minimize equation
4 K-optimal Preconditioner Based on Inverse Matrix for Sparse Matrices
In this section, we analyze the preconditioner quality functional:
min,
(6)
where L sparse lower triangular matrix with predetermined sparsity pattern. We will derive an explicit solution to this optimization problem.
4.1 Solution of the Optimization Problem
Let us demonstrate how minimize the K-condition number in the general case, then apply the results to obtain explicit. formulas for K-optimal preconditioners. Consider the optimization problem:
K(G BG) - min, G
(7)
where G belongs to some linear subspace of triangular matrices:
g = vec = vzz
where V is an n² x matrix, with m being the subspace dimension. For sparse matrices, m equals the number of non-zero elements in G.
Instead of directly minimizing functional equation we minimize its logarithm:
Φ (G) = log KGG BG) = log Th(G BG) log det(G)2 log det B).
The third term is independent of and can be omitted. For the first term:
Tr(GT BG) = (BG,G), G)
where (5)) denotes the Frobenius inner product. Therefore:
Tr(GT BG) = (Bg.g).
with = 7 B, leading to:
Tr(GT BG) (Bg. .) (v) (B,,,) z) (Cz, z);
5

---

Published in Transactions on Machine Learning Research (eebruar/2024
where C BV. To express the elements of matrix C, use three indices for V elements, Mar
Cw Σ Bis E Vin = (B, V.V.)), 1,=1
where V1, = 1,..., are n x matrices obtained from corresponding rows of V. Our task reduces to minimizing with respect to 2. Since is symmetric, C is also symmetric, yielding the gradient:
2(C2); 2 Tr(G-) (Vo(2)), = (C2,2) 2) n
derived using the formula for the logarithm matrix determinant derivative.
Special case: = is a sparse matrix If G L, where L sparse lower triangular matrix, then matrix C is a bloc-ddiagonal matrix of the form
C1
C2
==
Cn
where blocks C, are given by formulas
(C.)a = B (0 (9
(6) where s are indices of non-zero elements in the i-th column of matrix L, and matrix G- has non- zero diagonal elements only for j corresponding to diagonal elements of matrix L. For these elements Tr(G-10,) म.. = 1,...,
The problem reduces to n independent optimization problems on values non-zero elements in the i-th column of matrix L, i = I...., n. Let us consider each subproblem separately. The optimality condition for the i-th subproblem has the form
C 7el
(Ca.2) is number, is constant that does not depend on 2, e1 is the first column of the where 70 St
identity matrix corresponding size. Hence
2, 70%
0/ = == en.
and using the fact that K does not depend on multiplication by a number we get an equation for the first component of vector (which is the diagonal element of matrix L)
(vx)
(z))
(2:)1
from which
(x)) ((:)
The vector 2; contains the non-zero elements of i-th column of Therefore, the algorithm for finding the
sparse lower triangular matrix is summarized in Algorithm 1 We refer to preconditioners constructed
using this approach as K-oplimal preconditioners.

---

Published in Transactions on Machine Learning Research (February/002)
Algorithm Construction of K-optimal preconditioner
Require: Symmetric positive definite matrix A, sparsity pattern for L Ensure: Lower triangular matrix L
1: Compute B = A-1
2: for i- to n do
3: Find indices of non-zero elements in column of L
4: Extract submatrix B, using rows and columns from 8;
Compute 1/ B, le
Dr e1 is first unit vector Store as i-th column of L
Set L.., = ((v))) -1/2)
end for
5 Benchmark Construction
To robustly evaluate the fundamental limitations of messago-passing GNNs for sparse factorization, we constructed a two-part benchnark. One part is based on synthetic matrices engineered to have non-local dependencies. The other part is drawn from real-world sparse matrix problems. This dual approach ensures our benchmark covers both theoretically motivated and practically challenging instances.
5.1 Synthetic Benchmark
Our synthetic benchmark targets the core locality limitation of GNNs in a controlled setting. We generate tridiagonal matrices whose Cholesky factors L are sparse but whose construction requires non-local infor- mation. Through empirical investigation, we found that simply fixing the diagonal elements of T. and sampling the off-diagonal elements from a normal distribution does not produce the desired non-local be- havior, as the resulting matrices A = LLT tend to exhibit primarily local dependencies. Our key insight is that non-locality emerges when the inverse matrix L is dense.
We exploit the property that the inverse of bidiagonal matrix L is rank-1 semiscparable, where the elements are given by the formula L uey for S J, representing part of a rank-1 matrix. This relationship is bidirectional, meaning that given vectors u and v, compute L as its inverse. Our benchmark generation process exploits this property by randomly sampling appropriate vectors and u to create matrices with guaranteed non-local dependencies. Poor performance on this benchmark would raise serious concerns about the fundamental suitability of current GNN architectures for matrix factorization tasks.
construct L with this structure and then
5.2 Matrices from the SuiteSparse Collection
To complement synthetic cases with practical relevance, curated collection of 150 symmetric positive definite matrices from the SuiteSparse matrix collection (Davis) 2024 for which dense inverse computation was feasible. For each, we computed K-optimal preconditioners by explicitly solving the optimization prob- lem equation 7 under a fixed sparsity pattern matching the lower triangle of the original matrix, similar to incomplete Cholesky with zero fill-in (IC(O)) preconditioners (Saad) 2003). Our experimental results showed that K-optimal preconditioners generally perforrmde traditional TC(O) preconditioners. In many cases, IC(0) either did not exist or required excessive itcrations (> 10000) for convergence. However, we observed that for small subset of matrices, IC(0) achieved better convergence rates.
The final benchmark consists of (A,, L:) pairs, where each A, comes from SuiteSparse and L, represents factor of superior performing preconditioner, either IC(0) or K-optimal preconditioner. Note that constructing the K-optimal preconditioner requires access to the A, so it is used here as an oflline benchmark rather than as a practical method. Matrices where neither approach succeeded were excluded to ensure meaningful evaluation. The relative performance distribution between the K-optimal and IC(0) preconditioners visualized Figure 2 The K-optimal preconditioners are generally superior, though there are also cases in which IC(O) remains competitive in which both methods fail to converge.

---

Published in Transactions on Machine Learning Research (obruary/2020
In experiments, we consider only matrices for which high-quality sparse preconditioner is known to iee. cases where the K-optimal method succeeds- so that the task remains feasible in principle and negative GNN results are meaningful.
K- optimal less iteration
24.4%
K-optimal only
58.1%
9.3%
Both Failed
8.1%
IC(0) less iterations
Figure 2: The performance of K-optimal preconditioner and IC(0) preconditioner during solution of SuiteS- parse subset. K-optimal only: IC(0) preconditioner failed. K-optimal less iterations: both precondi- tioners were successful, with superior K-optimal performance. IC(O) less iterations: both preconditioners were successful, with superior TC(00) performance. Both failed: both preconditioners failed.
6 Experiments
6.1 GNN Architectures
Most classical GNNs either do not consider edges (e.g., GraphSAGE (Hamilton et al. 2017)) or consider them as scalar weighted adjacency matrix (e-g., Graph attention network (Veličković et al. 2017 To
enable edge updates during message-passing we use Graph Network (Battaglia et al. 2018 block a message-passing layer.
To validate GNNs on the proposed benchmarks, we use the popular Prnoodee-Proesso--Deaaaa configura- tion. The Encoder consists of two separate MLPs, one for nodes and one for edges. The processor consists of multiple blocks of Graph Networks. First, the Graph Network updates the edge represenlations with Edge Model. Then, the nodes are update by the Node Model with a message sasisggg-assinggggg mechanism. In our work, we do not consider models that achieve larger receptive field through graph coarsening or updates graph-level information. Hence, the Global Model in the Graph Network is omitted. The decoder single ML.P that decodes the hidden representations of the edges into single value per edge in resulting factors L.
Graph coarsening strategies in GNN architectures can be naturally rolated to classical domain decomposition and multigrid methods. Moreover, these methods are well represented in the application of deep learning to numerical methods, e.g., works Taghibakhshi et al. (2022 2023); Katrutsa et al. (2020); Luz et al. (2020). We intentionally omit coarsening techniques since their use would significantly shift the process training incomplete factors toward other methods from classical preconditioner approaches (e.g., multigrid). Instead, explicitly incorporate global information propagation within our setting, we employ the popular virtual node mechanism (Southern al. 2024h. This involves adding extra node to the graph that is connected to every other node.
As a neural network baseline that does not propagate information between nodes, we use simple two- layer MLP as the Node Model in Graph Network (MLPNodeModel), The following messago-passing GNNs are used as the Node Model: (i) graph attention network (GAT) (Brody et al. 2021), (ii) generalized
aggregation network (GEN) (Li et al. 2020) and (iii) message-passing [MessagePassingML]] (Gimer al,
2017) with two MLPs fe, and fez:
Σ Sor (hi, (haree))) JEN((1)
h; for hhi N
8

---

Published in Transactions on Machine Learning Research (ebruary/202)
Finally, tested two graph transformers Node Models: (i) graph transformer operator (GraphTrans- former) from (Shi et al. (2020) and (ii) global graph transformer operator (ClobalGraph Transformer) from [u et al. (2023). The Broballr Network (GN) that preserves local graph structure:
integrates an all-pair node Attention Network (AN) with Graph
Zont = (1 -1 AN(Z AN + a GN A),
where ( denotes the input node features and A the graph adjacency matrix. Following the original paper's design (Wu al., 2023), we employ a simple graph convolution layer as the GN component (graph attention network v2 in ours experiments) and fix the mixing coefficient at 0.5 in all experiments.
Note that the TranaGGrrhTTTaaooooaia achieves global all-pair attention with quadratic complexity in the number of nodes. This can be prohibitively expensive since medium-sized linear problems are of the size N = 10
In our experiments, we set the encoders for nodes and edges to two layer MLPs with 16 hidden and output features. The Node Model is single-layer model from following list: MLPNodeModel, GAT, GEN, Mes- sagelasssnngMLL GraphTransforme 16 hidden fcatures. The Node Model and Edge Model form the Graph Network, which is used to combine multiple pessagg-passing layers in the Processor. The Edge decoder is two-layer MLP with 16 hidden features and a single output feature. We use the all-ones vector [1,...,1]] 1] E RR- as the input node features,
ToobalGraphhTaassfoooo
The Edge Model is two-layer MLP with
which was shown to be effective in [rifonnv et al. (2024).
We trained the model for 300 epochs, reaching the convergence in each experiment. We used an initial learning rate of 10-3 and decreased it by factor of 0.6 every 50 epochs. We also used early stopping with 50 epoch patience. For the synthetic dataset, generated 1000 training samples and 200 test samples. The batch size was 16 and 8 for the synthetic and K-optimal datasets respectively.
We use the PyTorc Geometric Foey & Lenssen 2019 framework for training GNNs with main layers implementation. For the CaabbllGrrarrrrsoorreeeet we use the source implementation from Wu et al. (2023 We used single Nvidia A40 48Gb GPU for training.
To examine the importance of receptive field, experiment with various numbers of layers within the Processor block. The maximum depth of the mussage-passing layers within the Processor block varies across different Node Models and is determined by GPU memory allocation for each Node Model but not greater than 7 layers.
6.2 Loss Function
For the GNN-based preconditioner, can use training with objectives equation and equation 5 to ap- proximate true factor L with L(0) obtained by GNNs. Fortunately, we can avoid this since the optimization problem has an exact solution. Given the optimal factors L, the problem we want to solve regression on the edges of the graph, for which the most natural loss is the L2 loss || I((0)ll- However, we observed unstable training in our experiments and we could not achieve convergence with such a loss. Therefore, we switched cosine similarity loss, which is well-suited to preconditioner learning. For sparse factorized preconditioners, the matrix L is defined up to scaling factor. For both preconditioners X LL" and x = (oL)(oL)T it is easy to show that the preconditioned systems X-1 A- X-1 will be equivalent since X-1 -===x-+ X for a + O.
In general, a good preconditioner does not need to be an exact approximation of A, since iterative methods depend on the spectral propertics of A Besides empirical stability, the cosine similarity loss only penalizes mismatched direction but not mismatched scale. This allows for more possible variants of L(0) and eases training. We expect good approximation of L in cosine similarity to be very close to 1.
9

---

Published in Transactions on Machine Learning Research (February/21) 2026)
MLPNodeModel
MessagePassingMLL
smilariy
0.9
2 deptt
Casine siillariy
0.9
2 blocks
alocss
0.0
4 depth
0.0
3 MP blacks
0.7
5 deptt
0.7
4 MP slocks
0.6
deptth
0.6
www.
blocks
7 depth
MP alocks
0.5
MP alaciss
0.4
0.4
20
80
100
50
100
150
200
250
300
Epoch
Epoch
GAT
GEN
MP blocks
MP blocks
similaity
0.9
Blocks
Cosine similarity
0.9
blocks
MP Blocks
0.8
MP Blacks
0.7
MP elarkes
0.7
MP blacies
blacks
0.6
MP blocks
MP blockss
MP blocks
0.5
7 MP blocks
0.4
0.4
50
100
150
203
250
50
1000
150
200
Epach
Epach
GraphTransformr
GidbalGraphhlansttomer
MP blacks
MP blacks
similarity
0.9
blocks
Cosine similrity
0.9
blocks
0.8
MP blocks
0.7
MP blocks
0.7
4 MP blocks
5/107 blocks
0.6
MP blocks
blacks
blocks
0.5
0.5
MP blacks
7 MP blacks
0.4
50
100
150
200
250
300
0
50
100
150
200
250
Epach
Figure 3: Experiments on the synthetic dataset. Cosine similarity between true T. and predicted T.(0) factors of preconditioners during training. Higher is better.
6.3 Learning Triangular Factorization
Synthetic tridiagonal benchmark generated as described in Section 5.1 The modified training pairs (A,"L are obtained follows:
We begin our experiments with a synthetic benchmark, which is
ATT A t
L," cho)(A
(8)
where chol is a Cholesky factorization.
A trivial empirical justification of the non-local behaviour of the considered problem is performed using deep feed-forward network, MLPNodeModee, which has information about the neighbourhood context (Figure 3) Surprisingly, the classical graph network layers GAT and GEN achieve slightly higher final accu- racy than MLPNodeModel. We assume that this behaviour is explained by the fact that these architectures are not designed to properly pass edge-level information, a primary goal of our work. The GNN with Mes- sagePassingMT.P layer, on the other hand, directly uses edge features, allowing to produce satisfactory results with a number of rounds greater than one.
The GrodaalGaaTTTaaasoooooo straightforward global information propagation not necessarily lead to proper preconditioner approxima- tion. Even global all-pair attention via node features does not enable the GlobalGraph Transformer to learn a good triangular factorization.
is designed for global all-pair attention. However, one can observe that
K-optimal preconditioners on SuiteSparse subset Experiments with K-optimal preconditioners (Fig- ure 4 show that none of the models can achieve a higher accuracy than the bascline food-forwart network, except MessagePassingMLPP However, MessagePassinggNe performs slightly better than the baseline model. Although GAT, GEN and ChohalGraph do not explicitly edge uratures in their layers, infor-
10

---

Published in Transactions on Machine Learning Research (February22022
mation should still propagate through the sender-receiver connection in the edge model. However, did not observe any sat satistactory approxination with these models.
MI PNndeModel
MessagaePscinngMie
similarky
0.9
2 depth
depth
Cosine simllarky
0.9
blocks
blocks
4 depth
0.8
MP blocks
0.7
5 depth
0.7
blocks
depth
0.6
7 depth
0.4
0.4
100
200
250
50
100
150
200
250
300
Epoch
Epoch
GAT
GEN
blocks
blocks
similarity
0.9
blocks
Cosine similarity
0.9
MP blocks
0.8
MP blocks
0.7
0.7
0.6
0.5
0.1
0.4
100
150
200
250
330
0
50
100
150
200
250
300
Epoch
Epoch
GraphTiassfommer ansformer
GlobalG: aph anrfommer
MP blacks
MP alacks
0.9
blacks
Cosine similarity
0.9
blocks
MP blocks
MP bloces
0.7
0.7
0.6
0.5
0.4
50
100
150
200
250
300
50
100
150
200
250
300
Epoch
Epach
Figure 4: Experiments on the K-optimal preconditioners for the SuiteSparse subset. Cosine similarity between true L and predicted L(0) factors of preconditioners during training. Higher is better.
Experiments with Virtual Node We also explored a simple yet popular form of explicit global infor- mation propagation by augmenting the graph with virtual node, i.c., an additional node connected to every other node in the graph. This mechanism is often used in graph learning way to aggregate and redistribute global context. In our setting, however, it introduces strong bottleneck: heterogeneous and multiscale features of the operator must pass through single low-dimensional representation, which is closely related to the oversquashing phenomenon discussed the GNN literature. Consistent with this intuition, our experiments show that adding a virtual node does not improve the quality of the learned precondi- tioners, while substantially increasing the number of edges and thus the computational cost. These results also suggest that naive global aggregation is insufficient overcome the non-local dependencies inherent in high-quality triangular factors. The corresponding experimental results reported in Appendix A
Order-dependence of the GNN-based preconditioners While a standard message-passing GNNs are permutation equivariant when applied to edge features of A (Satorras et al., 2021), the overall mapping from our implementation, the GNN predicts values on
A a triangular factor inherently order-dependent. the sparsity pattern of A, and we then construct the factor by retaining only j. does not commute with arbitrary permutations. In general,
(CNN( AP)) # (GNN((()
Consequently, the learned preconditioner should be interpreted as order-dependent. This behavior is con- sistent with classical incomplete factorizations, which routinely employ reorderings to reduce fill to improve offectiveness. Rather than viewing the lack of strict permutation equivariance as limitation, motivates future work on incorporating ordering information more explicitly. The corresponding experimental results with positional encodings in work Häusner et al. (0224) reported in Appendix B
11

---

Published in Transactions on Machine Learning Research (February/200)
7 Discussion
We have demonstrated that GNNs cannot recover the Cholesky factors for tridiagonal matrices, for which perfect sparse preconditioners exist. Under fully controlled conditions-whee exact factorizations exist for synthetic matrices the models failed to approximate the preconditioners. For the real-world matrices, one could argue that the GNNs can learn better preconditioners then K-optimal IC(0) preconditioners in terms of quality. This is a subject of future work, but we believe that current benchmarks and the quality of computed preconditioners are quite challenging for statc-of-the-art methods, even when using functionals other than cosine similarily. This claim is supported by the fact that current GNN-based precondit ioners often do not outperform their classical analogues in terms of their effect on the spectrum of A and are usually not universal.
For problems whose graphs can have very large diameter (up to O(N)), an exact simulation of the underlying algorithm with peessage-psssing would in principle require the same order of depth. In this regime, depths equals to 1 and 7 are both very small compared to what would be needed to show GNN the entire graph. There is no reason to expect performance to improve smoothly with depth. Instead, we observe an almost flat plateau corresponding to the best achievable local approximation, which persists until the depth becomes comparable to the graph diameter. In other words, we are likely approximating a function that does not admit increasingly accurate local approximations. Further work includes understanding the effective depth required by novel architectures, possibly with different aggregation mechanisms or global memory, to break the quality barrier observe in our experiments.
A natural question is whether these limitations could be overcome by using deeper GNNs or archilect with global information propagation. However, our results show that even graph transformers with all-pairs attention and virtual node do not resolve the underlying non-locality barrier. This suggests that the problem is not only an insufficient receplive field, but also requires a dedicated architecture and shifts problem to designing architectures with the right algebraic inductive bias. The locality bias of message- saassag-ppssing is fundamentally incompatible with the non-local structure of sparse preconditioners.
These findings have two important implications. First, claims about GNNs' ability to learn matrix factor- izations for scientific computing must be tempered by these structural limits. Second, progress will likely require architectural innovations that go beyond standard message- passigg,,psnn,,,. potentially drawing insights from numerical linear algebra.
Another important design choice in classical sparse factorization is matrix reordering (Amestoy et al., 1996; Davis et al.) 2004). Symmetric permutations can dramatically reduce fill-in and improve the structure of the elimination tree. In the present study, we keep the ordering fixed and focus the difficulty of learning preconditioners for given sparsity pattern. For our synthetic benchmark, the sparsity pattern is tridiagonal and the fill-in is already minimal by construction, so standard bandwidth-roducing permutations cannot further improve the structure. For real-world matrices, however, reorderings may alter the effective graph distances over which crucial dependencies propagate. Exploring the interaction between reordering strategies and learned proconditioners an interesting direction for future work that could complement the limitations identified in this study.
While our experiments show that standard message-passing GNNs struggle to learn high-quality incomplete factorizations in the regimes we study, these findings should not be interpreted a general failure of neural networks in scientific computing. The practical success of neural network-based solvers depends on many additional factors, including the structure of the matrix family under consideration, numerical conditioning, architectural and implementation choices, training strategies, etc. A deeper investigation of these aspects represents an important direction for future research in applying deep learning to numerical mathematics.
Limitations We restricted our attention to symmetric positive definite matrices. The section on the tridiagonal matrices remains unchanged, but the K-optimality does not apply to non-symmetric matrices. Hence, other approaches are necessary for constructing the corresponding benchmarks. We will address this issue in future work.
12

---

Published in Transactions on Machine Learning Research (obbrarr//2200
References
Jan Ackmann, Peter D Düben, Tim N Palmer, and Piotr K Smolarkiewicz. Machine-learned preconditioners for lincar solvers in geophysical fluid flows. arXiv preprint arXiv:2010.0880 2020.
Patrick R Amestoy, Timothy A Davis, and lain S Duff. An approximate minimum degree ordering algorithm. SIAM Journal on Matriz Analysis and Applications, 17(4))886 905, 1996.
Peter W Ballaglia, Jessica B Hamrick, Victor Bapsi, Alvaro Sanchez-Gonzalez, Vinicius Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, et al. Relational inductive biases, deep learning, and graph networks. ar preprint arXiv: 01261, 2018.
Michele Benzi. Preconditioning techniques for large linear systems: a survey. Journal of computational Physics, 182(2):418 477, 2002.
Joshua Dennis Booth, Hongyang Sun, and Trevor Garnett. Neural acceleration of incomplete cholesky preconditioners. arXiv preprint or 0am:2433.00743 2024.
Shaked Brody, Uri Alon, and Eran Yahav. How attentive are graph attention networks? arXiv preprint arXiv:2105.1449h 2021.
Salvatore Cali, Daniel C Hackett, Yin Lin, Phiala E Shanahan, and Brian Xiao. Neural-network precondi- tioners for solving the dirac equation in lattice gauge thcory. Physical Review D, 107(3):034508, 2023.
Jie Chen. Graph neural preconditioners for iterative solutions of sparse linear systems. arXie preprint
arXiv:2406. 00809, 2024.
Jiong Chen, Florian T Schäfer, and Mathieu Desbrun. Lightning- method of fundamental solutions.
ACM Transactions on Grophics, 43(4):77, 2024.
Timothy A. Davis. Suitesparse: A suite of sparse matrix algorithms. httpps//github.cco 2024. GitHub repository.
Timothy A Davis, John R Gilbert, Stefan I Larimore, and Esmond G Ng. A column approximate minimum degree ordering algorithm. ACM Transactions on Mathematical Software (TOMS), 30(3):353 376, 2004.
Matthias Fey and Jan Eric Lenssen. Fast graph representation learning with pytorch geometric. arXiv
preprint Xrv: 02428, 2019.
Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. In International conference on machine learning, pp. 1263-1272 Pmlr, 2017.
Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive represcntation learning on large graphs. Advances in neural information processing systems, 30, 2017.
Paul Häusner, Ozan Öktem, and Jens Sjölund. Neural incomplete factorization: learning preconditioners for the conjugate gradient method. arXiv preprint arXiv:2305.16363, 2023.
Paul Häusner, Aleix Nieto Juscafresa, and Jens Sjölnd. Learning incomplete factorization preconditioners for gmres. orXiv preprint orXio:2409.08920 2024.
Tgor Kaporin. New convergence results and preconditioning strategies for the conjugate gradient method. Numerical linear algebra with opplications, 1(2):179 210, 1994.
Alexandr Katrutsa, Talgat Daulbacv, and Ivan Oscledets. Black-box lcarning of multigrid parameters. Journal of Compulational and Applied Mathematics, 368:112524, 2020.
Guohao Li, Chenxin Xiong, Ali Thabet, and Bernard Ghanem. Deepergen: All you need to train deeper gens. Xio preprint ar arXiv:2006.077390 2020.
13

---

Published in Transactions on Machine Learning Research (obbuaayy/2200
Mou Li, He Wang, and Peter K Jimack. Generative modeling of sparse approximate inverse preconditioners In International Conference on Compulational Science, pp.
Springer, 2024.
Yichen Li, Peter Yichen Chen, Tao Du, and Wojciech Matusik. Learning preconditioners for conjugate gradient pde solvers. In International Conference on Machine Learning, pp. 19425-19439. PMT.R, 2023.
Jian Luo, Jie Wang, Hong Wang, Zijie Geng, Hanzhu Chen, Yufei Kuang, et al. Neural krylov iteration for accelerating linear system solving. Advances in Neural Information Processing Systems, 37:128636 1286677 2024.
lay Luz, Meirav Galun, Haggai Maron, Ronen Basri, and Irad Yavneh. Learning algebraic multigrid using graph neural networks. In International Conference on Machine Learning, pp. 6489 6499. PMLR, 2020.
Nicholas S Moore, Eric C Cyr, Peter Ohm, Christopher M Sicfert, and Raymond S Tuminaro. Graph neural networks and applied linear algebra. STAM review, 67(1):141-177 2025.
T Konstantin Rusch, Michael M Bronstein, and Siddhartha Mishra. A survey on oversmoothing in graph neural networks. arXic preprint orXiv:2303.10993 2023.
Yousef Saad. Neralive methods for sparse linear systems. STAM, 2003.
Johannes Sappl, Laurent Seiler, Matthias Harders, and Wolfgang Rauch. Deep learning of preconditioners for conjugate gradient solvers in urban water related problems. arXiv preprint ar arXic:1906.06925, 2019.
Victor Garcia Satorras, Emiel Hoogeboom, and Max Welling. E (n) equivariant graph neural networks. Tn International conference on machine learning, pp. 9323 9332. PMLR, 2021.
Yunsheng Shi, Zhengjie Huang, Shikun Feng, Hui Zhong, Wenjin Wang, and Yu Sun. Masked label prediction: Unified message passing model for semi-supervised classification. arXiv preprint arXiv:2009. (rXx:2200003300,, 2020.
Joshua Southern, Francesco Di Giovanni, Michael Bronstein, and Johannes F Lutzeyer. Understanding virtual nodes: Oversmoothing, oversquashing, and node heterogencity. arXiv e-prints, pp. arXiv-2405, 2024.
Ali Taghibakhshi, Nicolas Nytko, Tareq Uz Zaman, Scott. MacLachlan, Luke Olson, and Matthew West. Learning interface conditions in domain decomposition solvers. Advances in Newral Information Processing Systems, 35:7222-7235, 2022.
Ali Taghibakhshi, Nicolas Nytko, Tareq Uz Zaman, Scott MacLachlan, Luke Olson, and Matthew West. Mg-gnm: Multigrid graph neural networks for learning multilevel domain decomposition methods. In International Conference on Machine Learning, pp. 33381-33395. PMLR, 2023.
Jake Topping, Francesco Di Giovanni, Beujamin Paul Chamberlain, Xiaowen Dong, and Michael M Bronstein. Understanding over-squashing and bottlenecks on graphs via curvature. arXio preprint arXiv:211111422,, 2021.
Vladislav Trifonov, Alexander Rudikov, Oleg Nliev, Yuri M Laevsky, Ivan Oseledets, and Ekaterina Muravl- eva. Learning from linear algebra: graph neural network approach to preconditioner design for conjugate gradient solvers. Xrxiv preprint ar Xri:2244 15557, 2024.
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. arXiv preprint arXiv: 10903, 2017.
Qitian Wu, Wentao Zhao, Chenxiao Yang, Hengrui Zhang, Fan Nie, Haitian Jiang, Yalao Bian, and Junchi Yan. Simplifying and empowering transformers for large-graph representations. In Thirty-seventh Con- ference on Neural Information Processing Systems, 2023.
Keyulu Xu, Weihua Hu, Jure Leskovee, and Stefanie Jegelka. How powerful are graph neural networks?
arXiv preprint ar Xiv: 181:1810. 0..0000000 2018
14

---

Published in Transactions on Machine Learning Research (February/200)
A Experimental Results with Virtual Node
MLPNNdeModel
Messageaassing
1.0
cepth
0.9
copth
0.9
copth
0.7
Ecpco
cepth
0.6
cepeo
Mleccs
04
0.4
Epoch
GAT
GEN
1.0
MP blocks
MD Meccs
0.-
0.5
blocks
Codlee
Coatee
0.0
0.4
0.4
spach
epoch
Grasshiransfommer
Gicbe@@rghhTronsffmmee
MP blocks
Mocks
11
0.00
Cuate amilany
Conteet
Mocks
00-
0.6
0.4
0.
130
Spach
Spach
Figure 5: Experiments with virtual node on the synthetic dataset. Cosine similarity between true L and predicted L(O) factors of preconditioners during training. Higher is better.
15

---

Published in Transactions on Machine Learning Research (February/200) 2026)
M.PhhdeMaddl
MessagelasssingMi
7:0pp
MP block
04
0.0
MP BROCS
Cod sinibrter
Coasso sinvaarey
blocks
0.
Sepey
0.6
0.1
0.4
GAT
GEN
NP blocks NP NNMbioe
MP blode
0.4
0.0
0.0
0.4
0.4
150
Rpocn
Graph Transforner
GiobalGraph
NP blocks
MP block
0.
0.5
XD Blod
Coshe Si-ibrit
Coamro simitaryy
0.0
0.4
250
Opocn
Figure 6: Experiments with virtual node on the K-optimal preconditioners for the SuiteSparse subset. Cosine
similarity between true and predicted L(0) factors of preconditioners during training. Higher is better.
B Experimental Results with Positional Encodings
The positional encoding is an additional edge feature that has values -1 and +1 for the lower and upper triangular part of the matrix respectively (Hänsner et al. 2024
Synthetic dataset
Koptimml preconditioners for the SuiteSparse
blocks
0.9
0.9
blocks
Cosine similanty
similarity
MP blocks
0.8
0.8
blocks
MP blocks
Blocks
0.8
MP blocks
0.6
blocks
MP blocks
0.4
Epoch
Epoch
Figure 7: Experiments with positional encodings the MessgacPassingMLL model. Cosine similarity between true L and predicted L(0) factors of preconditioners during training. Higher is better.
16


Reviews:
[{"review_id": "Q2azdqGUiX", "reviewer": "Reviewer_kLEU", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "There are two main contributions of this work. First, a dataset of SPD matrices for which an “optimal” (in the K-condition number sense) preconditioner can be found with the same sparsity pattern as the original matrix. This dataset contains two types of matrices: a synthetic dataset of tridiaganal matrices and a set of matrices from the SuiteSparse collection. In both cases, the matrices are such that an “optimal” preconditioner can be found, thereby giving the graph neural networks a target to learn.\n\n\nThe second contribution is experimental evidence that graph neural networks (GNNs) cannot be used to approximate such preconditioners. Several common graph neural network layers are trained in encoder-process-decode models to maximize cosine similarity. In general, the experiments provide evidence that in cases where good preconditioner construction requires non-local information, GNNs are not able to approximate such optimal preconditioners.", "claims_and_evidence_yn": "Yes", "claims_explanation": "In general, the analysis for the fact that a change in a single entry of a matrix can significantly affect many entries of the corresponding Cholesky factor appears accurate, as does the construction approach for building K-optimal preconditioners. The construction of the synthetic dataset appears sound, both from a theoretical standpoint and supported by the computational experiments. Based on the completed experiments, both components of the suggested dataset do appear to be adequate problems for testing how well a potential model may be able to approximate a preconditioner of the type studied in this work (an approximate Cholesky factorization with the same sparsity pattern as the original matrix). That being said, I do have a couple of minor concerns which is given in the “requested changes” response below.", "audience_yn": "Yes", "audience_explanation": "Construction of effective preconditioners for linear systems is an ongoing research area with broad appeal and the potential to improve computational science along with many other scientific areas which depend on efficient solutions to large, sparse linear systems. The construction of an effective preconditioner is highly problem-dependent. Therefore, the hope is that machine learning methods can provide a method which is able to construct preconditioners from the entries of the matrix. As the author(s) mention, this has been attempted in prior literature and while the preconditioners show some promise, most fall short of well -tuned, human-designed preconditioners. One of the strengths of this paper is the construction of a database of problems for which good preconditioners are known, but that current GNN methods are unable to replicate (at least in practical use, see my note in the “requested changes” response.) This should give researchers in this area a benchmark that future methods and architectures can target as a proxy for problems where non-local information is required for building effective preconditioners.", "requested_changes": "While I agree that the computational experiments presented in the paper provide evidence from a practical standpoint that GNNs can’t approximate K-optimal preconditioners, I disagree that this proves that GNNs can’t, at least in theory, compute these factorizations. Since the calculation of a Cholesky factorization can be completed using matrix multiplications, I believe the methods described in [1] could be used to construct a GNN which computes a Cholesky factorization exactly. While this wouldn’t be sparse for a general SPD matrix, it would of course be sparse for the tridiagonal matrices in the synthetic dataset.\n\nThis counters the author’s assertion that widening the receptive field is not enough to be able to construct an effective preconditioner. However, the resulting GNN would have more layers than the number of rows in the matrix. Therefore, the required theoretical GNN would be unreasonably large and even if it was feasible to use such a network, the probability that an optimizer could locate the precise weights needed to attain the necessary theoretical network would be small.\n\nTherefore I don’t believe the experiments in the paper are misleading or incorrect and I agree with the author(s) that new architectures are needed if reasonably-sized GNNs are to be used to construct effective preconditioners, but the discrepancy between the theoretical possibility and the practical implementation needs to be addressed.\n\nAlso, graph algorithms that enlarge the receptive field by coarsening or updating graph-level information aren’t considered at all, but no explanation is given for why they are not considered. If the reason they are disregarded is out of belief that the size of the receptive field is immaterial, that argument is more cloudy given the argument above. Thus some experimental evidence would be useful for firming up the argument that these methods are still insufficient.\n\n[1] Moore NS, Cyr EC, Ohm P, Siefert CM, Tuminaro RS, \"Graph Neural Networks and Applied Linear Algebra\", SIAM Rev, 2025", "broader_impact_concerns": "None.", "additional_comments": "- From a grammatical perspective, all important words in titles of sections, subsections, etc. should be capitalized.\n- Typo Page 1: “A well-established choice for symmetric positive definite matrices is to use an incomplete Cholesky **factorization**\n- Typo Page 3: line after eq (1) “biadiagonal” should be **bidiagonal**", "extra_scores": {}, "extra_text": {}}, {"review_id": "NXKAjAGmbe", "reviewer": "Reviewer_ri4G", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper’s title, “Message-Passing GNNs Fail to Approximate Sparse Triangular Factorizations”, summarizes the paper’s main content well. The paper studies the recent trend of approximating the inverse of a sparse matrix with a graph neural network. Noting that the inverse of a sparse matrix may introduce global dependencies among its nonzero entries, the paper constructs synthetic and practical matrices to demonstrate that existing GNN frameworks struggle to learn a good approximation of matrix factorization. Based on the experiment results, the paper concludes that existing GNNs have major structural limitations for learning matrix factorization in scientific computing and suggests that future research in this area requires innovative designs in the network architecture.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The paper’s motivation is well grounded in known facts in applied math, e.g., the properties of the factorization of a tridiagonal matrix, the Green's function of Poisson’s equation, and the closed-form solution to Kaporin preconditioning. The central claim is well motivated and follows naturally from these known facts. Overall, based on my research experience in this area over the past few years, I am generally OK with most of the claims and the experimental evidence reported in the paper.\n\nA few claims in the paper could benefit from further clarification, which I list in the requested changes below.", "audience_yn": "Yes", "audience_explanation": "Yes, I believe the topic of this paper will be of interest to researchers working on AI for linear solvers/PDEs. While multiple papers have attempted to develop network methods for solving matrices, few have conducted comprehensive comparisons between neural network solvers and classical solvers on large-scale problems. In fact, many classical solvers remain highly competitive, or even superior, in solving large-scale problems [1]. This TMLR submission makes a good effort to calibrate our expectations for a network method in solving classic numerical problems. Although it does not directly address solving PDEs/preconditioning linear systems (approximating the inverse of a matrix is, of course, highly relevant), the research direction it advocates for is critical for the healthy development of AI for linear solvers/PDEs.\n\n[1] Weak baselines and reporting biases lead to overoptimism in machine learning for fluid-related partial differential equations. Nature Machine Intelligence 2024.", "requested_changes": "Overall, this is good work. I have only two things to suggest:\n\n1. The derivation of the closed-form solution for the Kaporin preconditioner in Section 4 reminds me of results reported in previous work, particularly Appendix A of Chen et al. [2]. Please clarify the novelty of the results in Section 4 relative to it.\n\n2. While the paper provides convincing evidence that GNNs struggle to approximate matrix factorizations in some instances, it occasionally generalizes such limitations to broader scientific computing problems. In practice, many other factors play critical roles in determining whether a network can actually work well for a problem, e.g., the matrix pattern, implementation details, and system engineering/optimization, which are not fully covered in this paper’s discussion. I suggest the paper also inform readers of these factors and refrain from overgeneralizing their findings to (general) scientific computing before we see a more comprehensive analysis, which is perhaps outside the scope of this work.\n\n[2] “Lightning-fast method of fundamental solutions” (ACM TOG 2024) by Chen et al.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "E4PB4jfM9Z", "reviewer": "Reviewer_UMo9", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper investigates a fundamental limitation of using graph neural networks (GNNs) to learn sparse matrix factorizations. The authors show that (approximate) factorizations of sparse matrices can exhibit non-local structure, making them challenging for local message-passing architectures to approximate. Using a constructed example based on tridiagonal matrices, they demonstrate that the exact bidiagonal Cholesky factorization can behave non-locally, which message-passing GNNs with a fixed number of steps are theoretically incapable of capturing. Empirically, they observe that GNNs do not recover the lower-triangular factors, as reflected by low cosine similarity between predictions and ground truth.", "claims_and_evidence_yn": "No", "claims_explanation": "The theoretical results are interesting and relevant, but the empirical evaluation does not yet fully support the strong claims made in the paper. In particular, two standard architectural choices are not included in the analysis, and considering them could significantly strengthen the conclusions especially those involving the graph transformer baseline:\n- **positional encodings**: GNNs and transformers are inherently permutation invariant so they do not know the output order which is, however, highly relevant for determining the lower triangular factor of the factorization. It would strengthen the paper to test whether positional encodings of each node, as in [1], mitigate some of the limitations.\n- **virtual nodes**: according to the findings in the paper, the non-locality is an issue which could potentially easily resolved using a global node that is used to communicate between distant nodes which is a standard construction block in graph neural networks as recently studied for long-rage dependencies in [2], is it enough to add a single virtual node to learn the factorization?\n\nBeyond these main concerns, I have several additional questions and suggestions that could help strengthen the empirical support for the claims.  I do not require the authors to address all of these points but clarifying some would be valuable:\n- The theoretical results suggest that perfect performance requires O(n) steps. Given that, could you comment on why preconditioner performance does not improve when increasing the number of message-passing blocks beyond 2? One might expect incremental improvement even if exact recovery is not possible.\n- Plotting performance as a function of the graph diameter would directly support the claim that locality is the limiting factor. Larger diameters should correlate with reduced performance for fixed depth. When depth equals the graph diameter, the paper’s claims are less applicable, so illustrating this boundary condition could add clarity.\n- Could the authors elaborate why the experimental evaluation is conducted with respect to the K-optimal sparse matrices rather than the IC(0) matrices? Can GNNs learn IC(0) preconditioners? For the tridiagonal matrices, the IC(0) and K-optimal should coincide since the inverse is uniquely determined in the given sparsity pattern but for the suite sparse systems, computing the dataset is prohibitively expensive for large systems making the approach limited to small scale problems. What is the largest matrix in the suitsparse system that is considered?\n\n[1] Häusner, Paul, Aleix Nieto Juscafresa, and Jens Sjölund. \"Learning incomplete factorization preconditioners for GMRES\", NLDL, 2025.\n[2] Joshua Southern, Francesco Di Giovanni, Michael Bronstein, Johannes F. Lutzeyer. \"Understanding virtual nodes: Oversmoothing, oversquashing, and node heterogeneity\", ICLR, 2025.", "audience_yn": "Yes", "audience_explanation": "The work addresses a foundational question about the capabilities and limitations of GNN-based approaches for linear algebraic tasks, which should be of interest to the community.", "requested_changes": "Overall, the paper tries to answer an interesting and important question. The theoretical results indicate that previous methods have fundamental limitations that need to be addressed. However, the experimental section could more clearly support the full strength of the claims as I have outlined in my answer above. A number of clarifications and refinements would greatly improve readability and rigor:\n\n- The condition number does not necessarily be small for preconditioning to work, rather the spectrum of the matrix is required to be clustered.\n- The K-condition number needs a positive definite matrix so it is not clear how it is defined for $K(XA)$ in the text which is generally non-symmetric.\n- Figure 2 may give the impression that computing the KKK-preconditioner is straightforward, although it requires access to $A^{−1}$; explaining this connection may help avoid confusion.\n###### Questions\n- How do eq. 5 and eq. 4 relate?\n- Should Equation (8) involve the Cholesky factorization of the perturbed system rather than the original one?\n- Does the K-optimal preconditioner always work when the IC(0) preconditioner works? \n- Will the benchmark problems created for the paper be released?\n- Because K-optimal factors are not unique, do you normalize them to make the learning target well-defined?\n- What are the inputs to the node features of the GNN? \n- What is the cosine similarity loss used for training? Does it only compare the L factors of the matrix M?\n- What are the sizes of the considered matrices from the synthetic and suitesparse datasets?\n###### Writing \n\nThe manuscript would benefit from improved clarity and consistent notation. Some suggestions:\n\n- The notation for preconditioners shifts across sections: $X^{-1}$ in the introduction, $X$ in Section 3, lower-triangular $X$ in Section 4, and $M$ in Section 6. Unifying notation would improve readability.\n- Referring to exact inverses of tridiagonal matrices as “preconditioners” may be confusing; using more precise terminology could help.\n- Distinguish clearly between a _single_ message-passing step (local) and _multiple_ steps (expanded receptive field), particularly in the problem formulation.\n- In Section 6.2, scaling the system does not produce an identical system, although it may not affect preconditioning performance; clarifying this point would be helpful.\n\nOverall, I encourage the authors to provide additional technical details, ensure consistent notation, and refine the presentation for clarity.", "broader_impact_concerns": "There are no broader impact concerns for the paper.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "zkGJvCtiRf", "reviewer": "Reviewer_Z8yc", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper investigates the fundamental limitations of message-passing GNNs in learning sparse triangular preconditioners for linear systems. The key contributions are: (i) theoretical and empirical evidence that non-local dependencies in high-quality preconditioners cannot be captured by standard message-passing architectures; (ii) a benchmark dataset (synthetic and SuiteSparse-derived) to evaluate GNN-based approaches; and (iii) systematic evaluation across multiple GNN variants showing consistently low approximation quality (cosine similarity ≤0.7).\n\n**Strengths**: Clear motivation, technically sound methodology, rigorous mathematical analysis, and reproducible experimental design. The benchmark provides a valuable resource for the community.\n\n**Areas for further elaboration**: The negligible performance gap between shallow and deep architectures needs some additional discussion. The potential role of matrix reordering techniques (e.g., AMD, COLAMD) in graph construction could also enhance the technical depth. (see Requested Changes)", "claims_and_evidence_yn": "Yes", "claims_explanation": "The authors deliver compelling evidence through extensive evaluation across diverse matrix classes and GNN architectures. The theoretical framework correctly identifies non-local dependencies as a principal challenge. The benchmark construction follows standard practices and is carefully executed. The empirical findings could be further enriched by additional discussion of the observed depth-independence phenomenon, which would provide a more complete understanding of the architectural limitations.", "audience_yn": "Yes", "audience_explanation": "The findings directly impact ML + scientific computing research by highlighting structural barriers in applying GNNs to matrix factorization. The benchmark enables future architectural innovation, making it relevant to both ML and numerical methods communities.", "requested_changes": "**Important clarifications:**\nThe negligible performance gains from increased depth across most experiments (e.g., 1-layer vs. 7-layer models achieving nearly identical cosine similarity on SuiteSparse matrices) is surprising given the paper's emphasis on non-locality. The authors attribute this to \"architectural mismatch,\" but this remains descriptive rather than diagnostic. The paper would be significantly strengthened by either (a) a hypothesis for why depth fails, supported by preliminary evidence (e.g., analyzing error patterns), or (b) explicitly framing this as an open problem for future work.\n\n**Recommended enhancements:**\nConsider discussing the potential role of matrix reordering techniques (e.g., AMD, COLAMD) in the graph representation. Since reordering is fundamental to classical sparse Cholesky factorization, examining whether directed graph structures or order-aware message passing could address some identified limitations would substantially enhance the paper's technical depth.", "broader_impact_concerns": "None. This work addresses a fundamental problem in numerical linear algebra with no apparent negative societal implications.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)