Transolver++: An Accurate Neural Solver for PDEs on Million-Scale Geometries
Authors: Huakun Luo
Haixu Wu
Hang Zhou
Lanxiang Xing
Yichen Di
Jianmin Wang
Mingsheng Long
Venue: ICML 2025 poster (2025)
Keywords: PDE;Neural Operator;Eidetic Physical States;Local Adaptive mechanism;Parallelism optimization;large-scale industrial dataset;
Primary area: deep_learning->algorithms
TL;DR: We present Transolver++, which enables PDE-solving on million-scale geometries for the first time with a parallelism framework and a local adaptive mechanism.

Abstract: Although deep models have been widely explored in solving partial differential equations (PDEs), previous works are primarily limited to data only with up to tens of thousands of mesh points, far from the million-point scale required by industrial simulations that involve complex geometries. In the spirit of advancing neural PDE solvers to real industrial applications, we present Transolver++, a highly parallel and efficient neural solver that can accurately solve PDEs on million-scale geometries. Building upon previous advancements in solving PDEs by learning physical states via Transolver, Transolver++ is further equipped with an extremely optimized parallelism framework and a local adaptive mechanism to efficiently capture eidetic physical states from massive mesh points, successfully tackling the thorny challenges in computation and physics learning when scaling up input mesh size. Transolver++ increases the single-GPU input capacity to million-scale points for the first time and is capable of continuously scaling input size in linear complexity by increasing GPUs. Experimentally, Transolver++ yields 13\% relative promotion across six standard PDE benchmarks and achieves over 20\% performance gain in million-scale high-fidelity industrial simulations, whose sizes are 100$\times$ larger than previous benchmarks, covering car and 3D aircraft designs.

Transolver++: An Accurate Neural Solver for PDEs on Million-Scale Geometries
Huakun Luo" *1
Haixu Wu
Hang Zhou'
Lanxiang Xing 1
Yichen Di
Jianmin Wang'
Mingsheng Long'
Abstract
industrial manufacturing (Kopriva, 2009; Roubiček, 2013). However, it is hard and usually impossible to obtain the analytic solution of PDEs. Thus, numerical methods (Solín, 2005) have been widely explored, whose typical solving pro- cess is first discretizing PDEs into computation meshes and then approximating solution on discretized meshes (Gross- mann et al., 2007). In real applications, such as simulating a driving car or aircraft during takeoff, it will take several days or even months for calculation, and the simulation accuracy is highly affected by the fitness of discretized computation mesh (Solanki et al., 2003; Elrefaie et al., 2024). To speed up this process, deep models have been explored as efficient surrogates of numerical methods, known as neural PDE solvers (Wang et al., 2023). Training from pre-collected simulation data, neural solvers can learn to approximate the mapping between input and output of numerical methods and directly infer new samples in flash (Li al., 2021), posing a promising direction for industrial simulation.
Although deep models have been widely explored in solving partial differential equations (PDEs), previous works are primarily limited to data only with up to tens of thousands of mesh points, far from the million-point scale required by indus- trial simulations that involve complex geometries. In the spirit of advancing neural PDE solvers to real industrial applications, we present Tran- solver++, a highly parallel and efficient neural solver that can accurately solve PDEs on million- scale geometries. Building upon previous ad- vancements in solving PDEs by learning phys- ical states via Transolver, Transolver++ is fur- ther equipped with an extremely optimized par- allelism framework and a local adaptive mecha- nism to efficiently capture cidetic physical states from massive mesh points, successfully tackling the thorny challenges in computation and physies learning when scaling up input mesh size. Tran- solver++ increases the single-GPU input capacity to million-scale points for the first time and is capable of continuously scaling input size in lin- car complexity by increasing GPUs. Experimen- tally, Transolver++ yields 13% relative promotion across six standard PDE benchmarks and achieves over 20% performance gain in million-scale high- fidelity industrial simulations, whose sizes are 100x larger than previous benchmarks, covering car and 3D aircraft designs. Code is available at http://giituubooo/uuuurm// http://tu/////sleerrrrrrr plus.
Usually, industrial applications involve large and complex geometries, requiring the model to capture intricate physics processes underlying geometries efficiently. Although pre- vious works have made some progress in handling complex geometries (Li et al., 2023b; Hao et al., 2023) and attempted to speed up model efficiency, they still far from real ap- plications, especially in handling large geometries. Specif- ically, as illustrated in Figure (a)), existing models fail to scale up beyond 400k points, while real applications typi- cally involves million-scale mesh points or even more. Note that, as aforementioned, the fineness of computation meshes is the foundation of simulation accuracy. As the comparison of car mesh shown in Figure ((b), limited mesh size will seriously sacrifice the precision of the geometry, where the originally streamlined surface becomes rough and uneven. This will further limit the simulation accuracy of physics processes, especially for aerodynamic applications (Elrefaic et al., 2024). Thus, the capability of handling large geome- tries indispensable for practical neural PDE solvers.
1. Introduction
Extensive physics processes can be precisely described by partial differential equations (PDEs) (Wazwaz, 2002; Evans, 2010), such as air dynamics of driving cars or internal stress of buildings. Accurately solving these PDEs is essential to
As the latest progress in neural PDE solver architectures, Transolver (Wu et al., 2024) proposes to learn intrinsic phys- ical states underlying complex geometries and apply the at- tention mechanism among learned physical states to capture intricate physics interactions, which frees model capacities from unwieldy mesh points and achieves outstanding perfor- mance in car and airfoil simulations. Although Transolver
Eqqual contribution 'School of Software, BNRist, Tsinghua University. Huakun Luo <luohk24@@mall mails. wiwgsssa.nssssssseaeaae ed". Cor- respondence Mingssngnng Long boogg=-mngssengeee wsinghua.cuu.ccmm
Proceedings of the nnnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 20255 by the author(s).

---

Transolver++: An Accurate Neural Solver for PDEs on Million-Seale Geometries
[a) Maximum points single A100 GPU
(b) Scale of benchmarks adopted by different studies
50-
DrivAarnatt
GPU Mamory Usage (GB)
AirCroft
40
10-100x
ShapaNaiCar AirRANS
30-
2+
20-
Pipe
Airaol
Transiverr
10
GNOT
NS2D
Transo varee (Our))
Golerie
Plastictyy
Trano Others
Bleaticity
5k10k 50k
100k
200k
300k
400k
500k
600k
700k
1M
300k
700k
2.5M
#Points
#Mashes
Figure 1. (a) Comparison of model capability in handling large geometries. We plot the GPU memory change of each model when increasing input mesh points. The upper bound single A100 40GB GPU is depicted in the dotted line. (b) Comparison of experiment benchmarks. Transolver++ experiments on high-fidelity tasks with up to 2.5 million points, which is 100x larger than previous works.
demonstrates favorable capability in handling complex ge- ometries, its maximum input size remains limited to 700k points (Figure ((a)), and its experiment data is much simpler than real applications. Building on the essential step made by Transolver, we attempt to advance it to million-scale or even larger geometries in pursuing practical neural solvers.
from six standard benchmarks and over 20% improve- ment on high-fidelity million-scale industrial datasets, covering practical car and 3D aircraft design tasks.
2. Related Work
2.1. Neural PDE Solver
When scaling Transolver to million-scale high-fidelity PDE- solving tasks, we observe its bottlenecks in physics learning and computation efficiency. Firstly, as Transolver highly relies on the learned physical states, massive mesh points may overwhelm their learning process, resulting in homoge- neous physical states and model degeneration. Secondly, it is observed that even deep representations of million-scale points without considering intermediate calculations will consume considerable GPU memory, which is strictly con- strained by total resources on a single GPU. This drives us to elaboratively optimize the model architecture and unlock the power of multi-GPU parallelism. Thus, in this paper, we present Transolver++, which upgrades Transolver with an extremely optimized parallelism framework and a local adaptive mechanism to efficiently capture eidetic physical states from massive mesh points. Based on the co-design with model architecture, our parallelism framework signifi- cantly reduces the communication overhead and achieves linear scalability with resources without sacrificing perfor- mance. As result, Transolver++ achieves consistent state- of-the-art on six standard PDE datasets and successfully ex- tends Transolver to high-fidelity car and aircraft simulation tasks with million-scale points. Here are our contribations:
Traditional numerical methods for solving PDEs often re- quire high computational costs to achieve accurate solu- tions (Solanki et al., 2003). Recently, deep learning methods have demonstrated remarkable potential as efficient surro- gate models for solving PDEs due to their inherent non- linear modeling capability, known as neural PDE solvers.
As a typical paradigm. operator learning has been widely studied for solving PDEs by learning the mapping between input functions and solutions. FNO (Li et al., 2020a) first proposes to approximate integral in the Fourier domain for PDE solving. Subsequently, Gco-FNO (Li et al., 2021) ex- tends FNO to irregular meshes by transforming them into regular grids in the latent space. To further enhance the capabilities of FNO, U-FNO Wen et al. (2022) and U-NO Rahman et al. (2023) are presented by leveraging U-Net (Ronneberger et al., 2015) to capture multiscale properties. Considering the high dimensionality of real-world PDEs, LSM (Wu et al., 2023) applies spectral methods in a learned lower-dimensionall latent space to approximate input-output mappings. Afterward, LNO (Wang & Wang. 2024) adopts the attention mechanism to effectively map data from geo- metric space to latent space for complex geometries.
To ensure reliable modeling for complex physical inter- actions, we introduce Transolver++ with eidetic states, which can adaptively aggregate information from mas- sive mesh points to distinguishable physical states.
Recently, Transformers (Vaswani al., 2017) have achieved impressive progress in extensive fields and have also been applied to solving PDEs, where the attention mechanism has been proven as a Monte-Carlo approximation for global integral (Kovachki et al., 2023). However, standard atten- tion suffers from quadratic complexity. Thus, many mod- els like Galerkin (Cao, 2021), OFormer (Li et al., 2023c) and FactFormer (Tran et al., 2023) propose different effi- cient attention mechanisms. Among them, GNOT (Hao
We present an efficient and highly parallel implemen- tation of Transolver++ with linear scalability and mini- mal overhead across GPUs, affording mesh size of 1.2 million on a single GPU while maintaining accuracy.
Transolver++ achieves 13% relative gain averaged
2

---

Transolver++: An Accurate Neural Solver for PDEs on Million-Seale Geometries
et al., 2023) utilizes well-established linear attention, like Reformer or Performer (Kitaev et al., 2020; Choromanski et al., 2021), and separately encodes geometric information, achieving favorable performance. However, linear attention often suffers from degraded performance as an approxima- tion of standard attention (Qin et al., 2022). Moreover, these Transformer-basee methods treat input geometries as a se- quence of mesh points and directly apply attention among mesh points, which may fall short in geometrie learning and computation efficiency. As a significant advancement in PDE solving, Transolver (Wu et al., 2024) introduces the Physics-Attention mechanism that groups massive mesh points into multiple physical states and applies attention among states, thereby enabling more effective and intrinsic modeling of complex physical correlations. However, it still faces challenges in degenerated physics learning and a high computation burden under million-scale geometries. These challenges will be well addressed in our paper.
the communication volume remains excessive. In contrast, Transolver++ leverages its unique physics-leaning design and presents highly optimized parallelism framework tai- lored to PDE solving, enabling minimal communication overhead and allowing for meshes of million-scale points.
3. Revisiting Transolver
Before Transolver++, we would like to briefly introduce Physics-Attention, the key design of Transolver (Wu et al., 2024), and discuss its scaling issues for million-scale inputs.
Given meshes with N nodes, to capture intrinsic physics interactions under unwieldy mesh points, Physics-Attemt Attention will first assign C-channel input points x
| E
R- to M physical states S = { MM
1 xC based
on the slice weights w = =[w E R N learned from inputs, and each W; E RIIM represents the possibility that x; belongs to each state. Specifically, physical states are aggregated from all point representations based learned slice weights, which can be formalized as:
In addition to Transformers, graph neural networks (GNNs) (Hamilton et al., 2017: Gao & Ji. 2019; Pfaff et al., 2021) are also inherently suitable to process unstructured meshes by explicitly modeling the message passing among nodes and edges. GNO (Li et al., 2020b) first implements neural op- erator with GNN. Later, GINO (Li et al., 2023a) combines GNO with Geo-FNO to encode the geometry information. Recently, 3D-GeoCA (Deng et al., 2024) integrates pre- trained 3D vision models to achieve a better representation learning of geometries. However, prone to geometric insta- bility (Klabunde & Lemmerich, 2023), GNNs can lead to unstable results and may be insufficient in capturing global physics interactions, especially when handling large-scale unstructured meshes (Morris et al., 2023).
Slice weights: w = Softmax (Linear(x)/To)
N Σ Ei=1 Wi
M
(1)
Physical states: {,,}
Wij
j=1
where To is the temperature constant. Next, the canonical attention mechanism is applied to learned physical states to capture underlying physics interactions:
qk VC
q. k, v - Linear(s), s' Softmax
(2)
employs the deslice operation to map updated states s' back to mesh space using slice weights, i.e. x' = [x]],== = { w By re- placing standard attention in Transformer (Vaswani et al., 2017) with Ptysics-Attention, we obtain the Transolver.
Finally,
2.2. Parallelism of Deep Models
Despite processing large-scale geometries being crucial in industrial design, this problem has not been explored in previous research. This paper breaks the computation bot- tleneck by leveraging the parallelism framework, which is related to the following seminal works. Towards general needs in handling the high GPU memory usage caused by large-scale inputs, several parallel frameworks have been proposed, such as tensor parallelism (Shoeybi et al., 2019) and model parallelism (Huang et al., 2019). However, these methods are highly model-dependent and request significant communication overhead (Zhuang et al., 2023). Another di- rection is to optimize attention mechanisms. Ring attention (Liu et al., 2023), inspired by FltehAttention (Dao et al., 2022), uses a ring topology between multi-GPUs, achiev- ing quadratic communication complexity with respect to mesh points. Besides, DeepSpeed-Ulyssee (Jacobs et al., 2023) splits the deep representation along the channel di- mension and employs the All2All communication approach, reducing complexity to linear. Despite these improvements,
Although Transolver successfully reduces canonical com- putation complexity from O(N2) to O(M²) by learning physical states (M is a constant and usually set as 32 or 64 in practice), Transolver still face the following challenges when scaling input size to million-scale, i.e. N 2 10
Homogeneous physical states Eq. (1) shows that physical states are highly affected by slice weights w. If w tends to be uniform, attention in Eq. (2) will degenerate average pooling, losing the physics modeling capability. As shown in Figure 2(b), we find that Transolver may generate less distinguishable weights in some cases, especially in large- scale meshes, leading to homogeneous physical states.
Efficiency bottleneck Although Physics-Attention cost is nearly constant when scaling the input, the feedforward layer to embed million-scale points by x, will consume huge GPU memory, forming Transolver's stability bottleneck.
3

---

Transolver++: An Accurate Neural Solver for PDEs on Million-Scale Geometries
Slow-chogging region
Lineor
Transolver
FeedkForward
Sicee Weiges
Geoeiity Repoarttters
loyerhlorm
Deslice
Might
Broadest
Rep Slice
1200
Attention omong
Plysicndttattinn
Edetic States
Tramsllvrt++
with States
Tagg
Ado Temp
Bdatic Siice
Trencherr
LoyerNorn
Fast-changing
Linea
Geometry Representations
Region
(a) Physics with Eidetic States
(b) Slice Weights Visualization
[c) Physical Quantity Change Ratio
Figure 2. (a) Overall design of Transolver++ block. Blocks highlighted in red represent modifications compared to the original Transolver. (b) Visualization of slice weights. Transolver++ Iearns more diverse and eidetic physical states. (c) Visualizations of physical quantity change ratio (difference between each point and its neighbors) and slice weights Iearned by models. The lighter color means faster change.
4. Transolver++
distribution more concentrated on crucial states. Through a learnable linear projection layer, we can dynamically adjust the state distribution based on each point's local properties.
To tackle the scaling issues of Transolver in physics learning and computation efficiency, we present Transolver++, which can effectively avoid attention degeneration by learning eidetic physical states and successfully break the efficiency bottleneck with a highly optimized parallelism framework.
Slice reparameterization As mentioned above, we try to lear eidetic physical states and assign mesh points to physical states with possibility w. In the canonical design of Transolver, it uses Softmax to form categorical distribu- tion across states. However, simply generating a categorical distribution is not enough, as we have not completely mod- eled the assignment process from points to certain physical states. Considering that direct sampling via Argmax is non- differentiable, we propose to adopt the Gumbel-Softmax (Jang et al., 2017) to perform differentiable sampling from the discrete categorical distribution, which is accomplished with the following reparameterization technique:
4.1. Physics-Attention with Eidetic States
As aforementioned, if Physics-Attention is based on ho- mogeneous physical states, it will degenerate to average pooling, which will damage the model's performance. Thus, as the authors mentioned in Transolver (Wu et al., 2024), they adopt the softmax function in calculating slice weights in Eq. (1), which can alleviate the indistinguishable states to some extent by guaranteeing peakier distribution (Wu et al., 2022). However, we still observe the degeneration phenomenon when the model depth increases, as shown in Figure 2(b), which may result from the excessive focus on global information. Note that in industrial applications, the model's capability of learning subtle physics is essen- tial, which may highly affect the evaluation in industrial small diversion device
Rep-Slice(x, r) Softmax Linear(x) - -log(- log e)
(4)
where E N is the local adaptive temperature and {e)) E; ~ ((, 1). Here log(- e.) Gumbel (0, 1). where Gumbel is type of generalized extreme value dis- tribution. Replacing Transolver's slice weights w in Eq. (1) by our new design in Eq. (4), Transolver++ is able to learn eidetic physical states under complex geometries, offering the possibility to handle much larger-scale datasets.
design (Elrefaie et al., 2024), can drastically change the wind drag of driving cars.
To capture detailed physics phenomena, we propose to learn eidetic states by augmenting Transolver's state learning with a local-adaptive mechanism and slice reparameterization to carefully control the learned slice weight distribution.
As shown in Figure 2(c), the slice weights in Transolver++ can perfectly adapt the intricate physies fields on com- plex geometries. Specifically, regions with slow-changing physics quantities are assigned to one certain eidetic state as these areas are governed by one pure physical state. In con- trast, regions with fast-changing physics quantities exhibit a multimode distribution across physics states, reflecting that these areas are influenced by mixture of multiple states.
Local adaptive mechanism As shown in Figure 2(b), modeling the distribution of each mesh point using a non- parametric approach has been proven to be impractical (Ye et al., 2024). Thus, we introduce a local adaptive mech- anism that incorporates local information as pointwise adjustment the physics learning process. Specifically, we change the sharpness of the state distribution by learning to adjust the temperature To in the Softmax function, namely Ada-Temp: 7 = {T = Ioo + Lincar(x;)) (3)
4.2. Parallel Transolver++
where T E Nx1 and a higher temperature forms a more uniform distribution, while a lower temperature makes the
To break the GPU memory bottleneck caused by feedfor- ward layers of million-scale point representations, we design
4

---

Transolver+:: An Accurate Neural Solver for PDEs on Million-Scale Geometries
(a) Comparison with other parallel methods
(b) Scalability of Transferred Package
&GPU
#GPU
AGFU
#GPU-1
#GPU-1
#GPU-1
Tonal Transferred Padkage (GB)
(Oou))) DeepSpeedUllues
2.0
GPU
GPU
GPU
ring
Trersformer Block
Transforer Block
Transohvcrt Block
states
1.0
Transformer
Transformar
Transslvarme
Block
Black
Block
0.5
0.25MB
Trondformer
transferred
Transforer
transferred
Transover
transferred
Block
OIN'1
OIN)
g O[K)
6.0 18k 100k 2008 soos 4o0k soot eodk 70ck
ring attention
DeapSpeackUlysses
Transalver**
fimash points
Figure 3. (a) Comparison with other parallel methods. Tailored to the unique physics learning design, our method only communicates physical states with an all-reduce operation. (b) Scalability of the communication overhead to the number of mesh points with 32 GPUs. Our parallel method stands out by only transferring 0,25MB of data, which does not scale with the size of input mesh points.
highly optimized framework for PDE solving, which is based on the unique physics-leaming design of Transolver.
Algorithm Parallel Physics-Attention with Eidetic States
Input: Input features (x) E RNNxC on the k-th GPU, Output: Updated output features x((A) E RN
Parallelism formulation Through careful analysis, we ob- serve that in addition the point-wise feedforward layer, the computation of eidetic states within Physics-Attentionn can be efficiently distributed across multiple GPUs. Specifically, operations in Eq. (1), such as the weighted sum Σ WiyX; and the normalization denominator Σ Wij. can be easily dispatched to multiple GPUs and calculated separately in parallel. Thus, we first separate the input mesh into multiple GPUs for parallel computing and only communicate when calculating attention among eidetic states.
I/ drop f to save 50% memory.
Compute eift (A)) + Project(x(A))
Compute (8) + To + Ada-Temp( (x
Compute weights w (R) + Rep-SSiceex(tt ((k))
Compute weights norm w wnomm f 2i== w
Reduce slice norm norm AllReduce (w (8)
O(Ar)
Compute eidetic states s (R)
Woom
Reduce cidetic states s + AllReduce(s!))
O(MO)
Update eidetic states s' +- Attention(s) Deslice back to (*) Deslice(s', w (()) Return (k)
Suppose that the initial separation splits input mesh into #gpu GPUs and the integral representation x NxC is split into {x(2).... x # where x(k) xC and N& denotes the number of mesh points dispatched to the k-th GPU. Correspondingly, the point-wise slice weights w E N is also separated into #gpu compo- ments (w() ..... ( with ((k) E R.NeXM For clar- ity. we treat both multi-node moooee-uti--GUU and single-node- MuU configurations as a unified case. The calculation of eidetic states S, in Eq. (1) can be equivalently rewritten into the following parallelism formalization:
parameters in linear layers, tensor parallelism partitions the model parameters along the channel dimension. It reduces the memory consumption linearly but introduces an increase in communication overhead of O(N). Attention-optimized methods like RingAttention leverage the FlashAttention concept to distribute the outer loop using a ring topology, which results in O communication complexity, while DeepSpeed-Ulysee similar to tensor parallelism, partitions the data along feature dimensions and yields an O(N) com- munication volume. However, all of the above methods are not feasible when handling million-scale meshes as the com- munication overhead shown in Figure 3(b) is unacceptable.
(1)
. Ware (gguu) (pp))
1j
w
ij
11)
Σ Natar #epu)
1=1
(5)
where @ denotes the AllReduce operation (Patarasuk & Yuan, 2009), which aggregates the results from all processes. In practice, the k-th GPU first independently computes its partial sums for the numerator (E,,,
In parallel Transolver++, each GPU computes two partial sums of size (MCC) and (Ad) separately, which are then synchronized across GPUs and cause a total communication volume of O(#gpu x MCC + 1)), invariant to input size.
(k)) and de- (b)) of N& points. Next, these partial nominator (EN bf
x;
results are synchronized across GPUs to compute the eidetic states 8,, highlighted by blue-marked steps in Algorithm 1.
Further speedup Also, we found that Transolver's code implementation is somewhat pee-pparaeterized. Specifi- cally, Transolver projects the data onto both x and f, where x is used to generate slice weights, and f is combined with weights to generate physical states. This repetitive design
Overhead analysis Consider the input x E R NxC with N mesh points and C channels. To handle large model
5

---

Transolver+4: An Accurate Neural Solver for PDEs Million-Scale Geometries
[a]
Elasticity
Plasticity
Airfoil
Pipe
NS2d
Darey
Benchmorks
[b)
7,06
9:4g
0.76
9.7
11:00
Relative Error (1e2)
0.45
0.29
-10.2%
-1.99%
0.42
-18.8%
-12.3%
0.6-
0.00
110
-1.4%
-15.3%
0.48
0.52
0.49
0.52
GNOT
LNO
Tronsolvor
Trasoleer+1(Ov)
Figure 4. (a) Visualization of standard benchmarks covering wide range of physies scenarios, from solid physics to fluid dynamies. (b) Relative errors on standard benehmarks of the top-4 models selected based on overall performance. Full results can be found in Table 8,
Table 1. Summary of experimental datasets, where #Mesh denotes the size of mesh points each sample. Geo Type denotes the geometric structure of the mesh
approaches. These baselines include 12 neural opera- tors, such as Galerkin (2021), LNO (2024), and GINO (2023a), some of which are specifically designed for irreg- ular meshes; 4 Transformer-basee PDE solvers, including OFormer (2023c), FactFormer (2023d), GNOT (2023), and Transolver (2024); and 4 heaph-neural-network.
TYPE
BENCHMARKS
GEO TYPE
#MESH
NS2D
STRUCTURE
4,096
meth- ods: GraphSAGE (2017), PointNet (2017), Graph U-Net (2019), and MeshGraphNet (2021). Transolver is the previ- ous state-of-the-ar model. During experiments, we found that some neural operators designed for grids perform poorly for large-scale irregular meshes. Therefore, only report their performance on standard benchmark datasets.
PIPE DARCY
STRUCTURE
16,641
STANDARD BENCHMARKS
STRUCTURE
7,225
AIRFOIL
STRUCTURE
11,271
PLASTICITY
STRUCTURE
3,131
ELASTICITY
UNSTRUCTURE
972
INDUSTRIAL
AIRCRAFT
300K
UNSTRUCTURE
APPLICATIONS
DRIVAERNET++
2.5M
brings double memory costs. In this paper, we find that eliminating (marked gray in Algorithm 1) can simply and successfully increase the single-GPU input point capacity 1.2 million, without sacrificing the model's performance.
5.1. Standard Benchmarks
Setups As shown in Figure 4(a), compare the latest state-of-the-arr neural PDE solvers with Transolver++ on six standard datasets covering wide range of physics sce- narios. For a fair comparison, we keep all model parameters within a fixed range. Specifically, in Transolver++, we set the model's depth as eight layers, and the feature dimension is 128 or 256, depending on the scale of the data. The num- ber of slices chosen from {32, 64) to trade-off between computational cost and model performance.
5. Experiments
We extensively evaluate Transolver++ on six standard bench- marks and two industrial-level datasets with million-scall meshes, covering both physics and design-oriented metrics.
Benchmarks As summarized in Table 1, our experiments include both standard benchmarks and industrial simula- tions, which cover a broad range of mesh sizes. Specifi- cally, the standard benchmarks include Elasticity, Plasticity, Airfoil, Pipe, NS2d, and Darcy, which are widely used in previous studies (Wu et al., 2024). To further evaluate the model's efficacy in real applications, experiments on industrial design tasks, where utilized DrivAerNet++ (Elrefaie et al., 2024) for car design and newly simulated AirCraft dataset for 3D aircraft design. In addition to the error of predicted physics fields, we also measure the model performance for design by calculating drag and lift coefficients from predicted physics fields.
Results As presented in Figure 4(b), Transolver++ yields over 13% improvement w.r.t. the dataset-specifie second- best baseline averaged from all six standard benchmarks, showing the efficacy of our proposed methods in handling complex geometries. To highlight the comparison, we only present top-3 baselines and Transolver++ in Figure 4, which are chosen based on the overall performance. Full results for other baselines are provided in Table 8 of the Appendix.
also perform
It worth noticing that Transolver and Transolver++ signif- icantly outperform other models in Elasticity, whose geome- try is recorded as an unstructured point cloud. Going beyond Transolver, Transolver++ further boosts performance by learning eidetic physical states. As shown in Figure 2, Tran- solver++ can learn more diverse slice partitioning, thereby enabling more accurate physies learning. More analyses on
Baselines We widely compare Transolvert+ against more than 20 advanced baselines, covering various types of
6

---

Transolver++: An Accurate Neural Solver for PDEs Million-Scale Geometries
Table Comparison on large geometries benchmarks. Relative L2 the surrounding area (Volume) and surface (Swrf) physies as well as drag and lift coefficient (Cn. Ct.) is recorded, along with their coefficient of determination R' and H2. The closer H² is to 1, the better.
DRIVAERNET++ + FULL
DRIVAERNET++ SURF
AIRCRAFT
MODEL
VOLUME 4
SURF
CD
Rİ f
SURF
CL 4
SURF
GRAPHSAGE (2017)
0.328
0.284
0.282
0.859
0.294
0.040
0.988
0.109
POINTNET (2017)
0.285
0.478
0.301
0.831
0.237
0.095
0.982
0.169
GRAPH U-NET" (2019)
0.241
0.260
0,272
0.876
0.193
0.063
0.953
0.161
MESHGRAPHNET" (2021)
0.529
0.422
0.260
0.870
0.209
0.038
0.993
0.113
GNO" (2020A)
0.510
0.664
0,252
0.882
0.196
0.031
0.991
0.129
GALERKIN" (2021)
0.234
0.274
0.267
0.792
0.235
0.069
0.879
0.118
GEO-FNO" (2022)
0.718
0.892
0,288
0.831
0.291
0.243
0.903
0.395
GINO (2023A)
0.586
0.638
0.323
0.725
0.220
0.047
0.983
0.133
GNOT* (2023)
0.174
0.171
0.158
0.901
0.167
0.033
0.991
0.093
LNO" (2024)
0.180
0.203
0.208
0.855
0.195
0.091
0.992
0.137
3D-GEOCA" (2024)
0.389
0.224
0.205
0.883
0.175
0.022
0,993
0.097
TRANSOLVER" (2024)
0.173
0.167
0.061
0.931
0.145
0.037
0.994
0.092
TRANSOLVER++ (OURS)
0.154
0.146
0.036
0.997
0.110
0.014
0.999
0.064
RELATIVE PROMOTION
11.0%
12.6%
41.0%
24.1%
36.3%
30.4%
These models cannot directly handle million-scale meshes as the model input. Thus, enable comparison, split the input mesh of these models into several pieces, independently infer them, and concatenate separately inferred outputs as their final results.
Table 3. Data distribution across different car types in the Dri- vAerNet++ dataset. Each category is sampled in proportion to its original distribution both train set and test set. The other fea- tures, e.g. detailed smooth wheel, not under consideration.
Drag
Lift
DRIVAERNET+
TYPE
FASTBACK
ESTATERACK
NOTCHBACK
Wind
Wind
# TRAIN CASES
54
72
54
# TEST CASES
8
6
(a) DrivAerNet++
(b) AirCraft
TOTAL
60
80
60
Figure 5. Car and aircraft design predict drag and lift coefficient under extremely complex geometries with million-scale meshes.
ensures a fair and balanced evaluation. The detailed sam- pling statistics are presented in Table
learned physical states can also be found in Appendix C.
Table demonstrates that Transolver++ achieved consistent state-of-the-art in all datasets with an average promotion of over 20%. In Driv AerNettt, our model is capable of handling 2.5 million meshes within 4 A100 GPUs and surpassing all other models by relative promotion of 1 100% and 12.6% volume and surface field separately. In the surface field, our model still shows a prominent lead of 24. 1% in Driv-erNet++ Surface and 30.4% in AirCraft.
Results
5.2. PDEs on Large Geometries
Setups To evaluate the performance in practical applica- tions, conduct experiments two industrial datasets with million-scale meshes: DrivAerNet++ (Elrefaie al.,, 2024) and Aircraft. The latter is newly presented by which is of high quality and simulated by aerodynamicists. To fully evaluate the model performance under different scales, we split DrivAerNet++ into two scales, one only with surface pressure (~700k mesh points in each sample) and the other with full physies fields (2.5M mesh points).
We also observe that GNNs often degenerate quickly when handling large-scale meshes due to their geometric insta- bility (Klabunde & Lemmerich, 2023). Additionally, Geo- FNO also performs poorly across nearly all datasets, attempts to map irregular meshes to uniform latent grids, task that becomes especially difficult when the number of meshes scales up. These failures of our baselines further highlight the challenge of physics learning on million-scall geometries, while Transolver++ provides a practical solu- tion, advancing an essential step to industrial applications.
In DrivAerNet++, the dataset is categorized into several distinct car types, including fastback, estateback, and notch- back. To better demonstrate the generalizability of our pro- posed method across different geometries, we construct a representative subset by sampling from each category in proportion to its original distribution the full dataset and

---

Transolver++: An Accurate Neural Solver for PDEs Million-Scale Geometries
[b.1] AirCraft Pressure Coefficiant
[c]] Elficeecy Analysa
Transolves veo
Running Timels)
8.0
%
Tranachvrt* Transolver
GNOT
[b.2) DrivAernett Surface Pressure
(0cl)
INC
Dilerc Se
[c.2] Why we need large geometries
Transolvert+
200
Transolvert+ Error
Transolver Error
GNOT Error
250.
(a) Visualization of Slices Distribution
(b) Case Study of Industrial Datasets
(c) Statistics Analysis
Figure (a) Visualization of slice weight distributions. Transolver++ demonstrates more diverse pattern than Transolver. (b) Error map of t3p-3 models on DrivAerNet++ and AirCraft. Transolver++ outperforms other models and captures some subtle variations. (c) Statistics analysis of efficiency in terms running time and model performance under different scales of input geometries.
Table Performance comparison Transolver++ under patch and unpatch settings on the AirCraft dataset. Cz, Rī and relative error on surface are all recorded. The hest result for each metric is highlighted in bold, while the second- becond-bestae is underlined.
Table 5. Ablations AirCraft. Relative 1.2 and R of lift coeffi- cient are both recorded. Ada-Temp refers to adaptive mechanism in Eq. (3), Reparameter represents the gumbel-softmax softmax operation with reparameterization trick for Eq. (4) and Speedup represents removing in Algorithm for accelerationl.
AIRCRAFT RE
MODEL
CL
SURF
AIRCRAFT R2.
METHOD
3D-GEOCA
0.022
0.993
0.097
CL. 4
SURF
TRANSOLVER
0.037
0.994
0.092
TRANSOLVER
0.037
0.994
0.092
TRANSOLVER++ (PATCH)
0.017
0.997
0.072
+ ADA-TEMP
0.020
0.995
0.080
TRANSOLVER++ (UNPATCH)
0.014
0.999
0.064
+ ADA-TEME SPEEDUP
0.018
0.995
0,075
+ ADA-TEMP, REPARAMETER
0.016
0.998
0.069
TRANSOVLER++ (OURS)
0.014
0.999
0.064
To ensure fair and consistent comparison with existing methods, we additionally evaluate our model under the same pasch-based input setting footnoted in Table 2, which is primarily adopted in other models to address memory constraints during training and inference phase. Specifically, we conduct experiments under both settings to assess the impact of spatial partitioning on model performance. As shown in Table 4, our model achieves better results in the unpatched setting, indicating that the parallelism framework effectively enhances overall performance with full data in- put. Notably, even when operating under the same patch setting, our model still outperforms other baselines, further validating the robustness and scalability of our approach across different computational settings and input scale.
Ablations We conducted elaborative ablations on every component of Transolver+++ As shown in Table 5, intro- ducing the local adaptive mechanism significantly improves performance, which validates our motivation to learn eidetic states. With speed-up optimization and reparameterization techniques, the model achieves its best performance, fully demonstrating the effectiveness of our proposed design.
Slice Analysis As shown in Figure 6(a), Transolver++ can extract more diverse physical states than Transolver in million-scale meshes, enabling fine modeling for complex physics fields of driving cars. More visualization can be found in Appendix C. Moreover, in Table 6, also calcu- late the KL divergence between learned slice weights and uniform distribution in different layers. These statistical re- sults further demonstrate that Transolver++ can learn more diverse and varying slice distribution across all the layers.
As shown in Figure 6(b), our model has a lower relative error in most cases and excels in capturing the intrinsic physics variations in those regions with drastic changes, while other models tend to generate an over-smooth prediction, validat- ing the effectiveness of learning cidetic states.
Efficiency analysis In addition to GPU memory (Fig- ure 1), we also measured the running time of different mod- els on DrivAerNete datasets in Figure (c.1). To ensure a fair comparison, all models' parameter sizes are well aligned in our experiments and we only compare Transolver++ with
5.3. Model Analysis
In addition to model comparisons, we also conduct a series of analysis experiments to provide an in-depth understand- ing of our model and the necessity of large geometries.
8

---

Transolver+:: An Accurate Neural Solver for PDEs on Million-Seale Geometries
(o) Data
(b) Parameters
Table -ll-dieegeeece between learned slice weights and uniform distribution in different layers on Elasticity. We choose 3, 7} layers and a higher value means more diverse slice distribution generated by our method.
0.30
0.064
0.25
0.063
Relative
0.20
0.062
LAYER NUMBER
MODEL
AVG
0.10
3
5
7
0.061
20
100 140
12
16
TRANSOLVER
0.056
5.599
3.474
0.383
3.885
#troining cases
#layers
TRANSOLVER++
2.122
8.193
5.583
1.473
5.297
Figure 7. Evaluation of the model scalability in terms of data size and parameter size. Data size refers to the number of training cases, while parameter size is controlled by changing the number of network layers. Our default setting is 150 cases and layers.
Transformer-biie methods here, since GNNs struggle to handle million-scan meshes. We can find that Transolver++ strikes favorable balance between performance and effi- ciency, outperforming most models in terms of speed. More- over, at the same size of input mesh points, our method ex- hibits the lowest memory usage, highlighting its efficiency in handling large-scale data without sacrificing accuracy.
We evaluate the scalability of Transolver++ across different numbers of training samples and model pa- rameters of different sizes by altering the number of layers. From Figure 7, we can find that our model can consistently benefit from large data and large models, revealing its poten- tial to be the backbone of PDE-solving foundation models.
Scalability
As shown in Table 7, further highlight the advantages of deep learning-based approaches over traditional solvers, compare our model with traditional solvers, Un- sCFD(McKenna et al., 1998), on the AirCraft dataset. De- spite the time required for the entire training process, our model still proves to be more efficient when handling multi- ple simulation cases and consistently achieves better perfor- mance than traditional methods. Specifically, once trained, the model can generate high-fidelity predictions for new simulation cases within one second, bypassing the iterative convergence procedures inherent to traditional methods.
6. Conclusion
In pursuit of practical neural solvers, this paper presents Transolver++, which enables the first success in accu- rately solving PDEs discretized on million-scale geometries. Specifically, we upgrade the vanilla Transolver by introduc- ing eidetic states and highly optimized parallel framework, empowering Transolver++ with better physics learning and computation efficiency. As a result, our model achieves significant advancement industrial design tasks, demon- strating favorable efficiency and scalability, which can serve as a neat backbone of PDE-solving foundation models.
Table Performance comparison of different models on the Air- Craft dataset. Tase denotes the time consumption per inference case, Tran represents the total training time, and Surf indicates the relative surface prediction error. mcans that this model does not involve a training process and has no associated training time.
Impact Statement
This paper aims at advancing neural PDE solvers to indus- trial applications. Building on the previous advancement of Transolver, we present Transolver++ as an accurate neural solver for PDEs on million-scale geometries, which achieves consistent state-of-the-an the- art in both standard benchmarks and industrial-level tasks with significant promotion. Also, Tran- solver++ pushes the single-GP GPU input size capacity to the million scale for the first time and achieves linear scalability with resources, showing favorable practicability to indus- trial manufacturing. Our work only focuses on the scientific problem, so we believe there is no potential ethical risk.
AIRCRAFT
MODEL
Teasu J
Timas&
SURFI
UNSCFD
5-6 HOURS SECOND
0.173
TRANSOLVER++
10 HOURS
0.064
Why we need large geometries From the car mesh com- parison in Figure 1(b), we can observe that large geome- tries differ significantly from smaller-scale ones in terms of details, which directly affects the accuracy of the simula- tion. While previous models (Li al., 2021) claim to be isolution-invariane and aim apply directly to large-scale geometries, our experiments in Figure 6 (c.2) show that without training on large meshes, the model will fall short in the fine-gran -raiiid physics modeling, resulting in an insuffi- cient and limited performance, which further necessitates the capability of handling larger geometries.
A Acknowledgements
This work was supported by the National Natural Science Foundation of China (U2342217 and 62021002), the BNRist Innovation Fund (BNR2024RC01010, and the National Engineering Research Center for Big Data Software

---

Transolver++: An Accurate Neural Solver for PDEs on Million-Scale Geometries
References
Kopriva, D. A. Implementing spectral methods for par- tial differential equations: Algorithms for scientists and engineers. Springer Science & Business Media, 2009.
Cao. S. Choose transformer: Fourier or galerkin. In NeurlPS, 2021.
Kovachki, N., Li, Z., Liu, B., Azizzadenesheli, K., Bhat- tacharya, K., Stuart, A., and Anandkumar, A. Neural operator: Learning maps between function spaces with applications to pdes. JMLR, 2023.
Choromanski, K., Likhosherstov, V., Dohan, D., Song, X., Gane, A., Sarlós, T., Hawkins, P., Davis, J., Mohiuddin, A., Kaiser, L., Belanger, D., Colwell, L. J., and Weller, A. Rethinking attention with performers. ICLR, 2021.
Li, Z., Kovachki, N., Azizzadenesheli, K., Liu, B., Bhat- tacharya, K., Stuart, A., and Anandkumar, A. Neural operator: Graph kernel network for partial differential equations. arXiv preprint (0Xi::2003003885, 2020a.
Dao. T., Fu, D., Ermon, S., Rudra, A., and Ré, C. Flashat- tention: Fast and memory-efficien exact attention with io-awareness. NeurlPS, 2022.
Deng, J., Li, X., Xiong, H., Hu, X., and Ma, J. Geometry- guided conditional adaption for surrogate models of large- scale 3d PDEs on arbitrary geometries. In IJCAI, 2024.
Li, Z., Kovachki, N. B., Azizzadeneshelii K., liu, B., Bhat- tacharya, K., Stuart, A., and Anandkumar, A. Fourier neural operator for parametric partial differential equa- tions. In ICI.R, 2021.
Elrefaie, M., Morar, F., Dai, A., and Ahmed, Drivaer- net++: A lcage-scale multimodal car dataset with com- putational fluid dynamics simulations and deep learning benchmarks. arXiv preprint urXiv:2406.09622 2024.
Li, 7.,, Kovachki, N. B., Choy, C., Li, B., Kossaifi, J., Otta, S. P., Nabian, M. A., Stadler, M., Hundt, C., Azizzade- nesheli, K., and Anandkumar, A. Geometry-informed neural operator for large-scale 3d PDEs. In NeurlPS, 2023a.
Evans, L. C. Partial differential equations. American Math- ematical Soc., 2010.
Gao, H. and Ji. S. Graph u-nets. In ICML, 2019.
Li, 7.,, Kovachki, N. B., Choy, C., Li, B., Kossaifi, J., Otta, S. P., Nabian, M. A., Stadler, M., Hundt, C., Azizzade- nesheli, K., et al. Geometry-informed neural operator for large-scale 3d pdes. arXiv preprint arXiv:2309.00588 2023b.
Grossmann, C., Roos, H.-G., and Stynes, M. Numerical treatment of partial differential equations. Springer, 2007.
Hamilton, W., Ying, Z., and Leskovec, J. Inductive repre- sentation learning on large graphs. NeurIPS, 2017.
Li, Z., Meidani, K., and Farimani, A. B. Transformer for partial differential equations' operator learning. TMLR, 2023c.
Hao, Z., Ying, C., Wang, Z., Su, H., Dong, Y., Liu, S., Cheng, Z., Zhu, J., and Song, Gnot: general neural operator transformer for operator learning. ICML., 2023.
Li, Z., Shu, D., and Farimani, A. B. Scalable transformer for pde surrogate modeling. NeurlPS, 2023d.
Huang. Y., Cheng, Y.. Bapna, A., Firat, O., Chen, D., Chen, M., Lee, H., Ngiam, J., Le, Q. V., Wu, Y., et al. Gpipe: Efficient training of giant neural networks using pipeline parallelism. NeurlPS, 2019.
Li, Z.-Y., Kovachki, N. B., Avizzaaenesheli, K., Liu, B., Bhattacharya, K., Stuart, A., and Anandkumar, A. Neural operator: Graph kernel network for partial differential equations. arXiv preprint arXiv:2003.0344 03400000.035.... 2020b.
Jacobs, S. A., Tanaka, M., Zhang, C., Zhang, M., Song, S. I.., Rajbhandari, S., and He, Y. Deepspeed ulysses: System optimizations for enabling training of extreme long sequence transformer models. arXiv preprint arXiv:2309. 14509, 2023.
Li, Z.-Y.. Huang, D. Z.. Liu, B.. and Anandkumar, A. Fourier neural operator with learned deforma- tions for pdes on general geometries. arXiv preprint arXi::2207.05209 2022.
Jang, E., Gu, S., and Poole, B. Categorical reparameteriza-- tion with gumbel-softmax. ICLR, 2017.
Liu, H., Zaharia, M., and Abbeel, P. Ring attention with blockwise transformers for near-infinite context. arXiv preprint arXiv:231000/889 2023.
Kingma, D. and Ba, J. Adam: A method for stochastic optimization. In ICLR, 2015.
Kitaev, N., Kaiser, L., and Levskaya, A. Reformer: The efficient transformer. In ICLR, 2020.
Liu, X., Xu, B., and Zhang. L. HT-net: Hierarchical transformer based operator learning model for multiscale PDEs. arXiv preprini arXiv:22 22. 10890, 2022.
Klabunde, M. and Lemmerich, F. On the prediction instabil- ity of graph neural networks. In Machine Learning and Knowledge Discovery in Databases, Cham, 2023.
Loshchilov, and Hutter, Decoupled weight decay regu- larization. ICLR, 2019.
10

---

Transolver++:: An Accurate Neural Solver for PDEs Million-Scale Geometries
McKenna, T., Cokljat, D., and Wild, P. Cfd modelling of heat transfer during gas phase olefin polymerisation. Computers Chemical Engineering, 22:$285-S292, 1998.
Wang, T. and Wang, C. Latent neural operator for solving forward and inverse pde problems. In NeurlPS, 2024.
Wazwaz, A. M. Partial differential equations methods and applications. 2002.
Morris, E., Shen, H., Du, W., Sajjad, M. H., and Shi, B. Geometric instability of graph neural networks large graphs. arXiv preprinl urXiv:2308. 10099, 2023.
Wen, G., Li, Z.,, Azizzadenesheli, K., Anandkumar, A., and Benson, S. M. U-fno-an enhanced fourier neural operator- based deep-learning model for multiphase flow. Advances Water Resources, 2022.
Patarasuk, P. and Yuan, X. Bandwidth optimal all-reduce al- gorithms for clusters of workstations. Journal of Parallel and Distributed Computing, 2009.
Wu, H., Wu, J., Xu, J., Wang. J., and Long, M. Flowformer: Linearizing transformers with conservation flows. In ICML, 2022.
Peterson, E. K-nearest neighbor. Scholarpedia, 2009.
Pfaff, T., Fortunato, M., Gonzaez-Gozzaezz A., and Battaglia, P. Learning mesh-based simulation with graph networks. In ICLR, 2021.
Wu, H., Hu, T., Luo, H., Wang. J., and Long, M. Solving high-dimensional pdes with latent spectral models. In ICML, 2023.
Qi, R., Su, H., Mo, K., and Guibas, L. Pointnet: Deep learning on point sets for 3d classification and segmenta- tion. In CVPR, 2017.
Wu, H., Luo, H., Wang, H., Wang, J., and Long, M. Tran- solver: A fast transformer solver for pdes on general geometries, In ICML, 2024.
Qin, Z., Han, X., Sun, W., Li, D., Kong, L., Barnes, N., and Zhong, Y. The devil in linear transformer. arXiv preprint arXiv:2210.10340. 2022.
Xiao, Z., Hao, Z., Lin, B., Deng, Z., and Su, H. Improved operator learning by orthogonal attention. In ICML, 2024.
Rahman, M. A., Ross, Z. E., and Azizzadenesheli, K. U-no: U-shaped neural operators. TMLR, 2023.
Ye, T., Dong, L., Xia, Y., Sun, Y., Zhu, Y., Huang, G., and
Wei, F. Differential transformer, 2024. URL https: //arxiv. org abs 2410. 05258.
Ronneberger, O., Fischer, P., and Brox, T. U-net: Convolu- tional networks for biomedical image segmentation. In MICCAI, 2015.
Zhuang, Y., Zheng, L., Li, Z., Xing, E., Ho, Q. Gonzalez, J., Stoica, I., Zhang, H., and Zhao, H. On optimizing the communication model parallelism. Proceedings of Machine Learning and Systems, 5, 2023.
Roublček, T. Nonlinear partial differential equations with applications. Springer Science Business Media, 2013.
Shoeybi, M., Patwary, M., Puri, R., LeGresley, P., Casper, J., and Catanzaro, B. Megatron-Im: Training multi- billion parameter language models using model paral- lelism. arXiv preprint arXiv: 08053, 2019.
Solanki, K., Daniewicz, S., and Newman Jr, Finite ele- ment modeling of plasticity-induce crack closure with emphasis on geometry and mesh refinement effects. En- gineering Fracture Mechanics, 2003.
Solín, P. Purtial differential equations and the finite element method. John Wiley & Sons, 2005.
Tran, A., Mathews, A., Xie, L., and Ong, C. S. Factorized fourier neural operators. In ICLR, 2023.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, I.,, Gomez, A. N., Kaiser, L.. u. and Polosukhin, Atten tion all you need. NeurIPS, 2017.
Wang, H., Fu, T., Du, Y., Gao, W., Huang, K., Liu, Z., Chandak, P., Liu, S., Van Katwyk, P., Deac, A., et al. Scientific discovery in the age of artificial intelligence. Nature, 2023.

---

Transolver++: An Accurate Neural Solver for PDEs Million-Scale Geometries
A. Full Results on Standard Benchmarks
Due to the space limitation of the main text, we present the full results of model performance on standard benchmarks here, as supplement Figure As shown in Table 8. Transolver++ demonstrates superior performance across six standard PDE benchmarks, achieving the lowest relative L2 error in all PDE-solving tasks with averaged relative promotion of 13%.
For models marked with asterisk (*), we carefully reproduced the results by running the models more than three times and enabled fair and reliable comparison by maintaining the model parameter counts within closely comparable range with other models. Specifically, we adjust LNO and reduce its default dimension from 256 to 192 in elasticity, as its parameter count already three times larger than the other largest models. As discussed in previous studies (Wu et al., 2024), Transformer- basedd models usually benefit from large parameter sizes. This approach minimizes the potential influence of parameter variations and highlights the performance improvements achieved by our method under similar computation costs.
Table 8. Model performance on six standard PDE benchmarks is evaluated using the relative 1.2 error. The result marked with asterisk (*) indicates a reproduced outcome, where the parameter counts and configurations a fair comparison. "f" means that the baseline cannot apply to this benchmark.
haseline methods are carefully aligned to ensure
RELATIVE
MODEL
ELASTICITY
PLASTICITY
AIRFOIL
PIPE
NS2D
DARCY
FNO (2021)
/
0,1556
0.0108
U-FNO (2022)
0.0239
0,0039
0,0269
0.0056
0,2231
0.0183
GEO-FNO (2022)
0.0229
0.0074
0.0138
0.0067
0.1556
0.0108
U-NO (2023)
0.0258
0,0034
0.0078
0,0100
0.1713
0.0113
F-FNO (2023)
0.0263
0,0047
0.0078
0.0070
0,2322
0.0077
LSM (2023)
0.0218
0.0025
0,0059
0.0050
0,1535
0.0065
LNO* (2024)
0.0069
0.0029
0.0053
0.0031
0.0830
0.0063
GALERKIN (2021)
0.0240
0,0120
0,0118
0.0098
0,1401
0.0084
HT-NET (2022)
/
0,0333
0.0065
0.0059
0.1847
0.0079
OFORMER (2023C)
0.0183
0.0017
0.0183
0.0168
0.1705
0.0124
GNOT (2023)
0.0086
0.0336
0.0076
0.0047
0.1380
0.0105
FACTFORMER (2023D)
/
0,0312
0.0071
0.0060
0.1214
0.0109
ONO (2024)
0.0118
0,0048
0,0061
0.0052
0.1195
0.0076
TRANSOLVER (2024)
0.0064
0.0013
0.0053
0.0033
0.0900
0,0058
TRANSOLVER++ (OURS)
0.0052
0.0011
0.0048
0.0027
0.0719
0.0049
RELATIVE PROMOTION
18.8%
15.3%
10.2%
12.9%
13.4%
12.3%
B. Implementation Details
In this section, we provide detailed descriptions the benchmarks, baseline methods, and implementation setups to ensure reproducibility and facilitate comparisons.
B.1. Benchmarks
We evaluate our method on six standard PDE benchmarks, including Elasticity, Plasticity, Airfoil, Pipe. NS2D, and Darcy, as well two industrial datasets, Der and AirCraft. These benchmarks cover a wide range of physics simulation tasks, varying in complexity and geometry, and serve comprehensive benchmarks for assessing the effectiveness of neural PDE solvers shown in Table 9. Here the details of these datasets.
This benchmark generated by the simulations of the stress field in a hyper-elastic solid body under tension, which is governed by stress-straie relationship using the Sivlin-Saunders material model (Li et al., 2022). Each case involves unit cell of 972 points with a void in the middle clamped at the bottom edge and subjected to tension on the top. Elasticity contains a total of 1200 samples, with 1000 for training and 200 for testing.
Elasticity
Plasticity This benchmark is generated by the simulation of a plastic forging problem. where block is impacted by frictionless die moving at a constant speed (Li et al., 2022). An elastoplastic constitutive model is adopted to model this physical system with 900 training and 80 testing samples. The input the external force on every mesh with the shape of 101 x 31, and the output the dime-dependent deformation and mesh grid over 20 timesteps.
12

---

Transolver+:: An Accurate Neural Solver for PDEs on Million-Seale Geometries
Table Details of different benchmarks, including geometric type, number of mesh points, as well as the type of input and output, etc. The split of the dataset is also provided to ensure reproducibility, which is listed in the order of (training samples, test samples).
TYPE
BENCHMARK
#DIM
#MESHES
#INPUT
#OUTPUT
SPLIT
ELASTICITY
2D
972
STRUCTURE
INNER STRESS
(1000, 200)
PLASTICITY
2D TIME
3131
EXTERNAL FORCE
MESH DISPLACEMENT
(900, 80)
STANDARD BENCHMARK
AIRFOIL
2D
11271
STRUCTURE
MACH NUMBER
(1000, 200)
PIPE
2D
16641
STRUCTURE
VELOCITY
(1000, 200)
NAVIER-STOKER DARCY
2D + TIME
4096
VELOCITY
VELOCITY
(1000, 200)
2D
7225
POROUS MEDIUM
PRESSURE
(1000, 200)
INDUSTRIAL APPLICATIONS
DRIVÄERNET++
3D
~700K
STRUCTURE
SURFACE PRESSURE
(190. 10)
~..5
STRUCTURE STRUCTURE
PRESSURE & VELOCITY
(190. 10)
AIRCRAFT
3D
~300K
QUANTITIES
(140, 10)
Airfoil This benchmark is generated from simulations of transonic flow over an airfoil, governed by Euler's equations (Li
al., 2022). The whole field is discretized to unstructured meshes in the shape of 221 x 51 the input and the output is the corresponding Mach number on these meshes. The dataset includes 1000 training samples and 200 test samples that are based on the initial NACA-0012 shape.
Pipe This benchmark consists of simulations of incompressible flow in pipe, governed by the Navier-Stokes equation with viscosity / = 0.005 (Li et al., 2022). The pipe has length of 10 and a width of 1, with its centerline parameterized by cubic polynomials determined by five control nodes. The dataset also contains 1000 training and 200 testing samples, whose inputs are the mesh point locations 129 x 129 and outputs are the horizontal velocity field.
Navier-Stokes This benchmark consists of simulations of the 2D Navier-Stoker equations in vorticity form on the unit torus (0, 1)2 (Li et al., 2021). The objective is, given the past velocity, to predict future velocity for 10 steps on discretized meshes in the shape of 64 x 64. The inputs are the velocity for the past 10 steps, while the outputs provide the future velocity for 10 timesteps. The dataset also includes 1000 samples for training and 200 for testing.
Darcy This benchmark consists of simulations of the steady-state Darey Flow in two dimensions, governed by second- order elliptic equation on the unit square (Li et al., 2022). The input is the structure of the porous medium and the output is the corresponding fluid pressure. The dataset contains 1000 training samples and 200 testing samples, generated using a second-order finite difference scheme on 421 x 421 uniform grids and later downsampled to 85 x 85.
DriyAerNet++ This benchmark (Elrefaie et al., 2024) is large and comprehensive dataset for aerodynamic car design, featuring high-fidelity computational fluid dynamics (CFD) simulations. It includes over 8,000 car designs with various configurations of wheel and underbody design. To ensure efficiency while maintaining diversity, we select 200 representative cases from these designs. Notably, in our experiments, DrivAerNet++ is then divided into two subsets with different levels of resolution. The first subset, as shown in Table 9, only consists of surface meshes, where each mesh point is characterized by its 3D position (Ir, y, v),,, surface normal vector (uz, Ny uz)). and signed distance function (SDF). The output for this subset is the surface pressure on these meshes. The second subset provides a full 3D pressure and velocity field, significantly increasing the dataset's 's complexity, with the number of mesh points reaching approximately 2.5 million and requiring the model to predict surface pressure, velocity and pressure of the surrounding area. The dataset consists of 190 training samples and 10 test samples, offering a rigorous benchmark for evaluating aerodynamic modeling at different levels of fidelity.
This benchmark includes simulations of over 30 aircraft designs under 5 different incoming flow conditions, varying in Mach number, angle of attack, and sideslip angle. Unlike commonly used aerodynamics datasets such as Airfoil, the AirCraft dataset discretizes each aircraft into approximately 300,000 3D mesh points, offering significantly higher resolution for capturing complex aerodynamic phenomena. Each mesh point is characterized by its spatial coordinates (xr, y, and surface normal vector, serving as the input. These high-fidelity computational fluid dynamics (CFD) simulations, conducted by aerodynamicists in an aircraft design institution, ensure precise and realistic aerodynamic modeling. The dataset requires predicting six key physical quantities: pressure coefficient Cp., fluid density p. velocity components (u, v, w) and pressure p. With 140 training cases and 10 test cases, this dataset presents intricate aerodynamic interactions, making it a rigorous benchmark for assessing model scalability and accuracy.
AirCraft
13

---

Transolver++: An Accurate Neural Solver for PDEs Million-Scale Geometries
B.2. Metrics
To comprehensively evaluate model performance across different datasets and prediction tasks, we adopt relative L2 error as the primary evaluation metric. For large-scale datasets, evaluate both field and coefficient predictions separately. And further introduce R-squared (R²) score an additional metric assess the accuracy of coefficient predictions. In this section, we would explain how these metrics are applied to different dataset categories in detail.
B.2.1. STANDARD BENCHMARKS
For standard benchmark datasets, model performance is assessed using the relative L2 error, which directly measures the difference between the predicted output and the ground truth. Given an output field ý predicted by the model and the ground truth y, the relative L2 error is computed as:
11 ylle lyy
Relative L.2
(6)
This metric standard benchmarks is reported in the Table 8, providing direct comparison across different models.
B.2.2. LARGE-SCARR DATASETS
Field and Coefficient Errors For large-scale datasets, we decompose the relative L2 error into two components to gain deeper insights into model performance. Since large-scale datasets often involve both surface and volume (refers to the surrounding area) data, we separately compute the relative 1.2 error for surface fields and volume fields. This distinction allows us to evaluate how well the model captures different types of physical information. In addition predicting field values, some datasets require models to infer physical coefficients that characterize the feature of the system. The relative 1.2 error is also applied to these coefficients to measure prediction accuracy.
For example, in AirCraft, lift coefficient is calculated to measure the aerodynamic lift force on the body of an aircraft, which a key metric to assess the lift performance of an aircraft under a certain flow condition. The lift coefficient is defined as:
1 (-mm-i+ 1n--i+rn--- .iddddds dd. C7 - 100 A S (-mm pm i+
where p is the pressure field on the surface, n is the unit normal vector of the surface, i is the unit vector the lift direction, the shear stress tensor and S is the surface of the aircraft.
Also in car design, drag coefficient is crucial metric quantify the aerodynamic drag force the body of vehicle, which can be used improve fuel efficiency and the performance of vehicles. The drag coefficient is defined as:
-D=TARA A (-pm- d
1 (- rn d) ds,
Cp =
pu
where is the surface pressure, is the unit normal vector of the surface, d is the unit vector in the drag direction, T is the shear stress tensor and S is the surface of the vehicle.
R-squared Score for Coefficient Predictions To further evaluate the model's ability to predict physical coefficients, we introduce the R-squared (Ri2) score as an additional metric. The R²2 score is computed as:
R - E, - 0.)2 E, 7)2
(9)
where ÿü is the i-th of the predicted coefficients, is the i-th ground truth, and ÿ is the mean of the ground truth values. An R2 score closer to indicates better model performance, while lower values suggest less accurate predictions. This metric measures how well the model learns the physical quantitiy fields among all samples.
B.3. Baselines and Implementations
We conduct extensive comparisons between Transolver++ and over 20 ofate-of-the-eeee art baselines, encompassing a diverse range of methods for solving partial differential equations (PDEs). These baselines include typical neural operators,
14

---

Transolver++: An Accurate Neural Solver for PDEs Million-Scale Geometries
Table 10. Implementation details of Transolver++ including training and model configuration. Training configurations are identical to previous methods (Wu al., 2024; Hao al., 2023; Deng al., 2024; Elrefaic al., 2024) and shared all baselines. Ly and Lg refer to the loss on volume (surrounding area) and surface physics fields respectively.
TRAINING CONFIGURATION (SHARED IN ALL BASELINES)
MODEL CONFIGURATION
BENCHMARKS
Loss
EPOCHS INITIAL LR OPTIMIZER BATCH SIZE LAYERS L HEADS CHANNELS C SLICES M
ELASTICITY
128
64
PLASTICITY
128
64
AIRFOIL
RELATIVE
500
10-3
ADAMW (2019)
8
128
64
PIPE
L2
128
64
NAVIER- STOKES
8
256
32
DARCY
128
64
DRIVÄERNET++ FULL
Lu + L.
ADAM (2015)
DRIVAERNET++ SURF
L. Lv + L.
200
10-3
8
256
32
AIRCRAFT
Transformer- PDE solvers, and graph neural networks (GNNs) and tested under the same training configurations as shown in Table 10. To ensure a rigorous comparison, we obtain the open-source implementations of these models and carefully verify their consistency with the original papers before training.
B.3.1. STANDARD BENCHMARKS
We try to be as loyal to the original settings of baselines as possible. However, when the model parameters count is too large to conduct a fair comparison, we would change either the number of blocks or the hidden dimension to ensure fair comparison. As we mentioned before, Transformer- based models usually benefit from large parameter sizes (Wu et al., 2024). Thus, the alignment of parameter size is essential to control the variable and highlight the performance difference caused by architecture design. Here are detailed implementations.
Specifically, we adjust LNO and reduce its default dimension from 256 to 192 in elasticity, as its parameter count is already three times larger than the other largest models, while other models' settings remain unchanged.
Besides, we carefully review the original papers of these baseline models to make sure that they are fully tuned on their syper-parameters. For all the models whose data settings align exactly with ours, such FNO (Li et al., 2021). Geo-FNO (Li al., 2022), GNOT (Hao al., 2023), OFormer (Li et al., 2023c), Transolver (Wu al., 2024), directly adopt the results reported in their respective publications. For other models, refer to the results presented in LSM (Wu et al., 2023) and vra (Wu et al., 2024), as these works have conducted comprehensive and rigorous hyper-parameter tuning to ensure fair and reliable comparison.
B.3.2. INDUSTRIAL APPLICATIONS
For AerNAttt and AirCraft datasets, most the baseline models are unable to handle million-scale meshes directly, To address this, we subsample the dataset to 50k meshes point and reconstruct the meshes using K-Nearest Neighbors (KNN) (Peterson, 2009) method to preserve essential geometric relationships.
Specifically during training, apply random subsampling strategy at the beginning of each epoch and reconstruct meshes for only once in each case to maintain consistency of the total step count compared to other baselines capable of handling million-scale meshes. For testing, we perform a complete subsampling operation, obtaining multiple partial predictions across different subsampled sets. These predictions are then aggregated and concatenated to reconstruct the full output for each test ease. The relative 1.2 error is subsequently computed on the concatenated predictions, and the final results are reported in Table Furthermore, lift and drag coefficient are also computed based on the reconstructed outputs following their respective formulations in Eq. (8) and Eq. (9).
In terms of model configurations, each model has been extensively tuned on their hyperparameters, especially graph neural networks (GNNs) for their extreme instability (Klabunde & Lemmerich, 2023) when dealing with million-scale meshes. To minimize human bias, we employ a standardized tuning strategy by systematically changing the hidden dimension from {64, 128, 256, 512). the number layers from (2, 4, 6, 8, 10) along with tuning model-specific hyperparameters. During this process, we observe that almost all GNNs experienced geometric instability (Morris et al., 2023) when applied the
15

---

Transolver++:: An Accurate Neural Solver for PDEs Million-Scale Geometries
DrivAerNet++ Full dataset with approximately 2.5 million mesh points per case. Out of all baselines, Transolver achieves the second-best performance, benefiting from its Slice-Desliie mechanism, which improves its stability and scalability, while Transolver++ surpasses it by introducing eidetic physical states and parallelism framework that can directly handle million-scale meshes with enhanced efficiency. In Transolver++, we set the number of slices to 32 and its channels to 256 with 4 layers of Transolver++ block to balance efficieney and performance. And we only need to adopt parallelism to Transolver++ for the DrivAerNet++ Full benchamrk on only 4 A100 GPUs at most.
C. More Visualizations
As supplement to Figure 6 of the main text, in this section, will provide detailed visualization of the eidetic physical states of Transolver+++ well as case study showcases different datasets.
C.1. Eidetic States Visualization
Here, for simplicity and clarity, present visualizations of the eidetic physical states on four representative benchmarks: DrivAerNet++ Surf, Airplane, Airfoil, and Elasticity. We further compare these states with those learned by Transolver, demonstrating the effectiveness of our approach in capturing eidetic states under complex physical geometries. All visualizations are extracted from the final layer of each model to provide a direct comparison their learned representations.
DrivAerNert+ Surf
Transolver
Transolver+ (Ours)
Figure 8. Visualizations of 32 physical or eidetic states learned in the final layer of models on DrivAetNet++ Surface. Transolver and Transolver++ both plotted for clear comparison. The lighter color means higher weight the corresponding physical state.
AirCraft
Transolver
Transolvert+(Oun)
Figure Visualizations 32 physical eidetic states learned the final layer of models on AirCraft. Transolver and Transolver++ are both plotted for a clear comparison. The lighter color means higher weight in the corresponding physical state.
C.2. Case Studies
In addition to the Figure 6 in the main text, here provide showcase comparisons in Figure 14, 15 and 16. Specifically, compare model with the strong baselines in the respective datasets, where Transolver is selected for the comparison in standard benchmarks and both Transolver and GNOT are compared in industrial datasets.
16

---

Transolver++:: An Accurate Neural Solver for PDEs Million-Seale Geometries
Transolvert+
Figure 10. Visualizations of 64 learned cidetic states of the final layer in Transolver++ on Airfoil.
Transolver
Figure 11. Visualizations of 64 learned physical states of the final layer in Transolver on Airfoil.
Transolver++ (Ours)
Figure 12. Visualizations of 64 learned eidetic states of the final layer in Transolver++ on Elasticity.
Transolver
Figure 13. Visualizations of 64 learned states of the final layer in Transolver on Elasticity.
17

---

Transolver++: An Accurate Neural Solver for PDEs Million-Scale Geometries
Ground Truth
Transolver++ Prediction
Transolver Prediction
GNOT Prediction
0.1
0.1
0.1
AirCraft Error Map
0.11
Figure Showease comparison with Transolver and GNOT AirCraft. A lighter color the error map indicates better performance.
Ground Truth
Transolvert+ Prediction
Transolver Prediction
GNOT Prediction
500
-600
-1700
100
DrivAerNet++ Surface
Error Map
-100
Figure 15. Showcase comparison with Transolver and GNOT on DrivAerNet++ Surface. A lighter error map means better performance.
Ground Truth
Transolvert+ Prediction
Transolver Prediction
Ground Truth
Transolver+* Prediction Transolver Prediction
Elasticity Error Map
Airfoil
Error Map
Figure 16. Showease comparison with Transolver on Elasticity and Airfoil. A lighter color in the error map indicates better performance.
18


Reviews:
[{"review_id": "zDg4vFxlB8", "reviewer": "Reviewer_dnan", "summary": "This paper extends Transolver, which is a efficient transformer that predict the PDE solution of the input discretized geometry and physical quantities. The original Transolver weighted average the intermediate features at each grid node into several physical states tokens for efficient self-attention (physics-attention), and this weighted averaging can suffer from the over-smoothing problem. To address this issue, this paper propose to learn adaptive temperature at each node and use gumbel softmax trick. To further accelerate the computation of the physical states tokens, this paper also distribute the computation onto multi-gpus. To demonstrate the effectiveness of the proposed method, this paper conducted experiments on various synthetic benchmarks and real-world industrial applications, which proves that the two improvements introduced in this paper can handle mesh girds in higher resolution, which is essential to generate better results.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Many thanks to Reviewer dnan for providing a detailed and in-depth review.\n\n>**Q1:** Reclarification of the concept of \"Physical States,\" particularly in early layers.\n\nThank you for your rigorous review.\n\n**(1) \"Physical states\" is from Transolver (Wu et al., 2024)**\n\nIn Transolver, \"physical states\" are defined as **physical internal-consistent representations** associated with specific geometric regions (e.g., windshield or sunroof of a driving car). Extensive visualizations in their work demonstrate its ability to learn such states. Besides, after confirming with its authors that **all the state visualizations in Transolver are based on the first layer representation**, we think this concept is reasonable for early layers.\n\nSince Transolver++ is an upgraded version of Transolver, we would like to keep this concept to ensure consistency.\n\n**(2) New experiments**\n\nTo address your concern, in DrivAerNet++, we randomly sample 100 points across various regions and calculate JS divergence between **first-layer slice weights** among different areas. We can find that the slice weights present clear internal consistency and external inconsistency in different regions.\n\nThus, slicing operation is not a trivial weighted sum. It can ensure the model to capture physical internal-consistent representations.\n\n|JS divergence|Sunroof-Part2|Windshield|Front|\n|-|-|-|-|\n|w.r.t. Sunroof-Part1|0.041|0.576|0.378|\n\n**(3) Representation learning perspective**\n\nNote that all model parameters will be simultaneously updated during training. In Transformer-based models, residual connections blur the distinction between early and ending layers, as gradients from the final loss can directly affect the first layer. Therefore, we cannot simply assume that early-layer representations are less related to physics.\n\nConsidering visualizations of Transolver and concept consistency, we think \"physical state\" an appropriate term. It also clarifies our paper's core idea in \"learning distinguishable states\".\n\n>**Q2:** Reclarification of the novelty and significance of our work.\n\nTo our best knowledge, Transolver++ is the first to design and validate deep models on million-scale industrial datasets and to verify an effective parallelism framework for PDEs, which is **meaningful to this community** in paving the way to practical neural solvers for real-world applications.\n\nOur technical contributions are summarized as follows.\n|Our design|New insights|\n|-|-|\n|local adaptive mechanism|PDE-solving requires local-adaptive representations|\n|Parallelism framework|Parallelism is an effective way to large-scale geometries|\n|Industrial-level experiment|Encourage the community to tackle large geometries|\n\n**(1) Local adaptive mechanism is better in learning physical states**\n\nBeyond only considering oversmoothing, our approach tries to enable local-adaptive for each point, where learned distributions are expected to be uniform for high-uncertainty areas and peaky for high-confident areas.\n\nTo further demonstrate that our design is more than avoiding oversmoothing, we also provide ablation of \"global annealing\" temperature (a well-established way for oversmoothing attention). Results below confirm the effectiveness of our method. This also delivers the insight that local-adaptive representations are crucial in PDE solving.\n\n| AirCraft|$C_p$|$R_L$|Surf|\n|-|-|-|-|\n|Transolver + global annealing|0.034|0.993|0.093|\n|Transolver + Ada-Temp|0.020|0.995|0.080|\n|Transolver++|**0.014**|**0.999**|**0.064**|\n\n**(2) Highly parallel framework to enable million-scale geometries**\n\nWhile Eq (5) seems straightforward, it stems from our in-depth insights for Transolver and PDE-solving tasks, where we comprehensively investigated which representation should be parallel computed or communicated. With these efforts, Transolver++ successfully achieves the lowest communication overhead and presents linear scalability.\n\nIt is also the first efficient parallelism framework in neural operators for PDE solving, which serves as a starting point for exploring model-system co-design in solving PDEs. Thus, the contribution of the parallelism framework should not be underestimated.\n\n**(3) Outstanding performance on industrial tasks**\n\nOur model achieves 13% and 20% promotion on standard benchmarks and industrial tasks. The new AirCraft dataset will also be made public to facilitate future research. More results are in Q1 and Q3 of Reviewer Ma1w.\n\n>**Q3:** How does Gumbel-softmax affect the deslice process?\n\nAs shown in Eq.(5) and Algo 1, we generate new slice weights using Gumbel-Softmax rather than sampling a specific state, which allows the generated weights to be directly used for deslice.\n\n>**Q4:** f is not explained anywhere in the paper?\n\nAs stated in Lines 267-270, we have stated that \"Also, we found that ..., where x is used to generate slice weights, and f is combined with weights to form physical states.\" f represents geometric feature. We will highlight this in the revision.", "claims_and_evidence": "The claims made this paper is clear and convincing.", "methods_and_evaluation_criteria": "The methods and evaluation criteria make sense in general, except for the adaptive temperature in Eq.3 is not guaranteed to be positive.", "theoretical_claims": "This paper is about practical implementation improvements, there is no theoretical claim made.", "experimental_designs_or_analyses": "The experimental designs and analysis are solid.", "relation_to_broader_scientific_literature": "The key contribution of this paper is leveraging previous techniques such as gumbel-softmax trick", "essential_references_not_discussed": "To the best of my knowledge, references are sufficient.", "other_strengths_and_weaknesses": "- Strengths: This paper is well organized and generally well written.\n- Weakness: \n  - This paper contains some jargons make simple things complicated to understand: why it is called physical states while it is a weighted average of input features? __s__ is a weighted average of  __x__. They should have same physical interpretation, so it is confusing to name __s__ using additional term without any elaboration.  Although eventually this __s__ will be desliced and give the final output, it is really confusing to name __s__ through all layers as \"state\" since __s__ in early layers does not contain sufficient information that can be directly considered as a state of the unknown function to solve.\n  - The improvement is so incremental that can hardly provide any new insights. \n    - Oversmoothing is a well-known problem of attention, and the solution provided in the paper are a combination of existing techniques. Additionally, the authors did not mention how gumbel-softmax affect the deslice process during inference.\n    - The acceleration in Sec.4.2 is also straightforward, additionally the __f__ is not explained anywhere in the paper.", "other_comments_or_suggestions": "I do not have additional comments.", "questions_for_authors": "Please refer to the weakness. The major concern I have is that the improvement is so incremental and does not provide sufficient insights.\nTherefore, I would expect authors could justify the contribution better and explain why the contributions are not incremental in the rebuttal.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TIdXO8jdKJ", "reviewer": "Reviewer_Ma1w", "summary": "Authors improve the scaling characteristics of a previously proposed model - Transolver - by analyzing computational and performative bottlenecks of the original model:\n \n- homogeneous latent tokens (physical states) => inability to capture mesh details;\n- memory bottleneck caused by processing large-scale meshes (~1M points) => inability to scale to large meshes,\n\nwhich authors address by:\n- computing projection weights by sampling the categorical distribution (Rep-Slice) with learnable temperature (Ada-Temp);\n- distributing projection weights computation across multiple GPUs (as the operation is mainly point-wise).", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Many thanks to Reviewer Ma1w for your invaluable suggestions.\n\n>**Q1:** Lack of baseline range on large geometries benchmarks, such as UPT, GraphViT, and PointTransformer. Also, compare with imperfect baseline traditional solvers.\n\n**(1) New baselines.**\n\nThank you for your constructive and helpful suggestions. Due to the limited rebuttal time, we were only able to carefully tune and compare several models on the **AirCraft** dataset, as summarized below. We will include these baselines along with brief model descriptions in the revised version. Notably, our model consistently achieves the best performance under this setting, demonstrating its effectiveness and robustness across strong baselines.\n\n|Model|Cp↓|$R_L^2$↑|Surf↓|\n|-|-|-|-|\n|UPT (Alkin et al., NeurIPS 2024)|0.035|0.994|0.112|\n|GraphViT (Janny et al., ICLR 2023)|0.041|0.990|0.130|\n|Point Transformer v3 (Wu et al., CVPR 2024)|0.045|0.987|0.145|\n|Transolver++|**0.014**|**0.999**|**0.064**|\n\n**(2) Imperfect baseline traditional solvers**\n\nFor completeness, we also used a different traditional PDE solver (UnsCFD) to re-simulate the AirCraft data (smallest mesh scale in all three datasets). Each case took 5–6 hours on average, requiring 3–4 days to complete the full test set. In contrast, our neural model trains in under 10 hours and infers each case in less than 1 second, offering a significant efficiency advantage.\n\nMoreover, as shown below, **Transolver++ significantly outperforms UnsCFD in accuracy**, showing its potential as a practical surrogate for large-scale simulations. These results will be included in the appendix of the updated version.\n\n|AirCraft|Surf|\n|-|-|\n|UnsCFD|0.173|\n|Transolver++|**0.064**|\n\n> **Q2:** No error bars in Table2.\n\nThank you for pointing this out, and we sincerely apologize for the oversight. All experiments for Transolver and Transolver++ were run at least three times. We provide the error bars (mean ± std) for Table 2 below and will include them in the revised version for clarity.\n\n|model|Volume|Surf|$C_D$|$R_L^2$|Surf|$C_l$|$R_L^2$|Surf|\n|-|-|-|-|-|-|-|-|-|\n|Transolver|0.173±0.003|0.167±0.002|0.061±0.002|0.931±0.005|0.145±0.008|0.037±0.003|0.994±0.001|0.092±0.003|\n|Transolver++|0.154±0.002|0.146±0.002|0.036±0.001|0.997±0.001|0.110±0.004|0.014±0.001|0.999±0.001|0.064±0.002|\n\n> **Q3:** Is it possible that the method improves a single model, but the analysis is strictly limited to that particular model? In particular, perhaps the strongest contribution of the paper is the slice reparameterization. I am not sure how much knowledge it adds to the community.\n\nThanks for your detailed and instructive comments. As you mentioned, Transolver++ is specific to Transolver, which is why we name it \"Transolver++,\" not others. However, the considered questions and design principles of our paper are valuable to this community. Here is the summarization.\n\n|Our design|Transferable Insights|\n|-|-|\n|local adaptive mechanism|PDE needs local-adaptive representations|\n|Parallelism framework|Parallelism is an effective way to large-scale tasks|\n|Industrial-level experiment|Encourage community to tackle large geometries|\n\nWe would like to further clarify our contributions to this work. Our contributions lie in the following aspects.\n\n**1. A local adaptive mechanism to better learn the physical states.**\n\nThe idea of **learning hidden physical states** is not specific to our model, which is widely shared in this community as a means of compression without explicit modeling. Our contribution lies in improving **how** such states are learned. The underlying principle is general and can be extended to other models with similar goals.\n\n**2. An end-to-end model with highly parallel implementation**\n\nTo our best knowledge, Transolver++ is the first to validate end-to-end deep models and verify an effective parallelism framework for industrial datasets with million-scale geometries, which is **meaningful to the community**. Though tailored to Transolver, it provides insights into scalable architecture design and low overhead parallelism, which is ensuring the smallest communicating representation.\n\nIt is also the first efficient parallelism framework in neural operators for PDE solving, which can serve as a starting point for exploring model-system co-design in solving PDEs. Thus, the contribution of the parallelism framework should not be underestimated.\n\n**3. Outstanding performance on industrial datasets**\n\nOur model consistently achieves SOTA results on both standard benchmarks and industrial datasets. As shown above, we further validated other models on the AirCraft dataset and our model on the newly-released DrivAerML (Ashton et al., CoRR 2024, 800 million mesh points per sample) datasets below.\n\nThis outstanding performance firmly demonstrates the potential of advancing neural PDE solvers to real applications, which is a strong encouragement to the community.\n\n|Model|Cp↓|pPrime↓|\n|-|-|-|\n|Transolver|7.94|5.94|\n|Transolver++|**6.75**|**5.14**|", "claims_and_evidence": "Authors claim to achieve state-of-the-art performance on all the datasets, but on large geometries benchmarks (Table 2), the baseline range is quite limited. Mainly, there are graph neural network-based baselines, which are known to perform poorly on large meshes due to limited receptive field (mentioned by authors in the related works section). To support the claim, it would be beneficial to compare Transolver++ against more transformer-based architectures such as UPT (Alkin et al. 2024), GraphViT (Janny et al. 2023) and PointTransformer (Wu et al. 2024).\n\nAnother claim I find somewhat controversial is the high computational cost of traditional numerical methods. I know it is somewhat taken for granted within the community, but there are reports that the claim is based on rather imperfect baseline traditional solvers. I believe that a fair comparison with traditional solvers (given that they do not need dataset accumulation as neural-based approaches) would strengthen the claim.", "methods_and_evaluation_criteria": "The benchmarks make perfect sense and are complete dataset-wise.", "theoretical_claims": "There are no theoretical claims except overhead analysis (Algorithm 1), and I believe it is correct.", "experimental_designs_or_analyses": "The experimental design is comprehensive and covers against-baselines comparisons as well as ablations and empirical scalability analysis.", "relation_to_broader_scientific_literature": "Key contributions are aligned with the overall trend for scalability in the DL community.", "essential_references_not_discussed": "As in the original Transolver paper, authors explore linear-time attention computed on latent tokens (physical states). This is very related to Universal Physics Transformers (Alkin et al. 2024), and I believe the comparison with UPT would improve the paper. Similarly, GraphViT (Janny et al. 2023) is another instance of attention over coarse-grained mesh representation which is not presented in the baselines. Additionally, linear-time transformers such as PointTransformer v3 (Wu et al. 2024) are not compared against.", "other_strengths_and_weaknesses": "Strengths: \n- The paper is well written, the structure is easy to follow, and the notation is clear.\n- I particularly appreciate the visual aspect of the paper and find the presentation impeccable and really well done.\n- Authors focused on the scalability of the original paper and managed to improve it significantly by identifying key bottlenecks.\n- The solutions to the issues are simple, elegant and scalable.\n- Benchmarks are comprehensive, and the experimental design (up to a couple of additional baselines) is extensive.\n\nWeaknesses:\n- There are no error-bars in experiments. If that is possible to include them (at least for Transolver and Transolver++), that would be great.\n- While the paper is a strong contribution application-wise, it is rather weak theoretically. My main concern is that the paper essentially improves a single model, but the analysis is strictly limited to that particular model. In a way, the paper is a significant engineering effort with next-to-perfect delivery, but the scientific contribution does not seem to be significant to me. \n    - In particular, perhaps the strongest contribution of the paper is the slice reparameterization. It definitely works for Transolver, but I am not sure how much knowledge it adds to the community.\n   - For comparison, next iterations of PointNet and PointTransformer significantly change respective architectures and are effectively different models derived from their predecessors. Transolver++ does not do that; if anything, it polishes the existing framework.", "other_comments_or_suggestions": "No suggestions other than including a couple more baselines mentioned above.", "questions_for_authors": "Is it possible to include error bars for Transolver and Transolver++ in Table 2?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "j5a7aTAumF", "reviewer": "Reviewer_VjMs", "summary": "This paper introduces Transolver++, an extension of Transolver, designed to handle million-scale geometries in PDE solution operator learning. Building on the physics attention mechanism proposed in Transolver, which learns underlying physical states, this work presents two key advancements-a local adaptive mechanism and a parallelized implementation. The local adaptive mechanism enables Transolver++ to learn a per-point temperature used in the softmax function to determine the probability of a point belonging to a particular (learned) physical state. The parallelized implementation leverages Transolver’s design by synchronizing only the normalizer and physical states computed from a batch of points assigned to a single GPU, minimizing inter-GPU communication overhead. Transolver++ achieves a 13% performance improvement on standard PDE benchmarks compared to previous methods and a 20% performance gain on more challenging benchmarks with large-scale geometries, which are 100 times larger than those used in prior studies.\n\n## Update After Rebuttal\n\nI believe the concerns raised in my initial review are adequately addressed in the author rebuttal. While I acknowledge the points made in other reviews regarding the limited technical novelty, I find that this work offers a valuable perspective on scaling neural operators to handle PDEs on larger and more complex domains—a step toward real-world applicability. Therefore, I will maintain my original rating.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Many thanks to Reviewer VjMs for your detailed and instructive review.\n\n> **Q1:** How generalizable is the proposed method across different geometries, particularly in experiments involving car designs from the DrivAerNet++ dataset? Could you provide more details on the statistical distribution of design parameters for car shapes in the training and test splits?\n\nThank you for your constructive question, which helps to improve our work and analysis. We follow the sampling ratios provided in Figure 8 of the original DrivAerNet++ paper, using equal numbers of samples for both WW and WWC (wheels open detailed/closed):\n\n|Type|Fastback|Estateback|Notchback|\n|-|-|-|-|\n|#train cases|54|72|54|\n|#test cases|6|8|6|\n|Total|60|80|60|\n\n**Training and testing share the same distribution**, with no car-type shift between splits.\n\nTransolver++ shows **consistently strong performance across all categories**, confirming its robustness and generalization. A summary will be added in the revised version.\n\n|Type|Fastback|Estateback|Notchback|\n|-|-|-|-|\n|GNOT|0.167|0.143|0.190|\n|Transolver++|**0.112**|**0.108**|**0.109**|\n\n> **Q2:** UPT is a conceptually similar method and is not mentioned in the related work. How does this Transolver-based method differentiate itself from UPT and its follow-up NeuralDEM? Have the authors experimented with UPT, or do they have a high-level understanding of why one method might outperform the other?\n\nThank you for your valuable suggestion. It prompted us to reflect on the distinction between Transolver-based methods and UPT, representing important advances in neural solvers for scientific problems.\n\n**(1) Conceptual Difference**\n\nFirst, we note that Transolver (ICML 2024) predates UPT (NeurIPS 2024).\n\nSecond, the modeling paradigms differ: UPT decouples the encoder and decoder, enabling flexible querying across arbitrary scales, while Transolver-based models are fully end-to-end, enabling direct supervision on physical states and achieving stronger performance on fixed meshes (e.g., AirCraft).\n\nThird, **UPT compresses geometry via supernodes** into a global latent without explicit state assignments. **NeuralDEM** builds on this with primary quantity modeling and scalar control. In contrast, **Transolver++ learns soft point-to-state assignments**, enabling **interpretable adaptive assignments** based on shared physical behavior.\n\nFourth, UPT evolves dynamics entirely in the global latent space. Transolver-based models apply **slice-deslice at every layer**, supporting **progressive refinement of physical states** and stronger geometry-physics coupling, with enhanced interpretability and control.\n\n**(2) Empirical Comparison**\n\nAs our focus was on end-to-end architectures, UPT was not included in our original experiments. During the rebuttal period, we trained UPT on the AirCraft dataset.\n\nWe observed that UPT has ~**10× more parameters** than Transolver++ (14.3M vs. 1.7M), which may lead to **slower convergence**. Additionally, its lack of explicit physical modeling for nodes may limit its ability to capture fine-grained structures. As a result, UPT underperforms compared to Transolver++ in this setting.\n\n|Model|Cp↓|$R_L^2$↑|Surf↓|\n|-|-|-|-|\n|UPT (Alkin et al., NeurIPS 2024)|0.035|0.994|0.112|\n|GraphViT (Janny et al., ICLR 2023)|0.041|0.990|0.130|\n|Point Transformer v3 (Wu et al., CVPR 2024)|0.045|0.987|0.145|\n|Transolver++|**0.014**|**0.999**|**0.064**|\n\nWe sincerely apologize for not including UPT in the related work section earlier. A detailed comparison and proper citation will be added in the revised version.\n\n> **Q3:** Is this sampling performed only during training, or does it also occur during inference?\n\nAs shown in Eq.(5) and Algo 1, stochasticity occurs in both training and inference, which enables the model to explore a more diverse state assignment distribution. Also, we provide train log plots at https://anonymous.4open.science/r/ICML_rebuttal_3035-FD1C to show the stability of Transolver++.\n\n> **Q4:** What does the variable f represent? \n\nAs we stated in Lines 267-270 right column of main text, we have stated that \"Also, we found that ..., where x is used to generate slice weights, and f is combined with weights to generate physical states.\" In all, f represents geometric features while x, as other features, is used to generate weights in the github implementation of Transolver. We will make this clearer in the revised paper. Notably, this design significantly reduces computational complexity and enables the first practical application of deep learning to large-scale industrial simulations.", "claims_and_evidence": "Yes, this paper identifies two major limitations of its predecessor, Transolver: the degeneration of learned physical states when applied to large-scale geometries and the lack of multi-GPU support. These issues are addressed in the main text with a detailed exposition. The authors' claims are well-supported by experimental results, which demonstrate clear performance improvements on standard PDE benchmarks as well as more challenging benchmarks, including the recently proposed DrivAerNet++ dataset, which involve complex irregular geometries.", "methods_and_evaluation_criteria": "Yes, the proposed method aligns with the neural operator literature and is validated using well-known benchmarks, employing the relative $L_2$ error metric, which is standard in the field.", "theoretical_claims": "This paper does not present any theoretical claims requiring review.", "experimental_designs_or_analyses": "This paper validates the proposed method using commonly used PDE benchmarks, advancing the state of the art. The use of PDE dataset containing large-scale geometries is appropriate given the core objective of this work—developing a neural operator capable of handling such geometries. The authors include a wide range of baselines in qualitative and quantitative comparisons, covering both graph-based and transformer-based methods. Most of the experimental details are explained in the appendix.\n\nWhile the text is well-written overall, there are some points that remain unclear to me:\n\n1. How generalizable is the proposed method across different geometries, particularly in experiments involving car designs from the DrivAerNet++ dataset? While the paper mentions that 200 representative designs were selected, it would be beneficial to provide a statistical analysis of these shapes, including the distribution of car types (e.g., estateback, fastback, or notchback), to better assess the method's generalization capability.\n2. Regarding the baselines, I am curious how this method compares to Universal Physics Transformers (UPT) [1], another recent transformer-based neural operator that claims to handle large-scale geometries. UPT aggregates local physics fields at a small number of supernodes and applies an attention mechanism among them, which, in my view, is conceptually similar to Transolver and its extension in this work. However, UPT is not mentioned in either the related work or experiment sections. I would like to hear the authors' perspective (*e.g.*, how these methods are different) on this method.", "relation_to_broader_scientific_literature": "The idea proposed in this work could advance deep learning for large-scale scientific computing, which often involves simulating complex systems discretized at high spatial and temporal resolutions.", "essential_references_not_discussed": "As noted in the “Experimental Designs or Analyses” section, I would like to see the authors' discussion on UPT [1] and its follow-up [2], which extends the method to industry-level simulations. While I do not expect an empirical comparison, these works appear highly relevant to the scope of this study. Including them in the discussion would improve the completeness of the paper.\n\n**References**\n\n1. Universal Physics Transformers: A Framework For Efficiently Scaling Neural Operators, Alkin *et al.*, NeurIPS 2024\n2. NeuralDEM-Real-time Simulation of Industrial Particulate Flows, Alkin *et al.*, arXiv 2024", "other_strengths_and_weaknesses": "I have no further comments on the strengths and weaknesses.", "other_comments_or_suggestions": "In line 300, the text states “previous studies,” but only a single work is cited.", "questions_for_authors": "I would like to asks the authors the following questions:\n1. Could you provide more details on the statistical distribution of design parameters for car shapes in the training and test splits? A summary of how the shapes differ between these splits would help evaluate the generalizability of the proposed method.\n2. How does this Transolver-based method differentiate itself from the Universal Physics Transformers (UPT) [1] approach? Have the authors experimented with UPT, or do they have a high-level understanding of why one method might outperform the other?\n3. Based on my understanding, Transolver++ introduces a certain degree of stochasticity when mapping a point to multiple physical states, as described in lines 190–193 and Equation (4). Is this sampling performed only during training, or does it also occur during inference?\n4. In line 267, what does the variable $\\mathbf{f}$ represent? I could not find any mention of its meaning or usage in the Transolver paper either.\n\n**References**\n1. Universal Physics Transformers: A Framework For Efficiently Scaling Neural Operators, Alkin *et al*., NeurIPS 2024", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zcu5cVKYGC", "reviewer": "Reviewer_oAjG", "summary": "This paper presents two extensions to the Transolver architecture, for learning PDEs in systems with high resolution data. First, it introduces changes in how the weights for the \"physical states\" are computed, to achieve more peaky distributions. And second, it presents a Multi-GPU implementation of Transolver which reduces the communication overhead.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Many thanks to Reviewer oAjG for your instructive reviews.\n\n> **Q1:** It's not clear what the goal exactly is. The problem might be something else entirely: the distribution over output physical states.\n\n**(1) Our goal is to local-adaptively control each point’s state distribution.**\n\nAs stated in lines 210-218, neither overly peaky nor smooth weights are desirable. As shown in Fig.3(c) and stated in its caption, points in slow varying regions should have sharp distribution, while those in fast varying regions with higher uncertainty expect a relatively smooth distribution.\n\n**(2) We respectfully disagree with “the problem might be something else.\"**\n\nWe appreciate the reviewer’s careful observation. However, as shown in Figure 13 of Appendix (overly uniform assignments) and Figure 8 left of Appendix (Collapsed into two states), both cases illustrate unreasonable point-wise state assignments, failing to meet our goal above.\n\nThese cases reveal an underlying issue: **the lack of adaptive point-wise assignment**, rather than the \"distribution over output physical states.\" We believe our work directly addresses this core limitation of Transolver.\n\n**(3) New experiments on annealing.**\n\nAs per your request, we conduct ablation studies (see Q2(1) of Reviewer dnan) with annealing strategies. This further verifies the statements above.\n\n> **Q2:** Why spread some input features over several states, not others?\n\nAgain, our goal is point-wise adaptive distributions, not just peaky ones. A more uniform distribution is more suitable, especially for fast-changing regions with higher uncertainty.\n\n> **Q3:** Why use sampling? Is the sampling seed fixed? Stability\n\nAs shown in Eq.(5) and Algo 1, we don't perform explicit sampling during training or inference. Instead, we generate new state weights using Gumbel-softmax, which enables the model to explore more diverse state assignment distribution. With a fixed seed at the beginning, each training step yields different distributions for the same element. This will not affect the training stability. Please see: https://anonymous.4open.science/r/ICML_rebuttal_3035-FD1C\n\n> **Q4:** The contributions are quite minor, cannot transfer. Multi-GPU optimization is nice, but a bit of an implementation detail.\n\nAs you mentioned, Transolver++ is specific to Transolver, which is why we name it \"Transolver++,\" not others. However, the considered questions and design principle of our paper are valuable to this community. Here is the summary.\n\n|Our design|Transferable Insights|\n|-|-|\n|local adaptive mechanism|PDE needs local-adaptive representations|\n|parallelism framework|Parallelism is an effective way to large-scale tasks|\n|Industrial-level experiment|Encourage the community to tackle large geometries|\n\n**(1) Q1 can demonstrate insights behind local adaptive mechanisms.**\n\n**(2) Parallelism framework stems from in-depth investigation.**\n\nThe \"multi-GPU optimization\" design stems from our in-depth insights into Transolver and PDE-solving tasks, not only \"implementation detail,\" where we carefully investigated which representation should be parallel computed or communicated. Without these efforts, Transolver++ cannot achieve the lowest communication overhead and presents linear scalability now.\n\nIt is also the first efficient parallelism framework in neural operators for PDE solving, which can serve as a starting point for exploring model-system co-design in solving PDEs. Thus, the contribution of the parallelism framework should not be underestimated.\n\n**(3) Outstanding performance on industrial tasks can advance research to real applications. A new dataset of AirCraft will be made public.**\n\n> **Q5:** New datasets analysis is a bit misleading, baselines are on patches and stitched.\n\nThanks for your detailed review. We will highlight our training setting in the revision.\n\n**(1) We have to conduct patch-wise training, otherwise all prior methods are inapplicable.**\n\nAs noted in Table 2's footnote, prior methods cannot handle million-scale data and require patch-wise training. Since this is the only way (we can figure out) to measure other models on DrivAerNet++, we believe this comparison exactly shows their capability in large-scale tasks. \n\nThis unaligned setting highlights the advantage of Transolver++ in large-scale tasks (parallelism framework is our key contribution), instead of being considered meaningless.\n\n**(2) Comparison under the same setting** \n\nWe also provide results of aligned settings in six standard benchmarks and AirCraft. We will further clarify this in the revision. We also trained Transolver++ under the patch setting, which is still the best compared to other models.\n\n|DrivAerNet++|$C_D$|$R_L^2$|Surf|\n|-|-|-|-|\n|Transolver++ (patch)|0.017|0.997|0.072|\n|Transolver++ (unpatch)|0.014|0.999|0.064|\n\n> **Q6:** Confusing/vague terms and phrases.\n\nMany thanks. \"Mesh points\" are common in PDE-solving and refer to vertices. We will carefully revise other terms in the revision.", "claims_and_evidence": "Yes. The main claims are extending Transolver to work with larger mesh sizes, and multi-GPU optimizations. The paper shows evidence for both.", "methods_and_evaluation_criteria": "The paper largely reuses the baseline methods and datasets from the Transolver paper, and adds two further datasets. While not adding that much new over the previous paper, this is a broad coverage of different PDE systems and many relevant transformer and GNN based competing architectures which is nice to see.\n\nI'm less sure about the method; the changes to weight computation seem to be a bit unprincipled. \nIt's not really clear to me what the goal exactly is-- from the text it sounds like the Transolver weights are too uniform for large meshes, and a more peaky distribution is desired (ideally assigning each mesh point to exactly one \"physical state\"). The obvious solution for this would be an annealing schedule for the temperature during training.\nInstead, there's a complicated two-stage solution, a learned temperature per input point, and then differentiable sampling. The text doesn't quite explain why this makes sense. \n1. Why would we want to spread some input features over several physical states, but not others?\n2. Why would we want to use sampling for this? There's only one set of weights per input point.\n3. Is the sampling seed fixed (i.e. each training step gets the same element given an unchanged weight distribution) or not?\n4. What about stability? Both learning temperature and differentiable sampling seem like they would make training much more instable and seed-dependent.\nAnd finally, from a look at the weight visualization in paper and appendix is seems like the problem might be something else entirely-- not the uniform distribution of a given input state, but the distribution over output _physical states_. Transolver doesn't seems to use all the physical states available, and instead assigns the entire input to 2 states. Transolver++ seems marginally better at this, but there isn't anything in the proposed algorithm which directly addresses the issue. This is a much harder (but also much more interesting) problem, if this indeed turns out to be the issue there's plenty of literature on differentiable class assignment to look at.", "theoretical_claims": "No theoretical claims/proofs in this paper.", "experimental_designs_or_analyses": "The most important experiments are there, all claims and novel method components have matching experiments. \n\nOne issue I see it that the analysis on the new datasets is a bit misleading-- here, baselines are inferred (and maybe trained?) on patches and stitched together, while Transolver++ is not. The paper does mention this, but a) it's easy to miss on a quick glance and b) since the impact to MSE is very much unclear, it's hard to know what to even take from this comparison.\n\nAnd, as stated above, the paper could use clearer analysis to what exactly the issue with weight assignment is, and how or how not Transolver++ solves this. The hypothesis provided doesn't seem to be backed by the visualizations shown, and the only evidence is improved downstream performance.", "relation_to_broader_scientific_literature": "The paper is an extension to the Transolver paper. It shows some performance improvement over Transolver, particularly for high-res problems. However, both contributions (improved weight computation, and multi-GPU optimization) are relatively minor, and quite niche-- the only really apply to the Transolver architecture and would not transfer to any other model for learned PDEs.", "essential_references_not_discussed": "References are fine.", "other_strengths_and_weaknesses": "Strengths: The paper does show empirical performance improvement on a wide set of benchmarks, including very large domains.\nWeaknesses: The contributions are quite minor, and I'm not sure if we can learn much from the paper which will transfer to other work. As stated above, the weight computation is rather unprincipled and it's unclear why and how it works. The multi-GPU optimization is nice to see, but a bit of an implementation detail, and highly specific to this exact method.", "other_comments_or_suggestions": "The paper uses quite a few confusing/vague terms and phrases, e.g. \"massive mesh points\", \"overwhelm the learning process\", \"completely model the assignment process from points to certain physical states\". In particular, \"eidetic states\" is an odd choice, I had never heard of this word before, and the dictionary definition doesn't match what the paper likely wants to convey.\nSo if the paper is accepted, I would advise an editing pass with this in mind.\n\n# After rebuttal\nThe rebuttal did add a few new datapoints and explanations, which is appreciated. But the main issues I pointed out in my review still stand, and based on the rebuttal it didn't sound like the authors will address these in an updated version of the paper. So I think it's probably better if the paper went for a major revision & resubmit. Hence I will keep my score.", "questions_for_authors": "See \"methods and evaluation\".", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: **(a) Summary**
This paper presents Transolver++, an extension of the original Transolver neural PDE solver, designed to scale to million-point mesh geometries via two key innovations: (1) a local adaptive mechanism using point-wise learned temperatures with Gumbel-Softmax to improve representation of physical states, and (2) an optimized multi-GPU parallelism framework that enables linear scalability and low communication overhead. The paper provides extensive experiments demonstrating significant performance gains on both standard PDE benchmarks and new industrial-scale datasets.

**(b) Strengths**
The reviewers generally agree that Transolver++ effectively tackles scalability challenges in neural PDE solvers and demonstrates state-of-the-art performance. Reviewer Ma1w highlighted the paper as "a valuable submission" and praised the authors for addressing bottlenecks with “simple, elegant and scalable” solutions. Reviewer VjMs emphasized the paper’s practical significance for large-scale scientific computing and acknowledged the empirical rigor and relevance of the proposed improvements. Reviewer dnan noted that the claims are convincing, and the experimental setup is solid. Despite the lack of theoretical novelty, the paper was seen as a significant engineering contribution by most reviewers.

**(c) Weaknesses**
Several concerns were raised:
* Reviewer oAjG and Reviewer dnan criticized the work for lacking conceptual clarity. Reviewer oAjG questioned the motivation for the adaptive temperature mechanism and found the terminology and methodology insufficiently explained, suggesting the solution may be unprincipled and model-specific.
* Reviewer dnan felt the contribution was incremental and not insightful, while also pointing out minor notational and explanation gaps.
* Reviewer Ma1w and Reviewer VjMs raised concerns about the limited range of baselines on large-scale benchmarks. Reviewer Ma1w suggested including comparisons to models like UPT, GraphViT, and PointTransformer. Both Ma1w and VjMs also noted the lack of theoretical depth, suggesting the work is mostly a well-executed extension rather than a groundbreaking methodological innovation.

**(d) Decision: Weak accept**
Overall, despite some disagreements about novelty and generality, the consensus favors acceptance due to the paper’s practical contributions, strong empirical results, and successful rebuttal. The authors provided additional baselines, clarified terminology and goals, and validated the adaptive and parallel mechanisms through ablations and new datasets. Reviewer Ma1w increased their score and expressed appreciation for the thorough responses, while Reviewer VjMs maintained support after the rebuttal. Reviewer oAjG acknowledged the rebuttal but remained skeptical. Reviewer dnan also acknowledged the authors' clarifications without updating their score. Given the strength of the practical contributions and the compelling experimental support, I recommend acceptance.

**Additional Comments on Reviewer Discussion**
Ma1w and VjMs were actively engaged in the rebuttal and updated or confirmed their positive assessments. dnan and oAjG acknowledged the rebuttal but did not fully engage or update their reviews. The authors made commendable efforts to clarify misunderstandings and add new experiments, which were appreciated by the more responsive reviewers.
Decision: Accept (poster)