# A Preliminary Study on Performance Modeling at Scale for Geophysical Applications

Fabrice Dupros, Sylvain Jubertie, Hideo Aochi

► To cite this version:

Fabrice Dupros, Sylvain Jubertie, Hideo Aochi. A Preliminary Study on Performance Modeling at Scale for Geophysical Applications. 33rd Euromicro International Conference on Parallel, Distributed, and Network-based Processing (PDP 2025), Mar 2025, Turin (IT), Italy. hal-04938571

HAL Id: hal-04938571

https://brgm.hal.science/hal-04938571v1

Submitted on 10 Feb 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# A Preliminary Study on Performance Modeling at Scale for Geophysical Applications.

Fabrice Dupros

Intel

Orléans, France

fabrice.dupros@intel.com

Sylvain Jubertie

Intel

Orléans, France

sylvain.jubertie@intel.com

Hideo Aochi

Brgm

Orléans, France

h.aochi@brgm.fr

**Abstract**—Modeling the performance of real-world applications at scale is essential for designing next-generation platforms and shaping the development of future algorithms. However, accurately capturing the complexity of application execution graphs and their interaction with large-scale hardware environments remains a significant challenge. In recent years, several frameworks have been developed to tackle this issue by providing tools to simulate and analyze complex workloads on distributed systems. This paper focuses on seismic wave propagation problems as a representative use case to explore the challenges of modeling at scale. We employ the SimGrid simulation toolkit, a versatile framework for simulating distributed systems, to analyze the performance of large-scale applications. Particular emphasis is placed on the role of critical networking characteristics, such as bandwidth, latency, and topology, in influencing overall scalability and performance.

**Index Terms**—Finite-Differences, Performance Model, Networking.

## I. INTRODUCTION

Performance modeling of realistic workloads remains one of the most challenging areas in High Performance Computing (HPC), where large-scale exploration is crucial for both hardware design and long-term software engineering strategies. Additionally, the rising costs associated with building real-world platforms for accelerated computing make running experiments at scale increasingly prohibitive. Advanced modeling approaches are therefore necessary to identify the most beneficial hardware features efficiently. Due to the complexity of creating accurate models, multi-level strategies are often employed to meet diverse requirements, such as scale, accuracy, efficiency, or cross-validation. Networking capabilities are becoming increasingly critical for scaling both traditional HPC applications and emerging workloads in artificial intelligence. Standard supercomputing systems connect compute nodes—composed of CPUs, GPUs, or both—via network interface controllers (NICs) and large switches configured in various topologies. In large-scale supercomputers designed for Artificial Intelligence or HPC, which feature up to $O(100,000)$ compute units, the efficient deployment of realistic applications is essential for addressing scientific challenges. Choosing the appropriate network topology, communication algorithms, and transport protocols can significantly improve the performance of distributed applications. While advanced topologies are being explored for specific workloads such as

artificial intelligence [1], [2], Fat-Trees [3] remain the dominant topology for HPC systems, with Infiniband as the most common transport protocol. Modern systems also incorporate high-bandwidth regions to connect multiple accelerators (e.g., PODs) using proprietary technologies.

This paper explores the impact of various networking parameters on the performance of a key numerical kernel in Earth Sciences. Specifically, we examine the finite-differences formulation of the full elastodynamics equation using the SimGrid modeling framework. Building upon earlier work [4], we adapt the Ondes3D software package for performance exploration. Our goal is to evaluate the influence of key networking characteristics on simulations at scale.

The remainder of this paper is organized as follows: Section II addresses the challenges associated with modeling at scale and describes the key features of the SimGrid modeling framework. Section III provides an overview of the numerical background related to seismic wave propagation and standard parallel implementations. After presenting a detailed description of the hardware and software setup, we discuss the results in Section IV, including the impact of various networking parameters.

## II. PERFORMANCE MODELING AT SCALE

### A. Context

Building an accurate performance model at scale for a specific application requires a detailed representation of both the application's execution graph and the underlying hardware platform. However, extrapolating results from small-scale synthetic platforms (e.g., those with only a few tens of nodes) to predict behavior on larger systems often falls short. This challenge is particularly pronounced in high-performance computing (HPC), where the effects of Amdahl's Law can transform otherwise regular and symmetric parallel patterns into complex and asymmetric behaviors as the number of computing units increases.

Analytical models are frequently used in performance studies, leveraging high-level abstractions of both workloads and platforms. These models are particularly effective for exploring performance with simplified workloads or mini-applications, offering rapid insights without the need for exhaustive computation. However, their utility diminishes when applied to full-scale, real-world applications, as they
often lack the granularity needed to account for intricate behaviors and interactions.

To overcome these limitations, simulation-based approaches provide a complementary and more precise solution. These methods involve intercepting, replaying, and emulating critical phases of execution, allowing for a deeper and more accurate analysis of performance at scale. By simulating the interactions between workloads and hardware in a controlled environment, these approaches enable researchers to uncover performance bottlenecks and validate scaling strategies. This makes simulation an invaluable tool for guiding the design and optimization of next-generation HPC systems and algorithms.

## B. Simgrid Perspective

Introduced over two decades ago, SimGrid [5] is an open-source modeling framework designed for simulating distributed systems. Although it was not initially developed with high-performance computing (HPC) systems in mind, SimGrid has been widely adopted in this domain due to its flexibility and robust feature set [6]. The framework provides users with a collection of predefined components that simplify the creation of hypothetical computing platforms. These platforms can be constructed using an XML file format or through C++ functions. Users can model essential resources such as hosts, disks, and networking elements, as well as various activities like computation, communication, and I/O operations.

One of SimGrid's key distinctions lies in its approach to simulation. Unlike other reference tools, such as the Structural Simulation Toolkit (SST) [7] or NS-3 [8], which focus on simulating packet-level interactions, SimGrid mainly adopts a flow-level simulation approach. This higher level of abstraction simplifies the modeling process and enhances scalability, making it particularly well-suited for larger and more complex systems.

In addition to its core functionality, SimGrid offers built-in support for various network topologies commonly used in HPC environments, including Torus, Fat-Tree, and Dragonfly.

# III. TARGET APPLICATION

## A. Numerical background

Seismic wave propagation from an earthquake in the Earth has been always numerical challenge, because of its dimension, resolution and medium complexity with respect to the geoscientific knowledge. Finite-differences method, in particular, the fourth-order approximation in space on staggered grids is the most popular option because of its efficiency and numerical stability [9]. In the case of an elastic material, the equation is:

$$ \rho \frac{\partial v_i}{\partial t} = \frac{\partial \sigma_{ij}}{\partial j} + F_i \quad (1) $$

and the constitutive relation in the case of an isotropic medium is:

$$ \frac{\partial \sigma_{ij}}{\partial t} = \lambda \delta_{ij} \left( \frac{\partial v_x}{\partial x} + \frac{\partial v_y}{\partial y} + \frac{\partial v_z}{\partial z} \right) + \mu \left( \frac{\partial v_i}{\partial j} + \frac{\partial v_j}{\partial i} \right) \quad (2) $$

where indices i, j, k represent a component of a vector or tensor field in cartesian coordinates (x, y, z), $v_i$ and $\sigma_{ij}$ represent the velocity and stress field respectively, and $F_i$ denotes an external source force. $\rho$ is the material density and $\lambda$ and $\mu$ are the elastic coefficients known as Lamé parameters. A time derivative is denoted by $\frac{\partial}{\partial t}$ and a spatial derivative with respect to the i-th direction is represented by $\frac{\partial}{\partial i}$. The Kronecker symbol $\delta_{ij}$ is equal to 1 if $i = j$ and zero otherwise.

## B. Standard parallel implementations

The parallel implementation of algorithms for seismic wave propagation has been extensively studied [10]. Typically for a two-dimensional problem, parallelism is achieved by dividing the spatial domain into regular subdomains, which are then distributed among all worker processes. In this approach, each MPI process is responsible for a subdomain of size $(n_x/P) \times (n_y/Q)$ where P and Q are the total number of MPI processes along the x- and y-axes, respectively. This domain decomposition strategy ensures an even distribution of computational workload across processes.

Explicit parallel elastodynamics applications typically demonstrate excellent weak and strong scaling performance, often scaling efficiently to several tens of thousands of computing nodes. This scalability is primarily due to the nature of these applications, which involve a relatively limited amount of point-to-point communication between neighboring subdomains. This minimal communication overhead typically enables the compute-intensive phases to take precedence, leading to highly efficient parallel execution.

Additionally, substantial efforts have been made to extend the parallel efficiency of these applications to heterogeneous and low-power processor architectures. For instance, studies [11], [12] have explored optimizing these algorithms for systems comprising a mix of CPUs, GPUs, and other specialized processors, as well as energy-efficient platforms. These advancements are critical for adapting elastodynamics simulations to emerging hardware technologies and improving their overall energy efficiency, while maintaining their high performance on large-scale systems.

# IV. PERFORMANCE EXPLORATION

## A. Experimental Setup

Our experiments are based on the definition of three hypothetical clusters using the built-in capabilities of SimGrid. First, we model a system where all computing nodes are interconnected via a crossbar switch. In this configuration used only for validation, the number of ports matches the number of computing units, providing full connectivity without contention. For the second system, we implement a two-level fat-tree topology. In this setup, hosts are grouped in clusters of eight, each connected to a dedicated router via a single link. At the next level, these routers are interconnected with two links, forming a scalable and efficient hierarchical structure. The final platform under study uses a hierarchical topology inspired by modern accelerated platforms, where different
network technologies are employed for scale-up (within nodes) and scale-out (between nodes) communication. In our case, the innermost level corresponds to a high-bandwidth region where hosts are typically interconnected via proprietary technologies. For this setup, we leverage the star zone topology available in SimGrid, which enables flexible interconnections between subzones and complex hierarchical layouts.

For all these configurations, we define a reference value of 50 GB/s and 200 ns for the link bandwidth and the latency, respectively. This corresponds to standard network characteristics. We discuss results obtained on a hypothetical cluster of 1,024 hosts corresponding to the number of MPI ranks used for each run.

The Ondes3D software package [13] is used for simulations, with MPI communication phases modeled using the Simgrid SMPI interface. We target a synthetic test case representative of regional seismic wave modeling, where the computational domains span several hundred kilometers in each direction with spatial resolutions of a few hundred meters. Overall, this represents 1,000 × 1,000 × 325 grid points. The focus of this paper is not on the physical results of the simulations but rather on the evaluation of key hardware parameters and their influence on parallel efficiency.

One of the key feature of Simgrid is the ability to model compute phases involving the evaluation of velocity and stress components. We define the compute power available on a dual-socket Intel Xeon Platinum 8490H platform (single core execution) as the reference. This enables the results to be easily scaled to other target platforms.

To thoroughly analyze performance, we perform a parameter sweep over both compute power and network bandwidth. This systematic exploration allows us to evaluate the compute-communication tradeoffs for the target application and study the impact of various network topologies on overall performance.

## B. What-if Analyses

Figure 1 illustrates the performance results obtained for the last two systems described in the previous section. The left panel depicts the performance gain relative to the reference hardware configuration (fat-tree topology), while the right panel presents the ratio of the elapsed time between the fat-tree and hierarchical topologies. In these experiments, we vary the compute power to evaluate potential speedups or slowdowns, scaling up to a factor of 1000x.

This high scaling factor could correspond to the capabilities of next-generation accelerators with advanced memory bandwidth capabilities. In such cases, a roofline model provides a straightforward way to project the compute-intensive phases onto future platforms. Similarly, we explore the effect of bandwidth scaling up to 10x relative to the reference configuration, allowing us to analyze the impact of increased networking capabilities.

**Compute-Communication ratio:** We observe a strong correlation between the increase in compute power and the

improvement in elapsed time, particularly for configurations with a small scaling factor for compute capabilities. In these cases, the total volume of computation significantly outweighs the cost of communication, resulting in nearly perfect overlapping of communication and computation phases. This overlap minimizes the impact of higher bandwidth values, while the compute power scaling factor becomes the primary determinant of performance.

However, the situation changes for regions of the graph where there is an unfavorable ratio between communication and computation. In such cases, the limited opportunities for overlap lead to noticeable performance dependence on networking parameters, such as bandwidth and topology. For example, when compute power is scaled by a factor of 100x, doubling the link bandwidth results in a linear increase in performance up to a threshold of 50 GB/s. Beyond this point, the cost of communication becomes negligible and is fully overlapped by computation.

In a hypothetical scenario where compute power is scaled by a very large factor, we observe a nonlinear improvement in elapsed time as bandwidth increases, with performance gains leveling off as the upper limit of bandwidth values is reached. This behavior can likely be attributed to application-specific factors, such as load imbalance or algorithmic characteristics that limit scaling efficiency.

**Network Topology:** The right panel of Figure 1 compares the performance of fat-tree and hierarchical topologies as the compute-communication ratio is varied. Both topologies exhibit similar trends for small scaling factors, as these configurations minimize the potential influence of networking parameters.

Overall, the fat-tree topology demonstrates slightly better performance than the hierarchical topology as compute power increases. This result warrants further investigation, as the fat-tree topology is primarily optimized for efficient collective communications. In our case, we implemented a typical scale-up to scale-out bandwidth ratio of three. However, these features may not align well with the requirements of the Ondes3D software package, which predominantly relies on point-to-point communications between neighboring processes.

Finally, increasing the bandwidth leads to nonlinear performance improvements, particularly in the context of multi-level topologies. This behavior highlights the importance of additional parameters that may need to be explored. For instance, factors such as message routing, the number of network hops involved, and potential congestion mechanisms must be carefully analyzed to fully understand the observed performance trends.

## V. CONCLUSION AND FUTURE WORKS

This paper presents our research on performance modeling for a representative application in the Earth Sciences domain, focusing on the computational demands of regional-scale seismic simulations. By utilizing two classical network
<figure><img src="image_2.png" /><figcaption>Fig. 1. Gain relative to the defined reference hardware configuration for the fat-tree topology (left panel) - Ratio of total elapsed time between the fat-tree and the hierarchical topologies (right panel).</figcaption></figure>

topologies, we evaluate and characterize the parallel behavior
of the Ondes3D software package across a range of scales. To
achieve this, we employed the SimGrid framework to simulate
standard runs on hypothetical parallel platforms, providing a
controlled environment to analyze performance dynamics.
Our findings underscore the benefits of advanced modeling
strategies in identifying critical inflection points where parallel
efficiency begins to degrade. By leveraging these insights,
we offer an in-depth evaluation of the interplay between
application-specific communication patterns and key hardware
characteristics, such as networking bandwidth and latency.
Looking ahead, we plan to extend this work to emerging
artificial intelligence (AI) workloads, particularly focusing
on large language models (LLMs) and other deep learning
frameworks that exhibit distinct parallelization patterns and
communication demands.

By adapting our approach to simulate AI workloads, we aim to
explore how network bandwidth, latency, and topology impact
the training and inference of LLMs at scale and further explore
results from analytical models described for instance in [14].
Furthermore, we intend to investigate strategies to optimize
resource allocation and communication in mixed workloads
combining HPC and AI applications.
An essential aspect of our future work involves validating the
simulation results on real hardware. This validation process
will allow us to refine our modeling framework further

REFERENCES

[1] K. Lakhotia, M. Besta, L. Monroe, K. Isham, P. Iff, T. Hoefler, and F. Petrini, "Polarfly: A cost-effective and flexible low-diameter topology," in *SC22: TX, USA, November 13-18, 2022*. IEEE, 2022, pp. 12:1–12:15.

[2] T. Hoefler, T. Bonato, D. D. Sensi, S. D. Girolamo, S. Li, M. Heddes, J. Belk, D. Goel, M. Castro, and S. Scott, "Hammingmesh: A network topology for large-scale deep learning," in *SC22: Dallas, TX, USA, November 13-18, 2022*. IEEE, 2022, pp. 11:1–11:18.

[3] C. E. Leiserson, "Fat-trees: Universal networks for hardware-efficient supercomputing," in *International Conference on Parallel Processing, ICPP'85, University Park, PA, USA, August 1985*. IEEE Computer Society Press, 1985, pp. 393–402.

[4] R. K. Tesser, L. M. Schnorr, A. Legrand, F. C. Heinrich, F. Dupros, and P. O. A. Navaux, "Performance modeling of a geophysics application to accelerate over-decomposition parameter tuning through simulation," *Concurr. Comput. Pract. Exp.*, vol. 31, no. 11, 2019.

[5] H. Casanova, A. Giersch, A. Legrand, M. Quinson, and F. Suter, "Versatile, scalable, and accurate simulation of distributed applications and platforms," *Journal of Parallel and Distributed Computing*, vol. 74, no. 10, pp. 2899-2917, Jun. 2014.

[6] L. Stanisic, S. Thibault, A. Legrand, B. Videau, and J. Méhaut, "Faithful performance prediction of a dynamic task-based runtime system for heterogeneous multi-core architectures," *Concurr. Comput. Pract. Exp.*, vol. 27, no. 16, pp. 4075-4090, 2015.

[7] A. F. Rodrigues, K. S. Hemmert, B. W. Barrett, C. D. Kersey, R. A. Oldfield, M. Weston, R. Risen, J. Cook, P. Rosenfeld, E. CooperBalls, and B. L. Jacob, "The structural simulation toolkit," *SIGMETRICS Perform. Evaluation Rev.*, vol. 38, no. 4, pp. 37-42, 2011.

[8] G. F. Riley and T. R. Henderson, "The ns-3 network simulator," in *Modeling and Tools for Network Simulation*, K. Wehrle, M. Günes, and J. Gross, Eds. Springer, 2010, pp. 15-34.

[9] J. Virieux and R. Madariaga, "Dynamic faulting studied by a finite difference method," *Bulletin of the Seismological Society of America*, vol. 72, no. 2, pp. 345-369, 1982.

[10] T. Furumura and L. Chen, "Large scale parallel simulation and visualization of 3D seismic wavefield using the earth simulator," *Computer Modeling in Engineering and Sciences*, vol. 6, pp. 153-168, 2004.

[11] D. Michéa and D. Komatitsch, "Accelerating a 3D finite-difference wave propagation code using GPU graphics cards," *Geophysical Journal International*, vol. 182, no. 1, pp. 389-402, 2010.

[12] M. Castro, E. Francesquini, F. Dupros, H. Aochi, P. O. A. Navaux, and J. Méhaut, "Seismic wave propagation simulations on low-power and performance-centric manycores," *Parallel Computing*, vol. 54, pp. 108-120, 2016.

[13] H. Aochi, T. Ulrich, A. Ducellier, F. Dupros, and D. Michea, "Finite difference simulations of seismic wave propagation for understanding earthquake physics and predicting ground motions: Advances and challenges," in *Journal of Physics: Conference Series*, vol. 454, no. 1. IOP Publishing, 2013, p. 012010.

[14] M. Isaev, N. McDonald, L. Dennison, and R. W. Vuduc, "Calculon: a methodology and tool for high-level co-design of systems and large language models," in *Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2023, Denver, CO, USA, November 12-17, 2023*, D. Arnold, R. M. Badia, and K. M. Mohror, Eds. ACM, 2023, pp. 71:1-71:14.