row show the expected behaviour : the execution time follows a $1/(\delta \times N_{\text{core}})$ slope (with $\delta$ often very close to 1). On the other hand, the first three benchmarks of the bottom row show bad OpenMP scalability, as they all lose performance when the number of threads is greater than a given threshold (specific to each application). We can see on some benchmarks (LU-MZ and Lulesh) that the last point of the curve shows a greater speedup than the previous points. However, since those points are outliers, we consider they do not impact the general slope of the curve.

One can observe that the Kripke benchmark displays the behaviour we described in section 6.1.1: two set of input parameters cause different OpenMP scalability. When executed with small sizes (i.e. x=16, y=8 and z=8, in figure 6.1g), the bad OpenMP scalability causes a slowdown instead of a speedup when large number of threads are used. However, when executed with large sizes (i.e. x=400, y=200, z=200, in figure 6.1h), we observe speedup even with a large number of threads, and the scalability follows the $1/N_{\text{core}}$ slope.

We consider that taking a core from the parallel computation will cause slowdown induced by the loss of computational power. As we have seen, if the OpenMP scalability is not good, taking a core from the computation will not cause a slowdown, but a speedup! Hence, since the impact on computation time is supposed to hinder the speedup gained thanks to the communication progression, we will consider the worst-case scenario for our model : good OpenMP scalability. This case is the one actually producing a slowdown which may prevent a dedicated core for progression to be effective.

As we have seen, most applications with a good OpenMP scalability exposes a $1/(\delta \times N_{\text{core}})$ slope (with $\delta$ very close to 1). Hence, in our model, we will approximate the computation slowdown induced by the removal of one computational core with the linear equation $1/N_{\text{core}}$.

## Modeling manycore cpu impact on thread scalability

The overhead expectation $t_{\text{overhead}}$ due to the core stealing impact can be modelled using the computation time ($t_{\text{comp}}$) from no progression execution. With our approximation, we estimate that the new computation time when removing $N_{\text{dedcore}}$ cores among $N_{\text{core}}$ is $t_{\text{newcomp}} = t_{\text{comp}} \times \frac{N_{\text{core}}}{N_{\text{core}} - N_{\text{dedcore}}}$. Specifically, it means that when taking one dedicated core for progression, the expected computation time is $t_{\text{newcomp}} = t_{\text{comp}} \times \frac{N_{\text{core}}}{N_{\text{core}} - 1}$. The induced slowdown is then computed with $t_{\text{overhead}} = t_{\text{newcomp}} - t_{\text{comp}}$. This equation can be simplified to $t_{\text{overhead}} = \frac{t_{\text{comp}}}{N_{\text{core}} - 1}$. For a 16-core node, taking a computational core will cause a $\frac{t_{\text{comp}}}{15}$ slowdown, and on a 64-core node, the slowdown will be $\frac{t_{\text{comp}}}{63}$. Thus, the more core a processor has, the more negligible the slowdown will be.

## Link to the measures and metrics in chapter 4

The model for computation is thus constructed using the $t_{\text{comp}}$ get from the execution. The chapter 4 introduced a timing for computation called $t_{\text{comp}}$ that also represent computation. However, these two timings do not represent the same computation part: $t_{\text{comp}}$ represent the computation part to be overlapped and placed between MPI call and MPI Wait. Thus, $t_{\text{comp}}$ is a sub part of $t_{\text{comp}}$ and is interesting for modelling MPI part as the
amount of computation between initialisation and wait impact the overlap capacity. This timing is however hard to measure as it have sense only from the MPI runtime point of view: i.e, this is a computation part defined by its both MPI calls bounds, but is also code executed outside the runtime. An envisaged solution would be to instrument the runtime to calculate this time from its two bounds.

The section 4.1.2 presented a concurrency ratio designed to measure the direct impact from the progression mechanism on the benchmark parallel computation. For dedicated core, this impact is minimal as resources are not shared and each core is not disturbed. However, the real slowdown caused by the dedicated core is the removal of one core to the application. This slowdown has the advantage of being much easier to model and predict than a slowdown due to threads interactions, as these interaction prediction would require to simulate the scheduling of each thread, the effects on caches and more.

### 6.1.3 Modeling MPI performance with dedicated core

The second part of the model estimates the decrease of the time spent in MPI functions thanks to a dedicated core.

An MPI distributed application usually includes different MPI calls. These MPI calls will not display the same behaviour when a core dedicated for background progression is used. To build an accurate model for the communication part when using a dedicated core, we need to classify the MPI functions regarding their response to the use of said dedicated core.

#### Classification of MPI calls

For our model we distinguish four types of MPI calls:

* blocking communication calls

* nonblocking communication initialisation calls

* nonblocking communication completion calls (e.g. MPI\_Test, MPI\_Wait)

* other MPI calls (e.g. runtime initialisation, communicators management, datatypes management, etc.)

With this classification, we decompose the total time spent in MPI in the following categories:

$$t_{MPI} = t_{MPIblocking} + t_{MPInonblocking} + t_{MPItest} + t_{MPIwait} + t_{MPIother}$$

with the following definitions:

* $t_{MPIblocking}$ is the time spent in blocking communication calls;

* $t_{MPInonblocking}$ is the time spent in nonblocking initialisation calls such as MPI\_Isend, MPI\_Irecv or MPI\_Iallgather;
* $t_{MPItest} + t_{MPIwait}$, the time spent in active progression functions such as `MPI Test` and `MPI Wait`;

* $t_{MPIother}$ is the remaining time spent in the MPI runtime (e.g. initialisation, communicationers management, datatypes management, etc.).

In the following sections, we will describe how each category of MPI call will behave when a dedicated core is used for background progression.

## Impact of dedicated core on each MPI call category

**Impact on nonblocking initialisation and completion calls** As we presented in section 5.1, when the MPI library implements no progression mechanisms, then nonblocking communications only progress inside MPI calls. These calls can be nonblocking initialisation calls (e.g. `MPI_Isend`, `MPI_Irecv`, `MPI_Iallgather`), completion calls (e.g. `MPI`, `MPI_Wait`), or any MPI calls involved in communications (even blocking primitives).

With a dedicated core, nonblocking communications are expected to progress on said dedicated core, thus not consuming computational power from the other cores. However, even if the communication itself takes place on the dedicated core, the time taken by the actual involved MPI calls is not completely nullified: the requests still have to be initialized and the associated operation registered in the MPI runtime for initialisation calls, ; the status of the request has to be fetched, with the relevant synchronisation, for completion calls.

We define $t_{minMPInonblockinit}$, $t_{minMPItest}$ and $t_{minMPIwait}$ to be the minimum time required to execute a nonblocking initialisation call, a test call and a wait call respectively, without performing any progression.

The minimum time for nonblocking initialisation calls and `MPI Test` calls is easy to meet. When an asynchronous progression mechanism is involved, the initialisation call job is to just fill in the request argument, then let the progression happen in background. So no extra time is taken for progression. For a `MPI Test` call, its job is just to test if the associated operation is finished. If not, it will let the background mechanism to progress the operation, and will not take extra time to realize such progression.

However, a call to `MPI_Wait` actually has to *wait* until the operation is done. If the operation is not finished when the call is performed, then it will block until the operation completes. Even if the progression happens in background, it cannot be overlapped by computations and it will not be hidden. Our model aims at predicting if an application will benefit from the use of nonblocking communications with a dedicated core for asynchronous progress. Because of the semantics of the `MPI_Wait` procedure, nonblocking communications by themselves are useful only if there is enough computation to hide the communications. Hence, in our model, we consider the best-case scenario for using nonblocking communications. We assume that, when transformed to use said nonblocking communications, the application exhibits enough computations between an initialisation call and its corresponding completion call to completely overlap the communication time. In this case, when using a dedicated core for progression, the execution time for the `MPI_Wait` calls will always be the minimum time.

Thus, if we consider that the application embeds $N_{nonblocking}$ nonblocking initialisation calls, $N_{test}$ `MPI Test` calls and $N_{wait}$ `MPI_Wait` calls, we model the time for all calls
involved in nonblocking communications to be:

$$t_{MPI_{\text{nonblock dedicated}}} = N_{\text{nonblocking}} \times t_{\text{minMPI_{nonblock}}} + N_{\text{test}} \times t_{\text{minMPItest}} + N_{\text{wait}} \times t_{\text{minMPIwait}}$$

**Impact on blocking calls** We have seen so far how the nonblocking calls already in the application will behave with a dedicated core. We will now see what would happen if blocking calls would be changed into nonblocking calls to benefit from the use of a dedicated core.

Due to algorithmic constraints, it is expected that even with a large refactoring of the application, not all blocking calls may be changed to nonblocking with overlap. Thus blocking communications in the original application will either remain blocking communications, or be changed in their nonblocking counterparts (hence adding the necessary initialisation and completion calls). Both cases will behave differently when a dedicated core is used.

For the first case, one can think that it is trivial, as dedicated core is used to progress nonblocking calls and not blocking calls. This is wrong. As we said in the previous part, blocking calls may help progress nonblocking communications. We decided to decompose the time of a blocking call $t_{MPI_{\text{blocking}}}$ in the time of the actual blocking communication $t_{MPI_{\text{blockingcom}}}$ and the time spent to progress pending nonblocking communications $t_{MPI_{\text{blockprogress}}}$:

$$t_{MPI_{\text{blocking}}} = t_{MPI_{\text{blockingcom}}} + t_{MPI_{\text{blockprogress}}}$$

If a dedicated core is used for progress, then blocking calls will not progress nonblocking communications anymore. Thus, $t_{MPI_{\text{blockprogress}}}$ becomes null, and we have:

$$t_{MPI_{\text{blocking dedicated}}} = t_{MPI_{\text{blocking com}}}$$

For the second case, as the blocking communication is transformed in a nonblocking communication, its time becomes similar to nonblocking communications. The most direct way to change a blocking communication to its nonblocking counterpart is to call the corresponding initialisation call, then `MPI_Wait` as its completion call. Thus, the new time when using a dedicated core for progression is: $t_{MPI_{\text{blocking transformed}}}$ = $t_{\text{minMPI_{nonblock}}} + t_{\text{minMPIwait}}$.

Let us consider an application with $N_{\text{blocking}}$ MPI blocking communications, and that a ratio $\alpha$ of these blocking communications are transformed in their nonblocking counterparts, the new time for these communications are:

$$t_{MPI_{\text{old blocking}}} = \alpha \times N_{\text{blocking}} \times t_{MPI_{\text{blocking transformed}}} + (1 - \alpha) \times N_{\text{blocking}} \times t_{MPI_{\text{blocking dedicated}}}$$

which can be developed in:

$$t_{MPI_{\text{old blocking}}} = \alpha \times N_{\text{blocking}} \times (t_{\text{minMPI_{nonblock}}} + t_{\text{minMPIwait}}) + (1 - \alpha) \times N_{\text{blocking}} \times t_{MPI_{\text{blocking com}}}$$
**Impact on the other MPI calls and full MPI model** The last type of MPI calls in our classification is other, which are the MPI calls that handle the library and internal structures management, but perform no communications and no progression. These calls are not impacted by the use of a dedicated core for progress, and the time $t_{MPIother}$ remains the same.

Thus, when putting together all parts of the MPI model and considering that a ratio $\alpha$ of blocking calls will be changed between the initial version of the application and the version we want to model, the MPI time when using a dedicated core is:

$$t_{MPIdedicated} = t_{MPInonblockdedicated} + t_{MPIoldblocking} + t_{MPIother}$$

which can be developed in:

$$t_{MPIdedicated} = N_{nonblocking} \times t_{minMPInonblock} + N_{test} \times t_{minMPItest} + N_{wait} \times t_{minMPIwait} + \alpha \times N_{blocking} \times (t_{minMPInonblock} + t_{minMPIwait}) + (1 - \alpha) \times N_{blocking} \times t_{MPIblockingcom} + t_{MPIother}$$

## Link to the measures in chapter 4

The chapter 4 defined the MPI time in two parts: the $t_{call}$ is the time spent in the MPI initialisation call and the $t_{wait}$ time spent in the MPI Wait. These two parts defined to compute metrics represent the duration on a single place of overlap. In real applications, overlap situations can be multiple. The definition of $t_{MPInonblocking}$ can then be defined as the sum of the $t_{call}$ from each MPI initialisation call in the application. Similarly, the $t_{MPIwait}$ can be defined as the sum of all $t_{wait}$.

## 6.1.4 Global model with MPI and computation

The final model is the combination of the estimated computation slowdown modelled in section 6.1.2, and the communication evolution modelled in section 6.1.3:

$$t_{dedicated} = t_{newcomp} + t_{MPIdedicated}$$

which can be developed in:

$$t_{dedicated} = t_{comp} \times \frac{N_{core}}{N_{core} - 1} + N_{nonblocking} \times t_{minMPInonblock} + N_{test} \times t_{minMPItest} + N_{wait} \times t_{minMPIwait} + \alpha \times N_{blocking} \times (t_{minMPInonblock} + t_{minMPIwait}) + (1 - \alpha) \times N_{blocking} \times t_{MPIblockingcom} + t_{MPIother}$$

The good cases where the dedicated core brings some improvement in the execution time should have a gain on the MPI part (coloured part in the model) higher than the overhead estimated in the computation part (black part in the model).
## 6.2 Gathering applications information to use the model

The whole model is based on timings for each part of the original application. These timings need to be measured on an execution of the original application. Several methods can be used to gather these timings, such as using a tool wrapping MPI calls, or having probes inside the MPI runtime.

In our experiments, we used the former method with the `mpiP` [68] tool. `mpiP` is a light-weight profiling library for MPI applications.

It generates logs gathering time spent in each MPI functions, number of calls, message sizes with minimum, maximum and average value. Information are collected for each MPI rank. This tool is generic enough to gather information on any tested application. However, because we only have application-level timings and not runtime-level timings, we will approximate some terms of our model.

First, we have to approximate the minimum time for nonblocking initialisation calls, `MPI Test` and `MPI Wait` calls. To do so, we consider the minimum measured time of each type of call when executing the application. Even when no dedicated core is used for progress, completion calls may not perform progress. When a completion call is used on a request, the associated operation may already be done, because it has been progressed by other previous MPI calls (blocking calls, or completion calls for other operations). If there is no other pending operation, then the completion call will just check the status of the request, and its timing will be the minimum. We assume this case happens at least once in an MPI run for nonblocking initialisation, `MPI Test` and `MPI Wait` calls. Thus, we take the minimum measured time for each of these as the respective timings for

$$t_{\min{MPI_{\text{nonblock}}}}, t_{\min{MPI_{\text{test}}}} \text{ and } t_{\min{MPI_{\text{wait}}}}$$

Second, without having probes in the MPI runtime, it is not possible to know which part of the time spend in a blocking call actually relates to the execution of the associated blocking operation, or if some of the measured times correspond to progression of pending nonblocking operations. For this reason, we consider that the time measured for each blocking call is fully dedicated to the associated operation, and that $t_{MPI_{blockprogress}}$ is always null.

These approximations may cause some discrepancies between our model forecasts and measured runs, but we will show that it remains accurate enough for our purpose.

## 6.3 Evaluation of the model

In this section, we evaluate how close to reality the model we propose is. We run the applications presented in section 6.1.2 on various data sets and number of nodes; it is especially important to use various data sets since the behaviour of some applications depends on inputs. Each run is performed in hybrid MPI+OpenMP mode, with one MPI process per node, and each compute resources of a node used by the OpenMP threads for computation. We performed our tests with the MPI implementation `MadMPI`, as it allows activating and deactivating asynchronous progression and use of a dedicated core [30]. The machines used are `inti/sandy-bridge` presented in section 6.1.2.

To evaluate the model, we compare its output to the real results obtained when running the applications with a dedicated core. This evaluation is limited to applications that
<figure><img src="image_52.png" /><figcaption>Figure 6.2: Comparison of the model prediction and the dedicated core case (Speedup to the time with no progression) for the Kripke application for different problem sizes on *inti/sandy-bridge*.</figcaption></figure>
<figure><img src="image_53.png" /><figcaption>Figure 6.3: Comparison of the model prediction and the dedicated core case (Speedup to the time with no progression) for NAS BT-MZ for different configurations on *inti/sandy-bridge*.</figcaption></figure>
<figure><img src="image_54.png" /><figcaption>Figure 6.4: Comparison of the model prediction and the dedicated core case (Speedup to the time with no progression) MiniMD for different configurations on *inti/sandy-bridge*.</figcaption></figure>
already leverage nonblocking communications. Since we only run unmodified applications,
in this section we have $\alpha = 0$.

Each combination of applications/parameters were run twice:

* First, we collect data to calibrate our model. We run the application without a dedicated core, and without any progression mechanisms for communications. This run is similar to what a user obtains with a regular MPI library without progression. We measure $t_{nopprogress}$ and we gather all the input parameters needed by the MPI model as explained in section 6.1.3, with the help of mpiP. We used bash scripts to get the data necessary to compute our predicted $t_{model}$ with an R script<fnref n="1" /> building our model presented in section 6.1.4.

* The second run uses one dedicated core mapped on the first logical core (e.g. core #0). We configure the OpenMP runtime to use $N_{core} - 1$ threads and they are bound on all other available cores (1 to $N_{core} - 1$). All progression mechanisms are enabled. This execution gives us the real value for $t_{dedicated}$.

The closer $t_{model}$ is to $t_{dedicated}$, the better the model is.

We run the model on three applications: BT-MZ from the NAS Parallel Benchmark, and Kripke and MiniMD from the CORAL benchmarks. The results are shown in Figure 6.2 for Kripke, Figure 6.3 for NAS BT-MZ, and Figure 6.4 for MiniMD. On these figures, the x-axis corresponds to executions on various data sets, and various number of nodes for BT-MZ and MiniMD. The y-axis shows the performance represented as a speedup compared to the basic execution without dedicated core. For each configuration, we display two values. The red cross corresponds to the real execution with a dedicated core, defined as $s_{dedicated} = \frac{t_{nopprogress}}{t_{dedicated}}$; the green bullet represents the performance predicted by the model, defined as $s_{model} = \frac{t_{nopprogress}}{t_{model}}$. Thus, the closer the red cross is from the green bullet, the more accurate our model is.

Kripke was run with 52 different data sets, depicted in Figure 6.2. We distinguish two types of behaviour: cases where the dedicated core brings a significant speedup (> 1.1), and cases where the dedicated core does not bring any benefit (speedup below or very close to 1). Note that beneficial cases are not grouped together, so it may be hard for a user to know specifically which case is a good candidate for running with a dedicated core. We observe that our model accurately predicts the behaviour of the application for each data set. For all cases where the model predicts a significant speedup, it is confirmed by the experimental execution. And, for all cases where the predicted speedup is close to 1, we see this exact behaviour with the experimental run.

For NAS BT-MZ (figure 6.3) and MiniMD (figure 6.4), the results are less identical. On NAS BT-MZ, the results for class C and D on 16 nodes, the predictions from the model are correct — the dedicated core brings no gain. However, for class D on 128 nodes, the model predicts a speedup higher than 3 where the reality is a mere 0.96. This is due to the communication scheme. Our model makes the assumption that nonblocking operations are always overlapped by computation. Unfortunately, in this application the nonblocking operations are not designed to overlap communications and computation;

<footnote n="1">¹The source code is available at <suppress for double-blind evaluation></footnote>
they are used to overlapping multiple communications. Indeed, the code calls a series of `MPI_Isend` and `MPI_Irecv` with a final `MPI_Waitall`, and no computation in between.

For MiniMD we observe that the first two predictions are correct and the third is much too optimistic. For the first two cases, the execution with a dedicated core is slower than without (see Fig 6.1a) and the model is accurate. For the last cases, when looking at the detailed values for all parameters of the model and compare them to the real execution, we observe an increased time spent in `MPI_Wait` with dedicated core (4,19s with no progression against 8,69s with dedicated core). This behaviour is sometimes observed when there is an interaction between the dedicated core and the `MPI_Wait`: they try to progress simultaneously. In those cases, the progression is not just done by the dedicated core and hence the gain is less than expected by the model.

We have shown here that the proposed model successfully predicts performance of a hybrid MPI application using a core dedicated to communication when the hypotheses are fulfilled. It exhibits a very good precision in this case, as seen with the Kripke application.

However, the user should be wary of using it blindly. It may give wrong results when hypotheses are not fulfilled: when no computation is executed at the same time as nonblocking communications, like in NAS BT-MZ; when the progression is still performed in `MPI_Wait` like in MiniMD, and when the OpenMP scalability is too far from linear.

In conclusion, the model is strong enough so as to be used to predict performance of dedicated core, but the user should always check all terms of the model and not only rely blindly on the total. These differences due to the approximation made in the model can help application developers find suboptimal behaviour related to the use of nonblocking communications in their applications.

## 6.4 Using the model

In this section, we will detail how to use the model to understand the compromise between computation slowdown and the communication speedup, to evaluate the effectiveness of nonblocking communication usage, and to predict the maximum performance we would get if blocking communications were converted to nonblocking.

### 6.4.1 Understanding the computation-slowdown/communication-speedup ratio

As we have seen for Kripke in figure 6.2, the application exhibits two types of behaviour with a dedicated core: either a significant speedup, or no gain at all. The beneficial cases are spread along the x-axis, showing that it does not depend on the problem sizes but instead on the communication scheme.

To understand this specific behaviour, we analyse all the different timings gathered in the basic run displayed in table 6.1, and the real value for $t_{dedicated}$ in addition. The case #1 is an example of successful use of dedicated core with 20% speedup; case #2 is typical of situations where the dedicated core does not bring performance gain. In both cases, the prediction of the model is correct with an error less than 2%.

Kripke features some calls to `MPI_Isend` and `MPI_Irecv`; they are progressed using `MPI_Testany` between computation phases. The communication is ended by a final call
<table><caption>Table 6.1: Values of model parameters for two sets of parameters of Kripke (times in seconds).</caption><thead><tr><th></th><th>Case #1</th><th>Case #2</th></tr></thead><tbody><tr><td>t<sub>noprogress</sub></td><td>147</td><td>155</td></tr><tr><td>t<sub>MPI</sub></td><td>73.4</td><td>8.3</td></tr><tr><td>t<sub>model</sub></td><td>119.82</td><td>156.75</td></tr><tr><td>t<sub>dedicated</sub></td><td>117</td><td>154</td></tr><tr><td>N<sub>core</sub></td><td>16</td><td>16</td></tr><tr><td>t<sub>comp</sub></td><td>73.6</td><td>145.7</td></tr><tr><td>N<sub>nonblocking</sub></td><td>480000</td><td>96</td></tr><tr><td>t<sub>minMPInonblock</sub></td><td>2.14e-05</td><td>2.16e-05</td></tr><tr><td>N<sub>test</sub></td><td>1119810</td><td>109679</td></tr><tr><td>t<sub>minMPItest</sub></td><td>2.05e-05</td><td>2.2e-05</td></tr><tr><td>N<sub>wait</sub></td><td>10000</td><td>2</td></tr><tr><td>t<sub>minMPIwait</sub></td><td>3.2e-05</td><td>7.19e-05</td></tr><tr><td>t<sub>MPIblocking</sub></td><td>11.7</td><td>4.98</td></tr><tr><td>t<sub>MPIother</sub></td><td>0.0</td><td>0.34</td></tr></tbody></table>

to `MPI_Waitall`. We observe in the table the main differences between the two runs is $N_{nonblocking}$: the first case uses a lot of nonblocking operations while the second has very few of them. It should be noticed that both cases execute approximately in the same duration, even though once decomposed, the timing details are very different.

In the first case, the application uses enough nonblocking operations for the communication speedup to overcome the computation slowdown. On the contrary, in the second case, the small number of nonblocking operations does not successfully counterbalance the computation slowdown. With our model, an application developer can better understand the dynamic behaviour of its code, and know the cases where the nonblocking operations are used. The real impacting factor is the communication scheme and the amount of communications compared to computation.

### 6.4.2 Evaluating the effectiveness of nonblocking communication

An application may have everything theoretically to gain time in terms of MPI overlappable communication and computation, and structurally never put computation and nonblocking communication in parallel. This is the case for NAS BT-MZ. As we have seen in Figure 6.3, on 128 nodes the model is too optimistic. This is due to the nonblocking operations being used not to overlap with computation, but to overlap multiple communications. Hence, even if globally there would be enough computation to overlap all nonblocking communications, it is not placed at the same time as the nonblocking communication.

If an application already uses nonblocking communications, the model may be used to diagnose pathological cases: by comparing an actual run and the prediction of the model, we can check whether overlap performs as expected or not, whether the computation inserted between the nonblocking operation and the corresponding wait is long enough or not. If the actual run is far from the prediction, there may be room for improvement in
<table><caption>Table 6.2: Values of model parameters for three sets of parameters of MiniMD (times in seconds).</caption><thead><tr><th></th><th>Case #1</th><th>Case #2</th><th>Case #3</th></tr></thead><tbody><tr><td>number of nodes</td><td>16</td><td>128</td><td>128</td></tr><tr><td>problem size</td><td>200<sup>3</sup></td><td>200<sup>3</sup></td><td>300<sup>3</sup></td></tr><tr><td>t<sub>noprocess</sub></td><td>112</td><td>17.9</td><td>57.3</td></tr><tr><td>t<sub>MPI</sub></td><td>23.4</td><td>8.17</td><td>22.4</td></tr><tr><td>t<sub>model</sub></td><td>98</td><td>12.61</td><td>54.9</td></tr><tr><td>t<sub>dedicated</sub></td><td>122</td><td>16.9</td><td>63.7</td></tr><tr><td>N<sub>core</sub></td><td>16</td><td>16</td><td>16</td></tr><tr><td>t<sub>comp</sub></td><td>94.55</td><td>9.73</td><td>34.9</td></tr><tr><td>N<sub>blocking</sub></td><td>5863</td><td>5863</td><td>263</td></tr><tr><td>t<sub>minMPInonblock</sub></td><td>2.50e-05</td><td>7.49e-05</td><td>1.03e-05</td></tr><tr><td>N<sub>wait</sub></td><td>5862</td><td>5862</td><td>5862</td></tr><tr><td>t<sub>minMPIwait</sub></td><td>5.33e-06</td><td>5.47e-06</td><td>5.59e-06</td></tr><tr><td>t<sub>MPIblocking</sub></td><td>19.16</td><td>4.99</td><td>0.41</td></tr><tr><td>t<sub>MPIother</sub></td><td>2.20</td><td>1.42</td><td>17.8</td></tr></tbody></table>

the organization of communications and computations.

We have observed the symmetrical case, where the application performs actually better with a dedicated core than predicted by the model. It is especially the case with Kripke, on the points of figure 6.2 with a speedup higher than 1.1. When looking at the detailed parameters we measured, we observe that with the dedicated core, $N_{test}$ drops compared to the reference run. The model assumes that `MPI Test` will be shorter with dedicated core, because it will not have to make communication progress in the call itself. But, in addition, $N_{test}$ is decreased, because the communication finished earlier. A more progression-compliant version of Kripke should remove the calls to `MPI Test` and let the dedicated core do all the progress work in the background. However, this requires a refactoring of the application. The lesson learned is that we may gain more than the sole cost of communication progression. In addition, we may gain the cost of all tests scattered throughout the code.

### 6.4.3 Potential gain of transforming blocking call to nonblocking

Finally, the model is able to predict the potential gain of transforming blocking operations to nonblocking ones. As described in section 6.1.3, the model can take into account the behaviour of an application if a ratio $\alpha$ of blocking operations are changed to their nonblocking counterparts.

We apply the model on MiniMD, on the three configurations used in section 6.3. MiniMD uses mostly blocking calls, with very few nonblocking operations. The figures for the three configurations are gathered in table 6.2. The predicted speedups for varying $\alpha$ for each configuration are displayed in figure 6.5. A ratio $\alpha = 0$ is equivalent to the original unmodified application. A value $\alpha = 1$ means that all blocking communications are transformed to their nonblocking counterparts. However, the ability to transform
<figure><img src="image_55.png" /><figcaption>Figure 6.5: Evolution of predicted speedup relative to the proportion of blocking call converted to nonblocking for MiniMD</figcaption></figure>
blocking to nonblocking calls depends on having available data-independent computation between the call and the wait. For most applications, $\alpha = 1$ is not reachable; we need to study the full range of values for $\alpha$.

The results for the first and third sets of parameters (red and blue line) show that, even in the unlikely event of being able to convert all communications to nonblocking, the expected gain is poor. Values shown in table 6.2 reveal that the number of blocking calls $N_{blocking}$ is low and the time spent in blocking communication $t_{MPIblocking}$ is low compared to the 1/16th overhead on computation. In the first case, for realistic values of $\alpha$, no gain may be expected. In the third case (blue line), the model predicts a very small gain for any value of $\alpha$. However, we know that there would be actually no gain at all, considering that we have already seen in section 6.3 that our model is optimistic and overestimate by 12% the potential gain for this precise application.

The second set of parameters (green line) is the same as the first, except for the higher number of nodes. The predictions of the model, however, are very different, with a predicted significant performance improvement, even for moderate values of $\alpha$. With the higher number of nodes compared to the first case, the computation time is lower, and thus the communication time gets a higher proportion of the total time, which is enough to compensate for the slowdown cause by one less core for computation with a dedicated core.

The slope of each case plot is the most interesting metric as it represents the efficiency of blocking to nonblocking transformation. The first and the second cases have a positive slope since the time spent in blocking communications $t_{MPIblocking}$ is much larger than what is lost in computation speed with a dedicated core; in contrast, the third case has a quasi-neutral slope. Even with $t_{MPI}$ representing a large part of the total time, the proportion of $t_{MPIblocking}$ is negligible. As a consequence, $\alpha$ has a very low impact on the potential gain and the transformation is not efficient. Thus, for every case, we have to evaluate the potential gain by running the model with the parameters from the application.

This model is thus able to discriminate the cases where a transformed application will be effective or not. Taking into account the base point of the $\alpha$ plot, it can show the current state of the dedicated core effect on the application. At last, looking at the slope for used case of the application will show the impact of the transformation on the performances using a dedicated core.
# Chapter 7

## Conclusion

MPI is one of the most used programming model to exploit distributed memory systems. It allows the nodes composing the cluster to communicate using messages. However, performing communication has a cost and impact the overall performance. The overlap of computation and communication is a method used to reduce the impact of the communications MPI include notably nonblocking operations designed to execute independent computation during the communication on the network and enable overlap.

We showed that to efficiently have communication and computation operate simultaneously, the MPI runtime needs computation power to progress the communications. Moreover, MPI in its 3.0 version implement nonblocking collectives. The need for progression of these collectives is greater as these algorithms can't be only launched and let running on their own: they need to be run regularly all along the collective duration.

The necessity of progression to efficiently overlap nonblocking collectives is at the origin of new mechanisms developed for this purpose in MPI runtimes. Indeed, the application often rely on shared memory programming model such as OpenMP to benefit from multicore nodes. Thus associate a good overlap and a minimal impact on the global performance remains an issue. This thesis focused on the idea of progressing MPI communications using dedicated resources. The use of a dedicated core to remedy this problem come with multiple aspects that need to be taken into account such as:

* Creating a mechanism able to efficiently progress communications without degrading the performance

* Managing the resources required by both MPI runtime for the progression and the application computation.

* Understanding the causes and find solutions to diagnose bad overlap.

* Give a clue on the effect of a dedicated core in function of the application and system configuration.

We first presented the bases of high performance computing including hardware architectures and their evolution to the current problem we face up to. We saw the different
software technologies such as programming models used to operate these complex hardware
systems. We saw that benchmarking methods used to evaluate the MPI runtimes may
miss important behaviour to assess the performance of progression mechanisms.

In this thesis, we propose a global solution to efficiently progress communications and especially nonblocking collective in order to overlap communication and computation. The final purpose is then to benefit from this overlap to improve the performance of hybrid applications. This solution is composed of multiple axis: we propose a new benchmarking method and metrics focusing on the evaluation of nonblocking collective progression. This benchmarking method have for purpose to be use as a base to measure the overlap capability and monitor the effects of progression mechanism on overall performances. We thus propose a new mechanism to progress nonblocking collective, relying on task-based algorithm and dedicated core. The use of dedicated core have shown an efficient capability of gaining performances through overlap, but requires to be used in the good conditions concerning the system and application structure.
Thus, we finally propose a model to predict the effects of using this dedicated core mechanism on real hybrid MPI + OpenMP applications. This model can be used to simulate the potential gain when converting blocking communication application to use nonblocking operations.

# 7.1 Contributions

The contributions made in this thesis have followed three main axes: the benchmarking method for nonblocking collective, the conception of a progression mechanism based on a dedicated core and the modelling of this mechanism on the performance of hybrid MPI + OpenMP applications.

## 7.1.1 Nonblocking collective benchmarking

We first proposed a full methodology to not only measure the overlap capability of
mechanisms but also their impact on the overall system. This methodology relies first on
new metrics to accurately measure the performance of nonblocking collective with the
definition of an overhead ratio. They also assess the global behaviour of the runtime with
the computation and communication concurrency ratio.

We then proposed an implementation addressing multiple common issues encountered when benchmarking distributed operations. Such problems include the time variation between MPI processes, the management of time synchronisation between nodes clocks, the requirement of a simultaneous start, and finally the necessity of controlling the variable amount of computation and communication done in the overlapped situation compared to the sequential one.

Finally we used this benchmark to run a survey on the state-of-the-art most common MPI libraries, and the different progression mechanisms they use. This showed the impact of these mechanisms on the system and how they fail to manage resources in order to gain performances: the progression mechanism usually consume a non-negligible amount of computation power and hinder the application running on the whole system. This brings the necessity to create a new mechanism focusing on these points.
### 7.1.2 Dedicated core for progression

The chapter 5 presented an overview of software necessities required to progress communication using dedicated cores. We then illustrated this overview with the conception needed for adapting the NewMadeleine + Pioman runtime to use dedicated core for the progression of nonblocking collective. We first analysed the benefits of using a dedicated core for this purpose and how this mechanism fits for this work.

We then proposed an implementation of a specific worker in Pioman tuned and adapted to be executed on a dedicated core. We proposed an implementation of nonblocking collective algorithms in NewMadeleine based on event handlers to be efficiently executed by the Pioman dedicated core worker.

Finally, we evaluated the progression capacity of this global configuration using the BenchNBC benchmark on the madMPI MPI interface on top of NewMadeleine.

The dedicated core evaluation showed a good capability of progressing communications to gain performances. However, to be used in a real context, applications must respect constraints that need to be precisely defined to benefit from this solution.

### 7.1.3 Hybrid application with dedicated core modelling

The final chapter proposed a solution to classify applications following their relevance to be progressed with dedicated core in order to gain performances. We focused on the modelling of hybrid MPI + OpenMP applications performance using the dedicated core progression mechanism.

We defined a predicting model focusing on analysing the performance evolution of application using a dedicated core. The model is split in two parts: One computing the loss due to the removal of one core from the application, and the other the gain of overlapping the application computation and communication using the dedicated core.

We introduced the methodology used to gather the necessary input from application to feed the model. This model thus rely on universal input which can be found in every MPI + OpenMP applications such as MPI operation or computation duration.

We evaluated this model using multiple common and representative HPC applications, such as Kripke, MiniMD and Lulesh. Finally, we ran our model to predict the benefit of transforming blocking calls to nonblocking calls on these applications based on the understanding of computation slowdown and communication speed-up ratio.

The evaluation of the model demonstrated its capacity to distinguish cases to use dedicated core and cases to not use it. It also showed its usage to hint developers about the efficiency to re-design blocking applications to nonblocking ones.

### 7.1.4 Answers about the dedicated core usage

The set of contributions presented in this thesis bring answers to the problem of the effective progression of nonblocking collectives. The combination of these contributions not only helps the effective progression of nonblocking collective to gain performances, but also gives clues about the requirements to do so. It also defines an environment to use these primitives.
Globally, these contributions gives an answer to the questions of the nonblocking collective usage such as: How to gain performance using a dedicated core, what are the runtime requirements to gain performance with nonblocking collectives and when to use the dedicated core.

## 7.2 Perspectives

The contributions presented in this thesis open multiple opportunities and perspectives concerning MPI benchmarking and modelling. There are also works to continue concerning the dedicated core implementation.

### 7.2.1 Use of MPI nonblocking collectives benchmark for MPI library improvement

The benchmark method and implementation presented in the chapter 4 propose to diagnose the behaviour of MPI library. It could however be enhanced to fully cover the need of a complete benchmark.

**Implementation of all MPI nonblocking collectives benchmark**

The benchmark currently covers multiple collectives, however some remains not supported for now. Especially the support of collective with various data size (e.g, `MPI_Igatherv`, `MPI_Ialltoallv`) is an interesting point as these operations introduce the exchange of different data length. The main purpose of benchmarking these functions is to measure the impact of the communication unbalance on the progression, overlap, and overall behaviour of the system. To do so, the benchmark must introduce a new feature to assess the variation of different deltas between lengths used.

**Work on a microbenchmark feature**

The current implementation is conceived to test an overall set of computation and communication time, in order to give a general overview of the performance. Another perspective is to be able to test specific cases of overlap. These test cases could be an association of a computation time, a communication time and a collective. Such microbenchmark would be useful to stick to the numerous real cases of actual applications.

**Improve MPI libraries**

The use of this complete methodology has for purpose to be used to expose weak points of MPI libraries concerning their progress mechanisms, in order to improve these points. This benchmark has shown performance issues concerning these progress mechanisms and identified the causes of these issues.

The combinations of existing metrics and future ones presented in this section could then be used as a tool to correct future implementation of multiple MPI libraries such as MPICH or OpenMPI.
### 7.2.2 Evolution of the dedicated core impact model

The current model presented in chapter 6 uses the mpiP framework to gather the information needed as input for the model. mpiP is a generic tool and lacks some timings that could greatly improve the model.

#### Get more timing on the computation part

One point to be refined in the model is the management of computation time. Indeed, with the current instrumentation, we have no idea of the actual amount of the computation respecting the constraint to be overlapped with communication. The first mandatory one being the part of the computation located between the MPI initialisation call and the MPI Wait. Among this computation part, another constraint is to be sure that the data computed is actually independent of the data transferred over the network. However, this must already be the case from the conception of the code.

The section 4.4 showed that the computation have a direct impact on the overlap capability of a runtime. Especially in function of the unbalance between computation and communication time, the progression mechanism can be much less effective than with a balanced case.

This independent computation timing would thus be a precious addition to the model as it would allow it to simulate this efficiency loss calculated from the unbalance ratio.

#### Instrumenting the runtime

One solution envisaged to get these timings is to rely directly on a specific mode of an MPI runtime to gather the information we need. The modification of MPI initialisation calls and `MPI_Wait` would allow to measure the time between those two calls.

Coupled with the use of a dedicated core, we ensure that the time between these two bounds is not hindered by the MPI runtime.

#### Study on application transformation to overlap

The model in its current state is able to help to decide on the efficiency of transforming blocking application to nonblocking application. It could then be interesting to run a study on common and most used applications to evaluate the possibility of transforming these applications in order to gain performances.

Based on the proposed improvements, the enhanced model could gather quantitative information on the performance of these applications. Thus, it could be possible to measure the potential gain of this transformation to give a clearer hint on the cost/gain ratio of this operation.

### 7.2.3 Multiple runtime centralisation of resources management

The seek of performance in high performance computing imply to rely on clusters composed of numerous processors sharing memory and distributed nodes. To make the best of these complex hardware architectures, simulations and most widely applications may use and
associate multiple programming models. This combination of different runtimes on system
bring issues for the management of computation resources provided by the cluster.

A tool for global resource allocation

To distribute the resources between the runtime we used along this thesis, we relied on manually setting the placement of threads for each used runtime. Despite being a fully working solution, this has the disadvantage of requiring to resolve the resource problem with every different cases. The potential case of having more than two runtimes could make this resource issue even more complicated. To allow the good cooperation between those runtimes, an idea could be to develop a tool with the global overview of the available resources. From this global view, it could then take decision on how to allocate resources for the given runtimes used by applications.

A task based global runtime

Different runtimes may sometimes propose features with a similar workload management.
The use of tasks has shown a great capacity for the progression of communications.
Moreover, we presented hybrid MPI + OpenMP applications which represent a very
common association of runtimes. OpenMP, on its side, also features tasks to be executed
by its runtime. Thus, it is possible to have both OpenMP running a task engine to
execute parallel tasks, and an MPI runtime relying on tasks to achieve the communication
progression work.

The use of a global runtime implementing both OpenMP and MPI standard could thus unify the task engine to be used by all runtimes when needed. The conception of that solution would allow runtime to thus share resources for similar work. However, conceiving a global task engine requires combining the requirements for each runtime. For instance, OpenMP tasks usually rely on data dependencies to organise and execute the "taskified" workload. On its side, the MPI implementation presented in this thesis use events to trigger the available tasks. Thus, the design differences of each task engine is an issue in order to implement such global engine.

The notion of unifying runtimes into a global implementation have already been covered.
MPC is a framework including implementations for both MPI and OpenMP. It could be
used as a base to develop a task-based progression environment for MPI using a global
task engine for both the OpenMP and MPI runtimes.
Bibliography

[1] “Pvm website.” https://www.csm.ornl.gov/pvm/.

[2] D. J. Becker, T. Sterling, D. Savarese, J. E. Dorband, U. A. Ranawak, and C. V. Packer, “Beowulf: A parallel workstation for scientific computation,” in *Proceedings, International Conference on Parallel Processing*, vol. 95, pp. 11–14, 1995.

[3] N. Boden, D. Cohen, R. Felderman, A. Kulawik, C. Seitz, J. Seizovic, and W.-K. Su, “Myrinet: a gigabit-per-second local area network,” *IEEE Micro*, vol. 15, no. 1, pp. 29–36, 1995.

[4] M. Pérache, H. Jourdren, and R. Namyst, “Mpc: A unified parallel runtime for clusters of numa machines,” in *Proceedings of the 14th International Euro-Par Conference on Parallel Processing*, Euro-Par ’08, (Berlin, Heidelberg), p. 78–88, Springer-Verlag, 2008.

[5] F. Broquedis, J. Clet-Ortega, S. Moreaud, N. Furmento, B. Goglin, G. Mercier, S. Thibault, and R. Namyst, “hwloc: a Generic Framework for Managing Hardware Affinities in HPC Applications,” in *PDP 2010 - The 18th Euromicro International Conference on Parallel, Distributed and Network-Based Computing* (IEEE, ed.), (Pisa, Italy), Feb. 2010.

[6] “Gomp website.”

[7] C. Augonnet, S. Thibault, R. Namyst, and P.-A. Wacrenier, “StarPU: A Unified Platform for Task Scheduling on Heterogeneous Multicore Architectures,” in *Euro-Par 2009*, LNCS, (Delft, Netherlands), Aug. 2009.

[8] O. Aumage, E. Brunet, N. Furmento, and R. Namyst, “NewMadeleine: a Fast Communication Scheduling Engine for High Performance Networks,” in *Workshop on Communication Architecture for Clusters (CAC 2007)*, workshop held in conjunction with IPDPS 2007, (Long Beach, California, United States), Mar. 2007.

[9] S. Derradji, T. Palfer-Sollier, J.-P. Panziera, A. Poudes, and F. W. Atos, “The bxi interconnect architecture,” in *2015 IEEE 23rd Annual Symposium on High-Performance Interconnects*, pp. 18–25, 2015.

[10] “Mpich website.” https://www.mpich.org/.

[11] O. Aumage, E. Brunet, N. Furmento, and R. Namyst, “New madeleine: a fast communication scheduling engine for high performance networks,” in *2007 IEEE International Parallel and Distributed Processing Symposium*, pp. 1–8, March 2007.
[12] D. K. Panda, H. Subramoni, C.-H. Chu, and M. Bayatpour, “The mvapich project: Transforming research into high-performance mpi library for hpc community,” *Journal of Computational Science*, vol. 52, p. 101208, 2021. Case Studies in Translational Computer Science.

[13] “Intelmpi website.” https://www.intel.com/content/www/us/en/developer/tools/oneapi/mpi-library.html#gs.m7ly1i.

[14] Q. Snell, A. R. Mikler, and J. L. Gustafson, “Net-pipe: Network protocol independent performance evaluator,” 1997.

[15] Intel Corporation, “IMB-NBC benchmarks.” https://github.com/intel/mpi-benchmarks.

[16] T. O. S. University, “Omb (osu microbenchmarks).” http://mvapich.cse.ohio-state.edu/benchmarks/.

[17] R. Reussner, P. Sanders, and J. L. Traff, “SKaMPI: A comprehensive benchmark for public benchmarking of MPI,” *Sci. Program.*, vol. 10, p. 55–65, Jan. 2002.

[18] S. Hunold, A. Carpen-Amarie, and J. L. Träff, “Reproducible MPI micro-benchmarking isn’t as easy as you think,” in *Proceedings of the 21st European MPI Users’ Group Meeting*, EuroMPI/ASIA ’14, (New York, NY, USA), p. 69–76, Association for Computing Machinery, 2014.

[19] D. Becker, R. Rabenseifner, and F. Wolf, “Implications of non-constant clock drifts for the timestamps of concurrent events,” pp. 59–68, 09 2008.

[20] S. Hunold and A. Carpen-Amarie, “On the impact of synchronizing clocks and processes on benchmarking MPI collectives,” in *EuroMPI*, pp. 8:1–8:10, ACM, 2015.

[21] A. Nigay, L. Mosimann, T. Schneider, and T. Hoefler, “Communication and timing issues with mpi virtualization,” in *27th European MPI Users’ Group Meeting*, EuroMPI/USA ’20, (New York, NY, USA), p. 11–20, Association for Computing Machinery, 2020.

[22] T. Hoefler, A. Lumsdaine, and W. Rehm, “Implementation and performance analysis of non-blocking collective operations for mpi,” p. 52, 01 2007.

[23] T. Hoefler, T. Schneider, and A. Lumsdaine, “Accurately Measuring Overhead, Communication Time and Progression of Blocking and Nonblocking Collective Operations at Massive Scale,” *International Journal of Parallel, Emergent and Distributed Systems*, vol. 25, pp. 241–258, Jul. 2010.

[24] A. Denis and F. Trahay, “MPI Overlap: Benchmark and Analysis,” in *International Conference on Parallel Processing*, 45th International Conference on Parallel Processing, (Philadelphia, United States), Aug. 2016.

[25] MPI Forum, *MPI: a Message Passing Interface Standard V1.0*. 1993.
[26] D. R. Martinez, J. C. Cabaleiro, T. F. Pena, F. F. Rivera, and V. B. Perez, "Performance modeling of mpi applications using model selection techniques," in *2010 18th Euromicro Conference on Parallel, Distributed and Network-based Processing*, pp. 95-102, 2010.

[27] S. Sur, H.-W. Jin, L. Chai, and D. Panda, "R545250odma read based rendezvous protocol for mpi over infiniband," pp. 32-39, 01 2006.

[28] T. Hoefler and A. Lumsdaine, "Message progression in parallel computing - to thread or not to thread?," in *2008 IEEE International Conference on Cluster Computing*, pp. 213-222, Sep. 2008.

[29] M. Si, A. Peña, P. Balaji, M. Takagi, and Y. Ishikawa, "Mt-mpi: multithreaded mpi for many-core environments," 06 2014.

[30] A. Denis, "pioman: a pthread-based Multithreaded Communication Engine," in *Euromicro International Conference on Parallel, Distributed and Network-based Processing*, (Turku, Finland), Mar. 2015.

[31] M. Miwa and K. Nakashima, "Progression of mpi non-blocking collective operations using hyper-threading," pp. 163-171, 03 2015.

[32] M. Jiayin, S. Bo, Y. Wu, and Y. Guangwen, "Overlapping communication and computation in mpi by multithreading.," pp. 52-57, 01 2006.

[33] MPI Forum, *MPI - a Message Passing Interface Standard V3.0*. Ean 1114444410030. English Book Service, 2012.

[34] D. Buettner, J.-T. Acquaviva, and J. Weidendorfer, "Real asynchronous mpi communication in hybrid codes through openmp communication tasks," pp. 208-215, 12 2013.

[35] M. Wittmann, G. Hager, T. Zeiser, and G. Wellein, "Asynchronous mpi for the masses," 02 2013.

[36] G. Hager, G. Schubert, T. Schoenemeyer, and G. Wellein, "Prospects for truly asynchronous communication with pure mpi and hybrid mpi/openmp on current supercomputing platforms," 01 2011.

[37] S. Paul, M. Araya, J. Mellor-Crummey, and D. Hohl, "Performance analysis and optimization of a hybrid seismic imaging application," *Procedia Computer Science*, vol. 80, pp. 8-18, 06 2016.

[38] A. Ruhela, H. Subramoni, S. Chakraborty, M. Bayatpour, P. Kousha, and D. Panda, "Efficient asynchronous communication progress for mpi without dedicated resources," pp. 1-11, 09 2018.

[39] H.-V. Dang, M. Snir, and W. Gropp, "Towards millions of communicating threads," in *Proceedings of the 23rd European MPI Users' Group Meeting*, EuroMPI 2016, (New York, NY, USA), p. 1-14, Association for Computing Machinery, 2016.
[40] A. Friedley, G. Bronevetsky, T. Hoefler, and A. Lumsdaine, "Hybrid MPI: efficient message passing for multi-core systems," in *SC'13: Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis*, pp. 1-11, IEEE, 2013.

[41] M. Si, A. J. Peña, P. Balaji, M. Takagi, and Y. Ishikawa, "MT-MPI: Multithreaded MPI for many-core environments," in *Proceedings of the 28th ACM International Conference on Supercomputing, ICS '14*, (New York, NY, USA), p. 125-134, Association for Computing Machinery, 2014.

[42] P. Carribault, M. Pérache, and H. Jourdren, "Enabling low-overhead hybrid MPI/OpenMP parallelism with MPC," in *Beyond Loop Level Parallelism in OpenMP: Accelerators, Tasking and More, Proceedings of the 6th International Workshop on OpenMP (IWOMP 2010)* (M. Sato, T. Hanawa, M. Müller, B. Chapman, and B. de Supinski, eds.), vol. 6132 of *Lecture Notes in Computer Science*, pp. 1-14, Springer Berlin Heidelberg, 2010.

[43] S. White and L. V. Kale, "Optimizing point-to-point communication between adaptive mpi endpoints in shared memory," *Concurrency and Computation: Practice and Experience*, vol. 32, no. 3, p. e4467, 2020. e4467 cpe.4467.

[44] N. Hjelm, M. G. F. Dosanjh, R. E. Grant, T. Groves, P. Bridges, and D. Arnold, "Improving MPI multi-threaded RMA communication performance," in *Proceedings of the 47th International Conference on Parallel Processing*, ICPP 2018, (New York, NY, USA), Association for Computing Machinery, 2018.

[45] A. Ruhela, H. Subramoni, S. Chakraborty, M. Bayatpour, P. Kousha, and D. K. (DK) Panda, "Efficient design for MPI asynchronous progress without dedicated resources," *Parallel Computing*, vol. 85, pp. 13-26, 2019.

[46] R. E. Grant, M. G. F. Dosanjh, M. J. Levenhagen, R. Brightwell, and A. Skjellum, "Finepoints: Partitioned multithreaded mpi communication," in *High Performance Computing* (M. Weiland, G. Juckeland, C. Trinitis, and P. Sadayappan, eds.), (Cham), pp. 330-350, Springer International Publishing, 2019.

[47] J. H. Göbbert, H. Iliev, C. Ansorge, and H. Pitsch, "Overlapping of communication and computation in nb3dfft for 3d fast fourier transformations," pp. 151-159, 02 2017.

[48] T. Straatsma and D. Chavarría-Miranda, "On eliminating synchronous communication in molecular simulations to improve scalability," *Computer Physics Communications*, vol. 184, pp. 2634-2640, 12 2013.

[49] K. Al-Tawil and C. A. Moritz, "Performance modeling and evaluation of mpi," *Journal of Parallel and Distributed Computing*, vol. 61, no. 2, pp. 202-223, 2001.

[50] L. Adhianto and B. Chapman, "Performance modeling of communication and computation in hybrid mpi and openmp applications," *Simulation Modelling Practice and Theory*, vol. 15, no. 4, pp. 481-491, 2007. Performance Modelling and Analysis of Communication Systems.
[51] J. C. Sancho, K. J. Barker, D. J. Kerbyson, and K. Davis, "Quantifying the potential benefit of overlapping communication and computation in large-scale scientific applications," in *SC'06: Proceedings of the 2006 ACM/IEEE Conference on Supercomputing, pp. 17–17*, IEEE, 2006.

[52] R. Brightwell, R. Riesen, and K. Underwood, "Analyzing the impact of overlap, offload, and independent progress for message passing interface applications.," *IJHPCA*, vol. 19, pp. 103–117, 01 2005.

[53] V. Gupta, H. Kim, and K. Schwan, "Evaluating scalability of multi-threaded applications on a many-core platform," tech. rep., Center for Experimental Research in Computer Systems, Georgia Tech, 05 2012.

[54] A. Daumen, P. Carribault, F. Trahay, and G. Thomas, "Scalomp: Analyzing the scalability of openmp applications," in *OpenMP: Conquering the Full Hardware Spectrum* (X. Fan, B. R. de Supinski, O. Sinnen, and N. Giacaman, eds.), (Cham), pp. 36–49, Springer International Publishing, 2019.

[55] C. Iwainsky, S. Shudler, A. Calotoiu, A. Strube, M. Knobloch, C. Bischof, and F. Wolf, "How many threads will be too many? on the scalability of openmp implementations," in *Euro-Par 2015: Parallel Processing* (J. L. Träff, S. Hunold, and F. Versaci, eds.), (Berlin, Heidelberg), pp. 451–463, Springer Berlin Heidelberg, 2015.

[56] M. Woodyard, "An experimental model to analyze openmp applications for system utilization," in *OpenMP in the Petascale Era* (B. M. Chapman, W. D. Gropp, K. Kumaran, and M. S. Müller, eds.), (Berlin, Heidelberg), pp. 22–36, Springer Berlin Heidelberg, 2011.

[57] N. Drosinos and N. Koziris, "Performance comparison of pure mpi vs hybrid mpi-openmp parallelization models on smp clusters," in *18th International Parallel and Distributed Processing Symposium, 2004. Proceedings.*, pp. 15–, 2004.

[58] S. Derradji, T. Palfer-Sollier, J. Panziera, A. Poudes, and F. W. Atos, "The BXI inter-connect architecture," in *2015 IEEE 23rd Annual Symposium on High-Performance Interconnects*, pp. 18–25, 2015.

[59] D. L. Mills, "Internet time synchronization: the network time protocol," *IEEE Transactions on Communications*, vol. 39, no. 10, pp. 1482–1493, 1991.

[60] S. Hunold and A. Carpen-Amarie, "Hierarchical clock synchronization in mpi," in *2018 IEEE International Conference on Cluster Computing (CLUSTER)*, pp. 325–336, 2018.

[61] "Bench NBC website." http://pm2.gitlabpages.inria.fr/bench_nbc/.

[62] A. Denis, J. Jaeger, E. Jeannot, M. Pérache, and H. Taboada, "Study on progress threads placement and dedicated cores for overlapping mpi nonblocking collectives on manycore processor," *The International Journal of High Performance Computing Applications*, vol. 33, no. 6, pp. 1240–1254, 2019.
[63] M. P. I. Forum, “MPI: A message-passing interface standard – version 4.0,” June 2021.

[64] H. J. R.F. Van Der Wijngaart, “Nas parallel benchmarks, multi-zone versions,” tech. rep., NASA Ames Research Center, Moffett Field, CA, 2003.

[65] P. Crozier, H. Thornquist, R. Numrich, A. Williams, H. Edwards, E. Keiter, M. Rajan, J. Willenbring, D. Doerfler, and M. Heroux, “Improving performance via mini-applications,” 01 2009.

[66] I. Karlin, J. Keasler, and R. Neely, “Lulesh 2.0 updates and changes,” Tech. Rep. LLNL-TR-641973, August 2013.

[67] A. J. Kunen, T. S. Bailey, and P. N. Brown, “Kripke - a massively parallel transport mini-app,” 6 2015.

[68] J. Vetter and C. Chambreau, “mpip: Lightweight, scalable mpi profiling,” URL: http://www.llnl.gov/CASC/mpiP, 01 2005.