（様式 20）成果報告書

TSUBAME 共同利用 令和 4 年度 学術利用 成果報告書

利用課題名 MPI アプリケーションにおけるプロファイルおよびトレース予測手法の評価
英文： Evaluation of a Method for Predicting Profiles and Traces of MPI Applications
利用課題責任者 三輪忍First name Surname Shinobu Miwa所属 電気通信大学
Affiliation The University of Electro-Communications
URL http://www.hpc.is.uec.ac.jp/miwa_lab/index.html
邦文抄録（300 字程度）
本研究では，小規模実行によって得た性能メトリクスを用いて大規模実行時の並列アプリケーションの性能メトリクスを予測するツールである Extra-P を関数コール回数予測に応用した場合の評価を行う．我々の実験結果によると，Extra-P は 5 つの並列アプリケーションに対して関数コール回数を 2.23%以下の誤差で予測できた．また，我々は，関数コール回数予測の応用として，関数コール予測の結果をも言いて関数の実行時間を予測する手法を提案する．我々の実験結果によると，提案手法はさまざまな関数の総実行時間の予測精度を平均 1.41%改善した．
英文抄録（100 words 程度）
In this study, we provide the first evaluation of Extra-P, which is a tool for predicting the performance
metrics of a parallel application executed at a large scale using the performance metrics of theapplication executed at multiple small scales, in a use case of function call count prediction. Our
experimental results demonstrated that Extra-P predicted the function call counts for five parallel
applications within an error of 2.23%. Additionally, we propose a technique to predict the total
execution time of a function using the call count predicted as a use case of function call countprediction. Our experimental results demonstrated that the proposed technique improved theprediction accuracy of the total execution time of various functions by 1.41% on average.

Keywords: Extra-P, parallel applications, function call count, evaluation

1. Introduction

In this context, various methods for modeling

The analysis of the dynamic behavior of parallel

the performance of parallel applications haveapplications

bybeen developed. These methods enable us tosupercomputing users for various purposes (e.g.,collect the various performance metrics (e.g.,

performance tuning). Because an application's

execution time) of a given application executed at

behavior depends on the scale of execution (i.e.,a large scale with reasonable computing time andprocess count and input data), the collection of

resources, by creating their scalability models

performance metrics (e.g., function call, andempirically or analytically. Specifically, Extra-P,

memory and cache access counts) usually requires

which

the execution of a targeted application combined

performance analysis of parallel applications,with a profiling tool (e.g., TAU) at a targeted scale.

automatically generates a model to provide the

However,theextrapolated performance from a subset of metricperformance metrics of large-scale applications

data of a given application executed at small

consumes a large amount of computing time andscales. In previous work, the authors reported

resources. A more cost-effective approach tothat Extra-P is very useful for estimating somecollecting performance metrics is required for

performance

metricslarge-scale applications.

numberfloating-point

this

is

widelyapproachperformedforcollecting

is

the

of

most

powerful(e.g.,

tool

for

the

execution

time,

instructions,and（様式 20）成果報告書
communication

given

targeted application. Such metrics include theapplication executed at a large scale; however, but

number of function calls and the total execution

it is still unknown whether Extra-P can be used

time of functions. To collect performance metrics,

for scalability prediction for the other metrics. In

an instrumentation code is inserted into theparticular, the function call count is an important

targeted application using a compiler or dynamic

metric for understanding the application behavior

library, and the application is then executed on aand is widely used for the performance analysis oftargeted system. Profiling tools, such as TAU andparallel applications; however, to the best of our

Score-P, are available for the insertion of theknowledge, researchers have not reported the

instrumentation

effectiveness of Extra-P in function call countperformance metrics collected through profilingprediction.

are measured values. We believe that the costs

In this paper, we show the first evaluation of(computing

Extra-P

prediction.

performance-metric collection through profiling

Therefore, as a use case for function call countare high because this approach requires aprediction, we propose a method that predicts thetargeted application to be executed on a targeted

total execution time of function using the number

system. Many techniques, such as selectiveof predicted function calls. Our experimental

instrumentation, have been developed to reduce

results demonstrated that Extra-P predicted thethenumber of function calls with sufficiently highhowever, performance is limited by the execution

accuracy for various functions and our proposedof the application plus the required collection

method predicted the total execution time ofoverhead.

various functions with higher accuracy than the

2.2. Extra-P

conventional method.

Extra-P is a performance analysis tool for

We summarize the main contributions of thisparallel applications that can be used for the

study as follows:

quick analysis of parallel applications. Extra-P







for

messagefunction

size)

call

of

count

a

costs

code.

time

of

We

and

note

that

the

resources)

performance-metricofcollection;

We demonstrate that Extra-P is also usefulextrapolates a performance metric of a targeted

for creating models of function call counts.

application executed at a large scale from those of

We demonstrate that combining function callthe application executed at multiple small scales.

count prediction is effective in predicting theTototal execution time of functions.

scalability model for the performance metric

We demonstrate that Extra-P greatly reducesusing regression. The independent variables of

the cost of collecting the total execution timethe model are the process count and problem size,of functions.

and the dependent variable is the performanceaccomplish

this,

Extra-P

generatesametric to be predicted.
2. BackgroundThe general form of the model used in Extra-P is2.1. Profilingas follows:

Profiling parallel applications is widely used for
performance analysis and tuning in the field ofhigh-performance computing. Profiling involves
the collection of the performance metrics of awhere m is the number of independent variables,is an independent variable, andis a（様式 20）成果報告書
regression parameter. n is the number of product

Table 1. System configuration ofterms, each of which has a different combination

TSUBAME3.0

of. The exponents

andare called the hypotheses and characterize theform of the equation. Extra-P users can specify
the sets of(referred to as I and J,

andTable 2. Node configuration of TSUBAME3.0

respectively) used for the regression from thecommand line interface.
Using

Extra-P,wecancollect

variousperformance metrics of a targeted application

static code analysis. Thus, to obtain the exact

without executing the application at a targeted

number of function calls, we require a targeted

scale; however, the performance metrics reported

application to be executed at a targeted scale.

by Extra-P are predicted values. Additionally, the3.2. Experimental Methodologies

effectiveness of Extra-P has been proven for a few

In this subsection, we evaluate the accuracy of

performance metrics (e.g., total execution time ofExtra-Pa function and communication message size). It is

Specifically, we first collect sample data (i.e., callstill unknown whether Extra-P can also be usedcounts per function) by executing a targeted

for function call count prediction.

application at various small scales and then input

infunction

call

count

prediction.

them into Extra-P. Next, Extra-P automatically3.

Function Call Count Prediction using Extra-P

3.1. Function

CallCount and

Performance

Tuninggenerates an optimal model to predict thescalability of the call count for each function.
Finally, we predict the function call count at a

The function call count is helpful for tuning

large scale with using the model generated andthe performance of an application. Because

compare the result with the measured call count.

calling a function has a performance overhead,

To enable the generation of multi-variable

a reduction in the function call count may

models, we use multi-parameter for the modeler

improve overall application performance. For

option in Extra-P. Moreover, we use I = {-1, 0, 1, 2,

example, inlining a function can remove the3} for the exponents of the polynomial and use J =calling overhead from an application, but it{0, 1} for the exponents of the logarithm inincreases

Equationtheapplication

code

size.

The

(1).

We

function call count is fundamental information

force_combination_exponentsinallow_negative_exponents to True.

performancetuning;therefore,

many

also

set
and

profiling tools support functionality to reportWethe call counts of the functions executed.

supercomputer TSUBAME3.0. It consists of 540

The function call count is a dynamic attributecompute nodes, each of which has two CPUsof an application. It is usually affected by(Intel Xeon E5-2680 V4). The system and nodevarious factors of running applications (e.g.,

configurations of TSUBAME3.0 are shown in

inputs, process count, and results of branch

Tables 1 and 2, respectively. We used TAU as a

instructions) and cannot be computed using

profiling tool when collecting the actual number

conducted

ourexperiment

on

the

（様式 20）成果報告書
Table 3. Benchmark programs and scales

Table 5. Fitting errors in function call countused for model generation. Abbreviations in

prediction

italics in the Scales column represent the as
following: pc=process count, ps=problem size,

gs=grid size, it=iterations, kc=key count, andmv=max value.

the scales listed in Tables 3 and 4. Our
experimental results exclude a few functions
executed only at a specific scale.
We used the following two metrics to evaluate
prediction accuracy.
MAPE: The accuracy of a model is often
evaluated using the mean absolute percent

Table 4. Scales used for prediction

error (MAPE), which is the average of theabsolute error rate between the model's
output values ( ) and the observed values ().

It can be expressed by the following formula:of function calls.
We tested five HPC applications shown on
TSUBAME3.0. Four out of the five applications

where N is the number of data points.

(i.e., EP, FT, IS, and MG), which we selected from

Weighted MAPE: In the MAPE, the errors in

the NAS Parallel Benchmark suite, are simple

each data point contribute to the overall error

and stress particular properties. The remaining

equally; however, the more precise prediction

application, LULESH, is more complex and weof a large function call count is helpful for

selected it from the CORAL-2 Benchmark suite.

performance tuning. Therefore, we introduce

The scales used for model generation are

the weighted MAPE as the other metric forsummarizedanprediction accuracy. The weighted MAPE is aapplication for every combination of the scaleweighted average of the absolute error rate byparameters shown in the table and then entered

the number of function calls and can be

all the resulting function call counts into Extra-P

expressed by the following formula:

in

Table

3.

We

executed

to generate the models. For example, we executed
EP at 30 scales (6 types of process counts

5types of problem sizes) in total.

where

Using the models generated by Extra-P, wefunction calls measured for the function

predicted the function call counts of each$t$ and the total number of function calls

application for the three types of process counts

measured during the execution of a targeted

and input data shown in Table 4. We note thatapplication at a given scale, respectively.

andrepresent the number ofthese targeted scales are larger than the data

3.3. Experimental Results

collection scales shown in Table 3. In the next

Table 5 summarizes the fitting errors of thesection, we focus on the functions executed at allfunction-call-count models generated by Extra-P.

（様式 20）成果報告書(a) MAPE(b) MAPE(b) Weighted MAPE(b) Weighted MAPE

Figure 1. Accuracy of function call countFigure 2. Accuracy of function call countprediction for various process counts using

prediction for various input data using large

large input dataprocess counts

As shown in the table, Extra-P produced models

applications. We used large input data for thiswell -fitted to the collected data, in many cases. In

experiment.

particular,modelsFigure 1 (a) shows that two out of the fivegenerated by Extra-P had an error of 1.58% on

applications (i.e., IS, and LULESH) had small

average for FT. A few applications, such as EP

errors (within 10%). As shown in Table 5, Extra-P

and IS, had large MAPEs because they included

produced function-call-count models well -fitted tosome functions that had irregular patterns in thethese applications so that the generated models

function call count. However, the numbers of calls

had high prediction accuracy at the targeted

for such functions were relatively small. Extra-P

scales.

provided very accurate models for many functions

By contrast, FT and MG had large MAPEs

that were important to overall performance, evenbecause of the inaccuracy of the call-count models

in such applications.

for some functions. However, this inaccuracy may

Figure 1 shows the prediction accuracy of the

not become a problem in many use cases because

generated models for various process counts. The

such functions have relatively small call counts

x-axis represents the process count, and theand are therefore less important than the othery-axis represents MAPE or weighted MAPE. We

functions. Figure 1 (b) exemplifies this. The

note that the y-axis uses the logarithmic scale.

weighted MAPE was up to 2.23%.

The five bars on each x-label represent the five

Figure 2 shows the prediction accuracy of thethefunction-call-count（様式 20）成果報告書
i-th total execution time of function F.
Generally, Extra-P predicts the total execution
time of function directly. Specifically, we firstcollect the total execution time of each function() while executing a targeted application at

various small scales. Next, we enter the collected
data into Extra-P and then create a total
execution time of function model for each function.

Figure 3. Overview of prediction of total

Finally, we predict the total execution time of

execution time of functions

function executed at a targeted scale using thegenerated models for various input data. We usedgenerated model (the direct method in Figure 3).

large process counts for this experiment. Similar

Hereafter, we call this the direct approach

to Figure 1, the weighted MAPEs of the models

because we need to distinguish between thisgenerated were very small for all combinations ofapproach and our approach proposed in the next

applications and input data, whereas the MAPEs

section.

were large for a few applications. Thus, we

4.2. Combined Approachconclude that Extra-P produced models sufficient

Theoretically,for practical use in function call count prediction.

function can be expressed as follows.

4.

Combining Function Call Count Prediction and

where

Total Execution Time of Functions Prediction

function per call.

the

total

execution

time

of

is the average execution time of

As described in the previous section, Extra-P

Because the execution scales (i.e., process counts

produced accurate function-call-count models, inand input data) have different influences on the

many cases. In this section, as a use case forcall count (predicting function call counts, we combinefunction per call ( ), the direct modeling offunction call count prediction with the scalability

difficult for some functions. For such functions,prediction of the total execution time of aprediction

function.

developing individual models for4.1. Predicting the Total Execution Time ofthen combining the predicted results using the) and total execution time ofaccuracy

can

be

is

improved
and

by

, and

models (the combined method in Figure 3). We

Functions
The prediction of the total execution time of acall this the combined approach and use Extra-P

function

to model both

is

often

performed

toidentify

a

and

.

scalability bug in a targeted application. As

4.3. Experimental Methodologies

shown in previous work, it is the main use case of

Many profiling tools report two types of total

Extra-P.

execution time of function. One is inclusive time,

The total execution time of function () can beexpressed as follows.

which includes the time taken by all callees. The
other is exclusive time, which represents the time
taken by the targeted function itself. In this study,
we performed the prediction for both inclusive

where

is the function call count andis theand exclusive time.

（様式 20）成果報告書
Table 6. Fitting errors in the total execution time of function prediction. In "x (y-z)," x is the
MAPE, and y and z are the minimum and maximum MAPE per function, respectively

We

used

the

same

experimental

systemdescribed in Section 3.2. We also used thebenchmark programs and execution scales listed
in Tables 3 and 4.
We evaluated the cost of data collection, in
addition to fitting errors and prediction accuracy.
We define the cost of collecting data ( C) for an
application (app) executed on M cores as follows:(c) MAPE

Because many supercomputing services require
users to buy points, which are consumed based on
the product of the computing time and number ofcomputing resources.
Both direct and combined approaches require
model generation and prediction processes in
addition to data collection. These two processes
also consume computing time and resources;(b) Weighted MAPE

however, they can be executed very quickly, evenFigure 4. Accuracy of prediction of total

on a single compute node. Because the costs ofexecution time of functions for various

model generation and prediction are relatively

process counts using large input data

small compared with the cost of data collection,
we ignore them in this study.

the direct approach, the combined approach

4.4. Experimental Results

reduced the fitting error by 20,302% for FT. ThisThe fitting errors for the total execution time ofis because, with respect to the exclusive time,function models are shown in Table 6. Each entry

Extra-P can create more precise models forrepresents the MAPE of all models generated byandan approach for an application, and the range

Figure 4 shows the accuracy in predicting therepresents the minimum and maximum values intotal execution time of function. The x-axis

the case of computing the MAPE for each function.

represents the process count, and the y-axis

The table shows that the combined approach

represents the MAPE or weighted MAPE. We

produced models with higher accuracy than thenote that the y-axis uses logarithmic scale. We

direct approach when predicting the exclusive

used large input data.

time of functions. In particular, compared withAs shown in Figure 4 (a), Extra-P on average

than

.

（様式 20）成果報告書
large process counts. Similar to Figure 4, theweighted MAPEs were small, whereas the
MAPEs were large. The models generated had aweighted MAPE of 1.41%, on average. Thus, weconclude that Extra-P produced models sufficientfor practical use in total execution time offunction prediction.
Figure 6 shows the cost of collecting the total
execution time of functions. The x-axes represent(d) MAPE

the number of execution scales to be collected,
while the y-axes represent costs. The y-axis uses
the logarithmic scale. The blue lines represent thecase of predicting the total execution time offunction and the red lines represent the case ofcollecting the total execution time of function
based on actual runs (i.e., using profilers). We
note that the data collection time for the direct(b) Weighted MAPE
Figure 5. Accuracy of prediction of total
execution time of functions for various input
data using large process counts

approach was the same as that for the combined
approach.
The figure shows that the cost of predictively
collecting the total execution time of functions

produced inaccurate models in the total execution

was completely constant across the number oftime of function prediction. The smallest MAPE

predicted scales. This is because we ignored

was 58.95+%, even when we used the combined

Extra-P's execution cost, which was relatively

approach. In particular, FT had a MAPE of

small compared with the cost of data collection, as

10,000,000+%

count.

described in Section 4.3. By contrast, the cost of

all-to-allcollecting the total execution time of function

communication functions and they were calledbased on actual runs increased gradually as thefrom several locations within the code, it was

numberdifficult for Extra-P to precisely extrapolate theparticular, in the case of collecting the total

performance of such functions.

execution time of function at 27 scales, theBy contrast, we observed different properties ofpredictive approach reduced the cost of data

the models produced by Extra-P in view of thecollection by 99% for EP compared with theweighted MAPE. Figure 4 (b) shows that theapproach of actually-running the application.

models achieved very high accuracy (within 1% ofThus, the predictive approach was effective,

the weighted MAPE) for four out of five

particularly for collecting the total execution time

applications. Therefore, we consider that theof function at multiple scales.

Because

FT

for

the

large

executedprocessmanyofpredicted

scalesincreased.

Inmodels achieved sufficient accuracy in practice.
Figure 5 shows the prediction accuracy of the

5.

generated models for various input data. We used

In this study, we evaluated Extra-P in function

Conclusions（様式 20）成果報告書

Figure 6. Cost of prediction of total execution time of functions
call count prediction. Our experimental results

The predicted function call counts may be usefulshowed that Extra-P produced highly accurate

for the prediction of performance metrics, withscalability models for various functions in thethe exception of the total execution time ofcase of function call count prediction. Additionally,functions.

we showed that the predicted function call counts

In future work, we will extend the use case ofwerefunction call count prediction to the prediction of

helpfulfor

improvingtheprediction

accuracy of the total execution time of functions.

other metrics.