# Performance of RANDOM_NUMBER on Seaborg

#### Richard Gerber

##### *Lawrence Berkeley National Laboratory, NERSC Division, User Services Group, One Cyclotron Road, MS: 943R0256, Berkeley, CA 94720*

## Abstract

*The performance of the Fortran intrinsic function RANDOM_NUMBER is
shown to give very poor performance using 16 tasks/node on NERSC\'s IBM
SP Seaborg in the default runtime configuration. Setting the runtime
option intrinthds=16 improves runtime performance significantly and
gives good results for all possible numbers of tasks per node. It is
speculated that the cause of the problem is the creation of an excessive
number of threads in the default configuration. It is noted that these
threads appear to be created by default, without specifying a
\"thread-safe\" compiler invocation or other user interaction.*

## Introduction

The poor performance of the BeamBeam3D code of Ji Qiang and Robert Ryne
[\[1\]](../qiang2.html), [\[2\]](../qiang.html) on the current (May
2003) Seaborg system was caused by problems with the Fortran intrinsic
function RANDOM_NUMBER. Appropriate use of a runtime parameter improved
code performance significantly, improving runtime performance by a
factor of four. This report documents and quantifies the performance of
RANDOM_NUMBER on Seaborg, a POWER 3 (375 MHz) IBM SP with 16 CPUs/node
using the IBM XL Fortran compiler version 8.1.0.3.

Most of the use of RANDOM_NUMBER by the NERSC user community is assumed
to be in the context of a parallel code using MPI for communication.
RANDOM_NUMBER may be a minor component of the algorithm, but may be
called many times. In Ji Qiang's BeamBeam3D code, RANDOM_NUMBER was
called for each of millions of particles each time step. The most common
way to invoke the IBM XL Fortran compiler is via the mpxlf90 command;
therefore, this command was used to compile source code cited in this
report. Note this is not one of the \"thread-safe\" IBM compilers that
are used to create threaded programs.

For this investigation, a small test case was prepared in Fortran with
MPI initialized but otherwise unused.

## RANDOM_NUMBER Configuration Options

There are no documented options that affect the behavior of
RANDOM_NUMBER in the XLF 8.1 for AIX manuals. However, the following
excerpt was found in a README file in the XLF 8.1 installation directory
on Seaborg and in the version 8.1 *XL Fortran for Linux on pSeries
User\'s Guide*.

intrinthds={num_threads}

> Specifies the number of threads for parallel execution of the MATMUL
> and RANDOM_NUMBER intrinsic procedures. The default for *num_threads*
> equals the number of processors online.

As mentioned in the [report on the BeamBeam code](../qiang2.html), the
following was found via a web search at the URL
http://msgs.sp2.net/cgi-bin/get/sp0207/45.html:

> APAR: IY35729 COMPID: 5765F7100 REL: 810
>
> ABSTRACT: ADD A RUNTIME OPTION TO CONTROL THE NUMBER OF THREADS
>
> PROBLEM DESCRIPTION:
>
> ERROR DESCRIPTION:
>
> random_number() and matmul() are 2 Fortran instrinsic functions
>
> that are multi-threaded. It has been requested by
>
> customers to allow users to control the number of
>
> threads (eg. 1) used in these functions.
>
> PROBLEM CONCLUSION:
>
> A number of customers have experienced the problem that
>
> multi-threaded intrinsic function floods their systems with
>
> uncontrollable number of threads, which can result in
>
> unacceptable run-time performance. In order to
>
> solve this problem, we have to provide a new run-time option to
>
> give them the ability to control the number of threads used in
>
> those functions. Currently only matmul and random_number have
>
> the multi-threaded version.
>
> Syntax:
>
> intrinthds={num_threads}

This runtime option has a significant effect on the performance of
RANDOM_NUMBER on NERSC\'s SP. A test code was written and used to
investigate how execution times are affected by choices for the value of
intrinthds and the number of MPI tasks used per node.

## The Test Code

A test simple test code was created that does little but initialize MPI
and then call RANDOM_NUMBER a number of times. Here is the code:

program ranno

implicit none

include \'mpif.h\'

integer:: i

integer, parameter:: times=100

integer, parameter:: idim=10000000

real, dimension(idim):: realArray1,realArray2

integer:: ierr

integer:: totTasks, task_id

call MPI_INIT(ierr)

call MPI_COMM_SIZE( MPI_COMM_WORLD, totTasks, ierr)

call MPI_COMM_RANK( MPI_COMM_WORLD, task_id, ierr)

do i=1,times

call RANDOM_NUMBER(realArray1)

call RANDOM_NUMBER(realArray2)

end do

print \*, \"Task \",task_id,\" of \",totTasks,\" has a value:
\",realArray1(100),realArray2(100)

call MPI_FINALIZE(ierr)

end program ranno

The code initializes MPI so performance can be investigated as a
function of the number of MPI tasks created on a node. It loops through
enough calls to RANDOM_NUMBER to give a non-trivial, yet convenient run
time. It then prints out a value in an attempt to keep the compiler from
optimizing away the entire loop, which it might do if it realizes the
calculated values are never used or examined. With the array size given,
the maximum memory usage on a node with 16 tasks is approximately:

10,000,000 entries \* 4 bytes/entry \* 2 arrays \* 16 tasks = 1.28 GB.

This is less than the 16 GB of physical memory on the Seaborg node with
the least amount of memory. This was done to prevent the possibility of
memory paging, which would affect runtime performance.

The code was compiled with XLF 8.1 and the value of intrinthds set via
the XLFRTEOPTS environment variable.

**% mpxlf90 -O3 -qtune=pwr3 -qarch=pwr3**

**% setenv XLFRTEOPTS intrinthds=N**

N is varied for each run.

## Results

The code was run with 1, 2, 4, 8, 12, 13, 14, 15 and 16 MPI tasks on a
single Seaborg node in order to investigate the effect of running
various numbers of tasks on each node. Most NERSC user codes will run 16
tasks/node and users are charged for using all 16 CPUs.

The relevant IBM software versions were:

AIX 5.1

ppe.poe 3.2.0.13 poe Parallel Operating

LoadL.full 3.1.0.13 LoadLeveler

bos.rte.bind_cmds 5.1.0.35 Binder and Loader Commands

xlfcmp 8.1.0.3 XL Fortran Compiler

xlfrte 8.1.0.3 XL Fortran Runtime

xlfrte.aix51 8.1.0.3 XL Fortran Runtime Environment

Note that in the test code each task is essentially independent,
connected only by MPI initialization. If there were no interaction
between tasks, e.g. if each task was run on physically separate computer
hardware, the run time would be independent of the number of tasks.
However, this is not the case on SMP nodes like those on Seaborg; tasks
must compete for access to memory and CPUs for example. (Tasks are not
bound to CPUs on the SP SMPs; AIX schedules the tasks to run on CPUs
with available cycles.)

For each run, the value of intrinthds was either unset (termed the
\"default\" configuration) or had a value between 1 and 128. The NERSC
poe+ utility, which parses and concatenates output from IBM\'s hpmcount
program, was used to record run times, sys times, and user times. Use of
the poe+ utility to measure timings for the entire code, as opposed the
RANDOM_NUMBER calls only, was appropriate in this case since MPI startup
times are much shorter than the execution times reported here. (MPI
initialization times been measured by NERSC and are on the order of .017
seconds/task, or less than 1 second for 16 tasks or fewer.)

A striking result is that the program had a highly variable run time in
the default configuration when 16 MPI tasks were run on a single node.
Execution times varied unpredictably from 60 seconds to more than 30
minutes (jobs exceeding 30 minutes were killed). This result did not
depend on the memory resident on a node, the variability was observed on
both 16 and 32 GB nodes. Likewise the specific node used did not matter;
the variation was observed by running the executable multiple times in
succession in a single batch job running on one node.

The following table lists runs times for various combinations of total
tasks and values of intrinthds. A dedicated compute node was used for
all runs. When results were highly variable from run to run, multiple
timings are show below. Where single timings are given, a variation of
at most a few percent was observed for a minimum of three runs.

#### Run Times (seconds)

  ---------- --------------- ------- ------- ------- ------- -------- ----------- ----------- ----------
  ** **      **Value of                                                                       
             intrinthds**                                                                     

  **Number   **default**     **1**   **2**   **4**   **8**   **16**   **32**      **64**      **128**
  of tasks**                                                                                  

  **1**      11.3            51.4    30.5    19.3    13.6    10.7     10.9        11.4        11.8

  **2**      14.9            52.0    31.0    19.3    13.8    14.1     14.4        14.4        16.1

  **4**      21.1            52.4    31.0    19.9    20.2    20.6     20.8        93.1, 167,  \>1400
                                                                                  21.6^(1)^   

  **8**      35.5            55.0    33.6    33.4    33.5    34.3     35.0        \>720,      \>1000
                                                                                  \>1800,     
                                                                                  583^(1)^    

  **12**     50.7            57.8    49.0    49.0    48.8    49.0     50.6        \>1200      N/A^(2)^

  **13**     145, 55.1^(1)^  58.5    52.7    52.1    52.1    52.7     54.5, 112,  \>1200      N/A^(2)^
                                                                      54.1,                   
                                                                      54.0^(1)^               

  **14**     59.6, 641^(1)^  59.1    54.3    54.6    55.3    56.9     58.0, 207   \>1200      N/A^(2)^

  **15**     62.8, 62.2,     62.7    58.7    59.0    58.8    61.4     62.7, 456   \>1200      N/A^(2)^
             698^(1)^                                                                         

  **16**     1037, 233,      61.8    63.8    63.2    64.8    65.7     69.6,       \>1800,     N/A^(2)^
             63.2, 66.0,                                              \>1500,     \>1200      
             67.7, 857,                                               66.8^(1)^               
             807.5, \>1800,                                                                   
             \>720^(1)^                                                                       
  ---------- --------------- ------- ------- ------- ------- -------- ----------- ----------- ----------

^(1)^ Where results are highly variable, multiple measurements are
given.\
^(2)^ No runs were made with this combination of parameters.

These results are illustrated in the following graph. Configurations
that had highly variable long run times are shown as extending off
scale.

![](media/image1.wmf)

Wallclock, time in user mode, time in system mode, and MFlips/s as
reported by poe+ for some runs in the default configuration are reported
in the following table.

#### Default configuration measurements from poe+/hpmcount

  --------------- ----------------- ------------ ----------- --------------
  **Number of     **Wallclock       **user       **sys       **MFlips/s /
  tasks**         seconds**         seconds**    seconds**   task**

  1               11.3              46.0         17.1        1455

  2               14.9              50.4         8.25        1107

  4               21.1              52.1         4.65        708

  8               35.5              52.3         3.69        463

  16              varies; \>63      varies; \>53 varies;     Varies
                                                 \>2.8       
  --------------- ----------------- ------------ ----------- --------------

## Discussion

The code's performance in the default configuration with 16 tasks per
node is clearly unacceptable. The variation from run to run is great and
in many cases the programs is not observed to finish. When the runtime
variable intrinthds is set to have a value of 16, acceptable performance
is observed for any number of tasks per node.

If one examines the timings when intrinthds=32 in the first table and
graph, they are remarkably similar to the timings obtained when
intrinthds is unset (the \"default configuration\").

The second table above shows that in the default configuration 1 task
accrues 46 seconds of user time, but only takes 11 seconds of wallclock
time to run. This is an indication that the job is running multiple
threads that execute on the other 15 CPUs on the node. The finding that
multiple threads are being created by default, without using the
\"thread-safe\" compilers or any other user specification, is notable
and an unexpected result.

If we assume that by default each MPI task is spawning 32 threads then
there appears to be a threshold somewhere around 512 (or perhaps as low
as 256) spawned threads per node beyond which node performance becomes
inconsistent and significantly degraded. The uptime command was observed
to report a system load average of 99.99 during one of the long-running
jobs. LoadLeveler reported load averages up to 471 on the nodes with
jobs that were measured to have long run times.

Interestingly, a value of intrinthds=16, which would appear to be the
intended default from a reading of the IBM comments on the web site
referenced above, gives very good performance at all possible tasks per
node on Seaborg.

These results indicate that heavy use of the RANDOM_NUMBER Fortran
intrinsic can produce very poor performance in Seabborg\'s default
configuration. While RANDOM_NUMBER is seldom expected to be the major
component of a computational kernel, Seaborg users should be aware of
potential problems with this intrinsic. Failure to do so could cause a
minor component of the entire code to have a major impact on overall
performance.

Although not designed to investigate how to perform the maximum number
of RANDOM_NUMBER calls per unit time, the results of this paper can be
used to draw some tentative answers to that question. If one assumes
that a run with a single task will perform 16 times more RANDOM_NUMBER
calls in 16 times the measured execution time, etc., one can construct
the following table:

  ------------------------- ---------------------------------------------
  **Number of tasks per     **Time needed to execute RANDOM_NUMBER calls
  node**                    relative to the number performed with 16
                            tasks/node with intrinthds=16**

  1                         2.61

  2                         1.72

  4                         1.28

  8                         1.04

  12                        1.03

  16                        1.00
  ------------------------- ---------------------------------------------

Using 16 tasks/node with intrinthds=16 performs the most calculations in
the least amount of time.

## Conclusions

Calling the Fortran RANDOM_NUMBER intrinsic from 16 MPI tasks on a
single Seaborg SMP node gives unacceptable and unpredictable runtime
performance. NERSC users should be advised that heavy use of
RANDOM_NUMBER could adversely affect overall code performance. Setting
the runtime parameter intrinthds=16 gives good performance and should be
recommended to NERSC users.