Performance Optimization and Tuning
for NEC TX7
including
MOLPRO, MOLCAS, AMBER, GAMESS-USJan Fredin, Ph.D.
NEC Solutions America
March 5, 2004

© NEC Corporation 2004

1

1

Talk Overview
 NEC TX7 hardware General TX7 performance tuning techniques
 Case Studies: examples of tuning efforts on
Chemistry applications: MOLPRO MOLCAS AMBER GAMESS-US User Molecular Dynamics Code© NEC Corporation 2004

2

2

NEC TX7 Servers

 Second generation technical computing
server with Intel’s latest 64-bit CPU“Itanium® 2 processor”.

 Configured with up to 32 CPUs per
node. NEC was the first company torelease Itanium® 2 systems with 32
processors.

 Designed and built with supercomputer
technology from NEC.

 Ideal systems for scientific andengineering applications.
© NEC Corporation 2004

3

3

TX7 Internal Structure• ccNUMA architecture• Near-flat 32-way memory access• Flexible partitioning
Up

Cell Itanium2
Itanium2
Itanium2
Itanium2

Cell Photo
to

8C

Itanium2
Itanium2ells

Itanium2
Itanium2

MemoryMemory Controller
Controller

Cell
Cell
Controller
Controller
MemoryMemory Controller
Controller

DDR
DIMMs

Itanium2
NEC Chipset

Cross-bar interconnect
Cross
Cross-bar
interconnect

PCI-X
Up to 112slots

PCI-X bridge
PCI
PCI-X bridge
PCI-X
bridgePCI-X
bridgePCI-X
bridgePCI-X
bridgePCI-X
bridge PCI
PCI-X
bridgePCI-X
PCI-X
bridgePCI-X
bridge14bridge
PCI-X
slots
PCI-X
bridge
14
PCI-X
slots
PCI-X
bridgePCI-X
bridge
14
PCI-X
slots
PCI-X
bridge
PCI-X
14
slots
PCI-X
bridgePCI-Xbridge
bridge14PCI-X
PCI-X
slots

DIMMs memory

Total band width>100GB/s

1414PCI-X
slots
14PCI-X
PCI-Xslots
slots

TX7 /

© NEC Corporation 2004

i6010, i6510, i9510

4

4

Itanium 2 Processor Memory Hierarchy
1.5 GHz Itanium® 2 Processor

External
Memory6.4 GB/s

L3
6MB
24-way128B lines
7-14 CLKS

L1I
16KB
64B lines
1 CLK

48 GB/s

L2
48 GB/s
256KB
8-way
L1D
128B lines
16KB
5-7 CLKS
48 GB/s 64B lines
Banked
1 CLK

128 FP Registers
128 General Registers
Core Pipeline(functional units)© NEC Corporation 2004

5

5

NEC IPF Server Roadmap
Itanium®®
Itanium
Performance / Scalability

TX7 Series

Express5800/
1160Xb(AzusA)

16 way32 way4 way

2001© NEC Corporation 2004

Itanium®®22//Madison
Madison
Itanium
2002

2003
6

6

NEC Worldwide HPC Support andDevelopment Organizations

● ●
●
▲

CanadaGermany

Korea

NEC HPCE

NECSupport Office

●

JAPAN

▲
Cray

.
U.S.A.
■HMD
NECSAM●

BrasilFrance●Groupe Bull

Singapore

Australia

NEC France

NEC
Singapore
Pty. Ltd.

NEC Australia
Pty. Ltd.

© NEC Corporation 2004

● Subsidiaries▲ Partner■ Branch●

NEC doBrasilS.A.

7

7

Software Support Cycle Port - successfully run code for all QA available
 typical compile efc -O2 -ftz
 reduce optimization for problem areas
 Tune - improve performance identify time consuming routines for typical
workload measure performance using UNIX gprof, NEC
compiler add-of ftrace, Intel VTUNE or timing calls test impact of changes in compile options, compile
directives and source code restructure can be limited by ISV code maintenance rules© NEC Corporation 2004

8

8

Software Support Cycle Update, supporting software vendor and user sites
 test and verify full QA correctness
 provide changes through vendor distribution orrelease notes update for:- new vendor software release- compiler update- user problems, feedback and benchmarks- hardware upgrades Repeat the tune and update steps regularly© NEC Corporation 2004

99

Tuning Challenge keep fast functional units busy, providing all data required 2 FMA / clock - 6GFLOPS / sec for Itanium2 1.5GHz
 maximize loop optimization
 provide compiler with loops that can achieve highperformance software pipelining - similar to vectorization
 instructional parallelism - 6 (2 bundles) / clock prefetching - avoid waiting at the L3 cache bottleneck high performance is achieved by balance in the loops
 computational complexity
 data locality
 number of variables© NEC Corporation 2004

10

10

Software Tuning Techniques
 know the good application input and algorithm
choices memory, processors, site defaults
 use tuned kernels like NEC MathKeisan
 choose effective compile options
 restructure loops
 help compiler optimize through source-code
directives© NEC Corporation 2004

11

11

NEC MathKeisan
MathKeisan is a set of NEC highly tuned math libraries.
Name

Description

BLAS

Basic Linear Algebra Subprograms

LAPACK

Linear Algebra PACKage for high performance computers.

ScaLAPACK

Scalable Linear Algebra PACKage (contains PBLAS)

BLACS

Basic Linear Algebra Communication Subprograms. Uses MPI.

CBLAS

C interface to BLAS

SBLAS

Sparse BLAS (from ACM Algorithm 692)

FFT

Fast Fourier Transforms

METIS

Matrix/Graph ordering and partitioning library

ParMETIS

Parallel Matrix/Graph ordering and partition library. Uses MPI.

SOLVER

Direct solver for sparse symmetric systems

ARPACK

Solution of large scale eigenvalue problems© NEC Corporation 2004

12

12

MathKeisan 1.5 Improvements
DGEMM Performance

1.5 GHz Madison, 6 MB L3

MFLOP/s

6000
5000

Fortran -O3

4000

MathKeisan 1.3

3000

MathKeisan 1.5( target release
2Q CY2004 )

2000
1000
0
10

50

100

500

1000

1500

Matrix size

© NEC Corporation 2004

13

13

Compile Choices
 NEC compiler is a superset of the Intel compiler
 NEC compiler provides ftrace function for code
section performance tracing Compiler optimization level -O3 highest level of optimization-software pipelining on-activates prefetching-activates other loop optimizations
 -O2 safe optimization level- software pipelining on
 -O0 no optimization for debugging© NEC Corporation 2004

14

14

Other Compile Options
 -ftz flush-to-zero denormalized values
 should always use default at compiler level -O3
 -opt_report compiler optimization report messages about pipelining, dependencies,
unrolling, other loop restructuring
 -ip interprocedural optimization within object files
 -prof_gen and -prof_use profile generated
optimization
 -restrict / -ansi_alias enable strict pointer aliasing
 -i8/-r8 integer and real default word size
© NEC Corporation 2004

15

15

Loop Restructuring
 code developers know overall structure provide compiler with loops that can achieve maximum
performance multi-purpose scientific and engineering codes often
have heavy usage of long complex loops withconditionals and indirect indexing
 goal of hand recoding is to isolate sections that are
hard for the compiler to optimization
 too complex to analyze
 external calls data dependencies
 conditionals© NEC Corporation 2004

16

16

Restructuring Example - Loop Blocking
 breaks up multi-level loop into blocks that fit intocache
 best blocking sizes determined experimentally blocked sections are prime targets for parallelism
Original Code

Modified Codeparameter (kblock = 1024)
do ks = 1, m, kblock
do j = 1, n
do i = 1, p
do k = ks, min(ks+kblock-1,m)c(k,j)=c(k,j)-a(k,i)*b(i,j)
enddo
enddo
enddo
enddo

do j = 1, n
do i = 1, p
do k = 1, mc(k,j)=c(k,j)-a(k,i)*b(i,j)
enddo
enddo
enddo

© NEC Corporation 2004

17

17

Compile Automated Loop Restructuring
 Loop Unrolling Unroll and Jam Reduction Optimization
 Loop Interchange Loop Fusion
 Loop Fission© NEC Corporation 2004

18

18

Loop Unrolling perform n iterations of the inner loop on each pass
 actives at compile level -O3, compiler sets unroll depth n
 !DIR$ unroll (n) or !DIR$ nounroll
 directive controls unroll depth n or switchs off
Original Code

Modified Codesubroutine vadd(n, a, b)
integer n, i
double precision a(*),b(*)!DIR$ UNROLL(4)do i = 1,na(i) = a(i) + b(i)
enddo
end

subroutine vadd(n, a, b)
integer n, i
double precision a(*), b(*)
explicitely unrolled by 4
do i = 1,n,4a(i)= a(i)+ b(i)
a(i+1) = a(i+1) + b(i+1)
a(i+2) = a(i+2) + b(i+2)
a(i+3) = a(i+3) + b(i+3)
enddo
do i=i,na(i)= a(i)+ b(i)
enddo
end

c

Output from -O3 -opt_report
Block, Unroll, Jam Report:(loop line numbers, unroll factors and typeof transformation)
Loop at line 6 unrolled with remainder by 4
© NEC Corporation 2004

19

19

Unroll and Jam “jam” n iterations outer loop into each pass of inner loop actives at compile level -O3, no user control
Original Code

Modified Codedo j = 1, n-3, 4
do k = 1, p
do i = 1, mc(i,j+0)=c(i,j+0)+a(i,k)*b(k,j+0)c(i,j+1)=c(i,j+1)+a(i,k)*b(k,j+1)c(i,j+2)=c(i,j+2)+a(i,k)*b(k,j+2)c(i,j+3)=c(i,j+3)+a(i,k)*b(k,j+3)
enddo
enddo
enddo
do j = j, n
do k = 1, p
do i = 1, mc(i,j)=c(i,j)+a(i,k)*b(k,j)
enddo
enddo
enddo

do j = 1, n
do k = 1, p
do i = 1, mc(i,j)=c(i,j)+a(i,k)*b(k,j)
enddo
enddo
enddo
Output from -opt_report at -O3:
Block, Unroll, Jam Report:(loop line numbers, unroll factors
and type of transformation)
Loop at line 1 unrolled and jammed by 4
Loop at line 2 unrolled and jammed by 4

© NEC Corporation 2004

20

20

Reduction Optimization
 split sum into n partial sums then collect partial sums
 actives at compiler level -O3, no user control

Original Code

Modified Codes = 0.0d0
do i = 1, ns = s + a(i)enddo
Output from -opt_report at -O3
Block, Unroll, Jam Report:(loop line numbers, unroll factors and type oftransformation)
Loop at line 2 unrolled with remainder by 8
(reduction)© NEC Corporation 2004

s1 = 0.0d0
s2 = 0.0d0
s3 = 0.0d0
s4 = 0.0d0
do i = 1, n-3, 4
s1 = s1 + a(i)s2 = s2 + a(i+1)
s3 = s3 + a(i+2)
s4 = s4 + a(i+3)
enddo
do i = i, ns1 = s1 + a(i)enddos = s1 + s2 + s3 + s4

21

21

Loop Interchange change in nesting order of 2 or more loops
 unit stride on loads to reuse data in cache activates at compile level -O3, no user control
Original Code

Modified Codedo i = 1, m
do j = 1, n
do k = 1, pc(i,j)=c(i,j)-a(i,k)*b(k,j)
enddo
enddo
enddo

do k = 1, p
do j = 1, n
do i = 1, mc(i,j)=c(i,j)-a(i,k)*b(k,j)
enddo
enddo
enddo

Output from -opt_report using -O3
LOOP INTERCHANGE in test_ at line 1
LOOP INTERCHANGE in test_ at line 2
LOOP INTERCHANGE in test_ at line 3© NEC Corporation 2004

22

22

Loop Fusion
 combine 2 or more loops with same iteration count into 1
 reduces overhead, can promote data sharing andcomputational complexity
 can block pipelining if too many registers get used
 activates at compile level -O3, no user control
Original Code

Modified Codesubroutine vmul(n,alpha,x,y)
integer n, i
double precision x(*), y(*)
do i = 1, nx(i) = x(i) * alphaenddo
do i = 1, ny(i) = y(i) * alpha
enddo
end

subroutine vmul(n,alpha,x,y)
integer n, i
double precision x(*), y(*)
do i = 1, nx(i) = x(i) * alphay(i) = y(i) * alpha
enddo
end

Output from -opt_report at -O3
Fused Loops: ( 5 8 )© NEC Corporation 2004

23

23

Loop Fission
 also called loop distribution or loop splitting
 smaller loops for optimization like prefetching and pipelining
 activates at compile level -O3
 !DIR DISTRIBUTE POINT focuses compiler on trying fusion-outside loop, compiler sets location-inside loop, programmer sets location
Original Codedo i = lft, llt
fail(i)=1.0
hgener(i)=0.0
diagm(i)=1.e+31
sieu(i)=ies(nnm1+i)!dir$ distribute pointx1(i)=x(1,ix1(i))y1(i)=x(2,ix1(i))z1(i)=x(3,ix1(i))
enddo
Output of -opt_report for -O3
LOOP DISTRIBUTION in test_ at line 1
Distributed for large ii at __ performed in
test_ at line 7© NEC Corporation 2004

Modified Codedo i = lft, llt
fail(i)=1.0
hgener(i)=0.0
diagm(i)=1.e+31
sieu(i)=ies(nnm1+i)
enddo
do i = lft, llt
x1(i)=x(1,ix1(i))y1(i)=x(2,ix1(i))z1(i)=x(3,ix1(i))
enddo

24

24

Compile Directive - Variable Dependencies
 compiler cannot determine array references overlap
so pipelining is blocked !dir$ IVDEP - fortran loop level
 #pragma IVDEP - C loop level C restrict pointer - routine call level
subroutine test(nu1,nu2, tx, sxx, dd1, dd2, np)
integer nu1, nu2, i3, k, np(*)
double precision tx(3,*), sxx(9,*), dd1, dd2!dir$ ivdep
do k = nu1, nu2
i3 = np(k)tx(1,i3) = tx(1,i3) + sxx(1,k)*dd1 + sxx(2,k)*dd2
tx(2,i3) = tx(2,i3) + sxx(4,k)*dd1 + sxx(5,k)*dd2
tx(3,i3) = tx(3,i3) + sxx(7,k)*dd1 + sxx(8,k)*dd2
enddo
end
© NEC Corporation 2004

25

25

Other Compile Directives
 prefetch
 data values are loaded into cache ahead of access to
reduce latency in loading data too much prefetching can cause cache misses !dir$ prefetch a !dir$ noprefetch b software pipelining
 default in compile level -O2 and -O3
 directive to switch on/off software pipelining !dir$ swp
 !dir$ noswp

 loopcount hint for compiler to optimize loop
 !dir$ maxloopcount_value© NEC Corporation 200426

26

Case Studies
 Examples of tuning efforts on Chemistry
applications: MOLPRO MOLCAS AMBER GAMESS User Molecular Dynamics (MD) Code© NEC Corporation 2004

27

27

MOLPRO 2002.6
 MOLPRO is designed and maintained by H.-J. Werner(University of Stuttgart) and P. J. Knowles (University ofBirmingham)
http://www.molpro.net/
 MOLPRO is a complete system of ab initio programs formolecular electronic structure calculations with an emphasis on
highly accurate computations and extensive treatment of theelectron correlation.
 Mixed Fortran and C using PNNL Global Arrays (GA) for
shared memory and parallel MPI tasks. Driver program checks
license and exec’s computational executable. 64 bit real,
integer, pointer normal_dft dataset adrenaline SVWN, BLYP from MOLPRO bench directory© NEC Corporation 2004

28

28

MOLPRO 2002.6


upgrade source from MolPro2002.5 to MolPro2002.6
compiler initial optimization -O2 and < 20 routines –O0
upgrade MathKeisan for blas and lapack
profile top routines using MolPro development environment to
run computational unit stand-alone: normal_dft
 21.8% uncompress_double
 9.0% dfti_block 7.8% dft_rho
 5.2% dform 5.0% dgenrl aioint, compress=0 turns off compression
 at end of molpro2002.6/bin/molproi.rc add:
 nocompress
 dft/dfti.f -O3 gives correct numerical results for all QA
 another customer test profile showed top routines addmx,
mxmas, fzero tuned with -O3 (3 files) gave 22% improvement© NEC Corporation 2004

29

29

MOLPRO 2002.6
normal_dft
1200

real seconds

1000
800
1cpu
2cpu
4cpu

600
400
200
0
initial© NEC Corporation 2004

final

uncompress

30

30

MOLCAS 5.4
 MOLCAS is developed and distributed from the Department ofTheoretical Chemistry at Lund University
http://www.teokem.lu.se/molcas/
 MOLCAS is a quantum chemistry software package with a focus
on methods that will allow an accurate ab initio treatment of very
general electronic structure problems for molecular systems inboth ground and excited states.
 MOLCAS is a research product developed by scientists to be
used by scientists.
 Fortran with some C systems control routines, MPI parallel andshared memory through PNNL GA. 64 bit real, integer, pointer.
Independent modules are called in sequence from MOLCASgenerated scripts.
 test902 performance test provided in the MOLCAS distribution.
Disk-based Hartree-Fock using modules seward and scf© NEC Corporation 2004

31

31

MOLCAS 5.4
 initial compile optimization -O2 with 4 routines at-O0, static link
 serial focus due to limited parallel implementationin MOLCAS 5.4
 parallel support targeted for next release.

 4 main modules:
 46% seward 26% caspt2
 16% scf
 12% rasscf

 add MathKeisan blas – 38% improvement
 profile showed pack / nopack are in top routines
 added nopack keyword for Seward module input© NEC Corporation 2004

32

32

MOLCAS 5.4
test902900

real seconds

800
700

modules

600
500

rasscfcaspt2scf
seward

400
300
200
100
0
initial© NEC Corporation 2004

final

nopack33

33

AMBER 7.0
 Distributed by University of California, San Francisco
http://amber.scripps.edu/
 About 60 programs that work reasonably well together solving
molecular modeling and dynamics problems.
 sander- Simulated annealing with NMR-derived energy restraints.
- main program used for molecular dynamics simulations.
 Mostly Fortran with C for dynamic memory and file handling and MPI
parallel. Each module is a separate executable. Sander is built from
98 source files consisting of 408 subroutines.
 sander gb_mb dataset provided in AMBER benchmarks directory Generalized Born myoglobin simulation, 2492 atoms in 100,000
waters 20 Ang. cutoff, salt 0.2Molar, long-range forces every 4 cycles© NEC Corporation 2004

34

34

AMBER 7.0
 initial distribution compile optimization
 compile level2=-O2
 compile level3=-O2 used for 36 out of 96 files

 profile shows 98% of time spent in egb
 subroutine egb is composed of 3 large loops upgrade MathKeisan blas and lapack compile tuning level3=-O3, QA shows egb has numerical errors
 level3=-O3 used for all other level3 files, used -O2 for egb

 code restructuring
 tried splitting egb_loop1 and egb_loop2 - same performance tested impact of !dir$ IVDEP in many locations; no improvement© NEC Corporation 2004

35

35

AMBER 7.0
Sander gb_mb

Routines

70
real seconds

60

other

50

egb_loop3

4030egb_loop220egb_loop1

100initial© NEC Corporation 2004

final

36

36

GAMESS-US GAMESS is maintained by the members of the Gordon
research group at Iowa State University
http://www.msg.ameslab.gov/GAMESS/GAMESS.ht
ml The General Atomic and Molecular Electronic Structure
System (GAMESS) is a program for ab initio quantum
chemistry.
 Fortran single executable using MPI over sockets forparallel. 64bit real, integer, pointer gms_hf dataset
 nicotine Hartree Fock SCF single point calculation
 one of GAMESS benchmarks distributed in early 1990’s© NEC Corporation 2004

37

37

GAMESS-US
 profile gms_hf:
 42.0% hstar
 9.5% dspdfs
 3.7% forms

 initial compiler optimization -O2
 -O3 top 5 routines: slower
 -O1 hstar: slower hstar: ivdep on nonpipelining loops: slower
 turn off packing: slower use vector code for dspdfc & forms: slower update source, compiler, MathKeisan
 saw some tests with 12% time in __libc_read:comes from compile libraries on cross-mounted
disk: use static link and no shared object (lib*.so)© NEC Corporation 2004

38

38

GAMESS-US
gms_hf350

real seconds

300
250
1cpu
2cpu
4cpu

200
150
100
50
0
initial© NEC Corporation 2004

final

39

39

User MD Tuning - compile options
 customer provided molecular dynamics application, written in
about 12,000 lines of C with customer test input
 original compile flags

 ineffective with -mp (maintain precision) severly limits
performance ecc -O3 -tpp2 -Zp16 -static -mp -IPF_fma -IPF_fltacc
 tuned compile flags

 ecc -O3 -tpp2 -Zp16 -static -restrict -ftz -ip
 -restrict (or -ansi_alias) tells compiler that pointers labeled
restrict in routine call have independent locations- restrict pointers provides information about potential
dependencies that can enable pipelining- restrict pointers declared on most of MD’s malloc’ed locations

 -ip turns on inter-procedural optimizing within the file thatis being compiled© NEC Corporation 2004

40

40

User MD Tuning - code modifications
 use tuned library call
 replace original code with MathKeisanfunction vdSinCos
 computes (sin(a[i],cos(a[i]) for i=0…n pipelines multiple sin and cos calculations
 programmer level loop recoding
 split main loop of important routine into 3- non-bonded pair list generation- calculation of force potentials across all pairs- application of forces across all pairs enables pipelining© NEC Corporation 2004

41

41

User MD25

real seconds

20
original
compile flags
restrict pointers
vdSinCos()
split loop

15
10
5
0

© NEC Corporation 200442

42