# Presentation: 917680

## Getting Started on Jacquard

**Richard Gerber**
**NERSC User Services**
- _[RAGerber@lbl.gov](mailto:RAGerber@lbl.gov)_
**CS267**
**February 1, 2008**
**UC Berkeley**

## Presentation Overview

**How to get help**
**Cluster overview**
**Connecting to Jacquard**
**Nodes and processors**
**Operating system**
**File systems**
**Compilers and compiling**
**Running jobs/Batch system & queues**
**Optimization and debugging tools**

## How to Get Help

**NERSC web site**
_**[https://www.nersc.gov/](https://www.nersc.gov/)**_
_**[https://www.nersc.gov/nusers/systems/jacquard/](https://www.nersc.gov/nusers/systems/jacquard/)**_
**NERSC Consultants**
**510 486-8611 (800-66NERSC)**
**E-mail: **_**[consult@nersc.gov](mailto:consult@nersc.gov)**_
**Please use the e-mail address we have on file**
**Web: **_**[https://help.nersc.gov/](https://help.nersc.gov/)**_
**Submit questions on line**
**FAQs very much out of date! **
**NERSC office hours on campus**
**details TBD**

## Presentation Overview

**How to get help**
**Cluster overview**
**Connecting to Jacquard**
**Nodes and processors**
**Operating system**
**File systems**
**Compilers and compiling**
**Running jobs/Batch system & queues**
**Optimization and debugging tools**
**Third-party software**
**Node interconnect**
**Message passing interface**

## Introduction to Jacquard

**Named in honor of inventor Joseph Marie Jacquard, whose loom was the first machine to use punch cards to control a sequence of operations. **
***Jacquard***** is a 640-CPU ****Opteron**** cluster running a Linux operating system. **
**Integrated, delivered, and supported by ****Linux Networx**
**Jacquard has 320 dual-processor nodes available for scientific calculations. (Not ****dual-core**** processors.)**
**The nodes are interconnected with a high-speed ****InfiniBand**** network. **
**Global shared file storage is provided by a ****GPFS**** file system from IBM.**

## Jacquard

_**[http://www.nersc.gov/nusers/systems/jacquard/](http://www.nersc.gov/nusers/systems/jacquard/)**__** **_

## Jacquard Characteristics

|  |  |
| --- | --- |
| Processor type | Opteron 2.2 GHz |
| Processor theoretical peak | 4.4 GFlops/sec |
| Processors per node | 2 |
| Number of application nodes/processors | 320 / 640 |
| System theoretical peak (computational nodes) | 2.8 TFlops/sec |
| Physical memory per node (usable) | 6 (5.2) GBytes |
| Number of spare application nodes | 4 |
| Number of login nodes | 4 |
| Node interconnect | InfiniBand |
| Global shared disk | GPFS: 30 TBytes usable |
| Batch system | Torque (PBS)/ Moab |

## Jacquard’s Role

**Jacquard is meant to be for codes that do not scale well.**
**Most jobs use 64 or fewer CPUs. 25% of total hours used by 16-way jobs.**
**Increasingly being used for data analytics: e.g., post-processing**

## Presentation Overview

**Cluster overview**
**Connecting to Jacquard**
**Nodes and processors**
**Operating system**
**File systems**
**Compilers and compiling**
**Running jobs/Batch system & queues**
**Optimization and debugging tools**
**Third-party software**
**Node interconnect**
**Message passing interface**

## Connecting to Jacquard

**Interactive shell access is via SSH (v2)**
**UNIX: **
**ssh [–l *****login_name*****] –X jacquard.nersc.gov**
**PC: Various applications**
_**[https://www.nersc.gov/nusers/help/access/ssh_apps.php](https://www.nersc.gov/nusers/help/access/ssh_apps.php)**_
**MAC: **
**(Use X11) ****ssh [–l *****login_name*****] –Y jacquard.nersc.gov**
**Passwords**
**You need only one password to access all NERSC resources (except HPSS)**
**You must call NERSC account support office to get a password: 510 486-6820 (24x7, but best to call 9-4 Pacific)**
**Know your project PI/manager (Simon) and repo (mp309)**
**You change your password and default login shell via a web interface: **_**[https://nim.nersc.gov/](https://nim.nersc.gov/)**_
**Login Failures**
**Your account will be locked after 3 failed logins**
**Call the account support office to unlock the account**

## Connecting to Jacquard

**You log in to one of four login nodes (randomly). **
**Login nodes are for compiling, code development, etc. – not for running applications.**
**Outbound network services are open (e.g., ftp).**
**Most inbound network ports are blocked.**
**Including ftp: no clear-text password access.**
**Use hsi ****for interfacing with HPSS archival mass storage system.**

## Presentation Overview

**Cluster overview**
**Connecting to Jacquard**
**Nodes and processors**
**Operating system**
**File systems**
**Compilers and compiling**
**Running jobs/Batch system & queues**
**Optimization and debugging tools**
**Third-party software**
**Node interconnect**
**Message passing interface**

## jacquard.nersc.gov basics

**jacquard.nersc.gov basics**

| Resource | Speed | Bytes |
| --- | --- | --- |
| Registers |     0.5 ns |    2 KB |
| L1 Cache |    1.5 ns |   64  KB |
| L2 Cache |   45 ns |     1 MB |
| Main Memory | 70-117 ns |     6 GB |
| Remote Memory |   5 us |     2  TB |
| GPFS |   10 ms |   15  TB |
| HPSS |      5 s |     9  PB |

- 320 x

- HPSS

- IB

- 640 dedicated CPUs, 4 shared login CPUs
- Smaller caches, HT, Really Fast

- 2 way Opteron node

**Jacquard:**

**HT**

## Opteron Block Diagram : Not strictly SMP

**Opteron Block Diagram : Not strictly SMP**

- 1 TLB  per CPU
- 1K entries 4K pages  4MB coverage

- SDRAM

- SDRAM

- Switch, I/O

## Flops @ 2.2 GHz

**Peak Theoretical Flops**
- Double (64 bit) floats : 1 add + 1 mult = 2.2 GFlop/s
- Single (32 bit) floats   : 2 add + 2 mult = 4.4 GFlop/s
**Peak Realized Flops**
- Double (64 bit) floats : 1.9 GFlop/s
- Single (32 bit) floats   : 3.4 GFlop/s

- Memory BW
- 4 GB/sec per CPU

## Node Interconnect

**Nodes are connected by an InfiniBand high speed network from ****Mellanox.**
**Adapters and switches from Mellanox**
**Low latency: 7****μs vs. 25 μs on IBM SP Seaborg, 4.5μs**** on IBM p575 POWER 5 Bassi**
**Bandwidth ~ 2.5 GB/sec each direction**
**“****Fat tree”**

## Presentation Overview

**Cluster overview**
**Connecting to Jacquard**
**Nodes and processors**
**Operating system**
**File systems**
**Compilers and compiling**
**Running jobs/Batch system & queues**
**Optimization and debugging tools**
**Third-party software**
**Node interconnect**
**Message passing interface**

## Operating system

**Jacquard is running ****Novell SUSE Linux Enterprise Linux 9**
**Has all the “usual” Linux tools and utilities (gcc, GNU utilities, etc.)**
**It was the first “enterprise-ready” Linux for Opteron.**
**Novell (indirectly) provides support and product lifetime assurances (5 yrs).**

## User Environment

**Set your default login shell via the NERSC NIM web interface**
**https://nim.nersc.gov/**
**NERSC supplies $HOME directory “dot files”**
**Put all your extensions/customizations is *.ext files (e.g., .bashrc.ext, .login.ext)**
**DO NOT overwrite the default dot files!**
**Use /usr/common/usg/bin/fixdots if you do. **

## Modules

**Third-party software is accessed via the modules software package.**
_**[https://www.nersc.gov/nusers/resources/software/os/modules.php](https://www.nersc.gov/nusers/resources/software/os/modules.php)**_

**jacin04% module list**
**Currently Loaded Modulefiles:**
**  ****1) mvapich/0.9.5-mlx1.0.3.patched   3) null**
**  ****2) path/3.1                         4) acml/4.0**

**jacin04% module avail**
**(Big list of software!)**

**jacin04% ****which totalview**
**totalview: Command not found.**
**jacin04% ****module load totalview**
**jacin04% ****which totalview**
**/usr/common/usg/totalview/toolworks/totalview.8.1.0-0/linux-x86-64/bin/totalview**

## Modules

**Libraries are controlled by modules also.**

**jacin04% module load scalapack**
**jacin04% module help scalapack**
** **
**----------- Module Specific Help for 'scalapack/1.7' --------------**
** **
**        ****SCALAPACK 1.7 is a library of Parallel Linear Algebra routines**
** **
**        ****To link, you can do:**
** **
**                ****module load scalapack acml**
**                ****mpif90 code_f.o $SCALAPACK $FBLACS $ACML**
**                ****mpicc  code_c.o $SCALAPACK $CBLACS $ACML -lpathfortran**
** **
**        ****See http://acts.nersc.gov/scalapack/at-nersc.html for more**
**        ****information.**
**jacin04% module help scalapack**
**[ shows all the PATH and ENV variables set by the modules; useful if you want to see the full path to libraries and include files. ]**

## Modules

**jacin04% module help**
**  ****Modules Release 3.1.6 (Copyright GNU GPL v2 1991):**
**  ****Available Commands and Usage:**
**        ****+ add|load              modulefile [modulefile ...]**
**        ****+ rm|unload             modulefile [modulefile ...]**
**        ****+ switch|swap           modulefile1 modulefile2**
**        ****+ display|show          modulefile [modulefile ...]**
**        ****+ avail                 [modulefile [modulefile ...]]**
**        ****+ use [-a|--append]     dir [dir ...]**
**        ****+ unuse                 dir [dir ...]**
**        ****+ update**
**        ****+ purge**
**        ****+ list**
**        ****+ clear**
**        ****+ help                  [modulefile [modulefile ...]]**
**        ****+ whatis                [modulefile [modulefile ...]]**
**        ****+ apropos|keyword       string**
**        ****+ initadd               modulefile [modulefile ...] ****(load at login)**
**        ****+ initprepend           modulefile [modulefile ...]**
**        ****+ initrm                modulefile [modulefile ...]**
**        ****+ initswitch            modulefile1 modulefile2**
**        ****+ initlist**
**        ****+ initclear**

## Process Limits

**Interactive processes on login nodes**
**1 hour CPU time**
**131072 Kb of memory**
**You can not change these**
**Compute nodes**
**Default: 2621440 KB (2.6 GB) of memory**
**If using only 1 CPU/node, you can change**
**ulimit -v 5242880 (sh-like)**
**limit vmemoryuse 5242880 (csh-like)**

## Presentation Overview

**Cluster overview**
**Connecting to Jacquard**
**Nodes and processors**
**Operating system**
**File systems**
**Compilers and compiling**
**Running jobs/Batch system & queues**
**Optimization and debugging tools**
**Third-party software**
**Node interconnect**
**Message passing interface**

## Disks and file systems

**Three file systems**
**Home (5 GB, 15K inode quotas)**
**Scratch (50 GB, 50K inode quota)**
**Project (/project): allocated on a per project basis**
**$SCRATCH environment variable is defined to contain path to a user’s personal scratch space.**
**$SCRATCH gives far better performance, but may be purged if space is needed**

## Archival Storage

**HPSS (High Performance Storage System)**
**Your HPSS password is distinct from your NIM/LDAP/Jacquard password**
**Command-line FTP-like interface: hsi**
**HPSS is permanent storage; /scratch may be purged**
**Use HPSS to back up your data and store large files**
**Technically, there are quotas but you should not hit them in this class.**

## Presentation Overview

**Cluster overview**
**Connecting to Jacquard**
**Nodes and processors**
**Operating system**
**File systems**
**Compilers and compiling**
**Running jobs/Batch system & queues**
**Optimization and debugging tools**
**Third-party software**
**Node interconnect**
**Message passing interface**

## Compilers

**High performance Fortran/C/C++ compilers from ****Pathscale****.**
**Fortran compiler: ****pathf90**
**C/C++ compiler: ****pathcc, pathCC**
**MPI compiler scripts use Pathscale compilers “underneath” and have all MPI ****–I, -L, -l**** options already defined:**
**mpif90**
**mpicc**
**mpicxx**

## Pathscale Compilers

**Default compilers: Pathscale Fortran 90, C, and C++.**
**Module “path” is loaded by default and points to the current default version of the Pathscale compilers (currently 3.1).**
**Other versions available:  module avail path.**
**Extensive vendor documentation available on-line at **_**[http://pathscale.com/docs.html](http://pathscale.com/docs.html)**_**.**
**Commercial product:  well supported and optimized.**

## GCC Compilers

- GCC compilers are available. If you like them, use them but they probably won’t perform as well as the Pathscale compilers.

**jacin04% module avail gcc**
** **
**--- /usr/common/usg/Modules/modulefiles -----------------**
**gcc/3.3            gcc/4.0.1          gcc/4.0.2(default)**

## Compiling Code

**Compiler invocation:**
**No MPI:  pathf90, pathcc, pathCC.**
**MPI:  mpif90, mpicc, mpicxx**
**The mpi compiler invocation will use the currently loaded compiler version (including gcc, but no GNU F90).**
**The mpi and non-mpi compiler invocations have the same options and arguments.**

## Interlanguage

**C symbols are “as-is”**
**Pathscale Fortran adds one underscore to end of routine names**
**Consult manuals for name-mangling**
**Esp. C++, Fortran modules**
**Pathscale fortran options to control underscoring (see manual)**
**-fno-underscoring**
**-fno-double-underscoring**

## Interlanguage

**Link mixed-language codes with C compiler **
**add ****–lpathfortran ****to the link line**
**Use ****nm**** to examine object symbols**

**jacin04% nm matmul.o**
**0000000000000000 b A_112**
**0000000000483010 b B_113**
**0000000000906020 b C_114**
**                 ****U dgemm_desc**
**                 ****U drand48**
**                 ****U exit**
**                 ****U get_time**
**0000000000000000 T main**
**000000000000011c t matrix_clear**
**00000000000000e1 t matrix_init**
**                 ****U memset.pathscale.opteron**
**                 ****U printf**
**                 ****U square_dgemm**
**0000000000000000 R test_sizes**
**000000000000039a t time_dgemm**
**                 ****U timespec_diff**
**000000000000012b t validate_dgemm**

## Presentation Overview

**Cluster overview**
**Connecting to Jacquard**
**Nodes and processors**
**Operating system**
**File systems**
**Compilers and compiling**
**Running jobs/Batch system & queues**
**Optimization and debugging tools**
**Third-party software**
**Node interconnect**
**Message passing interface**

## Jacquard Overview

- Compute
-  Node

- Compute
-  Node

- Compute
-  Node

- Compute
-  Node

- Compute
-  Node

- Compute
-  Node

- Compute
-  Node

- etc....

- No
- local
- disk

- Login
-  Node

- Login
-  Node

- Login
-  Node

- etc....

**/home**

- Login
-  Node

- Login
-  Node

**/scratch**

- Login
-  Node

- etc....

**/project**

**HPSS**

- Full Linux OS

- Full Linux OS (no logins)

## Running a Job on Jacquard

- Login Node

- Log in from your desktop using SSH
- Compile your code or load a software module
- Write a job script
- Submit your script to the batch system
- Monitor your job’s progress
- Archive your output
- Analyze your results

**On a Jacquard login node:**

- Login nodes run full version of SUSE Linux

**www.nersc.gov/nusers/status/queues/jacquard**/

- NERSC Analytics server (DaVinci)

## Job Scripts

**#PBS –l walltime=01:00:00**
**#PBS –l nodes=4096**
**#PBS –l ppn=2**
**#PBS –q batch**
**#PBS –N MyJob**
**#PBS –V**
**cd $PBS_O_WORKDIR**
**echo “Starting at” `date`**
**mpirun –np 16 ./code.x**

- Directives specify how to run your job

**code.x** runs in parallel on compute nodes

- UNIX commands run on one of the compute nodes

## Directives

**#PBS –l nodes=*****num_nodes***

- Set nodes equal to the total number of nodes you want.

**#PBS –l ppn=*****procs_per_node***

- Set ppn equal to the # of tasks per node you want.

## NERSC Directives

**#PBS –q < *****batch | debug | premium | low >***

- Specify NERSC (charge) class of service.

**#PBS –A *****NERSC_repo_name***

- Specify one of your NERSC repositories to charge against.

**#PBS –V**

- Copy your current compute environment into batch environment.

- See https://www.nersc.gov/nusers/systems/jacquard/running_jobs.php

## Running 1 task per node

- Note that you never directly specify the total number of tasks.

- It is implicit in your settings for nodes and ppn. The default for ppn is 2.
- MPI tasks are mapped one-to-one to cores. You may want to run 1 task (core) per node to increase the memory a single task can access

**#PBS –l nodes=2**
**#PBS –l ppn=1**
**limit vmemoryuse 5242880 #csh,tcsh**
**ulimit –v 5242880        #sh,ksh,bash**

- This will allocate for you 2 nodes to run 2 tasks using one task per node. You must change the default ulimit to use more than 2.6 GB/nodes/task.

## Submitting Jobs

**jacin04% ****qsub script_name**

- Submit your job script with the **qsub** command.

- The batch script directives (#**PBS –whatever**) can be specified on the qsub command line. For example:

**jacin04% ****qsub –A mp309 script_name**

- I recommend putting everything you care about explicitly in the batch script to avoid ambiguity and to have a record of exactly how you submitted your job.

## Modifying Jobs

**qdel <jobid>: deletes queued job**
**qhold <jobid>: holds job in queue**
**qrls <jobid>: release held job**
**qalter <jobid> <options>**
**You can modify some parameters**
**See “****man qalter****”**

## Parallel Job Launch - mpirun

**remote SSH command**

- Node 1

- High Speed InfiniBand Network

**my_code**

- mpirun = commonly used parallel job launch command

**mpirun –np *****number_of_nodes [*****–ppn***** tasks_per_node******]***** *****executable***

- Node 2

- Node 3

- ...

- Node n

- Login

## Parallel Job Launch - mpiexec

**Uses PBS task manager**

- Node 1

- High Speed InfiniBand Network

**my_code**

- mpiexec = launch a parallel job from PBS environment

**mpiexec –n *****number_of_nodes***** *****executable***

- Node 2

- Node 3

- ...

- Node n

- Login

- Use mpirun instead of mpiexec unless you have problems with it.

## Job Scripts

**#PBS –l walltime=01:00:00**
**#PBS –l nodes=4**
**#PBS –l ppn=2**
**#PBS –q batch**
**#PBS –N MyJob**
**#PBS –V**
**#PBS –A mp309**
**cd $PBS_O_WORKDIR**
**echo “Starting at” `date`**
**mpirun –np 8./code.x**

**number_of_tasks must be consistent with**
**#PBS –l nodes=N:ppn=n **

## Interactive Jobs

- You can run interactive parallel jobs:

**jacin04% ****qsub –I –lnodes=1:ppn=1**
**qsub: waiting for job 481437 to start**
**Directory: /usr/common/homes/username**
**jaccn032-m%**

- When your prompt returns, you are on a compute node, and you have other compute nodes reserved for you so you can use mpirun at the command line. Convenient way to run while testing small programs. (But session will be terminated after 30 minute interactive queue limit)

**Login**
** ****node**

**jacin04% ****mpirun –np 4 ./mycode.x**

- From a login node you can use mpirun (but not mpiexec).

**Compute**
** ****node**

## Job Notes

** **The job script itself executes on one of the compute nodes allocated to your job.
-  All UNIX commands and serial programs (including hsi) therefore run on a single node.
- Propagating shell environment variables into the parallel environment is problematic. Runtime libraries that depend on LD_LIBRARY_PATH may not work. mpiexec may work better for this than mpirun does.
-  Don’t run anything non-trivial on login nodes.

## Batch queue limits

**Notes: **

- You can only specify these.

- Your job is routed to one of these based on your specified requirements

## Monitoring Jobs

**Web: **
**10 minute updates**
**https://www.nersc.gov/nusers/status/jacquard/qstat/**
**Monitoring commands – each shows something different**
**showq – moab**
**qstat – torque**
**showstart – moab**
**checkjob – moab**
**qs – NERSC’s concatenation **
**We think this is the most useful!**

## qs

- qs shows running jobs first and then the order in which queued jobs are scheduled to run

**jacin04% qs**
**  ****JOBID ST      USER        NAME   NDS       REQ      USED            SUBMIT**
** ****481337  R   schrier  mWfn_eps2_     1  48:00:00  01:49:07   Jan 31 12:15:27**
** ****481441  R   robsnoe    tio2.pbs     3  48:00:00  04:05:33   Jan 31 15:05:38**
** ****481460  R   pechkis         pbe     2  48:00:00  06:58:05   Jan 31 15:32:36**
** ****481151  R   robsnoe    vasp.pbs     3  48:00:00  09:22:41   Jan 31 11:12:22**
** ****481473  R     jacob   fe3_tight     1  48:00:00  15:45:17   Jan 31 16:47:30**
** ****481332  R   schrier  mWfn_eps2_     1  48:00:00  17:48:34   Jan 31 12:12:22**
** ****471016  Q  shirokov  9He_9_eff-   248  06:00:00         -    Jan 4 03:36:02**
** ****479706  Q    vardha      ch3_au    56  12:00:00         -   Jan 29 16:53:29**
** ****479669  Q    chencc  Chen_H_012    24  24:00:00         -   Jan 29 15:22:43**
** ****479939  Q      u344  st.r25tau1    32  24:00:00         -   Jan 30 12:24:54**
** ****480515  Q      u344  pedt.aLt1.    32  24:00:00         -   Jan 30 14:57:48**

## Presentation Overview

**Cluster overview**
**Connecting to Jacquard**
**Nodes and processors**
**Operating system**
**File systems**
**Compilers and compiling**
**Running jobs/Batch system & queues**
**Optimization and debugging **
**Third-party software**
**Node interconnect**
**Message passing interface**

## Optimization and Debugging

**Performance measurement**
**Not many tools on Jacquard**
**Put timers in your code**
**Use NERSC’s IPM **_**[http://www.nersc.gov/nusers/resources/software/tools/ipm.php](http://www.nersc.gov/nusers/resources/software/tools/ipm.php)**_
**See DOE ACTS tools at ****http://acts.nersc.gov/**
**Totalview debugger**
**module load totalview**
**X-windows based**
**GDB (GNU debugger)**

## Compiler Optimization Options

**Compilers have gotten very good at optimizing code for single processors**
**4 numeric levels ****–On**** where ****n**** ranges from 0 (no optimization) to 3.**
**Default level: -O2**
**–****g without a –O option changes the default to –O0.**

## -O1 Optimization

**Minimal impact on compilation time compared to –O0 compile.**
**Only optimizations applied to straight line code (basic blocks) like instruction scheduling.**

## -O2 Optimization

**Default when no optimization arguments given.**
**Optimizations that always increase performance.**
**Can significantly increase compilation time.**
**-O2 optimization examples:**
**Loop nest optimization.**
**Global optimization within a function scope.**
**2 passes of instruction scheduling.**
**Dead code elimination.**
**Global register allocation.**

## -O3 Optimization

**More extensive optimizations that may in some cases slow down performance.**
**Optimizes loop nests rather than just inner loops, i.e. inverts indices, etc.**
**“****Safe” optimizations – produces answers identical with those produced by –O0.**
**NERSC recommendation based on experiences with benchmarks.**

## -Ofast Optimization

**Equivalent to -O3 -ipa -fno-math-errno**
**-OPT:roundoff=2:Olimit=0:div_split=ON:alias=typed.**
**ipa – interprocedural analysis.**
**Optimizes across functional boundaries.**
**Must be specified both at compile and link time.**
**Aggressive “unsafe” optimizations:**
**Changes order of evaluation.**
**Deviates from IEEE 754 standard to obtain better performance.**

## NAS B Serial Benchmarks Performance (MOP/S)

|  | Seaborg
  Best | -O0 | -O1 | -O2 | -O3 | -Ofast |
| --- | --- | --- | --- | --- | --- | --- |
| BT |   99.6 | 157.2 | 348.1 | 633.6 | 739.8 | 750.9 |
| CG |   46.3 | 101.2 | 128.3 | 236.9 | 223.1 | 224.5 |
| EP |     3.7 |   15.1 |   17.5  |   21.9 |   21.8 |   21.8 |
| FT | 130.1 | 186.2 | 231.5 | 572.4 | 592.7 | did not compile |
| IS |     5.8 |   16.9 |   22.0 |   25.6 |   27.0 |   26.8 |
| LU | 169.8 | 129.0 | 342.4 | 700.0 | 809.9 | 903.2 |
| MG | 163.3 | 109.0 | 257.9 | 747.7 | 518.5 | 530.0 |
| SP |   78.2 | 104.7 | 225.7 | 507.3 | 462.9 | 516.6 |

## ACML Library

**AMD Core Math Library - set of numerical routines tuned specifically for AMD64 platform processors.**
**BLAS**
**LAPACK**
**FFT**
**To use with pathscale:**
**module load acml (built with pathscale compilers)**
**Compile and link with $ACML**
**To use with gcc:**
**module load acml_gcc **
**Compile and link with $ACML **

## Matrix Multiply Optimization Example

**3 ways to multiply 2 dense matrices**
**Directly in Fortran with nested loops**
**Matmul F90 intrinsic**
**dgemm from ACML**
**Example 2 1000 by 1000 double precision matrices.**
**Order of indices:  ijk means**
** ****do i=1,n**
** ****do j=1,n**
** ****do k=1,n**

##  Fortran Matrix Multiply MFLOPs

|  | Seaborg Best |   -O0 |   -O1 |   -O2 |   -O3 | -Ofast |
| --- | --- | --- | --- | --- | --- | --- |
| ijk |   693 |     65 |   117 |   148 | 1640 | 1706 |
| jik |   691 |     63 |   100 |   139 | 1640 | 1802 |
| ikj |   691 |     53 |     51 |     52 | 1471 | 1579 |
| kij |   691 |     48 |     48 |     53 | 1619 | 1706 |
| jki |   691 |     72 |   236 |   598 | 1153 | 1802 |
| kji |   691 |     72 |   183 |   385 | 1599 | 1706 |
| matmul |   946 |   561 |   554 |   564 | 1683 | 1706 |
| dgemm | 1310 | 3876 | 3763 | 3877 | 3877 | 3877 |

## Comments 

**Compiler optimization may interfere with your attempts to measure performance**
**-O0 will, in principle, turn off all optimizations**
**Otherwise, compilers will rearrange your coding constructs to increase performance.**
**Example: If you calculate something in your code, but never use the results, the compiler will eliminate your loop completely!**

## More Info

**Jacquard Training Day Presentations**
**https://www.nersc.gov/about/NUG/meeting_info/Oct05/**

**Jacquard Web Page**
**https://www.nersc.gov/nusers/systems/jacquard/**

**Pathscale Compiler Docs**
**http://pathscale.com/docs.html**