The Programming

Environment for Frontier
Luiz DeRose
Sr. Principal Engineer
Programming Environment Director© 2019 Cray, a Hewlett Packard Enterprise company

The Cray Programming Environment Mission▪ Provide scalable performance, portability, and programmability on
homogeneous and heterogeneous Cray systems▪ Provide the best environment to develop,

Debug

Port

debug, analyze, and optimize applications
for production supercomputing withtightly coupled compilers, libraries, andCompile
toolsCompiler
information

Export/Import
ProgramAnalyses

Debuginformation

application
information▪ Address issues of scale and complexity ofHPC systems

Application

PerformanceAnalysis

Analyze

Queries forApplication
Optimization▪ Close interaction with users and quick
turnaround on customer issues© 2019 Cray, a Hewlett Packard Enterprise company

2

Cray’s PE Vision for Accelerated Computing• Most important hurdle for widespread adoption of accelerated computing in HPC is programming difficulty• Need a single programming model that is portable across machine types• Portable expression of heterogeneity and multi-level parallelism• Programming model and optimization should not be significantly difference for “accelerated” nodes andmulti-core x86 processors• Allow users to maintain a single code base
• Cray’s approach to Accelerator Programming is to provide an ease of use tightly coupled high level
programming environment with compilers, libraries, and tools that can hide the complexity of the system• Ease of use is possible with• Compiler making it feasible for users to write applications in Fortran, C, and C++• Tools to help users port and optimize for hybrid systems• Auto-tuned scientific libraries© 2019 Cray, a Hewlett Packard Enterprise company

3Building a Software Stack for Frontier• ORNL, LLNL, Cray, and AMD working together todeliver full software stack• CORAL-2 NRE• Provides Compiler and library choice• Includes:• Multiple programming environments• Performance and correctness tools• Optimizations such as:• Cray MPI GPU-to-GPU data movement• libsci_acc• Cray PE DL Plugin• Compiler interoperability© 2019 Cray, a Hewlett Packard Enterprise company

Cray®
Programming
Environment

AMDProgramming
Environment

Cray Compiling
Environment(CCE)

AMD OpenMPcompiler(AOCC)
Heterogeneouscompute
Interface forPortability(HIP)

RevealGNUProgramming
Environment

GNU Compiler
Collection

Cray Message Passing Toolkit (CMPT)

Cray Performance Measurement & Analysis Tools (CPMAT)
Cray Debugger Support Tools (CDST)
Cray Scientific and Math Libraries (CSML)
Cray Environment Setup and Compiling Support (CENV)
Cray Chapel

Third Party Products4

The Cray Compiling Environment (CCE)• Cray technology designed for real scientific applications, not just for benchmarks• Arguably the most complete vectorization capabilities in the industry• Full automatic loop vectorization with automatic outer loop vectorization• no need for directives and source code modification• Automatic optimizations deliver performance for a new target through a simple recompile• Compiler optimization feedback for users with annotated listing of source code
• Customized for our users• Fully integrated heterogeneous optimization capability• Fully Integrated and optimized PGAS

• Support multiple platforms• X86 (Intel or AMD)
• ARM SVE+HBM (when available)• NSP-1 and ThunderX• AMD GPUs

© 2019 Cray, a Hewlett Packard Enterprise company

C++ 17OpenMP 5.0

Fortran 2018

C11

UPC 1.35

State of Cray’s Existing C++ Compiler• C++ Trends• C++ usage in HPC has accelerated• C++ language changes have accelerated too• Standards: 1998, 2003, 2011, 2014, 2017, 2020, …
• C++20 is expected to be a major change like C++11• Clang leads C++ compilers in standards support and diagnostics• Customer’s view
• Compile times are slow, especially when templates are used
• C++11 was provided late (2015), whereas other compilers had full C++11 support earlier (2013) and experimental support
before it was ratified• Cray’s view• Significantly more pressure to have a competitive C++ compiler!
• Difficult to reduce compile time while maintaining same performance• Dependent on EDG schedule for supporting new standards• Lots of developer time spent maintaining non-differentiating passes common to all compilers© 2019 Cray, a Hewlett Packard Enterprise company

6

Cray Compiling Environment – CCE 9.0
C and C++ Source

Fortran Source

Cray Libs (I/O, Math, …) ListingTools

C/C++ FE (EDG)

IPA

Optimizer

X86 CG(LLVM)

Aarch64 CG(LLVM)

3rd party with
Cray value added

C/C++ FE (CLANG)
Program Library

Cray developed

Fortran FEGPUCG

LLVM
CCE Differentiation:• Faster OpenMP 4.5• More vectorization• Loopmark• Reveal & CrayPat support• UPC support• C++17

Cray Libs (Math,OpenMP, PGAS)
Listing Tools

CCE-Clang (default)

CCE-Classic

Object Files

CCE 9.0 released on June 2019
Use $module swap cce cce/9.0-classic to change compilers© 2019 Cray, a Hewlett Packard Enterprise company

7CCE-Clang: What Will Be the Impact on Users?
• Cray will track Clang closely• Weekly or biweekly merges• CCE major releases will follow Clang releases (about 2 months lag)
• CCE will have more frequent major releases• Will follow the same version number (almost coincidentally)• Cray-Clang is not Clang• Cray will support Cray-Clang• What steps, if any, are we taking to open source programming environment tools
like compilers, debuggers and profilers for existing and future accelerators?
© 2019 Cray, a Hewlett Packard Enterprise company

8

Scalable Debugging on Cray Systems• Systems with thousands of threads of execution need a new debugging paradigm• Cray’s focus is to build tools around traditional debuggers with innovative techniques for productivity and scalability• Scalable Solutions based on MRNet from University of Wisconsin• STAT - Stack Trace Analysis Tool
• Scalable generation of a single merged stack backtrace
for the application• GUI based tool (stat-gui/stat-view) along with cli tools(stat-cl)• Gain insight into application behavior at a function level• ATP - Abnormal Termination Processing• Scalable core file generation and analysis whenapplication crashes• Generates a merged stack backtrace akin to stat
• Selection algorithm to dump unique core files© 2019 Cray, a Hewlett Packard Enterprise company• gdb4hpc• Conventional CLI based interactive parallel debugger• Look and feel of gdb – syntax is inspired by gdb!
• Debug your application at scale• CCDB - Comparative debugging• A data-centric paradigm• Compare two applications side-by-side• Focus on the data – not state and internal
operations• GUI tool that interacts with gdb4hpc• Valgring4hpc• Parallel valgrind based debugging tool (memcheck)• Aids in detection of memory leaks and errors inparallel applications• Aggregates like errors across PEs/threads
9Cray Performance Analysis Tools on GPUs
• Provide whole program performance analysis• Single tool for CPU and GPU performance
analysis• Single report can include statistics for boththe host and the accelerator• Performance statistics includes accelerator
time, host time, and amount of data copied
to/from the accelerator• Performance statistics mapped back to theuser source by line number• Performance statistics grouped by accelerator
directive

Cray Performance Tools profiled
production applications withover 256,000 ranks© 2019 Cray, a Hewlett Packard Enterprise company• Scaling (running big jobs with a large number ofGPUs)• Results summarized and consolidated in one
place10Simplifying the Parallelization Task with Reveal• Reduce effort associated withadding OpenMP to MPI programs• Identify parallelization and scopingissues• Get feedback on issues down thecall chain (shared reductions, etc.)• Get insight into optimizations
performed by the Cray compiler• Track requests to memory andevaluate the bandwidth contributionof objects within a program© 2019 Cray, a Hewlett Packard Enterprise company

11How About OpenACC?
• Are there any large applications in production with OpenMP foraccelerators?
• How are we going to support existing OpenACC codebases, when weconsider that an automatic tool will struggle to convert hundreds ofthousands of lines of Fortran full of hand-written, multi-gen OpenACC
directives to OpenMP?
• We are seeing that CUDA codes are more portable to AMD GPUs via HIP
than OpenACC codes. What does this tell us about the purported
portability benefits of directives?
© 2019 Cray, a Hewlett Packard Enterprise company

12

Developer Environment for Frontier
Programming
Languages

Programming
ModelsProgramming
Environments

Optimized
Libraries

Tools

Tools
(continued)

Analytics / AI

Languages

Distributed MemoryPrgEnv-

Scientific Libraries

Environment setup

Debuggers

AI Toolboxes

Fortran

Cray MPI
SHMEM

BLAS

Modules / Lmod

gdb4hpc

Cray Urika
AI - Analytics

LAPACKTool Enablement(Spack, CMake,EasyBuild, etc)

TotalView

Cray Compiling
Environment
PrgEnv-cray

C
ScaLAPACKC++

Shared Memory / GPU

ChapelOpenMP
MPI GPU2GPU

Python 3

HIP / CUDA

GNU
PrgEnv-gnu

3rd Party compilers
PrgEnv-AMD

Iterative
Refinement
Toolkit

Valgrind4hpc
STAT

Cray Apprentice2
FFTW

UPC
Fortran coarrays
Coarray C++
ChapelGA© 2019 Cray, a Hewlett Packard Enterprise companyDebugging Support

CrayPATlibsci_acc

PGAS & Global View

RPerformance Analysis

Chapel AI
DDTPorting

I/O Libraries
NetCDF
HDF5

RevealCCDBDL Frameworks
AbnormalTermination
Processing (ATP)Cray PE DL
Scalability Plugin

Tools Interface
CTI

ADIOS

Cray Developed
Cray added value to 3rd party

3rd party packaging
Licensed ISV SW

13

Summary• Cray/HPE will provide a high level programming environment for ORNL Frontier• Fortran, C, and C++ compilers• OpenMP Target directives• Compiler optimizations to take advantage of accelerator and multi-core X86 hardware appropriately• Support for new programming frameworks, such as RAJA and Kokkos• Cray Reveal• Scoping analysis tool to assist user in understanding their code and taking full advantage of SW andHW system• Cray Performance Measurement and Analysis toolkit• Single tool for GPU and CPU performance analysis with statistics for the whole application• Parallel debugger support withDDT,TotalView, or Cray gdb4hpc & CCDB

• optimized on-node, GPU-GPU data movement operations and data movement operations between CPU- andGPU-attached memory regions using the ROCm interface directly• Auto-tuned Scientific Libraries support• Getting performance from the system … no assembly required© 2019 Cray, a Hewlett Packard Enterprise company

14

F O RW A R D L O O K I N GS TAT E M E N T S
This presentation may contain forward-looking statements that involve risks,uncertainties and assumptions. If the risks or uncertainties ever materialize or theassumptions prove incorrect, the results of Hewlett Packard Enterprise Companyand its consolidated subsidiaries ("Hewlett Packard Enterprise") may differ
materially from those expressed or implied by such forward-looking statements andassumptions. All statements other than statements of historical fact are statementsthat could be deemed forward-looking statements, including but not limited to any
statements regarding the expected benefits and costs of the transaction
contemplated by this presentation; the expected timing of the completion of thetransaction; the ability of HPE, its subsidiaries and Cray to complete the transaction
considering the various conditions to the transaction, some of which are outside theparties’ control, including those conditions related to regulatory approvals;
projections of revenue, margins, expenses, net earnings, net earnings per share,
cash flows, or other financial items; any statements concerning the expecteddevelopment, performance, market share or competitive performance relating toproducts or services; any statements regarding current or future macroeconomic
trends or events and the impact of those trends and events on Hewlett Packard
Enterprise and its financial performance; any statements of expectation or belief;
and any statements of assumptions underlying any of the foregoing. Risks,
uncertainties and assumptions include the possibility that expected benefits of thetransaction described in this presentation may not materialize as expected; that thetransaction may not be timely completed, if at all; that, prior to the completion of thetransaction, Cray’s business may not perform as expected due to transaction-related
uncertainty or other factors; that the parties are unable to successfully implement
integration strategies; the need to address the many challenges facing Hewlett
Packard Enterprise's businesses; the competitive pressures faced by Hewlett
Packard Enterprise's businesses; risks associated with executing Hewlett Packard
Enterprise's strategy; the impact of macroeconomic and geopolitical trends andevents; the development and transition of new products and services and theenhancement of existing products and services to meet customer needs andrespond to emerging technological trends; and other risks that are described in ourFiscal Year 2018 Annual Report on Form 10-K, and that are otherwise described orupdated from time to time in Hewlett Packard Enterprise's other filings with theSecurities and Exchange Commission, including but not limited to our subsequent
Quarterly Reports on Form 10-Q. Hewlett Packard Enterprise assumes no obligationand does not intend to update these forward-looking statements.

© 2019 Cray, a Hewlett Packard Enterprise company

15

QUESTIONS?