第 3 回先進スーパーコンピューティング環境研究会（ASE 研究会，Advanced Supercomputing Environment）
中島研吾
東京大学情報基盤センター
2009 年 1 月 20 日（火）15 時から 18 時まで，東京大学情報基盤センター大会議室において，第 3 回先進スーパーコンピューティング環境研究会（ASE 研究会，Advanced SupercomputingEnvironment）が開催された。今回は，「ペタスケールアプリケーション開発支援環境に関する国際ワークショップ」と題して，科学技術振興機構 戦略的創造研究推進事業（CREST）との共催にて実施された。国内の大学，研究機関，企業から合計 20 名の出席者があり，活発な議論が行われた。
プログラムは表 1 に示すように，招待講演 1 件と情報基盤センタースーパーコンピューティ
ング研究部門に所属する教員による 3 件の講演から構成されている。
表 1 第 3 回 ASE 研究会
講演者
Jonathan Carter （ NERSC ，Lawrence Berkeley National
Laboratory, USA）（招待講演）
石川 裕（東大・情報理工／情報基盤センター）片桐孝洋（東大・情報基盤センター）
中島研吾（東大・情報基盤センター）

プログラム
講演タイトルCollaborative Benchmarking at NERSC and the T2K Open
SuperComputer (Tokyo)
An Overview of Seamless and Highly-Productive Parallel
Programming Environment Project
Towards Petascale Eigensolver and Its Auto-tuning Methodology
Parallel Multistage Preconditioners by Hierarchical Interface
Decomposition on T2K Open Supercomputer (Tokyo) with Hybrid
Parallel Programming Models

招待講演者の Jonathan T. Carter 博士1はローレンスバークレイ国立研究所（Lawrence BerkeleyNational Laboratory）の National Energy Research Scientific Computing (NERSC) Center で User
Service Group Leader を勤めており，スーパーコンピュータの導入，運用の責任者の一人である。
スパコンの導入，運用を円滑に実施するための評価手法，ベンチマーキングの研究を行っており，今回は NERSC における実例も交えてシステムの評価手法について紹介があった。今回はCarter 博士のご好意により，後掲のように当日の発表資料を掲載しているので，詳細についてはそちらをご覧いただきたいが，非常に興味深かったのは：・ 様々なユーザーアプリケーションの特徴を網羅し，実際のアプリケーションに基づいて作成された 7 種類のベンチマークテストによって評価を実施している。

1

http://www.nersc.gov/~jcarter/・ いわゆる「検収」を，運用前のテストだけでなく，実運用環境でも実施している。また，実行性能，入出力性能を常時モニターしており，例えばコンパイラやライブラリのバージョンアップ等による様々な影響についても考慮している。という点であり，今後の調達，センター運用においても参考になる点が多かった。
Carter 博士の今回の来日の目的は，これらのベンチマークを Hitachi

HA8000 クラスタシス

テム（T2K オープンスパコン（東大））に適用し，評価することである。現在 NERSC で稼動中の Cray XT4 システムは T2K オープンスパコンと同様，AMD Quad-core Opteron（Barcelona）2.3GHz をしており，両者の比較という点からも興味深い。結果については今後，本「スーパーコンピューティングニュース」で紹介して行きたい。
引き続いて実施された，センター3 教員による講演は，現在，筑波大学，京都大学と共同で実施されている「シームレス高生産・高性能プログラミング環境」（代表：石川裕教授（東大））に関連したものである。
本プロジェクトは文部科学省

次世代 IT 基盤構築のための研究開発「e-サイエンス実現のためのシステム統合・連携ソフトウェアの研究開発（テーマ①：高生産・高性能計算機環境実現のためのシステムソフトウェアの研究開発）」として実施されているものであり，本学では「高効率・高可搬性ライブラリ」を担当している。第 3 回 ASE 研究会では，プロジェクトの概要（石川），自動チューニング機構の開発（片桐），マルチコア環境での最適化（中島）について紹介した。
次回（第 4 回）ASE 研究会は 2009 年 3 月 27 日（金）14 時から 17 時を予定している。基調講演 1 件（Julien Langou 博士（University of Colorado, Denver）），招待講演 2 件（伊藤祥司博士（理化学研究所），村上弘博士（首都大学東京））が開催の予定である。

写真：Jonathan T. Carter 博士による講演の一コマ

Collaborative Benchmarking at NERSC and the T2K Open
SuperComputer (Tokyo)
Jonathan T. Carter
NERSC, Lawrence Berkeley National Laboratory
以下に第 3 回先進スーパーコンピューティング環境研究会（ ASE 研究会，Advanced
Supercomputing Environment）における Jonathan T. Carter 博士（NERSC，Lawrence Berkeley NationalLaboratory）の講演資料を掲載する。

NERSC Mission

Collaborative Benchmarking at NERSC andthe T2K Open SuperComputer (Tokyo)
Jonathan Carter
NERSC Division, Berkeley Lab
JTCarter@lbl.gov

The mission of the National Energy
Research Scientific Computing Center(NERSC) is to accelerate the pace of
scientific discovery by providing highperformance computing, information, data,and communications services for all DOE
Office of Science (SC) research.

Information Technology Center, University of Tokyo
January 20 2009

2Science Over the YearsNERSC is the Production
Facility for DOE SC• NERSC serves all areas~3000 users, ~400 projects• Allocations managed by DOE– 10% INCITE awards:• Large allocations, extra service• Used throughout SC; not just DOE mission– 70% Production (ERCAP) awards:• From 10K hour (startup) to 5M hour• Only available at NERSC– 10% each NERSC and DOE/SC reserve• Award mixture offers
NERSC is enabling new science in all disciplines,
with over 1,500 refereed publications in 200743

NERSC 2008 Configuration
Large-Scale Computing System
Franklin (NERSC-5): Cray XT4• 9,660 nodes; 38,640 cores• ~37 Tflop/s sustained SSP (355 Tflops/s peak)

Clusters

NERSC GlobalFilesystem (NGF)

Bassi (NCSb)• IBM Power5 (888 cores)
Jacquard (NCSa)• LNXI Opteron (712 cores)
PDSF (HEP/NP)• Linux cluster (~1K cores)

230 TB; 5.5 GB/s
HPSS Archival Storage• 44 PB capacity• 10 Sun robots• 130 TB disk cache

5–High impact through large awards–Broad impact across science domainsAnalytics /Visualization• Davinci (SGIAltix)

NERSC Global Filesystem (NGF)• After thorough evaluation andtesting phase in production• Based on IBM GPFS
• Seamless data access from allof NERSC’s computational andanalysis resources• Single unified namespacemakes it easier for users tomanage their data acrossmultiple system• First production globalfilesystem spanning fourplatforms, three architectures,and four different vendors

2005-2010 Plan

2005: NERSC Five-Year Plan
• Three trends identified:– widening gap between sustained
application performance and peak
– emergence of large, multidisciplinary
computational science teams– flood of scientific data from both
simulations and experiments• Requirements/Trend Analysis led
to the NERSC Five-Year Plan
• Approved by DOE and included:– NERSC-5 (Franklin, Cray XT4)
– NERSC-6: 3-4x NERSC-5 sustained
performance; initial delivery in 2009
8Science Benefits from Increased
Computational Capability

Nuclear Physics
Calculation: High accuracy ab initio calculations on O16 using
no-core shell model and no-core full configurationinteraction model• PI: James Vary, Iowa State• Science Results:•• 2007 Franklin experience– Accepted in October 2007
– DOE allocated the machine starting in January 2008
– Before acceptance, full NERSC workload was running– Before and after acceptance, all runs on Franklin were
“free” (not charged against user’s allocation)– Most accurate calculations todate on this size nuclei– Can be used to parametrize newdensity functionals for nuclearstructure simulations• Resulting experience:– Franklin was 80%-95% utilized within a week of acceptance– Users consumed 5x more compute time than they wereallocated in 2007 (14x for largest users)– Users took the opportunity to produce science results;
experiment with new algorithms, and scale to new levels•Scaling Results:– 4M hours used; 200K allocated– 12K cores; vs 2-4K before
Franklin uncharged time
– Diagonalize matrices ofdimension up to 1 billion

9Validating Climate Models•••

INCITE Award for “20th Century Reanalysis” using an Ensemble
Kalman filter to fill in missing climate data since 1892
PI: G. Compo, U. Boulder

Science Results:– Reproduced 1922
Knickerbocker storm– Data can be used tovalidate climate andweather models•

Modeling Dynamically and SpatiallyComplex Materials for Geoscience• Calculation: Simulation of seismic waves throughsilicates, which make up 80% of the Earth’s mantle• PI: John Wilkins, Ohio State University• Science Result– Seismic analysis shows jumps in wave velocity due tostructural changes in silicates under pressure• Scaling Resultí First use for elasticconstants
í 8K core vs. 128 on
allocated time

Scaling Results:– 3.1M CPU Hours inallocation– Scales to 2.4K cores– Switched to higher
resolution algorithmwith Franklin access

10Sea level pressures with color showing uncertainty(a&b); precipitation (c); temperature (d). Dots indicate
measurements locations (a).
1112

Concurrency Level is
Constrained by Availability

Nanoscience Calculations andScalable Algorithms

Median Job Concurrency(Job Size where 50% of Raw Hours Used)• Calculation: Linear Scaling 3D Fragment (LS3DF). Density
Functional Theory (DFT) calculation numerically equivalent tomore common algorithm, but scales with O(n) in number of atoms
rather than O(n3)• PI: L.W. Wang, LBNL
• Science Results• Calculated dipole moment on 2633
atom CdSe quantum rod,Cd961Se724H948 .
• Scaling Results• Ran on 2560 cores• Took 30 hours vs many months forO(n3) algorithm• Good parallel efficiency (80% on1024 relative to 64 procs)
1314

NERSC Allocation Breakdown

NERSC Response

Total NERSC allocation time divided into thirds
By TimeBy Number ofProjects

8 Projects
31 Projects

8 projects use 1/3 of the hours

1/31/3• Further progress in these key science
missions (and others) requires increased
computational capability.

2M - 9M hours each

31 projects use next 1/3

1/3356 Projects(including58 startupprojects)

1/3

500k - 1.8 M hours each

• NERSC Strategy: Increase user scientificproductivity via timely introduction of thebest new technologies designed to benefit
the broadest subset of the NERSC workload> 350 projects use bottom 1/3=> Upgrade Franklin=> Commence NERSC-6.

1000-500k hours eachStartup Projects

Middle projects have insufficient
resources for large
15 production runs

16

Maintaining Service While
Improving Service

Franklin Quad Core Upgrade• All 9,672 nodes change from 2.6-GHz
dual core to 2.3-GHz quad core.
• QC nodes have 8 GB memory, same
average GB/core as on DC Franklin.
• Memory from 667 MHz to 800 MHz.
17Nodes and Cores

Franklin System Size

• In-place, no-interruption upgrade taking
place between July and October, 2008.

Phase

StartDate

BeforeNumberof
Sustained
NumberofQuad
DualCore
PerformanceCoreRacks
Racks(SSPTflops/s)

SSPTflop/s
Days

July1,2008

102

019.2

115Jul08

78014.7425.8

2a13Aug08841822.2

177.3

2b21Aug08

54

1816.5330.4

3a10Sep08

5448

27.1

162.6

3b16Sep08

1248

19.2

403.2

4a7Oct08

092

32.5

454.6

4b21Oct08

0102

36.0

18

NERSC-5 Application Benchmarks

Measuring Success“For better or for worse,
benchmarks shape a field.”David Patterson, UCB CS267 2004

“Benchmarks are only useful insofar as
they model the intended computational
workload.”
Ingrid Bucher & Joanne Martin, LANL, 1982

Benchmark

Science Area

Algorithm Space

Base Case
ConcurrencyCAM

Climate (BER)

Navier Stokes
CFD56, 240
Strongscaling

D Grid, (~.5degresolution);
240 timesteps

GAMESS

Quantum
Chem (BES)

Dense linear
algebra64, 384 (Sameas Ti-06)

GTC

Fusion (FES)

PIC, finite
difference64, 256
Weak scaling

PMEMD

Life Science(BER)

Particle MeshEwald

64, 256
Strongscaling

PowerSpectrum
Estimation

64,256, 1024
Weak scaling

Vary Npix; 730
MB per task,
200 GB disk

F90

(HEP & NP)

MILC

Lattice Gauge
Physics (NP)Conjugate
gradient, sparse
matrix; FFT

64, 256, 2048
Weak scaling

16^4 Local
Grid, ~4,000
iters

C, assem.

PARATECMaterialScience(BES)

DFT; FFT,
BLAS364, 256
Weakscaling

250-686 Atoms,
1372 bands, 10iters

F90

MadBench Astrophysics

ProblemDescription

Lang

Libraries

F90netCDF

DFT gradient,
MP2 gradient

F77

DDI, BLAS

10 particles percell

F90

F90

Scalapack,
LAPACK

Scalapack,
FFTW

20

QC / DC Comparison

PARATEC: Parallel Total Energy CodeNERSC-5 Benchmarks• Captures the performance of ~70% of NERSC materialscience computation.
•Planewave DFT; calculation in both Fourier and real space;custom 3-D FFT to transform between.

•Uses MPI / SCALAPACK / FFTW / BLAS3•

All-to-all data transpositions dominate communications.

Dual Core faster
Quad Core faster

Compare time for n cores on DC socket to time for n cores on QC socket.
Data courtesy of Helen He, NERSC USG
2223

PARATEC: Performance
Medium Problem (64 cores)

FFTs1
Projectors1
Matrix-Matrix1
Overall2•••

Dual Core
425
4,600
4,750
2,900 (56%)

Quad Core
537
7,800
8,200
4,600 (50%)

Ratio
1.3
1.7
1.7
1.6

1 Rates in MFLOPS/core from PARATEC output.
2 Rates in MFLOPS/core from NERSC-5 reference count.

Projector/Matrix-Matrix rates dominated by BLAS3 routines.

=> SciLIB takes advantage of wider SSE in Barcelona-64.

24

Communication Topology for
PARATEC from IPM.

NERSC Sustained Performance• 7 application benchmarks• Two machines (DC & QC)• How do we summarize performance?
• How do we express computing
capability over time?

Sustained System Performance (SSP)• Aggregate, un-weighted measure of sustained
computational capability relevant to NERSC’sworkload.
• Geometric Mean of the processing rates of sevenapplications multiplied by N, # of cores in the system.
– Largest test cases used.

• Uses floating-point operation count predetermined on
a reference system by NERSC.

NERSC Composite SSP Metric
The time for the largest concurrency run of each full application
benchmark is used to calculate the SSP.

NERSC-5 SSPCAM
GAMESS
240
384
Climate Quantum
Modeling Chemistry

GTC
256
FusionMILCPMEMD MADBench
2048
256
1024
Lattice
MolDynAstro
QCD

PARATEC
256
MatSciDFTFor Franklin DualCore, (N = 19,344) 19.3 Tflop/s out of 101 Tflop/s peak
QuadCore, (N = 38,640) 37.6 Tflop/s out of 355 Tflop/s peak2728NERSC-6 Project Overview• Acquire the next major
NERSC computing system– Goal: 70-100 Sustained TF/s on
representative applications(NERSC-6 SSP)NERSC Next-Generation System– Fully-functional machine
accepted in FY10 and available
for DOE allocation– RFP release September 4, 2008.
– Approach designed to select thebest machine for science withgreatest flexibility for bothNERSC and vendors.
29

Base Case for Application RunsNERSC-6 Benchmarks• New codes/methods address evolution of theworkload, emerging programming models,algorithms– New SSP applications: MAESTRO and IMPACT-T– UPC, AMR, implicit and sparse methods– Comprehensive workload study:– http://www.nersc.gov/projects/procurements/NERSC6/NERSC6Workload.pdf• Largest concurrency increases from 2,048 to 8,196– Increased focus on strong scaling• Two ways for vendors to run benchmarks…

• LCD for comparison among proposed systems.
• Limits the scope of optimization.
– Modifications only to enable porting and correct execution.

•••••

Limits allowable concurrency to prescribed values.
MPI-only (even if OpenMP directives present).
Fully packed nodes.
Libraries okay (if generally supported).
Hardware multithreading okay, too.
– Expand MPI concurrency to occupy hardware threads.

Optimized Case for Application Runs

Use a Hierarchy of Benchmarks
Full Workload• Allow the Offeror to highlight features of theproposed system.
• Applies to seven SSP apps only, all test problems.
• Examples:– Unpack the nodes;– Higher (or lower) concurrency than reference base case;– Hybrid OpenMP / MPI;
– Source code changes for data alignment / layout;– Any / all of above.

composite tests
full application
stripped-down app

Integration(reality) increases

kernels
systemcomponent tests• Caveat: SSP based on total number of processors
blocked from other use.

NERSC-6 Application Benchmarks
Benchmark

Science Area

Algorithm Space

Base Case
Concurrency

CAM

Climate (BER)

Navier Stokes
CFD56, 240
Strongscaling

D Grid, (~.5degresolution);
240 timesteps

ProblemDescription

F90

Lang

netCDF

Libraries

GAMESS

Quantum
Chem (BES)

Dense linear
algebra256, 1024(Same as Ti09)

DFT gradient,
MP2 gradient

F77

DDI, BLAS

GTC

Fusion (FES)

PIC, finite
difference

512, 2048
Weak scaling

100 particles
per cell

F90

IMPACT-T

Accelerator
Physics(HEP)

PIC, FFT
component

256,1024
Strongscaling

50 particles percell

F90

FFTW

MAESTRO

Astrophysics(HEP)

Low Mach Hydro;
block structuredgrid multiphysics

512, 2048
Weak scaling

16 32^3 boxes
per proc; 10timesteps

F90

BoxlibMILC

Lattice Gauge
Physics (NP)Conjugate
gradient, sparse
matrix; FFT

256, 1024,8192
Weak scaling

8x8x8x9 Local
Grid, ~70,000
iters

C, assem.

PARATEC

MaterialScience(BES)

DFT; FFT, BLAS3256, 1024
Strongscaling

686 Atoms,
1372 bands, 20iters

F90

NERSC-6 Composite SSP Metric
The largest concurrency run of each full application benchmark
is used to calculate the composite SSP metric

NERSC-6 SSPCAM
240

Open Benchmark Suite– Compare multi-socket nodes to single
socket– Compare Myrinet to Cray HSN– Compare Hitachi compilers to PGI,
Pathscale

GAMESS
1024

GTC
2048

IMPACT-T MAESTRO
1024
2048

MILC
8192

PARATEC1024

For each benchmark measure•FLOP counts on a reference system•Wall clock run time on various systems

Scalapack,FFTW• Collect data from other HPC centersusing application benchmarks• Starting to run on T2K (Tokyo) cluster
this week

Understanding
Increases

Acknowledgements••••••

Kengo Nakajima, ITC, Tokyo University
Kathy Yelick, NERSC Director
John Shalf, Harvey Wasserman, NERSC SDSA
Nick Cardo, NERSC Franklin Project Lead
Helen He, Katie Antypas, NERSC USG
Joe Glenski, et al., Cray
This work was supported by Office of Advanced Scientific andComputational Research, Office of Science, US Department ofEnergy under Contract No. DE-AC02-05CH11231