2TSUBAME 2.0 Begins

The long road from TSUBAME1.0 to 2.0 (Part One)Multi-GPU Computing for
Next-generation Weather Forecasting- 145.0 TFlops with 3990 GPUs on TSUBAME 2.0 -Computer prediction of protein-protein
interaction network using MEGADOCK- application to systems biology -TSUBAME 2.0Begins

The long road from TSUBAME1.0 to 2.0 (Part One)
Satoshi Matsuoka** Global Scientific Information and Computing Center

The long-awaited TSUBAME2.0 will finally commence production operation in November, 2010.
However, the technological evolutionary pathway that stems from times even earlier than TSUBAME1,
its direct predecessor, was by no means paved smoothly.
In the Part one of this article, we discuss the pros and cons of TSUBAME1,
and how they have been addressed to achieve a 30-fold speedup in mere 4.5 years in TSUBAME2.0.

Introduction

1

newly developed with extreme computational density becomesvisible, augmented with mere few cables protruding from eachof its front panels. Overall performance contained in only a single
rack that looks more like a large refrigerator is 50 Teraflops, aperformance that would have had it ranked number one in the

Early October 2010—Opening the door to a room on the groundworld mere 8 years ago, comparable to the actual machine thefloor of our center, where our admin staff was kept busy handling

Earth Simulator that had occupied the entire facility of a large

paperwork, a brand new scenery from another world would

gymnasium, consisting of more than 600 racks that resembled

immediately jump into sight (Figure 1). To my ear that had beengroups of skyscrapers when viewed from its observatory. Overallaccustomed to the roaring and violent noise of TSUBAME1.0

TSUBAME2.0 performance of 2.4 petaflops is faster than thesolidly claiming its mighty presence, this time I could only hear

total of all the supercomputer performances of all such owned

the gentle sound of sound of air circulation in a new sealed

by public institutions in Japan circa 2010 — although we hadwater-cooled rack. TSUBAME2.0's soaring presence with deep

intentionally designed it so dense due to various technicalrows of racks would not initially present itself as the fastest

reasons, it is still is very amazing and exciting when one actually

supercomputer in Japan containing the world's most advanced

sees the result of such a technological feat.

supercomputing technologies, but would rather resemble rows

Before even becoming operational in November, there

of office filing cabinets. But once one opens the door of theseems to be almost never ending stream of visitors wishing for

rack, pristine rows of TSUBAME2.0 compute nodes that had beena tour of TSUBAME2.0. But just by observing the machine from

Figure 1
Newly Created
Computer RoomGSIC105 for TSUBAME2.0

02its exterior does not reveal its contents; moreover just learning

In the last volume of the Tsubame E-Science Journal, our article

the machine as is does not reveal the true nature of the design

described the characteristics of each component of TSUBAME2.0

decisions that had been made. The important question is, "why[1]. In this article, we attempt to provide the behind-the-scenes

was it designed so as a supercomputer", i.e., or what were thetechnical rationalization of the design. Part 1 of this article will

rationale of a particular component selection, how and why theyfirst cover the design of TSUBAME1, and briefly describe thatwere combined, and how the experiences and knowledge ofexperience that affected the TSUBAME2.0 design. In part 2,

TSUBAME1 had been technically reflected in the design. For users

latest technology trends from petascale towards exascale will

as well as for the future development of the supercomputers

be discussed, and how TSUBAME2.0 serves as a prototype, how

in general, these questions must be answered clearly, andsuch requirements are reflected in the design. More complete

furthermore, whether we have actually had conducted proper

introduction and evaluation of TSUBAME2.0 technologies would

design, engineering, and construction of TSUBAME2.0 given suchrequire a separate book, perhaps published after its operation

technical challenges, need to be assessed. Simple stable operation

begins. For the moment we hope that the this article will serve toas an infrastructure would not be sufficient for a machine to which

deepen users' understanding of TSUBAME2.0 as a reference to its

the taxpayers will be investing more than 3 billion yen for theeffective usage.

next 4 years as a machine in a national supercomputing center for
academia and the industry nationwide—its advance technologies
as well as their utility must be properly evaluated.
Given such a perspective, the predecessor TSUBAME1.0
had a unique standpoint as one of Japan's leading supercomputer.

Looking Back at TSUBAME 1.0

2

In particular, it was a machine that our center did not merely
buy out of a vendor product catalog; rather, similar to the

The predecessor of TSUBAME2.0, i.e., TSUBAME1.0, (Figure 2) was

Earth Simulator or the follow-ons of TSUBAME, such as the so-

inaugurated in April 2006. It was a combined fruition of a series

called T2K supercomputers, it was a procurement of a machineof various research on cluster computing at Tokyo Tech over thethat had been effectively jointly designed and developed withyears, as well experiences in fielding production supercomputers

supercomputing manufacturers. Moreover, the cycle consistingfor over 20 years, in addition to experimental productionof, (1) basic research in supercomputing and system software

operation projects such as Titech campus grid as mentioned

design, (2) experimental production of nascent hardware andearlier (April 2002 till March 2006). Based on the experiences,

software at GSIC and their assessment, (3) development andvarious design studies were conducted, resulting in a 655 node

determining the specifications of a production machine, as wellsupercomputer that embodied over 10,000 CPU cores, exhibiting

as feedback to a new cycle re-starting with (1), had proven to be

50 Teraflops in total with 21 terabytes of memory, in addition to 1.1

critical over the years. Such "waterfall" development model was

Petabytes of hard disk storage, interconnected by Infiniband. The

exactly our strategy for TSUBAME1.0: basic research of systemcompute nodes were augmented with 360 ClearSpeed Accelerator

software in clusters as well as various highly-parallel applications

cards (later expanded to 648 cards) well-suited for dense matrix

at various laboratories throughout Tokyo Institute of Technology

operations, providing additional 30 Teraflops of compute power.

constituted (1), while more than four years of operating a total of

TSUBAME in a way became a template architecture for large-scale

400 nodes in the "Titech Campus Grid Project" in experimental

cluster-based supercomputers for its follow-ons, as it possessed

production as group of remotely-managed clusters provided theexperiences for (2), and based on those experiences we were able
to determine the TSUBAME1 specifications (3). For TSUBAME2.0,
basic research (1) was often conducted not only on TSUBAME1
but on other research platforms, but as will be described later,
TSUBAME1 was retrofitted with additional equipments, andintegrated experimental operation was achieved alongsidenormal production as (2). The experiences drawn resulted in thespecifications of the successor model TSUBAME2.0 (3). Next, it
is TSUBAME2.0's turn to serve experimental roles for its future
successor, TSUBAME3.0!

Figure 2 The old Computer Room that housed TSUBAME1

03

TSUBAME 2.0 Begins

The long road from TSUBAME1.0 to 2.0 (Part One)architectural parameters had not been seen in clusters those days,

TSUBAME1.0's behavior was comprehensively recorded

and matched the best supercomputers in the world. In particular,

in detail over its 4+ years of operational experience, not only forcirca 2006, a typical cluster might have sported 2 to 4 CPU cores

large-scale applications and benchmarks that imposed heavy load

per node with 1-4 Gigabytes of memory, whereas Tsubame1on the system, but also "normal" activities involving hundreds ofutilized the "glue-less" multi-socket capability of the latest AMD

concurrent users launching over 1 million jobs per year, leading

Opteron 800 series to implement a so-called "fat node" withto the TSUBAME2.0 design. Below, we discuss the pros and16CPU cores and 32-128Gigabytes of memory each, with (80+80)

cons of TSUBAME1, that is "what went well" and "what had

gigaflops peak performance from the CPUs and the ClearSpeedproblems". What went well in design obviously carried over torespectively, interconnected by a dual-rail Infiniband fabric for

TSUBAME2.0, while the problems were the subject of R&D to be

20Gbps bandwidth. Such a configuration allowed for both stability

solved for TSUBAME2.0.

(smaller number of nodes) as well as provided extremely highease-of-use as a supercomputer, as well as offering an environment
where one's applications might not execute otherwise. The
evidence of this is that TSUBAME1 is still highly competitive and

TSUBAME1 – "The Bright Side"in very high demand, even as it nears its retirement well beyond

3

its initial designed shutdown in the Spring of 2010.
Based on these characteristics, TSUBAME1.0 [2] was the

By all means we lack the paper real-estate to cover ever y

first supercomputer in Japan to supersede the Earth Simulator

detailed aspect of TSUBAME1's strong points, but here we list theon the Top500, a former world champion, and sustained Asia's

technological points that lead to its success, and carried over toNo.1 supercomputer position for a year and a half. Moreover,

TSUBAME2.0:

the number of user base both inside and outside the Tokyo
Institute of Technology, quadrupled to 2,000, including the(A) Significant adoption of industry standard technologies such asindustry users. TSUBAME1.0 retained its popularity as "Everybody's

high-performance x86 processor and Linux-OS: Advantages

Supercomputer" being utilized by a highly diverse user base.

of PC clusters due to the adoption of x86 as well as its PC and

TSUBAME1 can thus can be objectively regarded as aserver industry "ecosystem" have been reported a number"success". Not only on for the "grand-challenge" performanceof times, such as just-in-time adoption of high-performancebenchmarks such as the Top500, compared to conventionaland cost-effective technologies, as well as stability of quality

supercomputers that had been specially designed, similar highachieved in the quantity due to mass production. In addition,

compute and storage capacity were achieved maintaining

continuity of the software and programming models fromthe properties of a large-scale supercomputer, but in a muchthe usual laboratory PCs, workstations, and small clusters all

more user-friendly environment. Not only these resulted in

the way up to supercomputing scale is a major benefit, as thehigh utilization as well as a significant increase in the number

increasing complexity of simulation software has transcendedof users, it had brought innumerable benefits to GSIC andsimulation work from being done on a single supercomputer,to the university, including being designated as one of the

but rather, the mainstream usage model is for the same

Ministry of Education's Global Center of Excellence Program

software to run seamlessly in a variety of environments"CompView"; officially becoming one of the so called coalition of

from personally small to extremely large ones. Finally, it

Major National Universities' Supercomputer Centers along withwould be desirable to offer a easy path for a typical novice

other 7 centers; officially being approved as one of the leading

PC simulation user to their eventual use of supercomputers

program members to promote industrial usage of advancedb y a l l o w i n g su c h s te p - b y - s te p a d v a n ce s i n a m o s t

research facilities, with nearly 50 well-known companies beingtransparent fashion. To achieve these goals of "Everybody's

selected to use TSUBAME1.0 for future industrial applications;Supercomputer" was the main objective of TSUBAME1, andand finally being selected to be major research collaborators

the most critical of the qualities to achieve such goals wereby major US IT companies such as Microsoft and NVIDIA, as

actually achieved through the aggressive adoption of both"Center of Technical Innovation" and "CUDA Center of Excellence"

hardware and software standards as much as possible.

respectively, demonstrating our new status as one of the leading
supercomputing centers being recognized not only within Japanbut also internationally.

04

(B) Implementation of a "fat-node" architecture with mainstream

SIMD accelerator that was just being productized by a UK

x86 CPUs: almost being an antithesis to the previous

company of the same name was chosen, due to its favorable

point, supercomputers are meaningless without being

performance in dense matrix-type operations at very low

associated with the unique value of its use. In particular,

power levels. For large dense matrix operations on using thea supercomputer would be a big attraction in solving not

BLAS library, user performance nearly doubled with a mere

just when it would be able to achieve acceleration in time-command line switch, without modifying the source code,

to-solution, but also being able to solve problems that hador adding significant power or space to the infrastructure.

been previously capacity-constrained. Thus, not only are total

However, although it also contributed significant performance

processor count and aggregated memory capacity for thegains for our Linpack it was necessary to conduct research

entire system important, but also core counts and sharedand development of a new heterogeneous algorithm tomemory capacity inside a node would also be of significancemaximize their contributions [3].

if problems were constrained at that level as they often would
be. In the past such "fat-node" design called for customized(D) Multi-rail InfiniBand-based Fat-tree network using large

and expensive supercomputer design; for TSUBAME1, we were

switches, and integration of I/O storage network.: Anotherable to exploit the first generation of the latest technology

important element of the supercomputer is the high-speed

x86 at the time that sported high-bandwidth processor-to-

network interconnections between the nodes. In particular,

processor interconnect as well as to memory in a gluelessin addition to bandwidth available to a single node (typicallyfashion, namely the AMD Opteron 800 series processors,called the injection bandwidth), latency between the nodes

which allowed up to 8 sockets or 16 CPU cores, largely

must be very low, in the order of several microseconds,

matching the numbers for dedicated supercomputer design

while the bisection bandwidth, or the bandwidth availableof the time.

to the entire sets of nodes when they conduct all-to-all

Fat node design not only benefits the users, but

communication, needs to be extremely high. By all means,also helps to reduce the number of "moving parts" in the

low space overhead, low number of cables, low cost, andsystem allowing for higher reliability. Moreover it eases thehigh reliability are required simultaneously, and many of these

redundancy and safety in the design, such as redundantelements are sometimes contradictory in nature. TSUBAME1

power supply and fans, numerous thermal as well as other

employed a set of eight large 288-port Infiniband switchessensors as well as comprehensive IPMI-based monitoring andconfigured in a two-tier fat-tree, six switches in the lower

control network intended for large-scale server farms, etc.

edge layer and two on the top layer. Also, each node had

Augmented with adoption of efficient cooling for the time

two rails of network ports. This allowed each node to achievewith clever configuration of servers versus the CRC units, as

20Gbps bandwidth with approximately 5ns end-to-end

well as well-attuned operations, it was possible to build andnetwork latency, matching the performance of supercomputers

operate a 80 teraflop supercomputer, that became the 7th

with dedicated networks. Moreover, the fat-tree configuration

fastest supercomputer in the world at the time. The TSUBAME

allowed the network to be symmetrical, making the operation

technology gave birth to many follow-ons with their unique,to be very flexible. For example, if one of the nodes would fail,further improvements, such as the "Ranger" cluster at the

it would have been possible to use an alternate node without

University of Texas TACC Supercomputer Center , the "Europa"

affecting the performances of other nodes.

		cluster at the Julich supercomputer center in Germany, andthe T2K supercomputers that were facilitated at centers in(E) High-performance storage achieving high-performance, high

Universities of Tokyo, Kyoto, and Tsukuba respectively in 2008.

density, and low cost, with a parallel file system: the oftforgotten part of supercomputers is storage; for TSUBAME1-(C) Accelerators for dense matrix computation: Based on thelevel simulation and processing capabilities in the 100

technology at the time of TSUBAME1, it was already difficult

Teraflops, it was judged from early on that sub-petabyte classto achieve 100 Teraflops given the constraints of space,data handling capabilities will be of absolute necessity. As

power, costs, etc. As a result, we had decided to experiment

was with compute nodes, storage subsystem required high-acceleration technologies that had demonstrated reasonable

performance, low cost and low power consumption, highresults within the research lab of our center. Considering

reliability and scalability that coincided the parallelism in theseveral candidates for actual production usage, ClearSpeedsystem. It became clear that traditional enterprise IT systems

05

TSUBAME 2.0 Begins

The long road from TSUBAME1.0 to 2.0 (Part One)technology-based storage systems were inadequate. Rather,a cluster-based approach was also taken, where a powerful

The Dark Side of TSUBAME1"fat" storage server that embedded 48 HDDs (24 Terabytes

and their Reflections in TSUBAME 2.0

raw capacity) along with a powerful storage and network

4

controller in a mere 4-U chassis, namely the Sun x4500"Thumper" served as the basic building block, and 42 of themSome of the problems identified in TSUBAME1 in its performance(later 20 more were added) were clustered and also connected

and operations were more fundamental, and found not todirectly to the main Infiniband network of the compute nodes

be solvable with only super ficial improvements describedallowing gigabyte level data transfer capability per eachabove. Moreover, supercomputing is a field where the averagenode. On top of this we implemented the Lustre parallel fileperformance of a machine increases by approximately 180% eachsystem so that the accesses can be done in parallel and very

year, well beyond the Moore's law. As such it was obvious thatlarge files could be handled. This allowed us to achieve overthere would be a need for continuous research and development

10 gigabytes per second I / O performance, in 1.0 (later 1.5)of various technologies merely to sustain such a growth level.

petabyte storage of amazingly compact dimensions.

Although some of the technological choices were done to thebest of our knowledge circa 2004-5 when TSUBAME1 was being(F) A Batch scheduler that was fair and easy to understand,designed, it became apparent not all choices we made were theassuming simultaneous use by hundreds of both Capacity

best leading to TSUBAME2.0's operations in 2010. Below we give

and Capability users: TSUBAME's user base of 2000 typicallysuch shortcomings, and how they were analyzed and attempts

had 100 or more users being logged into the system andwere made to solve the problem for TSUBAME2.0:

running jobs at the same time. The annual number ofjobs exceed a million in count, and varied greatly in their(A) Drastic improvement of power/performance: TSUBAME1.0

usage patterns — number, memory size, time (makespan),consumed at the peak of about 1MW electrical power, orparallelism, I / O performance, required QoS, I / Operformance,

10% or more of the entire power usage of the Tokyo Instituteetc. Moreover, there was a need to support both the expertsof Technology's Oo-okayama main campus, costing about

with highly-parallel, "capability" jobs, versus novice users

100 million yens per year. Since TSUBAME2.0 was initiallysometimes with parameter survey "capacity" jobs. Despiteplanned to be deployed in the early part of 2010, its target

the large computational capacity of Tsubame1, it possessed

performance would be (1.8)4 , or about 10 times speedup at 1

nowhere near the amount of resources to satisf y all of

petaflop while maintaining the power usage. However, recall

these requirements. So, it was essential to adopt a easy-to-that the power/performance ratio for TSUBAME1.0 was alreadyunderstand and seemingly-fair scheduling policy that would

very efficient for the supercomputer of the time, thanks tosatisfy the above goals, such as co-existence and QoS control

the ClearSpeed card for dense linear algebra operations.

of pay-per-use versus flat-rate usage models, pay-for-priority-

The University of Tokyo's T2K supercomputer that came out

QoS, as well as later introduction of job reservation for very

two years later but without using any type of accelerators,

large jobs, etc. All such requirements were incorporated

consumed about the same power as TSUBAME2.0, while inas customization modules of the Sun GridEngine batch

Linpack being only about 20% faster, instead of being overscheduler, and was continuously updated and improved due

3 times faster as the 180% growth should indicate. Thus, In

to user feedback.

order for TSUBAME1 to maintain international competitiveness,it became necessary to improve the power performance

06

Besides these issues, there were a variety of technical andratio significantly. Fortunately, the our project sponsored byoperational ideas we had incorporated into the system. In fact

the JST-CREST program, called "ULP-HPC: High Performance

every year we reviewed the hardware, software, and operational

Ultra Low-Power" was approved, with the aggressive goal of

aspects of the system and when necessary conducted additional

improving the power efficiency of supercomputers by x1000

procurements to augment the system. These were not merely

instead of x100 as is with the Moore's law, in 10 years. Such

constrained to simple issues such as additional software licenses,

boost in basic research accelerated our understanding of low

but some were directly relevant to the experimental operation we

power design of supercomputers, and some of the research

had mentioned as item (2) earlier, such as the addition of GPUs.

results were applied to TSUBAME2.0.

(B) Widening the applicability and use of Accelerators (GPUs): On

we were able to augment TSUBAME1 with 680 cards of therelated terms, we found that acceleration using GPUs werelatest NVIDIA Tesla GPU (Figures 3) in October 2008. The

the key to low power and high performance, however, early

resulting supercomputer, TSUBAME1.2 formed a platform

accelerators lacked the applicability to the wide range of

which allowed a variety of experiments in our preparations

existing applications. Some were resulting from the lack offor TSUBAME2.0 which was to become extremely GPU centricalgorithms and software, but some were from the hardware(Figure 4) due to many positive results we were obtaining andarchitecture itself. In fact, early accelerators were quite limited

the problems we encountered and solved in its operations.

in their applicability due to their hardware limitations; forexample, ClearSpeed excelled in dense matrix type problems,(C) Lack of memory and network bandwidth in the node: Not

but found to be poor at bandwidth-hungry problems. These

only the local memory bandwidth that the GPU resolved,

were principally due to the lack of memory capacity and aanother problem was the overall bandwidth of the systemshortage of available hardware memory bandwidth as wellbeing somewhat deficient in TSUBAME1 then we initiallyas programming difficulties; and as a result narrowed its

anticipated. CPU memory bandwidth was affected by two

applicability substantially. Fortunately, GPUs sporting both

negative factors, the decline of the memory bus clock withhigh computational density and high memory bandwidth,a larger number of multi-channel memory; and the inherent

and being low cost, were becoming rapidly general purpose

limit for the Socket 940 and Socket F generations of AMD

in terms of both its hardware and software, and in the labs

CPU design, where the total memory bandwidth of the

we had already demonstrated its advantages and flexibilities

shared memory was limited to be approximately 20GB/s due

in many real-world HPC situations. Therefore, the question

to memory coherency traffic. As for the network, Infiniband

was how much of GPU technology would be applicable

theoretical channel performance of 1GB/s was seeing strong

to TSUBAME2.0; fortunately, we had initiated technicaldecrease to be half or less due to overlap with other activities

partnerships and discussions in 2007 with two key companies,

within the node. As will be discussed in the part 2 of thisnamely NVIDIA and Microsoft, and we were able to conduct

article, bandwidth is necessary not only to significantly

projects which resulted initially in a 128GPU prototype cluster

improve acceleration of the current applications, but willin late 2007, and based on the results from that machine,

also be an important factor for ensuring future scalabilityof the system. Thus, in TSBAME2.0's design, increase of
bandwidth exceeding the increase in compute FLOPS was
absolutely necessary, and deemed to be the most important
technological improvement; this is one of the major reasons
why an entirely new node design was mandated, instead ofusing existing products.

Figure 3 NVIDIA Tesla s1070 GPU retrofitted to TSUBAME1(D) Lack of network bisection bandwidth: In addition, bisection
bandwidth in TSUBAME1 was lacking; whereas the aggregate
injection bandwidth at the endpoint was endpoint was 13
terabits/second, due to the constrained fat-tree bisection
bandwidth was approximately 2.8 terabits/second, or about 5
to 1 oversubscription ratio. This has resulted in the weakness(e.g., in comparison to the Earth Simulator) in supporting
various global-style algorithms such as global FFT, in which
the communication bottleneck would become the dominant
per formance inhibitor. Therefore, it was imperative toachieve high bandwidth across the network. A new network
design was needed to achieve full bisection in the samemanner as the Earth Simulator, while sporting more thanFigure 4 NVIDIA Tesla M2050 GPU used in TSUBAME2.0,
3 cards per node, over 4000 cards in total.

more than twice the number of nodes. As a result of various
design studies, combined use of a smaller leaf switches

07

TSUBAME 2.0 Begins

The long road from TSUBAME1.0 to 2.0 (Part One)with very large centralized core switch fabric that were

(E) Cooling efficiency issues: TSUBAME1.0 employed advanced

physically clustered, with optical fiber connection in-between

cooling strategy of the time after over a year of study andcomprising a full fat-tree configuration, was deemed to begiven the physical confines of relative small space we had

the smallest, most cost effective, and the most reliable (Figurefor our computer room, such as achieving the hot-row vs.

5). One requirement however was to make the system as

cold-row separation using specially placed CRC units as well as

small as possible to shorten the fiber cable length.

ducts in the ceiling instead of floor cooling etc. The achieved
PUE (Power Usage Effectiveness) was approximately 1.44, which
was a reasonably good measure at the time. However, it was afact that the cooling power consuming 44% beyond themachine power was not ideal indeed, and moreover, we knewthat some of the recent advances will allow us to achieve much
lower PUE. As a result, hybrid water- and air-cooled technology,
where the individual rack would be cooled by water, and thecirculatory air being sealed and contained to cool the nodes was
employed, with an aim to achieve PUE as good as 1.2.
(F) Machine size and weight problems: TSUBAME1 nearlyoccupied 80 racks to their entirety, occupying most of thecomputer room floorspace of GSIC's facilities building. This

Figure 5 Fiber optic cables are aggregatedinto the large Infiniband core switch fabricfrom the the lower-tier edge switches

seriously compromised our ability to scale further, as wellas complicating the wiring and control logistics spanning
two floors. These and for other reasons we have mentioned
constructing a machine that was dense with very small
resulting footprint was becoming an impor tant issue.
Making a machine smaller not only reflects in cost, butwould improve our ability to construct full bisection and fat
networks, as the machines need to be located to the central
switch fabric as close as possible. Fortunately, in order toaccommodate multiple GPU for high bandwidth fat-nodeconfiguration necessitated an entirely new node design, andmajor technological goal of node implementations was set
to achieve unprecedented levels of density (Figure 6). The
result achieved is 50 Teraflops per rack performance density,i.e. each rack being approximately the compute performanceof the entire Earth Simulator, while the entire TSUBAME2.0
occupying about 60 racks, or about 3/4th of TSUBAME1
despite the 30-fold increase in computational performance.
(G) Lack of operational storage capacity: The storage capacity
TSUBAME1 was significantly improved over its predecessor,
but still the actual production capacity was always lacking.
This depended on various factors: tertiary storage systems oftapes or MAIDs were lacking, and moreover, all the storage
was uniformly high-performance and somewhat expensive

Figure 6 Individual compute nodes in TSUBAME2.0 are
1/4th the size of TSUBAME1 nodes, despite being
10 times more computationally powerful.

08

as a result, despite some being purely used for backups.
Secondly, there were various single points of failures in thestorage, including the Lustre parallel file system, necessitating

duplication in various parts of the storage system. However,this resulted in an entire Thumper box used up for metadatamanagement, being expensive in terms of required capacity
and sacrifices in performance. Thumpers were consumed
for other service requirements; moreover, we had to deploy
RAID6 with relatively small number of stripes, fur ther
constraining space. As a result of all of these requirements,
the capacity of high performance Lustre storage area shrunk
down to be miniscule, less than 100 Terabytes out of theAoriginal 1 Petabytes. In 2007 we alleviated the problem
somewhat by adding 20 more Thumper units, and the total raw
physical capacity was raised to 1.6 petabytes; still the total
Lustre capacity including the scratch space for Gaussian was
only about 200 terabytes in total. Therefore, TSUBAME2.0
storage was designed to retain as much space as possible
for operational Lustre and GPFS storages by extensive
streamlining of the storage management, and improving
the reliability through redundancy and elimination of single

B

points of failures without sacrificing storage capacity. In
particular, we had started improving TSUBAME1's storage

A Figure 7

Various storage servers in TSUBAME 2.0 for Lustre,
GPFS, NFS, CIFS, etc.

B Figure 8

Each disk enclosure contains 60 units of
2 terabyte SATA HDDs.

C Figure 9

TSUBAME2's SL8500 tape drive system,
embodying over 10,000 tape slots and locatedin a separate building.

fabric in 2009 in preparation, incorporating dedicated storage
management servers, as well as introducing a large capacity
tape system that had the ability to be extended to beyond
10 petabytes, to better match the planned operational disk
capacity of 7 petabytes for TSUBAME2.0. (Figures 7, 8, 9)(H) Lack of storage bandwidth: In order to achieve the initiallydetermined per formance goal of 2-3 petaf lops, it was

D Figure 10 TSUBAME 2.0 embeds two or more SSD driveson each node 60GB, or 120GB dependingon the node memory capacity.

deemed that we would require I/O speeds of several hundred
gigabytes per second. However, implementing such I/O
performance would require a huge number of spinning disks
and storage servers. Fortunately, studies indicates that major
portions of I/O writes are stream writes for checkpoints, or
local scratch writes for simple scratch files or more serious
data structures for out-of-core algorithms [4][5]. Assumingthat approximately 80-90% of I/O workload would be ofsuch nature, the achieved speed of hundreds of MB/s bya new breed of SSDs with their improved capacity, costC

performances, and reliability seemed perfect for the task torelieve the parallel file systems of such loads and effectively
attaining x5-x10 effective improvement in I/O performance.
In practice, the aggregate I/O performances of two or more
SSDs equipped in every node of TSUBAME2.0 (Figure 10)
totals more than 660 gigabytes / second; work is underway
to achieve reliabilit y of local writes while maintaining
performance using redundant coding techniques.
D

09

TSUBAME 2.0 Begins

The long road from TSUBAME1.0 to 2.0 (Part One)

( I ) Further improvements in reliability: TSUBAME1 was already

References

designed with high reliability in mind. In fact, the entire
history of failures and repairs log was made constantly[1] Satoshi Matsuoka, Toshio Endo, Naoya Maruyama, Hitoshi Sato,

accessible via a web page of the GSIC center, and the only

Shin'Ichiro Takizawa. "The Total Picture of TSUBAME2.0", The

time the system was entirely down was twice in 4.5 years,

TSUBAME E-Science Journal, Tokyo Tech. GSIC, Vol. 1, pp. 16-18,

once when a major power outage occurred around a small

Sep. 2010

area South of Tokyo where Tokyo Tech, just happened to[2] Satoshi Matsuoka. Petascale Computing Algorithms andbe affected for two hours. However localized faults that

Applications --- Chapter 14 The Road to TSUBAME and Beyond,affect sizeable portions of the system did occur occasionally,

Chapman & Hall CRC Computational Science Series, pp.289-

including those that would affect one of the batch queues

310, 2009.

entirely. Significant lessons had been learned for TSUBAME2.0[3] Toshio Endo and Satoshi Matsuoka. "Massive Supercomputing

design as a result, eliminating wherever possible single

Coping with Heterogeneity of Modern Accelerators", In Proc.

points of failures in batch queues, storage/parallel file system,

22nd IEEE International Parallel & Distributed Processing

various service nodes, etc. Despite the costs incurred by

Symposium (IPDPS 2008), The IEEE Press, Miami, FL, April 2008,

such redundancies, it was deemed that the user's lost time

pp.1-10, DOI: 10.1109/IPDPS.2008.4536251

due to failures outweighed the costs, especially as the faults[4] Weikuan Yu, Jeffrey S. Vetter, H. Sarp Oral. "Performancethemselves would become bigger hindrances as we move on

Characterization and Optimization of Parallel I/O on the Cray

to future machines that will be more error prone.

XT", In Proc. 22nd IEEE International Parallel & Distributed
Processing Symposium (IPDPS 2008), The IEEE Press, Miami, FL,

TSUBAME 2.0's will now Begin— Towards Part 2

5April 2008, pp.1-10, DOI: 10.1109/IPDPS.2008.4536277[5] Julian Borrill, Leonid Oliker, John Shalf, Hongzhang Shan,
Andrew Uselton. "HPC Global File System Performance Analysis
Using A Scientic-Application Derived Benchmark", Parallel
Computing, Volume 35 , Issue 6 (June 2009), pp. 358-373.

We have outlined the evolutionary path from TSUBAME1 to
TSUBAME2.0. As of this writing TSUBAME1.0 continues to be
shrunk down due to the lack of power in the building, while
TSUBAME2.0 benchmarking and burn-in continues for services
to begin on Nov. 1, 2010, whence TSUBAME1.0 will be totally shut
down on Oct. 25th, 2010. Since much of the software stack will
be carried over from TSUBAME1, users should feel quite at homewith TSUBAME2.0, just bigger, better, and somewhat faster when
one would run non-GPU programs. When a user would port his
application to GPUs and exploit other performance capabilities
such as the intra-node SSDs, the applications would fully beable to utilize the 30-fold speedup it had achieved, perhaps tocompute in petaflops, along with other new features TSUBAME2.0
newly offers compared to TSUBAME1, some of which had beenoutlined in this article. Users should then should realize we would
be in the dawn of petaflops, moving towards exaflops in the
coming years.
Part 2 of this article will evaluate TSUBAME2.0 from theviewpoint of scaling towards exaflops, plus benchmarks toobserve the trends thereof. Please stay tuned.

10Multi-GPU Computing for
Next-generation Weather Forecasting- 145.0 TFlops with 3990 GPUs on TSUBAME 2.0 Takashi Shimokawabe*

Takayuki Aoki*** Interdisciplinary Graduate School of Science and Engineering, Tokyo Institute of Technology**Global Scientific Information and Computing Center, Tokyo Institute of Technology

In order to drastically shorten the runtime of the weather-prediction code ASUCA ,developed by the JMA (Japan Meteorological Agency) for the purpose of the next-generation weather forecasting service,
the entire parts of the huge code were rewritten for GPU computing from scratch.
By introducing many optimization techniques and several new algorithms, very high performance of
145 TFlops has been achieved with 3990 GPUs on TSUBAME 2.0 Supercomputer. It is quite meaningful to show that
the GPU supercomputing is really available for one of the major applications in the HPC field.

Introduction

1

NCAR (National Center for Atmospheric Research), UCAR (University
Corporation for Atmospheric Research) and others in the UnitedStates and supported by worldwide researchers.
Numerical weather models consist of a dynamical core andphysical processes. In the dynamical core, forecast variables such as

Weather forecasting is an indispensable part in our daily lives andwinds, atmospheric pressure and humidity are calculated by solving

business activities, needless to say for natural disaster preventions.

fluid dynamics equations. The physical processes strongly depend

The atmosphere has a very thin thickness compared with the Earthon parametrizations related to such microphysics as condensation

diameter. In the previous atmosphere code, the force balanceof water vapor, cloud physics, and rain. In the computation of the

between the gravity and the pressure gradient in the vertical

dynamics core, the memory access time is a major part of thedirection was used to produce a hydrostatic model. Recently it iselapsed time compared with the floating point calculations partwidely recognized that the vertical dynamical processes of theand therefore is hard to get close to the peak performance in any

water vapor should be taken into consideration in cloud formations.

computer. On the other hand, the physical processes include someA three-dimensional non-hydrostatic model describing up-and-computationally intensive parts demanding high performance ofdown movement of air has been developed in weather research.

floating point calculations.

For weather simulations, the initial data is produced byGraphics Processing Units (GPUs) have been developed

assimilating many kinds of observed data and simulation results

for the rendering purpose of computer graphics. The request forbased on the four-dimensional variational principle. Since thehigh-level computer visualization makes the GPUs to have highweather phenomena are chaotic, the predictability period is lessperformance of floating point calculation and wide memory

than several days for one set of initial data hence the jobs run

bandwidth. Recently, exploiting GPUs for general-purpose

sequentially updating the initial data.

computing, i.e. GPGPU, has emerged as an effective technique

In recent years, it is highly demanded to forecast detailed

to accelerate many applications. After CUDA [3] was released

weathers such as unexpected local heavy rain, and high resolution

by NVIDIA as the GPGPU-programming framework in 2006, it

non-hydrostatic models are desired to be carried on fine-grained

allowed us to use the GPU easily as an accelerator. In the area ofgrids.

high performance computing (HPC), it has been reported a lot of
successful applications in Computational Fluid Dynamics (CFD),

GPU computing forweather simulations

2

Fast Fourier Transform (FFT), molecular dynamics, astrophysics, bioinformatics and some others ran on GPUs dozens of times fasterthan on a conventional CPU.
In the numerical weather prediction, it was repor ted
that a computationally expensive module of the WRF model

A computational heavy load is required to run the high-resolution

was accelerated by means of a GPU[4, 5]. These efforts, however,weather models. As a reference the WRF [1] has been scored atonly result in a minor improvement (e.g., 1.3× in [4]) in the overall

50 TFlops on the current fastest supercomputer in the world [2].

application time due to the partial GPU porting of the entire code.

WRF is a next-generation atmosphere simulation model (Weather

Since the other functions (subroutines) run on the CPU and all the

Research and Forecasting), a world standard code developed at thevariables were allocated on the CPU main memory, it is necessary

11Multi-GPU Computing for
Next-generation Weather Forecasting- 145.0 TFlops with 3990 GPUs on TSUBAME 2.0 -to transfer the data between the host CPU memory and the GPU

cloud-microphysics parameterization describing the water vapor,

video memory through the PCI Express bus for every GPU kernel-

cloud water, and rain drops.

function call. They reported that the acceleration for the microphysics
module itself achieved a twenty times speedup. Physical processes
are composed of small and relatively independent modules and areSingle GPU Implementation

able to be easily replaced with other modules while a dynamicaland Performancecore computes time-integration of interdependent forecast

4variables thus GPU porting of parts of the dynamical core does not
Although our final destination is to develop the multi-GPU

contribute to improvement of the performance.
A s t h e s u c c e s s o r o f T S U B A M E 1. 2 , T S U B A M E 2 . 0

version of ASUCA, we start from the single GPU implementation

supercomputer equipped with more than 4000 GPUs has started

and show its performance. Figure 1 illustrates the computational

operating in November 2010 and has become the first petascale

flow diagram. In the beginning of the execution, the host CPU

supercomputer in Japan. Since TSUBAME 2.0 owes most of its

reads the initial data from the input files onto the host memory,computing performance to GPUs, it is a key issue to achieve highand then transfers it to the video memory on the GPU board.

performance on GPU in many applications. In this article, we show

The GPU carries out all the computational modules inside thethe process of porting an operational weather prediction code toshort time-step and long time-step loops. When the forecast

the GPU on TSUBAME 2.0 and demonstrate the performance for adata is completed on the GPU, a minimal data is transferred topractical operation size.

the host CPU memory to reduce the communication between
CPU and GPU.

Next-generation
weather prediction code ASUCA

3

4-1 Optimizations
In order to improve the per formance on the GPU, we haveintroduced several optimizations when implementing the code in
CUDA. We focus on two components as examples: (a) the advection

ASUCA (Asuca is a System based on a Unified Concept forcomputation (b) the 1-dimensional Helmholtz-type equation for

Atmosphere) is a next-generation high resolution mesoscale

the pressure.

atmospheric model being developed by the Japan Meteorological
Agency (JMA)[6]. ASUCA succeeds the Japan Meteorological
Agency Non-Hydrostatic Model (JMA-NHM) as an operational nonhydrostatic regional model at the JMA.
First, we have implemented the dynamical core as thefirst step toward developing the full GPU version of ASUCA. In
ASUCA, a generalized coordinate and flux-form non-hydrostaticbalanced equations are used for the dynamical core. The time
integration is carried out by a fractional step method with thehorizontally explicit and vertically implicit (HE-VI) scheme [7]. One
time step consists of short time sub-steps and a long time step.
The horizontal propagation of sound waves and the gravity waves
with implicit treatment for the vertical propagation are computedin the short time step with the second-order Runge-Kutta scheme.
The long time step is used for the advection of the momentum,
the density, the potential temperature, the water substances,
the Coriolis force, the diffusion and other effects in the physicalprocesses with the third-order Runge-Kutta method. The above
matters are almost as same as those employed in the WRF model.
In the present ASUCA, the physical core is still being developed
and a Kessler-type warm-rain model has been implemented for the

12

Figure 1 Computational flow diagram of the ASUCA.
All the modules inside the short time-stepand the long time-step loops are executed on the GPU.

(a) Implementation of the advection computational modules
The 3-dimensional advection computation is strongly
memory bound and it is very effective to reduce the access tothe video memory (called global memory in CUDA) in order toimprove the performance. We make use of the shared memory asa software-managed cache, which is shared among threads in a
block in the CUDA programming. For a given grid size (nx, ny, nz)
of the computational domain, the GPU kernel function is invokedwith (nx/64, nz/4, 1) blocks with (64, 4, 1) threads. The z directionin physical space is mapped to the y direction in the CUDA code.
Each thread specifies a point (x, z) and calculate the advection
equation on this grid point from j = 0 to j = ny − 1 marching in they direction as shown in Fig. 2(a). The block size of (64, 4, 1) threads is(a) Advection computation(b) Helmholtz-type
pressure equation

Figure 2 Configuration of the CUDA blocks
and marching directions of the threads.

derived from performance optimization.
The discretization of the advection equation has a fourpoint stencil in each direction on the mesh. Each block holds anarray of (64+3) x (4+3) elements on the shared memory. Whena block computes a xy plane of the computational domain, thevariable data on the global memory is copied to the sharedmemory to be shared among threads in the block. On the other
hand, the stencil access in the y-direction is closed in the thread bymarching the computation in the y-direction. The variable data of
the global memory are stored in the registers (temporal variables).
When we compute the j+1-th plane, the data have been alreadystored in the registers from the global memory in computing thej-th plane. In our implementation, we copy the data in the registers
to the shared memory and reuse them without same accesses to
the global memory [8].
(b) Implementation of the Helmholtz-type pressure equation

Figure 3 Usage of the shared memory and the registerto reduce the access to the global memory.

Due to the HE-VI splitting, the pressure equation reduces toa 1-dimensional Helmholtz-type elliptic equation in the vertical (z)direction. The discretization of the equation is expressed by a tridiagonal matrix. It is possible to apply the TDMA algorithm to solve
the matrix however we have to calculate the elements sequentiallyin the z-direction. Figure 2(b) shows the data parallel by marching
the sequential calculation in the z-direction.
4-2 Performance
Since ASUCA is being developed in FORTRAN language at
the JMA, the GPU code has to be developed from scratch in CUDA.
Before implementing ASUCA on GPU, we rewrote the Fortran
ASUCA code to C/C++ language because we changed the element
order of the 3-dimensional array to improve the memory accessperformance of the GPU computing. In order to measure theperformance of floating-point operations on a GPU, we count the
number of floating-point operations of the CPU-based ASUCA byrunning it on a CPU with a performance counter provided by PAPI

Figure 4 Single GPU performance on a NVIDIA
Tesla M2050 of TSUBAME 2.0 and CPU
performance of the Intel Xeon X5670.

13

Multi-GPU Computing for
Next-generation Weather Forecasting- 145.0 TFlops with 3990 GPUs on TSUBAME 2.0 -(Performance API) [9]. By using the counts and the GPU elapsed

5-1 Performance of Multi-GPU Computing

time, the performance of the GPU computing is evaluated.
In all the cases, we fixed the grid number nx = 256 andEach node of TSUBAME 2.0 has three NVIDIA GPU Tesla M2050

nz = 48 of the computational domain and varied the number ny

attached to the PCI Express bus 2.0x16, two QDR Infiniband and

32 to 208. The performance was measured in both single- andtwo sockets with Intel CPU Xeon X5670 (Westmere-EP) 2.93 GHz

double- precision floating-point calculation using a NVIDIA Tesla

6-core. The nodes are connected to the fat-tree interconnection

M2050 in TSUBAME 2.0. The results of the GPU performance arewith 200 Tbps bi-section bandwidth. Each GPU handles a domainshown in Figure 4. We achieved 49.1 GFlops in single precision forof 256 x 108 x 48 in double precision and 256 x 208 x 48 in single

256 × 208 × 48 mesh on a single GPU. In the double precision,

precision, respectively.

the performance has about half of that of the single precision.

The multi-GPU performance of ASUCA on TSUBAME 2.0 is

As references, the performances on an Intel CPU (Xeon X5670

shown in Figure 5. Using 3990 GPUs we achieved an extremely high(Westmere-EP) 2.93 GHz 6 core x2: total 12 cores and 1 core) are

performance of 145.0 TFlops for the domain of 14368 x 14284 x 48

plotted on the same graph. The original FORTRAN code wasin single precision. The double precision performance is 76.1 TFlops

compiled by the Intel ifort compiler. It is found that the single-GPU

for the domain of 10336 x 9988 x 48. It is confirmed to maintain aperformance achieved a six times speedups in comparison with 2

good weak scalability. To compare with the CPU performance, thesockets of the CPU performance of the Intel Xeon X5670 in double

performance of 3990 is compatible with 3990x50 CPU cores.

precision.

Figure 6 demonstrates a real case of ASUCA operation with both
the real initial and the boundary data used for the current weather

Multi-GPU Computing

5

forecast at the JMA. This simulation was performed with a 4792 ×
4696 × 48 mesh with horizontal mesh resolution of 500 meters
using 437 GPUs of TSUBAME 2.0 in single precision.

A GPU Tesla M2050 card on TSUBAME 2.0 has only 3-GB on-board
video memory, which can hold up to a grid size 256 × 208 × 48
when running ASUCA. For large-size problems, it is necessary to
use multiple GPUs beyond the video memory on a single GPU. The
current operation for the weather forecast at the JMA utilizes thegrid of size 721 x 577 x 50.
We decompose the given computational domain in boththe x- and y-directions (2-D decomposition) and allocate the subdomain to each GPU since the z-directional mesh size is relatively
small. Because GPUs cannot directly access the data stored on the
global memory of other GPUs, the host CPUs are used to bridgeGPUs for the exchange of the boundary data between the neighborGPUs. The process is composed of the following three steps: (1) thedata transfer from GPU to CPU using the CUDA runtime library, (2)the data exchange between nodes with the MPI library, and (3) thedata transfer back from CPU to GPU with the CUDA runtime library.
In the case of multi-GPU computing, the data communication timewith the neighbor GPUs is not ignored in the total execution time.
The overlapping technique with the computation is available tohide the communication costs and achieves better performancewith a large number of GPUs [10].

14

Figure 5 Multi-GPU performance of ASUCA
on TSUBAME 2.0 comparing with the CPU.

Figure 6 ASUCA real operation to describe a typhoonwith 4792 × 4696 × 48 mesh using 437 GPUs of TSUBAME 2.0

Summary

Summary

6
6This research was supported in part by the Global Centerof Excellence Program “Computationism as a Foundation for the
Sciences” and KAKENHI, Grant-in-Aid for Scientific Research (B)19360043 from the Ministry of Education, Culture, Sports, Scienceand Technology (MEXT) of Japan, and in part by the Japan Science

The full GPU implementation of the next-generation production

and Technology Agency (JST ) Core Research of Evolutional

weather code ASUCA was carried out on the TSUBAME 2.0

Science and Technology (CREST) research program “ULP-HPC:

supercomputer in the Tokyo Institute of Technology. GPU offers

Ultra Low-Power, High-Performance Computing via Modeling andnot only extremely huge computational performance but also

Optimization of Next Generation HPC Technologies”.

big advantages in respect of the cost and power consumption. It
is really meaningful that the numerical weather prediction, one

Referencesof the typical applications for a practical purpose, is provided as[1] W. C. Skamarock, J. B. Klemp, J. Dudhia, D. O. Gill, D. M. Barker, M.

a successful example of the GPU supercomputing. In China and

G. Duda, X. Y. Huang, W. Wang, and J. G. Powers, “A Description

other countries, large-scale GPU computing on supercomputersof the Advanced Research WRF Version 3,” National Center foris about to begin. TSUBAME 2.0 in the Tokyo Tech Global ScientificAtmospheric Research, (2008)

Information and Computing Center (GSIC) is the first over-petaflops[2] A. S. Bland, R. A. Kendall, D. B. Kothe, J. H. Rogers, and G. M.

supercomputer in Japan. In the GPU supercomputing era, it is

Shipman, “Jaguar: The world’s most powerful computer,” in

expected that research communities become large and active to2009 CUG Meeting, pp. 1‒7. (2009)

achieve great outcomes on GPU supercomputers.
Acknowledgements
We would like to thank Dr. Chiashi Muroi, Dr. Junichi Ishida and Dr.

[3] “CUDA

Programming Guide 3. 2,” ht tp: //d evel o p er.

download.nvidia.com/compute/cuda/ 3_2/toolkit/docs/
CUDA_C_Programming_Guide.pdf, NVIDIA, (2010)[4] J. Michalakes and M. Vachharajani, “GPU acceleration of

Kohei Kawano at the Japan Meteorological Agency for providing

numerical weather prediction.” in IPDPS. IEEE, pp. 1‒7, (2008)

the original ASUCA code and helping us develop the GPU version.

[5] J. C. Linford, J. Michalakes, M. Vachharajani, and A. Sandu,

We are grateful to Prof. Satoshi Matsuoka, Prof. Toshio Endo, Dr.

“Multi- core acceleration of chemical kinetics for simulation andAkira Nukada and Dr. Naoya Maruyama at the Tokyo Institute ofprediction,” in SC ’09: Proceedings of the Conference on HighTechnology for helping us to use TSUBAME 2.0.

Performance Computing Networking, Storage and Analysis.

15

Multi-GPU Computing for
Next-generation Weather Forecasting- 145.0 TFlops with 3990 GPUs on TSUBAME 2.0 -New York, NY, USA: ACM, pp. 1‒11, (2009)[6] J. Ishida, C. Muroi, K. Kawano, and Y. Kitamura, “Development of
a new nonhydrostatic model “ASUCA” at JMA,” CAS/JSC WGNE
Reserch Activities in Atomospheric and Oceanic Modelling,(2010)[7] W. C. Skamarock and J. B. Klemp, “Efficiency and Accuracy of
the Klemp-Wilhelmson Time-Splitting Technique,” Monthly
Weather Review, vol. 122, pp. 2623‒+, (1994)[8] P. Micikevicius, “3D finite difference computation on GPUs using
CUDA,” in GPGPU-2: Proceedings of 2nd Workshop on General
Purpose Processing on Graphics Processing Units. New York,NY, USA: ACM, pp. 79‒84, (2009)[9] S. Browne, J. Dongarra, N. Garner, G. Ho, and P. Mucci, “A
portable programming interface for performance evaluation
on modern processors,” Int. J. High Perform. Comput. Appl., vol.
14, no. 3, pp. 189‒204, (2000)[10] T. Shimokawabe, T. Aoki, C. Muroi, J. Ishida, K. Kawano, T. Endo, A.
Nukada, N. Maruyama, and S. Matsuoka, "An 80-Fold Speedup,
15.0 TFlops Full GPU Acceleration of Non-Hydrostatic Weather
Model ASUCA Production Code", in SC ’10: Proceedings of theConference on High Performance Computing Networking,Storage and Analysis. New York, NY, USA: ACM (2010) in press

16Computer prediction of protein-protein
interaction network using MEGADOCK- application to systems biology Yuri Matsuzaki*
Takashi Ishida*

Masahito Ohue*
Yutaka Akiyama*

Nobuyuki Uchikoga** Graduate School of Information Science and Engineering, Tokyo Institute of Technology

We have developed a high throughput and ultra-fast protein-protein interaction (PPI) prediction system“MEGADOCK” on TSUBAME. MEGADOCK showed significant speed-up in the rigid-body docking processthat leads a way of full utilization of protein tertiary structure data for large-scale and network-level problems
in systems biology. We then have explored biological PPI networks with the method of bioinformatics.
This is a grand challenge to solve the mega-order problems in medicinal science and drug discovery.

Introduction

1

Docking calculation

2

Living mat ters are maintaine d and develop e d by various

MEGADOCK includes a rigid-body docking system that searches

molecular interactions. Elucidation of the regulatory relations

relevant interacting protein pairs among the target protein

among the thousands of protein species working in a human cell

tertiary structures. The rigid-body docking process is mainly basedis crucial for understanding the mechanisms underlie diseaseson the shape complementarity of the target proteins without

and for development of drugs. We are working on the problem

considering conformational changes under the protein complex

of predicting Protein-Protein Interaction (PPI) network (Figure

formation event.

1) which is one of the main topics in systems biology, by using
bioinformatics methods.

Docking calculation part, the core of MEGADOCK system,
calculates docking scores by an equation consisting of the shape

Conventionally, computational methods have been usedcomplementarity term G and the electrostatic interaction term E.

mainly to analyze the mechanism of individual known protein

The target protein pair R (receptor) and L (lignad) are first allocatedinteractions in detail. Those methods are not applicable to aon the 3-D voxel space, 1.2 Å × 1.2 Å × 1.2 Å. Then the scores

large-scale analysis such as in systems biology field. In this work,

are assigned to each voxel according to the location in a protein,

we developed a method that can be applied to PPI prediction

such as surface and core. We use our original scoring function

problems of mega-order data.

called “real Pairwise Shape Complementarity” (rPSC) for the shape

We prop ose d a novel score f unc tion to re ducecomplementarity term G as follows [1]:calculations required for protein docking that incorporates shapecomplementarity and electrostatic interaction between the target
proteins. Using this simple model, we designed a PPI prediction
system “MEGADOCK” as to be capable of running on large-scaleparallel computing systems like TSUBAME.
The proposed method can be used as one of the basic
bioinformatics tools given the situation where we can use
thousands to ten thousands of CPU cores.
T h e a d v a nt a g e o f r P S C is t h a t i t is a r ea l n u m b e r
representation (Figure 2), thus we can put a physicochemicalparameter into the imaginary part. Then it is possible to calculate ascore with only one complex number for each voxel.
In order to search best docking poses, a ligand molecule
is moved through all voxel space (scores under translation are
effectively calculated as the convolution sum), and possible ligand

17

Computer prediction of protein-protein
interaction network using MEGADOCK- application to systems biology -Structure

Docking

PPI prediction

PPI network

Receptor

LigandPost-docking analysis

Figure 1 PPI network prediction.

Receptor rPSC

Ligand rPSC
Figure 2 rPSC and docking score.

orientations are exhaustively examined with 3,600 rotation angles
with 15 degree steps in default.
For a physicochemical parameter, we use electrostatic

By decreasing the number of required DFT/IFT operations withinteraction of each amino acid, ER ( l, m, n ) and E L ( l, m, n ) in Figure 2,

rPSC, MEGADOCK performs the docking process about four times

which are calculated using the interaction energy in each voxel,

faster than well-known protein-protein docking tool, ZDOCK[4,5],q( l, m, n ), with using CHARMM19[2] for the electrostatic charge of

which uses three complex numbers for the score function, as

each atom.

shown in Figure 2.

Considering these terms, the overall docking score S iscalculated as follows:

Large-scale parallelization

3

MEGADOCK is parallelized with the MPI library. Because calculations
for each pair are almost independent, we can parallelize an allIn a direct execution of simple convolution sum, O( N 6 )to-all exhaustive PPI prediction task in several ways on hundreds

calculations are required. On the other hand, the calculation order

or thousands of processors. User can specify number of receptor

using fast Fourier transform (FFT) algorithm both for discreteand ligand protein data assigned for a single processor with

Fourier transform (DFT) and inverse discrete Fourier transform (IFT)

considering memory capacity. When a processor is assigned for3

is reduced to O(N log N ) [3]. The score S for FFT is:18

m receptors and n ligands data it then calculates FFT for the firstligand with each possible rotation. The FFT results are repeatedlyused for docking with all m receptors, in order to avoid redundant

All-to-all calculation of 44 protein pairs(PPD Benchmark2.0)

calculation. Then the process is repeated for n times totally.
MEGADOCK has an option to avoid DFT calculation andupload pre-calculated DFT results from “FFT protein structure
library” on the hard disks. This approach is effective in a systemwith high I/O performance, and we recorded three times faster
speed than simple exhaustive calculation on TSUBAME 1.2.
The FF T routine in MEGADOCK uses not only base-2
logarithm, but uses 2, 3, 5 as bases to minimize volume of a target

Trypsinogen protein andtrypsin inhibitor(PDB id : 1CGI)β- actin and profilin(PDB id : 2BTF)

3-D cube. If we chose too many logarithm bases, we should
prepare so many pre-calculated FFT models in the library, because

Green: MEGADOCK2.1 predicted , Red: X-ray crystalline structure

protein paring is unknown a priori.
On the other hand, if we utilize GPU acceleration, it is betterto simply repeat FFT calculation on GPU with most adequate
combination of logarithm bases. We need to consider this point
when we implement our system on TSUBAME 2.0 system.

Application to systems biology

4

We applied MEGADOCK system to a known set of 44 protein
complex data adopted from a generally used benchmark of

Figure 3 PPI prediction results: benchmark dataprotein-protein docking, and evaluated the accuracy of theprediction. We predicted relevant interacting proteins from44 ×4 4 = 1,936 combinations by docking and post-docking
analysis. We obtained near-native complex structures as in the

Application to bacterial chemotaxis pathway

figure (Figure 3, above). And in the colored matrix (below), thered colored cells located on the diagonal line shows correctlypredicted interacting pairs.
To further demonstrate the potential of MEGADOCK system,
we evaluated our PPI prediction system by applying it to the data ofbacterial chemotaxis pathway, which is one of the typical problems
in systems biology (Figure 4) [6].

Red bold line : True Positive
Blue dashed line : False Negative
Black line : False Positive

Figure 4 Application to systems biology:
bacterial chemotaxis

19Computer prediction of protein-protein
interaction network using MEGADOCK- application to systems biology -Conclusion

5

We implemented a high-throughtput and ultra-fast PPI network
prediction system “MEGADOCK” on TSUBAME. MEGADOCK suits
well with the large-scale computing environment. And we showedthat our system could conduct network level analysis, whichis common in systems biology research, and is faster than theconventional method. We will further exploit the system as to makemega-scale analyses easier, involving analyses of proteins in cancer
Figure 5 Predicted interactions among
CheY-CheD-CheC
Enteric bacteria like Escherichia coli shows a behavior calledchemotaxis that leads the cell towards nutritional substances.

cells and microbes. Now we are analyzing 500 × 500 data using
protein data related to EGFR signaling which is important in humanlung cancer processes.
Acknowledgments

Because the signaling pathway of this system has been well-This work was supported in part by a Grant-in-Aid for Research andstudied for decades, almost all PPIs needed to explain theDevelopment of The Next-Generation Integrated Life Simulation

functionality of the system are known. We used these known

Software, and a Grant-in-Aid for Scientific Research (B), 19300102,

PPIs as relevant PPIs and evaluated the prediction performance ofboth from the Ministry of Education, Culture, Sports, Science and

MEGADOCK by investigating chemotaxis pathway proteins. We

Technology of Japan (MEXT).

conducted docking and post-docking analysis with MEGADOCK
ver.2.1 using protein data (13 protein species, 89 structures)

Referencescollected from a public protein structure database, PDB. We[1] Ohue M., Matsuzaki Y., Matsuzaki Y., Sato T., and Akiyama Y.,

analyzed 89 × 89 = 7,921 combinations of 3-D structure data. We

MEGADOCK: an all-to-all protein-protein interaction prediction

also discussed some unknown but interesting PPIs detected by thesystem using tertiary structure data and its application tosystem (Figure 5) [6].

systems biology study., IPSJ Transactions on Mathematical
Modeling and its Applications ( TOM), 3: 91-106, 2010.
[2] Katchalski-Katzir E, Shariv I, Eisenstein M, et al., Molecular surface
recognition: determination of geometric fit between proteins
and their ligands by correlation techniques., Proc Natl Acad Sci
U S A, 89:2195-9, 1992.
[3] Chen R., and Weng Z., Docking unbound proteins using shapecomplementarity, desolvation, and electrostatics., PROTEINS,
47: 281-294, 2002.
[4] Chen R., Li L., and Weng Z., ZDOCK: an initial-stage proteindocking algorithm., PROTEINS, 52: 80-87, 2003.
[5] Brooks BR, Bruccoleri RE, Olafson BD, et al., CHARMM: A programfor macromolecular energy, minimization, and dynamics
calculations., J Comput Chem, 4:187-217, 1983.
[6] Matsuzaki Y., Matsuzaki Y., Sato T and Akiyama Y., In silico
screening of protein-protein interactions with all-to-all rigid
docking and clustering: an application to pathway analysis., J
Bioinform Comput Biol, 7:991-1012, 2009.

20

● TSUBAME e-Science Journal No. 2
Published 11/10/2010 by GSIC, Tokyo Institute of Technology © ISSN 2185- 6028
Design & Layout: Caiba & Kick and Punch
Editor:TSUBAME e-Science Journal - Editorial room
Takayuki AOKI, Toshio WATANABE, Masakazu SEKIJIMA, Thirapong PIPATPONGSA, Fumiko MIYAMA
Address: 2-12-1 i7-3 O-okayama, Meguro-ku, Tokyo 152-8550
Tel:+81-3-5734-2087 Fax: +81-3-5734-3198
E-mail: tsubame_j@sim.gsic.titech.ac.jp
URL:
http://www.gsic.titech.ac.jp/

International Research Collaboration
The high performance of supercomputer TSUBAME has been extended to theinternational arena. We promote international research collaborations using
TSUBAME between researchers of Tokyo Institute of Technology and overseas
research institutions as well as research groups worldwide.

Recent research collaborations using TSUBAME
1. Simulation of Tsunamis Generated by Earthquakes using Parallel
Computing Technique
2. Numerical Simulation of Energy Conversion with MHD Plasma-fluid Flow
3. GPU computing for Computational Fluid Dynamics

Application Guidance
Candidates to initiate research collaborations are expected to conclude
MOU (Memorandum of Understanding) with the partner organizations/departments. Committee reviews the “Agreement for Collaboration” for joint
research to ensure that the proposed research meet academic qualifications
and contributions to international society. Overseas users must observe
rules and regulations on using TSUBAME. User fees are paid by Tokyo Tech’s
researcher as part of research collaboration. The results of joint research are
expected to be released for academic publication.

Inquiry
Please see the following website for more details.
http://www.gsic.titech.ac.jp/en/InternationalCollaboration