3

TSUBAME 2.0 Begins

The long road from TSUBAME1.0 to 2.0 (Part Two)GPU Computing for Interstellar
Atomic Hydrogen Turbulence- 11.5 TFlops with 120 GPU on TSUBAME 1.2 -Fast Fourier Transform using GPU

TSUBAME 2.0Begins

The long road from TSUBAME1.0 to 2.0 (Part Two)
Satoshi Matsuoka** Global Scientific Information and Computing Center

TSUBAME2.0 is Japan's first muti-petascale supercomputer with multitudes of innovative in architectural andsoftware features such as extensive use of GPUs, highly scalable and high-bandwidth node and network design,
as well as massive utilization of advanced I/O technologies such as SSD. TSUBAME2.0 had gone into production operation
as of early Nov., 2010; the part two of the article will cover the benchmarking activities conducted just prior tothat event, characterizing the overall performance of the machine in terms of dense compute-bound applications,power consumption, and high-bandwidth applications . TSUBAME2.0 became the fourth fastest supercomputerin the world on the Top500, as well as being awarded the "Greenest Production Supercomputer in the World" award on
the Green 500 in Nov. 2011, and at the sametime setting world record on bandwidth intensive ASUCA weather code,demonstrating its innovative performance and design.

The First Cries of a Newborn:
TSUBAME2.0

1

the lowest mutual common denominator between the end
points, so that communication can be established. However,when some glitch occurs that compromises a connection,and due to some reason the established communication is ofmuch lower specs, it would be more difficult to detect such

The first challenges for the newborn TSUBAME2.0, as its
installation was finishing in Oct., 2010, were large-scale benchmarks

anomalies. Automated means of detecting and compensating
for such performance anomalies are strongly required.

that would use the entire machine, such as Linpack. Such all-(3) In addition, large-scale supercomputers require constantmachine benchmark s in the early days of supercomputer

monitoring by tens to hundreds of thousands of sensors,

inception are fairly commonplace and important for the following

as well as proactive means for allocating resources to its

technical reasons:numerous simultaneous users with extremely large-scalerequests---users may submit tens of thousands of jobs at the(1) T h e n u m b e r o f c o m p o n e n t s e m b o d i e d i n a l a r g esame time, for example. If any portion of the job allocation

supercomputer such as TSUBAME2.0 is more than several

algorithm embodies O(n 2) behavior, then scaling of thethousands of factors greater than a standard PC. Even if we

machine would be catastrophic---a 100 times scaling would

count merely the number of sockets of compute elements,

manifest in 10,000-fold increase in overheads.

namely the GPUs and CPUs, TSUBAME embodies over
7000, whereas a standard PCs only would have one or two.

By all means, it is important not just for operations, but also

Memory on PCs would usually be a few gigabytes, whereas

from academic Computer Science perspective, to determine how

TSUBAME2.0 embodies nearly 100 Terabytes, several tens

much we would be on target with the designed performance,of thousands greater. For such numerous components toa n d /o r w h at t h e u n e x p e c te d o ve r h ea d w o u l d b e. T h eperform under prolonged stress is one of the most importantcomputational science users would only benefit from the exercise,factors in attaining stable and reliable operation, as the failure

as various factors including performance but also reliability andrates are roughly proportional to the number of components

usability at scale would greatly affect their actual usage.

in the system; that is to say, a PC which only fails once in theWith such a set of objectives, a series of large-scale,years would fail over 3 times a day if enlarged to the size ofwhole-machine benchmarks were conducted just before the

TSUBAME2.0.

operational commencement of TSUBAME2.0 on November 1, 2010,(2) At the same time, it is very important to confirm that expected

mostly throughout late October just after the machine was born:

design performance is being met in reality is especially02

important for supercomputers, as slightest deviation could(a) Linpack---the benchmark employed by the famous Top500[1]profoundly deteriorate overall performance of the system.

supercomputer performance ranking. Basically, it computes

It is quite common in a supercomputer for a component

the LU-decomposition of a very large dense matrix. For anwhich seemingly function but not performing up to its specs

n-by-n matrix, the computational complexity can be given

become the critical performance bottleneck. For example,

as 2/3 n 3+ O(n2). This means that, the larger we can makethe Infiniband network employed in TSUBAME has several

the problem, up to the point where the entire matrix fits within

speed specs, and the system automatically detunes itself tothe memory of the supercomputer, the more ef ficient thecomputation becomes due to the communication and othertop-level performance on TSUBAME2.0, as the theoreticalcosts becoming relatively minimized. As a result, top-level

memory bandwidth of TSUBAME2.0 is approximately six

supercomputers employ extremely large matrixes namely

times greater than the Earth Simulator, and ASUCA on GPU

n=several million, and subjects the CPUs/GPUs to the ultimately

was demonstrating to be quite efficient and scalable onhigh workload for a long duration. The solution algorithm isour preliminary tests on TSUBAME 1.2. Such tremendous

delicate with no margin of error---even a single numericalper formance was expected to allow real-time weather

error for its total O(1019~20) operations would result in an error
in the residual check, which would nullify the entire result.
On the other hand, since the communication complexity

prediction at unprecedented resolution and precision.
(c) Also, a set of benchmarks were performed as a part ofacceptance test of TSUBAME2.0.

being relatively low being O(n2), a reasonable supercomputer
network would incur less than 10% network overhead. An
extremely slow network such as the Gigabit Ethernet would
end up having the network cost being dominant. Finally,
memory bandwidth requirements are also fairly low.

TSUBAME2.0 Linpack---"The Greenest Production Supercomputerin the World"

2

Although there is no "of ficial" sof t ware for LUdecomposition, most supercomputers use the HPL(HighThe whole-machine benchmarks were commenced in mid-

Per formance Linpack)[2] which was designed for large

October 2010, immediately after the initial deployment tests ofparallel machines. However, on TSUBAME2.0, we employed

TSUABME2.0 were completed. We first commenced the Linpacka set of heavily customized versions of HPL for heterogeneous

efforts spearheaded by the two teams running two different

GPU-CPU computation. One was the Heterogeneous Linpack

heterogeneous Linpack programs as described above in (a).

that we had been developing on Linux over many years at

Due to the unprecedented load imposed on the machine not

the Matsuoka Laboratory, and other was the Windows HPC

possible on initial tests, we found a number of minor problems as

Linpack which was the result of our collaboration with andexpected, and resolved the issues one by one to attain stability.

developed by Microsoft, and naturally running on Windows

The Linux team at Tokyo Tech. and the Windows HPC team sent

HPC. The two programs took a very different approaches tofrom Microsoft took turns running their respective benchmarks.

cope with GPU/CPU heterogeneity, and it was quite interesting

Both were very closely matched, but in the end the Linux team's

to determine which would be advantageous for the future

heterogeneous edged the latter (Figure 1). It is important to noteadvancement of GPU computing.

however that, under slightly different conditions it would haveAlso, we conducted precise measurement of power
consumption of TSUBAME for being ranked high on the Green500[4], as the most important criteria of TSUBAME2.0 was tobecome ultimate green supercomputer of the time.

been be quite possible that the results could have been the
opposite.
As a result, TSUBAME2.0 recorded 1.192 Petaflops, achieving
approximately 52% of the theoretical peak performance. This is(b) GPU Version of ASUCA---ASCA is the next generation weather

lower than the typical 70-90% achieved by Linpack on CPU-based

forecast code for extremely large machines, being developed

supercomputers. However, it is not technically correct in simply

by Japan's Meteorological Agency. In collaboration with theassuming that GPU-based machines are inherently less efficient

Agency, a group at Professor Aoki's laboratory succeeded

compared to CPU-based ones. For our particular case, the lower

in full porting of ASUCA on a multi-GPU heterogeneous

efficiency is due to combination of the following performancesupercomputing environment [5][6]. The principal computational

degradation factors:

kernel of ASCA is a finite difference transport code, requiringextremely high memor y and network bandwidth, quite

1. Firstly, the current NVIDIA Fermi GPU as employed incontrasting to Linpack. Previously, such high-bandwidth

TUBAME2.0 embodies a set of design bottlenecks that are notapplication had been perceived to be best served by custom-

fundamental to GPU computing but rather results of particular

design, high-end vector supercomputers; so it was deemed

design decisions. Although sufficient for graphics as well as forimportant that such code would perform extremely well on

high-bandwidth applications where the GPUs are being used

TSUBAME2.0, whose chief compute element, GPU, embodies

in the similar manner as traditional vector processors, for dense

extremely high bandwidth as a modern-day vector processor

matrix multiply (Level 3 BLAS) which is the principal kernel ofIndeed we expected ASUCA on GPUs to achieve world's

Linpack, we only achieve 70-75% efficiency. This is substantially03

TSUBAME 2.0 Begins

The long road from TSUBAME1.0 to 2.0 (Part Two)

Figure 1 TSUBAME2.0 Linpack Execution Output.
Here we see that the Linpack run involved the matrix of n= 2.5 million squared elements ,and the run was completed in 2.4 hours, resulting in 1.192 Petaflops which ranked
TSUABME2.0 to be fourth fastest in the world on the Nov. 2011 edition of the Top500.
Notice that the residual computation is within the proper error bounds
which is a required property of the run.

lower than the efficiency achieved by CPUs that exhibit more

pragmatic applications would hide most of the transfer latency.

than 90% efficiency. However, with architectural as well as

However, for HPL the sub-matrix size is rather small, with nalgorithmic improvements we expect future GPUs to matchbeing 100~1000, resulting in non-negligible transfer overhead.

CPU efficiency in this regard, if not greatly exceed it.
2. For our Heterogeneous Linpack on Linux, CPUs are not utilized

The combination of all the factors above results infor BLAS kernel computation; however, the Top500 results

approximately 30% overhead. With advances in the GPU/CPU

mandate us to incorporate the CPU peak performance inarchitecture as well as algorithms to utilize them efficienctly wedetermination of the theoretical peak performance of thebelieve that the overhead could be effectively eliminated. By all

machine. For TSUBAME2.0, the two CPUs on each computemeans they are subject of future research.

node (Intel Xeon Westmere 2.93Ghz) constitute approximately

On the 36th edition of the Top 500 which was announced

8% of the peak performance, which we lose when we computeduring IEEE/ACM Supercomputing held in New Orleans, USA

measured versus theoretical performance ratio. By all means

during November, 2011, TSUBAME2.0 was ranked to be number 4

we could conceive an algorithm that does utilize the CPU, andin the world; this was higher than TSUBAME1.2's initial appearance

in fact we did so for TSUBAME1.2[3], but that particular version

in June 2006, which had been 7th in the world. The exhibited

proved to be less efficient due to various issues such as load

1.192 Petaflops was six times greater than the second ranking

balancing.

machine in Japan. Moreover, on the Green500, which ranks

3. Our Heterogeneous Linpack effectively utilizes GPUs as a matrix

supercomputers based on their power efficiency, the average

multiply engine, where we send the sub-matrices of the matrix

power consumption of 1243.80KW during the Top500 run resulted

which is stored in CPU memory in stream pipelined fashion,in 958.35 Flops/W, which ranked TSUBAME2.0 to be second in theperform the multiplication, and stream the matrix back. For

world on its initial November announcement. More importantly,

normal applications where we typically transfer data in bulk to

TSUBAME2.0 was recognized to be the "Greenest Production

the CPU, dense computing algorithms such as matrix multiply

Supercomputer in the World" (Figure 2), as other top machines on

where the compute overhead is O(n 3) as opposed to thethe Green500 were largely prototypes in nature.

2transfer overhead being O(n ), enlarging the matrix size n for04

T h e h i g h r a n k i n g s o f T SUA B M E 2 . 0 o n b o t h l i s t ssimultaneously have an important technological significance.
According to the current rules, it is difficult for a machine to behigh on both lists; in practice, the top supercomputers of theTop500 are extremely large-scale production supercomputers,
whereas the top rank s of the Green500 are smaller-scale
prototypes and/or special-purpose machines (Figure 3). Only
TSUBAME2.0 is ranked within world's top five on both lists.
Here are why such difficulty exists:(1) Since the top supercomputers on the Top500 are large scale,
general-purpose production machines worth 10s to 100s of
millions of dollars; as a result they typically embody numerous

Figure 2 November 2011 Green 500 Special Award for"Greenest Production Supercomputer in the World"

elements that are necessary for production runs but will be
detrimental to power efficiency. For example, such machines
incorporate 100s of terabytes of memory which is necessary
for practical high bandwidth/memory applications, but does
not contribute much to increasing the performance of Linpack.
On large machines DRAM power consumption could be as
much as 20-30% of the entire machine. On the other hand, toshoot for power efficiency on Linpack the best strategy would
be to have rather small memory, but this would limit the

TSUBAME2.0 World Rankings(Nov. 2010 Announcement Top500 and Green500)

The Top 500

Absolute Performance#1 ： 2.56 PetaFlops ： China Defense Univ.
Dawning Tianhe 1-A (11)#2 ： 1.76 Petaflops ： US ORNL Cray XT5 Jaguar (81)scope of the machine to a very small number of specialized#3 ： 1.27 PetaFlops ： China Shenzen SC Nebulae (13)(typically compute intensive) applications.

#4 ： 1.19 PetaFlops ： Japan Tokyo Tech. HP/NEC
TSUBAME2.0 ( 2)(2) The top rank s of Top500 of ten is of ex treme logisticalimportance for computing centers and even countries, and as aresult, all other factors could become sacrificed just to go up anotch in the rankings, including power efficiency in the Linpack#5 ：1.054 PetaFlops ： US LLBL Cray XE6 Hopper (30)#33 ： 0.191 Petaflops ： JAEA Fujitsu (95)(#2 Japan)(Green500 rank)algorithm and settings. On the other hand, if one would shootfor top ranks of the Green 500, going down in rankings on the
Top500 would not matter---all that matters would be that the

The Green 500

Performance/Power Efficiency

machine being on the Top500. Such difference of objectives#1 ： 1684.20 ： US IBM Research BG/Q Prototype (116)are difficult to coexist especially at the tops of each list.

#2 ： 958.35 ： Japan Tokyo Tech/HP/NEC TSUBAME 2.0 (4)(3) The Top500 runs of Linpack involve matrix size of n=a few
million, with tens of thousands of processor cores. Unfortunately,by the nature of the algorithm Linpack is inhererently more

#3 ： 933.06 ： US NCSA Hybrid Cluster Prototype (403)#4 ： 828.67 ： Japan Riken “K” Supercomputer
Prototype (170)

5-10% drop in performance just by going multi-node, with

#5-7 ： 773.38 ： Germany Julich etc.
IBM QPACE SFB TR (207-209)

increasing overhead with large to machine size [3].

#2+ ： 1448.03 ： Japan NAO Grape-DR Prototype (383)efficient on smaller machines. For example, one typically sees

Despite such disadvantageous, TSUBAME2.0 being ranked
highly on both lists was the reason for the award in Figure 2. By allmeans this was not achieved by simple employment of GPUs,
as other GPU machines did not achieve similar results. Rather thiswas the result of years of basic research on low power, highperformance computing at Tokyo Tech. GSIC, including the JSTCREST Ultra Low Power HPC (ULP-HPC) project.

(Added in Dec.)(Top500 rank)

Figure 3
The top ranking machine of November 2010 Top500 andGreen500 and their corresponding rankings on the other list.
(At the very end of 2010 a revised list which ranks Japan
NAO's Grape-DR to be ranked 2+ on the Top500 at 1448.03 andTop500 being 383; this did NOT change the status of
TSUBAME2.0 being the most power efficient production
supercomputer in the world.)

05

TSUBAME 2.0 Begins

The long road from TSUBAME1.0 to 2.0 (Part Two)

Memory and Network Benchmarking
in a Real Application--- GPU Porting of ASUCA Weather Code andits World Record Performance3

Exactly the same argument applies to GPUs versus CPUs,
but often the same mistake mistakes are made. As had been
mentioned in part one of this article, GPUs exhibit extremely
high memory bandwidth per socket compared to CPUs, but
at the same time, also embody much higher (and effectively

Although Linpack/Top500 is a significant metric for supercomputer

overprovision) compute as well. In fac t, on TSUBAME2.0,

performance measurement, for numerous applications thatthe p er-so cket p eak compute capabilit y of each GPU isare largely memory or network bandwidth bound the Top500

approximately 7 times that of CPU, but at the same time,numbers are not effective metrics. That is to say, for manymeasured effective memory bandwidth is also 6-7 times perimportant simulation applications such as computational fluid

socket. So if CPU-based implementation on TSUBAME2.0 would

dynamics, structural simulations, and even modern apps suchbe obtaining 5% of peak, then it is likely that a GPU-based

as Internet page rankings, how much ef fective bandwidth

implementation would be obtaining similar computationalis achieved governs the overall performance, not how muchefficiency, while being 6-7 times faster with the same number

Flops, and as a result, the baseline availability of theoreticalof sockets.

peak bandwidth as well as the ease at which major fraction ofThis in ef fec t allows us to largely conjec ture thatthe peak bandwidth could be achieved, become the dominant

TSUBAME2.0 would be comparable to x86 CPU-based machines

performance factor. Unfortunately, in recent supercomputerof similar peak performance, or those with socket counts thatarchitec tural trends, the amount of available bandwidth

are 7 times greater, if no artificial bottlenecks are imposed as

relative to the machine size, as well as its ratio to compute,

was the case for Linpack. That is to say TSUBAME2.0 with 4200

is on constant decrease due to various physical limitations.

GPU and 2800 CPU sockets would be roughly equivalent to

Vector supercomputers of the past, such as the Cray X series

a 30,000 socket / 200,000 core x86 CPU-based supercomputer,and the NEC SXes were built precisely for this purpose inwhich would largely equal the size of ORNL Jaguar. In fact wemind, i.e., increase the memory bandwidth during the days

can expect that Jaguar would have advantage in computewhere achieving high compute flops was technologically difficult,but might not perform as well on high bandwidth code, as

resulting in high computational efficiency as a result.

the older-generation x86 it employs is much less efficient in

Such glory days are over, and in fact we must really re-06

memory but similar in FLOPs utilization.

think our notion of efficiency for modern machines in terms

The ASUCA benchmark was important in this regardof efficiency over the dominant performance bottlenecks, into determine if such per formance estimations would holdthis case the memory/network bandwidth, but not compute.

for high-bandwidth applications. In par ticular, since finite

So, the true "efficiency" metric is how much the application isdifference solvers for transport codes are known to be mostly

utilizing the memory/network bandwidth in the system relative

pure memor y-bandwidth limited, the issues are whether

to the theoretical peak available, and has no correlations to the(1) GPUs would be able to ef ficiently utilize the available

peak FLOPS of the machine.

memory bandwidth, (2) whether we could achieve 6-7 times

In fact, in this regard we must point out that highspeedup per socket as discussed above, and finally (3) how thecomputational efficiency that was apparently achieved in such

performance would compare as a whole machine to Jaguar inclassic vector machines were rather artificial and misleading,executing the same or at least very similar weather code.

the result of technological trend of the times, and is not an

For details of the ASUCA code itself the readers are

effective metric in the modern times. That is to say, for suchreferred to [5]; the results achieved largely confirmed ourve c tor machines that under- provisione d the computing

conjec tures (1)-(3) above. The GPU version of ASUCA on

resources do not properly exploit the available opportunity

TSUBAME2.0 scaled up to 3990 GPUs[6] almost linearly in weak

presented by dense problems and the available locality (e.g.,scaling (the problem size proportionally increasing relative

due to the lack of cache memory), and as a result, its efficiency

to the machine size), and achieved 145 TeraFlops in single

might se em ar tif icially high, but the resulting absolute

precision, and 76.1 TeraFlops in double precision (Figure 4). The

performance being relatively low in relevance to the size/cost

per-socket performance is approximately 6 times that of CPUs,of the machine.

confirming our conjecture in real, production-level code.

Moreover, the previous world record holder was the
WRF code, a similar weather code to ASUCA, on Jaguar at

Final Words : Let More Petascale

approximately 50 TeraFlops (double precision); so ASUCA was

Applications Begin!

even faster, by a factor of 10 on a socket-to-socket basis,4

we might attribute this to less efficient memory bus of theolder-generation AMD processors in Jaguar, but since theT here were several other benchmark s that were run on

applications are dif ferent, this result should be taken as

TSUBAME2.0, including I/O, to confirm that the machine waspreliminary, and more rigorous benchmarking should be done

running correctly, after which the production operation began

using the same applications under a controlled environment.

in early November, 2010 as planned. At the time of this writing
numerous users are using TSUBAME2.0 daily imposing high loads.
Large-scale applications that utilize 5000 CPU cores or 1200 GPUs
(that are maximally allocatable under normal operations) are
fairly commonplace. Applications that utilize other aspects ofthe system for large-scale data processing, such as the SSD andthe LUSTRE parallel filesystem, are also substantially increasing
in number. Combined, the performance and scaling leap fromTSUBAME1.2 is very apparent, and users seem to be enjoying
the massive capability and relatively forgiving characteristics of
TSUBAME2.0
However, to best utilize the capability of TSUBAME2.0, onemust utilize its most advanced features. For example, in order toFigure 4 ASUCA Benchmarking on TSUBAME2.0 (1).
Notice that performance scales
almost linearly to 3990 GPUs.

best utilize the available memory bandwidth the use of GPUs area must, but not only this involves programming in extension of C
namely the CUDA language, but also involves fairly sophisticatedhybrid programming involving MPI and CUDA in CPUs andGPUs. Also GPU performance is more sensitive to algorithms,
programming methods, and data layouts, mandating more careful
tuning process to optimize the performance. This also applies
to other aspects of the system. Fortunately for TSUBAME2.0
networking is less troublesome as the network is configured as afull-bisection network, allowing the compute nodes to be muchless sensitive to its placement, but for I/O effective hybrid use of
SSDs and LUSTRE is sometimes required.
From a broader perspective, supercomputers composedfrom CPUs that consists of large many-core / multithread / vector
processor, combined with a small number of low latency scalar

Figure 5 ASUCA Benchmarking on TSUBAME2.0.
The per-socket performance difference isapproximately factor of six,
confirming the relative differencein peak achievable memory bandwidth.

cores in proximity, interconnected with multi-rail high-bandwidth
network and I/O capabilities would be the next trend in large
scale supercomputer as a continuum to TSUBAME2.0 architectural
trend. By all means in the not so distant future the two typesof cores would be bonded on the same die, sharing memory.
Such would alleviate many of the complexities as currently being
experienced in hybrid architectures such as TSUBAME2.0 today.
This would allow tremendous number of applications scale to bepetaflop-class, be it compute or memory bound, as we have seen
for Linpack and ASUCA, but with much less effort. In effect, in themost advanced large-scale GPU applications we are observing

07

TSUBAME 2.0 Begins

The long road from TSUBAME1.0 to 2.0 (Part Two)

the future, where petascale performance is being achieved with
substantial effort, but will be quite the norm, allowing high-end
science and engineering to progress much faster than in the past.
We will continue to strive in TSUBAME2.0 to demonstrate suchpossibilities, and prepare for TSUBAME3.0 to be designed andbecome operational in 2014, with possibly yet another great leapin performance.
References[1] The Top500 Supercomputing Sites http://www.top500.org/[2] A. Petitet, R. C. Whaley, J. Dongarra, A. Cleary. "HPL - A Portable
Implementation of the High-Performance Linpack Benchmark
for Distributed-Memory Computers v.2.0", http://www.netlib.
org/benchmark/hpl/[3] Endo, T.; Nukada, A.;

Matsuoka, S.;

Maruyama, N. "Linpack

Ev aluation on a Sup ercomputer with Hetero gene ous
Accelerators", Proc. IEEE Parallel & Distributed Processing (IPDPS)
2010, the IEEE Press, Apr. 2010, pp.1-8.
[4] The Green 500 http://www.green500.org/[5] Tak ashi Shimok aw ab e and Tak ay uk i Aok i. "Multi - GPU
Computing for Next Generation Weather Forecasting", The
TSUBAME E-Scinece Jounal, vol. 2, GSIC Center, Tokyo Instituteof Technology, Nov. 2010, pp.11-15.
[6] T. Shimokawabe, T. Aoki, C. Muroi, J. Ishida, K. Kawano, T. Endo, A.
Nukada, N. Maruyama, and S. Matsuoka. "An 80-Fold Speedup,
15.0 TFlops, Full GPU Acceleration of Non-Hydrostatic Weather
Model ASUCA Produc tion Code", Proc. 2010 ACM/IEEE
Supercomputing, The ACM Press, Nov. 2010, pp.1-11.

08

GPU Computing for Interstellar
Atomic Hydrogen Turbulence― 11.5 TFlops with 120 GPU on TSUBAME 1.2 ―Takayuki Muranushi **The Hakubi Center, Kyoto University

We have performed three-dimensional hydrodynamic simulations to study
the thermal instability of the interstellar medium. The instability is powered by phase-transition of
the atomic hydrogen gas and is one of the sources for turbulence in the interstellar atomic hydrogen gas.
We have also performed clump-detection and spectra analyses, as well as three-dimensional visualizations.
The analyses revealed multi-scale turbulent cascade connecting
the supersonic compressive turbulence at the large scale to the well-known Kolmogorov turbulence.
Such finding was enabled by the massive computational power of the GPU cluster supercomputers.

Introduction

1

On the JHPCN program2

My favorite word to describe our profession is “armchairFirst, we would like to mention the JHPCN Program (Joint Usage

astronaut.” Human beings have sent astronauts to the Moon and/ Research Center for Interdisciplinary Large-Scale Information

unmanned probes to the end of the Solar System. But to visit

Infrastruc tures) through which we gained access to thethe neighboring stars and collect the evidences will remain amachine located in Tokyo Institute of Technology. JHPCN is avery hard job for next generations. To overcome the limitations

network consisting of eight university-owned supercomputerand understand the mystery of the universe, we have to collectfacilities in Japan. We’ve been using TSUBAME since 2009

the observational evidences sitting on the Earth, and carefully

through this program.

construct the reasoning based on the scientific laws. Computer

We applied to the JHPCN program as the joint research

simulations are powerful tools for us. They help us understandbetween Takayuki Muranushi at The Hakubi Center, Kyoto

how the rich structures are formed out of the simple laws, andUniver sit y and Tsu yoshi Hamada at Nagas ak i Ad v ance d

how the structures will look like to our limited eyes. The beautiful

Computing Center, Nagasaki University. Hamada leads thevisualizations of today’s advanced simulations make us feel likedevelopment and operation of GPU cluster DEGIMA (DEstination

travelling the universe while sitting on armchairs.

of GPU Intensive Machines). Also there is a small GPU cluster

What we have studied using TSUBAME is one of the

TenGU (Tenmon GPU Cluster) in Kyoto University. It was useful toprocesses that will determine the mass of the stars. Stars arehave access to these different types of computers for different

born from fragmentation and condensation of the interstellar gas

stages of the code development. Joint research brought helpful

in the galaxy, which is as thin as one Hydrogen atom per cubic

communications with the researchers of var ying fields for

centimeter. If the typical stars are ten times lighter, the nuclear

code development and research. Also we had many technical,

fusion won’t ignite and our galaxy will remain dark. If they areadministrative and clerical supports from Tokyo Institute of

ten times heavier, they will much more rapidly burn Hydrogen

Technology and JHPCN staffs.

than they do today and our galaxy will be filled with black holes.
The mechanism that sets the adequate initial masses for the stars--- is the interstellar turbulence driven by the thermal instability.
Interstellar hydrogen has two stable phases determined by
the balance of various heating and cooling processes such as star

Simulating the Interstellar Medium

3

irradiation and molecular line emission. Triggered by supernova
shock waves, the interstellar medium makes phase transits fromWe have prepared several themes at the beginning of thethe warm, low density phase to numerous clumps of the cold,

JHPCN program. In this section we will describe one of thehigh density phase. It takes several more stories until stars andthemes, “Numerical Simulation and Analysis of The Two-Phased

planetary systems are formed from these clumps. There are lots

Hydrogen Gas Turbulence in Molecular Cloud Forming Region.”of detective works left to do in the universe.

As introduced in the first chapter, this study contributes to the09GPU Computing for Interstellar
Atomic Hydrogen Turbulence― 11.5 TFlops with 120 GPU on TSUBAME 1.2―

study of the interstellar medium, one of the various themes in

For the heating term Γ and the cooling term Λ, numerous

astrophysics.

heating and cooling processes contribute (e.g. Koyama & Inutsuka,

In the galaxy compressive and anisotropic turbulencesof the interstellar medium are obser ved. Considering the2000[3].) Based on that, we use the following fitting formula
proposed by Inoue & Inutsuka (2008)[4]:radiative equilibrium, the atomic interstellar hydrogen gas hasbistable equations of state. It is much different from that of theideal gas and has two stable phases, differing hundred times
in density (Field et al, 1969[1].) However, observed interstellar
turbulence is described by the homogeneous, isotropic andincompressive Kolmogorov turbulence spectra (power law index

Quantities are nondimensionalized based on the following threeof the velocity field αv =11/3.) Still, some observations reportscales typical to the interstellar phenomena:significant deviations from the Kolmogorov turbulence (αv=3.87±0.11) (Chepurnov et.al, 2010[2].) Understanding this turbulence
has been one of the grand challenges for interstellar physics,
requiring simulations of the resolutions of the order 10003. The
use of GPU computer lets us simulate the turbulence for time
much longer than before keeping the high resolutions, and drawout more detailed information on the turbulent statistics.
The basic equations to be simulated are the following

This means that the sound speed of the gas is about

Navier-Stokes equations coupled with the heating and cooling

10km per second, as fast as the speed of a rocket escaping theformulae.

Earth. Still, it takes about one hundred thousand years to crossa parsec (approximately 3.62 light years) with this sound speed.
The simulation region (Figure 1) was 20 parsec each side, and wehave simulated eight sound-crossing times to have the enoughdata to study the statistics of the turbulence. The total simulation
corresponds to whopping sixteen million physical years. It is, still
just an instance in the cosmological scales of time.

Figure 1
A visualization of the14403 simulation performedon TSUBAME GPU Cluster
at Tokyo Institute ofTechnology.

10Our code can be classified like “MPI GPU Full-Godunov
2nd order MUSCL 3- dimensional uniform mesh

Navier-Next, we have carried out long-run cariculations using
GPU supercomputer TSUBAME. We have reserved TSUBAME three

Stokes equations solver”. Godunov method is proposed by atimes: July 6-9,

July 20-23, August 20-24. The hpc1tes2 queue

Russian Mathematitian Godunov(1959) [5], and its higher-order

we have used consisted of 120 NVIDIA GT200 GPU chips and had

extensions are established by efforts of e.g. van Fig. The idea

124.2Tflops single precision peak performance. The maximumof Godunov’s method is to apply the analytic solutions of thenumber of threads available was 122’880. Total amount of the

Riemann problems (hydrodynamic problem starting with twovideo memory was 480GB and its bandwidth 1.224TB/s. It costed

constant initial conditions separated by a wall) to each mesh132’000 yen. Thus we obtained 14403 resolution simulated databoundary. This is necessary treatment to sharply resolve thefor about 8 sound crossing times, sufficient for the statisticalshock fronts. The second order version costs about 3000 floating

analyses of the turbulence. Also we obtained about 200 snapshot

operations per mesh update. Here, GPU’s high computation

data for the movie. The sustained performance was 11.5Tflops.

capability comes in handy.
Of course this code can handle multiple GPUs via MPI
framework. We have implemented checkponiting capabilities,
to overcome the accidental abort of the computing devices

Analyses and Visualization

and also overcome the time-limit applied to the job-queues

4

on supercomputers. We have also implemented irreversibly
compressed data outputs for movie visualizations.

Because we have achieved so high resolutions, the analyses andWe have carried out most of our code development

visualization of the data became challenging tasks and we had toand tuning on DEGIMA. On DEGIMA about 800 NVIDIA GT200

develop new tools for them. For example, each of the snapshot

GPU chips are available, and 576 of them was connected via

data was about 60GB, which is larger than the memory capacity

InfiniBand.It has 514.9Tflops single precision peak performance,of typical personal computer. So we have invented algorithms

and can handle 1769'4720 threads at most. Total amount of thethat consume memory only proportional to the 2-dimensional

video memory was 460GB, total bandwidth was 64.454TB/s. On

cross section of the data, and make minimum number of access

DEGIMA we have recorded 40.91Tﬂops sustained performance on

and only sequential access to the disk, while performing the12803 resolution simulations.

desired tasks.
For example, in clump detection analysis we need to detect
high density regions of the gas (Clump) and its surrounding
media (Veil), calculate their physical properties such as density,
sound speed, relative motion, and perform statistical analysis onthem. A naïve implementation for connected component finding
algorithm in three-dimensional grid will require random access on
the 3D grid. We have developed a new algorithm that performs
the above analysis within only (constant)×(two-dimensional crosssection) memory and twice sequential access of the whole data.
The algorithm enabled clump detection analysis in practical timeon PCs while keeping the main data on the disk.
The analysis revealed that the Cold Neutral Medium (CNM)
motion was actually faster than CNM sound speed. However,each CNM clump is surrounded by unstable medium. The CNM
motion was slower than the sound speed of the surrounding

Figure 2
The Clump Detection Analysis.
The horizontal axis is detected clump mass.
The vertical axis is the clump speed relative toits surroundings (veil) in Mach numbers. We have found thatclump motion is supersonic in terms of its own sound speed,
but is subsonic in terms of veil sound speed.

medium. This suggests that incompressive, Kolmogorov aspect of
the turbulence can be explained by the CNM motion subsonic in
the unstable media.

11

GPU Computing for Interstellar
Atomic Hydrogen Turbulence― 11.5 TFlops with 120 GPU on TSUBAME 1.2―

To study the turbulence, we have performed spectraet.al. 2010’s (αv =3.87±0.11) (αε=3.0±0.1. ) Supersonic turbulence

analyses of the 14403 simulation by the following algorithm.

regime has softer, and Kolmogorov regime has harder velocity

First, create two-dimensional projection for each of the three

power spectrum. These findings indicated that non-Kolmogorov

axes while scanning the snapshot once. Next, perform two-

turbulence observed like in Chepurnov et.al. may be explained as

dimensional Fourier analysis of the velocit y field. Finally,

superposition of the supersonic and Kolmogorov turbulences.

reconstruct the turbulent spectra using the same method used

To v i s u a l i z e t h e s i m u l a t i o n a s m o v i e s , w e h a v e

in reconstructing the astronomical observation. This algorithm

implemented an renderer that can create perspective or parallel

shortened the time required in Fourier analyses to practicalprojection plots of various physical quantities with only one orlevel and also meaningful in comparing the simulated data withseveral times of sequential access to the data. We can clearly see

observations.

in the movie the violent displacement of the shock front and

The spectra of the simulated turbulence showed the three-episodic penetration of streaming flows (Figure 1.)stage structure that consists of supersonic turbulence regime,

Also in collaboration with O yamada Lab. in Kyoto

Kolmogorov-like turbulence regime, and numerical dissipation

University, specialists of large -scale data visualization, we

regime (Figure 3). Overall fit of the density and velocity power

have made large and high-resolution visualization using thespectra showed significantly agreement with that of Chepurnov

40-side tiled display (Figure 4). We have also created a movieof our simulation on TSUBAME with Prof. Aoki’s help for use inadvertisement of TSUBAME.
We are continuing the research on this theme, changing
conditions and performing more analyses, to draw out more
physical knowledge.

From Hydrodynamics toMagnetohydrodynamics

5

Based on the experiences earned in the development andoperation of the hydrodynamics code on multi- GPUs, webegun development of magnetohydrodynamic (MHD) codesfrom August. MHD is one of the basic equations to describe
Figure 3 The result of velocity spectrum analysis.

behavior of ionized gas, plasma. Magnetic field takes the essential
roles in many active astrophysical phenomena including accretion
disks, jet activities, the Sun, and the Earth magnetosphere.
MHD is also used to control plasma e.g. in nuclear fusion
reactors studies. Use of GPGPUs in magnetohydrodynamics
will accelerate simulations, and make more high-resolution
simulations possible in those fields of science. It will also
help in engineering connected to our daily lives such as solar
activity prediction and design of spacecrafts.
Current version of the code is second-order in space
and time. We have adopted HLLD scheme by Miyoshi & Kusano
2005[7] for MHD Riemann solvers. As we develop the MHD code
we have performed various tests, and confirmed that our code
reproduces the known test results (Figure 5).

Figure 4 A visualization on 40-side tiled display
in collaboration with Oyamada Lab.
at Kyoto University.

12

From the development experiences of the multi-GPU
hydrodynamics codes, we have learned that simple C-likecoding style will require too many repetitions of lines and2010 [8]) --- the mechanism similar to lightning on Earth. Also
Inutsuka-Sano (2005) [9] proposed a discharge mechanism arisingfrom elementary processes of Resistive MHD. The paper above
treated lightning in a one-zone model, but recently Okuzumi
has predicted three-dimensional instability arising from themechanism. To study how these phenomena will be active andsaturate, we need three-dimensional Resistive MHD simulations,
possibly coupled with dust component and chemistry. I would
continue to develop codes, making use of the advanced
computing techniques, to meet such goals.
Acknowledgments
Takayuki Muranushi’s research is supported by The Hakubi Centerof Kyoto University. We are deeply grateful to Tsuyoshi Hamada,
Takayuki Aoki and many others for useful advices on coding.
We are also grateful to people working for JHPCN (Joint Usage /Research Center for Interdisciplinary Large-Scale Information
Infrastructures) Program through which this work was madepossible.
References[1] Field G. B., Goldsmith D. W., Habing H. J., (1969), AstrophysicalJournal Letters, 155, L149+[2] Chepurnov A., Lazarian A., (2010), Astrophysical Journal, 710,
853
[3] Koyama, H., & Inutsuka, S. (2000), Astrophysical Journal, 532,
Figure 5
An MHD shock-tube test problem,
corresponding to Miyoshi & Kusano
2005[7] Fig. 8. Equations of MHD is
integrated upto t = 0.1 with initial conditions(ρ, p, v x, v y, v z, By, Bz) = (1, 1, 0, 0, 0, 1, 0) on the left,(ρ, p, v x, v y, v z, By, Bz) = (0.125, 0.1, 0, 0, 0, -1, 0) on the right,
and Bx = 0.75. Each of the figure above contains 256 meshes.

980[4] Inoue, T. & Inutsuka, S. (2008), Astrophysical Journal, 687:303310[5] Godunov, S. K. (1959), Matematicheskii Sbornik, 47, 271-306[6] van Leer, B. (1979) Journal of Computational Physics 32, 101-136[7] Miyoshi & Kusano, Journal of Computational Physics 208 (2005)
315–344[8] Muranushi, T. (2010), Monthly Notices of the Royal Astronomical

inacceptable. CUDA has C++ capabilities; but we have also
learned limitations of abstractions in C++ style. So we havedesigned Cprb (https://github.com/nushio3/cprb,) a simple code

Society. 401, 2641–2664[9] Inutsuka & Sano, The Astrophysical Journal, Volume 628, Issue 2,
pp. L155-L158 (2005)

generator for C-like programming languages. Cprb is written in
Haskell and one can meta-program C++ codes in ruby style. We
are writing current version of MHD code in Cprb; it is very fluent.
We hope that Cprb contribute a step in search for new schemesof parallel programming.
I would definitely like to solve the mysterious discharge
phenomena in protoplanetary disks. Say, it’s study of spacethunderstorms that come after studying space clouds. I have infact proposed a discharge mechanism powered by collisionalcharging of ice particles in protoplanetary disks (Muranushi,

13

Fast Fourier Transformusing GPU
Akira Nukada**Global Scientific Information and Computing Center, Tokyo Institute of Technology

Fast Fourier Transform (FFT) is one of the most important computations used in various fields
from multimedia to large-scale simulation.
Therefore, speed-up of the FFT computation benefits many people.
TSUBAME 2.0 compute nodes equips GPUs which accelerates
many kinds of computations. In this column, latest status of GPU FFT is presented.

Introduction

1

GPU has very high floating point operation performance aswell as high memory bandwidth, which is a large advantage formemoryintensive computations. Indeed, many real applications inhigh-performance computing areas are categorized as memoryintensive computations.
The first line and third line correspond to L-point DFT and

Fast Fourier Transform

2

M-point DFT, respectively. The second line is called multiplicationby twiddle factors. Thus, an N-point DFT can be divided into L
M-point DFTs and M L-point DFTs and some additional operations.
This is the base of Fast Fourier Transform (FFT) algorithm tocalculate DFT efficiently. This transformation can be applied

The Discrete Fourier Transform (DFT) is used in many fields

recursively. If N is a highly composed number, the number ofof science and engineering. This is a transformation between

floating-point operations can be reduced to O(N logN). In thisphysical or time space and frequency space. DFT is used notcase, the computation of DFT is now memory-bound.

only in large-scale simulations such as Molecular Dynamics, but

Actually, FFT is well known as one of the most memory-also in commodity multimedia applications like audio and video

intensive computation in typical benchmark applications. Each

encoding/decoding.

compute nodes of TSUBAME 2.0 has two Intel Xeon X5670

DFT calculates N outputs Y(k) from N input values X (k) as
described below:

Processors (Westmere-EP 2.93GHz, six cores), which provides
32GB/s with triple-channel DDR3-1333 memory. However, thisbandwidth is not sufficient for the computation of FFT.
TSUBAME 2.0 compute node has three GPUs (NVIDIA Tesla
M2050) in addition to the CPUs. The memory bandwidth of theGPUs is about 150GB/s which is more than four times as much

X(k) and Y (k) are f loating-point complex numbers.
The computation of all outputs requires O(N2) floating-point
operations. However, this computation includes many redundant
operations when N is L multiplied by M. To clarify it, the formula
is transformed as follows.

14

as that of CPUs. The GPU is a main computation resource of
TSUBAME 2.0.

GPGPU

3

Performance of FFTusing CPU or GPU

4

One of the latest trends in HPC is generalpurpose computation

Figure 1 shows the performance of FFT using a CPU or a GPU

using GPU (GPGPU). Current GPUs have sufficient programmability

on TSUBAME 2.0 compute nodes. Performance is calculated in

to implement not only graphics operations but also more generic

GFLOPS, where the number of floating-point operations required

computations efficiently. Thus, now GPU is recognized as ain computing an N-point FFT is assumed to be 5N logN.

powerful, inexpensive, low-power computation resource.

We used Intel MKL Library 10.2.5 and FFTW 3.1.2 as FFT

In 2006, one of the major GPU vendor NVIDIA introduced

libraries using CPU. Both of them are installed on TSUBAME 2.0

CUDA, which is a new GPU architecture and programming tools

system by default. To measure the performance, all of the six coresfor GPGPU. Since it is much easier than prior conditions, manyof a CPU are used by multi threading. For FFT library for GPUs, weof compute intensive applications especially in HPC area wereused NVIDIA’s CUFFT library 3.1 and our NukadaFFT library 1.0. The

ported to GPU using CUDA and achieved drastic speed-ups.

CUFFT library is provided as a part of the CUDA toolkit packages as

In general, graphics operations do not require highwell as BLAS library and so on.

accurac y. Therefore, GPU suppor ted only single-precision

Performance of FFTW library is almost the same as thatfloating-point operations. For this reason, porting applicationsof MKL library. Of course, their implementation is quite different

to GPU is always involved with accuracy issues. But latest GPUs

from each other. However, the achieved performance is basicallysupport double-precision floatingpoint operations which enables

limited by the memory access.

compatible computations with the latest CPUs.
The advantage of GPU computing is not only the highOn the other hand, the performance of NukadaFFT libraryis much higher than CUFFT library.

performance floating-point operations. GPU has also high memory

This means, the FFT routines in double precision is not wellbandwidth to transfer the data required in the computation. The

optimized yet. Actually, its performance in single precision is very

memory bandwidth of the Tesla M2050 GPUs is about 150GB/s,

high. Of course, the performance using GPU is much higher thanwhich corresponds to more than four times as much as CPU. Thisthat using CPU.

will be a strong advantage for FFT computations.

Figure 1
This graph shows performance (GFLOPS) of
128-point, 256-point, and 512-point FFTwith batch=65,536, in double precision.
Intel MKL Library 10.2.5 and FFTW library 3.1.2
use a six-core CPU (Westmere-EP, 2.93GHz),
and NVIDIA CUFFT library 3.2 andNukadaFFT library 1.0
use a GPU (NVIDIA Tesla M2050).

15

Fast Fourier Transform using GPU

In the case of TSUBAME 2.0 compute nodes, Tesla M2050

T here can b e many k inds of GPU implementation

GPU has about 4.6 times theoretical peak memory bandwidth

to compute FFTs. To achieve high performance, we have tothan CPU. However, the speed-up by using GPU reaches about

implement programs for each transform size.

seven times in FFT computation. This is because there is a large

GPU model is another large fac tor. When we were

gap between theoretical peak and achieved memory bandwidth.

developing the essential part of the NukadaFFT library, neither

The memory bandwidth of the GPU is about 150GB/s, however

Tesla M2050 nor GeForce GTX 480 existed. The development

the achieved memory bandwidth in FFT is only 82GB/s. Since thewas done on older-generation GPUs such as GeForce GTX 280.

memory of the Tesla GPU is protected by ECC, additional data

Now the library shows competitive performance even on nexttransfer of ECC code consumes a part of the memory bandwidth.

generation GPUs. This is because of the auto-tuning features ofWrites to the GPU memory is slower than reads from the GPU

the library.

memory. In case of FFT computation, about 50% of memory

The auto-tuning strategy used in this library is exhaustive.

access is write access. This degrades the data transfer efficiency.

The library generates many kinds of FFT routines, executes them,This is common in the case of CPU. Simple memor y copy

and selects the best one. We employ three tuning parameters as

operation can achieve slightly less than 20GB/s, and the data

follows.

transfer ratio in FFT computation is less than 12GB/s.
Factorization The key of the FFT algorithm to reduce the number

Auto-tuning

5of floating-point operations is the factorization of the transformsize. Factorizing to prime numbers does not always show goodresults. In case of GPU implementation, it is better that the gap
between the largest and smallest factors is small. Otherwise, theSIMDization for GPU wastes many computation resources.

There are many kinds of GPU products. The number of CUDAcapable GPU models already exceeds a hundred including

Number of threads It is important to exploit the high memory

GeForce series, Quadro series, Tesla series, ION series and mobile

bandwidth of GPUs. It depends not only on the access patterns

GPUs. Table 1 shows specificaions of some major GPU models

but also on the number of threads running simultaneously. We

we used. Although the performance of GPUs rapidly increaseneed to choose the best number of threads.

year by year, they vary in number of cores, performance, memory
bandwidth, etc.

cache memory to store temporal data which will be used again

use, and have high memory bandwidth. On the other hand,

immediately. GPU implementation uses onchip shared memory

Tesla series are designed for high performance computing. Their

to exchange the data between threads. The shared memory candoubleprecision performance is much higher than GeForcebe accessed simultaneously by multiple threads since it consists

series. The memory capacity is also larger, and protected byof multiple banks. Therefore, we have to select the access pattern

ECC. Needless to say, the Tesla series is selected for TSUBAME 2.0

carefully to avoid a bank conflict. To do this, the library inserts

compute nodes.

padding automatically in specific patterns.

Table 1 The specifications of major CUDA-capable GPUs.

16

Shared memory access pattern CPU implementation of FFT uses

GeForce series are designed for gaming and desktop

Although the time spent for the auto-tuning essentiallydepends on the transform size, GPUs, CPUs, and so on, we

References

observed it completes within a minute in most cases. Generally,[1] James W. Cooley and John W. Tukey, “An Algorithm for theapplications repeatedly calls FFT functions with the same

Machine Calculation of Complex Fourier Series”, Math. Comput.

transform sizes. In such a case, the length of the auto-tuning timeVol. 19, pp. 297-301, 1965.

is acceptable enough. By default, the results of the autotuning
procedure, that is, the best parameters selected will be stored in
the database file. Therefore, applications start computation using[2] Charles Van Loan, “Computational frameworks for the fast
Fourier transform”, SIAM Press, Philadelphia, PA, 1992.
[3] Matteo Frigo and Steven G. Johnson, “ The Design and
Implementation of FFTW3”, Proceedings of the IEEE, Vol. 93,

them immediately them, except on the first time.
It is not true that GPU applications always require autotuning features to achieve high performance. In many cases, theNo. 2, pp. 216–231, 2005. special issue on ”Program Generation,
Optimization, and Platform Adaptation”.

GPU model used for the GPU application are limited and the[4] Akira Nukada, Yasuhiko Ogata, Toshio Endo, and Satoshi

same kernel can achieve good performance for all of them. Auto-

Matsuoka, “Bandwidth Intensive 3-D FFT kernel for GPUs

tuning mechanism is important especially for libraries which willusing CUDA”, In Proceedings of the ACM/IEEE conference

be publically available because these libraries should work on

on Supercomputing (SC08), Austin, IEEE Press, Pages 1- 11,

varying GPU models with acceptable performance.

November 2008.
[5] Akira Nukada and Satoshi Matsuoka, “Auto-Tuning 3-D FFT

Summary

6

Library for CUDA GPUs”, In Proceedings of the ACM/IEEE
conference on Supercomputing (SC09), Portland, ACM, Page
1-10, November 2009.

We introduced the latest status of FFT computation using GPUs.
In case of the CPUs and GPUs on TSUBAME 2.0 compute nodes,
the use of GPU resulted in about 7 times speedup due to thehigh memory bandwidth of the GPUs. This is not only for FFT but
also many kinds of memory-intensive computations like CFDs.
Needless to say, GPUs also accelerate many compute-intensive
applications. In the near future, we hope many new applications
will be ported to GPUs.
NukadaFFT library is still updated frequently. The latest
version is available at the following
URL. http://matsu-www.is.titech.ac.jp/˜nukada/nufft/
Acknowledgments
This work is partially supported by Core Research of Evolutional
Science and Technology (CREST) program of Japan Science andTechnology Agency (JST) “ULP-HPC: Ultra Low-Power, HighPerformance Computing via Modeling and Optimization of Next

● TSUBAME e-Science Journal No.3

Generation HPC Technologies”, by Microsoft Technical Computing

Published 02/25 / 2011 by GSIC, Tokyo Institute of Technology ©
ISSN 2185- 6028
Design & Layout: Kick and Punch
Editor: TSUBAME e-Science Journal - Editorial room
Takayuki AOKI, Toshio WATANABE, Masakazu SEKIJIMA,
Thirapong PIPATPONGSA, Fumiko MIYAMA
Address: 2-12-1 i7-3 O-okayama, Meguro-ku, Tokyo 152-8550
Tel:+81-3-5734-2087 Fax: +81-3-5734-3198
E-mail: tsubame_j@sim.gsic.titech.ac.jp
URL:
http://www.gsic.titech.ac.jp/

Initiative “HPC-GPGPU: LargeScale Commodity Accelerated
Clusters and its Application to Advanced Structural Proteomics”,
by NVIDIA CUDA Center of Excellence Program, and by MEXT
Grant-in-Aid for YoungScientists (A) 22680002.

17