CCS Report: PART I, Overview of Center for Computational Sciences
Summary of Activities 2004-2007

CCS Report: PART I

Overview of Center for Computational Sciences
Summary of Activities 2004 – 2007

October 2007

Center for Computational Sciences
University of Tsukuba-1-

CCS Report: PART I, Overview of Center for Computational Sciences
Summary of Activities 2004-2007-2-

CCS Report: PART I, Overview of Center for Computational Sciences
Summary of Activities 2004-2007

Table of Contents
1. Introduction ........................................................................................................ 4
2. Mission and Overview of the Center for Computational Sciences ............................. 5
3. Chronology of CCS ................................................................................................ 6
3.1 Brief history prior to 2004 .................................................................................. 6
3.2 Reorganization in 2004 ...................................................................................... 6
3.3 The PACS-CS Project........................................................................................ 8
3.4 Building ......................................................................................................... 11
4. Organization and Administration ........................................................................... 12
4.1 Organization ................................................................................................... 12
4.2 Director.......................................................................................................... 13
4.3 Faculty of the Center ....................................................................................... 13
4.4 Running of the Center...................................................................................... 13
4.5 Faculty Selection............................................................................................. 14
5. Overview of Research Activities from 2004 to 2007 ....................................... 15
5.1 Overview........................................................................................................ 15
5.2 PACS-CS Project ............................................................................................ 16
5.2.1 Design Concept of the PACS-CS System ..................................................... 16
5.2.2 Implementation of PACS-CS System ........................................................... 18
5.2.3 Operations of the PACS-CS System............................................................. 19
5.2.4 Scientific Research and Results using PACS-CS........................................... 20
5.3 FIRST Project ................................................................................................. 22
5.3.1 FIRST Simulator ....................................................................................... 22
5.3.2 Performance.............................................................................................. 23
5.4 ILFTNet and ILDG Project............................................................................... 25
5.4.1 ILDG: International Lattice Data Grid ......................................................... 25
5.4.2 LQA: Lattice QCD Archive ........................................................................ 26
5.4.3 JLDG: Japan Lattice Data Grid ................................................................... 26
5.5 Toward Peta-scale Computing .......................................................................... 27
5.6 Summary of Interdisciplinary Collaborations in the Center .................................. 28
6. Inter-university Activity, Education and Public Relations ......................................... 31
6.1 Interdisciplinary Computational Science Promotion Program............................... 31
6.2 Other Inter-university Activities and Contributions to Computational Science
Communities........................................................................................................ 33
Appendix A. Faculty Members of CCS................................................................ 35
Appendix B. Facilities of CCS ............................................................................. 48-3-

CCS Report: PART I, Overview of Center for Computational Sciences
Summary of Activities 2004-2007

1. Introduction
This report summarizes the main events and activities of the Center for Computational
Sciences, University of Tsukuba, since its foundation in April 2004 to the present. The
report has been prepared for a review of the Center, scheduled from 30 October to 1
November of 2007.
The purpose of the review is to receive an outside examination of the research activities
and their outcomes at the Center in view of the founding objectives, and to incorporate therecommendations from the review into future developments of the Center.
Since details of the research results are prepared as a separate report, this report focuses onan overview, which includes a brief overview and the mission of the Center (Section 2), achronology (Section 3), organizational aspects (Section 4), and a brief summary of theresearch activities and collaborations of the Center (Section 5).
More details of the research activities and results of each division and group are describedin CCS Report: PART II. The future plan and vision are described in CCS Report: PART
III.

-4-

CCS Report: PART I, Overview of Center for Computational Sciences
Summary of Activities 2004-2007

2. Mission and Overview of the Center for Computational Sciences
The mission of the Center of Computational Sciences is to promote scientific discovery bycomputational science through the application of advanced computing technologies, and tosupport research of computational science in Japanese universities and institutes by
operating leading-edge computing systems. The Center for Computational Sciences wasfounded in April 2004 as an inter-university research facility for computational science by
restructuring and expanding the Center for Computational Physics (April 1992-March
2004). The Center aims to carry out research on critical issues in fundamental science,material science, life science and environmental science by performing large-scale
simulations and large-scale data analyses. To realize this goal, the Center conducts researchand development of high-performance computing systems and networks, and advanced
studies in computer and information science. Because the Center is an inter-university
facility, it thus functions as an international as well as a national center for computational
sciences.
High-performance computing systems and a high-speed network infrastructure are
essential tools for carrying out successful computational science research. To develop
computational science, it is vital to establish an organization that combines the forefront ofscience research with that of computer science and information science. The Center for
Computational Sciences has been striving to carry out such collaborative research. For
example, by collaboration between physicists and computer scientists, we performed R&D
of a massively parallel computer CP-PACS (ranked as the No. 1 system in the Top 500 Listof November 1996) by collaboration between physicists and computer scientists.
Large-scale computations were carried out on CP-PACS resulting in significant progress inparticle physics and astrophysics. In July 2006, a new massively parallel cluster PACS-CS
started operation. This machine was built using special funds for education and researchfrom the Japanese Government. We aim to develop computational sciences for a wide rangeof scientific disciplines, from fundamental science to material, life and global
environmental sciences by making the best use of PACS-CS.

-5-

CCS Report: PART I, Overview of Center for Computational Sciences
Summary of Activities 2004-2007

3. Chronology of CCS
3.1 Brief history prior to 2004
The chronology of the Center for Computational Sciences goes back to 1992, when theCenter for Computational Physics (CCP) was founded at the University of Tsukuba. The
purpose of the CCP was (i) to carry out research and development of parallel computers
suitable for large-scale computational physics calculations, and (ii) to carry out researchwith the developed system in the area of particle physics, astrophysics, and condensed
matter physics. In particular, CCP was to serve as the base for the project entitled“Research of Field Physics with Dedicated Parallel Computers” (FY1992-FY1996) fundedby the Ministry of Education of the Japanese Government. This project developed amassively parallel system CP-PACS (Computational Physics with Parallel Array Computer
System). This system achieved 368.2 Gflops for the Linpack Benchmark in September
1996, and was ranked as the No. 1 system in the November 1996 Top 500 List.
CCP had a faculty of 10 researchers in four divisions: Computational Particle Physics,
Computational Astrophysics, Computational Condensed Matter Physics and Parallel
Computer Engineering. The simultaneous existence of physics and computer sciencefaculties within CCP, and actual close collaborations between them for the development ofparallel computer systems and their usage, marked the unique feature of the Center. In fact,
this interdisciplinary approach goes back to the early 1980s, when research anddevelopment of parallel computer systems started at University of Tsukuba under Professor
Tsutomu Hoshino.
From 1996 to 2005, CP-PACS served as the main computing facility of CCP. In the late
1990s, large-scale calculations using CP-PACS produced a number of well-known results.
The major scientific achievements are as follows:
1. Calculation of the quenched light hadron spectrum in lattice quantum
chromodynamics (QCD). ( S. Aoki et al., Physical Review Letters 84 (2000) 238)
2. First systematic lattice QCD calculation including light dynamical up and downquarks, showing that up and down quark masses are much lighter than previously
thought. (S. Aoki et al., Physical Review Letters 85 (2000) 4674)
3. First astrophysical radiation hydrodynamical calculation of the reionization of the
Universe including interaction of radiation and matter. ( Monthly Notice of the Royal
Astronomical Society 321 (2001) 593)
4. First many-body quantum mechanical calculation of the phases of solid hydrogen. (H.
Kitamura et al., Nature 404 (2000) 259)
3.2 Reorganization in 2004
In April 2004 the Japanese national university system, of which University of Tsukuba isa member, underwent a major transition. While previously the national universities were
-6-

CCS Report: PART I, Overview of Center for Computational Sciences
Summary of Activities 2004-2007

part of the Japanese government and were strictly controlled by the Ministry of Education,
Science, Culture and Sports, their legal status was changed to that of independent institutes.
While the budget still came from the government, each university was granted much wider
freedom of action on its own. The reason behind this change was pressure towardreformation of the Japanese government, including not only a reduction of employees andgovernment spending, but also reformation of the university system to be more effective inthe era of globalization and worldwide competition.
Preparations for the transition started in FY2002 at all national universities across Japan.
The Center for Computational Physics considered the transition as a welcome opportunity
to reexamine its future plan. While the Center for Computational Physics, with its small
faculty of a dozen members, had functioned well in pursuing research projects within alimited area of fundamental physics, it was strongly felt that the interdisciplinary approach
toward computational physics pursued at the Center could be extended to wider areas ofscience, and that the Center had a potential to significantly expand its activity by doing so.
Discussions with the Executive Office of the University on a possible reorganization andexpansion started in the early summer of 2002. The Executive Office basically welcomed
the Center’s proposal, and provided strong support during negotiations with the Ministry of
Education, Science, Culture and Sports to secure governmental approval from the winter of2002 to spring of 2003.
The Center for Computational Physics was formally reorganized as the Center for
Computational Sciences on 1 April 2004. The faculty expanded threefold from 11 to 31
academic members, and the following reorganization and expansion of divisions wereaccomplished:• Division of Particle Physics and Division of Astrophysics were combined into asingle division with an increased faculty of 7 members.
• Division of Condensed Matter Physics and Division of Biophysics were combined
and significantly expanded to 10 faculty members in the Division of Materials andLife Sciences.
• A new Division of Global Environment and Biological Sciences was introduced with
3 faculty members.
• Division of Parallel Computer Science was expanded from 3 to 5 faculty members,
and was renamed Division of High Performance Computing Systems.
• A new Division of Computational Informatics with 6 faculty members was set up to
carry out research on computational intelligence and media.
Of the 20 members added to the original 11 for the Center for Computational Sciences, 14
were faculty members who were actively working in various areas of computational scienceat graduate schools of the university, while the remaining 6 were approved by theuniversity and filled by new recruitments in 2004-2005.

-7-

CCS Report: PART I, Overview of Center for Computational Sciences
Summary of Activities 2004-2007

Founded in 1996, re-organized in 2004
Faculty
31
Professor
11
Associate Prof. 12
Assistant Prof. 8Center for
Computational Sciences
Division of Particle Physics
and Astrophysics

Graduate School of Pure andApplied Sciences• Dept. of Physics• Dept. of Frontier Science• Dept. of Materials Science

Division of Materials andLife Sciences

Graduate School of Life andEnvironmental Sciences

Division of Global Environmental
and Biological Sciences• Dept. of Geoenvironmental Sciences• Dept. of Structural Biosciences

Division of High Performance
Computing Systems

Graduate School of Systems andInformation Engineering• Dept. of Computer Science• Dept. of Intelligent Interaction Technologies

Division of Computational
Informatics

Figure 3.1: Overview of CCS Organization
3.3 The PACS-CS Project
Since advances in high-performance computing systems are rapid, planning toward
developing the successor to the CP-PACS system was started immediately after its
completion in 1996. One effort was the “Development of Next-Generation Massively
Parallel Computers'' project of “Research for the Future Program'' (FY1997-FY2001)
funded by the Japan Society for Promotion of Sciences. This project led to two importantresults: (1) the introduction of SCIMA, an on-chip memory architecture for controlling thememory wall problem in large-scale science and engineering calculations, and (2) the firstrealization of a heterogeneous multi-computer system to deal with calculations which
involve both local as well as long-range forces necessitating completely different
calculation schemes. This result later developed into the FIRST project.
By 2004, when the Center for Computational Sciences was founded, the CP-PACS system
was in its eighth year of operation. The needs to significantly enhance the computational
facility of the Center were dire both due to the aging of the CP-PACS and the expanded
faculty and research areas of the Center.
The PACS-CS project was an attempt to meet these needs. An intensive discussion by asmall group to formulate a basic architecture took place in the summer of 2003. In view of
the rapid progress of commodity technologies, a cluster approach based on a commodity
CPU and network was considered suitable for quick development of the system necessary
to meet the computational requirements of the Center’s researchers. At the same time, thesystem design had to achieve a good balance of floating point performance, memory
throughput and network throughput in order to secure a high overall sustained performancefor large-scale scientific calculations.
The basic design was drawn at the end of the summer of 2003. An initial presentation of-8-

CCS Report: PART I, Overview of Center for Computational Sciences
Summary of Activities 2004-2007

the project to the government was made in the winter of 2004. A formal funding requestfrom the university was made in the summer of 2004, and the government decision process
ran through the fall of 2004, and included a review by the Committee for Science andTechnology Planning (CSTP). CSTP is the highest office of the government for planningscience and technologies. The Committee recommended the funding of the project to startfrom FY2005, and a formal announcement from the government was made in December of
2004.
The entire process was not trivial, since 2004 was the first year after the change of thenational university system; so, the entire funding system for research was cast into a newform, and our proposal had to go through this new system.
Formally, the project has been funded by the ``Special Fund for Education and Researchfor National Universities'' for the period of FY2005–FY2007. The official title of the
project is “Discovery, Synthesis and Emergence of Novel Knowledge throughComputational Sciences.'' The goal of the project is to carry out leading-edge research incomputational sciences through a close collaboration of researchers of various science
disciplines and computer science. In particular, the project plans to foster aninterdisciplinary style of research in the area of materials and life sciences, and global
environment and biological sciences, which were either strengthened or newly introducedwith the reorganization of the Center. As a basic tool to carry out such research, the project
proposed to develop a massively parallel cluster system. The system is named Parallel
Array Computer System for Computational Sciences, or PACS-CS.
The project formally started in April 2005. A governmental bidding procedure wasfollowed to decide the manufacturer of the system, and Hitachi Ltd. was selected as themain contractor of the system hardware and software in July 2005, and Fujitsu Ltd. wasselected to develop special network software in August 2005. The development of
PACS-CS was carried out by a close collaboration of the researchers of the Center and thetwo vendors.
The PACS-CS system consists of 2560 nodes, each carrying a 5.6 Gflops Intel Xeon
processor and a 2 GBytes of memory. The nodes are connected by a 3-dimensional
hypercrossbar network, each link consisting of a dual Gigabit Ethernet, with a total of 750
MByte/sec bandwidth for each node. The Linpack Benchmark achieved 10.35 Tflops,
ranking 34th on the June 2006 Top 500 List. The full system started operation on 1 July
2006, and a formal commissioning was made on 3 September 2007. The system is now
fully saturated and running a variety of applications.

-9-

CCS Report: PART I, Overview of Center for Computational Sciences
Summary of Activities 2004-2007
1992 April

1996 OctoberNovember
1997 April

2002 April

2003 July-Dec.
2004 April
2005 April

Founding of Center for Computational Physics (CCP)(10 year term / 10 faculty members and 3 visiting faculties)
development of massively parallel computer CP-PACS begins
Massively parallel computer CP-PACS completed
Ranked as No. 1 in the Top 500 World Supercomputer List
JSPS research for the Future Project “Computational Science”「Development of Next-Generation Massively Parallel
Computers」 begins
The Second 10 year term of Center for Computational
Physics begins( 11 faculty members and 3 visiting faculties)
Planning on reorganization and expansion of CCP
Founding of Center for Computational Sciences (CCS)(31 faculty members and 3 visiting faculties)
Development of Massively Parallel Cluster PACS-CS in the project「Discovery, Synthesis and Emergence of Novel Knowledge throughComputational Sciences」 begins （3years, FY2005〜FY2007)

Figure 3.2: Brief chronology of CCS- 10 -CCS Report: PART I, Overview of Center for Computational Sciences
Summary of Activities 2004-2007

3.4 Building
Another serious problem which arose from the expansion of the Center was the shortageof office space. A budget request for an expansion of the main research building was
submitted to the government in the summer of 2005. The request was approved in the
supplementary budget of the government for FY2005. The research building, after acomplete remodeling, was completed in March 2007. The new building provides floor
space of 2356 m2, which is more than double the space of 1090 m2 of the old building.

Figure 3.3: Photo of CCS building- 11 -CCS Report: PART I, Overview of Center for Computational Sciences
Summary of Activities 2004-2007

4. Organization and Administration
4.1 Organization
The organization of the Center is shown in Fig. 4.1. The Center consists of five researchdivisions, a Division for Inter-university Activity and the Administrative Office under the
Director.

Director
Deputy director
Steering
CouncilSteering
Committee

Related
Graduate
Schools

Administrative
OfficeResearch
Evaluation board

Particle Physics and Astrophysics
Material and Life Sciences

Research
DivisionsGlobal Environment andBiological Sciences

Meeting of
Researchers

High Performance
Computing Systems

Collaborative
researchers

Computational Informatics
Division of Inter-university activities

Figure 4.1: Organization of the Center for Computational Sciences
The Steering Committee consists of full professors of the Center and representatives fromthe three graduate schools to which the faculty members of the Center belong (these are theGraduate School of Pure and Applied Sciences, Systems and Information Engineering, andGlobal environment and Life Sciences). This committee supervises the day-to-day runningof the Center.
The Steering Council consists of outside experts in various areas of computational
sciences appointed by the Director of the University, the Director and the division heads of
the Center. It supervises and coordinates the activities of the Center from a wider
perspective of computational sciences.

- 12 -CCS Report: PART I, Overview of Center for Computational Sciences
Summary of Activities 2004-2007

4.2 Director
The Director of the Center has to hold the rank of professor of University of Tsukuba. The
Director is appointed by the President of University of Tsukuba upon recommendationfrom the previous Director; in practice, a candidate is selected by a ballot of the Center
faculty. The appointment is for two years and can be repeated up to six years.
The past directors are Prof. Yoichi Iwasaki (physics) from April 1992 to March 1998 andProf. Akira Ukawa (physics) from April 1998 to March 2007. The current director sinceApril 2007 is Prof. Mitsuhisa Sato (computer science).
4.3 Faculty of the Center
At University of Tsukuba, all faculty members belong to one of seven graduate schools.
The faculty members of the Center ``belong'' to a graduate school and ``work'' at the Center.
The affiliation of a faculty member is written as ``Graduate School (Center for
Computational Sciences)''.
The faculty members as of 1 October 2007 are listed in Appendix A.
4.4 Running of the Center
The research activities of the Center are coordinated by the Director through the following
three meetings, and support is provided by the Administrative Office. The organization andthe roles of the three meetings are as follows.
•
Meeting of Researchers
This meeting consists of the entire Center faculty and the Associated Research Fellows,
who are collaborators of other universities and research institutes. This meeting ischaired by the Director of the Center and is held every month.
At the meeting, all aspects of research are discussed, such as the status of ongoingprojects, procurement of equipment and operation of the Center computer system.
•

Steering Committee
The Director of the Center chairs the committee, which holds meetings every month.
The committee discusses important issues for running the Center, which include mattersrelated to the organization of the Center, selection of faculty members, budget planningand confirmation of expenditures.

•

Steering Council
The council is held once a year, and is chaired by the Director of the Center.
The council hears annual reports of the Center research activities, and discusses the
direction of research of the Center and matters related to inter-university use of theCenter facility. Under the Steering Council, a Research Evaluation Board is organizedto evaluate the research activities annually.

- 13 -CCS Report: PART I, Overview of Center for Computational Sciences
Summary of Activities 2004-2007

4.5 Faculty Selection
The Center appoints new faculty members when openings become available among theCenter faculty positions. The selection procedure is as follows.
• Choice of field
When an opening of a position becomes available, the field of the position is chosen
by the Steering Committee based on considerations of the research purposes of theCenter and the balance among research fields.
•

Selection of candidate
Once the field is chosen, the actual selection process is handled by the Faculty
Selection Committee under the Steering Committee.
The opening is publicly announced to invite applications. Candidates are initially
screened by a Search Committee made of five professors appointed by the Faculty
Selection Committee. This screening includes reading the candidates' main papers
and interviews. When the screening at the subcommittee converges toward a
candidate, the conclusion is reported to the Faculty Selection Committee. If theFaculty Selection Committee approves the candidate, the result is reported both to
the Steering Committee and the University Personnel Committee for formal
approval.

- 14 -CCS Report: PART I, Overview of Center for Computational Sciences
Summary of Activities 2004-2007

5. Overview of Research Activities from 2004 to 2007
5.1 Overview
The reorganization of the Center in 2004 represented a major change in the way projects
are organized and executed. While particle physics and astrophysics already had a long
history of collaborating with computer science, other fields new to the Center, such as
materials and life sciences and Global environment and biological sciences, had no suchinteraction. Thus, a common target that could be shared by these new areas of science andcomputer science had to be formulated and the two groups had to learn to work together.
In this regard, the PACS-CS project (2005–2007) occupied the central place in the
research activities of the Center. The project aimed not only on developing a new clustercomputer for computational sciences, but it also tried to foster interdisciplinary
collaborations. The project involved all Center faculty members.
The research groups of the Center also pursued a number of large-scale projects of theirown. These projects attempted to take advantage of the coexistence of science andcomputer science faculties to advance research in the respective science fields. The science
projects that fall into this category are as follows:•
The FIRST project (2004–2007)
This project developed a large-scale, heterogeneous multi-computer system, which
integrates GRAPE-6, boards for gravity calculations into the node of a commodity
cluster. Using this system, the project attempted to clarify the physics of formation of
the first generation stars in the universe after the Big Bang.
•“Development of Nano-architecture through Computational Quantum Science”(2005–2009)This is a project in materials and life sciences which attempts to advance algorithmic
and computational study of the possible principles which determine the relation
between functions and spatial structure of nano- and bio-molecules. This project is
supported by the JST-CREST program.

The Center has been involved in the development of peta-scale computing. There are two
specific projects in this category:•“Development of Element Technologies for the Next-generation Supercomputers”(2005–2007)
With Hitachi Ltd., the project carries out research on possible CPU architecture forpeta-scale systems. The Center is attempting to further develop the on-chip memoryarchitecture along the line of SCIMA, and examine in more detail the low-power aspectof SCIMA as compared to vector architecture.
•“Development of the Next-generation Supercomputer” (2006–2012)- 15 -CCS Report: PART I, Overview of Center for Computational Sciences
Summary of Activities 2004-2007

This is a national project to develop a next-generation supercomputer reaching 10
petaflops. RIKEN (Research Institute for Science and Chemistry) has been selected forexecuting the project. In September 2006, RIKEN and University of Tsukuba signed acollaboration agreement on the next-generation supercomputer project, and a group ofcomputer scientists of the Center have been involved with the project as visiting fellows
under this agreement.
In the following, we summarize the main results obtained up to now within these projects.
We also summarize research activities carried out as interdisciplinary collaborations at theCenter.
5.2 PACS-CS Project
The official title of this project is “Discovery, Synthesis and Emergence of Novel
Knowledge through Computational Sciences.” It is funded by the “Special Fund for
Education and Research for National Universities” by the Ministry of Education, Culture,
Sports, Science and Technology (MEXT) over three years from FY 2005 to 2007. The total
amount is 2.3 billion Yen.
The project aims to advance the frontier of computational science through active
interdisciplinary collaboration between various areas of science and computer science. The
main component of the project is the research and development of the PACS-CS computer(Parallel Array Computer System for Computational Sciences) with which computations on
various large-scale computational science applications in advanced scientific research are to
be carried out.
The PACS-CS system is a massively parallel cluster system with 2560 computational
nodes. Unlike ordinary HPC cluster systems, the PACS-CS system is designed with a clear
view of the target applications.
This year is the last year of the project. The system itself has been completed and is underoperation for various important applications working toward scientific results.
5.2.1 Design Concept of the PACS-CS System
In the conceptual design of PACS-CS, we took the view that it is the main computational
engine to solve the fundamental issues of science in the areas covered at our Center, anddesigned the system as a semi-special-purpose system under the following concepts:
・ The PACS-CS system is the follow-up system of our previous CP-PACS system(1996–2005), and is to be designed for large-scale applications with massively parallel
algorithms.
・ The system is based on commodity technology both for CPU and the interconnection
network. However, we aim to realize an effective bandwidth for the memory andnetworking necessary for our important scientific calculations.
・ The primary scientific target of the project is real-space density function theory for- 16 -CCS Report: PART I, Overview of Center for Computational Sciences
Summary of Activities 2004-2007

material science and lattice QCD for particle physics, and the system is strongly
oriented to the fundamental algorithms for these problems.
Based on the above concepts, we decided to design the system based on “real-spacecomputational modeling” (see Fig. 5.1) with the following architectural features, which are
different from traditional HPC clusters:(1) Single processor node architecture
Each computational node is equipped with only a single CPU to utilize the fullbandwidth provided by the memory and network on the node. Although this differsfrom ordinary dual-socket HPC server nodes, we keep the same density by packing
two nodes on a 1U board in the system implementation.
(2) Multi-dimensional hyper-crossbar network with trunking of Gigabit Ethernet
For the interconnection network architecture, we selected the three-dimensional
hyper-crossbar network (3D-HXB), which allows direct mapping of physical lattices
to the array of nodes following real-space computational modeling. We applied adual-link Gigabit Ethernet for each dimension of the network link, which is controlled
by software. This inexpensive solution provides data transfer throughput similar to
expensive SAN such as Myrinet and Infiniband.
Direct Simulation of
Physical Space

Direct mapping fromdiscretization ofReal Space Modeling

Figure 5.1: Physics simulation by real-space computational modeling in which the physicalsystem is directly mapped onto a node array connected by a network.

- 17 -CCS Report: PART I, Overview of Center for Computational Sciences
Summary of Activities 2004-2007

5.2.2 Implementation of PACS-CS System
The basic specification of the PACS-CS system is as follows:
Table : Specification of PACS-CS
Number of nodes
Peak performance
Node construction
CPU
MemoryInterconnection
Link bandwidth
Local HDD
System size
Power (peak)
Shared File-Server

2560 (16x16x10)
14.3 TFLOPS
Single CPU / nodeIntel LV Xeon EM64T, 2.8 GHz, 1 MB L2 cache
DDR2 400 MHz, 2 GB/node (5.12 TB/system)
3D-HXB, GbE dual-link trunking/dimension
250 MB/s/dim.，750 MB/s/3-D
160 GB/node (RAID-1 mirroring)
59 racks
545 kW (full UPS support)
10 TB RAID-5 + external 100 TB RAID-5

SoftwareLinux Fedora Core 3, SCore ver.5, Intel Compiler, YAMPI &MPICH, PM-Ethernet/HXB driver

Following the basic design, we developed a new motherboard to implement the single
CPU/node configuration and a Gigabit Ethernet trunking to support the 3D-HXB
interconnection network with the same system density as the traditional IA-32 based HPC
clusters. Figure 5.2 shows the top-view of the motherboard (left) and the two-node
implementation in the 1U-sized rack mountable chassis (right) with 12 Gigabit Ethernet
ports.

Figure 5.2: PACS-CS motherboard (left) and mount on 1U chassis unit (right)- 18 -CCS Report: PART I, Overview of Center for Computational Sciences
Summary of Activities 2004-2007

・・・

10

・・・

・・・
・・・

・・・

・・・
・・・

・・・

・・・
・・・

16
・・・
・・・
16

Figure 5.3: 3D-HXB conceptual view (left) and the view of entire system
Figure 5.3 shows the conceptual view of the 3D-HXB network and a photo of the entire
PACS-CS system with 59 racks.
The 3D-HXB network can be configured as a 3-D torus of an arbitrary size that is
appropriate to map the real-space computational modeling. In actual operation, the entire
system is partitioned into multiple rectangular shapes to fit the logical node space requiredby the problem size, and multiple user jobs are performed simultaneously. The multi-link
trunking in each dimension, the multi-dimensional simultaneous data transfer, and therouting between different dimensions are performed by a specially developed network
driver for PACS-CS named PM-Ethernet/HXB. The maximum performance of the driveron 3-D simultaneous burst data transfer is up to approximately 80%. On MPI (YAMPI)
user-level communication, however, transferring messages over 100 KB is needed to hide
the latency caused by the Gigabit Ethernet protocol.
The implementation and installation of the hardware and software of the PACS-CS system,
including the first version of the PM-Ethernet/HXB driver, was completed at the end ofJune 2006, and its operation began in July 2006. The SCore cluster middleware employed
for the system has since been updated to include the check-pointing/restarting feature andimprovement of PM-Ethernet/HXB performance. The system went into full-specification
operation in the summer of 2007.
5.2.3 Operations of the PACS-CS System
We have been operating the PACS-CS system for more than one year, since July 2006. Inorder to avoid system failure and emergency system shutdown during the ordinary
operation time, the system is maintained every month mainly for pre-replacement ofmemory and hard disk drives based on system logging information. With this effort, theactual failure is a very low rate of 1~3 times per month on average.
While the system can be split into multiple partitions with various sizes, the basic partition- 19 -CCS Report: PART I, Overview of Center for Computational Sciences
Summary of Activities 2004-2007

is a 256-node size (1.4 TFLOPS) x 10 partitions. Depending on the demand, we configure
512 or 1024 node partitions by combining the basic ones or 64 or 128 node partitions bysubdividing them.
Since October 2007, the resource allocation of PACS-CS to users is regulated under theInterdisciplinary Computational Science Promotion Program. The partition size is changed
on a weekly or monthly basis to fully incorporate the demand for large-scale simulations
required by the program. Such fine-tuning of the system partitioning allows us to support
large-scale jobs that are difficult to execute at conventional supercomputer centers in Japan.
Because of the fine-tuned scheduling, the system always runs with a 100% workload.
5.2.4 Scientific Research and Results using PACS-CS
Research and results are as follows:・ Lattice QCD in particle physics
Lattice QCD calculation aims to elucidate the physics of the strong interactions basedon fundamental quarks and gluons. This field has been pursued since the early 1980s,
and a significant body of results already exists. However, one of the serious limiting
factors so far has been the rather heavy quark mass in the range of 50~100 MeV usedin the simulations. This necessitated a long chiral extrapolation to the physical up anddown quark mass of 3 MeV with ensuing systematic errors in the lattice QCD
predictions.
The specific goal of the PACS-CS project is to realize a realistic lattice QCD
calculation in the sense that (i) the dynamic effects of all three light quarks, up, down
and strange, are incorporated, and (ii) the masses of the three quarks, especially thoseof the lightest up and down quark, are reduced to their physical value. In order toachieve this goal, recent algorithmic developments based on domain decomposition
and multi-time step methods are incorporated, and a new set of lattice QCD codes
using the Wilson-clover quark formalism and optimized for PACS-CS are developed.
The PACS-CS calculations have already succeeded in reducing the up and down quark
mass from 70 MeV to 6 MeV on a large lattice of 3.2 fm with encouraging results forthe chiral behavior and hadron spectrum. Further efforts are underway to reduce thequark mass to the physical value.
・ Real-space density functional theory in materials science
We are developing a real-space density functional theory (RSDFT) method as apromising computational method for describing phenomena in real materials with avery large number of atoms, e.g., 10,000 or more, from the first principles of quantum
theory. RSDFT introduces a lattice in real space and calculates all quantities, such as
electron wave functions at each lattice point. This method has several advantages fornext-generation supercomputing: heavy all-to-all communications such as FFT are
unnecessary, and flexible boundary conditions on wave functions can be imposed at the- 20 -CCS Report: PART I, Overview of Center for Computational Sciences
Summary of Activities 2004-2007

boundaries of the computational cell, thereby expanding its applicability to various
materials. A newly developed code is now being fine-tuned on PACS-CS. Calculationsof ionization energies and electron affinities of 10,000-atom clusters are being
processed with the goal to pave the way toward a unified approach for molecules,
clusters and bulk materials.
・ Real-time density functional theory in materials science and many-body theory
We have succeeded in a first-principles calculation of the electron dynamics forlaser-induced breakdown of dielectrics. When an ultra-intense laser field is irradiated tothe dielectrics, a large number of electron-hole pairs are excited even for a visible lightwhich should be transparent in large-gap dielectrics. These excitations make theresponse of the dielectrics metallic, and are employed for laser machining and surgeries.
We have shown, for the first time, that this process of optical dielectric breakdown canbe naturally described by solving the time-dependent Kohn-Sham equation. The resultshows the energy transfer from the laser pulse to the electrons in diamond as a functionof the maximum intensity of the laser pulse. Around 1015 W/cm2, a substantial increaseof the energy transfer is seen where the dielectric breakdown occurs. A portion of thiscalculation is achieved by PACS-CS.
・ Global circulation model analysis on PACS-CS
We exchanged a research contract with the Center for Climate System Research(CCSR) of the University of Tokyo in order to collaborate on the development of thenext-generation cloud resolving non-hydrostatic icosahedral general circulation model(NICAM). In the NICAM of CCSR/FRCGC, the dynamic core of the model isformulated specifically for massively parallel processing using MPI, which fits
perfectly with the PACS-CS. The NICAM can extend the model resolution to a 3.5 km
global mesh with cloud resolving physical processes. The testing phase with an aqua
planet model (no topography) has been completed successfully, and full terrain has justbeen installed in the model. Using the latest version of the NICAM with topography,
near real-time prediction experiments are being conducted on PACS-CS using thereal-time GPV/JMA data archive developed and maintained at the Center.
・ Regional weather model analysis on PACS-CS
The Weather Research and Forecasting (WRF) regional model was progressively
developed under the PACS-CS project. Coupled with the real-time GPV/JMA data, thereal-time regional weather prediction system was constructed. A test run was conducted
for the explosive cyclogenesis event that occurred on 7 January 2007 to confirm that thesystem is useful for research as well as for educational purposes.
・ Phylogenetic multi-gene analyses on PACS-CS- 21 -CCS Report: PART I, Overview of Center for Computational Sciences
Summary of Activities 2004-2007

The MPI version of a phylogenetic search program, Tree-Puzzle, was developed for
PACS-CS. This program has been utilized for multi-gene analyses under separate
model conditions. By using MPI-Puzzle, the exact ML trees will be exhaustively
selected from all possible trees, and subsequently compared with the trees that are
heuristically searched for.
5.3 FIRST ProjectThis project is a collaboration between astrophysicists and computer scientists, aiming at
an elucidation of the evolution of first-generation objects through large-scale radiationhydrodynamic (RHD) simulations with a high-performance hybrid computer system. The
project is funded by the Specially Promoted Research in Grants-in-Aid for Scientific
Research over four years (2004~2007) with a budget of JPY 329.5 million (US $2.8
million), approved by the Ministry of Education, Culture, Sports, Science and Technology(MEXT) in Japan. Under this project, we have developed a new type of hybrid computer
dedicated for astrophysical RHD, called the FIRST simulator, in a collaboration with theDivision of High Performance Computing Systems.
5.3.1 FIRST Simulator
The FIRST (Fusional Integrator for Radiation-hydrodynamic Systems in Tsukuba
University) simulator is a hybrid PC cluster, in which a newly developed board for gravity
calculations, called Blade-GRAPE, is embedded in each node. The Blade-GRAPE is
composed of four GRAPE-6 chips which have dedicated pipelines for binary gravity
calculations, and is designed for a PCI-X bus in PC clusters. The theoretical peak
performance of a single Blade-GRAPE is 136.8 GFLOPS. Each board has 16 MB memoryand can calculate the self-gravity up to 260,000 particles. The Blade-GRAPE is directly
connected via PCI-X bus, and occupies the space of two PCI-X bus slots. The electric
power supply is from the PCI-X bus (3.3 V) as well as from the cluster server board, +12 V(54 W). Each server PC is equipped with multi-port Gigabit Ethernet NIC for connection toa special interconnection network with commodity Ethernet switches.
The first version of Blade-GRAPE was 32 bits and 33 MHz (PCI). It has been improved
to a 64 bit and 100 MHz version (PCI-X), Blade-GRAPE X64 (Fig. 5.4). Using
Blade-GRAPEs, we have constructed a 256-node hybrid PC cluster system, the FIRST
simulator. The system possesses 16 Blade-GRAPE boards and 224 Blade-GRAPE X64
boards. The node of the host PC cluster is a 2U-size 19-inch rack-mountable server PC (HP
ProLiant DL380 G4) that has two dual Xeon processors in SMP configuration. The peak
performance of the FIRST simulator is 36.1 TFLOPS, of which the host PC cluster
contributes 3.1 TFLOPS and the Blade-GRAPEs 33 TFLOPS. All nodes are connected
uniformly with each other via multi-port Gigabit Ethernet interconnect switches. The total
memory of the FIRST simulator is 1.6 TB. The Gfarm Grid file system, which is acommodity-based distributed file system that federates the local disk of each node
- 22 -CCS Report: PART I, Overview of Center for Computational Sciences
Summary of Activities 2004-2007(http://datafarm.apgrid.org/index.en.html), is installed. With Gfarm, the storage of 22 TB isavailable as a seamless file server. In Fig. 5.5, the 256 node FIRST simulator is shown.

Fig 5.4: Blade-GRAPE

Fig 5.5: FIRST Simulator (256node, 1TFLOPS)
5.3.2 PerformanceWith the FIRST simulator, we have investigated the acceleration of gravity calculations
by Blade-GRAPEs. First, we tested the two types of Blade-GRAPEs. The main difference
between Blade-GRAPE and Blade-GRAPE X64 is the bus clock; Blade-GRAPE works at
33 MHz and Blade-GRAPE X64 at 100 MHz. The dependence of performance on thenumber of particles for a single board is shown in Fig. 5.6. If 260,000 particles arecalculated with a single board, the performance of Blade-GRAPE is 93% of the peak speed,
and that of Blade-GRAPE X64 is 98%. This weak dependence shows that, for this numberof particles, the communication with the host computer is negligible compared to the force
calculations in GRAPE chips. The performance is reduced, however, with a decreasing
number of particles. For 16,000 particles, the performance of Blade-GRAPE is 44%, andthat of Blade-GRAPE X64 is 74%. For this number of particles, the communication withthe host computer is noticeable, and therefore Blade-GRAPE X64 with a three times fasterbus has a roughly double performance over Blade-GRAPE.
Next, in a simulation using 2563 particles, we measured the run time with and without
Blade-GRAPEs. The results are shown in Fig. 5.7. In this simulation, the dominant part of- 23 -CCS Report: PART I, Overview of Center for Computational Sciences
Summary of Activities 2004-2007

the calculations is the self-gravity calculations. The acceleration turns out to be 5 times
with Blade-GRAPE, and 10 times with Blade-GRAPE X64. In this simulation, 10,000
particles are sent to each Blade-GRAPE board on average. Therefore, the performance withBlade-GRAPE X64 is approximately twice that with Blade-GRAPE due to thecommunication bandwidth.

Blade-GRAPE X64

Blade-GRAPE

Number of Particles

Figure 5.6: The dependence of performance on the number ofparticles in a board is shown for Blade-GRAPE andBlade-GRAPE X64.

z=0 (NO GRAPE)

10 times faster ホスト部分(その他)

z=0 (Blade-GRAPE 64)

PM 部分
PP部分(GRAPE 以外)
GRAPE (重力計算)
GRAPE (通信)MPI 通信5 times fasterz=0 (Blade-GRAPE)

0

100

200

300

400

500

600

700

Figure 5.7: The acceleration of calculations for 2563
particle simulations.

- 24 -CCS Report: PART I, Overview of Center for Computational Sciences
Summary of Activities 2004-2007

5.4 ILFTNet and ILDG Project
The ILFTNet (International Lattice Field Theory Network) project (FY2004–FY2005)
aimed to construct a research network among leading institutes in Japan, USA, UK,
Germany and other countries in the field of lattice quantum field theories throughpromotion of international collaboration, a chain of workshops and seminars, andresearcher exchange. The infrastructure of the network was to be provided by theInternational Lattice Data Grid (ILDG) for sharing gluon configurations and lattice data.
The project was funded by JSPS and ran for two years, from April 2004 till March 2006.
The hub institute in Japan was the Center for Computational Sciences, Univ. of Tsukuba.
For more details, refer to http://www.rccp.tsukuba.ac.jp/projects/ILFTNet/.
UK
ＵＫＱＣＤ
network
University of Edinburgh
EPCC
University of Tsukuba
CCS

Fermi National Accelerator Lab
USA
ＳｃｉＤＡＣ
networkDESY
Von Neumann InstituteＷａｓｈｉｎｇｈｏｎ Ｕ
Indiana
Edinburgh
Glasgow
Liverpool
SwanseaSouthampton

Utah
Berlin/Zeuthen
DESY/Neumann
Bielefeld
Regensburg

Future expansion to Asia/Oceania

MIT/Boston U

BNL/Columbia

ＵＣＳＢ

St．Louise

JLAB

Arizona

Germany
LatFor
networkJapan
LFT Forum
network

KEK

Future expansion to EU network
Italy, France, Spain, Denmark,…

ＦＮＡＬ

京都大学

筑波大学

広島大学

Figure 5.8: Research network of ILDG
This activity has continued in FY2006 and beyond with the support from the CSI (Cyber
Science Infrastructure) project of the National Institute of Informatics (NII). The activity
involves constructing the lattice QCD archive and JLDG, the national data grid for thelattice QCD community in Japan.
5.4.1 ILDG: International Lattice Data Grid
ILDG is an international project to develop a grid of data grids for sharing lattice QCD
configurations worldwide. An XML-based markup language, QCDml, which describes
metadata for QCD configurations and ensembles (sets of configurations with common
physics parameters) have been developed. Middleware interfaces among collaborating grids
are defined with WSDL. The construction of regional grids was finalized in US, UK,
Germany, Australia and Japan. JLDG works as the ILDG Japan grid. The interoperabilityof the regional grids has been achieved for download operations, and valuable- 25 -CCS Report: PART I, Overview of Center for Computational Sciences
Summary of Activities 2004-2007

configurations have already been archived in the grid.
For more details, refer to http://www.lqcd.org/ildg
metadata-data cooperation

Middleware Components

mc://JLDG/nf2/b205k1356

ensembleXML

Storage
Elements

configuration
XML

Metadata Catalogue
SRM
SURL
TURL, data

<npClover><kappa>0.1354</kappa><cSW> 1.684 </cSW></npClover>

TURL
lfn://JLDG/nf2/b205k1356-A200

configuration filedata
GridFTP
httpservers

file-format XML size, precision
packedconfiguration binary data with LIME
lfn://JLDG/nf2/b205k1356-A200

web service interface

I
ath
UR
XP
a in
h
vC
rko
ma LFN
LFN
SUR
web service interface
L

VOMS

File Catalogue

Figure 5.9: QCDML and overview of ILDG architecture
5.4.2 LQA: Lattice QCD Archive
LQA is a database of QCD configurations maintained by the Center for Computational
Sciences (CCS) since December 2003. The archive currently stores i) two-flavor full QCD
configurations generated by the CP-PACS parallel computer at CCS (about 8000 files
stored in 1.5 TB disk space) and ii) 2+1 flavor full QCD configurations generated by the
CP-PACS and JLQCD collaborations (about 21000 files stored in 6 TB disk space) andmakes them available to the lattice QCD community worldwide. Configurations for muchlighter quarks, which will be generated with the PACS-CS computer, will be added to thearchive. One can use the ILDG standard interface to search ensembles and configurations
and to download configurations, or use an interactive Web interface. The LQA wasdesigned to serve as a Japan gateway to/from other sites of ILDG. The system will be
restructured as a gateway between JLDG and ILDG in 2007.
For more details, refer to http://www.jldg.org/lqa/index.html.
5.4.3 JLDG: Japan Lattice Data Grid
JLDG is a data-grid infrastructure for the lattice QCD (LQCD) community in Japan.
Several large LQCD collaborations in Japan have been working on lattice QCD simulations
using supercomputers. Outputs of simulations called "QCD configurations" are valuable,
because physicists can study various aspects of QCD using these configurations. JLDG
enables the community to share configurations and other data distributed over distant sites.
File sharing is realized with the GFarm global file system. GSI authentication is managed
by VOMS. JLDG utilizes the NII Super SINET MPLS VPN, HEPnet-J/sc, as a hardware
infrastructure. Part of the configurations can be accessed from all over the world throughthe ILDG interface. Full installation of the system was completed in FY2006. Mutual test
- 26 -CCS Report: PART I, Overview of Center for Computational Sciences
Summary of Activities 2004-2007

operations are ongoing.
For more details, refer to http://www.jldg.org.

Figure 5.10: Conceptual view of JLDG
5.5 Toward Peta-scale Computing
In 2005 MEXT initiated the “Elemental Supercomputing Technology” project within alarger project of “R&D to Build Future IT Infrastructure.” The elemental technologies
developed in these projects were expected to be used in future supercomputers.
Together with Hitachi Ltd. and University of Tokyo, we have been conducting the project“Low-Power and High-speed Device, Circuit and Logic for Next-generation
Supercomputer Systems” of the “Elemental Supercomputing Technology” project. Our task
in the project is to carry out research on “Research and Development on Fundamental
Logic on Processor Architecture and Memory Hierarchy.”
We have been studying the on-chip memory architecture and computing acceleration
mechanism for low-power and high-performance systems. In the next-generation
supercomputer, power consumption is one of the most important issues in the systemdesign. We proposed the on-chip memory architecture called SCIMA (Software-Controlled
integrated memory architecture) for multi-core processors and accelerators to makearithmetic units more efficient. We considered the SIMD arithmetic unit and Vector- 27 -CCS Report: PART I, Overview of Center for Computational Sciences
Summary of Activities 2004-2007

processing unit as accelerators, and compared the performance and power efficiency. We
hope these research results will be incorporated into the processor architecture of thenext-generation supercomputer.
In 2006, the Japanese government launched the national project (“Next-Generation
Supercomputer Project”) to develop a peta-scale computer system, which has theperformance of more than 10 PFLOPS (so-called “Keisoku” computer). The project spansseven years (2006–2012) and RIKEN (Research Institute for Science and Chemistry) has
been selected to conduct the project.
In September 2006, RIKEN and University of Tsukuba signed a collaboration agreement
for the “Next-Generation Supercomputer” project. Based on this agreement, several
members of the Center for Computational Sciences have been appointed as visiting fellows
of RIKEN and have been involved in the project.
The project is now in the stage of detailed designing. CCS will contribute not only to thedesigning of the system but also to the evaluation and performance improvement of realtarget application codes. The Center and RIKEN are setting up a research collaboration
program under the General Agreement for tuning QCD, RS-DFT and FFT codes for the
Next-Generation Supercomputer as peta-scale applications. We think that such acontribution is possible in the unique environment of CCS, where researchers of computerscience and computational science are working together.
5.6 Summary of Interdisciplinary Collaborations in the Center•

As described in Section 5.2, the PACS-CS project is an interdisciplinary program.
The Particle Physics Group, building upon their past experience in R&D of parallel
computing, has been involved in the development of the PACS-CS system from the
beginning with computer scientists in the High Performance Computing Group. It
has contributed significantly in the initial testing and improvement of the PACS-CS
system after the installation in July 2006. A new collaboration was also set up withthe experts in algorithms and programming, which helped in the optimization oflattice QCD codes and exploration of new methods such as the 32 bit acceleration
techniques.
The Materials Science Group set up a working group with the High Performance
Computing Group to develop the real-space and real-time density functional
calculations on PACS-CS. The performance of the RS- and RT-DFT codes wasanalyzed and optimizations were carried out. The benefit was twofold in thatcomputer scientists learned the key features of the RS-DFT method towardpeta-scale computing.
The Global Environment Science Group also set up a working group with computerscientists. In collaboration with the data-engineering experts in the Computational
Informatics Group, they set up the real-time weather data archive GPV/JMA and- 28 -CCS Report: PART I, Overview of Center for Computational Sciences
Summary of Activities 2004-2007

have been using it in conjunction with global and regional weather analyses using
PACS-CS.
Finally, the Biological Science Group has been working with the experts indistributed computing in the High Performance Computing Group to develop aparallelized phylogenetic search code in order to greatly expand the number ofanalyzed genes using PACS-CS•

As described in Section 5.3, the FIRST project is carried out through collaboration
between the Astrophysics Group and the High Performance Computing Group. In
addition to the systems development, the collaboration resulted in the installation of
the Gfarm Grid file system, which had been developed in the High Performance
Computing Group, in the FIRST simulator. As a result, the distributed storage has
been integrated as a seamless file server.

•

The ILDG (International Lattice Data Grid) project has been conducted by theParticle Physics Group and grid experts in the High Performance Computing Groupsince 2000. Jointly, they have contributed to a worldwide effort to standardize
QCDml, an XML for lattice QCD configurations, and the middleware architecture
for interfacing the regional lattice data grids and ILDG. They have also developed
JLDG, the Japanese regional grid. ILDG and JLDG are now in operation mode.

•

Within the Division of Material and Life Sciences, the Groups of Materials Science,
Life Science and Quantum Many-Body Systems have a tight collaboration. Our
stance is to recognize the common features in nano- and bio-materials, perform
calculations based on quantum theory, and clarify and predict various phenomena in
the two categories of materials from an integrated viewpoint. Collaboration with theComputational Life Science Group is therefore essential.
Time-dependent density functional theory is an important methodology for thequantum many-body group. This scheme has common aspects with the RS-DFT codeof the Materials Science Group. A considerable part of the source code of thereal-time calculation is common to the real-space density functional code. Joint
discussions, also involving researchers from the High Performance Computing
Group, have helped to locate the critical paths in the two types of programs and todevelop ways to optimize them.

•

The Global Environmental Science Group has a close collaboration with the High
Performance Computing Group and the Computational Intelligence Group under thecommon subject of constructing a large database of real-time weather data, named
the GPV/JMA project. This goal of this project is to archive real-time weather data of
the global spectral model (GSM), regional spectral model (RSM), meso-scale- 29 -CCS Report: PART I, Overview of Center for Computational Sciences
Summary of Activities 2004-2007

non-hydrostatic model (MSM) provided by the JMA. In addition to these model data,
weekly ensemble forecast data and monthly ensemble forecast data are also archived
and released to the public through the CCS web site. The weather map presented on
Google Earth was provided by a collaboration of the Computational Intelligence
Group.

•The groups in the Division of Computational Informatics organized the“Computational Informatics Seminar” in order to share awareness of the problems
involved in the leading-edge field of “Computational Informatics.” The seminarmembers include all the faculty in the Computational Informatics Division and active
researchers in related fields invited from other departments of the university andexternal organizations, such as the National Institute of Advanced Industrial Scienceand Technology (AIST).

•The Computational Media Group of Computational Informatics and the Grid
Computing Group of the High Performance Computing Division have jointly
planned a research proposal with other research groups of Kyoto University, etc. This
activity has led to a new project entitled “Sensing Web,” which aims at developing
the next generation of sensor grids and networking, for 2007–2009, and is supportedby the Special Coordination Funds for Promoting Science and Technology.

- 30 -CCS Report: PART I, Overview of Center for Computational Sciences
Summary of Activities 2004-2007

6. Inter-university Activity, Education and Public Relations
6.1 Interdisciplinary Computational Science Promotion Program
The Center of Computational Sciences is an inter-university facility, open to researchers ofuniversities and other research institutes in Japan. The Center solicits proposals for theInterdisciplinary Computational Science Promotion Program to encourage research
activities of computational sciences. The computational power of the PACS-CS is provided
to researchers throughout Japan through this program.
The Interdisciplinary Computational Science Promotion Program has two categories:
1. Interdisciplinary Collaboration Program
2. Large-scale Scientific Simulation Program.
The Interdisciplinary Collaboration Program promotes interdisciplinary research activities
of different disciplines. For example, trends in HPC architecture are such that, at thehighest end, systems are expected to consist of a huge number of processors, with eachprocessor containing multiple cores with small vector instructions. These features present a
programmer with a number of different ways to exploit different types of parallelism within
algorithms. Optimizing the performance and designing algorithms involve a number ofchallenges, including exploiting parallelism within codes and overlapping different types ofoperations. This requires tight collaboration of application scientists and computerscientists.
An area where innovative research may be expected from such a program is that ofalgorithm design. New modeling techniques enable researchers to take a morecomprehensive view of physical phenomena stretching across disparate length and timescales. We expect that similar numerical algorithms and modeling might be applied to the
problems in different fields.
A project can be proposed by a team that includes researchers from different fields. If one
team needs partners for the proposed collaborative research and it is accepted, CCS is readyto coordinate appropriate partners for the project through the CCS Interdisciplinary
Collaboration Program Support Committee. The fields inviting applications in this category
are particle physics and astrophysics, material sciences, life sciences, global environmental
sciences, biological science from application fields, and high-performance computing andcomputational informatics and numerical analysis from the computer science fields.
The Large-scale Scientific Simulation Program pushes forward the challenges in several
fields of computational sciences by providing the computational power of the PACS-CS.
Since this program solicits research proposals which are impossible to solve without thelarge amount of computing resources of PACS-CS, the proposal should describe the reason
why the use of PACS-CS is indispensable. The fields inviting applications in this category- 31 -CCS Report: PART I, Overview of Center for Computational Sciences
Summary of Activities 2004-2007

are expected to be particle physics and astrophysics, material sciences, life sciences, global
environmental sciences, biological science.

Interdisciplinary collaboration program

Support to establish a network of interdisciplinary
research projects and to encourage grand challenges
of interdisciplinary computational science.
To promote interdisciplinary research projects ofdifferent disciplines.
Between Computer science and application fields （e.g.
Applied Math and Particle Physics）
 Between different application fields（e.g. Modeling ofradiation in Astrophysics and Climate simulation）


Field ＡGroup Ｘ
Proposal
requestcoordinate

Field Ｃ・ Field Ｄ
Alliance ＺＷ

proposal
Proposal
requestsupportInterdisciplinary collaboration
program support committee（ review of proposal and coordinate collaboration）
Establish a network to promoteinterdisciplinary research projects

Matchmaking of research groups from different fields
Review of Proposals by external reviewer ofInterdisciplinary collaboration coordination committee
 Coordinate research groups from different fields
 Find partners to mach the request
 Advice with expertise of different fields

Field Ｂ
Group Ｙ

Follow up the research activity produced by theinterdisciplinary collaboration

Field ＣGroup Ｚ

Field ＡGroup Ｘ

CCS
Field Ｂ
Group Ｙ

Field ＤGroup Ｗ

Large-scale scientific simulation program
Push forward the grand challenge of several fields in computational sciences by providing thecomputational power of the PACS-CS.
 Review proposals and concentrate our computational power to make new scientific discoveries Follow up the scientific results

Figure 6.1 shows the organization for inter-university facilities operations. In order to
execute the Interdisciplinary Computational Science Promotion Programs, we organize theInter-university Research Review Committee, including external members, to evaluate theproposals. We also organize the Interdisciplinary Collaboration Program Support
Committee to review the proposals and coordinate collaborations between different fields.
We report the policy, results and status of inter-university activities to the annual Steering
Council meeting of the Center.

- 32 -CCS Report: PART I, Overview of Center for Computational Sciences
Summary of Activities 2004-2007
Researchers in University and InstituteProposalAcceptanceInter-university activity
committee

Director

Operation of inter-university facilities
Execute Interdisciplinary Computational
Science Promotion Programs

Computing facility
administration committee

Inter-university researchreview committee

Allocateresources

Review of proposal (including external reviewer)

Interdisciplinary collaboration
program support committee

Inter-university activity
support teamSupportstaff
from company

Steering committee
Steering council

Figure 6.1: organization for Inter-university facilities operations
The Interdisciplinary Computational Science Promotion Program started in October 2007.
Proposals from 13 research groups were accepted for the 2nd half of FY2007.
6.2 Other Inter-university Activities and Contributions to Computational Science
Communities•

Symposiums, workshops and colloquia organized by CCS
We organize an annual symposium on computational science, which covers all fields
related to computational science. The Center also organizes international symposia andinvites visitors from abroad in order to promote international exchange andcollaboration of scholars in computational sciences Workshops in each field andcolloquia are held occasionally.

•

Operation and support of scientific databases and data grids
Scientific databases are becoming increasingly important in computational science. We
support the following two databases for particle physics research and climate research:− Lattice QCD archive, Particle Physics (open since Feb 2004), extended into
Data Grid ILDG (2007)− GPV/JMA archive, Climate research (open since Jan 2005)•

Education, Outreach, and Public Relations
All the faculty members of the Center participate in the Graduate and Undergraduate
Program of University of Tsukuba, through which they provide education in various
aspects of computational sciences. The Center also has been hosting seminars andlectures for computational science and HPC technology. We hope that these activities- 33 -CCS Report: PART I, Overview of Center for Computational Sciences
Summary of Activities 2004-2007

educate the community to increase participation in advanced computing technology
careers.
Every year we receive a large number of visitors from high schools and other guests.
We attempt to inform society through the visitors about our activities and the impact ofadvanced computing in improving knowledge and the quality of life.
]- 34 -CCS Report: PART I, Overview of Center for Computational Sciences
Summary of Activities 2004-2007

Appendix A. Faculty Members of CCS
Director:

Mitsuhisa Sato

Deputy Director: Masayuki Umemura
Taisuke Boku

Division of Particle Physics and Astrophysics (Division Manager: Masayuki
Umemura)Particle Physics Group (Group Leader: Sinya Aoki)
Faculty Members

Tomoteru Yoshie, Associate Professor
Graduate School:
Physics , Graduate School of Pure and Applied Sciences
Research topics:
Elementary Particle Physics
Naruhito Ishizuka, Associate Professor
Graduate School:
Physics , Graduate School of Pure and Applied Sciences
Research topics:
Computational studies on strong interactions
using lattice field theories
Yoshinobu Kuramashi, Assoicate Professor
Graduate School
Physics , Graduate School of Pure and Applied Sciences
Research topics:
Theoretical and computational studies
on strong interact ions using lattice field theories
Yusuke Taniguchi, Assistant Professor
Graduate School:
Physics , Graduate School of Pure and Applied Sciences
Research topics:
Lattice gauge theory- 35 -CCS Report: PART I, Overview of Center for Computational Sciences
Summary of Activities 2004-2007

Akira Ukawa, Professor, Executive Advisor to the President
Graduate School
Physics , Graduate School of Pure and Applied Sciences ,Research topics
Computational particle physics, in particular lattice QCD,
and development of computers for such studies

Collaborative researchers
Sinya Aoki, Professor
Graduate School:
Physics, Graduate School of Pure and Applied Sciences
Research topics:
Elementary Particle Physics Theory, Lattice Field Theory
Kazuyuki Kanaya, Professor
Graduate School:
Physics, Graduate School of Pure and Applied Sciences
Research topics:
lattice field theory, parallel computers

Research staff

•
•
•
•

Kiyoshi SASAKI
Noriyoshi Ishii
Naoya Ukita
Daisuke Kadoh- 36 -CCS Report: PART I, Overview of Center for Computational Sciences
Summary of Activities 2004-2007

Astrophysics Group (Group Leader: Masayuki Umemura)
Faculty Members

Masayuki Umemura, Professor
Graduate School:
Physics , Graduate School of Pure and Applied Sciences
Research topics:
Theoretical Astrophysics.
In particular, the study with radiation hydrodynamics
on the formation of first generation objects and galaxies
Hiroyuki Hirashita, Assistant Professor
Graduate School:
Physics , Graduate School of Pure and Applied Sciences
Research topics:
Theoretical study of element production history
in galaxy formation and evolution
Kohji Yoshikawa, Assistant Professor
Graduate School:
Physics , Graduate School of Pure and Applied Sciences ,Research topics
Astrophysics.
Theoretical studies on observational cosmology,
the formation of galaxies and galaxy clusters, and intergalactic medium
Junichiro Makino, Visiting Associate Professor
Graduate School:
Graduate School of Science, University of Tokyo
Research topics:
Computational Astrophysics,
Development of dedicated computers for computational science

Research Staff•••

Yoshiaki Kato
Tamon Suwa

Takuya Akahori- 37 -CCS Report: PART I, Overview of Center for Computational Sciences
Summary of Activities 2004-2007

 Division of Material and Life Sciences (Division Manager: Atsushi Oshiyama)
Material Science Group (Group Leader: Atsushi Oshiyma)
Faculty Members

Atsushi Oshiyama, Professor
Graduate School:
Frontier Science, Graduate School of Pure and Applied Sciences
Research topics:
Nanoscience and Theoretical Condensed Matter Physics
Kenji Shiraishi, Professor
Graduate School:
Frontier Science , Graduate School of Pure and Applied Sciences
Research topics:
Nanoscience and Theoretical Condensed Matter Physics
Susumu Okada, Associate Professor
Graduate School:
Frontier Science , Graduate School of Pure and Applied Sciences
Research topics:
Nanoscience and Theoretical Condensed Matter Physics

Susumu Saito, Visiting Professor
Graduate School:
Department of Physics, Tokyo Institute of Technology
Research topics:
Nanoscience and Theoretical Condensed Matter Physics

Research Staff•

Jun'ichi Iwata- 38 -CCS Report: PART I, Overview of Center for Computational Sciences
Summary of Activities 2004-2007

Life Science Group (Group Leader: Masaru Tateno)
Faculty Members

Masaru Tateno, Associate Professor
Graduate School
Physics , Graduate School of Pure and Applied Sciences ,Research topics:
Computational analysesof functional controle mechanisms of biological macromolecules
Yasuteru Shigeta, Assistant Professor
Graduate School
Physics , Graduate School of Pure and Applied Sciences ,Research topics:
Theoretical studies on electron
and proton transfer reactions in biological systems

Collaborative researchers
Mauro Boero, Associate Professor
Graduate School:
Physics, Graduate School of Pure and Applied Sciences
Research topics:
First principles molecular dynamics simulations :checmical and biochemical reactions of organic and inorganic materials
in liquid and solid phases

Research Staff••

Kamiya Katsumasa
Yoshitaka Fujimoto- 39 -CCS Report: PART I, Overview of Center for Computational Sciences
Summary of Activities 2004-2007

Quantum Many-Body Systems Group (Group Leader: Kazuhiro Yabana)Faculty Members

Kazuhiro Yabana, Professor
Graduate School
Physics , Graduate School of Pure and Applied Sciences ,Research topics:
Nuclear physics, computational sciences on atomic,
molecular,and optical sciences
Yukio Hashimoto, Assistant Professor
Graduate School
Physics , Graduate School of Pure and Applied Sciences ,Research topics:
Microscopic theory of nuclear collective motions
Hiroyasu Koizumi, Associate Professor
Graduate School:
Materials Science, Graduate School of Pure and Applied Science
Research topics:

Tong Xiao-Min, Associate Professor
Graduate School:
Materials Science, Graduate School of Pure and Applied Science
Research topics:
Atoms, molecules interaction with intense laser field
and time-dependent density functional theory

Collaborative researchers
Kenichi Hino, Professor
Materials Science, Graduate School of Pure and Applied Science- 40 -CCS Report: PART I, Overview of Center for Computational Sciences
Summary of Activities 2004-2007

 Division of Global Environment and Biological Science (Division Manager: Hiroshi
Tanaka)Global Environmental Science Group (Group Leader: Hiroshi Tanaka)
Faculty Members

Hiroshi L. Tanaka, Professor
Graduate School:
Geoenvironmental Sciences,
Graduate School of Life and Environmental Sciences
Research topics:
General circulation, dynamics and energetics of the atmosphere
Kusaka Hiroyuki, Assistant Professor
Graduate School:
Geoenvironmental Sciences,
Graduate School of Life and Environmental Sciences
Research topics:
Urban climate, Applied Meteorology,
Numerical simulationof mesoscale weather using the WRF model

Collaborative researchers
Fujio Kimura, Professor
Graduate School:
Geoenvironmental Sciences
Graduate School of Life and Environmental Sciences
Research topics：- 41 -CCS Report: PART I, Overview of Center for Computational Sciences
Summary of Activities 2004-2007

Global Environmental Science Group (Group Leader: Tetsuo Hashimoto)
Faculty Members

Yuji Inagaki, Associate Professor
Graduate School:
Biological Sciences, Graduate School of Life and Environmental Sciences ,Research topics
Molecular phylogeny of eukaryotes Investigation of lateral genetransfers
Estimation of protein functions by combining structuraland evolutionary parameters Artifacts in molecular dataanalyses

Collaborative researchers
Tetsuo Hashimoto, Professor
Graduate School:
Biological Sciences, Graduate School of Life and Environmental Sciences ,Research topics
Molecular evolutionary studies
on the origin and early evolution of eukaryotes
Hideko Urushihara, Professor
Graduate School:
Functional Biosciences, Graduate School of Life and Environmental Science
Research topics
Studies on developmental programs described in the genome- 42 -CCS Report: PART I, Overview of Center for Computational Sciences
Summary of Activities 2004-2007

 Division of High Performance Computing Systems
Boku)(Division Manager: Taisuke

System Architecture Group (Group Leader: Taisuke Boku)
Faculty Members

Boku Taisuke, Professor
Graduate School:
Computer Science, Graduate School of Systems and Information Engineerin
Research topics:
High performance parallel computing,cluster computing, hybrid parallel processing system,
grid computing
Daisuke Takahashi, Assoicate Professor
Graduate School:
Computer Science, Graduate School of Systems and Information Engineerin
Research topics:
High performance parallel computing,High-performance numerical algorithms on parallel computersand performance evaluation
Hiroshi NAKAMURA, Visiting Assoicate Professor
Graduate School:
Graduate School of Information Science and Technology, University of Toky
Research topics:
High-Performance and Low-Power Processor Architecture,
High-Performance Computing, Dependable Computer System- 43 -CCS Report: PART I, Overview of Center for Computational Sciences
Summary of Activities 2004-2007

Collaborative researchers
Moritoshi Yasunaga, Professor
Graduate School:
Computer Science, Graduate School of Systems and Information Engineerin
Research topics:
VLSI Engineering, Evolvable Hardware, Dependable Systems

Kouichi Wada, Professor
Graduate School:
Computer Science, Graduate School of Systems and Information Engineerin
Research topics:
Numerical algorithms and simulation,
Mathematical software for GRID computing

Research Staff•

Toshihiro Hanawa- 44 -CCS Report: PART I, Overview of Center for Computational Sciences
Summary of Activities 2004-2007

Grid Computing Group (Group Leader: Mitsuhisa Sato)
Faculty Members

Mitsuhisa Sato, Professor, Director
Graduate School:
Computer Science, Graduate School of Systems and Information Engineerin
Research topics:
High performance parallel computing,compilers and performance evaluation, grid computing
Osamu Tatebe, Associate Professor
Graduate School:
Computer Science, Graduate School of Systems and Information Engineerin
Research topics:
High performance parallel computing,
Grid Computing, Distributed File System- 45 -CCS Report: PART I, Overview of Center for Computational Sciences
Summary of Activities 2004-2007

 Division of Computational Informatics(Division Manager: Yuichi Ohta)

Computational Intelligence Group (Group Leader: Hiroyuki Kitagawa)
Faculty Members

Hiroyuki Kitagawa, Professor
Graduate School:
Computer Science, Graduate School of Systems and Information Engineerin
Research topics:
Database systems, Data engineering Information integration,
WWW and databases, Knowledge discovery, XML databases,
Multimedia information retrieval, and DBMS architecture
Toshiyuki Amagasa, Assistant Professor
Graduate School:
Computer Science, Graduate School of Systems and Information Engineerin
Research topics:
Database Systems, Data Engineering.
XML Databases, Web Information Systems, Bioinformatics
Hideyuki Kawashima, Assistant Professor
Graduate School:
Computer Science, Graduate School of Systems and Information Engineerin
Research topics:
Database Systems and Data Engineering:
DBMS Architecture, Sensor Networks- 46 -CCS Report: PART I, Overview of Center for Computational Sciences
Summary of Activities 2004-2007

Computational Media Group (Group Leader: Yuichi Ohta)
Faculty Members

Yuichi Ohta, Professor
Graduate School:
Advanced Engineering Systems,
Graduate School of Systems and Information Engineering
Research topics:

Yoshinari Kameda, Assoicate Professor
Graduate School:
Research topics:
Massive Sensing, Intelligent Video Making based on Scene Understanding,Model Based Vision,Coorperative Distributed Vision, Lecture Archiving andDistance Learning, Visual Surveillance, Human Interface to Virtual Reality
Itaru Kitahara, Assistant Professor
Graduate School:
Advanced Engineering Systems,
Graduate School of Systems and Information Engineering
Research topics:
Image media with ubiquitous sensor networks

Collaborative researchers

Tomonori Shirakawa, Professor
Graduate School:
Advanced Engineering Systems,
Graduate School of Systems and Information Engineering
Research topics:- 47 -CCS Report: PART I, Overview of Center for Computational Sciences
Summary of Activities 2004-2007

Appendix B. Facilities of CCS
1. Computer Systems Overview
The computational facilities of the center mainly consist of a massively parallel cluster
system PACS-CS and its front-end computer system, and the new generation ofastrophysics simulator FIRST.
The front-end computer system consists of the system controlling servers to manage the
PACS-CS cluster and the small-sized mini-PACS-CS for program development, Magellan(Hitachi SR11000J) for post-processing and analysis on generated data by PACS-CS, 100
TByte of large capacity file server to hold all data of these machines. There are additional
medium to small size clusters for data analysis and general networking services. Beside ofthese main facilities in Computer Building, there are a number of graphic workstations andPCs to support daily research of researchers and students in Research Building. A central
switch in Computer Building connects all computational facilities by Gigabit Ethernet for
high-speed data exchange. It is also connected another central switch in Research Building
via 10 Gigabit Ethernet to support high-speed data access among all resources in the center.
Main computational facilities in the center are shifting from vector machines to large scale
high-end clusters including PACS-CS and FIRST to support various fields of computational
sciences and computer science in all research divisions in the center.
Since the PACS-CS system is the main central facility in the center, the operation, job
status and detailed system temperature such as CPU, chassis and network interfaces of eachcomputational node are monitored in real-time. The status of the machine and air
conditioners in Computer Building are remotely monitored to detect any failure.

- 48 -CCS Report: PART I, Overview of Center for Computational Sciences
Summary of Activities 2004-2007- 49 -CCS Report: PART I, Overview of Center for Computational Sciences
Summary of Activities 2004-2007

2. Network Environment
University of Tsukuba has been connected 10 Gbps of nation-wide network for researchand education named SINET3 under MEXT since April 2007. In addition, the university isalso connected to major research institutes in Tsukuba City by a high-speed WAN with 20
Gbps bandwidth named TSUKUBA-WAN. Our center is connected to SINET3 with 2 Gbps
of general purpose network and three of 1 Gbps special purpose research network.
Moreover, we are also connected to the nation-wide research network JGN2 with 10Gbps
link to stimulate various research activities in the center. In such network environment, weshare the valuable data computed by PACS-CS and other center facilities as well as
high-performance Grid computing research toward new generation of computational
sciences.

- 50 -