Joint Usage/Research Center for Interdisciplinary Large-scale Information Infrastructures
Final Report for JHPCN Joint Research of FY 2019jh190005-MDH

Combination of HPC and high-speed data transfer technologies for bigdata processing systems
Ken T. Murata (National Institute of Information and Communications Technology)
Abstract
In this research, we combine high performance computing (HPC) and high speed datatransfer technologies to improve big data processing systems. Three case studies onlarge-scale data processing and large-scale information systems are presented which
adopt high-performance and flexible protocol (HpFP) as a communication protocol.
The HpFP1 is designed for specified networks and puts more emphasis on latency
and packet loss tolerances than fairness and friendliness, while the HpFP2 is an
improved version of the HpFP1 and is more suitable for real network environments.
Based on HpFP2, we implement a file transfer tool, called high-performance copy(HCP), to improve data transfer performance on JHPCN. The performance of oursystems is evaluated using real data produced by HPC. The results show that theHCP outperforms the conventional tool for file transfer in JHPCN. We conclude thatour systems have significant potential for further researches in JHPCN.

1.

Basic Information

speed (30 fps) in sequence order.

(1) Collaborating JHPCN Centers

Data communication server (serverKyoto University

name: UV2000 login node): set up HCP,

Data communication server (server

receive

name: XC40, VM hosting): set up high-communication servers at high speed,performance copy (HCP), receive dataand save them on large-scale storage.

files

communication

Researchers at each university accessservers at high speed, and save themstored data at high speed from theon large-scale storage.

outside.

fromother

Large-scale storage (1PB): save each

files

fromother

Large-scale storage (0.5PB): save eachresearch domain data. Researchers at

research domain data. Researchers at

each university access the stored data

each university access the stored data

at high speed from the outside via theat high speed from the outside via thedata communication server.

data communication server.
Data processing server (server name:

Data processing server (server name:

XC40): perform data processing stored

UV2000 login node): perform datain large-scale storage.

processingNagoya University

datastoredKyushu University(display server): receive high-speed

storagelarge-scalestorage.

8K large-scale visualization system
externalin(especiallyhigh-The resources provided by Kyushu
Universityresolution time series of Himawari

intheinteractiveenvironmentimage data from Chiba University)and display it on an 8K display at high

Other facilities, the resources andmethods

1ofuseavailablefor

Joint Usage/Research Center for Interdisciplinary Large-scale Information Infrastructures
Final Report for JHPCN Joint Research of FY 2019

collaborative research: perform data

loss tolerance and is used in various domain

communication

science researches. The HpFP1 is designedexperimentusingHCP

for specified networks and focuses more on

Genomedataserver(Okawalatency and packet loss tolerances thanlaboratorymanagement,Kyushu

fairness and friendliness, while the HpFP2 isUniversity): read the genome data ofan improved version of the HpFP1 and is

Kyushu University and transmit themore suitable for real network environments.

datalarge-scale

Based on the HpFP2, we implement a filestorage via the data communicationtransfer tool, called HCP, to improve dataservertransfer performance on JHPCN. The HCP is

Genome storage (Kyushu UniversityusedOkawa laboratory management / 150

transmission/reception in the informationTB): save the genome data of Kyushu

infrastructure centers of universities viaUniversityScience Information NETwork 5 (SINET 5).

totheexternalTohoku University

3.

Supercomputer
SX-ACE):

perform

MHD

simulation

andJupitertoolforthedataSignificance as JHPCN Joint Research
SINET5 brings new possibilities with its 100

save

Gbps full-mesh network connecting more than

execution results in supercomputer

800 organizations, clouds, and academic

storage (cache area). The stored datacontents.

is transmitted to remote large-scale

performance computing (HPC), including Grid

storage via HCP.

computing, cloud computing, supercomputers,

Data communication server (server

an inter-university HPC system is one of thename: Express 5800): set up HCP andgoals

conduct

infrastructure (HPCI). As the high-speed datahigh-speed

experimentof

thenaProject(supercomputer

name:

as

data

transfermanagement

Inof

high

transfertheviewpoint

performancetechnology

of

high

computingdevelopedand(communication) server with large-

experimented on JHPCN has reached thescale storage from other institutions

practical level finally in 2019, it is installed in(2) Research Areasthe information infrastructure system of the☑

Very large-scale data processing

application

inparallelwiththebasic☑

Very large-scale information systems

experiment to perform file transfer of real

(3) Roles of Project Members

data. If this experiment succeeds, the way to

As shown in Section 1(1).

future inter-university HPC system will be

Purpose and Significance of Research

greatly opened up. By using this way, data

This research aims at conducting case studies

outputto demonstrate large-scale data processing

measuring instruments, etc. are stored in anand large-scale information systems. We

arbitrary storage system at high speed andadopt high-performance and flexible protocol

data processing is performed in an arbitrary(HpFP), which is a communication protocol

computing environment. In addition, the2.

with high delay tolerance and high packet

fromsupercomputers,

sensors,

processing results are displayed on a large-2

Joint Usage/Research Center for Interdisciplinary Large-scale Information Infrastructures
Final Report for JHPCN Joint Research of FY 2019

scale display installed in a specific institution,

domestic conference in 2019 are described inand it is possible to visualize and analyze

list of publications and presentations as below.

collaborative data by multiple researchers as4.

5.

DetailsofFY2019

Researchwell as demonstrations.

Achievements

Outline of Research Achievements up toWe present three case studies on large-scale

FY2018

data processing and large-scale information

TheJHPCN

appliedapplications(adopted)

asathatMuratasystems.

representative(1) ExtractionofMeteorological
Information
Using
Artificial
Intelligence (AI)applicant are as follows.
In2016,(jh150033-IS02)“Datatransmission experiment for realizing bigWe

data post processing environment using cloud”

information from two websites developedIn 2016, sprout (Kyushu University,by NICT. The first one is Himawari-8JPHCN-Q)“Epigenome

themeteorologicalDatareal-time web (https://himawari.asia/),

Visualization System Technology for SINET”

which provides Himawari-8 satellite

In 2016, sprout (Nagoya University, HPC

sensed images in real time and with fullscientificcomputingresearch)“ExperimentBig

considercollaborationofPJ

resolution, as shown in Fig. 1. The

large-scale

Himawari-8 real-time web is designed

visualization for remote data on cloud”

for personal computers and smartphones.

In 2017 (jh170034-ISH) “Mash-up ofThe second one is Amaterass web

high-performance numerical computing and(http://amaterass.nict.go.jp/),high-speed data transfer for large-scale data

provides ground weather data (e.g., windfile transfer between universities and theirspeed, solar radiation, air temperature,demonstration experiments with real dataset”

and rainfall) in real time, as shown in Fig.

In

2018,(jh180054-ISJ)

which“Inter-2. Both data are analyzed using AI in

Datacenter File Transfer Examinations for

NICT and Chiba University, and then

HPC Using Real Datasets”

are processed by supercomputer inAs for these achievements, the purpose of thisTohoku

research and development is to utilize

University.

computer(high-performance

displayed on tiled display wall (TDW) insupercomputers, large-scale storages, large-Chiba University, RIKEN and Kyushu

screenUniversity using STARS by NICT andresources

visualizationdevices/displays)ofUniversityandThevisualization

ChOWDERsensors (especially video IoT devices) on high-Currently, we transfer up to 371 MB (741

speed infrastructure networks (SINET, JGN,

pyramid tile image files) of the Himawari

and international networks). By combining

data every 10 minutes and 500 GB (5these resources, we conduct three casefiles) of the Amaterass data per day. In

studies of big data processing systems. The

addition, we also transfer the original

results adopted by international conference

PNG files of Himawari data to Kyoto
University.

3Kyushu

isinformation infrastructure centers, and IoTwith peer review in 2019 and presented inbyNagoya

University.

Joint Usage/Research Center for Interdisciplinary Large-scale Information Infrastructures
Final Report for JHPCN Joint Research of FY 2019

Fig. 1 Himawari-8 real-time web(https://himawari.asia/)

Fig. 3 Data transmission model in JHPCN fromKyushu University to Kyoto University

Fig. 4 Example of web application for viewing
Fig. 2 Amaterass web

epigenomic data

(http://amaterass.nict.go.jp/)(3) Extraction of Visual Internet of Things(2) Concurrent Processing System(IoT) Data using Deep Learning

We consider the Genomic information

We consider the visual IoT data collectedcollectedAnalyzer

from SmartSight camera, as shown in(Illumina Novaseq 6000) in Kyushu

Fig. 5(a). The image/video transmission

University. There are only five Geneticis fixed with 30 fps based on highAnalyzers in Japan. The running time isresolution (full HD). The SmartSight

about half a day to two days. Therefore,

cameras are set up in 20 locations ofit is necessary to transfer the Genomic

Kochi, Arctic region, Hirosaki University,

information

Genetic

Chikuma, and Tokyo, and Chiang Mai,

Analyzers to the large-scale storages atThailand. All of the data are transferredhigh speed. The data transmission isto Kyoto University, and then are

performed from Kyushu University toanalyzed and processed using deep

Kyoto University using HCP, as shown inlearning

Fig. 3. The visualization is displayed insupercomputer.

real time on the web application, as

displayed on TDW, as shown in Fig. 5(b).

shown in Fig. 4. Currently, we transfer

Moreover, the additional visual IoT data

up to 1.1 TB by demand.

from other locations are transferred tofromGenetic

collectedfromwith

TensorFlow
Thevisualization

onisChiba University. Currently, we transfer
images every 10 seconds and videos

4Joint Usage/Research Center for Interdisciplinary Large-scale Information Infrastructures
Final Report for JHPCN Joint Research of FY 2019

every 10 minutes.

traffic. The results show that we succeed thedata transmission in three case studies.
In the first case study, we demonstrate thatthe HpFP2 with fair mode outperforms thetraditionalTCP.

Figure6showsthecomparison of TCP and HpFP2 with fairmode in JHPCN from NICT to Kyoto
University. The throughput of HpFP2 withfair mode is higher than that of TCP. Figure
7 shows the Himawari data transferred toKyoto University. The data size changes
depending on the local time of day, since thePNG compression works particularly well forblack area. It is obvious that the data size inthe daytime is larger than that in thenighttime. Figure 8 shows the daily datatransmission from NICT to Kyoto University.
(a) Visual IoT device

The total size of the Himawari data is up to27 GB.

Fig. 6 Comparison of TCP and HpFP2 with fair(b) Example of real-time monitoring systemmode in JHPCN from NICT to Kyoto University

Fig. 5 SmartSight camera

6.

Progress during FY2019 and Future
Prospects
We evaluate the performance of our filetransfertoolfor

high-speed

datatransmission on JHPCN in three case studies.
The HCP is operated in fair mode to avoid
stressing transmission control protocol (TCP)

Fig. 7 Himawari data on April 26, 2020

5Joint Usage/Research Center for Interdisciplinary Large-scale Information Infrastructures
Final Report for JHPCN Joint Research of FY 2019

University are up to 96 GB and 82 GB per day,respectively. This implies that the monthly
data transmission from the SmartSight
cameras to the large-scale storages is morethan 5TB.

Fig. 8 Daily data transmission from NICT toKyoto University
In the second case study, we investigate theimprovement of data transmission overJHPCN

usingHCP,

comparedtotheFig. 9 Data transmission from Kyushu University

traditional tool, called rsync. The rsync

to Kyoto Universitycommand is run with the -a (archive mode)and -v (verbose) options. In this experiment,
the performance of the rsync and the HCPwith fair mode is evaluated over JHPCN fromKyushu University to Kyoto University using
epigenomic

data

collected

fromsupercomputer resources, as shown in Fig. 4.
The epigenomic data size in different data
series are 20 MB, 16 MB, 327 MB, 328 MB,
1.1 TB, 850 GB, 75 GB, and 54 GB,

Fig. 10 Daily data transmission from SmartSight

respectively. It is obvious that the throughput

cameras to Kyoto Universityof HCP with fair mode is higher than that ofrsync in every file transfer, as shown in Fig.
9. In other words, the HCP with fair modeachieves data transmission improvement,compared to the traditional tool.
In the last case study, we demonstrate thedaily data transmission from the SmartSight
cameras to the large-scale storages. Figures
10 and 11 show the daily data transmission
from the SmartSight cameras to Kyoto

Fig. 11 Daily data transmission from SmartSight

University and Chiba University, respectively.

cameras to Chiba University

The total sizes of the visual IoT datatransferred to Kyoto University and Chiba

In future prospects, we will evaluate the6Joint Usage/Research Center for Interdisciplinary Large-scale Information Infrastructures
Final Report for JHPCN Joint Research of FY 2019

performance of TCP and HpFP2 in a variety
of data sizes on Gfarm environment overJHPCN.

We

willalso

investigate

theperformance of the HCP in each mode andfine-tune the parameters of the HCP toachieve the best performance. Furthermore,
we will also investigate the performance ofthe HCP over collaborative international
networks,e.g.,

SINET5,

JapanGigabit

Network (JGN) and Asia Pacific Advanced
Network (APAN), as shown in Fig. 3. The
results of this research are expected to be
standardized for file transfer over JHPCN.

7.

List of Publications and Presentations(1) ProceedingsofConferences (Refereed)

International

P. Pavarangkoon, K. T. Murata, K. Yamamoto,K. Muranaga, T. Mizuhara, K. Fukazawa, R.
Egawa, T. Katagiri, M. Ogino, and T. Nanri,“Performance Improvement of High-Speed
File Transfer over JHPCN,” 5th IEEE
International Conference on Cloud and Big
Data Computing (CBDCom 2019), Aug. 2019.

(2) Presentations at domestic conference(Non-refereed)
P. Pavarangkoon, K. T. Murata, K. Yamamoto,T. Mizuhara, Y. Kagebayashi, A. Takaki, K.
Muranaga, and K. Fukazawa, “Experimental
Evaluation of High-Speed File Transfer over
JHPCN,” Japan Geoscience Union Meeting2019 (JpGU 2019), May 2019.

7