J. Plasma Fusion Res. SERIES, Vol.7 (2006) 361-364

Multi-Layer Distributed Storage of LHD Plasma Diagnostic Database
NAKANISHI Hideya, KOJIMA Mamoru, OHSUNA Masaki, NONOMURA Miki,IMAZU Setsuo and NAGAYAMA Yoshio
National Institute for Fusion Science, Toki 509-5292, Japan（Received: 4 October 2004 / Accepted: 4 September 2005）

Abstract
At the end of LHD experimental campaign in 2003, the amount of whole plasma diagnostics raw data hadreached 3.16 GB in a long-pulse experiment. This is a new world record in fusion plasma experiments, farbeyond the previous value of 1.5 GB/shot. The total size of the LHD diagnostic data is about 21.6 TB for thewhole six years of experiments, and it continues to grow at an increasing rate. The LHD diagnostic databaseand storage system, i.e. the LABCOM system, has a completely distributed architecture to be suﬃciently
flexible and easily expandable to maintain integrity of the total amount of data. It has three categories of thestorage layer: OODBMS volumes in data acquisition servers, RAID servers, and mass storage systems, suchas MO jukeboxes and DVD-R changers. These are equally accessible through the network. By data migration
between them, they can be considered a virtual OODB extension area. Their data contents have been listed ina “facilitator” PostgreSQL RDBMS, which now contains about 6.2 million entries, and informs the optimized
priority to clients requesting data. Using the “glib” compression for all of the binary data and applying thethree-tier application model for the OODB data transfer/retrieval, an optimized OODB read-out rate of 1.7MB/s and eﬀective client access speed of 3∼25 MB/s have been achieved. As a result, the LABCOM datasystem has succeeded in combination of the use of RDBMS, OODBMS, RAID, and MSS to enable a virtualand always expandable storage volume, simultaneously with rapid data access.
Keywords:
data acquisition, mass storage system, OODBMS, RAID, DVD changer, LHD, LABCOM
1．Introduction
In the 7th LHD campaign of 2003∼2004, quasisteady-state plasma experiments whose longest duration was ∼756 s were performed successfully. The primary LHD data acquisition system, named the LABCOM system, then established a new world record foracquisition data amount of 3.16 GB in one discharge.
This was far beyond the previous record of about 1.5
GB/shot by JET [1]. Even in the short-pulse operation, which usually repeats about 150 shots per day, thewhole acquisition data amount has been over 1 GB/shot.
Figure 1 shows the growth curve by shot number.
On the other hand, the LHD diagnostics have over40 kinds of plasma measurements with up to 2000 signal channels in total. A considerable number of theserequire fast data acquisition even in steady-state experiments. The greater part of the new world recordwas acquired by such fast sampling real-time digitizers, which provide quite diﬀerent capabilities from theconventional CAMAC digitizers. To realize fast realtime data acquisition, we have performed R&D for newdigitizer systems [2]. In the 7th campaign, we have be-1400

Acquired Data per Shot /MB

Up to 3.16 GB!

Raw Data Size
Compressed

1200

7c6c1000

5c4c800
3c600

2c
1c400
200
015000 10000 15000 20000 25000 30000 35000 40000 45000 50000
Shot NumberFig. 1 By-shot data growth in LABCOM data acquisitionsystem.

gun to operate the NI PXI/CompactPCI and Yokogawa
WE7000 digitizers, which can achieve 80 MB/s and 2.2
MB/s continuous data acquisition, respectively. A PXI
frame grabber can also deal with 16 MB/s video stream
for measurements using high-resolution CCD cameras.
As their cost-performance ratio is quite reasonable inC 2006 by The Japan Society of Plasma

Science and Nuclear Fusion Research

Corresponding author’s e-mail: nakanisi@nifs.ac.jp361

Multi-Layer Distributed Storage of LHD Plasma Diagnostic Databaseapparent read/write speed. The three-tier application
model for the OODB data was also applied for the transfer/retrieval programs. Thus, an optimized OODB readout rate of 1.7 MB/s and eﬀective client access speed of
3∼25 MB/s have been achieved.
Even though the OODB virtual space can containmany binary large objects (BLOB) inside, DBMS usually has less functionality to directly manage TB∼PB
huge virtual volumes. However, media library equipment, such as magnetic tape (MT) libraries or DVDchangers, are often used for mass storage systems. Ina similar way, hierarchical storage management (HSM)systems will be used, which will enable a huge virtual
file system.
HSM is a well-established method, which provides
automatic stage-in/stage-out file migration between adefinite logical file system and its front-end cache area.
When OODBMS volumes are held in files, however,their sizes can easily reach as large as 4 GB. Such large
file operations will cause longer time lags for any HSM
to complete the stage-in/-out processes. On the other
hand, the granularity of plasma diagnostic data is usually kS∼MS/channel, which is much smaller than popular storage media, such as 200 GB MT cartridges, and4.7 GB DVD-R. Therefore, the data access patterns willbe almost random. Based on examination of HSM with
the MT library, we concluded that randomly accessible
media, such as MO and DVD, are more appropriate forfusion experimental data [4].
Due to this mismatch between OODB and HSM, wehave developed a new OODB volume extension mechanism by translating their BLOBs into files and directories of the file system as explained in the next section.

25Raw data
Compressed

Total Data Amount /TB20151050010000

20000
30000
Shot Number

40000

50000

Fig. 2 By-shot data growth in LABCOM data archives.

comparison to CAMAC, their utilization is becoming
widespread in LHD. In the 7th and 8th campaigns, we
had ten and four new WE7000 and PXI installations,
respectively, with only one new CAMAC installation.
This technological shift to new digitizers has brought
about an explosion in output data quantity. The intense
increase in amount of diagnostic data inevitably leadsto larger storage volume requirements every year. As
shown in Fig. 2, the total size of the LHD diagnostic data for the previous six years is about 21.6 TB,
and it continues to grow at an increasing rate. Therefore, the data storage system must be suﬃciently flexible and easily expandable to allow maintenance of thewhole data integrity. However, large capacity and rapid
read/write performance are conflicting properties in a
mass storage system. For enormous databases, it isquite diﬃcult to maintain good responsiveness withouthighly sophisticated tuning and optimization.
Here, we describe the realization of the LABCOM
data system and discuss its achieved performance.

3．Multi-layer mass storage system
As the plasma diagnostics raw data usually consistof multiple channels of lengthy time series signals, its
occupied volume in data storage becomes much larger
than usual relational databases in other fields, even if
they have similar numbers of record entries.
The number of LHD data entries can be estimatedfrom the total shot number multiplied by the diagnostic varieties and the backup replications. At present,
the system contains about 6.2 million entries and theprimary part of 3.4 million entries is information for
distributed data locations. To promptly return a query
result, a fast index search of the relational databasemanagement system (RDBMS) will usually be applied.
Millions of record entries occupy a few GB of RDBMS
volume. Plasma raw data, therefore, should be stored
independently outside the database, to prevent any deceleration of its index searching.

2．Data acquisition and database
The database and storage system for LHD raw data
has three categories of storage layers. The first isthe 50∼250 GB local disk arrays for each data acquisition computer. Acquired raw data will be compressed by “zlib” and then stored in the virtual volume,
which is provided by the local object-oriented DBMS.
OODBMS was adopted because of the seamless connection between the volatile data objects in C++ applications and their persistent instances in OODB space[3].
The parts of the OODB client/server system, however, intrinsically share so much information with eachother that their communications often require excessive network bandwidth. Therefore, we first adopted“glib” compression of all the binary data to improve the362

Multi-Layer Distributed Storage of LHD Plasma Diagnostic Database

Table 1 Cost comparison of LHD mass storage systems. The 1st generation MO jukebox is about
20 times more expensive than the 2nd generation DVD changers. Prices include recording
media and management softwares.
Media
4.8GB MO
4.7GB DVD-R

The LABCOM data storage system, therefore, hasapplied a completely distributed architecture based on
fast network. It realizes data redundancy, fail-safe capability, and even load-balancing function by meansof replication pairs of every storage server, which
are equally accessible through the network. All oftheir contents are listed in a “facilitator” PostgreSQLRDBMS, and informed to any data retrieval clients on
demand. Figure 3 shows a schematic view of this system.
Storage servers in the latter two layers consist of files
and directories in the file system, not in the OODBMSvolume. To enable seamless extension from the threetier model of OODB, the same application server program runs in all of them and accesses the file system
instead for data retrieval. In addition, by means of thedata migration mechanism from OODB to file system,
they can be logically considered as an OODB extension
area.
The second layer consists of multiple sets of huge redundant disk array (RAID) servers, to provide fast data
retrieval to clients. The third has a few sets of so-called
mass storage systems (MSS). For the first four campaigns, three sets of 1.2 TB magneto-optical (MO) disk
jukeboxes were applied. Subsequently, 1.8 TB or 3.3
TB DVD-R changers were adopted until 2004. Figure 4
shows the storage structure. The numbers of runningservers in each layer are 40, 4 and 5, respectively.
Table 1 shows the cost comparison between the twokinds of third layer storage equipment. The recording media only account for a small part of the total
storage cost and the most expensive devices are libraries or changers with virtual volume management
software. Even though the prices of HDDs and theirarrays (RAID) always decrease rapidly, this hardly affects the per-byte cost as long as we continue to use
or reinforce the same equipment. With application of
next-generation DVD storage media, such as Blu-ray
Disc or HD DVD, the per-byte cost may again decrease
markedly.

Unit Price
20 M JPY
3 M JPY

Cost (/JPY)
17.5 M/TB
0.95 M/TB

Figure 5 shows the speed diﬀerences between each kindof storage server. Note that the multi-channel diagnostic
data were stored in one file per shot in 2nd and 3rd layerstorage. As designed, the 2nd layer RAIDs have beenshown to consistently provide a comfortable speed.
From OODBMS, the apparent speed of 31.5 MB raw
data retrieval was 2.1 MB/s, while the real I/O rate was
0.8 MB/s. The acceleration ratio was almost threefold,which was achieved by the data compression ratio. TheSequencer

mend

0. advertise

2.
rec

1.
queryFacilitatoromEquipment
HP SureStore 1200ex
Pioneer DRM-7000

Acquire!

ClientServerRetrieve
Data!

DAQ

I/O

Transd

3. request
4. answer

SHR
MEM

OODB

Archive!

Store

Fig. 3 Cooperation between data servers and the facilitator. The data clients never refer to the OODB directly, but their requests are sent to and answeredfrom the application server “Transd”, which can access the ﬁle system instead when running on the2nd or 3rd layer storage servers.

1st Layer(short-term)
50GB

Client computers

250GBAcquisition server w/ local RAID x 40

100Mbps
100Mbps

OmniCore5052
SSR8600

1Gbps
digital

2nd Layer 0.72
TB(middleterm)

1.6
TB

2.0
TB

3.5
TB

Large RAID-5 w/ replication x 4

1Gbps

3rd Layer (long-term)
100Mbps

3TB
480GB

4．Results and discussion
Data retrieval speed to the clients is the most important property to evaluate a database and storage system.

1.88TB x4
250GB

1.2TB

50GB

1.2TB

50GB

1.2TB

50GB

Pioneer DRM-7000 Kubota Comps BA-200

DVD-Rchangers

MO jukeboxes (HP Optical 1200ex x 3)

Fig. 4 Multi-layer structure of LABCOM data storage.

363

Multi-Layer Distributed Storage of LHD Plasma Diagnostic Databasepre-processing

easily understood as the robot moving time to pick up
and make the MO or DVD media ready. They cannot
respond quickly in random data access, whereas they
could provide vast online archive spaces instead.
This also provides insight into how it should be possible to optimize the facilitator’s recommendation priority for data retrieval requests; new or often referreddata must exist in RAID servers as soon or for as longas possible, while aged data, which will be referred toless, can be stored only in the 3rd layer. Here, the timeto search indexes in the facilitator RDBMS can always
be negligible (less than 1 s) as compared to the wholeelapsed time.
We conclude that the LABCOM database and storage system has succeeded in combination of RDBMS,OODBMS, RAID and MSS to realize a virtual and always expandable storage volume. It simultaneously enables rapid data retrieval with some optimization andacceleration mechanisms.

data transfer

45MO Jukebox
DVD changer

elapsed time /s

4035 OODBMS
301825 30

15RAID

20152817

105 11021520

17

2
10
(left/right: 1st/2nd retrieval)

17150Fig. 5 Data retrieval speeds from different kinds of storage. Elapsed times are for retrieving the same
12 MB of compressed data (raw size, 31.5 MB) of
126-channel Hα measurement. The client PC has
dual 450 MHz Pentium-III processors with 512 MB
memory and 100 Mbps Fast Ethernet port, while
OODBMS servers run on dual 200 MHz PentiumPro machines.

diﬀerence between the 1st and 2nd retrieval can be considered due to the internal cache mechanism. In general, internal OODBMS operations involve heavy address translating calculations between persistent object
images and volatile memory instances. Therefore, thedata retrieval speed would be considerably improved
with application of more powerful PCs. Roughly 2- or
3-fold increases in speed can be obtained easily by using ∼GHz Pentium 4 PCs, where the bottlenecks of data
retrieval may exist just in the transaction overheads ofboth HDD readout and TCP/IP telecommunications.
The preprocessing delays in 3rd layer storage can be

[1]

[2]
[3]
[4]

364

References
J.W. Farthing, Proc. 4th IAEA TM of Control,
Data Acquisition and Remote Participation forFusion Research, San Diego, 21-23 July 2003.
H. Nakanishi et al., Fusion Eng. Des. 56-57,
1011 (2001).
H. Nakanishi et al., Fusion Eng. Des. 48, 135(2000).
H. Nakanishi, PhD Thesis, The Graduate University for Advanced Studies, Hayama, Japan, 2003.