Distributed Mass Data Acquisition System Based on PCs and Windows NT for
LHD Fusion Plasma Experiment
Nakanishi H., Kojima M., Ohsuna M., Komada S., Emoto M., Sugisaki H., Sudo S.,
NIFS LABCOM, Toki 509-5292, Japan
Abstract
The new data acquisition and management system has beendeveloped in order to manage the diagnostics of the LHD
fusion plasma experiments. It is required to have the capability to process 100 MB  1 GB raw data rapidly within
the following few ten seconds after every plasma discharge
experiment. It has employed the wholly distributed andloosely-tied parallel-tasking structure through the fast network, and the cluster of the distributed database servers
seems to be a virtual macro-machine as a whole. The
PC/Windows NT computer is installed for each diagnostics
data acquisition of about 30 kinds, and it controls the CAMAC digitizers through the optical SCSI extenders. The
synchronous timing is also optically distributed as a moduleof the realtime remote-controlling VME computer which isinstalled nearby each diagnostic equipment. The data retrieving terminals behaving as the database access clientsare functionally separated through the switching Ethernet.

1 INTRODUCTION
The “Large Helical Device (LHD)” plasma fusion project
have started its discharge experiments in March 1998 [1],and now it is one of the world largest fusion experimental
device with the most up-to-date diagnostic facilities.
The LHD data acquisition and management system forplasma diagnostics, which is named as LABCOM system,has been designed to process  1 GB plasma raw data by
the typical discharge experiment of about 10 second shortpulse duration in the continuous 3-minute cyclic operation.
After half a year of the LHD experiment, the total amountof the raw data from kinds of the plasma diagnostics goesup to about 200 MB per each shot, than 20  30 MB in thestart-up experimental period.
As all of the LHD helical, poloidal and vertical field coils
are made by the superconducting magnets, the quasi steadystate operation of about 1 hour long will be also plannedin the near future experimental periods. Then, the total
amount of the plasma data will be certainly enlarged due tothe longer duration.
The huge amount of experimental data forces to adopt
the fully distributed data acquisition and processing system,
rather than the ordinary concentrated ones. Necessary conditions toward the LHD data processing system are to complete to acquire and process whole diagnostic data within
100 seconds after every discharge end. In order to satisfy
them, the LHD data acquisition system was obliged to utilize the massively parallel processing (MPP) structure andreduce the data processing load of the individual element,such as data I/O ports or cpus.
The principles of the new LHD data acquisition and management system are as follows [2];1. completely parallel distribution for each diagnostics
device2. functional separation by means of network clientserver model
3. 100Mbps FDDI-based fast switching network
4. object-oriented method for programming manner anddata storages
5. commercial-based distributed databases .
The first objective to apply the distributed structure is for
improving the data transfer throughput by using multiple
paths and cpus. Modern technologies of high-performance
personal computers (PC) are quite suitable for distributedand parallel data processing system.
In addition to the parallel distribution, the functional separation will be also effective to provide the advanced graphical user interface (GUI). The computer loads of the datavisualization or complicated analysis are often unrelated tothe data processing sequence following after a discharge experiment. Popular 2-dimensional or 3-dimensional analysis
and visualization makes heavy cpu loads, and a private cpu
for each user will become more desirable. As a good solution of this problem, the network client/server model canbe preferably applied between data acquisition servers anduser-interface clients.
Fast network connections are indispensable for the mutual data transfer between those separately arranged computers. Especially among the data acquisition server computers, the fast mutual linkage and close concurrent cooperation would realize a virtual macro-machine, which theclient computers can send the data requests to and receive
results from. In other words, this virtual machine uses fast
network links as its internal system bus. They exactly organize a massive parallel-tasking multi-processor system withloosely-tied communications [3].

2

INSTRUMENTATION OF LABCOMSYSTEM

The distributed structure is applied in both parallel andfunctional meanings for the LHD data management servers.
It utilizes two kinds of server computers shown in Fig. 1;
One is the data acquiring and storing computers which govern the CAMAC digitizers and databases, and the other isthe diagnostics controlling computers which interactively

manage and continuously monitor the diagnostic devices inreal-time. The former uses so-called AT-compatible PC andWindows NT OS, and the latter does 68040-based VME
and real-time OS Tornado, which is previously known as
VxWorks. Both kinds of server computers will be independently stationed for every diagnostics of 20 or 30 kinds.
Data Acquisition LAN (100Mbps)

AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAA
AAA
AA
A
AAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAA
AAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAA
AA
AA
A
A
AAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAA
AA
A
AA
A
A
A
ClientDiagnostics Control LAN (100Mbps)
Windows NT
ServerMagnetic Disk
(50GB)
Optical Disk
(600GB)

RM

10BASE-FL(10Mbps)

Reflective
Memory(256Mbps)(Optical Fiber)

Ctrl. Bldg.

LHD Bldg.

4. data cross-referencing by virtual shared memory
5. object-oriented database on every server computer
6. harddisk array and media-changeable mass-storage
As for the digitizer modules for LHD, we have to inherit a
lot of CAMAC properties which had been working in theprior experimental devices of our institute.
In order to manipulate the SCSI-connected CAMAC
crate controller from Windows NT and to acquire the datafrom their modules, we have newly developed the CAMAC
handling software[4]. This software consists of the following three parts;
1. CAMAC driver: the SCSI class-driver to control SCSI
crate controller
2. CAMAC library: the application programming interface (API) to the CAMAC driver
3. CAMAC list sequencer: the application program using the CAMAC library which manages the CAMAC
command lists.

Optical Transceiver

SCSI (2~10MByte/sec)(Optical Fiber)
CAMAC
ADC

VME (VxWorks)

CPU
RM
TRG
DAC

C
C

Diagnostics Control Server

ADC

C
C

PLC

ADC

C
C

Synchronous
Timing System

AAA
AAA
AA
A AAAA
AAAA
AAAA
AAAA
RS232C

VacuumSystemData Acquisition
Digitizer

Figure 1: A basic element of the CAMAC data acquisition
and VME device control/monitor system: This set will beapplied for each kind of the LHD diagnostics device.

2.1

CAMAC Data Acquisition on Windows NT

The primary purpose of the data acquisition part of the
LABCOM system is the plasma measurements of the physical analysis which generally requires as fast data sampling
as to follow the plasma intrinsic frequencies of its dynamic
behaviors. They manages the CAMAC digitizers and transfer, transform and store the experimental data, where all of
the processing sequences are executed in the pre- and postexperimental periods as they were pre-programmed. These
procedure flow of the data management are often recognized as so-called the batch processing.
Sub-components of the LABCOM data acquisition can
be classified into the categories as listed below,
1. digitizing by CAMAC modules
2. SCSI data transfer with optical extender
3. PC/Windows NT data acquisition server computers

In Windows NT, the SCSI device driver has a hierarchy ofseveral drivers. The CAMAC driver is an upper class driverof the SCSI port driver, and it translates the CAMAC commands into SCSI ones.
The CAMAC driver can deal with multiple SCSI ports,
and it has versions for both of Intel Pentium and DEC Alpha
processors. The CAMAC library is the interface between auser program and the CAMAC driver. It is provided as the
dynamic link library (DLL) in Windows NT. The function
names in this library are almost compatible with KineticSystems CAMAC libraries.
The CAMAC list sequencer runs as a server process
which executes the CAMAC command lists sequentially bycommunicating with the corresponding driver, and any process which wants to access the CAMAC can send it arbitrary requests. The list sequencer is implemented as a background service process of Windows NT.
The block data transfer between the list sequencer andthe application program will be executed efficiently throughthe double buffering mechanism. It improves the effective
transfer rate with using two buffers for avoiding the buffer
accessing conflicts between the list sequencer and the client
application. The total throughput of the block data transfer
is about 700kB/s between the CAMAC module and the application program.

2.2

Digitally-Synchronized Timing System

The timing distribution system is indispensable both for the
diagnostics control and the data acquisition digitizers in order to coordinate them to run synchronously on the experiment standard timing, especially if they are dispersedly
installed. For the purpose of the timing coordination, the
LABCOM system has developed the new digitally synchronized timing modules which applies a synchronous baseclock distribution and modulated trigger messages as digital patterns on it. Optical fibers are used for the distribu-

tion linkage because they provide the good accuracy of thetransfer delay. Their basic specifications are as follows;
Data Analysis LAN> Running Mode Setup> Event Type Setup
R> Manual Interactive
Operations
Diagnostics Control LAN> Running Mode Setup> Delayed Triggers Setup> Divided Clocks Setup> Manual Interactive
Operations

AAAAAAAAAAAAAAAAAAAAAAAAAAAA

Diag. Ctrl. VME #1
Diag. Ctrl. Master VME
Diag. Ctrl. Sub-Master VME
Delayed Triggers
C
MasterC
C
PPDivided Clocks
Opt.
Fibre
Trigger U
U
P
Opt. FibreU
SignalTiming Demodulator(fromCentral Ope.
Diag. Ctrl. VME #2
System)Timing Distributor(s)C
Timing Master Modulator
Delayed Triggers
P
Timing Relay Modulator
U
Divided Clocks
Event Signals

All of the optical fibre linkages are tuned to havethe same length from the master modulator to theend demodulator(s).

Figure 2: Digitally-Synchronized Timing System; It runs as
a part of the diagnostics control VMEs.
1. Synchronous 10 MHz clock modulated by the trigger
message patterns are distributed through optical fibers
2. Simple tree structure based on the modulatordemodulator pairs
3. Both of preprogrammed and user-interactive realtime
signal outputs are enabled
4. Delayed trigger output covers hours-long operations
5. System flexibility of expansion as the VME modules .
They are usually used as preprogrammed, however, it requires the realtime management when handling the realtime message distribution. It enables the realtime userinteractive operation in the quasi steady-state experimentsin the near future[5].

2.3Hierarchical Data Storage

The LABCOM data acquisition system has applied the hierarchical storage management (HSM) technology. It had
installed the 2 levels of the storage hierarchy in 1998: One
is the local RAID of 50 GB for each server computer for
30 kinds of diagnostics, and the other is the 3 MO jukebox
whose total volume goes up to 3.6 TB.
The 50 GB RAID is occupied by the database volume
file of the object-oriented database management system(ODBMS), and the LABCOM data acquisition program directly store the plasma data into the database volume as thedata object. The typical data storing rate into the ODBMS
is a few 100 kB/s, and in order to improve the effective rate
and reduce the storage volume size, the data compression
mechanism of GNU-zlib library is used toward those binary
mass objects. Owing to the data compression and multithread programming, almost all of the server PCs can finish
their acquisition and store within 40 seconds after every experiment end. The compression rates, however, tightly depend on the signal forms; some integrated signal or pulsecount measurements achieve fine rates about a few %, whilesome fluctuation signals never go under 50 %.

3 OPERATIONAL RESULTS ANDDISCUSSION
The LABCOM system has been successfully developedwith both of the strong capability of the mass data acquisition and the expansion flexibility by applying the MPP
structure based on PC and Windows NT.
At the end of the 2nd experimental term, the total amountof the plasma raw data acquired by the LABCOM system
was about 120 MB/shot, excepting some kinds of the standalone acquisition systems. In the following 1 or 2 years, itwill easily reach the initial estimate of  1 GB/shot.
Late in the 2nd period, however, we had experienced
the network rush during the experiments because both ofthe volume and number of the data retrieving by users increased much. The conventional network router between
the server LAN and client one became the bottleneck, we
decided to adopt the modern multi-layer switching fabrics
instead. Taking the data growth in the near future into account, the continuous improvement for each part of the dataacquisition system would be important to maintain the convenient speed of the total throughput.
Table 1: Operational achievement of the LABCOM dataacquisition system in the 1st and 2nd LHD experimental
terms; The 1st term has #1  #1888, and 2nd #1889 
#7132 discharge experiments.
1st term /MB 2nd term /MBHalpha4,231
41,674
Bolometer
8,276
51,114
Fast Ion Gauge
1,825
7,273
Impurity Monitor
2,169
21,568Magnetics
12,84790,300Langmuir Probe
6,706
58,050mm-wave Interfero.
4,361
ECH Power
3,380
Reflectmetry
23,587
Soft-X fluc.
25,314
X-ray PHA
9,420
ICH Power
15,577
Bremsstrahlung
28,162
Fast Ion
6,860
TOTAL
43,795
378,899

4

REFERENCES[1] A. Iiyoshi and K. Yamazaki, Phys. Plasmas 2, 2349 (1995).
[2] H. Nakanishi et al., J. Plasma Fusion Res. 72, 1362 (1996).
[3] J. Bacon, Concurrent Systems (Addison-Wesley, Reading,1993), [Japanese translation: Toppan, Tokyo (1996)].
[4] M. Kojima et al., NIFS-TECH 6, Nov. (1997).
[5] H. Nakanishi et al., J. Plasma Fusion Res. SERIES 1, 409(1998).