J. Plasma Fusion Res. SERIES, Vol.7 (2006) 343-347

Automation of Plasma-Process Fultext Bibliography Databases:
An On-Line Data-Collection, Data-Mining and Data-Input System
SUZUKI Manabu, PICHL Lukas1 , MURAKAMI Izumi2 , KATO Takako2 and SASAKI Akira3
University of Aizu, Aizu-Wakamatsu 965-8580, Japan
1 International Christian University, Tokyo 181-8585, Japan
2 National Institute for Fusion Science, Toki 509-5292, Japan
3 Japan Atomic Energy Research Institute, Kyoto 600-8216, Japan（Received: 4 October 2004 / Accepted: 6 December 2005）

Abstract
Searching for relevant data, information retrieval, data extraction and data input are time- and resourceconsuming activities in most data centers. Here we develop a Linux system automating the process in caseof bibliography, abstract and fulltext databases. The present system is an open-source free-software lowcost solution that connects the target and provider databases in cyberspace through various web publishingformats. The abstract/fulltext relevance assessment is interfaced to external software modules.
Keywords:
atomic and molecular data, bibliography database, download robot, data retrieval
1．Introduction
Comprehensive and reliable databases of atomic andmolecular data are indispensable, for instance, in accurate computer simulations of energy dissipation infusion edge plasmas. In addition, the range of users
is broadening, including astronomy and recently also
experimental medicine techniques (hadron therapy orpositron emission tomography). This brings new requirements on the scope of the databases and classification methods for the covered data.
The standard methodology for data collection has
been the literature search, classification of bibliography
output, paper retrieval, relevance evaluation, recognition of the atomic / molecular process and other classification fields, identification of relevant graphs andnumerical data tables, digitalization of numerical data(whether by retyping these manually or facilitated bycharacter recognition software) and data input. Data
production activities by theorists and experimentalists
in adjunct working groups at data centers exclude literature search and character recognition issues. Due toconsiderable human labor involved, the above described
process for maintaining a specialized database, such as
atomic and molecular processes in edge plasmas, hasbeen very costly. In addition, outsourcing the databasesolution to software companies associates with large extra costs, if the data structure is to be extended or altered(e.g. adding synonymous labels for chemical species involved in a collision process).

We therefore study the automation of database development and maintenance along two lines. The first isa test of the open-source and free-software database alternative to commercially maintained database systems.
The latter deals with automation of data collection, classification, and input, and builds on the former one, since
this kind of testing and automation is usually impossible within a commercial database product. The covered
data originate from our joint working group activities at
NIFS [1]. Our restriction to bibliography data, although
including abstract and fulltext fields, allows us to automate the entire data-handling process from the searchphase up to the data input. The system is designed for
network-accessible electronic articles and / or abstracts
from various subscribed publishers; the available datain this kind of cyberspace are almost complete for articles published since 2000.
The paper is organized as follows. In Section 2,
we discuss our open-source bibliographic databases inanalogy with the outsourced NIFS system. We firstprovide a prototype bibliography database for electronmolecule collisions that combines chemical species andcollision process search fields, then develop a source
fulltext database for abstract / paper relevance assessment. Section 3 discusses the full automation ofdata-handling process in dynamically updated fulltext
databases. In Section 4, automated data update, downC 2006 by The Japan Society of Plasma

Science and Nuclear Fusion Research
Corresponding author’s e-mail: lukas@icu.ac.jp

343Automation of Plasma-Process Fultext Bibliography Databases: An On-Line Data-Collection, Data-Mining and Data-Input Systemload robot protection and copyright issues are dealt
with. Concluding remarks close the paper.

the html files and input by using PHP string matching
scripts for automatic handling as described in the next
section.

2．Bibliographic databases
The basic bibliographic entries are authors, journal, volume, issue, page number range, publication
year; classification fields include method (theory, experiment), type of process, species involved, date ofdatabase entry and others; the fulltext feature covers abstracts (html formatted, as the special characters including Greek letters, superscripts etc. are very common)
and article pdf files (external link or local disk copy).
Due to the size of atomic and molecular physics
community, simultaneous database access rate is far below numbers common in electronic business etc. It israre that more than 50 users connect at a time. Within
this range, free software solutions fully compete withcommercial software; also the requirements on hardware parameters are modest.
It was therefore decided to use a custom-built PC asa dedicated server. The hardware specification is as follows: Pentium 4 (FSB 533) 2 GHz CPU, 1 GB RAM
and 120 GB HDD, which matches the needs of NIFS
database users (large data set, limited access rates). The
operating system is Linux (RedHat 9) running Apache 2
HTTP server. Database management system is MySQL
3.23.54 with the connecting logic layer for web interface written in PHP 4.2.2. MySQL was chosen since
it suﬃces for our purpose and for its ease of manipulation; more query-rich object relational database management system, such as PostgreSQL, can also be used.
The above operating system, web and database server,
and programming language compiler are free-software
open-source products. All databases discussed here areavailable online at the crdb.nifs.ac.jp server.
Figure 1 shows a snapshot of our test bibliography database [2] for electron-molecule collisions as described above. In addition, a combined search basedon molecular species and collision process has been inbuilt [3] and the source code is available online [4].
In order to facilitate development of relevanceassessment software for journal articles [5], we havealso created an abstract and fulltext database [6] whichstores 379 papers [7] previously evaluated for AMDIS
database at NIFS (includes both articles accepted andrejected for entry to AMDIS). The list serves for calibration of learning vector quantization method (LVQ)
by our collaborators [5]; the source code is also available on-line [4]. The major publishers covered in thelist are APS, IOP and EDP Sciences. The abstractsin html format have been downloaded from databasesof the three publishers; the data [7] are extracted from

3．Automated data handling
There are two principal stages in building and maintaining bibliography databases: (1) retrieve, evaluate
and input all relevant data until the set point t 0 , and (2)retrieve, evaluate, extract and input data in regular intervals t0 + nT . Focusing on the latter, the flow chart ofthe procedure we implemented follows:
1. Install initial skeleton for bibliography and fulltext
database (Linux system in Sec. 2)
2. Identify on-line publisher database locations andthe respective form of DB queries (APS, IOP, EDP
at present)
3. Create a set of boolean queries recognized as relevant for the maintained Linux database (updatedvia on-line query-creation and query-management
form interface; cf. Fig. 2)
4. Create interface script (PHP) that transforms
queries in 3 to the form of on-line queries in 2.
5. Send the transformed queries to publisher sources
and retrieve html output by using the wget command line software in regular time intervals (croncommand on Linux server).
6. Follow the abstract and fulltext links in query output forms, download,• Call the interfaced software module for relevance assessment judgement: accept / discard
the article entry;
and concatenate to MySQL database tables (wgetand PHP).
7. Extract relevant information from the query output
html-formatted source and concatenate to MySQL
data tables (PHP).
8. Wait time T , then continue from item 5.
In Fig. 2, the on-line database of APS (a) and IOP(b) are shown. Given the form of scientific articles andjournals, the structure of available queries is practically
identical. In addition to search fields and logical operations, there are search options: threshold, sorting, or
number of records per page. The query range can belimited by fixing the publication date period or volume /issue interval; hits can be specified either as a word stem
or exactly as a quote. The query transform script (item344

Automation of Plasma-Process Fultext Bibliography Databases: An On-Line Data-Collection, Data-Mining and Data-Input System

Fig. 1 Bibliography database for electron-molecule scattering designed by Prof. Y. Itikawa (NIFS) [2]. A combined searchby molecule species and / or collision process is enabled. The system is implemented as a free-software opensource solution (the source code and installation instructions are available form from [4]).

4 in the flow chart above) is rather straightforward.
Nonetheless, the query output in Fig. 2 varies greatly,
especially from the viewpoint of HTML code in output files (formatting, commercials, itemizing styles).
Therefore the extraction of relevant database fields fromthe pool of files downloaded across publishers is nontrivial. A part of PHP script to send the download
queries is shown at the bottom of Fig. 2; also the string
matching scripts to extract author, title, page number
and other fields are written in PHP language.
Since creating new scripts for diﬀerent publishers orafter any web-interface update would be cumbersome,
we have also created a simpler script-development interface in order to avoid programming in PHP at theuser stage. The user selects itemizing marks on the input site (e.g. a ”checkbox” field), and thus classifies
the beginning and the end of an entry. Thus is how
the particular parameters enter the PHP code skeleton.
To save the expert work on relevance assessment of thedownloaded article, a software module [5] interface isadded in item 6 of the flowchart above. Hence theentire procedure becomes automated; human intervention is required only to (1) create initial set of relevant
queries (select the keywords and logical operators) and(2) to use the PHP script development form (that analyzes html query output structure) when some new online database is added (or the html interface updates).
Based on the analyzer scripts, we finally developed ajoint database search form that sends query requests andcollects query answers across publisher web sites [8].

abstract and fulltext cannot be retrieved without intermissions. Although the retrieval rate is much restricted by query structure and generally fits within theconditions of subscription, zero-approaching interval ofsubsequent download requests often triggers download
robot protection scripts on the side of publisher and may
result in immediate suspension of the subscription service. We have therefore added artificial waiting intervals. The download interval τ is set byfunction τ{while(1){t ← t0 + 2T (r( ) − 0.5)c ← fm r( )if(c < f (t)) return t}}(1)

where 0 ≤ r( ) < 1 is a uniform random number and f ( )is a probability density function with maximum value
fm . We set tm = 10.0 s, δ = 3.5 s, and use the normal distribution for the download time envelope profile, f ( ) = N(tm , δ). Although this suﬃces for our purpose, an optimal solution would be to measure the access rate from within the particular subscription domain
and replace f ( ) with such a histogram of network connections.
5．Concluding remarks
We have reported our new free-software open-source
bibliography databases for the National Institute for Fusion Science. These include electron-molecule collision
process bibliography and a 379-item article databasewhich calibrates the include - reject scope of AMDIS
database at NIFS. In addition, we have developed an au-4．Scheduling of article retrieval
To avoid any kind of interference with download
robot protection software on the side of publisher, the345

Automation of Plasma-Process Fultext Bibliography Databases: An On-Line Data-Collection, Data-Mining and Data-Input SystemFig. 2 Automated data-handling. Data-provider query forms: APS (a) and IOP (b), structure of html output ﬁles (middle),trigger and analyzer PHP script (bottom).

tomated Linux system that fully handles data search, retrieval, extraction and input into fulltext databases. Article assessment programs, such as LVQ algorithm, areconveniently interfaced in the form of a software module. The present work is considered useful in saving
database maintenance costs. Further research will focus on the automation of data-handling for cross-section
databases with graphical user interface.

Program.

[1][2]
L. P. acknowledges partial support by a Grant-in-Aidof JSPS and Academic Frontier Research Program ofMEXT. This work was performed with the support andunder the auspices of the NIFS Collaborative Research[3]346

References
I. Murakami, T. Kato, A. Igarashi, M. Imai,
Y. Itikawa, D. Kato, M. Kimura, T. Kusakabe,
K. Moribayashi, T. Morishita, K. Motohashi, L.
Pichl, AMDIS and CHART update I, NIFS-DATA
Series (ISSN 0915-6364), Issue 70, Oct. 2002.
On-line database of electron-molecule scattering
bibliography,
http://crdb.nifs.ac.jp/bib/iti/top.php
Electron-molecule collision bibliography data,
Prof. Y. Itikawa, private communication.

Automation of Plasma-Process Fultext Bibliography Databases: An On-Line Data-Collection, Data-Mining and Data-Input System

[4]

[5]

Source code and installation instructions (freesoftware, GNU licence)
http://crdb.nifs.ac.jp/crdb/source/
A. Sasaki, K. Joe, H. Kashiwagi, C. Watanabe,
M. Suzuki, L. Pichl, M. Ohishi, D. Kato, M. Katoand T. Kato, Design and implementation of anevolutional data collecting system for the atomic
and molecular databases, Joint ICAMDATA and[6]

[7]
[8]

347

ITC14 conference 2004, Toki, Japan.
On-line database of input abstracts for relevanceassessment by LVQ method,
http://crdb.nifs.ac.jp/iti/list/list top.php
Y. Itikawa, ATOMIC DATA AND NUCLEAR
DATA TABLES 64, 151–315 (1996).
Joint search across publishers,
http://crdb.nifs.ac.jp/j search/js top.php